Spaces:
Running
Research context
InferScale-Sim is a portfolio-scale implementation situated within the research line on replacing expensive deployment sweeps with modeling, simulation, or emulation.
Vidur (2024)
Vidur combines experimental profiling, predictive models, and end-to-end inference simulation. Its paper reports under 9% inference-latency estimation error across the evaluated range and a configuration-search example where LLaMA2-70B exploration took about one CPU-hour versus an estimated 42K GPU-hours (~$218K) for deployment-based exploration.
InferScale inspiration: workload-aware configuration search, explicit performance-profile provenance, and separation between latency prediction and event simulation.
Source: https://arxiv.org/abs/2405.05465
SGLang / RadixAttention (2024)
SGLang introduced RadixAttention for automatic KV-cache reuse across shared prompt prefixes. A radix tree retains reusable prompt/generation state and combines prefix lookup with cache-aware scheduling and eviction.
InferScale v0.3 does not implement a radix tree. It adds a deliberately smaller controlled shared-prefix abstraction so the effect of avoiding repeated prefill and duplicate KV allocation can be studied independently.
Source: https://arxiv.org/abs/2312.07104
TokenSim (2025)
TokenSim emphasizes extensible exploration of scheduling and memory-management choices and reports sub-1% error in its evaluated workloads.
InferScale inspiration: modular scheduler/memory experiments.
Source: https://arxiv.org/abs/2503.08415
Revati (2026)
Revati avoids reimplementing every evolving serving-system control decision: it executes real vLLM/SGLang control paths while virtualizing CUDA time. The paper reports less than 5% prediction error across its evaluated scenarios and 5-17x faster execution than real GPU serving runs.
InferScale lesson: pure simulation has a control-path fidelity ceiling. Our schedulers therefore document their semantics rather than claiming framework equivalence.
Source: https://arxiv.org/abs/2601.00397
LLMServingSim 2.0 (2026)
LLMServingSim 2.0 focuses on heterogeneous and disaggregated infrastructure and integrates batching, routing, offloading, memory, communication, and power into a unified runtime loop. The 2026 paper reports about 0.97% average error in its validation.
InferScale v0.3 inspiration: role-specific resources and explicit data movement rather than assuming a monolithic replica.
Source: https://arxiv.org/abs/2602.23036
Frontier (May 2026)
Frontier models co-location, Prefill-Decode Disaggregation, Attention-FFN Disaggregation, runtime optimizations such as speculative decoding, and stateful workloads. It reports average throughput error below 4% on its 16-H800 evaluation and explicitly supports SLA-dependent Pareto-frontier exploration.
InferScale v0.3 inspiration:
- separate prefill and decode worker pools;
- communication/KV-transfer cost;
- role-specific bottleneck telemetry;
- an interactive bounded Pareto design-space study.
Source: https://arxiv.org/abs/2605.21312
Scope boundary
InferScale-Sim v0.3 is not intended to compete with these research systems on fidelity, hardware scale, or runtime compatibility. Its goal is an inspectable, dependency-light Python implementation with a zero-backend interactive interface for understanding the systems mechanisms that motivate them.
HeteroPanacea / When Does Disaggregation Pay? (Aug 2026)
When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference pushes the simulator design space toward heterogeneous stage-specific hardware and automated cross-stack exploration. InferScale-Sim v0.3 does not model Attention--FFN disaggregation or custom NPUs, but its P/D topology already permits independent prefill and decode accelerator profiles and reports resource-normalized efficiency so heterogeneous layouts can be compared without treating extra accelerators as free.