Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> | |
| <title>InferScale-Sim</title> | |
| <meta name="description" content="Interactive LLM serving simulator and SLO-aware capacity planner." /> | |
| <link rel="preconnect" href="https://cdn.jsdelivr.net" crossorigin> | |
| <link rel="stylesheet" href="styles.css" /> | |
| <script src="https://cdn.jsdelivr.net/npm/chart.js@4.5.1/dist/chart.umd.min.js"></script> | |
| </head> | |
| <body> | |
| <header class="topbar" id="topbar"> | |
| <div class="brand-wrap"> | |
| <div> | |
| <div class="brand">InferScale-Sim</div> | |
| <div class="subtitle">interactive serving-systems research prototype</div> | |
| </div> | |
| </div> | |
| <div class="runtime-pill" id="runtimePill"><span class="dot"></span><span id="runtimeText">Loading Python runtime...</span></div> | |
| </header> | |
| <main class="shell"> | |
| <section class="intro"> | |
| <div class="intro-copy"> | |
| <div class="eyebrow">InferScale-Sim / v0.3.0</div> | |
| <h1>Explore serving policies, cache reuse, and P/D disaggregation without provisioning a GPU.</h1> | |
| <p>Generate workloads, compare schedulers, model KV pressure and prefix reuse, separate prefill from decode, and search the SLO-constrained design space. The simulator is Python running locally in a Pyodide Web Worker.</p> | |
| </div> | |
| <dl class="project-facts"> | |
| <div><dt>Backend</dt><dd>None</dd></div> | |
| <div><dt>Runtime</dt><dd>Python / WASM</dd></div> | |
| <div><dt>Network data</dt><dd>None after load</dd></div> | |
| <div><dt>Profiles</dt><dd>Analytical reference</dd></div> | |
| </dl> | |
| </section> | |
| <div class="reference-note"><strong>Reference-profile mode.</strong> Absolute L4, A10G, and A100 latency values are analytical predictions, not measured GPU benchmarks. Queueing, scheduling, cache, transfer, and SLO behavior is simulated live.</div> | |
| <nav class="tabs" aria-label="InferScale sections"> | |
| <button class="tab active" data-tab="lab">Serving Lab</button> | |
| <button class="tab" data-tab="arena">Scheduler Arena</button> | |
| <button class="tab" data-tab="planner">Capacity Planner</button> | |
| <button class="tab" data-tab="modern">Modern Serving</button> | |
| <button class="tab" data-tab="design">Design Explorer</button> | |
| <button class="tab" data-tab="method">Methodology</button> | |
| </nav> | |
| <section id="lab" class="tab-panel active"> | |
| <div class="workspace"> | |
| <aside class="panel controls-panel"> | |
| <div class="panel-title-row"><h2>Experiment</h2><span class="tag">Deterministic seed</span></div> | |
| <div class="field-grid two"> | |
| <label>Model<select id="model"><option>Qwen2.5-3B</option><option>Llama-3.1-8B</option><option>Mistral-7B-v0.3</option></select></label> | |
| <label>Weight precision<select id="quantization"><option value="fp16">FP16</option><option value="int8" selected>INT8 scenario</option><option value="int4">INT4 scenario</option></select></label> | |
| </div> | |
| <div class="field-grid two"> | |
| <label>Topology<select id="topology"><option value="colocated">Colocated</option><option value="disaggregated_pd">P/D disaggregated</option></select></label> | |
| <label id="colocatedAcceleratorLabel">Accelerator<select id="accelerator"><option value="L4">NVIDIA L4</option><option value="A10G">NVIDIA A10G</option><option value="A100-40GB">NVIDIA A100 40GB</option></select></label> | |
| </div> | |
| <div class="field-grid two"> | |
| <label>Scheduler<select id="scheduler"><option value="continuous_fcfs">Continuous - FCFS</option><option value="continuous_sjf">Continuous - SJF</option><option value="continuous_slo">Continuous - SLO-aware</option><option value="chunked_slo">Chunked prefill - SLO-aware</option><option value="static_fcfs">Static batching - FCFS</option></select></label> | |
| <label>Prefix reuse<select id="prefixCache"><option value="off">Off</option><option value="on">On</option></select></label> | |
| </div> | |
| <div id="prefixControls" class="subcontrols hidden"> | |
| <div class="section-kicker">Shared-prefix scenario</div> | |
| <div class="field-grid two"> | |
| <label>Shared prefix<input id="sharedPrefix" type="number" min="0" step="32" value="256" /><span class="unit">tokens</span></label> | |
| <label>Reuse fraction<input id="prefixReuse" type="number" min="0" max="1" step="0.05" value="0.60" /><span class="unit">fraction</span></label> | |
| </div> | |
| </div> | |
| <div id="pdControls" class="subcontrols hidden"> | |
| <div class="section-kicker">P/D topology</div> | |
| <div class="field-grid two"> | |
| <label>Prefill accelerator<select id="prefillAccelerator"><option value="L4">NVIDIA L4</option><option value="A10G">NVIDIA A10G</option><option value="A100-40GB">NVIDIA A100 40GB</option></select></label> | |
| <label>Decode accelerator<select id="decodeAccelerator"><option value="L4">NVIDIA L4</option><option value="A10G">NVIDIA A10G</option><option value="A100-40GB">NVIDIA A100 40GB</option></select></label> | |
| <label>Prefill workers<input id="prefillWorkers" type="number" min="1" max="8" step="1" value="1" /></label> | |
| <label>Decode workers<input id="decodeWorkers" type="number" min="1" max="8" step="1" value="1" /></label> | |
| <label>Interconnect<input id="interconnect" type="number" min="1" step="1" value="50" /><span class="unit">GB/s</span></label> | |
| <label>Transfer base<input id="transferBase" type="number" min="0" step="0.05" value="0.20" /><span class="unit">ms</span></label> | |
| </div> | |
| </div> | |
| <hr /> | |
| <div class="section-kicker">Workload</div> | |
| <div class="field-grid two"> | |
| <label>Arrival process<select id="arrival"><option value="poisson">Poisson</option><option value="constant">Constant</option><option value="bursty">Bursty</option></select></label> | |
| <label>Request rate<input id="rate" type="number" min="0.1" step="0.1" value="4" /><span class="unit">req/s</span></label> | |
| <label>Duration<input id="duration" type="number" min="2" step="1" value="30" /><span class="unit">sim s</span></label> | |
| <label>Seed<input id="seed" type="number" step="1" value="7" /></label> | |
| </div> | |
| <div id="burstControls" class="field-grid two hidden"> | |
| <label>Burst multiplier<input id="burstMultiplier" type="number" min="1" step="0.25" value="3" /><span class="unit">x</span></label> | |
| <label>Burst period<input id="burstPeriod" type="number" min="0.5" step="0.5" value="10" /><span class="unit">s</span></label> | |
| </div> | |
| <div class="field-grid two"> | |
| <label>Prompt mean<input id="promptMean" type="number" min="16" value="512" /><span class="unit">tokens</span></label> | |
| <label>Prompt CV<input id="promptCv" type="number" min="0" max="2" step="0.05" value="0.50" /></label> | |
| <label>Output mean<input id="outputMean" type="number" min="1" value="64" /><span class="unit">tokens</span></label> | |
| <label>Output CV<input id="outputCv" type="number" min="0" max="2" step="0.05" value="0.60" /></label> | |
| </div> | |
| <hr /> | |
| <div class="section-kicker">Serving controls</div> | |
| <div class="field-grid two"> | |
| <label>Max batch size<input id="maxBatch" type="number" min="1" max="128" value="16" /></label> | |
| <label>Max batch tokens<input id="maxBatchTokens" type="number" min="128" step="128" value="8192" /></label> | |
| <label>Prefill chunk<input id="chunkSize" type="number" min="64" step="64" value="512" /><span class="unit">tokens</span></label> | |
| <label>KV block<input id="kvBlock" type="number" min="1" value="16" /><span class="unit">tokens</span></label> | |
| </div> | |
| <hr /> | |
| <div class="section-kicker">SLO</div> | |
| <div class="field-grid two"> | |
| <label>TTFT limit<input id="sloTtft" type="number" min="1" value="500" /><span class="unit">ms</span></label> | |
| <label>E2E limit<input id="sloE2e" type="number" min="100" value="5000" /><span class="unit">ms</span></label> | |
| </div> | |
| <button id="runBtn" class="primary" disabled>Run simulation</button> | |
| <div class="button-row"><button id="copyResultBtn" class="secondary" disabled>Copy JSON</button><button id="exportBtn" class="secondary" disabled>Download JSON</button></div> | |
| </aside> | |
| <div class="results-column"> | |
| <section class="panel result-panel"> | |
| <div class="panel-title-row"><h2>Run summary</h2><span id="runState" class="tag neutral">Waiting</span></div> | |
| <div id="emptyState" class="empty-state"><h3>Configure a workload and run it</h3><p>The Python simulator executes in a background Web Worker and returns request-level virtual timestamps.</p></div> | |
| <div id="resultContent" class="hidden"> | |
| <div class="metric-grid"> | |
| <div class="metric"><span>p95 TTFT</span><strong id="mTtft">N/A</strong></div> | |
| <div class="metric"><span>p95 E2E</span><strong id="mE2e">N/A</strong></div> | |
| <div class="metric"><span>Goodput</span><strong id="mGoodput">N/A</strong></div> | |
| <div class="metric"><span>SLO attainment</span><strong id="mSlo">N/A</strong></div> | |
| <div class="metric"><span>Throughput</span><strong id="mReq">N/A</strong></div> | |
| <div class="metric"><span>Peak KV</span><strong id="mKv">N/A</strong></div> | |
| </div> | |
| <section class="diagnostic-card" id="diagnosticCard"> | |
| <div class="diagnostic-head"><span>Simulator diagnosis</span><strong id="mBottleneck">N/A</strong></div> | |
| <p id="mDiagnosis">Run a simulation to generate a bottleneck explanation.</p> | |
| <p class="diagnostic-action" id="mRecommendation"></p> | |
| <div class="evidence-row" id="mEvidence"></div> | |
| </section> | |
| <div class="chart-grid"> | |
| <div class="chart-card" data-chart-card data-chart-name="latency-percentiles"> | |
| <div class="chart-head"><div class="chart-title">Latency percentiles</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div> | |
| <div class="chart-body"><canvas id="latencyChart"></canvas></div> | |
| </div> | |
| <div class="chart-card" data-chart-card data-chart-name="queue-decode-kv-timeline"> | |
| <div class="chart-head"><div class="chart-title">Queue, decode, and KV timeline</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div> | |
| <div class="chart-body"><canvas id="timelineChart"></canvas></div> | |
| </div> | |
| </div> | |
| <div class="chart-card full" data-chart-card data-chart-name="ttft-vs-prompt-length"> | |
| <div class="chart-head"><div class="chart-title">Request TTFT vs prompt length</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div> | |
| <div class="chart-body"><canvas id="scatterChart"></canvas></div> | |
| </div> | |
| <div id="warnings" class="warnings hidden"></div> | |
| </div> | |
| </section> | |
| </div> | |
| </div> | |
| </section> | |
| <section id="arena" class="tab-panel"> | |
| <div class="panel wide-panel"> | |
| <div class="panel-title-row arena-title-row"> | |
| <div><div class="section-kicker">Same workload - same seed</div><h2>Scheduler Arena</h2><p class="muted">Run every colocated v0.3 scheduler against the current Serving Lab workload and rank by SLO attainment, then goodput.</p></div> | |
| <button id="arenaBtn" class="primary compact" disabled>Compare schedulers</button> | |
| </div> | |
| <div id="arenaEmpty" class="empty-state small"><h3>No comparison yet</h3><p>Your Serving Lab workload and model controls are reused automatically.</p></div> | |
| <div id="arenaContent" class="hidden"> | |
| <div class="chart-card full" data-chart-card data-chart-name="scheduler-throughput-comparison"> | |
| <div class="chart-head"><div class="chart-title">Useful vs raw request throughput</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div> | |
| <div class="chart-body large"><canvas id="arenaChart"></canvas></div> | |
| </div> | |
| <div class="table-toolbar"><span>Scheduler comparison</span><div><button id="arenaCopyBtn" class="mini-button" disabled>Copy table</button><button id="arenaCsvBtn" class="mini-button" disabled>Download CSV</button></div></div> | |
| <div class="table-wrap"><table><thead><tr><th>Scheduler</th><th>Goodput</th><th>SLO attainment</th><th>p95 TTFT</th><th>p95 E2E</th><th>KV peak</th><th>Unfinished</th><th>Diagnosis</th></tr></thead><tbody id="arenaRows"></tbody></table></div> | |
| </div> | |
| </div> | |
| </section> | |
| <section id="planner" class="tab-panel"> | |
| <div class="workspace planner-grid"> | |
| <aside class="panel controls-panel"> | |
| <div class="panel-title-row"><h2>Capacity search</h2><span class="tag">Robust binary search</span></div> | |
| <p class="muted">Find the highest offered load where every repetition meets the SLO target and fully drains.</p> | |
| <label>Required SLO attainment<input id="targetSlo" type="number" min="0.5" max="1" step="0.001" value="0.99" /></label> | |
| <div class="field-grid two"> | |
| <label>Minimum rate<input id="minRate" type="number" min="0.05" step="0.1" value="0.25" /><span class="unit">req/s</span></label> | |
| <label>Maximum rate<input id="maxRate" type="number" min="0.1" step="1" value="20" /><span class="unit">req/s</span></label> | |
| <label>Search iterations<input id="searchIter" type="number" min="2" max="12" value="7" /></label> | |
| <label>Repetitions / rate<input id="repetitions" type="number" min="1" max="5" value="2" /></label> | |
| </div> | |
| <label>Safety headroom<input id="headroom" type="number" min="0" max="0.8" step="0.05" value="0.20" /><span class="unit">fraction</span></label> | |
| <button id="capacityBtn" class="primary" disabled>Find sustainable capacity</button> | |
| </aside> | |
| <section class="panel result-panel"> | |
| <div class="panel-title-row"><h2>Planner result</h2><span id="plannerState" class="tag neutral">Waiting</span></div> | |
| <div id="plannerEmpty" class="empty-state"><h3>No search yet</h3><p>The planner repeatedly runs the simulator at different offered loads.</p></div> | |
| <div id="plannerContent" class="hidden"> | |
| <div class="metric-grid four"> | |
| <div class="metric emphasis"><span>Estimated capacity</span><strong id="pCapacity">N/A</strong></div> | |
| <div class="metric"><span>Recommended load</span><strong id="pRecommended">N/A</strong></div> | |
| <div class="metric"><span>Safety headroom</span><strong id="pHeadroom">N/A</strong></div> | |
| <div class="metric"><span>Status</span><strong id="pStatus">N/A</strong></div> | |
| </div> | |
| <div class="planner-note" id="plannerCriterion">A rate passes only if every repetition meets the target and drains all generated requests.</div> | |
| <div class="chart-card full" data-chart-card data-chart-name="capacity-slo-curve"> | |
| <div class="chart-head"><div class="chart-title">SLO attainment across searched rates</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div> | |
| <div class="chart-body large"><canvas id="capacityChart"></canvas></div> | |
| </div> | |
| <div class="table-toolbar"><span>Search trace</span><div><button id="capacityCopyBtn" class="mini-button" disabled>Copy table</button><button id="capacityCsvBtn" class="mini-button" disabled>Download CSV</button></div></div> | |
| <div class="table-wrap"><table><thead><tr><th>Rate</th><th>Pass</th><th>Mean SLO</th><th>Worst repetition</th><th>Target</th><th>SLO range</th><th>Goodput</th><th>p95 TTFT</th><th>p95 E2E</th></tr></thead><tbody id="capacityRows"></tbody></table></div> | |
| </div> | |
| </section> | |
| </div> | |
| </section> | |
| <section id="modern" class="tab-panel"> | |
| <div class="panel wide-panel"> | |
| <div class="panel-title-row arena-title-row"> | |
| <div><div class="section-kicker">P/D disaggregation + prefix reuse</div><h2>Modern Serving Lab</h2><p class="muted">Compare colocated and prefill/decode-disaggregated serving, each with and without the current shared-prefix reuse scenario. P/D worker counts and interconnect settings come from Serving Lab.</p></div> | |
| <button id="topologyBtn" class="primary compact" disabled>Compare 4 scenarios</button> | |
| </div> | |
| <div id="topologyEmpty" class="empty-state small"><h3>No topology study yet</h3><p>Set shared-prefix and P/D parameters in Serving Lab, then run this controlled comparison.</p></div> | |
| <div id="topologyContent" class="hidden"> | |
| <div class="chart-card full" data-chart-card data-chart-name="topology-goodput-ttft"> | |
| <div class="chart-head"><div class="chart-title">Goodput vs p95 TTFT</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div> | |
| <div class="chart-body large"><canvas id="topologyChart"></canvas></div> | |
| </div> | |
| <div class="table-toolbar"><span>Topology and cache comparison</span><div><button id="topologyCopyBtn" class="mini-button" disabled>Copy table</button><button id="topologyCsvBtn" class="mini-button" disabled>Download CSV</button></div></div> | |
| <div class="table-wrap"><table><thead><tr><th>Scenario</th><th>GPU instances</th><th>Goodput</th><th>Goodput / GPU</th><th>SLO attainment</th><th>p95 TTFT</th><th>p95 E2E</th><th>p95 KV transfer</th><th>Cache hit</th><th>Prefill saved</th><th>Diagnosis</th></tr></thead><tbody id="topologyRows"></tbody></table></div> | |
| </div> | |
| </div> | |
| </section> | |
| <section id="design" class="tab-panel"> | |
| <div class="panel wide-panel"> | |
| <div class="panel-title-row arena-title-row"> | |
| <div><div class="section-kicker">Bounded what-if sweep</div><h2>Design Explorer</h2><p class="muted">Sweep 10 colocated candidates plus six P/D/cache variants, then identify the non-dominated goodput/TTFT frontier. This is an interactive design study, not an exhaustive optimizer.</p></div> | |
| <div class="action-stack"><label class="checkline"><input id="includePd" type="checkbox" checked /> Include P/D candidates</label><button id="designBtn" class="primary compact" disabled>Explore design space</button></div> | |
| </div> | |
| <div id="designEmpty" class="empty-state small"><h3>No design sweep yet</h3><p>The current workload, model, SLO, and P/D parameters are reused from Serving Lab.</p></div> | |
| <div id="designContent" class="hidden"> | |
| <div class="metric-grid four"><div class="metric emphasis"><span>Candidates</span><strong id="dCandidates">N/A</strong></div><div class="metric"><span>Performance Pareto</span><strong id="dPareto">N/A</strong></div><div class="metric"><span>Efficiency Pareto</span><strong id="dEfficiencyPareto">N/A</strong></div><div class="metric"><span>Profile</span><strong class="metric-small">Analytical</strong></div></div> | |
| <div class="chart-grid"> | |
| <div class="chart-card" data-chart-card data-chart-name="design-performance-pareto-frontier"> | |
| <div class="chart-head"><div class="chart-title">Raw goodput vs p95 TTFT</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div> | |
| <div class="chart-body large"><canvas id="designChart"></canvas></div> | |
| </div> | |
| <div class="chart-card" data-chart-card data-chart-name="design-efficiency-pareto-frontier"> | |
| <div class="chart-head"><div class="chart-title">Goodput per accelerator vs p95 TTFT</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div> | |
| <div class="chart-body large"><canvas id="designEfficiencyChart"></canvas></div> | |
| </div> | |
| </div> | |
| <div class="table-toolbar"><span>Candidate configurations</span><div><button id="designCopyBtn" class="mini-button" disabled>Copy table</button><button id="designCsvBtn" class="mini-button" disabled>Download CSV</button></div></div> | |
| <div class="table-wrap"><table><thead><tr><th>Candidate</th><th>Perf Pareto</th><th>Efficiency Pareto</th><th>SLO pass</th><th>GPU instances</th><th>Goodput</th><th>Goodput / GPU</th><th>p95 TTFT</th><th>p95 E2E</th><th>KV peak</th><th>Diagnosis</th></tr></thead><tbody id="designRows"></tbody></table></div> | |
| </div> | |
| </div> | |
| </section> | |
| <section id="method" class="tab-panel"> | |
| <div class="method-grid"> | |
| <article class="panel prose"><div class="section-kicker">Simulation core</div><h2>What is actually simulated?</h2><p>Requests are generated from deterministic workload distributions and advanced through virtual time. Colocated runs model admission, prefill, paged KV allocation, dynamic decode batches, and completion. P/D runs use separate prefill and decode worker pools plus an explicit serialized KV-transfer link.</p><div class="formula">request -> queue -> prefill -> [KV transfer] -> decode -> completion</div></article> | |
| <article class="panel prose"><div class="section-kicker">Prefix reuse</div><h2>Cache without pretending to implement a radix tree</h2><p>v0.3 models a single shared prompt prefix with configurable length and reuse fraction. Cache hits avoid redundant prefill work and share one persistent KV allocation. It is deliberately a controlled what-if abstraction, not a claim to reproduce SGLang's full RadixAttention policy.</p></article> | |
| <article class="panel prose"><div class="section-kicker">P/D disaggregation</div><h2>Role-specific resources and transfer cost</h2><p>Prefill and decode have separate accelerator profiles and worker counts. Prompt KV state crosses a modeled interconnect before decode admission. The simulator reports role utilization, transfer latency, and transfer pressure so the benefit of isolation can be weighed against data-movement overhead.</p></article> | |
| <article class="panel prose"><div class="section-kicker">Design search</div><h2>Pareto, not one magic configuration</h2><p>The Design Explorer reports both a raw-performance frontier and a resource-normalized frontier using goodput per accelerator. That prevents a multi-GPU P/D layout from looking unconditionally better merely because it uses more simulated hardware.</p></article> | |
| <article class="panel prose wide-method"><div class="section-kicker">Research lineage</div><h2>Why these features are in v0.3</h2><p>Vidur established the value of simulation for avoiding expensive deployment sweeps. Recent systems have pushed toward heterogeneous and disaggregated serving, communication-aware modeling, runtime optimizations, and SLA-dependent design-space exploration. InferScale-Sim remains intentionally smaller and inspectable.</p><div class="paper-grid"><div><strong>Vidur / 2024</strong><span>Predictive profiling, workload-aware serving simulation, configuration search.</span></div><div><strong>TokenSim / 2025</strong><span>Extensible scheduling and memory-management simulation.</span></div><div><strong>Revati / 2026</strong><span>GPU-free time-warp emulation of serving control logic.</span></div><div><strong>LLMServingSim 2.0 / 2026</strong><span>Heterogeneous and disaggregated infrastructure, memory and communication.</span></div><div><strong>Frontier / May 2026</strong><span>P/D disaggregation, runtime optimizations, stateful workloads, Pareto exploration.</span></div><div><strong>HeteroPanacea / Aug 2026</strong><span>Heterogeneous stage specialization motivates resource-aware P/D comparison.</span></div><div><strong>SGLang / RadixAttention</strong><span>Automatic shared-prefix KV reuse motivates the controlled cache scenario.</span></div></div></article> | |
| </div> | |
| </section> | |
| </main> | |
| <div id="toast" class="toast" role="status"></div> | |
| <script src="app.js"></script> | |
| </body> | |
| </html> | |