docs/architecture.md

Architecture

InferScale-Sim separates serving-system logic from latency estimation.

Main Python modules

1. workloads.py creates deterministic workload traces and shared-prefix hit assignments.
2. simulator.py implements the colocated serving loop.
3. disaggregated.py implements separate prefill/decode worker pools plus KV transfer.
4. kv_cache.py handles memory/admission and shared-prefix allocation.
5. latency.py predicts reference prefill/decode operation durations.
6. metrics.py derives request-level TTFT, TPOT, E2E, queueing, throughput, and goodput.
7. diagnostics.py converts simulated telemetry into explicit heuristic bottleneck labels.
8. optimizer.py implements capacity search, scheduler comparison, topology/cache comparison, and the bounded Pareto sweep.
9. api.py exposes JSON-like actions to both local Python and Pyodide.

Colocated path

arrival -> waiting -> prefill -> active decode batch -> complete

P/D path

arrival
  -> prefill queue
  -> prefill worker batch
  -> KV-transfer link
  -> decode-ready queue
  -> continuous decode worker
  -> complete

The P/D path is a genuine discrete-event loop: prefill workers, decode workers, and the transfer link can have overlapping virtual activity.

Browser execution

Canonical source lives in src/inferscale. scripts/sync_web_python.py mirrors it into py/inferscale. A module Web Worker loads Pyodide, writes those files into Pyodide's virtual filesystem, imports inferscale.api, and exchanges JSON messages with the UI.

The simulator therefore runs away from the browser main thread and uses the same Python implementation as the local tests.

Extension boundary

AnalyticalLatencyModel can later be replaced by an empirical profile interpolator exposing:

• prefill_seconds(token_counts)
• decode_step_seconds(context_lengths)
• model_weight_gb
• kv_bytes_per_token()

without rewriting scheduling, cache, P/D orchestration, SLO metrics, or search tooling.