docs/architecture.md

# Architecture

InferScale-Sim separates **serving-system logic** from **latency estimation**.

## Main Python modules

1. `workloads.py` creates deterministic workload traces and shared-prefix hit assignments.
2. `simulator.py` implements the colocated serving loop.
3. `disaggregated.py` implements separate prefill/decode worker pools plus KV transfer.
4. `kv_cache.py` handles memory/admission and shared-prefix allocation.
5. `latency.py` predicts reference prefill/decode operation durations.
6. `metrics.py` derives request-level TTFT, TPOT, E2E, queueing, throughput, and goodput.
7. `diagnostics.py` converts simulated telemetry into explicit heuristic bottleneck labels.
8. `optimizer.py` implements capacity search, scheduler comparison, topology/cache comparison, and the bounded Pareto sweep.
9. `api.py` exposes JSON-like actions to both local Python and Pyodide.

## Colocated path

```text
arrival -> waiting -> prefill -> active decode batch -> complete
```

## P/D path

```text
arrival
  -> prefill queue
  -> prefill worker batch
  -> KV-transfer link
  -> decode-ready queue
  -> continuous decode worker
  -> complete
```

The P/D path is a genuine discrete-event loop: prefill workers, decode workers, and the transfer link can have overlapping virtual activity.

## Browser execution

Canonical source lives in `src/inferscale`. `scripts/sync_web_python.py` mirrors it into `py/inferscale`. A module Web Worker loads Pyodide, writes those files into Pyodide's virtual filesystem, imports `inferscale.api`, and exchanges JSON messages with the UI.

The simulator therefore runs away from the browser main thread and uses the same Python implementation as the local tests.

## Extension boundary

`AnalyticalLatencyModel` can later be replaced by an empirical profile interpolator exposing:

- `prefill_seconds(token_counts)`
- `decode_step_seconds(context_lengths)`
- `model_weight_gb`
- `kv_bytes_per_token()`

without rewriting scheduling, cache, P/D orchestration, SLO metrics, or search tooling.