docs/methodology.md

# Methodology and limitations

## What v0.3 simulates

InferScale models request arrival, queueing, admission, prefill, autoregressive decode, dynamic batch membership, KV-cache memory, and request completion.

For P/D disaggregation it additionally models independent prefill/decode worker pools and explicit prompt-KV transfer before decode admission.

Metrics are computed from per-request virtual timestamps.

## Scheduler semantics

- `static_fcfs`: admits one colocated batch and drains it before admitting new requests.
- `continuous_fcfs`: admits FCFS work whenever decode slots become available.
- `continuous_sjf`: prioritizes shorter estimated jobs at admission.
- `continuous_slo`: uses a least-slack-style ordering based on E2E deadline and an analytical remaining-service estimate.
- `chunked_slo`: combines least-slack ordering with chunked prompt prefill.

These are transparent approximations, not line-by-line reproductions of vLLM or SGLang.

Static batching is intentionally excluded from the P/D comparison; if requested for P/D, the simulator warns and uses continuous semantics.

## Workload semantics

Poisson arrivals use exponentially distributed inter-arrival times. Constant arrivals are evenly spaced. Bursty arrivals alternate lower and higher rate periods.

Prompt/output lengths are sampled from log-normal distributions parameterized by mean and coefficient of variation (CV).

## Shared-prefix model

v0.3 models one reusable exact prefix:

1. cache hits are assigned deterministically with a separate seeded RNG so enabling cache does **not** change the underlying arrival/prompt/output trace;
2. a hit reduces prefill work by the configured reusable-prefix length, bounded by the request prompt;
3. shared prefix KV consumes one persistent allocation per serving worker rather than being duplicated per request;
4. decode still uses the full logical context length for attention-cost estimation.

This isolates prefix reuse without implementing radix-tree lookup, eviction, or prefix-aware scheduling.

## P/D disaggregation

P/D uses a global discrete-event queue with four main event classes:

```text
arrival
prefill_done
transfer_done
decode_done
```

Prefill workers batch queued requests independently. Completed prompt state is placed on a serialized transfer link. Decode workers admit transferred requests only between decode iterations, preserving continuous-batching semantics.

The transfer model is:

```text
base_latency + bytes / bandwidth
```

where transferred bytes correspond to newly computed prompt KV. With a cache hit, shared-prefix state is assumed resident in both role pools and only the uncached suffix is transferred.

The model does not claim protocol-level fidelity to PCIe, NVLink, RDMA, NIXL, or NCCL.

## Latency model

The default reference model is roofline-inspired:

- dense transformer FLOPs scale with parameter count and processed tokens;
- attention adds context-length-dependent work;
- decode includes model-weight traffic and context-dependent KV reads;
- time is approximated from compute/memory costs plus a launch/scheduling proxy;
- conservative efficiency factors keep accelerator peak specs from being treated as achieved throughput.

This creates useful qualitative dynamics but is **not empirically calibrated**.

## KV cache

KV bytes per token are approximated as:

```text
2 x layers x KV heads x head dimension x 2 bytes
```

for K and V with FP16 KV state. Paged allocation rounds live sequence lengths to `kv_block_tokens`. Static batching reserves full prompt+requested-output capacity.

## Capacity search

Each offered rate is evaluated over deterministic seed offsets and is feasible only when **every repetition**:

1. reaches the configured SLO-attainment target; and
2. fully drains all generated requests.

The result retains mean, worst, best, and standard-deviation evidence. A bounded binary search estimates the highest feasible rate and a recommended rate after user-selected headroom.

## Design-space explorer

The browser-safe v0.3 sweep evaluates:

- three colocated continuous schedulers over batch sizes 8/16/32;
- one additional cached SLO-aware colocated point at the user's current batch size;
- optionally P/D worker splits 1P:1D, 1P:2D, and 2P:1D, each with cache off/on.

A candidate is Pareto-optimal when no other candidate has both **at least as much goodput** and **no worse p95 TTFT**, with at least one strict improvement.

This is a bounded interactive study, not exhaustive global optimization.

## Validation status

v0.3 validates deterministic software invariants and expected qualitative behavior through tests. It does **not** provide held-out GPU calibration error. Empirical profile import/calibration remains a future extension rather than being fabricated for this release.


## Resource-normalized design comparison

P/D candidates can use more simulated accelerator instances than colocated candidates. v0.3 therefore reports both raw goodput and `goodput / accelerator`. The Design Explorer computes a separate Pareto frontier for each throughput objective against p95 TTFT. This avoids treating additional hardware as a free improvement and keeps raw performance distinct from resource efficiency.