docs/methodology.md

Methodology and limitations

What v0.3 simulates

InferScale models request arrival, queueing, admission, prefill, autoregressive decode, dynamic batch membership, KV-cache memory, and request completion.

For P/D disaggregation it additionally models independent prefill/decode worker pools and explicit prompt-KV transfer before decode admission.

Metrics are computed from per-request virtual timestamps.

Scheduler semantics

• static_fcfs: admits one colocated batch and drains it before admitting new requests.
• continuous_fcfs: admits FCFS work whenever decode slots become available.
• continuous_sjf: prioritizes shorter estimated jobs at admission.
• continuous_slo: uses a least-slack-style ordering based on E2E deadline and an analytical remaining-service estimate.
• chunked_slo: combines least-slack ordering with chunked prompt prefill.

These are transparent approximations, not line-by-line reproductions of vLLM or SGLang.

Static batching is intentionally excluded from the P/D comparison; if requested for P/D, the simulator warns and uses continuous semantics.

Workload semantics

Poisson arrivals use exponentially distributed inter-arrival times. Constant arrivals are evenly spaced. Bursty arrivals alternate lower and higher rate periods.

Prompt/output lengths are sampled from log-normal distributions parameterized by mean and coefficient of variation (CV).

Shared-prefix model

v0.3 models one reusable exact prefix:

1. cache hits are assigned deterministically with a separate seeded RNG so enabling cache does not change the underlying arrival/prompt/output trace;
2. a hit reduces prefill work by the configured reusable-prefix length, bounded by the request prompt;
3. shared prefix KV consumes one persistent allocation per serving worker rather than being duplicated per request;
4. decode still uses the full logical context length for attention-cost estimation.

This isolates prefix reuse without implementing radix-tree lookup, eviction, or prefix-aware scheduling.

P/D disaggregation

P/D uses a global discrete-event queue with four main event classes:

arrival
prefill_done
transfer_done
decode_done

Prefill workers batch queued requests independently. Completed prompt state is placed on a serialized transfer link. Decode workers admit transferred requests only between decode iterations, preserving continuous-batching semantics.

The transfer model is:

base_latency + bytes / bandwidth

where transferred bytes correspond to newly computed prompt KV. With a cache hit, shared-prefix state is assumed resident in both role pools and only the uncached suffix is transferred.

The model does not claim protocol-level fidelity to PCIe, NVLink, RDMA, NIXL, or NCCL.

Latency model

The default reference model is roofline-inspired:

• dense transformer FLOPs scale with parameter count and processed tokens;
• attention adds context-length-dependent work;
• decode includes model-weight traffic and context-dependent KV reads;
• time is approximated from compute/memory costs plus a launch/scheduling proxy;
• conservative efficiency factors keep accelerator peak specs from being treated as achieved throughput.

This creates useful qualitative dynamics but is not empirically calibrated.

KV cache

KV bytes per token are approximated as:

2 x layers x KV heads x head dimension x 2 bytes

for K and V with FP16 KV state. Paged allocation rounds live sequence lengths to kv_block_tokens. Static batching reserves full prompt+requested-output capacity.

Capacity search

Each offered rate is evaluated over deterministic seed offsets and is feasible only when every repetition:

1. reaches the configured SLO-attainment target; and
2. fully drains all generated requests.

The result retains mean, worst, best, and standard-deviation evidence. A bounded binary search estimates the highest feasible rate and a recommended rate after user-selected headroom.

Design-space explorer

The browser-safe v0.3 sweep evaluates:

• three colocated continuous schedulers over batch sizes 8/16/32;
• one additional cached SLO-aware colocated point at the user's current batch size;
• optionally P/D worker splits 1P:1D, 1P:2D, and 2P:1D, each with cache off/on.

A candidate is Pareto-optimal when no other candidate has both at least as much goodput and no worse p95 TTFT, with at least one strict improvement.

This is a bounded interactive study, not exhaustive global optimization.

Validation status

v0.3 validates deterministic software invariants and expected qualitative behavior through tests. It does not provide held-out GPU calibration error. Empirical profile import/calibration remains a future extension rather than being fabricated for this release.

Resource-normalized design comparison

P/D candidates can use more simulated accelerator instances than colocated candidates. v0.3 therefore reports both raw goodput and goodput / accelerator. The Design Explorer computes a separate Pareto frontier for each throughput objective against p95 TTFT. This avoids treating additional hardware as a free improvement and keeps raw performance distinct from resource efficiency.