v1.1 live-cell rebuild: README.md

README.md

@@ -8,12 +8,23 @@ tags:
   - mlx
 ---
 
-# coding-router — trained default artifact (v1)
-
-The default routing artifact for [experiential-labs/coding-router](https://github.com/experiential-labs/coding-router):
-given a coding task (or a whole agent conversation), pick the cheapest OpenAI/Anthropic
-model+effort arm predicted to solve it. Routing runs fully locally — the encoder below
-runs in-process; API keys are only ever used to dispatch the chosen model.
+# coding-router — trained default artifact (v1.1, live-cell rebuild)
+
+**Changelog v1.1 (2026-08-01):** the bank's published DeepSWE cells were replaced with
+the EXP-015 LIVE matrix (1,130 fresh trials, dense 10 arms × 113 tasks) after the June
+matrix was shown to be quality-stale live (e.g. `luna_max` 0.946 published → 0.687
+live, effort ladders inverted) — a router voting over stale cells never escalates.
+Roster 41 → 10 live arms; per-arm `med_cost` and fallback (`opus5_high`, live f2p
+0.951) from live cells; `lam` re-selected (0.005) on inner train-side splits under the
+same protocol; tuned encoder, bank embeddings, `T`, and `sim_floor` unchanged. Paired
+server fix: client system prompts are now excluded from the routing embedding
+(constant preamble bytes homogenized every request toward one arm).
+
+The default routing artifact for [coding-router](https://github.com/experientiallabs/coding-router):
+optimizes coding requests between big and small models — more usage at the same cost.
+Given a coding task (or a whole agent conversation), it picks the model+effort arm with
+the best measured cost/quality trade-off. Routing runs fully locally: the encoder below
+runs in-process.
 
 This release replaces the kNN-over-base-embeddings artifact with a **trained** router
 (EXP-012 winning recipe `reward_lcb_b0.2`), shipped in two encoder formats plus one
@@ -24,7 +35,7 @@ head/calibration payload.
 | path | what |
 |---|---|
 | `router.json` | arm list + per-arm request kwargs, decision-rule params (`T`, `lam`, `sim_floor`), provenance (`kind: "trained"`) |
-| `router.npz` | `emb` — 110-task DeepSWE evidence bank in the tuned space; `graded` — 41×110 outcome matrix (the calibration side of the vote); `med_cost` — per-arm median $/task |
+| `router.npz` | `emb` — 110-task DeepSWE evidence bank in the tuned space; `graded` — 10×110 LIVE outcome matrix (EXP-015; the calibration side of the vote); `med_cost` — per-arm median live $/task |
 | `encoder-fp16/` | merged (LoRA→base) Qwen3-Embedding-0.6B, fp16 safetensors — loads via sentence-transformers / transformers on CUDA or CPU |
 | `encoder-mlx-4bit/` | the same merged encoder, 4-bit MLX (group size 64) for Apple Silicon — loads via `mlx_embeddings` |
 
@@ -47,35 +58,51 @@ Final-mint policy: one training run with the sweep's exact code and seed; (check
 own selection discipline); the shipped bank is the full 110-task evidence. The artifact
 was never selected or early-stopped on the holdout numbers reported below.
 
-## Honest numbers (EXP-012, 6-seed repo-split holdout, DeepSWE v1.1)
+## Honest numbers
+
+**This shipped artifact, LIVE cells (EXP-015), 6-seed repo-split holdout:** routed
+graded **0.9276 at $98.47/split** vs always-`opus5_high` (the strongest live arm)
+0.9509/$139.69 — graded delta **−0.023, repo-clustered 95% CI [−0.066, +0.025]**
+(contains zero: statistical parity at n=110), cost ratio **1.42× [1.25, 1.65]**.
+`lam` was selected on inner train-side splits only; holdouts were scored afterwards.
 
-Two baselines, deliberately:
+The encoder recipe's original certification (EXP-012, June-matrix cells, 41 arms) for
+context — these numbers describe the RECIPE on the old cells, not this artifact:
 
 | policy | graded | cost/split | read |
 |---|---|---|---|
 | always-best-train arm (deployable baseline) | 0.9336 | $126.28 | what you could actually deploy without hindsight |
-| **this router (per-seed selection, sweep)** | **0.9484** | **$60.17** | **+0.015 graded, 2.1× cheaper** |
-| hindsight-best static arm (opus-5@high) | 0.9635 | $135.73 | the router is at parity with it, NOT better |
-
-Minted-artifact re-evaluation (this exact shipped artifact's encoder + single global
-(step=50, lam=0.01) instead of per-seed selection, same 6 splits, sweep eval code):
-graded **0.9353 at $50.21/split** — slightly below the per-seed-selected sweep mean
-(that gap is the per-seed selection's optimism, deliberately not shipped), cheaper,
-and still above the always-best-train baseline on both axes. The shipped checkpoint
-and `lam` were selected on inner train-side splits only; the holdout numbers above
-were computed after the artifact was frozen.
+| recipe, per-seed selection (sweep) | 0.9484 | $60.17 | +0.015 graded, 2.1× cheaper |
+| single-(step,λ) mint, same splits | 0.9353 | $50.21 | per-seed selection's optimism removed |
+| hindsight-best static arm (opus-5@high) | 0.9635 | $135.73 | parity, NOT better |
 
 Quality-BEATING routing on this data is dead (held-out oracle analysis, EXP-018): the
-honest framing is parity-quality at a large cost saving vs any deployable policy.
-**Cost figures are matrix-based (June 2026 collection) and pending live re-benchmark
-(EXP-014 measured live cost/quality drift on several arms).**
+honest framing is parity-quality at a cost saving vs the strongest deployable policy.
+Live cells make the cost claim current; models keep drifting monthly, so the bank is
+expected to be refreshed from live re-benchmarks (the EXP-014/015 harness) rather than
+treated as frozen.
+
+## Difficulty ladder (through the real serve path, MLX)
+
+7 rungs trivial → distributed-systems-hard, each run with and without a 12,000-char
+opencode-scale system preamble: decisions identical 7/7 (system messages are excluded
+from the routing embedding), 4 distinct arms across the ladder (`luna_high` on
+easy/medium, `sol_xhigh`/`terra_high` on hard in-bank tasks, `opus5_high`), trivial
+one-liners abstain to `opus5_high` by design (logged as `off_distribution`). Table in
+`provenance/ladder.json`.
 
 ## Backend agreement (fp16 vs MLX 4-bit)
 
-Over 20 probe texts (DeepSWE issues, LCB tasks, short interactive prompts): max
-per-arm |ΔP(solve)| = **0.0274**; decisions and abstentions agreed on **20/20**
-probes, no arm-ranking flips (`provenance/agreement.json`). Cross-backend embedding
-cosine runs 0.91–0.97 — the temperature-softmax vote absorbs the quantization noise.
+Over 20 probe texts (DeepSWE issues, LCB tasks, short interactive prompts) against
+the live-cell bank: max per-arm |ΔP(solve)| = **0.0340**; decisions and abstentions
+agreed on **18/20** probes (`provenance/agreement_live.json`). **The two
+disagreements are utility-ranking flips near the decision boundary** — adjacent
+tiers whose utility margin is smaller than the 4-bit quantization noise
+(luna_medium↔luna_high at |ΔP|=0.0005, and luna_high↔opus5_high on one probe):
+with only 10 live arms the utility gaps are tighter than under the old 41-arm bank
+(which agreed 20/20). Treat backend choice as part of the artifact's identity; both
+flips escalate to an equal-or-stronger arm on MLX, never a weaker one. Cross-backend
+embedding cosine runs 0.91–0.97.
 
 ## Scope warning