Add wikitext-2 perplexity vs BF16 source; revise file recommendation

README.md

@@ -37,21 +37,29 @@ a 34.66B-parameter MoE coding model (256 experts, 8 active, 256K context,
 > anyway, do not trust the output.
 
 > [!WARNING]
-> Both files load, generate coherent output, and were throughput-benchmarked
-> on RDNA4 `gfx1201`. That is the **only** validation performed. No Strix Halo
-> testing and **no quality evaluation of any kind** — see [What was not
+> Validation was performed on RDNA4 `gfx1201` only: both files load, generate
+> coherent output, were throughput-benchmarked, and were measured against the
+> BF16 source for wikitext-2 perplexity. No Strix Halo testing and **no
+> code-specific or agentic evaluation** — see [What was not
 > measured](#what-was-not-measured) before relying on either file.
 
 ## Which file?
 
-| File | Size | Effective BPW | Pick it if |
-| --- | ---: | ---: | --- |
-| `KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX_LEAN.gguf` | 17.32 GiB | 4.29 | You want the smaller, faster file. Recommended default. |
-| `KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4.gguf` | 21.18 GiB | 5.25 | You want more bits on the expert down-projections. |
+| File | Size | Effective BPW | Wikitext-2 PPL | Pick it if |
+| --- | ---: | ---: | ---: | --- |
+| `KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4.gguf` | 21.18 GiB | 5.25 | 6.9182 (+1.38%) | You care about output quality. **Recommended for coding.** |
+| `KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX_LEAN.gguf` | 17.32 GiB | 4.29 | 7.1079 (+4.16%) | You need the smaller file or the extra decode speed. |
 
-`STRIX_LEAN` is 18% smaller *and* measurably faster on both backends, so it is
-the recommended starting point despite the Strix-oriented name. The recipe was
-tuned on `gfx1151`; nothing about the file format is Strix-specific.
+This is a **real tradeoff, not a clean win for either file.** `STRIX_LEAN` is
+18% smaller and 13% faster at decode, but gives up three times as much
+perplexity against the BF16 source. For a coding model — where a single wrong
+token breaks a program — the plain `Q4_0_ROCMFP4` is the safer default, and
+21.18 GiB still fits a 32 GB card comfortably.
+
+Take `STRIX_LEAN` if you are memory-constrained (24 GB cards), or if you are
+throughput-bound and have validated that the quality holds on your own tasks.
+Its recipe was tuned on `gfx1151`; nothing about the file format is
+Strix-specific.
 
 ## Why the sizes differ from the nominal BPW
 
@@ -86,17 +94,41 @@ Two results worth acting on:
 - **Use Vulkan on this hardware.** Vulkan decodes roughly **2× faster** than
   HIP/ROCm for both files (122 vs 59 t/s on `STRIX_LEAN`) and also leads on
   prompt fill. This matches ROCmFPX's own Strix Halo findings.
-- **`STRIX_LEAN` wins on both axes.** It is 18% smaller *and* faster —
-  +13% decode and +5% prefill on Vulkan, +13% decode and +45% prefill on ROCm.
+- **`STRIX_LEAN` is the faster file** — +13% decode and +5% prefill on Vulkan,
+  +13% decode and +45% prefill on ROCm — but see the quality section below
+  before choosing it on speed alone.
 
 No control quant (Q4_K_M or similar) was benchmarked, so these numbers compare
 the two ROCmFP4 files against each other, not against ordinary GGUF quants.
 
+## Measured quality — wikitext-2 perplexity
+
+`llama-perplexity`, full wikitext-2 test set (580 chunks), `-c 512 -b 512`,
+FlashAttention on, Vulkan. The BF16 source GGUF was measured on the same host
+with the same settings, split across three GPUs.
+
+| File | BPW | PPL | Δ vs BF16 |
+| --- | ---: | ---: | ---: |
+| `KAT-Coder-V2.5-Dev-BF16.gguf` (source) | 16.01 | 6.8237 ± 0.04537 | — |
+| `Q4_0_ROCMFP4` | 5.25 | 6.9182 ± 0.04607 | **+1.38%** |
+| `Q4_0_ROCMFP4_STRIX_LEAN` | 4.29 | 7.1079 ± 0.04762 | **+4.16%** |
+
+Both quants land where you would expect for their bit budgets, and neither is
+degenerate. The gap between them is larger than the error bars, so it is a
+real difference and not measurement noise: `STRIX_LEAN` buys its 18% size
+reduction with roughly 3× the perplexity cost.
+
+Perplexity is a weak proxy for coding ability. It measures next-token
+prediction on English Wikipedia, not code correctness or tool-call formatting.
+Treat it as a floor check — it rules out a broken quantization, it does not
+establish that either file codes as well as the source.
+
 ## What was not measured
 
-- **Output quality.** No perplexity, KL-divergence, HumanEval, or MBPP
-  comparison against the BF16 source. Neither file has been quality-checked at
-  all.
+- **Coding ability.** No HumanEval, MBPP, or any code benchmark. Wikitext-2
+  perplexity was measured (see above), but it does not measure code
+  correctness.
+- **KL-divergence** against the BF16 source. Perplexity only.
 - **Agentic and tool-calling behavior**, which is the point of a coding model.
   Untested.
 - **Any hardware other than `gfx1201`.** Not tested on Strix Halo, RDNA3,
@@ -123,7 +155,7 @@ automatically. `gfx1200` builds are **not** interchangeable on these cards.
 
 ```bash
 ./build-rdna4/bin/llama-server \
-  -m KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
+  -m KAT-Coder-V2.5-Dev-Q4_0_ROCMFP4.gguf \
   -dev Vulkan0 \
   -ngl 999 \
   -fa on \
@@ -191,7 +223,7 @@ and the converted GGUFs contain no multimodal projector. Despite the
 
 - Requires the ROCmFPX fork; no upstream llama.cpp compatibility.
 - Validated on exactly one `gfx1201` host, batch 1, shallow context.
-- No quality evaluation of any kind has been published for these artifacts.
+- Quality evidence is wikitext-2 perplexity only; no code or agentic evals.
 - 34.66B MoE: needs ~18–22 GB for weights plus KV cache. Comfortable on a
   32 GB card, tight on 24 GB with meaningful context.