Upload README.md with huggingface_hub

README.md

@@ -13,66 +13,59 @@ library_name: pytorch
 
 # KB-Diffusion Model B — word-level masked diffusion
 
-A 4.75M-parameter masked diffusion language model trained on 8,000 five-letter
-English words. This is the "generalization companion" experiment suggested in
-[KB-Diffusion](https://github.com/PastelRuntime/KB-Diffusion-Optimized) (an
-educational masked-diffusion project by Bijan Bowen / OminousIndustries):
+Masked diffusion language models trained on 8,000 five-letter English
+words. The "generalization companion" experiment from
+[KB-Diffusion](https://github.com/PastelRuntime/KB-Diffusion-Optimized)
+(an educational masked-diffusion project by Bijan Bowen / OminousIndustries):
 swap the repo's four keyboard layouts for thousands of words and see if the
-same recipe still works.
+same recipe still works. It does — and iterating on decoding strategy turned
+out to matter more than more parameters.
 
-It does, with a much harder hypothesis space. Trained in ~11 minutes on a free
-Kaggle T4.
+Two checkpoints, same LLaDA-style recipe (t ~ U(0.05, 1) masking, 1/t-weighted
+CE, bidirectional transformer, no causal mask):
 
-## The recipe (identical in shape to the keyboard model)
-
-- Bidirectional transformer, no causal mask: vocab 27 (a-z + [MASK]), seq len 5,
-  dim 256, 6 layers, 4 heads, ff 1024
-- LLaDA-style training: masking ratio t ~ U(0.05, 1), cross-entropy on masked
-  positions only, weighted by 1/t
-- AdamW, lr 3e-4, 8000 steps, batch 1024
+| | v2 | v3 |
+|---|---|---|
+| Params | 4.75M (6 layers) | 6.33M (8 layers) |
+| Steps | 8,000 | 12,000 + cosine LR |
+| Best valid English | 95.5% | **98.4%** |
+| Unique words / 512 | 409 | **428** |
+| Unigram TV vs exact Bayes | 0.0374 | **0.0135** |
 
-## Results
+## The headline finding
 
-256 samples per sampler, checked against the full ~16k list of English
-5-letter words:
+Identical v2 weights, different decoding:
 
-| Sampler | Valid English | Unique |
+| Decoding | Valid English | Cost (passes) |
 |---|---|---|
-| One-shot parallel (all 5 letters at once) | 2.0% | 256/256 |
-| k=2 commits per step | 39.5% | 256/256 |
-| Ancestral (commit 1, re-condition, repeat) | 68.4% | 254/256 |
-| Greedy ancestral (no sampling) | 100% | 1/64 ("bales") |
+| Ancestral, temperature 1.0 | 68.8% | 5 |
+| Ancestral, temperature 0.5 | **95.5%** | 5 |
+| Revision-capable (re-mask weak commits) | 77.1% | 22.5 |
 
-The model's from-scratch letter predictions also track exact analytic unigram
-statistics to mean total variation 0.0374 — the "transformer learns Bayes'
-rule from corrupted examples" effect, at 8000 classes instead of 4.
+A 27-point validity jump from temperature alone, at zero extra compute. The
+sampler is half the model — twice over.
 
-Two takeaways:
-
-1. The sampler is half the model. Identical weights produce 2% or 68% valid
-   output depending only on how commitments are made. Coherence is bought
-   with iteration, not parameters.
-2. 68.4% of outputs were valid English but only 66.4% came from the training
-   list — it generalizes past its vocabulary, producing words it never saw.
+Full methodology, negative results (revision sampling not worth it at N=5;
+rare-prefix Bayes tracking degrades), prompting study, and per-sampler
+tables: see `docs/model-b.md` in the GitHub repo.
 
 ## Usage
 
 ```python
 import torch
-from model_b_word_diffusion import Net, CH, MASK, N  # from the GitHub repo
+from model_b_word_diffusion_v3 import Net, CH, MASK, N  # from the GitHub repo
 
-model = Net()
-sd = torch.load("modelb_v2.pt", map_location="cpu", weights_only=True)
+model = Net(layers=8)  # v3
+sd = torch.load("modelb_v3.pt", map_location="cpu", weights_only=True)
 model.load_state_dict(sd)
 model.eval()
-# then use the ancestral confidence-commit sampler from the repo's script
+# ancestral confidence-commit sampler, temperature 0.5 — see repo script
 ```
 
-See the GitHub repo for the full training/eval script and writeup.
-
 ## Intended use & limitations
 
-An educational experiment, not a production model. 27-token vocabulary,
-5-position sequences, memorization-adjacent scale on purpose. It exists to
-make the masked-diffusion mechanism (parallel prediction, confidence
-commits, re-masking, posterior sharpening) measurable and visible.
+Educational artifact, not a production model: 27-token vocab, 5-position
+sequences. It exists to make the masked-diffusion mechanism (parallel
+prediction, confidence commits, re-masking, posterior sharpening)
+measurable — and to show how much decoding strategy contributes to
+generation quality on frozen weights.