Update README.md

README.md

@@ -80,6 +80,112 @@ with torch.no_grad():
 print(tokenizer.decode(output_ids[0].tolist()))
 ```
 
+## Standardized base-model evaluation
+
+The checkpoint was evaluated as a base causal language model with
+EleutherAI LM Evaluation Harness `v0.4.10`.
+
+Evaluation protocol:
+
+- Hugging Face backend: `hf`
+- maximum context length: 1,024
+- `add_bos_token=False`
+- no chat template
+- deterministic likelihood-based evaluation
+- harness seeds: `0,1234,1234,1234`
+- base checkpoints only; no SFT or instruction checkpoints
+
+| Metric | Learned input table | Fixed Binary-16 | Affine GF(2), table-free | SmolLM2-135M | SmolLM2-360M |
+|---|---:|---:|---:|---:|---:|
+| HellaSwag acc | 28.49 ± 0.45 | 29.04 ± 0.45 | 29.04 ± 0.45 | 35.36 ± 0.48 | 43.05 ± 0.49 |
+| HellaSwag acc_norm | 31.32 ± 0.46 | 32.32 ± 0.47 | 31.80 ± 0.46 | 43.02 ± 0.49 | 56.28 ± 0.50 |
+| ARC-Easy acc | 46.38 ± 1.02 | 47.90 ± 1.03 | 47.64 ± 1.02 | 64.44 ± 0.98 | 70.24 ± 0.94 |
+| ARC-Easy acc_norm | 40.70 ± 1.01 | 40.87 ± 1.01 | 41.20 ± 1.01 | 58.75 ± 1.01 | 68.18 ± 0.96 |
+| ARC-Challenge acc | 20.39 ± 1.18 | 19.62 ± 1.16 | 21.33 ± 1.20 | 28.07 ± 1.31 | 36.26 ± 1.40 |
+| ARC-Challenge acc_norm | 25.85 ± 1.28 | 26.19 ± 1.28 | 24.83 ± 1.26 | 29.61 ± 1.33 | 38.05 ± 1.42 |
+| PIQA acc | 62.35 ± 1.13 | 62.57 ± 1.13 | 62.68 ± 1.13 | 68.44 ± 1.08 | 71.38 ± 1.05 |
+| PIQA acc_norm | 60.61 ± 1.14 | 62.08 ± 1.13 | 60.94 ± 1.14 | 68.39 ± 1.08 | 71.82 ± 1.05 |
+| WinoGrande acc | 50.20 ± 1.41 | 50.12 ± 1.41 | 50.43 ± 1.41 | 52.57 ± 1.40 | 59.35 ± 1.38 |
+| OpenBookQA acc | 18.40 ± 1.73 | 17.20 ± 1.69 | 17.60 ± 1.70 | 22.00 ± 1.85 | 24.80 ± 1.93 |
+| OpenBookQA acc_norm | 29.20 ± 2.04 | 31.00 ± 2.07 | 29.40 ± 2.04 | 32.60 ± 2.10 | 37.80 ± 2.17 |
+| CommonsenseQA acc | 20.31 ± 1.15 | 19.90 ± 1.14 | 20.23 ± 1.15 | 19.90 ± 1.14 | 21.05 ± 1.17 |
+| MMLU 0-shot | 24.13 ± 0.36 | 23.86 ± 0.36 | 24.11 ± 0.36 | 24.24 ± 0.36 | 25.47 ± 0.37 |
+| MMLU 5-shot | 25.68 ± 0.37 | 25.60 ± 0.37 | 25.66 ± 0.37 | 25.39 ± 0.37 | 25.05 ± 0.37 |
+| LAMBADA accuracy | 22.38 ± 0.58 | 21.23 ± 0.57 | 21.99 ± 0.58 | 42.97 ± 0.69 | 53.31 ± 0.70 |
+| LAMBADA perplexity | 95.14 ± 4.01 | 101.74 ± 4.27 | 100.61 ± 4.17 | 19.06 ± 0.63 | 9.38 ± 0.27 |
+| WikiText word perplexity | 81.04 | 74.87 | 76.17 | 25.53 | 18.84 |
+| WikiText byte perplexity | 2.27 | 2.24 | 2.25 | 1.83 | 1.73 |
+| WikiText bits/byte | 1.19 | 1.16 | 1.17 | 0.87 | 0.79 |
+
+The three paper checkpoints form the controlled architectural comparison.
+SmolLM2-135M and SmolLM2-360M are external reference models, not matched
+baselines: they use different architectures, tokenizers, training mixtures,
+and much larger pretraining budgets. SmolLM2-135M was trained on approximately
+2T tokens and SmolLM2-360M on approximately 4T tokens, whereas the paper
+checkpoints saw approximately 16–17B tokens. Their scores therefore provide
+context for absolute capability and must not be interpreted as isolating the
+effect of the input parameterization.
+
+Perplexity values should be interpreted especially cautiously across different
+tokenizers. The primary controlled comparison is among the three paper models,
+which share the same tokenizer, data pipeline, and architecture.
+
+
+## Input-interface audit
+
+This checkpoint stores a deterministic `65,536 × 16` binary codebook as a
+frozen `nn.Embedding` for computational convenience. During training, the
+table was initialized from the fixed token codes, marked with
+`requires_grad=False`, and excluded from the optimizer. It therefore contained
+1,048,576 stored but non-trainable values and contributed zero trainable input
+parameters.
+
+The released checkpoint can be audited directly:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM
+
+repo_id = "E6E831728/fixed-minimal-binary-code"
+
+model = AutoModelForCausalLM.from_pretrained(
+    repo_id,
+    trust_remote_code=True,
+    torch_dtype=torch.float32,
+).cpu().eval()
+
+embedding = model.get_input_embeddings()
+weight = embedding.weight.detach()
+
+vocab_size, code_bits = weight.shape
+ids = torch.arange(vocab_size, dtype=torch.long)
+positions = torch.arange(code_bits, dtype=torch.long)
+
+expected = ((ids[:, None] >> positions[None, :]) & 1).float()
+expected[model.config.pad_token_id].zero_()
+
+print("shape:", tuple(weight.shape))
+print("unique values:", torch.unique(weight).tolist())
+print("all entries binary:", bool(torch.all((weight == 0) | (weight == 1))))
+print("exact canonical-code match:", bool(torch.equal(weight, expected)))
+print("mismatching entries:", int((weight != expected).sum().item()))
+
+assert tuple(weight.shape) == (65536, 16)
+assert torch.all((weight == 0) | (weight == 1))
+assert torch.equal(weight, expected)
+```
+
+Expected audit properties:
+
+```text
+shape: (65536, 16)
+unique values: [0.0, 1.0]
+all entries binary: True
+exact canonical-code match: True
+mismatching entries: 0
+```
+
+
 ## Intended use
 
 This checkpoint is provided for anonymous review and reproducibility of the paper's main claim: a trainable input embedding table is not necessary for useful language modeling in the studied regime.