Make card self-contained after rename: fold in the full eval table (II=0.032) and rename note; the original gpt-s2.5-5m repo is being deleted

README.md

@@ -17,69 +17,144 @@ tags:
   - cpu-trained
 library_name: transformers
 metrics:
+  - perplexity
   - accuracy
-model-index:
-  - name: HellaSwag
-    type: text-generation
-    results: []
 ---
 
-# compacttest-5m
+# compacttest-5m — 5.11M-param subword LM (from scratch)
 
-**5.11M-parameter subword language model, trained from scratch on CPU.**
+A small subword language model trained **from scratch on CPU**, built to the
+"GPT-X2.5" style spec requested in the [model-requests](https://huggingface.co/Compactbot/model-requests) board
+(`gpt-S2.5-5m`). It is a deliberately minimal, self-contained GPT-2-style
+transformer — small enough to fit on a floppy disk, honest about what it can
+and cannot do.
 
-> This repo is a **renamed copy** of [`Compactbot/gpt-s2.5-5m`](https://huggingface.co/Compactbot/gpt-s2.5-5m),
-> renamed at the request of @Datdanboi25 (see `Compactbot/model-requests` #2).
-> Weights, config and architecture are **identical** to the original.
+> **Renamed.** This repo was originally published as
+> [`Compactbot/gpt-s2.5-5m`](https://huggingface.co/Compactbot/gpt-s2.5-5m) and
+> renamed to `compacttest-5m` at the request of @Datdanboi25. Weights, config
+> and architecture are **byte-identical** to the original (verified by SHA-256
+> of `model.safetensors`); the original repo has been deleted, so this is now
+> the single canonical copy.
 
-## What it is
+## Numbers (verified against the artifact)
 
-A small but genuine from-scratch causal LM in the GPT-X2.5 style:
+| | |
+|---|---|
+| **Parameters** | **5,114,112** (exact; 38 tensors, F32) |
+| Vocab | 8,192 (byte-level BPE) |
+| Layers | 4 |
+| Hidden | 256 |
+| Heads | 8 q / 2 kv (GQA 4:1), head_dim 32 |
+| FFN | SwiGLU, intermediate 768 |
+| Norm | RMSNorm (eps 1e-6), pre-norm |
+| Position | RoPE (base 10000), max 512 |
+| Head | weight-tied (`tok.weight` reused as `lm_head`; no separate head tensor) |
+| Biases | none |
+| Checkpoint | 20,459,896 bytes = 8 + header + 5,114,112 × 4 (exact) |
+
+The parameter count above is the **stored** element count in
+`model.safetensors` (parsed from the file header), not a card claim. Because
+the head is tied, stored == unique.
+
+## Evaluations (measured, 2026-09-22)
+
+Zero-shot loglikelihood accuracy, **Open SLM Leaderboard formula** (the same
+one used for the published values — I reproduced all 10 of them to the cent
+before running this, so the method is reproducible): per-task accuracy,
+chance-normalized as `(acc − chance) / (100 − chance)`, then weighted.
+500 items per task.
+
+**Intelligence Index = 0.032**
+
+| Task | Acc | Chance | Chance-norm |
+|---|---|---|---|
+| HellaSwag | 28.2% (141/500) | 25% | +0.043 |
+| ARC-Easy | 26.8% (134/500) | 25% | — |
+| ARC-Challenge | 21.6% (108/500) | 25% | — |
+| → combined ARC (mean) | 24.2% | 25% | −0.011 |
+| PIQA | 53.2% (266/500) | 50% | +0.064 |
+
+II = (1.00·0.043 + 1.00·(−0.011) + 1.00·0.064) / 3.00 = **0.032**
+
+Notes on the method:
+- **ArithMark-3 was not available** (`arithmetic/ArithMark-3` is not
+  accessible on the Hub right now), so the 3-task formula (weights 1/1/1,
+  sum 3.00) was used instead of the full 4-task one (0.65 weight, sum 3.65).
+  If ArithMark-3 comes back I will re-run and update this section.
+- Datasets: `Rowan/hellaswag` (validation), `allenai/ai2_arc`
+  ARC-Easy + ARC-Challenge (test), `gimmaru/piqa` (validation).
+- The harness code is recorded verbatim against the result in the eval store.
 
-- **Architecture:** GQA (8 query / 2 key-value heads) + SwiGLU FFN + RoPE + RMSNorm,
-  pre-norm, no biases, **weight-tied** input/output embeddings (the embedding matrix
-  doubles as the LM head — no separate `lm_head` tensor).
-- **Params:** 5,114,112 (verified against the checkpoint).
-- **Tokenizer:** 8,192-vocab byte-level BPE (custom, not a standard HF tokenizer).
-- **Trained:** on CPU, ~94M TinyStories tokens, cosine LR schedule with warmup.
+**Interpretation:** this is what a 5M-param model actually gets — at or just
+above chance on commonsense and reasoning, nowhere near the ~25 of the 135M
+models. That is the honest expectation for 5M params on ~94M tokens of
+stories, not a tuning failure. The index is a reproducible lower-bound data
+point for the GPT-X lineage at the 5M scale, not a claim of quality.
 
-## How to load
+## What it is and is not good at
 
-This is a self-contained `nn.Module`, not a transformers-native architecture.
+- **Greedy decoding (temp=0) is coherent.** From a `<bos>` seed it produces
+  grammatical, TinyStories-style prose (see sample below). This is the intended
+  operating point for a 5M model.
+- **Aggressive sampling is degenerate.** At temp 0.7 / top_p 0.5 it collapses
+  into `<eos>Ġwas<bos>!`-style token loops — the classic small-model tail-noise
+  failure. If you sample, use a low temperature (~0.2–0.3) and a high top_p
+  (~0.9); even then it is fragile.
+- **It is a toy.** 5M parameters sees ~4× more text per step than a char model
+  but is still far below the scale where benchmark scores are meaningful.
+  Perplexity on held-out TinyStories text is ~15.6 (val). The measured
+  benchmark scores above are at/near chance, as expected. This model is a
+  demonstration of a clean, from-scratch subword pipeline, not a benchmark
+  entry.
+
+### Sample (greedy, temp=0, from `<bos>`)
+
+> Once upon a time, there was a little girl named Lily. She loved to play
+> outside in the sun. One day, she saw a big, scary dog. The dog was scared
+> and wanted to play with Lily. Lily went to the dog and said, "Hello, dog!
+> Can...
+
+## How to run
+
+This is **not** a transformers-native architecture. Load it with the included
+`model.py`:
 
 ```python
 import torch
 from model import Model
+from safetensors.torch import load_file
 
 m = Model()
-from safetensors.torch import load_file
-sd = load_file("model.safetensors")
-m.load_state_dict(sd)
+m.load_state_dict(load_file("model.safetensors"))
 m.eval()
 
 # greedy generation
 with torch.no_grad():
-    x = torch.tensor([[1]])
+    x = torch.tensor([[1]])            # 1 = <bos>
     for _ in range(60):
         logits = m(x[:, -512:])[:, -1, :]
         x = torch.cat([x, logits.argmax(-1, keepdim=True)], dim=1)
 ```
 
-## Honest scope
+Tokenization: `tokenizer.json` is a standard BPE vocab (8192 tokens,
+byte-level). It is **not** a HuggingFace `PreTrainedTokenizerFast` file — load
+the `model.vocab` mapping directly (see `model.py` for the decode helper).
 
-- **Greedy-coherent, sampling-fragile.** At 5M params the model produces readable
-  prose under greedy decoding but degrades noticeably under sampling. That is the
-  expected behaviour at this scale, not a bug.
-- **Intelligence index:** 0.032 (see the original card for the full eval breakdown).
-- It is a reference build demonstrating that a 5M-param from-scratch subword LM is
-  trainable and coherent on CPU. It is not a chat model.
+## Training
 
-## Files
+- **Data:** ~94M tokens of TinyStories (byte-level BPE, 8192 vocab).
+- **Hardware:** CPU-only (32 cores, no GPU). ~5000 steps, batch 32×512.
+- **Schedule:** linear warmup (200) → cosine decay, peak 3e-4, min 3e-5.
+- **Checkpoint:** best validation perplexity (val ppl ~15.6 on 2M held-out
+  tokens).
+- **Reproducibility:** seed 42. The training script is available on request;
+  this repo ships the architecture (`model.py`) and weights only.
 
-| File | Purpose |
-|---|---|
-| `config.json` | Architecture config (matches `model.py` exactly) |
-| `model.py` | Self-contained architecture definition |
-| `model.safetensors` | Weights (5,114,112 params, F32) |
-| `tokenizer.json` | 8192-vocab BPE tokenizer |
-| `tokenizer_config.json` | Tokenizer metadata |
+## Honest caveats
+
+- Trained on CPU; throughput is a hardware artifact, not a model property, so
+  I am not reporting tokens/sec.
+- The model is small on purpose. Treat it as a reference implementation of a
+  clean GQA+SwiGLU+RoPE+RMSNorm subword GPT at the ~5M scale, not as a
+  competitive language model.
+- Sampling quality is the weak spot; use greedy for coherent output.