SlayerLab/gollem-v5-ckpts

GoLLeM-v5 — Tiny English Language Models (16M-128M)

Research checkpoints of sub-100M-parameter English language models, GPT-style decoders (nanoGPT lineage) trained for the Glint Tiny-ML Leaderboard. The repository holds a controlled scaling study (tokens, width, optimizer) and a set of data-attribution experiments on the 64M model (paired continued-training runs that differ only in data).

Repository map (what is where)

Every run folder holds plain PyTorch checkpoints (ckpt.pt or ckpt_<N>k.pt, where N is the training step in thousands). Folders with a README.md (all runs since 2026-09-25) contain the full specification: data pool and its hash, trainer version, seed, steps and tokens. "Result" is the checkpoint that the reported numbers come from; it is always the planned last step, never a selected "best" checkpoint.

Final GoLLeM-v5 models (training now)

folder	model	status	result checkpoint	live metrics
final_64m_14x576/	GoLLeM-v5 64M final, 14 layers / d_model 576 / 9 heads, 62.9M	training, checkpoint every 20k steps	ckpt_760k.pt (when finished)	track
final_128m_16x768/	GoLLeM-v5 128M final, 16 layers / d_model 768 / 12 heads, 122.8M	training, checkpoint every 20k steps	ckpt_760k.pt (when finished)	track

Published models (board entries and references)

folder	model	eff (canonical protocol)	result checkpoint	live metrics
v1_muon/	64M flagship v1 (Muon), 62.9M, ARC-MIX (e87f594f…), 400k steps / 13.1B tokens, seed 1337, v1 trainer	75.81	ckpt_400k.pt (also 40k, 80k, 160k, 320k, 360k)	track
v1_128m/	128M v1 (Muon), 122.8M, 16 × 768, same recipe and data as the 64M flagship, 400k steps / 13.1B tokens; resumed at 120k	75.84	ckpt_400k.pt (every 40k from 40k)	track
run_32m_16b/	32M board entry, ARC-MIX, 16B tokens	75.41	ckpt.pt	–
run_16m_expanded/	16M board entry, expanded 8.29B corpus, 16B tokens	74.33	ckpt.pt	–

Earlier scaling and optimizer studies (numbers in the Checkpoints table below)

folder	what it is
bpe16m_3.2B/, bpe16m_6B/, bpe16m_10B/	16M token scan (3.2B / 6B / 10B tokens)
bpe32m_baseline/	32M at 10B tokens
run_32m_18b/	32M at 18B tokens on ARC-MIX (slope check: more epochs did not help)
run_149m/	145.6M scaling reference (labelled 149M in earlier versions of this card; under-trained, 10B tokens, older recipe)
run_16m_arcmix/	16M on ARC-MIX instead of the expanded corpus (ARC unchanged at 16M; glint_16m_arcmix_results.json)
v2_muon/	64M on a FineWeb-Edu-dominant blend with a changed recipe (40k–160k); confounded (different document separator and recipe), stopped
v2b/	64M on ARC-MIX with the v2 recipe (z-loss, logit cap, WSD), 40k and 80k; diagnostic

Data experiments on the 64M flagship (branches from step 320k, 80k steps of the cosine tail) — see Data-attribution experiments below

folder	data	result checkpoint	live metrics
ctrl_arcmix_resume/	ARC-MIX (control)	ckpt_400k.pt	track
forkB_arcmix_edu/	ARC-MIX + educational (flawed first build)	ckpt_400k.pt	track
r3A_arcmix_edu_clean/	45% ARC-MIX + 55% educational, clean build	ckpt_400k.pt	track
r3B_arcmix_qa2x/	ARC-MIX with Q&A documents ×2	ckpt_400k.pt	track
r4K_anchor_arcmix_qa1/	ARC-MIX sample of the same pool size, Q&A ×1 (anchor for r3B)	ckpt_400k.pt	track
r4B_arcmix_qa2x_seed1338/	as r3B, seed 1338 (run-to-run noise)	ckpt_400k.pt	track

Constant-learning-rate building pair (WSD, from the 64M flagship at 400k; stopped after two segments, a tie)

folder	what it is	live metrics
s12A_arcmix_pool/	arm A, ARC-MIX sample (420k, 440k, 460k)	track
s12B_arcmix_edu/	arm B, segment 1, ARC-MIX + FineWeb-Edu (420k)	track
s12B2_arcmix_edu/	arm B, segment 2 (440k, 460k)	track
probe_s12A_420k/, probe_s12B_420k/, probe0_s12A_420k/, probe_s12A_440k/, probe_s12B2_440k/	4k-step learning-rate decay probes used to compare the arms	–

From-scratch 32M tests (r6 trainer, 150k steps / 4.9B tokens, checkpoints at 30k–150k; rules written before the results)

folder	test	outcome (selection sets, not the board test)	live metrics
z1_Z_fineweb_edu/, z1_Z_fineweb_edu_seed1338/	31.3M, 15 × 384; FineWeb-Edu (score ≥ 3), pool 877f80d1…; seeds 1337 and 1338	tie with ARC-MIX (Z − C = −0.24 eff, threshold 0.6): ARC-MIX kept	track, track
z1_C_arcmix/	31.3M, 15 × 384; ARC-MIX after the test/validation scan, pool 27bae5ad… (arm C)	reference arm of the test above	track
h4_fineweb_edu_22x320/	31.0M, 22 × 320 × 5 heads (deeper, narrower) on pool 877f80d1…	BLiMP +0.41 against a +1.0 threshold: shapes unchanged	track
z4_fineweb_edu_ge4/	31.3M, 15 × 384; FineWeb-Edu (score ≥ 4), pool d8d49b35…	evaluation pending a cleaned axis; this filter concentrates ARC-like quiz text	track

Other folders and root files

path	contents
vals/	validation streams (arcmix_val.bin, fwe_only_val.bin, v2_mix_val.bin)
eval/	older evaluation scripts (superseded by glint_parity_eval.py)
decontam_verify/	a FineWeb-Edu sample used to check the test-set overlap filter
tokenizer.json, train_gpt_ref.py, glint_parity_eval.py	tokenizer, model/trainer, board-protocol evaluation
glint_*_results.json, progress_v5.png, scaling_v5.png, board_overlay_v5.png	earlier result files and charts (from before the harness fix; the numbers in this card supersede them)
other *.py, *.sh	data-building and pod scripts used for the runs above

Related repositories: the ARC-MIX corpus card SlayerLab/gollem-v5-arcmix-9b and the live dashboard SlayerLab/gollem-dashboard.

Model details

• Architecture: 16M/32M = decoder-only Transformer (nanoGPT lineage), learned positional embeddings, tied input/output embeddings. 64M flagship = Qwen3-style decoder (RoPE θ=100k, SwiGLU, RMSNorm, QK-Norm, value residuals).
• Sizes: 16M = 6 layers / d_model 408 / 6 heads (17.4M); 32M = 6 layers / d_model 576 / 9 heads (31.6M); 64M flagship = 14 layers / d_model 576 / 9 heads (62.9M); 128M = 16 layers / d_model 768 / 12 heads (122.8M), same Qwen3-style stack as the 64M.
• Context length: 1024 tokens.
• Tokenizer: BPE, vocab 12288 (tokenizer.json), shared across all checkpoints.
• Training: 16M/32M: AdamW, lr 6e-4 -> 6e-5 (cosine), batch 64 x 1024, seed 1337, bf16 (RTX 5090). 64M flagship: Muon (muon-lr 0.02) + AdamW for non-matrix parameters, same cosine schedule, 400k steps.

Checkpoints

checkpoint	params	shape	tokens	BLiMP	ARC-Easy	WikiText-2 BPB
bpe16m_3.2B/ckpt.pt	17.4M	L6 d408 h6	3.2B	67.40⁺	38.22	1.2161
bpe16m_6B/ckpt.pt	17.4M	L6 d408 h6	6B	68.92⁺	39.10	1.1943
bpe16m_10B/ckpt.pt	17.4M	L6 d408 h6	10B	70.36°	39.52	1.1815
bpe32m_baseline/ckpt.pt	31.6M	L6 d576 h9	10B	70.08ˢ	42.59	1.124
run_16m_expanded/ckpt.pt (16M board entry)	17.4M	L6 d408 h6	16B†	70.08	40.91	1.4193
run_32m_16b/ckpt.pt (32M board entry)	31.6M	L6 d576 h9	16B‡	73.48	44.44	1.3441
run_149m/ckpt.pt (scaling ref, labelled 149M earlier)	145.6M	L12 d960 h12	10B§	76.68	49.66	1.2051
run_32m_18b/ckpt.pt (slope-check)	31.6M	L6 d576 h9	18B¶	72.38°	44.70	1.3431
Muon 32M (results only: glint_32m_muon_results.json, checkpoint not published)	31.6M	L6 d576 h9	16B‖	72.29°	42.89	1.3866
v1_muon/ckpt_400k.pt (64M flagship)	62.9M	L14 d576 h9	13.1B★	75.83	47.94	1.246

BLiMP harness fix (2026-09-25). Our evaluation script matched BLiMP configs by substring, so 6 phenomena were counted twice (73,000 pairs instead of 67,000). It is fixed in glint_parity_eval.py (exact config match + assertion of 67,000 pairs). The three board entries (bold BLiMP) were recomputed with the fixed script; for each we also reproduced the old 73,000-pair variant, which matches the previously published number to two decimals, so the difference comes from the loader alone (+0.16 to +0.46 pp before the fix). The 145.6M reference was recomputed later (see §). Other rows are earlier results, not recomputed:

• ° computed with the pre-fix 73,000-pair loader (pair count recorded in the results file); on the three recomputed models the bias was +0.16 to +0.46 pp upward, expect similar here;
• ˢ BLiMP on a 4,000-pair random sample, so the error is random (SE ~0.7 pp), not a systematic bias;
• ⁺ the results file is not archived, so the pair count is unknown; treat as indicative only.

† 16M board entry = expanded 8.29B-token corpus (~1.9 epochs). eff 74.33 with the fixed BLiMP (70.08 / 40.91 / byte_ppl 2.6746).

‡ 32M board entry = 32M at 16B tokens on the ARC-MIX corpus. eff 75.41 with the fixed BLiMP (73.48 / 44.44 / byte_ppl 2.5386). It broke the 16M BLiMP ceiling (~70) seen in the token scan: more capacity plus tokens moved both axes.

§ 145.6M scaling reference (labelled 149M in earlier versions of this card; 145.6M counts unique parameters with tied embeddings counted once, as for the 64M and 128M, and includes 0.98M learned positional embeddings). Older nanoGPT-lineage recipe with learned positions, trained to step 305,000 = 9.99B tokens, about 69 tokens per parameter (under-trained). Recomputed on 2026-09-26 with the fixed harness (67,000 BLiMP pairs): ARC-Easy 49.66 / BLiMP 76.68 / WikiText-2 byte_ppl 2.3055 → eff 74.26; the earlier BLiMP 76.99 came from the 73,000-pair loader. Highest raw scores among the older runs, but the size multiplier falls with size, so it ranks below the 64M flagship on eff. Because the recipe differs, it is a size reference, not a point on the final-run curves.

¶ Slope-check = 32M at 18B tokens on the same ARC-MIX corpus as the 32M entry. BLiMP 72.38 (lower than at 16B) with byte_ppl flat: more epochs over a fixed corpus did not help. Diagnostic run.

‖ 32M with Muon on the expanded corpus. This run changed optimizer and corpus at once, so it cannot isolate Muon. The clean optimizer comparison was later run at 64M (see ★). Diagnostic run.

★ 64M flagship (v1 Muon) = Qwen3-style decoder + value residuals, Muon optimizer, ARC-MIX 9.42B corpus, 400k steps = 13.1B tokens (~1.4 epochs). Recomputed with the fixed harness: ARC-Easy 47.94 / BLiMP 75.83 / WikiText-2 byte_ppl 2.372 (BPB 1.246) → eff 75.81. The maintainer's independent re-benchmark (PR #78 and discussion #1; checkpoint sha256 59f982c1… matched) reproduced ARC-Easy 47.94 exactly; its BLiMP 75.99 matches our pre-fix 73,000-pair variant to two decimals. Earlier versions of this card cited eff 77.51 (byte_ppl 2.016, from a wrong bytes-per-token factor 4.755 instead of 3.8605) and later eff ~75.9 (BLiMP 75.99); both are superseded. Muon was at least as good as AdamW in a clean 64M A/B (identical data, architecture and seed; only the optimizer differs); those numbers pre-date the BLiMP fix.

Final GoLLeM-v5 runs (in progress)

Two final models are training now with a recipe fixed before the runs started. Neither the shape nor the data is chosen by looking at intermediate results; the two candidate changes tested beforehand (FineWeb-Edu instead of ARC-MIX, a deeper shape) did not pass their pre-registered rules (see the repository map).

	GoLLeM-v5 64M final	GoLLeM-v5 128M final
folder	final_64m_14x576/	final_128m_16x768/
shape	14 layers / d_model 576 / 9 heads	16 layers / d_model 768 / 12 heads
parameters	62,867,021	122,759,951
steps / tokens	760,000 / 24.9B (≈2.65 passes over the pool)	760,000 / 24.9B (≈2.65 passes over the pool)
checkpoints	every 20,000 steps (38 in total)	every 20,000 steps (38 in total)
hardware	1× RTX 5090 (Xeon Gold 6530 host to step 300,000; new pod with Ryzen 9 9950X host after the resume)	1× RTX 5090 (Ryzen 9 9950X host; new pod after the resume)
measured speed	~202k tokens/s (to step 300,000), ~224k tokens/s after the resume	~140k tokens/s
started / expected end (UTC)	2026-09-26 09:09 / ~2026-09-27 19:40	2026-09-26 09:09 / ~2026-09-28 13:35
live metrics	a7e6791b-30c5-4546-b157-8aac12c4d6fd	d96091ad-922b-404c-8e8d-6ddaa5fa090a

• Architecture: Qwen3-style decoder as in the 64M flagship: RMSNorm, RoPE (θ = 100,000), SwiGLU (ratio 2.667), QK-norm, value residuals, context 1,024, vocabulary 12,288 (BPE).
• Optimization: Muon (lr 0.02) for the 2-D hidden weights and AdamW for the rest (betas 0.9 / 0.95, weight decay 0.1); peak learning rate 6e-4, 2,000 warmup steps, cosine decay to 6e-5 at step 760,000; batch 32 × 1,024 tokens; seed 1337; no z-loss, no logit cap.
• Trainer: each training window is drawn once per pass over the pool; the earlier runs sampled windows with replacement, and a resume could replay early windows. Resume from a checkpoint is exact. A supervisor restarts the trainer from the last checkpoint only after an out-of-memory exit (at most three times).
• Data (pool sha256 ecfd0a40…, 9,391,706,576 tokens): ARC-MIX, built from the base corpus (5,396,605,407 tokens) + FineWeb-Edu (2,756,248,149) + 58 OpenStax CC BY 4.0 textbooks ×4 (140,742,204) + extra copies of ARC-relevant FineWeb-Edu documents (1,123,440,072) = 9,417,035,832 tokens. From it we removed every document that matched WikiText-2 test/validation or ARC validation/test in a normalized 13-gram and short-question scan (4,362 documents); documents containing the marker ‘CC BY-NC-SA’ were also removed (1,175 documents). 55 OpenStax titles remain; they are listed in OPENSTAX_ATTRIBUTION.md in each final folder. The pool was built independently on both machines with identical hashes.
• Measurement: the Glint-1.3 benchmark.py protocol: BLiMP 67,000 pairs (first token not scored), ARC-Easy test 2,376 questions (bare prompt, LL(question + choice) − LL(question)), WikiText-2 test in 256-token windows, byte perplexity with 3.8605 bytes per token for this tokenizer. eff = mean(BLiMP, ARC-Easy, WikiScore) × size multiplier (1.03646 for the 64M, 1.00839 for the 128M, computed from the declared sizes 62.9M and 122.8M as the board does), with the board's own constants.
• Resume after the loss of the pods: the original pods were lost on 2026-09-26 at 22:38 UTC. Both runs were resumed on new pods on 2026-09-27 at 00:58 UTC from the step-300,000 (64M) and step-200,000 (128M) checkpoints on the Hub (SHA-256 checked against the Hub's LFS hash), with the same trainer, flags and pool (rebuilt on the new pods; SHA-256 ecfd0a40… checked before the start). Steps 300,000–301,000 (64M) and 200,000–208,000 (128M) were recomputed on the new hardware; the tracker keeps the points it already had for those steps. On those recomputed steps the training loss matches the original run to within 4×10⁻⁵ (5 logged steps for the 64M, 40 for the 128M), so the resumed runs see the same data in the same order as an uninterrupted run.
• Result = the last checkpoint (step 760,000). Intermediate checkpoints are public but are not used for selection or for reporting.

Progress of the final runs. Measured with the Glint-1.3 benchmark.py protocol (ARC-Easy test 2,376 questions, BLiMP 67,000 pairs, WikiText-2 test) on the checkpoints uploaded so far. Training in progress (step N of 760,000; learning rate still high); not a result. The recipe is fixed, so these numbers select nothing; the result will be the checkpoint at step 760,000. The table grows as new checkpoints are measured. The tracker's public view shows training metrics (loss, throughput); the evaluation numbers are in this table.

run	step (of 760,000)	ARC-Easy	BLiMP	WikiText-2 byte_ppl	eff	status
64M final (14×576)	20,000 (2.6%)	40.61	72.83	2.6070	71.66	training in progress; not a result
64M final (14×576)	40,000 (5.3%)	43.43	73.46	2.5388	73.01	training in progress; not a result
64M final (14×576)	60,000 (7.9%)	43.77	73.41	2.4992	73.21	training in progress; not a result
64M final (14×576)	80,000 (10.5%)	45.08	73.52	2.4777	73.75	training in progress; not a result
64M final (14×576)	100,000 (13.2%)	45.50	74.49	2.4733	74.24	training in progress; not a result
64M final (14×576)	120,000 (15.8%)	45.58	75.15	2.4593	74.53	training in progress; not a result
64M final (14×576)	140,000 (18.4%)	44.70	74.74	2.4501	74.11	training in progress; not a result
64M final (14×576)	160,000 (21.1%)	45.79	73.81	2.4416	74.19	training in progress; not a result
64M final (14×576)	180,000 (23.7%)	44.99	73.72	2.4344	73.90	training in progress; not a result
64M final (14×576)	200,000 (26.3%)	45.50	74.72	2.4302	74.43	training in progress; not a result
64M final (14×576)	220,000 (28.9%)	46.30	74.38	2.4184	74.62	training in progress; not a result
64M final (14×576)	240,000 (31.6%)	46.46	74.42	2.4185	74.69	training in progress; not a result
64M final (14×576)	260,000 (34.2%)	46.30	74.26	2.4077	74.61	training in progress; not a result
64M final (14×576)	280,000 (36.8%)	46.51	74.48	2.4016	74.77	training in progress; not a result
64M final (14×576)	300,000 (39.5%)	46.59	74.41	2.4003	74.78	training in progress; not a result
64M final (14×576)	320,000 (42.1%), first after the resume	46.97	74.56	2.3955	74.97	training in progress; not a result
64M final (14×576)	340,000 (44.7%)	46.80	75.05	2.3987	75.08	training in progress; not a result
64M final (14×576)	360,000 (47.4%)	45.79	75.26	2.3917	74.82	training in progress; not a result
64M final (14×576)	380,000 (50.0%)	46.72	75.14	2.3957	75.09	training in progress; not a result
64M final (14×576)	400,000 (52.6%)	46.97	75.31	2.3930	75.24	training in progress; not a result
64M final (14×576)	420,000 (55.3%)	47.56	75.46	2.3865	75.51	training in progress; not a result
128M final (16×768)	20,000 (2.6%)	43.90	73.91	2.5383	71.34	training in progress; not a result
128M final (16×768)	40,000 (5.3%)	46.68	74.81	2.4612	72.77	training in progress; not a result
128M final (16×768)	60,000 (7.9%)	47.43	75.24	2.4151	73.28	training in progress; not a result
128M final (16×768)	80,000 (10.5%)	46.59	77.57	2.3959	73.83	training in progress; not a result
128M final (16×768)	100,000 (13.2%)	47.81	77.35	2.3813	74.20	training in progress; not a result
128M final (16×768)	120,000 (15.8%)	48.82	77.51	2.3760	74.61	training in progress; not a result
128M final (16×768)	140,000 (18.4%)	49.71	77.58	2.3639	74.96	training in progress; not a result
128M final (16×768)	160,000 (21.1%)	49.62	77.14	2.3523	74.81	training in progress; not a result
128M final (16×768)	180,000 (23.7%)	49.41	77.65	2.3455	74.93	training in progress; not a result
128M final (16×768)	200,000 (26.3%)	48.36	78.65	2.3406	74.92	training in progress; not a result
128M final (16×768)	220,000 (28.9%), first after the resume	49.37	78.53	2.3275	75.26	training in progress; not a result
128M final (16×768)	240,000 (31.6%)	48.91	77.67	2.3287	74.81	training in progress; not a result
128M final (16×768)	260,000 (34.2%)	49.79	78.53	2.3150	75.43	training in progress; not a result
128M final (16×768)	280,000 (36.8%)	50.17	78.74	2.3145	75.63	training in progress; not a result

Reference, same protocol: the v1 models at the end of their runs (step 400,000) and at the same steps as the final runs.

model	step	ARC-Easy	BLiMP	WikiText-2 byte_ppl	eff
64M v1 (v1_muon/)	40,000	41.41	72.61	2.5404	72.02
64M v1 (v1_muon/)	80,000	43.48	75.24	2.4922	73.76
64M v1 (v1_muon/)	160,000	44.99	75.52	2.4436	74.50
64M v1 (v1_muon/)	320,000	47.22	75.89	2.3775	75.57
64M v1 (v1_muon/)	360,000	47.69	76.15	2.3746	75.83
64M v1 (v1_muon/), end of run	400,000	47.94	75.83	2.3718	75.81
128M v1 (v1_128m/)	40,000	45.83	73.21	2.4577	71.95
128M v1 (v1_128m/)	80,000	46.38	75.53	2.4096	73.04
128M v1 (v1_128m/), last checkpoint before its resume	120,000	48.02	75.19	2.3628	73.59
128M v1 (v1_128m/)	160,000	48.27	75.63	2.3471	73.86
128M v1 (v1_128m/)	200,000	49.62	76.78	2.3344	74.74
128M v1 (v1_128m/)	240,000	50.00	76.67	2.3094	74.89
128M v1 (v1_128m/)	280,000	51.14	77.10	2.2950	75.46
128M v1 (v1_128m/), end of run	400,000	51.94	77.26	2.2717	75.84

At step 40,000 the final 64M run was +0.99 eff ahead of v1 at the same step (ARC-Easy +2.02, BLiMP +0.85, byte_ppl −0.0016) and the final 128M run +0.81 eff (ARC-Easy +0.85, BLiMP +1.60, byte_ppl +0.0035). At step 80,000 the 64M runs are level (73.75 vs 73.76: ARC-Easy +1.60, BLiMP −1.72, byte_ppl −0.0145), and the final 128M run is +0.79 eff ahead of its v1 (73.83 vs 73.04: ARC-Easy +0.21, BLiMP +2.04, byte_ppl −0.0137). At step 160,000 the final 64M run is 0.31 eff behind v1 at the same step (74.19 vs 74.50: ARC-Easy +0.80, BLiMP −1.71, byte_ppl −0.0020). At step 120,000 the final 128M run is +1.02 eff ahead of its v1 (74.61 vs 73.59: ARC-Easy +0.80, BLiMP +2.32, byte_ppl +0.0132); for v1 128M this is the last checkpoint before its resume. At step 160,000 the final 128M run is +0.95 eff ahead of its v1 (74.81 vs 73.86: ARC-Easy +1.35, BLiMP +1.51, byte_ppl +0.0052). At step 200,000 it is +0.18 eff ahead (74.92 vs 74.74: ARC-Easy −1.26, BLiMP +1.87, byte_ppl +0.0062). At step 320,000 the final 64M run is 0.60 eff behind v1 at the same step (74.97 vs 75.57: ARC-Easy −0.25, BLiMP −1.33, byte_ppl +0.0180); there v1's learning rate was at about 19% of peak versus 66% for the final run. At step 360,000 it is 1.01 eff behind (74.82 vs 75.83: ARC-Easy −1.90, BLiMP −0.89, byte_ppl +0.0171), with v1's learning rate at about 12% of peak versus 59%. At step 240,000 the final 128M run is 0.08 eff behind its v1 (74.81 vs 74.89: ARC-Easy −1.09, BLiMP +1.00, byte_ppl +0.0193), with v1's learning rate at about 41% of peak versus 80%. At step 400,000, where v1 64M ended, the final 64M run is 0.57 eff behind v1's result (75.24 vs 75.81: ARC-Easy −0.97, BLiMP −0.52, byte_ppl +0.0212); v1 was then at its minimum learning rate (10% of peak), while the final run is at about 52% of peak with 360,000 steps still to go. At step 280,000 the final 128M run is +0.17 eff ahead of its v1 (75.63 vs 75.46: ARC-Easy −0.97, BLiMP +1.64, byte_ppl +0.0195), with v1's learning rate at about 29% of peak versus 73%. Read this with three caveats: each is a single measurement (one ARC-Easy point is about one standard error at n = 2,376); the v1 pool was not cleaned of the test-set overlaps listed under Benchmark contamination check, while the final pool was; and v1 used a shorter cosine schedule (400k steps; at step 160,000 its learning rate was at about 69% of peak versus 91% for the final run), so from roughly 100k steps on the same-step comparison increasingly favours v1.

Learning curves (training in progress; not a result; final 64M to step 240,000, 128M to step 160,000):

Figure 1. eff (canonical board protocol) against tokens seen, for v1 (400k-step cosine) and the final runs (760k-step cosine, in progress). Lines are fits within the measured range only, with no extrapolation; the band is ±2σ (seed and evaluation noise). The v1 128M points up to its resume (40k, 80k and 120k, grey) are not fitted.

Figure 2. (a) Training loss (5k-step centred mean), final runs up to their last evaluated step. Training losses are on each run's own pool; v1 sampled windows with replacement and, on its shorter schedule, had a lower learning rate at the same step; both lower its training loss, so (a) compares shape, not level. From about 3B tokens the final 64M run has a higher training loss than v1; v1 128M has no logged loss before its resume at 120k. (b) Bits per byte on the WikiText-2 test without the 7 near-copies found in the v1 pool (55 articles), the same text for all runs; the remaining articles overlap any pool by at most ~9% of their 13-grams. On these 55 articles the final and v1 runs are within ±0.01 bits per byte at the same step.

Data-attribution experiments (64M, 2026-09-25)

Question: does changing the training data in the late phase of training move the efficiency score? Method: two identical 64M models continue from the same flagship checkpoint (step 320k) with the same recipe, seed and number of steps (80k, to step 400k); they differ only in the data. Both are evaluated at 340k / 360k / 380k / 400k with the board protocol and a paired bootstrap over items. Each checkpoint folder has its own README with the full data specification.

run (folder)	data	ARC-E	BLiMP	wiki byte_ppl	eff @400k	note
flagship (v1_muon/)	ARC-MIX, full run 0→400k	47.94	75.83	2.3718	75.81	reference
control (ctrl_arcmix_resume/)	ARC-MIX	47.69	76.29	2.3717	75.88	re-draws the flagship's first 80k steps of data (the data sampler restarted from its seed on resume)
fork-B (forkB_arcmix_edu/)	ARC-MIX + educational	46.34	77.20	2.3982	75.66	flawed build: took only the head of the ARC-MIX file and a different document separator. Not a test of educational data
A (r3A_arcmix_edu_clean/)	45% ARC-MIX + 55% educational (clean build), 2.55B-token pool	47.31	76.22	2.3762	75.71	clean pair with B
B (r3B_arcmix_qa2x/)	ARC-MIX with Q&A documents ×2, 2.62B-token pool	47.26	75.90	2.3712	75.60	clean pair with A

• A − B: Δeff +0.11 (95% CI −0.30 to +0.51); at the four checkpoints +0.46 / −0.13 / −0.02 / +0.11. No difference on eff. Educational data consistently made WikiText-2 slightly worse (by more than the run-to-run spread); BLiMP was slightly higher but within noise.
• Conclusion: at these data doses and 80k steps of the cosine tail (learning rate 19% → 10% of peak), the effect of the data on eff is below ~0.5. None of the runs beats the flagship beyond noise, so none is submitted to the board. This is an upper bound on the effect at this scale of intervention, not evidence that data does not matter.
• Round 4: an anchor run (ARC-MIX sample of the same pool size, Q&A ×1; r4K_anchor_arcmix_qa1/) reached eff 75.53 at 400k and B with a second seed (r4B_arcmix_qa2x_seed1338/) eff 75.32. Q&A ×2 against the anchor stayed within noise at all four checkpoints; the run-to-run noise between seeds is about 0.3 eff.

Usage

These are raw nanoGPT-lineage checkpoints (plain torch state dicts), not transformers AutoModel weights. The model class and a board-scoring harness are included in this repo:

• train_gpt_ref.py — GPT definition (rebuild the GPT of the tabled shape, load_state_dict, trim logits to vocab 12288).
• glint_parity_eval.py — Glint board-scoring forward (256-token clip, raw log-prob) for BLiMP / ARC-Easy / WikiText-2 (BLiMP loader fixed 2026-09-25, see above).

import torch
from tokenizers import Tokenizer
tok   = Tokenizer.from_file("tokenizer.json")          # BPE-12k, vocab 12288
ckpt  = torch.load("bpe16m_10B/ckpt.pt", map_location="cpu")
state = ckpt.get("model", ckpt)                        # load into the GPT from train_gpt_ref.py

64M checkpoints (v1_muon/, and the data-attribution folders) — Qwen3-style arch, config embedded in the checkpoint. Load with the cfg from ckpt["config"], not the legacy 5-arg nanoGPT path. glint_parity_eval.py's loader is cfg-aware for both lineages.

import torch
from types import SimpleNamespace
from train_gpt_ref import GPT
ck  = torch.load("v1_muon/ckpt_400k.pt", map_location="cpu", weights_only=False)
c   = ck["config"]                        # self-describing: vocab 12288, n_layer 14, n_embd 576, n_head 9, block 1024
cfg = SimpleNamespace(**c)                # Qwen3 flags: rope θ100k / swiglu / qk_norm / value_residual
m   = GPT(c["vocab"], c["n_layer"], c["n_embd"], c["n_head"], c["block"], cfg)
m.load_state_dict(ck["model"], strict=False)   # tied head.weight
m.eval()

Reproducing training. train_gpt_ref.py is a general nanoGPT-style causal transformer whose vocabulary and bin dtype are CLI-parameterized. These checkpoints were trained in BPE-12k mode (--vocab 12288 --dtype uint16), not the script's byte-level defaults:

# 16M @ expanded corpus
python train_gpt_ref.py --data-dir <corpus> \
  --n-layer 6 --n-embd 408 --n-head 6 --block 1024 \
  --batch 64 --steps 244141 --lr 6e-4 --min-lr 6e-5 \
  --vocab 12288 --dtype uint16 --seed 1337
# 32M: --n-embd 576 --n-head 9 (same vocab 12288 / uint16 / BPE tokenizer)

The script's byte-level defaults (--vocab 256 --dtype uint8) and its header comment reflect its origin as a standard-GPT control compared against an experimental BDH (fast-weights) architecture; the leaderboard models here are the standard causal transformer in BPE mode and do not use BDH.

Training data

SlayerLab/minimal-en-corpus-5b — ~5.40B BPE-12k tokens, English, filtered against the benchmark test sets with 13-gram matching when it was built (see Benchmark contamination check below). A broad high-quality mix: FineWeb-Edu, DCLM, StackExchange, open-web-math, FineMath, scientific papers, books/Gutenberg, code, CC-News. An expansion to ~8.3B tokens (added FineWeb-Edu + OpenStax science) feeds later runs.

ARC-MIX (9.42B). The 32M board entry and the 64M runs use ARC-MIX: a reasoning/knowledge-enriched expansion of the base mixture (9,417,035,832 BPE-12288 tokens) with ARC-relevant science/reasoning/QA web content upweighted (gold ~3×, related ~2×) over the base. Same tokenizer and the same build-time 13-gram filtering.

Earlier FineWeb-Edu-dominant corpus (v2). A from-scratch 64M run on a FineWeb-Edu-dominant blend scored lower on ARC than ARC-MIX. That build used a different document separator than the rest of our data and a changed recipe (WSD schedule, z-loss, logit cap), so the comparison is confounded and is not evidence against educational data trained from scratch.

Benchmark contamination check (2026-09-25)

The ARC-MIX corpus was filtered against the benchmark test sets with 13-gram matching when it was built. A later scan of the finished corpus with normalized text found that the filter missed some overlap:

• WikiText-2 test: 7 of 62 test articles appear almost completely in ARC-MIX (66-99% of their 13-grams, as web copies with different formatting from WikiText). The remaining articles have at most ~9% overlap.
• ARC-Easy test: 30 of 2,376 questions have a match, almost all of them short factual sentences (e.g. "All matter is made up of atoms") rather than copies of the question.
• ARC-Challenge test (not part of the board metric): 13 questions matched, 2 of them appear as actual question copies.
• BLiMP was not part of this re-scan (its sentences are synthetic minimal pairs); build-time 13-gram filtering is the only check.
• The FineWeb-Edu blend used in the fork arms (forkB_arcmix_edu/, r3A_arcmix_edu_clean/): 0 WikiText-2 test overlap; 14 ARC test questions matched, mostly the same short fact.

We measured whether this inflates our numbers:

model	WikiText-2 byte_ppl, full test (published)	without the 7 articles	ARC-Easy, full test	without the 30 questions
64M	2.3718	2.3628	47.94	47.87
32M	2.5386	2.5323	44.44	44.63
16M	2.6746	2.6665	40.91	41.01

• On the 7 affected articles our models are about 2% worse than on the rest of the test (ratio 1.021-1.028). GPT-2 small, whose training data deliberately excluded Wikipedia (so it is unlikely to have seen these articles), shows the same ratio (1.021). The articles are simply harder; there is no measurable memorization signal.
• Removing the 30 ARC questions changes accuracy by −0.07 to +0.19 pp, within noise.
• The 16M entry was trained on the expanded corpus, which contains the same base corpus; the same 7 articles are present there, so the 16M row is affected in the same way (its ratio, 1.022, matches the other models and GPT-2).
• Origin: all of the WikiText-2 overlap and both ARC-Challenge question copies are already present in the base pretraining corpus; the ARC-MIX additions contributed only 7 more ARC test matches, mostly short fact sentences.
• Conclusion: the overlap is real and we disclose it, but none of our published numbers is inflated by it beyond noise; on the uncontaminated subsets WikiText-2 is better and ARC-Easy is unchanged within noise.

Scan method: 13-gram and exact short-question matching on normalized text over the full training corpus; per-article coverage for WikiText. Evaluation artifacts available on request.

Evaluation

All metrics use the Glint benchmark protocol (Glint-1.3/benchmark.py), i.e. the board-comparable definitions:

• BLiMP — 67 configs (train split, 67,000 pairs), each sentence clipped to the first 256 tokens, raw sentence log-prob preference (good > bad), no length normalization.
• ARC-Easy — test split, zero-shot, raw accuracy over LL(question + choice) - LL(question).
• WikiText-2 — byte-normalized (byte_ppl / BPB; bytes-per-token 3.8605 for this tokenizer on WikiText-2-raw-v1 test).

A generic lm-eval-harness run scores BLiMP about 2 pp higher on our 64M (77.84 vs 75.83); the numbers here are the board-comparable ones.

Board status. The Glint board still lists the earlier values for our entries (64M: BLiMP 77.84 / byte_ppl 2.016; 32M: BLiMP 73.77; 16M: BLiMP 70.53). The correct values are the ones in this card. With the board's own efficiency formula (eff = mean(BLiMP, ARC-Easy, normalized-WikiText-2) × size multiplier, checked line-for-line against the Space source) they give eff 75.81 (64M), 75.41 (32M), 74.33 (16M).

Key findings

• Tokens drive BLiMP at small size, up to a size-specific ceiling. At 16M, BLiMP rose with tokens (3.2B → 10B) and then flattened near ~70 on the expanded corpus; 32M at 16B tokens moved past that ceiling.
• Capacity and knowledge drive ARC. 16M → 32M at matched tokens lifted ARC-Easy by ~3pp; 64M lifted it further (47.94).
• Bigger is not automatically better on eff. Raw scores keep rising with size, but the efficiency score multiplies by a size bonus that shrinks with parameter count. In our runs 64M and 128M tie at the top (75.81 and 75.84); the 128M model gains about +4 ARC-Easy and +1.4 BLiMP points over the 64M model on the same data, but the smaller size multiplier cancels almost all of it.
• Muon was at least as good as AdamW at 64M in a clean A/B (same data, architecture, seed; numbers pre-date the BLiMP fix).
• Late-phase data changes move eff by less than ~0.5 (see Data-attribution experiments).
• Value residuals are part of the 64M stack. The board's #1 model attributes ~+6 ARC-Easy to them in its own card; we have not isolated this effect ourselves.

Roadmap

• Finish the final 64M and 128M runs (above) and report the last checkpoint with the canonical protocol.
• Not yet tested, each first as a small from-scratch run with a rule fixed in advance: a larger tokenizer vocabulary, question-answer formatted data from the start of training, and distillation from a larger model.

Limitations

Base (not instruction-tuned) research models at 16-128M parameters, English-only. Expect limited factual knowledge and coherence; not intended for production use.

Provenance

Trained on RunPod RTX 5090. Full evaluation artifacts and protocol details are available on request.