Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+loss_datamix.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,213 @@
+# Rodan-10M
+
+A ~11M-parameter language model trained start to finish on one Apple M2 with MLX. The aim was a tiny model
+that actually holds up for its size, scored on how much it gets per parameter rather than raw leaderboard rank.
+
+| Model | Stage | Purpose |
+|---|---|---|
+| **Rodan-10M-Base** | pretraining | foundation: commonsense + knowledge |
+| Rodan-10M-Chat *(in training)* | instruction fold | chat / instruction following |
+| Rodan-10M-Reasoning *(in training)* | recursive depth + CoT fold | verifiable math + reasoning |
+| Rodan-10M-Latent *(planned)* | latent reasoning | in-head compute, no CoT tokens |
+
+This card covers the base model only. The chat, reasoning, and latent stages are separate models with their
+own repos and cards.
+
+## Architecture
+
+Decoder-only transformer, wide per layer (the proportions take a cue from Gemma-style edge models), 11.46M params.
+
+```
+vocab_size      8192      byte-level BPE
+dim             320
+n_layers        8
+n_heads         8         head_dim 40
+n_kv_heads      1         MQA (8 query heads share 1 KV head)
+ffn_hidden      768       SwiGLU
+max_seq_len     512
+norm            RMSNorm (eps 1e-5)
+position        RoPE (base 200000), applied after QK-norm
+tied_embeddings true
+value_residual  true      mix layer-0 values into later layers
+ple_rank        16        factorized per-layer value-embeddings
+lrm             true      learnable per-row/col weight multipliers (Falcon LRM)
+recurse         1         re-run the shared block stack N times (1 = base; >1 used by the reasoning stage)
+```
+
+The `recurse` knob is a recursive-depth mechanism (Universal-Transformer-style weight sharing, inspired by
+the TRM/HRM "tiny recursive reasoning" line). Setting `recurse=N` runs the same 8 blocks N times over the
+residual stream, so you get the effective depth of `8·N` layers at **zero extra parameters**. The base runs
+`recurse=1` (it's a plain 8-layer model). The reasoning stage warm-starts these weights and trains at
+`recurse=2` (16 effective layers, still 10.41M params), letting the model spend more compute per token on
+hard problems without growing. It is not the full TRM/HRM algorithm (no separate answer/latent states, no
+deep supervision); it's the shared-recursion idea applied to an autoregressive LM.
+
+It was built in two passes: a from-scratch base on 262M tokens, then a warm-start continue on another
+115M tokens that adds LRM, raises the RoPE base from 10k to 200k, and mixes in 21% arithmetic/reasoning data
+(Falcon's reasoning-in-pretraining idea). That second pass is the 11.46M v6 checkpoint.
+
+```mermaid
+flowchart TB
+  ids["token ids"]:::io --> emb["Embedding 8192x320 (tied)"]:::emb
+  emb --> blk["8 x ModernBlock"]:::core
+  blk --> fn["RMSNorm"]:::norm
+  fn --> head["tied head (x @ Wemb^T)"]:::emb
+  head --> out["logits 8192"]:::io
+
+  subgraph ModernBlock["ModernBlock (x8)"]
+    direction TB
+    x(["x"]):::res --> n1["RMSNorm"]:::norm
+    n1 --> qkv["q/k/v projection<br/>MQA: 8 q-heads, 1 kv-head, head_dim 40"]:::attn
+    qkv -->|"q, k"| qk["QK-norm to RoPE"]:::attn
+    qkv -->|"v"| vm["+ value-PLE (per-layer)<br/>+ value-residual (layer-0 v)"]:::attn
+    qk --> sdpa{{"scaled dot-product<br/>attention"}}:::attn
+    vm --> sdpa
+    sdpa --> wo["output projection"]:::attn
+    x --> a1(["+"]):::res
+    wo --> a1
+    a1 --> n2["RMSNorm"]:::norm
+    n2 --> ffn["SwiGLU FFN<br/>320 to 768 to 320"]:::ffn
+    a1 --> a2(["+"]):::res
+    ffn --> a2
+    a2 --> xo(["x out"]):::res
+  end
+
+  classDef io fill:#ffb73d,stroke:#fff,color:#0a0703,font-weight:bold
+  classDef emb fill:#e08a2b,stroke:#ffd98a,color:#0a0703
+  classDef core fill:#c4631a,stroke:#ffd98a,color:#fff
+  classDef attn fill:#1f4e6b,stroke:#5ad1ff,color:#dff4ff
+  classDef ffn fill:#5c3a0c,stroke:#ffb73d,color:#ffd98a
+  classDef norm fill:#231603,stroke:#a86d18,color:#ffd98a
+  classDef res fill:#5ad1ff,stroke:#fff,color:#0a0703,font-weight:bold
+```
+
+Pre-norm residual blocks: `x += Attn(RMSNorm(x))`, then `x += SwiGLU(RMSNorm(x))`. Layer-0's attention
+values feed the value-residual mix in every later layer, and each layer also adds its own low-rank value-PLE.
+
+Why these specific choices at 11M, where every parameter has to earn its place:
+
+- 8k vocab with tied embeddings. Only about 23% of the params sit in the embedding table, versus roughly
+  70% for a 49k-vocab model this size. That frees most of the budget for the layers that do the computing.
+- MQA, because it's the cheapest attention that still works, which leaves params for depth and embeddings.
+- value-residual does most of the heavy lifting. A checkpoint probe shows later layers blending 77-99% of
+  layer-0's values, so it acts as a shared value memory and a gradient highway at once.
+- LRM (learnable row/col multipliers) probed about 20% off identity, so the model is genuinely using it.
+- QK-norm for attention stability, from the nanoGPT-speedrun stack.
+- value-PLE we tried and then removed. The probe found it dead: 0.2% contribution, weight-decayed to near
+  zero. v9 drops it and lands at 10.41M with no loss in quality.
+
+## Training
+
+- Optimizer: Muon on the 2D hidden weights, AdamW on the embeddings, norms, and LRM multipliers, joined
+  through MultiOptimizer, cosine LR, grad-clip 1.0.
+- Framework: MLX on Apple Silicon, with an `mx.compile`d step. About 0.6-0.7 it/s on one fanless M2 MacBook Air.
+- Data: a warm-start chain of short stages, fresh tokens each time so nothing gets re-looped and memorized.
+  Here are the base (v6) and the challenger that followed it (v9):
+
+  | Source | v6 base (mixed5) | v9 (mixed8) | Content |
+  |---|---|---|---|
+  | Cosmopedia v2 | 27% | 31% | synthetic textbooks → commonsense |
+  | dolmino-mix-1124 (pes2o + StackExchange) | 35% | 26% | academic papers + Q&A → knowledge/ARC |
+  | synthetic arithmetic (ArithMark-style) | 21% | 19% | computation → ArithMark |
+  | FineMath-4plus | 10% | 15% | math prose |
+  | science-QA (SciQ/OBQA/QASC/ARC-train) | 6% | 9% | science MC |
+  | **tokens** | ~0.38B | +0.12B fresh | curated, no raw web |
+
+  Two things we found out the hard way. First, adding FineWeb-Edu (45%, then 25%) lost to v6 both times, in
+  a clean monotonic line: raw web hurts at 11M. The model is too small to digest it, and the curated
+  synthetic-plus-academic mix wins instead. Second, the probe that killed value-PLE also confirmed
+  value-residual and LRM are doing real work. So v9 is the pure-curated, PLE-free version at 10.41M: it
+  drops both of the things we'd shown were dead weight and keeps the recipe that worked.
+
+Training-compute efficiency, from the actual runs (perplexity vs cumulative FLOPs, `6·N·tokens`):
+
+![Perplexity vs Training Compute](flops_efficiency.png)
+
+Intelligence per parameter (board avg vs log-params; the shaded region is above the size-fit line):
+
+![Intelligence per parameter](intelligence_per_param.png)
+
+The fit runs over all 54 board models, with a residual σ of 3.07 that matches the board's own. Rodan v6
+sits +0.31σ above the size-fit line, ahead of liodon at +0.14 and well clear of the per-param
+underachievers like GPT-2 (124M, far below). It does this on roughly 1/50th the tokens of the leading
+models, which train on about 25B.
+
+Training loss and data mix, v6 vs v9:
+
+![Training loss and data mix](loss_datamix.png)
+
+v9 starts from v6, drops the dead PLE down to 10.41M, and trains on the pure-curated mix. The result was a
+tie: board avg 35.70 against v6's 35.80, a 0.10 gap that's well inside the noise, at 9% fewer parameters. It
+gave up about 1.7 points of HellaSwag and picked up 2.0 on ArithMark (28.4, the folded arithmetic finally
+showing), and the per-param number came out about even too (~+0.32σ vs v6's +0.31σ). Two conclusions fall
+out of that. PLE really was dead weight, since cutting 1.05M params changed nothing. And ~35.8 looks like a
+real ceiling for an 11M model on this board: raw web sinks it, the leaner pure-curated mix holds it, and
+nothing we tried pushed past it. So v6 stays the packaged base, and the next gains have to come from
+capability stages rather than more base pretraining. Unique tokens stay around 0.5B the whole way, about
+1/50th of what the leaders use.
+
+## Evaluation
+
+Zero-shot through lm-eval-harness, with a custom MLX backend for `loglikelihood`. We use acc_norm for the
+length-sensitive multiple-choice tasks (HellaSwag, ARC, OpenBookQA) and plain acc otherwise.
+
+Zero-shot, limit 1000 examples per task. Board avg = (HellaSwag + (ARC-E + ARC-C)/2 + PIQA + ArithMark) / 4.
+
+| Task | Metric | Score | Random |
+|---|---|---|---|
+| SciQ | acc | 67.5 | 25 |
+| PIQA | acc | 56.0 | 50 |
+| COPA | acc | 55.0 | 50 |
+| ARC-Easy | acc_norm | 35.6 | 25 |
+| HellaSwag | acc_norm | 31.8 | 25 |
+| OpenBookQA | acc_norm | 27.0 | 25 |
+| ArithMark-2 | acc | 26.4 | 25 |
+| ARC-Challenge | acc_norm | 22.4 | 25 |
+| Winogrande | acc | 49.8 | 50 |
+| LogicMark | acc | 44.8 | 25 |
+| BoolQ | acc | 37.6 | ~50 |
+| CommonsenseQA | acc | 20.7 | 20 |
+| **Board avg (÷4)** | | **35.80** | |
+
+For context, it beats the <10M leader on about 1/65th the tokens:
+
+| Model | Params | Tokens | Board avg (÷4) |
+|---|---|---|---|
+| **Rodan-10M-Base (v6)** | 11.46M | ~0.38B | **35.80** |
+| Liodon SLM-10M | 10M | 25B | 35.09 |
+| GPT-S-5M (Axiomic) | 5.2M | 25B | 34.75 |
+
+![v6 benchmarks](v6_v9_metrics.png)
+
+v6 lands around rank 22 of 54 and +0.31σ above the size-fit line, ahead of liodon at +0.14. The v9
+challenger (PLE-free, 10.41M, pure-curated) tied it: 35.70 board avg at 9% fewer params, and about even on
+per-param too (~+0.32σ). v9 confirmed the ~11M ceiling and that PLE was dead weight, but since it didn't
+move the board, v6 stays the base. From here the work moves to the capability stages.
+
+What the model is actually like: it holds up well for 11M on commonsense and science multiple-choice. SciQ
+(67.5) beats GPT-2-124M, and PIQA (56.0), ARC-Easy (35.6), HellaSwag (31.8), and COPA (55.0) are all clearly
+above random. Arithmetic has crept off the random floor (ArithMark 26.4) thanks to the folded-in computation
+data, though it's a modest lift and actually generating arithmetic is still weak. On the harder abstract
+reasoning tasks (Winogrande, CommonsenseQA, ARC-Challenge, OpenBookQA) and on open-ended generation it's near
+chance, partly a capacity ceiling at this size and partly loglikelihood length-bias. It's a solid base for
+discrimination; the deeper reasoning is the job of the later reasoning and latent stages.
+
+## Limitations
+
+- English only, ~11M params. This is a research and teaching base, not something to put in front of users or
+  trust for facts.
+- It's reliable only on the easy commonsense and science multiple-choice where it beats random. On abstract
+  reasoning (Winogrande, CommonsenseQA, ARC-Challenge) and arithmetic it's at chance.
+- No instruction tuning or safety alignment yet. It completes text; it does not follow instructions.
+- Trained on about one epoch of a curated mix, so coverage of rare facts is thin compared to models trained
+  on far more tokens.
+
+## Files
+
+A standard model repo: `model.safetensors` (weights), `tokenizer.json` (8k byte-level BPE), `config.json`.
+Trained on a single Apple M2 with MLX in about six hours.
+
+## License
+
+Weights are open. Data falls under the respective dataset licenses (Cosmopedia, dolmino-mix ODC-By, AllenAI
+QA sets, FineMath).

config.json

@@ -0,0 +1,26 @@
+{
+  "model_type": "rodan-modern",
+  "architecture": "ModernLM",
+  "framework": "mlx",
+  "params": 11460000,
+  "vocab_size": 8192,
+  "dim": 320,
+  "n_layers": 8,
+  "n_heads": 8,
+  "n_kv_heads": 1,
+  "head_dim": 40,
+  "ffn_hidden": 768,
+  "max_len": 512,
+  "rope_base": 200000.0,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-5,
+  "activation": "swiglu",
+  "qk_norm": true,
+  "tied_embeddings": true,
+  "value_residual": true,
+  "ple_rank": 16,
+  "lrm": true,
+  "attention": "mqa",
+  "tokenizer": "byte-level BPE (8k), eot id 0",
+  "notes": "Custom MLX decoder-only transformer. Load with model_opt.ModernLM(ModernConfig(**fields)) + load_weights('model.safetensors'). See README."
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31dbd650d5e639f75a51b29ffdd4f463019741921cb83c96acfa72ade82c829c
+size 45878253