File size: 2,094 Bytes
7097693 e04d00b 7097693 e04d00b 7097693 e04d00b 7097693 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | # decoderstack-d12
Training-state checkpoints for the **DecoderStack d12** Colab notebook line
(`stacks/decoder-medium`, single-GPU 40GB-A100 configuration of the nanochat-style
d12: 286,261,730 params, 12 layers x 768, FA3 varlen, Muon + AdamW with per-step
schedule tables, bf16-live + uint16-mantissa fp32 masters).
## Layout
```
checkpoints/<run_name>/state_stepNNNNNN.pt
```
Each `state_step*.pt` is the **entire** training state at that step, written by the
walkthrough notebook's `write_state` and consumed by its `load_state`:
- `params[name]` — per-Param `w` (live weights), `mantissa` (uint16 low bits of the
fp32 master, where applicable), `first_mntm`, `scnd_mntm`
- `step`, `t_step` — schedule position
- `rng` — torch CPU + CUDA generator states
- `batch` — the next unconsumed training micro-batch (`inputs`, `targets`,
`cu_seqlens`), so the walkthrough dissects exactly the batch training would have
seen next
- `config` — the full `StackConfig` (asserted on load)
- `code` — the notebook source that produced the state
Sign convention (since 2026-09-01): the backward pass is seeded with
`logits_grad = (onehot - probs) * softcap' * loss_scale`, so `grad`/`first_mntm`
hold **update directions** (the negated loss gradient) and both optimizers apply
`w + update`. States saved before this date carry `first_mntm` under the old sign
and should not be loaded into the current notebook.
Produced by the `DecoderStack d12 Walkthrough` notebook: Part 1 trains under the
real 1680-step schedule, stops at `cfg.walkthrough_step`, saves + pushes here; the
walkthrough part reloads and spells the last layer's forward/backward and one Muon
step out flat.
## Current states (run `40GB-A100_d12_walkthrough`, 2026-09-01)
| file | where in the run | val bpb |
|---|---|---|
| `state_step000250.pt` | step 250 of 1680 (past all warmups, lr at peak) | 1.077130 |
| `state_step000015.pt` | step 15 (mid lr/scalar/lm-head warmup) | 1.905353 |
To walk through a different state, set `cfg.walkthrough_step` to its step number
and run the notebook's walkthrough part.
|