| # decoderstack-d12 |
|
|
| Training-state checkpoints for the **DecoderStack d12** Colab notebook line |
| (`stacks/decoder-medium`, single-GPU 40GB-A100 configuration of the nanochat-style |
| d12: 286,261,730 params, 12 layers x 768, FA3 varlen, Muon + AdamW with per-step |
| schedule tables, bf16-live + uint16-mantissa fp32 masters). |
|
|
| ## Layout |
|
|
| ``` |
| checkpoints/<run_name>/state_stepNNNNNN.pt |
| ``` |
|
|
| Each `state_step*.pt` is the **entire** training state at that step, written by the |
| walkthrough notebook's `write_state` and consumed by its `load_state`: |
|
|
| - `params[name]` — per-Param `w` (live weights), `mantissa` (uint16 low bits of the |
| fp32 master, where applicable), `first_mntm`, `scnd_mntm` |
| - `step`, `t_step` — schedule position |
| - `rng` — torch CPU + CUDA generator states |
| - `batch` — the next unconsumed training micro-batch (`inputs`, `targets`, |
| `cu_seqlens`), so the walkthrough dissects exactly the batch training would have |
| seen next |
| - `config` — the full `StackConfig` (asserted on load) |
| - `code` — the notebook source that produced the state |
|
|
| Sign convention (since 2026-09-01): the backward pass is seeded with |
| `logits_grad = (onehot - probs) * softcap' * loss_scale`, so `grad`/`first_mntm` |
| hold **update directions** (the negated loss gradient) and both optimizers apply |
| `w + update`. States saved before this date carry `first_mntm` under the old sign |
| and should not be loaded into the current notebook. |
|
|
| Produced by the `DecoderStack d12 Walkthrough` notebook: Part 1 trains under the |
| real 1680-step schedule, stops at `cfg.walkthrough_step`, saves + pushes here; the |
| walkthrough part reloads and spells the last layer's forward/backward and one Muon |
| step out flat. |
|
|
| ## Current states (run `40GB-A100_d12_walkthrough`, 2026-09-01) |
|
|
| | file | where in the run | val bpb | |
| |---|---|---| |
| | `state_step000250.pt` | step 250 of 1680 (past all warmups, lr at peak) | 1.077130 | |
| | `state_step000015.pt` | step 15 (mid lr/scalar/lm-head warmup) | 1.905353 | |
|
|
| To walk through a different state, set `cfg.walkthrough_step` to its step number |
| and run the notebook's walkthrough part. |
|
|
