Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -24,16 +24,22 @@ walkthrough notebook's `write_state` and consumed by its `load_state`:
|
|
| 24 |
- `config` — the full `StackConfig` (asserted on load)
|
| 25 |
- `code` — the notebook source that produced the state
|
| 26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
Produced by the `DecoderStack d12 Walkthrough` notebook: Part 1 trains under the
|
| 28 |
real 1680-step schedule, stops at `cfg.walkthrough_step`, saves + pushes here; the
|
| 29 |
walkthrough part reloads and spells the last layer's forward/backward and one Muon
|
| 30 |
step out flat.
|
| 31 |
|
| 32 |
-
## Current states (run `40GB-A100_d12_walkthrough`, 2026-
|
| 33 |
|
| 34 |
| file | where in the run | val bpb |
|
| 35 |
|---|---|---|
|
| 36 |
-
| `state_step000250.pt` | step 250 of 1680 (past all warmups, lr at peak) | 1.
|
| 37 |
| `state_step000015.pt` | step 15 (mid lr/scalar/lm-head warmup) | 1.905353 |
|
| 38 |
|
| 39 |
To walk through a different state, set `cfg.walkthrough_step` to its step number
|
|
|
|
| 24 |
- `config` — the full `StackConfig` (asserted on load)
|
| 25 |
- `code` — the notebook source that produced the state
|
| 26 |
|
| 27 |
+
Sign convention (since 2026-09-01): the backward pass is seeded with
|
| 28 |
+
`logits_grad = (onehot - probs) * softcap' * loss_scale`, so `grad`/`first_mntm`
|
| 29 |
+
hold **update directions** (the negated loss gradient) and both optimizers apply
|
| 30 |
+
`w + update`. States saved before this date carry `first_mntm` under the old sign
|
| 31 |
+
and should not be loaded into the current notebook.
|
| 32 |
+
|
| 33 |
Produced by the `DecoderStack d12 Walkthrough` notebook: Part 1 trains under the
|
| 34 |
real 1680-step schedule, stops at `cfg.walkthrough_step`, saves + pushes here; the
|
| 35 |
walkthrough part reloads and spells the last layer's forward/backward and one Muon
|
| 36 |
step out flat.
|
| 37 |
|
| 38 |
+
## Current states (run `40GB-A100_d12_walkthrough`, 2026-09-01)
|
| 39 |
|
| 40 |
| file | where in the run | val bpb |
|
| 41 |
|---|---|---|
|
| 42 |
+
| `state_step000250.pt` | step 250 of 1680 (past all warmups, lr at peak) | 1.077130 |
|
| 43 |
| `state_step000015.pt` | step 15 (mid lr/scalar/lm-head warmup) | 1.905353 |
|
| 44 |
|
| 45 |
To walk through a different state, set `cfg.walkthrough_step` to its step number
|