ChrisMcCormick commited on
Commit
e04d00b
·
verified ·
1 Parent(s): 9d22e28

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +8 -2
README.md CHANGED
@@ -24,16 +24,22 @@ walkthrough notebook's `write_state` and consumed by its `load_state`:
24
  - `config` — the full `StackConfig` (asserted on load)
25
  - `code` — the notebook source that produced the state
26
 
 
 
 
 
 
 
27
  Produced by the `DecoderStack d12 Walkthrough` notebook: Part 1 trains under the
28
  real 1680-step schedule, stops at `cfg.walkthrough_step`, saves + pushes here; the
29
  walkthrough part reloads and spells the last layer's forward/backward and one Muon
30
  step out flat.
31
 
32
- ## Current states (run `40GB-A100_d12_walkthrough`, 2026-08-31)
33
 
34
  | file | where in the run | val bpb |
35
  |---|---|---|
36
- | `state_step000250.pt` | step 250 of 1680 (past all warmups, lr at peak) | 1.076657 |
37
  | `state_step000015.pt` | step 15 (mid lr/scalar/lm-head warmup) | 1.905353 |
38
 
39
  To walk through a different state, set `cfg.walkthrough_step` to its step number
 
24
  - `config` — the full `StackConfig` (asserted on load)
25
  - `code` — the notebook source that produced the state
26
 
27
+ Sign convention (since 2026-09-01): the backward pass is seeded with
28
+ `logits_grad = (onehot - probs) * softcap' * loss_scale`, so `grad`/`first_mntm`
29
+ hold **update directions** (the negated loss gradient) and both optimizers apply
30
+ `w + update`. States saved before this date carry `first_mntm` under the old sign
31
+ and should not be loaded into the current notebook.
32
+
33
  Produced by the `DecoderStack d12 Walkthrough` notebook: Part 1 trains under the
34
  real 1680-step schedule, stops at `cfg.walkthrough_step`, saves + pushes here; the
35
  walkthrough part reloads and spells the last layer's forward/backward and one Muon
36
  step out flat.
37
 
38
+ ## Current states (run `40GB-A100_d12_walkthrough`, 2026-09-01)
39
 
40
  | file | where in the run | val bpb |
41
  |---|---|---|
42
+ | `state_step000250.pt` | step 250 of 1680 (past all warmups, lr at peak) | 1.077130 |
43
  | `state_step000015.pt` | step 15 (mid lr/scalar/lm-head warmup) | 1.905353 |
44
 
45
  To walk through a different state, set `cfg.walkthrough_step` to its step number