Document step-3000 weights and optimizer shards; extend checksum manifest

README.md

@@ -39,6 +39,8 @@ sentence.
 
 ## Files
 
+**To run the model — 6.0 GB.** This is what `setup.sh` fetches.
+
 | path | size | what |
 |---|---|---|
 | `checkpoints/t14b_b64/milestones/step002250_noema.pt` | 5.3 GB | **the model** — 825 tensors, bf16, no EMA copy |
@@ -47,7 +49,26 @@ sentence.
 | `out/world_p{0,44,60,82}.pt` | 4 × 59 MB | the four cached evaluation worlds — skip base-model generation entirely |
 | `samples/` | 16 MB | reference clips and analysis filmstrips from those worlds |
 
+**To continue the run — a further 28.3 GB.** Not needed for inference; do not
+download these unless you intend to train.
+
+| path | size | what |
+|---|---|---|
+| `checkpoints/t14b_b64/latest.pt` | 10.6 GB | step 3000, **carries its EMA copy**. The comparison arm, and the weights a resume starts from |
+| `checkpoints/t14b_b64/opt_rank00{0..7}.pt` | 8 × 2.2 GB | FSDP-sharded AdamW state at step 3000, one shard per rank |
+
+Nothing here duplicates a public download. The Wan2.1 base model, its VAE and the
+umt5-xxl encoder are **not** mirrored in this repository — `setup.sh` pulls
+[Wan-AI/Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) (17 GB)
+from its own repo, and the 14B teacher is only needed if you are distilling.
+
 The layout mirrors the code repository, so `setup.sh` downloads straight into place.
+To take only part of it:
+
+```bash
+hf download JonathanColetti/LiveWan --local-dir . \
+  --exclude "checkpoints/t14b_b64/latest.pt" "checkpoints/t14b_b64/opt_rank*"
+```
 
 ## Use
 
@@ -92,6 +113,10 @@ Checkpoint quality here is **not monotone** — an earlier run went −0.024 →
 counter-argument for step 3000, which is sharper and has more motion, is in the
 code repo's `docs/HANDOFF.md` §1.
 
+Step 3000 ships here as `checkpoints/t14b_b64/latest.pt` so the comparison is
+yours to make rather than to take on trust. Swap it into the command above and
+watch world 60 — that is where the two checkpoints actually differ.
+
 `_noema` means the EMA copy was stripped to halve the file, 10.6 → 5.3 GB. This
 changes nothing for inference: every evaluation in this project ran with
 `use_ema: false`.
@@ -104,6 +129,42 @@ at 66.5 s/it, zero interventions. Losses do not decrease in this trainer and sho
 not — the critic is retrained every step, so the generator holds position against a
 strengthening opponent.
 
+## Continuing the run
+
+The optimizer shards make extending the run a **resume rather than a restart**.
+Put `latest.pt`, all eight `opt_rank*.pt` and `history.json` in the same run
+directory and:
+
+```bash
+ITERS=4000 ./run_b64_resume.sh
+```
+
+Four things decide whether this works:
+
+- **Exactly 8 ranks.** These are per-rank FSDP shards, not a consolidated state
+  dict. World size 8 is what wrote them and world size 8 is what can read them;
+  fewer ranks means *larger* shards, not smaller, so there is no repacking
+  shortcut. `run_b64_resume.sh` checks for all eight up front and refuses rather
+  than half-starting.
+- **You resume from step 3000, not from the released step 2250.** Optimizer state
+  and the EMA copy exist only for step 3000 — 2250 was archived `_noema` and its
+  EMA is gone. The released checkpoint is the better *model*; `latest.pt` is the
+  only *resumable* one.
+- **`--resume` overwrites the checkpoint it resumed from.** It reads
+  `OUT/latest.pt` and writes that same path at the first save.
+  `run_b64_resume.sh` preserves it as `step003000.pt` first — if you drive the
+  trainer directly instead, copy it aside yourself.
+- **~30 GiB must be free transiently at each save**, because the shards are
+  written as `.tmp` and renamed, so old and new coexist. Prune before the save,
+  not after; a post-save janitor cannot help, since the save is what runs out of
+  space.
+
+The resume path is verified rather than assumed: killing a run at step 12 brought
+AdamW's internal counters back as 16/18 rather than reset, and the real 750→3000
+resume held `loss_gen`, `loss_critic`, `dmd_grad_norm` and `gn_gen` within one
+standard deviation of the 60-point pre-crash reference. A subtly broken restore
+shows up as a shifted mean or changed variance; neither appeared.
+
 ## Verified
 
 - all 825 tensors load, **zero non-finite values**, weight norms 0.018–115.4

checksums.sha256

@@ -1,5 +1,14 @@
-9f7619d2f96f253c9ef18c6c026e961a82c0da7798f098943325719121bcbb17  checkpoints/t14b_b64/milestones/step002250_noema.pt
 4256a6d4cedfc38fdbb4ccf9e21307f592737e844e0e05456ac9b6e8e493e43f  checkpoints/t14b_b64/history.json
+c4fe559eeaff4c9e545ca4c5a9fe7e39c1a51cc49e5e992a4697700ec9dc02a5  checkpoints/t14b_b64/latest.pt
+9f7619d2f96f253c9ef18c6c026e961a82c0da7798f098943325719121bcbb17  checkpoints/t14b_b64/milestones/step002250_noema.pt
+f0f9e7214e108f0cf63a4ea9039113b2438ac7911f1fc5a640fad8f79d2ec2b4  checkpoints/t14b_b64/opt_rank000.pt
+4f6925320fb258359e01eeec107699e3d6742c1141ccc6c1ac35400401721239  checkpoints/t14b_b64/opt_rank001.pt
+ef94e9c70f6cd0227f4a9f9437bc242846ab904760cbabbe02b4e2d0a4a1f024  checkpoints/t14b_b64/opt_rank002.pt
+d6831bfd1c73f45de07213cba5cb8db89cc3ed5b5e6ac11a7cc8244bc0d16c83  checkpoints/t14b_b64/opt_rank003.pt
+7212cb3da6624fc752c3861ff05c53b8d9aad9f515e872e126a49b98067ae80d  checkpoints/t14b_b64/opt_rank004.pt
+13e4ae9ff3479dd8f6937a19a2ce7935fe28fb059661f11541aff520180d2495  checkpoints/t14b_b64/opt_rank005.pt
+936f712c9d14ca762d93f3871c29a9360959cf5fff38ac7f609fbde8f93a5669  checkpoints/t14b_b64/opt_rank006.pt
+0e352a437b3881fd5de834d48008e573d906d1279476635bdb978a8e3e45e992  checkpoints/t14b_b64/opt_rank007.pt
 79a2a8edfcf34999b3928382e3b891cad0dbcbcf999c20433331cefea76ac631  data/prompts.pt
 4a0b0067309a143cee3b853b047041412b6e7556d5f668c97434a76cdd6f4b5e  out/world_p0.pt
 26de659980d9441766fd167aa1966e97e529579bd2552d1ac8721a55987f45f8  out/world_p44.pt