--- license: apache-2.0 base_model: Wan-AI/Wan2.1-T2V-1.3B pipeline_tag: text-to-video library_name: livewan tags: - text-to-video - streaming - real-time - video-generation - distillation - dmd - wan2.1 - wanstreamer --- # LiveWan: streaming text 2 video, 3000 steps > **Unofficial community project.** Not affiliated with, endorsed by, or produced > by Alibaba Group or the Wan-Video team. Built on their Apache-2.0 > [Wan2.1](https://github.com/Wan-Video/Wan2.1) release. A 1.3 B student distilled from a Wan2.1-14B teacher that generates video **continuously** rather than as a fixed clip: 750 ms of 640x368 video per block, generated in 428 ms on one A100 80GB, extended block by block for as long as you let it run. The text conditioning can be changed mid-stream without clearing the K/V cache, so the scene continues instead of cutting. **Code, setup and documentation: [JonathanColetti/LiveWan](https://github.com/JonathanColetti/LiveWan).** This repository holds weights and data only. It is not usable on its own. ## Not audio The model is text-to-video and was never trained to articulate speech. There is no audio, no lip sync and no speech anywhere in this project. Text steers **the scene**. A face that appears to talk is generating plausible mouth motion, not saying your sentence. ## Files **To run the model — 6.0 GB.** This is what `setup.sh` fetches. | path | size | what | |---|---|---| | `checkpoints/t14b_b64/milestones/step002250_noema.pt` | 5.3 GB | **the model** — 825 tensors, bf16, no EMA copy | | `checkpoints/t14b_b64/history.json` | 100 KB | the full training history of the run | | `data/prompts.pt` | 388 MB | umt5-xxl embeddings for 96 prompts, indices 0–95 | | `out/world_p{0,44,60,82}.pt` | 4 × 59 MB | the four cached evaluation worlds — skip base-model generation entirely | | `samples/` | 16 MB | reference clips and analysis filmstrips from those worlds | **To continue the run** Not needed for inference. do not download these unless you intend to train. | path | size | what | |---|---|---| | `checkpoints/t14b_b64/latest.pt` | 10.6 GB | step 3000, **carries its EMA copy**. The comparison arm, and the weights a resume starts from | | `checkpoints/t14b_b64/opt_rank00{0..7}.pt` | 8 × 2.2 GB | FSDP-sharded AdamW state at step 3000, one shard per rank | Nothing here duplicates a public download. The Wan2.1 base model, its VAE and the umt5-xxl encoder are not mirrored in this repository. `setup.sh` pulls [Wan-AI/Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) (17 GB) from its own repo, and the 14B teacher is only needed if you are distilling. The layout mirrors the code repository, so `setup.sh` downloads straight into place. To take only part of it: ```bash hf download JonathanColetti/LiveWan --local-dir . \ --exclude "checkpoints/t14b_b64/latest.pt" "checkpoints/t14b_b64/opt_rank*" ``` ## Use ```bash git clone https://github.com/JonathanColetti/LiveWan && cd LiveWan pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128 ./setup.sh # pulls this repo, checksum-verified python scripts/demo.py \ --weights checkpoints/t14b_b64/milestones/step002250_noema.pt \ --world-cache out/world_p60.pt --prompt-idx 60 \ --latent-norm 1.0 --block 3 --steps 2 --window 6 --units 60 --fps 16 \ --out out/demo_w60 ``` Needs ~10 GiB of VRAM. `scripts/live_demo.py` serves the same model as a steerable browser stream. **`--prompt-idx` must match the world**: `world_p60.pt` goes with `--prompt-idx 60`. Indices run 0–95; anything above fails. ## Training 3000 iterations of SF-DMD distillation from a Wan2.1-T2V-14B teacher into a Wan2.1-T2V-1.3B student, effective batch 64 (8×H200, accum 8, FSDP), **41.6 hours** at 66.5 s/it, zero interventions. Losses do not decrease in this trainer and should not (the critic is retrained every step, so the generator holds position against a strengthening opponent). ## Continuing the run The optimizer shards make extending the run a **resume rather than a restart**. Put `latest.pt`, all eight `opt_rank*.pt` and `history.json` in the same run directory and: ```bash ITERS=4000 ./run_b64_resume.sh ``` Four things decide whether this works: - **Exactly 8 ranks.** These are per-rank FSDP shards, not a consolidated state dict. World size 8 is what wrote them and world size 8 is what can read them; fewer ranks means *larger* shards, not smaller, so there is no repacking shortcut. `run_b64_resume.sh` checks for all eight up front and refuses rather than half-starting. - **You resume from step 3000, not from the released step 2250.** Optimizer state and the EMA copy exist only for step 3000 — 2250 was archived `_noema` and its EMA is gone. The released checkpoint is the better *model*; `latest.pt` is the only *resumable* one. - **`--resume` overwrites the checkpoint it resumed from.** It reads `OUT/latest.pt` and writes that same path at the first save. `run_b64_resume.sh` preserves it as `step003000.pt` first — if you drive the trainer directly instead, copy it aside yourself. - **~30 GiB must be free transiently at each save**, because the shards are written as `.tmp` and renamed, so old and new coexist. Prune before the save, not after; a post-save janitor cannot help, since the save is what runs out of space. The resume path is verified rather than assumed: killing a run at step 12 brought AdamW's internal counters back as 16/18 rather than reset, and the real 750→3000 resume held `loss_gen`, `loss_critic`, `dmd_grad_norm` and `gn_gen` within one standard deviation of the 60-point pre-crash reference. A subtly broken restore shows up as a shifted mean or changed variance; neither appeared. ## Verified - all 825 tensors load, **zero non-finite values**, weight norms 0.018–115.4 - three correctness gates pass with their deliberately broken controls failing, both before and after training - 24 streaming cells across every arm and world met the real-time budget - same box, same seed → **byte-identical** output ## Limits - **Quality holds for roughly a minute.** By 160 s sharpness has fallen to ~73% of the world's. - **Maximum 1024 latent frames** (4.3 minutes at block 3), set by `WanModel.freqs`. - **The pixel check has not been done.** Checkpoint selection rests on proxy metrics, and in this project nine automatic measurements have pointed the wrong way — every one caught by looking at pixels at 1:1. - **World 82 is unexplained**: every trained checkpoint drifts on it while the untrained baseline sits at −0.001. - **The batch-64 premise is unsettled** — confounded with having simply trained longer. ## Licence Apache-2.0. Derived from [Wan2.1](https://github.com/Wan-Video/Wan2.1) (Apache-2.0, Alibaba Group).