lyc commited on
Commit
affa705
·
1 Parent(s): 6d398f0

Rename TCR release and refresh model card

Browse files
README.md CHANGED
@@ -11,53 +11,53 @@ tags:
11
 
12
  <div align="center">
13
 
14
- # Separating What from When
15
-
16
- ### Fine-Grained Temporal Control for Joint Audio-Video Generation
17
 
18
  **Yichen Liu**<sup>1</sup> · **Quanwei Zhang**<sup>2</sup> · **Haozhe Wang**<sup>3</sup> · **Donghao Zhou**<sup>4</sup> · **Xiaojie Li** · **Yang Shi**<sup>2</sup> · **Jiaming Liu**<sup>2</sup> · **Ruihua Huang**<sup>2</sup> · **Yingtian Zou**<sup>5</sup> · **Daquan Zhou**<sup>1</sup>
19
 
20
- <sup>1</sup>Peking University · <sup>2</sup>Qwen Business Unit of Alibaba · <sup>3</sup>HKUST · <sup>4</sup>CUHK · <sup>5</sup>Shanghai Jiao Tong University
21
 
22
- [![Project Page](https://img.shields.io/badge/Project-Page-4d86c4)](https://dagroup-pku.github.io/SEPARATING-WHAT-FROM-WHEN.github.io/)
23
- [![GitHub](https://img.shields.io/badge/Code-GitHub-24292f?logo=github)](https://github.com/DAGroup-PKU/SEPARATING-WHAT-FROM-WHEN)
24
- [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97-this%20repo-ffbd2e)](https://huggingface.co/starry0929/Separating-What-From-When)
25
 
26
  </div>
27
 
28
- **Temporal Context Routing (TCR)** adds a timing term to text-attention logits so picture and speech are generated as two streams against one shared clock. This repo hosts the released **LoRA** (`separating-what-from-when.safetensors`, ~2.3 GB) and the **200 held-out test prompts** (`test_prompts_200.json`) used in the paper. Training and inference code is at [DAGroup-PKU/SEPARATING-WHAT-FROM-WHEN](https://github.com/DAGroup-PKU/SEPARATING-WHAT-FROM-WHEN). Demos: [project page](https://dagroup-pku.github.io/SEPARATING-WHAT-FROM-WHEN.github.io/).
 
 
29
 
30
  ## Files
31
 
32
  | File | What it is |
33
  | --- | --- |
34
- | `separating-what-from-when.safetensors` | TCR LoRA, rank 128, trained on LTX-2.3 22B (paper 7k-step recipe) |
35
  | `test_prompts_200.json` | 200 held-out script JSON files with per-shot / per-line `time_range` |
36
 
37
  You still need the **LTX-2.3 22B** checkpoint and the **Gemma 3 12B** text encoder locally. One 80 GB GPU is the working setup.
38
 
39
  ```bash
40
- huggingface-cli download starry0929/Separating-What-From-When \
41
- separating-what-from-when.safetensors test_prompts_200.json \
42
  --local-dir ./weights
43
  ```
44
 
45
- Then follow the [code README](https://github.com/DAGroup-PKU/SEPARATING-WHAT-FROM-WHEN#inference):
46
 
47
  ```bash
48
  bash scripts/infer.sh \
49
  --checkpoint /path/to/ltx-2.3-22b-dev.safetensors \
50
  --text-encoder-path /path/to/gemma-3-12b-it \
51
- --lora-path ./weights/separating-what-from-when.safetensors \
52
  --output outputs/tcr_infer.mp4
53
  ```
54
 
55
  ## Citation
56
 
57
  ```bibtex
58
- @inproceedings{liu2027separating,
59
- title = {Separating What from When: Fine-Grained Temporal Control
60
- for Joint Audio-Video Generation},
61
  author = {Liu, Yichen and Zhang, Quanwei and Wang, Haozhe and
62
  Zhou, Donghao and Li, Xiaojie and Shi, Yang and Liu, Jiaming and
63
  Huang, Ruihua and Zou, Yingtian and Zhou, Daquan},
@@ -68,4 +68,4 @@ bash scripts/infer.sh \
68
 
69
  ## License
70
 
71
- The LoRA is trained on LTX-2.3 and is intended to be used with that backbone under the [LTX-2 Community License](https://github.com/DAGroup-PKU/SEPARATING-WHAT-FROM-WHEN/blob/main/LICENSE).
 
11
 
12
  <div align="center">
13
 
14
+ ## The Missing Temporal Link:<br>Temporal Context Routing for Script-Driven Audio-Video Generation
 
 
15
 
16
  **Yichen Liu**<sup>1</sup> · **Quanwei Zhang**<sup>2</sup> · **Haozhe Wang**<sup>3</sup> · **Donghao Zhou**<sup>4</sup> · **Xiaojie Li** · **Yang Shi**<sup>2</sup> · **Jiaming Liu**<sup>2</sup> · **Ruihua Huang**<sup>2</sup> · **Yingtian Zou**<sup>5</sup> · **Daquan Zhou**<sup>1</sup>
17
 
18
+ <sup>1</sup>Peking University · <sup>2</sup>Qwen Applications · <sup>3</sup>HKUST · <sup>4</sup>CUHK · <sup>5</sup>Shanghai Jiao Tong University
19
 
20
+ [![Project Page](https://img.shields.io/badge/Project-Page-4d86c4)](https://dagroup-pku.github.io/Temporal-Context-Routing.github.io/)
21
+ [![GitHub](https://img.shields.io/badge/Code-GitHub-24292f?logo=github)](https://github.com/DAGroup-PKU/Temporal-Context-Routing)
22
+ [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97-this%20repo-ffbd2e)](https://huggingface.co/starry0929/Temporal-Context-Routing)
23
 
24
  </div>
25
 
26
+ **Temporal Context Routing (TCR)** aligns script-specified shot and dialogue timing with the shared video-audio timeline. Timing bypasses the text encoder and enters cross-attention logits as a routing score, separating *what* to generate from *when* it should appear. Built on LTX-2.3 22B, TCR reduces shot-boundary error from **1.11 s to 0.042 s**—about one frame at 24 fps—and raises dialogue timing accuracy from **28.3% to 84.1%**.
27
+
28
+ This repository hosts the released **LoRA** (`temporal-context-routing.safetensors`, ~2.3 GB) and the **200 held-out test prompts** (`test_prompts_200.json`) used in the paper. Training and inference code is at [DAGroup-PKU/Temporal-Context-Routing](https://github.com/DAGroup-PKU/Temporal-Context-Routing). Demos are available on the [project page](https://dagroup-pku.github.io/Temporal-Context-Routing.github.io/).
29
 
30
  ## Files
31
 
32
  | File | What it is |
33
  | --- | --- |
34
+ | `temporal-context-routing.safetensors` | TCR LoRA, rank 128, trained on LTX-2.3 22B (paper 7k-step recipe) |
35
  | `test_prompts_200.json` | 200 held-out script JSON files with per-shot / per-line `time_range` |
36
 
37
  You still need the **LTX-2.3 22B** checkpoint and the **Gemma 3 12B** text encoder locally. One 80 GB GPU is the working setup.
38
 
39
  ```bash
40
+ huggingface-cli download starry0929/Temporal-Context-Routing \
41
+ temporal-context-routing.safetensors test_prompts_200.json \
42
  --local-dir ./weights
43
  ```
44
 
45
+ Then follow the [code README](https://github.com/DAGroup-PKU/Temporal-Context-Routing#inference):
46
 
47
  ```bash
48
  bash scripts/infer.sh \
49
  --checkpoint /path/to/ltx-2.3-22b-dev.safetensors \
50
  --text-encoder-path /path/to/gemma-3-12b-it \
51
+ --lora-path ./weights/temporal-context-routing.safetensors \
52
  --output outputs/tcr_infer.mp4
53
  ```
54
 
55
  ## Citation
56
 
57
  ```bibtex
58
+ @inproceedings{liu2027missingtemporallink,
59
+ title = {The Missing Temporal Link: Temporal Context Routing
60
+ for Script-Driven Audio-Video Generation},
61
  author = {Liu, Yichen and Zhang, Quanwei and Wang, Haozhe and
62
  Zhou, Donghao and Li, Xiaojie and Shi, Yang and Liu, Jiaming and
63
  Huang, Ruihua and Zou, Yingtian and Zhou, Daquan},
 
68
 
69
  ## License
70
 
71
+ The LoRA is trained on LTX-2.3 and is intended to be used with that backbone under the [LTX-2 Community License](https://github.com/DAGroup-PKU/Temporal-Context-Routing/blob/main/LICENSE).
separating-what-from-when.safetensors → temporal-context-routing.safetensors RENAMED
File without changes