starry0929 commited on
Commit
c3cf3d8
verified
1 Parent(s): d6cfdac

Add project-style model card with code and page links

Browse files
Files changed (1) hide show
  1. README.md +69 -1
README.md CHANGED
@@ -1,3 +1,71 @@
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: other
3
+ license_name: ltx-2-community
4
+ tags:
5
+ - video-generation
6
+ - audio
7
+ - lora
8
+ - temporal-control
9
+ - text-to-video
10
  ---
11
+
12
+ <div align="center">
13
+
14
+ # Separating What from When
15
+
16
+ ### Fine-Grained Temporal Control for Joint Audio-Video Generation
17
+
18
+ **Yichen Liu**<sup>1</sup> 路 **Quanwei Zhang**<sup>2</sup> 路 **Haozhe Wang**<sup>3</sup> 路 **Donghao Zhou**<sup>4</sup> 路 **Yang Shi**<sup>2</sup> 路 **Jiaming Liu**<sup>2</sup> 路 **Ruihua Huang**<sup>2</sup> 路 **Yingtian Zou**<sup>5</sup> 路 **Daquan Zhou**<sup>1</sup>
19
+
20
+ <sup>1</sup>Peking University 路 <sup>2</sup>Qwen Business Unit of Alibaba 路 <sup>3</sup>HKUST 路 <sup>4</sup>CUHK 路 <sup>5</sup>Shanghai Jiao Tong University
21
+
22
+ [![Project Page](https://img.shields.io/badge/Project-Page-4d86c4)](https://dagroup-pku.github.io/SEPARATING-WHAT-FROM-WHEN/)
23
+ [![GitHub](https://img.shields.io/badge/Code-TCR-24292f?logo=github)](https://github.com/DAGroup-PKU/TCR)
24
+ [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97-this%20repo-ffbd2e)](https://huggingface.co/starry0929/Separating-What-From-When)
25
+
26
+ </div>
27
+
28
+ **Temporal Context Routing (TCR)** adds a timing term to text-attention logits so picture and speech are generated as two streams against one shared clock. This repo hosts the released **LoRA** (`separating-what-from-when.safetensors`, ~2.3 GB) and the **200 held-out test prompts** (`test_prompts_200.json`) used in the paper. Training and inference code is at [DAGroup-PKU/TCR](https://github.com/DAGroup-PKU/TCR). Demos: [project page](https://dagroup-pku.github.io/SEPARATING-WHAT-FROM-WHEN/).
29
+
30
+ ## Files
31
+
32
+ | File | What it is |
33
+ | --- | --- |
34
+ | `separating-what-from-when.safetensors` | TCR LoRA, rank 128, trained on LTX-2.3 22B (paper 7k-step recipe) |
35
+ | `test_prompts_200.json` | 200 held-out MTSS scripts with per-shot / per-line `time_range` |
36
+
37
+ You still need the **LTX-2.3 22B** checkpoint and the **Gemma 3 12B** text encoder locally. One 80 GB GPU is the working setup.
38
+
39
+ ```bash
40
+ huggingface-cli download starry0929/Separating-What-From-When \
41
+ separating-what-from-when.safetensors test_prompts_200.json \
42
+ --local-dir ./weights
43
+ ```
44
+
45
+ Then follow the [TCR README](https://github.com/DAGroup-PKU/TCR#inference):
46
+
47
+ ```bash
48
+ bash scripts/infer.sh \
49
+ --checkpoint /path/to/ltx-2.3-22b-dev.safetensors \
50
+ --text-encoder-path /path/to/gemma-3-12b-it \
51
+ --lora-path ./weights/separating-what-from-when.safetensors \
52
+ --output outputs/tcr_infer.mp4
53
+ ```
54
+
55
+ ## Citation
56
+
57
+ ```bibtex
58
+ @inproceedings{liu2027separating,
59
+ title = {Separating What from When: Fine-Grained Temporal Control
60
+ for Joint Audio-Video Generation},
61
+ author = {Liu, Yichen and Zhang, Quanwei and Wang, Haozhe and
62
+ Zhou, Donghao and Shi, Yang and Liu, Jiaming and
63
+ Huang, Ruihua and Zou, Yingtian and Zhou, Daquan},
64
+ booktitle = {Under review},
65
+ year = {2027}
66
+ }
67
+ ```
68
+
69
+ ## License
70
+
71
+ The LoRA is trained on LTX-2.3 and is intended to be used with that backbone under the [LTX-2 Community License](https://github.com/DAGroup-PKU/TCR/blob/main/LICENSE).