HeyGenAI/TAVR

TAVR

Generate Your Talking Avatar from Video Reference

Accepted to SIGGRAPH Asia 2026

This is not a drop-in Wan2.1 checkpoint. It is a TAVR transformer with 18 input channels and its own reference and audio conditioning, and it needs four companion models at inference time (Wan VAE, umT5, wav2vec 2.0, and a person detector plus a pose model) that are not bundled here. A stock diffusers Wan2.1 pipeline will not produce correct results.

Run it through the released inference code: https://github.com/heygen-com/TAVR

Model Description

TAVR replaces single-image avatar references with short video clips, enabling cross-scene generation with significantly better identity preservation. A three-stage training strategy bridges the domain gap between reference and target scenes.

Model Details

Base model	Wan-AI/Wan2.1-T2V-14B
Input	reference video, target still, caption, driving audio
Output	896x480, 25 fps, driving audio muxed in
Sampler	UniPC, 24 steps, flow shift 5.0; text guidance 5.0, audio guidance 1.8
Reference frames	Flexible (default 20)
Precision	bfloat16
Hardware	one Hopper-class CUDA GPU with at least 80 GB of memory

Usage

git clone https://github.com/heygen-com/TAVR
cd TAVR

python3.10 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu126

hf download HeyGenAI/TAVR --local-dir ./ckpt

PYTHONPATH=. python infer.py \
  --sample-dir samples/example1 \
  --dit-ckpt ckpt/tavr_transformer.safetensors \
  --ckpt-dir . \
  --output-dir outputs

flash_attn_3 has no PyPI wheel and is the only attention backend; it and the companion models are set up as described in the repository README.

Evaluation

Qualitative results and comparisons against baselines are on the project page.

On the cross-scene benchmark, TAVR yields the best identity similarity and an overall quality score of 16.42 against 14.13 for the next best method.

Cross-scene benchmark data

benchmark_data.json lists the 158 reference/target pairs of the cross-scene benchmark. The clips come from the public TalkVid dataset. Each sample has:

field	content
reference, target	video_id, video_url, start_time / end_time (seconds), start_frame / end_frame at the source fps, width, height
target_caption	scene caption used as the text prompt

Release Progress

• Model weights
• Inference code
• Cross-scene benchmark data

Citation

@inproceedings{guo2026generate,
     title={Generate Your Talking Avatar from Video Reference},
     author={Guo, Zujin and Ye, Zhenhui and Ren, Yi and Li, Yuanming and Chen, Ce and Hong, Zhibin and Loy, Chen Change},
      booktitle={SIGGRAPH Asia 2026 Conference Papers},
     year={2026}
}

License

Apache License 2.0, inherited from Wan2.1-T2V-14B.

Third-party weights at inference time

_{Running TAVR additionally requires DWPose pose weights (Apache-2.0) and Ultralytics YOLO weights (AGPL-3.0). Neither is redistributed here, and neither is contained in these weights; obtain them from their own channels and observe their terms.}

_{All videos and results shown here are for research demonstration purposes only.}