Add pipeline tag and improve model card

This PR adds the `text-to-video` pipeline tag to the model metadata to improve discoverability on the Hugging Face Hub. Additionally, the model card has been updated to include detailed information about the model's supported tasks (generation and editing) and providing clear inference examples sourced from the official repository.

README.md

@@ -1,14 +1,15 @@
 ---
+base_model:
+- Qwen/Qwen3-VL-8B-Instruct
+- Wan-AI/Wan2.2-TI2V-5B
 language:
-  - en
+- en
 tags:
-  - video-generation
-  - video-editing
-  - multi-modal
-  - diffusion
-base_model:
-  - Qwen/Qwen3-VL-8B-Instruct
-  - Wan-AI/Wan2.2-TI2V-5B
+- video-generation
+- video-editing
+- multi-modal
+- diffusion
+pipeline_tag: text-to-video
 ---
 
 <p align="center">
@@ -25,6 +26,8 @@ base_model:
   <a href="https://msalab-pku.github.io/projects/LoomVideo/index.html" target="_blank"><img src="https://img.shields.io/badge/Project%20Page-333399.svg?logo=homepage" height="22px"></a>
 </p>
 
+This repository contains the weights for **LoomVideo**, a compact 5B-parameter unified architecture for both video generation and editing. For more details, see the paper: [LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing](https://arxiv.org/abs/2606.06042).
+
 # 🔥 News
 
 - [2026-06-05] We release LoomVideo [paper](https://arxiv.org/abs/2606.06042) on Arxiv!
@@ -33,20 +36,17 @@ base_model:
 
 # 📌 TL;DR
 
-**The Problem:** Existing unified video generation & editing models are massive (13B+) and rely on token concatenation for source conditioning — doubling sequence length and quadrupling attention cost.
+LoomVideo is a compact **5B-parameter** unified architecture built on MLLM + DiT that introduces three key designs:
+- **Deepstack Injection** — extracts features from every MLLM layer and injects them into corresponding DiT layers via cross-attention.
+- **Scale-and-Add Conditioning** — a zero-overhead approach for video editing that eliminates the need for token concatenation.
+- **Negative Temporal RoPE** — seamlessly integrates multiple reference images without architectural modification.
 
-**The Method:** We present **LoomVideo**, a compact **5B-parameter** unified architecture built on MLLM + DiT that introduces three key designs:
-- **Deepstack Injection** — extracts features from every MLLM layer and injects them into corresponding DiT layers via cross-attention, enabling rich multi-granular semantic guidance.
-- **Scale-and-Add Conditioning** — a zero-overhead approach that scales the clean source latent by the current timestep and directly adds it to the noised target, completely bypassing token concatenation.
-- **Negative Temporal RoPE** — assigns negative temporal indices to reference images, seamlessly integrating multi-image conditions without architectural modification.
-
-**The Result:** Our 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, with at least **5.41×** inference speedup over models of similar capabilities — demonstrating that efficiency and quality can coexist.
+Our 5B model achieves state-of-the-art performance across benchmarks, with at least **5.41×** inference speedup over models of similar capabilities.
 
 <p align="center">
   <img src="assets/architecture.png" width="90%">
 </p>
 
-
 # 🎯 Supported Tasks
 
 LoomVideo supports **four** unified video generation and editing tasks within a single model:
@@ -60,54 +60,24 @@ LoomVideo supports **four** unified video generation and editing tasks within a
 
 # 🔧 Preparation
 
-## Step 1: Clone the Repository
+### 1. Clone the Repository
 
 ```bash
 git clone https://github.com/MSALab-PKU/LoomVideo
 cd LoomVideo
 ```
 
-## Step 2: Install Dependencies
-
-We recommend using [uv](https://github.com/astral-sh/uv) for a fast and fully reproducible environment setup.
+### 2. Install Dependencies
 
 ```bash
 uv sync
 source .venv/bin/activate
-
-# (Optional) Include evaluation dependencies
-uv sync --extra eval
-```
-
-Additionally, install [Flash Attention](https://github.com/Dao-AILab/flash-attention) for faster inference and reduced GPU memory consumption. (for reference, our environment uses v2.7.4)
-
-## Step 3: Download Model Weights
-
-Download the pretrained LoomVideo checkpoint from [Hugging Face](https://huggingface.co/MSALab/LoomVideo) and place it under `checkpoints/LoomVideo/`:
-
-```
-checkpoints/LoomVideo/
-└── gen_model.pth
+pip install flash-attn --no-build-isolation
 ```
 
-We provide a helper script to download the weights automatically:
-
-'''bash
-python hf_download.py
-'''
-
-You can also specify a custom path via the `--ckpt_path` argument at inference time.
-
-> 💡 Stage 3 model weights are now available. Higher-performance post-trained weights will be released as soon as possible!
-
 # 🎬 Inference
-LoomVideo provides a unified inference script that supports **four generation tasks** through a single entry point. Each task is selected via the `--task` flag.
 
-### 1. Text-to-Video / Text-to-Image (`t2v`)
-
-Generate a video from a text description. Default resolution is **480×832** at **81 frames**. When `--num_frames` is set to `1`, the pipeline automatically switches to **image generation** mode and saves the output as a `.jpg` file.
-
-**Required:** `--prompt`
+LoomVideo provides a unified inference script. Below is an example for **Text-to-Video** generation. For other tasks (editing, reference-guided editing), please refer to the [GitHub README](https://github.com/MSALab-PKU/LoomVideo).
 
 ```bash
 NUM_GPUS=1
@@ -117,409 +87,17 @@ accelerate launch --num_processes=${NUM_GPUS} \
     --config_path configs/inference/generation.yaml \
     --ckpt_path checkpoints/LoomVideo \
     --task t2v \
-    --prompt "Your prompt here" \
+    --prompt "Vampire makeup face of beautiful girl, red contact lenses." \
     --height 480 \
     --width 832 \
     --num_frames 97 \
     --num_inference_steps 50 \
     --seed 0 \
-    --output_path outputs/t2v.mp4
-```
-
-### 2. Instruction Editing (`edit`)
-
-Edit an existing image or video based on a text instruction. The source can be either an image file (`.jpg`, `.png`, etc.) or a video file (`.mp4`). Resolution and frame count are automatically inferred from the source when not specified.
-
-**Required:** `--prompt` `--source_video_path`
-
-```bash
-NUM_GPUS=1
-
-accelerate launch --num_processes=${NUM_GPUS} \
-    scripts/inference/generate.py \
-    --config_path configs/inference/generation.yaml \
-    --ckpt_path checkpoints/LoomVideo \
-    --task edit \
-    --prompt "Your editing instruction here" \
-    --source_video_path /path/to/source_video.mp4 \
-    --num_inference_steps 50 \
-    --seed 0 \
-    --output_path outputs/edit.mp4
+    --output_path outputs/t2v_demo.mp4
 ```
 
-### 3. Instruction-Image Editing (`ref_edit`)
-
-Edit a source video with guidance from one or more reference images along with a text instruction.
-
-**Required:** `--prompt` `--source_video_path` `--ref_image_paths`
-
-```bash
-NUM_GPUS=1
-
-accelerate launch --num_processes=${NUM_GPUS} \
-    scripts/inference/generate.py \
-    --config_path configs/inference/generation.yaml \
-    --ckpt_path checkpoints/LoomVideo \
-    --task ref_edit \
-    --prompt "Your editing instruction" \
-    --source_video_path /path/to/source_video.mp4 \
-    --ref_image_paths /path/to/ref1.jpg /path/to/ref2.jpg \
-    --num_inference_steps 50 \
-    --seed 0 \
-    --output_path outputs/ref_edit.mp4
-```
-
-### 4. Multi-Image-to-Video (`mi2v`)
-
-Generate a video conditioned on multiple reference images and a text prompt. We recommend using `@Image N` in the prompt to reference specific input images.
-
-**Required:** `--prompt` `--ref_image_paths`
-
-```bash
-NUM_GPUS=1
-
-accelerate launch --num_processes=${NUM_GPUS} \
-    scripts/inference/generate.py \
-    --config_path configs/inference/generation.yaml \
-    --ckpt_path checkpoints/LoomVideo \
-    --task mi2v \
-    --prompt "Your prompt here" \
-    --ref_image_paths /path/to/img1.jpg /path/to/img2.jpg /path/to/img3.jpg \
-    --num_frames 97 \
-    --num_inference_steps 50 \
-    --seed 0 \
-    --output_path outputs/mi2v.mp4
-```
-
-
-## Additional Arguments
-
-The following arguments can be appended to any task command for further customization:
-
-### Generation Control
-
-<table>
-  <thead>
-    <tr><th>Argument</th><th>Type</th><th>Default</th><th>Description</th></tr>
-  </thead>
-  <tbody>
-    <tr><td nowrap><code>--num_inference_steps</code></td><td>int</td><td><code>50</code></td><td>Number of denoising steps.</td></tr>
-    <tr><td nowrap><code>--guidance_scale</code></td><td>float</td><td><code>5.0</code> / <code>2.5</code></td><td>Text CFG scale. <code>5.0</code> for t2v/mi2v, <code>2.5</code> for edit/ref_edit.</td></tr>
-    <tr><td nowrap><code>--guidance_scale_visual</code></td><td>float</td><td><code>1.5</code></td><td>Visual CFG scale for source/reference conditioning.</td></tr>
-    <tr><td nowrap><code>--negative_prompt</code></td><td>str</td><td><em>(from config)</em></td><td>Negative prompt for quality improvement.</td></tr>
-    <tr><td nowrap><code>--seed</code></td><td>int</td><td><code>0</code></td><td>Random seed. Set to <code>-1</code> for random generation.</td></tr>
-  </tbody>
-</table>
-
-### Resolution & Frames
-
-<table>
-  <thead>
-    <tr><th>Argument</th><th>Type</th><th>Default</th><th>Description</th></tr>
-  </thead>
-  <tbody>
-    <tr><td nowrap><code>--height</code></td><td>int</td><td><em>auto</em></td><td>Output height. <code>480</code> for t2v; inferred from source for edit.</td></tr>
-    <tr><td nowrap><code>--width</code></td><td>int</td><td><em>auto</em></td><td>Output width. <code>832</code> for t2v; inferred from source for edit.</td></tr>
-    <tr><td nowrap><code>--num_frames</code></td><td>int</td><td><em>auto</em></td><td>Output frames. <code>81</code> for t2v/mi2v; inferred for edit.</td></tr>
-    <tr><td nowrap><code>--fps</code></td><td>int</td><td><code>24</code></td><td>Output video FPS.</td></tr>
-  </tbody>
-</table>
-
-
-# 📦 Data Preparation
-
-Since our training relies heavily on proprietary datasets, we are unable to release the original data directly. However, we provide a **flexible data organization framework** that makes it easy to plug in your own data or publicly available datasets.
-
-## Open-Source Datasets
-
-Below are the open-source datasets used in our training. You can download them or substitute with your own data:
-
-| Category | Dataset |
-|---|---|
-| Video Generation | [Koala-36M](https://huggingface.co/datasets/Koala-36M/Koala-36M-v1), [OpenVid-1M](https://huggingface.co/datasets/nkp37/OpenVid-1M) |
-| Image Editing | [CrispEdit-2M](https://huggingface.co/datasets/WeiChow/CrispEdit-2M), [OmniGen-2-Edit](https://huggingface.co/OmniGen2), [GPT-Image-Edit-1.5M](https://huggingface.co/datasets/UCSC-VLAA/GPT-Image-Edit-1.5M), [NHR-Edit](https://huggingface.co/datasets/iitolstykh/NHR-Edit), [Pico-Banana](https://github.com/apple/pico-banana-400k), [ShareGPT-4o-Image](https://huggingface.co/datasets/FreedomIntelligence/ShareGPT-4o-Image) |
-| Video Editing | [KIWI-Edit](https://huggingface.co/datasets/linyq/kiwi_edit_training_data) |
-| Video Ref Editing / MI2V | [RefVIE](https://huggingface.co/datasets/linyq/kiwi_edit_training_data), [Phantom-Data](https://huggingface.co/datasets/ZhuoweiChen/Phantom-data-Koala36M) |
-
-## Organize Data as Single JSON Files
-
-Each data sample should be stored as an **individual JSON file**, placed in a single directory (e.g., `single_jsons/`), and named sequentially starting from `0.json`:
-
-```
-your_dataset/
-└── single_jsons/
-    ├── 0.json
-    ├── 1.json
-    ├── 2.json
-    ├── ...
-```
-
-## JSON Format for Each Task
-
-Each task type expects a specific set of keys in its JSON file. Below are the templates — fill in according to your data:
-
-**Text-to-Video** (`process_t2v_data`):
-```json
-{
-    "text": "A caption describing the video content.",
-    "path": "relative/path/to/video.mp4"
-}
-```
-
-**Text-to-Image** (`process_t2i_data`):
-```json
-{
-    "caption": "A caption describing the image content.",
-    "image_path": "relative/path/to/image.jpg"
-}
-```
-
-**Video Editing** (`process_video_edit_data`):
-```json
-{
-    "source_video_path": "relative/path/to/source_video.mp4",
-    "instruction": "The editing instruction.",
-    "target_video_path": "relative/path/to/target_video.mp4"
-}
-```
-
-**Image Editing** (`process_image_edit_data`):
-```json
-{
-    "source_image_path": "relative/path/to/source_image.jpg",
-    "instruction": "The editing instruction.",
-    "target_image_path": "relative/path/to/target_image.jpg"
-}
-```
-
-**Multi-Image-to-Video** (`process_t2v_data_withref`):
-```json
-{
-    "instruction": "A prompt describing the video to generate with reference images.",
-    "reference_image_paths": [
-        "relative/path/to/ref1.jpg",
-        "relative/path/to/ref2.jpg"
-    ],
-    "target_video_path": "relative/path/to/target_video.mp4"
-}
-```
-
-**Reference-Guided Video Editing** (`process_video_edit_data_withref`):
-```json
-{
-    "source_video_path": "relative/path/to/source_video.mp4",
-    "reference_image_paths": [
-        "relative/path/to/ref1.jpg"
-    ],
-    "instruction": "The editing instruction with reference guidance.",
-    "target_video_path": "relative/path/to/target_video.mp4"
-}
-```
-
-> 💡 All paths in JSON files are **relative** to the `data_root` specified in the dataset config.
-
-## Custom Process Functions (Optional)
-
-You may also organize your JSON files in any format you prefer, as long as you implement a corresponding `process_*` function. We provide several reference implementations in `src/dataset/processors.py`. Each process function takes `(dataset_info, data_info)` and returns a list of segments describing the data flow. See the existing functions for examples.
-
-## Dataset Config
-
-Create a YAML config file to register your datasets. See `configs/dataset/train_demo.yaml` as a reference. The config is organized into `train`, `val`, and `eval` sections, each containing dataset entries with the following arguments:
-
-| Argument | Description |
-|---|---|
-| `task_weight` | Controls the sampling probability of this task group relative to others during training. |
-| `process_func_name` | Name of the processing function in `src/dataset/processors.py` that parses each JSON sample. |
-| `data_root` | Base directory for resolving relative paths in JSON files. |
-| `data_json_dir` | Directory containing the JSON files (`0.json`, `1.json`, ...). |
-| `num_samples` | Total number of samples in the directory. |
-| `sample_weight` | Sampling weight of this dataset within its task group. |
-
-
-# 🏋️ Training
-
-## Training Config
-
-The training behavior is fully controlled by a YAML config file (e.g., `configs/train/stage3.yaml`).
-
-**Key arguments:**
-
-| Argument | Description |
-|---|---|
-| `log_dir` | Directory for saving logs, checkpoints, and generated samples. |
-| `dataset_config_path` | Path to the dataset config YAML file. |
-| `train_steps` | Total number of training iterations. |
-| `checkpointing_interval` | Save a checkpoint every N steps. |
-| `validation_interval` | Run validation every N steps. |
-| `evaluation_interval` | Run evaluation benchmarks every N steps. |
-
-**Model settings:**
-
-| Argument | Description |
-|---|---|
-| `model.trainable_modules.gen_model` | Which modules to train. `"all"` trains the full generation model. |
-| `model.gradient_checkpointing` | Enable gradient checkpointing to reduce GPU memory usage. |
-| `model.und.pretrained_model_path` | Path to the pretrained understanding backbone. |
-| `model.gen.pretrained_model_path` | Path to the pretrained generation backbone. |
-| `model.pretrained_ckpt_path` | *(Optional)* Load weights from a previous training stage for continued training. |
-
-**Data settings:**
-
-| Argument | Description |
-|---|---|
-| `data.train.resolution_buckets` | List of resolution buckets for dynamic batching. |
-| `data.train.num_frames` | Number of frames per training sample. |
-| `data.train.fps` | Video FPS for frame sampling. |
-| `data.train.all_dropout_rate` | Probability of dropping all conditions (for unconditional training). |
-| `data.train.text_dropout_rate` | Probability of dropping text condition (for classifier-free guidance). |
-
-## Launch Training
-
-Once the data and configs are ready, you can simply start training with:
-
-```bash
-NUM_GPUS=8
-
-accelerate launch --num_processes=${NUM_GPUS} \
-    -m scripts.train.train \
-    --config_path path/to/your/config.yaml
-```
-
-> 💡 All training outputs — including checkpoints, EMA weights, logs, and generated samples — are saved under the `log_dir` directory specified in the config.
-
-
-# 📊 Evaluation
-
-## Environment Setup
-
-### Step 1: Prepare Benchmark Data
-
-We evaluate on the following benchmarks. Download each dataset and organize it into the same **single JSON** format used for training data (see [Data Preparation](#-data-preparation)):
-
-| Benchmark | Category | Samples |
-|---|---|---|
-| [GenEval](https://github.com/djghosh13/geneval) | Image Generation | 553 |
-| [ImgEdit-Bench](https://github.com/pku-yuangroup/imgedit) | Image Editing | 737 |
-| [VBench](https://github.com/Vchitect/VBench) | Video Generation | 165 |
-| [OpenVE-Bench](https://huggingface.co/datasets/Lewandofski/OpenVE-Bench) | Video Editing | 431 |
-| [RefVIE-Bench](https://huggingface.co/datasets/linyq/RefVIE-Bench) | Reference Video Editing | 120 |
-| [Intelligent-VBench-MI2V](https://github.com/Tencent-Hunyuan/OmniWeaving) | Multi-Image-to-Video | 320 |
-| [Intelligent-VBench-TIV2V](https://github.com/Tencent-Hunyuan/OmniWeaving) | Text-Image-Video-to-Video | 210 |
-
-> 💡 For **Intelligent-VBench**, we split the original benchmark into two subsets based on task type — **MI2V** and **TIV2V**. Their JSON files should be placed in separate directories.
-
-After downloading, update the `data_root` and `data_json_dir` paths in `configs/dataset/benchmarks.yaml` to point to your local directories.
-
-### Step 2: Install Evaluation Dependencies
-
-**VBench:**
-
-```bash
-mkdir -p libs && cd libs
-git clone https://github.com/Vchitect/VBench.git
-```
-
-Add the following to `libs/VBench/vbench/__init__.py`:
-
-```python
-import sys, os
-local_lib_path = os.path.abspath("libs/VBench")
-if local_lib_path not in sys.path:
-    sys.path.append(local_lib_path)
-```
-
-If you encounter a NumPy 2.0 compatibility error (`np.sctypes was removed`), modify lines 45–47 of `[YOUR_PYTHON_LIBS]/imgaug/imgaug.py`:
-
-```python
-# Replace:
-# NP_FLOAT_TYPES = set(np.sctypes["float"])
-# NP_INT_TYPES = set(np.sctypes["int"])
-# NP_UINT_TYPES = set(np.sctypes["uint"])
-
-# With:
-NP_FLOAT_TYPES = {np.float16, np.float32, np.float64, np.longdouble}
-NP_INT_TYPES = {np.int8, np.int16, np.int32, np.int64, np.longlong}
-NP_UINT_TYPES = {np.uint8, np.uint16, np.uint32, np.uint64, np.ulonglong}
-```
-
-To save disk space, remove unnecessary files:
-
-```bash
-rm -rf libs/VBench/VBench-2.0 libs/VBench/.git libs/VBench/asset libs/VBench/vbench2_beta_trustworthiness
-```
-
-**GenEval:**
-
-```bash
-cd libs
-git clone https://github.com/djghosh13/geneval.git
-cd geneval
-./evaluation/download_models.sh "../../checkpoints/"
-
-cd ..
-pip install mmcv-full
-git clone https://github.com/open-mmlab/mmdetection.git
-cd mmdetection && git checkout 2.x
-pip install -v -e . --no-build-isolation
-```
-
-The GenEval model paths are configured in `configs/evaluation/evaluation.yaml` under `model.evaluation.geneval`:
-
-```yaml
-model:
-  evaluation:
-    geneval:
-      model_path: checkpoints/evaluation/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco.pth
-      model_config_path: libs/mmdetection/configs/mask2former/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco.py
-      clip_path: checkpoints/evaluation/ViT-L-14.pt
-```
-
-### Step 3: Configure API Keys
-
-Some benchmarks (OpenVE-Bench, RefVIE-Bench, ImgEdit-Bench, Intelligent-VBench) require LLM API calls for metric computation. Configure your API keys in `configs/evaluation/evaluation.yaml` under `model.evaluation`:
-
-```yaml
-model:
-  evaluation:
-    # For OpenVE-Bench, RefVIE-Bench, Intelligent-VBench
-    gemini:
-      api_key: "YOUR_GEMINI_API_KEY"
-      base_url: "YOUR_GEMINI_BASE_URL"
-      model: "gemini-2.5-pro-06-17"
-    # For ImgEdit-Bench
-    openai:
-      api_key: "YOUR_OPENAI_API_KEY"
-      base_url: "YOUR_OPENAI_BASE_URL"
-      model: "gpt-4.1"
-```
-
-
-## Run Evaluation
-
-Once the environment is set up, you can simply run evaluation with:
-
-```bash
-NUM_GPUS=8
-
-accelerate launch --num_processes=${NUM_GPUS} \
-    -m scripts.evaluation.evaluate \
-    --config configs/evaluation/evaluation.yaml \
-    --checkpoint_dir checkpoints/LoomVideo \
-    --generation_configs configs/dataset/benchmarks.yaml \
-    --output_dir results/evaluation \
-    --calculate_metrics
-```
-
-
-# 📧 Contact
-
-Jianzong Wu (吴健宗): jzwu@stu.pku.edu.cn
-
-
 # 📄 Citation
 
-If you find our work helpful, please consider giving us a ⭐ on this repo and citing our paper as follows:
-
 ```bibtex
 @article{wu2026loomvideo,
   title={LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing},