File size: 13,192 Bytes

99f8f6e

---
base_model:
- ""
frameworks:
- ""
license: Apache License 2.0
tags: []
tasks:
- text-to-video-synthesis
---
# MiniMax-H3 Templates: Text Embeddings

This repository provides a collection of video effect templates for the [MiniMax-H3](https://modelscope.cn/models/MiniMax/MiniMax-H3) model. These templates encapsulate specific model capabilities as text embeddings, implemented via [Diffusion Templates](https://arxiv.org/abs/2604.24351).

## Showcase

### Effects Gallery

| **art_is_explosion** | **storm_magic** |
| --- | --- |
| <video src="assets/video_5.mp4" controls muted loop></video> | <video src="assets/video_6.mp4" controls muted loop></video> |
| **dark_magic** | **kiss_camera** |
| <video src="assets/video_1.mp4" controls muted loop></video> | <video src="assets/video_2.mp4" controls muted loop></video> |
| **bullet_time** | **truman_show** |
| <video src="assets/video_3.mp4" controls muted loop></video> | <video src="assets/video_4.mp4" controls muted loop></video> |
| **fire_breath** | **blooming_flowers** |
| <video src="assets/video_7.mp4" controls muted loop></video> | <video src="assets/video_8.mp4" controls muted loop></video> |
| **four_seasons** | **spiral_ascent** |
| <video src="assets/video_9.mp4" controls muted loop></video> | <video src="assets/video_10.mp4" controls muted loop></video> |

### Effect Combinations

Combining effects: `art_is_explosion` + `storm_magic`

| | | |
| --- | --- | --- |
| <video src="assets/extra_videos/video_1.mp4" autoplay muted loop></video> | <video src="assets/extra_videos/video_2.mp4" autoplay muted loop></video> | <video src="assets/extra_videos/video_3.mp4" autoplay muted loop></video> |

## How It Works

Each template consists of a single tensor. This tensor can either replace or be combined with the output of the Text Encoder, a mechanism very similar to [Textual Inversion](https://arxiv.org/abs/2208.01618).

While MiniMax-H3 boasts powerful base capabilities, its massive parameter count makes LoRA training challenging. Text Embeddings offer a lightweight alternative to LoRA with the following advantages:

*   **Modular Capabilities:** Text Embeddings act as carriers of model functionality. Like agent skills, they allow specific capabilities to be saved, reused, and distributed as atomic units via model platforms.
*   **Flexible Initialization:** Embeddings can be initialized from text prompts, images, or videos. Simply process the input data through the MiniMax-H3 Text Encoder to quickly generate a usable Text Embedding.
*   **Trainable:** Like LoRA, Text Embeddings support end-to-end training on video datasets, enabling targeted enhancement of specific generative capabilities.
*   **Composable:** Multiple templates can be combined to jointly influence generation, allowing you to create complex and stunning visual effects.

## Inference and Training

### Installation

Install [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio):

```shell
git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e ".[all]"
```

### Initialize a Text Embedding via Text Encoder

The following code has minimal VRAM requirements and can run with as little as 6GB of VRAM:

```python
import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from safetensors.torch import save_file

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[ModelConfig(
        model_id="MiniMax/MiniMax-H3",
        origin_file_pattern="FL2VA/text_encoder/model*.safetensors",
        offload_dtype="disk",
        offload_device="disk",
        onload_dtype="disk",
        onload_device="disk",
        preparing_dtype=torch.bfloat16,
        preparing_device="cuda",
        computation_dtype=torch.bfloat16,
        computation_device="cuda",
    )],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=0,
)
prompt = "xxx"
text_embedding = pipe.export_text_embedding(prompt)
save_file({"weight": text_embedding}, "model.safetensors")
```

### Inference with Text Embeddings via Diffusion Templates

You can load Text Embeddings through [Diffusion Templates](https://arxiv.org/abs/2604.24351) for inference. Optionally, load an acceleration LoRA to speed up inference (note: the Text Encoder is not required during this stage):

```python
import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.diffusion.template import TemplatePipeline
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from modelscope import snapshot_download
from PIL import Image

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
template = TemplatePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[ModelConfig(
        model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/art_is_explosion/",
    )],
)
snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = template(
    pipe,
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
    template_inputs=[{}],
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)
```

<details>
<summary>Inference with Multiple Text Embeddings</summary>

```diff
import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.diffusion.template import TemplatePipeline
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from modelscope import snapshot_download
from PIL import Image

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
template = TemplatePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
+       ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/art_is_explosion/"),
+       ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/storm_magic/"),
    ],
)
snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = template(
    pipe,
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
+   template_inputs=[{"model_id": 0}, {"model_id": 1}],
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)
```

</details>

<details>
<summary>Inference Using the Native Pipeline</summary>

```python
import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from diffsynth import load_state_dict
from modelscope import snapshot_download
from PIL import Image

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
text_embedding_config = ModelConfig(
    model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings",
    origin_file_pattern="models/art_is_explosion/model.safetensors",
)
text_embedding_config.download_if_necessary()
text_embedding = load_state_dict(text_embedding_config.path)["weight"]

snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = pipe(
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
    text_embedding=text_embedding,
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)
```

</details>

### Training Text Embeddings

```shell
modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "minimax_h3/MiniMax-H3-Text-Embeddings/*" --local_dir ./data/diffsynth_example_dataset

accelerate launch examples/minimax_h3/model_training/train.py \
  --dataset_base_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Text-Embeddings \
  --dataset_metadata_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Text-Embeddings/metadata.json \
  --data_file_keys "video,input_audio" \
  --extra_inputs "input_audio,input_image,template_inputs" \
  --height 832 \
  --width 480 \
  --num_frames 124 \
  --dataset_repeat 100 \
  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:video_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:audio_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-fl2va-pruned-nf4.safetensors" \
  --template_model_id_or_path "DiffSynth-Studio/MiniMax-H3-Text-Embeddings:models/art_is_explosion/" \
  --learning_rate 1e-4 \
  --num_epochs 2 \
  --remove_prefix_in_ckpt "pipe.template_model." \
  --output_path "./models/train/MiniMax-H3-Text-Embeddings-full" \
  --trainable_models "template_model" \
  --use_gradient_checkpointing
```