Upload 5 files

README.md

@@ -1,3 +1,151 @@
 ---
 license: apache-2.0
+base_model: MiniMaxAI/MiniMax-H3
+base_model_relation: adapter
+tags:
+  - text-to-video
+  - text-to-audio
+  - audio-video
+  - lora
+  - minimax-h3
+  - diffusers
+  - peft
+pipeline_tag: text-to-video
+library_name: diffusers
 ---
+
+# MiniMax-H3 Turbo LoRA — Diffusers
+
+Diffusers / PEFT conversion of [`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora): a LoRA that lets [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) render joint **video + synchronized stereo audio** in about **4 sampling steps** instead of the usual ~20.
+
+This repo ships:
+
+- converted LoRA weights in Diffusers PEFT layout (`transformer.*.lora_A/B.weight`)
+- `convert.py` to turn the original ComfyUI / `generate.py` safetensors into that layout
+
+> ⚠️ **Early prototype.** Same caveat as the upstream release: under-trained preview weights, not production quality. They already beat the base model at 4 steps (sharper detail, cleaner / better-synced audio), but treat this as a work-in-progress taste, not a finished product. Prefer the non-EMA `ckpt500` weights by default.
+
+## Weights
+
+Converted from the upstream Turbo LoRA (bf16, `W_eff = W + lora_B @ lora_A`, **alpha = rank** so scale is 1). QKV is split into `to_q` / `to_k` / `to_v`, and SwiGLU `fc1` halves are swapped to match Diffusers' `[value; gate]` layout.
+
+| file | source (ComfyUI layout) | notes |
+|---|---|---|
+| `minimax_h3_turbo_4step_ckpt500_diffusers.safetensors` | `minimax_h3_turbo_4step_ckpt500.safetensors` | **recommended default** — newest non-EMA @ ~500 steps, usually sharpest |
+| `minimax_h3_turbo_4step_ema_ckpt500_diffusers.safetensors` | `minimax_h3_turbo_4step_ema_ckpt500.safetensors` | EMA @ ~500 steps — smoother, but early EMA can show **ghosting / motion smear** |
+
+Ranks: attention / MLP = 64, AdaLN = 16. Keys are prefixed with `transformer.` for `MiniMaxH3Transformer3DModel.load_lora_adapter`.
+
+## Requirements
+
+MiniMax-H3 is not in a released Diffusers build yet. Install Diffusers from `main`, plus PEFT:
+
+```bash
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
+pip install -r requirements.txt
+pip install git+https://github.com/huggingface/diffusers.git
+```
+
+Base weights: Diffusers-format [MiniMaxAI/MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) (or your local conversion). Use a **non-pruned** DiT; pruned time-conditioning layouts are **not** compatible with this LoRA (same restriction as upstream).
+
+## Quick start (Diffusers)
+
+```python
+import torch
+from diffusers import ComponentsManager, ModularPipeline
+from diffusers.utils.export_utils import encode_video
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+
+def network_alphas_alpha_eq_rank(state_dict):
+    # Turbo LoRA: alpha == rank. Required when ranks differ (attn/mlp=64, adaln=16).
+    alphas = {}
+    for key, tensor in state_dict.items():
+        if key.endswith(".lora_B.weight") and tensor.ndim > 1:
+            base = key[: -len(".lora_B.weight")]
+            alphas[f"{base}.alpha"] = float(tensor.shape[1])
+    return alphas
+
+lora_path = hf_hub_download(
+    "InstantX/MiniMax-H3-Turbo-Lora-Diffusers",
+    "minimax_h3_turbo_4step_ckpt500_diffusers.safetensors",
+)
+
+manager = ComponentsManager()
+pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", components_manager=manager)
+pipe.load_components(dtype=torch.bfloat16)
+
+lora_sd = load_file(lora_path, device="cpu")
+pipe.transformer.load_lora_adapter(
+    lora_sd,
+    prefix="transformer",
+    adapter_name="turbo_4step",
+    network_alphas=network_alphas_alpha_eq_rank(lora_sd),
+)
+
+# Load LoRA *before* enabling offload so PEFT injects into resident modules.
+manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="12GB")
+
+# Optional: FlashAttention-3 on Hopper (kernels from the Hub).
+try:
+    pipe.transformer.set_attention_backend("_flash_3_hub")
+except Exception:
+    pipe.transformer.set_attention_backend("native")
+
+# MiniMaxH3Scheduler: num_inference_steps is the sigma grid length *including* terminal 0,
+# so it drives (num_inference_steps - 1) model evals.
+#   5 -> 4 evals  (matches upstream generate.py --steps 4)
+#   7–9 -> 6–8 evals  (upstream comfort zone for sharpness at this early checkpoint)
+results = pipe(
+    prompt="A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark.",
+    num_frames=124,   # 17*k+5, ~5.17s @ 24fps
+    height=768,
+    width=1344,
+    num_inference_steps=5,
+    generator=torch.Generator().manual_seed(42),
+    output=["videos", "audio", "sampling_rate"],
+)
+
+encode_video(
+    results["videos"][0],
+    fps=24,
+    output_path="out.mp4",
+    audio=results["audio"][0],
+    audio_sample_rate=results["sampling_rate"],
+)
+```
+
+Diffusers already runs **dual video / audio schedules** (`scheduler` shift 12, `audio_scheduler` shift 3). You do **not** need the ComfyUI Turbo custom sampler node; a wrong single-schedule sampler is what blows up audio at 4 steps in ComfyUI.
+
+## Convert from the original Turbo LoRA
+
+Original weights live in [`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora) (ComfyUI module names, fused `qkv_proj` / `mlp.fc1`).
+
+```bash
+pip install safetensors torch
+
+python convert.py \
+  --input minimax_h3_turbo_4step_ckpt500.safetensors \
+  --output minimax_h3_turbo_4step_ckpt500_diffusers.safetensors
+```
+
+What `convert.py` does:
+
+1. Renames ComfyUI paths onto `MiniMaxH3Transformer3DModel` (`blocks.*` → `transformer_blocks.*`, `mlp.fc*` → `ff.net.*`, `final_layer.adaln_proj` → `norm_out.linear`, …).
+2. Splits fused `attn.qkv_proj` LoRA into `to_q` / `to_k` / `to_v` (shared `A`, row-split `B` in `[q_all; k_all; v_all]` layout).
+3. Swaps `mlp.fc1` LoRA halves from `[gate; value]` to Diffusers SwiGLU `[value; gate]`.
+4. Writes keys with a `transformer.` prefix for `load_lora_adapter`.
+
+## Notes
+
+- **Steps**: 4 model evals (`num_inference_steps=5`) works; at this early checkpoint **6–8 evals** (`num_inference_steps=7…9`) are usually sharper. Any count ≥ 4 evals is valid; more steps look better.
+- **Resolution / duration**: `height` / `width` multiples of 32 (short edge typically 768). `num_frames` at 24 fps snaps up to the video VAE’s `17·k+5` grid (124 ≈ 5 s). Validated roughly 5–15 s.
+- **VRAM**: the base DiT is ~33B. An 80–96 GB GPU is comfortable with `ComponentsManager.enable_auto_cpu_offload`; smaller cards need quantization / group offload as in the [MiniMax-H3 Diffusers docs](https://huggingface.co/docs/diffusers/main/en/api/pipelines/minimax_h3).
+- **Audio**: 32 kHz stereo aligned to the video; video and audio ride different flow schedules inside one transformer call.
+- **ComfyUI**: for the original graph / custom Turbo sampler, use the [upstream repo](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora) and [Larryvrh/ComfyUI-MiniMax-H3-Turbo](https://github.com/Larryvrh/ComfyUI-MiniMax-H3-Turbo).
+
+## Credit
+
+- Turbo LoRA training & original release: [`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora)
+- Base model: [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)
+- Diffusers MiniMax-H3 integration: Hugging Face Diffusers (modular pipeline)

convert.py

@@ -0,0 +1,178 @@
+"""Convert a MiniMax-H3 Turbo LoRA (ComfyUI / generate.py layout) to Diffusers PEFT layout.
+
+The Turbo LoRA ships with ComfyUI module names and fused projections:
+
+* ``blocks.*.attn.qkv_proj``  — fused ``[q_all; k_all; v_all]`` (the in-memory layout after
+  ComfyUI's load-time QKV reorder), not the raw checkpoint's per-head interleave.
+* ``blocks.*.mlp.fc1``        — fused ``[gate; value]``; Diffusers' ``SwiGLU`` wants ``[value; gate]``.
+* ``alpha == rank``           — no extra scale (``W_eff = W + B @ A``).
+
+This script renames everything onto ``MiniMaxH3Transformer3DModel``, splits the fused QKV LoRA
+into ``to_q`` / ``to_k`` / ``to_v`` (shared ``A``, split ``B``), and swaps the ``fc1`` halves so the
+low-rank update matches the converted base weights.
+
+Usage:
+
+```bash
+python convert.py \
+  --input minimax_h3_turbo_4step_ckpt500.safetensors \
+  --output minimax_h3_turbo_4step_ckpt500_diffusers.safetensors
+```
+
+If ``--output`` is omitted, ``_diffusers`` is inserted before the ``.safetensors`` suffix.
+"""
+
+from __future__ import annotations
+
+import argparse
+from collections import defaultdict
+from pathlib import Path
+
+import torch
+from safetensors.torch import load_file, save_file
+
+# Must match MiniMaxH3Transformer3DModel / convert_minimax_h3_to_diffusers.py.
+NUM_ATTENTION_HEADS = 56
+ATTENTION_HEAD_DIM = 128
+INNER_DIM = NUM_ATTENTION_HEADS * ATTENTION_HEAD_DIM  # 7168
+FFN_DIM = 14336
+PREFIX = "transformer"
+
+
+def _module_names(state_dict: dict[str, torch.Tensor]) -> list[str]:
+    return sorted({key.rsplit(".lora_", 1)[0] for key in state_dict})
+
+
+def _rename_base(name: str) -> str:
+    """Map a ComfyUI module path (without ``.lora_*``) onto the Diffusers module path."""
+    if name.startswith("token_refiner.blocks."):
+        name = name.replace("token_refiner.blocks.", "token_refiner.refiner_blocks.", 1)
+    elif name.startswith("blocks."):
+        name = name.replace("blocks.", "transformer_blocks.", 1)
+
+    name = name.replace("final_layer.adaln_proj.linear", "norm_out.linear")
+    name = name.replace(".attn.out_proj", ".attn.to_out.0")
+    name = name.replace(".mlp.fc2", ".ff.net.2")
+    name = name.replace(".mlp.fc1", ".ff.net.0.proj")
+    return name
+
+
+def convert_lora_state_dict(
+    src: dict[str, torch.Tensor],
+) -> tuple[dict[str, torch.Tensor], dict[str, int]]:
+    """Convert one Turbo LoRA state dict into Diffusers PEFT keys (with ``transformer.`` prefix)."""
+    out: dict[str, torch.Tensor] = {}
+    counts: dict[str, int] = defaultdict(int)
+
+    for name in _module_names(src):
+        a = src[f"{name}.lora_A.weight"]
+        b = src[f"{name}.lora_B.weight"]
+        if a.ndim != 2 or b.ndim != 2:
+            raise ValueError(f"{name}: expected 2-D LoRA matrices, got A{tuple(a.shape)} B{tuple(b.shape)}")
+        if a.shape[0] != b.shape[1]:
+            raise ValueError(f"{name}: rank mismatch A{tuple(a.shape)} vs B{tuple(b.shape)}")
+
+        # qkv: split fused [q;k;v] B into three LoRAs that share A.
+        if name.endswith(".attn.qkv_proj"):
+            if b.shape[0] != 3 * INNER_DIM:
+                raise ValueError(
+                    f"{name}: fused qkv B has {b.shape[0]} rows, expected {3 * INNER_DIM} "
+                    f"(= 3 * {INNER_DIM})."
+                )
+            base = _rename_base(name[: -len(".attn.qkv_proj")])
+            bq, bk, bv = b.split(INNER_DIM, dim=0)
+            for suffix, b_part in (("to_q", bq), ("to_k", bk), ("to_v", bv)):
+                key = f"{PREFIX}.{base}.attn.{suffix}"
+                # Clone A so to_q/to_k/to_v do not share storage (safetensors forbids that).
+                out[f"{key}.lora_A.weight"] = a.detach().clone().contiguous()
+                out[f"{key}.lora_B.weight"] = b_part.detach().clone().contiguous()
+                counts["qkv_split"] += 1
+            continue
+
+        base = _rename_base(name)
+
+        # fc1 / SwiGLU: reference stores [gate; value], Diffusers wants [value; gate].
+        if name.endswith(".mlp.fc1"):
+            if b.shape[0] != 2 * FFN_DIM:
+                raise ValueError(
+                    f"{name}: fc1 B has {b.shape[0]} rows, expected {2 * FFN_DIM} (= 2 * {FFN_DIM})."
+                )
+            gate, value = b.chunk(2, dim=0)
+            b = torch.cat([value, gate], dim=0).contiguous()
+            counts["fc1_swap"] += 1
+        else:
+            counts["rename"] += 1
+
+        key = f"{PREFIX}.{base}"
+        out[f"{key}.lora_A.weight"] = a.detach().clone().contiguous()
+        out[f"{key}.lora_B.weight"] = b.detach().clone().contiguous()
+
+    return out, dict(counts)
+
+
+def network_alphas_from_state_dict(state_dict: dict[str, torch.Tensor]) -> dict[str, float]:
+    """``alpha == rank`` for every module (Turbo LoRA convention)."""
+    alphas: dict[str, float] = {}
+    for key, tensor in state_dict.items():
+        if key.endswith(".lora_B.weight") and tensor.ndim > 1:
+            base = key[: -len(".lora_B.weight")]
+            alphas[f"{base}.alpha"] = float(tensor.shape[1])
+    return alphas
+
+
+def default_output_path(input_path: str | Path) -> Path:
+    path = Path(input_path)
+    stem = path.stem
+    if stem.endswith("_diffusers"):
+        return path
+    return path.with_name(f"{stem}_diffusers{path.suffix}")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument(
+        "--input",
+        required=True,
+        help="ComfyUI / generate.py Turbo LoRA safetensors "
+        "(e.g. minimax_h3_turbo_4step_ckpt500.safetensors from larryvrh/MiniMax-H3-Turbo-Lora)",
+    )
+    parser.add_argument(
+        "--output",
+        default=None,
+        help="Diffusers PEFT LoRA safetensors (keys prefixed with transformer.). "
+        "Defaults to <input_stem>_diffusers.safetensors",
+    )
+    args = parser.parse_args()
+    output = Path(args.output) if args.output is not None else default_output_path(args.input)
+
+    print(f"loading {args.input}")
+    src = load_file(args.input, device="cpu")
+    dst, counts = convert_lora_state_dict(src)
+
+    ranks = sorted({int(v.shape[1]) for k, v in dst.items() if k.endswith(".lora_B.weight")})
+    print(
+        f"converted {len(src)} -> {len(dst)} tensors; "
+        f"qkv_split={counts.get('qkv_split', 0)} "
+        f"fc1_swap={counts.get('fc1_swap', 0)} "
+        f"rename={counts.get('rename', 0)}; ranks={ranks}"
+    )
+
+    # Keep the original dtype (bf16). Metadata is informational for humans / loaders.
+    metadata = {
+        "format": "pt",
+        "base_model": "MiniMax-H3",
+        "application": "W_eff = W + lora_B @ lora_A (alpha == rank)",
+        "sampler_steps": "4",
+        "converted_from": "larryvrh/MiniMax-H3-Turbo-Lora (ComfyUI layout)",
+    }
+    save_file(dst, str(output), metadata=metadata)
+    print(f"saved {output}")
+
+    # Print a tiny alpha hint so callers can wire network_alphas correctly.
+    alphas = network_alphas_from_state_dict(dst)
+    alpha_values = sorted(set(alphas.values()))
+    print(f"network alphas (== rank): {alpha_values} across {len(alphas)} modules")
+
+
+if __name__ == "__main__":
+    main()

minimax_h3_turbo_4step_ckpt500_diffusers.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad92b79a1626df5b55bbdd300eec9380ea75c952468d104601c9fdec42bebb9a
+size 851455256

minimax_h3_turbo_4step_ema_ckpt500_diffusers.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fed3db077957dc49385e36a7597d701cc2948ffa5c723d90788c373d54325dea
+size 851455248

requirements.txt

@@ -0,0 +1,43 @@
+# MiniMax-H3 inference (infer.py / convert.py)
+# Versions pinned to the working docker env (CUDA 12.6).
+#
+# Install torch first from the CUDA wheel index, then the rest:
+#   pip install torch==2.9.1 torchvision==0.24.1 --index-url https://download.pytorch.org/whl/cu126
+#   pip install -r requirements.txt
+#
+# MiniMax-H3 is not in a released diffusers build yet — install the local checkout
+# (or the PR) instead of the PyPI package:
+#   pip install -e ./diffusers
+#   # or: pip install git+https://github.com/huggingface/diffusers.git@refs/pull/14355/head
+
+# --- PyTorch (install via the CUDA index above; listed here for reference) ---
+# torch==2.9.1+cu126
+# torchvision==0.24.1+cu126
+
+torch==2.9.1
+torchvision==0.24.1
+
+# --- Hugging Face stack ---
+transformers==5.14.1
+accelerate==1.14.0
+huggingface-hub==1.26.0
+safetensors==0.8.0
+peft==0.20.0
+tokenizers==0.22.2
+sentencepiece==0.2.0
+protobuf==4.24.4
+
+# --- Diffusers extras used by infer.py ---
+kernels==0.16.0
+
+# --- Numerics / IO ---
+numpy==1.24.4
+scipy==1.12.0
+Pillow==10.2.0
+einops==0.7.0
+ftfy==6.2.0
+opencv-python==4.7.0
+av==17.1.0
+soundfile==0.12.1
+imageio==2.37.4
+imageio-ffmpeg==0.6.0