Instructions to use MATLOWAI/MiniMax-H3-Motion-Adapter with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use MATLOWAI/MiniMax-H3-Motion-Adapter with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image, export_to_video # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", dtype=torch.bfloat16, device_map="cuda") pipe.load_lora_weights("MATLOWAI/MiniMax-H3-Motion-Adapter") prompt = "A man with short gray hair plays a red electric guitar." input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png") image = pipe(image=input_image, prompt=prompt).frames[0] export_to_video(output, "output.mp4") - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
card: lead with two worked exhibits, name the panels, say it needs the de-rope pass
Browse files
README.md
CHANGED
|
@@ -14,9 +14,75 @@ pipeline_tag: image-to-video
|
|
| 14 |
|
| 15 |
# MiniMax-H3 motion adapter (pilot, r16)
|
| 16 |
|
| 17 |
-
A
|
| 18 |
-
[ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes) behave
|
| 19 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
38 clips we generated ourselves. It works, it is measurable, and it has
|
| 21 |
clear rough edges listed below. We are publishing it as an intermediate
|
| 22 |
release while we try something considerably more ambitious that may or may
|
|
|
|
| 14 |
|
| 15 |
# MiniMax-H3 motion adapter (pilot, r16)
|
| 16 |
|
| 17 |
+
A rank-16 LoRA that makes the de-rope pass in
|
| 18 |
+
[ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes) behave better
|
| 19 |
+
on fast motion. It needs that pass; it is not a general-purpose motion LoRA.
|
| 20 |
+
|
| 21 |
+
## What it looks like
|
| 22 |
+
|
| 23 |
+
One 5 s clip, four ways. Every panel has its numbers burned in.
|
| 24 |
+
|
| 25 |
+
<video controls muted playsinline preload="metadata" width="100%"
|
| 26 |
+
poster="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad_poster.jpg">
|
| 27 |
+
<source src="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad.mp4" type="video/mp4">
|
| 28 |
+
<a href="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad.mp4">Download the comparison clip (mp4, 2.9 MB)</a>
|
| 29 |
+
</video>
|
| 30 |
+
|
| 31 |
+
| | left | right |
|
| 32 |
+
|---|---|---|
|
| 33 |
+
| **top** | the source plate, straight out of H3 | the whole clip re-generated on a stretched clock, base model, 139.9 s |
|
| 34 |
+
| **bottom** | only the fast span re-generated, no adapter, inject 0.45. 63.6 s, alternation 0.370, rate 1.416 | the same span **with this adapter** at strength 0.75, inject 0.30. 49.9 s, alternation **0.134**, rate **1.011** |
|
| 35 |
+
|
| 36 |
+
Watch the bottom two panels, not the stills. Without the adapter the fighters
|
| 37 |
+
advance and snap, advance and snap, frame to frame. With it they keep moving.
|
| 38 |
+
Rate 1.011 means the motion comes out at the speed the shot called for; 1.416
|
| 39 |
+
means the pass invented about 40 percent more movement than it should have.
|
| 40 |
+
|
| 41 |
+
Two things this exhibit is not. It is not the adapter on its own: the stretched
|
| 42 |
+
clock comes from the [ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes)
|
| 43 |
+
de-rope pass, and the adapter only changes how the model behaves once that pass
|
| 44 |
+
has stretched it. Load this file on a plain text-to-video render and you get the
|
| 45 |
+
costs listed below with none of the benefit. And the settings shown here are
|
| 46 |
+
0.75 / inject 0.30, the lower-invention cell; see "Settings that work" for when
|
| 47 |
+
to prefer strength 1.0.
|
| 48 |
+
|
| 49 |
+
## How it is actually wired, and what the adapter is not responsible for
|
| 50 |
+
|
| 51 |
+
Same clip, same seed, same graph in all four tiles. An anime sword ring, which
|
| 52 |
+
is the content type it helps most.
|
| 53 |
+
|
| 54 |
+
<video controls muted playsinline preload="metadata" width="100%"
|
| 55 |
+
poster="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad_poster.jpg">
|
| 56 |
+
<source src="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad.mp4" type="video/mp4">
|
| 57 |
+
<a href="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad.mp4">Download the comparison clip (mp4, 2.7 MB)</a>
|
| 58 |
+
</video>
|
| 59 |
+
|
| 60 |
+
| | left | right |
|
| 61 |
+
|---|---|---|
|
| 62 |
+
| **top** | the source plate | the burst window re-generated at denoise 0.70 with **nothing anchoring it**, so it drifts off the plate and jumps at both splices |
|
| 63 |
+
| **bottom** | the same graph with the window's **first and last frames pinned**. Two extra wires, no extra time (10m41s either way): jitter 0.103 to **0.006**, entry jank 2.05 to 1.58, invented objects 109 to **60** | pins **plus this adapter** at 0.75, one `LoraLoaderModelOnly`: alternation 0.129 to **0.093**, objects 60 to 54, for 39 s more wall time |
|
| 64 |
+
|
| 65 |
+
The order matters, and it is the opposite of what people expect. **The pins do
|
| 66 |
+
most of the work**, and they are free: pinning the window's boundary frames to
|
| 67 |
+
the plate frames they must butt against is what stops the splice jump, adapter
|
| 68 |
+
or no adapter. The adapter then converts spare capacity into smoothness on top
|
| 69 |
+
of that. If your seams are jumping, pin first and only then reach for this file.
|
| 70 |
+
|
| 71 |
+
The same tile also shows the cost honestly: warm colour goes 1.04 to -1.59 with
|
| 72 |
+
the adapter on. That is the colour muting described below, visible in the one
|
| 73 |
+
exhibit where everything else improves.
|
| 74 |
+
|
| 75 |
+
Play to the end for a card listing every setting: window, hold map, denoise,
|
| 76 |
+
sampler, seed, quantisation and wall time per tile. (It labels the LoRA by its
|
| 77 |
+
training name, `p4_pilot_k100.safetensors`; that is this file, published here as
|
| 78 |
+
`minimax_h3_motion_adapter_pilot_r16.safetensors`.)
|
| 79 |
+
|
| 80 |
+
More before/after clips, including the two cases where it makes things worse,
|
| 81 |
+
are on the [demo page](https://matlowai.github.io/ComfyUI-MAINodes/#adapter).
|
| 82 |
+
|
| 83 |
+
## About this release
|
| 84 |
+
|
| 85 |
+
It is a pilot: rank 16, trained for a few hours on
|
| 86 |
38 clips we generated ourselves. It works, it is measurable, and it has
|
| 87 |
clear rough edges listed below. We are publishing it as an intermediate
|
| 88 |
release while we try something considerably more ambitious that may or may
|