matlod commited on
Commit
724293a
·
verified ·
1 Parent(s): bc18cd4

card: lead with two worked exhibits, name the panels, say it needs the de-rope pass

Browse files
Files changed (1) hide show
  1. README.md +69 -3
README.md CHANGED
@@ -14,9 +14,75 @@ pipeline_tag: image-to-video
14
 
15
  # MiniMax-H3 motion adapter (pilot, r16)
16
 
17
- A small LoRA for MiniMax-H3 that makes the de-rope pass in
18
- [ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes) behave
19
- better on fast motion. It is a pilot: rank 16, trained for a few hours on
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
  38 clips we generated ourselves. It works, it is measurable, and it has
21
  clear rough edges listed below. We are publishing it as an intermediate
22
  release while we try something considerably more ambitious that may or may
 
14
 
15
  # MiniMax-H3 motion adapter (pilot, r16)
16
 
17
+ A rank-16 LoRA that makes the de-rope pass in
18
+ [ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes) behave better
19
+ on fast motion. It needs that pass; it is not a general-purpose motion LoRA.
20
+
21
+ ## What it looks like
22
+
23
+ One 5 s clip, four ways. Every panel has its numbers burned in.
24
+
25
+ <video controls muted playsinline preload="metadata" width="100%"
26
+ poster="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad_poster.jpg">
27
+ <source src="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad.mp4" type="video/mp4">
28
+ <a href="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_t2c_insert_quad.mp4">Download the comparison clip (mp4, 2.9 MB)</a>
29
+ </video>
30
+
31
+ | | left | right |
32
+ |---|---|---|
33
+ | **top** | the source plate, straight out of H3 | the whole clip re-generated on a stretched clock, base model, 139.9 s |
34
+ | **bottom** | only the fast span re-generated, no adapter, inject 0.45. 63.6 s, alternation 0.370, rate 1.416 | the same span **with this adapter** at strength 0.75, inject 0.30. 49.9 s, alternation **0.134**, rate **1.011** |
35
+
36
+ Watch the bottom two panels, not the stills. Without the adapter the fighters
37
+ advance and snap, advance and snap, frame to frame. With it they keep moving.
38
+ Rate 1.011 means the motion comes out at the speed the shot called for; 1.416
39
+ means the pass invented about 40 percent more movement than it should have.
40
+
41
+ Two things this exhibit is not. It is not the adapter on its own: the stretched
42
+ clock comes from the [ComfyUI-MAINodes](https://github.com/matlowai/ComfyUI-MAINodes)
43
+ de-rope pass, and the adapter only changes how the model behaves once that pass
44
+ has stretched it. Load this file on a plain text-to-video render and you get the
45
+ costs listed below with none of the benefit. And the settings shown here are
46
+ 0.75 / inject 0.30, the lower-invention cell; see "Settings that work" for when
47
+ to prefer strength 1.0.
48
+
49
+ ## How it is actually wired, and what the adapter is not responsible for
50
+
51
+ Same clip, same seed, same graph in all four tiles. An anime sword ring, which
52
+ is the content type it helps most.
53
+
54
+ <video controls muted playsinline preload="metadata" width="100%"
55
+ poster="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad_poster.jpg">
56
+ <source src="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad.mp4" type="video/mp4">
57
+ <a href="https://huggingface.co/MATLOWAI/MiniMax-H3-Motion-Adapter/resolve/main/assets/adapter_mage_pinned_quad.mp4">Download the comparison clip (mp4, 2.7 MB)</a>
58
+ </video>
59
+
60
+ | | left | right |
61
+ |---|---|---|
62
+ | **top** | the source plate | the burst window re-generated at denoise 0.70 with **nothing anchoring it**, so it drifts off the plate and jumps at both splices |
63
+ | **bottom** | the same graph with the window's **first and last frames pinned**. Two extra wires, no extra time (10m41s either way): jitter 0.103 to **0.006**, entry jank 2.05 to 1.58, invented objects 109 to **60** | pins **plus this adapter** at 0.75, one `LoraLoaderModelOnly`: alternation 0.129 to **0.093**, objects 60 to 54, for 39 s more wall time |
64
+
65
+ The order matters, and it is the opposite of what people expect. **The pins do
66
+ most of the work**, and they are free: pinning the window's boundary frames to
67
+ the plate frames they must butt against is what stops the splice jump, adapter
68
+ or no adapter. The adapter then converts spare capacity into smoothness on top
69
+ of that. If your seams are jumping, pin first and only then reach for this file.
70
+
71
+ The same tile also shows the cost honestly: warm colour goes 1.04 to -1.59 with
72
+ the adapter on. That is the colour muting described below, visible in the one
73
+ exhibit where everything else improves.
74
+
75
+ Play to the end for a card listing every setting: window, hold map, denoise,
76
+ sampler, seed, quantisation and wall time per tile. (It labels the LoRA by its
77
+ training name, `p4_pilot_k100.safetensors`; that is this file, published here as
78
+ `minimax_h3_motion_adapter_pilot_r16.safetensors`.)
79
+
80
+ More before/after clips, including the two cases where it makes things worse,
81
+ are on the [demo page](https://matlowai.github.io/ComfyUI-MAINodes/#adapter).
82
+
83
+ ## About this release
84
+
85
+ It is a pilot: rank 16, trained for a few hours on
86
  38 clips we generated ourselves. It works, it is measurable, and it has
87
  clear rough edges listed below. We are publishing it as an intermediate
88
  release while we try something considerably more ambitious that may or may