Viggle-Animate

Character Replacement in Video from a Single Repainted Frame

Try the demo  ·  viggle.ai/h3  ·  Built on MiniMaxAI/MiniMax-H3

Viggle-Animate replaces the character in a video with one you paint into that video's own first frame, keeping the original motion, camera and timing.

It reads no pose skeleton, no mask and no prompt, and runs no model but itself. Two inputs, four sampling steps, one GPU, 26 seconds a shot.

Abstract

Controlled character replacement has been built on intermediate representations: pose skeletons for motion, segmentation masks and background plates for the scene, face crops for identity. Each is produced by a separate extractor, and each extraction is a place to lose information — and a model to load, schedule and wait for. Recent work has begun dropping the skeleton while keeping a mask channel. Viggle-Animate keeps neither. It takes two inputs — a driving video, and that video's own first frame with the character repainted — and its only task is to propagate that edit across the shot. There is no preprocessing pass, no second network, and no text encoder at inference.

The choice of reference is what makes this possible. Because the reference is a frame of the clip, pose, camera, framing and lighting have already been reconciled by the image editor that produced it, and nothing downstream has to solve them again. It also means the model is never told what the new character is: no prompt, no class, no identity encoder. Viggle-Animate is a 33.1 B full finetune of MiniMax-H3's ref2va transformer, distilled with DMD into a four-step sampler that renders 124 frames in 26 s on a single B200 — 6.1× faster per clip than Wan2.2-Animate-14B on the same hardware, with no preprocessing charged to either.

Method

Character replacement asks two questions at once: what does the new character look like, and how does it move through this shot. Systems that condition on a standalone character photograph must answer both, and reconciling a photograph with footage it was never part of is what the scaffolding exists for.

State-of-the-art image models have finished that job. Give gpt-image a frame and an instruction and it replaces the character while following the prompt exactly — transferring the pose, matching the lighting, preserving the background. The hard reconciliation is already solved, once, on one image. This model is the second half of that pipeline, not the whole of it.

driving.mp4
the performance

frame 0 →

any image editor
gpt-image-2 · not this model

→

ref.png
frame 0, repainted

driving.mp4  +  ref.png
↓

Viggle-Animate
33.1 B ref2va full finetune  +  DMD2 LoRA, rank 128
4 sampling steps · 3 forward passes · frozen text embedding, encoder never loads

↓

out.mp4
same length, same camera, new character

Never computed: pose skeleton · segmentation mask · face crops · background plate · depth · text prompt

Appearance enters only through the repainted frame; geometry enters only through the driving video. Conditioning is a single frozen embedding shipped with the weights (assets/fixed_prompt.txt), so the text encoder never loads at inference and every render is conditioned identically.

Left panel is the driving video, right panel is this model at --steps 4:

It is fast twice over. There is nothing else to run — no preprocessing pass, no second network, no text encoder. And the sampler is DMD-distilled, so a finished clip is three forward passes rather than thirty. The two compound: what is left is one model, one GPU, one pass, and no orchestration to get wrong.

It generalises past humans, because nothing in the loop assumes one. A pose skeleton has a neck and two arms; a mask has a person-shaped hole. We have neither, so the model holds no representation that a character must be a person. What it can animate is bounded by what you can paint.

Efficiency

26 s

per render
124 frames at 24 fps, 480×832, a single B200

3

forward passes
4 sampling steps, after DMD distillation

2

inputs
a clip, and that clip's own first frame repainted

0

other models
no pose estimator, no segmenter, no face tracker, no text encoder

One B200, 480×832, 124 frames at 24 fps, bf16, no compile, no offload. Wan ran its documented replacement recipe — 20 steps, sample_shift 5.0, --refert_num 1 --replace_flag --use_relighting_lora, --w_len 1 --h_len 1 — after its own preprocessing pass.

	Viggle-Animate	Wan2.2-Animate-14B
Inputs	driving video + one repainted frame	driving video + character image, then a preprocessing pass producing pose, face, mask and background tracks
Render, after weights load	26 s	160 s
— of which sampling	13.6 s	140 s
Forward passes	3	40 (20 steps × 2 chunks)
Parameters	33.1 B	17.3 B

6.1× faster per render, 10.3× on sampling alone, on a model with 1.9× the parameters. Wan's preprocessing pass is not counted in its 160 s. We compare against Wan2.2-Animate-14B as the released open-weight replacement model.

Qualitative comparison

Four panels each: painted reference · driving video · this model · Wan2.2-Animate-14B, the last rendered from the official unmodified weights at its documented replacement settings.

Generalization

The model is never told what it is animating, so how far the character can get from a person is an empirical question rather than a list of supported categories. Three panels each: painted reference · driving video · this model. Every clip below is one paint and one render at --steps 4 --seed 42 — no best-of-N.

Animals. Ears, eye patches and flippers move on limbs the driving clip does not have — the paint places them, the render animates them as if they had always been arms and a head.

Not humanoid. The airliner is the hardest case we have: the paint binds wings to arms and landing gear to legs, and the model's job is to keep that binding for 124 frames. The robot has to relight specular metal as it turns.

Stylised. A plasticine figure in a photographic kitchen holds its style boundary for the whole clip.

More than one character. The work moves into the paint prompt, which has to bind each character to a position — "the one on the left" — or the editor swaps one and leaves the other. The last clip is a wide arena shot where each figure is a few dozen pixels tall.

Limitations

It inherits the image edit. What the paint does not show, the model will not add, and where paint and video disagree the video wins. Appearance comes from the paint but shape comes from the driving pose: a LEGO minifigure kept its palette and yellow claw hands, yet reverted to human anatomy — the airliner held because the paint tied its wings to real arms.

Lip-sync is weak. Mouth shapes do not track speech closely in close-ups. Identity and expression hold; it is the sync that lags, and we believe that is a training-data limit rather than anything structural.

Complex scenes are harder than single subjects. Several characters at once, close interaction between them, and shots that cut are all cases where quality drops off — enough that we would not call them solved.

We are training a substantially better model right now, aimed squarely at these three. This release is the version we can ship today, not the ceiling.

What this repository contains

Two parts, both derived from MiniMaxAI/MiniMax-H3's ref2va transformer:

transformer/	33.1 B, bf16, 14 shards. A full finetune of the base transformer_ref on a character-replacement objective
lora/	rank 128 over 302 linear layers, 2.5 GB. A DMD2-distilled delta on that finetune — this is what turns 30 sampling steps into 4

The LoRA is a delta on the finetuned transformer — loading it onto stock transformer_ref produces garbage.

Quickstart

This repository ships only the transformer and the LoRA — the VAE, audio VAE and schedulers load from your own copy of the base model. Inference touches 11 GB of its 269 GB:

hf download MiniMaxAI/MiniMax-H3 --local-dir ./MiniMax-H3 \
  --include "modular_model_index.json" "vae/*" "audio_vae/*" \
            "scheduler/*" "audio_scheduler/*" "assets/ref2va.mp4"

hf download Viggle/Viggle-Animate --local-dir ./Viggle-Animate

pip install torch "git+https://github.com/huggingface/diffusers@d6726f3" av

python Viggle-Animate/inference/sample.py \
  --model-dir ./MiniMax-H3 \
  --cond driving.mp4 --ref repainted_first_frame.png --out swapped.mp4

d6726f3 is the tested diffusers commit; the upstream minimax_h3 modular pipeline is enough, no fork or patch. The last --include is the clip examples/demo.sh needs.

One 80 GB card is not enough at bf16 — the transformer is 62 GiB resident and a 480×832 / 124-frame render peaks at 80.1 GiB allocated. Use a card with ≥ 96 GB, or pass --offload to stream blocks from CPU (~12 GB resident, much slower).

It also runs quantized on consumer hardware. We deploy it on a single RTX 5090 (32 GB): NVFP4 weights — 4.5 bits/param, dispatching to the real sm_120 cutlass block-scaled kernel, 2.70× bf16 per compiled linear — plus a low-rank adaln_proj and torch.compile. Quantization alone is not enough for 32 GB: 13.0 B of the 33.1 B parameters sit in adaln_proj, which the linear-layer quantizer does not touch. That deployment path is not shipped in this repository.

Defaults are the evaluated configuration: --steps 4 --flow-shift 3 --num-frames 124 (≈ 5.2 s at 24 fps) --seed 42. Output geometry follows the driving clip and must be a multiple of 32 on both axes. Weights load in ~21 s, once per process. Four steps is the operating point, not a shortcut — the distilled model already renders sharper than its teacher, and raising the step count tips that into over-sharpening. The driving clip's audio is dropped at input; the model emits its own track, and the fixed prompt asks for silence.

Make the reference with ffmpeg -i driving.mp4 -frames:v 1 first_frame.png, then edit that frame at the same resolution. Name the change, and pin down what must not change — pose, hands, props, framing, background, light. An editor that quietly reframes the shot will fight the driving motion. Prefer clips that keep one side to camera, and bind any new limb to a real one. examples/demo.sh reproduces the clip above and needs no media from you.

Citation

@misc{viggle2026animate,
  title  = {Viggle-Animate: Character Replacement in Video from a Single Repainted Frame},
  author = {Viggle Research},
  year   = {2026},
  url    = {https://huggingface.co/Viggle/Viggle-Animate}
}

License

The weights are a Model Derivative of MiniMax H3, so the MiniMax H3 Community License applies to them — read it before you redistribute them or ship a product on them. Our changes are listed in MODIFICATIONS.md.

The code in inference/ and examples/ is Apache 2.0 (LICENSE-CODE).

Music in the teaser at the top of this page: "Electrodoodle" by Kevin MacLeod (incompetech.com), licensed under Creative Commons: By Attribution 4.0.

Intended use

This model exists to put a consenting performer into footage they did not shoot, and it will just as readily put someone into footage they never agreed to appear in. Note where that decision is made: the identity comes from the frame you paint, so an image editor's safeguards are upstream of this model and none of them are in it. It cannot verify identity or consent. Do not run it on people who have not agreed to it, label what you generate as AI-generated, and see Section V.5 of the Agreement if you offer this as a service.

Powered by MiniMax H3.