OzzyGT/motionbricks_soma_diffusers

MotionBricks SOMA

The start of a Kimodo run, then the 30 frames generated with the code below, then the standing end of another Kimodo take, placed 1.4 m ahead. The generated frames are drawn on SOMA77 with a relaxed finger pose. At normal playback speed; the cameras follow the root.

This is the SOMA model of nvidia/MotionBricks, repacked into the Diffusers format in float32. It runs as a Modular Diffusers pipeline with the MotionBricks SOMA custom blocks and generates the transition from the last poses of one motion to target poses in one forward pass, choosing the length of the transition itself.

The three components work together:

• root_model (40M parameters) chooses the length and predicts the root trajectory.
• pose_model (137M parameters) predicts the pose codes of every token of four frames.
• pose_decoder (12M parameters) decodes the codes to frames.

Note: You need diffusers>=0.41.0.

Usage

The custom blocks load the model code shipped in this repository, hence trust_remote_code and trust_components_code.

Note: both flags run Python code downloaded from the Hub: trust_remote_code=True runs the code of the blocks repository, and trust_components_code=True runs the code shipped in the components_repo. Only use them with repositories you trust, and review their code before passing different ones.

import numpy as np
import torch
from diffusers import ModularPipelineBlocks
from huggingface_hub import hf_hub_download

blocks = ModularPipelineBlocks.from_pretrained(
    "OzzyGT/motionbricks_soma_custom_blocks",
    trust_remote_code=True,
    components_repo="OzzyGT/motionbricks_soma_diffusers",
    trust_components_code=True,
)
pipe = blocks.init_pipeline()
pipe.load_components(dtype=torch.float32)
pipe.to("cuda")

first = hf_hub_download("OzzyGT/diffusers-examples", "motionbricks_soma_diffusers/first.npz", repo_type="dataset")
second = hf_hub_download("OzzyGT/diffusers-examples", "motionbricks_soma_diffusers/second.npz", repo_type="dataset")
context = dict(np.load(first))
target = dict(np.load(second))

motion = pipe(context=context, target=target, target_offset=(1.4, 0.0), num_target_poses=4, output="motion")[0]
np.savez_compressed("transition.npz", **{key: value.cpu().numpy() for key, value in motion.items()})

Both clips come from Kimodo SOMA Modular: the start of a run, and a standing pose that target_offset places 1.4 m ahead of it. The model accepts SOMA30 or SOMA77 poses at 30 FPS and returns a SOMA30 motion; here it brings the run to a stop in 30 frames. See the custom blocks for the options.

Float32 reproduces NVIDIA's ONNX graph with PyTorch's default settings; float16 halves the memory with small drift, while bfloat16 changes the motion noticeably.

Licenses

The model weights are licensed under the NVIDIA Open Model Agreement, the same as the original. The model code in this repository is Apache-2.0. See NOTICE for what was changed.