👏 CLAP: Cross-Embodiment Action-Conditioned Video World Models are Zero-Shot Physical Simulators

CLAP is a cross-embodiment, action-conditioned video generation framework that unifies disparate human and robot action spaces via end-effector poses, language, and latent actions. This repo hosts the pretrained/post-trained/adapted CLAP checkpoints; training code, configs, and full usage instructions live in the GitHub repo.

• Two-Stage Curriculum: Learns unsupervised physical priors from unlabeled video, then grounds them in end-effector space for zero-shot deployment.
• Sample-Efficient Adaptation: Facilitates few-shot transfer to target robots, matching or beating single-embodiment baselines like DROID (without post-training) and Bridge (with post-training).
• Zero-Shot Real-World Generalization: Generalizes out-of-the-box to power real-world inference planning and RL finetuning with robot policies like π₀.₅ and MolmoAct-2.

✨ Conditioning spaces

conditioning	Action representation	Typical use
🦾 "ee"	7-dim end-effector cartesian pose + gripper (or raw joint angles for joint-space embodiments)	Cross-embodiment, post-training, adaptation
🧬 "lam"	32-dim latent-action-model embedding (learned, not physical)	Curriculum cross-embodiment training, including egocentric human video
💬 "language"	Per-frame CLIP-encoded text captions of the action	Language-conditioned, post-training, adaptation

🗂️ Checkpoints

Each subfolder below is a complete checkpoint (loadable directly via CLAPModel.from_pretrained, see the GitHub repo for the clap package and CLI entrypoints).

🌐 Cross-embodiment models

Trained for 100K steps on a mix of OXE datasets (fractal/fmb/bc_z/taco_play/furniture_bench/bridge/droid, egodex too for lam). All ee-conditioned models use absolute actions, while the lang-conditioned model uses relative-to-anchor-frame language captions.

Model name	Conditioning	Use case
clap-curr	ee, continued from a LAM-pretrained checkpoint	Default — getting-started default; base checkpoint for novel-embodiment adaptation
clap-ee	ee — 7-dim end-effector cartesian	Cross-embodiment world model, EE-only (no LAM pretraining stage)
clap-lam	lam — 32-dim latent action	Curriculum LAM pretraining stage
clap-lang	language, relative-to-anchor-frame captions	Language-conditioned generation/eval

🎯 Post-trained on a single robot platform

Each finetunes the matching cross-embodiment checkpoint above onto one dataset alone (still ee-conditioned regardless of the base's own conditioning).

DROID

Model name	Conditioning	Use case
clap-curr-droid	ee — 7-dim EE cartesian	Post-trained on droid, from the clap-curr base
clap-ee-droid	ee — 7-dim EE cartesian	Post-trained on droid, from the clap-ee base
clap-lam-droid	ee — 7-dim EE cartesian	Post-trained on droid, from the clap-lam base
clap-lang-droid	ee — 7-dim EE cartesian	Post-trained on droid, from the clap-lang base

Bridge

Model name	Conditioning	Use case
clap-curr-bridge	ee — 7-dim EE cartesian	Post-trained on bridge, from the clap-curr base
clap-ee-bridge	ee — 7-dim EE cartesian	Post-trained on bridge, from the clap-ee base
clap-lam-bridge	ee — 7-dim EE cartesian	Post-trained on bridge, from the clap-lam base
clap-lang-bridge	ee — 7-dim EE cartesian	Post-trained on bridge, from the clap-lang base

🆕 Novel-embodiment adaptation

Finetuned on data for a new robot embodiment, building on physical priors from the cross-embodiment models (defaults to clap-curr).

Model name	Conditioning	Use case
adapt-yam	ee, 14-dim joint-space	Novel-embodiment adaptation target: bimanual_yam
adapt-g1	ee, 26-dim joint-space	Novel-embodiment adaptation target: g1_humanoid

🚀 Usage

Install the clap package from GitHub, then point any checkpoint path below at a --ckpt/checkpoint arg (see the GitHub repo's Getting started for clap-rollout-replay/clap-teleop/clap-rollout-deploy/clap-eval):

git clone https://github.com/omni-CLAP/clap.git && cd clap
uv venv .venv/clap --python 3.11 && source .venv/clap/bin/activate
uv pip install -e .

# download a checkpoint from this repo, e.g.:
hf download omni-CLAP/CLAP --local-dir ./checkpoints --include "cross_embodiment_oxe_curriculum_lam_ee/*"

📖 Glossary

Term	Meaning
EE	End-effector — the robot's gripper/hand; ee-conditioning uses its 7-dim cartesian pose (position + orientation) + gripper, as opposed to raw joint angles
LAM	Latent Action Model — a learned, unsupervised action representation extracted directly from video (not a physical quantity), letting CLAP pretrain on video with no paired action labels, including egocentric human video
OXE	Open X-Embodiment — the multi-robot, multi-dataset collection CLAP's cross-embodiment training draws from (fractal/fmb/bc_z/taco_play/furniture_bench/bridge/droid, egodex for lam)
DROID	A large-scale real-world Franka-arm manipulation dataset/platform; one of CLAP's core cross-embodiment/post-training targets
bimanual YAM / bimanual_yam	A dual-arm robot platform; one of CLAP's novel-embodiment adaptation targets (14-dim joint-space action)
G1 / g1_humanoid	Unitree G1, a bipedal humanoid robot (2 arms + 2 hands); the other novel-embodiment adaptation target (26-dim joint-space action)
SVD	Stable Video Diffusion — the pretrained video diffusion backbone CLAPModel's U-Net is built on

⚖️ License

MIT — see LICENSE.