arxiv:2602.10102

VideoWorld 2: Learning Transferable Knowledge from Real-world Videos

Published on Feb 10

· Submitted by

rzw on Feb 11

ByteDance Seed

Upvote

Authors:

Abstract

VideoWorld 2 enables transferable knowledge learning from raw videos through a dynamic-enhanced Latent Dynamics Model that decouples action dynamics from visual appearance, achieving improved task performance and long-horizon reasoning in real-world applications.

AI-generated summary

Learning transferable knowledge from unlabeled video data and applying it in new environments is a fundamental capability of intelligent agents. This work presents VideoWorld 2, which extends VideoWorld and offers the first investigation into learning transferable knowledge directly from raw real-world videos. At its core, VideoWorld 2 introduces a dynamic-enhanced Latent Dynamics Model (dLDM) that decouples action dynamics from visual appearance: a pretrained video diffusion model handles visual appearance modeling, enabling the dLDM to learn latent codes that focus on compact and meaningful task-related dynamics. These latent codes are then modeled autoregressively to learn task policies and support long-horizon reasoning. We evaluate VideoWorld 2 on challenging real-world handcraft making tasks, where prior video generation and latent-dynamics models struggle to operate reliably. Remarkably, VideoWorld 2 achieves up to 70% improvement in task success rate and produces coherent long execution videos. In robotics, we show that VideoWorld 2 can acquire effective manipulation knowledge from the Open-X dataset, which substantially improves task performance on CALVIN. This study reveals the potential of learning transferable world knowledge directly from raw videos, with all code, data, and models to be open-sourced for further research.

View arXiv page View PDF Project page GitHub 685 Add to collection

Community

maverickrzw

Paper submitter 1 day ago

•

edited 1 day ago

🤖Text is not enough, Visual is the key to AGI！Can Al learn transferable knowledge for complex tasks directly from videos? Just like a child learns to fold a paper airplane or build a LEGO from video tutorials👶

😎Thrilled to introduce VideoWorld 2, the successor of VideoWorld. Unlike Sora and Veo, it is the the first generative model that masters complex real-world knowledge solely through visual data, without any reliance on language models.

🙋 You might wonder: what knowledge remains out of reach for today’s AI? Try asking Sora 2 or Veo 3 to fold a coherent paper boat, or have Gemini describe every micro-fold and material change in text. Although any child can master this skill just by watching video tutorials, today's most advanced AI often fails at such tasks.

🚀To address this challenge, we propose VideoWorld 2. Unlike models that rely on language priors, it is the first to master complex, long-horizon real-world knowledge solely by "watching" raw videos and generalizing the skill to new environments.

🧑‍🏫 The "Cambrian Moment" for AI? As Dr. Feifei Li noted, vision-enabled perception and planning triggered the Cambrian Explosion 540 million years ago. VideoWorld 2 explores this frontier: Without any textual descriptions, it completes minute-long handcraft tasks like paper folding and block-building, which involve fine-grained manipulation and long-horizon planning that current AI fails to learn. Furthermore, it can generalize these skills across various unseen scenes and perform multi-task, cross-environment robotic manipulation.

Our main contributions are:
👉We explore, for the first time, how to learn complex, long-range skills from raw videos and generalize them to new environments. We find that disentangling visual appearance from core dynamics is the key to mastering world knowledge.

👉We propose VideoWorld 2, leveraging a dynamic-enhanced Latent Dynamic Model to extract task-relevant dynamics to boost long-horizon tasks success rates by up to 70%

👉We construct Video-CraftBench, a large-scale video-based handcraft dataset for training and evaluation, facilitating future research on knowledge learning from pure videos.