VideoWorld 2：从真实世界视频中学习可迁移知识
VideoWorld 2: Learning Transferable Knowledge from Real-world Videos
用预训练视频扩散模型承担视觉外观建模、把动作动力学剥离到紧凑潜变量码上再自回归建模，这一「外观-动力学解耦」思路可直接迁移到机器人操作策略学习与长时序视频预测中，也提示视觉研究者可把扩散模型当作可复用的外观先验而非端到端生成器。
原文依据 a pretrained video diffusion model handles visual appearance modeling, enabling the dLDM to learn latent codes that focus on compact and meaningful task-related dynamics
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从无标注视频数据中学习可迁移知识并将其应用于新环境，是智能体的基本能力。本工作提出 VideoWorld 2，它在 VideoWorld 的基础上进行扩展，首次研究了如何直接从原始真实世界视频中为复杂、长时序任务学习可迁移知识。VideoWorld 2 的核心是引入动力学增强的潜变量动力学模型（dLDM），该模型将动作动力学与视觉外观解耦：由预训练视频扩散模型负责视觉外观建模，使 dLDM 能够学习专注于紧凑且有意义的任务相关动力学的潜变量码。随后对这些潜变量码进行自回归建模，以学习任务策略并支持长时序推理。我们在具有挑战性的真实世界手工制作任务上评估 VideoWorld 2，此前视频生成模型与潜变量动力学模型在这些任务上难以稳定工作。值得注意的是，VideoWorld 2 将任务成功率最高提升 70%，并能生成连贯的长时执行视频。在机器人领域，我们表明 VideoWorld 2 能够从 Open-X 数据集中获取可迁移的操作知识，显著提升在 CALVIN 上的任务表现，展现出强大的跨域泛化能力。本研究揭示了直接从原始视频中学习可迁移世界知识的潜力，所有代码、数据与模型均已开源以促进后续研究。
英文原文摘要 1436 字符
Learning transferable knowledge from unlabeled video data and applying it in new environments is a fundamental capability of intelligent agents. This work presents VideoWorld 2, which extends VideoWorld and provides the first investigation of learning transferable knowledge for complex, long-horizon tasks directly from raw real-world videos. At its core, VideoWorld 2 introduces a dynamics-enhanced Latent Dynamics Model (dLDM) that decouples action dynamics from visual appearance: a pretrained video diffusion model handles visual appearance modeling, enabling the dLDM to learn latent codes that focus on compact and meaningful task-related dynamics. These latent codes are then modeled autoregressively to learn task policies and support long-horizon reasoning. We evaluate VideoWorld 2 on challenging real-world handcraft making tasks, where prior video generation and latent-dynamics models struggle to operate reliably. Remarkably, VideoWorld 2 achieves up to 70% improvement in task success rate and produces coherent long execution videos. In robotics, we show that VideoWorld 2 can acquire transferable manipulation knowledge from the Open-X dataset, which substantially improves task performance on CALVIN, demonstrating strong cross-domain generalization. This study reveals the potential of learning transferable world knowledge directly from raw videos, with all code, data, and models open-sourced for further research.