强化学习与具身智能 · CVPR2026 ·2026.06.03

Motus:统一的潜动作世界模型

Motus: A Unified Latent Action World Model

Hongzhe Bi、Hengkai Tan、Shenghao Xie、Zeyuan Wang、Shuhe Huang、Haitian Liu 等 16 人

与你有关的部分

把光流当作像素级「delta action」来学习潜动作,并用 UniDiffuser 式调度器在同一个 MoT 主干里切换世界模型、VLA、逆动力学与视频生成模式,为视觉研究者提供了「用视频生成先验做机器人动作预训练」的可复用范式。

原文依据 Motus further leverages the optical flow to learn latent actions and adopts a recipe with three-phase training pipeline and six-layer data pyramid
可迁移到 机器人操作策略学习视频生成视觉-语言-动作建模逆动力学

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

通用具身智能体必须作为一个统一系统运行,但现有方法却建立在理解、世界建模与控制相互割裂的模型之上。这种割裂阻碍了多模态生成能力的统一,也妨碍了从大规模异构数据中学习。本文提出 Motus,一个统一的潜动作世界模型,它利用现有的通用预训练模型与丰富且可共享的运动信息。Motus 引入混合 Transformer(MoT)架构来整合三个专家(即理解、视频生成与动作),并采用 UniDiffuser 式调度器,从而在不同建模范式(即世界模型、视觉-语言-动作模型、逆动力学模型、视频生成模型以及视频-动作联合预测)之间灵活切换。Motus 进一步利用光流学习潜动作,并采用包含三阶段训练流程与六层数据金字塔的方案,从而提取像素级的「delta action」并实现大规模动作预训练。实验表明,Motus 在仿真场景(相比 X-VLA 提升 15%,相比 Pi-0.5 提升 45%)与真实场景(提升 11%–48%)中均优于最优方法,证明对所有功能与先验的统一建模能显著受益于下游机器人任务。

英文原文摘要 1401 字符

While a general embodied agent must function as a unified system, current methods are built on isolated models for understanding, world modeling, and control. This fragmentation prevents unifying multimodal generative capabilities and hinders learning from large-scale, heterogeneous data. In this paper, we propose Motus, a unified latent action world model that leverages existing general pretrained models and rich, sharable motion information. Motus introduces a Mixture-of-Transformer (MoT) architecture to integrate three experts (ie, understanding, video generation, and action) and adopts a UniDiffuser-style scheduler to enable flexible switching between different modeling modes (ie, world models, vision-language-action models, inverse dynamics models, video generation models, and video-action joint prediction). Motus further leverages the optical flow to learn latent actions and adopts a recipe with three-phase training pipeline and six-layer data pyramid, thereby extracting pixel-level "delta action" and enabling large-scale action pretraining. Experiments show that Motus achieves superior performance against state-of-the-art methods in both simulation (a +15% improvement over X-VLA and a +45% improvement over Pi-0.5) and real-world scenarios(improved by +11 48%), demonstrating unified modeling of all functionalities and priors significantly benefits downstream robotic tasks.

标签

会议 CVPR2026
页码 35101-35113
Bibkey
来源 CVF Open Access