MM-ACT：从多模态并行生成中学习动作
MM-ACT: Learn from Multimodal Parallel Generation to Act
把文本、图像、动作放进共享词元空间并用 re-mask 并行解码统一生成，为视觉研究者提供了一个可复用的 VLA 架构范式：动作生成可借用扩散/掩码生成的并行解码思路，且跨模态监督能反哺动作预测精度。
原文依据 MM-ACT adopts a re-mask parallel decoding strategy for text and image generation, and employs a one-step parallel decoding strategy for action generation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论通用机器人策略既需要对任务规划进行语义理解，也需要通过预测能力与环境交互。为此，我们提出 MM-ACT，一个统一的视觉-语言-动作（VLA）模型，它在共享词元空间中整合文本、图像与动作，并对全部三种模态执行生成。MM-ACT 对文本与图像生成采用重掩码并行解码策略，对动作生成采用一步并行解码策略以提升效率。我们提出上下文共享多模态学习，这是一种统一的训练范式，从共享上下文对全部三种模态的生成进行监督，通过跨模态学习增强动作生成。实验分别在 LIBERO 仿真环境、Franka 真实机器人平台以及 RoboTwin2.0 上进行，以评估域内与域外性能。我们的方法在 LIBERO 上取得 96.3% 的成功率，在真实 Franka 的三个任务上取得 72.0% 的成功率，在 RoboTwin2.0 的八个双臂任务上取得 52.38% 的成功率，并从跨模态学习中获得额外 9.25% 的提升。我们已在 https://github.com/HHYHRHY/MM-ACT 发布代码、模型与数据。
英文原文摘要 1199 字符
A generalist robotic policy needs both semantic understanding for task planning and the ability to interact with the environment through predictive capabilities. To tackle this, we present MM-ACT, a unified Vision-Language-Action (VLA) model that integrates text, image, and action in shared token space and performs generation across all three modalities. MM-ACT adopts a re-mask parallel decoding strategy for text and image generation, and employs a one-step parallel decoding strategy for action generation to improve efficiency. We introduce Context-Shared Multimodal Learning, a unified training paradigm that supervises generation in all three modalities from a shared context, enhancing action generation through cross-modal learning. Experiments were conducted on the LIBERO simulation and Franka real-robot setups as well as RoboTwin2.0 to assess in-domain and out-of-domain performances respectively. Our approach achieves a success rate of 96.3% on LIBERO, 72.0% across three tasks of real Franka, and 52.38% across eight bimanual tasks of RoboTwin2.0 with an additional gain of 9.25% from cross-modal learning. We release our codes, models and data at https://github.com/HHYHRHY/MM-ACT.