Mantis：一种具有解耦视觉预见的通用视觉-语言-动作模型
Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresight
把视觉预见预测从 VLA 骨干网络中解耦，用元查询加 DiT 头在残差连接下做下一状态预测，从而隐式捕获潜动作；这一「解耦式视觉预见」结构可直接迁移到任何需要视觉前瞻辅助动作学习的具身策略中。
原文依据 Mantis decouples visual foresight prediction from the backbone with the combination of meta queries and a diffusion Transformer (DiT) head.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期视觉-语言-动作（VLA）模型的进展表明，视觉信号能够有效补充稀疏的动作监督。然而，让 VLA 直接预测高维视觉状态会分散模型容量并带来高昂的训练成本，而将视觉状态压缩为更紧凑的监督信号又不可避免地引入信息瓶颈。此外，现有方法往往因忽视语言监督而导致理解与推理能力不足。本文提出 Mantis，一个以解耦视觉预见（DVF）为核心的新框架来解决上述问题。具体而言，Mantis 通过元查询与扩散 Transformer（DiT）头的组合，将视觉预见预测从骨干网络中解耦。在通过残差连接将当前视觉状态提供给 DiT 的情况下，一个简单的下一状态预测目标使元查询能够自动捕获刻画视觉轨迹的潜动作，从而促进显式动作的学习。这种解耦减轻了 VLA 骨干网络的负担，使其能够通过语言监督保持理解与推理能力。在实验中，Mantis 在人类操作视频、机器人演示与图文对上预训练，经微调后在 LIBERO 基准上达到 96.5% 的成功率，超越强基线并展现出较快的收敛速度。真实世界评估显示，Mantis 优于领先的开源 VLA 模型 π_0.5，尤其在指令遵循能力、对未见指令的泛化以及推理能力方面表现突出。我们还引入自适应时序集成（ATE）策略，以在推理过程中平衡计算效率与运动稳定性，得到 Mantis-ATE 变体，在保持性能的同时将推理次数减少 45%。我们已将代码与模型权重开源发布，地址为 https://github.com/SJTU-DENG-Lab/Mantis。
英文原文摘要 2061 字符
Recent advances in Vision-Language-Action (VLA) models demonstrate that visual signals can effectively complement sparse action supervisions. However, letting VLA directly predict high-dimensional visual states can distribute model capacity and incur prohibitive training cost, while compressing visual states into more compact supervisory signals inevitably incurs information bottlenecks. Moreover, existing methods often suffer from poor comprehension and reasoning capabilities due to the neglect of language supervision. This paper introduces Mantis, a novel framework featuring a Disentangled Visual Foresight (DVF) to tackle these issues. Specifically, Mantis decouples visual foresight prediction from the backbone with the combination of meta queries and a diffusion Transformer (DiT) head. With the current visual state provided to the DiT via a residual connection, a simple next-state prediction objective enables the meta queries to automatically capture the latent actions that delineate the visual trajectory, and hence boost the learning of explicit actions. The disentanglement reduces the burden of the VLA backbone, enabling it to maintain comprehension and reasoning capabilities through language supervision. Empirically, pretrained on human manipulation videos, robot demonstrations, and image-text pairs, Mantis achieves a 96.5% success rate on LIBERO benchmark after fine-tuning, surpassing powerful baselines while exhibiting high convergence speed. Real-world evaluations show that Mantis outperforms \pi_ 0.5 , a leading open-source VLA model, particularly in instruction-following capability, generalization to unseen instructions, and reasoning ability. We also introduce the adaptive temporal ensemble (ATE) strategy to balance computational efficiency and motion stability during inference, yielding the Mantis-ATE variant, which reduces inference counts by 45% while maintaining performance. We have released the code and model weights to the open-source community, which are available at https://github.com/SJTU-DENG-Lab/Mantis.