强化学习与具身智能 · CVPR2026 ·2026.06.03

StaMo:从紧凑状态表征中无监督学习可泛化的机器人运动

StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representation

Mingyu Liu、Jiuhe Shu、Hui Chen、Zeju Li、Canyu Zhao、Jiange Yang 等 9 人

与你有关的部分

用预训练 DiT 作为解码器,从静态图像编码出仅两个 token 的紧凑状态表征,并通过潜变量插值得到可直接解码为机器人动作的潜动作;对做 VLA、世界模型或从视频/图像中挖掘动作监督的视觉研究者,这是一个可复用的轻量表征与潜动作提取方案。

原文依据 the difference between these tokens, obtained via latent interpolation, naturally serves as a highly effective latent action
可迁移到 vision-language-actionlatent action learningworld modelrobot manipulation

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

具身智能中的一个根本性挑战是为高效的世界建模与决策构建富有表达力且紧凑的状态表征。然而,现有方法往往难以取得这种平衡,所得表征要么过度冗余,要么缺乏任务关键信息。我们提出一种无监督方法,利用轻量编码器与预训练的扩散 Transformer(DiT)解码器,借助其强大的生成先验,学习一种高度压缩的双 token 状态表征。我们的表征高效、可解释,并能无缝集成到现有的基于 VLA 的模型中,在 LIBERO 上性能提升 11.6%,在真实世界任务中成功率提升 31%,且推理开销极小。更重要的是,我们发现通过潜变量插值得到的这两个 token 之间的差异,自然构成了一种极为有效的潜动作,可进一步解码为可执行的机器人动作。这一涌现能力表明,我们的表征在无显式监督的情况下捕捉到了结构化的动力学。我们将方法命名为 StaMo,因其能够从紧凑的状态表征中学习可泛化的机器人运动,而该表征由静态图像编码而来,这挑战了当前普遍依赖复杂架构与视频数据来学习潜动作的做法。所得的潜动作还增强了策略协同训练,相较先前方法提升 10.4%,并具有更好的可解释性。此外,我们的方法可有效扩展到多种数据来源,包括真实世界机器人数据、仿真以及人类第一人称视频。

英文原文摘要 1623 字符

A fundamental challenge in embodied intelligence is developing expressive and compact state representations for efficient world modeling and decision making. However, existing methods often fail to achieve this balance, yielding representations that are either overly redundant or lacking in task-critical information. We propose an unsupervised approach that learns a highly compressed two-token state representation using a lightweight encoder and a pre-trained Diffusion Transformer (DiT) decoder, capitalizing on its strong generative prior. Our representation is efficient, interpretable, and integrates seamlessly into existing VLA-based models, improving performance by 11.6% on LIBERO and 31% in real-world task success with minimal inference overhead. More importantly, we find that the difference between these tokens, obtained via latent interpolation, naturally serves as a highly effective latent action, which can be further decoded into executable robot actions. This emergent capability reveals that our representation captures structured dynamics without explicit supervision. We name our method StaMo for its ability to learn generalizable robotic Motion from compact State representation, which is encoded from static images, challenging the prevalent dependence to learning latent action on complex architectures and video data. The resulting latent actions also enhance policy co-training, outperforming prior methods by 10.4% with improved interpretability. Moreover, our approach scales effectively across diverse data sources, including real-world robot data, simulation, and human egocentric video.

标签

会议 CVPR2026
页码 35014-35024
Bibkey
来源 CVF Open Access