通过人类视频视觉-物理对齐的空间感知VLA预训练
Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
从大规模人类演示视频中自动抽取3D视觉与3D动作标注，作为对齐2D观测与3D空间推理的监督信号；该「视觉-物理对齐预训练」思路可迁移到任何需要从2D视觉输入推断3D动作的视觉-语言-动作模型，为视觉表征注入3D感知特征。
原文依据 leverage large-scale human demonstration videos to extract 3D visual and 3D action annotations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉-语言-动作（VLA）模型通过将视觉感知与语言引导的策略学习相结合，为机器人学习提供了一种有前景的范式。然而，现有方法大多依赖2D视觉输入在3D物理环境中执行动作，造成感知与动作视觉定位之间的显著鸿沟。为弥合这一鸿沟，我们提出一种空间感知VLA预训练范式，使模型在机器人策略学习之前获得3D空间理解。从预训练的视觉-语言模型出发，我们利用大规模人类演示视频提取3D视觉与3D动作标注，形成一种新的监督来源，将2D视觉观测与3D空间推理对齐。我们以VIPA-VLA实例化该范式，这是一种双编码器架构，引入3D视觉编码器以3D感知特征增强语义视觉表征，并通过视觉-物理对齐预训练使二者对齐。当下游机器人任务适配时，VIPA-VLA显著改善了2D视觉与3D动作之间的视觉定位，从而得到更鲁棒、更具泛化性的机器人策略。
英文原文摘要 1161 字符
Vision-Language-Action (VLA) models provide a promising paradigm for robot learning by integrating visual perception with language-guided policy learning. However, most existing approaches rely on 2D visual inputs to perform actions in 3D physical environments, creating a significant gap between perception and action grounding. To bridge this gap, we propose a Spatial-Aware VLA Pretraining paradigm that enables models to acquire 3D spatial understanding before robot policy learning. Starting from pretrained vision-language models, we leverage large-scale human demonstration videos to extract 3D visual and 3D action annotations, forming a new source of supervision that aligns 2D visual observations with 3D spatial reasoning. We instantiate this paradigm with VIPA-VLA, a dual-encoder architecture that incorporates a 3D visual encoder to augment semantic visual representations with 3D-aware features, and aligns the two through visual-physical alignment pretraining. When adapted to downstream robot tasks, VIPA-VLA achieves significantly improved grounding between 2D vision and 3D action, resulting in more robust and generalizable robotic policies.