SPEAR-1：通过 3D 理解超越机器人演示数据规模
SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding
将 2D 图像-语言预训练的 VLM 通过 3D 标注数据增强为能单图预测物体 3D 坐标的 3D 感知 VLM，并作为机器人基础模型的感知前端，为视觉研究者提供了用廉价非机器人数据注入 3D 理解以提升具身控制可靠性的可复用范式。
原文依据 we propose to enrich easy-to-collect non-robotic image data with 3D annotations and enhance a pretrained VLM with 3D understanding capabilities
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论机器人基础模型（RFMs）作为通用、端到端的机器人控制系统具有巨大潜力。然而，它们在新环境、新任务和新本体上的泛化能力仍然有限。我们认为主要瓶颈在于其基础：大多数 RFMs 是通过微调互联网预训练的视觉-语言模型（VLMs）构建的。然而，这些 VLMs 是在 2D 图像-语言任务上训练的，缺乏在 3D 世界中进行具身控制所固有的 3D 空间推理能力。直接用大规模机器人数据弥合这一差距成本高昂且难以扩展。相反，我们提出用 3D 标注丰富易于收集的非机器人图像数据，并增强预训练 VLM 的 3D 理解能力。遵循这一策略，我们训练了 SPEAR-VLM，一个 3D 感知的 VLM，能够从单张 2D 图像推断物体在 3D 空间中的坐标。基于 SPEAR-VLM，我们引入了主要贡献 SPEAR-1：一个将接地的 3D 感知与语言指令的具身控制相结合的机器人基础模型。在来自 24 个 Open X-Embodiment 数据集的约 4500 万帧上训练，SPEAR-1 在机器人演示数据少 20 倍的情况下，性能优于或匹配最先进的模型，如 π0-FAST 和 π0.5。这一精心设计的训练策略解锁了新的 VLM 能力，从而将具身控制的可靠性提升到仅用机器人数据无法达到的水平。我们公开模型权重和 3D 标注数据集。
英文原文摘要 1489 字符
Robotic Foundation Models (RFMs) hold great promise as generalist, end-to-end systems for robot control.Yet their ability to generalize across new environments, tasks, and embodiments remains limited.We argue that a major bottleneck lies in their foundations: most RFMs are built by fine-tuning internet-pretrained Vision-Language Models (VLMs).However, these VLMs are trained on 2D image-language tasks and lack the 3D spatial reasoning inherently required for embodied control in the 3D world.Bridging this gap directly with large-scale robotic data is costly and difficult to scale.Instead, we propose to enrich easy-to-collect non-robotic image data with 3D annotations and enhance a pretrained VLM with 3D understanding capabilities.Following this strategy, we train SPEAR-VLM, a 3D-aware VLM that infers object coordinates in 3D space from a single 2D image.Building on SPEAR-VLM, we introduce our main contribution, SPEAR-1: a robotic foundation model that integrates grounded 3D perception with language-instructed embodied control.Trained on ~45M frames from 24 Open X-Embodiment datasets, SPEAR-1 outperforms or matches state-of-the-art models such as \pi_0-FAST and \pi_ 0.5 , while it uses 20xfewer robot demonstrations.This carefully-engineered training strategy unlocks new VLM capabilities and as a consequence boosts the reliability of embodied control beyond what is achievable with only robotic data.We make our model weights and 3D-annotated datasets publicly available.