面向双臂操作的动作-几何预测与3D几何先验
Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation
把预训练3D几何基础模型作为策略的感知骨干，用扩散模型联合预测动作块与未来3D latent（解码为稠密点图），为视觉研究者提供了一条仅用RGB观测就获得几何感知与场景演化预测的可复用路线，可迁移到其他需要空间推理的视觉-动作任务。
原文依据 uses a diffusion model to jointly predict an action chunk and a future 3D latent, which decodes into a dense point map
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论双臂操作要求策略能够对3D几何进行推理、预判其在动作作用下如何演化，并生成平滑且协调的运动。然而，现有方法通常依赖空间感知能力有限的2D特征，或需要难以在真实场景中可靠获取的显式点云。与此同时，近期的3D几何基础模型表明，可以从RGB图像中以快速且鲁棒的方式重建出准确而多样的3D结构。基于这些进展，我们提出一个直接构建在预训练3D几何基础模型之上的双臂操作框架。我们的策略将几何感知隐变量、2D语义特征与本体感知融合为统一的状态表示，并使用扩散模型联合预测动作块与未来的3D隐变量，后者可解码为稠密点图。通过显式预测3D场景将如何随动作序列一同演化，该策略仅使用RGB观测便获得了强大的空间理解与预测能力。我们在RoboTwin基准的仿真环境以及真实机器人执行中评估了该方法。我们的方法持续优于基于2D和基于点云的基线，在操作成功率、双臂协调性以及3D空间预测精度上均达到最优性能。代码见 https://github.com/Chongyang-99/GAP.git。
英文原文摘要 1452 字符
Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions. However, existing methods typically rely on 2D features with limited spatial awareness, or require explicit point clouds that are difficult to obtain reliably in real-world settings. Meanwhile, recent 3D geometric foundation models show that accurate and diverse 3D structure can be reconstructed directly from RGB images in a fast and robust manner. Building on these advances, we propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model. Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses a diffusion model to jointly predict an action chunk and a future 3D latent, which decodes into a dense point map. By explicitly predicting how the 3D scene will evolve together with the action sequence, the policy gains strong spatial understanding and predictive capability using only RGB observations. We evaluate our method both in simulation on the RoboTwin benchmark and in real-world robot executions. Our approach consistently outperforms 2D-based and point-cloud-based baselines, achieving state-of-the-art performance in manipulation success, inter-arm coordination, and 3D spatial prediction accuracy. Code is available at https://github.com/Chongyang-99/GAP.git.