G$^2$VLM：统一三维重建与空间推理的几何基础视觉语言模型
G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
把前馈三维重建模型学到的几何特征直接接入 VLM 的表征空间，用交错推理与上下文学习让语言模型「读懂」几何特征；做空间理解或多模态三维任务的可以借鉴这种统一训练范式，用易得的多视图/视频数据替代难采集的三维标注。
原文依据 G^2VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在空间智能方面仍缺乏鲁棒性，在空间理解与推理任务上表现不佳。我们将这一差距归因于缺少一个能够从二维图像重建三维空间的视觉几何学习过程。我们提出 G^2VLM，一个几何基础的视觉语言模型，它连接了空间智能的两个基本方面：空间三维重建与空间理解。G^2VLM 原生地利用学习到的三维视觉几何特征直接预测三维属性，并通过上下文学习与交错推理增强空间推理任务。我们的统一设计在空间理解方面具有高度可扩展性：它在丰富的多视图图像与视频数据上训练，同时受益于通常只能从难以采集的标注中获得的三维视觉先验。实验结果表明，G^2VLM 在这两项任务上均表现出色，取得了与最优前馈三维重建模型相当的结果，并在空间理解与推理任务上取得更好或具有竞争力的结果。通过将语义能力强的 VLM 与低层三维视觉任务统一起来，我们希望 G^2VLM 能成为社区的强基线，并开启更多未来应用，例如三维场景编辑。
英文原文摘要 1334 字符
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G^2VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G^2VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations.Experimental results demonstrate G^2VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks.By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G^2VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.