SpaceMind：面向视觉语言模型空间推理的相机引导模态融合
SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models
把相机表示当作主动引导模态、在语言模型前用轻量融合模块对空间 token 做相机条件化偏置与门控，这一思路可直接迁移到任何需要几何 grounding 的 RGB-only VLM 中，替代浅层特征拼接。
原文依据 treat the camera representation as an active guiding modality rather than passive metadata
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（VLM）展现出强大的多模态理解能力，但在距离估计、尺寸比较和跨视角一致性等 3D 空间推理任务上仍存在困难。现有的 3D 感知方法要么依赖辅助 3D 信息，要么通过浅层特征融合为仅使用 RGB 的 VLM 增强几何编码器。我们提出 SpaceMind，一个在 SpaceMind-700K 上训练、从 RGB 输入进行空间推理的多模态大语言模型。该模型采用双编码器架构，集成 VGGT 作为空间理解编码器、InternViT 作为 2D 视觉编码器。其核心思想是将相机表示视为主动的引导模态，而非被动的元数据。具体而言，SpaceMind 在语言模型之前引入轻量的相机引导模态融合模块，以取代浅层融合。该模块对空间 token 施加相机条件化偏置，为其分配反映几何重要性的与查询无关的权重，并利用相机嵌入对融合后的表示进行门控。实验上，SpaceMind 在 VSI-Bench、SQA3D 和 SPBench 上取得新的最优结果，在 VSI-Bench 和 SPBench 上以较大幅度超越开源与专有系统，并在 SQA3D 上达到最优性能。这些结果表明，相机引导模态融合是一种有效且实用的归纳偏置，可为 VLM 赋予真正具有空间 grounding 的智能。我们将发布代码与模型检查点以支持后续研究。
英文原文摘要 1521 字符
Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning, such as distance estimation, size comparison, and cross-view consistency. Existing 3D-aware methods either depend on auxiliary 3D information or enhance RGB-only VLMs with geometry encoders through shallow feature fusion. We propose SpaceMind, a multimodal large language model trained on SpaceMind-700K for spatial reasoning from RGB inputs. The model adopts a dual-encoder architecture integrating VGGT as a spatial understanding encoder and InternViT as a 2D visual encoder. The key idea is to treat the camera representation as an active guiding modality rather than passive metadata. Specifically, SpaceMind introduces a lightweight Camera-Guided Modality Fusion module before the language model to replace shallow fusion. It applies camera-conditioned biasing to spatial tokens, assigns query-independent weights reflecting their geometric importance, and uses the camera embedding to gate the fused representation. Empirically, SpaceMind establishes new state-of-the-art results on VSI-Bench, SQA3D and SPBench, surpassing both open and proprietary systems on VSI-Bench and SPBench by large margins and achieving state-of-the-art performance on SQA3D. These results demonstrate that camera-guided modality fusion is an effective and practical inductive bias for equipping VLMs with genuinely spatially grounded intelligence. We will release code and model checkpoints to support future research.