S$^2$-MLLM：以结构引导提升多模态大语言模型空间推理能力用于3D视觉定位
S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance
用前馈三维重建的结构感知作为隐式空间引导，替代逐视角点云渲染，让 MLLM 在不重建点云的情况下获得 3D 空间推理能力；其 intra/inter-view 注意力加多级位置编码的结构增强模块可直接迁移到任何需要把多视角 2D 特征对齐到 3D 空间的 VLM 任务。
原文依据 we propose S^2-MLLM, an efficient framework that enhances spatial reasoning in MLLMs through implicit spatial reasoning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论3D视觉定位（3DVG）旨在根据自然语言描述在三维场景中定位物体，是具身智能与机器人领域的基础任务。多模态大语言模型（MLLM）的最新进展推动了将其扩展到3DVG的研究。然而，MLLM 主要处理二维视觉输入，仅凭这些有限视角难以理解场景的三维空间结构。现有方法主要利用重建点云的视角相关渲染，为 MLLM 在 3DVG 任务中提供显式的结构引导，导致效率低下且空间推理能力受限。为解决该问题，我们提出 S^2-MLLM，一个通过隐式空间推理增强 MLLM 空间推理能力的高效框架。我们引入一种空间引导策略，利用前馈三维重建的结构感知能力。通过在训练中获得三维结构理解，模型无需依赖低效的点云重建即可隐式推理三维场景。此外，我们提出结构增强模块（SE），该模块首先采用视图内与视图间注意力机制，捕获视图内的依赖关系与跨视图的对应关系。该模块进一步整合多级位置编码，将视觉表示与空间位置及视角信息关联，从而实现更准确的结构理解。大量实验表明，S^2-MLLM 统一了优越的性能、泛化性与效率，在 ScanRefer、Nr3D 和 Sr3D 数据集上相较现有方法取得显著性能提升。代码将在论文录用后公开。
英文原文摘要 1711 字符
3D Visual Grounding (3DVG) focuses on locating objects in 3D scenes based on natural language descriptions, serving as a fundamental task for embodied AI and robotics. Recent advances in Multi-modal Large Language Models (MLLMs) have motivated research into extending them to 3DVG. However, MLLMs primarily process 2D visual inputs and struggle with understanding 3D spatial structure of scenes solely from these limited perspectives. Existing methods mainly utilize viewpoint-dependent rendering of reconstructed point clouds to provide explicit structural guidance for MLLMs in 3DVG tasks, leading to inefficiency and limited spatial reasoning. To address this issue, we propose S^2-MLLM, an efficient framework that enhances spatial reasoning in MLLMs through implicit spatial reasoning. We introduce a spatial guidance strategy that leverages the structure awareness of feed-forward 3D reconstruction. By acquiring 3D structural understanding during training, our model can implicitly reason about 3D scenes without relying on inefficient point cloud reconstruction. Moreover, we propose a structure-enhanced module (SE), which first employs intra-view and inter-view attention mechanisms to capture dependencies within views and correspondences across views. The module further integrates multi-level position encoding to associate visual representations with spatial positions and viewpoint information, enabling more accurate structural understanding. Extensive experiments demonstrate that S^2-MLLM unifies superior performance, generalization, and efficiency, achieving significant performance over existing methods across the ScanRefer, Nr3D, and Sr3D datasets. Code will be available upon acceptance.