面向视觉语言模型空间智能的抽象3D感知
Abstract 3D Perception for Spatial Intelligence in Vision-Language Models
通过抽象边界框将3D几何结构编码给VLM，无需额外训练即可提升空间推理能力，为视觉语言模型中的3D理解提供了一种可复用的零样本增强手段。
原文依据 we introduce SandboxVLM, a simple yet effective framework that leverages abstract bounding boxes to encode geometric structure for VLMs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在空间认知和物理理解等3D相关任务上表现不佳，而这些任务对于机器人和具身智能体等现实应用至关重要。我们将此归因于3D任务与VLM的2D训练之间的模态鸿沟，这导致从2D输入中检索3D信息的效率低下。为弥合这一鸿沟，我们提出了SandboxVLM，一个简单而有效的框架，利用抽象边界框为VLM编码几何结构。具体而言，我们设计了一个包含四个阶段的3D沙盒重建与感知流程：生成具有抽象控制的多视图先验、代理高程、多视图投票与聚类，以及3D感知推理。在零样本设置下跨多个基准和VLM骨干进行评估，我们的方法持续提升了空间智能，例如在SAT Real上相比基线方法取得了8.3%的提升。这些结果表明，向VLM呈现3D抽象可以在无需额外训练的情况下显著增强其3D推理能力，为通用具身智能开辟了新的可能性。
英文原文摘要 1166 字符
Vision-language models (VLMs) struggle with 3D-related tasks such as spatial cognition and physical understanding, which are crucial for real-world applications like robotics and embodied agents. We attribute this to a modality gap between the 3D tasks and the 2D training of VLM, which led to inefficient retrieval of 3D information from 2D input. To bridge this gap, we introduce SandboxVLM, a simple yet effective framework that leverages abstract bounding boxes to encode geometric structure for VLMs. Specifically, we design a 3D Sandbox reconstruction and perception pipeline comprising four stages: generating multi-view priors with abstract control, proxy elevation, multi-view voting and clustering, and 3D-aware reasoning. Evaluated in zero-shot settings across multiple benchmarks and VLM backbones, our approach consistently improves spatial intelligence, achieving an 8.3% gain on SAT Real compared with baseline methods for instance. These results demonstrate that presenting VLMs with 3D abstractions substantially enhances their 3D reasoning ability without additional training, suggesting new possibilities for general-purpose embodied intelligence.