UZ3DVG:基于生成语言条件的无辅助零样本 3D 视觉定位
UZ3DVG: Unaided Zero-Shot 3D Visual Grounding with Generated Language Conditions
用 VLM 离线生成物体级空间伪标签与推理链,再蒸馏到仅吃点云的轻量 3DVG 学生网络,配合 GeoSM 对齐文本推理与 3D 几何;这条「大模型造标签 + 蒸馏 + 推理时零外部依赖」的路线可直接迁移到其他需要空间推理的 3D 任务。
原文依据 a VLM is employed solely to produce object-wise descriptions (pseudo labels) and reasoning chains for training a lightweight 3DVG model
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论零样本 3D 视觉定位(Zero-Shot 3DVG)旨在不依赖实例级描述标注的情况下,根据自然语言描述在 3D 场景中定位目标物体。现有方法在推理阶段依赖额外的 2D 图像,和/或需要与大语言模型(LLM)或视觉语言模型(VLM)进行多轮交互,这增加了延迟、计算成本与部署复杂度。为克服这些局限,我们提出基于生成语言条件的无辅助零样本 3D 视觉定位方法(UZ3DVG),其在推理阶段仅输入 3D 点云与文本描述,且不依赖外部模型。这是一种新的训练范式:VLM 仅用于生成物体级描述(伪标签)与推理链,以训练一个具备鲁棒空间推理能力的轻量 3DVG 模型。具体而言,所引入的开放词表多源空间标注与推理链生成器处理来自开放世界场景的 RGB-D 图像或 3D 投影得到的 2D 图像,以生成用于训练的空间伪标签与推理链。随后,我们提出推理链蒸馏,将大型教师网络提取的推理知识迁移到轻量学生网络。为同时表示全局与局部几何关系,我们引入几何感知空间建模(GeoSM)模块,将文本推理与 3D 空间结构对齐。实验表明,UZ3DVG 在 ScanRefer 与 NR3D 上取得 SOTA 零样本性能,推理速度最高达 7.69 FPS,约为 SOTA 方法的 38 倍。代码已发布于 https://github.com/tanwb/UZ3DVG。
英文原文摘要 1690 字符
Zero-Shot 3D Visual Grounding (Zero-Shot 3DVG) aims to localize target objects in 3D scenes from natural language descriptions without relying on instance-wise description annotations. Existing methods rely on extra 2D images during inference and/or require multi-turn interactions with large language models (LLMs) or vision-language models (VLMs), which increase latency, computational cost, and deployment complexity. To overcome these limitations, we propose Unaided Zero-shot 3D Visual Grounding with generated language conditions (UZ3DVG), which is fed with 3D point clouds and textual descriptions only during inference and does not depend on external models. This is a new training paradigm: a VLM is employed solely to produce object-wise descriptions (pseudo labels) and reasoning chains for training a lightweight 3DVG model with robust spatial reasoning. Specifically, the introduced Open-Vocabulary Multi-Source Spatial Annotation and Reasoning Chain Generator processes RGB-D images or 3D-projected 2D images from open-world scenes to generate spatial pseudo-labels and reasoning chains for training. Then, we propose Reasoning Chain Distillation, which transfers reasoning knowledge extracted by a large teacher network to a lightweight student network. To represent both global and local geometric relationships, the Geometry-Aware Spatial Modeling (GeoSM) module is introduced to align textual reasoning with 3D spatial structures. Experiments show that UZ3DVG achieves SOTA zero-shot performance on ScanRefer and NR3D, with inference speeds up to 7.69 \mathrm FPS , approximately 38 times faster than SOTA methods. The code is available at https://github.com/tanwb/UZ3DVG.