LocateAnything3D:基于视线链的视觉语言 3D 检测
LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
将 3D 检测转化为 VLM 的下一词元预测问题,通过显式的 Chain-of-Sight 序列(先 2D 检测再 3D 框)实现开放词表多目标 3D 检测,为视觉研究提供了一种无需专用检测头的统一 VLM 接口。
原文依据 casts 3D detection as a next-token prediction problem
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论为了在世界中行动,模型必须能说出它看到了什么,并知道它在 3D 中的位置。当今的视觉语言模型擅长开放式的 2D 描述和定位,但多目标 3D 检测在很大程度上仍缺失于 VLM 工具箱。我们提出 LocateAnything3D,一种 VLM 原生的方法,将 3D 检测转化为下一词元预测问题。关键在于一个简短、显式的视线链(Chain-of-Sight, CoS)序列,它模仿人们从图像中推理的方式:先在 2D 中找到物体,然后推断其距离、大小和姿态。解码器首先作为视觉链式思维输出 2D 检测,然后在由易到难的课程下预测 3D 框:在物体之间,由近到远的顺序减少了早期歧义并匹配以自我为中心的效用;在每个物体内部,从相机中心、尺寸和旋转的分解按稳定性和可学习性对信息进行排序。这种 VLM 原生接口保留了开放词表和视觉提示能力,无需专用检测头。在具有挑战性的 Omni3D 基准上,我们的模型取得了最优结果,达到 49.89 AP,即使基线被给予真实 2D 框,也以 +15.51 的绝对提升超越了此前最佳。它还能零样本泛化到留出类别,并具有强校准性和鲁棒性。通过将 3D 检测转化为一个受约束的下一词元问题,LocateAnything3D 为模型在 3D 中感知提供了实用的基础。
英文原文摘要 1411 字符
To act in the world, a model must name what it sees and know where it is in 3D. Today's vision-language models excel at open-ended 2D description and grounding, yet multi-object 3D detection remains largely missing from the VLM toolbox. We present LocateAnything3D, a VLM-native recipe that casts 3D detection as a next-token prediction problem. The key is a short, explicit Chain-of-Sight (CoS) sequence that mirrors how people reason from images: find an object in 2D, then infer its distance, size, and pose. The decoder first emits 2D detections as a visual chain-of-thought, then predicts 3D boxes under an easy-to-hard curriculum: across objects, a near-to-far order reduces early ambiguity and matches ego-centric utility; within each object, a center-from-camera, dimensions, and rotation factorization ranks information by stability and learnability. This VLM-native interface preserves open-vocabulary and visual-prompting capability without specialized heads. On the challenging Omni3D benchmark, our model achieves state-of-the-art results, with 49.89 AP, surpassing the previous best by +15.51 absolute improvement even when baseline is given ground-truth 2D boxes. It also generalizes zero-shot to held-out categories with strong calibration and robustness. By turning 3D detection into a disciplined next-token problem, LocateAnything3D offers a practical foundation for models to perceive in 3D.