方向感知的 3D 大型多模态模型
Direction-aware 3D Large Multimodal Models
PoseRecover 用 RGB-D 视频外参、物体视锥求交与 Z-buffer 可见性检查自动为点云基准补全自我位姿，PoseAlign 则直接变换点云坐标使其与位姿对齐——这套「把位姿注入数据而非提示词或投影层」的做法，可复用于任何需要空间/方向推理的 3D 视觉-语言任务。
原文依据 PoseRecover, a fully automatic pose recovery pipeline that matches questions with ego poses from RGB-D video extrinsics
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论3D 大型多模态模型（3D LMM）在实现方向性问答与空间推理时高度依赖自我位姿。然而，现有点云基准大多包含丰富的方向性查询，却缺少相应的自我位姿，这使得它们在 3D 大型多模态建模中本质上是病态的。在本工作中，我们重新定义了一个新的严格范式：通过识别并向点云基准中补充自我位姿，并依据识别出的自我位姿对相应的点云数据进行变换，从而实现方向感知的 3D LMM。我们通过两项新颖设计实现方向感知的 3D LMM。第一项是 PoseRecover，一个全自动的位姿恢复流程，它借助 RGB-D 视频外参，通过物体视锥求交与基于 Z-buffer 的可见性检查，将问题与自我位姿进行匹配。第二项是 PoseAlign，它直接变换点云数据使其与识别出的自我位姿对齐，而不是将自我位姿注入文本提示，也不是在投影层中引入位姿编码特征。大量实验表明，我们的设计在 LL3DA、LL3DA-SONATA、Chat-Scene 和 3D-LLAVA 等多个 3D LMM 骨干上均带来一致的提升，将 ScanRefer 的平均交并比提升 30.0%，将 Scan2Cap 的 LLM-as-judge 准确率提升 11.7%。此外，我们的方法简单、通用且训练高效，仅需指令微调，并为方向感知的 3D LMM 建立了一个强基线。
英文原文摘要 1464 字符
3D large multimodal models (3D LMMs) rely heavily on ego poses for enabling directional question-answering and spatial reasoning. However, most existing point cloud benchmarks contain rich directional queries but lack the corresponding ego poses, making them inherently ill-posed in 3D large multimodal modelling. In this work, we redefine a new and rigorous paradigm that enables direction-aware 3D LMMs by identifying and supplementing ego poses into point cloud benchmarks and transforming the corresponding point cloud data according to the identified ego poses. We enable direction-aware 3D LMMs with two novel designs. The first is PoseRecover, a fully automatic pose recovery pipeline that matches questions with ego poses from RGB-D video extrinsics via object-frustum intersection and visibility check with Z-buffers. The second is PoseAlign that transforms the point cloud data to be aligned with the identified ego poses instead of either injecting ego poses into textual prompts or introducing pose-encoded features in the projection layers. Extensive experiments show that our designs yield consistent improvements across multiple 3D LMM backbones such as LL3DA, LL3DA-SONATA, Chat-Scene, and 3D-LLAVA, improving ScanRefer mIoU by 30.0% and Scan2Cap LLM-as-judge accuracy by 11.7%. In addition, our approach is simple, generic, and training-efficient, requiring only instruction tuning while establishing a strong baseline for direction-aware 3D-LMMs.