D3D-VLP：面向具身视觉定位与导航的动态三维视觉-语言-规划模型
D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation
把规划、视觉定位、导航与问答统一进单个 3D-VLM 的 3D 思维链管线，并用掩码自回归损失从部分标注的混合数据中互相隐式监督，对做具身导航与 3D 场景理解的人提供了一个可复用的多任务联合训练范式。
原文依据 A Dynamic 3D Chain-of-Thought (3D CoT) that unifies planning, grounding, navigation, and question answering within a single 3D-VLM
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论具身智能体面临一个关键困境：端到端模型缺乏可解释性与显式的三维推理能力，而模块化系统则忽略了各组件之间的相互依赖与协同效应。为弥合这一鸿沟，我们提出动态三维视觉-语言-规划模型（D3D-VLP）。该模型引入两项关键创新：1）动态三维思维链（3D CoT），在单一 3D-VLM 与思维链管线中统一规划、视觉定位、导航与问答；2）碎片化监督下的协同学习（SLFS）策略，利用掩码自回归损失从海量且部分标注的混合数据中学习，使不同思维链组件相互增强并隐式地互相监督。为此，我们构建了一个大规模数据集，包含来自 5K 真实扫描与 20K 合成场景的 1000 万条混合样本，且兼容强化学习与 DAgger 等在线学习方法。D3D-VLP 在多个基准上取得最优结果，包括视觉-语言导航（R2R-CE、REVERIE-CE、NavRAG-CE）、目标导航（HM3D-OVON）以及任务导向的序列视觉定位与导航（SG3D）。真实世界移动操作实验进一步验证了其有效性。
英文原文摘要 1246 字符
Embodied agents face a critical dilemma that end-to-end models lack interpretability and explicit 3D reasoning, while modular systems ignore cross-component interdependencies and synergies. To bridge this gap, we propose the Dynamic 3D Vision-Language-Planning Model (D3D-VLP). Our model introduces two key innovations: 1) A Dynamic 3D Chain-of-Thought (3D CoT) that unifies planning, grounding, navigation, and question answering within a single 3D-VLM and CoT pipeline; 2) A Synergistic Learning from Fragmented Supervision (SLFS) strategy, which uses a masked autoregressive loss to learn from massive and partially-annotated hybrid data. This allows different CoT components to mutually reinforce and implicitly supervise each other. To this end, we construct a large-scale dataset with 10M hybrid samples from 5K real scans and 20K synthetic scenes that are compatible with online learning methods such as RL and DAgger. Our D3D-VLP achieves state-of-the-art results on multiple benchmarks, including Vision-and-Language Navigation (R2R-CE, REVERIE-CE, NavRAG-CE), Object-goal Navigation (HM3D-OVON), and Task-oriented Sequential Grounding and Navigation (SG3D). Real-world mobile manipulation experiments further validate the effectiveness.