3D-IDE:三维隐式深度涌现
3D-IDE: 3D Implicit Depth Emergent
通过信息瓶颈和几何自监督让MLLM在统一视觉表征中隐式涌现3D感知,推理时无需深度和位姿输入且零延迟开销,为视觉语言模型融合3D信息提供了免外部嫁接的新范式。
原文依据 eliminating depth and pose dependencies during inference with zero latency overhead
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在多模态大语言模型(MLLMs)中利用3D信息近来在室内场景理解方面展现出显著优势。然而,现有方法——包括使用显式真值3D位置编码的方法以及嫁接外部3D基础模型以获取隐式几何的方法——在2D-3D表征融合的权衡上表现不佳,导致部署效果次优。为此,我们提出3D隐式深度涌现(3D-Implicit Depth Emergence),一种将3D感知重新定义为源自几何自监督的涌现属性而非显式编码的方法。我们的核心洞见是隐式几何涌现原理:通过细粒度几何验证器和全局表征约束等机制策略性地利用特权几何监督,我们构建了一个信息瓶颈。该瓶颈迫使模型最大化视觉特征与3D结构之间的互信息,使3D感知在统一视觉表征中自然涌现。与现有方法不同,我们的方法使3D感知隐式涌现,在密集区域解耦特征,并且关键的是,在推理阶段消除对深度和位姿的依赖且零延迟开销。这种从外部嫁接到隐式涌现的范式转变代表了对视觉-语言模型中3D知识整合的根本性重新思考。大量实验表明,我们的方法在多个3D场景理解基准上超越SOTA。我们的方法在保持多样化下游任务强劲性能的同时,实现了55%的推理延迟降低,凸显了精心设计的辅助目标对于无依赖3D理解的有效性。源代码见\href https://chushanzhang.github.io/3D-IDE/ \textcolor[HTML] E8477A \small\texttt github.com/ChushanZhang/3D-IDE 。
英文原文摘要 1927 字符
Leveraging 3D information within Multimodal Large Language Models (MLLMs) has recently shown significant advantages for indoor scene understanding. However, existing methods, including those using explicit ground-truth 3D positional encoding and those grafting external 3D foundation models for implicit geometry, struggle with the trade-off in 2D-3D representation fusion, leading to suboptimal deployment. To this end, we propose 3D-Implicit Depth Emergence, a method that reframes 3D perception as an emergent property derived from geometric self-supervision rather than explicit encoding. Our core insight is the Implicit Geometric Emergence Principle: by strategically leveraging privileged geometric supervision through mechanisms like a fine-grained geometry validator and global representation constraints, we construct an information bottleneck. This bottleneck forces the model to maximize the mutual information between visual features and 3D structures, allowing 3D awareness to emerge naturally within a unified visual representation. Unlike existing approaches, our method enables 3D perception to emerge implicitly, disentangling features in dense regions and, crucially, eliminating depth and pose dependencies during inference with zero latency overhead. This paradigm shift from external grafting to implicit emergence represents a fundamental rethinking of 3D knowledge integration in visual-language models. Extensive experiments demonstrate that our method surpasses SOTA on multiple 3D scene understanding benchmarks. Our approach achieves a 55% reduction in inference latency while maintaining strong performance across diverse downstream tasks, underscoring the effectiveness of meticulously designed auxiliary objectives for dependency-free 3D understanding. Source code can be found at \href https://chushanzhang.github.io/3D-IDE/ \textcolor[HTML] E8477A \small\texttt github.com/ChushanZhang/3D-IDE .