PromptDepth：面向具身智能的高效可提示几何三维视觉模型
PromptDepth: Efficient and Promptable Geometric 3D Vision Model for Embodied Intelligence
PromptDepth 将深度估计与实例级分割统一到一个前馈网络中，通过提示（prompt）灵活切换全景/实例/跟踪深度图输出，为具身智能提供实时 3D 感知与交互能力；其 ILDS 损失和 Gram Anchoring 技术可缓解密集与离散表示冲突，可复用于其他多任务密集预测模型。
原文依据 PromptDepth is a feed-forward network that quickly yields panoptic, instanced, or tracked depth map from two corresponding frames
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论面向具身智能的视觉模型需要高效的三维理解以及与场景中物体的交互能力。现有的三维重建模型要么忽略实例级感知，要么依赖耗时的离线推理，在实时具身场景中适应性较差。本文提出 PromptDepth，这是首个专为具身智能设计的、兼具几何三维理解与实例级交互能力的可提示视觉模型。PromptDepth 是一个前馈网络，能够从两帧对应图像中快速生成全景、实例或跟踪深度图，从而支持具身智能体实时推理序列。具体而言，遵循最小预测问题，我们设计了一个可提示的密集预测 Transformer，使其能够根据特定提示灵活地与统一的密集预测进行交互。考虑到全景深度图与实例深度图之间的显著差异，我们进一步引入了一种新颖的实例标签分布平滑（ILDS）损失，并辅以 Gram Anchoring，以缓解密集表示与离散表示之间的内在冲突。仅在合成数据上训练，我们的模型在公开基准上的深度估计和交互式分割任务中均取得了最优结果。大量实验表明，与当前的基础模型相比，该模型在具身任务中展现出更优的视觉效率。我们相信，这一高效灵活的几何三维模型为具身智能中的视觉任务提供了新的基础。
英文原文摘要 1561 字符
Vision models for embodied intelligence require efficient 3D comprehension and interaction with objects within the scene. Existing 3D reconstruction models either overlook instance-level perception or rely on time-consuming offline reasoning, showing a less adaptability in real-time embodied scenario. In this paper, we present PromptDepth, the first promptable vision model that features both geometric 3D understanding and instance-level interaction especially designed for embodied intelligence. PromptDepth is a feed-forward network that quickly yields panoptic, instanced, or tracked depth map from two corresponding frames, enabling the real-time infer sequences from embodied agents. Specifically, following the minimal prediction problem, we design a promptable Dense Prediction Transformer, making it flexible to interact with unified dense prediction according to a specific prompt. Considering the substantial discrepancy between panoptic and instanced depth map, we further introduce a novel Instanced Label Distribution Smoothing (ILDS) loss, followed by Gram Anchoring, to mitigate the inherent conflict between dense and discrete representation. Trained on synthetic data only, our model achieves state-of-the-art results in both depth estimation and interactive segmentation on public benchmarks. Extensive experiments demonstrate superior visual efficiency in embodied tasks compared to current fundamental models. We believe that our efficient and flexible geometric 3D model offers a new foundation for vision tasks in embodied intelligence.