基于视觉语言模型的零样本深度补全
Zero-Shot Depth Completion with Vision-Language Model
把稀疏深度以视觉 token、文本提示和文本监督三种方式注入 VLM，使通用视觉语言模型无需改结构即可输出带绝对尺度的稠密深度，为利用 VLM 语义先验做几何感知提供了一条可复用的路径。
原文依据 we propose a sparse depth injection mechanism that extends the capability of VLM toward 3D perception
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在语言引导下的语义理解任务中已取得显著成功，但其在几何感知方面的潜力仍鲜有探索。本文提出首个基于 VLM 的深度补全框架。在几乎不修改网络结构的前提下，我们提出一种稀疏深度注入机制，通过视觉 token 化、文本提示与文本监督三个关键方面将 VLM 的能力扩展至三维感知。在视觉输入侧，稀疏深度被 token 化，以提供绝对尺度和准确的几何线索，从而缓解仅依赖 RGB 输入时的尺度与相机歧义。在文本输入侧，由稀疏深度导出的二值掩码作为提示，指示模型在何处补全、在何处保留。在监督侧，模型使用由稀疏深度生成的文本标签进行微调，无需真实深度。得益于 VLM 强大的语义先验与跨模态表达能力，我们的框架在多种传感器、不同稀疏程度和场景下均取得优异的零样本性能。
英文原文摘要 1140 字符
Vision language models (VLMs) have achieved remarkable success in semantic understanding tasks under language guidance, yet their potential for geometric perception remains largely underexplored. This paper introduces the first VLM-based depth completion framework. With almost no architectural modifications, we propose a sparse depth injection mechanism that extends the capability of VLM toward 3D perception through three key aspects: visual tokenization, textual prompt, and textual supervision. At the visual input side, sparse depth is tokenized to provide absolute scale and accurate geometric cues, alleviating the scale and camera ambiguities of RGB-only inputs. At the textual input side, a binary mask derived from sparse depth serves as a prompt, instructing the model where to complete and where to preserve. At the supervision side, the model is fine-tuned using text labels generated from sparse depth, requiring no ground-truth depth. Benefiting from the strong semantic priors and cross-modal expressiveness of VLM, our framework achieves superior zero-shot performance across diverse sensors, sparsity levels, and scenes.