Iris：将语言融入基于扩散的单目深度估计
Iris: Integrating Language into Diffusion-based Monocular Depth Estimation
把文本描述作为额外条件注入扩散式单目深度估计器（Marigold、Lotus、E2E-FT），可缩小深度解空间、提升小区域精度，并支持通过追加场景描述迭代细化深度；同时文本条件能加速训练收敛与推理扩散轨迹，是一种可直接复用的条件化与加速手段。
原文依据 language can improve the fidelity of estimates by providing additional information through text as a condition
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论传统单目深度估计器受视觉歧义与干扰影响。我们证明，语言可以通过以文本作为条件提供额外信息来提升估计的保真度，从而缩小深度估计的解空间。该条件分布是在扩散模型的文本到图像预训练过程中学习得到的。为了生成反映文本描述的各种视角与布局的图像，模型会隐式编码物体形状及其空间关系，即三维（3D）场景结构。我们研究了将文本描述融入基于扩散的单目深度估计器（MDE）的训练与推理所带来的收益。我们在三种不同的基于扩散的 MDE 上进行了实验，即 Marigold、Lotus 和 E2E-FT 及其变体。通过在 HyperSim 和 Virtual KITTI 上训练，并在 NYUv2、KITTI、ETH3D、ScanNet 和 DIODE 上评估，我们发现将文本融入 MDE 的策略提升了深度估计的整体精度，尤其是在小区域上。该效果还具有针对性：在文本所描述的特定区域，改进更为显著。因此，这自然支持通过提供关于 3D 场景的额外描述来迭代细化深度估计。另外，我们发现引入文本可以加速训练的收敛以及推理的扩散轨迹。
英文原文摘要 1443 字符
Conventional monocular depth estimators suffer from visual ambiguities and nuisances. We demonstrate that language can improve the fidelity of estimates by providing additional information through text as a condition, thereby reducing the solution space for depth estimates. This conditional distribution is learned during text-to-image pre-training of diffusion models. To generate images under various viewpoints and layouts that reflect textual descriptions, the model implicitly encodes object shapes and their spatial relationships, comprising the 3-dimensional (3D) scene structure. We investigate the benefits of integrating text descriptions into the training and inference of the diffusion-based monocular depth estimators (MDE). We experiment with three different diffusion-based MDEs, namely Marigold, Lotus, and E2E-FT, and their variants. By training on HyperSim and Virtual KITTI, and evaluating on NYUv2, KITTI, ETH3D, ScanNet, and DIODE, we find that our strategy of integrating text into MDEs improves the overall accuracy of depth estimates, especially in small areas. The effect is also targeted in that improvements are more pronounced in specific regions described in the text. Naturally, this lends to iterative refinement of the depth estimate by providing additional descriptions of the 3D scene. Separately, we find that incorporating text can accelerate convergence of training and the inference diffusion trajectory.