FE2E:从编辑器到密集几何估计器
FE2E: From Editor to Dense Geometry Estimator
将预训练的图像编辑模型(DiT 架构)微调为密集几何预测器,利用其固有结构先验,并通过一致速度训练目标和对数量化解决精度冲突,为单目深度/法线估计提供新范式。
原文依据 editing models possess inherent structural priors, which enable them to converge more stably by "refining" their innate features
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论预训练的文本到图像(T2I)生成先验在深度和法线预测中已展现出成功。然而,密集预测本质上是一项图像到图像的任务,这表明图像编辑模型而非 T2I 生成模型可能是更适合微调的基础。受此启发,我们对编辑器和生成器在密集几何估计中的微调行为进行了系统分析。我们的发现表明,编辑模型具有固有的结构先验,这使其能够通过“精炼”其内在特征而更稳定地收敛,并最终获得比生成模型更高的性能。基于这些发现,我们提出了 FE2E,这是一个开创性地将基于扩散 Transformer(DiT)架构的先进编辑模型适配于密集几何预测的框架。具体而言,为了将该编辑器定制用于这一确定性任务,我们将编辑器原始的流匹配损失重新表述为“一致速度”训练目标。并且我们使用对数量化来解决编辑器原生 BFloat16 格式与本任务高精度需求之间的精度冲突。此外,我们重新利用了编辑器被丢弃的区域,以无成本地联合估计深度和法线,从而提高了推理效率。在不扩大训练数据的情况下,FE2E 在多个数据集上的零样本单目深度和法线估计中取得了令人印象深刻的性能提升。值得注意的是,它在 ETH3D 数据集上实现了超过 35% 的性能提升,并超越了在 100 倍数据上训练的 DepthAnything 系列。
英文原文摘要 1638 字符
Pre-trained text-to-image (T2I) generative priors have shown success in depth and normal prediction. However, dense prediction is inherently an image-to-image task, suggesting that image editing models, rather than T2I generative models, may be a more suitable foundation for fine-tuning. Motivated by this, we conduct a systematic analysis of the fine-tuning behaviors of both editors and generators for dense geometry estimation. Our findings show that editing models possess inherent structural priors, which enable them to converge more stably by "refining" their innate features, and ultimately achieve higher performance than their generative counterparts. Based on these findings, we introduce FE2E, a framework that pioneeringly adapts an advanced editing model based on Diffusion Transformer (DiT) architecture for dense geometry prediction. Specifically, to tailor the editor for this deterministic task, we reformulate the editor's original flow matching loss into the "consistent velocity" training objective. And we use logarithmic quantization to resolve the precision conflict between the editor's native BFloat16 format and the high precision demand of our tasks. Additionally, we repurpose the editor's discarded region for a cost-free joint estimation of depth and normals, which improves the inference efficiency. Without scaling up the training data, FE2E achieves impressive performance improvements in zero-shot monocular depth and normal estimation across multiple datasets. Notably, it achieves over 35% performance gains on the ETH3D dataset and outperforms the DepthAnything series, which is trained on 100xdata.