ResDiT:激发扩散 Transformer 内在的分辨率可扩展性
ResDiT: Evoking the Intrinsic Resolution Scalability in Diffusion Transformers
对做 DiT 高分辨率生成的人,这篇给出了免训练的分辨率外推方案:修正位置编码在分辨率变化下的外推误差,并用基分辨率局部注意力做细节增强,可直接套用到 SD3/FLUX 类 DiT 的高分辨率推理与空间可控生成。
原文依据 we introduce a PE scaling technique that rectifies positional encoding under resolution changes
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论利用预训练的扩散 Transformer(DiT)进行高分辨率(HR)图像合成,往往会导致空间布局崩塌与纹理保真度下降。已有工作通过复杂流程缓解这些问题,即先执行基分辨率(即训练分辨率)的去噪过程来引导高分辨率生成。本文转而探究 DiT 内在的生成机制,提出 ResDiT——一种无需训练、可高效扩展分辨率的方法。我们识别出支配空间布局的核心因素是位置编码(PE),并表明原始位置编码在外推到高分辨率时编码了错误的位置信息,从而引发布局崩塌。为此,我们引入一种位置编码缩放技术,在分辨率变化下校正位置编码。为进一步修复低保真度的细节,我们设计了一种基于基分辨率局部注意力的局部增强机制。我们设计了一个 patch 级融合模块来聚合全局与局部线索,并提出一种高斯加权拼接策略以消除网格伪影。综合评测表明,ResDiT 能够稳定地实现高保真、高分辨率的图像合成,并可无缝集成到下游任务中,包括空间可控生成。
英文原文摘要 1290 字符
Leveraging pre-trained Diffusion Transformers (DiTs) for high-resolution (HR) image synthesis often leads to spatial layout collapse and degraded texture fidelity. Prior work mitigates these issues with complex pipelines that first perform a base-resolution (i.e., training-resolution) denoising process to guide HR generation. We instead explore the intrinsic generative mechanisms of DiTs and propose ResDiT, a training-free method that scales resolution efficiently. We identify the core factor governing spatial layout, position embeddings (PEs), and show that the original PEs encode incorrect positional information when extrapolated to HR, which triggers layout collapse. To address this, we introduce a PE scaling technique that rectifies positional encoding under resolution changes. To further remedy low-fidelity details, we develop a local-enhancement mechanism grounded in base-resolution local attention. We design a patch-level fusion module that aggregates global and local cues, together with a Gaussian-weighted splicing strategy that eliminates grid artifacts. Comprehensive evaluations demonstrate that ResDiT consistently delivers high-fidelity, high-resolution image synthesis and integrates seamlessly with downstream tasks, including spatially controlled generation.