免训练混合分辨率潜变量上采样用于空间加速扩散 Transformer
Training-free Mixed-Resolution Latent Upsampling for Spatially Accelerated Diffusion Transformers
提出一种免训练的空间加速框架，通过区域自适应潜变量上采样和噪声-时间步匹配，在保持生成质量的同时显著加速扩散 Transformer，可复用于高分辨率图像生成等任务。
原文依据 RALU achieves artifact-free, efficient acceleration with early upsampling only on artifact-prone edge regions and noise-timestep matching for different latent resolutions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩散 Transformer（DiTs）为高保真生成提供了出色的可扩展性，但其计算开销对实际部署构成了巨大挑战。现有加速方法主要利用时序维度，而空间加速仍未得到充分探索。在本工作中，我们通过潜变量上采样研究 DiTs 的空间加速。我们发现，用于空间加速的朴素潜变量上采样会引入伪影，主要归因于高频边缘区域的混叠以及噪声-时间步不匹配。然后，基于这些发现和分析，我们提出了一个免训练的空间加速框架，称为区域自适应潜变量上采样（RALU），通过我们的混合分辨率潜变量上采样来缓解这些伪影，同时实现 DiTs 的空间加速。RALU 通过仅在易产生伪影的边缘区域进行早期上采样以及针对不同潜变量分辨率的噪声-时间步匹配，实现了无伪影的高效加速，在 FLUX-1.dev 上最高获得 7.0 倍加速，在 Stable Diffusion 3 上获得 3.0 倍加速，且质量下降可忽略不计。此外，我们的 RALU 可互补地应用于现有的时序加速方法和时间步蒸馏模型，最高实现 15.9 倍加速。
英文原文摘要 1275 字符
Diffusion transformers (DiTs) offer excellent scalability for high-fidelity generation, but their computational overhead poses a great challenge for practical deployment. Existing acceleration methods primarily exploit the temporal dimension, whereas spatial acceleration remains underexplored. In this work, we investigate spatial acceleration for DiTs via latent upsampling. We found that naive latent upsampling for spatial acceleration introduces artifacts, primarily due to aliasing in high-frequency edge regions and mismatching from noise-timestep discrepancies. Then, based on these findings and analyses, we propose a training-free spatial acceleration framework, dubbed Region-Adaptive Latent Upsampling (RALU), to mitigate those artifacts while achieving spatial acceleration of DiTs by our mixed-resolution latent upsampling. RALU achieves artifact-free, efficient acceleration with early upsampling only on artifact-prone edge regions and noise-timestep matching for different latent resolutions, leading to up to 7.0xspeedup on FLUX-1.dev and 3.0xon Stable Diffusion 3 with negligible quality degradation. Furthermore, our RALU is complementarily applicable to existing temporal acceleration methods and timestep-distilled models, leading to up to 15.9xspeedup.