DiverseDiT：面向扩散 Transformer 中多样化表征学习
DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformers
对做 DiT 图像生成的人，这篇给出了一个可直接插入任意 DiT 主干的正则手段：跨块长残差连接 + 表征多样性损失，用来替代或补充 REPA 式外部编码器对齐，并报告了收敛加速与一步生成下的收益。
原文依据 we propose DiverseDiT, a novel framework that explicitly promotes representation diversity
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩散 Transformer（DiT）近期的突破凭借其优越的可扩展性革新了视觉合成领域。为促进 DiT 捕获有意义的内部表征，REPA 等近期工作引入外部预训练编码器进行表征对齐。然而，DiT 内部表征学习的底层机制尚未被充分理解。为此，我们首先系统性地研究了 DiT 的表征动态。通过分析不同设置下内部表征的演化与影响，我们揭示出跨块的表征多样性是有效学习的关键因素。基于这一关键洞见，我们提出 DiverseDiT，一个显式促进表征多样性的新框架。DiverseDiT 引入长残差连接以多样化各块的输入表征，并采用表征多样性损失以鼓励各块学习彼此不同的特征。在 ImageNet 256x256 与 512x512 上的大量实验表明，DiverseDiT 应用于不同规模的不同主干时均能带来一致的性能提升与收敛加速，即使在具有挑战性的一步生成设置下测试亦如此。此外，我们表明 DiverseDiT 与现有表征学习技术互补，可带来进一步的性能增益。我们的工作为 DiT 的表征学习动态提供了有价值的洞见，并给出了一种提升其性能的实用方法。代码见 https://github.com/kobeshegu/DiverseDiT。
英文原文摘要 1635 字符
Recent breakthroughs in Diffusion Transformers (DiTs) have revolutionized the field of visual synthesis due to their superior scalability. To facilitate DiTs' capability of capturing meaningful internal representations, recent works such as REPA incorporate external pretrained encoders for representation alignment. However, the underlying mechanisms governing representation learning within DiTs are not well understood. To this end, we first systematically investigate the representation dynamics of DiTs. Through analyzing the evolution and influence of internal representations under various settings, we reveal that representation diversity across blocks is a crucial factor for effective learning. Based on this key insight, we propose DiverseDiT, a novel framework that explicitly promotes representation diversity. DiverseDiT incorporates long residual connections to diversify input representations across blocks and a representation diversity loss to encourage blocks to learn distinct features. Extensive experiments on ImageNet 256x256 and 512x512 demonstrate that our DiverseDiT yields consistent performance gains and convergence acceleration when applied to different backbones with various sizes, even when tested on the challenging one-step generation setting. Furthermore, we show that DiverseDiT is complementary to existing representation learning techniques, leading to further performance gains. Our work provides valuable insights into the representation learning dynamics of DiTs and offers a practical approach for enhancing their performance. Our code is available at https://github.com/kobeshegu/DiverseDiT.