面向高效流匹配与扩散模型的多 patch 全局到局部 Transformer 架构
Multi-Patch Global-to-Local Transformer Architecture For Efficient Flow Matching and Diffusion Model
把 DiT 各向同性的等尺寸 patch 处理改为「浅层大 patch 抓全局、深层小 patch 修细节」的层次化多 patch 设计，可直接迁移到任何基于 Transformer 主干的扩散/流匹配生成器中，在 GFLOPs 减半的同时保持生成质量；其时间/类别嵌入改进也可作为加速收敛的通用技巧复用。
原文依据 early blocks operate on larger patches to capture coarse global context, while later blocks use smaller patches to refine local details
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论Transformer 架构，尤其是扩散 Transformer（DiT），由于性能优于卷积 UNet，已被广泛用于扩散模型与流匹配模型。然而，DiT 的各向同性设计在每个模块中处理相同数量的 patch 化 token，导致训练过程计算量相对偏大。在本工作中，我们引入一种多 patch Transformer 设计：浅层模块在较大的 patch 上运算以捕获粗粒度的全局上下文，而深层模块使用较小的 patch 来细化局部细节。这种层次化设计可将计算成本（以 GFLOPs 计）最多降低 50%，同时取得良好的生成性能。此外，我们还提出了对时间嵌入与类别嵌入的改进设计，可加速训练收敛。在 ImageNet 数据集上的大量实验验证了我们架构选择的有效性。
英文原文摘要 915 字符
Transformer architectures, particularly Diffusion Transformers (DiTs), have become widely used in diffusion and flow-matching models due to their strong performance compared to convolutional UNets. However, the isotropic design of DiTs processes the same number of patchified tokens in every block, leading to relatively heavy computation during training process. In this work, we introduce a multi-patch transformer design in which early blocks operate on larger patches to capture coarse global context, while later blocks use smaller patches to refine local details. This hierarchical design could reduces computational cost by up to 50% in GFLOPs while achieving good generative performance. In addition, we also propose improved designs for time and class embeddings that accelerate training convergence. Extensive experiments on the ImageNet dataset demonstrate the effectiveness of our architectural choices.