一个模型，多种预算：面向扩散 Transformer 的弹性潜变量接口
One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformers
把 DiT 的计算量与输入分辨率解耦：用可学习、可变长的潜变量序列作为中间接口，配合轻量 Read/Write 交叉注意力在空间词元与潜变量之间搬运信息，并通过随机丢弃尾部潜变量训练出重要性有序的表示，推理时按算力预算截断潜变量数量即可实现连续的延迟-质量权衡。做高分辨率生成或需要在固定算力下部署扩散模型的工作可直接把该模块插进现有 DiT/U-ViT/MM-DiT 主干。
原文依据 By training with random dropping of tail latents, ELIT learns to produce importance-ordered representations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩散 Transformer（DiT）能够取得很高的生成质量，但其浮点运算量被锁定在图像分辨率上，限制了有原则的延迟-质量权衡；同时它在所有输入空间词元上均匀分配计算，把资源浪费在不重要的区域。我们提出弹性潜变量接口 Transformer（ELIT），这是一种可直接替换、与 DiT 兼容的机制，将输入图像尺寸与计算量解耦。该方法插入一个潜变量接口，即一段可学习、长度可变的词元序列，标准 Transformer 块可在其上运行。轻量的 Read 与 Write 交叉注意力层在空间词元与潜变量之间搬运信息，并优先关注重要的输入区域。通过在训练中随机丢弃尾部潜变量，ELIT 学会产生按重要性排序的表示：靠前的潜变量捕捉全局结构，靠后的潜变量则包含用于细化细节的信息。在推理时，潜变量的数量可以动态调整，以匹配计算约束。ELIT 刻意保持极简，仅增加两个交叉注意力层，而整流流目标与 DiT 堆栈保持不变。在多个数据集与多种架构（DiT、U-ViT、HDiT、MM-DiT）上，ELIT 都带来一致的提升。在 ImageNet-1K 512 像素上，ELIT 在弗雷歇初始距离与 FDD 分数上分别取得 35.3% 与 39.6% 的平均提升。
英文原文摘要 1307 字符
Diffusion transformers (DiTs) achieve high generative quality but lock FLOPs to image resolution, limiting principled latency-quality trade-offs, and allocate computation uniformly across input spatial tokens, wasting resource allocation to unimportant regions. We introduce Elastic Latent Interface Transformer (ELIT), a drop-in, DiT-compatible mechanism that decouples input image size from compute. Our approach inserts a latent interface, a learnable variable-length token sequence on which standard transformer blocks can operate. Lightweight Read and Write cross-attention layers move information between spatial tokens and latents and prioritize important input regions. By training with random dropping of tail latents, ELIT learns to produce importance-ordered representations with earlier latents capturing global structure while later ones contain information to refine details. At inference, the number of latents can be dynamically adjusted to match compute constraints. ELIT is deliberately minimal, adding two cross-attention layers while leaving the rectified flow objective and the DiT stack unchanged. Across datasets and architectures (DiT, U-ViT, HDiT, MM-DiT), ELIT delivers consistent gains. On ImageNet-1K 512px, ELIT delivers an average gain of 35.3% and 39.6% in FID and FDD scores.