RecTok:沿整流流的重建蒸馏
RecTok: Reconstruction Distillation along Rectified Flow
把视觉基础模型的语义蒸馏进流匹配的前向流轨迹而非潜空间,使高维视觉分词器不再受维度-生成质量权衡限制,可直接替换扩散 Transformer 的 tokenizer 以提升重建与生成质量。
原文依据 distill the semantic information in VFMs into the forward flow trajectories in flow matching
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉分词器在扩散模型中起着关键作用。潜空间的维度同时决定重建保真度与潜特征的语义表达能力。然而,维度与生成质量之间存在固有的权衡,使现有方法受限于低维潜空间。尽管近期工作利用视觉基础模型(VFM)来丰富视觉分词器的语义并加速收敛,高维分词器仍然不如其低维对应版本。本文提出 RecTok,通过两项关键创新克服高维视觉分词器的局限:流语义蒸馏与重建对齐蒸馏。我们的核心洞见是让流匹配中的前向流具备丰富语义,并将其作为扩散 Transformer 的训练空间,而非像以往工作那样关注潜空间。具体而言,本方法将 VFM 中的语义信息蒸馏到流匹配的前向流轨迹中,并进一步通过掩码特征重建损失增强语义。RecTok 在图像重建、生成质量与判别性能上均表现优异,在有分类器无关引导与无分类器无关引导两种设置下于 gFID-50K 上均取得最优结果,同时保持语义丰富的潜空间结构。此外,随着潜变量维度增加,我们观察到性能持续提升。代码与模型将公开。
英文原文摘要 1595 字符
Visual tokenizers play a crucial role in diffusion models. The dimensionality of latent space governs both reconstruction fidelity and the semantic expressiveness of the latent feature. However, a fundamental trade-off is inherent between dimensionality and generation quality, constraining existing methods to low-dimensional latent spaces. Although recent works have leveraged vision foundation models (VFMs) to enrich the semantics of visual tokenizers and accelerate convergence, high-dimensional tokenizers still underperform their low-dimensional counterparts. In this work, we propose RecTok, which overcomes the limitations of high-dimensional visual tokenizers through two key innovations: flow semantic distillation and reconstruction-alignment distillation. Our key insight is to make the forward flow in flow matching semantically rich, which serves as the training space of diffusion transformers, rather than focusing on the latent space as in previous works. Specifically, our method distill the semantic information in VFMs into the forward flow trajectories in flow matching. And we further enhance the semantics by introducing a masked feature reconstruction loss. Our RecTok achieves superior image reconstruction, generation quality, and discriminative performance. It achieves state-of-the-art results on the gFID-50K under both with and without classifier-free guidance settings, while maintaining a semantically rich latent space structure. Furthermore, as the latent dimensionality increases, we observe consistent improvements. Code and model will be publicly available.