视觉基础模型可以成为潜扩散模型的优秀分词器
Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models
把冻结的视觉基础模型（如 DINO 类语义特征）直接当作潜扩散模型的分词器，并配一个从语义特征重建真实图像的解码器，可显著加快扩散训练收敛；做 LDM 的人可以据此替换 VAE 分词器，做表征学习的人可借鉴其双侧对齐训练策略。
原文依据 we bypass the distillation by proposing a more direct approach by leveraging the frozen VFM for the LDMs tokenizer, named VFM Variational Autoencoder (VFM-VAE)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论潜扩散模型（LDM）的性能关键取决于其视觉分词器的质量。已有工作尝试通过蒸馏将视觉基础模型（VFM）引入分词器训练，但我们通过实验发现，这种做法不可避免地削弱了原始 VFM 所学表征的鲁棒性。本文绕过蒸馏，提出一种更直接的方法，即利用冻结的 VFM 作为 LDM 的分词器，命名为 VFM 变分自编码器（VFM-VAE）。为充分挖掘冻结 VFM 作为 LDM 分词器的潜力，我们设计了一个新的解码器，以从 VFM 语义丰富的表征中重建逼真图像。借助所提出的 VFM-VAE，我们系统研究了不同分词器的表征如何影响整个扩散训练过程中的表征学习过程，从而实现分词器与扩散模型双侧对齐的协同增益。我们在分词器设计与训练策略上的努力带来了更优的性能与效率：我们的系统仅在 80 个 epoch 内就达到 2.22 的 gFID（不使用 CFG）（相较先前分词器实现 10 倍加速）。继续训练至 640 个 epoch 后，其 gFID（不使用 CFG）进一步达到 1.62。这些结果为 VFM 作为视觉分词器以加速 LDM 训练进程的巨大潜力提供了有力证据。
英文原文摘要 1427 字符
The performance of Latent Diffusion Models (LDMs) is critically dependent on the quality of their visual tokenizers. While recent works have explored incorporating Vision Foundation Models (VFMs) into the tokenizers training via distillation, we empirically find this approach inevitably weakens the robustness of learnt representation from original VFM. In this paper, we bypass the distillation by proposing a more direct approach by leveraging the frozen VFM for the LDMs tokenizer, named VFM Variational Autoencoder (VFM-VAE). To fully exploit the potential to leverage frozen VFM for the LDMs tokenizer, we design a new decoder to reconstruct realistic images from the semantic-rich representation of VFM. With the proposed VFM-VAE, we conduct a systematic study on how the representation from different tokenizers impact the representation learning process throughout diffusion training, enabling synergistic benefits of dual-side alignment on both tokenizers and diffusion models. Our effort in tokenizer design and training strategy lead to superior performance and efficiency: our system reaches a **gFID (w/o CFG) of 2.22 in merely 80 epochs** (a 10xspeedup over prior tokenizers). With continued training to **640 epochs**, it further attains a **gFID (w/o CFG) of 1.62**. These results offer solid evidence for the substantial potential of VFMs to serve as visual tokenizers to accelerate the LDM training progress.