用方差扩张损失驯服潜扩散模型中的采样扰动
Taming Sampling Perturbations with Variance Expansion Loss for Latent Diffusion Models
针对潜空间扩散模型的 tokenizer 训练环节，提出方差扩张损失来抑制 b-VAE 潜空间的方差坍缩，使潜流形对扩散采样中的随机扰动更鲁棒；做 LDM/DiT 类生成时可直接替换或叠加到 VAE 微调目标中，无需改动扩散主干。
原文依据 introducing a Variance Expansion loss that counteracts variance collapse
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论潜扩散模型因能在紧凑潜空间中学习扩散过程，已成为高保真、高效图像生成的主流框架。然而，以往研究主要关注潜空间的重建精度与语义对齐，我们观察到另一个关键因素——对采样扰动的鲁棒性——同样在决定生成质量中起重要作用。通过经验与理论分析，我们表明潜扩散模型中常用的基于 b-VAE 的 tokenizer 往往产生过于紧凑的潜流形，该流形对扩散采样过程中的随机扰动高度敏感，从而导致视觉质量退化。为解决该问题，我们提出一种简单而有效的方案，在保持强重建保真度的同时构建对采样扰动鲁棒的潜空间。这通过引入方差扩张损失实现：该损失抵消方差坍缩，并利用重建与方差扩张之间的对抗性相互作用，达成自适应平衡，从而在保持重建精度的同时提升对随机采样的鲁棒性。大量实验表明，我们的方法在不同潜扩散架构上均能持续提升生成质量，证实潜空间的鲁棒性是稳定、忠实扩散采样所缺失的关键要素。项目主页：https://github.com/CVL-UESTC/VE-Loss 。
英文原文摘要 1536 字符
Latent diffusion models have emerged as the dominant framework for high-fidelity and efficient image generation, owing to their ability to learn diffusion processes in compact latent spaces. However, while previous research has focused primarily on reconstruction accuracy and semantic alignment of the latent space, we observe that another critical factor, robustness to sampling perturbations, also plays a crucial role in determining generation quality. Through empirical and theoretical analyses, we show that the commonly used b-VAE-based tokenizers in latent diffusion models, tend to produce overly compact latent manifolds that are highly sensitive to stochastic perturbations during diffusion sampling, leading to visual degradation. To address this issue, we propose a simple yet effective solution that constructs a latent space robust to sampling perturbations while maintaining strong reconstruction fidelity. This is achieved by introducing a Variance Expansion loss that counteracts variance collapse and leverages the adversarial interplay between reconstruction and variance expansion to achieve an adaptive balance that preserves reconstruction accuracy while improving robustness to stochastic sampling. Extensive experiments demonstrate that our approach consistently enhances generation quality across different latent diffusion architectures, confirming that robustness in latent space is a key missing ingredient for stable and faithful diffusion sampling. Our project page: https://github.com/CVL-UESTC/VE-Loss .