缓解基于扩散的数据集蒸馏中的分布偏移
Mitigating The Distribution Shift of Diffusion-based Dataset Distillation
对做数据集蒸馏或需要合成小规模训练集的人，这篇给出了两个可直接复用的手段：在扩散训练时加 L1 稀疏正则让模型学紧凑流形，以及在采样时对整个合成集同步去噪并加分布正则，替代逐样本顺序采样。
原文依据 we abandon the flawed sequential sampling paradigm and instead synchronously denoises the entire synthetic dataset with distribution regularizers
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论数据集蒸馏（DD）旨在构建小型合成数据集，以实现高效的模型训练。尽管扩散模型是强大的生成器，但其在 DD 中的应用受制于合成数据与理想蒸馏数据之间的分布偏移，从而导致性能次优。我们识别出两种关键的偏移。首先，考虑到合成数据的容量较小，DD 的最优合成分布应当是对真实数据分布的简化，而非复现原始数据的复杂度。其次，由于数据采样过程，合成数据集相对于该学习到的分布存在有害的经验偏差。为解决这些问题，我们提出一种两阶段方法。在扩散训练阶段，我们通过引入 L1 稀疏正则项来缓解分布偏移，迫使扩散模型学习一个紧凑且语义稀疏的流形。随后在采样阶段，我们放弃有缺陷的顺序采样范式，转而借助分布正则项对整个合成数据集进行同步去噪。该框架系统性地缓解了上述两种已识别的分布偏移。实验表明，我们的方法在取得最优性能的同时具备更优的计算效率。
英文原文摘要 1266 字符
Dataset Distillation (DD) seeks to create small, synthetic datasets for efficient model training. While diffusion models are powerful generators, their use in DD is hampered by distribution shifts between synthetic and ideal distilled data, leading to suboptimal performance. We identify two critical shifts. First, considering the small capacity of the synthetic data, an optimal synthetic distribution for DD should be a simplification of the real data distribution, rather than replicating the original data's complexity. Second, there is a hazardous empirical deviation in the synthetic dataset from this learned distribution due to the data sampling process. To address these, we introduce a two-stage approach. During diffusion training time, we mitigate the distribution shift by employing an L1 sparsity regularizer, compelling the diffusion model to learn a compact and semantically sparse manifold. Then, during sampling time, we abandon the flawed sequential sampling paradigm and instead synchronously denoises the entire synthetic dataset with distribution regularizers. This framework systematically mitigates both identified distribution shifts. Experiments show our method achieves state-of-the-art performance with superior computational efficiency.