重新审视数据集蒸馏：关于软标签的硬真相
Rethinking Dataset Distillation: Hard Truths about Soft Labels
论文揭示了软标签评估下数据集蒸馏方法性能饱和的问题，并提出计算感知的样本难度选择指标 CAD-Prune 与对齐方法 CA2D，为视觉数据高效训练提供了可复用的样本筛选与评估协议。
原文依据 we introduce CAD-Prune, a compute-aware pruning metric that efficiently identifies samples of optimal difficulty for a given compute budget
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管大规模数据集蒸馏（DD）方法被认为取得了成功，但近期证据[??]发现，由于在下游模型训练中使用了软标签，简单的随机图像基线能够与最优的DD方法如SRe2L[??]表现相当。这与核心集文献中的发现形成对比：在硬标签（HL）设置下，高质量核心集始终优于随机子集。为了理解这一差异，我们进行了详细的可扩展性分析，考察不同标签机制下数据质量的作用，范围从充足的软标签（称为SL+KD机制）到固定软标签（SL）和硬标签（HL）。我们的分析表明，在SL和SL+KD两种机制下，高质量核心集均无法令人信服地超越随机基线。在SL+KD设置中，对于给定的计算预算，无论子集大小或质量如何，性能进一步接近相对于完整数据集的近最优水平。这种性能饱和现象对广泛使用软标签进行模型评估的做法提出了质疑：与HL设置不同，子集质量的影响微乎其微。随后在HL设置下对五种大规模和四种小规模DD方法进行的系统评估表明，只有RDED[??]在ImageNet-1K上能够可靠地超越随机基线，但由于其过度依赖简单样本块，仍可能落后于强核心集方法。基于此，我们引入了CAD-Prune，一种计算感知的剪枝指标，能够高效识别给定计算预算下具有最优难度的样本，并利用它开发了CA2D，一种计算对齐的DD方法，在各种IPC设置下于ImageNet-1K上优于当前DD方法。总之，我们的发现揭示了当前DD研究的许多洞见，并建立了有用的工具，以推进核心集和DD两者的数据高效学习。
英文原文摘要 1890 字符
Despite the perceived success of large-scale dataset distillation (DD) methods, recent evidence [??] finds that simple random image baselines perform on-par with state-of-the-art DD methods like SRe2L [??] due to the use of soft labels during downstream model training. This is in contrast with the findings in coreset literature, where high-quality coresets consistently outperform random subsets in the hard-label (HL) setting. To understand this discrepancy, we perform a detailed scalability analysis to examine the role of data quality under different label regimes, ranging from abundant soft labels (termed as SL+KD regime) to fixed soft labels (SL) and hard labels (HL). Our analysis reveals that high-quality coresets fail to convincingly outperform the random baseline in both SL and SL+KD regimes. In the SL+KD setting, performance further approaches near-optimal levels relative to the full dataset, regardless of subset size or quality, for a given compute budget. This performance saturation calls into question the widespread practice of using soft labels for model evaluation, where unlike the HL setting, subset quality has negligible influence. A subsequent systematic evaluation of five large-scale and four small-scale DD methods in the HL setting reveals that only RDED [??] reliably outperforms random baselines on ImageNet-1K, but can still lag behind strong coreset methods due to its over-reliance on easy sample patches. Based on this, we introduce CAD-Prune, a compute-aware pruning metric that efficiently identifies samples of optimal difficulty for a given compute budget, and use it to develop CA2D, a compute-aligned DD method, outperforming current DD methods on ImageNet-1K at various IPC settings. Together, our findings uncover many insights into current DD research and establish useful tools to advance data-efficient learning for both coresets and DD.