面向动态数据剪枝的批损失分数
Batch Loss Score for Dynamic Data Pruning
对视觉模型训练而言，BLS 提供了一种无需逐样本损失即可估计样本重要性的三行代码级方案，可直接嵌入现有动态数据剪枝流程，在检测、分割、视频等逐样本损失难以获取的复杂损失场景下实现 20%-50% 的无损剪枝加速。
原文依据 BLS demonstrates remarkable code integration simplicity (three-line injection) and readily adapts existing per-sample loss-based methods (one-line proxy).
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论动态数据剪枝通过在训练过程中有选择地省略信息量较少的样本，从而加速深度学习。虽然逐样本损失是一种常用的重要性度量，但对于复杂模型或损失函数而言，获取该损失可能十分困难甚至不可行，往往需要大量的实现工作。本文提出批损失分数（Batch Loss Score，BLS），这是一种计算高效的替代方案，利用易于获取的批损失的指数移动平均（EMA）为单个样本分配分数。我们从单个样本的视角出发，将批损失视为该样本缩放后个体损失的一次含噪测量，其噪声来源于随机的批次组成。文中形式化地证明了 EMA 机制起到一阶低通滤波器的作用，能够衰减高频的批次组成噪声。由此得到的分数近似于该样本对损失所做的平滑且持续性的贡献，为 BLS 作为样本重要性的代理指标提供了理论依据。BLS 在代码集成上极为简便（三行注入），并且可以方便地适配现有的基于逐样本损失的方法（一行代理）。其有效性通过增强两种此类方法得到验证：在 14 个数据集、11 项任务和 18 个模型上无损地剪枝 20%-50% 的样本，凸显了其实用性与广泛适用性，尤其是在逐样本损失难以获取的复杂场景中。代码见 https://github.com/mrazhou/BLS。
英文原文摘要 1494 字符
Dynamic data pruning accelerates deep learning by selectively omitting less informative samples during training. While per-sample loss is a common importance metric, obtaining it can be challenging or infeasible for complex models or loss functions, often requiring significant implementation effort. This work proposes the Batch Loss Score (BLS), a computationally efficient alternative using an Exponential Moving Average (EMA) of readily available batch losses to assign scores to individual samples. We frame the batch loss, from the perspective of a single sample, as a noisy measurement of its scaled individual loss, with noise originating from stochastic batch composition. It is formally shown that the EMA mechanism functions as a first-order low-pass filter, attenuating high-frequency batch composition noise. This yields a score approximating the smoothed and persistent contribution of the individual sample to the loss, providing a theoretical grounding for BLS as a proxy for sample importance. BLS demonstrates remarkable code integration simplicity (three-line injection) and readily adapts existing per-sample loss-based methods (one-line proxy). Its effectiveness is demonstrated by enhancing two such methods to losslessly prune 20%-50% of samples across 14 datasets, 11 tasks and 18 models, highlighting its utility and broad applicability, especially for complex scenarios where per-sample loss is difficult to access. Code is available at https://github.com/mrazhou/BLS.