从有偏教师中蒸馏平衡知识
Distilling Balanced Knowledge from a Biased Teacher
把蒸馏目标拆成跨组损失与组内损失，并分别做组级校准与重加权，可直接用于任何长尾视觉分类任务中从有偏教师迁移平衡知识，也可迁移到检测/分割的长尾蒸馏。
原文依据 reformulates the conventional objective into two components: a cross-group loss
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论面向模型压缩的传统知识蒸馏在长尾分布上失效，因为教师模型往往偏向头部类别，对尾部类别提供的监督有限。我们提出长尾知识蒸馏（LTKD），该框架将传统目标重构为两个部分：跨组损失，刻画不同类别组（头部、中部、尾部）之间预测分布的失配；组内损失，刻画每个组内部分布之间的差异。这一分解揭示了教师偏差的具体来源。为缓解继承而来的偏差，LTKD 引入（1）重新平衡的跨组损失，用于校准教师的组级预测；（2）重新加权的组内损失，确保所有组的贡献相等。在 CIFAR-100-LT、TinyImageNet-LT 和 ImageNet-LT 上的大量实验表明，LTKD 在总体精度和尾部类别精度上均显著优于现有方法，从而表明其能够从有偏教师中蒸馏出平衡知识，适用于真实世界应用。
英文原文摘要 1137 字符
Conventional knowledge distillation, designed for model compression, fails on long-tailed distributions because the teacher model tends to be biased toward head classes and provides limited supervision for tail classes. We propose Long-Tailed Knowledge Distillation (LTKD), a novel framework that reformulates the conventional objective into two components: a cross-group loss, capturing mismatches in prediction distributions across class groups (head, medium, and tail), and a within-group loss, capturing discrepancies within each group's distribution. This decomposition reveals the specific sources of the teacher's bias. To mitigate the inherited bias, LTKD introduces (1) a rebalanced cross-group loss that calibrates the teacher's group-level predictions and (2) a reweighted within-group loss that ensures equal contribution from all groups. Extensive experiments on CIFAR-100-LT, TinyImageNet-LT, and ImageNet-LT demonstrate that LTKD significantly outperforms existing methods in both overall and tail-class accuracy, thereby showing its ability to distill balanced knowledge from a biased teacher for real-world applications.