精简知识蒸馏
Streamlined Knowledge Distillation
提供了一种仅用 logit 的蒸馏损失设计：用归一化 logit 的 Gram 矩阵加马氏距离（Tikhonov 正则化 + Cholesky 分解稳定）传递方向性知识，可即插即用地替换视觉模型压缩中常用的特征蒸馏，省去多层特征对齐的工程开销。
原文依据 we propose Streamlined Knowledge Distillation (SKD), a simple yet effective logit-based method that transfers only two essential forms of knowledge
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于 logit 的知识蒸馏（KD）已成为基于特征的知识蒸馏的一种轻量替代方案。近期的基于 logit 的方法往往依赖多知识对齐与关系建模。由于目标冗余、变换次优以及损失函数设计不佳，这些方法通常效率低下。受这些问题启发，我们提出精简知识蒸馏（SKD），一种简单而有效的基于 logit 的方法，它只传递两种本质形式的知识，无需额外的对齐或关系建模。具体而言，SKD 通过 Kullback-Leibler 散度传递实例级知识，并通过对齐归一化 logit 的 Gram 矩阵传递方向级知识。对于后者，我们引入基于马氏距离的方向级损失，并通过 Tikhonov 正则化与 Cholesky 分解使其稳定。该方向级损失考虑了输出空间中的方差与相关性，并且正如我们形式化证明的那样，它等价于协方差白化空间中的 L2 范数。大量实验表明，尽管设计更为简单，SKD 始终优于现有的基于 logit 的方法，甚至超越基于特征的方法。代码见 https://github.com/HyunJunSik/StreamLined。
英文原文摘要 1276 字符
Logit-based Knowledge Distillation (KD) has emerged as a lightweight alternative to feature-based KD. Recent logit-based methods often rely on multi-knowledge alignment and relational modeling. These methods are often inefficient due to redundant objectives, suboptimal transformations, and poorly designed loss functions. Motivated by these issues, we propose Streamlined Knowledge Distillation (SKD), a simple yet effective logit-based method that transfers only two essential forms of knowledge without requiring additional alignment or relational modeling. Specifically, SKD transfers instance-wise knowledge via Kullback-Leibler divergence and direction-wise knowledge by aligning the Gramian matrix of normalized logits. For the latter, we introduce a Mahalanobis distance-based direction-wise loss stabilized through Tikhonov regularization and Cholesky decomposition. This direction-wise loss accounts for variance and correlation in the output space and, as we formally show, is equivalent to the L2-norm in a covariance-whitened space. Extensive experiments demonstrate that SKD consistently outperforms existing logit-based methods and even surpasses feature-based methods, despite its simpler design. Code is available at https://github.com/HyunJunSik/StreamLined.