SelecTKD：面向大语言模型的选择性词元加权知识蒸馏
SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs
对做多模态大模型压缩的视觉研究者，可把教师验证的接受/拒绝词元掩码机制直接套到 VLM 蒸馏上，替代对所有词元一视同仁的 KL 损失，在容量差距大时抑制高熵噪声监督。
原文依据 Accepted tokens receive full loss, while rejected tokens are masked or down-weighted.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论知识蒸馏（KD）是将大语言模型（LLM）压缩为紧凑学生模型的标准途径，然而大多数流程无论教师置信度如何都统一施加词元级损失。这种不加区分的监督会放大带噪的高熵信号，在教师与学生容量差距较大时尤为有害。我们提出 SelecTKD，一个即插即用的选择性词元加权蒸馏框架，它将关注点从「如何度量散度」转向「在何处施加学习」。在每一步中，学生提出词元，教师通过一种稳健的「提议-验证」流程对其进行验证，该流程有两种变体：贪心 Top-k 与非贪心 Spec-k。被接受的词元获得完整损失，被拒绝的词元则被掩码或降权。这一与目标函数无关的设计可同时适用于同策略与异策略数据，诱导出一种由词元接受率（TAR）量化的隐式课程，并稳定优化过程。在指令遵循、数学推理、代码生成以及一个视觉语言模型设定中，SelecTKD 一致地改进了强基线，并在无需架构改动或额外参考模型的情况下，为小模型取得了最优结果。
英文原文摘要 1189 字符
Knowledge distillation (KD) is a standard route to compress Large Language Models (LLMs) into compact students, yet most pipelines uniformly apply token-wise loss regardless of teacher confidence. This indiscriminate supervision amplifies noisy, high-entropy signals and is especially harmful under large teacher-student capacity gaps. We introduce SelecTKD, a plug-and-play Selective Token-Weighted distillation framework that shifts the focus from "how to measure divergence" to "where to apply learning". At each step, the student proposes tokens that are verified by the teacher through a robust propose-and-verify procedure with two variants: greedy Top-k and non-greedy Spec-k. Accepted tokens receive full loss, while rejected tokens are masked or down-weighted. This objective-agnostic design works with on- and off-policy data, induces an implicit curriculum quantified by Token Acceptance Rate (TAR), and stabilizes optimization. Across instruction following, mathematical reasoning, code generation, and a VLM setting, SelecTKD consistently improves strong baselines and achieves state-of-the-art results for small models without architectural changes or extra reference models.