面向多模态大语言模型的不确定性感知知识蒸馏
Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
把教师监督视为学生激活上的 Gibbs 先验，可导出闭式的不确定性感知权重，可直接套用到任意蒸馏损失上；做 VLM 蒸馏或小模型压缩时，可用它按样本自适应调节教师信号强度，避免噪声样本与教师不确定样本的过度影响。
原文依据 we interpret teacher supervision as a Gibbs prior over student activations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论知识蒸馏建立了一种同时利用数据监督与教师指导的学习范式。然而，在从数据学习和从教师学习之间确定最优平衡是困难的，因为某些样本可能带有噪声，而另一些样本则受制于教师的不确定性。这促使我们需要自适应地平衡数据监督与教师监督。我们提出 Beta 加权知识蒸馏（Beta-KD），一个不确定性感知的蒸馏框架，能够自适应地调节学生依赖教师指导的程度。具体而言，我们从统一的贝叶斯视角对教师-学生学习进行建模，并将教师监督解释为学生激活上的 Gibbs 先验。由此得到一个闭式的不确定性感知加权机制，并支持任意蒸馏目标及其组合。我们在多模态 VQA 基准上开展了大量实验，通过从大型教师视觉语言模型蒸馏出一个学生视觉语言模型。结果表明，Beta-KD 持续优于现有的知识蒸馏方法。代码见 https://github.com/Jingchensun/beta-kd。
英文原文摘要 1164 字符
Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is challenging, as some samples may be noisy while others are subject to teacher uncertainty. This motivates the need for adaptively balancing data and teacher supervision. We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that adaptively modulates how much the student relies on teacher guidance. Specifically, we formulate teacher-student learning from a unified Bayesian perspective and interpret teacher supervision as a Gibbs prior over student activations. This yields a closed-form, uncertainty-aware weighting mechanism and supports arbitrary distillation objectives and their combinations. Extensive experiments are conducted on multimodal VQA benchmarks by distilling a student Vision-Language Model from a large teacher VLM. The results demonstrate that Beta-KD consistently outperforms existing knowledge distillation methods. Code is available at https://github.com/Jingchensun/beta-kd.