面向来自不同领域教师的持续蒸馏
Continual Distillation of Teachers from Different Domains
把「教师模型流式到达、且无法访问旧教师与教师训练数据」的持续蒸馏设定引入视觉模型压缩，用外部无标签数据上的 logits 保持来缓解跨域知识遗忘，可直接用于多源教师（如不同域检测/分割模型）合并到单一学生网络的场景。
原文依据 we propose Self External Data Distillation (SE2D), a method that preserves logits on external data to stabilize learning across heterogeneous teachers
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论深度学习模型规模持续扩大，部分模型的存储需求甚至超过许多大规模数据集。为此，我们提出一种新范式：持续蒸馏（Continual Distillation, CD），即学生模型从一系列教师模型中顺序学习，且不再保留对先前教师的访问权限。CD 面临两个挑战：教师训练数据不可得，且各教师专长不同。我们表明，外部无标签数据能够实现未见知识迁移（Unseen Knowledge Transfer, UKT），使学生模型获取训练数据中不存在、但教师已知的信息。我们还表明，顺序蒸馏会导致未见知识遗忘（Unseen Knowledge Forgetting, UKF），即在后续教师上训练后已迁移的知识被丢失。为更好地权衡 UKT 与 UKF，我们提出自外部数据蒸馏（Self External Data Distillation, SE2D），该方法在外部数据上保留 logits，以稳定跨异构教师的学习。在多个基准上的实验表明，SE2D 减少了 UKF 并提升了跨域泛化能力。本工作的代码与实现已公开于：https://github.com/Nicolas1203/continual_distillation。
英文原文摘要 1151 字符
Deep learning models continue to scale, with some requiring more storage than many large-scale datasets. Thus, we introduce a new paradigm: Continual Distillation (CD), where a student learns sequentially from a stream of teacher models without retaining access to earlier teachers. CD faces two challenges: teacher training data is unavailable, and teachers have varying expertise. We show that external unlabeled data enables Unseen Knowledge Transfer (UKT), allowing the student to acquire information from domains not present in the training data, while known to the teacher. We also show that sequential distillation causes Unseen Knowledge Forgetting (UKF) when transferred knowledge is lost after training on later teachers. To better trade off between UKT and UKF, we propose Self External Data Distillation (SE2D), a method that preserves logits on external data to stabilize learning across heterogeneous teachers. Experiments on multiple benchmarks show that SE2D reduces UKF and improves cross-domain generalization. The code and implementation for this work are publicly available at: https://github.com/Nicolas1203/continual_distillation