自我学习：从视觉语言模型内部挖掘知识用于持续学习
Learning from Itself: Mining Internal Knowledge from Vision Language Models for Continual Learning
把 CLIP 的文本编码器当作可挖掘的内部监督源：通过优化可学习 token 生成伪描述来弥合预训练-下游分布差距，并用按瞬时性能加权的双向蒸馏在视觉分类器与文本编码器之间迁移知识，推理时零额外开销——这套「模型自教」流程可直接迁移到任何需要持续适配的视觉-语言下游任务。
原文依据 we generate pseudo-captions by optimizing learnable tokens to minimize CLIP's contrastive loss
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论CLIP 等视觉语言模型在零样本识别上表现优异，但在持续学习中却举步维艰，原因有两个关键问题：(1) 预训练描述与后训练类别名称之间存在严重的分布差距；(2) 纯视觉方法与双编码器方法之间存在性能失配——纯视觉方法在细粒度任务上准确率高出 20%，而 CLIP 在自然图像上占优。我们提出「自我学习」（Learning from Itself, LfI），通过挖掘 CLIP 的内部知识来同时应对这两个挑战。首先，我们通过优化可学习 token 以最小化 CLIP 的对比损失来生成伪描述，从而构造出无需外部模型的辅助训练信号，弥合预训练与微调之间的分布差距。其次，我们引入自适应互蒸馏，依据 CLIP 文本编码器与一个临时视觉分类器的瞬时表现，动态加权二者之间的知识迁移——更强的分支教得更多，更弱的分支学得更多。推理阶段仅使用原始 CLIP 架构，此时它已吸收了来自两个分支的判别性知识。LfI 在多个持续学习基准上取得最优结果，表明 CLIP 能够有效地自我教导以持续学习新任务。代码见：https://github.com/jordangong/continual-learning。
英文原文摘要 1343 字符
Vision-language models like CLIP excel at zero-shot recognition but struggle with continual learning due to two critical issues: (1) severe distribution gap between pretraining captions and post-training class names, and (2) performance mismatch between vision-only and dual-encoder approaches--vision-only methods achieve 20% higher accuracy on fine-grained tasks while CLIP dominates on natural images. We propose Learning from Itself (LfI), which mines CLIP's internal knowledge to address both challenges. First, we generate pseudo-captions by optimizing learnable tokens to minimize CLIP's contrastive loss, creating auxiliary training signals that bridge the pretraining-finetuning distribution gap without external models. Second, we introduce adaptive mutual distillation that dynamically weights knowledge transfer between CLIP's text encoder and a temporary vision classifier based on their instantaneous performance--stronger branches teach more, weaker ones learn more. At inference, only the original CLIP architecture is used, having absorbed discriminative knowledge from both branches. LfI achieves state-of-the-art results across multiple continual learning benchmarks, demonstrating that CLIP can effectively teach itself to continually learn new tasks. Code is available at: https://github.com/jordangong/continual-learning.