基于典型相关分析的 CLIP 持续学习子空间对齐
Subspace Alignment for CLIP-based Continual Learning via Canonical Correlation Analysis
针对 CLIP 单模态持续学习中视觉分支漂移强于文本分支的「非对称漂移」问题，用跨任务累积的视觉-文本协方差求解 CCA 共享子空间，在不改动 CLIP 参数、不用样本回放的前提下恢复跨模态对齐；这一子空间对齐思路可直接迁移到任何需要冻结主干、只做后处理投影的持续学习或跨模态检索场景。
原文依据 we propose CCA-CL, a framework that accumulates visual-textual covariance statistics across tasks and solves Canonical Correlation Analysis to compute a shared subspace
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期基于 CLIP 的持续学习进展显示了利用预训练视觉-语言模型处理序列任务的潜力。然而，现有方法忽略了一个我们称之为非对称漂移的关键问题。在单模态 CLIP 持续学习中，由于视觉分布发生显著偏移，视觉分支经历了更强的适应，而文本分支因文本提示的低方差而保持相对稳定。这种不平衡增大了模态间距离，并随时间推移削弱跨模态对齐。为解决该问题，我们提出 CCA-CL 框架，它跨任务累积视觉-文本协方差统计量，并求解典型相关分析以计算共享子空间。在该子空间中，视觉与文本特征之间的距离被最小化，从而在不修改 CLIP 参数的情况下实现更好的对齐。这也使我们的方法天然兼容无样本回放的持续学习设定。为进一步捕捉线性典型相关分析难以建模的非线性关系，我们引入随机傅里叶投影作为扩展。实验结果表明，CCA-CL 有效缓解了非对称漂移问题，并在多个基准上取得最优性能。
英文原文摘要 1287 字符
Recent advances in CLIP-based continual learning have shown the potential of leveraging pre-trained vision-language models for sequential tasks. However, existing methods overlook a key problem we call Asymmetric Drift. In unimodal CLIP-based continual learning, the visual branch undergoes stronger adaptation because the visual distribution shifts significantly, whereas the text branch remains relatively stable due to the low variance of textual prompts. This imbalance increases the modality distance and degrades cross-modal alignment over time. To address this issue, we propose CCA-CL, a framework that accumulates visual-textual covariance statistics across tasks and solves Canonical Correlation Analysis to compute a shared subspace. In this subspace, the distance between visual and textual features is minimized, enabling better alignment without modifying CLIP parameters. This also makes our method naturally compatible with exemplar-free CL settings. To further capture nonlinear relationships that linear Canonical Correlation Analysis is hard to model, we introduce Random Fourier Projection as an extension. Experimental results demonstrate that CCA-CL effectively mitigates the asymmetric drift problem and achieves state-of-the-art performance on several benchmarks.