应对无源跨域少样本学习中加剧的注意力汇聚问题
Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning
揭示了CLIP类VLM在目标域少样本微调时注意力汇聚被加剧、导致类间判别性下降的问题，并给出按token与目标类相关性动态重加权的通用微调策略，可直接用于任何需要把CLIP迁移到新域的少样本适配场景。
原文依据 standard target-domain few-shot fine-tuning in CDFSL significantly exacerbates the attention sink problem
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论以CLIP为代表的视觉语言模型（VLM）已展现出令人印象深刻的泛化能力，但其在跨域少样本学习（CDFSL）中的潜力仍未被充分探索；在该设定下，模型需要将源域信息迁移到训练数据稀缺的目标域。尽管注意力汇聚现象已在VLM的某些任务中被观察到，但其在CDFSL场景中的作用尚未得到研究。本文揭示了一个被先前工作忽视的关键问题：CDFSL中标准的目标域少样本微调会显著加剧注意力汇聚问题，导致类间判别性变差。为理解这一现象，我们通过大量实验将其解释为模型为域自适应而进行的捷径学习：为克服源域与目标域之间的巨大域差距，模型表现出很强的倾向，将初始就与目标域类别更接近的token（即简单token）进一步推向这些类别，从而加剧注意力汇聚，并浪费了学习其他具有判别性但初始较远的token（即困难token）的能力。为解决该问题，我们提出一种新方法，在目标域微调过程中依据token与目标域类别的相关性对其动态重加权，从而显式抑制模型对这些简单token的依赖并增强对困难token的学习，减少汇聚token并提升判别性。在四个基准数据集上的大量实验验证了本文方法的合理性，并展示了新的最优性能。代码已发布于 https://github.com/shuaiyi308/TIR。
英文原文摘要 1675 字符
Vision-language models (VLMs) like CLIP have shown impressive generalization capabilities, yet their potential for Cross-Domain Few-Shot Learning (CDFSL) remains underexplored, where the model needs to transfer source-domain information to target domains with scarce training data. While the attention sink phenomenon has been observed in VLMs for certain tasks, its role in CDFSL scenarios has not been studied. In this paper, we uncover a critical issue overlooked by prior works: standard target-domain few-shot fine-tuning in CDFSL significantly exacerbates the attention sink problem, leading to poor discriminability across classes. To understand this phenomenon, through extensive experiments, we interpret it as the model's shortcut learning for domain adaptation: to overcome the huge domain gap between the source and target domains, the model shows a high tendency to push tokens that are initially closer to target-domain classes (i.e., simple tokens) to be even closer to these classes, exacerbating the attention sink and wasting the capability of learning other discriminative but initially further tokens (i.e., hard tokens). To address this, we propose a novel approach to dynamically re-weight tokens according to their relevance with target-domain classes during the target-domain finetuning, which explicitly suppresses the model's reliance on these simple tokens and enhances the learning of hard tokens, reducing sink tokens and enhancing discriminability. Extensive experiments on four benchmark datasets validate the rationale of our method, demonstrating new state-of-the-art performance. Our codes are available at https://github.com/shuaiyi308/TIR.