警惕无源跨域少样本学习中的判别性陷阱
Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning
揭示了 VLM 在无源跨域少样本微调中「视觉判别性越强反而越差」的捷径现象：交叉熵损失中的视觉学习项会绕过跨模态对齐，因此做 CLIP/SigLIP 下游适配时应显式扰动视觉分支并利用图文语义关系逐步恢复模态对齐，而不是一味增强视觉特征判别性。
原文依据 the visual learning essentially acts as a shortcut that encourages the model to reduce L_ vlm without considering the cross-modal part
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论无源跨域少样本学习（SF-CDFSL）关注在仅有目标域少量训练数据（如医学或卫星图像）的条件下进行微调，其中 CLIP、SigLIP 等视觉语言模型（VLM）已展现出良好效果。传统视觉模型的相关工作表明，提升视觉判别性可以改善性能。然而，在基于 VLM 的 SF-CDFSL 任务中，我们发现增强视觉模态的判别性实际上会抑制 VLM 的性能。本文旨在深入剖析这一现象并给出解释与解决方案。通过理论证明与实验验证，我们的研究表明，使用典型交叉熵损失（L_vlm）进行微调时，损失中天然包含一个视觉学习部分和一个跨模态学习部分，其中跨模态部分对于纠正 SF-CDFSL 中被严重破坏的模态错位至关重要。但我们发现，视觉学习本质上起到捷径的作用，它促使模型在不考虑跨模态部分的情况下降低 L_vlm，从而阻碍跨模态对齐并损害性能。基于这一解释，我们进一步提出一种解决该问题的方法：首先，对视觉学习进行扰动，引导模型关注跨模态对齐；随后，在微调过程中利用视觉-文本语义关系逐步对齐视觉与文本模态。在多种设置、多种骨干网络（CLIP、SigLIP、PE-Core）以及多种任务（4 个 CDFSL 数据集和 11 个 FSL 数据集）上的大量实验表明，我们一致地刷新了最优结果。代码将开源。
英文原文摘要 1652 字符
Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where Vision-Language Models (VLMs) such as CLIP and SigLIP have shown promising results. Current works in traditional visual models suggest that improving visual discriminability enhances performance. However, in VLM-based SF-CDFSL tasks, we find that strengthening visual-modal discriminability actually suppresses VLMs' performance. In this paper, we aim to delve into this phenomenon for an interpretation and a solution.By both theoretical and experimental proofs, our study reveals that fine-tuning with the typical cross-entropy loss (L_ vlm ) inherently includes a visual learning part and a cross-modal learning part, where the cross-modal part is crucial for rectifying the heavily disrupted modality misalignment in SF-CDFSL.However, we find that the visual learning essentially acts as a shortcut that encourages the model to reduce L_ vlm without considering the cross-modal part, therefore hindering the cross-modal alignment and harming the performance.Based on this interpretation, we further propose an approach to address this problem: first, we perturb the visual learning to guide the model to focus on the cross-modal alignment. Then, we use the visual-text semantic relationships to gradually align the visual and textual modalities during the fine-tuning. Extensive experiments on various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 11 FSL datasets) show that we consistently set new state-of-the-art results. We will release the code.