重新利用丢失的文本层用于无源跨域少样本学习
Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learning
揭示了CLIP文本编码器中间层在跨域少样本学习中的潜在价值，并提出通过层级别和编码器级别的引导来重新利用这些信息，为视觉分支在域偏移下的再学习提供了新思路。
原文依据 removing certain middle layers of the text encoder can effectively improve performance in SF-CDFSL, which we call the Lost Layers.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论无源跨域少样本学习（SF-CDFSL）专注于利用目标域（如医学或卫星图像）中有限的训练数据进行微调，其中CLIP由于其对下游任务的泛化能力而最近显示出有希望的结果。当前工作表明CLIP的文本编码器更适合跨域任务，然而，我们发现移除文本编码器的某些中间层可以有效提升SF-CDFSL的性能，我们将这些层称为丢失层。本文深入研究了这一现象以加深理解。我们发现，这些层中的信息实际上对SF-CDFSL任务有益而非有害，但视觉差距阻止了这些有用信息被充分利用，使得这些层看起来冗余。基于这一理解，与当前简单移除这些层的工作不同，我们提出了一种方法，在层级别和编码器级别上教导模型重新利用这些丢失层中的信息，引导视觉分支在域偏移下的再学习。我们的方法有效解决了文本编码器中信息未充分利用的问题。在各种设置、骨干网络（CLIP、SigLip、PE-Core）和任务（4个CDFSL数据集和10个Meta-dataset数据集）上的大量实验证明了我们方法的有效性。我们将发布代码。
英文原文摘要 1377 字符
Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where CLIP has recently shown promising results due to its generalizability to downstream tasks. Current works indicate CLIP's text encoder is more suitable for cross-domain tasks, however, we find that removing certain middle layers of the text encoder can effectively improve performance in SF-CDFSL, which we call the Lost Layers. In this paper, we delve into this phenomenon for a deeper understanding. We discover that instead of being harmful for the SF-CDFSL task, the information in these layers is actually beneficial, but visual gaps prevent this useful information from being fully utilized, making these layers seem redundant.Based on this understanding, unlike current works that simply remove these layers, we propose a method to teachs the model to re-utilize information in these lost layersat both the layer and encoder levels, guiding the re-learning of the visual branch under domain shifts. Our approach effectively addresses the issue of underutilized information in the text encoder. Extensive experiments across various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 10 Meta-dataset datasets) demonstrate the effectiveness of our method. We will release the code.