可解释的跨域少样本学习与校正目标域局部对齐
Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment
针对CLIP在跨域少样本场景下局部视觉-文本对齐失效的问题，提出基于循环一致性的自监督对齐方法，并引入语义锚框机制过滤噪声，可提升医学等细粒度领域的可解释识别性能。
原文依据 the domain gap and scarce training data further exacerbate such shortcomings, much more than that of holistic patterns
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论跨域少样本学习（CDFSL）将在大规模通用数据（源域）上训练的模型适配到仅有稀缺训练数据的下游目标域，其中对视觉-语言模型（如CLIP）的研究仍处于早期阶段。典型的下游领域（如医学诊断）需要细粒度视觉线索以实现可解释识别，但我们发现当前微调的CLIP模型几乎无法关注这些线索，尽管它们能大致关注源域中的重要区域。虽然现有工作已证明CLIP在捕捉局部细微模式方面的不足，但在本文中我们发现，域间差距和稀缺训练数据进一步加剧了这种不足，且远甚于对整体模式的不足，我们将其称为基于CLIP的CDFSL中的局部错位问题。为解决该问题，由于缺乏对齐局部视觉特征与文本语义的监督，我们转向自监督信息。受翻译任务启发，我们提出具有循环一致性的CC-CDFSL方法，该方法将局部视觉特征翻译为文本特征再翻译回视觉特征（反之亦然），并约束原始特征接近翻译回的特征。为减少视觉模态中更丰富信息引入的噪声，我们进一步提出语义锚框机制，该机制首先增强视觉特征以为文本到图像映射提供更大语料，然后收缩图像特征以过滤不相关的图像到文本映射。在各种基准、骨干网络和微调方法上的大量实验表明，我们能够（1）有效改善局部视觉-语言对齐，（2）通过可视化图像块增强所学模式和模型决策的可解释性，（3）达到最优性能。我们的代码将公开。
英文原文摘要 1907 字符
Cross-Domain Few-Shot Learning (CDFSL) adapts models trained with large-scale general data (source domain) to downstream target domains with only scarce training data, where the research on vision-language models (e.g., CLIP) is still in the early stages. Typical downstream domains, such as medical diagnosis, require fine-grained visual cues for interpretable recognition, but we find that current fine-tuned CLIP models can hardly focus on these cues, albeit they can roughly focus on important regions in source domains. Although current works have demonstrated CLIP's shortcomings in capturing local subtle patterns, in this paper, we find that **the domain gap and scarce training data further exacerbate such shortcomings, much more than that of holistic patterns**, which we call the local misalignment problem in CLIP-based CDFSL. To address this problem, due to the lack of supervision in aligning local visual features and text semantics, we turn to self-supervision information. Inspired by the translation task, we propose the CC-CDFSL method with cycle consistency, which translates local visual features into text features and then translates them back into visual features (and vice versa), and constrains the original features close to the translated back features.To reduce the noise imported by richer information in the visual modality, we further propose a Semantic Anchor mechanism, which first augments visual features to provide a larger corpus for the text-to-image mapping, and then shrinks the image features to filter out irrelevant image-to-text mapping. Extensive experiments on various benchmarks, backbones, and fine-tuning methods show we can (1) effectively improve the local vision-language alignment, (2) enhance the interpretability of learned patterns and model decisions by visualizing patches, and (3) achieve state-of-the-art performance. Our codes will be released.