语言在跨域少样本视觉特征增强中至关重要
Language Does Matter for Cross-Domain Few-Shot Visual Feature Enhancement
该论文提出用图像描述模型和LLM分别提取图像级与域级语言属性，并通过轻量残差交叉注意力将语言描述嵌入视觉特征，以增强跨域少样本场景下的视觉特征泛化能力。这一思路可复用于任何需要提升预训练视觉特征跨域迁移性的视觉任务，如分类、分割和检测。
原文依据 A lightweight residual cross-attention scheme is developed to seamlessly embed linguistic descriptions of image attributes into visual feature representations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论跨域少样本图像解译（CD-FSII）通过使用目标域中有限的标注样本微调预训练视觉特征模型而取得了显著进展。然而，巨大的跨域分布差异，以及大量物体视觉外观变化与有限标注之间的内在冲突，使现有的纯视觉特征表示陷入不可迁移的捷径模式，从而降低了其跨域泛化能力。为缓解这一问题，我们提出一个简单而有效的跨模态视觉特征增强框架，其主要贡献体现在以下三个方面。1）我们首次尝试引入图像属性的语言描述来调控预训练视觉特征模型，以实现对特定目标图像的适应。具体而言，分别使用预训练图像描述模型和大语言模型（LLM）提取图像级属性（例如单张图像中的物体外观）和域级属性（例如数据集的整体风格和背景特征），以构建全面的语言表征。2）开发了一种轻量级残差交叉注意力方案，将图像属性的语言描述无缝嵌入视觉特征表示中，从而弥补纯视觉线索在捕获跨域可迁移高层语义特征方面的局限性。3）所提框架与任务无关，可无缝集成到现成的预训练视觉特征模型中。在多个CD-FSII基准（包括图像分类、语义分割和目标检测）上，它展现出优于多种最优方法的泛化性能。我们将发布所有代码和数据以促进进一步研究。
英文原文摘要 1904 字符
Cross-domain few-shot image interpretation (CD-FSII) has been significantly advanced by fine-tuning pre-trained visual feature models using limited labeled samples in target domains. However, profound cross-domain distribution discrepancies, along with inherent conflicts between extensive object visual appearance variations and limited annotations, trap those existing pure visual feature representations into some non-transferable short-cut patterns, thus degrading their cross-domain generalization capacity. To mitigate this problem, we present a simple yet effective cross-modal visual feature enhancement framework which primarily contributes in the following three aspects. 1) We make the first attempt to introduce linguistic descriptions of image attributes to regulate the pre-trained visual feature model for specific target image adaptation. Specifically, image-level attributes (e.g., object appearance in individual images) and domain-level attributes (e.g., overall style and background characteristics of the dataset) are extracted using a pre-trained image captioning model and a large language model (LLM), respectively, to construct comprehensive linguistic characterizations. 2) A lightweight residual cross-attention scheme is developed to seamlessly embed linguistic descriptions of image attributes into visual feature representations, thereby compensating for the limitations of purely visual cues in capturing cross-domain transferable high-level semantic characteristics. 3) The proposed framework is task-agnostic and can be seamlessly integrated with off-the-shelf pre-trained visual feature models. It demonstrates superior generalization performance compared to several state-of-the-art methods across multiple CD-FSII benchmarks, including image classification, semantic segmentation, and object detection. We will release all code and data to facilitate further research.