适配上下文生成以增强组合图像检索
Adapting In-context Generation for Enhanced Composed Image Retrieval
利用预训练文本到图像模型的上下文生成能力，在少样本下合成无偏的组合图像检索三元组，并通过分布鲁棒合成预训练和细粒度真实世界适配两阶段框架提升检索性能，为视觉-语言检索任务提供可复用的数据增强与微调策略。
原文依据 we shift the focus to developing robust CIR models under limited labeled data and propose Domain-Adaptive In-context Generation (DAIG)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论作为一项具有挑战性的视觉-语言任务，组合图像检索（CIR）旨在整合双模态查询（图像+文本）的信息以检索目标图像。监督式CIR在特定领域场景中已取得显著成功，但其对人工标注三元组的依赖限制了可扩展性和应用。零样本CIR通过利用无标签数据或自动收集的三元组来缓解这一问题，但往往遭受难以处理的域间差距。为此，我们将重点转向在有限标注数据下开发鲁棒的CIR模型，并提出域自适应上下文生成（DAIG），该方法利用少样本样本将预训练文本到图像（T2I）模型的上下文能力适配到目标域和CIR任务，然后将LLM生成的文本三元组转换为无偏的CIR三元组作为额外训练数据。之后，我们提出一个适用于任何监督式CIR方法的两阶段框架。第一阶段，分布鲁棒合成预训练（DRSP），通过扰动视觉特征来扩展合成数据的分布并提高在其上的训练鲁棒性。第二阶段，细粒度真实世界适配（FRA），通过在匹配对上施加角度间隔来微调人工标注三元组，以促进细粒度学习。在两个基准上的实验验证了我们的方法的有效性，即在少样本和全监督CIR设置下，DAIG相较于CLIP4CIR、BLIP4CIR和SPRC均取得了显著的性能提升。代码可在 https://github.com/JThuge/DAIG 获取。
英文原文摘要 1653 字符
As a challenge vision-language task, Composed Image Retrieval (CIR) aims to integrate information from a bi-modal query (image + text) to retrieve target images. While supervised CIR has achieved notable success in domain-specific scenarios, its reliance on manually annotated triplets restricts its scalability and application. Zero-shot CIR alleviates this by leveraging unlabeled data or automatically collected triplets, yet it often suffers from an intractable domain gap. To this end, we shift the focus to developing robust CIR models under limited labeled data and propose Domain-Adaptive In-context Generation (DAIG), which adapts the in-context capability of a pretrained Text-to-Image (T2I) model to the target domain and the CIR task using few-shot samples and then transforms the LLM-generated textual triplets into unbiased CIR triplets as additional training data. After that, we present a two-stage framework applicable to any supervised CIR approach. The first stage, Distributionally Robust Synthetic Pretraining (DRSP), perturbs visual features to expand the distribution of synthetic data and improve training robustness on it. The second stage, Fine-grained Real-world Adaptation (FRA), fine-tunes on manually annotated triplets by imposing an angular margin on matching pairs to facilitate fine-grained learning. Experiments on two benchmarks validate the effectiveness of our method, i.e., under both few-shot and fully supervised CIR settings, DAIG yields substantial performance gains over CLIP4CIR, BLIP4CIR, and SPRC. The code is available at \href https://github.com/JThuge/DAIG \color blue!37https://github.com/JThuge/DAIG .