面向零样本组合图像检索的自引导语义检查
Self-guided Semantic Inspection for Zero-Shot Composed Image Retrieval
DiffComp 用「短语引导掩码」主动制造图文语义差异，再以差异感知组合模块按差异程度自适应融合特征，这套「先制造差异再组合」的自监督训练范式可直接迁移到任何需要图文联合推理的检索或组合式生成任务中。
原文依据 Phrase-guided Masking that selectively removes text-aligned visual cues to induce controlled cross-modal discrepancies
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论零样本组合图像检索（ZS-CIR）旨在使用由参考图像和文本修改组成的组合查询来检索目标图像，而不依赖基于三元组的监督。由于两种输入描述的是相关但语义未对齐的信息，关键挑战在于解读它们之间的跨模态差异，以推断用户意图的语义修改。现有的 ZS-CIR 方法主要采用一致性驱动的范式，在语义对齐的图像-文本对上以对齐或重建目标进行训练。该范式强制跨模态一致，却忽略了推理过程中自然出现的模态间语义差异。为解决这一问题，我们提出 DiffComp（先区分后组合），一个差异驱动的自监督框架，在训练中主动诱导并利用跨模态差异。它激励模型感知并调和视觉与文本模态之间的语义差异，从而提升训练与推理之间的一致性。该框架由三个组件构成：上下文语义超图块，为下游感知与组合提供局部且连贯的视觉表示；短语引导掩码，选择性地移除与文本对齐的视觉线索以诱导受控的跨模态差异；差异感知组合，根据语义差异程度自适应地整合视觉与文本特征。在四个 ZS-CIR 基准上的大量实验表明，DiffComp 达到最优性能并具有强泛化能力。我们的代码将发布于 https://github.com/Orange1999/DiffComp。
英文原文摘要 1702 字符
Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve target images using a composed query of a reference image and a textual modification, without relying on triplet-based supervision. As the two inputs describe related but semantically unaligned information, the key challenge lies in interpreting their cross-modal discrepancy to infer the user's intended semantic modification. Existing ZS-CIR methods mainly adopt a consistency-driven paradigm, training on semantically aligned image-text pairs with alignment or reconstruction objectives. This paradigm enforces cross-modal agreement but overlooks the semantic discrepancies between modalities that naturally arise during inference. To address this issue, we propose DiffComp (Differentiate-then-Compose), a difference-driven self-supervised framework that actively induces and exploits cross-modal discrepancies during training. It stimulates the model to perceive and reconcile semantic differences across visual and textual modalities, thereby improving consistency between training and inference. The framework consists of three components: Contextual Semantic Super-patches that provide localized and coherent visual representations for downstream perception and composition; Phrase-guided Masking that selectively removes text-aligned visual cues to induce controlled cross-modal discrepancies; and Difference-aware Composition that adaptively integrates visual and textual features according to their degree of semantic difference. Extensive experiments on four ZS-CIR benchmarks show that DiffComp achieves state-of-the-art performance and strong generalization. Our code will be released at https://github.com/Orange1999/DiffComp.