STiTch：协同语义过渡与传输的免训练零样本组合图像检索
STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrieval
把检索从点对点匹配改成基于最优传输的集合到集合对齐，并用嵌入空间中的过渡向量修正 LLM 生成的描述，这两点可直接迁移到任何图文检索或跨模态匹配任务中。
原文依据 we model the caption and image as discrete distributions and reformulate the retrieval task as a set-to-set alignment task
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论免训练零样本组合图像检索模型因其在未见多模态检索中的泛化性与灵活性，近来受到越来越多的研究关注。近期基于大语言模型的进展侧重于挖掘大语言模型背后的组合能力来生成期望的目标描述。尽管高效，我们发现：1）由于输入图像与文本修改之间存在语义鸿沟——图像包含的细节远多于文本——生成的描述往往会引入来自参考图像的意外特征；2）检索阶段的点对点对齐无法捕捉多样的组合。为解决这些挑战，我们提出一种新颖的协同语义过渡与传输框架，用于免训练零样本组合图像检索任务。具体而言，给定由大语言模型推断出的组合描述，我们旨在通过嵌入空间中的过渡向量对其进行精炼，使其更接近目标图像。将大语言模型与用户指令相结合，精炼后的描述更聚焦于核心修改意图，从而滤除不必要的噪声。此外，为在检索阶段探索多样的对齐，我们将描述与图像建模为离散分布，并将检索任务重新表述为集合到集合的对齐任务。最后，我们提出一种双向传输距离，以考虑跨模态的细粒度对齐并计算检索得分。大量实验表明，我们的方法具有通用性、有效性，并对许多组合图像检索任务有益。代码见补充材料。
英文原文摘要 1639 字符
Training-free zero-shot composed image retrieval models are recently gaining increasing research interest due to their generalizability and flexibility in unseen multimodal retrieval. Recent LLM-based advances focus on generating the expected target caption by exploring the compositional ability behind the LLMs. Although efficient, we find that 1) the generated captions tend to introduce unexpected features from the reference image due to the semantic gap between the input image and text modification, where the image contains much more details than the text; 2) the point-to-point alignment during the retrieval stage fails to capture diverse compositions.To address these challenges, we introduce a novel Semantic Transition and Transportation in Collaboration framework for training-free zero-shot CIR tasks. Specifically, given the composed caption inferred by an LLM, we aim to refine it through a transition vector in the embedding space and make it closer to the target image. Combining LLMs with user instruction, the refined caption concentrates more on the core modification intent and thus filters out unnecessary noise. Moreover, to explore diverse alignment during the retrieval stage, we model the caption and image as discrete distributions and reformulate the retrieval task as a set-to-set alignment task. Finally, a bidirectional transportation distance is developed to consider fine-grained alignments across modalities and calculate the retrieval score.Extensive experimentsdemonstrate that our method can be general, effective, and beneficial for many CIR tasks.The code is attached in the supplementary material.