ReMatch：通过匹配提升多模态检索中的表征
ReMatch: Boosting Representation through Matching for Multimodal Retrieval
将MLLM的生成能力用于多模态检索，通过生成式匹配阶段提供实例级判别监督，增强对比学习在困难负样本上的梯度；同时使用多个可学习token生成细粒度正交嵌入，为视觉-语言检索提供可复用的训练配方。
原文依据 We train the embedding MLLM end-to-end with a chat-style generative matching stage.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出ReMatch，一个利用多模态大语言模型（MLLM）的生成能力进行多模态检索的框架。先前的方法将MLLM视为简单的编码器，忽略了其生成特性，未能充分利用其组合推理和世界知识。我们通过聊天风格的生成式匹配阶段端到端地训练嵌入MLLM。匹配阶段使用相同的MLLM，从多视角输入（包括原始数据及其为每个查询和文档投影的嵌入）自回归地判断相关性。它提供实例级判别监督，补充标准对比损失，在困难负样本上提供更强的梯度，并保留原始MLLM的组合优势。为了获得语义更丰富的多模态嵌入，我们使用多个可学习token增强每个输入，以低推理成本生成细粒度上下文、相互正交的嵌入。利用我们建立的高性能基线，我们将上述想法整合为一个强大的训练配方，并在大规模多模态嵌入基准（MMEB）上取得了新的最优性能。我们的实验在五个数据集上显示出特别强的零样本泛化结果，突显了ReMatch的鲁棒性和可迁移性。
英文原文摘要 1307 字符
We present ReMatch, a framework that leverages the generative strength of MLLMs for multimodal retrieval. Previous approaches treated an MLLM as a simple encoder, ignoring its generative nature, and under-utilising its compositional reasoning and world knowledge. We train the embedding MLLM end-to-end with a chat-style generative matching stage. The matching stage uses the same MLLM to autoregressively decide relevance from multi-view inputs, including both raw data and its own projected embeddings for each query and document. It provides instance-wise discrimination supervision that complements a standard contrastive loss, offering stronger gradients on hard negatives and preserving the compositional strengths of the original MLLM. To obtain semantically richer multimodal embeddings, we use multiple learnable tokens to augment each input, generating fine-grained contextual, mutually orthogonal embeddings with low inference cost. Leveraging our established high-performance baseline, we assemble the ideas mentioned above into a powerful training recipe and achieve a new state-of-the-art on the Massive Multimodal Embedding Benchmark(MMEB). Our experiments show particularly strong zero-shot generalization results on five datasets, highlighting the robustness and transferability of ReMatch.