MuCo：面向多模态嵌入模型的多轮对比学习
MuCo: Multi-turn Contrastive Learning for Multimodal Embedding Model
MuCo 将多轮对话结构引入对比学习，在单次前向中处理多个相关查询-目标对，可复用于视觉-语言检索模型的训练效率提升与嵌入一致性增强。
原文依据 MuCo leverages the conversational nature of MLLMs to process multiple, related query-target pairs associated with a single image within a single forward pass.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于多模态大语言模型（MLLMs）构建的通用多模态嵌入模型传统上采用对比学习，以对齐跨不同模态的查询-目标对的表示。然而，尽管其取得了经验上的成功，这些模型主要建立在“单轮”范式之上，即每个查询-目标对被当作独立的数据点。这种范式在扩展时导致计算效率低下，因为它需要对每一对进行单独的前向传播，并且忽略了多个查询之间可能存在的、与同一上下文相关的潜在上下文关系。在本工作中，我们引入了多轮对比学习（MuCo），一个受对话启发的框架，重新审视了这一过程。MuCo 利用 MLLMs 的对话特性，在单次前向传播中处理与单张图像相关的多个相关查询-目标对。这使我们能够同时提取一组多个查询和目标嵌入，并以共享的上下文表示为条件，从而放大有效批大小和整体训练效率。实验展示了 MuCo 与一个新构建的 500 万多模态多轮数据集（M3T）的结合，在 MMEB 和 M-BEIR 基准上取得了最优的检索性能，同时显著提升了训练效率和跨模态的表示一致性。代码和 M3T 可在 https://github.com/naver-ai/muco 获取。
英文原文摘要 1426 字符
Universal Multimodal embedding models built on Multimodal Large Language Models (MLLMs) have traditionally employed contrastive learning, which aligns representations of query-target pairs across different modalities. Yet, despite its empirical success, they are primarily built on a "single-turn" formulation where each query-target pair is treated as an independent data point. This paradigm leads to computational inefficiency when scaling, as it requires a separate forward pass for each pair and overlooks potential contextual relationships between multiple queries that can relate to the same context. In this work, we introduce Multi-Turn Contrastive Learning (MuCo), a dialogue-inspired framework that revisits this process. MuCo leverages the conversational nature of MLLMs to process multiple, related query-target pairs associated with a single image within a single forward pass. This allows us to extract a set of multiple query and target embeddings simultaneously, conditioned on a shared context representation, amplifying the effective batch size and overall training efficiency. Experiments exhibit MuCo with a newly curated 5M multimodal multi-turn dataset (M3T), which yields state-of-the-art retrieval performance on MMEB and M-BEIR benchmarks, while markedly enhancing both training efficiency and representation coherence across modalities. Code and M3T are available at https://github.com/naver-ai/muco