高效高保真的全模态检索
Efficient and High-Fidelity Omni Modality Retrieval
提出基于注意力的重采样模块将多模态编码器的长词元序列压缩为固定长度表示，并用 Attention Sliced Wasserstein Pooling 保留细粒度信息，可直接用于视觉-语言检索中降低 LLM 输入开销并提升嵌入保真度。
原文依据 We therefore introduce an attention-based resampling mechanism to generate compact, fixed-size representations from these sequences.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态检索的任务是聚合来自异构模态查询的信息以检索所需目标。最优的多模态检索模型能够理解复杂查询，但通常局限于文本和视觉两种模态。这一局限阻碍了能够理解结合两种以上模态的查询的通用检索系统的发展。为推进这一目标，我们提出 OmniRet，这是首个能够处理跨越文本、视觉和音频三个关键模态的复杂组合查询的检索模型。我们的 OmniRet 模型解决了通用检索的两个关键挑战：计算效率和表示保真度。首先，将来自模态特定编码器的大量词元序列馈入大语言模型（LLM）在计算上是低效的。因此，我们引入一种基于注意力的重采样机制，从这些序列生成紧凑的固定大小表示。该共享模块旨在保持表示多样性和泛化能力，同时对模态特定信息保持敏感。其次，将丰富的全模态数据压缩为单个嵌入向量不可避免地会造成信息损失并丢弃细粒度细节。我们提出注意力切片 Wasserstein 池化来保留这些细粒度细节，从而改进全模态表示。OmniRet 在约 600 万查询-目标对上训练，涵盖 30 个数据集。我们在 13 个检索任务和 MMEBv2 子集上对模型进行基准测试。我们的模型在组合查询、音频和视频检索任务上表现出显著改进，而在其他任务上达到与最优模型相当的性能。此外，我们策划了一个新的以音频为中心的多模态基准（ACM）。该新基准引入了两个关键的、此前缺失的任务——组合音频检索和音视频检索——以更全面地评估模型的全模态嵌入能力。我们相信我们的基准将促进通用检索系统的发展。
英文原文摘要 2139 字符
Multimodal retrieval is the task of aggregating information from queries across heterogeneous modalities to retrieve desired targets. State-of-the-art multimodal retrieval models can understand complex queries, yet they are typically limited to two modalities: text and vision. This limitation impedes the development of universal retrieval systems capable of comprehending queries that combine more than two modalities. To advance toward this goal, we present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio. Our OmniRet model addresses two critical challenges for universal retrieval: computational efficiency and representation fidelity. First, feeding massive token sequences from modality-specific encoders to Large Language Models (LLMs) is computationally inefficient. We therefore introduce an attention-based resampling mechanism to generate compact, fixed-size representations from these sequences. This shared module is designed to maintain representational diversity and generalization capabilities while remaining sensitive to modality-specific information. Second, compressing rich omni-modal data into a single embedding vector inevitably causes information loss and discards fine-grained details. We propose Attention Sliced Wasserstein Pooling to preserve these fine-grained details, leading to improved omni-modal representations. OmniRet is trained on an aggregation of approximately 6 million query-target pairs spanning 30 datasets. We benchmark our model on 13 retrieval tasks and a MMEBv2 subset. Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others. Furthermore, we curate a new Audio-Centric Multimodal Benchmark (ACM). This new benchmark introduces two critical, previously missing tasks--composed audio retrieval and audio-visual retrieval--to more comprehensively evaluate a model's omni-modal embedding capacity. We believe our benchmark will facilitate the development of universal retrieval systems.