多多益善：面向高阶多模态对齐的对比融合
THE MORE, THE MERRIER: CONTRASTIVE FUSION FOR HIGHER-ORDER MULTIMODAL ALIGNMENT
ConFu 把「模态对」与「模态对+第三模态」的融合嵌入一起拉进同一对比空间，可直接用于视觉-语言-音频等三模态检索与分类；做 VLM 时若需建模图像、文本、音频之间的 XOR 式互补关系，可把该融合对比项接到现有 CLIP 式双塔上，而不必放弃成对对齐。
原文依据 ConFu extends traditional pairwise contrastive objectives with an additional fused-modality contrastive term
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论学习跨多个模态的联合表示仍然多模态机器学习的核心挑战。主流方法大多在成对设定下运行，每次对齐两个模态。尽管近期一些方法试图捕捉多个模态之间的高阶交互，它们往往忽略或未能充分保留成对关系，从而限制了其在单模态任务上的效果。在本工作中，我们提出对比融合（ConFu），一个将各单个模态及其融合组合共同嵌入统一表示空间的框架，在该空间中模态与其融合对应物相互对齐。ConFu 在传统成对对比目标的基础上增加了一个融合模态对比项，促使模态对与第三个模态的联合嵌入相互靠近。该形式使 ConFu 能够捕捉仅靠成对对齐无法恢复的高阶依赖关系（如 XOR 式关系），同时仍保持较强的成对对应性。我们在合成与真实世界多模态基准上评估 ConFu，考察其利用跨模态互补性、捕捉高阶依赖关系以及随多模态复杂度提升而扩展的能力。在这些设定下，ConFu 在检索与分类任务上展现出有竞争力的性能，并在单一对比框架内同时支持一对一与二对一检索。代码与数据集发布于 https://github.com/estafons/confu。
英文原文摘要 1566 字符
Learning joint representations across multiple modalities remains a central challenge in multimodal machine learning. Prevailing approaches predominantly operate in pairwise settings, aligning two modalities at a time. While some recent methods aim to capture higher-order interactions among multiple modalities, they often overlook or insufficiently preserve pairwise relationships, limiting their effectiveness on single-modality tasks. In this work, we introduce Contrastive Fusion (ConFu), a framework that jointly embeds both individual modalities and their fused combinations into a unified representation space, where modalities and their fused counterparts are aligned. ConFu extends traditional pairwise contrastive objectives with an additional fused-modality contrastive term, encouraging the joint embedding of modality pairs with a third modality. This formulation enables ConFu to capture higher-order dependencies, such as XOR-like relationships, that cannot be recovered through pairwise alignment alone, while still maintaining strong pairwise correspondence. We evaluate ConFu on synthetic and real-world multimodal benchmarks, assessing its ability to exploit cross-modal complementarity, capture higher-order dependencies, and scale with increasing multimodal complexity. Across these settings, ConFu demonstrates competitive performance on retrieval and classification tasks, while supporting unified one-to-one and two-to-one retrieval within a single contrastive framework. We release our code and dataset at https://github.com/estafons/confu.