Easy2Hard：从部分不匹配到完全不匹配的模态作为对比学习负样本
Easy2Hard: From Partially to Fully Unmatched Modalities as Negative Samples in Contrastive Learning
对做多模态视觉表征（如图文、视频-音频-文本）的研究者，这篇论文给出了一个可直接替换的负样本构造与加权策略：把部分不匹配与完全不匹配的模态对分开，并用 sigmoid 权重课程从易到难过渡，可嵌入现有对比学习损失中提升细粒度对齐效果。
原文依据 Easy2Hard further introduces a sigmoid weighting curriculum that smoothly transitions the learning process from easy (partially unmatched) to hard (fully unmatched) contrasts.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论对比学习通过对齐同一数据样本不同模态的嵌入，被广泛用于生成多模态数据表征。这种对齐通过一个损失函数实现，该损失函数将数据批次内匹配与不匹配的模态对分别视为正样本和负样本。然而，当把对比学习扩展到涉及两个以上模态的场景时，现有方法要么仅依赖完全不匹配的模态作为负样本，要么无法区分部分不匹配与完全不匹配的模态，从而忽略了细粒度的对比关系。为解决这一局限，我们提出 Easy2Hard，一个显式分离部分不匹配与完全不匹配模态的新框架。为了以更细的粒度从负样本中学习，Easy2Hard 进一步引入 sigmoid 加权课程，使学习过程从简单（部分不匹配）对比平滑过渡到困难（完全不匹配）对比。在跨多个领域的五个多模态数据集上的全面评估证明了我们方法的优越性。
英文原文摘要 1120 字符
Contrastive learning is widely used for generating multimodal data representations by aligning embeddings of different modalities of the same data samples. This alignment is achieved through a loss function that treats matched and unmatched modality pairs as positive and negative samples within a data batch. However, when extending contrastive learning to scenarios involving more than two modalities, existing approaches either rely solely on fully unmatched modalities as negative samples, or fail to distinguish between partially and fully unmatched modalities, thereby overlooking the fine-grained contrasting relationships. To address this limitation, we propose Easy2Hard, a novel framework that explicitly separates partially and fully unmatched modalities. To learn from negative samples at improved granularity, Easy2Hard further introduces a sigmoid weighting curriculum that smoothly transitions the learning process from easy (partially unmatched) to hard (fully unmatched) contrasts. Comprehensive evaluations on five multimodal datasets across diverse domains demonstrate the superiority of our approach.