无需硬负样本：以概念为中心的学习在不损害对比模型零样本能力的前提下实现组合性
No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models
用无参数的跨模态注意力池化替代文本/图像编码器的全局池化，从图像中抽取概念级视觉嵌入，并与短概念化文本片段对齐；这套改动可直接插到 CLIP 类双塔模型上，在不增加推理成本的前提下提升组合性理解，同时不牺牲零样本与检索性能。
原文依据 we introduce a parameter-free cross-modal attention-pooling to obtain concept centric visual embeddings from the image encoder
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论对比式视觉语言（V&L）模型仍是各类应用中的热门选择。然而，其若干局限也逐渐显现，其中最突出的是 V&L 模型学习组合式表征的能力有限。以往方法常通过生成定制训练数据来获得硬负样本，以缓解这一局限。硬负样本已被证明能提升组合性任务上的表现，但往往只针对单一基准、缺乏泛化性，并且可能显著损害零样本或检索等基础 V&L 能力，因而难以实用。本工作采用不同的思路。我们识别出限制 V&L 组合性表现的两个根本原因：1）长训练描述并不要求组合式表征；2）文本与图像编码器末端的全局池化直接导致学习绑定关系所需的信息完全丢失。作为补救，我们提出两个简单方案：1）使用标准 NLP 软件获得以概念为中心的短描述片段，并将其与图像对齐；2）引入无参数的跨模态注意力池化，从图像编码器获得以概念为中心的视觉嵌入。借助这两项改动与简单的辅助对比损失，我们在标准组合性基准上取得 SOTA 性能，同时保持或提升了强劲的零样本与检索能力，且不增加推理成本。本工作代码发布于 https://github.com/SamsungLabs/concept_centric_clip。
英文原文摘要 1605 字符
Contrastive vision-language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard-negative samples. Hard-negatives have been shown to improve performance on compositionality tasks, but are often specific to a single benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or retrieval performance, rendering them impractical. In this work we follow a different approach. We identify two root causes that limit compositionality performance of V&Ls: 1) Long training captions do not require a compositional representation; and 2) The final global pooling in the text and image encoders leads to a complete loss of the necessary information to learn binding in the first place. As a remedy, we propose two simple solutions: 1) We obtain short concept centric caption parts using standard NLP software and align those with the image; and 2) We introduce a parameter-free cross-modal attention-pooling to obtain concept centric visual embeddings from the image encoder. With these two changes and simple auxiliary contrastive losses, we obtain SOTA performance on standard compositionality benchmarks, while maintaining or improving strong zero-shot and retrieval capabilities. This is achieved without increasing inference cost. We release the code for this work at https://github.com/SamsungLabs/concept_centric_clip.