CLIP 目光短浅：把注意力放到第一句之外
CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
揭示了长描述图文对中「首句摘要」造成的注意力捷径，并给出无需额外参数的训练期去偏方案（去掉摘要句＋句子子采样＋词元填充），可直接用于改进 CLIP 类图像编码器在密集描述与长文本检索上的对齐质量。
原文依据 this acts as a shortcut during training, concentrating attention on the opening sentence and early tokens
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论CLIP 模型通过在大规模互联网数据上进行图文对比学习，学到可迁移的多模态特征。它们被广泛用于零样本分类、多模态检索、文本到图像扩散，以及作为大型视觉语言模型中的图像编码器。然而，CLIP 的预训练以图像与短标题配对为主，这使模型偏向于编码对显著物体的简单描述，从而在复杂场景和密集描述上产生粗粒度对齐。尽管近期工作通过在小规模长标题数据集上微调来缓解这一问题，我们指出了一个重要的共性偏置：无论由人类还是大语言模型生成的长标题，通常都以一句摘要开头，随后才是详细描述。我们表明，这在训练中充当了一种捷径，使注意力集中在开篇句和靠前的词元上，并削弱了对标题其余部分的对齐。为解决这一问题，我们提出 DeBias-CLIP，它在训练时移除摘要句，并采用句子子采样与文本词元填充，将监督分布到所有词元位置。DeBias-CLIP 取得了最优的长文本检索性能，提升了短文本检索效果，并且对句子顺序置换更不敏感。它可作为 Long-CLIP 的直接替代方案，且不引入额外的可训练参数。
英文原文摘要 1343 字符
CLIP models learn transferable multi-modal features via image-text contrastive learning on internet-scale data. They are widely used in zero-shot classification, multi-modal retrieval, text-to-image diffusion, and as image encoders in large vision-language models. However, CLIP's pretraining is dominated by images paired with short captions, biasing the model toward encoding simple descriptions of salient objects and leading to coarse alignment on complex scenes and dense descriptions. While recent work mitigates this by fine-tuning on small-scale long-caption datasets, we identify an important common bias: both human- and LLM-generated long captions typically begin with a one-sentence summary followed by a detailed description. We show that this acts as a shortcut during training, concentrating attention on the opening sentence and early tokens and weakening alignment over the rest of the caption. To resolve this, we introduce DeBias-CLIP, which removes the summary sentence during training and applies sentence sub-sampling and text token padding to distribute supervision across all token positions. DeBias-CLIP achieves state-of-the-art long-text retrieval, improves short-text retrieval, and is less sensitive to sentence order permutations. It is a drop-in replacement for Long-CLIP with no additional trainable parameters.