FALCON：视觉语言对齐中假负样本感知的对比负样本学习
FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment
针对视觉语言预训练中图文多对多对应导致的假负样本问题，提出用可学习的负样本挖掘调度器在 mini-batch 构建阶段为每个锚框实例动态选择合适难度的负样本；做图文检索、CLIP 类对齐训练的工作可以直接把这一负样本调度策略插到自己的对比损失里，替代固定的难负样本启发式。
原文依据 a learning-based mini-batch construction strategy that adaptively balances the trade-off between hard and false negatives during VLP
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论由于大规模数据集中图像与文本之间存在多对多对应关系，假负样本给视觉语言预训练（VLP）带来了严峻挑战。这些假负样本引入相互冲突的监督信号，损害了所学嵌入空间的质量，并削弱了难负样本采样的效果。本文提出 FALCON（假负样本感知的对比负样本学习），一种基于学习的 mini-batch 构建策略，在 VLP 过程中自适应地平衡难负样本与假负样本之间的权衡。FALCON 不依赖固定的启发式规则，而是采用负样本挖掘调度器，在 mini-batch 构建过程中以跨模态对齐提升的代理指标为指导，为每个锚框实例动态选择难度合适的负样本。实验结果表明，FALCON 在三种视觉语言学习框架（ALBEF、BLIP-2、SigLIP-2）以及广泛的下游任务和评测设置上均显著提升了性能，凸显了其在缓解假负样本影响方面的有效性与鲁棒性。
英文原文摘要 1128 字符
False negatives pose a critical challenge in vision-language pretraining (VLP) due to the many-to-many correspondence between images and texts in large-scale datasets. These false negatives introduce conflicting supervision signals that degrade the learned embedding space and diminish the effectiveness of hard negative sampling. In this paper, we propose FALCON (False-negative Aware Learning of COntrastive Negatives), a learning-based mini-batch construction strategy that adaptively balances the trade-off between hard and false negatives during VLP. Rather than relying on fixed heuristics, FALCON employs a negative mining scheduler that dynamically selects negative samples of appropriate hardness for each anchor instance during mini-batch construction, guided by a proxy for cross-modal alignment improvement. Experimental results demonstrate that FALCON significantly improves performance across three vision-language learning frameworks (ALBEF, BLIP-2, SigLIP-2) and a broad range of downstream tasks and evaluation settings, underscoring its effectiveness and robustness in mitigating the impact of false negatives.