CoV-Align：以连贯视觉语义优先的高效细粒度跨模态对齐
CoV-Align: Efficient Fine-grained Cross-Modal Alignment with Cohesive Visual Semantics Priority
用无文本引导的语义收敛注意力先聚合视觉 patch，再与文本对齐，可把图文检索/跨模态检索中的 patch-word 冗余相关性计算量降下来，3-5x 加速对大规模多模态训练与推理有直接复用价值。
原文依据 Through a semantically convergent attention mechanism, it progressively aggregates meaningful visual patches in a text-free manner.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论跨模态对齐旨在跨不同模态学习语义一致的潜变量表征。主流方法依赖文本引导的聚合范式来实现细粒度对齐，但存在冗余的 patch-词 相关性和高昂的计算开销。为解决这些问题，我们提出 CoV-Align，一个以连贯视觉语义优先的高效且有效的细粒度跨模态对齐框架。它通过语义收敛注意力机制，以无文本的方式逐步聚合有意义的视觉 patch。我们设计了一个粗粒度视觉语义特征提取器，融合可变形注意力与一致性分配注意力，将语义一致的 patch 分组。我们提出连贯且具判别性的特征优化，以增强视觉区域特征的语义内聚性与语义间判别性，从而显式提升跨模态对齐效果。大量实验表明，CoV-Align 在 Flickr30K 与 MS-COCO 基准上取得最优性能。值得注意的是，相比先驱方法它带来 3-5 倍的计算加速，为大规模多模态任务提供了显著优势。
英文原文摘要 1249 字符
Cross-modal alignment aims to learn semantically consistent latent representations across diverse modalities. Prevailing methods rely on a text-guided aggregation paradigm to achieve fine-grained alignment, while they suffer from redundant patch-word correlations and high computational costs. To address these issues, we propose CoV-Align, an effective and efficient fine-grained cross-modal alignment framework with cohesive visual semantics priority. Through a semantically convergent attention mechanism, it progressively aggregates meaningful visual patches in a text-free manner. We design a coarse visual semantic feature extractor that integrates deformable attention and consistent assign attention to group patches with semantic consistency. A cohesive and discriminative feature optimization is presented to enhance intra-semantic cohesion and inter-semantic discriminability of visual region features, resulting in explicit improvements in cross-modal alignment. Extensive experiments demonstrate that CoV-Align achieves state-of-the-art performance on the Flickr30K and MS-COCO benchmarks. Notably, it delivers a 3-5xcomputational speedup compared to pioneer approaches, offering compelling advantages for large-scale multi-modal tasks.