视觉在哪里与语言相遇？通过对比注意力理解并改进 MLLM 中的视觉融合
Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention
提供了一种免训练的对比注意力机制，通过建模早期融合层与最终层之间的注意力变换来突出有意义的注意力转移，可直接插入任意 MLLM 的推理过程以提升多模态推理，无需微调；其逐层掩码分析范式也可用于诊断其他视觉-语言模型的融合位置。
原文依据 we introduce a training-free contrastive attention framework that models the transformation between early fusion and final layers
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLM）在视觉-语言理解方面已取得显著进展，但它们内部如何整合视觉与文本信息仍鲜为人知。为弥补这一空白，我们对多种架构开展了系统的逐层掩码分析，揭示了视觉-文本融合在 MLLM 内部如何演化。结果表明，融合出现在若干特定层，而非均匀分布于整个网络；部分模型还表现出一种晚期“复核”现象，即在输出生成之前视觉信号被重新激活。此外，我们进一步分析了逐层注意力演化，观察到无关区域上持续存在高注意力噪声，同时文本对齐区域的注意力逐渐增强。基于这些发现，我们提出了一种免训练的对比注意力框架，对早期融合层与最终层之间的变换进行建模，以突出有意义的注意力转移。在多种 MLLM 与基准上的大量实验验证了我们的分析，并表明所提方法提升了多模态推理性能。代码将公开。
英文原文摘要 1182 字符
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in vision-language understanding, yet how they internally integrate visual and textual information remains poorly understood. To bridge this gap, we perform a systematic layer-wise masking analysis across multiple architectures, revealing how visual-text fusion evolves within MLLMs. The results show that fusion emerges at several specific layers rather than being uniformly distributed across the network, and certain models exhibit a late-stage "review" phenomenon where visual signals are reactivated before output generation. Besides, we further analyze layer-wise attention evolution and observe persistent high-attention noise on irrelevant regions, along with gradually increasing attention on text-aligned areas. Guided by these insights, we introduce a training-free contrastive attention framework that models the transformation between early fusion and final layers to highlight meaningful attention shifts. Extensive experiments across various MLLMs and benchmarks validate our analysis and demonstrate that the proposed approach improves multimodal reasoning performance.Code will be released.