跨层重分配注意力以减少多模态幻觉
Reallocating Attention Across Layers to Reduce Multimodal Hallucination
提出一种免训练、轻量级的注意力重分配插件，通过识别感知与推理功能头并类条件缩放其层间贡献，可直接用于缓解多模态大模型在视觉问答等任务中的幻觉，提升视觉忠实度。
原文依据 we propose Functional Head Identification and Class-Conditioned Rescaling, a lightweight, training-free plugin that identifies perception- and reasoning-oriented heads and adaptively rebalances their layerwise contributions.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大型推理模型（MLRMs）常出现幻觉，其根源不仅在于视觉定位不足，还在于感知与推理过程之间的分配失衡。基于近期关于注意力跨层分阶段分工的可解释性发现，我们分析了这种功能错位如何导致两种互补的失效模式：浅层中的感知偏差和深层中的推理漂移。为缓解这些问题，我们提出了功能头识别与类条件缩放，这是一种轻量级、免训练的插件，能够识别面向感知和推理的注意力头，并自适应地重新平衡它们的逐层贡献。我们的方法无需重新训练或任何架构修改，即可提升推理一致性和视觉忠实度。在三个代表性MLRM和五个多模态推理基准上的评估显示，平均提升4.2个百分点，额外计算量不到1%，仅增加9%的基线延迟。除实证改进外，我们的研究为调控跨层功能动态以增强多模态推理的可靠性提供了可解释的视角。
英文原文摘要 1220 字符
Multimodal large reasoning models (MLRMs) often suffer from hallucinations that stem not only from insufficient visual grounding but also from imbalanced allocation between perception and reasoning processes. Building upon recent interpretability findings suggesting a staged division of attention across layers, we analyze how this functional misalignment leads to two complementary failure modes: perceptual bias in shallow layers and reasoning drift in deeper layers. To alleviate these issues, we propose Functional Head Identification and Class-Conditioned Rescaling, a lightweight, training-free plugin that identifies perception- and reasoning-oriented heads and adaptively rebalances their layerwise contributions. Our method improves reasoning consistency and visual faithfulness without retraining or any architectural modification. Evaluations across three representative MLRMs and five multimodal reasoning benchmarks show an average 4.2 percentage-point gain, with less than 1% additional computation and only 9% baseline latency. Beyond empirical improvements, our study provides an interpretable perspective on regulating cross-layer functional dynamics to enhance the reliability of multimodal reasoning.