同样的注意力，不同的真相：用 Logit Lens 审视视觉注意力以检测和缓解 LVLM 物体幻觉
Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
提出用 Logit Lens 解码高注意力区域的视觉特征，以检测 LVLM 物体幻觉，并针对视觉不确定性和上下文先验两种机制分别设计免训练缓解策略（HARM 和 VEED），可直接用于提升多模态模型生成可靠性。
原文依据 we decode the visual features of high-attention regions using Logit Lens
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（LVLM）常出现物体幻觉，即生成图像中不存在的物体。先前工作大多将此归因于视觉注意力不足。然而，本研究发现，在模型的中后层，真实物体和幻觉物体获得的视觉注意力强度相当。这表明关键问题可能不在于模型关注了多少，而在于它关注了什么以及为什么关注。为此，我们使用 Logit Lens 解码高注意力区域的视觉特征，观察到对应真实物体的高注意力区域能被正确解码为目标物体词元，而对应幻觉物体的区域则不能。基于此，我们识别出两种不同的幻觉机制：（i）视觉不确定性，由语义相似或易混淆的区域触发，掩蔽这些区域可消除幻觉；（ii）上下文先验，由强共现先验触发，即使掩蔽最初关注的区域，幻觉仍持续存在且注意力漂移至其他区域。基于这些发现，我们提出一个简单而有效的免训练检测-缓解框架，包含用于检测幻觉的 Logit-Lens 一致性检查以及针对性补救措施：针对视觉不确定性幻觉的高注意力区域掩蔽（HARM），以及针对上下文先验幻觉的视觉证据增强解码（VEED），后者利用真实视觉证据抑制错误先验。我们的方法在多个幻觉基准上取得了最优结果。
英文原文摘要 1621 字符
Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, in this work, we are surprised to find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers. This indicates that the key issue may not be how much the model attends, but **what it attends to and why**. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that high-attention regions corresponding to real objects can be correctly decoded to the target object token, whereas those for hallucinated objects cannot. Building on this, we identify two distinct hallucination mechanisms: **(i) visual uncertainty**, triggered by semantically similar or confusable regions, masking these regions eliminates the hallucination. **(ii) contextual prior**, triggered by strong co-occurrence priors, even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free **Detect-Mitigate framework** comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination, which leverages genuine visual evidence to suppress erroneous priors. Our approach achieves state-of-the-art results on multiple hallucination benchmarks.