AdaIAT：自适应增强对生成文本的注意力以缓解 LVLM 中的幻觉
AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
提供了一种无需训练的推理时注意力干预手段：在解码时放大对已生成文本 token 的注意力权重，并按层阈值与注意力头粒度自适应调节，可直接插入 LLaVA 等 LVLM 的解码流程以降低幻觉且不引入重复描述。
原文依据 real object tokens tend to assign higher attention to the generated text than hallucinated ones
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论幻觉一直是当前大型视觉语言模型（LVLM）发展与应用的重大障碍。为缓解幻觉，一种直观且有效的方式是在推理时直接提高对图像 token 的注意力权重。尽管这能有效降低幻觉率，却常常引发重复性描述。为此，我们首先对注意力模式进行分析，并揭示出真实物体 token 往往比幻觉 token 对生成文本分配更高的注意力。这启发我们利用生成文本——其中包含与指令相关的视觉信息和上下文知识——在保持语言连贯性的同时缓解幻觉。我们据此提出「增强对生成文本的注意力」（IAT），并证明其能显著降低幻觉率且避免重复性描述。为防止朴素放大损害 LVLM 固有的预测能力，我们进一步探索了自适应 IAT（AdaIAT），它采用逐层阈值来控制干预时机，并依据每个注意力头的特性施加细粒度的放大强度。分析与实验均证明了 AdaIAT 的有效性。在多个 LVLM 上的结果表明，AdaIAT 有效缓解了幻觉（在 LLaVA-1.5 上将幻觉率 C_S 与 C_I 分别降低 35.8% 与 37.1%），同时保持语言表现与预测能力，实现了有吸引力的权衡。
英文原文摘要 1554 字符
Hallucination has been a significant impediment to the development and application of current Large Vision-Language Models (LVLMs). To mitigate hallucinations, one intuitive and effective way is to directly increase attention weights to image tokens during inference. Although this effectively reduces the hallucination rate, it often induces repetitive descriptions. To address this, we first conduct an analysis of attention patterns and reveal that real object tokens tend to assign higher attention to the generated text than hallucinated ones. This inspires us to leverage the generated text, which contains instruction-related visual information and contextual knowledge, to alleviate hallucinations while maintaining linguistic coherence. We therefore propose Increase Attention to Generated Text (IAT) and demonstrate that it significantly reduces the hallucination rate while avoiding repetitive descriptions. To prevent naive amplification from impairing the inherent prediction capabilities of LVLMs, we further explore Adaptive IAT (AdaIAT) that employs a layer-wise threshold to control intervention time and fine-grained amplification magnitude tailored to the characteristics of each attention head.Both analysis and experiments demonstrate the effectiveness of AdaIAT. Results of several LVLMs show that AdaIAT effectively alleviates hallucination (reducing hallucination rates C_S and C_I on LLaVA-1.5 by 35.8% and 37.1%, respectively) while preserving linguistic performance and prediction capability, achieving an attractive trade-off.