对齐视觉语言模型所见与所感：自适应信息流
Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow
提出一种基于词元动态的视觉词元重要性度量，在解码阶段抑制文本词元对无关视觉词元的注意力，可作为即插即用的推理时干预手段，用于提升 VLM 的视觉定位与抗幻觉能力。
原文依据 text tokens distribute too much attention to irrelevant visual tokens, leading to incorrect answers
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在视觉识别、文档解析和视觉定位等广泛任务中展现出强大能力。然而，近期工作表明，尽管 VLM 往往能够捕捉到与问题对应的正确图像区域，却不一定给出正确答案。在本工作中，我们证明这种错位可归因于 VLM 内部次优的信息流：文本词元将过多注意力分配给无关的视觉词元，从而导致错误答案。基于这一观察，我们表明在推理阶段调制信息流可以提升 VLM 的感知能力。其核心思想是：在解码过程中，文本词元应只与重要的视觉词元相关联，从而消除无关区域的干扰。为此，我们提出一种基于词元动态的方法来确定视觉词元的重要性，其中在不同解码阶段表现出显著不同激活模式的视觉词元被视为重要词元。我们将该方法应用于具有代表性的开源 VLM，并在多个数据集上进行评估，涵盖视觉问答、视觉定位与计数、光学字符识别以及物体幻觉。结果表明，我们的方法显著提升了基线模型的性能。项目主页：https://cxliu0.github.io/AIF/。
英文原文摘要 1423 字符
Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding. Nevertheless, recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers. In this work, we demonstrate that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens, leading to incorrect answers. Based on the observation, we show that modulating the information flow during inference can improve the perception capability of VLMs. The idea is that text tokens should only be associated with important visual tokens during decoding, eliminating the interference of irrelevant regions. To achieve this, we propose a token dynamics-based method to determine the importance of visual tokens, where visual tokens that exhibit distinct activation patterns during different decoding stages are viewed as important. We apply our approach to representative open-source VLMs and evaluate on various datasets, including visual question answering, visual grounding and counting, optical character recognition, and object hallucination. The results show that our approach significantly improves the performance of baselines. Project page: https://cxliu0.github.io/AIF/.