多模态大模型关注何处、依赖什么：解释自回归词元生成
Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation
提供一种黑盒、低显存的归因框架，可把 MLLM 生成的任意词元追溯到紧凑的图像区域，并量化语言先验与视觉证据的相对贡献；做 VLM 幻觉诊断或视觉 grounding 分析时，可直接复用其充分性/必要性目标与稀疏区域贪心搜索来定位模型真正依赖的图像证据。
原文依据 EAGLE attributes any selected tokens to compact perceptual regions while quantifying the relative influence of language priors and perceptual evidence.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）在将视觉输入与自然语言输出对齐方面展现出显著能力。然而，生成的词元在多大程度上依赖视觉模态仍鲜有理解，这限制了模型的可解释性与可靠性。本文提出 EAGLE，一个轻量级黑盒框架，用于解释 MLLMs 中的自回归词元生成。EAGLE 将任意选定的词元归因到紧凑的感知区域，同时量化语言先验与感知证据的相对影响。该框架引入一个目标函数，统一了充分性（洞察分数）与不可或缺性（必要性分数），并通过在稀疏化图像区域上的贪心搜索进行优化，从而实现忠实且高效的归因。除空间归因外，EAGLE 还执行模态感知分析，解耦词元所依赖的对象，为模型决策提供细粒度的可解释性。在多个开源 MLLMs 上的大量实验表明，EAGLE 在忠实性、定位与幻觉诊断方面持续优于现有方法，同时所需 GPU 显存大幅减少。这些结果凸显了其在推进 MLLMs 可解释性方面的有效性与实用性。
英文原文摘要 1262 字符
Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in aligning visual inputs with natural language outputs. Yet, the extent to which generated tokens depend on visual modalities remains poorly understood, limiting interpretability and reliability. In this work, we present EAGLE, a lightweight black-box framework for explaining autoregressive token generation in MLLMs. EAGLE attributes any selected tokens to compact perceptual regions while quantifying the relative influence of language priors and perceptual evidence. The framework introduces an objective function that unifies sufficiency (insight score) and indispensability (necessity score), optimized via greedy search over sparsified image regions for faithful and efficient attribution. Beyond spatial attribution, EAGLE performs modality-aware analysis that disentangles what tokens rely on, providing fine-grained interpretability of model decisions. Extensive experiments across open-source MLLMs show that EAGLE consistently outperforms existing methods in faithfulness, localization, and hallucination diagnosis, while requiring substantially less GPU memory. These results highlight its effectiveness and practicality for advancing the interpretability of MLLMs.