基于梯度自反思的多模态幻觉缓解
Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection
用一阶泰勒梯度估计每个视觉 token 与文本 token 对当前输出的贡献，从而在解码时抑制虚假视觉特征并重新平衡跨模态贡献；这是一种无需训练、无需辅助模型的推理期去幻觉手段，可直接套用到任意 MLLM 的解码环节。
原文依据 uses first-order Taylor gradients to understand the contribution of individual tokens--visual features and text tokens--to the current output
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLM）在多种任务上表现强劲，但仍易产生幻觉，即输出并未建立在视觉输入之上。该问题可归因于两种主要偏差：文本—视觉偏差，即过度依赖提示词与先前输出；以及共现偏差，即频繁成对出现的物体之间的虚假关联。我们提出基于梯度的影响感知约束解码（GACD），这是一种基于推理的方法，无需辅助模型即可同时应对上述两种偏差，并且无需微调即可直接应用于现有模型。该方法的核心是偏差估计，即利用一阶泰勒梯度理解各个 token——视觉特征与文本 token——对当前输出的贡献。基于这一分析，GACD 通过两个组件缓解幻觉：（1）抑制与输出物体相关的虚假视觉特征；（2）通过相对文本增强视觉特征来重新平衡跨模态贡献。在多个基准上的实验表明，GACD 有效减少了幻觉，并提升了 MLLM 输出的视觉定位能力。
英文原文摘要 1178 字符
Multimodal large language models (MLLMs) achieve strong performance across diverse tasks but remain prone to hallucinations, where outputs are not grounded in visual inputs. This issue can be attributed to two main biases: text-visual bias, the overreliance on prompts and prior outputs, and co-occurrence bias, spurious correlations between frequently paired objects. We propose Gradient-based Influence-Aware Constrained Decoding (GACD), an inference-based method, that addresses both biases without auxiliary models, and is readily applicable to existing models without finetuning. The core of our approach is bias estimation, which uses first-order Taylor gradients to understand the contribution of individual tokens--visual features and text tokens--to the current output. Based on this analysis, GACD mitigates hallucinations through two components: (1) suppressing spurious visual features correlated with the output objects, and (2) rebalancing cross-modal contributions by strengthening visual features relative to text. Experiments across multiple benchmarks demonstrate that GACD effectively reduces hallucinations and improves the visual grounding of MLLMs outputs.