残差解码:通过历史感知残差引导缓解大型视觉语言模型的幻觉
Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance
ResDec 是一种免训练的解码策略,利用 LVLM 内部词元 logits 的演化历史来校正语言先验带来的偏置;做视觉问答或多模态生成时可直接套用到现有模型的解码阶段,无需重训即可提升视觉 grounding、减少物体幻觉。
原文依据 It is a novel training-free method that uses historical information to aid decoding.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型(LVLM)能够从图像-文本输入进行推理,并在各类多模态任务中表现良好。尽管取得了这些成功,它们仍受语言先验影响,常常产生幻觉。幻觉指生成的内容在语法和句法上连贯,却与视觉输入不匹配或没有直接关联。为解决这一问题,我们提出残差解码(ResDec)。这是一种新颖的免训练方法,利用历史信息辅助解码。该方法依赖 LVLM 内部的隐式推理机制与词元 logits 演化机制来校正偏置。大量实验表明,ResDec 能有效抑制由语言先验引发的幻觉,显著提升视觉 grounding,并减少物体幻觉。除缓解幻觉外,ResDec 在综合性 LVLM 基准上也表现优异,凸显了其广泛的适用性。
英文原文摘要 974 字符
Large Vision-Language Models (LVLMs) can reason from image-text inputs and perform well in various multimodal tasks. Despite this success, they are affected by language priors and often produce hallucinations. Hallucinations denote generated content that is grammatically and syntactically coherent, yet bears no match or direct relevance to visual input. To address this problem, we propose Residual Decoding (ResDec). It is a novel training-free method that uses historical information to aid decoding. The method relies on the internal implicit reasoning mechanism and token logits evolution mechanism of LVLMs to correct biases. Extensive experiments demonstrate that ResDec effectively suppresses hallucinations induced by language priors, significantly improves visual grounding, and reduces object hallucinations. In addition to mitigating hallucinations, ResDec also performs exceptionally well on comprehensive LVLM benchmarks, highlighting its broad applicability.