KVSmooth：通过键值平滑缓解多模态大语言模型中的幻觉
KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothing
提出一种免训练、即插即用的推理时方法，通过对 KV-Cache 中的键和值施加指数移动平均（EMA）平滑，并利用注意力熵动态调节平滑强度，来缓解多模态大语言模型在解码过程中的语义漂移和幻觉。该方法可直接集成到现有 MLLM 的推理流程中，无需重新训练或修改模型结构，为视觉-语言生成任务提供了一种高效的后处理手段。
原文依据 KVSmooth applies an exponential moving average (EMA) to both keys and values in the KV-Cache while dynamically quantifying the sink degree of each token through its attention distribution entropy to adaptively adjust the smoothing strength.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管多模态大语言模型（MLLMs）在多种任务上取得了显著进展，但幻觉——即生成与视觉输入不一致的物体、属性或关系——仍然是其可靠部署的主要障碍。与纯语言模型不同，MLLMs 必须将其生成过程建立在视觉输入之上；然而，现有模型在解码过程中常常遭受语义漂移，导致输出随着序列长度的增加而偏离视觉事实。为解决这一问题，我们提出了 KVSmooth，一种免训练、即插即用的方法，通过对隐藏状态进行注意力熵引导的自适应平滑来缓解幻觉。具体而言，KVSmooth 对 KV-Cache 中的键和值均应用指数移动平均（EMA），同时通过每个词元的注意力分布熵动态量化其汇聚程度，以自适应地调整平滑强度。与计算昂贵的重新训练或对比解码方法不同，KVSmooth 在推理过程中高效运行，无需额外训练或模型修改。大量实验表明，KVSmooth 显著减少了幻觉（CHAIR_S 从 41.8 降至 18.2），同时提升了整体性能（F_1 分数从 77.5 升至 79.2），实现了精确率和召回率的同步提高，而先前方法往往以牺牲其中一个为代价，从而验证了我们方法的有效性和通用性。
英文原文摘要 1486 字符
Despite the significant progress of Multi-modal Large Language Models (MLLMs) across diverse tasks, hallucination, which corresponds to the generation of visually inconsistent objects, attributes, or relations, remains a major obstacle to their reliable deployment. Unlike pure language models, MLLMs must ground their generation process in visual inputs; However, existing models often suffer from semantic drift during decoding, causing outputs to diverge from visual facts as the sequence length increases.To address this, we propose KVSmooth, a training-free, plug-and-play method that mitigates hallucination by performing attention-entropy-guided adaptive smoothing on hidden states. Specifically, KVSmooth applies an exponential moving average (EMA) to both keys and values in the KV-Cache while dynamically quantifying the sink degree of each token through its attention distribution entropy to adaptively adjust the smoothing strength. Unlike computationally expensive retraining or contrastive decoding methods, KVSmooth operates efficiently during inference without additional training or model modification. Extensive experiments demonstrate that KVSmooth significantly reduces hallucination (\mathit CHAIR _ S from 41.8 -> 18.2) while improving overall performance (F_1 score from 77.5 -> 79.2), achieving higher precision and recall simultaneously, whereas prior methods often sacrifice one for the other, thereby validating the effectiveness and generality of our method.