面向大型视觉语言模型幻觉缓解的预填充阶段干预
Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
在 prefill 阶段一次性修改 KV cache 来抑制 LVLM 幻觉，且对 key/value 分别施加视觉与文本方向，可与任意解码阶段方法正交叠加，为视觉语言模型的推理期干预提供了新的可复用切入点。
原文依据 we propose Prefill-Time Intervention (PTI), a novel steering paradigm that intervenes only once during the prefill stage, enhancing the initial Key-Value (KV) cache
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（LVLM）在视觉-文本理解方面取得了显著进展，但其可靠性因幻觉而受到严重削弱，即生成与事实不符或不一致的回复。尽管近期使用引导向量的研究在减少幻觉方面展现出潜力，但仍存在一个显著挑战：这些方法会无意中加剧残余幻觉的严重程度。我们将此归因于它们仅关注解码阶段，而在该阶段误差会自回归地累积，并逐步恶化后续的幻觉输出。为解决这一问题，我们提出预填充阶段干预（PTI），一种新颖的引导范式，仅在预填充阶段干预一次，在误差累积发生之前增强初始的键值（KV）缓存。具体而言，PTI 具备模态感知能力，为视觉与文本表征推导出不同的方向。该干预是解耦的，将键引导至有视觉依据的对象，并让值过滤背景噪声，从而在源头纠正易产生幻觉的表征。大量实验表明，PTI 在缓解幻觉方面具有显著性能，并且在多种解码策略、LVLM 与基准上具有泛化性。此外，PTI 与现有的解码阶段方法正交，可实现即插即用集成并进一步提升性能。
英文原文摘要 1405 字符
Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual-textual understanding, yet their reliability is critically undermined by hallucinations, i.e., the generation of factually incorrect or inconsistent responses.While recent studies using steering vectors demonstrated promise in reducing hallucinations, a notable challenge remains: they inadvertently amplify the severity of residual hallucinations. We attribute this to their exclusive focus on the decoding stage, where errors accumulate autoregressively and progressively worsen subsequent hallucinatory outputs.To address this, we propose Prefill-Time Intervention (PTI), a novel steering paradigm that intervenes only once during the prefill stage, enhancing the initial Key-Value (KV) cache before error accumulation occurs.Specifically, PTI is modality-aware, deriving distinct directions for visual and textual representations. This intervention is decoupled to steer keys toward visually-grounded objects and values to filter background noise, correcting hallucination-prone representations at their source.Extensive experiments demonstrate PTI's significant performance in mitigating hallucinations and its generalizability across diverse decoding strategies, LVLMs, and benchmarks. Moreover, PTI is orthogonal to existing decoding-stage methods, enabling plug-and-play integration and further boosting performance.