一个Token，两种命运：通过视觉Token操控对抗多模态大模型幻觉的统一框架
One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination
对做多模态大模型幻觉的研究者，这篇给出了一个免训练、只在潜空间操作的视觉token双向用法：用增强图像的token强化视觉表征，用剪枝后的token构造潜空间负样本校正语言先验偏置，可直接插到LLaVA类模型的推理流程里，代价仅1.06倍延迟。
原文依据 removing vision tokens (information-gap) isolates hallucination tendencies more precisely than distorting images (modality-gap)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当前免训练方法以彼此分离的策略应对多模态大模型幻觉：或增强视觉信号，或抑制文本惯性。然而这些分离方法因关键的权衡而效果不足：单纯增强视觉往往难以对抗强语言先验，而抑制语言又会引入与图像无关的额外噪声。此外，我们发现二者的简单组合同样无效，因而需要一个统一框架。我们通过聚焦核心资产——视觉token——提出了这样一个框架。我们的设计基于两个关键洞见：(1) 增强图像提供互补的视觉语义；(2) 移除视觉token（信息差）比扭曲图像（模态差）能更精确地隔离幻觉倾向。基于此，我们的框架以两种不同方式使用视觉token，且二者均作用于潜变量表征：我们的协同视觉校准（SVC）模块引入增强token以强化视觉表征，而我们的因果表征校准（CRC）模块利用剪枝后的token在潜变量空间构造负样本，以校正模型内部偏置。通过协调这两种角色，我们的框架有效恢复了视觉-语言平衡，显著减少物体幻觉，在多个基准上使LLaVA-1.5的POPE准确率平均绝对提升2%，而推理延迟开销仅为1.06倍。
英文原文摘要 1431 字符
Current training-free methods tackle MLLM hallucination with separate strategies: either enhancing visual signals or suppressing text inertia. However, these separate methods are insufficient due to critical trade-offs: simply enhancing vision often fails against strong language prior, while suppressing language can introduce extra image-irrelevant noise. Moreover, we find their naive combination is also ineffective, necessitating a unified framework. We propose such a framework by focusing on the core asset: the vision token. Our design leverages two key insights: (1) augmented images offer complementary visual semantics, and (2) removing vision tokens (information-gap) isolates hallucination tendencies more precisely than distorting images (modality-gap). Based on these, our framework uses vision tokens in two distinct ways, both operating on latent representations: our Synergistic Visual Calibration (SVC) module incorporates augmented tokens to strengthen visual representations, while our Causal Representation Calibration (CRC) module uses pruned tokens to create latent-space negative samples for correcting internal model biases. By harmonizing these two roles, our framework effectively restores the vision-language balance, significantly reducing object hallucinations, improving POPE accuracy by an average of 2% absolute on LLaVA-1.5 across multiple benchmarks with only a 1.06x inference latency overhead.