以反事实对抗幻觉：扩散引导扰动用于抑制 LVLM 幻觉
Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression
用扩散模型编辑图像构造反事实样本，提取视觉诱发幻觉的低秩表征子空间，并在推理时把中间隐状态投影出该子空间——这套「反事实扰动+子空间投影」的免训练流程可直接迁移到其他多模态模型的忠实性修正与表征编辑研究。
原文依据 Contrasting these representations with those from authentic (image, caption) pairs reveals structured, systematic shifts spanning a low-rank subspace
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管大型视觉语言模型（LVLM）在多模态任务上表现优异，它们仍频繁产生幻觉——即与视觉输入不一致的不忠实输出。为解决该问题，我们提出 CIPHER（Counterfactual Image Perturbations for Hallucination Extraction and Removal），一种免训练方法，通过轻量的特征级校正抑制视觉诱发的幻觉。与以往主要关注文本诱发幻觉的免训练方法不同，CIPHER 明确针对源自视觉模态的幻觉。CIPHER 分两个阶段运行。在离线阶段，我们构建了 OHC-25K（Object-Hallucinated Counterfactuals，25,000 个样本），这是一个由扩散编辑图像构成的反事实数据集，这些图像有意与原始真实描述相矛盾。我们将这些编辑图像与未改动的真实描述配对，并输入 LVLM 以提取与幻觉相关的表征。将这些表征与来自真实（图像，描述）对的表征进行对比，可揭示出跨越一个低秩子空间的结构化、系统性偏移，该子空间刻画了视觉诱发的幻觉。在推理阶段，CIPHER 通过将中间隐状态投影远离该子空间来抑制幻觉。在多个基准上的实验表明，CIPHER 显著降低了幻觉率，同时保持任务性能，证明了反事实视觉扰动在提升 LVLM 忠实性方面的有效性。
英文原文摘要 1548 字符
While large vision-language models (LVLMs) achieve strong performance on multimodal tasks, they frequently generate hallucinations--unfaithful outputs misaligned with the visual input. To address this issue, we introduce CIPHER (Counterfactual Image Perturbations for Hallucination Extraction and Removal), a training-free method that suppresses vision-induced hallucinations via lightweight feature-level correction. Unlike prior training-free approaches that primarily focus on text-induced hallucinations, CIPHER explicitly targets hallucinations arising from the visual modality. CIPHER operates in two phases. In the offline phase, we construct OHC-25K (Object-Hallucinated Counterfactuals, 25,000 samples), a counterfactual dataset consisting of diffusion-edited images that intentionally contradict the original ground-truth captions. We pair these edited images with the unchanged ground-truth captions and process them through an LVLM to extract hallucination-related representations. Contrasting these representations with those from authentic (image, caption) pairs reveals structured, systematic shifts spanning a low-rank subspace characterizing vision-induced hallucination. In the inference phase, CIPHER suppresses hallucinations by projecting intermediate hidden states away from this subspace. Experiments across multiple benchmarks show that CIPHER significantly reduces hallucination rates while preserving task performance, demonstrating the effectiveness of counterfactual visual perturbations for improving LVLM faithfulness.