COPO：面向多模态大语言模型幻觉的因果导向策略优化
COPO: Causal-Oriented Policy Optimization for Hallucinations of MLLMs
COPO 通过因果完备性奖励和 token 级充分性/必要性约束，在 GRPO 框架内构造因果感知的优势函数，为视觉语言模型提供了一种可复用的幻觉缓解训练策略，可直接用于改进多模态对齐与生成忠实度。
原文依据 we propose Causal-Oriented Policy Optimization (COPO) to mitigate these spurious correlations, thus addressing the issue of hallucinations.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管多模态大语言模型（MLLMs）已展现出令人印象深刻的能力，但它们仍可能产生幻觉。根据经验，我们发现与纯文本大语言模型相比，MLLMs 会不成比例地关注与任务无关的背景区域，这暗示了虚假的背景-答案关联。我们主张并分析：(i) 基于结果的奖励可能是导致虚假关联的重要因素；(ii) 虚假关联可能是导致幻觉的重要因素。基于这些结果，我们提出因果导向策略优化（COPO）来缓解这些虚假关联，从而解决幻觉问题。它施加 token 级的充分性和必要性约束，以衡量每个推理 token 的因果贡献，从而确保输出正确且有证据支撑。具体而言，我们首先通过新提出的因果完备性奖励来评估每个 token 的因果贡献。然后，该奖励被用于在 GRPO 优化框架内构建因果感知的优势函数，鼓励模型关注对准确生成具有因果充分性和必要性的 token。跨多个基准的实验结果证明了 COPO 的优势。
英文原文摘要 1252 字符
Despite Multimodal Large Language Models (MLLMs) having shown impressive capabilities, they may suffer from hallucinations. Empirically, we find that MLLMs attend disproportionately to task-irrelevant background regions compared with text-only LLMs, implying spurious background-answer correlations. We claim and analyze that (i) outcome-based rewards can be an important factor leading to spurious correlations, and (ii) spurious correlations can be an important factor leading to hallucinations. Based on these results, we propose Causal-Oriented Policy Optimization (COPO) to mitigate these spurious correlations, thus addressing the issue of hallucinations. It imposes token-level sufficiency and necessity constraints to measure each inference token's causal contribution, thus ensuring correct and evidence-grounded output. Specifically, we first evaluate each token's causal contribution via a newly proposed causal completeness reward. This reward is then used to construct a causally informed advantage function within the GRPO optimization framework, encouraging the model to focus on tokens that are causally sufficient and necessary for accurate generation. Experimental results across various benchmarks demonstrate the advantages of COPO.