理解幻觉在多模态推理模型强化学习后训练中的作用
Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models
提出一种通过模态特定扰动（移除或替换关键视觉信息）来诱导幻觉的分析框架，可用于诊断多模态大模型在强化学习后训练中是否真正依赖视觉信息，为设计模态感知的RL训练策略提供可复用工具。
原文依据 we introduce hallucination-inductive, modality-specific corruptions that remove or replace essential information required to derive correct answers
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论强化学习（RL）在大型推理模型中的近期成功，激发了越来越多将其用于多模态大语言模型（MLLMs）后训练以增强视觉推理能力的实践。尽管许多研究报告了性能提升，但RL训练是否真正使模型从视觉信息中学习仍不清楚。在本工作中，我们提出“幻觉作为线索”框架，这是一个分析框架，旨在从模型幻觉的角度研究基于RL的后训练对多模态推理模型的影响。具体而言，我们引入幻觉诱导的、模态特定的损坏，这些损坏移除或替换推导正确答案所需的关键信息，从而迫使模型通过幻觉进行推理。通过在训练和评估期间应用这些损坏，我们的框架为诊断RL训练动态和理解数据集的内在属性提供了独特视角。通过在多个多模态推理基准上进行大量实验和分析，我们揭示模型幻觉对RL训练的作用比先前认识更为显著。例如，我们发现，在纯幻觉诱导设置下进行RL后训练仍能显著提升模型的推理性能，在某些情况下甚至优于标准训练。这些发现挑战了关于MLLM推理训练的普遍假设，并推动开发更具模态感知的基于RL的训练设计。
英文原文摘要 1583 字符
The recent success of reinforcement learning (RL) in large reasoning models has inspired the growing adoption of RL for post-training Multimodal Large Language Models (MLLMs) to enhance their visual reasoning capabilities. Although many studies have reported improved performance, it remains unclear whether RL training truly enables models to learn from visual information. In this work, we propose the Hallucination-as-Cue Framework, an analytical framework designed to investigate the effects of RL-based post-training on multimodal reasoning models from the perspective of model hallucination. Specifically, we introduce hallucination-inductive, modality-specific corruptions that remove or replace essential information required to derive correct answers, thereby forcing the model to reason by hallucination. By applying these corruptions during both training and evaluation, our framework provides a unique perspective for diagnosing RL training dynamics and understanding the intrinsic properties of datasets. Through extensive experiments and analyses across multiple multimodal reasoning benchmarks, we reveal that the role of model hallucination for RL-training is more significant than previously recognized. For instance, we find that RL post-training under purely hallucination-inductive settings can still significantly improve models' reasoning performance, and in some cases even outperform standard training. These findings challenge prevailing assumptions about MLLM reasoning training and motivate the development of more modality-aware RL-based training designs.