让VLM给自己的思维打分：一种面向推理感知奖励建模的自量化方法
Let VLMs Grade Their Own Thoughts: A Self-Quantification Approach to Reasoning-Aware Reward Modeling
提出利用VLM自生成置信度作为奖励信号，替代外部约束进行强化学习对齐，为视觉语言模型的推理优化提供了无需人工标注的奖励建模思路，可迁移至其他多模态推理任务。
原文依据 we propose leveraging the model's intrinsic self-evaluation to guide its optimization
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论思维链（CoT）是增强视觉语言模型（VLM）推理能力的关键技术。现有方法通常采用带有外部约束的强化学习（RL）来使模型的推理过程与人类认知模式对齐。然而，我们认为模型的内在推理路径可能与人类认知不同，强制这种对齐会限制模型的潜力，甚至降低其性能。为此，我们提出利用模型内在的自我评估来指导其优化。我们假设模型自生成的置信度分数是其推理质量的有效指标。基于这一评估指标，我们设计了两种新颖的奖励函数：（1）序列置信度严格评估（SCRE），用于需要严格逻辑推理的挑战性问题；（2）组内分数重排序（IGSR），用于通用、开放式的场景。我们将我们的方法命名为Video-RAISE（通过内在自我评估实现推理对齐）。在六个视频理解基准上的综合实验表明，Video-RAISE达到了最优（SOTA）性能，显著优于先前的方法甚至专有模型，例如GPT-4o和Gemini 1.5 pro。例如，在VideoMMMU基准上，我们的Video-RAISE达到了52.8%的新SOTA准确率，比之前的最佳模型显著高出3.0%。此外，我们的方法实现了90%的推理路径一致性，是Qwen2.5-VL-Instruct的两倍，甚至超过了监督微调的性能。
英文原文摘要 1629 字符
Chain-of-Thought (CoT) is a key technique for enhancing the reasoning capabilities of Vision Language Models (VLMs). Existing methods often employ Reinforcement Learning (RL) with external constraints to align the model's reasoning process with human cognitive patterns. However, we argue that the model's intrinsic reasoning paths may differ from human cognition, and that forcing such alignment can constrain the model's potential and even degrade its performance. To address this, we propose leveraging the model's intrinsic self-evaluation to guide its optimization. We hypothesize that a model's self-generated confidence scores are effective indicators of its reasoning quality. Based on this evaluation metric, we design two novel reward functions: (1) Sequential Confidence Rigorous Evaluation (SCRE) for challenging problems that demand strict logical reasoning, and (2) Intra-group Score Re-ranking (IGSR) for general-purpose, open-ended scenarios. We name our method Video-RAISE (Reasoning Alignment through Intrinsic Self-Evaluation). Comprehensive experiments on six video understanding benchmarks demonstrate that Video-RAISE achieves state-of-the-art (SOTA) performance, significantly outperforming previous methods and even proprietary models, e,g. GPT-4o and Gemini 1.5 pro. For instance, on the VideoMMMU benchmark, our Video-RAISE achieves a new SOTA accuracy of 52.8%, outperforming the previous best model by a significant 3.0%. In addition, our method achieves a reasoning path consistency of 90%, which is double that of the Qwen2.5-VL-Instruct and even surpasses the performance of supervised fine-tuning.