强化结构化思维链以提升视频理解
Reinforcing Structured Chain-of-Thought for Video Understanding
提出无需 SFT 的单阶段强化学习框架 SDRL，用「总结-思考-回答」结构化思维链和两个自监督机制（CVK 约束视觉事实一致性、DVR 按组准确率调节推理多样性）来抑制视频推理中的思维漂移；做视频 MLLM 推理增强的工作可直接复用该 GRPO 目标改造思路。
原文依据 we introduce Summary-Driven Reinforcement Learning (SDRL), a novel single-stage RL framework that obviates the need for SFT
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）在视频理解方面展现出潜力。然而，即便借助组相对策略优化（GRPO）等强化学习（RL）技术进行增强，其推理仍常出现思维漂移与时序理解薄弱的问题。此外，现有 RL 方法通常依赖监督微调（SFT），这需要昂贵的思维链（CoT）标注与多阶段训练，并强制固定推理路径，从而限制了 MLLMs 的泛化能力，并可能引入偏差。为克服这些局限，我们提出总结驱动强化学习（SDRL），这是一种新颖的单阶段 RL 框架，通过采用「总结—思考—回答」的结构化 CoT 格式，无需 SFT。SDRL 在 GRPO 目标中引入两种自监督机制：1）视觉知识一致性（CVK）通过降低所生成总结之间的 KL 散度来强化事实视觉定位；2）推理动态多样性（DVR）基于组准确率动态调节思考多样性以促进探索。这种新颖的集成有效平衡了对齐与探索，同时对最终答案与推理过程进行监督。我们的方法在七个公开 VideoQA 数据集上取得了最优性能。
英文原文摘要 1317 字符
Multi-modal Large Language Models (MLLMs) show promise in video understanding. However, their reasoning often suffers from thinking drift and weak temporal comprehension, even when enhanced by Reinforcement Learning (RL) techniques like Group Relative Policy Optimization (GRPO). Moreover, existing RL methods usually depend on Supervised Fine-Tuning (SFT), which requires costly Chain-of-Thought (CoT) annotation and multi-stage training, and enforces fixed reasoning paths, limiting MLLMs' ability to generalize and potentially inducing bias. To overcome these limitations, we introduce Summary-Driven Reinforcement Learning (SDRL), a novel single-stage RL framework that obviates the need for SFT by utilizing a Structured CoT format: Summarize - Think - Answer. SDRL introduces two self-supervised mechanisms integrated into the GRPO objective: 1) Consistency of Vision Knowledge (CVK) enforces factual grounding by reducing KL divergence among generated summaries; and 2) Dynamic Variety of Reasoning (DVR) promotes exploration by dynamically modulating thinking diversity based on group accuracy. This novel integration effectively balances alignment and exploration, supervising both the final answer and the reasoning process. Our method achieves state-of-the-art performance on seven public VideoQA datasets.