通过数据高效强化学习激发多模态大模型的多功能视频推理能力
Incentivizing Versatile Video Reasoning in MLLMs via Data-Efficient Reinforcement Learning
对做视频多模态大模型的人，这篇给出了用多任务冷启动加多任务强化学习、再配合推理阶段多模态推理与动态采样的低成本训练配方，仅用 3k 冷启动数据与 5k RL 数据即可显著提升视频推理表现，可作为替代高成本长链语言 RL 的可复用方案。
原文依据 we leverage multimodal reasoning and dynamic sampling to further improve the performance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）在视频理解任务上已取得很大进展。然而，当需要理解复杂或长视频时，MLLMs 往往忽略细节或产生幻觉。为缓解这些问题，近期工作尝试利用强化学习（RL）来提升模型对复杂视频的深度语言推理能力。但这些方法存在两个主要问题：第一，其所用的 RL 框架训练不稳定、训练成本高，难以训练出令人满意的视频推理模型；第二，语言推理过程难以保证视觉信息的可靠性。为缓解这些问题，我们提出使用多模态元素进行推理，并设计了一个新框架，在多模态大模型上构建并增强多功能视频推理能力。我们精心设计了多任务冷启动与多任务强化学习，以提升模型的视觉感知能力与多种能力上的熟练度。在推理阶段，我们利用多模态推理与动态采样进一步提升性能。我们在基础多模态大模型（Qwen2-VL-7B-Base）上验证了该框架的效率。通过 3k 数据的冷启动与 5k 数据的强化学习训练，结合推理设计，我们的最终模型在七个公开视频基准上显著优于基础模型，甚至超越或接近以大规模数据训练的最优指令模型，如 Qwen2.5-VL-7B-Instruct。我们的代码将发布于 https://github.com/Wang-Xiaodong1899/VideoReasoner。
英文原文摘要 1624 字符
Multimodal Large Language Models (MLLMs) have made great progress in video understanding tasks. However, when it comes to understanding complex or lengthy videos, MLLMs tend to overlook details or produce hallucinations. To alleviate these issues, recent work has attempted to leverage reinforcement learning (RL) to boost models' deep linguistic reasoning of complex videos. But these methods have two main problems: First, the RL framework they used has unstable training, high training costs, and is difficult to train satisfactory video reasoning models; Second, the linguistic reasoning process is difficult to guarantee the reliability of visual information. To alleviate these problems, we propose to use multimodal elements for reasoning, and we design a novel framework to build and enhance versatile video reasoning capabilities on MLLMs. We carefully design a multi-task cold start and multi-task reinforcement learning to improve the model's visual perception and proficiency in multiple capabilities. In the inference phase, we leverage multimodal reasoning and dynamic sampling to further improve the performance. We verified the efficiency of the framework on a base MLLM (Qwen2-VL-7B-Base). Through cold-start with 3k data and reinforcement learning training with 5k data, combined with inference design, our final model significantly outperforms the base model on seven public video benchmarks, even surpassing and approaching the state-of-the-art Instruct Models such as Qwen2.5-VL-7B-Instruct trained with large-scale data. Our code will be available at https://github.com/Wang-Xiaodong1899/VideoReasoner.