通过强化学习实现长视频理解中的高效帧选择
Efficient Frame Selection for Long Video Understanding via Reinforcement Learning
把「选哪些帧」当作可学习策略，用分层奖励的强化学习训练轻量选择器，可直接插到任意 MLLM 前端替代均匀采样，是长视频理解中低成本提升输入质量的通用手段。
原文依据 we propose a lightweight frame selection method to identify keyframes and train it via a two-stage strategy
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLM）的最新进展推动了视频理解的显著进步。由于上下文窗口有限且计算开销较大，多数 MLLM 采用均匀帧采样。这种做法极易遗漏关键视觉信息，尤其在长视频中会制约性能。为解决该问题，我们提出一种轻量级帧选择方法以识别关键帧，并通过两阶段策略对其进行训练。在预训练阶段，帧选择器学习建模单帧与查询之间的相关性。在强化学习（RL）阶段，我们采用分层奖励，在帧组合层面与单帧层面评估选择质量。通过对帧组合的随机探索，选择器学会识别并保留能够提升任务性能的帧，而非仅仅最大化查询相关性——后者可能产生误导。所选帧作为下游 MLLM 的输入，用于视频理解与推理。实验结果表明，所提出的选择器在涵盖中长视频的多个基准上提升了多种下游 MLLM 的性能。
英文原文摘要 1185 字符
Recent advances in Multimodal Large Language Models (MLLMs) have led to significant progress in video understanding. Due to limited context windows and computational overhead, most MLLMs adopt uniform frame sampling. This approach is at high risk of missing critical visual information and constrains performance especially for long videos. To address this problem, we propose a lightweight frame selection method to identify keyframes and train it via a two-stage strategy. In the pre-training stage, the frame selector learns to model relevance between individual video frames and queries. In the reinforcement learning (RL) stage, we employ a hierarchical reward that evaluates selection quality at combination and frame levels. Through stochastic exploration of frame combinations, the selector learns to identify and retain frames that improve task performance rather than merely maximizing query relevance, which can be misleading. The selected frames serve as input to downstream MLLMs for video understanding and reasoning. Experimental results demonstrate the proposed selector improves performance of diverse downstream MLLMs across benchmarks spanning medium to long videos.