视觉语言模型 · CVPR2026 ·2026.06.03

VideoSSR:视频自监督强化学习

VideoSSR: Video Self-Supervised Reinforcement Learning

Zefeng He、Xiaoye Qu、Yafu Li、Siyuan Huang、Daizong Liu、Yu Cheng

与你有关的部分

提出三种视频自监督前置任务(异常定位、物体计数、时序拼图)来自动生成可验证的训练数据,为视频多模态大模型提供无需人工标注的强化学习训练信号,可直接用于提升视频时序定位与复杂推理能力。

原文依据 we first introduce three self-supervised pretext tasks for video understanding: Anomaly Grounding, Object Counting, and Temporal Jigsaw.
可迁移到 视频问答时序定位视频理解

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

基于可验证奖励的强化学习(RLVR)显著提升了多模态大语言模型(MLLMs)的视频理解能力。然而,MLLMs 的快速发展正超越现有视频数据集的复杂度,而人工标注新的高质量数据仍然成本高昂。本文研究一个关键问题:能否利用视频中丰富的内在信息来自动生成高质量、可验证的训练数据?为探究此问题,我们首先为视频理解引入三个自监督前置任务:异常定位、物体计数和时序拼图。为验证这些任务的难度,我们构建了视频内在理解基准(VIUBench),揭示当前最优 MLLMs 在这些任务上表现显著不足。基于这些前置任务,我们开发了 VideoSSR-30K 数据集,并提出 VideoSSR,一种用于 RLVR 的新型视频自监督强化学习框架。在涵盖四个主要视频领域(通用视频问答、长视频问答、时序定位和复杂推理)的 17 个基准上的大量实验表明,VideoSSR 持续提升模型性能,平均提升超过 5%。这些结果确立了 VideoSSR 作为开发 MLLMs 更高级视频理解能力的有力基础框架。

英文原文摘要 1429 字符

Reinforcement Learning with Verifiable Reward (RLVR) has substantially advanced the video understanding capabilities of Multimodal Large Language Models (MLLMs). However, the rapid progress of MLLMs is outpacing the complexity of existing video datasets, while the manual annotation of new, high-quality data remains prohibitively expensive.This work investigates a pivotal question: Can the rich, intrinsic information within videos be harnessed to self-generate high-quality, verifiable training data?To investigate this problem, we first introduce three self-supervised pretext tasks for video understanding: Anomaly Grounding, Object Counting, and Temporal Jigsaw. To validate the difficulty of these tasks, we construct the Video Intrinsic Understanding Benchmark (VIUBench), revealing that current state-of-the-art MLLMs struggle significantly on these tasks. Building upon these pretext tasks, we develop the VideoSSR-30K dataset and propose VideoSSR, a novel video self-supervised reinforcement learning framework for RLVR. Extensive experiments across 17 benchmarks, spanning four major video domains (General Video QA, Long Video QA, Temporal Grounding, and Complex Reasoning), demonstrate that our VideoSSR consistently enhances model performance, yielding an average improvement of over 5%. These results establish VideoSSR as a potent foundational framework for developing more advanced video understanding in MLLMs.

标签

会议 CVPR2026
页码 26530-26540
Bibkey
来源 CVF Open Access