OmniVTG：面向开放世界视频时序定位的大规模数据集与训练范式
OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding
提出自校正思维链训练范式，让多模态大模型先预测再反思修正，可迁移到其他需要精确定位的视觉语言任务；同时发布大规模开放世界视频时序定位数据集，为后续研究提供基准。
原文依据 we propose a Self-Correction Chain-of-Thought (CoT) training paradigm. We train the MLLM to first predict, then use its understanding capabilities to reflect on and refine its own predictions.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频时序定位（VTG）任务旨在根据文本查询定位视频片段，但在开放世界场景中，由于数据集规模和语义多样性有限，常见概念与罕见概念之间存在性能差距。为克服这些限制，我们提出了 OmniVTG，一个面向开放世界 VTG 的新大规模数据集，并配套一种自校正思维链（CoT）训练范式，旨在增强多模态大语言模型（MLLMs）的定位能力。我们的 OmniVTG 通过一种新颖的语义覆盖迭代扩展流程构建，该流程首先识别现有数据集词汇表中的空白，并收集极有可能包含这些目标概念的视频。为实现高质量标注，我们利用现代 MLLMs 在密集描述方面优于直接定位的洞察，设计了一个以描述为中心的数据引擎，提示 MLLMs 生成密集的、带时间戳的描述。除数据集外，我们观察到简单的监督微调（SFT）并不充分，因为罕见概念与常见概念之间的性能差距仍然存在。我们发现 MLLMs 的视频理解能力显著超过其直接定位能力。基于此，我们提出了一种自校正思维链（CoT）训练范式。我们训练 MLLM 先进行预测，然后利用其理解能力反思并精炼自己的预测。这一能力通过 SFT、CoT 微调和强化学习的三阶段流程来灌输。大量实验表明，我们的方法不仅在 OmniVTG 数据集的开放世界定位中表现出色，而且在四个现有 VTG 基准上实现了最优的零样本性能。代码见 https://github.com/oceanflowlab/OmniVTG。
英文原文摘要 1786 字符
Video Temporal Grounding (VTG), the task of localizing video segments from text queries, struggles in open-world settings due to limited dataset scale and semantic diversity, causing performance gaps between common and rare concepts. To overcome these limitations, we introduce OmniVTG, a new large-scale dataset for open-world VTG, coupled with a Self-Correction Chain-of-Thought (CoT) training paradigm designed to enhance the grounding capabilities of Multimodal Large Language Models (MLLMs). Our OmniVTG is constructed via a novel Semantic Coverage Iterative Expansion pipeline, which first identifies gaps in the vocabulary of existing datasets and collects videos that are highly likely to contain these target concepts. For high-quality annotation, we leverage the insight that modern MLLMs excel at dense captioning more than direct grounding and design a caption-centric data engine to prompt MLLMs to generate dense, timestamped descriptions. Beyond the dataset, we observe that simple supervised finetuning (SFT) is insufficient, as a performance gap between rare and common concepts still persists. We find that MLLMs' video understanding ability significantly surpasses their direct grounding ability. Based on this, we propose a Self-Correction Chain-of-Thought (CoT) training paradigm. We train the MLLM to first predict, then use its understanding capabilities to reflect on and refine its own predictions. This capability is instilled via a three-stage pipeline of SFT, CoT finetuning, and reinforcement learning. Extensive experiments show our approach not only excels at open-world grounding in our OmniVTG dataset but also achieves state-of-the-art zero-shot performance on four existing VTG benchmarks. Code is available at https://github.com/oceanflowlab/OmniVTG.