先分流,再定位:面向长视频理解的按查询类型自适应帧选择
Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding
提出按查询类型(全局/局部)分流的免训练帧选择策略:全局查询用均匀采样、局部查询才启用查询感知检索,可直接嵌入现有 LMM 长视频推理管线,在 256 帧输入下仍稳定提升,为长视频 token 预算分配提供可复用思路。
原文依据 we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论将大型多模态模型(LMM)应用于长视频理解,受限于有限的上下文长度以及处理稠密视频 token 在计算上难以承受的开销。因此,近期研究集中于查询感知的帧选择,但这类方法往往带来显著的计算开销。本文质疑了「如此复杂的搜索机制普遍必要」这一假设。我们首先识别并验证了一种查询类型划分,区分全局查询与局部查询。我们证明,均匀采样对全局查询既有效又高效,而局部查询确实需要查询感知的选择才能达到最优性能。基于这一洞见,我们提出 DIG,一个免训练的帧选择框架,根据查询类型自适应调整策略。具体而言,DIG 对全局查询采用高效的均匀采样,而对局部查询则激活一条专门流程来提取与查询相关的帧。在三个长视频理解基准上的实验表明,DIG 一致优于现有基线,并能稳健地提升 LMM 性能,即使将输入帧数扩展至 256 帧亦然。
英文原文摘要 1234 字符
The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically, DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.