基于小波变换的语义边界检测帧选择用于长视频理解
Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding
提出一种免训练的小波变换帧选择方法，通过多分辨率分析从噪声相似度信号中提取语义边界，为长视频理解中的关键帧选择提供了可复用的信号去噪与分段思路。
原文依据 we leverage the wavelet transform, which provides an ideal solution through its multi-resolution analysis in both time and frequency domains.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在将大型视觉语言模型（LVLM）应用于长视频时，由于帧冗余度高且上下文窗口有限，帧选择至关重要。当前方法通常选择与给定查询高度相关的帧，导致所选帧集合不连贯，忽视了视频的叙事结构。本文提出基于小波变换的语义边界检测帧选择（WFS-SB），这是一个免训练框架，提出了新视角：有效的视频理解不仅依赖于高相关性，更重要的是捕捉语义变化——即对理解视频整体故事情节至关重要的叙事转折点。然而，由于模型不确定性和瞬态视觉变化引起的高频噪声，直接检测查询-帧相似度信号中的突变往往不可靠。为此，我们利用小波变换，其通过时频域的多分辨率分析提供了理想解决方案。通过应用该变换，我们将噪声信号分解为多个尺度，并从最粗尺度中提取干净的语义变化信号。我们将该信号的局部极值识别为语义边界，从而将视频分割为连贯片段。在此基础上，WFS-SB包含两阶段策略：首先，基于复合重要性分数自适应地为每个片段分配帧预算；其次，在每个片段内采用最大边际相关性方法选择多样且相关的帧集合。大量实验表明，WFS-SB显著提升了LVLM性能，例如在VideoMME上准确率提升5.5%，在MLVU上提升9.5%，在LongVideoBench上提升6.2%，持续优于最优方法。代码见https://github.com/MAC-AutoML/WFS-SB。
英文原文摘要 1871 字符
Frame selectoin is crucial due to high frame redundancy and limited context windows when applying Large Vision-Language Models (LVLMs) to long videos. Current methods typically select frames with high relevance to a given query, resulting a disjointed set of frames that disregard the narrative structure of video. In this paper, we introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework that presents a new perspective: effective video understanding hinges not only on high relevance but, more importantly, on capturing semantic shifts--pivotal moments of narrative change that are essential to comprehending the holistic storyline of video. However, a direct detection of abrupt changes in the query-frame similarity signal is often unreliable due to high-frequency noise arising from model uncertainty and transient visual variations.To address this, we leverage the wavelet transform, which provides an ideal solution through its multi-resolution analysis in both time and frequency domains. By applying this transform, we decompose the noisy signal into multiple scales and extract a clean semantic change signal from the coarsest scale. We identify the local extrema of this signal as semantic boundaries, which segment the video into coherent clips. Building on this, WFS-SB comprises a two-stage strategy: first, adaptively allocating a frame budget to each clip based on a composite importance score; and second, within each clip, employing the Maximal Marginal Relevance approach to select a diverse yet relevant set of frames. Extensive experiments show that WFS-SB significantly boosts LVLM performance, e.g., improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench, consistently outperforming state-of-the-art methods. Our code is available at https://github.com/MAC-AutoML/WFS-SB.