FlexiVideo：面向高效视频理解的变异感知时序动态建模
FlexiVideo: Variation-Aware Temporal Dynamics Modeling for Efficient Video Understanding
面向视频 MLLM 的视觉 token 冗余问题，提出按帧间差异自适应分段并动态调整时序编码窗口，可直接复用到任何视频 LLM 的视觉编码前端以降低 token 数；其「高动态帧对导致视觉混淆」的发现也提示做视频 token 合并/采样时需按运动强度而非固定间隔处理。
原文依据 FlexiVideo-3B reduces visual tokens by 43.5% compared with Qwen2.5-VL-3B while achieving a 1.3% performance gain
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论自然视频呈现出异质的时序动态：某些片段发生高动态的场景转换，另一些片段则以低动态的视觉变化为主。然而，多数多模态大模型（MLLM）对所有帧一视同仁，导致冗余的视觉编码，带来显著的计算开销。近期最优模型 Qwen2.5-VL 采用固定的双帧编码方案，但我们的先导实验表明，它在高动态帧对下会出现视觉混淆问题。为解决该问题，我们提出 FlexiVideo，一种利用视觉变异来建模时序动态的高效 MLLM。FlexiVideo 首先采用自适应时序分段模块估计帧间差异，将连续帧划分为视觉变化细微的场景片段。随后，动态时空嵌入模块调整用于场景级编码的时序窗口。通过在结构化的时序组织中重构场景级视觉表示，我们的方法能更有效地建模动态，并在保留细粒度视觉变异的同时降低编码负担。大量实验表明，FlexiVideo-3B 在 6 个通用视频基准上持续优于 Qwen2.5-VL-3B。值得注意的是，在 10 FPS 下于 MotionBench 上评测时，FlexiVideo-3B 相比 Qwen2.5-VL-3B 减少 43.5% 的视觉 token，同时取得 1.3% 的性能提升，在效率与效果之间实现了显著更优的平衡。代码与检查点见 github.com/AI9Stars/FlexiVideo。
英文原文摘要 1612 字符
Natural videos exhibit heterogeneous temporal dynamics, with certain segments undergoing high-dynamic scene transitions and others dominated by low-dynamic visual changes. However, treating all frames identically, a common practice in most MLLMs, leads to redundant visual encoding, which results in significant computational overhead. The recent state-of-the-art model, i.e., Qwen2.5-VL, adopts a fixed two-frame encoding scheme, but our pilot experiments indicate that it encounters a visual confusion problem under high-dynamic frame pairs. To address this issue, we propose FlexiVideo, an efficient MLLM that models temporal dynamics leveraging visual variation. FlexiVideo first employs an adaptive temporal segmentation module to estimate inter-frame differences, grouping consecutive frames into scene segments with subtle visual changes. Subsequently, a dynamical spatio-temporal embedding module adjusts the temporal window for scene-level encoding. By restructuring scene-level visual representations within a structured temporal organization, our approach models dynamics more effectively and reduces the encoding burden while preserving fine-grained visual variations. Extensive experiments show that FlexiVideo-3B consistently outperforms Qwen2.5-VL-3B across 6 general video benchmarks. Notably, when evaluated on MotionBench at 10 FPS, FlexiVideo-3B reduces visual tokens by 43.5% compared with Qwen2.5-VL-3B while achieving a 1.3% performance gain, striking a significantly better balance between efficiency and effectiveness. Code and checkpoints are available at github.com/AI9Stars/FlexiVideo.