面向长视频理解的视频面板
Video Panels for Long Video Understanding
提出一种免训练、参数无关的视觉提示策略，通过将多帧拼接为面板图像来权衡空间细节与时序分辨率，可直接集成到现有视频语言模型中提升长视频理解性能。
原文依据 By combining multiple frames as panels into one image, we effectively trade off spatial details for temporal resolution.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期视频语言模型（VLM）在长视频理解上取得了有希望的结果，但其性能仍落后于图像或短视频任务。这导致大量研究通过引入新模块和额外复杂度来改进VLM的长上下文建模。本文采取不同方法：不利用有限数据微调VLM，而是尝试最大化现有模型的性能。为此，我们提出一种专为长视频理解设计的新视觉提示策略。通过将多帧作为面板组合成一幅图像，我们有效权衡了空间细节与时序分辨率。我们的方法免训练、无参数且模型无关，可无缝集成到现有VLM中。在五个基准、多种模型架构、规模和上下文窗口上的广泛实验证实了我们方法的一致性。对于视频最长的TimeScope（Long）数据集，视频问答准确率最高提升19.4%。总体而言，我们的方法提升了长视频理解模型的性能标准。代码见 https://fedespu.github.io/Video-Panels。
英文原文摘要 1271 字符
Recent Video-Language Models (VLMs) achieve promising results on long-video understanding, but their performance still lags behind that achieved on tasks involving images or short videos. This has led to great interest in improving the long context modeling of VLMs by introducing novel modules and additional complexity. In this paper, we take a different approach: rather than fine-tuning VLMs with the limited data available, we attempt to maximize the performance of existing models.To this end, we propose a novel visual prompting strategy specifically designed for long-video understanding. By combining multiple frames as panels into one image, we effectively trade off spatial details for temporal resolution.Our approach is training-free, parameter-free, and model-agnostic, and can be seamlessly integrated into existing VLMs.Extensive experiments on five established benchmarks across a wide range of model architectures, sizes, and context windows confirm the consistency of our approach. For the TimeScope (Long) dataset, which has the longest videos, the accuracy for video question answering is improved by up to 19.4%. Overall, our method raises the bar for long video understanding models. The code is available at https://fedespu.github.io/Video-Panels.