基于记忆反馈的问题引导视觉压缩用于长视频理解
Question-guided Visual Compression with Memory Feedback for Long-Term Video Understanding
提出问题引导的多模态选择性注意力（QMSA），将上下文记忆中的历史帧反馈到当前感知，用于压缩长视频中的视觉 token；该机制可复用于任何需要长时序视觉 token 压缩的 VLM 推理流程。
原文依据 Question-guided Visual Compression with Memory Feedback (QViC-MF), a framework for long-term video understanding
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在基于大型多模态模型的长期视频理解背景下，已有许多框架被提出。尽管基于 Transformer 的视觉压缩器和记忆增强方法常被用于处理长视频，但它们通常独立压缩每一帧，因而在需要理解完整事件的任务（如 MLVU 和 VNBench 中的时序排序任务）上无法取得强劲性能。这促使我们重新思考从感知到记忆的传统单向方案，转而建立一个反馈驱动过程，使存储在上下文记忆中的过往视觉上下文能够有益于正在进行的感知。为此，我们提出问题引导的视觉压缩与记忆反馈（QViC-MF），一个用于长期视频理解的框架。其核心是问题引导的多模态选择性注意力（QMSA），它学习从当前片段和记忆中过往相关帧中保留与给定问题相关的视觉信息。压缩器与记忆反馈对整个视频的每个片段迭代工作。这种简单而有效的设计在长期视频理解任务上带来大幅性能提升。大量实验表明，我们的方法在 MLVU 测试集上比当前最优方法显著提升 6.1%，在 LVBench 上提升 8.3%，在 VNBench Long 上提升 18.3%，在 VideoMME Long 上提升 3.7%。代码将在 https://github.com/FujitsuResearch/QViC-MF/ 发布。
英文原文摘要 1482 字符
In the context of long-term video understanding with large multimodal models, many frameworks have been proposed. Although transformer-based visual compressors and memory-augmented approaches are often used to process long videos, they usually compress each frame independently and therefore fail to achieve strong performance on tasks that require understanding complete events, such as temporal ordering tasks in MLVU and VNBench. This motivates us to rethink the conventional one-way scheme from perception to memory, and instead establish a feedbackdriven process in which past visual contexts stored in the context memory can benefit ongoing perception. To this end, we propose Question-guided Visual Compression with Memory Feedback (QViC-MF), a framework for long-term video understanding. At its core is a Question-guided Multimodal Selective Attention (QMSA), which learns to preserve visual information related to the given question from both the current clip and the past related frames from the memory. The compressor and memory feedback work iteratively for each clip of the entire video. This simple yet effective design yields large performance gains on longterm video understanding tasks. Extensive experiments show that our method achieves significant improvement over current state-of-the-art methods by 6.1% on MLVU test, 8.3% on LVBench, 18.3% on VNBench Long, and 3.7% on VideoMME Long. The code will be available at https://github.com/FujitsuResearch/QViC-MF/.