通过视觉记忆机制扩展多模态大语言模型的长视频理解能力
Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism
提出免训练的视觉记忆机制，将视觉KV缓存作为记忆源，通过双路径压缩和多样化读取策略，使多模态大模型能处理无限长度视频，为长视频理解提供可复用的记忆管理方案。
原文依据 FlexMem first consider the visual KV caches as the memory sources, and realize the effective memory transfer and writing via a dual-pathway compression design.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论长视频理解是阻碍多模态大语言模型（MLLMs）发展的关键挑战。本文从视觉记忆机制的角度研究该问题，提出了一种新颖且免训练的方法，称为灵活记忆（FlexMem）。原则上，FlexMem旨在模仿人类观看视频的行为，即持续观看视频内容并回忆最相关的记忆片段来回答问题。通过这种方式，FlexMem可以帮助MLLMs实现无限长度的视频理解，而以往方法一次性处理所有视频信息且存在输入上限。具体而言，FlexMem首先将视觉KV缓存视为记忆源，并通过双路径压缩设计实现有效的记忆传递与写入。随后，FlexMem还探索了针对多样化视频理解任务的不同记忆读取策略，包括流行的流式任务。为验证FlexMem，我们将其应用于两个流行的视频MLLMs，并在五个长视频任务和一个流式视频任务上进行了大量实验。实验结果表明，在单块3090 GPU上，FlexMem相比现有高效视频理解方法取得了明显改进，并能处理超过1k帧，还帮助基础MLLMs在某些基准上达到与SOTA MLLMs（如GPT-4o和Gemini-1.5 Pro）相当甚至更好的性能。代码已发布于：https://github.com/city1517/FlexMem。
英文原文摘要 1513 字符
Long video understanding is a key challenge that plagues the advancement of Multimodal Large language Models (MLLMs). In this paper, we study this problem from the perspective of visual memory mechanism, and proposed a novel and training-free approach, termed Flexible Memory (FlexMem). In principle, FlexMem aims to mimic human behavior of video watching, i.e., continually watching video content and recalling the most relevant memory fragments to answer the question. In this way, FlexMem can help MLLMs achieve video understanding of infinite lengths, unlike previous methods that process all video information at once and have input upper-limit. Concretely, FlexMem first consider the visual KV caches as the memory sources, and realize the effective memory transfer and writing via a dual-pathway compression design. Afterwards, FlexMem also explores different memory reading strategies for the diverse video understanding tasks, including the popular streaming one. To validate FlexMem, we apply it to two popular video-MLLMs, and conduct extensive experiments on five long video and one streaming video task. The experimental results show that on a single 3090 GPU, our FlexMem can achieve obvious improvements than existing efficient video understanding methods and process more than 1k frames, which also helps the base MLLMs achieve comparable or even better performance than SOTA MLLMs on some benchmarks, e.g. , GPT-4o and Gemini-1.5 Pro. Our code is released at: https://github.com/city1517/FlexMem.