WorldMM：面向长视频推理的动态多模态记忆智能体
WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning
把长视频理解从纯文本摘要记忆扩展到文本+视觉的多源记忆库，并用自适应检索代理按查询迭代选择记忆源与时间粒度；对做长视频 VQA、视频检索或记忆增强 VLM 的工作，这套「多粒度情景记忆+语义记忆+视觉记忆」的组织与检索策略可直接复用。
原文依据 an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期视频大语言模型的进展已展现出理解短视频片段的强大能力。然而，由于上下文容量有限，且在抽象过程中会丢失关键视觉细节，将其扩展到数小时乃至数天的长视频仍极具挑战。现有的记忆增强方法通过利用视频片段的文本摘要来缓解这一问题，但它们严重依赖文本，在对复杂场景进行推理时未能利用视觉证据。此外，从固定时间尺度进行检索进一步限制了其捕捉持续时间可变的事件的灵活性。为此，我们提出 WorldMM，一种新颖的多模态记忆智能体，它构建并检索多种互补的记忆，涵盖文本与视觉两种表示。WorldMM 包含三类记忆：情景记忆在多个时间尺度上索引事实性事件，语义记忆持续更新高层概念知识，视觉记忆则保留场景的细节信息。在推理时，一个自适应检索智能体根据查询迭代地选择最相关的记忆源并利用多种时间粒度，持续进行直至判定已收集到足够信息。WorldMM 在五个长视频问答基准上显著优于现有基线，相较此前最优方法平均取得 8.4% 的性能提升，显示出其在长视频推理上的有效性。
英文原文摘要 1539 字符
Recent advances in video large language models have demonstrated strong capabilities in understanding short clips. However, scaling them to hours- or days-long videos remains highly challenging due to limited context capacity and the loss of critical visual details during abstraction. Existing memory-augmented methods mitigate this by leveraging textual summaries of video segments, yet they heavily rely on text and fail to utilize visual evidence when reasoning over complex scenes. Moreover, retrieving from fixed temporal scales further limits their flexibility in capturing events that span variable durations. To address this, we introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories, encompassing both textual and visual representations. WorldMM comprises three types of memory: episodic memory indexes factual events across multiple temporal scales, semantic memory continuously updates high-level conceptual knowledge, and visual memory preserves detailed information about scenes. During inference, an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query, continuing until it determines that sufficient information has been gathered. WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks, achieving an average 8.4% performance gain over previous state-of-the-art methods, showing its effectiveness on long video reasoning.