ReMoRa：基于精细化运动表征的多模态大语言模型用于长视频理解
ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
用块级运动矢量作为光流式的紧凑时序代理，替代逐帧 RGB 输入，并配去噪模块提升运动表征保真度；对做长视频理解的人，这是一条把解码与序列长度成本从 RGB 帧转移到压缩域的可复用路径。
原文依据 These motion representations act as a compact proxy for optical flow, capturing temporal dynamics without full frame decoding.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管多模态大语言模型（MLLMs）在广泛任务上取得了显著成功，长视频理解仍是一项重大挑战。本研究聚焦于利用 MLLMs 进行视频理解。该任务之所以困难，是因为处理完整的 RGB 帧流在计算上不可行且高度冗余，而自注意力相对于序列长度具有二次复杂度。本文提出 ReMoRa，一种直接基于视频压缩表征进行处理的视频 MLLM。模型保留一小组稀疏的 RGB 关键帧以表征外观，同时将时序动态编码为运动表征，从而无需顺序的 RGB 帧。这些运动表征充当光流的紧凑代理，在无需完整帧解码的情况下捕获时序动态。为消除基于块的运动所固有的噪声与低保真度，我们引入一个模块对其进行去噪并生成细粒度的运动表征。此外，我们的模型以随序列长度线性扩展的方式压缩这些特征。我们通过在一整套全面的长视频理解基准上的大量实验验证了 ReMoRa 的有效性。ReMoRa 在包括 LongVideoBench、NExT-QA 和 MLVU 在内的多个挑战性基准上优于基线方法。项目主页见 https://remora-v1rcm.kinsta.page/。
英文原文摘要 1404 字符
While multimodal large language models (MLLMs) have shown remarkable success across a wide range of tasks, long-form video understanding remains a significant challenge.In this study, we focus on video understanding by MLLMs.This task is challenging because processing a full stream of RGB frames is computationally intractable and highly redundant, as self-attention have quadratic complexity with sequence length.In this paper, we propose ReMoRa, a video MLLM that processes videos by operating directly on their compressed representations.A sparse set of RGB keyframes is retained for appearance, while temporal dynamics are encoded as a motion representation, removing the need for sequential RGB frames.These motion representations act as a compact proxy for optical flow, capturing temporal dynamics without full frame decoding.To refine the noise and low fidelity of block-based motions, we introduce a module to denoise and generate a fine-grained motion representation.Furthermore, our model compresses these features in a way that scales linearly with sequence length.We demonstrate the effectiveness of ReMoRa through extensive experiments across a comprehensive suite of long-video understanding benchmarks.ReMoRa outperformed baseline methods on multiple challenging benchmarks, including LongVideoBench, NExT-QA, and MLVU. Our project page is available at https://remora-v1rcm.kinsta.page/.