增强视频视觉语言模型的海马体感知
Enhancing Video Vision Language Model with Hippocampal Sensing
提出跨模态预测感知范式与对比强化学习算法 VANAO，可借鉴用于视频-音频联合推理与选择性信息获取，提升视频 VLM 的推理效率与准确性。
原文依据 we introduce Hippocampal Sensing (HippoSense), a learning paradigm inspired by hippocampus that shifts the focus from temporal predictive sensing to cross-modal predictive sensing.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当前的视频视觉语言模型（VLM）被动地处理信息，缺乏动态规划其分析或跨视频与音频等关键模态进行联合推理的能力。为解决此问题，我们引入了海马体感知（HippoSense），一种受海马体启发的学习范式，将焦点从时序预测感知转向跨模态预测感知。HippoSense 的核心目标是训练模型从视频预测当前状态的音频-字幕摘要，反之亦然。我们提出了 HippoVLM，一个实施该范式的视频 VLM。HippoVLM 不是被动地摄入所有数据，而是利用思维链（CoT）主动推理其信息需求。我们的训练过程分为两步：首先用 HippoSense 微调 HippoVLM，然后应用一种新颖的对比强化学习（RL）算法——视频-音频负感知优化（VANAO）来优化这种选择性联合推理过程。该方法被证明非常有效：尽管模型规模显著更小，我们的 HippoVLM 在多个视频 VQA 基准上达到了与 GPT-4o 和 Gemini 1.5 Pro 等大型多模态大模型相竞争的性能。
英文原文摘要 1204 字符
Current video vision language models (VLMs) process information passively, lacking the ability to dynamically plan their analysis or perform joint reasoning across crucial modalities such as video and audio. To address this, we introduce Hippocampal Sensing (HippoSense), a learning paradigm inspired by hippocampus that shifts the focus from temporal predictive sensing to cross-modal predictive sensing. The core objective of HippoSense is to train the model to anticipate current status's audio-caption summarizations from video and vice versa. We present HippoVLM, a video VLM that operationalizes this paradigm. Instead of passively ingesting all data, HippoVLM actively reasons about its information needs using Chain-of-Thought (CoT). Our training process is twofold: we first finetune HippoVLM with HippoSense, and then apply a novel contrastive Reinforcement Learning (RL) algorithm, Video-Audio Negative-aware Optimization (VANAO), to optimize this selective co-reasoning process. This approach proves highly effective: despite their significantly smaller size, our HippoVLM model achieve competitive performance to massive MLLMs like GPT-4o and Gemini 1.5 Pro on multiple video VQA benchmarks.