面向流式指代视频分割的大语言模型方法
Towards Streaming Referring Video Segmentation via Large Language Model
把图像级指代分割模型通过语义嵌入复用与在线掩码一致性感知扩展成流式视频分割，无需额外参数即可在单卡上以 7 FPS 流式推理，为实时指代视频分割与 MLLM 的在线部署提供可复用范式。
原文依据 we propose a simple but efficient MLLM-based framework StreamingRVOS, which can extend image-level segmentation to video-level via a streaming pipeline without introducing extra parameters
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当前的指代视频分割方法通常以离线方式运行：先选取稀疏帧进行图像级指代分割，再将得到的掩码在视频中传播。尽管视频采样能够捕获全局上下文，但其彼此孤立的处理步骤不仅使优化变得复杂，也限制了其在真实流式场景中的适用性。本文提出一个简单而高效、基于多模态大语言模型的框架 StreamingRVOS，它能够通过流式流水线将图像级分割扩展到视频级，且不引入额外参数。具体而言，我们采用语义嵌入复用（SER）方法在帧间传播时序上下文，使模型能够感知视频中的语义表示。随后，我们提出在线掩码一致性感知（OMCP）策略，自适应地调用多模态大语言模型重新感知当前场景并重新生成语义嵌入。我们在多个下游数据集上开展了大量实验，以证明 StreamingRVOS 的有效性。与以往方法相比，我们的方法在指代视频分割上取得了优异性能（1B 变体在 MeViS 数据集上较 Sa2VA 提升 19.2），同时在单张 A800 GPU 上进行流式推理时平均速度达到 7 FPS。
英文原文摘要 1328 字符
Current referring video segmentation methods typically operate in an offline manner, where sparse frames are first selected for image-level referring segmentation, and the resulting masks are then propagated across the video. Although video sampling captures global context, its isolated processing steps not only complicate optimization but also restrict applicability to real-world streaming scenarios. In this paper, we propose a simple but efficient MLLM-based framework StreamingRVOS, which can extend image-level segmentation to video-level via a streaming pipeline without introducing extra parameters. Specifically, we employ a Semantic Embedding Recycling (SER) method to propagate temporal context across frames, enabling the model to perceive semantic representation in the video. Then, we propose an Online Mask Consistency Perception (OMCP) strategy to adaptively invoke the MLLM to re-perceive the current scene and regenerate the semantic embedding. We conduct extensive experiments on multiple downstream datasets to prove the effectiveness of StreamingRVOS. Compared to previous methods, our method achieves excellent performance in referring video segmentation (1B variant improves upon Sa2VA by 19.2 on the MeViS dataset), while operating at an average speed of 7 FPS under streaming inference on 1 xA800 GPU.