SPARROW：在像素级视频多模态大模型中学习空间精度与时序指代一致性
SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs
针对视频 MLLM 中静态 [SEG] 词元缺乏时序上下文导致的漂移与身份切换问题，提出目标特定跟踪特征（TSF）注入时序对齐的指代线索，并用 [BOX]/[SEG] 双提示解码融合几何先验与语义定位；这套即插即用设计可直接迁移到任意视频 MLLM 的像素级指代任务上。
原文依据 a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLM）已从图像级推理推进到像素级定位，但将这些能力扩展到视频仍然困难，因为模型必须同时实现空间精度与时序一致的指代跟踪。现有视频 MLLM 往往依赖静态分割词元（[SEG]）进行逐帧定位，该词元提供了语义但缺乏时序上下文，导致物体移动或重新出现时出现空间漂移、身份切换与初始化不稳定。我们提出 SPARROW，一个像素级视频 MLLM，通过两个关键组件统一空间准确性与时序稳定性：(i) 目标特定跟踪特征（TSF），在训练期间注入时序对齐的指代线索；(ii) 双提示设计，解码框（[BOX]）与分割（[SEG]）词元，以融合几何先验与语义定位。SPARROW 由一个精心构建的指代视频数据集支撑，包含 30,646 个视频与 45,231 个问答对，并通过基于 SAM2 的类别无关提议器实现端到端运行，无需外部检测器。将 SPARROW 集成到三个近期开源视频 MLLM（UniPixel、GLUS 与 VideoGLaMM）后，在六个基准上取得一致增益：RVOS 上 J&F 最高提升 +8.9，视觉定位上平均交并比提升 +5，GCG 上 CLAIR 提升 +5.4。这些结果表明，SPARROW 显著提升了像素级视频理解中的指代稳定性、空间精度与时序连贯性。项目主页：https://risys-lab.github.io/SPARROW/
英文原文摘要 1521 字符
Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: https://risys-lab.github.io/SPARROW/