GroundVTS：多模态大语言模型中的视觉词元采样用于视频时序定位
GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
提出查询引导的视觉词元采样机制，在输入LLM前过滤冗余视觉词元，可复用于其他视频-语言任务（如视频问答、时刻检索）以提升时序建模效率。
原文依据 GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频时序定位（VTG）是视频理解中的关键任务，也是将视频大语言模型（Vid-LLMs）扩展到更广泛应用的关键能力。然而，现有的Vid-LLMs依赖均匀帧采样来提取视频信息，导致关键帧分布稀疏并丢失关键的时序线索。为解决这一局限，我们提出了Grounded Visual Token Sampling（GroundVTS），一种专注于最具信息量时序片段的Vid-LLM架构。GroundVTS采用细粒度的、查询引导的机制，在将视觉词元输入LLM之前对其进行过滤，从而保留必要的时空信息并保持时序连贯性。此外，我们引入了一种渐进式优化策略，使LLM能够有效适应视觉特征的非均匀分布，增强其建模时序依赖关系的能力并实现精确的视频定位。我们在三个标准VTG基准上全面评估了GroundVTS，其性能优于现有方法，在时刻检索任务上平均交并比（mIoU）提升了7.7个百分点，在高光检测任务上平均精度均值（mAP）提升了12.0个百分点。代码可在https://github.com/Florence365/GroundVTS获取。
英文原文摘要 1298 字符
Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at the \href https://github.com/Florence365/GroundVTS GroundVTS code repository .