T2SGrid：面向视频时序定位的时序到空间网格化
T2SGrid: Temporal-to-Spatial Gridification for Video Temporal Grounding
将视频帧按时间顺序以行主序拼成网格图像，把时序定位转化为二维空间理解任务，为视觉语言模型处理长视频提供了一种无需额外时间戳词元的可复用输入组织方式。
原文依据 reformulates video temporal understanding as a spatial understanding task
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频时序定位（VTG）旨在定位与自然语言查询相对应的视频片段，这要求对复杂的时序动态有全面的理解。现有的视觉语言大模型通常通过位置编码、基于文本的时间戳或视觉帧编号来感知时序动态。然而，这些方法存在明显局限：为每一帧分配一个基于文本的时间戳词元会引入额外的计算开销，并导致视觉注意力稀疏；位置编码难以捕捉绝对时序信息；视觉帧编号则常常损害空间细节。为解决这些问题，我们提出时序到空间网格化（T2SGrid），一种将视频时序理解重新表述为空间理解任务的新框架。T2SGrid 的核心思想是以片段而非单帧的方式处理视频内容。我们采用重叠滑动窗口机制将视频切分为时序片段。在每个窗口内，各帧按时间顺序以行主序排列成一张复合网格图像，从而有效地将时序序列转换为结构化的二维布局。网格化不仅编码了时序信息，还增强了每个网格内部的局部注意力。此外，T2SGrid 能够使用复合文本时间戳来建立全局时序感知。在标准 VTG 基准上的实验表明，T2SGrid 取得了更优的性能。
英文原文摘要 1496 字符
Video Temporal Grounding (VTG) aims to localize the video segment that corresponds to a natural language query, which requires a comprehensive understanding of complex temporal dynamics. Existing Vision-LMMs typically perceive temporal dynamics via positional encoding, text-based timestamps, or visual frame numbering. However, these approaches exhibit notable limitations: assigning each frame a text-based timestamp token introduces additional computational overhead and leads to sparsity in visual attention, positional encoding struggles to capture absolute temporal information, and visual frame numbering often compromises spatial detail. To address these issues, we propose Temporal to Spatial Gridification (T2SGrid), a novel framework that reformulates video temporal understanding as a spatial understanding task. The core idea of T2SGrid is to process video content in clips rather than individual frames. we employ a overlapping sliding windows mechanism to segment the video into temporal clips. Within each window, frames are arranged chronologically in a row-major order into a composite grid image, effectively transforming temporal sequences into structured 2D layouts. The gridification not only encodes temporal information but also enhances local attention within each grid. Furthermore, T2SGrid enables the use of composite text timestamps to establish global temporal awareness. Experiments on standard VTG benchmarks demonstrate that T2SGrid achieves superior performance.