基于视觉几何Transformer的快速空间跟踪
Fast Spatial Tracking with Visual Geometry Transformer
提出一种基于前馈视觉几何Transformer的实时空间点跟踪框架，其双分支嵌入设计与交互模块可复用于其他需要全局-局部特征融合的视频理解任务，如视频目标分割与光流估计。
原文依据 we present a novel spatial tracker that leverages a feed-forward visual geometry transformer to predict the trajectories of arbitrary query points from monocular videos in real time.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有的三维点跟踪方法大多依赖启发式设计或场景重建，这带来了显著的计算开销，难以满足实时应用的需求。为解决该问题，本文提出了一种新颖的空间跟踪器，利用前馈视觉几何Transformer从单目视频中实时预测任意查询点的轨迹。具体而言，我们采用查询初始化机制，为每个查询点维护并更新一个全局特征向量和一组帧级特征向量。然后，我们提出了一种新的空间跟踪框架，由视觉几何Transformer骨干网络、全局嵌入分支、帧级嵌入分支和跟踪头组成。关键创新在于双分支嵌入设计：全局嵌入分支将整个视频的几何基础特征整合为全局查询特征，以优化整个序列上的轨迹信息；帧级分支将各帧的几何基础特征整合为帧级查询特征，以细化细粒度的轨迹坐标预测。此外，为促进全局分支与帧级分支之间的协作，我们引入了一个交互模块，实现全局查询特征与帧级查询特征之间的单向或双向信息交换。在多个点跟踪基准数据集上的大量实验表明，与最优方法相比，我们的方法在保持相当跟踪精度的同时，实现了显著更快的空间跟踪速度。
英文原文摘要 1690 字符
Existing 3D point tracking methods mostly rely on heuristic designs or scene reconstruction, which incurs significant computational overhead and makes it difficult to meet the demands of real-time applications. To address this problem, in this work, we present a novel spatial tracker that leverages a feed-forward visual geometry transformer to predict the trajectories of arbitrary query points from monocular videos in real time. Specifically, we employ a query initialization mechanism to maintain and update a global feature vector and a set of frame-level feature vectors for each query point. Then, we propose a new spatial tracking framework, which consists of a visual geometry transformer backbone, a global embedding branch, a frame-level embedding branch, and a tracking head. The key innovation lies in the dual-branch embedding design, where the global embedding branch integrates geometry-grounded features of the entire video into global query features to optimize track information across the entire sequence and the frame-level branch combines geometry-grounded features of each respective frame into frame-level query features to refine fine-grained track coordinate predictions. Furthermore, to facilitate collaboration between the global branch and the frame-level branch, we introduce an interaction module which enables unidirectional or bidirectional information exchange between the global query features and frame-level query features. Extensive experiments on various point tracking benchmark datasets show that our approach achieves significantly fast spatial tracking speed compared with state-of-the-art methods, while maintaining comparable tracking accuracy.