基于单句自然语言描述的实例跟踪学习
Learning to Track Instance from Single Nature Language Description
提出动态词元聚合模块，通过锚框词元选择模板帧中的目标词元并与语言词元融合，为视觉语言跟踪提供了一种无需边界框标注的自监督训练范式，可复用于其他多模态时序任务。
原文依据 we propose an efficient Dynamic Token Aggregation Module, which treats each visual token unequally.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论如何在不依赖任何边界框真值的情况下，利用视频序列中的自然语言描述实现视觉语言（VL）跟踪？在本工作中，我们通过研究自监督视觉语言跟踪来实现这一目标，该任务旨在评估由自然语言描述引导的跟踪能力。我们提出了\tracker，一种新颖的自监督视觉语言跟踪器，能够根据语言描述跟踪任意被指代的目标。与传统方法平等融合所有语言和视觉词元不同，我们提出了一种高效的动态词元聚合模块，该模块对每个视觉词元进行非平等处理。该模块包含三个主要步骤：i）基于锚框词元，从模板帧中选择多个重要的目标词元。ii）根据注意力分数合并选中的目标词元，并将其聚合到语言词元中，从而消除冗余的视觉词元噪声并增强语义对齐。iii）最后，融合后的语言词元作为引导信号，从搜索帧中提取潜在的目标词元并将其传播到后续帧，增强时序提示并鼓励跟踪器从无标签视频中自主地学习实例跟踪。这种新的建模方法使得无需大规模边界框标注即可有效自监督学习语言引导的跟踪表示。在视觉语言跟踪基准上的大量实验表明，\tracker超越了最先进的自监督方法。
英文原文摘要 1530 字符
How to achieve vision-language (VL) tracking using natural language descriptions from a video sequence without relying on any bounding-box ground truth? In this work, we achieve this goal by tackling self-supervised VL tracking, which aims to evaluate tracking capabilities guided by natural language descriptions. We introduce \tracker, a novel self-supervised VL tracker that is capable of tracking any referred object by a language description. Unlike traditional methods that equally fuse all language and visual tokens, we propose an efficient Dynamic Token Aggregation Module, which treats each visual token unequally. The module consists of three main steps: i) Based on an anchor token, it selects multiple important target tokens from the template frame. ii) The selected target tokens are merged according to their attention scores and aggregated into the language tokens, thereby eliminating redundant visual token noise and enhancing semantic alignment. iii) Finally, the fused language tokens serve as guiding signals to extract potential target tokens from the search frame and propagate them to subsequent frames, enhancing temporal prompts and encouraging the tracker to autonomously learn instance tracking from unlabeled videos. This new modeling approach enables the effective self-supervised learning of language-guided tracking representations without the need for large-scale bounding box annotations. Extensive experiments on VL tracking benchmarks show that \tracker surpasses SOTA self-supervised methods.