TrackMAE：通过轨迹掩码与预测进行视频表征学习
TrackMAE: Video Representation Learning via Track Mask and Predict
把现成点跟踪器产生的稀疏轨迹当作显式重建目标与掩码先验，为视频自监督预训练注入细粒度运动信号；做视频表征学习或运动中心任务时，可直接复用其运动感知管状掩码与轨迹监督设计。
原文依据 we propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论掩码视频建模（MVM）已成为一种简单且可扩展的自监督预训练范式，但它仅隐式地编码运动信息，限制了所学表征对时序动态的编码能力。因此，这类模型在需要细粒度运动感知的运动中心任务上表现不佳。为解决这一问题，我们提出 TrackMAE，一种简单地将运动信息用作重建信号的掩码视频建模范式。在 TrackMAE 中，我们使用现成的点跟踪器对输入视频中的点进行稀疏跟踪，生成运动轨迹。此外，我们利用提取的轨迹以运动感知的掩码策略改进随机管状掩码。通过以运动目标的形式提供互补的监督信号，我们在像素空间与特征语义重建空间中同时增强所学习的视频表征。我们在六个数据集上跨多种下游设置进行评估，发现 TrackMAE 一致优于最优的视频自监督基线，从而学习到更具判别性与泛化性的表征。代码见 https://github.com/rvandeghen/TrackMAE
英文原文摘要 1184 字符
Masked video modeling (MVM) has emerged as a simple and scalable self-supervised pretraining paradigm, but only encodes motion information implicitly, limiting the encoding of temporal dynamics in the learned representations. As a result, such models struggle on motion-centric tasks that require fine-grained motion awareness. To address this, we propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal. In TrackMAE, we use an off-the-shelf point tracker to sparsely track points in the input videos generating motion trajectories. Furthermore, we exploit the extracted trajectories to improve the random tube masking with a motion-aware masking strategy. We enhance video representations learned in both pixel and feature semantic reconstruction space by providing a complementary supervision signal in the form of motion targets. We evaluate on six datasets across diverse downstream settings and find that TrackMAE consistently outperforms the state-of-the-art video SSL baselines, therefore learning more discriminative and generalizable representations Code available at https://github.com/rvandeghen/TrackMAE