从静态到动态：探索自监督图像到视频表征迁移学习
From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning
提出在冻结的图像预训练编码器上只加一个轻量投影层，用时序循环一致性与语义可分性约束做自监督迁移，为视觉研究者提供了一种低成本把图像骨干转成视频表征的通用方案，可直接套用到自有图像预训练模型上。
原文依据 applies a lightweight projection layer on top of the frozen image-pretrained encoder
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期研究通过将图像预训练模型迁移到视频任务，在视频表征学习上取得了显著进展，但通常依赖复杂的时序模块与视频微调。然而，微调重型模块可能损害视频间语义可分性，即区分不同视频中物体的基本能力；而减少可调参数又会妨碍视频内时序一致性，而该一致性是同一视频中同一物体获得稳定表征所必需的。这一困境表明，在图像到视频迁移过程中，视频内时序一致性与视频间语义可分性之间存在潜在的权衡。为此，我们提出一致性—可分性权衡迁移学习（Co-Settle）框架，在冻结的图像预训练编码器之上施加一个轻量投影层，以时序循环一致性目标和语义可分性约束来调整表征空间。我们进一步给出理论支持，证明在适当条件下优化后的投影能在上述两种性质之间取得更好的权衡。在八个图像预训练模型上的实验表明，仅需五个轮次的自监督训练，即可在多个层次的视频任务上取得一致的提升。代码已发布于 https://github.com/yafeng19/Co-Settle。
英文原文摘要 1376 字符
Recent studies have made notable progress in video representation learning by transferring image-pretrained models to video tasks, typically with complex temporal modules and video fine-tuning. However, fine-tuning heavy modules may compromise inter-video semantic separability, i.e., the essential ability to distinguish objects across videos. While reducing the tunable parameters hinders their intra-video temporal consistency, which is required for stable representations of the same object within a video. This dilemma indicates a potential trade-off between the intra-video temporal consistency and inter-video semantic separability during image-to-video transfer. To this end, we propose the Consistency-Separability Trade-off Transfer Learning (Co-Settle) framework, which applies a lightweight projection layer on top of the frozen image-pretrained encoder to adjust representation space with a temporal cycle consistency objective and a semantic separability constraint. We further provide a theoretical support showing that the optimized projection yields a better trade-off between the two properties under appropriate conditions. Experiments on eight image-pretrained models demonstrate consistent improvements across multiple levels of video tasks with only five epochs of self-supervised training. The code is available at https://github.com/yafeng19/Co-Settle.