面向高效视频-语言预训练的簇级时空掩码
Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining
提出按帧内语义聚类、每簇保留时序密度最高词元的掩码策略，可直接迁移到任何视频-语言或视频自监督预训练流程中，在极高掩码率下同时缓解视觉信息丢失与帧间信息泄漏，并配合视频-文本相关性重建目标提升跨模态语义对齐。
原文依据 ClusterSTM first performs intra-frame clustering to partition visual tokens into multiple semantically independent clusters
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大规模视频-语言预训练使模型在多模态任务上具备强泛化能力，但往往带来难以承受的计算开销。尽管近期掩码视觉建模的进展有助于缓解这一问题，它们仍受两个根本性限制：高掩码率下严重的视觉信息丢失，以及帧间相关性导致的时序信息泄漏。为解决这些挑战，我们提出 ClusterSTM，一种用于高效视频-语言预训练的簇级时空掩码策略。ClusterSTM 首先执行帧内聚类，将视觉词元划分为多个语义独立的簇，然后进行簇级掩码，在每个簇内保留时序密度最高的词元。我们的掩码策略确保被保留的词元既能捕捉整体视频内容，又表现出强时序相关性。此外，我们引入视频-文本相关性重建目标，在传统视觉重建之外对齐高层多模态语义。在多个基准上的大量实验表明，ClusterSTM 在视频-文本检索、视频问答和视频描述生成任务上取得更优性能，在高效视频-语言模型中确立了新的最优水平。
英文原文摘要 1327 字符
Large-scale video-language pretraining enables strong generalization across multimodal tasks but often incurs prohibitive computational costs. Although recent advances in masked visual modeling help mitigate this issue, they still suffer from two fundamental limitations: severe visual information loss under high masking ratios and temporal information leakage caused by inter-frame correlations. To address these challenges, we propose ClusterSTM, a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining. ClusterSTM first performs intra-frame clustering to partition visual tokens into multiple semantically independent clusters, then conducts cluster-wise masking by retaining the token with the highest temporal density within each cluster. Our masking strategy ensure that the retained tokens capture holistic video content while exhibit strong temporal correlation. Additionally, we introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction. Extensive experiments across multiple benchmarks demonstrate that ClusterSTM achieves superior performance on video-text retrieval, video question answering, and video captioning tasks, establishing a new state-of-the-art among efficient video-language models.