AdapTok：在一维潜空间中学习自适应且时序因果的视频分词
AdapTok: Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space
提出一种内容感知的视频分词器，通过块因果评分器和整数线性规划在总预算内动态分配每帧词元数，可直接用于视频生成模型的前端，提升词元效率与重建质量。
原文依据 an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出 AdapTok，一种自适应时序因果视频分词器，能够根据视频内容为不同帧灵活分配词元。AdapTok 配备了一种块级掩码策略，在训练时随机丢弃每个块尾部的词元，并配备了一个块因果评分器，用于预测视频帧在使用不同数量词元时的重建质量。在推理阶段，进一步提出了一种基于整数线性规划的自适应词元分配策略，根据预测分数调整词元使用量。这种设计允许在可控的总体预算下实现样本级、内容感知且时序动态的词元分配。在 UCF-101 和 Kinetics-600 上进行的大量视频重建与生成实验证明了我们方法的有效性。在没有额外图像数据的情况下，AdapTok 在不同词元预算下持续提升重建质量和生成性能，从而实现更具可扩展性和词元高效的生成式视频建模。
英文原文摘要 1014 字符
We propose AdapTok, an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content. AdapTok is equipped with a block-wise masking strategy that randomly drops tail tokens of each block during training, and a block causal scorer to predict the reconstruction quality of video frames using different numbers of tokens. During inference, an adaptive token allocation strategy based on integer linear programming is further proposed to adjust token usage given predicted scores. Such design allows for sample-wise, content-aware, and temporally dynamic token allocation under a controllable overall budget. Extensive experiments for video reconstruction and generation on UCF-101 and Kinetics-600 demonstrate the effectiveness of our approach. Without additional image data, AdapTok consistently improves reconstruction quality and generation performance under different token budgets, allowing for more scalable and token-efficient generative video modeling.