PyraTok：面向视频理解与生成的语言对齐金字塔分词器
PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation
PyraTok 把视频 VAE 的编码器特征在多个深度上用共享大码本做二值量化，并联合多尺度文本引导量化与自回归目标训练，可直接替换现有视频生成/理解管线中的离散分词器，提升跨模态对齐与零样本下游迁移。
原文依据 a novel Language aligned Pyramidal Quantization (LaPQ) module that discretizes encoder features at several depths using a shared large binary codebook
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论离散视频变分自编码器是当代文本到视频生成与视频理解系统的基础，然而现有分词器通常只在单一尺度上学习视觉码本，词表规模有限且语言监督浅薄，导致跨模态对齐与零样本迁移效果不佳。本文提出 PyraTok，一种语言对齐的金字塔分词器，在多个时空分辨率上学习语义结构化的离散潜变量。PyraTok 建立在预训练视频变分自编码器之上，并引入一种新的语言对齐金字塔量化（LaPQ）模块，该模块利用共享的大型二值码本在多个深度上对编码器特征进行离散化，从而得到紧凑而富有表现力的视频词元序列。为使视觉词元与语言紧密耦合，PyraTok 联合优化多尺度文本引导量化以及词元层级上的全局自回归目标。在十项基准上，PyraTok 取得了最优的视频重建效果，稳定提升文本到视频生成质量，并在视频实例分割、时序动作定位与视频理解上刷新零样本最优性能，且可稳健扩展至 4K/8K 分辨率。
英文原文摘要 1179 字符
Discrete video VAEs underpin modern text-to-video generation and video understanding systems, yet existing tokenizers typically learn visual codebooks at a single scale with limited vocabularies and shallow language supervision, leading to poor cross-modal alignment and zero-shot transfer. We introduce PyraTok, a language-aligned pyramidal tokenizer that learns semantically structured discrete latents across multiple spatial-temporal resolutions. PyraTok builds on a pretrained video VAE and a novel Language aligned Pyramidal Quantization (LaPQ) module that discretizes encoder features at several depths using a shared large binary codebook, yielding compact yet expressive video token sequences. To tightly couple visual tokens with language, PyraTok jointly optimizes multi-scale text-guided quantization and a global autoregressive objective over the token hierarchy. Across ten benchmarks, PyraTok delivers state-of-the-art (SOTA) video reconstruction, consistently improves text-to-video quality, and sets new SOTA zero-shot performance on video instance segmentation, temporal action localization, and video understanding, scaling robustly to up to 4K/8K resolutions.