生成模型与扩散 · CVPR2026 ·2026.06.03

一帧只需一个词元:用 Delta 词元进行高效生成式世界建模

A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens

Tommie Kerssies、Gabriele Berton、Ju He、Qihang Yu、Wufei Ma、Daan de Geus 等 8 人

与你有关的部分

把相邻帧的 VFM 特征差压缩成单个连续 delta 词元,将视频从三维时空表示降为一维时序序列,这一分词思路可直接复用于视频预测、特征空间生成建模与高效多假设采样。

原文依据 a tokenizer that encodes the feature difference between consecutive frames into a single continuous "delta" token
可迁移到 视频预测世界模型视频生成稠密预测

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

预测多样化的未来状态是视频世界建模的核心挑战。判别式世界模型产生确定性预测,会隐式地对可能的未来取平均,而现有的生成式世界模型计算开销仍然很高。近期工作表明,在视觉基础模型(VFM)的特征空间中预测未来,而非在针对像素重建优化的潜空间中预测,所需的 world model 参数量显著更少。然而,这类方法大多仍是判别式的。本文提出 DeltaWorld,一种基于 VFM 的生成式世界模型,能够高效生成多样且合理的未来。DeltaWorld 的核心是 DeltaTok,一种分词器,它将相邻帧之间的特征差异编码为单个连续的“delta”词元,从而把视频从三维时空表示降为一维时序序列。例如,在 512x512 帧下可实现 1,024 倍的词元缩减。delta 词元使高效且有效的多假设训练成为可能:并行生成大量多样的未来,只对其中最优者进行监督。在推理时,这使得单次前向传播即可产生多样化预测。在稠密预测任务上的实验表明,DeltaWorld 预测的未来更贴近真实世界结果,同时相比现有生成式世界模型参数量减少 35 倍以上、浮点运算量减少 2,000 倍。项目主页:https://deltatok.github.io。

英文原文摘要 1521 字符

Anticipating diverse future states is a central challenge in video world modeling. Discriminative world models produce deterministic predictions that implicitly average over possible futures, while existing generative world models remain computationally expensive. Recent work demonstrates that predicting the future in the feature space of a vision foundation model (VFM), rather than a latent space optimized for pixel reconstruction, requires significantly fewer world model parameters. However, most such approaches remain discriminative. In this work, we introduce DeltaWorld, a generative VFM-based world model that efficiently generates diverse plausible futures. At the core of DeltaWorld is DeltaTok, a tokenizer that encodes the feature difference between consecutive frames into a single continuous "delta" token, reducing video from a three-dimensional spatio-temporal representation to a one-dimensional temporal sequence. For example, this yields a 1,024x token reduction with 512x512 frames. Delta tokens enable efficient and effective multi-hypothesis training, where many diverse futures are generated in parallel and only the best is supervised. At inference, this leads to diverse predictions in a single forward pass. Experiments on dense forecasting tasks demonstrate that DeltaWorld forecasts futures that more closely align with real-world outcomes, while having over 35x fewer parameters and using 2,000x fewer FLOPs than existing generative world models. Project page: https://deltatok.github.io.

标签

会议 CVPR2026
页码 27978-27988
Bibkey
来源 CVF Open Access