面向高效视频生成的双粒度记忆
Dual-Granularity Memory for Efficient Video Generation
针对循环式视频生成模型的分块处理导致时序不连续问题，提出上下文记忆（sink 列+边界缓冲）与潜变量记忆（跨段检索历史 latent）两种机制，可直接迁移到任何 chunked 自回归/循环视频生成器以提升长程一致性，且无需相机标注或帧重建。
原文依据 Latent Context-as-Memory (LCaM) extends memory across video segments by storing and retrieving historical latent embeddings
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于循环架构的视频生成相比基于注意力的 Transformer 具有显著的效率优势，尤其适用于长序列生成。然而，循环模型中的分块处理会造成时序不连续，损害长程一致性。我们引入两种互补的记忆机制，在不同粒度上应对这一挑战：（1）上下文记忆通过可学习的 sink 列与边界缓冲，在注意力块内维持持久的全局上下文，仅增加 15 万参数（低于 0.1% 的开销）；（2）潜变量上下文即记忆（LCaM）通过存储与检索历史潜变量嵌入，将记忆扩展到视频片段之间，在无需相机标注或帧重建的情况下实现跨片段一致性。将上述双记忆方法应用于广义时空传播网络（GSTPN）后，我们的方法推理速度比基于注意力的 Transformer 快 1.54 倍，同时在视觉质量指标上表现优异。我们的方法在仅有预提取潜变量嵌入可用的知识蒸馏场景中尤为有效。本工作为实际长视频生成展示了颇具吸引力的效率与质量权衡。
英文原文摘要 1234 字符
Video generation using recurrent architectures offers compelling efficiency advantages over attention-based transformers, particularly for long-sequence generation. However, chunked processing in recurrent models creates temporal discontinuities that harm long-range consistency. We introduce two complementary memory mechanisms to address this challenge at different granularities: (1) Context Memory maintains persistent global context within attention chunks through learnable sink columns and boundary buffers, adding only 150K parameters (\textless 0.1% overhead); (2) Latent Context-as-Memory (LCaM) extends memory across video segments by storing and retrieving historical latent embeddings, enabling cross-segment consistency without requiring camera annotations or frame reconstruction. Applied to Generalized Spatial-temporal Propagation Networks (GSTPN), our dual-memory approach achieves 1.54xfaster inference than attention-based transformers, while excelling in visual quality metrics. Our approach is particularly effective for knowledge distillation scenarios where only pre-extracted latent embeddings are available. This work demonstrates compelling efficiency-quality trade-offs for practical long video generation.