M4V：面向高效文本到视频生成的多模态 Mamba
M4V: Multimodal Mamba for Efficient Text-to-Video Generation
用双向词元重排加视觉寄存器把 Mamba 的线性复杂度序列建模接入扩散式文生视频主干，在 768x1280 分辨率下比注意力方案省 45% 浮点运算量，可作为高分辨率视频生成中替代全局注意力的可复用模块。
原文依据 the MM-DiM blocks in M4V reduce FLOPs by 45% compared with the attention-based alternative
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论文本到视频生成显著丰富了内容创作，并有望演化为强大的世界模拟器。然而，对庞大时空空间的建模在计算上仍然代价高昂，尤其是在采用 Transformer 时，其序列处理具有二次复杂度，从而限制了实际应用。近期线性时间序列建模的进展，特别是 Mamba 架构，提供了更高效的替代方案。但其朴素设计限制了其直接应用于多模态与时空视频生成任务。为应对这些挑战，我们提出 M4V，一个用于高效文本到视频生成的多模态 Mamba 框架。具体而言，该框架内设计了一个多模态扩散 Mamba（MM-DiM）模块，以实现多模态信息的无缝集成与时空建模。详细来说，我们引入了一种新颖的多模态词元重组合设计，它通过简单的词元排布采用双向方案进行多模态信息集成，并结合视觉寄存器以增强时空一致性。因此，在生成 768x1280 分辨率视频时，M4V 中的 MM-DiM 模块相比基于注意力的替代方案将浮点运算量降低了 45%。此外，本工作探索了若干训练策略，以更好地理解仅使用公开数据集训练文本到视频模型的过程。在文本到视频基准上的大量实验表明，M4V 能够生成高质量视频，同时显著降低计算成本。代码将公开。
英文原文摘要 1666 字符
Text-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators. However, modeling the vast spatiotemporal space remains computationally demanding, particularly when employing Transformers, which incur quadratic complexity in sequence processing and thus limit practical applications. Recent advancements in linear-time sequence modeling, particularly the Mamba architecture, offer a more efficient alternative. Nevertheless, its plain design limits its direct applicability to multimodal and spatiotemporal video generation tasks. To address these challenges, we introduce M4V, a multimodal Mamba framework for efficient text-to-video generation. Specifically, a MultiModal diffusion Mamba (MM-DiM) block is designed within the framework to enable seamless integration of multimodal information and spatiotemporal modeling. In detail, we introduce a novel multimodal token re-composition design, which employs a bidirectional scheme for multimodal information integration through simple token arrangement, along with visual registers to enhance spatial-temporal consistency. As a result, the MM-DiM blocks in M4V reduce FLOPs by 45% compared with the attention-based alternative when generating videos at 768x1280 resolution. Additionally, several training strategies are explored in this work to provide a better understanding of training text-to-video models using only publicly available datasets. Extensive experiments on text-to-video benchmarks demonstrate M4V's ability to produce high-quality videos while significantly lowering computational costs. Code will be made publicly available.