UnityVideo：面向世界感知视频生成的统一多模态多任务学习
UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation
将分割掩码、人体骨架、DensePose、光流、深度图等多种异构模态作为统一条件输入视频扩散模型，并用动态加噪统一不同训练范式、用模态切换器+上下文学习器实现模块化参数共享，为视觉研究者提供了一套可复用的多模态条件注入与跨模态泛化方案。
原文依据 dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期的视频生成模型展现出令人印象深刻的合成能力，但仍受限于单模态条件，制约了其对世界的整体理解。这源于跨模态交互不足以及模态多样性有限，难以全面表征世界知识。为解决这些局限，我们提出 UnityVideo，一个面向世界感知视频生成的统一框架，在多种模态——分割掩码、人体骨架、DensePose、光流和深度图——以及多种训练范式上进行联合学习。该方法包含两个核心组件：(1) 动态加噪，用于统一异构的训练范式；(2) 模态切换器与上下文学习器，通过模块化参数与上下文学习实现统一处理。我们贡献了一个包含 130 万样本的大规模统一数据集。通过联合优化，UnityVideo 加速了收敛，并显著增强了对未见数据的零样本泛化能力。我们证明 UnityVideo 实现了更优的视频质量、一致性，以及与物理世界约束更好的对齐。
英文原文摘要 1133 字符
Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation.To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities--segmentation masks, human skeletons, DensePose, optical flow, and depth maps--and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints.