UniAVGen：基于非对称跨模态交互的统一音频与视频生成
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
提出非对称跨模态交互机制与模态感知无分类器引导，可复用于任何需要时空对齐的多模态生成任务（如视频配音、说话人脸生成），其双分支 DiT 架构与 Face-Aware 调制为视觉-音频联合建模提供了新思路。
原文依据 UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论由于缺乏有效的跨模态建模，现有开源音视频生成方法往往表现出唇形同步不佳和语义一致性不足的问题。为缓解这些缺陷，我们提出 UniAVGen，一个面向以人为中心的音视频联合生成的统一框架。UniAVGen 基于双分支联合合成架构，包含两个并行的扩散 Transformer（DiT），以构建一个紧密的跨模态潜变量空间。其核心在于非对称跨模态交互机制，该机制实现了双向、时间对齐的交叉注意力，从而确保精确的时空同步和语义一致性。此外，这种跨模态交互通过人脸感知调制（FAM）模块得到增强，该模块在交互过程中动态地优先处理显著区域。为提高推理时的生成保真度，我们还引入了模态感知无分类器引导（MA-CFG），这是一种显式放大跨模态相关信号的新策略。值得注意的是，UniAVGen 的鲁棒联合合成设计能够将关键音视频任务无缝统一到单一模型中。此外，我们证明联合多任务训练可以进一步提升联合生成的性能。综合实验验证表明，在训练样本远少的情况下（130 万 vs. 3010 万），UniAVGen 在音视频同步、音色一致性和情感一致性方面均展现出整体优势。
英文原文摘要 1517 字符
Due to the lack of effective cross-modal modeling, existing open-source audio-video generation methods often exhibit compromised lip synchronization and insufficient semantic consistency. To mitigate these drawbacks, we propose UniAVGen, a unified framework for human-centric joint audio and video generation. UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which enables bidirectional, temporally aligned cross-attention, thus ensuring precise spatiotemporal synchronization and semantic consistency. Furthermore, this cross-modal interaction is augmented by a Face-Aware Modulation (FAM) module, which dynamically prioritizes salient regions in the interaction process. To enhance generative fidelity during inference, we additionally introduce Modality-Aware Classifier-Free Guidance (MA-CFG), a novel strategy that explicitly amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint synthesis design enables the seamless unification of pivotal audio-visual tasks within a single model. Furthermore, we demonstrate that joint multi-task training can further boost the performance of joint generation. Comprehensive experiments validate that, with far fewer training samples (1.3M vs. 30.1M), UniAVGen delivers overall advantages in audio-video synchronization, timbre consistency, and emotion consistency.