Harmony：通过跨任务协同实现音视频生成和谐化
Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy
提出跨任务协同训练与SyncCFG，可迁移到视频生成中增强跨模态对齐；全局-局部解耦交互模块为时序细粒度对齐提供高效注意力设计思路。
原文依据 we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论同步音视频内容的合成是生成式人工智能中的一个关键挑战，开源模型在稳健的音视频对齐方面面临困难。我们的分析表明，这一问题根源于联合扩散过程的三个基本挑战：（1）对应漂移，即同时演化的噪声潜变量阻碍了对齐的稳定学习；（2）低效的全局注意力机制无法捕捉细粒度的时序线索；（3）传统无分类器引导（CFG）的模态内偏差，它增强了条件性但未增强跨模态同步。为克服这些挑战，我们提出了Harmony，一个从机制上强制音视频同步的新框架。我们首先提出跨任务协同训练范式，通过利用音频驱动视频生成和视频驱动音频生成任务中的强监督信号来缓解漂移。然后，我们设计了一个全局-局部解耦交互模块，用于高效且精确的时序风格对齐。最后，我们提出了一种新的同步增强CFG（SyncCFG），在推理过程中显式地分离并放大对齐信号。大量实验表明，Harmony建立了新的最先进水平，在生成保真度以及关键的细粒度音视频同步方面显著优于现有方法。
英文原文摘要 1410 字符
The synthesis of synchronized audio-visual content is a key challenge in generative AI, with open-source models facing challenges in robust audio-video alignment. Our analysis reveals that this issue is rooted in three fundamental challenges of the joint diffusion process: (1) Correspondence Drift, where concurrently evolving noisy latents impede stable learning of alignment; (2) inefficient global attention mechanisms that fail to capture fine-grained temporal cues; and (3) the intra-modal bias of conventional Classifier-Free Guidance (CFG), which enhances conditionality but not cross-modal synchronization.To overcome these challenges, we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization. We first propose a Cross-Task Synergy training paradigm to mitigate drift by leveraging strong supervisory signals from audio-driven video and video-driven audio generation tasks. Then, we design a Global-Local Decoupled Interaction Module for efficient and precise temporal-style alignment. Finally, we present a novel Synchronization-Enhanced CFG (SyncCFG) that explicitly isolates and amplifies the alignment signal during inference. Extensive experiments demonstrate that Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization.