Omni2Sound：迈向统一的视频-文本到音频生成
Omni2Sound: Towards Unified Video-Text-to-Audio Generation
提出三阶段多任务渐进训练调度，将视频到音频与文本到音频的跨任务竞争转化为联合优化，并缓解VT2A任务中的模态偏置；该调度策略可迁移到其他多模态条件扩散生成任务（如视频配音、音效生成）中平衡异构条件。
原文依据 we design a three-stage multi-task progressive training schedule that converts cross-task competition into joint optimization
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论训练一个统一模型来整合视频到音频（V2A）、文本到音频（T2A）以及联合视频-文本到音频（VT2A）生成，能带来显著的应用灵活性，但面临两个尚未探索的基础性挑战：（1）缺乏高质量且视频-音频-文本（V-A-T）紧密对齐的音频描述，导致多模态条件之间出现严重的语义冲突；（2）跨任务与任务内竞争，表现为V2A与T2A性能之间不利的权衡，以及VT2A任务中的模态偏置。首先，为解决数据稀缺问题，我们引入了SoundAtlas，一个大规模数据集（47万对），其质量显著优于现有基准，甚至超越人类专家。该数据集由一种新颖的智能体流水线驱动，集成了视觉到语言压缩以缓解多模态大语言模型（MLLM）的视觉偏置、初级-高级智能体交接机制以实现五倍成本降低，以及严格的事后过滤以确保保真度。因此，SoundAtlas提供了语义丰富、时间细节详尽且V-A-T紧密对齐的描述。其次，我们提出了Omni2Sound，一个支持灵活输入模态的统一VT2A扩散模型。为解决固有的跨任务与任务内竞争，我们设计了一种三阶段多任务渐进训练调度，将跨任务竞争转化为联合优化，并缓解VT2A任务中的模态偏置，同时保持视听对齐与画外音频生成的忠实度。最后，我们构建了VGGSound-Omni，一个用于统一评估的综合基准，包含具有挑战性的画外音轨。在标准DiT骨干网络下，Omni2Sound在单一模型内实现了全部三个任务的统一最先进性能，并在具有异构输入条件的基准上展现出强大的泛化能力。
英文原文摘要 1788 字符
Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) generation offers significant application flexibility, yet faces two unexplored foundational challenges: (1) the scarcity of high-quality audio captions with tight V-A-T alignment, leading to severe semantic conflict between multimodal conditions, and (2) cross-task and intra-task competition, manifesting as an adverse V2A-T2A performance trade-off and modality bias in the VT2A task. First, to address data scarcity, we introduce SoundAtlas, a large-scale dataset (470k pairs) that significantly outperforms existing benchmarks and even human experts in quality. Powered by a novel agentic pipeline, it integrates Vision-to-Language Compression to mitigate visual bias of MLLMs, a Junior-Senior Agent Hand-off for a five times cost reduction, and rigorous Post-hoc Filtering to ensure fidelity. Consequently, SoundAtlas delivers semantically rich and temporally detailed captions with tight V-A-T alignment. Second, we propose Omni2Sound, a unified VT2A diffusion model supporting flexible input modalities. To resolve the inherent cross-task and intra-task competition, we design a three-stage multi-task progressive training schedule that converts cross-task competition into joint optimization and mitigates modality bias in the VT2A task, maintaining both audio-visual alignment and off-screen audio generation faithfulness. Finally, we construct VGGSound-Omni, a comprehensive benchmark for unified evaluation, including challenging off-screen tracks. With a standard DiT backbone, Omni2Sound achieves unified SOTA performance across all three tasks within a single model, demonstrating strong generalization across benchmarks with heterogeneous input conditions.