OmniSonic：面向视频与文本的通用整体音频生成
OmniSonic: Towards Universal and Holistic Audio Generation from Video and Text
TriAttn-DiT 用三路交叉注意力分别处理屏内环境声、屏外环境声与语音条件，并以 MoE 门控自适应加权，这种「多条件分支 + 门控融合」的结构可直接迁移到视频到音频、视频到音乐等多模态条件生成任务中；UniHAGen-Bench 也提供了评估视频条件音频生成的评测协议。
原文依据 a TriAttn-DiT architecture that performs three cross-attention operations to process on-screen environmental sound, off-screen environmental sound, and speech conditions simultaneously
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文提出通用整体音频生成（Universal Holistic Audio Generation, UniHAGen）任务，旨在合成包含屏内与屏外声音的综合听觉场景，覆盖环境事件、乐器、人声等多种领域。此前的视频条件音频生成模型通常只生成与可见发声事件对应的屏内环境声，忽略了屏外听觉事件。近期的整体式文本-视频联合到音频生成模型虽能产生同时包含屏内与屏外声音的听觉场景，但仅限于非语音声音，缺乏生成或整合人声的能力。为克服这些局限，我们提出 OmniSonic，一个同时以视频和文本为条件的、基于流匹配的扩散框架。它采用 TriAttn-DiT 架构，执行三次交叉注意力操作以同时处理屏内环境声、屏外环境声与语音条件，并借助混合专家（MoE）门控机制在生成过程中自适应地平衡三者的贡献。此外，我们构建了 UniHAGen-Bench，这是一个包含一千多个样本、覆盖三种代表性屏内/屏外语音-环境场景的新基准。大量实验表明，OmniSonic 在客观指标与人工评测上均持续优于最优方法，为通用整体音频生成建立了强有力的基线。项目主页：https://weiguopian.github.io/OmniSonic_webpage/
英文原文摘要 1580 字符
In this paper, we propose Universal Holistic Audio Generation (UniHAGen), a task for synthesizing comprehensive auditory scenes that include both on-screen and off-screen sounds across diverse domains (e.g., ambient events, musical instruments, and human speech). Prior video-conditioned audio generation models typically focus on producing on-screen environmental sounds that correspond to visible sounding events, neglecting off-screen auditory events. While recent holistic joint text-video-to-audio generation models aim to produce auditory scenes with both on- and off-screen sound but they are limited to non-speech sounds, lacking the ability to generate or integrate human speech. To overcome these limitations, we introduce OmniSonic, a flow-matching-based diffusion framework jointly conditioned on video and text. It features a TriAttn-DiT architecture that performs three cross-attention operations to process on-screen environmental sound, off-screen environmental sound, and speech conditions simultaneously, with a Mixture-of-Experts (MoE) gating mechanism that adaptively balances their contributions during generation. Furthermore, we construct UniHAGen-Bench, a new benchmark with over one thousand samples covering three representative on/off-screen speech-environment scenarios. Extensive experiments show that OmniSonic consistently outperforms state-of-the-art approaches on both objective metrics and human evaluations, establishing a strong baseline for universal and holistic audio generation. Project page: https://weiguopian.github.io/OmniSonic_webpage/