FoleyDesigner：面向电影片段的沉浸式立体声拟音生成与精确时空对齐
FoleyDesigner: Immersive Stereo Foley Generation with Precise Spatio-Temporal Alignment for Film Clips
论文将视频帧的时空线索作为条件输入潜变量扩散模型，实现与画面精确对齐的立体声拟音生成，其多智能体分析架构和时空条件注入方式可迁移到视频到音频生成、音视觉对齐等跨模态生成任务。
原文依据 It achieves spatio-temporal alignment through latent diffusion models trained on spatiotemporal cues extracted from video frames
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论拟音艺术在提升电影沉浸式听觉体验方面起着关键作用，然而人工制作时空对齐的音频仍然劳动密集。我们提出 FoleyDesigner，一个受专业拟音工作流程启发的新颖框架，集成了电影片段分析、时空可控的拟音生成以及专业混音能力。在技术上，FoleyDesigner 采用多智能体架构进行精确的时空分析。它通过基于从视频帧中提取的时空线索训练的潜变量扩散模型，结合大语言模型驱动的混合机制来模拟电影工业级后期制作实践，从而实现时空对齐。为解决电影领域缺乏高质量立体声拟音数据集的问题，我们引入了 FilmStereo，这是首个包含空间元数据、精确时间戳和八种常见拟音类别语义标注的专业立体声拟音数据集。在应用方面，该框架支持交互式用户控制，同时保持与专业流程的无缝集成，包括符合 ITU-R BS.775 标准的 5.1 声道杜比全景声系统，从而提供广泛的创作灵活性。大量实验表明，与现有基线相比，我们的方法实现了更优越的时空对齐，并且其集成在电影工业级工作流程中得到了验证。
英文原文摘要 1416 字符
Foley art plays a pivotal role in enhancing immersive auditory experiences in film, yet manual creation of spatio-temporal aligned audio remains labor-intensive. We propose FoleyDesigner, a novel framework inspired by professional Foley workflows, integrating film clip analysis, spatio-temporal controllable Foley generation, and professional mixing capabilities. Technically, FoleyDesigner employs a multi-agent architecture for precise spatio-temporal analysis. It achieves spatio-temporal alignment through latent diffusion models trained on spatiotemporal cues extracted from video frames, combined with large language model (LLM)-driven hybrid mechanisms that emulate film industry-grade post-production practices. To address the lack of high-quality stere Foley datasets in film, we introduce FilmStereo, the first professional stereo Foley dataset containing spatial metadata, precise timestamps, and semantic annotations for eight common Foley categories. For application, the framework supports interactive user control while maintaining seamless integration with professional pipelines, including 5.1-channel Dolby Atmos systems compliant with ITU-R BS.775 standards, thereby offering extensive creative flexibility. Extensive experiments demonstrate that our method achieves superior spatio-temporal alignment compared to existing baselines, with integration validated in film industrial-grade workflows.