利用视觉线索的电影音频源分离
Cinematic Audio Source Separation Using Visual Cues
把视觉线索作为条件注入条件流匹配生成过程，为视觉研究者提供了一个可复用的范式：用视频帧（人脸、场景）作为条件信号去引导音频生成/分离，可迁移到其他以视觉为条件的生成任务（如视觉引导的语音增强、音效合成）。
原文依据 Our method formulates CASS as a conditional generative modeling problem using conditional flow matching, enabling multimodal audio source separation.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论电影音频源分离（CASS）旨在将混合的电影音频分解为语音、音乐和音效，从而支持配音和重制等应用。现有的 CASS 方法仅基于音频，忽略了电影固有的音视频特性，即声音往往与视觉线索对齐。我们提出了首个音视频 CASS 框架（AV-CASS），利用视觉上下文提升分离质量。我们的方法将 CASS 建模为条件生成建模问题，采用条件流匹配，从而实现多模态音频源分离。为解决缺乏带独立音轨的电影数据集的问题，我们提出了一套训练数据合成流程，将野外音频流与视频流配对（例如用面部视频对应语音、用场景视频对应音效），并为这一双流设置设计了专用的视觉编码器。我们的模型完全在合成数据上训练，却能有效泛化到真实电影内容，并在合成、真实以及纯音频 CASS 基准上取得强劲性能。代码与演示见 https://cass-flowmatching.github.io。
英文原文摘要 1126 字符
Cinematic Audio Source Separation (CASS) aims to decompose mixed film audio into speech, music, and sound effects, enabling applications like dubbing and remastering. Existing CASS approaches are audio-only, overlooking the inherent audio-visual nature of films, where sounds often align with visual cues. We present the first framework for audio-visual CASS (AV-CASS), leveraging visual context to enhance separation quality. Our method formulates CASS as a conditional generative modeling problem using conditional flow matching, enabling multimodal audio source separation. To address the lack of cinematic datasets with isolated sound tracks, we introduce a training data synthesis pipeline that pairs in-the-wild audio and video streams (e.g., facial videos for speech, scene videos for effects) and design a dedicated visual encoder for this dual-stream setup. Trained entirely on synthetic data, our model generalizes effectively to real-world cinematic content and achieves strong performance on synthetic, real-world, and audio-only CASS benchmarks. Code and demo are available at https://cass-flowmatching.github.io.