以大规模多模态对应学习推进音视频感知前沿
Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
把视频-文本对比学习扩展到音视频-文本三模态联合嵌入，并提供用大模型为亿级音视频对合成高质量描述的数据引擎；做视频理解或多模态检索时，可直接复用其十组成对对比目标与跨模态/描述类型配对的扩展策略来提升零样本迁移。
原文依据 We exploit ten pairwise contrastive objectives, showing that scaling cross-modality and caption-type pairs strengthens alignment and improves zero-shot performance.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出 Perception Encoder-Audiovisual（PE-AV），这是一族用于音频与视频理解的新型编码器，采用规模化对比学习训练。在 PE 的基础上，PE-AV 做出若干关键贡献，将表征扩展到音频，并原生支持音频-视频、音频-文本与视频-文本模态之间的联合嵌入。PE-AV 的统一跨模态嵌入使语音检索等新任务成为可能，并在标准音频与视频基准上刷新了当前最优结果。我们通过构建一个强大的音视频数据引擎来实现这一点：该引擎为约一亿（O(100M)）个音视频对合成高质量描述，从而提供跨模态一致的大规模监督。我们的音频数据涵盖语音、音乐与一般音效，避免了此前工作中常见的单一领域局限。我们利用十组成对对比目标，表明扩展跨模态与描述类型的配对能够增强对齐并提升零样本性能。模型与代码已公开。
英文原文摘要 1010 字符
We introduce Perception Encoder-Audiovisual, PE-AV, a new family of encoders for audio and video understanding trained with scaled contrastive learning. Building on PE, PE-AV makes several key contributions to extend representations to audio, and natively support joint embeddings across audio-video, audio-text, and video-text modalities. PE-AV's unified cross-modal embeddings enable novel tasks such as speech retrieval, and set a new state of the art across standard audio and video benchmarks. We unlock this by building a strong audiovisual data engine that synthesizes high-quality captions for O(100M) audio-video pairs, enabling large-scale supervision consistent across modalities. Our audio data includes speech, music, and general sound effects--avoiding single-domain limitations common in prior work. We exploit ten pairwise contrastive objectives, showing that scaling cross-modality and caption-type pairs strengthens alignment and improves zero-shot performance. Models and code are available.