自举你的音视频代理：面向音视频分割的自适应对比与原型学习
Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation
论文提出的自监督音频增强模块通过对比学习对齐音频与梯度阻断的视觉嵌入，以及动态原型约束模块利用动量更新原型增强像素级类别感知，这两者均可作为跨模态对齐与分割的通用技术手段，尤其适用于需要抑制融合噪声的多模态分割任务。
原文依据 we propose a novel adaptive contrastive and prototype learning framework, BYOAVP, for AVS
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论音视频分割（AVS）旨在利用音视频对应线索准确分割视频帧中的发声物体。然而，由于单一模态内固有的语义不完整性以及音频与视觉表示之间的语义鸿沟，该任务仍具挑战性。传统的特征融合解码方法难以有效抑制融合噪声，而近期引入数据依赖先验的方法往往增加了建模音视频关联的复杂度，导致跨域泛化能力较差。为解决这些问题，我们提出了一种新颖的自适应对比与原型学习框架BYOAVP用于AVS。具体而言，我们设计了一个自监督音频增强（SSAE）模块，引入对比学习以自适应地将音频表示与梯度阻断的视觉嵌入对齐，从而缩小模态间的语义鸿沟。此外，我们开发了动态原型约束（DPC）模块，通过基于动量的原型更新来细化像素级类别感知，同时通过跨模态交互增强发声区域的定位。大量实验表明，我们的方法在两个AVS基准和六个子任务上取得了最先进的性能，展现出强大的鲁棒性和泛化能力。
英文原文摘要 1344 字符
Audio-Visual Segmentation (AVS) aims to accurately segment sounding objects in video frames by leveraging audio-visual correspondence cues. However, it remains challenging due to the intrinsic semantic incompleteness within a single modality and the semantic gap between audio and visual representations. Traditional feature-fusion decoding approaches struggle to suppress fusion noise effectively, while recent methods that incorporate data-dependent priors often increase the complexity of modeling audio-visual correlations, leading to poor cross-domain generalization. To address these issues, we propose a novel adaptive contrastive and prototype learning framework, BYOAVP, for AVS. Specifically, we design a Self-Supervised Audio Enhancement (SSAE) module that introduces contrastive learning to adaptively align audio representations with gradient-blocked visual embeddings, thus narrowing the semantic gap between modalities. Furthermore, a Dynamic Prototype Constraint (DPC) module is developed to refine pixel-wise category perception via momentum-based prototype updating, while enhancing localization of sounding regions through cross-modal interaction. Extensive experiments demonstrate that our method achieves SOTA performance across two AVS benchmarks and six sub-tasks, exhibiting strong robustness and generalization ability.