通过教师引导的双路径音视频表征学习实现语义噪声消减
Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning
把对比对齐与掩码重建拆成两条独立优化路径、各自使用不同的可见 patch 模式，并用教师模型组织对比分支的可见 token——这套解耦思路可直接迁移到图像/视频的自监督预训练中，缓解多目标联合优化时的语义噪声与梯度干扰。
原文依据 decouples reconstruction and alignment into separate optimization paths
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期音视频表征学习的进展显示了将对比对齐与掩码重建相结合的价值。然而，在单次前向传播中联合优化这两个目标，会迫使对比分支依赖于为重建而设计的随机可见 patch，而非服务于跨模态对齐，从而引入语义噪声与优化干扰。我们提出 TG-DP，一个教师引导的双路径框架，将重建与对齐解耦为相互独立的优化路径。通过分离两个分支的掩码机制，TG-DP 使对比路径能够采用更适合跨模态对齐的可见性模式。教师模型进一步为该分支中可见 token 的组织提供辅助引导，有助于减少干扰并稳定跨模态表征学习。TG-DP 在零样本检索上取得了最优性能。在 AudioSet 上，视频到音频检索的 R@1 从 35.2% 提升至 37.4%，音频到视频检索的 R@1 从 27.9% 提升至 37.1%。所学表征也保持语义鲁棒性，在 AS20K 与 VGGSound 上取得最优的线性探测性能。综上，我们的结果表明，解耦多模态目标并在对比路径中引入教师引导的结构，为改进大规模音视频预训练提供了一个有效框架。
英文原文摘要 1433 字符
Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2% to 37.4% for video-to-audio retrieval and from 27.9% to 37.1% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining.