SOUPLE：以可学习提示词上下文增强音视频定位与分割
SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts
用可学习的上下文词元替代 CLIP 中固定的文本提示词，并把视觉特征注入其中生成条件上下文，为视觉侧提供了一种可复用的提示词调优范式，可直接迁移到开放词表检测、指代分割等需要文本-视觉对齐的任务。
原文依据 we propose sound-aware prompt learning (SouPLe), which replaces fixed prompts with learnable context tokens
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大规模预训练图文模型展现出稳健的多模态表征能力，然而将对比语言-图像预训练（CLIP）应用于音视频定位仍然充满挑战。用音频嵌入词元（[V_A]）替换分类词元（[CLS]）难以捕捉语义线索，而提示词「a photo of a [V_A]」也无法在音频嵌入与上下文词元之间建立有意义的联系。为解决这些问题，我们提出声音感知提示词学习（SouPLe），以可学习的上下文词元替代固定提示词。这些词元融合视觉特征，为掩码解码器生成条件上下文，从而有效弥合音频与视觉输入之间的语义对应关系。在 VGGSound、SoundNet 与 AVSBench 上的实验证实，SouPLe 显著提升了定位与分割性能。
英文原文摘要 869 字符
Large-scale pre-trained image-text models exhibit robust multimodal representation, yet applying contrastive language-image pretraining (CLIP) to audio-visual localization remains challenging. Replacing the classification token ([CLS]) with an audio-embedded token ([V_A])struggles to capture semantic cues, and the prompt "a photo of a [V_A]" fails to establish meaningful connections between audio embeddings and context tokens. To address these issues, we propose sound-aware prompt learning (SouPLe), which replaces fixed prompts with learnable context tokens. These tokens incorporate visual features to generate conditional context for a mask decoder, effectively bridging semantic correspondence between audio and visual inputs. Experiments on VGGSound, SoundNet, and AVSBench confirm that SouPLe significantly improves localization and segmentation performance.