SAVE：面向视频-文本检索的语音感知视频表征学习
SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval
为视频-文本检索引入语音分支和软对齐策略，视觉研究者可借鉴其音视频早期对齐与融合机制，提升多模态检索性能。
原文依据 we propose SAVE, a Speech Aware Video rEpresentation learning method
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在视频-文本检索中，CLIP的使用已成为事实标准。然而，由于CLIP仅提供图像和文本编码器，这一共识导致了一种完全忽略视频音轨的偏置范式。尽管已有若干尝试重新引入音频——通常通过加入音频编码器并将其输出与视觉特征融合——但这些方法面临两个挑战：语音内容表征效果不佳以及视音频融合欠优。为联合解决这些问题，我们提出SAVE，一种语音感知视频表征学习方法。SAVE在先进音视频方法AVIGATE的基础上改进，通过专用的语音分支实现更有效的语音嵌入。此外，我们引入soft-ALBEF进行早期视音频对齐以促进融合。在五个基准上的大量实验表明，SAVE优于当前最优方法，在SumR指标下，于MSRVTT-1k上超过AVIGATE +4.1%，MSRVTT-3k上+1.9%，VATEX上+2.5%，Charades上+9.8%，LSMDC上+2.1%。
英文原文摘要 1051 字符
For video-text retrieval, the use of CLIP has been a de facto standard. However, as CLIP provides only image and text encoders, this consensus has led to a biased paradigm that entirely ignores the sound track of videos. While several attempts have been made to reintroduce audio -- typically by incorporating an audio encoder and fusing its output with visual features -- these methods face two challenges: ineffective representation of speech content and suboptimal vision-audio fusion. To address these issues jointly, we propose SAVE, a Speech Aware Video rEpresentation learning method. SAVE improves upon AVIGATE, a SOTA audiovisual method, with a dedicated speech branch for more effective speech embedding. Furthermore, we introduce soft-ALBEF for early vision-audio alignment that facilitates fusion. Extensive experiments on five benchmarks show that SAVE compares favorably against the SOTA, outperforming AVIGATE by +4.1% on MSRVTT-1k, +1.9% on MSRVTT-3k, +2.5% on VATEX, +9.8% on Charades, and +2.1% on LSMDC, in light of the SumR metric.