探究自监督表征在音视频深度伪造检测中的作用
Investigating Self-Supervised Representations for Audio-Visual Deepfake Detection
系统评测了音频、视频、多模态三类自监督特征在音视频深度伪造检测中的表现，并给出「音频引导表征泛化最好」的结论，可直接作为视觉研究者挑选伪造检测骨干特征、以及设计跨模态互补融合时的选型依据。
原文依据 audio-informed representations generalize best and achieve state-of-the-art results
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论自监督表征在许多视觉与语音任务中表现优异，但其在音视频深度伪造检测中的潜力仍未被充分探索。与以往将这些特征孤立使用或埋没于复杂架构中的工作不同，我们系统地在模态（音频、视频、多模态）与域（唇部运动、通用视觉内容）两个维度上对它们进行评估。我们从三个关键维度进行考察：检测有效性、所编码信息的可解释性，以及跨模态互补性。我们发现，大多数自监督特征都捕获了与深度伪造相关的信息，且这些信息之间具有互补性。此外，模型主要关注语义上有意义的区域，而非虚假的伪影（例如开头的静音段）。在所考察的特征中，音频引导的表征泛化能力最好，并取得了最优结果。然而，面向真实野外数据的泛化仍然具有挑战性。我们的分析表明，这一差距源于数据集本身的内在难度，而非特征依附于表层模式。
英文原文摘要 1090 字符
Self-supervised representations excel at many vision and speech tasks, but their potential for audio-visual deepfake detection remains underexplored. Unlike prior work that uses these features in isolation or buried within complex architectures, we systematically evaluate them across modalities (audio, video, multimodal) and domains (lip movements, generic visual content). We assess three key dimensions: detection effectiveness, interpretability of encoded information, and cross-modal complementarity. We find that most self-supervised features capture deepfake-relevant information, and that this information is complementary. Moreover, models primarily attend to semantically meaningful regions rather than spurious artifacts (such as the leading silence). Among the investigated features, audio-informed representations generalize best and achieve state-of-the-art results. However, generalization to realistic in-the-wild data remains challenging. Our analysis indicates this gap stems from intrinsic dataset difficulty rather than from features latching onto superficial patterns.