面向高迁移性视觉语言攻击的语义增强动态对比交互方法
Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction
对做视觉语言模型鲁棒性评测的人，这篇给出了一个可复用的攻击侧思路：用对抗/正/负样本三元组的动态对比机制持续破坏跨模态对齐，并把传统迁移攻击里的输入变换增强搬到 VLP 上做语义增广，可直接用于检验自家 VLM 的迁移攻击抵抗力。
原文依据 input transformations commonly used in traditional transfer-based attacks also benefit VLPs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论随着视觉语言预训练（VLP）模型的快速发展和广泛应用，其对对抗攻击的脆弱性已成为一个关键问题。一般而言，对抗样本通常被设计为具备迁移能力，不仅能攻击不同模型，还能跨多种任务生效。然而，现有针对语言-视觉模型的攻击主要依赖静态的跨模态交互，并且只关注破坏正图文对，导致跨模态破坏有限、迁移性较差。为解决该问题，我们提出语义增强动态对比攻击（SADCA），通过渐进式、语义引导的扰动提升对抗迁移性。SADCA 借助对抗图像与文本之间的动态交互，逐步破坏跨模态对齐。具体而言，SADCA 建立了一个涉及对抗样本、正样本与负样本的对比学习机制，以强化所得扰动的语义不一致性。此外，我们通过实验发现，传统基于迁移的攻击中常用的输入变换同样对 VLP 有益，这促使我们设计了一个语义增广模块，以提升对抗样本的多样性与泛化能力。在多个数据集和模型上的大量实验表明，SADCA 显著提升了对抗迁移性，并持续超越最优方法。代码已发布于 https://github.com/LiYuanBoJNU/SADCA。
英文原文摘要 1561 字符
With the rapid advancement and widespread application of vision-language pre-training (VLP) models, their vulnerability to adversarial attacks has become a critical concern. In general, the adversarial examples can typically be designed to exhibit transferable power, attacking not only different models but also across diverse tasks. However, existing attacks on language-vision models mainly rely on static cross-modal interactions and focus solely on disrupting positive image-text pairs, resulting in limited cross-modal disruption and poor transferability. To address this issue, we propose a Semantic-Augmented Dynamic Contrastive Attack (SADCA) that enhances adversarial transferability through progressive and semantically guided perturbation. SADCA progressively disrupts cross-modal alignment through dynamic interactions between adversarial images and texts. This is accomplished by SADCA establishing a contrastive learning mechanism involving adversarial, positive and negative samples, to reinforce the semantic inconsistency of the obtained perturbations. Moreover, we empirically find that input transformations commonly used in traditional transfer-based attacks also benefit VLPs, which motivates a semantic augmentation module that increases the diversity and generalization of adversarial examples. Extensive experiments on multiple datasets and models demonstrate that SADCA significantly improves adversarial transferability and consistently surpasses state-of-the-art methods. The code is released at https://github.com/LiYuanBoJNU/SADCA.