面向多模态大语言模型的多范式协同对抗攻击
Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models
对做多模态大模型红队与鲁棒性评测的人，这篇给出了一个可复用的思路：不要只用单一范式的代理模型做特征空间优化，而是把视觉与语言两个范式的语义表征聚合起来，用对比匹配自适应加权后再做全局扰动优化，从而缓解表征偏置、提升对抗样本的跨模型迁移性。
原文依据 MPCAttack aggregates semantic representations, from both visual images and language texts, to facilitate joint adversarial optimisation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）的快速进展显著推动了下游应用的发展。然而，这一进展也暴露出严重的可迁移对抗脆弱性。总体而言，现有针对 MLLMs 的对抗攻击通常依赖在单一学习范式内训练的代理模型，并在各自的特征空间中独立优化。这种直接的设定自然限制了特征表征的丰富性，使搜索空间受限，进而阻碍了对抗扰动的多样性。为解决这一问题，我们提出了一种新的多范式协同攻击（MPCAttack）框架，以提升针对 MLLMs 的对抗样本的可迁移性。在原理上，MPCAttack 聚合来自视觉图像与语言文本的语义表征，并通过多范式协同优化（MPCO）策略在聚合特征上进行联合对抗优化。MPCO 通过对多范式特征执行对比匹配，自适应地平衡不同范式表征的重要性，并引导全局扰动优化，从而有效缓解表征偏置。在多个基准上的大量实验结果表明了 MPCAttack 的优越性，说明我们的方案在开源与闭源 MLLMs 的定向攻击与非定向攻击中均持续优于最优方法。代码已发布于 https://github.com/LiYuanBoJNU/MPCAttack。
英文原文摘要 1564 字符
The rapid progress of Multi-Modal Large Language Models (MLLMs) has significantly advanced downstream applications. However, this progress also exposes serious transferable adversarial vulnerabilities. In general, existing adversarial attacks against MLLMs typically rely on surrogate models trained within a single learning paradigm and perform independent optimisation in their respective feature spaces. This straightforward setting naturally restricts the richness of feature representations, delivering limits on the search space and thus impeding the diversity of adversarial perturbations. To address this, we propose a novel Multi-Paradigm Collaborative Attack (MPCAttack) framework to boost the transferability of adversarial examples against MLLMs. In principle, MPCAttack aggregates semantic representations, from both visual images and language texts, to facilitate joint adversarial optimisation on the aggregated features through a Multi-Paradigm Collaborative Optimisation (MPCO) strategy. By performing contrastive matching on multi-paradigm features, MPCO adaptively balances the importance of different paradigm representations and guides the global perturbation optimisation, effectively alleviating the representation bias. Extensive experimental results on multiple benchmarks demonstrate the superiority of MPCAttack, indicating that our solution consistently outperforms state-of-the-art methods in both targeted and untargeted attacks on open-source and closed-source MLLMs. The code is released at https://github.com/LiYuanBoJNU/MPCAttack.