通过分歧引导的后缀优化与图像短语注入越狱视觉语言模型
Jailbreaking Vision-Language Models via Dissonance-Guided Suffix Optimization and Image-Phrase Injection
对做 VLM 安全与多模态对齐的研究者，这篇给出了一个可复用的攻击信号：用目标模型与未对齐模型之间的预测分歧（predictive dissonance）来定位被安全对齐压制的词元，替代离散词元空间上的梯度近似；同时把短语的内容与呈现方式一起嵌入图像做联合优化，利用跨模态敏感性提升迁移性，可作为红队评测 VLM 防护的基线。
原文依据 leverages predictive dissonance between the target model and an unaligned model to identify tokens suppressed by safety alignment
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）中视觉与语言的融合在赋予其多模态能力的同时，也天然扩大了其攻击面。在现有的白盒越狱方法中，基于后缀优化的方法往往依赖离散词元空间上的梯度近似，引导信号不足，导致优化停滞于局部最优；而基于图像扰动的方法则常常表现出较差的跨模型迁移性。在本工作中，我们提出 DGSIP，一个分歧引导的后缀优化与图像短语注入框架。DGSIP 利用目标模型与未对齐模型之间的预测分歧来识别被安全对齐所压制的词元，并将其作为比基于梯度的线索更有效的信号用于后缀优化。它进一步通过联合优化嵌入图像中的短语的内容与呈现方式来强化攻击，以利用 VLM 的跨模态敏感性。我们的大量实验表明，DGSIP 在多个安全基准和一系列开源 VLM（如 MiniGPT-4、InstructBlip 和 LLaVA）上均优于此前的基线方法。值得注意的是，与基线相比，我们的方法对商用黑盒 VLM（如 GPT-4o-Mini、Gemini 2.0 Flash 和 Qwen 2.5-VL）表现出强得多的迁移性。基于 DGSIP，我们从实验上揭示了当前 VLM 防护机制中的关键脆弱性，凸显了对更鲁棒防御策略的需求。实现代码已发布于 https://github.com/Trusted-LLM/DGSIP。
英文原文摘要 1561 字符
The integration of vision and language in Vision-Language Models (VLMs), while enabling multimodal capabilities, inherently expands their attack surface. Among existing white-box jailbreak methods, suffix-optimization-based approaches often rely on gradient approximations over discrete token spaces, yielding insufficient guidance and causing optimization to stagnate in local optima, while image-perturbation-based ones frequently exhibit poor cross-model transferability. In this work, we introduce DGSIP, a Dissonance-Guided Suffix Optimization and Image-Phrase Injection framework. DGSIP leverages predictive dissonance between the target model and an unaligned model to identify tokens suppressed by safety alignment, using them as a more effective signal than gradient-based cues for suffix optimization. It further reinforces the attack by jointly optimizing the content and presentation of phrase embedded in images to leverage VLMs' cross-modal sensitivity. Our extensive experiments demonstrate that DGSIP outperforms prior baselines across multiple safety benchmarks and a range of open-source VLMs (e.g., MiniGPT-4, InstructBlip and LLaVA). Notably, compared to baselines, our method exhibits much stronger transferability to commercial black-box VLMs, such as GPT-4o-Mini, Gemini 2.0 Flash and Qwen 2.5-VL. Based upon DGSIP, we empirically reveal critical vulnerabilities in the safeguard mechanisms of current VLMs, highlighting the need for more robust defense strategies. The implementation is available on https://github.com/Trusted-LLM/DGSIP.