IAG：面向基于 VLM 的视觉定位的输入感知后门攻击
IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding
对做视觉定位或多模态大模型落地的人，这篇给出了一个可复用的攻击面：用文本条件 UNet 把目标语义隐写进图像，使任意指定目标都能被诱导定位，可作为评估 grounding 系统可信度的红队基线。
原文依据 we propose IAG, a method that dynamically generates input-aware, text-guided triggers conditioned on any specified target object description to execute the attack
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）的最新进展显著提升了视觉定位任务，即根据自然语言查询在图像中定位物体。尽管取得了这些进展，基于 VLM 的定位系统的安全性尚未得到充分研究。本文揭示了一种新颖且现实的脆弱性：首个针对基于 VLM 的视觉定位的多目标后门攻击。与依赖静态触发器或固定目标的先前攻击不同，我们提出 IAG，该方法以任意指定的目标物体描述为条件，动态生成输入感知的、文本引导的触发器以执行攻击。这通过一个文本条件 UNet 实现，它将不可察觉的目标语义线索嵌入视觉输入，同时在良性样本上保持正常的定位性能。我们进一步设计了联合训练目标，平衡语言能力与感知重建，以确保不可察觉性、有效性和隐蔽性。在多种 VLM（如 LLaVA、InternVL、Ferret）和基准（RefCOCO、RefCOCO+、RefCOCOg、Flickr30k Entities 和 ShowUI）上的大量实验表明，与其它基线相比，IAG 在几乎所有设置下都取得了最佳 ASR，同时不损害干净准确率，对现有防御保持鲁棒性，并展现出跨数据集和跨模型的迁移性。这些发现凸显了具备定位能力的 VLM 中的关键安全风险，并强调需要进一步研究可信多模态理解。代码见补充材料。
英文原文摘要 1588 字符
Recent advances in vision-language models (VLMs) have significantly enhanced the visual grounding task, which involves locating objects in an image based on natural language queries. Despite these advancements, the security of VLM-based grounding systems has not been thoroughly investigated. This paper reveals a novel and realistic vulnerability: the first multi-target backdoor attack on VLM-based visual grounding. Unlike prior attacks that rely on static triggers or fixed targets, we propose IAG, a method that dynamically generates input-aware, text-guided triggers conditioned on any specified target object description to execute the attack. This is achieved through a text-conditioned UNet that embeds imperceptible target semantic cues into visual inputs while preserving normal grounding performance on benign samples. We further develop a joint training objective that balances language capability with perceptual reconstruction to ensure imperceptibility, effectiveness, and stealth. Extensive experiments on multiple VLMs (e.g., LLaVA, InternVL, Ferret) and benchmarks (RefCOCO, RefCOCO+, RefCOCOg, Flickr30k Entities, and ShowUI) demonstrate that IAG achieves the best ASRs compared with other baselines on almost all settings without compromising clean accuracy, maintaining robustness against existing defenses, and exhibiting transferability across datasets and models. These findings underscore critical security risks in grounding-capable VLMs and highlight the need for further research on trustworthy multimodal understanding. Code is in the supplementary material.