面向被植入后门的大型视觉语言模型的测试时注意力净化
Test-Time Attention Purification for Backdoored Large Vision Language Models
对做多模态大模型安全与部署的研究者，这篇给出了一个可直接复用的测试时检测信号——跨模态融合层中视觉-文本注意力比值，以及一个免训练、即插即用的输入净化流程（剪除高注意力可疑视觉 token），无需重训 LoRA 或适配器即可中和后门激活。
原文依据 the trigger does not influence prediction through low-level visual patterns, but through abnormal cross-modal attention redistribution
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管大型视觉语言模型（LVLM）具备强大的多模态性能，但它们在微调阶段易受后门攻击：攻击者将嵌入触发器的样本插入训练数据，从而植入可在测试时被恶意激活的行为。现有防御通常依赖用干净数据重新训练被植入后门的参数（如适配器或低秩适配模块），计算开销大且往往导致模型性能下降。在本工作中，我们从机制上对 LVLM 中的后门行为给出了新的理解：触发器并非通过低层视觉模式影响预测，而是通过异常的跨模态注意力再分配起作用——携带触发器的视觉 token 会从文本上下文中夺走注意力，我们将这一现象称为注意力窃取。受此启发，我们提出 CleanSight，一种免训练、即插即用的防御方法，完全在测试时运行。CleanSight（i）基于选定跨模态融合层中的视觉-文本相对注意力比值检测被投毒输入，（ii）通过有选择地剪枝可疑的高注意力视觉 token 来净化输入，从而中和后门激活。大量实验表明，CleanSight 在多种数据集和后门攻击类型上显著优于现有的基于像素的净化防御，同时在干净样本和被投毒样本上均保持模型的效用。
英文原文摘要 1433 字符
Despite the strong multimodal performance, large vision-language models (LVLMs) are vulnerable during fine-tuning to backdoor attacks, where adversaries insert trigger-embedded samples into the training data to implant behaviors that can be maliciously activated at test time. Existing defenses typically rely on retraining backdoored parameters (e.g., adapters or LoRA modules) with clean data, which is computationally expensive and often degrades model performance. In this work, we provide a new mechanistic understanding of backdoor behaviors in LVLMs: the trigger does not influence prediction through low-level visual patterns, but through abnormal cross-modal attention redistribution, where trigger-bearing visual tokens steal attention away from the textual context -- a phenomenon we term attention stealing. Motivated by this, we propose CleanSight, a training-free, plug-and-play defense that operates purely at test time. CleanSight (i) detects poisoned inputs based on the relative visual-text attention ratio in selected cross-modal fusion layers, and (ii) purifies the input by selectively pruning the suspicious high-attention visual tokens to neutralize the backdoor activation. Extensive experiments show that CleanSight significantly outperforms existing pixel-based purification defenses across diverse datasets and backdoor attack types, while preserving the model's utility on both clean and poisoned samples.