动态词元重加权：面向鲁棒视觉语言模型
Dynamic Token Reweighting for Robust Vision-Language Models
对做视觉语言模型安全的研究者，这篇给出了一个无需额外训练、不改图像输入的推理时防御手段：通过优化 KV 缓存动态调整视觉词元权重，可直接插到现有 VLM 的推理流程里，用来抵御对抗性视觉输入，同时保持正常任务性能。
原文依据 a novel inference-time defense that mitigates multimodal jailbreak attacks through optimizing the model's key-value (KV) caches
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（VLM）极易受到多模态越狱攻击，这类攻击利用视觉与文本的交互绕过安全护栏。本文提出 DTR，一种新颖的推理时防御方法，通过优化模型的键值（KV）缓存来缓解多模态越狱攻击。该方法不依赖精心整理的安全专用数据，也无需代价高昂的图像到文本转换，而是对视觉模态所引发的安全相关分布偏移给出一种新的刻画。基于该刻画，DTR 能够动态调整视觉词元权重，在最小化对抗性视觉输入影响的同时，保持模型的通用能力与推理效率。在多种 VLM 与攻击基准上的大量评估表明，DTR 在攻击鲁棒性与良性任务性能两方面均优于现有防御方法，这是 KV 缓存优化首次成功应用于多模态基础模型的安全增强。复现 DTR 的代码已发布于：https://github.com/TanqiuJiang/DTR。
英文原文摘要 1088 字符
Large vision-language models (VLMs) are highly vulnerable to multimodal jailbreak attacks that exploit visual-textual interactions to bypass safety guardrails. In this paper, we present DTR, a novel inference-time defense that mitigates multimodal jailbreak attacks through optimizing the model's key-value (KV) caches. Rather than relying on curated safety-specific data or costly image-to-text conversion, we introduce a new formulation of the safety-relevant distributional shift induced by the visual modality. This formulation enables DTR to dynamically adjust visual token weights, minimizing the impact of adversarial visual inputs while preserving the model's general capabilities and inference efficiency. Extensive evaluation across diverse VLMs and attack benchmarks demonstrates that DTR outperforms existing defenses in both attack robustness and benign-task performance, marking the first successful application of KV cache optimization for safety enhancement in multimodal foundation models. The code for replicating DTR is available at: https://github.com/TanqiuJiang/DTR.