一种可证明的能量引导测试时防御方法提升大型视觉语言模型的对抗鲁棒性
A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models
ET3 提供了一种无需训练、在推理阶段即可使用的防御手段：通过对输入样本做能量最小化的测试时变换来削弱对抗扰动，可直接套用到 CLIP 零样本分类以及 LVLM 的图像描述、VQA 等下游流程中，为视觉模型部署增加一层轻量防护。
原文依据 we propose Energy-Guided Test-Time Transformation (ET3), a lightweight, training-free defense that enhances the robustness by minimizing the energy of the input samples
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管多模态模型与大型视觉语言模型（LVLM）取得了快速进展，它们仍然极易受到对抗扰动的影响，这引发了人们对其在真实场景中使用可靠性的严重担忧。对抗训练已成为构建对抗攻击鲁棒模型的主流范式，而测试时变换（TTT）则作为一种在推理阶段提升鲁棒性的有前景策略而出现。鉴于此，我们提出能量引导测试时变换（ET3），这是一种轻量、无需训练的防御方法，通过最小化输入样本的能量来增强鲁棒性。我们的方法建立在一个理论基础之上，该理论证明在合理假设下我们的变换能够成功完成分类。我们给出了大量实验，表明 ET3 为分类器、基于 CLIP 的零样本分类提供了强有力的防御，并且在图像描述与视觉问答等任务中提升了 LVLM 的鲁棒性。代码见 github.com/OmnAI-Lab/Energy-Guided-Test-Time-Defense。
英文原文摘要 1070 字符
Despite the rapid progress in multimodal models and Large Visual-Language Models (LVLM), they remain highly susceptible to adversarial perturbations, raising serious concerns about their reliability in real-world use. While adversarial training has become the leading paradigm for building models that are robust to adversarial attacks, Test-Time Transformations (TTT) have emerged as a promising strategy to boost robustness at inference.In light of this, we propose Energy-Guided Test-Time Transformation (ET3), a lightweight, training-free defense that enhances the robustness by minimizing the energy of the input samples.Our method is grounded in a theory that proves our transformation succeeds in classification under reasonable assumptions. We present extensive experiments demonstrating that ET3 provides a strong defense for classifiers, zero-shot classification with CLIP, and also for boosting the robustness of LVLMs in tasks such as Image Captioning and Visual Question Answering. Code is available at github.com/OmnAI-Lab/Energy-Guided-Test-Time-Defense .