TTP：面向视觉语言模型对抗检测与鲁棒自适应的测试时填充
TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models
对做 CLIP 类视觉语言模型鲁棒性的人，这篇给出了一个无需重训的推理期防御范式：用空间填充前后 CLIP 特征的余弦相似度漂移做对抗检测，并对检测出的对抗样本用可训练填充修复注意力模式，可直接套用到自己的 VLM 安全部署流程中。
原文依据 TTP identifies adversarial inputs via the cosine similarity shift between CLIP feature embeddings computed before and after spatial padding
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（如 CLIP）已取得令人瞩目的零样本识别性能，但仍极易受到对抗扰动的影响，在安全关键场景中构成重大风险。此前的训练期防御依赖对抗微调，需要标注数据和昂贵的重训练；而现有的测试时策略无法可靠区分干净输入与对抗输入，导致对抗鲁棒性与干净准确率都无法达到最优。为解决这些局限，我们提出测试时填充（TTP），一个轻量级防御框架，在推理阶段先进行对抗检测，再进行针对性自适应。TTP 通过计算空间填充前后 CLIP 特征嵌入之间的余弦相似度漂移来识别对抗输入，从而得到一个可在不同架构与数据集上可靠检测的通用阈值。对于检测出的对抗样本，TTP 采用可训练填充来恢复被破坏的注意力模式，并结合相似度感知的集成策略以获得更鲁棒的最终预测。对于干净输入，TTP 默认保持不变，也可选择性地集成现有的测试时自适应技术以进一步提升准确率。在多种 CLIP 骨干网络和细粒度基准上的大量实验表明，TTP 持续优于最优的测试时防御方法，在对抗鲁棒性上取得显著提升，同时不损害干净准确率。本文代码即将发布。
英文原文摘要 1580 字符
Vision-Language Models (VLMs), such as CLIP, have achieved impressive zero-shot recognition performance but remain highly susceptible to adversarial perturbations, posing significant risks in safety-critical scenarios. Previous training-time defenses rely on adversarial fine-tuning, which requires labeled data and costly retraining, while existing test-time strategies fail to reliably distinguish between clean and adversarial inputs, thereby preventing both adversarial robustness and clean accuracy from reaching their optimum. To address these limitations, we propose Test-Time Padding (TTP), a lightweight defense framework that performs adversarial detection followed by targeted adaptation at inference. TTP identifies adversarial inputs via the cosine similarity shift between CLIP feature embeddings computed before and after spatial padding, yielding a universal threshold for reliable detection across architectures and datasets. For detected adversarial cases, TTP employs trainable padding to restore disrupted attention patterns, coupled with a similarity-aware ensemble strategy for a more robust final prediction. For clean inputs, TTP leaves them unchanged by default or optionally integrates existing test-time adaptation techniques for further accuracy gains. Comprehensive experiments on diverse CLIP backbones and fine-grained benchmarks show that TTP consistently surpasses state-of-the-art test-time defenses, delivering substantial improvements in adversarial robustness without compromising clean accuracy. The code for this paper will be released soon.