以攻为守：赋能分割一切模型的点提示优化对抗智能体
Attack for Defense: Adversarial Agents for Point Prompt Optimization Empowering Segment Anything Model
把点提示词的选择建模为双空间图上的强化学习博弈，用攻击者-防御者范式自动精化粗提示集，可直接套用到 SAM 类分割模型的提示词工程与鲁棒性提升上。
原文依据 an attacker agent learns to activate a subset of prompts that maximally degrade SAM's segmentation performance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论提示词质量对分割一切模型（SAM）的性能起着关键作用，然而现有方法往往依赖启发式或人工设计的提示词，限制了可扩展性与泛化能力。本文提出 Point Prompt Defender，一种采用「以攻为守」范式的对抗强化学习框架，用于自动优化点提示词。我们将图像块表示为双空间图中的节点，边同时编码物理距离与语义距离，从而构建与任务无关的点提示环境。在该环境中，攻击者智能体学习激活一组能最大程度削弱 SAM 分割性能的提示词，而防御者智能体学习抑制这些破坏性提示词并恢复精度。两个智能体均使用深度 Q 网络训练，奖励信号基于分割质量的变化。在推理阶段仅部署防御者，用于精化任意粗糙的提示词集合，从而无需重新训练即可在多种任务上提升 SAM 的分割性能。大量实验表明，Point Prompt Defender 有效提升了 SAM 的鲁棒性与泛化能力，为基于提示词的分割建立了一个灵活、可解释且即插即用的框架。
英文原文摘要 1285 字符
Prompt quality plays a critical role in the performance of the Segment Anything Model (SAM), yet existing approaches often rely on heuristic or manually crafted prompts, limiting scalability and generalization. In this paper, we propose Point Prompt Defender, an adversarial reinforcement learning framework that adopts an attack-for-defense paradigm to automatically optimize point prompts. We construct a task-agnostic point prompt environment by representing image patches as nodes in a dual-space graph, where edges encode both physical and semantic distances. Within this environment, an attacker agent learns to activate a subset of prompts that maximally degrade SAM's segmentation performance, while a defender agent learns to suppress these disruptive prompts and restore accuracy. Both agents are trained using Deep Q-Networks with a reward signal based on segmentation quality variation. During inference, only the defender is deployed to refine arbitrary coarse prompt sets, enabling enhanced SAM segmentation performance across diverse tasks without retraining. Extensive experiments show that Point Prompt Defender effectively improves SAM's robustness and generalization, establishing a flexible, interpretable, and plug-and-play framework for prompt-based segmentation.