SAGE:面向隐私约束下跨域语义分割的风格自适应泛化框架
SAGE: Style-Adaptive Generalization for Privacy-Constrained Semantic Segmentation Across Domains
在模型权重冻结、无法微调的黑盒设定下,用输入级视觉提示词做域泛化:先以风格迁移构造覆盖广的风格表征库,再按每张输入的视觉上下文动态融合成提示词,为视觉研究者提供了一条不碰骨干网络即可对齐跨域特征分布的思路,可直接迁移到分割、检测等密集预测任务的隐私受限部署场景。
原文依据 SAGE learns to synthesize visual prompts that implicitly align feature distributions across styles instead of directly fine-tuning the backbone.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论语义分割的域泛化旨在缓解域偏移导致的模型性能下降。然而在许多真实场景中,出于隐私与安全约束,我们无法获取模型参数与架构细节,传统微调或自适应方法因此受阻,这就催生了对无需修改模型权重即可增强泛化的输入级策略的需求。为此,我们提出风格自适应泛化框架(SAGE),在隐私约束下提升冻结模型的泛化能力。SAGE 学习合成视觉提示词,以隐式对齐不同风格间的特征分布,而非直接微调骨干网络。具体而言,我们首先利用风格迁移构建源域的多样化风格表征,从而学习一组能够覆盖广泛视觉特征的风格特性;随后,模型根据每个输入的视觉上下文自适应地融合这些风格线索,形成动态提示词,在不触及模型内部的前提下协调图像外观。通过这一闭环设计,SAGE 有效弥合了冻结模型的不变性与未见域多样性之间的差距。在五个基准数据集上的大量实验表明,在隐私约束下 SAGE 取得了与最优方法相当或更优的性能,并在所有设置下均优于全量微调基线。
英文原文摘要 1533 字符
Domain generalization for semantic segmentation aims to mitigate the degradation in model performance caused by domain shifts. However, in many real-world scenarios, we are unable to access the model parameters and architectural details due to privacy concerns and security constraints. Traditional fine-tuning or adaptation is hindered, leading to the demand for input-level strategies that can enhance generalization without modifying model weights. To this end, we propose a Style-Adaptive GEneralization framework (SAGE), which improves the generalization of frozen models under privacy constraints. SAGE learns to synthesize visual prompts that implicitly align feature distributions across styles instead of directly fine-tuning the backbone. Specifically, we first utilize style transfer to construct a diverse style representation of the source domain, thereby learning a set of style characteristics that can cover a wide range of visual features. Then, the model adaptively fuses these style cues according to the visual context of each input, forming a dynamic prompt that harmonizes the image appearance without touching the interior of the model. Through this closed-loop design, SAGE effectively bridges the gap between frozen model invariance and the diversity of unseen domains. Extensive experiments on five benchmark datasets demonstrate that SAGE achieves competitive or superior performance compared to state-of-the-art methods under privacy constraints and outperforms full fine-tuning baselines in all settings.