SALMUBench:敏感关联级多模态遗忘基准
SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning
为视觉-语言模型(如 CLIP)的敏感信息遗忘提供了首个细粒度关联级评测基准,包含受污染/干净模型对和结构化留出集,可直接用于诊断遗忘方法的有效性与附带损伤。
原文依据 We introduce SALMUBench (Sensitive Association-Level Multimodal Unlearning), a benchmark built upon a synthetic dataset of 60K persona-attribute associations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论随着 CLIP 等多模态模型成为下游系统不可或缺的组成部分,移除敏感信息的需求变得至关重要。然而,针对对比学习训练编码器的机器遗忘仍未得到充分探索,现有评估无法诊断细粒度的关联级遗忘。我们提出 SALMUBench(敏感关联级多模态遗忘),这是一个基于包含 6 万条人物-属性关联的合成数据集构建的基准,并包含两个基础模型:被该数据污染的受损模型和未包含该数据的干净模型。为隔离遗忘效果,两者均在相同的 4 亿对保留基集上从头训练,其中受损模型额外在敏感集上训练。我们提出一种新颖的评测协议,采用结构化留出集(holdout_identity、holdout_association)来精确测量遗忘效果和附带损伤。我们的基准揭示:尽管高效用的删除是可行的,但当前方法表现出不同的失效模式:它们要么未能有效遗忘,要么过度泛化而擦除了超出预期的信息。SALMUBench 为全面的遗忘评估设立了新标准,我们公开数据集、模型、评估脚本和排行榜以促进未来研究。
英文原文摘要 1320 字符
As multimodal models like CLIP become integral to downstream systems, the need to remove sensitive information is critical. However, machine unlearning for contrastively-trained encoders remains underexplored, and existing evaluations fail to diagnose fine-grained, association-level forgetting. We introduce SALMUBench (Sensitive Association-Level Multimodal Unlearning), a benchmark built upon a synthetic dataset of 60K persona-attribute associations and two foundational models: a Compromised model polluted with this data, and a Clean model without it. To isolate unlearning effects, both are trained from scratch on the same 400M-pair \texttt retain base, with the Compromised model additionally trained on the \texttt sensitive set. We propose a novel evaluation protocol with structured holdout sets (\texttt holdout_identity , \texttt holdout_association ) to precisely measure unlearning efficacy and collateral damage. Our benchmark reveals that while utility-efficient deletion is feasible, current methods exhibit distinct failure modes: they either fail to forget effectively or over-generalize by erasing more than intended. SALMUBench sets a new standard for comprehensive unlearning evaluation, and we publicly release our dataset, models, evaluation scripts, and leaderboards to foster future research.