面向社会公平的可解释视觉语言模型去偏
Interpretable Debiasing of Vision-Language Models for Social Fairness
用稀疏自编码器在多模态编码器上定位可解释的「社会属性神经元」，再通过选择性关闭这些神经元实现去偏，为视觉研究者提供了一种无需重训练、可审计的模型内部干预手段，可直接迁移到人脸属性识别、跨群体公平性分析等场景。
原文依据 localizes social attribute neurons in VLMs through sparse autoencoders (SAEs) applied to multimodal encoders
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）的快速发展引发了日益增长的担忧：其黑箱推理过程可能导致非预期的社会偏见形式。当前的去偏方法侧重于通过事后学习或测试时算法来缓解表层的偏见信号，而模型的内部动态在很大程度上仍未被探索。在本工作中，我们提出了 DeBiasLens，一个可解释、模型无关的偏见缓解框架，它通过将稀疏自编码器（SAE）应用于多模态编码器，在 VLM 中定位社会属性神经元。基于 SAE 的解耦能力，我们在没有对应社会属性标签的人脸图像或描述文本数据集上训练它们，以发现对特定人群（包括那些代表性不足的人群）高度响应的神经元。通过选择性地停用对每个群体而言与偏见关联最强的社会神经元，我们有效缓解了 VLM 的社会偏见行为，同时不损害其语义知识。我们的研究为未来的审计工具奠定了基础，在新兴的现实世界 AI 系统中优先考虑社会公平。
英文原文摘要 1149 字符
The rapid advancement of Vision-Language models (VLMs) has raised growing concerns that their black-box reasoning processes could lead to unintended forms of social bias. Current debiasing approaches focus on mitigating surface-level bias signals through post-hoc learning or test-time algorithms, while leaving the internal dynamics of the model largely unexplored. In this work, we introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute neurons in VLMs through sparse autoencoders (SAEs) applied to multimodal encoders. Building upon the disentanglement ability of SAEs, we train them on facial image or caption datasets without corresponding social attribute labels to uncover neurons highly responsive to specific demographics, including those that are underrepresented. By selectively deactivating the social neurons most strongly tied to bias for each group, we effectively mitigate socially biased behaviors of VLMs without degrading their semantic knowledge. Our research lays the groundwork for future auditing tools, prioritizing social fairness in emerging real-world AI systems.