鲁棒性与安全 · CVPR2026 ·2026.06.03

偏见是子空间而非坐标:对视觉语言模型事后去偏的几何再思考

Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models

Dachuan Zhao、Weiyue Li、Zhenda Shen、Yushu Qiu、Bowen Xu、Haoyu Chen 等 7 人

与你有关的部分

把「偏见分布在低维线性子空间而非个别坐标」这一几何视角,转化为可直接套用的后处理去偏算子:在嵌入空间做子空间投影并回填中性均值,可复用于CLIP类模型的零样本分类、图文检索与生成,无需重训练。

原文依据 bias is not localized to a few coordinates but is instead distributed across a few linear subspaces
可迁移到 零样本图像分类文本到图像检索文本到图像生成公平性评测

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

视觉语言模型(VLM)已成为多模态推理不可或缺的工具,然而其表征往往编码并放大人口统计学偏见,从而在下游任务中产生有偏关联与失准预测。此类行为损害公平性,并扭曲视觉与语言之间本应具有的对齐。近期的事后方法试图通过将最与属性相关的嵌入坐标替换为中性值来缓解偏见。然而,我们的系统分析揭示了这种逐坐标方法的三个关键局限:特征纠缠、跨数据集泛化差以及偏见去除不彻底。我们发现,偏见并非局限于少数坐标,而是分布在少数线性子空间之中。为应对这些局限,我们提出子空间投影去偏(SPD),这是一个具有几何原理的框架,能够识别并移除可线性解码的偏见的整个子空间,同时重新插入中性均值分量以保持语义保真度。在零样本分类、文本到图像检索与图像生成上的大量实验验证了 SPD 的有效性:与最佳去偏基线相比,我们的方法实现了更稳健的去偏,在四项公平性指标上平均提升 18.5%,同时任务性能损失极小。

英文原文摘要 1373 字符

Vision-Language Models (VLMs) have become indispensable for multimodal reasoning, yet their representations often encode and amplify demographic biases, resulting in biased associations and misaligned predictions in downstream tasks. Such behavior undermines fairness and distorts the intended alignment between vision and language. Recent post-hoc approaches attempt to mitigate bias by replacing the most attribute-correlated embedding coordinates with neutral values. However, our systematic analysis reveals three critical limitations of this coordinate-wise approach: feature entanglement, poor cross-dataset generalization, and incomplete bias removal. We find that bias is not localized to a few coordinates but is instead distributed across a few linear subspaces. To address these limitations, we propose Subspace Projection Debiasing (SPD), a geometrically principled framework that identifies and removes the entire subspace of linearly decodable bias while reinserting a neutral mean component to preserve semantic fidelity. Extensive experiments across zero-shot classification, text-to-image retrieval, and image generation validate the effectiveness of SPD: our method achieves more robust debiasing with an average improvement of 18.5% across four fairness metrics, while maintaining minimal loss in task performance compared to the best debiasing baseline.

标签

会议 CVPR2026
页码 10230-10240
Bibkey
来源 CVF Open Access