分数末端的偏差
Bias at the End of the Score
对做 T2I 生成与奖励引导优化（如 RLHF/DRaFT 类微调、数据集筛选）的研究者，这篇审计提示：奖励模型本身携带人口统计偏差，用它做监督信号会系统性地放大性别/种族刻板印象并压缩多样性；在搭建奖励反馈管线时，应把奖励模型的偏差审计与去偏作为必要环节，而非只关注 reward hacking 与模式坍塌。
原文依据 RMs encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female image subjects, reinforce gender/racial stereotypes, and collapse demographic diversity.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论奖励模型（RM）本质上是非中立的价值函数，其设计与训练旨在编码特定目标，例如人类偏好或文本-图像对齐。奖励模型已成为文本到图像（T2I）生成系统中的关键组件，被用于数据集筛选、作为评价指标、在参数优化过程中作为监督信号，以及用于生成后对 T2I 输出进行安全与质量过滤。尽管将奖励模型整合进 T2I 流程所引发的具体问题已有研究（例如奖励黑客或模式坍塌），但其作为评分函数的鲁棒性与公平性在很大程度上仍属未知。我们对 T2I 模型训练与生成过程中奖励模型在人口统计偏差方面的鲁棒性开展了大规模审计。我们提供了定量与定性证据，表明奖励模型虽然最初是作为质量度量而开发的，却编码了人口统计偏差，这些偏差导致奖励引导的优化不成比例地将女性图像主体性化、强化性别与种族刻板印象，并使人口统计多样性坍塌。这些发现凸显了当前奖励模型的缺陷，对其作为质量指标的可靠性提出了质疑，并强调了改进数据收集与训练流程以实现更鲁棒评分的必要性。
英文原文摘要 1307 字符
Reward models (RMs) are inherently non-neutral value functions designed and trained to encode specific objectives, such as human preferences or text-image alignment. RMs have become crucial components of text-to-image (T2I) generation systems where they are used at various stages for dataset filtering, as evaluation metrics, as a supervisory signal during optimization of parameters, and for post-generation safety and quality filtering of T2I outputs. While specific problems with the integration of RMs into the T2I pipeline have been studied (e.g. reward hacking or mode collapse), their robustness and fairness as scoring functions remains largely unknown. We conduct a large scale audit of RM robustness with respect to demographic biases during T2I model training and generation. We provide quantitative and qualitative evidence that while originally developed as quality measures, RMs encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female image subjects, reinforce gender/racial stereotypes, and collapse demographic diversity. These findings highlight shortcomings in current reward models, challenge their reliability as quality metrics, and underscore the need for improved data collection and training procedures to enable more robust scoring.