SoliReward：缓解视频生成奖励模型对奖励攻击与标注噪声的敏感性
SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models
为视频生成模型的偏好对齐提供了一套可复用的奖励模型训练范式：用单条二值标注加跨提示配对构造偏好数据，用分层渐进查询注意力聚合特征，并改造 BT 损失以容纳平局样本、正则化正样本分数分布，从而缓解奖励攻击；做视频生成后训练或构建 VLM 奖励模型时可直接借鉴这套数据与损失设计。
原文依据 we introduce a modified BT loss that explicitly accommodates win-tie scenarios
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论将视频生成模型与人类偏好进行后训练对齐是一个关键目标。为此过程开发有效的奖励模型（RM）面临显著的方法学障碍。当前的数据收集范式依赖同一提示内的成对标注，存在标注噪声问题。同时，基于视觉语言模型（VLM）的奖励模型的架构设计，尤其是其输出机制，仍缺乏充分探索。此外，奖励模型在后训练中易受奖励攻击影响。为缓解上述局限，我们提出 SoliReward，一个用于视频奖励模型训练的系统化框架。该框架首先通过单条二值标注获取高质量、低成本的数据，随后采用跨提示配对策略构建偏好对。在架构上，我们采用分层渐进查询注意力机制以增强特征聚合。最后，我们引入一种改进的 BT 损失，显式容纳胜-平局情形。该方法对正样本的奖励模型分数分布进行正则化，提供更细致的偏好信号，以缓解对少数最高分样本的过度关注。我们的方法在评估物理合理性、主体畸形和语义对齐的基准上得到验证，在奖励模型直接评测指标以及视频生成模型后训练效果上均展现出改进。代码与基准见 https://github.com/lian700/SoliReward。
英文原文摘要 1455 字符
Post-training alignment of video generation models with human preferences is a critical goal. Developing effective Reward Models (RMs) for this process faces significant methodological hurdles. Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise. Concurrently, the architectural design of VLM-based RMs, particularly their output mechanisms, remains underexplored. Furthermore, RM is susceptible to reward hacking in post-training. To mitigate these limitations, we propose SoliReward, a systematic framework for video RM training. Our framework first sources high-quality, cost-efficient data via single-item binary annotations, then constructs preference pairs using a cross-prompt pairing strategy. Architecturally, we employ a Hierarchical Progressive Query Attention mechanism to enhance feature aggregation. Finally, we introduce a modified BT loss that explicitly accommodates win-tie scenarios. This approach regularizes the RM's score distribution for positive samples, providing more nuanced preference signals to alleviate over-focus on a small number of top-scoring samples. Our approach is validated on benchmarks evaluating physical plausibility, subject deformity, and semantic alignment, demonstrating improvements in direct RM evaluation metrics and in the efficacy of post-training on video generation models. Code and benchmark are available at https://github.com/lian700/SoliReward.