看见关键之处：面向视觉生成的视觉偏好策略优化
Seeing What Matters: Visual Preference Policy Optimization for Visual Generation
把 GRPO 的单一标量奖励升级为基于预训练视觉骨干构建的像素级/时序级优势图，可直接复用到任何扩散或视频生成模型的 RL 后训练中，用于抑制局部伪影、强化细粒度感知区域。
原文依据 ViPO employs a Perceptual Structuring Module that uses pretrained vision backbones to construct spatially and temporally aware advantage maps
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论强化学习已成为视觉生成模型后训练的有力工具，其中组相对策略优化（GRPO）被越来越多地用于使生成器与人类偏好对齐。然而，现有的 GRPO 流程依赖每个样本的单一标量奖励，将每张图像或每段视频视为整体，忽略了视觉内容丰富的空间与时序结构。这种粗粒度监督阻碍了对局部伪影的纠正以及对细粒度感知线索的建模。我们提出视觉偏好策略优化（ViPO），一种将标量反馈提升为结构化像素级优势的 GRPO 变体。ViPO 采用感知结构化模块，利用预训练视觉骨干网络构建具有空间与时序感知能力的优势图，将优化压力重新分配到感知上重要的区域，同时保持标准 GRPO 的稳定性。在图像与视频基准上，ViPO 均持续优于原始 GRPO，提升了域内与人类偏好奖励的对齐程度，并增强了域外评测上的泛化能力。该方法与架构无关、轻量，且与现有 GRPO 训练流程完全兼容，为视觉生成提供了更具表现力、信息更丰富的学习信号。
英文原文摘要 1330 字符
Reinforcement learning (RL) has become a powerful tool for post-training visual generative models, with Group Relative Policy Optimization (GRPO) increasingly used to align generators with human preferences. However, existing GRPO pipelines rely on a single scalar reward per sample, treating each image or video as a holistic entity and ignoring the rich spatial and temporal structure of visual content. This coarse supervision hinders the correction of localized artifacts and the modeling of fine-grained perceptual cues. We introduce Visual Preference Policy Optimization (ViPO), a GRPO variant that lifts scalar feedback into structured, pixel-level advantages. ViPO employs a Perceptual Structuring Module that uses pretrained vision backbones to construct spatially and temporally aware advantage maps, redistributing optimization pressure toward perceptually important regions while preserving the stability of standard GRPO. Across both image and video benchmarks, ViPO consistently outperforms vanilla GRPO, improving in-domain alignment with human-preference rewards and enhancing generalization on out-of-domain evaluations. The method is architecture-agnostic, lightweight, and fully compatible with existing GRPO training pipelines, providing a more expressive and informative learning signal for visual generation.