PaCo-RL：以成对奖励建模推进一致性图像生成的强化学习
PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modeling
把成对一致性奖励模型与 GRPO 结合，用无标注数据对生成模型做 RL 微调，并给出分辨率解耦的优化策略来降低 RL 训练成本；做多图一致性生成或想把主观视觉偏好转成奖励信号的工作可直接复用这套奖励建模与多奖励聚合思路。
原文依据 we introduce PaCo-RL, a comprehensive framework that combines a specialized consistency reward model with an efficient RL algorithm
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论一致性图像生成要求在生成多张图像时忠实保持身份、风格与逻辑连贯性，这对故事叙述和角色设计等应用至关重要。监督训练方法在该任务上存在困难，一方面缺乏刻画视觉一致性的大规模数据集，另一方面难以建模人类的感知偏好。本文认为，强化学习（RL）提供了一条有前景的替代路径，使模型能够以无数据的方式学习复杂且主观的视觉准则。为此，我们提出 PaCo-RL，一个将专用一致性奖励模型与高效 RL 算法相结合的综合框架。第一个组件 PaCo-Reward 是一个成对一致性评估器，其训练所用的大规模数据集通过自动子图配对构建；它借助任务感知指令与思维链（CoT）理由增强的生成式自回归打分机制来评估一致性。第二个组件 PaCo-GRPO 采用一种新的分辨率解耦优化策略，大幅降低 RL 开销，并配合对数约束的多奖励聚合机制，确保奖励优化均衡且稳定。在两个代表性子任务上的大量实验表明，PaCo-Reward 显著提升了与人类视觉一致性感知的对齐程度，PaCo-GRPO 在提升训练效率与稳定性的同时取得最优的一致性性能。这些结果共同表明，PaCo-RL 是一种实用且可扩展的一致性图像生成方案。我们将公开代码、数据集与模型。
英文原文摘要 1672 字符
Consistent image generation requires faithfully preserving identities, styles, and logical coherence across multiple images,which is essential for applications such as storytelling and character design.Supervised training approaches struggle with this task due to the lack of large-scale datasets capturing visual consistency and the complexity of modeling human perceptual preferences.In this paper, we argue that reinforcement learning (RL) offers a promising alternative by enabling models to learn complex and subjective visual criteria in a data-free manner.To achieve this, we introduce PaCo-RL, a comprehensive framework that combines a specialized consistency reward model with an efficient RL algorithm.The first component, PaCo-Reward, is a pairwise consistency evaluator trained on a large-scale dataset constructed via automated sub-figure pairing.It evaluates consistency through a generative, autoregressive scoring mechanism enhanced by task-aware instructions and CoT reasons.The second component, PaCo-GRPO,leverages a novel resolution-decoupled optimization strategy to substantially reduce RL cost,alongside a log-tamed multi-reward aggregation mechanism that ensures balanced and stable reward optimization.Extensive experiments across the two representative subtasks show that PaCo-Reward significantly improves alignment with human perceptions of visual consistency,and PaCo-GRPO achieves state-of-the-art consistency performance with improved training efficiency and stability.Together, these results highlight the promise of PaCo-RL as a practical and scalable solution for consistent image generation.We will release the code, dataset, and models.