Style-GRPO:由奖励建模引导的语义感知图像风格迁移偏好优化
Style-GRPO: Semantic-Aware Preference Optimization for Image Style Transfer Guided by Reward Modeling
提供了一套可复用的「奖励模型 + 在线 RL」范式:先构建 8 万级空间关系偏好对训练 SpatialScore 奖励模型,再用其驱动 GRPO 式在线强化学习提升扩散模型的空间关系生成能力,可直接迁移到其他需要精确空间布局的可控生成任务。
原文依据 We further demonstrate that this reward model effectively enables online reinforcement learning for the complex spatial generation.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论文本到图像生成领域的最新进展极大提升了视觉保真度与创造力,但同时也对提示词的复杂度提出了更高要求——尤其是在编码复杂空间关系方面。在此类情况下,往往需要多次采样尝试才能获得满意结果。为应对这一挑战,我们提出一种新方法,以增强现有图像生成模型的空间理解能力。我们首先构建了包含 8 万余对偏好样本的 SpatialReward-Dataset。基于该数据集,我们构建了 SpatialScore,这是一个用于评估文本到图像生成中空间关系准确性的奖励模型,其性能在空间评估上甚至超越了领先的专有模型。通过严格的数据整理与筛选,SpatialScore 在空间评估上超越了多个领先的视觉语言模型(VLM)与现有奖励模型。我们进一步证明,该奖励模型能够有效支撑面向复杂空间生成的在线强化学习。在多个基准上的大量实验表明,我们专用的奖励模型为图像生成的空间理解带来了显著且一致的提升。所有模型与数据集将开源发布。
英文原文摘要 1285 字符
Recent progress in text-to-image generation has greatly advanced visual fidelity and creativity, but it has also imposed higher demands on prompt complexity--particularly in encoding intricate spatial relationships. In such cases, achieving satisfactory results often requires multiple sampling attempts.To address this challenge, we introduce a novel method that strengthens the spatial understanding of current image generation models. We first construct the **SpatialReward-Dataset** with over 80k preference pairs. Building on this dataset, we build **SpatialScore**, a reward model designed to evaluate the accuracy of spatial relationships in text-to-image generation, achieving performance that even surpasses leading proprietary models on spatial evaluation.% Through rigorous data curation and filtering, **SpatialScore** surpasses several leading vision-language models (VLMs) and existing reward models on spatial evaluation. We further demonstrate that this reward model effectively enables online reinforcement learning for the complex spatial generation. Extensive experiments across multiple benchmarks show that our specialized reward model yields significant and consistent gains in spatial understanding for image generation. All models and datasets will be released.