DiverseGRPO：通过多样性感知GRPO缓解图像生成中的模式坍缩
DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPO
针对GRPO微调扩散模型时出现的模式坍缩问题，提出基于语义聚类的分布级奖励和结构感知正则化，可直接用于提升文生图模型的输出多样性，同时保持生成质量。
原文依据 we propose a distributional creativity bonus based on semantic grouping
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论强化学习（RL），特别是GRPO，通过比较同一组内生成图像的相对表现，显著提升了图像生成质量。然而，在训练后期，模型倾向于产生同质化输出，缺乏创造力和视觉多样性，限制了模型的应用场景。该问题可以从奖励建模和生成动力学两个角度进行分析。首先，传统GRPO依赖单样本质量作为奖励信号，驱使模型收敛到少数高奖励生成模式，而忽视了分布级多样性。其次，传统GRPO正则化忽略了早期去噪在保持多样性中的主导作用，导致正则化预算分配失当，限制了可达到的质量-多样性权衡。受这些见解启发，我们从奖励建模和生成动力学两方面重新审视多样性退化问题。在奖励层面，我们提出了一种基于语义分组的分布级创造力奖励。具体而言，我们通过对同一标题生成的样本进行谱聚类来构建分布级表示，并根据组大小自适应分配探索性奖励，以鼓励发现新的视觉模式。在生成层面，我们引入了一种结构感知正则化，它施加更强的早期约束以保持多样性，同时不损害奖励优化效率。实验表明，在匹配质量分数下，我们的方法在语义多样性上实现了13%~18%的提升，为基于GRPO的图像生成在图像质量与多样性之间建立了新的帕累托前沿。
英文原文摘要 1757 字符
Reinforcement learning (RL), particularly GRPO, improves image generation quality significantly by comparing the relative performance of images generated within the same group. However, in the later stages of training, the model tends to produce homogenized outputs, lacking creativity and visual diversity, restricting the application scenarios of the model. This issue can be analyzed from both reward modeling and generation dynamics perspectives. First, traditional GRPO relies on single sample quality as the reward signal, driving the model to converge toward a few high reward generation modes while neglecting distribution-level diversity. Second, conventional GRPO regularization neglects the dominant role of early stage denoising in preserving diversity, causing a misaligned regularization budget that limits the achievable quality-diversity trade-off.Motivated by these insights, we revisit the diversity degradation problem from both reward modeling and generation dynamics. At the reward level, we propose a distributional creativity bonus based on semantic grouping. Specifically, we construct a distribution-level representation via spectral clustering over samples generated from the same caption, and adaptively allocate exploratory rewards according to group sizes to encourage the discovery of novel visual modes. At the generation level, we introduce a structure-aware regularization, which enforces stronger early-stage constraints to preserve diversity without compromising reward optimization efficiency. Experiments demonstrate that our method achieves an 13%~18% improvement in semantic diversity under matched quality scores, establishing a new Pareto frontier between image quality and diversity for GRPO-based image generation.