RebRL：以重平衡时间步信用强化离散视觉扩散模型
RebRL: Reinforcing Discrete Visual Diffusion Models with Rebalanced Timestep Credits
针对离散扩散（掩码生成）模型在 RL 微调时各时间步信用分配不均的问题，提出按掩码阶段重新平衡时间步权重的即插即用策略，可直接用于改进视觉生成模型的 RLHF/偏好对齐训练流程，并减少约 40% 训练步数。
原文依据 severe imbalance of credit assignment across timesteps during training
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论离散扩散模型（DDM）在图像生成中展现出巨大潜力，尤其是在结合强化学习（RL）技术之后。然而，我们的实验揭示了一个基础却长期被忽视的局限：训练过程中各时间步之间的信用分配严重失衡。其结果是，具有更高探索潜力并决定全局结构的早期生成时间步对策略优化的贡献反而更小。为解决这一问题，我们提出一种简单而有效的方法——强化学习时间步信用重平衡（RebRL），以实现更好的探索-利用权衡，并更高效地训练 DDM。RebRL 即插即用——只需沿掩码阶段以策略性重平衡替代均匀的时序策略。RebRL 在分析上是合理的——推导与分析表明其具有均匀的 token 级策略梯度，有利于策略优化。在文本到图像生成基准上的实验表明，RebRL 在 GenEval 上达到最优性能，并将人类偏好得分最高提升 3.40，同时有效减少约 40% 的训练步数。
英文原文摘要 1183 字符
Discrete Diffusion Models (DDMs) have shown great potential in image generation, especially when equipped with reinforcement learning (RL) techniques.However, a fundamental yet overlooked limitation is revealed in our experiments: severe imbalance of credit assignment across timesteps during training. As a result, early generation timesteps, which carry higher exploration potential and determine the global structure, provide a smaller contribution to policy optimization.To conquer this, we propose a simple-yet-effective approach, to Re-balance timestep credit of Reinforcement Learning (RebRL) for better exploration-exploitation trade-off and more efficient training of DDMs.RebRL is plug-and-play\textemdash simply replacing uniform temporal policy with strategic rebalancing along masking stages. RebRL is analytically plausible\textemdashderivation and analysis show that it enjoys a uniform token-level policy gradient, which benefits policy optimization.Experiments on text-to-image generation benchmarks show that RebRL achieves state-of-the-art performance on GenEval and improves human preference score by up to 3.40 while effectively reducing training steps by ~40\%.