少做多得：扩散模型强化学习微调需要每一步优化吗？
Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?
提出一种自适应确定RL微调最佳时间窗口的插件方法，可减少计算成本并提升生成质量，为扩散模型RL微调提供高效策略。
原文依据 we propose \ourmethod , an RL-enhanced plug-in that improves generation quality while reducing computational cost.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩散模型在图像生成方面取得了显著成功，但其目标通常局限于重建，难以直接与人类偏好对齐。强化学习（RL）通过使用显式奖励信号优化模型，为解决这一问题提供了有前景的方法。然而，大多数研究在整个去噪过程中应用RL，这不仅计算成本高，而且往往会削弱偏好对齐，即“做得多但收获少”。我们观察到，RL微调的影响在不同去噪阶段差异显著。在早期阶段，图像结构不稳定且远离最终奖励信号，在此阶段应用RL会导致奖励延迟和动作-奖励不匹配，从而产生高方差和低效更新。相反，在后期阶段，奖励增益饱和，持续训练容易过拟合局部细节，加剧奖励黑客。为应对这些挑战，我们提出了\ourmethod，一种RL增强插件，在提高生成质量的同时降低计算成本。具体而言，\ourmethod通过感知去噪过程中的结构演变和语义一致性，自适应地确定RL的最佳时机，并在去噪收敛且奖励增益饱和时动态终止训练。因此，它实现了罕见的“双重收益”：计算成本降低的同时性能显著提升。从熵角度的理论分析和大量实验验证了我们的主张：与最优方法相比，\ourmethod在性能上提升了xx%，同时将计算成本降低了xx%。
英文原文摘要 1694 字符
Diffusion models have achieved outstanding success in image generation, yet their objectives are often limited to reconstruction, making it difficult to align with human preferences directly. Reinforcement learning (RL) offers a promising approach to address this by optimizing models using explicit reward signals. However, most studies apply RL across the entire denoising process, which is both computationally expensive and tends to weaken preference alignment, i.e., doing more but achieving less. We observe that the impact of RL fine-tuning varies significantly across denoising stages. In the early stage, image structures are unstable and distant from the final reward signal. Applying RL at this stage leads to delayed rewards and action-reward mismatching, resulting in high variance and inefficient updates. Conversely, in the later stage, reward gains saturate, and continued training tends to overfit local details, intensifying reward hacking. To tackle these challenges, we propose \ourmethod , an RL-enhanced plug-in that improves generation quality while reducing computational cost. Specifically, \ourmethod adaptively identifies the optimal timing for RL by perceiving the structural evolution and semantic consistency during denoising, and dynamically terminates training once the denoising converges and reward gains saturate. As a result, it achieves a rare `dual benefit': a reduction in computational costs alongside a significant performance improvement. Theoretical analysis from an entropy perspective and extensive experiments verify our claims: compared with state-of-the-art methods, \ourmethod improves performance by xx% while cutting computational cost by xx%.