正确的VAR强化学习：解决视觉自回归生成中的异步策略冲突
VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation
针对VAR模型在强化学习微调中因异构输入结构导致的异步策略冲突问题，提出稳定中间奖励、动态时间步重加权和掩码传播算法，为视觉自回归生成模型的RL对齐提供了可复用的技术方案。
原文依据 VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉生成领域主要由三种范式主导：自回归（AR）、扩散和视觉自回归（VAR）模型。与AR和扩散不同，VAR在其生成步骤中操作异构的输入结构，这会导致严重的异步策略冲突。这一问题在强化学习（RL）场景中尤为突出，导致训练不稳定和对齐效果欠佳。为解决此问题，我们提出了一种新颖的框架，通过显式管理这些冲突来增强组相对策略优化（GRPO）。我们的方法集成了三个协同组件：1）一个稳定的中间奖励，用于引导早期生成；2）一种动态时间步重加权方案，用于精确的信用分配；3）一种新颖的掩码传播算法，源自奖励反馈学习（ReFL）原理，旨在在空间和时间上隔离优化效果。我们的方法在样本质量和目标对齐方面相较于原始GRPO基线表现出显著提升，为VAR模型实现了稳健且有效的优化。我们的官方页面为 https://github.com/ByteVisionLab/NextFlow。
英文原文摘要 1169 字符
Visual generation is dominated by three paradigms: AutoRegressive (AR), diffusion, and Visual AutoRegressive (VAR) models. Unlike AR and diffusion, VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts. This issue becomes particularly acute in reinforcement learning (RL) scenarios, leading to unstable training and suboptimal alignment. To resolve this, we propose a novel framework to enhance Group Relative Policy Optimization (GRPO) by explicitly managing these conflicts. Our method integrates three synergistic components: 1) a stabilizing intermediate reward to guide early-stage generation; 2) a dynamic time-step reweighting scheme for precise credit assignment; and 3) a novel mask propagation algorithm, derived from principles of Reward Feedback Learning (ReFL), designed to isolate optimization effects both spatially and temporally. Our approach demonstrates significant improvements in sample quality and objective alignment over the vanilla GRPO baseline, enabling robust and effective optimization for VAR models. Our official page is https://github.com/ByteVisionLab/NextFlow.