我们准备好将强化学习用于文本到3D生成了吗？一项渐进式探究
Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation
把 GRPO 这类 RL 后训练从 2D 图像生成迁移到文本到 3D 自回归生成，并给出奖励设计（多模态模型作奖励、人类偏好对齐）与 token 级优化的可复用经验；Hi-GRPO 的全局到局部层级奖励集成思路可直接用于任何分阶段生成的 3D/视频模型。
原文依据 we propose Hi-GRPO, which optimizes the global-to-local hierarchical 3D generation through dedicated reward ensembles
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论强化学习（RL）此前已被证明在大语言模型和多模态模型中有效，近期也被成功扩展用于增强二维图像生成。然而，由于三维物体具有更高的空间复杂度，需要全局一致的几何结构与细粒度的局部纹理，将 RL 应用于三维生成仍基本未被探索。这使得三维生成对奖励设计和 RL 算法显著敏感。为应对这些挑战，我们从多个维度首次对面向文本到三维自回归生成的 RL 进行了系统性研究。(1) 奖励设计：我们评估了奖励维度与模型选择，表明与人类偏好对齐至关重要，且通用多模态模型能为三维属性提供稳健的信号。(2) RL 算法：我们研究了 GRPO 的多种变体，凸显了 token 级优化的有效性，并进一步考察了训练数据与迭代次数的扩展规律。(3) 文本到三维基准：由于现有基准无法衡量三维生成模型中的隐式推理能力，我们提出了 MME-3DR。(4) 进阶 RL 范式：受三维生成天然层级结构的启发，我们提出 Hi-GRPO，通过专门的奖励集成来优化从全局到局部的层级式三维生成。基于上述洞见，我们开发了 AR3D-R1，这是首个经 RL 增强的文本到三维模型，从粗粒度形状逐步精化到纹理。我们希望本研究能为 RL 驱动的三维生成推理提供启示。代码已发布于 https://github.com/Ivan-Tang-3D/3DGen-R1。
英文原文摘要 1592 字符
Reinforcement learning (RL), earlier proven to be effective in large language and multi-modal models, has been successfully extended to enhance 2D image generation recently. However, applying RL to 3D generation remains largely unexplored due to the higher spatial complexity of 3D objects, which require globally consistent geometry and fine-grained local textures. This makes 3D generation significantly sensitive to reward designs and RL algorithms. To address these challenges, we conduct the first systematic study of RL for text-to-3D autoregressive generation across several dimensions. (1) Reward designs: We evaluate reward dimensions and model choices, showing that alignment with human preference is crucial, and that general multi-modal models provide robust signal for 3D attributes. (2) RL algorithms: We study GRPO variants, highlighting the effectiveness of token-level optimization, and further investigate the scaling of training data and iterations. (3) Text-to-3D Benchmarks: Since existing benchmarks fail to measure implicit reasoning abilities in 3D generation models, we introduce MME-3DR. (4) Advanced RL paradigms: Motivated by the natural hierarchy of 3D generation, we propose Hi-GRPO, which optimizes the global-to-local hierarchical 3D generation through dedicated reward ensembles. Based on these insights, we develop AR3D-R1, the first RL-enhanced text-to-3D model, expert from coarse shape to texture refinement. We hope this study provides insights into RL-driven reasoning for 3D generation. The code is released at https://github.com/Ivan-Tang-3D/3DGen-R1.