Circular-DPO：通过偏好反馈循环对齐多阶段 3D 生成模型
Circular-DPO: Aligning Multi-Stage 3D Generative Models via Preference Feedback Loop
将 DPO 扩展到多阶段 3D 生成流程，通过采样-解码构造跨阶段偏好对，绕过不可微操作实现端到端偏好对齐；质量感知加权机制可复用于其他带噪声偏好数据的对齐训练。
原文依据 we introduce Circular-DPO, which builds a preference feedback loop to align multi-stage 3D generation models to human preference.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多阶段生成模型因在不同阶段分别聚焦生成结构或纹理，在 3D 内容创作中展现出巨大潜力，但其输出往往难以与人类偏好对齐。应用对齐方法的关键瓶颈在于生成阶段之间存在不可微操作。这种断连使得施加于最终输出的偏好信号无法反向传播至关键的早期生成阶段，而简单的分阶段独立对齐又会导致纹理与几何不一致。为解决这一挑战，我们提出 Circular-DPO，通过构建偏好反馈循环将多阶段 3D 生成模型对齐至人类偏好。该方法首先应用直接偏好优化（DPO）精炼最终 3D 资产。随后，通过对优化后模型生成的资产进行采样与解码来构建新的偏好对。这些新形成的偏好对用于训练前序生成阶段，有效创建了一个桥接不可微鸿沟的反馈循环。此外，为增强对噪声数据的鲁棒性，我们引入质量感知加权机制，在训练中优先采用可靠的偏好对。实验表明，该方法通过实现整体性的多阶段优化，提升了生成 3D 内容与人类偏好的对齐程度。
英文原文摘要 1389 字符
Multi-stage generative models have shown great promise in 3D content creation due to focused generation of structure or texture in different stages, but their outputs often fail to align with human preferences. The key bottleneck to apply alignment methods is the presence of non-differentiable operations between generative stages.This disconnection stops preference signals applied to the final output from being backpropagated to the crucial, early stages of generation, while simple separated stage-wise alignment leads to texture-geometry inconsistency.To address this challenge, we introduce Circular-DPO, which builds a preference feedback loop to align multi-stage 3D generation models to human preference.Our method first applies Direct Preference Optimization (DPO) to refine the final 3D asset.We then construct new preference pairs by sampling and decoding the assets generated by the optimized model.These newly-formed pairs are used to train the preceding generative stage, effectively creating a feedback loop that bridges the non-differentiable gap. Furthermore, to enhance robustness against noisy data, we introduce a quality-aware weighting mechanism that prioritizes reliable preference pairs during training. Experiments demonstrate that our approach improves the alignment of generated 3D content with human preferences by enabling holistic, multi-stage optimization.