面向视觉语言模型的统一生成与自验证：基于优势解耦的偏好优化
Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization
把生成与自验证统一进单个 VLM 策略，用解耦优势与词元掩码隔离梯度，可在不额外训练验证模型的前提下实现并行测试时扩展，直接降低多模态推理的推理成本；其偏好验证奖励设计可迁移到任意需要自洽性打分的视觉语言任务。
原文依据 ADPO achieves up to +34.1% higher verification AUC and -53.5% lower inference time
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论并行测试时扩展通常需要分别训练生成模型与验证模型，带来高昂的训练与推理成本。本文提出优势解耦偏好优化（ADPO），这是一个统一的强化学习框架，在单一策略内联合学习答案生成与自验证。ADPO 引入两项创新：提升验证能力的偏好验证奖励，以及使生成与验证协同优化的解耦优化机制。具体而言，偏好验证奖励以正样本与负样本的平均验证分数作为判定阈值，当预测正确性与答案正确性一致时给出正向反馈。同时，优势解耦优化分别为生成与验证计算各自的优势，施加词元掩码以隔离梯度，并组合掩码后的 GRPO 目标，从而在保持生成质量的同时校准验证分数。ADPO 的验证 AUC 最高提升 34.1%，推理时间降低 53.5%，并在 MathVista/MMMU 上取得 +2.8%/+1.4% 的准确率提升，在 ReasonSeg 上提升 1.9 cIoU，在 AndroidControl/GUI Odyssey 上分别提升 1.7%/+1.0% 的步骤成功率。代码见 https://github.com/ZJUSCL/ADPO。
英文原文摘要 1308 字符
Parallel test-time scaling typically trains separate generation and verification models, incurring high training and inference costs. We propose Advantage Decoupled Preference Optimization (ADPO), a unified reinforcement learning framework that jointly learns answer generation and self-verification within a single policy. ADPO introduces two innovations: a preference verification reward improving verification capability and a decoupled optimization mechanism enabling synergistic optimization of generation and verification. Specifically, the preference verification reward computes mean verification scores from positive and negative samples as decision thresholds, providing positive feedback when prediction correctness aligns with answer correctness. Meanwhile, the advantage decoupled optimization computes separate advantages for generation and verification, applies token masks to isolate gradients, and combines masked GRPO objectives, preserving generation quality while calibrating verification scores. ADPO achieves up to +34.1% higher verification AUC and -53.5% lower inference time, with significant gains of +2.8%/+1.4% accuracy on MathVista/MMMU, +1.9 cIoU on ReasonSeg, and +1.7%/+1.0% step success rate on AndroidControl/GUI Odyssey. Code is available at https://github.com/ZJUSCL/ADPO.