VOLD：通过在线策略蒸馏将推理能力从大语言模型迁移到视觉语言模型
VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation
把纯文本 LLM 的推理能力通过 on-policy 蒸馏迁移到 VLM，视觉研究者可直接复用其「SFT 冷启动对齐 + GRPO 在线蒸馏」流程来缓解图文推理数据稀缺问题。
原文依据 VOLD combines reinforcement learning via Group Relative Policy Optimization (GRPO) with on-policy distillation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论训练视觉语言模型（VLM）进行复杂推理仍是一项具有挑战性的任务，原因之一是高质量图文推理数据的稀缺。相反，基于文本的推理资源丰富且可扩展，但如何将其用于 VLM 推理仍是一个开放问题。为解决该问题，我们提出 VOLD，一个将纯文本教师模型的推理能力迁移到 VLM 学生模型的框架。为此，VOLD 将基于组相对策略优化（GRPO）的强化学习与在线策略蒸馏相结合，使学生模型的推理轨迹能够接受教师模型的引导，相比单独使用 GRPO 取得显著增益。我们进一步表明，在此场景下冷启动对齐对于在线训练阶段的有效迁移至关重要，若教师与学生之间缺乏充分的分布对齐，在线策略蒸馏无法提供有意义的引导。我们在包括 MMMU-Pro、MathVision、MathVista 和 LogicVista 在内的多个基准上评估 VOLD，结果表明 VOLD 显著优于基线模型，并以一定幅度超越当前最优方法。我们的消融实验表明，在使用纯文本教师进行在线策略蒸馏时，通过 SFT 实现冷启动对齐十分重要。
英文原文摘要 1310 字符
Training vision-language models (VLMs) for complex reasoning remains a challenging task, i.a. due to the scarcity of high-quality image-text reasoning data. Conversely, text-based reasoning resources are abundant and scalable, but it is still an open question how to leveraging them for VLM reasoning. To address this problem, we propose VOLD, a framework to transfer reasoning capabilities from text-only teacher models to VLM student models. To this end, VOLD combines reinforcement learning via Group Relative Policy Optimization (GRPO) with on-policy distillation, which allows the student reasoning traces to be guided by the teacher model, resulting in a significant gain over using GRPO alone.We further show that a cold-start alignment is essential for an effective transfer during the online training phase in this scenario and that without sufficient distributional alignment between teacher and student, on-policy distillation fails to provide meaningful guidance.We evaluate VOLD across diverse benchmarks including MMMU-Pro, MathVision, MathVista, and LogicVista, showing that VOLD outperforms the baseline model significantly and improves over the state of the art by a margin.Our ablation shows the importance of a cold-start alignment via SFT for on-policy distillation with a text-only teacher