GTR-Turbo：合并检查点竟是智能体 VLM 训练中的免费教师
GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training
对做多模态智能体 RL 的人，这篇给出一个免特权教师信号的方案：把 RL 训练过程中产生的多个 checkpoint 权重合并成一个「免费教师」，再用 SFT 或软 logit 蒸馏回灌到策略上，可直接替换 GTR/On-Policy Distillation 里对 GPT、Gemini 的调用，同时缓解熵坍缩。
原文依据 merges the weights of checkpoints produced during ongoing RL training and then uses the resulting merged model as a "free" teacher
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论面向基于视觉语言模型（VLM）构建的多模态智能体，多轮强化学习（RL）受稀疏奖励与长程信用分配问题制约。近期方法通过查询提供步骤级反馈的教师来稠密化奖励，例如引导式思维强化（GTR）与在线策略蒸馏，但这些方法依赖昂贵且往往需要特权访问的模型作为教师，限制了其实用性与可复现性。我们提出 GTR-Turbo，这是对 GTR 的高效升级，无需在昂贵教师模型上训练或查询该教师即可达到与之相当的性能。具体而言，GTR-Turbo 将正在进行的 RL 训练过程中产生的多个检查点的权重进行合并，然后把得到的合并模型作为「免费」教师，通过监督微调或软 logit 蒸馏来指导后续的 RL。该设计消除了对特权 VLM（如 GPT 或 Gemini）的依赖，缓解了此前工作中观察到的「熵坍缩」现象，并保持训练稳定。在多种视觉智能体任务上，GTR-Turbo 将基线模型的准确率提升 10-30%，同时相对 GTR 将实际训练时间缩短 50%、计算成本降低 60%。
英文原文摘要 1161 字符
Multi-turn reinforcement learning (RL) for multi-modal agents built upon vision-language models (VLMs) is hampered by sparse rewards and long-horizon credit assignment. Recent methods densify the reward by querying a teacher that provides step-level feedback, e.g., Guided Thought Reinforcement (GTR) and On-Policy Distillation, but rely on costly, often privileged models as the teacher, limiting practicality and reproducibility. We introduce GTR-Turbo, a highly efficient upgrade to GTR that matches its performance without training on or querying an expensive teacher model. Specifically, GTR-Turbo merges the weights of checkpoints produced during ongoing RL training and then uses the resulting merged model as a "free" teacher to guide subsequent RL via supervised fine-tuning or soft logit distillation. This design removes dependence on privileged VLMs (e.g., GPT or Gemini), mitigates the "entropy collapse" observed in prior work, and maintains stable training. Across diverse visual agentic tasks, GTR-Turbo improves the accuracy of the baseline model by 10-30% while reducing wall-clock training time by 50% and compute cost by 60% relative to GTR.