CGL:通过强化微调推进持续 GUI 学习
CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning
对做多模态 GUI 智能体或 VLM 智能体持续学习的人,这篇给出了 SFT 与 RL(GRPO)混合训练时避免灾难性遗忘的具体手段:用策略熵调节 SFT 比例,并把 SFT 梯度投影到 GRPO 锚框梯度上做梯度手术,可直接迁移到其他 VLM 智能体的增量适配场景。
原文依据 we further develop a specialized gradient surgery strategy. By projecting exploratory SFT gradients onto GRPO-based anchor gradients
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论图形用户界面(GUI)智能体受益于多模态大语言模型(MLLM)的最新进展,已取得显著发展。然而,由于 GUI 应用频繁更新,在 GUI 持续学习中适应新任务而不遗忘旧任务仍是一个开放问题。在本工作中,我们揭示出:监督微调(SFT)虽然有助于快速适应,却常常引发知识覆盖,而强化学习(RL)则表现出一种内在的韧性,能够保护先前的交互逻辑不被抹除。基于这一洞见,我们提出持续 GUI 学习(CGL)框架,通过增强 SFT 与 RL 之间的协同,动态平衡适应效率与技能保持。具体而言,我们引入一种由策略熵引导的 SFT 比例调整机制,以动态控制 SFT 与 RL 训练阶段之间的权重分配。为解决显式的梯度干扰,我们进一步设计了一种专门的梯度手术策略。通过将探索性的 SFT 梯度投影到基于 GRPO 的锚框梯度上,我们的方法显式裁剪了 SFT 梯度中与 GRPO 冲突的分量。在此基础上,我们建立了 AndroidControl-CL 基准,将 GUI 应用划分为不同的任务组,以有效模拟和评估持续 GUI 学习的性能。实验结果表明,我们提出的 CGL 框架在各类持续学习场景中均有效。基准、代码与模型将公开。
英文原文摘要 1576 字符
Graphical User Interface (GUI) Agents, benefiting from recent advances in multimodal large language models (MLLM), have achieved significant development. However, due to the frequent updates of GUI applications, adapting to new tasks without forgetting old tasks in GUI continual learning remains an open problem. In this work, we reveal that while Supervised Fine-Tuning (SFT) facilitates fast adaptation, it often triggers knowledge overwriting, whereas Reinforcement Learning (RL) demonstrates an inherent resilience that shields prior interaction logic from erasure. Based on this insight, we propose a Continual GUI Learning (CGL) framework that dynamically balances adaptation efficiency and skill retention by enhancing the synergy between SFT and RL. Specifically, we introduce an SFT proportion adjustment mechanism guided by policy entropy to dynamically control the weight allocation between the SFT and RL training phases. To resolve explicit gradient interference, we further develop a specialized gradient surgery strategy. By projecting exploratory SFT gradients onto GRPO-based anchor gradients, our method explicitly clips the components of SFT gradients that conflict with GRPO. On top of that, we establish an AndroidControl-CL benchmark, which divides GUI applications into distinct task groups to effectively simulate and evaluate the performance of continual GUI learning. Experimental results demonstrate the effectiveness of our proposed CGL framework across continual learning scenarios. The benchmark, code, and model will be made publicly available.