DEVA：面向视觉感知任务的多模态大语言模型微调
DEVA: Fine-tuning Multimodal Large Language Models for Visual Perception Tasks
对做视觉感知任务的 MLLM 微调者，这篇给出了 GRPO 训练动力学的失效分析（多样性下降、探索受限、奖励聚合次优），并给出可插拔的改进：基于流的训练目标提升多样性、全局熵正则扩大探索、以对齐度作为非可验证奖励配合调和聚合，可直接套用到检测/定位等视觉感知的 RL 微调流程中。
原文依据 we propose DEVA: a framework that enhances Diversity via a flow-based training objective, encourages broader policy Exploration through global entropic regularization
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论使用强化学习目标微调大语言模型已获得广泛关注，在标注数据有限的场景中尤其如此。得益于其在语言领域的成功，近期工作将基于强化学习的微调扩展到多模态任务。例如 Visual-RFT 将组相对策略优化（GRPO）用于在多个视觉感知基准上微调多模态大语言模型，相较标准监督微调取得了显著提升。然而，其研究范围受限于对强化学习适配策略较为狭窄的评估。在本工作中，我们在相同基准上引入新的基于强化学习的基线，并对 GRPO 的训练动力学进行更深入的分析，从而拓展了这一研究图景。我们识别出若干关键局限，例如生成多样性下降、策略探索受限，以及奖励构造与聚合不够理想。为解决这些问题，我们提出 DEVA：该框架通过基于流的训练目标提升多样性，通过全局熵正则鼓励更广泛的策略探索，并利用对齐度作为非可验证奖励并结合调和聚合。将 DEVA 应用于 GRPO 及其他强化学习方法后，在定量指标（提升 5 至 13 个点）与定性指标上均取得一致增益。我们进一步提供可视化、消融与分析，以揭示框架中各组成部分的贡献。
英文原文摘要 1444 字符
Fine-tuning large language models (LLMs) using reinforcement learning (RL) objectives has gained traction, especially in scenarios where labeled data is limited. Building on its success in the language domain, recent efforts have extended RL-based fine-tuning to multimodal tasks. Visual-RFT, for instance, applied Group Relative Policy Optimization (GRPO) to fine-tune multimodal LLMs (MLLMs) across various visual perception benchmarks, achieving notable improvements over standard supervised fine-tuning (SFT). However, its scope was limited by a narrow evaluation of RL adaptation strategies. In this work, we expand the landscape by introducing new RL-based baselines on the same benchmarks and conducting a deeper analysis of GRPO's training dynamics. We identify key limitations--such as reduced generation diversity, constrained policy exploration, and suboptimal reward formulation and aggregation. To address these, we propose DEVA: a framework that enhances Diversity via a flow-based training objective, encourages broader policy Exploration through global entropic regularization, and leverages alignment Volume as a non-verifiable reward combined with harmonic Aggregation. Applied to GRPO and other RL methods, DEVA delivers consistent gains in both quantitative (+5 to +13 points) and qualitative metrics. We further provide visualizations, ablations, and analyses to unpack the contributions of each component in our framework.