面向强化微调的动态重要样本挖掘
Dynamic Important Example Mining for Reinforcement Finetuning
对做视觉推理模型（如多模态大模型、视觉问答）的研究者，DIEM 提供了一种在 RFT 阶段按梯度对齐动态估计样本边际贡献并做约束式批重加权的通用手段，可直接迁移到视觉推理数据的采样与课程设计上。
原文依据 a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论强化微调（RFT）正越来越多地被用于增强大模型的推理能力，但其效果受制于训练数据的选择与使用方式。多数以数据为中心的 RFT 方法依赖静态或启发式的样本选择，隐含地假设样本的价值在训练过程中是固定的。这忽视了策略学习的非平稳动态特性，可能导致次优的更新。我们提出动态重要样本挖掘（DIEM），一个原理清晰且完全自动化的框架，使数据利用在 RFT 全程保持自适应。DIEM 在每一步优化中集成两个组件：（i）一个梯度对齐的重要性估计器，高效地近似每个样本对策略改进的边际贡献；（ii）一个带约束的批重加权方案，在最大化总体效用的同时保持更新的梯度幅值以稳定优化。在多个推理基准上，DIEM 一致优于强静态与动态基线。代码将通过 https://github.com/hrtan/DIEM 发布。
英文原文摘要 1082 字符
Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.