为什么RL比SFT泛化更好？基于数据视角的VLM后训练分析
Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training
提出基于样本难度的数据筛选策略（DC-SFT），可替代RL微调以更低成本提升VLM的OOD泛化，为视觉语言模型后训练提供数据侧改进思路。
原文依据 RL's generalization advantage arises from an implicit data filtering mechanism that inherently prioritizes medium-difficulty training samples.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大规模视觉语言模型（VLM）的后训练揭示了一个显著的泛化差距：与监督微调（SFT）训练的模型相比，使用强化学习（RL）微调的模型始终在分布外（OOD）性能上表现更优。本文从数据角度为这一现象提供解释，认为RL的泛化优势源于一种隐式的数据筛选机制，该机制本质上优先选择中等难度的训练样本。为验证这一假设，我们系统评估了SFT模型在不同难度训练数据集上的OOD泛化能力。结果证实数据难度是关键因素，并揭示在困难样本上训练会显著降低OOD性能。基于这一发现，我们提出了难度筛选SFT（DC-SFT），一种根据样本难度显式过滤训练集的简单方法。实验表明，DC-SFT不仅在标准SFT基础上大幅提升了OOD泛化能力，而且超越了基于RL的训练性能，同时提供了更高的稳定性和计算效率。这项工作从数据角度解释了VLM中的OOD泛化差距，并建立了一条实现鲁棒泛化的更高效路径。代码见 https://github.com/byyx666/DC-SFT。
英文原文摘要 1393 字符
Post-training of large-scale Vision-Language Models (VLMs) reveals a pronounced generalization gap: models fine-tuned with Reinforcement Learning (RL) consistently achieve superior out-of-distribution (OOD) performance compared to those trained with Supervised Fine-Tuning (SFT). This paper posits a data-centric explanation for this phenomenon, contending that RL's generalization advantage arises from an implicit data filtering mechanism that inherently prioritizes medium-difficulty training samples. To test this hypothesis, we systematically evaluate the OOD generalization of SFT models across training datasets of varying difficulty levels. Our results confirm that data difficulty is a critical factor, revealing that training on hard samples significantly degrades OOD performance. Motivated by this finding, we introduce Difficulty-Curated SFT (DC-SFT), a straightforward method that explicitly filters the training set based on sample difficulty. Experiments show that DC-SFT not only substantially enhances OOD generalization over standard SFT, but also surpasses the performance of RL-based training, all while providing greater stability and computational efficiency. This work offers a data-centric account of the OOD generalization gap in VLMs and establishes a more efficient pathway to achieving robust generalization. Code is available at https://github.com/byyx666/DC-SFT.