SketchVL：通过细粒度信用分配的策略优化用于图表理解及其他任务
SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More
把中间推理步骤以「在图上画标记」的形式外化并回灌给模型，再用过程奖励模型对每一步打分做细粒度信用分配，这套「视觉标注式思维链 + 步级 RL」可直接迁移到需要精确空间定位的视觉推理任务（图表、几何、文档、计数）。
原文依据 drawing its intermediate reasoning steps as markers on the image and feeding the annotated image back to itself
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论图表是复杂数据的高密度视觉载体，也是信息提取与分析的媒介。由于需要精确且复杂的视觉推理，自动化图表理解对现有多模态大语言模型（MLLM）构成了重大挑战。许多经强化学习（RL）训练的 MLLM 面临信用分配的难题。其优势估计通常在轨迹层面进行，无法区分单条生成回复内部正确与错误的推理步骤。为应对这一局限，我们提出 SketchVL，一种新型 MLLM，采用 FinePO 进行优化，后者是一种为每条轨迹内部细粒度信用分配而设计的新 RL 算法。SketchVL 的方法是将其中间推理步骤作为标记绘制在图像上，并把标注后的图像重新输入自身，从而构建稳健的多步推理过程。在训练中，FinePO 算法利用细粒度过程奖励模型（FinePRM）对轨迹中的每个绘制动作打分，从而精确地为每一步分配信用。该机制使 FinePO 在轨迹整体成功时更有力地奖励正确 token，而在轨迹整体次优时更重地惩罚错误 token，从而实现细粒度的强化信号。实验表明，SketchVL 学会使其步级行为与 FinePRM 对齐，在图表数据集、自然图像数据集和数学任务上相较其基座模型平均获得 7.23% 的性能提升，为训练强大的推理模型提供了有前景的新方向。
英文原文摘要 1657 字符
Charts are high-density visual carriers of complex data and medium for information extraction and analysis. Due to the need for precise and complex visual reasoning, automated chart understanding poses a significant challenge to existing Multimodal Large Language Models (MLLMs). Many MLLMs trained with reinforcement learning (RL) face the challenge of credit assignment. Their advantage estimation, typically performed at the trajectory level, cannot distinguish between correct and incorrect reasoning steps within a single generated response. To address this limitation, we introduce SketchVL, a novel MLLM that optimized with FinePO, a new RL algorithm designed for fine-grained credit assignment within each trajectory. SketchVL's methodology involves drawing its intermediate reasoning steps as markers on the image and feeding the annotated image back to itself, creating a robust, multi-step reasoning process. During training, the FinePO algorithm leverages a Fine-grained Process Reward Model (FinePRM) to score each drawing action within a trajectory, thereby precisely assigning credit for each step. This mechanism allows FinePO to more strongly reward correct tokens when a trajectory is globally successful, and more heavily penalize incorrect tokens when the trajectory is globally suboptimal, thus achieving fine-grained reinforcement signals. Experiments show that SketchVL learns to align its step-level behavior with the FinePRM, achieving an average performance gain of 7.23% over its base model across chart datasets, natural image datasets, and mathematics, providing a promising new direction for training powerful reasoning models.