ARM-Thinker：通过智能体工具使用与视觉推理强化多模态生成式奖励模型
ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
提出了一种可自主调用外部工具（如图像裁剪、文档检索）的多模态奖励模型，通过多阶段强化学习联合优化工具调用决策与判断准确性，为视觉语言模型的奖励建模提供了可验证证据和可解释性，可复用于需要细粒度视觉验证的视觉语言任务。
原文依据 We present ARM-Thinker, an Agentic multimodal Reward Model that autonomously invokes external tools (e.g., image cropping, doc page retrieval) to ground judgments in verifiable evidence
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论奖励模型对于使视觉语言系统与人类偏好对齐至关重要，然而当前方法存在幻觉、视觉定位能力弱以及无法使用工具进行验证等问题，限制了其在复杂多模态推理任务上的可靠性。我们提出 ARM-Thinker，一种智能体多模态奖励模型，能够自主调用外部工具（如图像裁剪、文档页面检索）将判断建立在可验证的证据之上，取代静态、非交互式的奖励评分。这使得模型能够验证细粒度视觉细节、交叉引用多页证据并验证推理主张，这些能力是现有奖励模型所不具备的。我们通过多阶段强化学习训练 ARM-Thinker，联合优化工具调用决策与判断准确性。为了评估智能体奖励建模，我们引入了 ARMBench-VL，包含三个基准，分别评估细粒度视觉定位（图像级工具）、多页文档理解（检索工具）和指令遵循（文本级验证）。ARM-Thinker 在奖励建模基准上平均提升 16.2%，在工具使用任务上提升 9.6%，并在多模态数学和逻辑推理基准上优于基线。我们的结果表明，智能体能力显著提升了奖励模型的准确性和可解释性。
英文原文摘要 1407 字符
Reward models are critical for aligning vision-language systems with human preferences, yet current approaches suffer from hallucination, weak visual grounding, and an inability to use tools for verification, limiting their reliability on complex multimodal reasoning tasks.We present **ARM-Thinker**, an **A**gentic multimodal **R**eward **M**odel that autonomously invokes external tools (e.g., image cropping, doc page retrieval) to ground judgments in verifiable evidence, replacing static, non-interactive reward scoring.This enables the model to verify fine-grained visual details, cross-reference multi-page evidence, and validate reasoning claims, which are capabilities absent in existing reward models.We train ARM-Thinker with multi-stage reinforcement learning, jointly optimizing tool-calling decisions and judgment accuracy.To evaluate agentic reward modeling, we introduce **ARMBench-VL**, comprising three benchmarks that assess fine-grained visual grounding (image-level tools), multi-page document understanding (retrieval tools), and instruction following (text-level verification).ARM-Thinker achieves +16.2% average improvement on reward modeling benchmarks, +9.6% on tool-use tasks, and outperforms baselines on multimodal math and logical reasoning benchmarks.Our results demonstrate that agentic capabilities significantly enhance both accuracy and interpretability of reward models.