CodeV：以图像代码实现忠实视觉推理的工具感知策略优化
CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
对做多模态视觉推理智能体的人，这篇提供了可复用的过程级奖励设计思路：把奖励直接定义在图像裁剪等工具输入/输出上而非思维链 token 上，并用可执行 Python 代码表示视觉工具，从而抑制奖励作弊、提升工具使用的证据一致性；其忠实性评测协议也可直接用于诊断自家 VLM 的"猜对答案但推理不忠实"问题。
原文依据 a process-level RL framework that augments GRPO with dense rewards defined directly on visual tool inputs and outputs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论具备智能体能力的视觉语言模型正越来越多地通过调用图像操作来训练"用图像思考"。然而，我们表明，较高的最终答案准确率往往掩盖了不忠实的视觉推理：模型可能在无关区域调用工具，或完全忽略工具输出，却仍能猜出正确答案。在本工作中，我们首先提出一种忠实性评测协议，用于衡量中间视觉工具输出（例如裁剪图像）是否真正包含所查询的证据。该协议揭示，近期的视觉智能体在视觉搜索基准上取得了较高的最终答案准确率，但忠实工具使用率却很低。随后我们提出 CodeV，一个基于代码的视觉智能体，采用工具感知策略优化（TAPO）训练。TAPO 是一个过程级强化学习框架，它在 GRPO 基础上增加了直接定义在视觉工具输入与输出上的稠密奖励，而非定义在思维链 token 上，从而使监督更易验证、也更不易受到奖励作弊的影响。CodeV 将视觉工具表示为可执行的 Python 代码，TAPO 仅依据问题和工具输出分配逐步奖励，鼓励既必要又与证据一致的工具使用。在 SFT+RL 两阶段流程中，CodeV 在相关视觉搜索基准上取得了有竞争力或更优的准确率，同时大幅提升了忠实工具使用率。除视觉搜索之外，CodeV 在一系列多模态推理与数学基准上也表现强劲，这表明对中间工具行为进行显式监督，对于构建可信的智能体式视觉推理系统至关重要。
英文原文摘要 1597 字符
Agentic vision-language models are increasingly trained to "think with images" by calling image operations. However, we show that high final-answer accuracy often hides unfaithful visual reasoning: models may invoke tools on irrelevant regions or ignore tool outputs entirely, yet still guess the correct answer. In this work, we first propose a faithfulness evaluation protocol that measures whether intermediate visual tool outputs (e.g., crops) actually contain the queried evidence. This reveals that recent visual agents achieve high final-answer accuracy but exhibit low rates of faithful tool-use on visual search benchmarks. We then introduce CodeV, a code-based visual agent trained with Tool-Aware Policy Optimization (TAPO). TAPO is a process-level RL framework that augments GRPO with dense rewards defined directly on visual tool inputs and outputs, rather than on chain-of-thought tokens, making supervision easier to verify and less susceptible to reward hacking. CodeV represents visual tools as executable Python code, and TAPO assigns step-wise rewards based solely on the question and tool output, encouraging both necessary and evidence-consistent tool use. In a two-stage SFT+RL pipeline, CodeV achieves competitive or superior accuracy while substantially increasing faithful tool-use rates on related visual search benchmarks. Beyond visual search, CodeV attains strong performance on a range of multimodal reasoning and math benchmarks, suggesting that explicitly supervising intermediate tool behavior is crucial for building trustworthy, agentic visual reasoning systems.