ChartR：评估图表问答中的推理准确性与鲁棒性
ChartR: Evaluating Reasoning Accuracy and Robustness in Chart Question Answering
对做多模态大模型视觉推理的人，它提供了把问题拆成 4-10 步子问题、并对同一图表施加模糊/加噪/水印/去标注四种视觉扰动来定位「哪一步开始错、错误如何向后传播」的评测协议与八项指标，可直接用于诊断自家模型是依赖文本线索还是真正读图。
原文依据 Each question is decomposed into 4-10 sub-questions covering key reasoning types, and each chart includes four visually perturbed variants
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论图表问答（CQA）基准是评估多模态大语言模型（MLLM）在视觉数据推理方面表现的关键手段。现有基准主要关注最终答案的正确性，忽略了中间推理步骤以及多步过程中错误的传播。为解决这一问题，我们提出 ChartR，一个旨在同时评估图表理解任务中推理准确性与鲁棒性的基准。每个问题被分解为 4 至 10 个子问题，覆盖关键推理类型；每张图表包含四种视觉扰动变体（模糊、加噪、加水印、去除标注），以系统评估鲁棒性。ChartR 包含 200 张基础图表、800 个变体、1,652 个问题以及 8,260 个图像-问题对。我们进一步提出一个包含八项指标的全面评估框架，用于评估推理链准确性、视觉扰动下的鲁棒性，并支持分析潜在的错误传播模式。在包括通用模型与图表专用模型在内的十二个 MLLM 上的实验揭示了推理可靠性低、早期步骤错误可能向后传播、数值提取是主要瓶颈，以及扰动下性能急剧下降等问题，凸显出模型对文本线索的依赖超过了对真实视觉理解的使用。
英文原文摘要 1305 字符
Chart Question Answering (CQA) benchmarks are critical for evaluating Multimodal Large Language Models (MLLMs) on visual data reasoning. Existing benchmarks focus mainly on final-answer correctness, ignoring intermediate reasoning steps and the propagation of errors in multi-step processes. To address this, we introduce ChartR, a benchmark designed to assess both the accuracy and robustness of reasoning in chart-understanding tasks. Each question is decomposed into 4-10 sub-questions covering key reasoning types, and each chart includes four visually perturbed variants (blurred, noise-added, watermark-added, annotation-removed) to systematically evaluate robustness. ChartR contains 200 base charts, 800 variants, 1,652 questions, and 8,260 image-question pairs. We further propose a comprehensive evaluation framework with eight metrics that evaluate reasoning-chain accuracy, robustness under visual perturbations, and enable analysis of potential error propagation patterns. Experiments on twelve MLLMs, including general-purpose and chart-specialized models, reveal low reasoning reliability, early-step errors that may propagate, value extraction as the primary bottleneck, and sharp performance drops under perturbations, highlighting reliance on textual cues over true visual understanding.