TableMix：从数据中心视角增强多模态大语言模型的表格推理能力
TableMix: Enhancing Multimodal Table Reasoning in MLLMs from a Data-Centric Perspective
对做多模态表格理解或文档 VLM 的研究者，这篇给出了一个可复用的数据配方：把多模态表格数据、纯文本数学推理数据和简单多模态感知数据按比例混合，再用难度感知的 GRPO 奖励塑形，缓解预训练造成的推理能力退化；这套数据混合与奖励设计思路可以直接迁移到图表问答、文档理解等视觉推理任务的后训练中。
原文依据 TableMix constructs a hybrid dataset that combines: (1) multimodal table reasoning data to improve task-specific reasoning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）的最新进展使基于视觉表格输入的表格推理取得了可观进步。尽管这类模型能够捕捉颜色、布局等丰富的视觉线索，其表现仍逊于纯文本模型。我们认为，一项主要限制在于预训练过程，它无意中削弱了模型的内在推理能力，进而阻碍了强化微调在表格推理任务上的效果。本文提出 TableMix，一个从数据中心视角应对该挑战的新框架。TableMix 的核心是一种有原则的数据混合策略。具体而言，TableMix 构建了一个混合数据集，融合了：（1）多模态表格推理数据，以提升任务特定的推理能力；（2）纯文本数学推理数据，以恢复模型的逻辑能力；（3）简单的多模态感知数据，以保持视觉定位能力。考虑到混合数据难度的非均匀性，我们进一步提出难度感知奖励塑形（DRS）机制，使组相对策略优化（GRPO）算法能够对简单问题自适应地奖励简洁推理，同时对复杂问题鼓励更细致的推理，从而减少冗余计算与错误。大量实验表明，TableMix 显著增强了 MLLMs 的推理能力，优于强多模态基线，甚至可与最优的纯文本模型相媲美。
英文原文摘要 1518 字符
Recent advances in Multimodal Large Language Models (MLLMs) have enabled promising progress in table reasoning from visual table inputs. Despite their ability to capture rich visual cues such as color and layout, MLLMs still underperform compared to text-only models.We argue that a major limitation lies in the pre-training process, which inadvertently weakens the model's intrinsic reasoning ability and consequently hinders the effectiveness of reinforcement fine-tuning on table reasoning tasks.In this paper, we introduce TableMix, a novel framework that tackles this challenge from a data-centric perspective. At the core of TableMix is a principled data mixing strategy. Specifically, TableMix constructs a hybrid dataset that combines: (1) multimodal table reasoning data to improve task-specific reasoning, (2) text-only mathematical reasoning data to revive the model's logical competence, and (3) simple multimodal perception data to preserve visual grounding.Recognizing the non-uniform difficulty of mixed data, we further propose a Difficulty-Aware Reward Shaping (DRS) mechanism, which enables the Group Relative Policy Optimization (GRPO) algorithm to adaptively reward concise reasoning for easy problems while encouraging more elaborate reasoning for complex ones, thereby reducing redundant computation and errors.Extensive experiments show that TableMix markedly enhances the reasoning ability of MLLMs, outperforming strong multimodal baselines and even rivaling state-of-the-art text-only models.