面向图表识别的短语定位感知监督微调：基于侧路掩码注意力
Phrase-Grounding-Aware Supervised Fine-Tuning for Chart Recognition via Side-Masked Attention
把短语级视觉定位作为辅助监督信号插入 LLM 每一层 Transformer 的侧路掩码注意力中，为 VLM 微调提供区域级参考 logit；做文档/图表理解或需要细粒度区域对齐的 VLM 微调时可直接复用该侧路注意力设计。
原文依据 we introduce phrase-grounding-aware SFT via a Side-Masked Attention Module (SMAM) inserted into each transformer layer of the LLM
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期图表识别的进展主要由视觉语言模型（VLM）的监督微调（SFT）推动，该方法统一了多个相关任务，并得益于训练语料的多样化。与此同时，关于利用大语言模型（LLM）进行目标检测的研究表明，在 SFT 的同时联合训练短语定位能够增强模型的生成能力。受此启发，我们假设图表识别同样可以从短语定位中获益，即把文本短语与图表区域对齐。然而，由于缺乏相应数据集，这一设定仍未得到充分探索。在本工作中，我们通过一个插入到 LLM 每一层 Transformer 中的侧路掩码注意力模块（SMAM）来实现短语定位感知的 SFT。SMAM 在与每个短语对齐的标注区域内执行掩码注意力，以产生一个额外的 logit。该 logit 受到监督，并在微调过程中作为参考信号，与标准 SFT 目标一起引导 LLM 的输出预测。为支持该方法，我们还开发了一套自动化的短语到区域对齐生成流程，用于扩充现有数据集。实验表明，我们的方法有效地将短语定位融入面向图表识别的 VLM 微调之中。
英文原文摘要 1292 字符
Recent advances in chart recognition have been driven by supervised fine-tuning (SFT) of vision-language models (VLMs), which unify multiple related tasks, and by diversifying training corpora. In parallel, research on leveraging large language models (LLMs) for object detection has shown that jointly training phrase grounding alongside SFT enhances a model's generative capabilities. Inspired by this, we hypothesize that chart recognition can also benefit from phrase grounding, which aligns textual phrases with chart regions. However, this setting remains underexplored due to the lack of corresponding datasets. In this work, we introduce phrase-grounding-aware SFT via a Side-Masked Attention Module (SMAM) inserted into each transformer layer of the LLM. SMAM performs masked attention within the annotated region aligned with each phrase to produce an additional logit. This logit is supervised and serves as a reference signal to guide the LLM's output prediction during fine-tuning, alongside the standard SFT objective. To enable this approach, we also develop an automated pipeline for generating phrase-to-region alignments, which augments existing datasets. Experiments show that our method effectively incorporates phrase grounding into VLM fine-tuning for chart recognition.