VinQA：面向真实世界多模态文档问答的视觉元素交错长答案生成
VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
提出将文档页面中的视觉元素（表格、图表、照片等）与文本交错生成长篇答案的任务，并系统比较了整页编码与模态编码两种视觉元素输入方式，为多模态文档理解与生成提供了可复用的数据构建、编码方案和评测框架（M-GroSE、Visual Source F1）。
原文依据 We introduce VinQA, a dataset designed for long-form answer generation where cited visual elements are explicitly interleaved with their supporting text
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论真实世界文档将文本与表格、图表、照片和示意图以多样布局组合在一起，然而现有多模态大语言模型（MLLM）的文档问答研究主要产生纯文本回答，未能充分利用这些视觉元素。我们提出 VinQA，一个面向长答案生成的数据集，其中被引用的视觉元素与其支撑文本显式交错，并锚定在相关文档页面上。为支持该任务，我们研究了两种将原始文档页面图像输入 MLLM 的编码方法及其视觉元素引用机制：(1) 页面编码，直接编码带有视觉元素边界框的整页图像，并将这些框定区域视为可引用单元；(2) 模态编码，解析每页以提取文本并裁剪视觉元素，分别编码，并将这些裁剪元素作为可引用单元。在实验中，我们提出 M-GroSE，一个扩展 GroUSE 的多模态评估框架，从完整性、答案相关性、忠实性和不可回答性四个维度评估此类答案。我们还报告视觉来源 F1 以直接衡量视觉引用准确性。尽管专有前沿模型在 VinQA 测试集上仍取得最佳总体分数，但在 VinQA 训练集上微调开源 Qwen2.5-VL 模型可显著提升其性能并大幅缩小这一差距。对于长文本、多视觉元素和多样视觉引用需求的复杂文档，模态编码最初比页面编码更稳健。然而，在 VinQA 上训练后，页面编码达到相当的性能水平，表明即使没有模态编码中使用的显式解析，它也能有效竞争。最后，基于 MLLM 的评判器 Visual G-Eval 确认，微调后的模型将视觉元素插入语义适当的位置，并配有忠实的支撑文本。
英文原文摘要 1984 字符
Real-world documents combine text with tables, charts, photographs, and diagrams arranged in diverse layouts, yet existing research on multimodal large language models (MLLMs) for document QA predominantly produces text-only responses, underutilizing these visual elements. We introduce VinQA, a dataset designed for long-form answer generation where cited visual elements are explicitly interleaved with their supporting text and grounded in relevant document pages. To support this task, we study two encoding methods for feeding raw document page images into an MLLM, along with their visual-element citation mechanisms: (1) Page Encoding, which directly encodes full-page images with bounding boxes of visual elements and treats these boxed regions as citable units; and (2) Modality Encoding, which parses each page to extract text and crop visual elements, encodes them separately, and uses these cropped elements as citable units. In our experiments, we propose M-GroSE, a multimodal evaluation framework extending GroUSE to assess such answers along four dimensions: completeness, answer relevancy, faithfulness, and unanswerability. We additionally report Visual Source F1 to directly measure visual citation accuracy. Although proprietary frontier models still achieve the best overall scores on the VinQA test split, fine-tuning open Qwen2.5-VL models on the VinQA training split substantially improves their performance and markedly narrows this gap. Modality Encoding is initially more robust than Page Encoding for complex documents with long text, many visual elements, and diverse visual citation requirements. After training on VinQA, however, Page Encoding reaches a comparable performance level, showing that it can compete effectively even without the explicit parsing used in Modality Encoding. Finally, Visual G-Eval, an MLLM-based judge, confirms that fine-tuned models insert visual elements at semantically appropriate positions with faithful supporting text.