SO-Bench：多模态大语言模型的结构化输出评测
SO-Bench: A Structural Output Evaluation of Multimodal LLM
为视觉研究者提供了一个以 JSON schema 约束 MLLM 输出的评测与训练框架，可直接用于评估和提升模型在 UI、文档、图表等图像上的结构化信息抽取能力，也可作为 schema 约束解码或结构化微调的数据来源。
原文依据 there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLM）正越来越多地部署于真实世界的智能体场景中，在这些场景里输出不仅需要正确，还必须符合预定义的数据模式。尽管文本领域的结构化生成已取得进展，目前仍缺乏系统评估视觉输入上基于模式的信息抽取与推理的基准。在本工作中，我们通过精心设计的 SO-BENCH 基准，对 MLLM 的视觉结构化输出能力进行了全面研究。SO-BENCH 覆盖四个视觉领域，包括 UI 屏幕、自然图像、文档和图表，由超过 6.5K 个多样化的 JSON 模式以及 1.8K 个经人工核验质量、精心筛选的图像-模式对构建而成。对开源模型和前沿闭源模型的基准实验揭示了在预测准确且符合模式的输出方面持续存在的差距，凸显了对更好的多模态结构化推理的需求。除基准评测之外，我们还开展了训练实验，大幅提升了模型的结构化输出能力。我们将该基准公开于 https://github.com/apple/ml-sobench。
英文原文摘要 1153 字符
Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to pre-defined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-BENCH benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-BENCH is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced andfrontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We make the benchmark publicly available at https://github.com/apple/ml-sobench.