MarkushGrapher-2：端到端多模态化学结构识别
MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures
该工作将文本、图像与版面信息通过视觉-文本-版面编码器与光学化学结构识别视觉编码器联合编码，并采用两阶段训练策略融合，为文档图像中的多模态结构识别提供了可复用的编码与融合范式。
原文依据 the text, image, and layout information are jointly encoded through a Vision-Text-Layout encoder and an Optical Chemical Structure Recognition vision encoder
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从文档中自动提取化学结构对于化学文献的大规模分析至关重要。目前已开发出自动流程来分别识别以图形或文本形式表示的分子。然而，从多模态描述（Markush结构）中识别化学结构的方法在精度上仍然落后，无法用于大规模自动处理。在本工作中，我们提出了MarkushGrapher-2，一种端到端的文档中化学结构多模态识别方法。首先，我们的方法采用专用的OCR模型从化学图像中提取文本。其次，文本、图像和版面信息通过视觉-文本-版面编码器和光学化学结构识别视觉编码器进行联合编码。最后，通过两阶段训练策略有效融合所得编码，并用于自回归生成Markush结构的表示。为解决训练数据缺乏的问题，我们引入了一个自动流程来构建大规模真实世界Markush结构数据集。此外，我们提出了IP5-M，一个大型人工标注的真实世界Markush结构基准，旨在推动这一挑战性任务的研究。大量实验表明，我们的方法在多模态Markush结构识别上大幅优于最优模型，同时在分子结构识别上保持强劲性能。代码、模型和数据集将公开释放。
英文原文摘要 1567 字符
Automatically extracting chemical structures from documents is essential for the large-scale analysis of the literature in chemistry. Automatic pipelines have been developed to recognize molecules represented either in figures or in text independently. However, methods for recognizing chemical structures from multimodal descriptions (Markush structures) lag behind in precision and cannot be used for automatic large-scale processing. In this work, we present MarkushGrapher-2, an end-to-end approach for the multimodal recognition of chemical structures in documents. First, our method employs a dedicated OCR model to extract text from chemical images. Second, the text, image, and layout information are jointly encoded through a Vision-Text-Layout encoder and an Optical Chemical Structure Recognition vision encoder. Finally, the resulting encodings are effectively fused through a two-stage training strategy and used to auto-regressively generate a representation of the Markush structure. To address the lack of training data, we introduce an automatic pipeline for constructing a large-scale dataset of real-world Markush structures. In addition, we present IP5-M, a large manually-annotated benchmark of real-world Markush structures, designed to advance research on this challenging task. Extensive experiments show that our approach substantially outperforms state-of-the-art models in multimodal Markush structure recognition, while maintaining strong performance in molecule structure recognition. Code, models, and datasets will be released publicly.