面向多模态大语言模型的接地思维链
Grounded Chain-of-Thought for Multimodal Large Language Models
提出 Grounded Chain-of-Thought 任务与 MM-GCoT 基准，要求 MLLM 在逐步推理中显式输出视觉线索的 grounding 坐标，并引入答案-定位一致性指标，可用于诊断和缓解多模态模型在视觉空间推理中的幻觉问题。
原文依据 GCoT aims to improve the visual-spatial reasoning capabilities of MLLMs via recognizing and grounding the relevant visual cues step by step
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管取得了巨大进展，现有多模态大语言模型（MLLMs）在视觉空间推理方面仍然表现不佳，这极大地阻碍了其在具身智能等场景中的可信应用。为促进相关研究，本文提出了一项新的 MLLM 任务，称为接地思维链（Grounded Chain-of-Thought, GCoT）。与近期更关注视觉知识推理的视觉思维链研究不同，GCoT 旨在通过逐步识别并定位相关视觉线索来提升 MLLMs 的视觉空间推理能力，并以逐步的定位坐标作为直观依据。为便于开展该任务，我们还精心设计并构建了一个名为多模态接地思维链（MM-GCoT）的基准。此外，还引入了一套全面的一致性评估体系，包括答案准确率、定位准确率以及答案-定位一致性等指标。我们进一步在 12 个先进 MLLMs 上设计并开展了一系列实验，揭示了一些值得注意的发现：i. 大多数 MLLMs 在一致性评估中表现不佳，表明存在明显的视觉幻觉；ii. 视觉幻觉与参数规模和通用多模态性能并无直接关联；iii. 更大更强的 MLLM 并未更少受到该问题的影响。
英文原文摘要 1321 字符
Despite great progress, existing multimodal large language models (MLLMs) are still inferior in visual-spatial reasoning, which greatly impedes their trustworthy applications in scenarios such as Embodied AI. To facilitate the research, we propose a new MLLM task in this paper, called Grounded Chain-of-Thought (GCoT). Different from recent visual CoT studies, which focus more on visual knowledge reasoning, GCoT aims to improve the visual-spatial reasoning capabilities of MLLMs via recognizing and grounding the relevant visual cues step by step, which are also supported by step-vise grounding coordinates as the intuitive basis. To facilitate this task, we also carefully design and construct a benchmark called multimodal grounded chain-of-thought (MM-GCoT). Besides, a comprehensive consistency evaluation system is also introduced, including the metrics of answer accuracy, grounding accuracy and answer-grounding consistency. We further design and conduct a bunch of experiments on 12 advanced MLLMs, and reveal some notable findings: i. most MLLMs performs poorly on the consistency evaluation, indicating obvious visual hallucination; ii. visual hallucination is not directly related to the parameter size and general multimodal performance; iii. a larger and stronger MLLM is not less affected by this issue.