相同内容，不同答案：多模态大语言模型中的跨模态不一致性
Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
该论文提出的REST/REST+基准可用于系统评估多模态大语言模型的跨模态一致性，其发现视觉特征（文本颜色、分辨率）和视觉token数量影响模型性能，为设计更鲁棒的视觉语言模型提供了可复用的评测方法和分析视角。
原文依据 We introduce two new benchmarks REST and REST+ (Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in multimodal large language models (MLLMs).
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们引入了两个新的基准测试 REST 和 REST+（渲染等价压力测试），用于系统评估多模态大语言模型（MLLMs）中的跨模态不一致性。MLLMs 被训练为在同一嵌入空间中表示视觉和语言，然而它们无法在两种模态下执行相同的任务。我们的基准包含在三种模态（图像、文本、混合）中具有相同语义信息的样本，并且我们展示了最优的 MLLMs 无法在这些不同模态上进行一致的推理。我们评估了 15 个 MLLMs，发现模态不一致的程度差异很大，即使考虑了文本识别（OCR）的问题。无论是将文本渲染为图像还是将图像渲染为文本，都无法解决不一致性。即使 OCR 正确，我们发现视觉特征（文本颜色和分辨率，但不包括字体）以及视觉 token 的数量对模型性能有影响。最后，我们发现我们的不一致性分数与文本和图像之间的模态差距相关，突出了对跨模态不一致 MLLMs 的机制性解释。
英文原文摘要 1129 字符
We introduce two new benchmarks REST and REST+ (Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in multimodal large language models (MLLMs). MLLMs are trained to represent vision and language in the same embedding space, yet they cannot perform the same tasks in both modalities. Our benchmarks contain samples with the same semantic information in three modalities (image, text, mixed) and we show that state-of-the-art MLLMs cannot consistently reason over these different modalities. We evaluate 15 MLLMs and find that the degree of modality inconsistency varies substantially, even when accounting for problems with text recognition (OCR). Neither rendering text as image nor rendering an image as text solves the inconsistency. Even if OCR is correct, we find that visual characteristics (text colour and resolution, but not font) and the number of vision tokens have an impact on model performance. Finally, we find that our consistency score correlates with the modality gap between text and images, highlighting a mechanistic interpretation of cross-modal inconsistent MLLMs.