VisualOverload：探测视觉语言模型在真正密集场景中的视觉理解
VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
该基准揭示了当前VLM在密集场景下细节编码与推理的严重不足，为视觉编码器的高分辨率细节保留和推理模块的细粒度逻辑一致性提供了明确的评测靶点。
原文依据 even the best model (o3) out of 37 tested models only achieves 19.6% accuracy on our hardest test split
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基础视觉理解在最先进的视觉语言模型（VLM）中真的已经解决了吗？我们提出 VisualOverload，一个略有不同的视觉问答（VQA）基准，包含 2,720 个问答对，其真实答案私下保存。与以往通常关注近全局图像理解的 VQA 数据集不同，VisualOverload 挑战模型在密集（或过载）场景中执行简单的、无需知识的视觉任务。我们的数据集由公共领域绘画的高分辨率扫描组成，这些绘画中充满了多个人物、动作和展开的子情节，背景细节繁复。我们手动为这些图像标注了跨越六个任务类别的问答，以探查对场景的全面理解。我们假设当前基准高估了 VLM 的性能，而细节的编码与推理对它们来说仍然是一项具有挑战性的任务，尤其是当它们面对密集场景时。确实，我们观察到在 37 个测试模型中，即使是最好的模型（o3）在我们最难的测试划分上也仅达到 19.6% 的准确率，在所有问题上的总体准确率为 69.5%。除了全面的评估，我们还通过错误分析补充了基准，揭示了多种失败模式，包括计数能力不足、OCR 失败以及在复杂任务下显著的逻辑不一致。总之，VisualOverload 暴露了当前视觉模型的一个关键缺口，并为社区开发更好的模型提供了关键资源。
英文原文摘要 1543 字符
Is basic visual understanding really solved in state-of-the-art VLMs? We present VisualOverload, a slightly different visual question answering (VQA) benchmark comprising 2,720 question-answer pairs, with privately held ground-truth responses. Unlike prior VQA datasets that typically focus on near global image understanding, VisualOverload challenges models to perform simple, knowledge-free vision tasks in densely populated (or, overloaded) scenes. Our dataset consists of high-resolution scans of public-domain paintings that are populated with multiple figures, actions, and unfolding subplots set against elaborately detailed backdrops. We manually annotated these images with questions across six task categories to probe for a thorough understanding of the scene. We hypothesize that current benchmarks overestimate the performance of VLMs, and encoding and reasoning over details is still a challenging task for them, especially if they are confronted with densely populated scenes. Indeed, we observe that even the best model (o3) out of 37 tested models only achieves 19.6% accuracy on our hardest test split and overall 69.5% accuracy on all questions. Beyond a thorough evaluation, we complement our benchmark with an error analysis that reveals multiple failure modes, including a lack of counting skills, failure in OCR, and striking logical inconsistencies under complex tasks. Altogether, VisualOverload exposes a critical gap in current vision models and offers a crucial resource for the community to develop better models.