Multi-Crit：面向多元准则遵循能力的多模态评判者基准
Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
为视觉生成与多模态理解提供了一套以细粒度准则为条件的评判基准与三项新指标（多元准则遵循度、准则切换灵活性、准则级偏好冲突识别），可直接用于诊断和选择 LMM 作为视觉生成质量评估器，并揭示整体性评判微调无法泛化到准则级判断这一关键失效模式。
原文依据 critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型多模态模型（LMM）凭借强大的指令遵循能力以及与人类偏好的一致性，正越来越多地被用作多模态评估系统中的评判者。然而，它们遵循多样且细粒度评价准则的能力仍缺乏充分探索。我们构建了 Multi-Crit，这是一个用于评估多模态评判者遵循多元准则并给出可靠准则级判断能力的基准。Multi-Crit 同时覆盖开放式生成任务与可验证推理任务，通过严格的数据整理流程构建，收集了带有多种准则人工标注的困难回复对。该基准进一步引入三项新指标，用于系统评估多元准则遵循度、准则切换灵活性以及识别准则级偏好冲突的能力。对 25 个 LMM 的全面分析揭示：1）专有模型在保持对多元准则的一致遵循方面仍存在困难，在开放式评估中尤为明显；2）开源模型在灵活遵循多样准则方面落后更多；3）使用整体性判断信号进行评判者微调虽能增强视觉定位，却无法泛化到多元准则级判断。围绕推理微调、测试时扩展以及开源与专有模型之间的边界一致性所做的进一步分析，进一步探查了当前多模态评判者的能力边界。作为一项开创性研究，Multi-Crit 为构建可靠且可调控的多模态 AI 评估奠定了基础。
英文原文摘要 1559 字符
Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.