FINER：多模态大语言模型在细粒度负查询下产生幻觉
FINER: MLLMs Hallucinate under Fine-grained Negative Queries
FINER 提供了针对细粒度负查询（多物体、多属性、多关系）的幻觉基准与 DPO 微调数据构造方式，可直接用于诊断和缓解自研 MLLM 在细粒度视觉判别上的幻觉，其负查询构造思路也可迁移到 captioning、VQA 的数据筛选与偏好对构建。
原文依据 MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）难以应对幻觉问题，尤其是在细粒度查询下，而现有基准多聚焦于粗粒度的图像相关问题，对这一挑战覆盖不足。我们提出细粒度负查询（FINER），并配套两个基准：FINER-CompreCap 与 FINER-DOCCI。借助 FINER，我们从四种设定分析幻觉：多物体、多属性、多关系以及“什么”类问题。我们的基准揭示：当细粒度不匹配与图像中真实存在的元素同时出现时，MLLMs 会产生幻觉。为此，我们提出 FINER-Tuning，在受 FINER 启发的数据上利用直接偏好优化（DPO）。用 FINER-Tuning 微调四个前沿 MLLM，在我们基准的幻觉指标上最高取得 24.2% 的提升（InternVL3.5-14B），同时改善了在八个现有幻觉测试集上的表现，并在六个基准上增强了通用多模态能力。代码、基准与模型见 https://explainableml.github.io/finer-project/。
英文原文摘要 1096 字符
Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coarse image-related questions. We introduce **FI**ne-grained **NE**gative que**R**ies (**FINER**), alongside two benchmarks: **FINER-CompreCap** and **FINER-DOCCI**. Using FINER, we analyze hallucinations across four settings: multi-object, multi-attribute, multi-relation, and "what" questions. Our benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image. To address this, we propose **FINER-Tuning**, leveraging Direct Preference Optimization (DPO) on FINER-inspired data. Finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2% gains (InternVL3.5-14B) on hallucinations from our benchmarks, while simultaneously improving performance on eight existing hallucination suites, and enhancing general multimodal capabilities across six benchmarks. Code, benchmark, and models are available at https://explainableml.github.io/finer-project/.