PinPoint：面向显式负样本、多图查询与改写测试的组合图像检索评测
PinPoint: Evaluation of Composed Image Retrieval with Explicit Negatives, Multi-Image Queries, and Paraphrase Testing
为组合图像检索提供了带显式难负样本、多正确答案、指令改写与多图查询的评测基准，并给出一个可直接套用到任意现有检索系统上的免训练 MLLM 重排序方案，可用于诊断和缓解假阳性与指令敏感性问题。
原文依据 we propose a training-free reranking method based on an off-the-shelf MLLM that can be applied to any existing system
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论组合图像检索（CIR）已取得显著进展，但现有基准仅限于单一标准答案，且缺乏评估假阳性规避、鲁棒性与多图推理所需的标注。我们提出 PinPoint，一个全面的真实世界基准，包含 7,635 条查询与跨 23 个查询类别的 329K 条相关性判断。PinPoint 通过提供以下内容推动该领域发展：（1）多个正确答案（平均每条查询 9.1 个）；（2）显式难负样本；（3）每条查询六个指令改写版本以用于鲁棒性测试；（4）多图组合支持（13.4% 的查询）；（5）用于公平性评估的人口统计元数据。基于对 4 种主要范式下 20 多种方法的分析，我们揭示了三个显著缺陷：最佳方法虽达到 28.5% 的 mAP@10，但仍有 9% 的时间检索到不相关结果（难负样本）；最佳模型在不同改写版本间还表现出 25.1% 的性能波动，表明当前 CIR 技术仍有显著提升空间；多图查询在不同方法上性能下降 40% 至 70%。为克服我们的评测框架所揭示的这些新问题，我们提出一种基于现成 MLLM 的免训练重排序方法，可应用于任何现有系统以弥合这一差距。我们发布了完整数据集，包括全部图像、查询、标注、检索索引与基准测试代码。
英文原文摘要 1443 字符
Composed Image Retrieval (CIR) has made significant progress, yet current benchmarks are limited to single ground-truth answers and lack the annotations needed to evaluate false positive avoidance, robustness and multi-image reasoning. We present PinPoint, a comprehensive real world benchmark with 7,635 queries and 329K relevance judgments across 23 query categories. PinPoint advances the field by providing: (1) multiple correct answers (averaging 9.1 per query) (2) explicit hard negatives, (3) six instruction paraphrases per query for robustness testing, (4) multi-image composition support (13.4% of queries), and (5) demographic metadata for fairness evaluation. Based on our analysis of 20+ methods across 4 different major paradigms, we uncover three significant drawbacks: The best methods while achieving mAP@10 of 28.5%, still retrieves irrelevant results (hard negatives) 9% of the time. The best models also exhibit 25.1% performance variation across paraphrases, indicating significant potential for enhancing current CIR techniques. Multi-image queries performs 40 to 70% worse across different methods. To overcome these new issues uncovered by our evaluation framework, we propose a training-free reranking method based on an off-the-shelf MLLM that can be applied to any existing system to bridge the gap. We release the complete dataset, including all images, queries, annotations, retrieval index, and benchmarking code.