超越单张图像：面向相册级视觉语言理解的综合基准
Beyond Single Images: A Comprehensive Benchmark for Album-Level Vision-Language Understanding
为多图/相册级视觉语言理解提供了带用户意图标注的评测基准，可直接用于检验和训练 VLM 的跨图上下文聚合与意图对齐能力，例如在相册内做照片筛选、评分与分组。
原文依据 we introduce AlbumBench, the first comprehensive benchmark for automatic album organization
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论过去几十年间，得益于数字摄影技术的显著进步，自动相册组织得到了广泛研究。近期视觉语言模型（VLM）在多图理解上展现出强劲性能，使其成为自动化相册组织流程的天然候选方案。尽管 VLM 的多图理解能力已被广泛研究，但其在相册组织上的表现仍缺乏充分探索。为填补这一空白，我们提出 AlbumBench，这是首个面向自动相册组织的综合基准。具体而言，我们（1）将相册组织任务定义为：针对相册特定用户目标的照片筛选、依据用户意图满足程度进行的照片评分，以及在给定用户查询下进行的相册特定照片分组，后者要求对相册的上下文理解；（2）构建 AlbumBench，一个包含 641 个相册、共 27051 张图像、每张图像 5 条标注的基准数据集；（3）在 AlbumBench 上评估主流开源与专有 VLM。我们表明，由于 AlbumBench 要求理解相册上下文与用户意图，相比传统多图理解基准，它呈现出独特的挑战。我们的发现揭示了开源与专有 VLM 在相册组织任务上存在显著性能差距。尽管存在这一差距，即便是表现最好的专有模型有时也会在人类认为相对简单的任务上遇到困难。我们希望 AlbumBench 能成为统一相册组织研究的基础，并推动 VLM 在这些任务上的性能提升。
英文原文摘要 1613 字符
Automatic album organization has been studied extensively over the past decades due to significant progress in digital photography. Recent vision-language models (VLMs) have shown strong performance on multi-image understanding, making them natural candidates for automating album organization workflows. While VLMs' abilities in multi-image understanding have been widely studied, their performance on album organization remains underexplored. To bridge this gap, we introduce AlbumBench, the first comprehensive benchmark for automatic album organization. Specifically, we (1) define album organization tasks as photo selection for album-specific user objectives, photo rating according to how well user intents are fulfilled, and album-specific photo grouping given a user query which requires contextual understanding of the album; (2) establish AlbumBench, a benchmark dataset containing 27051 images across 641 albums with 5 annotations per image; and (3) evaluate mainstream open-source and proprietary VLMs on AlbumBench. We show that AlbumBench presents unique challenges compared to traditional multi-image understanding benchmarks due to its requirement for understanding album context and user intent. Our findings reveal a significant performance gap between open-source and proprietary VLMs on album organization tasks. Despite this gap, even the best-performing proprietary models sometimes struggle with tasks that humans find relatively easy. We hope that AlbumBench can serve as a foundation for unifying album organization research and motivate improvements in VLMs' performance on these tasks.