CaptionQA：你的描述能像图像本身一样有用吗？
CaptionQA: Is Your Caption as Useful as the Image Itself?
把「图像描述能否替代图像」转化为可用 LLM 直接作答的多选题效用探针，为评估 captioning 与多模态描述模型提供了可复用的下游效用评测协议，也提示视觉研究者关注 caption 在检索、推荐、agent 流水线中的信息保真度。
原文依据 Can captions stand-in for images in real downstream tasks?
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论图像描述在多模态系统（如检索、推荐、多步智能体推理流水线）中充当视觉内容的高效替代物。然而当前的评测实践忽略了一个根本问题：描述能否在下游任务中替代图像？我们提出基于效用的基准 CaptionQA，用于评估模型生成的描述，其质量以对下游任务的支持程度来衡量。CaptionQA 是一个可扩展、依赖领域的基准，覆盖自然、文档、电商与具身智能四个领域，每个领域都有细粒度分类体系（25 个顶层类别与 69 个子类别），用于识别对领域特定任务有用的信息。CaptionQA 构建了 33,027 道密集标注的多选题（平均每张图像 50.3 道），这些题目明确需要视觉信息才能作答，从而全面探测描述的效用。在我们的评测协议中，由 LLM 仅凭描述回答这些问题，直接衡量描述是否保留了图像级效用、能否被下游 LLM 所利用。对最优多模态大模型的评估揭示出图像与其描述效用之间存在显著差距。值得注意的是，在传统图像问答基准上表现几乎相同的模型，其描述效用最多低 32%。我们发布 CaptionQA 以及一个开源流水线，以便扩展到新的领域。
英文原文摘要 1379 字符
Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains, Natural, Document, E-commerce, and Embodied AI, each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains.