对话式图像分割:以可扩展监督对抽象概念进行视觉定位
Conversational Image Segmentation: Grounding Abstract Concepts with Scalable Supervision
把「意图/可供性/物理常识」这类抽象概念作为分割查询,并用无需人工标注的数据引擎自动生成 prompt-mask 对,为语言引导分割提供了可扩展的监督构造范式,可直接迁移到开放词表与指令式分割的数据扩增。
原文依据 an AI-powered data engine that generates prompt-mask pairs without human supervision
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论对话式图像分割将抽象的、由意图驱动的概念定位为像素级精确的掩码。此前的指代图像视觉定位工作主要关注类别性与空间性查询(例如「最左边的苹果」),而忽略了功能性与物理性推理(例如「我可以把刀安全地放在哪里?」)。我们针对这一空白,提出对话式图像分割(CIS)与 ConvSeg——一个涵盖实体、空间关系、意图、可供性、功能、安全性与物理推理的基准。我们还提出 ConvSeg-Net,它将强分割先验与语言理解相融合,并提出一个由 AI 驱动的数据引擎,可在无人工监督的情况下生成提示-掩码对。我们表明,当前的语言引导分割模型难以胜任 CIS,而在我们的数据引擎上训练的 ConvSeg-Net 在 ConvSeg 上取得显著提升,同时在现有的语言引导分割基准上保持强劲性能。项目主页:https://glab-caltech.github.io/converseg
英文原文摘要 1001 字符
Conversational image segmentation grounds abstract, intent-driven concepts into pixel-accurate masks. Prior work on referring image grounding focuses on categorical and spatial queries (e.g., "left-most apple") and overlooks functional and physical reasoning (e.g., "where can I safely store the knife?"). We address this gap and introduce Conversational Image Segmentation (CIS) and ConvSeg, a benchmark spanning entities, spatial relations, intent, affordances, functions, safety, and physical reasoning. We also present ConvSeg-Net, which fuses strong segmentation priors with language understanding, and an AI-powered data engine that generates prompt-mask pairs without human supervision. We show that current language-guided segmentation models are inadequate for CIS, while ConvSeg-Net trained on our data engine achieves significant gains on ConvSeg and maintains strong performance on existing language-guided segmentation benchmarks. Project webpage: https://glab-caltech.github.io/converseg