检索与分割:少量示例足以弥合开放词表分割中的监督差距吗?
Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
提出检索增强的测试时适配器,通过融合文本与视觉支持特征学习轻量级逐图像分类器,为开放词表分割提供可复用的少样本融合思路,并支持支持集持续扩展与个性化分割。
原文依据 we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论开放词表分割(OVS)将视觉语言模型(VLM)的零样本识别能力扩展到像素级预测,使得能够分割由文本提示指定的任意类别。尽管近期取得了进展,但由于两个挑战,OVS 仍落后于全监督方法:用于训练 VLM 的粗粒度图像级监督以及自然语言的语义歧义。我们通过引入少样本设置来解决这些限制,该设置用一组像素标注的支持图像来增强文本提示。在此基础上,我们提出了一种检索增强的测试时适配器,通过融合文本和视觉支持特征来学习轻量级的逐图像分类器。与先前依赖后期手工融合的方法不同,我们的方法执行基于学习的逐查询融合,实现了模态之间更强的协同。该方法支持持续扩展的支持集,并适用于细粒度任务,例如个性化分割。实验表明,我们显著缩小了零样本分割与监督分割之间的差距,同时保持了开放词表能力。
英文原文摘要 1133 字符
Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.