缓解开放词表全景分割中的物体性偏差与区域-文本错位
Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation
针对开放词表全景分割中掩码选择偏差和区域-文本对齐不足的问题，提出模块化的 COAT 和 OVR 组件，可复用于其他开放词表分割任务，以较低内存成本提升 CLIP 的区域级对齐能力。
原文依据 Open-vocabulary panoptic segmentation remains hindered by two coupled issues: (i) mask selection bias, where objectness heads trained on closed vocabularies suppress masks of categories not observed in training, and (ii) limited regional understanding in vision-language models such as CLIP
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论开放词表全景分割仍受两个相互关联的问题困扰：(i) 掩码选择偏差，即在闭词表上训练的物体性头会抑制训练中未观察到的类别的掩码；(ii) 视觉语言模型（如 CLIP）的区域理解能力有限，这些模型针对全局图像分类而非局部化分割进行优化。我们提出 OVRCOAT，一个简单、模块化的框架来解决这两个问题。首先，CLIP 条件化的物体性调整（COAT）更新背景/前景概率，保留词汇外物体的高质量掩码。其次，开放词表掩码到文本精炼（OVR）增强 CLIP 的区域级对齐，以显著低于先前微调方案的内存成本改进已见和未见类别的分类。两个组件结合，共同改进物体性估计和掩码识别，带来一致的全景分割增益。尽管简单，OVRCOAT 在 ADE20K 上创造了新的最先进水平（+5.5% PQ），并在 Mapillary Vistas 和 Cityscapes 上取得明显增益（分别 +7.1% 和 +3% PQ）。代码已公开。
英文原文摘要 1160 字符
Open-vocabulary panoptic segmentation remains hindered by two coupled issues: (i) mask selection bias, where objectness heads trained on closed vocabularies suppress masks of categories not observed in training, and (ii) limited regional understanding in vision-language models such as CLIP, which were optimized for global image classification rather than localized segmentation. We introduce OVRCOAT, a simple, modular framework that tackles both. First, a CLIP-conditioned objectness adjustment (COAT) updates background/foreground probabilities, preserving high-quality masks for out-of-vocabulary objects. Second, an open-vocabulary mask-to-text refinement (OVR) strengthens CLIP's region-level alignment to improve classification of both seen and unseen classes with markedly lower memory cost than prior fine-tuning schemes. The two components combine to jointly improve objectness estimation and mask recognition, yielding consistent panoptic gains. Despite its simplicity, OVRCOAT sets a new state of the art on ADE20K (+5.5% PQ) and delivers clear gains on Mapillary Vistas and Cityscapes (+7.1% and +3% PQ, respectively). The code is available here.