看清且自信推理：面向视觉语言模型盲区的即插即用补救方案
Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness
提出一种免微调的即插即用模块：用视觉基础模型先验加同义词增强文本学出稀有物体的多模态类别嵌入，再用轻量注意力模块精炼 VLM 的视觉 token，并把嵌入当作物体感知检测器生成提示注入文本 prompt——这套「视觉 token 精炼 + 文本提示注入」的双路增强思路可直接迁移到任何冻结 VLM 的细粒度/长尾识别与 grounding 场景。
原文依据 an efficient plug-and-play module that substantially improves VLMs' reasoning over rare objects by refining visual tokens and enriching input text prompts, without VLMs finetuning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在广泛的视觉理解任务上取得了显著成功，但由于预训练数据中此类实例稀缺，它们在稀有物体上的以物体为中心的推理仍然受到挑战。已有工作通过检索额外数据或引入更强的视觉编码器来缓解这一问题，但这些方法在微调 VLM 时计算开销仍然很大，且未能充分利用原始训练数据。本文提出一个高效的即插即用模块，通过精炼视觉 token 并丰富输入文本提示，在无需微调 VLM 的情况下显著提升其对稀有物体的推理能力。具体而言，我们利用视觉基础模型的先验知识与同义词增强的文本描述，为稀有物体学习多模态类别嵌入，以补偿训练样本的不足。这些嵌入通过一个轻量的基于注意力的增强模块精炼 VLM 中的视觉 token，从而改善细粒度物体细节。此外，我们将学习到的嵌入用作物体感知检测器以生成信息性提示，并将其注入文本提示中，帮助引导 VLM 的注意力指向相关图像区域。在两个基准上的实验表明，预训练 VLM 在稀有物体识别与推理上获得了一致且显著的提升。进一步分析揭示了本方法如何增强 VLM 聚焦并推理稀有物体的能力。代码见 https://github.com/XinHu98/seeing。
英文原文摘要 1548 字符
Vision language models (VLMs) have achieved remarkable success in broad visual understanding, yet they remain challenged by object-centric reasoning on rare objects due to the scarcity of such instances in pretraining data. While prior efforts alleviate this issue by retrieving additional data or introducing stronger vision encoders, these methods are still computationally intensive during finetuning VLMs and don't fully exploit the original training data. In this paper, we introduce an efficient plug-and-play module that substantially improves VLMs' reasoning over rare objects by refining visual tokens and enriching input text prompts, without VLMs finetuning. Specifically, we propose to learn multi-modal class embeddings for rare objects by leveraging prior knowledge from vision foundation models and synonym-augmented text descriptions, compensating for limited training examples. These embeddings refine the visual tokens in VLMs through a lightweight attention-based enhancement module that improves fine-grained object details. In addition, we use the learned embeddings as object-aware detectors to generate informative hints, which are injected into the text prompts to help guide the VLM's attention toward relevant image regions. Experiments on two benchmarks show consistent and substantial gains for pretrained VLMs in rare object recognition and reasoning. Further analysis reveals how our method strengthens the VLM's ability to focus on and reason about rare objects. Code is available at https://github.com/XinHu98/seeing