PET-DINO：通过提示词增强训练将视觉线索统一到 Grounding DINO 中
PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training
在 Grounding DINO 这类文本提示检测器上，用轻量视觉提示生成模块（AFVPG）把视觉线索注入文本提示路径，并配合批内并行提示与动态记忆提示两种训练策略，为开放词表/零样本检测提供了一条无需复杂多阶段优化的视觉提示融合方案。
原文依据 we propose PET-DINO, a universal detector supporting both text and visual prompts
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论开放集目标检测（OSOD）能够识别固定类别之外的新类别，但在将文本表示与复杂视觉概念对齐、以及稀有类别图文对稀缺方面面临挑战，导致在专业领域或复杂目标上性能欠佳。近期的视觉提示方法部分缓解了这些问题，但往往涉及复杂的多模态设计和多阶段优化，延长了开发周期。此外，面向数据驱动的 OSOD 模型的有效训练策略仍鲜有探索。为应对这些挑战，我们提出 PET-DINO，一个同时支持文本提示与视觉提示的通用检测器。我们的对齐友好视觉提示生成（AFVPG）模块构建于先进的文本提示检测器之上，缓解了文本表示引导的局限并缩短了开发周期。我们引入两种提示词增强训练策略：迭代层面的批内并行提示（IBP）与整体训练层面的动态记忆驱动提示（DMD）。这些策略能够同时建模多条提示路径，促进与多样真实使用场景的并行对齐。大量实验表明，PET-DINO 在多种基于提示的检测协议下展现出有竞争力的零样本目标检测能力。这些优势可归因于基于继承的理念与提示词增强训练策略，它们在构建有效的通用目标检测器中起着关键作用。项目主页：https://fuweifuvtoo.github.io/pet-dino。
英文原文摘要 1644 字符
Open-Set Object Detection (OSOD) enables recognition of novel categories beyond fixed classes but faces challenges in aligning text representations with complex visual concepts and the scarcity of image-text pairs for rare categories. This results in suboptimal performance in specialized domains or with complex objects. Recent visual-prompted methods partially address these issues but often involve complex multi-modal designs and multi-stage optimizations, prolonging the development cycle. Additionally, effective training strategies for data-driven OSOD models remain largely unexplored. To address these challenges, we propose PET-DINO, a universal detector supporting both text and visual prompts. Our Alignment-Friendly Visual Prompt Generation (AFVPG) module builds upon an advanced text-prompted detector, addressing the limitations of text representation guidance and reducing the development cycle. We introduce two prompt-enriched training strategies: Intra-Batch Parallel Prompting (IBP) at the iteration level and Dynamic Memory-Driven Prompting (DMD) at the overall training level. These strategies enable simultaneous modeling of multiple prompt routes, facilitating parallel alignment with diverse real-world usage scenarios. Comprehensive experiments demonstrate that PET-DINO exhibits competitive zero-shot object detection capabilities across various prompt-based detection protocols. These strengths can be attributed to inheritance-based philosophy and prompt-enriched training strategies, which play a critical role in building an effective generic object detector. Project page: https://fuweifuvtoo.github.io/pet-dino.