Thermal-Det:面向开放词表热成像目标检测的语言引导跨模态蒸馏
Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detection
把冻结的 RGB 开放词表检测器当作教师,为成对但无标注的 RGB-热成像数据提供几何与语义伪监督,这一跨模态蒸馏范式可直接迁移到其他缺少标注的成像域(如近红外、深度、事件相机)的开放词表检测。
原文依据 A frozen RGB teacher provides geometric and semantic pseudo-supervision for paired but unlabeled RGB-thermal data
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有的开放词表检测器主要面向 RGB 图像,难以泛化到热成像影像,因为热成像中纹理匮乏且发射率变化大,对基于 RGB 的语义构成挑战。本文提出 Thermal-Det,这是首个专为热成像图像设计、由大语言模型(LLM)监督的开放词表检测器。为支持大规模训练,我们构建了一个合成数据集:将 GroundingCap-1M 转换到热成像域,并过滤掉包含 RGB 特有词汇的文本描述,从而得到超过一百万条热成像对齐样本,包含边界框、视觉定位文本和详细描述。Thermal-Det 联合优化检测、描述生成与跨模态蒸馏三个目标。一个冻结的 RGB 教师模型为成对但无标注的 RGB-热成像数据提供几何与语义伪监督,从而在无需人工标注的情况下迁移开放词表知识。该模型进一步采用热成像-文本对齐检测头进行文本校准,并使用模态融合交叉注意力模块进行双模态推理。与以往的域自适应方法不同,该检测器经过完全微调,以内化热成像对比度模式,同时保持语言对齐。在公开基准上的实验表明,相较现有开放词表检测器,其 AP 稳定提升 2-4%,为可扩展的、语言驱动的热成像感知奠定了坚实基础。
英文原文摘要 1346 字符
Existing open-vocabulary detectors focus on RGB images and fail to generalize to thermal imagery, where low texture and emissivity variations challenge RGB-based semantics. We present Thermal-Det, the first large language model (LLM) supervised open-vocabulary detector tailored for thermal images. To enable large-scale training, we construct a synthetic dataset by converting GroundingCap-1M into the thermal domain and filtering captions to remove RGB-specific terms, yielding over one million thermally aligned samples with bounding boxes, grounding texts, and detailed captions. Thermal-Det jointly optimizes detection, captioning, and cross-modal distillation objectives. A frozen RGB teacher provides geometric and semantic pseudo-supervision for paired but unlabeled RGB-thermal data, transferring open-vocabulary knowledge without manual annotation. The model further employs a Thermal-Text Alignment Head for text calibration and a Modality-Fused Cross-Attention module for dual-modality reasoning. Unlike prior domain-adaptation methods, the detector is fully fine-tuned to internalize thermal contrast patterns while preserving language alignment. Experiments on public benchmarks show consistent 2-4% AP gains over existing open-vocabulary detectors, establishing a strong foundation for scalable, language-driven thermal perception.