EXOTIC:外部视觉驱动的非完整多视图分类
EXOTIC: External Vision-driven Incomplete Multi-view Classification
利用预训练视觉语言模型的外部知识作为监督信号来补全缺失视图,为多视图分类中的不完整数据问题提供了新思路,可迁移到其他需要跨模态对齐与补全的视觉任务。
原文依据 we propose an EXternal visiOn-driven incomplete mulTi-vIew Classification (EXOTIC) paradigm that incorporates external vision knowledge as semantic guidance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论由于数据采集过程中的传感器故障和遮挡,多视图数据常常存在部分样本缺失,从而产生非完整多视图数据。近年来,非完整多视图分类(IMVC)已成为研究热点之一,涌现出大量IMVC方法。尽管这些方法通过利用部分观测数据中的内部语义信息取得了良好性能,但它们主要依赖有限的内部监督进行视图补全。显然,这很大程度上限制了其性能上限。为克服这一局限,我们提出了一种外部视觉驱动的非完整多视图分类(EXOTIC)范式,该范式引入外部视觉知识作为语义指导,从而辅助补全非完整视图。据我们所知,这是首个利用外部视觉知识作为监督信号来指导缺失视图补全的工作。具体而言,我们首先基于预训练的视觉语言模型构建了一个外部视觉知识库。然后,我们设计了一个知识过滤模块来自适应地选择任务相关知识。接着,我们提出了一个知识净化模块,将外部知识与内部表示对齐。最后,我们提出了外部补全方法,利用精炼后的知识来填补缺失视图,从而增强分类决策能力。在多个非完整多视图数据集上的大量实验表明,所提出的EXOTIC方法持续优于现有方法,尤其是在高缺失率情况下。
英文原文摘要 1630 字符
Due to sensor failures and occlusions during data acquisition, multi-view data often suffer from partial missing samples, thereby producing incomplete multi-view data. Recently, Incomplete Multi-View Classification (IMVC) has become one of the research hot topics, where numerous IMVC methods have been proposed. Although these methods have achieved promising performance by exploiting internal semantic information from partially observed data, they primarily rely on limited internal supervision for view completion. Clearly, this largely constrains their performance ceiling. To overcome this limitation, we propose an EXternal visiOn-driven incomplete mulTi-vIew Classification (EXOTIC) paradigm that incorporates external vision knowledge as semantic guidance, thereby assisting in imputing incomplete views. To the best of our knowledge, it is the first work that leverages external vision knowledge as supervision signals, thereby guiding missing-view completion. Specifically, we first introduce an external vision knowledge library based on a pre-trained vision-language model. Then, we design a Knowledge Filtering module to adaptively select task-relevant knowledge. Afterwards, we present a Knowledge Purification module to align external knowledge with internal representations. Finally, we propose External Completion that leverages the refined knowledge to impute missing views, thereby enhancing the classification decision ability. Extensive experiments on multiple incomplete multi-view datasets demonstrate that the proposed EXOTIC consistently outperforms existing methods, especially under high missing rates.