RetFormer：用于增强图像识别的多模态检索
RetFormer: Multimodal Retrieval for Enhancing Image Recognition
将外部多模态知识库与检索交叉融合模块引入视觉识别，为长尾分布和噪声标签下的图像分类提供了一种不依赖增大模型参数的增强思路。
原文依据 we introduce RetFormer, a model enhanced with a multimodal knowledge base for storing world knowledge, and a retrieval cross-fusion module
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论Transformer 的扩展与高质量多模态数据集的收集，推动深度神经网络在视觉和语言任务中取得了前所未有的性能。然而，在现实应用中应用这些进展并非易事。庞大的参数量使模型更新变得复杂，而真实世界数据往往呈现长尾分布并伴有噪声标签。为解决上述问题，我们提出探索神经网络的内部结构，以样本关系进行学习，而非仅仅增加模型参数数量。具体而言，我们引入 RetFormer，一个通过多模态知识库（用于存储世界知识）增强的模型，以及一个检索交叉融合模块，旨在利用知识库中的内容建立稳健的多模态样本关系。RetFormer 通过将外部知识库的信息整合到模型的决策过程中，在图像和文本模态之间建立稳健的关系，从而克服传统方法在模型规模和数据集上的局限。我们的实验证明了将大规模图像-文本数据集整合到视觉任务中的益处，并例证了建模图像和文本模态之间关系的重要性。我们在长尾识别和噪声标签学习任务上评估了我们的方法，并表明其达到了最优的准确率。
英文原文摘要 1493 字符
The expansion of Transformers and the collection of high-quality multimodal datasets have propelled deep neural networks to achieve unprecedented performance in vision and language tasks. However, applying these advances is non-trivial in real-world applications. The extensive number of parameters complicates model updates, and real-world data often features a long-tailed distribution along with noisy labels. To address the above issues, we propose to explore the internal structure of the neural network for learning with sample relationships, rather than just increasing the number of model parameters. Specifically, we introduce RetFormer, a model enhanced with a multimodal knowledge base for storing world knowledge, and a retrieval cross-fusion module designed to establish robust multimodal sample relationships by leveraging content from the knowledge base. RetFormer establishes a robust relationship between image and text modalities by integrating information from external knowledge bases into the model's decision-making process, thus overcoming the limitations of traditional approaches on model size and datasets. Our experiments demonstrate the benefits of integrating large-scale image-text datasets into vision tasks and exemplify the importance of modeling the relationship between image and text modalities. We have evaluated our approach on the task of long-tailed recognition and learning with noisy labels and have shown that it achieves state-of-the-art accuracies.