大规模地理空间图像数据集中的数据泄漏检测与去重
Data Leakage Detection and De-duplication in Large Scale Geospatial Image Datasets
提出基于感知哈希的数据验证流水线，可自动检测大规模视觉数据集中的重复与泄漏问题，为训练可靠模型提供数据质量保障。
原文依据 we present a data validation pipeline to address these issues of duplication and data leakage
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在本研究中，我们对深度神经网络建筑物轮廓提取领域中三个广泛使用的数据集进行了全面分析：INRIA航空图像标注数据集、SpaceNet 2：建筑物检测v2以及AICrowd地图挑战数据集。实验揭示了AICrowd地图挑战数据集中存在的若干问题，其中近90%（约25万）的训练集图像存在完全相同的副本，表明数据重复程度很高。此外，我们发现验证集中的6万张图像中约有5.6万张也出现在训练集中，数据泄漏高达93%。进一步地，我们提出了一种数据验证流水线，以解决这些重复和数据泄漏问题，这些问题会妨碍在此类数据集上训练的模型性能。该流水线采用感知哈希技术，旨在实现高效的去重和泄漏识别。其目标是在使用数据集之前全面评估其质量，从而确保训练模型的可靠性和鲁棒性。
英文原文摘要 1085 字符
In our study, we conducted a comprehensive analysis of three widely used datasets in the domain of building footprint extraction using deep neural networks: the INRIA Aerial Image Labelling dataset, SpaceNet 2: Building Detection v2, and the AICrowd Mapping Challenge datasets. Our experiments revealed several issues in the AICrowd Mapping Challenge dataset, where nearly 90% (about 250k) of the training split images had identical copies, indicating a high level of duplicate data. Additionally, we found that approximately 56k of the 60k images in the validation split were also present in the training split, amounting to a 93% data leakage.Furthermore, we present a data validation pipeline to address these issues of duplication and data leakage, which hinder the performance of models trained on such datasets. Employing perceptual hashing techniques, this pipeline is designed for efficient de-duplication and leakage identification. It aims to thoroughly evaluate the quality of datasets before their use, thereby ensuring the reliability and robustness of the trained models.