KaLOS 寻找共识：一种评估复杂视觉任务中标注者间一致性的元算法
KaLOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks
为视觉基准的标注质量提供可复用的评估流程：先做空间对应匹配再计算一致性，把框、体素分割、姿态等异构标注统一成名义可靠性矩阵，可直接用于清洗检测/分割数据集并诊断标注者与协作结构。
原文依据 By resolving spatial correspondence before assessing agreement, our framework transforms complex spatio-categorical problems into nominal reliability matrices.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论目标检测基准的进展正陷入停滞。其瓶颈不在架构，而在于无法区分模型改进与标签噪声。为重建对基准评测的信任，该领域需要对标注一致性进行严格的量化，以确保评测数据的可靠性。然而，标准统计指标无法处理视觉任务固有的实例对应问题。此外，验证新的一致性指标本身仍是循环论证，因为一致性并不存在客观真值，这迫使研究者依赖无法验证的启发式做法。我们提出 KαLOS，一种统一的元算法，它将「定位优先」原则推广开来，用以标准化数据集质量评估。通过在评估一致性之前先解决空间对应关系，我们的框架将复杂的时空-类别问题转化为名义可靠性矩阵。与以往的启发式实现不同，KαLOS 采用有原则的、数据驱动的配置方式；通过将定位参数依据内在的一致性分布进行统计校准，它可推广到从边界框到体素分割或姿态估计的多种任务。这种标准化使得超越单一分数的细粒度诊断成为可能，包括标注者活跃度、协作聚类与定位敏感性。为验证该方法，我们引入了一种新颖的、由经验推导的噪声生成器。以往的验证依赖均匀误差假设，而我们的可控测试平台能够建模复杂且非各向同性的人类变异性。这为该指标的性质提供了证据，并使 KαLOS 成为在现代计算机视觉基准中区分信号与噪声的稳健标准。
英文原文摘要 1763 字符
Progress in object detection benchmarks is stagnating. It is limited not by architectures but by the inability to distinguish model improvements from label noise. To restore trust in benchmarking the field requires rigorous quantification of annotation consistency to ensure the reliability of evaluation data. However, standard statistical metrics fail to handle the instance correspondence problem inherent to vision tasks. Furthermore, validating new agreement metrics remains circular because no objective ground truth for agreement exists. This forces reliance on unverifiable heuristics. We propose K\alphaLOS, a unified meta-algorithm that generalizes the "Localization First" principle to standardize dataset quality evaluation. By resolving spatial correspondence before assessing agreement, our framework transforms complex spatio-categorical problems into nominal reliability matrices. Unlike prior heuristic implementations, K\alphaLOS employs a principled, data-driven configuration; by statistically calibrating the localization parameters to the inherent agreement distribution, it generalizes to diverse tasks ranging from bounding boxes to volumetric segmentation or pose estimation. This standardization enables granular diagnostics beyond a single score. These include annotator vitality, collaboration clustering, and localization sensitivity. To validate this approach, we introduce a novel and empirically derived noise generator. Where prior validations relied on uniform error assumptions, our controllable testbed models complex and non-isotropic human variability. This provides evidence of the metric's properties and establishes K\alphaLOS as a robust standard for distinguishing signal from noise in modern computer vision benchmarks.