HAD：面向终身异质学习的异质性感知蒸馏
HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning
提出无样本回放的异质知识蒸馏框架，通过分布平衡和显著性引导的蒸馏损失，在密集预测任务序列中保留不同输出结构的异质知识，可复用于多任务或持续学习场景。
原文依据 an exemplar-free approach that preserves previously gained heterogeneous knowledge by self-distillation in each training phase
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论终身学习旨在保留从先前任务中获得的知识，同时融入一系列新任务的知识。然而，大多数先前工作仅探索同质任务流（例如仅分类任务），忽略了跨异质任务学习的场景，这些任务具有不同的输出结构。在本工作中，我们将这一更广泛的设定形式化为终身异质学习（LHL）。与传统终身学习不同，LHL的任务序列跨越不同任务类型，学习者需要为不同输出空间结构保留异质知识。为了实例化LHL，我们聚焦于密集预测背景下的LHL（LHL4DP），这是一个现实且具有挑战性的场景。为此，我们提出了异质性感知蒸馏（HAD）方法，这是一种无样本回放的方法，通过在每个训练阶段进行自蒸馏来保留先前获得的异质知识。所提出的HAD包含两个互补组件，包括一个分布平衡的异质性感知蒸馏损失，以缓解预测分布的全局不平衡，以及一个显著性引导的异质性感知蒸馏损失，将学习集中在由Sobel算子提取的信息性边缘像素上。大量实验表明，所提出的HAD方法在这一新场景中显著优于现有方法。
英文原文摘要 1421 字符
Lifelong learning aims to preserve knowledge acquired from previous tasks while incorporating knowledge from a sequence of new tasks. However, most prior work explores only streams of homogeneous tasks (*e.g.*, only classification tasks) and neglects the scenario of learning across heterogeneous tasks that possess different structures of outputs. In this work, we formalize this broader setting as lifelong heterogeneous learning (LHL).Departing from conventional lifelong learning, the task sequence of LHL spans different task types, and the learner needs to retain heterogeneous knowledge for different output space structures.To instantiate the LHL, we focus on LHL in the context of dense prediction (LHL4DP), a realistic and challenging scenario.To this end, we propose the Heterogeneity-Aware Distillation (HAD) method, an exemplar-free approach that preserves previously gained heterogeneous knowledge by self-distillation in each training phase.The proposed HAD comprises two complementary components, including a distribution-balanced heterogeneity-aware distillation loss to alleviate the global imbalance of prediction distribution and a salience-guided heterogeneity-aware distillation loss that concentrates learning on informative edge pixels extracted with the Sobel operator.Extensive experiments demonstrate that the proposed HAD method significantly outperforms existing methods in this new scenario.