把弹性权重巩固做对：面向持续学习的正确实现
Elastic Weight Consolidation Done Right for Continual Learning
对做视觉模型持续学习/增量训练的人，这篇给出了一个几乎零成本的改动：在计算 Fisher 信息矩阵时把 logit 取反，即可同时缓解梯度消失与冗余保护，可直接替换现有 EWC 正则项用于视觉骨干网络的顺序多任务训练。
原文依据 reversing the logit values during the calculation of the FIM can effectively prevent both the gradient vanishing and the redundant protection
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论持续学习（CL）中的权重正则化方法通过评估并惩罚对重要模型权重的改动来缓解灾难性遗忘。弹性权重巩固（EWC）是该框架内一项基础且被广泛使用的方法，它基于梯度估计权重重要性。然而，它的表现一直不够理想。本文从基于梯度的视角对 EWC 中的重要性估计进行了系统分析。我们首次发现，EWC 对 Fisher 信息矩阵（FIM）的依赖会在某些情形下导致梯度消失与重要性估计不准确。我们的分析还揭示，作为 EWC 变体的记忆感知突触（MAS）会对与先前任务无关的参数施加不必要的约束，我们称之为冗余保护。因此，EWC 及其变体在估计权重重要性时都存在根本性的错位，从而导致性能不佳。为解决这些问题，我们提出 Logits Reversal（LR）操作，这是一种简单而有效的修改，可纠正 EWC 的重要性估计。具体而言，在计算 FIM 时对 logit 值取反，能够有效同时避免梯度消失与冗余保护。在多种持续学习任务和数据集上的大量实验表明，所提方法显著优于现有的 EWC 及其变体。因此，我们将其称为 EWC Done Right（EWC-DR）。
英文原文摘要 1499 字符
Weight regularization methods in continual learning (CL) alleviate catastrophic forgetting by assessing and penalizing changes to important model weights. Elastic Weight Consolidation (EWC) is a foundational and widely used approach within this framework that estimates weight importance based on gradients. However, it has consistently shown suboptimal performance.In this paper, we conduct a systematic analysis of importance estimation in EWC from a gradient-based perspective. For the first time, we find that EWC's reliance on the Fisher Information Matrix (FIM) results in gradient vanishing and inaccurate importance estimation in certain scenarios.Our analysis also reveals that Memory Aware Synapses (MAS), a variant of EWC, imposes unnecessary constraints on parameters irrelevant to prior tasks, termed the redundant protection. Consequently, both EWC and its variant exhibit fundamental misalignments in estimating the importance of weights, leading to inferior performance. To tackle these issues, we propose the Logits Reversal (LR) operation, a simple yet effective modification that rectifies the importance estimation of EWC. Specifically, reversing the logit values during the calculation of the FIM can effectively prevent both the gradient vanishing and the redundant protection. Extensive experiments across various CL tasks and datasets show that the proposed method significantly outperforms existing EWC and its variants. Therefore, we refer to it as EWC Done Right (EWC-DR).