从遗忘中学习：通过遗忘回溯预测知识溢出权重
Learning from Oblivion: Predicting Knowledge-Overflowed Weights via Retrodiction of Forgetting
提出一种通过元学习建模权重演化、逆转结构化遗忘过程来合成知识溢出权重的通用框架，可复用于视觉模型预训练权重的增强与下游任务微调初始化。
原文依据 employ meta-learning to model weight prediction effectively
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论预训练权重已成为现代深度学习的基石，能够实现高效的知识迁移并提升下游任务性能，尤其是在数据稀缺场景下。然而，一个根本性问题仍然存在：如何获得更好的预训练权重，使其封装超出给定数据集的知识？在本工作中，我们引入知识溢出权重（KNOW）预测，这是一种利用结构化遗忘及其逆过程来合成知识增强权重的新策略。我们的关键洞察是，在逐步缩小的数据集上进行顺序微调会诱发结构化的遗忘过程，该过程可以被建模并逆转，从而恢复如同在更大数据集上训练所获得的知识。我们构建了一个由这种受控遗忘所支配的权重转移数据集，并采用元学习来有效建模权重预测。具体而言，我们的知识溢出权重预测器（KNOWN）充当一个超模型，学习权重的一般演化规律并预测具有改进泛化能力的增强权重。在不同数据集和架构上的大量实验表明，KNOW预测始终优于朴素微调和简单的权重预测，从而带来更优的下游性能。我们的工作为重新解释遗忘动态以推动知识迁移的极限提供了新视角。代码和预训练模型可在 https://github.com/jjh6297/KNOW 获取。
英文原文摘要 1486 字符
Pre-trained weights have become a cornerstone of modern deep learning, enabling efficient knowledge transfer and improving downstream task performance, especially in data-scarce scenarios. However, a fundamental question remains: how can we obtain better pre-trained weights that encapsulate more knowledge beyond the given dataset? In this work, we introduce KNowledge-Overflowed Weights (KNOW) prediction, a novel strategy that leverages structured forgetting and its inversion to synthesize knowledge-enriched weights. Our key insight is that sequential fine-tuning on progressively downsized datasets induces a structured forgetting process, which can be modeled and reversed to recover knowledge as if trained on a larger dataset. We construct a dataset of weight transitions governed by this controlled forgetting and employ meta-learning to model weight prediction effectively. Specifically, our KNowledge-Overflowed Weights Nowcaster (KNOWN) acts as a hyper-model that learns the general evolution of weights and predicts enhanced weights with improved generalization. Extensive experiments across diverse datasets and architectures demonstrate that KNOW prediction consistently outperforms Naive fine-tuning and simple weight prediction, leading to superior downstream performance. Our work provides a new perspective on reinterpreting forgetting dynamics to push the limits of knowledge transfer. The code and pre-trained model are available at https://github.com/jjh6297/KNOW