AdaBet：面向深度神经网络高效训练的无梯度层选择方法
AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks
用Betti数分析激活空间拓扑、仅靠前向传播即可选出高学习能力的层与通道，为视觉模型在边缘设备上的端侧微调提供了一种免梯度、免标签的层选择手段，可直接嵌入现有迁移学习/域自适应流程以降低显存与反向传播开销。
原文依据 a gradient-free layer selection approach to rank important layers, followed by important channels of these layers, by analyzing topological features of their activation spaces through Betti Numbers
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论为了在边缘与移动设备上使用预训练神经网络，我们通常需要在有限的计算与内存资源下高效地适应用户特定的运行时数据分布。使用目标数据集进行端侧再训练可以促成此类适应；然而，由于现代神经网络深度不断增加，且对所有层进行基于梯度的优化会带来计算开销，这一做法仍不切实际。现有方法通过选择一部分层进行再训练来降低训练成本；但它们依赖标注数据、至少一次全模型反向传播或服务端元训练，因而限制了其在资源受限设备上的适用性。我们提出 AdaBet，一种无梯度的层选择方法，通过 Betti 数分析各层激活空间的拓扑特征，仅依靠前向传播即可对重要层进行排序，并进一步排序这些层中的重要通道。AdaBet 能够选出具有高学习能力的层与通道，这些层与通道对再训练和适应至关重要，且无需标签或梯度。在十六组基准模型与数据集上评估 AdaBet 的结果显示，相比基于梯度的基线方法，AdaBet 平均提升 2.5% 的分类准确率，同时将平均峰值内存消耗降低 40%。代码已开源于 https://github.com/Nokia-Bell-Labs/efficient_layer_selection
英文原文摘要 1445 字符
To utilize pre-trained neural networks on edge and mobile devices, we often require efficient adaptation to user-specific runtime data distributions while operating under limited compute and memory resources. On-device retraining with a target dataset can facilitate such adaptations; however, it remains impractical due to the increasing depth of modern neural nets, as well as the computational overhead associated with gradient-based optimization across all layers. Current approaches reduce training cost by selecting a subset of layers for retraining; however, they rely on labeled data, at least one full-model backpropagation, or server-side meta-training, limiting their suitability for constrained devices. We introduce AdaBet, a gradient-free layer selection approach to rank important layers, followed by important channels of these layers, by analyzing topological features of their activation spaces through Betti Numbers and using forward passes alone. AdaBet allows selecting layers and channels with high learning capacity, which are important for retraining and adaptation, without requiring labels or gradients. Evaluating AdaBet on sixteen pairs of benchmark models and datasets shows AdaBet achieves an average gain of 2.5% more classification accuracy over gradient-based baselines while reducing average peak memory consumption by 40%. We open-source our code at https://github.com/Nokia-Bell-Labs/efficient_layer_selection