在线梯度学习中的动态动量重校准
Dynamic Momentum Recalibration in Online Gradient Learning
SGDF 把梯度动量重解释为信号处理中的最优线性滤波，在线计算时变增益来平衡噪声抑制与信号保留；做视觉模型训练时可直接替换 SGD/Adam 类优化器，尤其适合梯度噪声大的小 batch 或高方差任务。
原文依据 SGDF computes an online, time-varying gain to dynamically refine gradient estimation by minimizing the mean-squared error
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论随机梯度下降（SGD）及其动量变体构成了深度学习优化的骨干，然而其梯度行为背后的动力学仍未被充分理解。在本工作中，我们从信号处理的视角重新诠释梯度更新，并揭示固定动量系数会固有地扭曲偏差与方差之间的平衡，导致参数更新出现偏斜或次优。为解决这一问题，我们提出 SGDF（带滤波的 SGD），一种受最优线性滤波原理启发的优化器。SGDF 通过最小化均方误差来计算一个在线的、时变的增益，从而动态地精化梯度估计，进而在噪声抑制与信号保留之间实现最优权衡。此外，我们的方法可以扩展到其他优化器，展现出其对优化框架的广泛适用性。在多种架构与基准上的大量实验表明，SGDF 超越了传统动量方法，并达到与最优优化器相当或更优的性能。
英文原文摘要 1094 字符
Stochastic Gradient Descent (SGD) and its momentum variants form the backbone of deep learning optimization, yet the underlying dynamics of their gradient behavior remain insufficiently understood. In this work, we reinterpret gradient updates through the lens of signal processing and reveal that fixed momentum coefficients inherently distort the balance between bias and variance, leading to skewed or suboptimal parameter updates. To address this, we propose SGDF (SGD with Filter), an optimizer inspired by the principles of Optimal Linear Filtering. SGDF computes an online, time-varying gain to dynamically refine gradient estimation by minimizing the mean-squared error, thereby achieving an optimal trade-off between noise suppression and signal preservation. Furthermore, our approach could extend to other optimizers, showcasing its broad applicability to optimization frameworks. Extensive experiments across diverse architectures and benchmarks demonstrate SGDF surpasses conventional momentum methods and achieves performance on par with or surpassing state-of-the-art optimizers.