GR-Gauge：通过度量梯度冗余实现低成本训练配置
GR-Gauge: Cost-efficient Training Configuration By Gauging the Gradient Redundancy
提出以梯度冗余度作为轻量级信号来指导超参数搜索和试验终止，可替代传统基于验证集精度的昂贵评估，为视觉模型训练节省大量计算开销。
原文依据 to attain cost-efficient training is to ensure that the gradient redundancy is within a proper range
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期人工智能的成功促使许多非专业用户训练自己的模型。这些用户通常求助于云训练服务，希望以适中的成本获得足够准确的模型，为此正确设置学习率和批量大小至关重要。尽管在这方面已经提出了各种超参数优化（HPO）方法，但它们大多基于重量级的验证信号来行动，在总体成本上效率低下。我们发现模型训练过程可以视为一个二维投票过程——不同迭代和不同样本的梯度参与投票；此外，要实现成本高效的训练，就是要确保梯度冗余处于一个合适的范围内，而这个范围在不同模型之间是相似的。基于这一洞察，我们进一步引入了GR-Gauge，一种通过度量梯度冗余来指导HPO决策（如配置搜索和试验终止）的通用方法。大量实验表明，GR-Gauge能够帮助在比现有方法少得多的时间内达到接近最优的准确率。
英文原文摘要 1117 字符
The recent success of artificial intelligence motivates many non-professional users to train their own models. Those users often resort to cloud training services, seeking to obtain a sufficiently accurate model at a modest cost, for which properly setting up the learning rate and batch size is crucial. While various Hyper-parameter Optimization (HPO) methods have been proposed in that regard, they largely act based on heavy-weight validation signals, being inefficient in the overall cost. We find that the model training process can be viewed as a two-dimensional voting process---with gradients for different iterations and from different samples; moreover, to attain cost-efficient training is to ensure that the gradient redundancy is within a proper range which is similar across diverse models. Based on that insight, we further introduce GR-Gauge, a general method that gauges the gradient redundancy to instruct HPO decisions like configuration searching and trial termination. Extensive experiments demonstrate that GR-Gauge can help attain near-optimal accuracy in much less time than existing methods.