OS-Fed:单张快照足矣
OS-Fed: One Snapshot Is All You Need
把每轮累积梯度编码成一张紧凑的类图像快照(图像加可学习标签)来传输,这种「用图像式表示承载梯度/更新」的思路可迁移到视觉模型的分布式训练与梯度压缩,也可反向用于从图像式表示重建更新信号的表征学习研究。
原文依据 the entire gradient information from local training can be effectively reconstructed from a compact, image-like representation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论降低联邦学习中的通信开销具有挑战性,但对大规模分布式隐私保护机器学习至关重要。然而,直接压缩模型更新往往因信息损失而导致次优收敛,而增加本地计算量又可能引起模型发散。为此,本文提出一种截然不同的方法,遵循「一图胜千言」的原则。我们观察到,本地训练产生的全部梯度信息都可以从一个紧凑的、类图像的表示中有效重建。基于这一观察,我们提出一种新方法 OS-Fed,它通过每轮仅传输单张紧凑快照(由一张图像和一组可学习标签构成)来实现单样本联邦学习。为实现这一方法,OS-Fed 提出了新的快照合成技术:(1) 以一段轨迹的累积更新为目标,以应对梯度噪声;(2) 设计多网格快照,以解耦相互冲突的梯度方向;(3) 引入误差补偿,以在极端压缩下保持训练稳定。在计算机视觉与自然语言处理基准上的大量实验表明,与最优算法相比,OS-Fed 将通信成本降低 1.5 至 16 倍,并使收敛速度提升 18% 至 45%。
英文原文摘要 1344 字符
Reducing communication overhead in federated learning (FL) is challenging but crucial for large-scale distributed privacy-preserving machine learning. Unfortunately, directly compressing model updates often leads to sub-optimal convergence due to information loss, while increasing local computation can cause model divergence. Hence, this paper proposes a drastically different approach that adheres to the maxim that "a picture is worth a thousand words". We observe that the entire gradient information from local training can be effectively reconstructed from a compact, image-like representation. Based on this observation, we propose a novel approach, OS-Fed, which performs One-Shot Federated Learning by transmitting only a single, compact snapshot (comprising an image and a set of learnable labels) per round. To realize this approach, OS-Fed presents new snapshot synthesis techniques to (1) target the accumulated update of a trajectory segment to tackle gradient noise, (2) design a multi-grid snapshot that decouples conflicting gradient directions, and (3) incorporate error compensation to maintain training stability under extreme compression. Extensive experiments on CV and NLP benchmarks show that OS-Fed reduces communication costs by 1.5-16xcompared to state-of-the-art algorithms , resulting in 18-45% faster convergence.