DP-FedAdamW：面向差分隐私联邦大模型的高效优化器
DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models
为在隐私约束下用视觉 Transformer/ResNet 做联邦训练提供可直接替换的优化器：修正二阶矩估计的隐私噪声偏差并抑制客户端漂移，视觉研究者可在 DP 联邦微调大模型时直接套用该优化器与收敛分析框架。
原文依据 DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83%
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在联邦学习中，平衡差分隐私（DP）下的收敛效率与鲁棒性是一项核心挑战。尽管 AdamW 加速了大规模模型的训练与微调，我们发现将其直接应用于差分隐私联邦学习（DPFL）会带来三个主要问题：(i) 数据异构性与隐私噪声共同放大了二阶矩估计器的方差；(ii) DP 扰动使二阶矩估计器产生偏差；(iii) DP 放大了 AdamW 对局部过拟合的敏感性，加剧了客户端漂移。我们提出 DP-FedAdamW，这是首个面向 DPFL 的基于 AdamW 的优化器。它通过稳定二阶矩方差、消除 DP 引起的偏差、并使本地更新与全局下降方向对齐以抑制客户端漂移，从而在 DP 下恢复 AdamW 的有效性。在理论上，我们建立了一个无偏的二阶矩估计器，并在不依赖任何异构性假设的情况下证明了线性加速的收敛速率，同时给出了更紧的 (ε,δ)-DP 保证。实验结果证明了 DP-FedAdamW 在语言与视觉 Transformer 以及 ResNet-18 上的强劲性能。在 Tiny-ImageNet（Swin-Base，ε=1）上，DP-FedAdamW 以 5.83% 的优势超越最优方法。代码已发布于 https://github.com/junkangLiu0/DP-FedAdamW。
英文原文摘要 1340 字符
Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). Although AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of the second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplifies AdamW's sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores the effectiveness of AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates with the global descent direction to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter (\varepsilon,\delta)-DP guarantees. Our empirical results demonstrate the strong performance of DP-FedAdamW across language and vision Transformers, as well as ResNet-18. On Tiny-ImageNet (Swin-Base, \varepsilon=1), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83%. The code is available at https://github.com/junkangLiu0/DP-FedAdamW.