面向显存高效测试时自适应的曲率感知零阶优化
Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
用仅前向的零阶梯度估计替代反向传播来做测试时自适应，并借助对角 Hessian 的滑动平均构造各向异性扰动协方差，为显存受限的端侧视觉模型提供了一条可复用的免 BP 适配路径。
原文依据 we propose a loss-landscape curvature-aware zeroth-order (CAZO) method, which leverages a sliding-average estimation of the diagonal Hessian
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论测试时自适应（TTA）旨在通过适应无标签测试数据来提升预训练模型的跨域性能。现有大多数 TTA 方法依赖反向传播（BP）进行微调，而零阶（ZO）等免 BP 方法在实际的端侧场景中更为理想。ZO 方法仅依赖前向计算，可大幅降低端侧部署的复杂度与显存开销。然而，与一阶方法相比，ZO 方法在梯度估计上的方差要高得多。为解决这一问题，我们提出一种改进的 ZO 方法，以显著提升基于 ZO 优化的 TTA 性能。首先，我们给出一个观察，揭示适应过程中损失持续存在的低秩 Hessian 结构。基于这一洞见，我们进而提出损失景观曲率感知的零阶（CAZO）方法，它利用对角 Hessian 的滑动平均估计来构造协方差矩阵，用于各向异性扰动采样。CAZO 通过冻结预训练权重，并基于仅前向传播的梯度估计来优化极少量适配器参数，相比基于 BP 的方法可大幅降低显存开销。大量实验表明，CAZO 显著优于现有 TTA 方法，在达到最优性能的同时，在精度与显存效率之间保持了极佳的平衡。代码见 https://github.com/Hollyming/CAZO。
英文原文摘要 1521 字符
Test-time adaptation (TTA) aims to enhance the cross-domain performance of pre-trained models by adapting to unlabeled test data.While most existing TTA methods rely on backpropagation (BP) for finetuning, BP-free methods such as zeroth-order (ZO) methods are more desired in practical on-device scenarios. ZO methods rely only on forward computation, which can largely reduce the complexity and memory overhead of on-device deployment.However, ZO methods suffer from much higher variance compared with first-order methods in estimating the gradient.To address this, we propose an improved ZO method to substantially boost the performance of ZO optimization based TTA.First, we provide an observation to reveal the persistent low-rank Hessian structure of the loss during the adaptation process. Based on this insight, we then propose a loss-landscape curvature-aware zeroth-order (CAZO) method, which leverages a sliding-average estimation of the diagonal Hessian to construct a covariance matrix for anisotropic perturbation sampling. CAZO operates by freezing pretrained weights and optimizing minimal adapter parameters via forward-only passes based gradient estimation, which can substantially reduce the memory overhead compared to BP-based methods. Extensive experiments demonstrate that CAZO significantly outperforms existing TTA methods, achieving state-of-the-art performance while maintaining an excellent balance between accuracy and memory efficiency. Code is available at https://github.com/Hollyming/CAZO.