稳定含噪预训练模型的特征几何以提升下游任务鲁棒性
Stabilizing Feature Geometry in Noisy Pretrained Models for Robust Downstream Tasks
对做预训练-微调视觉模型的人，这篇给出一个可复用的诊断量 PDA 来量化噪声预训练模型的主特征子空间旋转，并用扰动一致性、方差-激活正则与特征一致性蒸馏把子空间拉回干净模型的方向，可直接用于清洗噪声数据后的模型修复或鲁棒微调。
原文依据 even mild pretraining noise can cause a clear rotation of the dominant feature subspace
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在大规模数据上预训练再微调已成为视觉模型的标准范式。然而，预训练数据中的噪声会被模型吸收并带入下游任务，造成灾难性继承。已有研究主要将该问题归因于特征谱的变化，认为噪声削弱了关键特征分量的强度，并据此通过放大这些分量来提升可迁移性。但这些方法只关注谱能量，并隐含假设特征方向保持不变，而这一假设在实践中并不成立。本文重新审视该观点，揭示了一个被忽视的效应：即使轻微的预训练噪声也会导致主特征子空间发生明显旋转，而谱能量的退化却可以忽略不计。为定量刻画这一现象，我们提出使用主方向夹角（PDA）来度量干净模型与含噪模型之间的方向偏移。基于该观察，我们提出特征几何稳定（FGS）框架，通过扰动一致性、方差-激活正则化与特征一致性蒸馏的协同作用增强特征空间的几何稳定性，以抵消 PDA 所揭示的子空间旋转。在多个视觉基准上的实验验证了 FGS 的有效性，并证实稳定特征几何对于缓解灾难性继承的重要性。
英文原文摘要 1535 字符
Pretraining on large-scale data followed by fine-tuning has become a standard paradigm for visual models. However, noise in the pretraining data can be absorbed by the model and carried into downstream tasks, causing catastrophic inheritance. Prior studies mainly link this issue to changes in the feature spectrum, arguing that noise reduces the strength of key feature components. Following this view, they aim to improve transferability by amplifying these components. However, these approaches focus only on spectral energy and implicitly assume that the feature directions remain fixed, which does not hold in practice. In this work, we revisit this view and reveal an overlooked effect: even mild pretraining noise can cause a clear rotation of the dominant feature subspace, despite negligible spectral energy degradation. To quantitatively characterize this phenomenon, we propose using the Principal Directional Angle (PDA) to measure the directional shift between the clean and noisy models. Building on this observation, we introduce the Feature Geometry Stabilization (FGS) framework, which aims to counteract the subspace rotation revealed by PDA by enhancing the geometric stability of the feature space through the synergistic interaction of perturbation consistency, variance-activation regularization, and feature consistency distillation. Experiments across multiple visual benchmarks demonstrate the effectiveness of FGS and verify the importance of stabilizing feature geometry to mitigate catastrophic inheritance.