残差连接损害生成式表征学习
Residual Connections Harm Generative Representation Learning
对 MAE、扩散模型等生成式表征学习框架，把残差恒等捷径的权重随深度单调衰减，可显著提升线性探测精度与生成质量；做自监督预训练或扩散骨干网络时，这是一个几乎零成本、可直接替换的架构改动。
原文依据 increasing ImageNet accuracy from 67.8% to 72.7% for MAEs with a VIT-B/16 backbone
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们表明，在残差网络中引入权重因子以削弱恒等捷径的影响，能够显著增强掩码自编码器（MAE）与扩散模型等生成式表征学习框架的语义特征学习。该改动同时提升了二者的线性探测精度，其中以 ViT-B/16 为骨干的 MAE 在 ImageNet 上的精度从 67.8% 提升至 72.7%，扩散模型的生成质量也得到提升。这一显著差距表明，残差连接结构虽然在促进梯度传播方面起着关键作用，但可能带来有害的副作用：它把较浅层表征的回声注入深层，从而削弱了抽象学习的能力。我们通过一个固定公式缓解这一缺陷，使恒等连接的贡献随层深增加而单调递减。该设计促进了特征抽象能力的逐步发展，同时不影响网络的可训练性。通过分析经我们改动的残差网络所学到的表征，我们发现有效特征秩偏低与下游任务性能之间存在相关性。
英文原文摘要 1185 字符
We show that introducing a weighting factor to reduce the influence of identity shortcuts in residual networks significantly enhances semantic feature learning in generative representation learning frameworks, such as masked autoencoders (MAEs) and diffusion models. Our modification improves linear probing accuracy for both, notably increasing ImageNet accuracy from 67.8% to 72.7% for MAEs with a VIT-B/16 backbone, while also boosting generation quality for diffusion models. This significant gap suggests that, while residual connection structure serves an essential role in facilitating gradient propagation, it may have a harmful side effect of reducing capacity for abstract learning by virtue of injecting an echo of shallower representations into deeper layers. We ameliorate this downside via a fixed formula for monotonically decreasing the contribution of identity connections as layer depth increases. Our design promotes the gradual development of feature abstractions, without impacting network trainability. Analyzing the representations learned by our modified residual networks, we find correlation between low effective feature rank and downstream task performance.