更强的无归一化 Transformer
Stronger Normalization-Free Transformers
Derf 可作为 LayerNorm/RMSNorm/DyT 的即插即用替代，用于视觉识别与生成任务中的无归一化 Transformer，降低归一化层开销并可能提升泛化。
原文依据 Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including visual recognition and generation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管归一化层长期以来被视为深度学习架构中不可或缺的组件，但最近引入的动态 Tanh（DyT）表明替代方案是可能的。逐点函数 DyT 约束极端值以实现稳定收敛，并达到归一化级别的性能；本研究旨在进一步寻找能够超越它的函数设计。我们首先研究逐点函数的内在属性如何影响训练和性能。基于这些发现，我们进行了大规模搜索以寻找更有效的函数设计。通过这一探索，我们引入了 \mathrm Derf (x) = \mathrm erf (\alpha x + s)，其中 \mathrm erf (x) 是重新缩放的高斯累积分布函数，并确定其为性能最佳的设计。Derf 在包括视觉识别与生成、语音表示和 DNA 序列建模在内的广泛领域中优于 LayerNorm、RMSNorm 和 DyT。我们的分析还表明，Derf 的性能提升主要源于其改进的泛化能力，而非更强的拟合能力。其简单性和更强的性能使 Derf 成为无归一化 Transformer 架构的实用选择。
英文原文摘要 1248 字符
Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that alternatives are possible. The point-wise function DyT constrains extreme values for stable convergence and reaches normalization-level performance; this work seeks further for function designs that can surpass it. We first study how the intrinsic properties of point-wise functions influence training and performance. Building on these findings, we conduct a large-scale search for a more effective function design. Through this exploration, we introduce \mathrm Derf (x) = \mathrm erf (\alpha x + s), where \mathrm erf (x) is the rescaled Gaussian cumulative distribution function, and identify it as the most performant design. Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including visual recognition and generation, speech representation, and DNA sequence modeling. Our analysis also suggests that the performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity. Its simplicity and stronger performance make Derf a practical choice for normalization-free Transformer architectures.