TWEO：无极端离群值的 Transformer 实现 FP8 训练与傻瓜式量化
TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
TWEO 通过一个简单的损失项消除激活离群值，使 ViT 和 LLM 无需工程技巧即可进行全模型 FP8 预训练，并支持硬件友好的 W8A8 静态量化，为视觉 Transformer 的低精度训练与部署提供了可直接复用的方法。
原文依据 TWEO then enables full-model FP8 pre-training with neither engineering tricks nor architectural changes for both LLM and ViT.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现代硬件原生支持 FP8 对于训练大型 Transformer 至关重要，但受到极端激活离群值的严重阻碍。现有解决方案要么依赖复杂的混合精度工程，要么依赖侵入式架构修改。本文从根本上挑战了离群值由数据驱动的传统观点。我们证明极端离群值是训练过程中机械产生的、与数据无关的伪影，源于权重矩阵的特定结构特性（即共线性）。基于这一洞见，我们提出 TWEO（无极端离群值的 Transformer），一种新颖的非侵入式损失函数。TWEO 通过一个非常简单的损失项有效防止极端离群值，将离群值从 10000+ 降低到 20 以下。随后，TWEO 无需工程技巧或架构改动即可为 LLM 和 ViT 实现全模型 FP8 预训练。当标准 FP8 训练灾难性崩溃时，TWEO 达到与 BF16 基线相当的性能，同时训练吞吐量提升 36%。此外，TWEO 还实现了一种新的量化范式。此前因离群值而被认为完全不可用的、硬件友好的 LLM 逐张量静态 W8A8 量化，首次在 TWEO 训练的模型上达到 SOTA 性能。
英文原文摘要 1314 字符
Native FP8 support in modern hardware is essential for training large Transformers, but is severely hindered by extreme activation outliers. Existing solutions either rely on complex mixed-precision engineering or invasive architectural modifications. This paper fundamentally challenges the conventional wisdom that outliers are data-driven. We demonstrate that extreme outliers are a data-independent, mechanically-produced artifact of training, originating from specific structural properties of the weight matrices (i.e., colinearity). Based on this insight, we propose TWEO (Transformers Without Extreme Outliers), a novel, non-invasive loss function. TWEO effectively prevents extreme outliers via a very simple loss term, which reduces outliers from 10000+ to less than 20. TWEO then enables full-model FP8 pre-training with neither engineering tricks nor architectural changes for both LLM and ViT. When standard FP8 training catastrophically collapses, TWEO achieves performance comparable to the BF16 baseline while delivering a 36% increase in training throughput. Also, TWEO enables a new quantization paradigm. Hardware-friendly W8A8 per-tensor static quantization of LLMs, previously considered completely unusable due to outliers, achieves SOTA performance for the first time on TWEO-trained models.