Bag of Tricks for Optimizing Transformer Efficiency (2021)

Abstract: Improving Transformer efficiency has become increasingly attractive recently. A wide range of methods has been proposed, e.g., pruning, quantization, new architectures and etc. But these methods are either sophisticated in implementation or dependent on hardware. In this paper, we show that the efficiency of Transformer can be improved by combining some simple and hardware-agnostic methods, including tuning hyper-parameters, better design choices and training strategies. On the WMT news translation tasks, we improve the inference efficiency of a strong Transformer system by 3.80x on CPU and 2.52x on GPU.

## introduction standard implementation of transformer @xcite is not efficient for inference. researchers have explored more efficient architectures @xcite @xcite or break the auto-regressive constraint in sequence generation @xcite . but most of these require significant updates of the model or hardware-dependent designs. it is still natural to ask whether the transformer system can be optimized in a simple way @xcite @xcite . in this paper we show that transformer can be optimized for efficiency by a bag of techniques. these techniques are easy to implement and some of them have been tested in related studies. here we focus on using them in combination for transformer speedup which has not been well investigated. in particular, our work is based on the following facts: • the default byte-pair encoding (bpe) setting @xcite has a great impact on efficiency but is generally not optimal. removing some regularization measures like label smoothing @xcite also helps when training such models. all these methods are compatible with popular transformer codebases. in this work, we implement them on the decoder side because it occupies the inference time in many sequence generation tasks @xcite . the end result is a simplified and fast transformer decoder (see speedup on gpu with performance on par with the baseline. the speedup obtained is available on most modern hardware, as it does not depend on specific hardware or library, e.g., quantization @xcite and unstructured pruning (hoefler et al., 2021) require the support of the latest hardware-dependent and acceleration libraries. ## conclusion in this work, we present a bag of tricks to optimize the efficiency of the standard transformer. the resulting model achieves a 3.61× speedup on cpu and a 2.62× speedup on gpu without loss in performance.