L ight S eq: A High Performance Inference Library for Transformers (2021)

Abstract: Transformer and its variants have achieved great success in natural language processing. Since Transformer models are huge in size, serving these models is a challenge for real industrial applications. In this paper, we propose , a highly efficient inference library for models in the Transformer family. includes a series of GPU optimization techniques to both streamline the computation of Transformer layers and reduce memory footprint. supports models trained using PyTorch and Tensorflow. Experimental results on standard machine translation benchmarks show that achieves up to 14x speedup compared with TensorFlow and 1.4x speedup compared with , a concurrent CUDA implementation. The code will be released publicly after the review.

## introduction sequence processing and generation have been fundamental capabilities for many natural language processing tasks, including machine translation, summarization, language modeling, etc @xcite @xcite . in recent years, with the introduction of transformer model @xcite , many pre-trained language models such as bert, gpt, and mrasp have also been widely used in these tasks @xcite @xcite . however, the parameters of these models become increasingly large, which causes the high latency of inference and brings great challenges to the deployment @xcite . the current popular inference systems are not necessarily the best choice for the online service of sequence processing problems. first, training frameworks, such as tensorflow and pytorch, require accommodating flexible model architectures and backward propagation, which introduce additional memory allocation and extra overhead of using fine-grain kernel functions. therefore, the direct deployment of the training framework is not able to make full use of the hardware resource. taking an example of machine translation, the transformer big model currently takes roughly 2 seconds to translate a sentence, which is unacceptable in both academia and industry @xcite . second, current optimizing compilers for deep learning such as tensorflow xla @xcite , tvm @xcite and tensor rt @xcite are mainly designed for fixed-size inputs. however, most nlp problems enjoy variable-length inputs, which are much more complex and require dynamic memory allocation. therefore, a high-performance sequence inference library for variable-length inputs is required. there are several concurrent cuda libraries which share a similar idea with our project, such as faster-transformer foot_0 and turbotransformers @xcite . we will highlight three innovative features that make lightseq outperforms similar projects. first, we replace a straightforward combination of finegrained gpu kernel functions in tensorflow or pytorch implementations with coarse-grain fused ones, which avoid high time cost introduced by a mass of kernel function launches and gpu memory i/o for intermediate results. as a result, light-seq reduces the atomic kernel functions by four times compared with tensorflow approaches. second, we specially design a hierarchical auto regressive search method to speed up the auto-regressive search. third, we propose a dynamic gpu memory reuse strategy. different from fixed-length inputs, sequence processing tackles the variable-length inputs, which bring difficulty for memory allocation. lightseq proposes to pre-define the maximal memory for each kernel function and shares the gpu memory across non-dependent ones. as a result, lightseq reduces eight times memory allocation without loss of inference speed. as a benefit, light-seq enjoys several advantages: efficient lightseq shows better inference performance for generation tasks. for example, in machine translation benchmarks, lightseq achieves up to 14 times speedup compared with tensorflow and 1.4 times speedup compared with fastertransformer. functional lightseq supports more architecture variants, such as bert, gpt, transformer, and variational autoencoders (vaes). further, lightseq provides different search algorithms, such as beam search, diverse beam search and probabilistic sampling @xcite . convenient lightseq is easy to use, which contains a serving system and efficient cuda implementations. the popular models, such as bert, roberta, gpt, vaes, mt transformer, and speech transformer can be directly deployed online without code modification. for user-specific architectures, lightseq supports multiple model reuse, which can be easily adapted with only a few lines of code modification. ## lightseq approach transformer-based nlp models mainly consist of two components during inference: the feature calculation layer and the output layer, as shown in figure 1 . 2 as of this writing, we use fastertransformer v2.1 for comparison. the feature calculation layer is mainly based on self-attention mechanism and feature transformation, which is actually implemented by matrix multiplication and a series of i/o-intensive operations such as element-wise (e.g., reshape) and reduce (e.g., layer normalization). the output layer slightly changes in different tasks, such as classification in nlu tasks or search (e.g., beam search) in nlg tasks. this layer is usually composed of the softmax over vocabulary, probability sorting, cache refreshing, etc., which are essentially i/o-intensive. these two components pose challenges for efficient inference: lightseq employs a series of innovative methods to address these challenges to accelerate model development, such as fusion of multiple kernel functions to reduce i/o overhead, hierarchical optimization of search algorithms to erase redundant calculations, and reuse of dynamic gpu memory to avoid run-time allocation. the following is a detailed introduction to these methods. ## experiments in this section, we will show the improvements of lightseq with different gpu hardware and precisions. we first analyze the gpu occupation of lightseq during inference to investigate if light-seq can make full use of gpu resources. then, we make a fair comparison with tensorflow, pytorch, fastertransformer, and turbotransformers on machine translation and text generation to show the efficiency of lightseq. ## conclusion in this paper, we address the deployment problem of expensive sequence models and present an efficient inference library lightseq for sequence processing and generation, reducing the gap between the performance of big models and the requirement of online services. comparisons with faster-transformer show that we perform better in both machine translation and text generation. in future work, we will focus on exploring more techniques to achieve a more significant speedup, including efficient integer-arithmetic-only inference and sparse gemm computations.