F ast S eq: Make Sequence Generation Faster (2021)

Abstract: Transformer-based models have made tremendous impacts in natural language generation. However the inference speed is a bottleneck due to large model size and intensive computing involved in auto-regressive decoding process. We develop FastSeq framework to accelerate sequence generation without accuracy loss. The proposed optimization techniques include an attention cache optimization, an efficient algorithm for detecting repeated n-grams, and an asynchronous generation pipeline with parallel I/O. These optimizations are general enough to be applicable to Transformer-based models (e.g., T5, GPT2, and UniLM). Our benchmark results on a set of widely used and diverse models demonstrate 4-9x inference speed gain. Additionally, FastSeq is easy to use with a simple one-line code change. The source code is available at https://github.com/microsoft/fastseq .

## introduction transformer-based model architectures have made tremendous impact in multiple domains. however, due to large model size and intensive computing involved in the decoding process, the inference speed is still a bottleneck for long sequences applications @xcite . a variety of model architectural innovations have been proposed to increase the generation speed from different perspectives. one trend is to change the model architectures, like model distillation @xcite and sparse attention @xcite . although these techniques can alleviate the performance issue, there may be still some tradeoff between model accuracy and speed. on the other hand, efficient infrastructures have been de- * equal contribution † corresponding author veloped to accelerate the inference speed, e.g., ten-sorrt @xcite and fastertransformers foot_0 . in this paper, we present fastseq framework to make sequence generation faster. fastseq can accelerate the sequence generation by 4x to 9x with a simple one-line code change for models in fairseq @xcite and huggingface-transformers @xcite . the design principle of fastseq is to improve the inference speed without losing model accuracy and usability. our optimization approaches include an attention cache optimization, an efficient algorithm for detecting repeated n-grams, and an asynchronous generation pipeline with parallel i/o. these optimizations are general enough for a wide range of transformer-based model @xcite architectures, including the encoder-decoder architecture (e.g., @xcite @xcite , the decoder-only architecture (e.g., @xcite , and the encoder-only architecture (e.g., @xcite . fastseq is also designed to be flexible for extension on supporting other models and frameworks. our technologies are partially adopted by fairseq foot_1 . a demo video can be found at https: // www.youtube.com/watch?@xmath0. ## preliminary analysis for models with similar size, the sequence generation is much slower than classification, regression or language score computation. why is the generation so time-consuming? before analyzing the reasons, let's recap the generation algorithms first. ## design in order to address above bottlenecks, optimizations need to be done at multiple levels, including operations, models, and pipelines, which basically touch every component of a sequence generation framework. it is a non-trivial burden for researchers and practitioners. as a result, we develop this fast-seq library to address these barriers and speed up end-to-end inference in sequence generation. fastseq is designed with following features: (i) speed up the inference of sequence models without any accuracy loss; (ii) easy to use and compatible python apis with fairseq and huggingface-transformers; (iii) flexible to be extended to support new models and frameworks. fastseq is written in pytorch @xcite and composed of (1) ops module: provide efficient implementations of kernels (e.g., block n-gram repeats); (2) optimizer module: optimize model implementations in run-time, where more efficient implementations will be automatically patched to replace the ones in existing nlp toolkits (e.g., fairseq and huggingface-transformers) or the deep learning libraries (e.g., pytorch); (3) models module: define the model architectures (e.g., prophetnet, unilm). it is noteworthy that the models in fairseq and huggingface-transformers are natively supported as well. only one-line code change is needed to make them work with fast-seq; (4) command line interfaces (clis) module: run the inference via commands with an asynchronous pipeline, including preprocess (e.g., tokenization), generation process, and post-process (e.g., detokenization). these clis are compatible with fairseq and huggingface-transformers as well. users can use the same parameters to run their end-to-end inferences. fastseq is designed to be easy to use. existing model usages (e.g., model content and parameter settings) in fairseq and huggingface-transformers do not need to be changed. the example code can be found in below: ## optimizations to address the bottlenecks discovered in section 2.2, we develop following optimizations. ## evaluation in the benchmarks, fairseq and huggingface-transformers are used as the baseline to evaluate the performance. the selected models cover different kinds of architectures, including the encoderdecoder models (e.g., bart, distilbart, t5, prophetnet), the decoder-only models (e.g., gpt2), and the encoder-only models (e.g., unilm). cnn / daily mail dataset @xcite and wmt'16 @xcite are used as the benchmark datasets. the benchmark experiments are split into two groups 1) huggingface-transformers with/without fastseq; 2) fairseq with/without fastseq. if both fairseq and huggingface-transformers have implemented the model, we choose the faster result as the baseline. hardware the experiments are conducted on a node with 1 gpu (nvidia tesla v100 pcie 16gb ) and 24 cores cpu (intel(r) xeon(r) cpu e5-2690 v4 @ 2.60ghz). ## related work a variety of efforts have been developed to improve the efficiency of transformer models. from the perspective of model architectures, there are efforts on reducing attention matrix size by chunking input sequences into blocks @xcite , or using strided convolution over the keys and queries to compress memory @xcite . another kind of approaches focus on reducing model size and memory consumption by weight quantization @xcite , weight sharing @xcite , and weight pruning @xcite . knowledge distillation is another popular approach @xcite . on the other hand, a dozen of innovations on infrastructure side have been conducted to speed up serving of transformer models. the fused chains of basic operators in the attention layers have been widely adopted in many frameworks (e.g., onnx runtime foot_4 , deep speed foot_5 ). it is also performance critical to optimize data layout and movement among the connected operations @xcite . in situation of varied input lengths, turbotransformers @xcite is developed to better serve online models by using dynamic batch scheduler, more efficient memory allocation and deallocation algorithms. fastertransformers foot_6 deeply optimizes kernels of encoder, decoder and beam search to better utilize computer power of tensor core. ## conclusion in this work, we present fastseq, which provides general solutions for speeding up the sequence generation without accuracy loss. the proposed optimizations include an attention cache optimization, an gpu-based n-grams blocking algorithm, and an