  The ultra-large-scale pre-training model can effectively improve the effect
of a variety of tasks, and it also brings a heavy computational burden to
inference. This paper introduces a series of ultra-large-scale pre-training
model optimization methods that combine algorithm characteristics and GPU
processor hardware characteristics, and on this basis, propose an inference
engine -- Easy and Efficient Transformer (EET), Which has a significant
performance improvement over the existing schemes.
  We firstly introduce a pre-padding decoding mechanism that improves token
parallelism for generation tasks. Then we design high optimized kernels to
remove sequence masks and achieve cost-free calculation for padding tokens, as
well as support long sequence and long embedding sizes. Thirdly a user-friendly
inference system with an easy service pipeline was introduced which greatly
reduces the difficulty of engineering deployment with high throughput. Compared
to Faster Transformer's implementation for GPT-2 on A100, EET achieves a
1.5-15x state-of-art speedup varying with context length.EET is available
https://github.com/NetEase-FuXi/EET.
