The N iu T rans System for the WMT 2021 Efficiency Task (2021)

Abstract: This paper describes the NiuTrans system for the WMT21 translation efficiency task. Following last year’s work, we explore various techniques to improve the efficiency while maintaining translation quality. We investigate the combinations of lightweight Transformer architectures and knowledge distillation strategies. Also, we improve the translation efficiency with graph optimization, low precision, dynamic batching, and parallel pre/post-processing. Putting these together, our system can translate 247,000 words per second on an NVIDIA A100, being 3 \times faster than our last year’s system. Our system is the fastest and has the lowest memory consumption on the GPU-throughput track. The code, model, and pipeline will be available at NiuTrans.NMT.

## introduction large and deep transformer models have dominated machine translation (mt) tasks in recent years @xcite @xcite . despite their high accuracy, these models are inefficient and difficult to deploy @xcite @xcite . many efforts have been made to improve the translation efficiency, including efficient architectures @xcite , quantization @xcite , and knowledge distillation @xcite . this work investigates efficient transformers architectures and optimizations specialized for different hardware platforms. in particular, we study deep encoder and shallow decoder transformer models and optimize them for both gpus and cpus. starting from an ensemble of three deep transformer teacher models, we train various student models via sequence-level knowledge distil-1 http://statmt.org/wmt21/ efficiency-task.html 2 https://github.com/niutrans/niutrans. nmt lation (skd) @xcite @xcite and data augmentation @xcite . we find that using a deep encoder (6 layers) and a shallow decoder (1 layer) gives reasonable improvements in speed while maintaining high translation quality. we improve the student model's efficiency by removing unimportant components, including the ffn sub-layers and multi-head mechanism. we also explore other model-agnostic optimizations, including graph optimization, dynamic batching, parallel pre/postprocessing, 8-bit matrix multiplication on cpus, and 16-bit computation on gpus. section 2 describes the training procedures of the deep teacher models. then, section 3 presents various optimizations for reducing the model size, improving model performance and efficiency. finally, section 4 details the accuracy and efficiency results of our submissions for the shared efficiency task. ## model overview following @xcite , @xcite and @xcite , we use the skd method to train our models. our experiments also show that the skd method can obtain better performance than the word-level knowledge distillation (wkd) method, similar to @xcite . therefore, all of student models are optimized by using the interpolated skd method @xcite , and trained on data generated from the teacher models. ## optimizations for decoding our optimizations for decoding are implemented with niutensor foot_3 . the optimizations can be divided into three parts, including optimizations for cpus, gpus, and device-independent techniques. ## conclusion we have described our systems for the wmt21 shared efficiency task. we have explored various efficient transformer architectures and optimizations specialized for both cpus and gpus. we have shown that a lightweight decoder and proper optimizations for different hardware can significantly accelerate the translation process with slight or no loss of translation quality. our fastest gpu system with three encoder layers and one decoder layer is 11× faster than the deep ensemble model and lose 1.9 bleu points.