T en T rans High-Performance Inference Toolkit for WMT 2021 Efficiency Task (2021)

Abstract: The paper describes the TenTrans’s submissions to the WMT 2021 Efficiency Shared Task. We explore training a variety of smaller compact transformer models using the teacher-student setup. Our model is trained by our self-developed open-source multilingual training platform TenTrans-Py. We also release an open-source high-performance inference toolkit for transformer models and the code is written in C++ completely. All additional optimizations are built on top of the inference engine including attention caching, kernel fusion, early-stop, and several other optimizations. In our submissions, the fastest system can translate more than 22,000 tokens per second with a single Tesla P4 while maintaining 38.36 BLEU on En-De newstest2019. Our trained models and more details are available in TenTrans-Decoding competition examples.

## introduction we participate in the gpu throughput track of the workshop on machine translation (wmt) 2021 efficiency shared task. the efficiency task aims at exploring the different techniques for training and optimizing gpu models for high throughput while preserving the highest possible accuracy. while we do not pay more attention to training techniques, we apply a variety of optimizations to improve the computation efficiency of our gpu models in the inference phase. in terms of the training phase, we trained a variety of smaller compact student models using the common teacher-student training approach @xcite on our opensource multilingual training platform tentrans-py. all of them are based on the deep transformer which has proven more effective and has lower training costs than the wide transformer models @xcite . for the inference phase, our strategy for the shared task includes attention caching, kernel fusion, early-stop, and several other optimizations. all of these optimizations are employed in a high-optimized and c++-based inference engine tentrans-decoding. the paper is structured as follows: section 2 describes the data preparation and the training details, then section 3 presents the variety of ours optimizations to improve decoding efficiency. the detailed accuracy and efficiency results are shown in section 4. finally, we conclude our work in section 5. ## teacher-student training to train smaller compact student models, the teacher-student training approach @xcite ) is adopted. first, a large model (the teacher) is trained on all available bilingual data, included synthetic data generated by the back-translation @xcite ) method. multiple model ensembles are also typically used to build stronger teacher systems. then, all our small optimized models (the student) are created using sequence-level knowledge distillation @xcite and trained on data generated from the teacher model. the sequencelevel knowledge distillation is a common technique that has proven successful for reducing the size of neural models, especially in nmt tasks. ## optimization results in the future version, we plan to optimize these points mentioned above. ## conclusion this work presents the tentrans's submissions to the 2021 efficiency shared task of wmt. we show the deep encoder and shallow decoder student models that training with sequence-level distillation can achieve a competitive performance both in speed and accuracy compared with the teacher baseline. to further improve computation efficiency, we combine several optimizations including attention caching, kernel fusion, early-stop and sorted batch. finally, our fastest student model achieves a speedup of 3.58x times, while only has one-sixth parameters of the teacher baseline. in the future, we will apply low-precision inference (e.g., fp16) and more kernel fusion techniques to improve the computation efficiency of our gpu systems. furthermore, we will continue to explore a more efficient teacher-student training approach to obtain compact student models with competitive performance both in quality and speed.