Efficient and High-Quality Neural Machine Translation with O pen NMT (2020)

Abstract: This paper describes the OpenNMT submissions to the WNGT 2020 efficiency shared task. We explore training and acceleration of Transformer models with various sizes that are trained in a teacher-student setup. We also present a custom and optimized C++ inference engine that enables fast CPU and GPU decoding with few dependencies. By combining additional optimizations and parallelization techniques, we create small, efficient, and high-quality neural machine translation models.

## introduction this paper describes the opennmt @xcite submissions to the workshop on neural generation and translation 2020 efficiency shared task. for wnmt 2018, we explored training and optimizations of small lstm translation models combined with a customized runtime @xcite . while this resulted in interesting decoding speed, there was still room for improvements in terms of quality, memory usage, and overall efficiency. for this 2020 edition, we focus on the standard transformer architecture @xcite that is now commonly used in production machine translation systems. similar to our first participation, we train smaller models using the teacherstudent technique @xcite . we experiment with several encoder and decoder sizes following the work by @xcite which shows that reducing the number of decoder layers can improve decoding speed at a very limited accuracy cost. we also keep the approach of running the models with a custom c++ runtime. this year we present ctranslate2 1 , an optimized and production-grade 1 https://github.com/opennmt/ ctranslate2 inference engine for opennmt models that enables fast cpu and gpu decoding with few dependencies. this library implements several optimizations for decoding neural machine translation models such as 8-bit quantization, parallel translations, caching, and dynamic target vocabulary reduction. section 2 of this paper describes the data preparation and the training procedures we apply to train the candidate models. section 3 presents the various optimizations we implemented to reduce model size and improve runtime efficiency. finally, section 4 details the accuracy and efficiency results achieved by the submitted models. ## teacher-student training we train our systems using a teacher-student approach @xcite . first, a large model (the teacher) is trained on all available bilingual data, including synthetic data such as backtranslations of monolingual target sentences @xcite and translations of monolingual source sentences @xcite . model ensembles are also typically used to build stronger teacher systems. then, a small model (the student) is trained by means of minimizing the loss between the student and teacher systems with the goal of distilling the knowledge of the teacher @xcite into a smaller model with comparable accuracy results. @xcite show that student models can even outperform to some extent their teacher counterparts. knowledge distillation is an effective approach to reduce the model size, thus lowering memory and computation requirements. ## inference optimizations all models are converted and executed with ctrans-late2. we use the version 1.10.0 of the library. ## optimization results finally, ## conclusion we demonstrated that the opennmt ecosystem can be used to train efficient and high-quality neural machine translation models. the training frameworks-opennmt-tf and opennmt-pyinclude all features and procedures that are commonly applied to reach competitive translation scores. this year we presented ctranslate2, an optimized and production-grade inference engine for opennmt models that enables fast cpu and gpu decoding with few dependencies. by combining several optimizations and parallelization techniques, the library can drastically improve decod-