Efficient Inference for Multilingual Neural Machine Translation (2021)

Abstract: Multilingual NMT has become an attractive solution for MT deployment in production. But to match bilingual quality, it comes at the cost of larger and slower models. In this work, we consider several ways to make multilingual NMT faster at inference without degrading its quality. We experiment with several “light decoder” architectures in two 20-language multi-parallel settings: small-scale on TED Talks and large-scale on ParaCrawl. Our experiments demonstrate that combining a shallow decoder with vocabulary filtering leads to almost 2 times faster inference with no loss in translation quality. We validate our findings with BLEU and chrF (on 380 language pairs), robustness evaluation and human evaluation.

## introduction multilingual machine translation @xcite @xcite @xcite has made a lot of progress in the last years. it is attractive because it allows handling multiple language directions within a single model, thus significantly reducing training and maintenance costs. however, to preserve good performance across all the language pairs, both the vocabulary size and model size have to be increased compared to bilingual nmt, which hurts inference speed. for example, the recently released m2m-100 @xcite has 15b parameters and needs multiple gpus for inference. the problem of inference speed has been well studied in bilingual settings @xcite @xcite @xcite . one line of work consists in using lighter decoder architectures (e.g., shallow decoder: @xcite @xcite . these works demonstrate that it is possible to significantly speed up inference with almost no loss in translation quality as measured by bleu. figure 1 compares the inference time spent in each nmt component by bilingual and multilingual models of the same architecture but with different vocabulary sizes. the decoder is also the bottleneck in the multilingual model which suggests that we can expect similar speed gains with a lighter decoder. it also indicates that some speed gain could be obtained by reducing vocabulary size (which impacts both beam search and softmax). however, it is not so obvious that lighter decoder architectures would preserve translation quality in multilingual settings, where the decoder may need more capacity to deal with multiple languages. therefore, the goal of this work is to benchmark different architectures in terms of inference speed/translation quality trade-off and to identify the best combination for multilingual nmt. the contributions of this paper are: nmt architectures (deep encoder / shallow decoder, @xcite on two multilingual datasets (ted talks and paracrawl) in both englishcentric and multi-parallel settings. it demonstrates that the previous findings transfer to multilingual models. although most of the above works report speed improvements with similar bleu scores as transformer baselines, it is uncertain that the same will hold in multilingual many-to-many settings, where the decoder may need more capacity to deal with multiple languages. in particular, @xcite observe that single shallow decoders degrade oneto-many mt quality and propose to train shallow language-specific decoders or decoders that are specific to a language family or group of languages. @xcite ; @xcite propose modular mt models with jointly trained language-specific encoders and decoders. such models have higher per-language capacity, increasing their performance without hurting inference speed (contrary to the common approach of training bigger multilingual models). they are also more flexible for adding new languages. @xcite study how languagespecific and language-independent parameters naturally emerge in multilingual nmt. their findings indicate that language-independent parameters can be distributed within the encoder and decoder and benefit final nmt performance. ## conclusion on one hand, multilingual nmt saves training and deployment costs. on the other hand, larger architectures (required to keep performance on a par with bilingual mt) and large shared vocabularies penalize inference speed and user latency. in this work, we study various approaches to improve the speed of multilingual models without degrading translation quality. we find that transformers with a deep encoder and a shallow decoder can outperform a baseline transformer at a much faster decoding speed. this can be combined with per-language vocabulary filtering to reach a global 2× speed-up with no loss in bleu. a careful analysis of the results on different aspects such as sequence length, robustness to noise, and human evaluation validates this finding. additionally, language-specific shallow decoders can be trained to get even better performance at the same speed. and finally, hybrid models with a shallow rnn decoder offer an excellent bleu-speed trade-off (3× faster than baseline with a minor drop in bleu). we also provide supplementary material to facilitate reproducibility.