Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders (2021)

Abstract: Recent work in multilingual translation advances translation quality surpassing bilingual baselines using deep transformer models with increased capacity. However, the extra latency and memory costs introduced by this approach may make it unacceptable for efficiency-constrained applications. It has recently been shown for bilingual translation that using a deep encoder and shallow decoder (DESD) can reduce inference latency while maintaining translation quality, so we study similar speed-accuracy trade-offs for multilingual translation. We find that for many-to-one translation we can indeed increase decoder speed without sacrificing quality using this approach, but for one-to-many translation, shallow decoders cause a clear quality drop. To ameliorate this drop, we propose a deep encoder with multiple shallow decoders (DEMSD) where each shallow decoder is responsible for a disjoint subset of target languages. Specifically, the DEMSD model with 2-layer decoders is able to obtain a 1.8x speedup on average compared to a standard transformer model with no drop in translation quality.

## introduction encoder-decoder based neural machine translation (nmt) systems have achieved great success on bilingual translation tasks @xcite @xcite @xcite . recently, multilingual neural machine translation (mnmt) has also attracted much attention because of its ease of deployment, knowledge transfer among languages and the potential to enable zero-shot translation (dong et al., 2015; firat et al., 2016; ha et al., 2016; johnson et al., 2017; arivazhagan et al., * work done at facebook ai. 2019; @xcite . while mnmt can support translations in several directions, not all of them have better performance when compared to their corresponding bilingual models. suspecting that poor performance in some directions is due to the limited model capacity, many prior works adopt deeper encoder and decoder @xcite @xcite . however, increasing the number of layers, especially in the decoder, deteriorates the latency of translation and memory costs. recently, @xcite show that given a fixed capacity budget, as measured by the number of layers, models with a deep encoder and a shallow decoder (desd) are faster at inference time when compared to standard models with an equal number of encoder and decoder layers while maintaining translation quality. inspired by findings from @xcite , in this work, we explore the speed-accuracy trade-off in multilingual machine translation systems. given the same model capacity budget, we experiment various layer allocation strategies. we analyze multilingual models in the one-to-many (o2m) setting and many-to-one (m2o) setting. in the one-tomany setting, there are numerous target languages from a single source language (limited to english in this study); and in the many-to-one setting, several possible source languages are translated into a single target language (again, english in this study). in the many-to-one scenario, we find that allocating more capacity to the encoder reduces the latency while achieving comparable performance. we hypothesize that a deeper encoder helps the model accommodate multiple source languages, while a shallow decoder is sufficient to support a single target language. however, in the one-to-many translation setting, speed-accuracy trade-off is complicated. we observe a performance drop as the decoder depth is reduced. we hypothesize that the shallow decoder can no longer model several different target languages adequately. with the goal of obtaining low latency while maintaining translation quality, we propose using multiple shallow decoders where each decoder is responsible for a subset of the target languages. clearly, the introduction of multiple shallow decoders increases the size of our model. however, at inference time only one shallow decoder will be used, thus not adding latency or memory costs. with multiple target languages and decoders, one natural question is how to assign each target language to one of these decoders. we investigate several methods to assign each target language to one of these shallow decoders. more details are in the section 3. experimental results on three multilingual translation corpora show the effectiveness of our method to improve translation accuracy with lower latency at the same time. our main contributions are summarized as follows: however, the computation in the encoder is still parallelized given the source sentence. therefore, the main latency of the transformer at inference time happens in the decoder, especially translating long sentences. recently, @xcite find that on bilingual machine translation tasks, putting more capacity of the transformer model to the encoder substantially reduces the decoding time and maintain the performance at the same time. because this deep encoder and shallow decoder model achieves a superior speed-accuracy trade-off on bilingual translation tasks, in this section, we try to understand the layer allocations of transformer on the multilingual neural machine translation task given the same capacity budget which is measured by the number of layers in the encoder and decoder. we first experiment with three multilingual translation corpora. instead of just trying the shallowest possible decoder (1-layer), we train models with various configurations on each of these three corpora. other than the layer allocation, all the other hyperparameters and model configurations are the same among these models and the same training procedure is applied to these models (odel and training details are listed in appendix a.). to understand the speed and accuracy trade-off of the layer allocation, two metrics are reported: the results are shown in figure 1 . models with fewer decoder layers obtain higher decoding speed. ## deep encoder and multiple shallow decoders (demsd) we have seen that in one-to-many translation, desd models have a performance drop compared to the standard transformer. in order to preserve translation quality and low latency at the same time, we propose a model with a shared encoder and multiple shallow decoders (demsd), each of which is used to decode a subset of target languages. although this will introduce more parameters, at inference time only one shallow decoder is needed for a given translation (since the output language is fixed) thus the model incurs no extra latency or memory costs. one natural question that arises when using this multiple-decoder approach is how to assign output languages to each of the decoders. in this section, we explore several language assignment methods to assign each target language (or language group) to one of these multiple decoders. as a result, each decoder only needs to handle a disjoint subset of target languages. ## related work speed and accuracy are two important metrics to evaluate a machine translation system. in this work, we mainly discuss the transformer architecture @xcite . a number of works have explored various ways to improve its inference speed. @xcite adopt shallow decoder and layer trying to speed up the inference on cpus. @xcite and @xcite employee vocabulary reduction to speed up the softmax layer. @xcite employ a latent depth transformer model which prune layers during inference time to reduce the inference cost. there are also some works optimizing attention computations to speed up the inference speed @xcite @xcite . recently, @xcite places more capacity to the encoder side and keep an extremely shallow (one-layer) decoder to achieve a superior speed-accuracy trade-off. multilingual neural machine translation (mnmt) is an attractive field recently @xcite @xcite because mnmt tries to employ one model to translate more than one language pair, even including ones unseen during training (zero-shot translation). knowledge transfer among languages boosts the performance of low-resource languages. however, many works @xcite @xcite have shown the capacity bottleneck of translation when modeling many languages. therefore, before simply stacking more layers in the encoder and decoder, it is crucial to first understand how to balance the speed and accuracy given a fixed capacity budget. therefore, in this work, we try to understand various capacity allocations to achieve the best speed-accuracy trade-off. ## conclusion in this work, we study speed-accuracy trade-offs using various layer configurations for multilingual neural machine translation. we find that for manyto-one translation, deep encoder and shallow decoder (desd) models improve decoding speed while maintaining translation quality with the same model capacity. however, for one-to-many translation we do observe a drop in quality when the decoder depth is reduced. to mitigate the performance drop of desd models in one-to-many translation, we proposed using a shared encoder and multiple shallow decoders (demsd). our best demsd models with 2-layer decoders are capable of speeding up decoding by 1.8 times while achieving the same quality compared to a standard transformer. our work can be combined with techniques mentioned in section 6 such as optimized attention computation, vocabulary reduction, knowledge distillation, etc. we expect that these combinations will further improve the decoding speed and obtain a better speed-accuracy trade-off. this work can also be extended to other encoder-decoder applications beyond translation, such as question answering, dialogues, and so on. we will explore these directions in the future work. group id languages 0 ta, ml, te 1 de, nl, af 2 hi, si, ne, ur, mr, gu, bn 3 ps, fa 4 zh 5 lv 6 km, id, xh, he 7 ja 8 ko 9 th, my, vi 10 pl, sv, ar 11 fr, es, lt, ro, it, pt, gl 12 ka, mn, cs, ru, hr, uk, mk, sl 13 tr, kk, az 14 fi, et