Revisiting M odularized M ultilingual NMT to M eet I ndustrial D emands (2020)

Abstract: The complete sharing of parameters for multilingual translation (1-1) has been the mainstream approach in current research. However, degraded performance due to the capacity bottleneck and low maintainability hinders its extensive adoption in industries. In this study, we revisit the multilingual neural machine translation model that only share modules among the same languages (M2) as a practical alternative to 1-1 to satisfy industrial requirements. Through comprehensive experiments, we identify the benefits of multi-way training and demonstrate that the M2 can enjoy these benefits without suffering from the capacity bottleneck. Furthermore, the interlingual space of the M2 allows convenient modification of the model. By leveraging trained modules, we find that incrementally added modules exhibit better performance than singly trained models. The zero-shot performance of the added modules is even comparable to supervised models. Our findings suggest that the M2 can be a competent candidate for multilingual translation in industries.

## introduction with the current increase in the demand for neural machine translation (nmt), serving an increasing number of languages poses a practical problem for the industry. a naive approach for multilingual nmt is to have multiple single-directional models, which is unsustainable owing to the quadratic increase of models as more languages are introduced. a more practical approach is to limit the number of models by sharing the components among the models @xcite @xcite . in addition to reducing the number of parameters, sharing the components is also regarded as an effective method to enhance the performance. a fully shared model (henceforth 1-1), which only uses one encoder and one decoder to translate all directions @xcite , has been the most popular method because of its compactness. however, introduction of a significant number of tasks into a 1-1 model is known to cause capacity bottleneck. @xcite suggested that, given a fixed model capacity, a 1-1 model is bound to the tradeoff between the number of languages and translation accuracy. zhang et al. (2020) explicitly identified the capacity bottleneck problem of the 1-1 model by showing a clear decrease in performance when translation directions are doubled. moreover, data unbalance complicates the problem. @xcite presented the transfer and interference dilemma among low and high resource languages in an unbalanced environment. the capacity bottleneck observed in the 1-1 model is particularly undesirable for the industry. unlimited scaling of the model size @xcite is impossible in practice, where inference cost and latency are crucial. with limited capacity, gain from multilingual translation training (henceforth multi-way training) without being subject to the losses of the capacity bottleneck is difficult to achieve. furthermore, modification of the 1-1 model such as simple addition of a language is troublesome because the entire model must be retrained from the beginning as a single module, thus requiring a considerable amount of time and effort. this low maintainability makes 1-1 less attractable for industrial use. still, the benefit from multi-way training is difficult to miss. these problems lead us to revisit the multilingual neural machine translation model that share left is a collection of single models for 6 translation directions. middle is the 1-1 model that share the whole parameters of the model for 6 directions. right is the m2 model that only share language-specific modules. figure 1 illustrates the architectural overview of multilingual translation using single models, the 1-1 and the m2. although the m2 has not been given substantial attention owing to the linear increase in its parameters as the number of languages increases, it is relatively free from the capacity bottleneck problem while maintaining a reasonable inference cost. in this study, we explore the possibility of m2 as an alternative to the 1-1 model in industrial settings. to resolve the capacity bottleneck problem while enjoying the benefits, we identify the effects of multi-way training in a carefully controlled environment. we find that the data-diversification and regularization of multi-way training enable the m2 to outperform both single and 1-1 models with less suffering from capacity bottlenecks. additionally, the m2 demonstrates a comparable performance increase to 1-1 for low resource pairs in an unbalanced environment. combined with its modularizable architecture, interlingual space learned by the m2 allows convenient and effective modification of the model. the simple addition of language-specific modules to the m2 outperformed an individually trained model. the zero-shot learning of the incremented language module outperforms english pivoted translation and is even comparable to a supervised model. finally, we show that the language invariance of such space improves with more languages. in summary, our contribution is threefold. 1) we conceptually specify the effects of multi-way training and verified them with comprehensive experiments. 2) we show that the m2 can leverage those effects as the 1-1 without the constraint of the capacity bottleneck. 3) finally, we find that multi-way training of the m2 forms interlingual space which allows simple yet effective extension of languages. ## effects of multi-way training because of its complexity, the effects of multi-way training are yet to be identified. various factors may affect the performance of multilingual translation: model size compared to the amount of data, the number of training directions, the degree of data imbalance among different directions, and the portion of multi-parallel data. in this section, we discuss the possible effects on performance resulting from these factors. capacity bottleneck a capacity bottleneck is the most plausible cause of performance degradation in multi-way training. for a fixed size model, the capacity bottleneck is more prominent with the increase in training directions (especially target languages) and the amount of data @xcite @xcite . cross-language effect cross-language effect occurs when multiple languages are shared in a module. low resource languages reportedly benefit from multi-way training when trained along with high resource pairs @xcite @xcite . the interaction among languages in a module can either be positive (transfer) or negative (interference) on the performance according to their similarity in linguistic patterns. data-diversification data-diversification is associated with the portion of multi-parallel among multi-way data. if either the source-side or the target-side language is shared across two directions and data of the directions is not multi-parallel, the shared module learns more diverse samples of the language. for example, if an english encoder is shared between en-de and en-fr directions (and english sentences of two are not completely shared), the encoder learns more diverse english sentences from both pairs. few studies distinguished this effect @xcite . we refer to the improvement resulting from this factor as the data-diversification effect. regularization learning to encode or decode the same language in various directions may result in better representation learning and less overfitting in a single direction. this effect has already been observed by @xcite as the benefit of generalization and suggested by @xcite to benefit many-to-many models compared to many-to-one models. ## comparison of single models, 1-1 and m2 we compared the models with the same inference capacity in a series of conditions. note that most of the multilingual nmt research was conducted in a joint one-to-many and many-to-one environment (jm2m): collected data are english-centered. despite its simplicity, observations under such setting may be unreliable to speak for many-to-many (m2m) environment, which is also clearly in demand in the industry. therefore, we set m2m training as the default. we also distinguish between two different dataset compositions: the sharing case where all language pairs share the same sentence set, and the non-sharing case where there is no overlap between different pairs. to illustrate, a multiparallel set 'en -es -ko' can be shared for all possible three pairs (en -es, en -ko, es -ko) or used only once for one pair. considering that multiparallel data is rare in practice, we compared the models in a strictly non-sharing environment. ## interlingual space of the m2 creating interlingual space has been an active research area @xcite @xcite @xcite because it is critical to scaling out languages, such as incremental learning. because input of m2 does not contain any information regarding the target language, encoders need to encode it so that any decoder can translate. at the same time, decoders of the m2 should be able to generate from output of any m2 encoder. for this reason, we assume that the output space of m2 encoders is interlingual. figure 2 illustrates the interlingual space of a m2. multi-way training of 3 languages (en, es and ko) forms the interlingual space which is shared by 6 modules. this space is preserved as long as the weights of the m2 are frozen. training a new module (ja) with a single parallel corpus (en -ja) using one of the frozen modules (en) adapt the module to the interlingual space. we speculate that the new module (ja) would be compatible with the other modules (es and ko) if the interlingual space is formed well. we verify this using incremental zero-shot learning. additionally, we measure how the language invariance of the space changes as the number of languages involved in the m2 varies. since maintainability is one of the critical needs in practice, high performance on incremental learning would be a desirable trait in industrial settings. ## conclusion in this study, we re-evaluate the m2 model and suggest it as an appropriate choice for multilingual translation in industries. by extensively comparing the single models, 1-1 model, and m2 in varying conditions, we find that the m2 can benefit from multi-way training through data-diversification and regularization while suffering less from capacity bottlenecks. additionally, we demonstrate that the m2 can also benefit low resource pairs in an unbalanced environment as a 1-1 model without being subject to cross-language effect. next, we suggest that the m2 model is easily maintainable because of its interlingual space. the interlingual space not only enables incremental training in a simple manner, but also accompanies competitive incremental zero-shot performance. furthermore, we validate that the language invariance of the space enhances as the number of languages in the m2 increases. we hope that this study sheds light on the relatively disregarded m2 model and provide a benchmark for selecting a model among varying levels of shared components. resource pairs 1:1:1 1:2:4 1:5:25