Modularized Multilingual NMT with Fine-grained Interlingua (2024)

Abstract: Recently, one popular alternative in Multilingual NMT (MNMT) is modularized MNMT that has both language-specific encoders and decoders. However, due to the absence of layer-sharing, the modularized MNMT failed to produce satisfactory language-independent (Interlingua) features, leading to performance degradation in zero-shot translation. To address this issue, a solution was proposed to share the top of language-specific encoder layers, enabling the successful generation of interlingua features. Nonetheless, it should be noted that this sharing structure does not guarantee the explicit propagation of language-specific features to their respective language-specific decoders. Consequently, to overcome this challenge, we present our modularized MNMT approach, where a modularized encoder is divided into three distinct encoder modules based on different sharing criteria: (1) source language-specific ( Enc_{s} ); (2) universal ( Enc_{all} ); (3) target language-specific ( Enc_{t} ). By employing these sharing strategies, Enc_{all} propagates the interlingua features, after which Enc_{t} propagates the target language-specific features to the language-specific decoders. Additionally, we suggest the Denoising Bi-path Autoencoder (DBAE) to fortify the Denoising Autoencoder (DAE) by leveraging Enc_{t} . For experimental purposes, our training corpus comprises both En-to-Any and Any-to-En directions. We adjust the size of our corpus to simulate both balanced and unbalanced settings. Our method demonstrates an improved average BLEU score by "+2.90” in En-to-Any directions and by "+3.06” in zero-shot compared to other MNMT baselines.

## introduction neural machine translation (nmt) has been a cornerstone of machine translation success @xcite @xcite . the success of nmt systems lies in their end-to-end encoder-decoder architecture @xcite @xcite . furthermore, multilingual neural machine translation (mnmt) has gained significant attention due to its ability to perform multitasking by translating multiples language pairs using a single model @xcite . @xcite propose encoder-decoder full layer sharing where a single encoder-decoder translates multiple language pairs. they also prefix the input sentence tokens with a target language token and perform zero-shot translation, where the model trains with the source and target languages paired to a pivot language rather than directly paired with each other. modularized mnmt @xcite utilizes language-specific encoders and decoders to alleviate the capacity constraints in mnmt. by doing so, it demonstrates its potential to address the capacity limitations in the mnmt domain. nevertheless, it possesses certain drawbacks such as poor zero-shot performance and increased training complexity @xcite . @xcite argued that the lack of layer sharing hinders encoders from producing language-independent (interlingua) features, leading them to develop a modularized encoder consisting of two distinct encoder modules based on sharing strategies: (i) source language-specific (enc s ); (ii) universal (enc all ), as illustrated in figure 1 @xcite . to enhance the zero-shot performance for english-centric directions, they also introduced the denoising autoencoder (dae), as illustrated in figure 2 (a) . however, according to @xcite , the interlingua feature generated by enc all might lose the language signal, which can potentially lead to overall possible performance deterioration. moreover, the dae of the complete sharing of the upper encoder layers scarcely produces language-specific figure 2: illustration of different denoising autoencoders. (a) denoising autoencoder (dae) with enc s and enc all . (b) our denoising bi-path autoencoder (dbae) where output of enc all bifurcates toward both enc t ′′ (e.g., english in our case) and enc t ′ . outputs from both enc t ′′ and enc t ′ are weighted-summed before the decoder. in this work, to alleviate these shortcomings, we propose a new modularized encoder that is divided into three encoder modules with different sharing strategies as follows: (i) source languagespecific (enc s ); (ii) universal (enc all ); and (iii) target language-specific (enc t ). our proposed modularized encoder propagates interlingua features through enc all , and the target languagespecific language features to the language-specific decoders through enc t . besides, to resolve the alignment issue from the dae, we propose a denoising bi-path autoencoder (dbae) as illustrated in figure 2 (b). in the dbae, the output of enc all bifurcates and is propagated to both the target language-specific (enc t ′ ) and the pivot language-specific (enc t ′′ ) encoder modules. the outputs of both enc t ′ and enc t ′′ are weighted-summed, and the decoder utilizes the weighted-summed features. by incorporating the enc t ′′ , we attempt to align the encoder output feature with the pivot languagespecific feature that the non-pivot language-specific decoders prefer. we provide further details regarding the implementation of dbae in 3.2. in this work, we use an english-centric dataset to train all models for experiments. to assess our method, we make use of other mnmt methods as baselines for the experiment, which include (i) @xcite , (ii) @xcite , (iii) @xcite , and (iv) @xcite . through the experiments, we demonstrate that our modularized mnmt outperforms other mnmt baselines across both english-centric and zero-shot test sets. ## method this section introduces how to implement our modularized mnmt. we provide details about our modularized encoder and then explain our denoising bi-path autoencoder (dbae). lastly, we put forth different inference hypotheses for zero-shot. ## result in this section, we evaluate our proposed method against other baselines. for all experiments, the model configurations are provided in appendix c for all experiments. the complete bleu scores for all the experiments in this section are presented in appendix d. ## conclusion in this work, we propose a new modularized mnmt with a modularized encoder consisting of the encoder modules of which each implements effective sharing strategies. for the zero-shot, we develop denoising bi-path autoencoder (dbae) to train the non-pivot language-specific decoders to be robust to the output by the non-pivot language. in the experiment, we demonstrate that our modularized encoder enhances the overall performance of the english-centric tests. the dbae also improves the zero-shot performance. ultimately, we hope that our work sheds light on the modularized mnmt once more and believe that the modularized mnmt can serve as a path to the entrance of the optimal state of mnmt.