Counter-Interference Adapter for Multilingual Machine Translation (2021)

Abstract: Developing a unified multilingual model has been a long pursuing goal for machine translation. However, existing approaches suffer from performance degradation - a single multilingual model is inferior to separately trained bilingual ones on rich-resource languages. We conjecture that such a phenomenon is due to interference brought by joint training with multiple languages. To accommodate the issue, we propose CIAT, an adapted Transformer model with a small parameter overhead for multilingual machine translation. We evaluate CIAT on multiple benchmark datasets, including IWSLT, OPUS-100, and WMT. Experiments show that the CIAT consistently outperforms strong multilingual baselines on 64 of total 66 language directions, 42 of which have above 0.5 BLEU improvement.

## introduction machine translation (mt) is a core task in natural language processing. in recent years, neural machine translation (nmt) approaches have made tremendous progress and takes the lead in the field @xcite @xcite . conventionally, each nmt model only tackles a single language direction (e.g. english ñ german). a commonly used model like transformer has s " 240 million parameters. therefore, translating n language pairs requires training models separately for each direction, resulting in s ˆn total parameters. the huge size of all models for every language direction can be too costly to deploy, considering more than 100 popular languages worldwide. hence, developing a single unified yet parameter-efficient model for multilingual machine translation, enabling the translation of multiple directions, becomes crucially important. there is much effort towards multilingual ˚work is done while at bytedance. machine translation. @xcite first proposed training a single multilingual model via an additional target language tag, which became the paradigm for training multilingual mt models henceforth @xcite @xcite . their approach is simple and parameter-efficient, however, it often lags behind separately trained bilingual models, especially on resource-rich language pairs, where the phenomenon is known as performance degradation @xcite . this paper analyzes the performance degradation between multilingual mt models and separate bilingual ones. prior research suggests that unified embedding on a joint vocabulary leads to meaning conflation deficiency in multilingual training @xcite . for example, words with identical spelling may have distinct meanings in different languagesbride in english refers to a woman soon to get married while in french it means horse bridle. when it comes to machine translation, the effect goes beyond word embedding. as a single model has bounded capacity, the multilingual learning may cause negative influences among shared parameters @xcite . we conjecture that such a performance degradation is due to the interference across languages brought by joint training on multiple language directions. such interference affects both joint token embedding and representations from intermediate layers. we argue that resolving the interference is critical to improving multilingual translation performance. inspired by the insights above, we propose counter-interference adapter for multilingual machine translation (ciat). the ciat includes a major multilingual base model (i.e., a multilingual transformer), which is universally pre-trained on multiple language directions, and two kinds of designed adapter modules, which are trained on specific language directions. specifically, we propose embedding adapter and layer adapter to reduce multilingual interference among embedding and intermediate layers, respectively. we also seek a new parallel connection method for adapter units, which is more effective for multilingual mt than the previous series connection. we validate ciat among three real-world datasets and observe that ciat gains significant improvements in translation performance over all other multilingual baselines. our contributions are as follows: a) we analyze performance degradation in multilingual nmt and formulate it as two issues; b) we propose ciat, an adapter-based framework to tackle the two issues and enhance the performance of the multilingual model with small amounts of extra parameters; c) we demonstrate the efficacy of ciat through extensive experiments on iwslt, opus-100, and wmt benchmark datasets, surpassing other multilingual models over most of the translation directions. ## related work multilingual machine translation. the multilingual mt enjoys a rich research history, dating back to the age of statistical machine translation @xcite @xcite . in recent years, the prosperity of neural machine translation (nmt) has led to the growing prominence and popularity of multilingual mt systems. the encoder-decoder framework has made the de facto standard for nmt @xcite . @xcite did the pioneering work on extending conventional nmt to one-to-many translation, where the authors added a distinct decoder for each target language. @xcite further extended such framework into many-to-many settings by building exclusive encoders and decoders for each language. those attempts still faced problems such as low parameter utilization. on the other hand, @xcite treated all sources as the same language by translating on a character level. however, it only meets the many-to-one scenarios. @xcite managed to train a single model that applied to multiple translation directions. their solution is relatively simple: they attached a dedicated token at the beginning of the source sentence to specify the target language, while the rest of the model was shared among all languages. the paper has set a milestone of multilingual mt and has become the basis for most subsequent work. recent studies paid more attention to the performance improvement of multilingual models based on @xcite 's effort. several improved the model with external knowledge from human or other models: @xcite boosted the multilingual model by knowledge distillation, @xcite pre-clustered languages to assist similar languages. several studies enhance the model from data: @xcite and @xcite conducted data augmentation to low-resource languages via related high-resources or monolingual data. @xcite improved translation with relevant auxiliary languages. some other studies enhanced the transformer model by introducing language-aware modules and learning languagespecific representation @xcite . ## challenges on multilingual machine translation the ultimate goal of multilingual machine translation is to build a universal model to achieve mutual translation among all natural languages. that is, given a source sentence s and the target language l, the multilingual mt system shall output a sentence that resembles human reference t. currently, transformer @xcite ) gains popularity and becomes the paradigm for state-of-the-art nmt systems. here, we follow the recent implementations @xcite of the pre-norm transformer, whose layer normalization is applied to the input of each sub-layer. the transformation of i-th sublayer taking x i as input can be formulated as: the classic multilingual translation approach @xcite takes sentences from all language pairs and results in a universal model that translates across various languages. however, as previous researches have discovered, the multilingual model yields an inferior performance on high-resource languages compared to the bilingual models under the same configuration. in this paper, we reconsider the limitations of multilingual models and attribute the performance degradation to the following two issues: multilingual embedding deficiency camacho-collados and pilehvar (2018) addressed the meaning conflation deficiency problem of the word embedding as a single vector is limited for representing polysemy. we extend the meaning conflation deficiency into the multilingual scenario. generally, words/tokens may have unrelated or even opposite meanings in different languages. for example, "娘" denotes mother in chinese but daughter in japanese. as the word embeddings are usually jointly trained on a multilingual corpus, representing a multilingual word with just one single vector may burden the model's semantic representation. we refer to the problem as multilingual embedding deficiency. multilingual interference effects besides the word embedding, the insufficient capacity of a single nmt model also bottlenecks its performance on multilingual tasks @xcite . the parameter-sharing among different languages may be a potential cause of negative interference @xcite . we here formulate this phenomenon as multilingual interference effects; that is, when a single model tries to learn multiple languages simultaneously, the extracted language features interfere with each other impose adverse effects upon overall performance. accordingly, we regard the model trained on bilingual data offers the approximately optimal solution on language representation, compared to which the representation of the multilingual model is bias-influenced: ## proposed method we propose a counter-interference adapter for multilingual machine translation (ciat) to address the two issues mentioned above with adapterbased architectures: embedding adapter and layer adapter. figure 1 illustrates the overall architecture of ciat, which we will describe in detail. ## experiments we conducted experiments on three multilingual translation datasets to show the effectiveness of ciat. ## conclusion this work analyzes the performance degradation problem in multilingual nmt systems and decomposes into multilingual embedding deficiency and multilingual interference effects. we then propose a novel framework to deal with degradation, named counter-interference adapter for multilingual machine translation (ciat). ciat alleviates two issues above respectively by introducing two kinds of adapters. we validate the effectiveness of ciat on three multilingual translation datasets, where the results show that ciat improves the performance of the multilingual nmt model on various translation directions. the experiments also demonstrate that ciat variants outperform several strong baselines, approving our analysis and framework design. furthermore, we investigate the behavior and utility of each component via empirical studies. ciat-basic: besides removing all embedding adapters, this variant introduces only one layer adapter for each attention block. we design this variant to compare with serial @xcite under a similar amount of parameters to determine the effectiveness of our parallel connection further. we present the detailed results on two ciat variants on