Is Robustness Transferable across Languages in Multilingual Neural Machine Translation? (2023)

Abstract: Robustness, the ability of models to maintain performance in the face of perturbations, is critical for developing reliable NLP systems. Recent studies have shown promising results in improving the robustness of models through adversarial training and data augmentation. However, in machine translation, most of these studies have focused on bilingual machine translation with a single translation direction. In this paper, we investigate the transferability of robustness across different languages in multilingual neural machine translation. We propose a robustness transfer analysis protocol and conduct a series of experiments. In particular, we use character-, word-, and multi-level noises to attack the specific translation direction of the multilingual neural machine translation model and evaluate the robustness of other translation directions. Our findings demonstrate that the robustness gained in one translation direction can indeed transfer to other translation directions. Additionally, we empirically find scenarios where robustness to character-level noise and word-level noise is more likely to transfer.

## introduction the research interest in multilingual neural machine translation (mnmt) has been increasing in recent years @xcite @xcite . this is partially due to the capability of mnmt in translating multiple language pairs, including both high-and lowresource languages, with a single model, easing the deployment of machine translation systems. despite this capability, mnmt, like bilingual neural machine translation, is still vulnerable towards domain shift @xcite , perturbations and noises in words and sentences @xcite . furthermore, low-resource language * corresponding author. pairs in mnmt are more vulnerable to perturbations than high-resource language pairs, which is observed in our preliminary experiments. in the context of nlp, transfer learning has proven highly effective in various tasks, including sentiment analysis @xcite , named entity recognition @xcite , and question answering @xcite . existing research demonstrates that transfer learning is also present in mnmt models @xcite , e.g., knowledge transfer from high-resource language translation to low-resource language translation. while previous research has shed light on the effectiveness of knowledge transfer within different language pairs, the robustness transfer across diverse translation directions remains an open question. in this work, we aim to investigate whether and how robustness is transferable across languages in mnmt. to achieve this purpose, we attack the inputs for a particular translation direction and then observe the difference in performance of other translation directions on the noise test set. the model that we use is a standard multilingual translation model which shares the transformer encoderdecoder between language pairs. the adversarial attack is conducted in both one-to-many and manyto-one translation tasks. meanwhile, we select languages from the same and different language families for our experiments. for each language pair, we generate 3 types of noises (character-, word-, and multi-level noise) in the source language. after attacking the mnmt model with different noises in the source language, we evaluate the translation quality of the language pair that is attacked as well as that of other language pairs that are not attacked in terms of bleu. we are especially interested in the impact of the adversarial attack to an mnmt model on language pairs that are not attacked. to summarize, the contributions in this paper are: mul�lingual transformer en fr en de en es en it en fr en de en es en it train trained mul�lingual transformer fr de es it fr de es it test en en en en character noise word noise mul�-level noise character noise word noise mul�-level noise • we propose a robustness transfer analysis protocol to investigate the robustness transfer phenomenon in mnmt models. ## related work mnmt supports one-to-many, many-to-one, and many-to-many translation via partial or full parameter sharing @xcite @xcite . massively mnmt has been explored, where a single model supports a large number of languages, including high-and lowresource languages @xcite . in the context of massively mnmt, m2m-100, a nonenglish-centric mnmt model is proposed @xcite , which is capable of translating 100 language pairs. the successor to m2m-100, nllb-200, hugely extends the number of languages to 200 via model scaling @xcite . nmt is vulnerable towards both naturally occurring and synthetic noise, e.g., human typos, character swaps @xcite . many works have been done to improve the robustness of nmt, including black-box approaches @xcite @xcite and white-box methods @xcite . black-box attack methods include both characterand word-level attacks @xcite . character-level attacks are obtained usually by random character insertion, deletion, replacement and swapping, and word-level attacks can be performed via similar word insertion and replacement @xcite and so on. differently, white-box attacks are usually gradient-based @xcite . recent research has attempted to address the challenging task of enhancing the robustness of mnmt models. @xcite propose robust optimization methods to tackle the issue of imbalanced data distribution between high-and low-resource languages. @xcite attempt to address variations in data distributions across different domains by introducing robust domain adaptation techniques. different from the above works, our work focuses on the robustness of mnmt models towards noise and the transferability of robustness across languages in mnmt models. ## robustness transfer analysis protocol in order to explore the phenomenon of robustness transfer across translation directions, we propose a robustness transfer analysis protocol. as shown in figure 1 , the protocol consists of 2 phases, a training phase and a testing phase. during the training phase, we conduct blackbox attacks on the source side within a specific translation direction in a multilingual training corpus. this entails carrying out character-level, wordlevel, or multi-level black-box attacks. specifically, character-level black-box attacks involve random character insertion, deletion, replacement, and adjacent character swap operations. word-level blackbox attacks consist of random word swapping, deletion, insertion and replacement operations. for word replacement, we use pre-trained word embeddings to find words semantically related to the target word for replacement. the combination of character-level and word-level black-box methods constitutes the multi-level black-box approach. it is important to note that while we attack the source data in a particular translation direction, the target data in the same translation direction remains unchanged, as well as the data in other translation directions. lastly, the combined multilingual data are fed into the multilingual transformer for training. during the testing phase, we conduct a black-box attack on the source side of all translation directions on the multilingual test corpus. to maintain consistency in the distribution of the training and test data, we apply the same black-box attack operations to the test corpus as we do on the training corpus. the attacked source data are then fed into the multilingual transformer, which is trained in the previous phase. we measure the translation performance of the translation directions that are not attacked during training. additionally, we compare the translation performance of the model in this training setting with the translation performance of the model under other training settings. for instance, this can involve comparing it to the performance of a model trained on a clean corpus on the same test set, or to the performance of a model trained to perform other levels of black-box attacks on the same test set. ## experiments with the robustness transfer analysis protocol, we conducted extensive experiments to investigate whether and how robustness transfers across languages in multilingual nmt. ## conclusions in this paper, we have conducted an extensive and in-depth investigation on robustness transfer in multilingual neural machine translation models with the proposed robustness transfer analysis protocol. our empirical findings demonstrate that robustness can indeed transfer across languages. furthermore, we find that robustness towards character-level noise is more likely to transfer across related language while robustness towards word-level noise is more likely transfer across distant languages.