An Empirical Investigation of Word Alignment Supervision for Zero-Shot Multilingual Neural Machine Translation (2021)

Abstract: Zero-shot translations is a fascinating feature of Multilingual Neural Machine Translation (MNMT) systems. These MNMT models are usually trained on English-centric data, i.e. English either as the source or target language, and with a language label prepended to the input indicating the target language. However, recent work has highlighted several flaws of these models in zero-shot scenarios where language labels are ignored and the wrong language is generated or different runs show highly unstable results. In this paper, we investigate the benefits of an explicit alignment to language labels in Transformer-based MNMT models in the zero-shot context, by jointly training one cross attention head with word alignment supervision to stress the focus on the target language label. We compare and evaluate several MNMT systems on three multilingual MT benchmarks of different sizes, showing that simply supervising one cross attention head to focus both on word alignments and language labels reduces the bias towards translating into the wrong language, improving the zero-shot performance overall. Moreover, as an additional advantage, we find that our alignment supervision leads to more stable results across different training runs.

## introduction multilingual neural machine translation (mnmt) focuses on translation between multiple language pairs through a single optimized neural model, and has been explored from different angles witnessing a rapid progress in recent years @xcite @xcite . besides the great flexibility mnmt models offer, they are also highlighted by their so called zero-shot translation capabilities, i.e., translating between all combinations of languages available in the training data, including those with no parallel data seen at training time @xcite @xcite . many studies have investigated this feature, focusing on the impact of both, the model architecture design @xcite and data pre-processing @xcite @xcite . broadly speaking, mnmt architectures are categorized according to their degree of parameter sharing, from fully shared @xcite to the use of language-specific components @xcite @xcite . @xcite mnmt model is widely used, due to its simplicity and good translation quality. it uses the fully shared parameters setting, and relies on appending an artificial language label to each input sentence to indicate the target language. while this method allows for zeroshot translation, several works have highlighted two major flaws: i) its failure to reliably generalize to unseen language pairs, ending up with the so called off-target issue, where the language label is ignored and the wrong target language is produced as a result @xcite , ii) its lack of stability in translation results between different training runs @xcite . in this work, we investigate the role of guided alignment in the @xcite setting, by jointly training one cross attention head to explicitly focus on the target language label. we show that alignment supervision mitigates the off-target translation issue in the zero-shot case. our method improves the zero-shot translation performance and results in more stable results across different training runs. a ⊆ {(j, i) : @xmath0, ..., n ; @xmath1, ..., m } (1) we study three different settings: (a) standard word alignment between corresponding words, (b) alignments between all target words and the language label in the input string, and (c) the union between the former two. figure 1 shows an example of those approaches. to produce word alignments between parallel sentences, i.e., figure 1 (a) , we use the awesome-align tool @xcite , a recent work that leverages multilingual bert @xcite to extract the links. 1 models. to train many-to-many mnmt models, we use a 6-layer transformer architecture @xcite , prepending a language label in the input to indicate the target language @xcite . following @xcite , given an alignment matrix am m,n and an attention matrix computed by a cross attention head ah m,n , for each target word i, we use the following cross-entropy loss l a to minimize the kullback-leibler divergence between ah and am : ## experimental setup we use three highly multilingual mt benchmarks: following related work @xcite , we apply joint byte-pair encoding (bpe) segmentation @xcite , with a shared vocabulary size of 32k symbols for ted talks and 64k for wmt-2018 and opus-100. as evaluation measure, we use tokenized bleu @xcite to be comparable with @xcite for the ted talks benchmark, and sacrebleu 5 @xcite for wmt-2018 and opus-100. 6 as an additional evaluation, we report the target language identification accuracy score for the zeroshot cases @xcite , called acc zero . we use fasttext as a language identification tool @xcite , counting how many times the translation language matches the reference target language. the transformer models follow the base setting of @xcite , with three different random seeds in each run. all of them are trained on the many-to-many english-centric scenario, i.e., on the concatenation of the training data having english either as the source or target language. details about data and model settings in the appendix. ## results and discussion throughout this section we refer to our baseline mnmt models by the labels 1 and 2 , while 3 , 4 , and 5 mark the models trained with the auxiliary alignment supervision task, (a), (b), (c) from figure 1 respectively (see section 2). ## conclusions and future work in this work we present an empirical comparative evaluation of integrating different alignment methods in transformer-based models for highly multilingual english-centric mt setups. our extensive evaluation over three alignment variants shows that adding alignment supervision between corresponding words and the language label consistently improves the stability of the models, resulting in stable performance across different runs and mitigating the off-target translation issue in the zero-shot scenario. we believe that our work will pave the way for designing new and better multilingual mt models to improve their generalization in zero-shot setups. as future work, we intend to analyze the quality of the learned alignments and their effect on the other attention weights in both supervised and zeroshot evaluation data @xcite @xcite . finally, we plan to explore other mechanisms to inject prior knowledge to better handle zero-shot translations @xcite @xcite .