Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer (2021)

Abstract: Multilingual pre-trained models have achieved remarkable performance on cross-lingual transfer learning. Some multilingual models such as mBERT, have been pre-trained on unlabeled corpora, therefore the embeddings of different languages in the models may not be aligned very well. In this paper, we aim to improve the zero-shot cross-lingual transfer performance by proposing a pre-training task named Word-Exchange Aligning Model (WEAM), which uses the statistical alignment information as the prior knowledge to guide cross-lingual word prediction. We evaluate our model on multilingual machine reading comprehension task MLQA and natural language interface task XNLI. The results show that WEAM can significantly improve the zero-shot performance.

## introduction large-scale multilingual pre-trained language models such as mbert @xcite , xlm @xcite and xlm-r @xcite have shown significant effectiveness in transfer learning on various cross-lingual tasks. the pre-training methods of the multilingual language models can be divided into two groups: unsupervised pre-training like multilingual masked language model (mmlm) @xcite , and supervised pretraining like translation language model (tlm) @xcite . in the mmlm, the model predicts the masked tokens with the monolingual context; in the tlm, the model can attend to both the contexts in the source language and target language. variations of tlm model can be found in @xcite ; @xcite ; @xcite . while it is possible for the model to learn the alignment knowledge by itself, some works have investigated injecting prior knowledge to the model to help it to align better. @xcite proposed a bilingual pre-training model for mbert, where it identifies matched word pairs in parallel bilingual corpora using unsupervised standard techniques such as fastalign @xcite , and aligns the contextual representations between the matched words with a similarity loss function. the previous works focus on aligning the contextual representations of the pre-trained models. in this paper, we propose a new cross-lingual pre-trained model called word-exchange aligning model (weam). different from previous works, we align the static embeddings and the contextual representations of different languages in the multilingual pre-trained models. specifically, in the pre-training stage, we first use fastalign to identify bilingual word pairs in parallel bilingual sentence pairs as our prior knowledge. then we randomly mask some tokens in the bilingual sentence pairs. for each masked token, weam performs two kinds of predictions: a multilingual prediction and a cross-lingual prediction. the multilingual prediction task predicts the original masked word in the standard way. while the cross-lingual task predicts the corresponding word from the representations in the other language. for example, if the words apple and apfel (german for apple) appear in the the english-german parallel sentence and apple is masked in the sentence, weam takes the representation of the masked apple and apfel for multilingual prediction and crosslingual prediction respectively to recover the original word apple. through the two ways of prediction, both the contextual representations from the last transformer layer and the static embeddings from the embedding layer can be aligned. we evaluated our method on the word-level machine reading comprehension task mlqa @xcite and the sentencelevel classification task xnli @xcite al., cross-lingual prediction mask multilingual prediction word-exchange multilingual pre-trained language model bilingual parallel sentences word alignment matrix it raining is 吗 [cls] 下⾬ [sep] 今天 today ? [sep] ？ [pad] it [mask] [mask] 吗 [cls] 下⾬ [sep] [mask] today ? [sep] ？ [pad] raining is 今天 raining is 今天 1 1 1 今天 下⾬ 吗 ？ [pad] is it raining today ? figure 1 : an overview of the word-exchange aligning model (weam). for each language pair, there are two tasks. the multilingual prediction task predicts the masked tokens. the cross-lingual prediction task utilizes a word alignment matrix to swap the representations of aligned words in parallel sentences, then predicts the masked tokens in the swapped sentences. ## visualization the effect of contextual alignment has been well studied in @xcite , where the authors demonstrate that the contextual alignment is powerful in improving the transferability of mbert. but the effect of the word-level information alignment is still unclear. to further explore this problem, we use t-sne @xcite to visualize the distances between embeddings of word alignment pairs with the highest frequencies (excluding stop words). the result is illustrated in figure 2 . the left panel shows word pairs in the embedding layer of mbert without weam pre-training, we can see that these word pairs are partly aligned. for example, the pairs today-heute, council-rat are aligned well, but beriche-report, mr-herr are distant away. as a comparison, we show the word pairs from the embedding layer of mbert with weam pre-training in the right panel, where most of the word pairs are aligned much better. there are also words that remained poorly aligned even with weam. for example, our-uns, which may be due to that they are not the exact translation pair (us-uns are more exact pairs in this case). in general, the embeddings are aligned much better after the weam pre-training procedure. ## conclusion in this paper, we propose a new pre-training task named weam to align the contextual representations and static word embeddings from multilingual pre-trained models. weam consists of a multilingual prediction task and a cross-lingual prediction task. as a supplement to previous works mmlm or tlm, weam introduces the statistic alignment information as prior knowledge to guide the cross-lingual prediction. through the experiments on mlqa and xnli, we show that weam can improve the transfer performance significantly and align the word embeddings within the models much better. in the future, we plan to extend our method to other multilingual models like xlm-r.