Multilingual BERT Post-Pretraining Alignment (2021)

Abstract: We propose a simple method to align multilingual contextual embeddings as a post-pretraining step for improved cross-lingual transferability of the pretrained language models. Using parallel data, our method aligns embeddings on the word level through the recently proposed Translation Language Modeling objective as well as on the sentence level via contrastive learning and random input shuffling. We also perform sentence-level code-switching with English when finetuning on downstream tasks. On XNLI, our best model (initialized from mBERT) improves over mBERT by 4.7% in the zero-shot setting and achieves comparable result to XLM for translate-train while using less than 18% of the same parallel data and 31% fewer model parameters. On MLQA, our model outperforms XLM-R_Base, which has 57% more parameters than ours.

## introduction building on the success of monolingual pretrained language models (lm) such as bert @xcite and roberta @xcite , their multilingual counterparts mbert @xcite and xlm-r @xcite are trained using the same objectives-masked language modeling (mlm) and in the case of mbert, next sentence prediction (nsp). mlm is applied to monolingual text that covers over 100 languages. despite the absence of parallel data and explicit alignment signals, these models transfer surprisingly well from high resource languages, such as english, to other languages. on the natural language inference (nli) task xnli @xcite , a text classification model trained on english training data can be directly applied to the other 14 languages and achieve respectable performance. having a single model that can serve over 100 languages also has important business applications. recent work improves upon these pretrained models by adding cross-lingual tasks leveraging parallel data that always involve english. @xcite pretrain a new transformerbased @xcite model from scratch with an mlm objective on monolingual data, and a translation language modeling (tlm) objective on parallel data. @xcite align mbert embeddings in a post-hoc manner: they first apply a statistical toolkit, fastalign @xcite , to create word alignments on parallel sentences. then, mbert is tuned via minimizing the mean squared error between the embeddings of english words and those of the corresponding words in other languages. such post-hoc approach suffers from the limitations of word-alignment toolkits: (1) the noises from fastalign can lead to error propagation to the rest of the pipeline; (2) fastalign mainly creates the alignments with word-level translation and usually overlooks the contextual semantic compositions. as a result, the tuned mbert is biased to shallow cross-lingual correspondence. importantly, both approaches only involve word-level alignment tasks. in this work, we focus on self-supervised, alignment-oriented training tasks using minimum parallel data to improve mbert's cross-lingual transferability. we propose a post-pretraining alignment (ppa) method consisting of both wordlevel and sentence-level alignment, as well as a finetuning technique on downstream tasks that take pairs of text as input, such as nli and question answering (qa). specifically, we use a slightly different version of tlm as our word-level alignment task and contrastive learning @xcite ) on mbert's @xcite tokens to align sentence-level representations. both tasks are self-supervised and do not require pre-alignment tools such as fastalign. our sentence-level alignment is implemented using moco @xcite , an instance discrimination-based method of contrastive learning that was recently proposed for self-supervised representation learning in computer vision. lastly, when finetuning on nli and qa tasks for non-english languages, we perform sentence-level codeswitching with english as a form of both alignment and data augmentation. we conduct controlled experiments on xnli and mlqa @xcite , leveraging varying amounts of parallel data during alignment. we then conduct an ablation study that shows the effectiveness of our method. on xnli, our aligned mbert improves over the original mbert by 4.7% for zero-shot transfer, and outperforms @xcite while using the same amount of parallel data from the same source. for translate-train, where translation of english training data is available in the target language, our model achieves comparable performance to xlm while using far fewer resources. on mlqa, we get 2.3% improvement over mbert and outperform xlm-r base for zero-shot transfer. ## method this section introduces our proposed post-pretraining alignment (ppa) method. we first describe the moco contrastive learning framework and how we use it for sentence-level alignment. next, we describe the finer-grained word-level alignment with tlm. finally, when training data in the target language is available, we incorporate sentence-level code-switching as a form of both alignment and data augmentation to complement ppa. figure 1 shows our overall model structure. background: contrastive learning instance discrimination-based contrastive learning aims to bring two views of the same source image closer to each other in the representation space while encouraging views of different source images to be dissimilar through a contrastive loss. recent advances in this area, such as simclr @xcite and moco @xcite have bridged the gap in performance between self-supervised representation learning and fully-supervised methods on the imagenet @xcite dataset. as a key feature for both methods, a large number of negative examples per instance are necessary for the models to learn such good representations. simclr uses in-batch negative example sampling, thus requiring a large batch size, whereas moco stores negative examples in a queue and casts the contrastive learning task as dictionary (query-key) lookup. in what follows, we first describe moco and then how we use it for sentence-level alignment. concretely, moco employs a dual-encoder architecture. given two views v 1 and v 2 of the same image, v 1 is encoded by the query encoder f q and v 2 by the momentum encoder f k . v 1 and v 2 form a positive pair. negative examples are views of different source images, and are stored in a @xmath0, which is randomly initialized. k is usually a large number (e.g., @xmath1, 536 for imagenet). negative pairs are formed by comparing v 1 with each item in the queue. similarity between pairs is measured by dot product. moco uses the infonce loss @xcite to bring positive pairs closer to each other and push negative pairs apart. after a batch of view pairs are processed, those encoded by the momentum encoder are added to the queue as negative examples for future queries. during training, the query encoder is updated by the optimizer while the momentum encoder is updated by the exponential moving average of the query encoder's parameters to maintain queue consistency: ## results we report results on the test set of xnli and mlqa and we do hyperparameter searching on the development set. all the experiments for translatetrain were done using the code-switching technique introduced in section 2. xnli compared to @xcite , which use 250k parallel sentences per language from the same sources as we do for post-pretraining alignment, compared to xlm, our 250k, 600k and 2m settings represent 3.1%, 7% and 17.8% of the parallel data used by xlm, respectively (see our alignment-oriented method is, to a large degree, upper-bounded by the english performance, since all our parallel data involve english and all the other languages are implicitly aligning with english through our ppa objectives. our 2m model is able to improve the english performance to 82.4 from the mbert baseline, but it is still lower than xlm (mlm), and much lower than xlm (mlm+tlm). we hypothesize that more highquality monolingual data and model capacity are needed to further improve our english performance, thereby helping other languages better align with it. mlqa comparing our model performance using vary ablation comparing our model without word-level alignment, i.e., -tlm, to the baseline mbert in in the zero-shot setting, removing moco (-moco) performs similarly to -tlm, where we observe an accuracy drop of about 1% compared to our full system. in translate-train, -moco outperforms -tlm and even matches the full system performance for 250k. finally, we show ablation result for our codeswitching in translate-train. on average, codeswitching provides an additional gain of 1%. ## conclusion post-pretraining embedding alignment is an efficient means of improving cross-lingual transferability of pretrained multilingual lms, especially when pretraining from scratch is not feasible. we showed that our self-supervised sentence-level and word-level alignment tasks can greatly improve mbert's performance on downstream tasks of nli and qa, and the method can potentially be applied to improve other pretrained multilingual lms. in addition to zero-shot cross-lingual transfer, we also showed that code-switching with english during finetuning provides additional alignment signals, when training data is available for the target language.