Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment (2021)

Abstract: The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. Specifically, the model first self-label word alignments for parallel sentences. Then we randomly mask tokens in a bitext pair. Given a masked token, the model uses a pointer network to predict the aligned token in the other language. We alternately perform the above two steps in an expectation-maximization manner. Experimental results show that our method improves cross-lingual transferability on various datasets, especially on the token-level tasks, such as question answering, and structured prediction. Moreover, the model can serve as a pretrained word aligner, which achieves reasonably low error rate on the alignment benchmarks. The code and pretrained parameters are available at github.com/CZWin32768/XLM-Align.

## introduction despite the current advances in nlp, most applications and resources are still english-centric, making non-english users hard to access. therefore, it is essential to build cross-lingual transferable models that can learn from the training data in highresource languages and generalize on low-resource languages. recently, pretrained cross-lingual language models have shown their effectiveness for cross-lingual transfer. by pre-training on monolingual text and parallel sentences, the models provide significant improvements on a wide range of crosslingual end tasks @xcite @xcite . cross-lingual language model pre-training is typically achieved by learning various pretext tasks on monolingual and parallel corpora. by simply learning masked language modeling (mlm; @xcite on monolingual text of multiple languages, the models surprisingly achieve competitive results on cross-lingual tasks @xcite . besides, several pretext tasks are proposed to utilize parallel corpora to learn better sentence-level cross-lingual representations @xcite @xcite . for example, the translation language modeling (tlm; conneau and lample 2019) task performs mlm on the concatenated parallel sentences, which implicitly enhances cross-lingual transferability. however, most pretext tasks either learn alignment at the sentence level or implicitly encourage cross-lingual alignment, leaving explicit fine-grained alignment task not fully explored. in this paper, we introduce a new cross-lingual pre-training task, named as denoising word alignment. rather than relying on external word aligners trained on parallel corpora @xcite @xcite , we utilize self-labeled alignments in our task. during pretraining, we alternately self-label word alignments and conduct the denoising word alignment task in an expectation-maximization manner. specifically, the model first self-labels word alignments for a translation pair. then we randomly mask tokens in the bitext sentence, which is used as the perturbed input for denosing word alignment. for each masked token, the model learns a pointer network to predict the self-labeled alignments in the other language. we repeat the above two steps to iteratively boost the bitext alignment knowledge for cross-lingual pre-training. we conduct extensive experiments on a wide range of cross-lingual understanding tasks. experimental results show that our model outperforms the baseline models on various datasets, particularly on the token-level tasks such as question answer-ing and structured prediction. moreover, our model can also serve as a multilingual word aligner, which achieves reasonable low error rates on the bitext alignment benchmarks. our contributions are summarized as follows: cross-lingual lm pre-training pretrained with masked language modeling (mlm; devlin et al. 2019) on monolingual text, multilingual bert (mbert; devlin et al. 2019) and xlm-r (conneau et al., 2020) produce promising results on cross-lingual transfer benchmarks @xcite . mt5 @xcite learns a multilingual version of t5 @xcite ) with text-totext tasks. in addition to monolingual text, several methods utilize parallel corpora to improve crosslingual transferability. xlm @xcite presents the translation language modeling (tlm) task that performs mlm on concatenated translation pairs. alm @xcite introduces code-switched sequences into cross-lingual lm pre-training. unicoder @xcite employs three cross-lingual tasks to learn mappings among languages. from an information-theoretic perspective, infoxlm @xcite proposes the cross-lingual contrastive learning task to align sentence-level representations. additionally, am-ber @xcite introduces an alignment objective that minimizes the distance between the forward and backward attention matrices. more recently, ernie-m @xcite presents the back-translation masked language modeling task that generates pseudo parallel sentence pairs for learning tlm, which provides better utilization of monolingual corpus. veco @xcite ) pretrains a unified cross-lingual language model for both nlu and nlg. mt6 @xcite improves the multilingual text-to-text transformer with translation pairs. notably, word-aligned bert models @xcite finetune mbert by an explicit alignment objective that minimizes the distance between aligned tokens. @xcite exploit contrastive learning to improve the explicit alignment objectives. however, @xcite show that these explicit alignment objectives do not improve cross-lingual representations under a more extensive evaluation. moreover, these models are restricted to stay close to their original pretrained values, which is not applicable for large-scale pre-training. on the contrary, we demonstrate that employing our explicit alignment objective in large-scale pre-training can provide consistent improvements over baseline models. word alignment the ibm models @xcite are statistical models for modeling the translation process that can extract word alignments between sentence pairs. a large number of word alignment models are based on the ibm models @xcite @xcite . recent studies have shown that word alignments can be extracted from neural machine translation models @xcite @xcite or from pretrained cross-lingual lms @xcite . ## analysis in this section, we conduct comprehensive ablation studies for a better understanding of our xlm-align. to reduce the computational cost, we reduce the batch size to 256, and pretrain models with 50k steps in the following experiments. ## discussion in this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. by alternately self-labeling and predicting word alignments, our xlm-align model learns transferable cross-lingual representations. experimental results show that our method improves the cross-lingual transferability on a wide range of tasks, particularly on the token-level tasks such as question answering and structured prediction. despite the effectiveness for learning crosslingual transferable representations, our method also has the limitation that requires a cold-start pre-training to prevent the model from producing low-quality alignment labels. in our experiments, we also try to pretrain xlm-align from scratch, i.e., without cold-start pre-training. however, the dwa task does not work very well due to the lowquality of self-labeled alignments. thus, we recommend continue-training xlm-align on the basis of other pretrained cross-lingual language models. for future work, we would like to research on removing this restriction so that the model can learn word alignments from scratch. ## ethical considerations despite the current advances in nlp, most nlp research works and applications are english-centric, making none-english users hard to access to nlprelated services. our method aims to pretrain cross-lingual language models that transfer supervision signals from high-resource languages to lowresource languages, which makes the nlp services and applications more accessible for low-resourcelanguage speakers. furthermore, our method can build multilingual models that serve on different languages at the same time, reducing the computational resources for building multilingual models separately for each language.