ERNIE - M : Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2021)

Abstract: Recent studies have demonstrated that pre-trained cross-lingual models achieve impressive performance in downstream cross-lingual tasks. This improvement benefits from learning a large amount of monolingual and parallel corpora. Although it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for low-resource languages. In this paper, we propose Ernie-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. Our key insight is to integrate back-translation into the pre-training process. We generate pseudo-parallel sentence pairs on a monolingual corpus to enable the learning of semantic alignments between different languages, thereby enhancing the semantic modeling of cross-lingual models. Experimental results show that Ernie-M outperforms existing cross-lingual models and delivers new state-of-the-art results in various cross-lingual downstream tasks. The codes and pre-trained models will be made publicly available.

## introduction recent studies have demonstrated that the pretraining of cross-lingual language models can significantly improve their performance in crosslingual natural language processing tasks @xcite @xcite . existing pretraining methods include multilingual masked language modeling (mmlm; @xcite ) and translation language modeling (tlm; @xcite , of which the key point is to learn a shared language-invariant feature space among multiple languages. mmlm implicitly models the semantic representation of each language in a unified feature space by learning them separately. tlm is an extension of mmlm that is trained with a parallel corpus and captures semantic alignment by learning a pair of parallel sentences simultaneously. this study shows that the use of parallel corpora can significantly improve the performance in downstream cross-lingual understanding and generation tasks. however, the sizes of parallel corpora are limited @xcite , restricting the performance of the cross-lingual language model. to overcome the constraint of the parallel corpus size on the model performance, we propose ernie-m, a novel cross-lingual pre-training method to learn semantic alignment among multiple languages on monolingual corpora. specifically, we propose cross-attention masked language modeling (camlm) to improve the cross-lingual transferability of the model on parallel corpora, and it trains the model to predict the tokens of one language by using another language. then, we utilize the transferability learned from parallel corpora to enhance multilingual representation. we propose back-translation masked language modeling (btmlm) to train the model, and this helps the model to learn sentence alignment from monolingual corpora. in btmlm, a part of the tokens in the input monolingual sentences is predicted into the tokens of another language. we then concatenate the predicted tokens and the input sentences as pseudo-parallel sentences to train the model. in this way, the model can learn sentence alignment with only monolingual corpora and overcome the constraint of the parallel corpus size while improving the model performance. ernie-m is implemented on the basis of xlm-r @xcite , and we evaluate its performance on five widely used cross-lingual benchmarks: xnli @xcite for crosslingual natural language inference, mlqa @xcite for cross-lingual question answering, conll @xcite for named entity recognition, cross-lingual paraphrase adversaries from word scrambling (paws-x) @xcite for cross-lingual paraphrase identification, and tatoeba @xcite for cross-lingual retrieval. the experimental results demonstrate that ernie-m outperforms existing cross-lingual models and achieves new state-of-the-art (sota) results. ## methodology in this section, we first introduce the general workflow of ernie-m and then present the details of the model training. ## experiments we consider five cross-lingual evaluation benchmarks: xnli for cross-lingual natural language inference, mlqa for cross-lingual question answering, conll for cross-lingual named entity recognition, paws-x for cross-lingual paraphrase identification, and tatoeba for cross-lingual retrieval. next, we first describe the data and pre-training details and then compare the ernie-m with the existing state-of-the-art models. ## conclusion to overcome the constraint that the parallel corpus size places on the cross-lingual models performance, we propose a new cross-lingual model, ernie-m, which is trained using both monolingual and parallel corpora. the contribution of ernie-m is to propose two training objectives. the first objective is to enhance the multilingual representation on parallel corpora by applying camlm, and the second objective is to help the model to align cross-lingual semantic representations from a monolingual corpus by using btmlm. experiments show that ernie-m achieves sota results in various downstream tasks on the xnli, mlqa, conll, paws-x, and tatoeba datasets.