I nfo XLM : An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2021)

Abstract: In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm .

## introduction learning cross-lingual language representations plays an important role in overcoming the language barrier of nlp models. the recent success of crosslingual language model pre-training @xcite @xcite @xcite significantly improves the cross-lingual transferability in various downstream tasks, such as cross-lingual classification, and question answering. state-of-the-art cross-lingual pre-trained models are typically built upon multilingual masked language modeling (mmlm; @xcite , and translation language modeling (tlm; @xcite . the goal of both pretext tasks is to predict masked tokens given input context. the difference is that mmlm uses monolingual text as input, while tlm feeds bilingual parallel sentences into the model. even without explicit encouragement of learning universal representations across languages, the derived models have shown promising abilities of cross-lingual transfer. in this work, we formulate cross-lingual pretraining from a unified information-theoretic perspective. following the mutual information maximization principle @xcite , we show that the existing pretext tasks can be viewed as maximizing the lower bounds of mutual information between various multilingualmulti-granularity views. specifically, mmlm maximizes mutual information between the masked tokens and the context in the same language while the anchor points across languages encourages the correlation between cross-lingual contexts. moreover, we present that tlm can maximize mutual information between the masked tokens and the parallel context, which implicitly aligns encoded representations of different languages. the unified informationtheoretic framework also inspires us to propose a new cross-lingual pre-training task, named as cross-lingual contrast (xlco). the model learns to distinguish the translation of an input sentence from a set of negative examples. in comparison to tlm that maximizes token-sequence mutual information, xlco maximizes sequence-level mutual information between translation pairs which are regarded as cross-lingual views of the same meaning. we employ the momentum contrast @xcite to realize xlco. we also propose the mixup contrast and conduct the contrast on the universal layer to further facilitate the cross-lingual transferability. under the presented framework, we develop a cross-lingual pre-trained model (infoxlm) to leverage both monolingual and parallel corpora. we jointly train infoxlm with mmlm, tlm and xlco. we conduct extensive experiments on several cross-lingual understanding tasks, including cross-lingual natural language inference @xcite , cross-lingual question answering @xcite , and cross-lingual sentence retrieval @xcite . experimental results show that infoxlm outperforms strong baselines on all the benchmarks. moreover, the analysis indicates that infoxlm achieves better cross-lingual transferability. ## information-theoretic framework for cross-lingual pre-training in representation learning, the learned representations are expected to preserve the information of the original input data. however, it is intractable to directly model the mutual information between the input data and the representations. alternatively, we can maximize the mutual information between the representations from different views of the input data, e.g., different parts of a sentence, a translation pair of the same meaning. in this section, we start from a unified information-theoretic perspective, and formulate cross-lingual pre-training with the mutual information maximization principle. then, under the information-theoretic framework, we propose a new cross-lingual pre-training task, named as crosslingual contrast (xlco). finally, we present the pre-training procedure of our infoxlm. ## experiments in this section, we first present the training configuration of infoxlm. then we compare the finetuning results of infoxlm with previous work on three cross-lingual understanding tasks. we also conduct ablation studies to understand the major components of infoxlm. ## conclusion in this paper, we present a cross-lingual pre-trained model infoxlm that is trained with both monolingual and parallel corpora. the model is motivated by the unified view of cross-lingual pretraining from an information-theoretic perspective. specifically, in addition to the masked language modeling and translation language modeling tasks, infoxlm is jointly pre-trained with a newly introduced cross-lingual contrastive learning task. the cross-lingual contrast leverages bilingual pairs as the two views of the same meaning, and encourages their encoded representations to be more similar than the negative examples. experimental results on several cross-lingual language understanding tasks show that infoxlm can considerably improve the performance.