Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only (2023)

Abstract: Recent studies have revealed the remarkable cross-lingual capability of multilingual pre-trained language models (mPLMs), even when pre-trained without parallel corpora (mono-mPLMs). Intuitively, semantic alignments may be the reason behind such capability but remain under-explored. In this work, we investigate the alignment properties from the token perspective in mono-mPLMs and find that the alignments correspond to the geometric similarity of embedding space across different languages. Nevertheless, mono-mPLMs tend to damage this geometric similarity at the higher layers due to the lack of cross-lingual interactions, thus limiting their cross-lingual transfer capabilities. To address this issue, we introduce token-level and semantic-level code-switched masked language modeling, employing the self-induced token alignments to explicitly improve cross-lingual interactions over layers of mono-mPLMs without relying on parallel sentences. We evaluate our method on various natural language understanding tasks and unsupervised machine translation tasks. The results demonstrate that our methods outperform the strong baselines and achieve comparable performance with mPLMs trained with parallel corpora.

## introduction recent studies show that multilingual pre-trained language models (mplms) significantly improve the performance of cross-lingual natural language processing tasks. conventional mplms @xcite @xcite typically adopt multiple monolingual corpora (mono-mplms) to perform masked language modeling during pretraining, obtaining impressive and stable multilingual capabilities, which may intuitively result from the semantic alignments but remains underexplored. another line of research involves improving the multilingual pre-training by incorporating the cross-lingual parallel corpora (para-mplms) into pre-training @xcite @xcite @xcite . however, parallel sentences are not always available, especially for low-resource languages @xcite . and collecting such data often entails substantial costs. therefore, exploring approaches to improve multilingual pre-training without using parallel corpora is important and worthy of study. to achieve this, we first conduct analyses to investigate the token alignment properties of xlm-r, a strong mono-mplm that only uses multiple monolingual corpora for pre-training. our empirical study demonstrates that the cross-lingual token alignments occur at the embedding layer (alignment location) with surprisingly high alignment accuracy (alignment degree) but they become weaker at the higher layers (figure 1 (a) ). we also find that the alignments are geometrically aligned instead of absolutely aligned (alignment format), as shown in figure 1 @xcite . the phenomenon shows that token embeddings of different languages are separately distributed but geometrically similar. we further compare the differences in geometric similarities of representations from the bottom layer to the top layer using mono-mplms and para-mplms. and we find that the representations be-come geometrically dissimilar at higher layers of mono-mplms while para-mplms would alleviate the problem by using parallel sentences, obtaining better cross-lingual transfer capability. it shows the necessity of explicit cross-lingual interactions. based on the above observation, we propose self-improving methods to encourage cross-lingual interactions using self-induced token alignments. intuitively, the masked tokens can be predicted with semantic-equivalent but slightly language-mixed contexts. therefore, we first utilize self-induced alignments to perform token-level code-switched masked language modeling (tcs-mlm), which requests the model to predict original masked tokens with the semantic-equivalent but code-switched surrounding text. considering that vanilla replacements usually lack diversity, we further propose a novel semantic-level code-switched masked language modeling (scs-mlm), which replaces the context tokens with a weighted combination of multiple semantically similar ones in other languages. scs-mlm involves on-the-fly semantic replacements during training, further enhancing the diversity of code-switched examples and cross-lingual interactions. we evaluate our methods on various crosslingual transfer tasks. specifically, we conduct experiments on natural language understanding tasks, including xnli @xcite and paws-x @xcite for sentence-pair classification, wikiann @xcite and udpos @xcite for structural prediction, mlqa @xcite for question answering, and tatoeba (artetxe and schwenk, 2019) for sentence retrieval. we also perform experiments on unsupervised machine translation to evaluate the performance of the generation task. experimental results demonstrate that our methods significantly improve the performance compared with the strong baselines, even surpassing some mplms pre-trained using parallel corpora. further analysis demonstrates that our methods improve the geometric similarity of representations for different languages, and thus promoting the cross-lingual transfer capability. our contributions are summarized as follows: ## a closer look at multilinguality in this section, we take the commonly used xlm-r, the strong mono-mplm, as an example to show our observation foot_0 . specifically, we first investigate the properties of cross-lingual token alignments in mono-mplms, showing their relation to geometric similarity. then, we explore the variation of geometric similarity of representations across different layers and demonstrate that the geometric similarity at higher layers would be disturbed due to the lack of cross-lingual interactions, hindering the cross-lingual transfer capability. ## pre-training settings pre-training data we collect monolingual corpora from common crawl corpus, which contains about 890gb data for 50 languages. different from previous studies, we do not use any bilingual corpus. following @xcite , we sample multilingual data according to a multinomial distribution with probabilities. considering the pre-training corpora in n languages with n i training instances for the i-th language, the probability for i-th language can be calculated as: ## related work multilingual pre-trained language models begin with mbert @xcite and xlm (conneau and lample, 2019), which learn the shared feature space among languages using multiple monolingual corpora. xlm-r @xcite shows the effects of models when trained on a large-scale corpus, establishing strong baselines for subsequent studies. based on the observation that parallel corpora would help cross-lingual alignment in @xcite , many studies pay attention to the usage of the parallel corpus. unicoder @xcite employs a multi-task learning framework to learn cross-lingual semantic representations. alm @xcite and paradise (reid and artetxe, 2022) uses parallel sentences to construct code-switch sentences. infoxlm @xcite and hictl @xcite respectively employ sentence-level and token-level contrastive learning for cross-lingual semantic alignments. veco @xcite proposes a variable framework to enable the model to process understanding and generation tasks. @xcite generates pseudo-training examples, further improving the overall performance on downstream tasks. @xcite explores the mask strategy (prototype-word) to improve cross-lingual pre-training. besides the above discriminative models, generative models, such as mass @xcite , mbart @xcite , mt5 @xcite , xglm @xcite , bloom @xcite , also show impressive performance on generation tasks. mass, mt5 and mbart pre-train the denoising seq2seq model, handling both nlu and nlg tasks. xglm and bloom pre-train extremely large auto-regressive models using extremely large multiple monolingual corpora, showing surprising in-context learning @xcite capability on cross-lingual tasks. different from previous studies, our work first investigates the properties of token alignments behind multilinguality and then proposes selfimproving methods for multilingual pre-training with only monolingual corpora, alleviating the need for parallel sentences. ## conclusion in this work, we first investigate the properties of cross-lingual token alignments in mono-mplms, and then make a comparison between mono-mplms and para-mplms, demonstrating that geometric similarities of higher-layer representations would be damaged without explicit cross-lingual interactions, hindering the multilinguality. therefore, we propose token-level and semantic-level codeswitch masked language modeling to improve the cross-lingual interactions without relying on parallel corpora. empirical results on language understanding and generation tasks demonstrate the effectiveness of our methods. future work would adapt our methods to much larger language models. language understanding and generation. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3980-3994, online. association for computational linguistics. joakim nivre, mitchell abrams, željko agić, lars ahrenberg, lene antonsen, maria jesus aranzabe, gashaw arutie, masayuki asahara, luma ateyah, mohammed attia, et al. 2018. universal dependencies 2.2. xuan ouyang, shuohuan wang, chao pang, yu sun, hao tian, hua wu, and haifeng wang. 2021. ernie-m: enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 27-38, online and punta cana, dominican republic. association for computational linguistics. xiaoman pan, boliang zhang, jonathan may, joel nothman, kevin knight, and heng ji. 2017. cross-lingual name tagging and linking for 282 languages. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1946-1958, vancouver, canada. association for computational linguistics. telmo pires, eva schlinger, and dan garrette. 2019. how multilingual is multilingual bert? in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4996-5001, florence, italy. association for computational linguistics. machel reid and mikel artetxe. 2022. paradise: exploiting parallel data for multilingual sequenceto-sequence pretraining. in proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 800-810, seattle, united states. association for computational linguistics. shuo ren, yu wu, shujie liu, ming zhou, and shuai ma. 2019. explicit cross-lingual pre-training for unsupervised machine translation. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 770-779, hong kong, china. association for computational linguistics. shuo ren, long zhou, shujie liu, furu wei, ming zhou, and shuai ma. 2021. semface: pre-training encoder and decoder with a semantic interface for neural machine translation. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 4518-4527, online. association for computational linguistics.