On-the-fly Cross-lingual Masking for Multilingual Pre-training (2023)

Abstract: In multilingual pre-training with the objective of MLM (masked language modeling) on multiple monolingual corpora, multilingual models only learn cross-linguality implicitly from isomorphic spaces formed by overlapping different language spaces due to the lack of explicit cross-lingual forward pass. In this work, we present CLPM (Cross-lingual Prototype Masking), a dynamic and token-wise masking scheme, for multilingual pre-training, using a special token [\mathcal{C}]_{x} to replace a random token x in the input sentence. [\mathcal{C}]_{x} is a cross-lingual prototype for x and then forms an explicit cross-lingual forward pass. We instantiate CLPM for the multilingual pre-training phase of UNMT (unsupervised neural machine translation), and experiments show that CLPM can consistently improve the performance of UNMT models on \{De, Ro, Ne \} \leftrightarrow En . Beyond UNMT or bilingual tasks, we show that CLPM can consistently improve the performance of multilingual models on cross-lingual classification.

## introduction with tied weights across the languages and the help of language identifications @xcite , multilingual models only have access to monolingual corpora in different languages. stemming from bert/mlm @xcite and gpt @xcite , for cross-lingual knowledge, multilingual pre-training with the objective of mlm on multiple monolingual corpora is introduced by xlm @xcite , explored by mass @xcite and mbart @xcite , and scaled by xlm-r @xcite and mt5 @xcite . essentially, in multilingual mlm pre-training, models are encouraged to learn implicit crosslinguality from both linguistic similarities and shared tokens @xcite @xcite ) for translation and cross-lingual transfer. however, it does not learn any explicit and principled cross-lingual forward pass from inputs to outputs, only relying on the isomorphic space that emerged from multilingual mlm pretraining by overlapping language spaces agnostically. given the nature of translation and crosslingual transfer, models should understand explicit cross-lingual forward passes initiating crosslingual knowledge directly. considering this aspect, beyond the implicit and agnostic cross-linguality, we are interested in the question: can models learn explicit and principled cross-linguality in multilingual pre-training without any supervision? following this idea, for multilingual pre-training, we present a dynamic and token-wise masking scheme, clpm (cross-lingual prototype masking), to compute a special token @xcite x representing a cross-lingual prototype for a selected token x and then replace x with @xcite x instead of the standard token [m] in multilingual mlm pre-training. we present an example in ## cross-lingual prototype masking notation l x is the language id of language lang x . p n stands for positions. e r is the embedding for r. d is the model/embedding dimension. ## empirical study and experiment all the links of datasets, libraries, scripts, and tools marked with ⋄ are listed in appendix f. a preview version of the code is submitted, and we will open the source code on github. pre-training setting we use adam optimizer (kingma and ba, 2015) with β 1 = 0.9,β 2 = 0.999, ϵ = 1e -8, warm_up step @xcite and @xmath0. dropout regularization is set to @xmath1.1. readers can refer to appendix d.1 for details. model configuration our transformer model @xcite is identical to xlm @xcite , which consists of a 6-layer encoder and 6-layer decoder with 1024 word embedding size and hidden size and 4096 feed-forward filter size. we add a learnable language embedding and a learnable position embedding to each token of the input sentence for the encoder and decoder (p and l in eq.1 ). we have some default configurations for our method based on the study of model robustness (see §robustness and model variation): multilingual task we consider three multilingual tasks: 1) unmt for evaluation on translation tasks, 2) cross-lingual word similarity for evaluation on cross-lingual embedding tasks, and 3) zeroshot cross-lingual classification for evaluation on cross-lingual transfer tasks. ## conclusion in this work, we present clpm, an alternative masking scheme, to compute special tokens @xcite x for masking in multilingual mlm pre-training. [c] x is the cross-lingual prototype for the selected word x, computed from multiple candidates dynamically and token-wise. compared to the standard masking scheme [m], @xcite x automatically forms an explicit cross-lingual forward pass in attention mechanism, consistently improving cross-linguality in multilingual mlm pre-training. experiments show that clpm can consistently improve the performance of translation and cross-lingual transfer. ivan vulić, goran glavaš, roi reichart, and anna korhonen. 2020. do we really need fully unsupervised cross-lingual embeddings? in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, pages 4407-4418. yiren wang, yingce xia, fei tian, fei gao, tao qin, cengziang zhai, and tie-yan liu. 2019. neural machine translation with soft prototype. in advances in neural information processing systems. guillaume wenzek, marie anne lachaux, alexis conneau, vishrav chaudhary, francisco guzmán, armand joulin, and edouard grave. 2020. ccnet: extracting high quality monolingual datasets from web crawl data. in proceedings of the 12th language resources and evaluation conference, pages 4003-4012. shijie wu and mark dredze. 2019. beto, bentz, becas: the surprising cross-lingual effectiveness of bert. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, pages 833-844, hong kong, china. association for computational linguistics. linting xue, noah constant, adam roberts, mihir kale, rami al-rfou, aditya siddhant, aditya barua, and colin raffel. 2021. mt5: a massively multilingual pre-trained text-to-text transformer. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 483-498. biao zhang, ankur bapna, rico sennrich, and orhan firat. 2021. share or not? learning to schedule language-specific capacity for multilingual translation. in international conference on learning representations.