XLM - E : Cross-lingual Language Model Pre-training via ELECTRA (2022)

Abstract: In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.

## introduction it has become a de facto trend to use a pretrained language model @xcite @xcite for downstream nlp tasks. these models are typically pretrained with masked language modeling objectives, which learn to generate the masked tokens of an input sentence. in addition to monolingual representations, the masked language modeling task is effective for learning cross-lingual representations. by only using multilingual corpora, such pretrained models perform well on zero-shot cross-lingual transfer @xcite , i.e., fine-tuning with english training data while directly applying the model to other target languages. the cross-lingual transferability can be further improved by introducing external pre-training tasks using parallel corpus, such as translation language modeling @xcite , and crosslingual contrast @xcite . however, previous cross-lingual pre-training based on masked language modeling usually requires massive computation resources, rendering such models quite expensive. as shown in figure 1 , our proposed we also present xlm-r @xcite , infoxlm @xcite , and xlm-align @xcite . the compared models are all in base size. xlm-e achieves a huge speedup compared with well-tuned pretrained models. in this paper, we introduce electra-style tasks @xcite to cross-lingual language model pre-training. specifically, we present two discriminative pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. rather than recovering masked tokens, the model learns to distinguish the replaced tokens in the corrupted input sequences. the two tasks build input sequences by replacing tokens in multilingual sentences, and translation pairs, respectively. we also describe the pretraining algorithm of our model, xlm-e, which is pretrained with the above two discriminative tasks. it provides a more compute-efficient and sampleefficient way for cross-lingual language model pretraining. we conduct extensive experiments on the xtreme cross-lingual understanding benchmark to evaluate and analyze xlm-e. over seven datasets, our model achieves competitive results with the baseline models, while only using 1% of the computation cost comparing to xlm-r. in addition to the high computational efficiency, our model also shows the cross-lingual transferability that achieves a reasonably low transfer gap. we also show that the discriminative pre-training encourages universal representations, making the text representations better aligned across different languages. our contributions are summarized as follows: the discriminator d is trained with the replaced token detection task. specifically, the discriminator takes the corrupted sentences x corrupt as input, which is constructed by replacing the tokens in m with the tokens sampled from the generator g: then, the discriminator predicts whether x corrupt i is original or sampled from the generator. the loss function of the discriminator d is compared to generative pre-training, electra uses more model parameters and training flops per step, because it contains a generator and a discriminator during pre-training. however, only the discriminator is used for fine-tuning on downstream tasks, so the size of the final checkpoint is similar to bert-like models in practice. ## methods figure 2 shows an overview of the two discriminative tasks used for pre-training xlm-e. similar to electra described in section 2, xlm-e has two transformer components, i.e., generator and discriminator. the generator predicts the masked tokens given the masked sentence or translation pair, and the discriminator distinguishes whether the tokens are replaced by the generator. ## related work learning self-supervised tasks on large-scale multilingual texts has proven to be effective for pretraining cross-lingual language models. masked language modeling (mlm; @xcite ) is typically used to learn cross-lingual encoders such as multilingual bert (mbert; @xcite and xlm-r @xcite . the crosslingual language models can be further improved by introducing external pre-training tasks using parallel corpora. xlm @xcite introduces the translation language modeling (tlm) task that predicts masked tokens from concatenated translation pairs. alm @xcite utilizes translation pairs to construct codeswitched sequences as input. infoxlm @xcite considers an input translation pair as crosslingual views of the same meaning, and proposes a cross-lingual contrastive learning task. several pre-training tasks utilize the token-level alignments in parallel data to improve cross-lingual language models @xcite @xcite . in addition, parallel data are also employed for cross-lingual sequence-to-sequence pre-training. xnlg @xcite presents cross-lingual masked language modeling and cross-lingual autoencoding for cross-lingual natural language generation, and achieves the cross-lingual transfer for nlg tasks. veco @xcite utilizes crossattention mlm to pretrain a variable cross-lingual language model for both nlu and nlg. mt6 @xcite improves mt5 @xcite by learning the translation span corruption task on parallel data. ∆lm @xcite proposes to align pretrained multilingual encoders to improve cross-lingual sequence-to-sequence pre-training. ## conclusion we introduce xlm-e, a cross-lingual language model pretrained by electra-style tasks. specifically, we present two pre-training tasks, i.e., multilingual replaced token detection, and translation replaced token detection. xlm-e outperforms baseline models on cross-lingual understanding tasks although using much less computation cost. in addition to improved performance and computational efficiency, we also show that xlm-e obtains the cross-lingual transferability with a reasonably low transfer gap.