VECO : Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation (2021)

Abstract: Existing work in multilingual pretraining has demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages. However, much of this work only relies on the shared vocabulary and bilingual contexts to encourage the correlation across languages, which is loose and implicit for aligning the contextual representations between languages. In this paper, we plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. More importantly, when fine-tuning on downstream tasks, the cross-attention module can be plugged in or out on-demand, thus naturally benefiting a wider range of cross-lingual tasks, from language understanding to generation. As a result, the proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark, covering text classification, sequence labeling, question answering, and sentence retrieval. For cross-lingual generation tasks, it also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and English-to-French translation datasets, with gains of up to 1 2 BLEU.

## introduction cross-lingual pre-trained models like mbert @xcite , xlm @xcite and xlm-r @xcite that target providing contextualized representations for the inputs across languages, have shown large potential on a variety of cross-lingual understanding and generation tasks. behind the great success, two major factors play the role of aligning the contextual representations between languages: 1) build the shared vocabulary across languages through subword tokenization, which supports the simple extension of masked language modeling (mlm) from english corpus to multilingual corpus; 2) capture the alignment in parallel data via concatenating two sentences as input, called translation language modeling (tlm). however, both of these two mechanisms rely on the self-attention module (@xmath0, which may lead to few attention patterns across languages. taking figure 1 as an example, even though inputting a pair of parallel sentences, both models only attend to the english context to build the representation of english tokens, while ignoring the se-b) translation language modeling (tlm) a) masked language modeling (mlm) ## fine-tuning veco for downstream cross-lingual understanding and generation tasks as figure 3 illustrated, when fine-tuning on various downstream tasks, one advantage of veco is its flexibility for initializing both the encoder-only transformer for understanding tasks and encoderdecoder transformer for generation tasks. beyond it, we also explore a fine-tuning approach combined with the characteristics of veco . ## pre-training setup model configuration we pre-train a 24-layer model with 1024 embedding/hidden size and 4096 feed-forward size. we do not use language embeddings to allow our model to better deal with downstream tasks of unseen languages. we adopt the same 250k vocabulary that is also used by xlm-r @xcite . pre-training data we collect monolingual and bilingual corpus covering 50 languages. for monolingual training datasets, we reconstruct common-crawl corpus used in xlm-r @xcite . we extract 1.36tb data in 50 languages, which contains 6.5g sentences and 0.4g documents. we up/down-sample the monolingual text like xlm from each language with a smoothing parameter α = 0.5. for bilingual data, we collect from the opus website foot_3 like previous works @xcite . there are 6.4g parallel sentences, covering 879 language pairs across 50 languages. see more statistics of training data in appendix a. ## analysis and ablation study we perform an ablation study to investigate where the improvement in cross-lingual nlu and nlg tasks mainly comes from. specifically, there are three main aspects we have studied: to figure out these questions, we train xlm, mbart and veco model from scratch using the same datasets and parameter settings (see appendix a for more details). all of them is pre-trained via mlm and tlm tasks. note that the mlm task generally refers to predict the masked words of source language, while the tlm task generally refers to predict the words of the target language. specifically for mbart that is under the framework of encoder-decoder, the input of encoder is masked sequence x, and the target of decoder is the masked words of source input x (for mlm task), or the parallel sentence y (for tlm task). moreover, we find that pre-training a sequenceto-sequence model like mbart @xcite performs worst on nlu tasks like xnli foot_5 , almost 6 points worse than veco and near 2 points worse than xlm. one possible explanation could be that the unidirectional language modeling in the decoder might be sub-optimal for nlu tasks. and even on the machine translation task, mbart still performs worse than veco when pre-training on the same bilingual datasets. we conclude that it is because that veco can do better in the contextual modeling of source input x via a explicit masked language modeling objective in eq 10 applied to a feasible solution for cross-language generation is to pre-train a denoising auto-encoder like mbart @xcite . it extends bart @xcite to the multilingual setting, demonstrating significant gains in low/medium-resource machine translation, but with a decrease in high resource languages. unlike mbart, @xcite first trains an encoder via mlm and then frozen the encoder to train the decoder only via two generative tasks. a similar approach is also proposed in @xcite and @xcite , with the main difference in the joint training of encoder-decoder with code-switch tricks. however, all these cross-lingual models emphasize training a dedicated model for nlg. thus they may hurt the nlu capabilities of the model. the ablation study in section 7 also validates that it is sub-optimal to train an encoder-encoder network for nlu tasks. this paper endeavors to build a unified crosslingual model for nlu and nlg tasks via a plugand-play cross-attention module. more importantly, the cross-attention module plays a role in the explicit alignment of encoded representations of different languages, thus largely contributing to building a unified cross-lingual model. ## conclusion we present veco, a variable and flexible crosslingual pre-training model, targets at explicitly capturing the interdependence between languages via a plug-and-play cross-attention module. based on the flexible characteristics, veco can initialize both nlu preferred encoder-only and nlg specialized encoder-decoder transformer. moreover, we also introduce a plug-in fine-tuning approach to encourage the fusion between languages, combining the feature of veco and cross-language downstream tasks. taken together, veco achieves consistent improvements on various language understanding and generation tasks, broadening the way of thinking about pre-trained backbone architecture and finetuning methods under the cross-lingual scenario.