VECO: V ARIABLE E NCODER - DECODER P RETRAINING FOR C ROSS - LINGUAL U NDERSTANDING AND G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent studies about learning multilingual representations have achieved significant performance gains across a wide range of downstream cross-lingual tasks.
They train either an encoder-only Transformer mainly for understanding tasks, or an encoder-decoder Transformer speciﬁcally for generation tasks, ignoring the correlation between the two tasks and frameworks.
In contrast, this paper presents a variable encoder-decoder (VECO) pre-training approach to unify the two mainstreams in both model architectures and pre-training tasks.
VECO splits the standard Transformer block into several sub-modules trained with both innersequence and cross-sequence masked language modeling, and correspondingly reorganizes certain sub-modules for understanding and generation tasks during inference.
Such a workﬂow not only ensures to train the most streamlined parameters necessary for two kinds of tasks, but also enables them to boost each other via sharing common sub-modules.
As a result, VECO delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark covering text classiﬁcation, sequence labeling, question answering, and sentence retrieval.
For generation tasks, VECO also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and English-to-French translation datasets, with gains of up to 1 ∼ 2 BLEU.
I NTRODUCTION
Driven by the striking success of pre-trained language models (Devlin et al., 2019), recent crosslingual pre-training (Lample & Conneau, 2019; Liu et al., 2020b) has attracted an increasing of attention.
It provides cross-lingual contextualized representations for the inputs of different languages, which signiﬁcantly advances performance in both natural language understanding (NLU) and generation (NLG) tasks.
There are two mainstream architectures in current cross-lingual pre-training literature: encoderonly and encoder-decoder .
The former like XLM (Lample & Conneau, 2019) focuses on conducting masked language modeling (MLM) with a single Transformer (Vaswani et al., 2017) encoder.
This paradigm is naturally compatible with various NLU tasks, but tends to suffer from limited gains on cross-lingual generation tasks (e.g., machine translation) due to the lack of effective decoder initialization.
In contrast, the latter like mBART (Liu et al., 2020b) attempts to pre-train the encoder-decoder Transformer via denoising auto-encoding tasks to provide complete initialization for downstream generation tasks.
However, when applied in NLU scenarios, it usually requires more computation and memory to match the performance of the encoder-only models.
In light of the above pros and cons, this work presents Variable Encoder-deCOder (VECO) pretraining, which targets at providing pre-trained model initialization for both the encoder-only and encoder-decoder Transformer with the most streamlined parameters.
We observe that Transformer encoder and decoder blocks have two common modules: SelfAttention and FFN (feed-forward network), with the main difference that the latter introduces an extra CrossAttention (attention across from the encoder to the decoder) module.
Inspired by the lottery ticket hypothesis (Frankle & Carbin, 2018), we split the standard Transformer block into three independent modules FFN Cross-attention Self-attention × 𝑁 !
𝑥 ! 𝑦 ̅𝑥 %𝑦 FFN Self-attention
FFN
Cross-attention
Self-attention
Self-attention
FFN × 𝑁 × 𝑁 IS-MLM
CS-MLM (
[Figure]
Pre-training
FFN
Cross-attention
Self-attention × 𝑁 !
𝑥 !
𝑦 ̅𝑥 %𝑦
FFN
Self-attention FFN Cross-attention Self-attention
Self-attention
FFN × 𝑁 × 𝑁 IS-MLM
CS
[Figure]
MLM
[Figure]
FFN
Cross-attention
Self-attention ×
[Figure]
𝑥 !
𝑦 ̅𝑥 %𝑦
FFN
Self-attention
FFN
Cross-attention
Self-attention
Self-attention
FFN × 𝑁 × 𝑁
[Figure]
{ SelfAttention , CrossAttention , FFN } to be collaboratively trained via two speciﬁc MLM tasks.
After that, we rebuild the desired complete architecture applicable for NLU or NLG with different speciﬁc combinations of these modules during ﬁne-tuning.
1 Speciﬁcally, to be equipped with the ability of language understanding during pre-training, SelfAttention and FFN are assembled into a standard Transformer encoder for conducting inner-sequence masked language modeling (IS-MLM).
In terms of generation, SelfAttention , CrossAttention , and FFN act together as the decoder in the standard sequence-to-sequence model, and are trained by the elaborately designed cross-sequence masked language modeling (CS-MLM) task.
When applied to downstream ﬁnetuning, both SelfAttention and FFN modules constitute the Transformer encoder for contextual modeling in NLU or NLG, or cooperate with additional CrossAttention to provide the effective initialization of Transformer decoder.
With such kind of workﬂow, VECO can be applied to both
NLU and NLG tasks with the most streamlined parameters, which signiﬁcantly reduces computational overhead and memory costs.
Moreover, IS-MLM is speciﬁcally designed for understanding of individual sequences, while both understanding and generation tasks can beneﬁt from CS-MLM.
With such parameter sharing, VECO enables SelfAttention and FFN modules to be jointly trained by the two MLMs, which boosts both NLU and NLG performance.
We validate VECO on a variety of representative cross-lingual NLU and NLG benchmarks.
For cross-lingual understanding tasks, we conduct experiments on the XTREME benchmark consisting of 9 cross-lingual tasks, including text classiﬁcation, sequence labeling, question answering, and sentence retrieval.
VECO ranks ﬁrst at the XTREME leaderboard 2 at the submission deadline and obtains new state-of-the-art results on most of the tasks.
For cross-lingual generation tasks, we validate VECO on the widely used WMT14 English-German and English-French machine translation benchmarks.
VECO obtains 44.4 and 31.5 BLEU scores, consistently outperforming existing crosslingual pre-training approaches and state-of-the-art Transformer variants by around 1 ∼ 2 BLEU.
V ARIABLE E NCODER - DECODER P RE - TRAINING 2. B ACKBONE N ETWORK
The backbone network of VECO is composed of a stack of N identical layers.
Each layer has three modules, consisting of a required self-attention module, an optional cross-attention module, and a required feed-forward linear module.
Both self-attention and cross-attention modules are based on the
[Equation]
where W O , W Q , W K and W V are parameter matrices.
Attention ( a, b, c ) represents the attention operation with a as query, b as key, and c as value.
We refer the readers to Vaswani et al.
(2017) for more details.
The main difference of the self-attention module and cross-attention module is that Q = K
= V holds in the self-attention module while only K = V exists in the cross-attention module.
We formalize these two modules as:
[Equation]
where θ s and θ c are the corresponding parameters, and AddNorm denotes a residual connection (He et al., 2016) with a post layer normalization (Ba et al., 2016).
After that, a fully connected feed-forward network is applied to each element of input independently:
[Equation]
where θ f = { W 1 ; W 2 } are matrices of parameters.
2. P RE - TRAINING O BJECTIVES
In cross-lingual pre-training scenarios, we can utilize both monolingual and bilingual data widely used in previous works (Lample & Conneau, 2019; Chi et al., 2020b; Yang et al., 2020).
We formalize both the two adjacent segments in the monolingual corpus and a pair of parallel sentence in the bilingual corpus as ( x , y ) .
We ﬁrstly adopt the same mask strategy like BERT (Devlin et al., 2019) to construct the masked input (ˆ x , ˆ y ) .
Then, the backbone Transformer takes the input to perform the inner-sequence and cross-sequence masked language modeling, which enables the model to be optimized jointly for cross-lingual language understanding and cross-sequence generation.
IS-MLM: Inner-Sequence Masked Language Modeling To be equipped with the ability of language understanding, we perform masked language modeling of a single-sequence on the selfattention and FFN modules, while skipping the cross-attention modules.
As shown in Figure 1, the purple lines show the forward process of the IS-MLM task in each layer.
To be speciﬁc, the embeddings of the masked input sequence ˆ x are fed into the self-attention and FFN modules in each layer to get a contextual presentation
X ( i ) :
[Equation]
The contextual presentations X ( N ) of the last layer is used to recover the masked tokens ¯ x .
Thus the training loss of inner-sequence masked language modeling can be formalized as:
[Equation]
CS-MLM: Cross-Sequence Masked Language
Modeling
In order to fully train the crossattention module that plays a primary role in semantic mapping between sentences in cross-lingual generation tasks (e.g., machine translation), we simulate a decoder by reusing the self-attention and
FFN modules to cooperate with the cross-attention module.
As shown in Figure 1, the green lines depict the forward process of the CS-MLM task in each layer.
Speciﬁcally, we ﬁrst extract the contextual representation of ˆ y via the SelfAttention module 3 ; then a cross-attention module is employed to model an interactive representation of (ˆ x , ˆ y ) :
[Equation]
Finally, Y ( N ) , considering both the context of the semantic-related sequence ˆ x and its left segments ˆ y <t , is used to predict the masked tokens ¯ y :
[Equation]
Note that when optimizing the CS-MLM objective, we detach X ( N ) in Eq.
(6) from the computation graph (i.e., stop the gradients back-propagation from “virtual” decoder to encoder) to let the two objectives optimized in isolation.
It also speeds up and stabilizes the training of this “virtual” decoder model, since very deep encoder-decoders are typically hard to train.
To conclude, the total MLM loss for a training instance ( x , y ) , by exchanging the ˆ x and ˆ y in Eq.
(5) and Eq.
(7), can be further formalized as:
[Equation]
Several monolingual pre-training models such as MASS (Song et al., 2019), BART (Lewis et al., 2019) and PALM (Bi et al., 2020) also present similar unidirectional language modeling tasks. Except that VECO focuses on the multilingual scenario, there are some major differences in terms of both model architecture and task design: 1) In terms of model architecture, the core difference is that we share the self-attention and FFN modules in the encoder and decoder. We ﬁnd that such parameter sharing can enhance the semantic mapping between different languages not only at the embedding-level (e.g, shared BPE vocabulary), but also at the module-level.
Moreover, it also acts as a form of regularization that stabilizes the training and helps with generalization (Xia et al., 2019;
Lan et al., 2019).
In terms of task design, we differ in several ways.
The proposed IS-MLM task forces the model to bidirectionally comprehend the source input (good for NLU), which is a shortage of mBART.
Meanwhile, CS-MLM predicts the masked words other than generating the next word (adopted by MASS and mBART), thus keeping in line with IS-MLM towards a more consistent optimization direction (predicting masked words) on the shared parameters.
2. F INE - TUNING ON D OWNSTREAM NLU AND NLG T ASKS
VECO for Cross-lingual Natural Language Understanding Since the mainstream framework for NLU is an encoder-only Transformer, we only keep the self-attention and FFN modules while throwing out the cross-attention module in each layer (Figure 1(b)).
Note that the cross-attention modules only occupy less than 20% of the total parameters, which is smaller than the discarded generator of ELECTRA (Clark et al., 2020b).
VECO for Cross-lingual Natural Language Generation Considering the most popular backbone network for cross-lingual generation tasks like machine translation is the standard Transformer encoder-decoder model, we reorganize the VECO modules to act like that.
As shown in Figure 1(c), the self-attention and FFN modules constitute the Transformer encoder for contextual modeling, and the three modules work together to act as a decoder for both inner-sequence and cross-sequence contextual modeling.
Due to the training difﬁculty and inference speed of the deep network, we can choose a subset (e.g., 6 layers) of all layers to assemble the decoder.
More in-depth analysis can be found in Table 4 and Section 5.2.
P RE - TRAINING S ETUP
Model Conﬁguration We pre-train a 24-layer model with 1024 embedding/hidden size andfeed-forward size ( ∼ 662M parameters).
We do not use language embeddings to allow our model to better deal with downstream tasks of unseen languages.
We adopt the same 250K vocabulary that is also used by XLM-R (Conneau et al., 2019) and mBART (Liu et al., 2020b).
Pre-Training Datasets For monolingual training datasets
, we reconstruct Common-Crawl Corpus used in XLM-R (Conneau et al., 2019).
We extract 1.36TB data in 50 languages, which contains 6.5G sentences and 0.4G documents.
We up/down-sample the monolingual text like XLM from each language with a smoothing parameter α = 0 .
For bilingual data, we collect from the OPUS website 4 like previous works (Lample & Conneau, 2019; Chi et al., 2020b).
There are 6.4G parallel sentences, covering 879 language pairs across 50 languages.
More details about the languages and statistics of our training corpus can be founded in Appendix A.
Optimization Settings For each training iteration, we alternately sample a batch of adjacent segments from monolingual corpus and a batch of parallel sentences from bilingual datasets to conduct a pair of masked input (ˆ x , ˆ y ) .
We ﬁrstly perform IS-MLM for both ˆ x and ˆ y .
Then, we reuse their hidden states from the last layer to perform CS-MLM.
Meanwhile, we also adopt the translation masked language modeling (TLM) proposed in XLM (Lample & Conneau, 2019) when the inputs are parallel bilingual sentences.
Thus the overall training objective is the sum of three MLM objectives.
During training, the model parameters except for cross-attention are initialized by XLM-R.
We ﬁrst freeze the parameters of XLM-R and only update the cross-attention parameters for faster convergence.
Then, we jointly train the whole model.
We pre-train our model with mixed-precision training using 64 Nvidia Telsa V100 32GB GPUs.
More hyperparameters can be founded in Table 7.
E XPERIMENTS ON L ANGUAGE U NDERSTANDING 4. E XPERIMENTAL S ETUP
Downstream Tasks We conduct NLU evaluations on XTREME (Hu et al., 2020), a representative massively multilingual multi-task benchmark.
It consists of various NLU tasks over 40 languages.
XTREME tasks can be classiﬁed into four different categories: (1) sentence-pair classiﬁcation: XNLI (Conneau et al., 2018), PAWS-X (Yang et al., 2019); (2) structured prediction: UDPOS (Nivre et al., 2018), Wikiann NER (Pan et al., 2017); (3) question answering: XQuAD (Artetxe et al., 2020), MLQA (Lewis et al., 2020), TyDiQA
(Clark et al., 2020a); (4) sentence retrieval: BUCC 2018 (Zweigenbaum et al., 2018), Tatoeba (Artetxe & Schwenk, 2019).
Tasks in the ﬁrst three categories only provide golden training corpus in English and dev/test set in different target languages.
For the two zero-shot sentence retrieval tasks, no training datasets are provided.
We refer the reader to Hu et al.
for additional details about the datasets.
Fine-tuning Setting
Following previous works (Conneau et al., 2019; Hu et al., 2020), we consider two typical ﬁne-tuning settings: (1) Cross-lingual Transfer which ﬁne-tunes the pre-trained model only using English golden data and directly performs inference on the test data of different target languages; (2) Translate-Train-All which ﬁrst automatically translates the English golden data to the remaining target languages and ﬁne-tunes a multilingual model on the concatenation of all data.
We use the machine-translated data released by XTREME, except for two sequence-labeling tasks (POS, NER) since the golden token labels in the target language are unavailable.
To have a fair comparison with the strong baseline XLM-R (Conneau et al., 2019) under the translate-train-all setting, we also show the results of XLM-R using the same ﬁne-tuning hyperparameters as VECO.
4. E XPERIMENTAL R ESULTS
The detailed test results of 9 tasks on the XTREME benchmark are shown in Table 1.
It demonstrates that the proposed VECO outperforms previous cross-lingual models on most of the datasets.
Compared to XLM-R, it averagely scores 5.0 and 2.8 points higher under the cross-lingual transfer and translation-train-all settings, respectively.
It is worth noting that, VECO delivers a large improvement on zero-shot sentence retrieval tasks (BUCC, Tatoeba).
This phenomenon reﬂects that our model has a strong cross-lingual modeling ability, thus it can better mine parallel sentences in a multilingual corpus.
The reasons are two-folds.
First is the introduction of more bilingual data during pre-training, which is a direct and effective way to enhance the cross-lingual ability of the model.
Second is the mutual improvement between two pre-training tasks and the superiority of model design.
To analyze whether bilingual data plays a leading role in improving performance, we conduct a set of more fair experiments.
We train small-sized XLM and VECO models from scratch using a subset of full training data and hyperparameters (see Appendix A for details).
Table 2 shows the results of XNLI, the most widely used cross-lingual evaluation dataset in the XTREME leaderboard.
[Table] [Table] corpus only, VECO can outperform XLM by 0.8 points.
It suggests that our models can beneﬁt from adjacent sentences used by the CS-MLM task to be equipped with a stronger ability of contextual modeling.
Moreover, when trained on both the monolingual and bilingual corpus, VECO can achieve a larger improvement compared to XLM.
It reveals that VECO can better utilize the bilingual corpus, compared to only-optimized translation language modeling (TLM) in XLM.
E XPERIMENTS ON L ANGUAGE G ENERATION 5. E XPERIMENTAL S ETUP
Datasets We choose machine translation (MT) task, a typical cross-lingual generation scenario.
In order to illustrate the generality of our approach and have a fair comparison with the most recent state-of-the-art Transformer works (Liu et al., 2020a), we choose two most widely used datasets:
WMT14 English → German (En-De) and English → French (En-Fr) translation.
WMT14 En-De is a medium-resource dataset which provides 4.5M pairs for training and validation.
We adopt standard newstest2014 as the test set.
WMT14 En-Fr is a high-resource dataset which contains 36M pairs of parallel sentences.
We use newstest2012+newstest2013 for validation and newstest2016 for test.
We measure case-insensitive tokenized BLEU with multi-bleu.perl to have a fair comparison with previous Transformer variants and de-tokenized SacreBLEU 5 to avoid the inﬂuence of different tokenization between models (Post, 2018).
Fine-tuning Setting
In theory, VECO can initialize a 24-layer encoder and 24-layer decoder
Transformer model.
However, since VECO follows the post-layernorm design of vanilla Transformer (Vaswani et al., 2017) like XLM-R, we ﬁnd it is hard to train such a deep post-layernorm
Transformer variant without careful parameter searching.
This phenomenon is consistent with the ﬁndings in recent researches about deep transformer (Liu et al., 2020a; Bachlechner et al., 2020).
Since these works also show that deeper encoders are more worthwhile than deeper decoders, thus [Table] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
methods.
the main results of VECO and XLM-R are based on the most normal 6-layer decoder with the full 24-layer encoder.
The batch size is 64k and 256k for En-De and En-Fr respectively.
The total training updates are set to 100k.
The learning rate is 1e-4/2e-4, with linear warmup over the ﬁrst 16k steps and linear decay.
We run En-De and En-Fr MT experiments on 16 and 32 V100 GPUs respectively.
We average the last 10 checkpoints and use beam search with a beam size of 5.
Baselines We consider two types of Transformer baselines: randomly initialized and cross-lingual models initialized.
For random initialization, we take the original Transformer-big and the stateof-the-art Deep Transformer (Liu et al., 2020a) into consideration.
Besides, we also reproduce a
Transformer baseline which adopts the same ﬁne-tuning hyperparameters as VECO but with random initialization.
For cross-lingual pre-training methods, we include mBART, which shows impressive results on MT.
We also conduct the WMT experiments for XLM-R, following the totally same ﬁnetuning settings as VECO.
5. E XPERIMENTAL R ESULTS
Table 3 shows the comparison between VECO and baselines.
Compared to existing cross-lingual models, VECO can consistently outperform the best models about 1 BLEU.
We can also observe that VECO can largely outperform randomly initialized same-sized Transformer-xBig model by
BLEU.
And it even beats the state-of-the-art Deep-Transformer with a 60-layer encoder and 12-layer decoder on both datasets.
Table 4 (left) contrasts two ways initializing a Transformer with n ( < 24) decoder layers via selecting: (1) the ﬁrst n layers; (2) the last n layers from a 24-layer pre-trained VECO model.
We consider n = { 3 , 6 } to conduct experiments.
The predominant method adopted in this paper, the straightforward strategy of selecting the last n layers, exhibits better performance.
It is possibly because the last several layers play a more important role in making predictions over the whole vocabulary.
Moreover, we can ﬁnd that there is 0.2 ∼ 0.3 BLEU improvement when increasing the decoder layers from 3 to 6.
6 Regardless of the initialization method, the VECO-initialized model can gain consistent 1 ∼ 2 BLEU improvement over the randomly-initialized model.
Moreover, Table 4 (right) displays the sacreBLEU scores of same-sized (24-layer encoder and 6layer decoder) models during training.
We ﬁnd that VECO-initialized model can get a surprising more than 28 sacreBLEU score just after 10 epochs, which is better than the ﬁnal score of the randomly initialized model at 35 epochs.
It reveals that VECO can provide a fairly good initialization for the machine translation model, which can converge quickly and further boost the results.
To investigate whether the exciting improvement in MT mainly comes from a) the use of parallel corpus during pre-training or b) the superiority of the designed model and pre-training tasks, we also conduct a more comparable experiment.
We ﬁrst train an out-of-domain Transformer-xBig model using the whole En-De parallel data ( ∼ 68M) used in VECO pre-training, and then continue to train the model on the in-domain WMT14 En-De training dataset.
Results are shown in Table 4 (left) marked with *.
Under this set of a totally fair comparison, VECO still maintains a lead of 1.1 BLEU score.
This directly conﬁrms that the improvement on MT is not only due to the use of bilingual data.
More importantly, reasonable pre-training tasks and model design ensure better use of bilingual and large-scale unlabeled multilingual corpus.
R ELATED W ORK
Encoder-only Cross-lingual Pre-training mBERT (Devlin et al., 2019) is the ﬁrst work to pretrain a Transformer encoder over multiple languages.
There have been several extensions that use the same encoder-only backbone as mBERT, with the main difference is the introduction of more training corpus and pre-training tasks.
XLM (Lample & Conneau, 2019) utilizes both monolingual and bilingual corpus to perform mask language modeling.
XLM-R (Conneau et al., 2019) extends to be built on RoBERTa (Liu et al., 2019) using larger monolingual training data.
Unicoder (Huang et al., 2019), ALM (Yang et al., 2020), and InfoXLM (Chi et al., 2020b) propose new pre-training tasks to better utilize the bilingual data.
These works deliver impressive performance on crosslingual understanding tasks, while only marginal improvement has been gained on cross-lingual generation tasks like machine translation, especially on high-resource languages.
Encoder-Decoder Cross-lingual Pre-training BART (Lewis et al., 2019) pre-trains denoising auto-encoder built with a standard Transformer-based encoder-decoder architecture.
And mBART (Liu et al., 2020b) extends it to the multilingual setting, demonstrating signiﬁcant gains in low/medium-resource machine translation, but with a decrease in high resource languages.
Chi et al.
(2020a)
ﬁrst trains an encoder via MLM and then frozen the encoder to train the decoder only via two generative tasks.
A similar approach is also proposed in Liang et al.
(2020) that extends
Unicoder (Huang et al., 2019) to generation tasks, with the main difference exists in the joint training of encoder-decoder.
All these cross-lingual models emphasize training a dedicated model for
NLG, thus these didn’t improve or even hurt the NLU capabilities of the encoder.
Or they require more computation and memory to match the performance of the encoder-only models.
Uniﬁed Language Representation for NLU and NLG BART (Lewis et al., 2019) and
UNILM (Dong et al., 2019) also endeavor to build a uniﬁed model for NLU and NLG tasks, with the core idea that allows the model to see both unidirectional and bidirectional context.
UNILM pre-trains a Transformer encoder with an ensemble of attention masks, while BART pre-trains a
Transformer encoder-decoder model with arbitrary noising functions.
Our work differs from them in several ways.
Firstly, they only focus on the monolingual (English) domain, while we dedicate to the more challenging multilingual domain.
Secondly, since multiple languages cannot be completely mapped to the same space at every layer only via self-attention like UNILM, the crossattention module is important to model the cross-lingual mapping.
Thus we drop the way to pre-train an encoder-only model under the multilingual scenario.
Thirdly, previous work (Xia et al., 2019) have shown that sharing the encoder and decoder of the Transformer can strengthen the semantic correlation between languages and regularize the encoder-decoder with high capacity on machine translation.
Due to the above reasons, VECO varies during training to train speciﬁc modules via a bidirectional task (IS-MLM) and unidirectional task (CS-MLM).
C ONCLUSION
We present VECO, a variable cross-lingual pre-training model, targeted at initializing both NLU preferred encoder-only and NLG specialized encoder-decoder Transformer.
We analyze three core modules in standard Transformer and propose two masked language modeling tasks to train the reasonable combinations of them.
The two tasks jointly optimize for inner-sequence understanding and cross-sequence generation, enabling them to boost each other via strong regularization from module sharing.
On this account, VECO achieves consistent improvements on various language understanding and generation tasks compared to existing cross-lingual encoder-only and encoderdecoder approaches, opening up new ways of thinking about pre-trained backbone architecture
A P RE -T RAINING D ETAILS
For monolingual data following XLM-R (Conneau et al., 2019) we build a clean CommonCrawl Corpus using an open-source tool CCNet (Wenzek et al., 2019).
Table 5 reports the language codes and statistics of datasets.
There are 1.36TB monolingual data in 50 languages before up/downsampling.
We collect bilingual corpus in 50 languages from the OPUS website 7 , including MultiUN, UNPC,
Bombay, EU-bookshop, OpenSubtitles2018, Tanzil, GlobalVoices, ParaCrawl, MultiParaCrawl, DGT, Tilde, Europarl, Wikipedia, ECB, TED2013, News-Commentary, Ubuntu, Books, UN, infopankki-v1, EUconst, and Bianet.
In total, there are 1TB bilingual training corpus before preprocessing, covering 879 language pairs. Table 6 lists the statistics for each language pair. [Table] [Table]
[Table]
247M [Table] We then apply subword tokenization directly on raw text data using Sentence Piece Model (Kudo & Richardson, 2018) without any additional preprocessing.
We use the whole corpus to train VECO, while using a subset ( ∼ 1 / 4 ) that contains 33 languages to train XLM S MALL and VECO S MALL .
The full set of pre-training hyperparameters for small-sized and large-sized VECO (default) are listed in Table 7.
The comparisons between VECO and other cross-lingual models are shown in Table 8.
B NLU F INE -T UNING D ETAILS
We consider two typical ﬁne-tuning settings:
• Cross-lingual Transfer : For all tasks except sentence retrieve tasks (BUCC and Tatoeba), we select the model with the best average result over all the languages on the dev sets, by searching the learning rate over [1e-5,2e-5,3e-5], training epoch over [3,5,10], and batch size over [16,32,64].
As for sentence retrieve tasks without ﬁne-tuning on any parallel sentences at all, we use the average word embeddings in the middle layer to extract the sentence representations.
• Translate-Train-All : We found that the large-sized training datasets can beneﬁt from a smaller learning rate.
Therefore, for translate-train-all, we select the model on the dev sets, by searching the learning rate over [3e-6,5e-6,1e-5].
For XNLI, PAWS-X, XQuAD and MLQA, we ﬁne-tune our model directly on the translation data provided by the ofﬁcial XTREME repo 8 .
Following the participants (FILTER and Anonymous1) on the XTREME leaderboard, we use several methods to further improve the scores.
For TyDiQA, we start ﬁne-tuning based on the best XQuAD translate-train-all model.
For the sentence retrieve tasks, BUCC and Tatoeba, we use the averaged representation in the middle layer of the best XNLI model.
Note that we only use these tricks under the translate-train-all setting, wishing to have a possible fair comparison on the XTREME leaderboard.
However, we still use less ﬁne-tuned data compared to concurrent works FILTER and Anonymous1.
Speciﬁcally, FILTER, a model-agnostic ﬁne-tune method, uses more translation data on
POS and NER tasks.
Anonymous1 uses other labeled data outside of XTREME.
The full set of ﬁne-tuning hyperparameters is listed in Table 9.
Learning Rate [1e-5,2e-5,3e-5] for Cross-lingual Transfer [3e-6,5e-6,1e-5] for Translate-Train-All
[Table]
[Table]
C D ETAILED R ESULTS ON XTREME
The detailed results of each XTREME task under the cross-lingual transfer and translate-train-all settings on all languages are listed in the following tables. [Table] [Table] [Table]
[Table]
ar de el
[Table]
hi
[Table] [Table] [Table]
en ar bn ﬁ id ko ru sw te
[Table] [Table]
[Table] [Table]