F ROM E NGLISH T O F OREIGN L ANGUAGES : T RANSFERRING P RE - TRAINED L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Pre-trained models have demonstrated their effectiveness in many downstream natural language processing
(NLP) tasks.
The availability of multilingual pre- trained models enables zero-shot transfer of
NLP tasks from high resource languages to low resource ones.
However recent research in improving pre-trained models focuses heavily on English.
While it is possible to train the latest neural architectures for other languages from scratch it is undesirable due to the required amount of compute
In this work
, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget
With a single
GPU
, our approach can obtain a foreign
BERT
BASE model within a day and a foreign BERT
LARGE within two days.
Furthermore, evaluat
- ing our models on six languages, we demonstrate that our models are better than multilingual
BERT on two zero-shot tasks: natural language inference and dependency parsing.
I NTRODUCTION
Pre-trained models (Devlin et al., 2019; Peters et al., 2018) have received much of attention recently thanks to their impressive results in many down stream NLP tasks.
Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer.
Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages.
Wu & Dredze (2019) show the potential of multilingual-BERT (Devlin et al., 2019) in zero-shot transfer for a large number of languages from different language families on ﬁve NLP tasks, namely, natural language inference, document classiﬁcation, named entity recognition, part-of-speech tagging, and dependency parsing.
Although multilingual models are an important ingredient for enhancing language technology in many languages, recent research on improving pre-trained models puts much emphasis on English (Radford et al., 2019; Dai et al., 2019; Yang et al., 2019).
The current state of affairs makes it difﬁcult to translate advancements in pre-training from English to non-English languages.
To our best knowledge, there are only three available multilingual pre-trained models to date: (1) the multilingualBERT (mBERT) 1 that supports 104 languages, (2) cross-lingual language model (XLM; Lample &
Conneau, 2019) 2 that supports 100 languages, and (3) Language Agnostic SEntence Representations (LASER; Artetxe & Schwenk, 2019) 3 that supports 93 languages.
Among the three models, LASER is based on neural machine translation approach and strictly requires parallel data to train.
Do multilingual models always need to be trained from scratch?
Can we transfer linguistic knowledge learned by English pre-trained models to other languages?
In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efﬁcient way (Strubell et al., 2019).
As the ﬁrst step, we focus on building a bilingual language model (LM) of English and a target language.
Starting from a pre-trained English LM, we learn the target language speciﬁc parameters ( i .
e ., word embeddings), while keeping the encoder layers of the pre-trained English LM ﬁxed.
We then ﬁne-tune both English and target model to obtain the bilingual LM.
We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer.
Our main contributions in this work are:
• We propose a fast adaptation method for obtaining a bilingual BERT BASE of English and a target language within a day using one Tesla V100 16GB GPU.
• We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks, namely natural language inference (XNLI; Conneau et al., 2018) and universal dependency parsing.
We show that our models offer competitive performance or even better that mBERT.
• We illustrate that our bilingual LMs can serve as an excellent feature extractor in supervised dependency parsing task.
B ILINGUAL P RE - TRAINED LM S
We ﬁrst provide some background of pre-trained language models.
Let E e be English wordembeddings and Ψ( θ ) be the Transformer (Vaswani et al., 2017) encoder with parameters θ .
Let e w i denote the embedding of word w i ( i .
e ., e w i =
E e [ w 1 ] ).
We omit positional embeddings and bias for clarity.
A pre-trained LM typically performs the following computations: (i) transform a sequence of input tokens to contextualized representations [ c w 1 , . .
c w n ] =
Ψ( e w 1 , . .
, e w n ; θ ) , and (ii) predict an output word
y i at i th position p ( y i | c w i )
∝ exp( c (cid:62) w i e y i ) .
Autoencoding LM (BERT; Devlin et al., 2019) corrupts some input tokens w i by replacing them with a special token [MASK] .
It then predicts the original tokens
y i
=
w i from the corrupted tokens.
Autoregressive LM (GPT-2; Radford et al., 2019) predicts the next token ( y i =
w i +1 )
given all the previous tokens.
The recently proposed XLNet model (Yang et al., 2019) is an autoregressive
LM that factorizes output with all possible permutations, which shows empirical performance improvement over GPT-2 due to the ability to capture bidirectional context.
Here we assume that the encoder performs necessary masking with respect to each training objective.
Given an English pre-trained LM, we wish to learn a bilingual LM for English and a given target language (cid:96) under a limited computational resource budget.
To quickly build a bilingual LM, we directly adapt the English pre-traind model to the target model.
Our approach consists of three steps.
First, we initialize target language word-embeddings E (cid:96) in the English embedding space such that embeddings of a target word and its English equivalents are close together ( § 2.1).
Next, we create a target LM from the target embeddings and the English encoder Ψ
( θ ) .
We then ﬁne-tune target embeddings while keeping Ψ( θ ) ﬁxed ( § 2.2).
Finally, we construct a bilingual LM of E e ,
E (cid:96) , and Ψ( θ ) and ﬁne-tune all the parameters ( § 2.3).
Figure 1 illustrates the last two steps in our approach.
[Figure]
2. I NITIALIZING T ARGET E MBEDDINGS
Our approach to learn the initial foreign word embeddings E (cid:96) ∈ R | V (cid:96) |× d is based on the idea of mapping the trained English word embeddings E e ∈ R | V e |× d to E (cid:96) such that if a foreign word and an English word are similar in meaning then their embeddings are similar.
Borrowing the idea of universal lexical sharing from Gu et al.
(2018), we represent each foreign word embedding
E (cid:96) [ i ] ∈ R d as a linear combination of English word embeddings E e [ j ] ∈ R d
E
[Equation] [Equation]
] =
[Equation]
|
[Equation]
j = α ij E
[Equation] [Equation]
] =
[Equation]
E e (1) where α i ∈ R | V e | is a sparse vector and (cid:80) | V e | j α ij = 1 .
In this step of initializing foreign embeddings, having a good estimation of α could speed of the convergence when tuning the foreign model and enable zero-shot transfer ( § 5).
In the following, we discuss how to estimate α i ∀ i ∈ { 1 , 2 , . . .
, | V (cid:96) |} under two scenarios: (i) we have parallel data of
English-foreign, and (ii)
we only rely on English and foreign monolingual data.
Learning from Parallel Corpus Given an English-foreign parallel corpus, we can estimate word translation probability p ( e j | (cid:96) i ) for any (English-foreign) pair ( e j , (cid:96) i ) using popular word-alignment (Brown et al., 1993) toolkits such as fast-align (Dyer et al., 2013).
We then assign:
[Equation]
Since α i is estimated from word alignment, it is a sparse vector.
Learning from Monolingual Corpus For low resource languages, parallel data may not be available.
In this case, we rely only on monolingual data ( e .
g ., Wikipedias).
We estimate word translation probabilities from word embeddings of the two languages.
Word vectors of these languages can be learned using fastText (Bojanowski et al., 2017) and then are aligned into a shared space with English (Lample et al., 2018b; Joulin et al., 2018).
Unlike learning contextualized representations, learning word vectors is fast and computationally cheap.
Given the aligned vectors ¯ E (cid:96) of foreign and ¯ E e of English, we calculate the word translation matrix A ∈ R | V
[Equation]
|×|
[Equation]
e | as
[Equation]
Here, we use sparsemax (Martins & Astudillo, 2016) instead of softmax.
Sparsemax is a sparse version of softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to a given foreign word.
This property is desirable in our approach since it leads to a better initialization of the foreign embeddings.
2. F INE - TUNING T ARGET E MBEDDINGS
After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM.
We then ﬁne-tune only foreign word-embeddings on target monolingual data.
The training objective is the same as the training objective of the English pre-trained LM ( i .
e ., masked LM for BERT).
Since the trained encoder Ψ( θ ) is good at capturing association, the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task.
For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned.
2. F INE - TUNING B ILINGUAL LM
We create a bilingual LM by plugging foreign language speciﬁc parameters to the pre-trained English LM (Figure 1).
The new model has two separate embedding layers and output layers, one for
English and one for foreign language.
The encoder layer in between is shared.
We then ﬁne-tune this model using English and foreign monolingual data.
Here, we keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer ( § 3).
In this step, the encoder parameters are also updated so that in can learn syntactic aspects ( i .
e ., word order, morphological agreement) of the target languages.
Z ERO - SHOT E XPERIMENTS
In the scope of this work, we focus on transferring autoencoding LMs trained with masked language model objective.
We choose BERT and RoBERTa (Liu et al., 2019) as the source models for building our bilingual LMs, named RAMEN 4 for the ease of discussion.
For each pre-trained model, we experiment with 12 layers (BERT BASE and RoBERTa BASE ) and 24 layers (BERT LARGE and
RoBERTa LARGE ) variants.
Using BERT BASE allows us to compare the results with mBERT model.
Using BERT LARGE and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source pre-trained model.
RoBERTa is a recently published model that is similar to BERT architecturally but with an improved training procedure.
By training for longer time, with bigger batches, on more data, and on longer sequences, RoBERTa matched or exceed previously published models including XLNet.
We include RoBERTa in our experiments to validate the motivation of our work: with similar architecture, does a stronger pre-trained English model results in a stronger bilingual LM?
We evaluate our models on two cross-lingual zero-shot tasks: (1) Cross-lingual Natural Language Inference (XNLI) and (2) dependency parsing.
3. D ATA
We evaluate our approach for six target languages: French ( fr ), Russian ( ru ), Arabic ( ar ), Chinese ( zh ), Hindi ( hi ), and Vietnamese ( vi ).
These languages belong to four different language families.
French, Russian, and Hindi are Indo-European languages, similar to English.
Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively.
The choice of the six languages also reﬂects different training conditions depending on the amount of monolingual data.
French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource.
For experiments that use parallel data to initialize foreign speciﬁc parameters, we use the same datasets in the work of Lample & Conneau (2019).
Speciﬁcally, we use United Nations Parallel
Corpus (Ziemski et al., 2016) for en-ru , en-ar , en-zh , and en-fr .
We collect en-hi parallel data from IIT Bombay corpus (Kunchukuttan et al., 2018) and en-vi data from OpenSubtitles 2018 5 .
For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vectors 6 from fastText (Bojanowski et al., 2017) to estimate word translation probabilities (Eq. 3).
We align these vectors into a common space using orthogonal Procrustes (Artetxe et al., 2016; Lample et al., 2018b; Joulin et al., 2018).
We only use identical words between the two languages as the supervised signal.
We use
WikiExtractor 7 to extract extract raw sentences from Wikipedias as monolingual data for ﬁne-tuning target embeddings and bilingual LMs ( § 2.3).
We do not lowercase or remove accents in our data preprocessing pipeline.
We tokenize English using the provided tokenizer from pre-trained models 8 .
For target languages, we use fastBPE 9 to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and
RoBERTa respectively.
We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models.
Precisely, the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa.
We use XNLI dataset (Conneau et al., 2018) for classiﬁcation task and Universal Dependencies v2.4 (UD; Nivre et al., 2019) for parsing task.
Since a language might have more than one treebank in Universal Dependencies, we use the following treebanks: en ewt (English), fr gsd (French), ru syntagrus (Russian) ar padt (Arabic), vi vtb (Vietnamese), hi hdtb (Hindi), and zh gsd (Chinese).
Remark on BPE Lample et al.
(2018a) show that sharing subwords between languages improves alignments between embedding spaces.
Wu & Dredze (2019) observe a strong correlation between the percentage of overlapping subwords and mBERT’s performances for cross-lingual zero-shot transfer.
However, in our current approach, subwords between source and target are not shared.
A subword that is in both English and foreign vocabulary has two different embeddings.
3. E STIMATING TRANSLATION PROBABILITIES
Since pre-trained models operate on subword level, we need to estimate subword translation probabilities.
Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align (Dyer et al., 2013).
Estimating subword translation probabilities from aligned word vectors requires an additional processing step since the provided vectors from fastText are not at subword level 10 .
We use the following approximation to obtain subword vectors: the vector e s of subword s is the weighted average of all the aligned word vectors e w i that have s as an subword
[Equation]
where p ( w j ) is the unigram probability of word w j and n s = (cid:80) w j : s ∈ w j p ( w j ) .
We take the top 50,000 words in each aligned word-vectors to compute subword vectors.
In both cases, not all the words in the foreign vocabulary can be initialized from the English wordembeddings.
Those words are initialized randomly from a Gaussian N (0 , 1 / d 2 ) .
3. H YPER - PARAMETERS
In all the experiments, we tune RAMEN BASE for 175,000 updates and RAMEN LARGE for 275, updates where the ﬁrst 25,000 updates are for language speciﬁc parameters.
The sequence length is set to 256.
The mini-batch size are 64 and 24 when tuning language speciﬁc parameters using
RAMEN BASE and RAMEN LARGE respectively.
For tuning bilingual LMs, we use a mini-batch size of 64 for RAMEN BASE and 24 for RAMEN LARGE where half of the batch are English sequences and the other half are foreign sequences.
This strategy of balancing mini-batch has been used in multilingual neural machine translation (Firat et al., 2016; Lee et al., 2017).
We optimize RAMEN BASE using Lookahead optimizer (Zhang et al., 2019) wrapped around Adam with the learning rate of 10 − 4 , the number of fast weight updates k = 5 , and interpolation parameter α = 0 .
We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam).
For Adam optimizer, we linearly increase the learning rate from 10 − 7 to 10 − 4 in the ﬁrst 4000 updates and then follow an inverse square root decay.
All
RAMEN LARGE models are optimized with Adam due to memory limit 11 .
When ﬁne-tuning RAMEN on XNLI and UD, we use a mini-batch size of 32, Adam’s learning rate of 10 − 5 .
The number of epochs are set to 4 and 50 for XNLI and UD tasks respectively.
All experiments are carried out on a single Tesla V100 16GB GPU.
Each RAMEN BASE model is trained within a day and each RAMEN LARGE is trained within two days 12 .
R ESULTS
In this section, we present the results of out models for two zero-shot cross lingual transfer tasks:
XNLI and dependency parsing.
4. C ROSS - LINGUAL N ATURAL L ANGUAGE I NFERENCE
Table 1 shows the XNLI test accuracy.
For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM (Lample & Conneau, 2019).
Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN BASE +BERT trained with monolingual only.
[Table]
We ﬁrst discuss the transfer results from BERT.
Initialized from fastText vectors, RAMEN slightly outperforms mBERT by 1.9 points on average and widen the gap of 3.3 points on Arabic.
RAMEN BASE gains extra 0.8 points on average when initialized from parallel data.
With triple number of parameters, RAMEN LARGE offers an additional boost in term of accuracy and initialization with parallel data consistently improves the performance.
It has been shown that BERT LARGE signiﬁcantly outperforms BERT BASE on 11 English NLP tasks (Devlin et al., 2019), the strength of
BERT LARGE also shows up when adapted to foreign languages.
Transferring from RoBERTa leads to better zero-shot accuracies.
With the same initializing condition, RAMEN BASE +RoBERTa outperforms RAMEN BASE +BERT on average by 2.9 and 2.3 points when initializing from monolingual and parallel data respectively.
This result show that with similar number of parameters, our approach beneﬁts from a better English pre-trained model.
When transferring from RoBERTa LARGE , we obtain state-of-the-art results for ﬁve languages.
Currently, RAMEN only uses parallel data to initialize foreign embeddings.
RAMEN can also exploit parallel data through translation objective proposed in XLM.
We believe that by utilizing parallel data during the ﬁne-tuning of RAMEN would bring additional beneﬁts for zero-shot tasks.
We leave this exploration to future work.
In summary, starting from BERT BASE , our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI.
Our approach shows the accuracy gains when adapting from a better pre-trained model.
4. U NIVERSAL D EPENDENCY P ARSING
We build on top of RAMEN a graph-based dependency parser (Dozat & Manning, 2016).
For the purpose of evaluating the contextual representations learned by our model, we do not use part-ofspeech tags.
Contextualized representations are directly fed into Deep-Biafﬁne layers to predict arc and label scores.
Table 2 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing.
Unlabeled Attachment Scores are provided in Appendix A.1.
We ﬁrst look at the fairest comparison between mBERT and monolingually initialized
RAMEN BASE +BERT.
The latter outperforms the former on ﬁve languages except Arabic.
We observe the largest gain of +5.2 LAS for French.
Chinese enjoys +3.1 LAS from our approach.
With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RAMEN+RoBERTa performs better than RAMEN+BERT for most of the languages.
Arabic and Hindi beneﬁt the most from bigger models.
For the other four languages, RAMEN LARGE renders a modest improvement over RAMEN BASE .
[Table]
A NALYSIS 5. I MPACT OF INITIALIZATION
Initializing foreign embeddings is the backbone of our approach.
A good initialization leads to better zero-shot transfer results and enables fast adaptation.
To verify the importance of a good initialization, we train a RAMEN BASE +RoBERTa with foreign word-embeddings that are initialized randomly from N (0 , 1 / d 2 ) .
For a fair comparison, we use the same hyper-parameters in § 3.3.
Table 3 shows the results of XNLI and UD parsing of random initialization.
In comparison to the initialization using aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN BASE by 15.9% for XNLI and 27.8 points for UD parsing on average.
We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization.
[Table]
We also compare RAMEN to a bilingual BERT trained from scratch at 1,000,000 updates.
On average, RAMEN outperforms bilingual BERT by 10.5% accuracy on XNLI task and 13.9 LAS on dependency parsing.
The detail scores are provided in Appendix A.2.
5. A RE CONTEXTUAL REPRESENTATIONS FROM RAMEN ALSO GOOD FOR SUPERVISED PARSING ?
All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks.
It is reasonable to expect RAMENs do well in those tasks as we have shown in our experiments.
But are they also a good feature extractor for supervised tasks?
We offer a partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets
[Table]
We used train/dev/test splits provided in UD to train and evaluate our RAMEN-based parser.
Table 4 summarizes the results (LAS) of our supervised parser.
For a fair comparison, we choose mBERT as the baseline and all the RAMEN models are initialized from aligned fastText vectors.
With the same architecture of 12 Transformer layers, RAMEN BASE +BERT performs competitive to mBERT and outshines mBERT by +1.2 points for Vietnamese.
The best LAS results are obtained by RAMEN LARGE +RoBERTa with 24 Transformer layers.
Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks.
5. H OW DOES LINGUISTIC KNOWLEDGE TRANSFER HAPPEN THROUGH EACH TRAINING STAGES ?
We evaluate the performance of RAMEN+RoBERTa BASE (initialized from monolingual data) at each training steps: initialization of word embeddings (0K update), ﬁne-tuning target embeddings (25K), and ﬁne-tuning the model on both English and target language (at each 25K updates).
The results are presented in Figure 2.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
Zero
[Table]
parsing RoBERTA Without ﬁne-tuning, the average accuracy of XLNI is 39.7% for a three-ways classiﬁcation task, and the average LAS score is 3.6 for dependency parsing.
We see the biggest leap in the performance after 50K updates.
While semantic similarity task proﬁts signiﬁcantly at 25K updates of the target embeddings, syntactic task beneﬁts with further ﬁne-tuning the encoder.
This is expected since the target languages might exhibit different syntactic structures than English and ﬁne-tuning encoder helps to capture language speciﬁc structures.
We observe a substantial gain in LAS from 25 to
LAS for all languages just after 25K updates of the encoder.
Language similarities have more impact on transferring syntax than semantics.
Without tuning the English encoder, French enjoys 50.3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4.2 and 6.4 points using the SVO encoder.
Although
Chinese has SVO order, it is often seen as head-ﬁnal while English is strong head-initial.
Perhaps, this explains the poor performance for Chinese.
C ONCLUSIONS
In this work, we have presented a simple and effective approach for rapidly building a bilingual LM under a limited computational budget.
Using BERT as the starting point, we demonstrate that our approach performs better than mBERT on two cross-lingual zero-shot sentence classiﬁcation and dependency parsing.
We ﬁnd that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models.
We also ﬁnd that RAMEN is also a powerful feature extractor in supervised dependency parsing.
Finally, we hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained English models to other languages.
A A PPENDIX A. U NLABELED A TTACHMENT S CORES AND L ABEL A TTACHMENT S CORES
[Table]
A. C OMPARISION TO B ILINGUAL BERT
For each language pair, we train a bilingual BERT (b-BERT) from scratch using the hyperparameters provided in section § 3.3.
We learn 60,000 BPE codes on a concatenation of English and foreign language monolingual data.
All b-BERT models are trained for 1,000,000 updates, which is signiﬁcantly longer than RAMEN.
Table 6 show the results of XNLI and UD.
[Table]