O N L EARNING U NIVERSAL R EPRESENTATIONS A CROSS L ANGUAGES
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent studies have demonstrated the overwhelming advantage of cross-lingual pre-trained models (PTMs), such as multilingual BERT and XLM
, on crosslingual NLP tasks.
However existing approaches essentially capture the cooccurrence among tokens through involving the masked language model (
MLM
) objective with token-level cross entropy.
In this work, we extend these approaches to learn sentence-level representations and show the effectiveness on cross-lingual understanding and generation.
We propose Hi erarchical C on t rastive
L earning (H ICTL ) to (1) learn universal representations for parallel sentences distributed in one or multiple languages and (2) distinguish the semantically-related words from a shared cross-lingual vocabulary for each sentence
We conduct evaluations on two challenging cross-lingual tasks, XTREME and machine translation.
Experimental results show that the H ICTL outperforms the state of the art
XLM
-R by an absolute gain of 1.3% accuracy on XTREME as well as achieves substantial improvements of +1.7 ∼ +3.6 BLEU on both the high-resource and lowresource English → X translation tasks over strong baselines
I NTRODUCTION
Pre-trained models (PTMs) like ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and
BERT (Devlin et al., 2019) have shown remarkable success of effectively transferring knowledge learned from large-scale unlabeled data to downstream NLP tasks, such as text classiﬁcation (Socher et al., 2013) and natural language inference (Bowman et al., 2015; Williams et al., 2018), with limited or no training data.
To extend such pretraining-ﬁnetuning paradigm to multiple languages, some endeavors such as multilingual BERT (Devlin et al., 2019) and XLM (Conneau & Lample, 2019) have been made for learning cross-lingual representation.
More recently, Conneau et al.
(2020) present XLM-R to study the effects of training unsupervised cross-lingual representations at a huge scale and demonstrate promising progresses on cross-lingual tasks.
However, all of these studies only perform masked language model (MLM) with token-level (i.e., subword ) cross entropy, which limits PTMs to capture the co-occurrence among tokens and consequently fail to understand the whole sentence.
It leads to two major shortcomings for current crosslingual PTMs, i.e., the acquisition of sentence-level representations and semantic alignments among parallel sentences in different languages .
Considering the former, Devlin et al.
(2019) introduced the next sentence prediction (NSP) task to distinguish whether two input sentences are continuous segments from training corpus.
However, this simple binary classiﬁcation task is not enough to model sentence-level representations (Joshi et al., 2020; Yang et al., 2019; Liu et al., 2019; Lan et al., 2020; Conneau et al., 2020).
For the latter, (Huang et al., 2019) deﬁned the cross-lingual paraphrase classiﬁcation task, which concatenates two sentences from different languages as input and classiﬁes whether they are with the same meaning.
This task learns patterns of sentence-pairs well, but fails to distinguish the exact meaning of each sentence.
In response to these problems, we propose to strengthen PTMs through learning universal representations among semantically-equivalent sentences distributed in different languages.
We introduce a novel Hi erarchical C on t rastive L earning (H ICTL ) framework to learn language invariant sentence representations via self-supervised non-parametric instance discrimination.
Speciﬁcally, we use a
BERT-style model to encode two sentences separately, and the representation of the ﬁrst token (e.g., [CLS] in BERT) will be treated as the sentence representation.
Then, we conduct instance-wise comparison at both sentence-level and word-level, which are complementary to each other.
At the sentence level, we maximize the similarity between two parallel sentences while minimize which among non-parallel ones.
At the word level, we maintain a bag-of-words for each sentence-pair, each word in which is considered as a positive sample while the rest words in vocabulary are negative ones.
To reduce the space of negative samples, we conduct negative sampling for word-level contrastive learning.
With the H ICTL framework, the PTMs are encouraged to learn language agnostic representation, thereby bridging the semantic discrepancy among cross-lingual sentences.
The H ICTL is conducted on the basis of XLM-R (Conneau et al., 2020) and experiments are performed on several challenging cross-lingual tasks: language understanding tasks (e.g., XNLI,
XQuAD and MLQA) in the XTREME (Hu et al., 2020) benchmark, and machine translation in the IWSLT and WMT benchmarks.
Extensive empirical evidences demonstrate that our approach can achieve consistent improvements over baselines on various tasks of both cross-lingual language understanding and generation.
In more detail, our H ICTL obtains absolute gains of 1.3% (up to 2.4% on zero-shot sentence retrieval tasks, e.g. BUCC and Tatoeba) accuracy on XTREME over XLMR. For machine translation, our H ICTL achieves substantial improvements over baselines on both low-resource (IWSLT English → X) and high-resource (WMT English → X) translation tasks.
R ELATED W ORK
Pre-trained Language Models.
Recently, substantial work has shown that pre-trained models (PTMs) (Peters et al., 2018; Radford et al., 2018;
Devlin et al., 2019) on the large corpus are beneﬁcial for downstream NLP tasks, like in GLUE (Wang et al., 2018) and XNLI (Conneau et al., 2018).
The application scheme is to ﬁne-tune the pre-trained model using the limited labeled data of speciﬁc target tasks
Lample (2019) trained a transformer-based model on multilingual Wikipedia which covers various languages, while XLM-R (Conneau et al., 2020) studied the effects of training unsupervised crosslingual representations at a very large scale.
For sequence-to-sequence pre-training, UniLM (Dong et al., 2019) ﬁne-tuned BERT with an ensemble of masks, which employs a shared Transformer network and utilizing speciﬁc self-attention mask to control what context the prediction conditions on.
Song et al.
(2019) extended BERT-style models by jointly training the encoder-decoder framework.
XLNet (Yang et al., 2019) trained by predicting masked tokens auto-regressively in a permuted order, which allows predictions to condition on both left and right context.
Raffel et al.
(2019) uniﬁed every NLP problem as a text-to-text problem and pre-trained a denoising sequence-to-sequence model at scale.
Concurrently, BART (Lewis et al., 2020) pre-trained a denoising sequence-to-sequence model, in which spans are masked from the input but the complete output is auto-regressively predicted.
Previous work have explored using pre-trained models to improve text generation, such as pretraining both the encoder and decoder on several languages (Song et al., 2019; Conneau & Lample, 2019; Raffel et al., 2019) or using pre-trained models to initialize encoders (Edunov et al., 2019;
Zhang et al., 2019a).
Zhu et al.
(2020) proposed a BERT-fused NMT model, in which the representations from BERT are treated as context and fed it into all layers of both the encoder and decoder, rather than served as input embeddings only.
Zhong et al.
(2020) formulated the extractive summarization task as a semantic text matching problem and proposed a Siamese-BERT architecture to compute the similarity between the source document and the candidate summary, which leverages the pre-trained BERT in a Siamese network structure.
Our approach also belongs to the contextual pre-training so it could been applied to various downstream NLU and NLG tasks.
Contrastive Learning.
Contrastive learning (CTL) (Saunshi et al., 2019) aims at maximizing the similarity between the encoded query q and its matched key k
+ while keeping randomly sampled keys { k − 0 , k − 1 , k − 2 , ... } faraway from it.
With similarity measured by a score function s ( q, k ) , a form of a contrastive loss function, called InfoNCE (Oord et al., 2018), is considered in this paper:
[Equation]
where the score function s ( q, k ) is essentially implemented as the cosine similarity
[Equation]
and k are often encoded by a learnable neural encoder, such as BERT (Devlin et al., 2019) or ResNet (He
[Figure]
et al., 2016).
k
+ and k − are typically called positive and negative samples.
In addition to the form illustrated in Eq.
(1), contrastive losses can also be based on other forms, such as margin-based loses (Hadsell et al., 2006) and variants of NCE losses (Mnih & Kavukcuoglu, 2013).
Contrastive learning is at the core of several recent work on unsupervised or self-supervised learning from computer vision (Wu et al., 2018;
Oord et al., 2018; Ye et al., 2019; He et al., 2019;
Chen et al., 2020) to natural language processing (Mikolov et al., 2013; Mnih & Kavukcuoglu, 2013; Devlin et al., 2019; Clark et al., 2020b).
Kong et al.
(2020) improved language representation learning by maximizing the mutual information between a masked sentence representation and local n-gram spans.
Clark et al.
(2020b) utilized a discriminator to predict whether a token is replaced by a generator given its surrounding context.
Iter et al.
(2020) proposed to pre-train language models with contrastive sentence objectives that predicts the surrounding sentences given an anchor sentence.
In this paper, we propose H ICTL to encourage parallel cross-lingual sentences to have the identical semantic representation and distinguish whether a word is contained in them as well, which can naturally improve the capability of cross-lingual understanding and generation for PTMs.
M ETHODOLOGY 3. H IERARCHICAL C ONTRASTIVE L EARNING
We propose hierarchical contrastive learning (H ICTL ), a novel comparison learning framework that uniﬁes cross-lingual sentences as well as related words.
H ICTL can learn from both non-parallel and parallel multilingual data, and the overall architecture of H ICTL is illustrated in Figure 1.
We represent a training batch of the original sentences as x = { x 1 , x 2 , ..., x n } and its aligned counterpart is denoted as y
= { y 1 , y 2 , ..., y n } , where n is the batch size.
For each pair (cid:104) x i , y i (cid:105) , y i is either the translation in the other language of x i when using parallel data or the perturbation through reordering tokens in x i when only monolingual data is available 2 .
x \ i is denoted as a modiﬁed version of x where the i -th instance is removed.
Sentence-Level CTL.
As illustrated in Figure 1a, we apply the XLM-R as the encoder to represent sentences into hidden representations.
The ﬁrst token of every sequence is always a special token (e.g., [CLS] ), and the ﬁnal hidden state corresponding to this token is used as the aggregate sentence representation for pre-training,
that is, r x = f ◦ g
( M ( x )) where g ( · ) is the aggregate function and f ( · ) is a linear projection, ◦ denotes the composition of operations.
To obtain universal representation among semantically-equivalent sentences, we encourage r x
i
the query, denoted as q ) to be which has little effect on performance.
as similar as possible to r y i
(the positive sample, denoted as k + ) but dissimilar to all other instances (i.e., y \
i ∪ x \ i , considered as a series of negative samples 3 , denoted as { k − 1 , k − 2 , ..., k − 2 n − 2 } ) in a training batch.
Formally, the sentence-level contrastive loss for x i is deﬁned as
[Equation]
Symmetrically, we also expect r y i (the query, denoted as ˆ q ) to be as similar as possible to r x i (the positive sample, denoted as ˆ k + ) but dissimilar to all other instances in the same training batch, thus,
[Equation]
The sentence-level contrastive loss over the training batch can be formulated as
[Equation]
Word-Level CTL.
The motivations of introducing the word-level contrastive learning are in two folds.
First, a sentence can be in several correct literals expressions and most of them share the similar bag-of-words (Ma et al., 2018).
Thus, it is beneﬁcial for sentence understanding by distinguishing its bag-of-words from the vocabulary.
Second, there is a natural gap between the word embeddings of different languages.
Intuitively, predicting the related words in other languages for each sentence can bridge the representations of words in different languages.
As shown in Figure 1b, we concatenate the sentence pair (cid:104) x
i , y i (cid:105) as x i ◦ y i : [CLS] x i [SEP]
y i [SEP] and the bag-ofwords of which is denoted as B .
For word-level contrastive learning, the ﬁnal state of the ﬁrst token is treated as the query ( ˜ q ), each word w t ∈ B is considered as the positive sample and all the other words ( V\B , i.e., the words in V that are not in B where V indicates the overall vocabulary of all languages) are negative samples.
As the vocabulary usually with large space, we propose to only use a subset S ⊂ V\B sampled according to the normalized similarities between ˜ q and the embeddings of the words.
As a result, the subset S naturally contains the hard negative samples which are beneﬁcial for learning high quality representations (Ye et al., 2019).
Speciﬁcally, the word-level contrastive loss for (cid:104) x i , y i (cid:105) is deﬁned as
[Equation]
where e ( · ) is the embedding lookup function and |B| is the number of unique words in the concatenated sequence x i ◦ y i .
The overall word-level conrastive loss can be formulated as:
[Equation]
Multi-Task Pre-training.
Both MLM and translation language model (TLM) are combined with
H ICTL by default, as the prior work (Conneau & Lample, 2019) have veriﬁed the effectiveness of them in XLM.
In summary, the model can be optimized by minimizing the entire training loss:
[Equation]
where L LM is implemented as either the TLM when using parallel data or the MLM when only monolingual data is available to recover the original words of masked positions given the contexts.
3. C ROSS - LINGUAL F INE - TUNING
Language Understanding.
The representations produced by H ICTL can be used in several ways for language understanding tasks whether they involve single text or text pairs.
Concretely, ( i ) the [CLS] representation of single-sentence in sentiment analysis or sentence pairs in paraphrasing and entailment is fed into an extra output-layer for classiﬁcation.
( ii ) The pre-trained encoder can be used to assign POS tags to each word or to locate and classify all the named entities in the sentence for structured prediction, as well as ( iii ) to extract answer spans for question answering.
[Table]
Language Generation.
We also explore using H to improve machine translation.
In the previous work,
Conneau & Lample (2019) has shown that the pre-trained encoders can provide a better initialization of supervised and unsupervised neural machine translation (NMT) systems.
Liu et al.
(2020) has shown that NMT models can be improved by incorporating pre-trained sequence-tosequence models on various language pairs but highestresource settings.
As illustrated in Figure 2, we use the model pre-trained by H ICTL as the encoder, and add a new set of decoder parameters that are learned from scratch.
To prevent pre-trained weights from being washed out by supervised training, we train the encoder-decoder model in two steps.
At the ﬁrst step, we freeze the pre-trained encoder and only update the decoder.
At the second step, we train all model parameters for a relatively small number of iterations.
In both cases, we compute the similarities between the [CLS] representation of the encoder and all target words in advance.
Then we aggregate them with the logits before softmax of each decoder step through an element-wise additive operation.
The encoder-decoder model is optimized by maximizing the log-likelihood of bitext at both steps.
E XPERIMENTS
We consider two evaluation benchmarks: cross-lingual language understanding tasks in XTREME and machine translation tasks (IWSLT’14 English ↔{ German } , IWSLT’17 English
→{ French, Chinese } , WMT’14 English →{ German, French } and WMT’18 English →{ Chinese, Russian } ).
In this section, we describe the data and training details, and provide detailed evaluation results.
4. D ATA AND M ODEL
Our model is pre-trained on 15 languages, including English (en), French (fr), Spanish (es), German (de), Greek (el), Bulgarian (bg), Russian (ru), Turkish (tr), Arabic (ar), Vietnamese (vi), Thai (th),
Chinese (zh), Hindi (hi), Swahili (sw) and Urdu (ur).
For monolingual data, we use the Wikipedia from these languages.
For bilingual data, we use the same ( English-to-X ) MT dataset as (Conneau &
Lample, 2019), which are collected from MultiUN (Eisele & Yu, 2010) for French, Spanish, Arabic and Chinese, the IIT Bombay corpus (Kunchukuttan et al., 2018) for Hindi, the OpenSubtitlesfor Turkish, Vietnamese and Thai, the EUbookshop corpus for German, Greek and Bulgarian, Tanzil for both Urdu and Swahili, and GlobalVoices for Swahili.
Table 1 lists the statistics.
We adopt the T RANSFORMER -E NCODER (Vaswani et al., 2017) as the backbone with 12 layers and 768 hidden units for H ICTL Base , and 24 layers and 1024 hidden units for H ICTL .
We initialize the parameters of H ICTL with XLM-R (Conneau et al., 2020).
During pre-training, a training batch for
H ICTL covers 15 languages with equal probability, each instance with two sentences as input and the max sequence length is set to 128.
We use Adam optimizer to train the model, and learning rate starts from 2 .
5 e − 5 with invert linear decay.
We run the pre-training experiments on 8 V100 GPUs, batch size 32.
The number of negative samples m = 512 for word-level contrastive learning.
4. E XPERIMENTAL E VALUATION
Cross-lingual Language Understanding (XTREME)
There are nine tasks in XTREME that can be grouped into four categories: ( i ) sentence classiﬁcation consists of Cross-lingual Natural Language Inference (XNLI)
(Conneau et al., 2018) and Cross-lingual Paraphrase Adversaries from
[Table]
[Table] [Table]
Word Scrambling (PAWS-X) (Zhang et al., 2019b).
( ii ) Structured prediction includes POS tagging and NER.
We use POS tagging data from the Universal Dependencies v2.5 (Nivre et al., 2018) treebanks.
Each word is assigned one of 17 universal POS tags.
For NER, we use the Wikiann dataset (Pan et al., 2017).
( iii ) Question answering includes three tasks: Cross-lingual Question Answering (XQuAD) (Artetxe et al., 2019),
Multilingual Question Answering (MLQA) (Lewis et al., 2019), and ( iii ) the gold passage version of the Typologically Diverse Question Answering dataset (TyDiQA-GoldP)
(Clark et al., 2020a).
( iv ) Sentence retrieval includes two tasks: BUCC (Zweigenbaum et al., 2017) and Tatoeba (Artetxe & Schwenk, 2019), which aims to extract parallel sentences between the English corpus and target languages.
As XTREME provides no training data, thus we directly evaluate pre-trained models on test sets.
Table 2 provides detailed results on four categories in XTREME.
First, compared to the state of the art XLM-R baseline, H ICTL further achieves signiﬁcant gains of 1.43% and 1.30% on average on nine tasks with cross-lingual zero-shot transfer and translate-train settings, respectively.
Second,
H ICTL outperforms XLM-R by signiﬁcant margins of 2.6% and 2.2% on two sentence retrieval tasks even with no training data, which fully depends on the output representations of H ICTL without the ﬁne-tuning stage.
Table 3 shows XNLI results on each language.
We evaluate our model on cross-lingual transfer from English to other languages (denoted as C ROSS - LINGUAL T EST ).
It means that the pre-trained model is ﬁne-tuned on English MultiNLI, and then evaluated on the foreign language XNLI test.
In addition, we also consider two machine translation 4 baselines:
(1) T RANSLATE -T RAIN :
we ﬁnetune a multiligual model using the training set machine-translated from English for each language (2) T RANSLATE - TRAIN - ALL : we ﬁne-tune a multilingual model on the concatenation of all training sets from T RANSLATE -T RAIN .
H ICTL achieves remarkable results in all ﬁne-tuning settings.
On cross-lingual transfer (C ROSSLINGUAL T EST ), H ICTL Base obtains 77.3% accuracy, outperforming the state-of-the-art XLMR Base , Unicoder and XLM models by 1.1%, 1.9% and 2.2% average accuracy.
Compared to mBERT, H ICTL Base obtains substantial gains of 11.4%, 12.7% and 13.8% on Urdu, Arabic and
Chinese respectively.
Using the multilingual pre-training of T RANSLATE - TRAIN - ALL , H ICTL further improves performance and reaches 83.8% accuracy, outperforming XLM-R and Unicoder by 0.9% and 5.3% average accuracy respectively.
Machine Translation
The main idea of H ICTL is to summarize cross-lingual parallel sentences into a shared representation that we term as semantic embedding, using which semantically related words can be distinguished from others.
Thus it is natural to apply this global embedding to text generation.
To that end, we ﬁne-tune the pre-trained H ICTL on machine trans-
[Table]
lation tasks with both low-resource and high-resource settings.
For the low-resource scenario, we choose IWSLT’14 English ↔ German
(En ↔ De) 5 , IWSLT’17 English → French (En → Fr) and
English → Chinese (En → Zh) translation 6 .
There are 160k, 183k, 236k, 235k bilingual sentence pairs for En ↔ De, En
→
Fr and En → Zh tasks.
For the rich-resource scenario, we work on WMT’
En →{ De, Fr } and WMT’18 En →{ Zh, Ru } .
For WMT’
En →{ De, Fr } , the corpus sizes are 4.5M and 36M respectively, and we concatenate newstest 2012 and newstest 2013 as the validation set and use newstest 2014 as the test set.
For WMT’
En →{ Zh, Ru } , there are 24M and 8M sentence pairs respectively, we select the best models on newstest 2017 and report results on newstest 2018 .
During ﬁne-tuning, we use the pre-trained model to initialize the encoder and introduce a randomly initialized decoder.
We develop a shallower decoder with 4 identical layers (with 768 hidden units, 12 heads) to reduce the computation overhead.
At the ﬁrst ﬁne-tune step, we concatenate the datasets of all language pairs in either low-resource or high-resource setting to optimize the decoder only until convergence.
Then we tune the whole encoder-decoder model using per-language corpus at the second step.
The initial learning rate is 2 e − 5 and inverse sqrt learning rate (Vaswani et al., 2017) scheduler is also adopted.
For WMT’
En → De, we use beam search with widthand length penalty 0.6 for inference.
For other tasks, we use width 5 and length penalty 1.0.
We use multi-bleu.perl to evaluate IWSLT’14 En ↔ De and WMT tasks, but sacreBLEU for the remaining tasks, for fair comparison with previous work.
Results are reported in Table 4.
We implemented standard Transformer (apply the base and big setting for IWSLT and WMT tasks respectively) as baseline.
The proposed H ICTL can improve the BLEU scores of the eight tasks by 1.93, 1.76, 2.7, 2.2, 1.43, 1.25, 1.87 and 1.67.
As the taskadaptive pre-training (T APT ) (Gururangan et al., 2020) can be applied to our H ICTL with minimal modiﬁcations, thus we introduce a second phase of pre-training for H ICTL on IWSLT or WMT parallel corpora (denoted as H ICTL ∗ ), which can obtain additional gains of 0.7 BLEU on average.
Our approach also outperforms the BERT-fused model
(Yang et al., 2020), a method treats BERT as an extra context and fuses the representations extracted from BERT with each encoder and decoder layer.
Note we achieve new state-of-the-art results on IWSLT’
En → De, IWSLT’
En →{ Fr,
[Table]
[Table]
[Table]
replace H ICTL with XLM-R 30.
Table 5 provides results of ablation study on I WSLT ’
En → De and
W MT ’
En → De tasks.
( i )
We remove the [CLS] aggregation operation, which leads to consistent drops of 0.45 and 0.64 BLEU on both tasks.
( ii ) We also try to directly ﬁnetune the whole NMT until convergence.
We ﬁnd that the BLEU of I WSLT ’
En → De slightly decreases, while the one of W MT ’
En → De signiﬁcantly decreases.
Thus pre-trained weights can be washed out by supervised training with rich resources.
( iii ) Instead of H ICTL , we use XLM-R that is further pre-trained on IWSLT and WMT parallel corpora to initialize the encoder.
The BLEU scores are 30.72 and 30.04, which are better than the vanilla T RANSFORMER but signiﬁcantly worse than leveraging H ICTL . [Figure]
Visualization We collect 10 sets of samples from
W MT ’14-19, each of them contains 100 parallel sentences distributed in 5 languages.
As the t-SNE visualization in Figure 3, a set of sentences under the same meaning are clustered more densely for H ICTL than XLM-R, which reveals the strong capability of H ICTL on learning universal representations across different languages.
Note that the t-SNE visualization of H ICTL still demonstrates some noises, we attribute them to the lack of hard negative examples for sentence-level contrastive learning and leave this to future work for consideration.
C ONCLUSION
We have demonstrated that pre-trained language models (PTMs) trained to learn commonsense knowledge from large-scale unlabeled data highly beneﬁt from hierarchical contrastive learning (H ICTL ), both in terms of cross-lingual language understanding and generation.
Learning universal representations at both word-level and sentence-level bridges the semantic discrepancy across languages.
As a result, our H ICTL sets a new level of performance among cross-lingual PTMs, improving on the state of the art by a large margin.
We have also presented that by combing our method with task-adaptive pre-training, the better results cab be obtained.
Even rich-resource languages also have been improved.