E NHANCING C ROSS - LINGUAL T RANSFER BY M ANIFOLD M IXUP
Anonymous authors
Paper under double-blind review
A BSTRACT
Based on large-scale pre-trained multilingual representations, recent cross-lingual transfer methods have achieved impressive transfer performances.
However, the performance of target languages still lags far behind the source language.
In this paper, our analyses indicate such a performance gap is strongly associated with the cross-lingual representation discrepancy.
To achieve better cross-lingual transfer performance, we propose the cross-lingual manifold mixup (X-Mixup) method, which adaptively calibrates the representation discrepancy and gives compromised representations for target languages.
Experiments on the XTREME benchmark show X-Mixup achieves 1.8% performance gains on multiple text understanding tasks, compared with strong baselines, and reduces the cross-lingual representation discrepancy signiﬁcantly.
I NTRODUCTION
Many natural language processing tasks have shown exciting progress utilizing deep neural models.
However, these deep models often heavily rely on sufﬁcient annotation data, which is not the case in the multilingual setting.
The fact is that most of the annotation data are collected for popular languages like English and Spanish (Ponti et al., 2019; Joshi et al., 2020), while many long-tail languages could hardly obtain enough annotations for supervised training.
As a result, cross-lingual transfer learning (Prettenhofer & Stein, 2011; Wan et al., 2011; Ruder et al., 2019) is crucial, transferring knowledge from the annotation-rich source language to low-resource or zero-resource target languages .
In this paper, we focus on the zero-resource setting, where labeled data are only available in the source language.
Recently, multilingual pre-trained models (Conneau & Lample, 2019; Conneau et al., 2020a; Xue et al., 2020) offer an effective way for cross-lingual transfer, which yield a universal embedding space across various languages.
Such universal representations make it possible to transfer knowledge from the source language to target languages through the embedding space, signiﬁcantly improving the transfer learning performance (Chen et al., 2019;
Zhou et al., 2019; Keung et al., 2019; Fang et al., 2020).
Besides, Conneau et al.
(2018) proposes translate-train , a simple yet effective crosslingual data augmentation method, which constructs pseudo-training data for each target language via machine translation.
Although these works have achieved impressive improvements in cross-lingual transfer (Hu et al., 2020; Ruder et al., 2021), signiﬁcant performance gaps between the source language and target languages still remain (see Table 1).
Hu et al.
(2020) refers to the gap as the cross-lingual transfer gap , a difference between the performance on the source and target languages.
To investigate how the cross-lingual transfer gap emerges, we perform relevant analyses, demonstrating that transfer performance correlates well with the cross-lingual representation discrepancy (see
Section 3 for details).
Here the cross-lingual representation discrepancy means the degree of difference between the source and target language representations in the universal embedding space.
As shown in Figure 1(a), in translate-train, the representation distribution of Spanish almost overlaps with English, while Arabic shows a certain representation discrepancy compared with English and
Swahili performs larger discrepancy, where translate-train achieves 88.6 accuracy on English, 85.7 on
Spanish, 82.2 on Arabic and 77.0 on Swahili.
Intuitively, a larger representation discrepancy could lead to a worse cross-lingual transfer performance.
In this paper, we propose the Cross-Lingual Manifold Mixup (X-Mixup) approach to ﬁll the crosslingual transfer gap.
Based on our analyses, reducing the cross-lingual representation discrepancy is
[Figure]
a promising way to narrow the transfer gap.
Given the cross-lingual representation discrepancy is hard to remove, X-Mixup directly faces the issue and explicitly accommodates the representation discrepancy in the neural networks, by mixing the representation of the source and target languages during training and inference.
With X-Mixup, the model itself can learn how to escape the discrepancy, which adaptively calibrates the representation discrepancy and gives compromised representations for target languages to achieve better cross-lingual transfer performance.
X-Mixup is motivated by robust deep learning (Vincent et al., 2008), while X-Mixup adopts the mixup (Zhang et al., 2018) idea to handle the cross-lingual discrepancy.
Speciﬁcally, X-Mixup is designed upon the translate-train approach, faced with the exposure bias (Ranzato et al., 2016) problem and data noise problem.
During training, the source sequence is a real sentence and the target sequence is a translated one, while situations are opposite during inference.
Besides, the translated text often introduces some noises due to imperfect machine translation systems.
To address them, we further impose the Scheduled Sampling (Bengio et al., 2015) and Mixup Ratio in
X-Mixup to handle the distribution shift problem and data noise problem, respectively.
We verify X-Mixup on the cross-lingual understanding benchmark XTREME (Hu et al., 2020), which includes several understanding tasks and covers 40 languages from diverse language families.
Experimental results show X-Mixup achieves 1.8% performance gain across different tasks and languages, comparing with strong baselines.
It also reduces the cross-lingual representation discrepancy signiﬁcantly, as Figure 1(b) shows.
R ELATED W ORK
Multilingual Representation Learning Recent studies have demonstrated the superiority of largescale pre-trained multilingual representations on downstream tasks.
Multilingual BERT (mBERT;
Devlin et al., 2019) is the ﬁrst work to extend the monolingual pre-training to the multilingual setting.
Then, several extensions achieve better cross-lingual performances by introducing more monolingual or parallel data and new pre-training tasks, such as Unicoder (Huang et al., 2019), XLM-R (Conneau et al., 2020a), ALM (Yang et al., 2020), MMTE (Siddhant et al., 2020), InfoXLM (Chi et al., 2020),
HICTL (Wei et al., 2020), ERNIE-M (Ouyang et al., 2020), mT5 (Xue et al., 2020), nmT5 (Kale et al., 2021), AMBER (Hu et al., 2021) and VECO (Luo et al., 2021).
They have been the standard backbones of current cross-lingual transfer methods.
Cross-lingual Transfer Learning Cross-lingual transfer learning (Prettenhofer & Stein, 2011;
Wan et al., 2011; Ruder et al., 2019) aims to transfer knowledge learned from source languages to target languages.
According to the type of transfer learning (Pan & Yang, 2010), previous crosslingual transfer methods can be divided into three categories: instance transfer, parameter transfer, and feature transfer.
The cross-lingual transferability improves a lot when engaged with the instance
The data resource refers to resource language
Germanic
so we
Germanic languages
[Table] [Table]
by translation (i.e. translate-train, translate-test) or other cross-lingual data augmentation methods (Singh et al., 2019; Bornea et al., 2020; Qin et al., 2020; Zheng et al., 2021).
Chen et al.
and
Zhou et al.
(2019) focus on the parameter transfer to learn a share-private model architecture.
Besides, other works implement the feature transfer to learn the language-invariant features by adversarial networks (Keung et al., 2019; Chen et al., 2019) or re-alignment (Libovick ´ y et al., 2020; Zhao et al., 2020).
X-Mixup utilizes both the instance transfer and feature transfer, which is based on the translate-train data augmentation approach and implements the feature transfer by cross-lingual manifold mixup.
Mixup and Its Variants Mixup (Zhang et al., 2018) proposes to train models on the linear interpolation at both the input level and label level, which is effective to improve the model robustness and generalization.
Generally, the interpolated pair is selected randomly.
Manifold mixup (Verma et al., 2019) performs the interpolation in the latent space by conducting the linear combinations of hidden states.
Previous mixup methods (Chen et al., 2020; Jindal et al., 2020) focus on the monolingual setting.
However, X-Mixup focuses on the cross-lingual setting and faces many new challenges (see Section 4 for details).
Besides, in contrast to previous mixup methods, X-Mixup mixes the parallel pairs, which share the same semantics across different languages.
As a result, the choice of parallel pairs for interpolation can build a smart connection between the source and target languages.
A NALYSES OF THE C ROSS - LINGUAL T RANSFER P ERFORMANCE
In this section 1 , we concentrate on the cross-lingual transfer performance and ﬁnd it is strongly associated with the cross-lingual representation discrepancy.
Firstly, we observe the cross-lingual transfer performance on different target languages and propose an assumption.
Then we conduct qualitative and quantitative analyses to verify it.
Although previous studies (Hu et al., 2020; Ruder et al., 2021) have shown impressive improvements on cross-lingual transfer, the cross-lingual transfer gap is still pretty large, more than 16 points in
Hu et al.
Furthermore, results in Table 1 show the performance of low-resource languages and dissimilar languages fall far behind other languages in cross-lingual transfer tasks.
Compared with English, the representations of other languages, especially low-resource languages, are not well-trained (Lauscher et al., 2020; Wu & Dredze, 2020), because high-resource languages dominate the representation learning process, which results in the cross-lingual representation discrepancy.
Besides, dissimilar languages often show differences in language characteristics (like vocabulary, word order), which also leads to the representation discrepancy.
As a result, we assume that the cross-lingual transfer performance is closely related to the representation discrepancy between the source language and target languages.
Following Conneau et al.
(2020b), we utilize the linear centered kernel alignment (CKA; Kornblith et al., 2019) score to indicate the cross-lingual representation discrepancy
[Equation]
where X and Y are parallel sequences from the source and target languages, respectively.
A higher
CKA score denotes a smaller cross-lingual representation discrepancy.
To verify our assumption, we perform qualitative and quantitative analyses on the relationship between the CKA score and cross-lingual transfer performance.
Figure 3 in Appendix B indicates a higher CKA score tends to induce better cross-lingual transfer performance.
We also calculate the
Spearman’s rank correlation between the CKA score and the transfer performance in Table 2, which shows a strong correlation between them.
Both the trend and correlation score conﬁrm the crosslingual transfer performance is highly related to the cross-lingual representation discrepancy .
M ETHODOLOGY : X-M IXUP
Based on the aforementioned analyses, we believe that reducing the cross-lingual representation discrepancy is the key to ﬁlling the cross-lingual transfer gap.
In this section, we propose X-Mixup to explicitly reduce the representation discrepancy by implementing the manifold mixup between the source language and target language.
With X-Mixup, the model can adaptively calibrate the representation discrepancy and give compromised representations for target languages.
This section will ﬁrst introduce the overall architecture of X-Mixup and its details.
After that, the training objectives and inference process will be shown.
4. O VERALL A RCHITECTURE
Figure 2 illustrates the overall architecture of X-Mixup.
Sequences from the source and target languages are ﬁrst encoded separately.
Then within the encoder, X-Mixup implements the manifold mixup between the paired sequences (original sequence and its translation) within a speciﬁc layer, where Mixup Ratio controls the degree of mixup and Scheduled Sampling schedules the data sampling process during training.
Notations We use S to denote the source language and T to denote the target language.
h l denotes the hidden states of a sequence in layer l .
D denotes the real text data collection and ˜ D denotes the translation data collection.
For downstream understanding tasks, there are annotation data in the source language D Train S =
( X Train S , Y Train S ) and raw test data in the target language D Test T =
( X Test T ) .
Through translate-train, we can get pseudo-training data in the target language ˜ D Train T =
( ˜ X Train T , ˜ Y Train T ) .
Similarly, through translate-test, we can get pseudo-test data in the source language ˜ D Test S =
( ˜ X Test S ) .
During training, the Scheduled Sampling process uses translation data 2 ˜ D Train S =
( ˜ X Train S ) from the source language. Note that we use translation data ( ˜ X Train T and ˜ X Test S ) and translatetrain labels ( ˜ Y Train T ) from the ofﬁcial XTREME repository, which is in the same setting as baselines.
Basic Model We use mBERT (Devlin et al., 2019) or XLM-R (Conneau et al., 2020a)
as the backbone model.
Within each layer, there are two sub-layers: the multi-head attention layer and the feed-forward layer 3 , followed by the residual connection and layer norm.
We use the same multi-head attention layer (see details in Appendix A.1) as BERT (Devlin et al., 2019), where inputs are query, key, and value respectively.
In layer l + 1 , the hidden states of the source sequence x S and target sequence x T are acquired by the multi-head attention h l +1 S = MultiHead ( h lS , h lS , h lS
[Equation]
× n × n
Multi-head Attention
LN
Feed Forward
Add & LN 𝐡 !
𝐡 "
Q K V
Multi-head Attention
Feed Forward
Add & LN
Q K V
Add & LN
Multi-head Attention
Q K V
Add & LN 𝐡 "|!
𝐡 " ’𝐡 "
Train:
Inference:
X-Mixup
Output layer
Output layer ℒ $%&’(’) ℒ )*’+" ℒ )*’+! 𝐱 ! ∈ +𝒟 !",’)
𝐱 " ∈ +𝒟 ""-*(& 𝐱 !
∈ 𝒟 !"
-*(& ∪ +𝒟 !"
-*(& 𝐱 " ∈ 𝒟 "",’) λ
[Figure]
language
Target language 𝐀
Share parameters
Manifold Mixup To reduce the cross-lingual representation discrepancy, a straightforward idea is to ﬁnd compromised representations between the source and target languages.
It’s difﬁcult to ﬁnd such representations because of varying degrees of differences across languages, like vocabulary and word order.
However, manifold mixup provides an elegant way to get intermediate representations by conducting linear interpolation on hidden states.
To extract target-related information from the source hidden states, the target hidden states are used as the query, and the source hidden states are used as the key and value.
This cross-attention process is computed as
[Equation]
(3) which shares parameters with the multi-head attention.
The manifold mixup process mixes the target hidden states h l +1 T and the source-aware target hidden states h l +1 T | S based on the mixup ratio λ
[Equation]
where λ is an instance-level parameter, ranging from 0 to 1, and indicates the degree of manifold mixup.
LN denotes the layer norm operation.
Mixup Ratio
The machine translation process may change the original semantics and introduce data noises in varying degrees (Castilho et al., 2017; Fomicheva et al., 2020).
Thus we introduce the translation quality modeling in the mixup process to handle this problem.
Following Fomicheva et al.
(2020), we use the entropy of attention weights to measure the translation quality
[Equation]
where
[Equation]
I is the number of target tokens and J is the number of source tokens.
Lower entropy implies better cross-lingual alignment and higher translation quality.
To introduce the translation quality modeling into the manifold mixup process, we compute the mixup ratio as λ = λ 0 · σ [( H ( A ) + H
( A (cid:62) ))
W + b ] , where σ is the sigmoid function, and W , b are trainable parameters.
λ 0 is the max value of the mixup ratio, which is set to 0.5 in this paper.
We consider two-way alignment in the translation quality modeling, i.e. H ( A ) and H ( A (cid:62) ) .
Scheduled
Sampling
The source sequences utilized in training and inference are drawn from different distributions.
During training, the source sequence is a real text from D Train S , while during inference, the source sequence is a translation from ˜ D Test S .
This discrepancy, commonly called the exposure bias, leads to a gap between training and inference.
Motivated by the scheduled sampling approach (Bengio et al., 2015) in NMT, we sample the source sequence dynamically during training.
Speciﬁcally, the source sequence fed into the manifold mixup is either a real text from D Train S or translation from ˜ D Train S with a certain probability p
[Equation]
where p ∗ is decreasing during training to match the situation of inference.
We utilize the inverse sigmoid decay (Bengio et al., 2015), which decreases p ∗ as a function of the index of mini-batch.
4. F INAL T RAINING O BJECTIVE
The training loss is composed of two parts: the task loss and the consistency loss
[Equation]
where MSE ( · ) is Mean Squared Error and KL ( · ) is Kullback-Leibler divergence.
r ∗ is the sequence representation 4 and p ∗ is the predicted probability distribution of downstream tasks.
The task loss L task is the sum of the source language task loss L S task and target language one L T task , weighted by the hyper-parameter α , which is utilized to balance the training process
[Equation]
For classiﬁcation, structured prediction, and span extraction tasks, the task loss is the cross-entropy loss (see details in Appendix A.2).
For structured prediction tasks, it is non-trivial to implement the token-level label mapping across different languages.
Thus we use the label probability distribution, predicted by the source language task model, as the pseudo-label for training, where tokens and labels are corresponding.
The consistency loss is composed of two parts: the representation consistency loss and the prediction consistency loss.
The ﬁrst loss is a regularization term and provides a way to align representations across different languages (Ruder et al., 2019).
The second loss is to make better use of the supervision of downstream tasks. It only exists in the classiﬁcation task, as the translation process does not change the label of this task, while in other tasks, it does.
4. I NFERENCE
During inference, the manifold mixup process is the same as training, except for the Scheduled
Sampling process.
Concretely, for the source language, only translation data are available in the inference stage, without real data, so we use x s ∈ ˜ D Test S .
For classiﬁcation tasks, we synthesize the predictions of both the source and target sequences by taking the mean of the predicted probability distributions as the ﬁnal prediction.
For structured prediction and QA tasks, we only consider the prediction of the target sequence.
E XPERIMENTS
This section ﬁrst introduces the cross-lingual understanding benchmark, XTREME.
Then brieﬂy introduces the conﬁgurations of downstream tasks and baselines.
Finally, shows the main results of baselines and X-Mixup on XTREME.
[Table]
5. T ASKS AND S ETTINGS
Tasks In our experiments, we focus on three types of tasks in XTREME: (1) sentence pair classiﬁcation task:
XNLI (Conneau et al., 2018) and PAWS-X (Yang et al., 2019); (2) structured prediction task:
POS (Nivre et al., 2018) and NER (Pan et al., 2017); (3) question answering task:
XQuAD (Artetxe et al., 2020), MLQA (Lewis et al., 2020) and TyDiQA (Clark et al., 2020).
The details of these datasets can refer to Hu et al.
We utilize the translate-train and translate-test data from the XTREME repo 5 , which also provide the pseudo-label of translate-train data for classiﬁcation tasks and question answering tasks.
The rest translation data are from Google Translate 6 .
Models Experiments are based on two multilingual pre-trained models: mBERT and XLM-R. We use the pre-trained models of Huggingface Transformers 7 as the backbone model.
Hyper-parameters We select XNLI, POS, and MLQA as representative tasks to search for the best hyper-parameters.
The ﬁnal model is selected based on the averaged performance of all languages on the dev set.
We perform grid search over the balance training parameter α and learning rate from [0.2, 0.4, 0.6, 0.8] and [3e-6, 5e-6, 2e-5, 3e-5].
We also search for the best manifold mixup layer from [1, 4, 8, 12, 16, 20, 24].
In ﬁnal results, we implement mixup in the ﬁrst layer for classiﬁcation tasks, 4th layer for structured prediction tasks.
For QA tasks, we implement mixup in the 16th layer for large model, 8th layer for base model.
Concrete details of experiments are presented in Appendix C.1.
5. B ASELINES
We conduct experiments on two strong multilingual pre-trained models to verify the generality of methods: (1) mBERT Multilingual BERT is a 12-layer transformer model pre-trained on the
Wikipedias of 104 languages.
(2) XLM-R XLM-R-large is a 24-layer transformer model pretrained on 2.5T data extracted from Common Crawl covering 100 languages.
Based on them, these are some strong baselines: (1) Trans-train Abbreviation for Translate-train.
The training set of the source language is machine-translated to each target language and then the model is trained on the concatenation of all training sets.
(2) Joint-Align Zhao et al.
(2020) aligns the monolingual sub-spaces of the source and target language by minimizing the distances of embeddings for matched word pairs.
(3) Filter Fang et al.
(2020) splices the representation of the target sequence and its translation in intermediate layers to extract multilingual knowledge.
(4) X T UNE
Zheng et al.
(2021) uses two types of consistency regularization based on four types of data augmentation.
5. M AIN R ESULTS
Results on the XTREME benchmark are shown in Table 3.
Concrete results for each task are presented in Appendix C.2.
Compared with strong baselines, X-Mixup shows its superiority across [Table] [Table] [Table]
different backbones and tasks, which indicates its generality.
For the mBERT backbone, X-Mixup outperforms Trans-train by 2.2%.
For the XLM-R backbone, X-Mixup outperforms Filter by 1.5%.
The superiority of X-Mixup over Filter is that X-Mixup gives a calibrated representation for target languages, not just the concatenation of two representations.
Besides, X-Mixup considers the noise of translation data and limits the noise propagation by the mixup ratio.
Compared with methods in Table 3, X T UNE uses additional three data augmentation approaches.
To make a fairer comparison, we conduct experiments under the same setting in Table 4.
Results indicate
X-Mixup outperforms X T UNE on three types of tasks.
Besides, X-Mixup only needs one-stage training, while X T UNE implements a two-stage training algorithm.
However, X-Mixup and X T UNE are complementary, where the former focuses on ﬁnding better representations for target languages but the latter concentrates on the cross-lingual data augmentation and consistency regularization.
X-Mixup can also implement the cross-lingual manifold mixup based on the four cross-lingual data augmentation methods and the model consistency regularization proposed by X T UNE .
A NALYSIS AND D ISCUSSION
To better understand X-Mixup and explore how X-Mixup inﬂuences the cross-lingual transfer performance, we conduct analyses 8 on several questions.
Results show X-Mixup achieves performance improvements across different languages and it also reduces the cross-lingual representation discrepancy obviously.
Table 7 in Appendix B veriﬁes the effectiveness of X-Mixup on both seen and unseen languages.
Besides, ablation results show the cross-lingual manifold mixup training contributes a lot to cross-lingual transfer.
(Q1)
How X-Mixup inﬂuences the overall cross-lingual transfer gap?
We compare the crosslingual transfer gap of X-Mixup with baselines in Table 5.
Compared with Trans-train, X-Mixup reduces the averaged gap by 39.8% and shows its superiority across three types of tasks, reducing the averaged gap by 38.0% on classiﬁcation tasks, 20.7% on structured prediction tasks and 52.8% on QA tasks.
Compared with state-of-the-art methods, Filter and X T UNE , X-Mixup achieves the smallest cross-lingual transfer gap across different tasks (except for TyDiQA) without performance drop on the source language, which suggests X-Mixup achieves better cross-lingual transfer performance.
(Q2)
How X-Mixup inﬂuences the cross-lingual representation discrepancy?
Language centroid (Rosenberg & Hirschberg, 2007) is the mean of the representations within each language. We plot the language centroid of different methods (see Figure 4 in Appendix B), which indicates X-Mixup [Table]
brings closer language centroids signiﬁcantly.
We also calculate the CKA scores of the XNLI dataset (see Table 8 in Appendix B).
Results show X-Mixup reduces the cross-lingual representation discrepancy evenly across different target languages, improving the CKA score by 10.4% on average.
In conclusion, both the language centroids visualization and the CKA score improvement indicate
X-Mixup reduces the cross-lingual representation discrepancy effectively.
(Q3)
What is the essential component of X-Mixup?
There are ﬁve major components of X-Mixup: cross-lingual manifold mixup training, mixup inference, Mixup Ratio, Scheduled Sampling, and consistency loss.
To better understand X-Mixup, we implement ablation studies in Table 6.
Comparisons between X-Mixup and w/o mixup show the effectiveness of cross-lingual manifold mixup across different tasks, and even without mixup inference (translate-test data), the mixup training can also achieve 2.6% improvements on average.
Besides, comparisons between X-Mixup and λ = λ 0 show the effectiveness of introducing the translation quality modeling in the mixup process.
Scheduled sampling achieves more performance improvements on the classiﬁcation task, as the task shares labels across languages, and scheduled sampling can prevent the model from solely relying on the gold source sequence to make predictions.
In addition, the consistency loss is also more effective on the classiﬁcation task, because there is additional prediction consistency loss which can transfer the task capability from the source language to target languages.
Detailed ablation results on the consistency loss are shown in Table 9 in Appendix B, which shows the KL consistency loss contributes more than the MSE consistency loss on the classiﬁcation task.
(Q4)
Which layer is the best to implement the manifold mixup?
We implement the cross-lingual manifold mixup in different layers (see Figure 5 in Appendix B for details) and ﬁnd different tasks prefer different mixup layers.
Although different tasks have their own preferences, no matter which layer we mix, the cross-lingual transfer performance can be improved, except for mixing within a higher layer on classiﬁcation tasks.
The drop in classiﬁcation task is mainly because the source and target sequences share the same task label.
Performing mixup in a higher layer may make the model rely on the source sequence and ignore the target sequence.
The structured prediction task is not sensitive to the mixup layer, mainly because this task relies on both the short and long dependence.
For QA tasks, the cross-lingual transfer performance shows a trend from rise to decline as the mixup layer increases.
The QA task needs higher-level understanding, but higher layers are more language-speciﬁc, where sequences from different languages have different gold answers.
C ONCLUSION
This paper focuses on enhancing the cross-lingual transfer in cross-lingual understanding tasks.
Considering the large cross-lingual transfer gap in recent works, this paper ﬁrst analyses related factors and ﬁnds such a performance gap is strongly associated with the cross-lingual representation discrepancy.
Then X-Mixup is proposed to alleviate the discrepancy, which gives compromised representations for target languages by implementing the manifold mixup between the source and target languages.
Empirical evaluations on XTREME verify the effectiveness of X-Mixup across different tasks and languages.
Besides, both the visualization and quantitative analyses show XMixup reduces the representation discrepancy effectively.
Findings on the relationship between the cross-lingual transfer performance and representation discrepancy shed light on a promising way to boost cross-lingual transfer for future research. Furthermore, we can also apply X-Mixup to the multilingual pre-training process by implementing the cross-lingual manifold mixup on parallel data. We leave it for future work.
A M ETHOD D ETAILS A. M ULTI - HEAD A TTENTION
In the multi-head attention layer, multiple attention heads are concatenated
MultiHead ( Q, K, V ) =
Concat
[Equation]
(9) and each head is the scaled dot-product attention
[Equation]
) where W O , W q , W k and W v are trainable parameters.
A. T RAINING O BJECTIVE
For classiﬁcation tasks (e.g. NLI), the task loss is the cross-entropy loss
[Equation]
12) where C is the size of the label set.
For structured prediction tasks (e.g. POS) and span extraction tasks (e.g. QA), the task loss is also the cross-entropy loss
[Equation]
where n is the sequence length.
B A NALYSIS R ESULTS
[Figure]
[Table] [Figure] [Table] af ar bg de el en es et eu fa fi
fr he hi hu id
it ja kk ko mr nl pt ru ta te th tl tr ur vi zh (a) w/o X-Mixup 0.4 0.2 0.0 0.2 0.4 0.6 0.
af ar
bg deel en es eteu fa fi
fr he hi huidit ja kk ko mr nlptruta te th tl tr ur vi zh (b) X-Mixup
[Figure]
[Table]
C E XPERIMENTAL D ETAILS C. H YPER - PARAMETERS
For all tasks, we ﬁne-tune on 8 Nvidia V100-32GB GPU cards with the batch size 64.
For XQuAD and MLQA, we ﬁnetune 2 epochs.
For other tasks, we ﬁnetune 4 epochs.
There is no dev set in
XQuAD, so we use the dev set of MLQA for the model selection.
Table 10 shows hyper-parameters used for X-Mixup.
[Table]
C. D ETAILED R ESULTS
Detailed results of each tasks and languages are shown below.
Results of mBERT, XLM, MMTE and XLM-R are from XTREME (Hu et al., 2020).
Results of Filter is the best results of Fang et al.
[Table] [Table]
[Table] [Table]
[Table]
el es hi ru th tr vi
[Table]
[Table] [Table]