L EARNING I NVARIANT R EPRESENTATIONS ON M ULTILINGUAL L ANGUAGE M ODELS FOR U NSUPERVISED C ROSS -L INGUAL T RANSFER
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts, evidenced by their decent zero-shot transfer performance.
While studies have attributed this success to cross-lingually shared representations, quantitative analyses are sparse.
Towards a better understanding of the role of multilingual representations, in this work, we ﬁrst make the following observations through empirical analysis: (1) invariance of the feature representations strongly correlates with transfer performance, and (2) distributional shift in class priors between data in the source and target languages negatively affects performance—an issue that is largely overlooked in prior work.
Based on our ﬁndings, we propose an unsupervised cross-lingual learning method, called importance-weighted domain alignment (IWDA), that performs representation alignment, prior shift estimation, and correction.
Experiment results demonstrate its superiority under large prior shifts.
In addition, our method delivers further performance gains when combined with existing semi-supervised learning techniques.
I NTRODUCTION
Many recent state-of-the-art results in natural language processing (NLP) have been achieved on Transformer-based deep neural language models (LMs) using the “pre-train then ﬁne-tune” paradigm (Devlin et al., 2019; Conneau & Lample, 2019).
Multilingual versions of these LMs are pre-trained on unannotated and unparallel texts from more than one language, such as multilingual
BERT (mBERT) and XLM-R (Conneau et al., 2020a), and it is found that the learned representations are shared across languages.
In particular, the cross-lingual knowledge acquired by these models has enabled zero-shot cross-lingual transfer, where after ﬁne-tuned on data in a source language, not only do they work well when evaluated on the source, but also quite decently for the same task on almost all languages they have seen during pre-training (Wu & Dredze, 2019).
Their success on zero-shot cross-lingual transfer has prompted numerous studies on their multilingual abilities.
Among the many explanations, a recent hypothesis suggests that the deep representation combined with parameter sharing induced intermediate feature representations that are shared across languages (K et al., 2019; Conneau et al., 2020b; Muller et al., 2021).
Following this, we begin by studying the following question for mutilingual neural LMs:
What is the role of shared representations in unsupervised cross-lingual learning?
In an effort to answer this question in a quantitative manner, through empirical analysis on mBERT, we ﬁnd that (1) its cross-lingual transfer performance strongly correlates with the alignment of feature representations for the NLP task at hand between the source and target languages (Section 2.1).
In addition, (2) distributional shift in class priors between data in the source and target languages negatively affects transfer performance (sections 2.2 and 4.2).
Despite being a common issue with real-world data under unsupervised settings, it has been largely overlooked in prior work.
Based on these ﬁndings on the cross-lingual transfer of mutilingual neural LMs, towards a principled approach for the unsupervised setting where access to unlabeled data in the target language for the downstream task at hand is available in addition to labeled ones in the source (hereafter referred to as UCL 1 ), we propose importance-weighted domain alignment (IWDA). This method performs representation alignment, class prior shift estimation, and correction (Section 3).
Experiment results show that IWDA outperforms the more popular semi-supervised learning methods under large prior shifts, and can be additionally combined with them for further performance gains (Section 4).
1. R ELATED W ORK
Our proposed method is heavily inspired by Tachet des Combes et al.
(2020), which studied the issues with alignment-based domain adaptation (DA) under the presence of class prior shifts, and proposed the inclusion of prior shifts estimation and correction procedures in DA.
Our method also shares similar components with the DA-based UCL method by Keung et al.
(2019), while highlighting its limitations discovered in the present work (Section 2.2).
Analyzing Multilingual Neural LMs. Most studies analyze mBERT’s cross-lingual ability via probing experiments, and identiﬁed inﬂuencing factors including but not limited to linguistic similarity, size of pretraining corpus, domain similarity, parameter sharing and model depth (Pires et al., 2019; K et al., 2019; Lauscher et al., 2020; Conneau et al., 2020b; Dufter & Sch¨utze, 2020). Another line of research analyze intermediate representations and observed that mBERT learns crosslingually shared representations (Wu & Dredze, 2019; Conneau et al., 2020b; Muller et al., 2021). Cross-Lingual Representation Alignment.
The idea of learning invariant representations has powered a long history of approaches for cross-lingual transfer on neural models.
Under zero-shot settings, it is achieved by aligning word embeddings on bilingual dictionaries that could be induced via adversarial training (Mikolov et al., 2013; Smith et al., 2017; Zhang et al., 2017; Artetxe & Schwenk, 2019; Cao et al., 2019), a method that has powered many machine translation techniques (Artetxe et al., 2018; Lample et al., 2018; Arivazhagan et al., 2019; Pham et al., 2019; Hu et al., 2021)
It is found on static embedding models that the goodness of alignment affects transfer performance (Patra et al., 2019; Dubossarsky et al., 2020).
On deep models such as mBERT, it could also take the form of shared feature representations in the Transformer body (Artetxe et al., 2020b; Pfeiffer et al., 2020).
Under UCL and supervised settings, techniques based on representation alignment were proposed for models ranging from distributed word embeddings (Joty et al., 2017; Chen et al., 2018), RNN (Kim et al., 2017; Huang et al., 2019), to mBERT (Keung et al., 2019; Zheng &
Lapata, 2019; Xia et al., 2021).
For a survey, see Ruder et al. and Ruder (2019).
Connection to Domain Adaptation.
Domain adaptation is a popular framework for handling domain shifts between training and test data, e.g. learning a POS tagger on news articles but to be deployed on biomedical papers with a signiﬁcantly different word distribution (Ben-David et al., 2007). The concept of domain is extended naturally to cross-lingual learning by treating the distributions of sentence and document pertaining to the same task in each language as separate domains(Ruder, 2019).
Thereby, adaptation techniques proposed for domain shift can be applied to UCL.
Besides methods aiming at extracting shared representations (Li et al., 2020; Vernikos et al., 2020), another family of successful approaches is semi-supervised learning, which include self-training (Dong & de Melo, 2019), tri-training (Ruder & Plank, 2018), knowledge distillation (Wu et al., 2020), and data augmentation (Wang et al., 2018; Maharana & Bansal, 2020; Bari et al., 2021).
F ACTORS A FFECTING C ROSS -L INGUAL T RANSFER
To develop and provide evidence to the hypothesis that attributes the cross-lingual transfer success of multilingual neural LMs to shared representations, as well as to understand its implications on the design of unsupervised transfer algorithms, we attempt to quantify the relation between representation invariance and transfer performance in this section.
We brieﬂy summarize the “pre-train then ﬁne-tune” training paradigm and introduce notation along the way. Let g : X → Z denote the feature mapping provided by the pre-trained multilingual LM, and z = g ( x ) ∈ Z ⊆ R d the feature vector computed on an input sentence or token x ∈ X . On
BERT-Base models, d = 768 . In practice as well as in our studies and experiments, the feature is the last-layer wordpiece embeddings for token classiﬁcation tasks, and the embedding of a special start-of-sentence marker (symbolized as [CLS] ) for sequence classiﬁcation tasks. Let h : Z → Y denote the output layer(s) added on top of g with an appropriate activation, usually just a single linear layer with Softmax activation for classiﬁcation tasks in practice.
Hence, an end-to-end model is obtained, which computes predictions via ˆ y =
h ( g ( x )) .
By ﬁne-tuning both g, h on labeled task data ( x, y ) sampled from the source language domain p S only, we get a zero-shot transfer model for the downstream task at hand,
where the term zero-shot points to the fact that the model does not use any task data from the target language ( x (cid:48) , y (cid:48) ) ∼ p T during training.
Based on the following decomposition of the joint feature-label distribution into the product of class-conditional feature distribution and marginal prior distribution,
[Equation]
we examine how cross-lingual transfer performance is inﬂuenced by (1) invariance of the feature representation of data between the source and target languages, with perfect invariance being p S ( z | y = j ) =
p T ( z | y = j ) for all j ∈ Y , and (2) the distributional shift in the class priors, with no shift being p S ( y = j ) =
p T ( y = j ) for all j ∈ Y .
On classiﬁcation tasks, Y = { 1 , 2 , · · · , k } is discrete.
For legibility in later sections, we use the shorthands to denote the class-conditional feature distribution and the marginal prior distribution: p Z | Y and p Y . Setup.
Our empirical analysis is performed on the representations learned by mBERT zero-shot ﬁne-tuned on English for two multilingual tasks: sentiment analysis on the Multilingual Amazon Reviews Corpus (MARC), and named-entity recognition (NER) on the WikiANN dataset (see descriptions in Section 4). To simulate the effects of varying amount of representation misalignment and class prior difference, evaluation is performed on 500 smaller datasets randomly subsampled from the original MARC according to different class priors, and roughly 700 datasets from WikiANN.
MARC covers ﬁve high-resource languages (excluding English), and (our selected subset of) WikiANN covers 39 languages of distinct linguistic properties and varying resourcefulness.
2. I NVARIANCE OF F EATURE R EPRESENTATIONS
For studying the effects of representation invariance, we compare model performance to the discrepancy between the class-conditional feature distributions on the source and target domains, p Z | Y S and p Z | Y T . We take the average to aggregate the discrepancies computed when conditioned on each class,k (cid:80) kj =1 D ( p Z | Y = j S , p Z | Y =
j T ) , and refer to this quantity by conditional feature shift .
The discrepancy measure we use computes the (cid:96) 1 -distance between the mean feature vectors 3 ,
[Equation]
Measuring the distance between the means of representations for the purpose of analyzing mBERT has appeared in prior work.
A similar quantity is referred to as language centroids by Libovick´y et al.
The important distinction here is that we explicitly remove the effects of class prior shift by conditioning on the class labels, which we argue is an artifact of the dataset since arbitrary amounts of unconditioned feature shift can be generated by altering the class priors.
First, we consider zero-shot transfer.
In Fig.
1a, we plot the F 1 score 4 attained by the zero-shot model against conditional feature shift between English features and those of the target language, and observe that better representation alignment is indicative of better transfer performance.
Findings in prior work on the cross-lingual ability of mBERT are reﬂected.
For instance, linguistic similarity, a factor commonly argued to be important for cross-lingual transfer, could be related to the alignment
[Figure]
i
[Figure]
n a l f e a t u r e s h
i f t
[Figure]
of features: Spanish, French and German
are more linguistically similar to English than Japanese and Chinese, whose features are less aligned with those of English.
These results also reﬂects the successes of zero-shot learning approaches on mBERT that aim to achieve representation invariance
While the above ﬁnding shows that representation alignment is desirable for zero-shot transfer, is it also suitable for UCL settings where task data from the target language are available?
We start by recognizing that the objective function of most unsupervised transfer algorithms consists of two parts: loss on labeled source data, and terms involving unlabeled data that in theory are linked to the loss on target domain.
Based on this, we compared against conditional feature shift the performance of the model ﬁne-tuned supervisedly on labeled data from source and target languages, as it would be the ideal objective for unsupervised algorithms.
The results are plotted in Fig.
1b.
Instead of observing increases in conditional feature shifts, which would have implied that language-speciﬁc features are preferred by over shared ones for good performance on each language, we observe much-improved representation alignments between English and each of the target languages.
This means that achieving representation invariance is also appropriate under UCL settings.
To demonstrate that the correlation between last-layer class-conditional alignment and performance is not an artifact of the linear classiﬁcation head, we show that model performance is also correlated with (1) conditional feature shift in intermediate layers, plotted on the bottom panels of Fig. 1, and (2) class-balanced feature shift, plotted in Fig.
8 in the appendix, where the removal of prior shift effects is achieved by reweighting the class priors to be uniform (eq. 1).
Furthermore, we include results on the XNLI dataset, a higher-level semantic task (Fig. 6), and results on MARC with the more powerful XLM-R Large model (Fig. 7).
2. D ISTRIBUTIONAL S HIFT IN C LASS P RIORS
Most benchmarks for evaluating cross-lingual transfer are constructed so that the class prior distributions between data in the source and target languages are the same.
This is desired for assessing zero-shot learning approaches, as the focus should not be on “‘tricks’ for how to best handle the class imbalance” (Schwenk & Li, 2018).
However, for UCL settings with unlabeled target data, we argue that making the best use of the additional data involves the detection and correction of prior shift. Otherwise, as we show in this and later sections, negative transfer could occur. [Figure] [Figure]
n
[Figure]
i t i o n a l f
[Figure]
a t u r e s h
i f t ( l a s t l a y e r )
[Figure]
F 1 s
[Figure]
r e Stronger correlations of each factor with performance than those in ﬁgs. 1 and 2 are revealed. To examine the effects of class prior shift on cross-lingual transfer, we compare zero-shot transfer performance to the difference in prior distributions, D ( p YS , p YT ) , equivalent to total variation (TV).
The results are presented in Fig.
We also plot the effects of representation alignment and prior shift jointly in Fig. 3, where stronger correlations are revealed: (1) transfer performance generally takes a hit when prior shifts are large, regardless of how well the features are aligned, and (2) on target domains of the same amount of prior shift, performance decreases when features are less aligned.
The ﬁndings suggest that in addition to achieving representation invariance, good transfer performance also requires the detection and handling of potential prior shifts. While this is not possible under zero-shot settings without any data from the target domain, the unlabeled target task data provided under UCL settings could be leveraged toward this goal.
We examined existing approaches proposed for multilingual neural LMs for the UCL settings, but did not ﬁnd procedures for prior shift correction, and moreover, there is a lack of attention to the potential issues and effects of class prior shift to the algorithms. Speciﬁcally, for alignment-based methods, Zhao et al.
and Tachet des Combes et al.
(2020) showed that adaptation could lead to negative transfer when prior shifts are not accounted for due to erroneous alignment. For methods based on semi-supervised learning, negative transfer is observed empirically under large shifts in our studies (Section 4.2). These cases of negative transfer call for more considerations to the effects of class prior shift in future work on cross-lingual transfer that leverage target data.
Interestingly, although the alignment-based method proposed by Keung et al.
(2019) did not account for prior shift, it avoided negative transfer on the CoNLL NER task (which exhibits shifts) by performing feature alignment on the [CLS] sentence embeddings instead of the word embeddings (recall that predictions are made on word embeddings on token classiﬁcation tasks).
Since sentence embeddings on Transformer-based models could be loosely related to a weighted sum of the word embeddings, the procedure of Keung et al. (2019) is performing coarse-grained representation alignment, which avoids negative transfer while providing decent performance gain.
However, as we compare in Table 1, this misses out opportunities for larger gains from word embedding alignment. If alignment were performed on word embeddings without prior shift correction, deteriorating performance would have been observed (Fig. 10).
I MPORTANCE -W EIGHTED D OMAIN A LIGNMENT
For unsupervised cross-lingual learning on multilingual neural LMs with access to unlabeled target language data (which we refer to as the UCL setting), the ﬁndings in Section 2 suggest that good performance can be achieved via aligning representations and correcting class prior shifts.
Based on these principles, we propose class-importance-weighted domain alignment (IWDA) for UCL, following closely the work by Tachet des Combes et al.
In this section, we provide a highlevel description of the algorithm and its connection to the ﬁndings above, and refer interested readers to the aforementioned work and Appendix C for further implementation details.
We have seen that class-conditional feature alignment is desired for cross-lingual transfer, however, this is not possible to achieve under the UCL setting since the target labels are unknown.
Assume for now that the target class prior distribution is known, then we can compute the true ratio of the class priors, w ∗ j := p T ( y = j ) /p S ( y = j ) , also known as the importance weights (IWs), and aim for a weaker importance-weighted feature alignment, p w ∗ ,Z S = p ZT , where
[Equation]
Besides as shown in Fig.
8 that importance-weighted feature alignment also correlates with transfer performance, it is also a necessary condition for the ideal goal of class-conditional alignment (Tachet des Combes et al., 2020). Since the true w ∗ are unknown under UCL, they are estimated by matching ﬁrst-moment statistics using the confusion matrix (Saerens et al., 2002; Lipton et al., 2018).
Assume now that the features are conditionally aligned, p Z | Y =
j S =
p Z | Y
[Equation]
j T for all j ∈ Y (known as the label shift assumption), and the label space is discrete, Y = { 1 , · · · , k } .
Then for each class i ∈ [ k ] , we have the following relation between predicted target prior distribution, the source confusion matrix, and w ∗ :
[Equation]
where ˆ y = h ( g ( x )) are predicted labels. Since we have the same number of equations as unknowns, w ∗ could be recovered under mild assumptions (Lipton et al., 2018).
Careful readers may have noticed that achieving either goal above requires having the other as an assumption. Indeed, attempting to estimate IW without conditional alignment could fail, as well as alignment without accurate IWs. The hope is that the optimization will be predominantly guided by the implicit bias of the pre-trained multilingual LM for selecting largely invariant features (hence conditionally aligned), so that small improvements to feature alignment and updates to prior shift estimates could bring incremental improvements to both components. Optimization.
The two goals above deﬁne an optimization problem over g, h and w ,
[Equation]
where alignment objective minimizes source task loss and the discrepancy of w -importanceweighted feature distributions (the metric should be more ﬁne-grained than the ﬁrst-moment metric used in our analysis), and the IW estimation objective solves eq. 2 using current statistics based on h, g with the constraints w ≥ 0 and w (cid:62) p YS = 1 .
In our implementation, we use the minimax formulation of the alignment objective optimized with adversarial training, where D is replaced by a shallow ReLU network Wasserstein-1 critic and zero-centered gradient penalty (Arjovsky et al., 2017; Thanh-Tung et al., 2019).
We found the Wasserstein-1 adversarial loss to be more responsive to importance weighting than cross-entropy loss (Ganin et al., 2016). Improving Stability of IW Estimates.
In Tachet des Combes et al.
(2020), the IWs estimation objective is solved by quadratic programming (QP), and updated once every epoch of training in their implementation. Since accurate IWs estimates are integral to successful adaptation, we (1) update w using projected gradient descent at every training step so to maintain their optimality.
This, however, led to the discovery of previously unexplored practical issues with the IW estimator. In our preliminary experiments, we observed that w tends overestimate the majority class on the target domain, and diverges during longer training episodes. These issues are unique to unsupervised adaptation settings, and are due to inefﬁciencies of the critic and noises introduced by the alignment objective. They could also be reproduced with the previous
QP procedure.
We address these issues by (2) (cid:96) 2 -regularizing w (Azizzadenesheli et al., 2018), and (3) using a decaying learning rate schedule for w .
These modiﬁcations are necessary for achieving best performance, as long training episodes are required for ﬁne-tuning on many NLP tasks, and also needed to maintain the critic near optimality for guiding properly the alignment.
[Table]
[Table]
E XPERIMENTS
In our experiments, we compare importance-weighted domain alignment (IWDA) to two baseline methods for cross-lingual transfer on three NLP tasks, under the unsupervised setting where access to unlabeled task data from the target language is provided (UCL setting).
Besides supporting our ﬁndings and our motivated algorithm, these experiments are aimed at investigating the limitations of existing approaches under distributional shifts in prior distributions between source and target. We also illustrate the potential beneﬁts of IWDA in light of stronger cross-lingual learning methods.
Datasets.
The tasks considered are named-entity recognition (NER), sentiment analysis, and textual entailment.
NER: Two benchmark multilingual datasets are used, CoNLL (Tjong Kim Sang, 2002; Tjong Kim Sang & De Meulder, 2003) and WikiANN (Pan et al., 2017; Rahimi et al., 2019).
The former includes English, German, Spanish and Dutch, and the latter covers over 200 languages, from which we select a subset of 40 according to the XTREME benchmark (Hu et al., 2020).
Sentiment analysis: Multilingual Amazon Reviews Corpus (MARC) is used (Keung et al., 2020), a balanced dataset with product reviews in English, German, Spanish, French, Japanese and Chinese.
Textual entailment: The XNLI dataset is used (Conneau et al., 2018), but due to space constraints, its setup and results are relegated to Appendix B.
As in most work on cross-lingual transfer, English is used as the source language for transfer in our evaluations.
Baselines.
We perform extensive comparative experiments with two semi-supervised learning (SSL) techniques.
Both require having a model ﬁne-tuned on source data only.
Knowledge distillation (KD): It distills and transfers knowledge of the source teacher model to the target domain by training a newly initialized student network to mimic teacher outputs on target data (Hinton et al., 2015).
A variant of KD was proposed for NER by Wu et al.
(2020), which also freezes the bottom three layers of mBERT during training (Wu & Dredze, 2019). Self-training (ST): It iteratively assigns pseudolabels to unlabeled target data and train on the most conﬁdent ones.
Our implementation follows closely the work by Dong & de Melo (2019), which considered sequence classiﬁcation. For IWDA experiments, the model is ﬁrst ﬁne-tuned on source only, then continue ﬁne-tuned with the IWDA objective on both source and unlabeled target data. We also include results for when the true IWs are known and hard-coded as an upper bound on IWDA performance ( IWDA (oracle) ).
Experiments are performed on BERT-Base Multilingual Cased and XLM-R Large models. More details on IWDA and our baselines are included in Appendices C and D along with hyperparameter settings. Unless otherwise noted, results are from our implementation. [Table]
4. R ESULTS ON S TANDARD B ENCHMARKS
To begin, through evaluations on standard benchmarks
[Table]
we demonstrate that IWDA brings performance gains via improving representations to be better-aligned across languages.
Results on CoNLL and low-resource WikiANN NER are presented in Tables 1 and [Table] and those on MARC in Table 3.
Remaining results on WikiANN with mBERT are included in Table 6 in the appendix.
In all mBERT IWDA experiments, improvements upon zero-shot baseline are observed, and are accompanied by improved representation alignment. The average decrease in conditional feature shift is 21.60% and 22.33% on CoNLL with IWDA and IWDA (oracle) respectively (with 100% indicating perfect alignment), 51.53% and 64.77% on MARC, and 32.91% and 36.34% on WikiANN.
However, improvements sometimes fall short of SSL baselines, which we suspect is due to the noise introduced during the alignment process. Potential sources include ﬂuctuations in IW estimates and insufﬁciently trained critics.
By training IWDA with more passes on the data and improving the optimality of the critic, further improvements on CoNLL NER are observed (Table 5).
Another way to reduce noise is to perform a ﬁnal-stage SSL training on the IWDA-aligned model.
These experiments are labeled IWDA + KD and IWDA + ST , on which we observe some best results with mBERT on both tasks.
This is an appealing strategy, because SSL algorithms theoretically depend on the assumption that the source and target domains are the same, IWDA training can be viewed as a preprocessing step on the feature representations to satisfy this assumption.
We will revisit this point in the next section where we consider cross-lingual transfer under prior shifts. On XLM-R models, IWDA also brings improvements on NER. However for MARC, drops in performance are observed.
Here, we remark that the performance gain from IWDA (if any) will come from enhancements to the alignment of last-layer features in their current conﬁgurations, but not from learning entirely different representations with fundamentally better class-conditional alignment.
This cannot be achieved absent target labels, additional supervision or resources.
Hence, if the representations learned by the powerful XLM-R model during zero-shot training are already efﬁcient, then the potentials of IWDA will be limited. Opportunities for further improvement in this case could come from additional supervision, e.g. Bari et al. (2021) achieve signiﬁcant improvement via data augmentation and tri-training by leveraging the multilingual mask-ﬁlling ability of XLM-R.
4. R ESULTS U NDER C LASS P RIOR S HIFTS
In this section, we evaluate on datasets with large prior shifts, a likely property of real-world data that is insufﬁciently investigated for UCL, and contrast the brittleness of SSL baselines with the robustness of IWDA.
Because of the lack of multilingual benchmarks exhibiting prior shifts, similar to Section 2.2 , we evaluate on smaller datasets subsampled from the German portion of CoNLL and the Japanese portion of MARC according to varying amounts of shifts. Results plotting percent improvements over zero-shot baselines against the amount of prior shift are presented in Fig.
Although SSL baselines generally perform better than IWDA under mild to no shifts, their performances quickly deteriorate as prior shift increases, until eventually resulting in
[Figure]
negative transfer.
IWDA on the other hand does not cause negative transfer from prior shifts owing to its estimation and correction capabilities.
The relative success of IWDA demonstrates that it is possible to estimate prior shift from unlabeled target data.
We also include results on MARC with
XLM-R in Fig.
In mBERT experiments, the average percent correction of prior shift, computed from the decrease in TV between true target domain priors and end-of-training estimates, normalized by the amounts of shift, is 87.33% on CoNLL (with 100% indicating perfect IW estimation), and 28.08% on MARC under shifts larger than 0.8 in TV.
On MARC with smaller shifts, we observe estimates that are worse than assuming a uniform prior distribution (-335.80% in correction). This is because these are harder problems with noisy labels (as it is naturally difﬁcult to distinguish a 2-star review from a 3-star one), the suboptimality of the source model from which IWDA is trained means that the initial representations may be conditionally misaligned, thereby affecting the accuracy of the initial as well as subsequent IW estimates. Since IWDA brings improvements to class-conditional alignment, as already pointed out in the previous section, it is appealing to complement SSL and potentially more elaborated methods with IWDA to improve robustness to prior shift.
As demonstrated in our IWDA + KD and ST experiments, SSL no longer suffer from negative transfer when preceded by IWDA, and altogether provide further performance gains.
C ONCLUSION AND F UTURE W ORK
From the observations on multilingual neural LMs that cross-lingual transfer performance is strongly correlated with representation invariance and negatively affected by class prior shift, we proposed importance-weighted domain alignment (IWDA) for unsupervised cross-lingual learning.
Experiment results support our ﬁndings, and show that IWDA is effective at the estimation and correction of class prior shift from unlabeled data, and robust to large shifts, under which in contrast existing UCL approaches based on SSL could result in negative transfer.
As prior shift is a prevalent issue under unsupervised settings, more care to it is necessary in the design of future UCL algorithms for them to achieve good performance not only on curated benchmarks, but also on real-world data.
While our present implementation of IW estimation with the moment-matching approach is largely effective, future work on the search and study of potentially more stable estimation methods would enable IWDA to achieve better and more consistent transfer performance.
Since the strength of
IWDA lies in its ability to detect prior shifts and improve on class-conditional representation alignment, with our experiments on combining IWDA and SSL baselines as a preliminary demonstration of its potential usefulness as a feature preprocessing step and in improving robustness to prior shifts, it would be of interest to investigate its beneﬁts on stronger models and methods towards better cross-lingual transfer.
A A DDITIONAL A NALYSES AND F INDINGS FROM S ECTION
In this section, we provide descriptions to additional results for the analysis in Section 2, and additional ﬁndings for interested readers.
Similar to the plots included in Section 2, by plotting zero-shot transfer performance against representation alignment and prior shift on 39 target languages from the WikiANN dataset in Fig.
5, we aim to demonstrate that the ﬁndings on the relation are consistent regardless of linguistic properties or pre-training resourcefulness.
[Figure]
[Figure]
(cf. class-conditional alignment in Fig. 1).
These are intended to illustrate that the correlation between performance and feature alignment is not an artifact of having a linear predictor on last-layer features.
In Fig. 6, performance is compared against representation alignment on the XNLI textual entailment dataset, which is a high-level task.
As the same conclusions are supported by these results, this shows that the ﬁndings are also consistent across tasks.
Finally, in Fig. 7, performance on MARC is compared against representation alignment and prior shift using XLM-R as the transfer model.
The result shows that the ﬁndings are furthermore consistent across models.
A. E VALUATING M ODELS T RAINED ON S KEWED S OURCE D ATA
In Section 2, the relation between feature invariance, class prior shift and zero-shot transfer performance is studied when transferring from a relatively balanced source dataset.
Here, we show that when the model is ﬁne-tuned on source datasets with skewed prior distributions, the effects of class prior shift are more pronounced.
For this study, on MARC, we undersample from the English portion a source dataset according to the prior distribution of 0.1486, 0.0643, 0.0803, 0.1928 and 0.5141, reﬂective of the distribution [Figure] C o n d i t i o n a l f e a t u r e s h
i f t ( l a s t l a y e r ) [Figure] of product ratings on Amazon.com. We produce the same plot of Fig.
3 for a mBERT model ﬁnetuned on this skewed dataset in Fig.
9a.
Compared to when the source is class-balanced, here, the correlation between performance and conditional feature alignment has weakened, and that with class prior shift has strengthened.
On WikiANN, potentially due to NER being a structured prediction problem where there exist dependencies between tokens in the same sentence, although the original English portion of the dataset is considerably skewed (with 50.75% of the tokens belonging to class O , i.e. not part of an entity mention), there appears to be little to no correlation between class prior shift and model performance (Fig. 5b); not until we evaluated an mBERT that was zero-shot ﬁne-tuned on undersampled English
WikiANN where only sequences containing tokens with label O are kept, so that 80.92% of the tokens belong to class O (Fig. 9b).
B A DDITIONAL E XPERIMENT R ESULTS FROM S ECTION
In this section, we provide a description of the setup and discuss the results of XNLI experiments.
Because XNLI does not provide human-generated but only machine-translated training data for languages other than English (Conneau et al., 2020a), similar to the experiment considered in Bari et al.
(2021), we partition the XNLI training split into portions of size 20,000 (equal to 5% of the original size), and randomly select disjoint partitions to use as source and target data.
To avoid domain shift between train and test due to translation artifacts (Artetxe et al., 2020a), our test set is also selected from the machine-translated portion of the dataset.
Note that due to our setting, our results are not comparable to those evaluated on the standard XNLI benchmark.
Instead, the goal here is to evaluate the beneﬁts of improving feature alignment for higher-level tasks.
The results are included in Table 4, and it is observed that IWDA only brings slight improvements.
This may also be due to the small transfer gap between zero-shot and supervised results under our setting, which means that representations are already largely aligned.
C I MPLEMENTATION D ETAILS OF IWDA
Feature Alignment.
In our implementations, we used Wasserstein-1 loss for the adversarial loss (cid:96) ad (Arjovsky et al., 2017), which we observed to be more responsive to importance-weighted class prior correction than cross-entropy loss, which corresponds to Jensen-Shannon divergence under optimality (Goodfellow et al., 2014; Ganin et al., 2016).
The Lipschitz requirement on the adver-
[Table]
[Table]
sarial function class F (cid:51) f is satisﬁed with the soft constraints provided by zero-centered gradient penalty, which penalizes the gradient of f evaluated on random points on the feature space between source and target inputs (Thanh-Tung et al., 2019).
(Pseudo-)Class conditioned feature alignment, proposed by Long et al.
(2018), is performed for added stability, under which the outer product between features z = g ( x ) and the output probability produced by the current classiﬁer h ( z ) , z ⊗ h ( z ) , is taken and fed to the adversary f .
Combining everything, the joint loss computed on a minibatch of source examples ( x i , y i ) ni
=1 and target examples ( x (cid:48)
j ) mj
=1 is given by the following (cf.
the minimax objective in Section 3)
[Equation]
where ˜ n = min( n, m ) and
˜ z i is a random point on the line connecting g ( x i ) and g ( x (cid:48) i ) ( λ da = 0 .
005 , λ gp = 10 in our experiments).
The adversary f is optimized to maximize this loss, Trans- [Figure] Wasserstein Because of increasingly erroneous alignment as result of prior shift, DA performance deteriorates as training progresses.
Performing (pseudo)classconditional feature alignment (CDAN) protects
DA from the effects of prior shift to some degree (Long et al., 2018), compared to DANN (Ganin et al., 2016).
[Figure]
formers feature mapping g is optimized to minimize the ﬁrst two terms, and the linear classiﬁer h is optimized to minimize the ﬁrst term only.
Prior Shift Estimation.
On the estimation of importance weights (IWs), for added stability, we use the regularized estimator inspired by (Azizzadenesheli et al., 2018) that penalizes the amount of estimated shift.
Here, for simplicity, we use (cid:96) 2 -squared penalty implemented as weight decay, (cid:80) ki
=1 ( w i − w 0 ) 2 .
By default, w 0 = 1 , which corresponds to no prior shift.
Given ﬁrst moment statistics of source confusion, C ij := p S (ˆ y
=
i, y = j ) , and target output distribution, ν i :=
p T (ˆ y = i ) , the IWs are estimated by minimizing the following objective w.r.t.
w ,
[Equation]
( λ iw = η iw in our experiments, where η iw is the learning rate for w ).
Because obtaining the ﬁrst moment statistics from the entire training set is expensive, they are updated with exponential decay on (unbiased) estimates ˆ C, ˆ ν obtained from minibatches in each forward-pass, C t
= (1 − η iw-stats ) C t − 1 + η iw-stats ˆ C , and similarly for ˆ ν ( η iw-stats = 0 .
005 in our experiments).
To minimize potential effects from the miscalibration of neural models (Guo et al., 2017; Alexandari et al., 2020), hard confusion matrix and target output distribution are used in the estimation of IWs (Lipton et al., 2018;
Garg et al., 2020).
For initialization, before the start of IWDA ﬁne-tuning, C 0 , ν 0 are warmed up with data from the training sets, and w is computed using C 0 , ν 0 with the quadratic program described in Tachet des
Combes et al.
Since w will be likely be incorrect as features are not yet aligned, we set w 0 =
(1 − λ iw-init )
w + λ iw-init 1 ( λ iw-init = 0 .
25 in our experiments, and linearly decays to 0 over the ﬁrst 10% of optimization steps).
Optimization.
AdamW optimizer is applied to all parameters, with learning rate of 5 × 10 − 4 for the adversary and IW, and 1 × 10 − 5 for mBERT.
Linear learning rate schedule is applied to mBERT and IW, and 10% warm up is applied to mBERT and λ da .
The batch size is set to 8.
No truncation on the input texts is performed (although sequence length is limited to 512 due to modeling constraints of mBERT).
For each minibatch of examples from the source and target domains, one gradient step is applied on updating h , w (followed by a projection to satisfy w ≥ 0 and w (cid:62)
p
YS = 1 ), and the ﬁrst moment statistics C, ν are updated.
When simultaneous adversary updates are used, one gradient step is also applied on g and f .
Otherwise, with alternating updates, it is applied on either g or f .
This set of hyperparameters was found to work well for IWDA on most datasets and languages evaluated in the present work.
For discussions on hyperparameter tuning for unsupervised crosslingual learning, we refer interested readers to Artetxe et al.
(2020c).
D D ESCRIPTION OF SSL B ASELINE A PPROACHES
Knowledge Distillation (Our KD).
In our implementation, we follow the original paper by Hinton et al.
(2015) more closely.
Where the KD method by Wu et al.
(2020) sets Softmax temperature to 1 and does not use examples from the source domain for distillation, we set the temperature toand use the source labeled data for distillation for both soft targets and hard targets with a relative weight of 0.1.
Same as Wu et al.
(2020), the losses on the soft targets are computed with MSE for numerical stability.
The learning rate is 1 × 10 − 4 , and the batch size is 32.
In addition, Wu et al.
(2020) observed that when distilling from a zero-shot ﬁne-tuned teacher mBERT, better performance is achieved when the teacher is also trained with bottom three layers frozen.
Therefore, for fair comparison, the zero-shot models on which our KD implementation is applied are also ﬁne-tuned with frozen bottom layers (Table 1).
We did not freeze any layers during the training of the models involved in IWDA experiments.
Self-Training (ST).
Our implementation follows Dong & de Melo (2019) closely, in which we ﬁrst obtain a zero-shot ﬁne-tuned model, and then continue to ﬁne-tune it on source and target data with high-conﬁdence pseudolabels.
Self-training is performed for 4 epochs.
In each epoch, on
MARC, we ﬁne-tune on 50000 examples randomly selected on the source domain, and 4000 examples from each class on the target domain by conﬁdence with their predicted labels.
For experiments in Section 4.2, the sample sizes are scaled down to 5000 and 400, as the size of undersampled MARC datasets is 20000.
The learning rate is 1 × 10 − 5 , and the batch size is 128.
[Table]