L EARNING I NVARIANT R EPRESENTATIONS ON M ULTILINGUAL L ANGUAGE M ODELS FOR U NSUPERVISED C ROSS -L INGUAL T RANSFER
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts, evidenced by their decent zero-shot transfer performance.
While studies have attributed this success to cross-lingually shared representations, quantitative analyses are sparse.
Towards a better understanding of the role of multilingual representations, in this work, we ﬁrst make the following observations through empirical analysis: (1) invariance of the feature representations strongly correlates with transfer performance, and (2) distributional shift in class priors between data in the source and target languages negatively affects performance—an issue that is largely overlooked in prior work.
Based on our ﬁndings, we propose an unsupervised cross-lingual learning method, called importance-weighted domain alignment (IWDA), that performs representation alignment, prior shift estimation, and correction.
Experiment results demonstrate its superiority under large prior shifts.
In addition, our method delivers further performance gains when combined with existing semi-supervised learning techniques.
I NTRODUCTION
Many recent state-of-the-art results in natural language processing (NLP) have been achieved on Transformer-based deep neural language models (LMs) using the “pre-train then ﬁne-tune” paradigm (Devlin et al., 2019; Conneau & Lample, 2019).
Multilingual versions of these LMs are pre-trained on unannotated and unparallel texts from more than one language, such as multilingual
BERT (mBERT) and XLM-R (Conneau et al., 2020a), and it is found that the learned representations are shared across languages.
In particular, the cross-lingual knowledge acquired by these models has enabled zero-shot cross-lingual transfer, where after ﬁne-tuned on data in a source language, not only do they work well when evaluated on the source, but also quite decently for the same task on almost all languages they have seen during pre-training (Wu & Dredze, 2019).
Their success on zero-shot cross-lingual transfer has prompted numerous studies on their multilingual abilities.
Among the many explanations, a recent hypothesis suggests that the deep representation combined with parameter sharing induced intermediate feature representations that are shared across languages (K et al., 2019; Conneau et al., 2020b; Muller et al., 2021).
Following this, we begin by studying the following question for mutilingual neural LMs:
What is the role of shared representations in unsupervised cross-lingual learning?
In an effort to answer this question in a quantitative manner, through empirical analysis on mBERT, we ﬁnd that (1) its cross-lingual transfer performance strongly correlates with the alignment of feature representations for the NLP task at hand between the source and target languages (Section 2.1).
In addition, (2) distributional shift in class priors between data in the source and target languages negatively affects transfer performance (sections 2.2 and 4.2).
Despite being a common issue with real-world data under unsupervised settings, it has been largely overlooked in prior work.
Based on these ﬁndings on the cross-lingual transfer of mutilingual neural LMs, towards a principled approach for the unsupervised setting where access to unlabeled data in the target language for the downstream task at hand is available in addition to labeled ones in the source (hereafter referred to as UCL 1 ), we propose importance-weighted domain alignment (IWDA).
This method performs representation alignment, class prior shift estimation, and correction (Section 3).
Experiment results show that IWDA outperforms the more popular semi-supervised learning methods under large prior shifts, and can be additionally combined with them for further performance gains (Section 4).
1. R ELATED W ORK
Our proposed method is heavily inspired by Tachet des Combes et al.
(2020), which studied the issues with alignment-based domain adaptation (DA) under the presence of class prior shifts, and proposed the inclusion of prior shifts estimation and correction procedures in DA.
Our method also shares similar components with the DA-based UCL method by Keung et al.
(2019), while highlighting its limitations discovered in the present work (Section 2.2).
Analyzing Multilingual Neural LMs.
Most studies analyze mBERT’s cross-lingual ability via probing experiments, and identiﬁed inﬂuencing factors including but not limited to linguistic similarity, size of pretraining corpus, domain similarity, parameter sharing and model depth (Pires et al., 2019; K et al., 2019; Lauscher et al., 2020; Conneau et al., 2020b; Dufter & Sch¨utze, 2020).
Another line of research analyze intermediate representations and observed that mBERT learns crosslingually shared representations (Wu & Dredze, 2019; Conneau et al., 2020b; Muller et al., 2021).
Cross-Lingual Representation Alignment.
The idea of learning invariant representations has powered a long history of approaches for cross-lingual transfer on neural models.
Under zero-shot settings, it is achieved by aligning word embeddings on bilingual dictionaries that could be induced via adversarial training (Mikolov et al., 2013; Smith et al., 2017; Zhang et al., 2017; Artetxe &
Schwenk, 2019; Cao et al., 2019), a method that has powered many machine translation techniques (Artetxe et al., 2018; Lample et al., 2018; Arivazhagan et al., 2019; Pham et al., 2019; Hu et al., 2021)
It is found on static embedding models that the goodness of alignment affects transfer performance (Patra et al., 2019; Dubossarsky et al., 2020).
On deep models such as mBERT, it could also take the form of shared feature representations in the Transformer body (Artetxe et al., 2020b;
Pfeiffer et al., 2020).
Under UCL and supervised settings, techniques based on representation alignment were proposed for models ranging from distributed word embeddings (Joty et al., 2017; Chen et al., 2018), RNN (Kim et al., 2017; Huang et al., 2019), to mBERT (Keung et al., 2019; Zheng &
Lapata, 2019; Xia et al., 2021).
For a survey, see Ruder et al.
and Ruder (2019).
Connection to Domain Adaptation.
Domain adaptation is a popular framework for handling domain shifts between training and test data, e.g. learning a POS tagger on news articles but to be deployed on biomedical papers with a signiﬁcantly different word distribution (Ben-David et al., 2007).
The concept of domain is extended naturally to cross-lingual learning by treating the distributions of sentence and document pertaining to the same task in each language as separate domains(Ruder, 2019).
Thereby, adaptation techniques proposed for domain shift can be applied to UCL.
Besides methods aiming at extracting shared representations (Li et al., 2020; Vernikos et al., 2020), another family of successful approaches is semi-supervised learning, which include self-training (Dong & de Melo, 2019), tri-training (Ruder & Plank, 2018), knowledge distillation (Wu et al., 2020), and data augmentation (Wang et al., 2018; Maharana & Bansal, 2020; Bari et al., 2021).
F ACTORS A FFECTING C ROSS -L INGUAL T RANSFER
To develop and provide evidence to the hypothesis that attributes the cross-lingual transfer success of multilingual neural LMs to shared representations, as well as to understand its implications on the design of unsupervised transfer algorithms, we attempt to quantify the relation between representation invariance and transfer performance in this section.
We brieﬂy summarize the “pre-train then ﬁne-tune” training paradigm and introduce notation along the way.
Let g : X → Z denote the feature mapping provided by the multilingual LM, and
z = g ( x ) ∈ Z ⊆ R d the feature vector computed on an input sentence or token x ∈ X .
In practice as well as in our studies and experiments, the feature is the last-layer wordpiece embeddings for token classiﬁcation tasks, and the embedding of a special start-of-sentence marker (symbolized as [CLS] ) for sequence classiﬁcation tasks. Hence, on BERT-Base models, d = 768 and Z ≈ S d − 1 sphere due to LayerNorm.
Let h : Z → Y denote the output layer(s) added on top of g with an appropriate activation, usually a single linear layer with Softmax for classiﬁcation tasks in practice.
Hence, an end-to-end model is obtained, which computes predictions via ˆ y =
h ( g ( x )) .
By ﬁne-tuning both g, h on labeled task data ( x, y ) sampled from the source language domain p S only, we get a zeroshot transfer model for the downstream task at hand,
where the term zero-shot points to the fact that the model does not use any task data from the target language ( x (cid:48) , y (cid:48) ) ∼ p T during training.
Based on the following decomposition of the joint feature-label distribution into the product of class-conditional feature distribution and marginal prior distribution,
[Equation]
we examine how cross-lingual transfer performance is inﬂuenced by (1) invariance of the feature representation of data between the source and target languages, with perfect invariance being p S ( z | Y = j ) =
p T ( z | Y = j ) for all j ∈ Y , and (2) the distributional shift in the class priors, with no shift being p S ( Y = j ) =
p T ( Y = j ) for all j ∈ Y .
On classiﬁcation tasks, Y = { 1 , 2 , · · · , k } is discrete.
For legibility in later sections, we use the shorthands to denote the class-conditional feature distribution and the marginal prior distribution: p Z | Y and p Y .
Setup.
Our empirical analysis is performed on the representations learned by mBERT zero-shot ﬁne-tuned on English for two multilingual tasks: sentiment analysis on the Multilingual Amazon
Reviews Corpus (MARC), and named-entity recognition (NER) on the WikiANN dataset (see descriptions in Section 4).
To simulate the effects of varying amount of representation misalignment and class prior difference, evaluation is performed on 500 smaller datasets randomly subsampled from the original MARC according to different class priors, and roughly 700 datasets from
WikiANN.
MARC covers ﬁve high-resource languages (excluding English), and (our selected subset of) WikiANN covers 39 languages of distinct linguistic properties and varying resourcefulness.
2. I NVARIANCE OF F EATURE R EPRESENTATIONS
For studying the effects of representation invariance, we compare model performance to the discrepancy between the class-conditional feature distributions on the source and target domains, p Z | Y S and p Z | Y T .
We take the average to aggregate the discrepancies computed when conditioned on each class,k (cid:80) kj =1 D ( p Z | Y =
j S , p Z | Y =
j T ) , and refer to this quantity by conditional feature shift .
The discrepancy measure we use computes the (cid:96) 1 -distance between the mean feature vectors 3 ,
[Equation]
Measuring the distance between the means of representations for the purpose of analyzing mBERT has appeared in prior work.
A similar quantity is referred to as language centroids by Libovick´y et al.
The important distinction here is that we explicitly remove the effects of class prior shift by conditioning on the class labels, which we argue is an artifact of the dataset since arbitrary amounts of unconditioned feature shift can be generated by altering the class priors.
First, we consider zero-shot transfer.
In Fig.
1a, we plot the F 1 score 4 attained by the zero-shot model against conditional feature shift between English features and those of the target language, and observe that better representation alignment is indicative of better transfer performance.
Findings in prior work on the cross-lingual ability of mBERT are reﬂected.
For instance, linguistic similarity, a factor commonly argued to be important for cross-lingual transfer, could be related to the alignment
[Figure]
i t i o n a l f e a t u r e
s h
i f t
[Figure]
of features: Spanish, French and German
are more linguistically similar to English than Japanese and Chinese, whose features are less aligned with those of English.
These results also reﬂects the successes of zero-shot learning approaches on mBERT that aim to achieve representation invariance.
While the above ﬁnding shows that representation alignment is desirable for zero-shot transfer, is it also suitable for UCL settings where task data from the target language are available?
We start by recognizing that the objective function of most unsupervised transfer algorithms consists of two parts: loss on labeled source data, and terms involving unlabeled data that in theory are linked to the loss on target domain.
Based on this, we compared against conditional feature shift the performance of the model ﬁne-tuned supervisedly on labeled data from source and target languages, as it would be the ideal objective for unsupervised algorithms.
The results are plotted in Fig.
1b.
Instead of observing increases in conditional feature shifts, which would have implied that language-speciﬁc features are preferred by over shared ones for good performance on each language, we observe much-improved representation alignments between English and each of the target languages.
This means that achieving representation invariance is also appropriate under UCL settings.
To demonstrate that the correlation between last-layer class-conditional alignment and performance is not an artifact of the linear classiﬁcation head, we show that model performance is also correlated with (1) conditional feature shift in intermediate layers, plotted on the bottom panels of Fig. 1, and (2) class-balanced feature shift, plotted in Fig.
5 in the appendix, where the removal of prior shift effects is achieved by reweighting the class priors to be uniform (eq. 1).
Furthermore, we include results on the XNLI dataset, a higher-level semantic task (Fig. 7), and results on MARC with the more powerful XLM-R Large model (Fig. 8).
2. D ISTRIBUTIONAL S HIFT IN C LASS P RIORS
Most benchmarks for evaluating cross-lingual transfer are constructed so that the class prior distributions between data in the source and target languages are the same.
This is desired for assessing zero-shot learning approaches, as the focus should not be on “‘tricks’ for how to best handle the class imbalance” (Schwenk & Li, 2018).
However, for UCL settings with unlabeled target data, we argue that making the best use of the additional data involves the detection and correction of prior shift.
Otherwise, as we show in this and later sections, negative transfer could occur.
[Figure]
[Figure]
e
Stronger correlations of each factor with performance than those in ﬁgs.
1 and 2 are revealed.
To examine the effects of class prior shift on cross-lingual transfer, we compare zero-shot transfer performance to the difference in prior distributions, D ( p YS , p YT ) , equivalent to total variation (TV).
The results are presented in Fig.
We also plot the effects of representation alignment and prior shift jointly in Fig.
3, where stronger correlations are revealed: (1) transfer performance generally takes a hit when prior shifts are large, regardless of how well the features are aligned, and (2) on target domains of the same amount of prior shift, performance decreases when features are less aligned.
The ﬁndings suggest that in addition to achieving representation invariance, good transfer performance also requires the detection and handling of potential prior shifts.
While this is not possible under zero-shot settings without any data from the target domain, the unlabeled target task data provided under UCL settings could be leveraged toward this goal.
We examined existing approaches proposed for multilingual neural LMs for the UCL settings, but did not ﬁnd procedures for prior shift correction, and moreover, there is a lack of attention to the potential issues and effects of class prior shift to the algorithms.
Speciﬁcally, for alignment-based methods, Zhao et al.
and Tachet des Combes et al.
(2020) showed that adaptation could lead to negative transfer when prior shifts are not accounted for due to erroneous alignment.
For methods based on semi-supervised learning, negative transfer is observed empirically under large shifts in our studies (Section 4.2).
These cases of negative transfer call for more considerations to the effects of class prior shift in future work on cross-lingual transfer that leverage target data.
Interestingly, although the alignment-based method proposed by Keung et al.
(2019) did not account for prior shift, it avoided negative transfer on the CoNLL NER task (which exhibits shifts) by performing feature alignment on the [CLS] sentence embeddings instead of the word embeddings (recall that predictions are made on word embeddings on token classiﬁcation tasks).
Since sentence embeddings on Transformer-based models could be loosely related to a weighted sum of the word embeddings, the procedure of Keung et al.
(2019) is performing coarse-grained representation alignment, which avoids negative transfer while providing decent performance gain.
However, as we compare in Table 1, this misses out opportunities for larger gains from word embedding alignment.
If alignment were performed on word embeddings without prior shift correction, deteriorating performance would have been observed (Fig. 11).
I MPORTANCE -W EIGHTED D OMAIN A LIGNMENT
For unsupervised cross-lingual learning on multilingual neural LMs with access to unlabeled target language data (which we refer to as the UCL setting), the ﬁndings in Section 2 suggest that good performance can be achieved via aligning representations and correcting class prior shifts.
Based on these principles, we propose class-importance-weighted domain alignment (IWDA) for UCL, following closely the work by Tachet des Combes et al.
In this section, we provide a highlevel description of the algorithm and its connection to the ﬁndings above, and refer interested readers to the aforementioned work and Appendix C for further implementation details.
We have seen that class-conditional feature alignment is desired for cross-lingual transfer, however, this is not possible to achieve under the UCL setting since the target labels are unknown.
Assume for now that the target class prior distribution is known, then we can compute the true ratio of the class priors, w ∗ j :=
p T ( Y = j ) /p S ( Y = j ) , also known as the importance weights (IWs), and aim for a weaker importance-weighted feature alignment, p w ∗ ,Z S = p ZT , where
[Equation]
Besides as shown in Fig.
5 that importance-weighted feature alignment also correlates with transfer performance, it is also a necessary condition for the ideal goal of class-conditional alignment (Tachet des Combes et al., 2020).
Since the true w ∗ are unknown under UCL, they are estimated by matching ﬁrst-moment statistics using the confusion matrix (Saerens et al., 2002; Lipton et al., 2018).
Assume now that the features are conditionally aligned, p Z | Y =
j S =
p Z | Y
[Equation]
j T for all j ∈ Y (known as the label shift assumption), and the label space is discrete, Y = { 1 , · · · , k } .
Then for each class i ∈ [ k ] , we have the following relation between predicted target prior distribution, the source confusion matrix, and w ∗ : p T [Equation] ˆ Y
[Equation]
i ) = [Equation] where ˆ Y := h ( g ( X )) are label predictions.
Since we have the same number of equations as unknowns, w ∗ could be recovered under mild assumptions (Lipton et al., 2018).
Careful readers may have noticed that achieving either goal above requires having the other as an assumption.
Indeed, attempting to estimate IW without conditional alignment could fail, as well as alignment without accurate IWs.
The hope is that the optimization will be predominantly guided by the implicit bias of the pre-trained multilingual LM for selecting largely invariant features (hence conditionally aligned), so that small improvements to feature alignment and updates to prior shift estimates could bring incremental improvements to both components.
Optimization.
The two goals above deﬁne an optimization problem over g, h and w ,
[Equation]
where alignment objective minimizes source task loss and the discrepancy of w -importanceweighted feature distributions (the metric should be more ﬁne-grained than the ﬁrst-moment metric used in our analysis), and the IW estimation objective solves eq.
2 using current statistics based on h, g with the constraints w ≥ 0 and w (cid:62) p
YS = 1 .
In our implementation, we use the minimax formulation of the alignment objective optimized with adversarial training, where D is replaced by a shallow ReLU network Wasserstein-1 critic and zero-centered gradient penalty (Arjovsky et al., 2017; Thanh-Tung et al., 2019).
We found the
Wasserstein-1 adversarial loss to be more responsive to importance weighting than cross-entropy loss (Ganin et al., 2016).
Improving Stability of IW Estimates.
In Tachet des Combes et al.
(2020), the IWs estimation objective is solved by quadratic programming (QP), and updated once every epoch of training in their implementation.
Since accurate IWs estimates are integral to successful adaptation, we (1) update w using projected gradient descent at every training step for maintaining their optimality.
This, however, led to the discovery of previously unexplored practical issues with the IW estimator.
In our preliminary experiments, we observed that w tends overestimate the majority class on the target domain, and diverges during longer training episodes.
These issues are unique to unsupervised adaptation settings, and are due to inefﬁciencies of the critic and noises introduced by the alignment objective.
They could also be reproduced with the previous
QP procedure.
We address these issues by (2) (cid:96) 2 -regularizing w (Azizzadenesheli et al., 2018), and (3) using a decaying learning rate schedule for w .
These modiﬁcations are necessary for achieving best performance, as long training episodes are required for ﬁne-tuning on many NLP tasks, and also needed to maintain the critic near optimality for guiding properly the alignment.
[Table] [Table]
E XPERIMENTS
In our experiments, we compare importance-weighted domain alignment (IWDA) to two baseline methods for cross-lingual transfer on three NLP tasks, under the unsupervised setting where access to unlabeled task data from the target language is provided (UCL setting).
Besides supporting our ﬁndings and our motivated algorithm, these experiments are aimed at investigating the limitations of existing approaches under distributional shifts in prior distributions between source and target.
We also illustrate the potential beneﬁts of IWDA in light of stronger cross-lingual learning methods.
Datasets.
The tasks considered are named-entity recognition (NER), sentiment analysis, and textual entailment.
NER:
Two benchmark multilingual datasets are used, CoNLL (Tjong Kim Sang, 2002;
Tjong Kim Sang & De Meulder, 2003) and WikiANN (Pan et al., 2017; Rahimi et al., 2019).
The former includes English, German, Spanish and Dutch, and the latter covers over 200 languages, from which we select a subset of 40 according to the XTREME benchmark (Hu et al., 2020).
Sentiment analysis: Multilingual Amazon Reviews Corpus (MARC) is used (Keung et al., 2020), a balanced dataset with product reviews in English, German, Spanish, French, Japanese and Chinese.
Textual entailment: The XNLI dataset is used (Conneau et al., 2018), but due to space constraints, its setup and results are relegated to Appendix B.
As in most work on cross-lingual transfer, English is used as the source language for transfer in our evaluations.
Baselines.
We perform extensive comparative experiments with two semi-supervised learning (SSL) techniques.
Both require having a model ﬁne-tuned on source data only.
Knowledge distillation (KD): It distills and transfers knowledge of the source teacher model to the target domain by training a newly initialized student network to mimic teacher outputs on target data (Hinton et al., 2015).
A variant of KD was proposed for NER by Wu et al.
(2020), which also freezes the bottom three layers of mBERT during training (Wu & Dredze, 2019).
Self-training (ST): It iteratively assigns pseudo-labels to unlabeled target data and train on the most conﬁdent ones.
Our implementation follows closely the work by Dong & de Melo (2019), which considered sequence classiﬁcation.
For IWDA experiments, the model is ﬁrst ﬁne-tuned on source only, then continue ﬁne-tuned with the IWDA objective on both source and unlabeled target data.
We also include results for when the true IWs are known and hard-coded as an upper bound on IWDA performance ( IWDA (oracle) ).
Experiments are performed on BERT-Base Multilingual Cased and XLM-R Large models.
More details on IWDA and our baselines are included in Appendices C and D along with hyperparameter settings.
Unless otherwise noted, results are from our implementation.
[Table]
4. R ESULTS ON S TANDARD B ENCHMARKS
To begin, through evaluations on standard benchmarks
[Table]
we demonstrate that IWDA brings performance gains via improving representations to be better-aligned across languages.
Results on CoNLL and low-resource WikiANN NER are presented in Tables 1 and
[Table]
and those on MARC in Table 3.
Remaining results on WikiANN with mBERT are included in Table 4 in the appendix.
In all mBERT IWDA experiments, improvements upon zero-shot baseline are observed, and are accompanied by improved representation alignment.
The average decrease in conditional feature shift is 21.60% and 22.33% on CoNLL with IWDA and IWDA (oracle) respectively (with 100% indicating perfect alignment), 51.53% and 64.77% on MARC, and 32.91% and 36.34% on WikiANN.
However, improvements sometimes fall short of SSL baselines, which we suspect is due to the noise introduced during the alignment process.
Potential sources include ﬂuctuations in IW estimates and insufﬁciently trained critics.
By training IWDA with more passes on the data and improving the optimality of the critic, further improvements on CoNLL NER are observed (Table 6).
Another way to reduce noise is to perform a ﬁnal-stage SSL training on the IWDA-aligned model.
These experiments are labeled IWDA + KD and IWDA + ST , on which we observe some best results with mBERT on both tasks.
This is an appealing strategy, because SSL algorithms theoretically depend on the assumption that the source and target domains are the same, IWDA training can be viewed as a preprocessing step on the feature representations to satisfy this assumption.
We will revisit this point in the next section where we consider cross-lingual transfer under prior shifts.
On XLM-R models, IWDA also brings improvements on NER.
However on MARC, gains are only observed for languages distant from English.
Here, we remark that the performance gain from IWDA (if any) will come from enhancements to the alignment of last-layer features in their current conﬁgurations, but not from learning entirely different representations with fundamentally better class-conditional alignment.
This cannot be achieved absent target labels, additional supervision or resources.
If the representations learned by the powerful XLM-R model during zero-shot training are already efﬁcient, then the potentials of IWDA will be limited.
Opportunities for further improvement in this case could come from additional resources, e.g. Bari et al.
(2021) perform unsupervised data augmentation by leveraging the multilingual mask-ﬁlling ability of XLM-R.
4. R ESULTS U NDER C LASS P RIOR S HIFTS
In this section, we evaluate on datasets with large prior shifts, a likely property of real-world data that is insufﬁciently investigated for UCL, and contrast the brittleness of SSL baselines with the robustness of IWDA.
Because of the lack of multilingual benchmarks exhibiting prior shifts, similar to Section 2.2 , we evaluate on smaller datasets subsampled from the German portion of CoNLL and the Japanese portion of MARC according to varying amounts of shifts.
Results plotting percent improvements over zero-shot baselines against the amount of prior shift are presented in Fig.
Although SSL baselines are generally better than IWDA under mild to no shifts,
[Figure]
their performances quickly deteriorate as prior shift increases, until eventually resulting in negative transfer.
IWDA on the other hand does not cause negative transfer from prior shifts owing to its estimation and correction capabilities.
The relative success of IWDA demonstrates that it is possible to estimate prior shift from unlabeled target data.
We include XLM-R MARC results in Fig. 10.
In mBERT experiments, the average percent correction of prior shift, computed from the decrease in
TV between true target domain priors and end-of-training estimates, normalized by the amounts of shift, is 87.33% on CoNLL (with 100% indicating perfect IW estimation), and 28.08% on MARC under shifts larger than 0.8 in TV.
On MARC with smaller shifts, we observe estimates that are worse than assuming a uniform prior distribution (-335.80% in correction).
This is because these are harder problems with noisy labels (as it is naturally difﬁcult to distinguish a 2-star review from a 3-star one), the suboptimality of the source model from which IWDA is trained means that the initial representations may be conditionally misaligned, thereby affecting the accuracy of the initial as well as subsequent IW estimates.
Since IWDA brings improvements to class-conditional alignment, as already pointed out in the previous section, it is appealing to complement SSL and potentially more elaborated methods with
IWDA to improve robustness against prior shift.
As demonstrated in our IWDA + KD and ST experiments, SSL no longer suffer from negative transfer when preceded by IWDA, and altogether provide further performance gains.
C ONCLUSION AND F UTURE W ORK
From the observations on multilingual neural LMs that cross-lingual transfer performance is strongly correlated with representation invariance and negatively affected by class prior shift, we proposed importance-weighted domain alignment (IWDA) for unsupervised cross-lingual learning.
Experiment results support our ﬁndings, and show that IWDA is effective at the estimation and correction of class prior shift from unlabeled data, and robust to large shifts, under which in contrast existing
UCL approaches based on SSL could result in negative transfer.
As prior shift is a prevalent issue under unsupervised settings, more care to it is necessary in the design of future UCL algorithms for them to achieve good performance not only on curated benchmarks, but also on real-world data.
While our present implementation of IW estimation with the moment-matching approach is largely effective, future work on the search and study of potentially more stable estimation methods would enable IWDA to achieve better and more consistent transfer performance.
Since the strength of
IWDA lies in its ability to detect prior shifts and improve on class-conditional representation alignment, with our experiments on combining IWDA and SSL baselines as a preliminary demonstration of its potential usefulness as a feature preprocessing step and in improving robustness to prior shifts, it would be of interest to investigate its beneﬁts on stronger models and methods towards better cross-lingual transfer.
A A DDITIONAL A NALYSES AND F INDINGS FROM S ECTION
In this section we provide additional results to support our ﬁndings by considering more languages (WikiANN NER), a higher-level task (XNLI) and a more powerful model (XLM-R Large)
First and foremost, to demonstrate that the correlation between class-conditional representation alignment and performance is not confounded by the clustering structure induced by the linear classiﬁer trained on last-layer features, we plot and observe similar correlation between transfer performance and class-balanced representation alignment in Fig. WikiANN. As an attempt to minimize the effects from potential confounders on the empirical ﬁndings in the main sections, we perform the same experiments for studying the relation between transfer performance, representation alignment and prior shift by zero-shot transferring mBERT from English to 39 target languages included in the WikiANN NER dataset. The results are provided in Fig.
These languages have distinct linguistic properties and varying degrees of resourcefulness. A summary of them can be found in Table 5 of Hu et al.
(2020), and sizes of their pretraining corpora in Table 1 of Wu & Dredze (2020). It is observed that the same correlation between representation alignment and performance still holds, but the effects of prior shift are mostly mild. We suspect that this is because the task we evaluated on, named-entity recognition, is a structured prediction problem, hence mBERT’s ability to leverage dependencies between words made it robust to prior shifts on these tasks. We later observed that if the source dataset on which the model is trained on has a skewed prior distribution, then the effects of prior shift will be pronounced.
The mBERT WikiANN results under this setting are shown in Fig.
9b, and more discussions are coming up in Section A.1. XNLI.
So far, we have seen empirical evidence on sentiment analysis and NER tasks, and one may wonder if the same conclusions still hold when considering cross-lingual transfer to high-level semantic tasks. For thoroughness, we include a plot in Fig. 7 comparing zero-shot transfer performance to representation alignment when mBERT is trained on a textual entailment task, from
English to 14 languages included in the XNLI dataset. Here, the same observations are made, demonstrating that the ﬁndings hold across NLP tasks. XLM-R. Another variable in our empirical analyses is the model used for evaluation. While the lightness of mBERT afforded us to perform extensive experiments, it would be of practical interest to see whether our ﬁndings also hold on larger and more powerful models.
In Fig. 8, we compared performance to alignment and prior shift on XLM-R Large model on MARC.
Again, the observations support our ﬁndings, and show that they also hold across models. [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] FROM S KEWED S OURCE P RIOR D ISTRIBUTIONS
As brieﬂy mentioned above, we have only considered the cross-lingual transfer from relatively balanced source datasets in previous sections.
Here, we show that when the model is ﬁne-tuned on source datasets with skewed prior distributions, the effects of class prior shift are more pronounced.
We ﬁrst experiment on MARC. We generate an skewed English source dataset via subsampling according to the prior distribution of 14.86%, 6.43%, 8.03%, 19.28% and 51.41% 6 , corresponding to the proportion of 1- through 5-star reviews.
Then, we repeat the previous empirical analysis by evaluating an mBERT ﬁne-tuned on this skewed English dataset, and the results are shown in Fig. 9a.
Compared to when the source is class-balanced (Fig. 3), here the correlation between performance and class prior shift is much stronger.
On WikiANN, although the original English portion of the dataset is already considerably skewed (with class O makes up 50.75% of the labels), the correlation between prior shift and model performance is weak (Fig. 6b), likely due to NER being a structured prediction problem where the dependencies between words in the same sentence helped the model to be more robust. Repeating the same experiment above with the skewed English source dataset generated by keeping only sentences containing tokens with an O label (so that now 80.92% of the tokens belong to class O ), we again observe pronounced effects of prior shift on performance (results in Fig. 9b). [Table]
[Table] [Figure] [Figure] We compare domain alignment techniques applied on last-layer word embeddings with importanceweight estimation (IWDA) and without (DA).
B A DDITIONAL E XPERIMENT R ESULTS FROM S ECTION
By comparing the transfer performance of IWDA to SSL on standard benchmarks as well as datasets exhibiting prior shifts in the main sections, we demonstrated that better performance can be achieved with better representation alignment under the UCL setting, as well as the limitations with existing approaches under prior shifts.
While the prior shifts experiments in Section 4.2 are performed on mBERT, in Fig. 10, we provide XLM-R Large results on MARC Japanese prior shift experiments. Furthermore, we provide full mBERT WikiANN results of applying IWDA (Table 4), and results on the XNLI dataset, a high-level semantic task of textual entailment. XNLI.
Because XNLI does not provide human-generated training data for languages other than English, to obtain unlabeled target language task data for our studies on the UCL setting, we use the machine-translation of the English training set provided by Conneau et al. (2020a). To remove potential effects due to parallelism, similar to the setup considered in Bari et al.
(2021), we partition the dataset into equal portions of size 20,000 (5% of the original size) and randomly select disjoint partitions to use as source and target data.
In addition, since the original XNLI test set is humantranslated but our target data is machine-translated, to avoid the effects from this discrepancy (or domain shift) between train and test, evaluations are performed on the machine-translated portion of the dataset in our experiments (we refer interested readers to Artetxe et al. (2020a) for a discussion on XNLI translation artifacts).
Due to our setup, our results are non-benchmark and cannot be compared to models evaluated on the original XNLI test set.
However, the goal of this experiment is to study whether our principle of achieving domain alignment is beneﬁcial on high-level tasks.
The results are included in Table 5. It is observed that IWDA generally improves transfer performance upon the zero-shot baseline, although the gains are not signiﬁcant. As discussed in Section 4.1, the potentials of IWDA will be limited if the zero-shot representations are already largely aligned, which is reﬂected by the small performance gap between zero-shot and supervised results in our setup. IWDA vs. DA.
Besides demonstrating that SSL suffers from negative transfer under large prior shifts in our experiments in Section 4.2, we also discussed in Section 2.2 that domain alignment without prior shift correction leads to negative transfer. In Fig. 11, we compare the performance on benchmark CoNLL German, which exhibits a prior shift from CoNLL English, from applying IWDA and domain alignment without prior shift correction (DA). CDAN refers to the conditioning technique by Long et al.
(2018), which we use in our IWDA implementation by default. It is observed that without prior shift correction, as training progresses, [Table] transfer performance of vanilla DA deteriorates as the model makes more and more erroneous feature alignments caused by the shift.
Better Training of Adversary.
As brieﬂy mentioned in Section 4, while the IWDA models for the experiments in the main sections are trained for four epochs with simultaneous adversary and model updates for fair comparisons with SSL techniques, better performance could be obtained from
IWDA if the optimality of the critic (adversary) can be better maintained throughout training with more passes over the data and alternating updates.
In Table 6, we observe that IWDA delivers much-improved performance on CoNLL when we perform four adversary updates per model update and allow 40 passes over the dataset (we also set lambda iw to 1).
Although this setting would lengthen the training time, this could be of practical interest for getting the best IWDA performance.
C I MPLEMENTATION D ETAILS OF IWDA
A high-level description of IWDA was given in Section 3 with discussions that connect it to the motivating ﬁndings in Section 2.
This section provides more details on the implementation of the algorithm along with the hyperparameter settings 7 used in our experiments (Table 7).
First, recall the joint objective, reproduced below, which consists of an alignment objective and an importance weight (IW) estimation objective,
[Equation]
Note that the alignment objective contains the discrepancy between (important-weighted) source and target feature distributions and source task loss, added to prevent g from degenerating to a trivial representation with perfect source and target alignment, e.g. constant maps x (cid:55)→ c .
Alignment Objective.
We reformulate the alignment objective in eq.
3 as a minimax objective, and replace explicit evaluation of the discrepancy D with a critic function f (also known as discriminator or adversary; Ganin et al. 2016;
Goodfellow et al. 2014). This turns the alignment objective into [Equation] where (cid:96) ad is a speciﬁc choice of adversarial loss accompanied by an adversary function class F , that takes as input the scores computed by the adversary and the domain labels (here, source domain is labeled 0 ).
The role of the adversary is to distinguish features computed from inputs sampled on the source domain from those on the target domain, and output a score on each feature that reﬂects its domain identity. When the features are aligned, the adversary would not be able to distinguish source and target features. In our implementation, we use Wasserstein-1 loss for (cid:96) ad , whose corresponding adversary function class F is 1-Lipschitz functions (Arjovsky et al., 2017). It is shown that when f attains its maximum, the term ( (cid:52) ) exactly computes the Wasserstein-1 distance between the importance-weighted feature
[Table]
distributions
[Table]
and p ZT . We parameterize F as a shallow ReLU network of width 2048, and enforce the Lipschitz condition using soft constraints in the form of zero-centered gradient penalty, which penalizes the gradient of f between pairs of source and target features (Thanh-Tung et al., 2019).
Last but not least, for added stability, we condition the alignment of features on their (pseudo-)classes as proposed by Long et al.
(2018) (CDAN). This is implemented as follows. Instead of feeding the adversary directly with features output by g , it receives the outer product between features and classiﬁer output distributions, g ( x ) ⊗ h ( g ( x )) .
This means that the input dimension of f is increased to the dimension of features multiplied by the number of output classes k . When k is large,
Long et al. (2018) used random projections to reduce dimensionality, which we did not explore. Putting everything together—minimax reformulation, Wasserstein-1 loss, zero-centered gradient penalty, and conditional alignment—upon receiving a mini-batch of source labeled examples, ( x i , y i ) ni =1 , and target unlabeled examples, ( x (cid:48) j ) mj
=1 , the empirical alignment loss is
[Equation]
where ˜ n = min( n, m ) and
˜ z i is a random point on the line connecting g ( x i ) and g ( x (cid:48) i ) .
Here, we have two hyperparameters: λ ( lambda da ) controls the strength of alignment, and µ ( lambda gp ) controls the strength on enforcing the Lipschitz condition.
The learning rate of the adversary ( lr ad ) is set to be larger than that of the model ( lr ; Heusel et al. 2017). AdamW optimizer is used throughout our experiments.
On this loss, we back-propagate gradients to f on maximizing the second and third terms, g on minimizing the ﬁrst two terms, and h on minimizing the ﬁrst term only. In our experiments in the main sections, all three models are optimized with simultaneous updates, but we note that alternating updates (standard in GAN literature) could allow IWDA to deliver further performance gains with longer training time.
In Table 6, we observe improved mBERT CoNLL results when the training alternates between four updates to f and one update to g for 40 epochs.
IW Estimation Objective.
We proposed several modiﬁcations to the IW estimation objective in Section 3 to improve its stability, and one of them is adding (cid:96) 2 -regularization to w (Azizzadenesheli et al., 2018).
Given ﬁrst moment statistics in the form of the source confusion matrix, C ij := p S ( ˆ Y =
i, Y = j ) , and target output distribution, ν i :=
p T ( ˆ Y = i ) , the regularized IWs objective minimizes k (cid:88) i = [Equation] (cid:18) ν [Equation]
−
[Equation]
(cid:88) [Equation]
C [Equation] +
[Equation] (4) where w (0) is an initial IW estimate (set to all-ones vector by default, i.e. no prior shift).
Here, one hyperparameter that controls the strength of the regularization is introduced, ξ ( lambda iw ). For simplicity, the (cid:96) 2 -regularization is implemented with and documented as weight decay.
Hence, ξ equals to lambda iw multiplied by 2 times the learning rate of w .
Instead of initializing w (0) = 1 , because IWDA ﬁne-tuning is preceded by zero-shot ﬁne-tuning in our experiments and its decent performance reﬂects partial class-conditional feature alignment, we initialize it to a crude IW estimate w (cid:48) obtained from the zero-shot model using the QP by Tachet des
Combes et al.
Since the initial alignment will not be perfect and w (cid:48) not entirely reliable, we set w (0) =
(1 − ρ ) w (cid:48)
+ ρ 1 , where ρ ( lambda iw init ) is a hyperparameter.
The statistics C, ν evolve during training as the feature representation computed by g changes, so it needs to be updated throughout the learning process.
Re-evaluating them at each optimization step on a held-out set would make training prohibitively slow, hence we update them with the statistics ˆ C, ˆ ν computed on training mini-batches: C ( t ) =
(1 − η ) C ( t − 1) + η ˆ C , and similarly for ν ( t ) .
This update rate η ( lr iw stats ) is not the same as IW learning rate ( lr iw ). Lastly, to minimize potential effects from the miscalibration of neural models (Guo et al., 2017;
Alexandari et al., 2020), hard confusion matrix and target output distribution are used in the estimation of IWs (Lipton et al., 2018; Garg et al., 2020).
D D ESCRIPTION OF SSL B ASELINES
Knowledge Distillation (KD).
Our implementation follows the original paper by Hinton et al.
(2015) closely (from which the Wu et al.
(2020) variant is moderately different). Given a teacher LM ﬁne-tuned either under zero-shot or using IWDA, we distill its knowledge on a newly initialized pre-trained student LM by minimizing the mean-squared-error difference of the teacher and student Softmax outputs on source and target data without labels.
If labeled data from the source is available, we also train the student to minimize source task loss.
We set the temperature to 3, and the weight of the source task loss to 0.1.
Learning rate is 1e-4, batch size is 32, and we train for 4 epochs.
In addition, Wu et al.
(2020) observed that when distilling to pretrained LMs, better performance can be achieved if the bottom three layers of the student is frozen (motivated by ﬁndings from Wu & Dredze (2019)).
We also apply this technique in our KD experiments.
Self-Training (ST).
Our implementation follows Dong & de Melo (2019) closely. Given an LM ﬁne-tuned on source domain only, ST continues ﬁne-tuning the model by ﬁrst assigning pseudolabels to target data, collecting the ones with the highest conﬁdence, and then train on them with the pseudo-labels.
Source task loss is also minimized during self-training.
The hyperparameters of ST involve selecting how any pseudo-labeled examples to train on. On
MARC with 200,000 training examples, at each epoch, we select top 4,000 most conﬁdent pseudolabeled examples from the target domain conditioned on each (pseudo-)class, so that 20,000 examples are selected each round (MARC has ﬁve classes). These numbers are scales proportionally on datasets of other sizes.
We perform self-training for 4 epochs, and set learning rate to 1e-5 and batch size to 128.
On IWDA + ST prior shift experiments, we select much less examples for self-training to avoid training on low-conﬁdence examples from minority classes. This adjustment of hyperparameters relies on knowledge about the prior distribution on the target domain, which is approximated by the IW estimates from IWDA.