UXLA : A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP (2021)

Abstract: Transfer learning has yielded state-of-the-art (SoTA) results in many supervised NLP tasks. However, annotated data for every target task in every target language is rare, especially for low-resource languages. We propose UXLA, a novel unsupervised data augmentation framework for zero-resource transfer learning scenarios. In particular, UXLA aims to solve cross-lingual adaptation problems from a source language task distribution to an unknown target language task distribution, assuming no training label in the target language. At its core, UXLA performs simultaneous self-training with data augmentation and unsupervised sample selection. To show its effectiveness, we conduct extensive experiments on three diverse zero-resource cross-lingual transfer tasks. UXLA achieves SoTA results in all the tasks, outperforming the baselines by a good margin. With an in-depth framework dissection, we demonstrate the cumulative contributions of different components to its success.

## introduction self-supervised learning in the form of pretrained language models (lm) has been the driving force in developing state-of-the-art nlp systems in recent years. these methods typically follow two basic steps, where a supervised task-specific finetuning follows a large-scale lm pretraining @xcite . however, getting labeled data for every target task in every target language is difficult, especially for low-resource languages. recently, the pretrain-finetune paradigm has also been extended to multi-lingual setups to train effective multi-lingual models that can be used for zero-shot cross-lingual transfer. jointly trained deep multi-lingual lms like mbert @xcite and xlm-r @xcite ) coupled * equal contribution with supervised fine-tuning in the source language have been quite successful in transferring linguistic and task knowledge from one language to another without using any task label in the target language. the joint pretraining with multiple languages allows these models to generalize across languages. despite their effectiveness, recent studies @xcite ) have also highlighted one crucial limiting factor for successful crosslingual transfer. they all agree that the crosslingual generalization ability of the model is limited by the (lack of) structural similarity between the source and target languages. for example, for transferring mbert from english, k et al. (2020) report about 23.6% accuracy drop in hindi (structurally dissimilar) compared to 9% drop in spanish (structurally similar) in cross-lingual natural language inference (xnli). the difficulty level of transfer is further exacerbated if the (dissimilar) target language is low-resourced, as the joint pretraining step may not have seen many instances from this language in the first place. in our experiments ( §3.2), in cross-lingual ner (xner), we report f1 reductions of 28.3% in urdu and 30.4% in burmese for xlm-r, which is trained on a much larger multilingual dataset than mbert. one attractive way to improve cross-lingual generalization is to perform data augmentation @xcite , and train the model on examples that are similar but different from the labeled data in the source language. formalized by the vicinal risk minimization (vrm) principle @xcite , such data augmentation methods have shown impressive results in vision @xcite . these methods enlarge the support of the training distribution by generating new data points from a vicinity distribution around each training example. for images, the vicinity of a training image can be defined by a set of operations like rotation and scaling, or by linear mixtures of features and labels @xcite . however, when it comes to text, such unsupervised augmentation methods have rarely been successful. the main reason is that unlike images, linguistic units are discrete and a smooth change in their embeddings may not result in a plausible linguistic unit that has similar meanings. in nlp, to the best of our knowledge, the most successful augmentation method has so far been back-translation @xcite which paraphrases an input sentence through round-trip translation. however, it requires parallel data to train effective machine translation systems, acquiring which can be more expensive for low-resource languages than annotating the target language data. furthermore, back-translation is only applicable in a supervised setup and to tasks where it is possible to find the alignments between the original labeled entities and the back-translated entities, such as in question answering @xcite . other related work includes contextual augmentation @xcite , conditional bert @xcite and aug-bert @xcite . these methods use a constrained augmentation that alters a pretrained lm to a label-conditional lm for a specific task. since they rely on labels, their application is limited by the availability of enough task labels. in this work, we propose uxla, a robust unsupervised cross-lingual augmentation framework for improving cross-lingual generalization of multilingual lms. uxla augments data from the unlabeled training examples in the target language as well as from the virtual input samples generated from the vicinity distribution of the source and target language sentences. with the augmented data, it performs simultaneous self-learning with an effective distillation strategy to learn a strongly adapted cross-lingual model from noisy (pseudo) labels for the target language task. we propose novel ways to generate virtual sentences using a multilingual masked lm @xcite , and get reliable task labels by simultaneous multilingual co-training. this co-training employs a twostage co-distillation process to ensure robust transfer to dissimilar and/or low-resource languages. we validate the effectiveness and robustness of uxla by performing extensive experiments on three diverse zero-resource cross-lingual transfer tasks-xner, xnli, and paws-x, which posit different sets of challenges, and across many (14 in total) language pairs comprising languages that are similar/dissimilar/low-resourced. uxla yields impressive results on xner, setting sota in all tested languages outperforming the baselines by a good margin. the relative gains for uxla are particularly higher for structurally dissimilar and/or low-resource languages: 28.54%, 16.05%, and 9.25% absolute improvements for urdu, burmese, and arabic, respectively. for xnli, with only 5% labeled data in the source, it gets comparable results to the baseline that uses all the labeled data, and surpasses the standard baseline by 2.55% on average when it uses all the labeled data in the source. we also have similar findings in paws-x. we provide a comprehensive analysis of the factors that contribute to uxla's performance. we open-source our framework at https://ntunlpsg.github.io/project/uxla/ . ## uxla framework while recent cross-lingual transfer learning efforts have relied almost exclusively on multi-lingual pretraining and zero-shot transfer of a fine-tuned source model, we believe there is a great potential for more elaborate methods that can leverage the unlabeled data better. motivated by this, we present uxla, our unsupervised data augmentation framework for zero-resource cross-lingual task adaptation. figure 1 gives an overview of uxla. let d @xmath0, y s ) and d @xmath1, respectively. uxla augments data from various origins at different stages of training. in the initial stage (epoch 1), it uses the augmented training samples from the target language (d t ) along with the original source (d s ). in later stages (epoch 2-3), it uses vicinal sentences generated from the vicinity distribution of source and target examples: ϑ(x s n |x s n ) and ϑ(x t n |x t n ), where x s n ∼ x s and x t n ∼ x t . it performs self-training on the augmented data to acquire the corresponding pseudo labels. to avoid confirmation bias with self-training where the model accumulates its own errors, it simultaneously trains three task models to generate virtual training data through data augmentation and filtering of potential label noises via multi-epoch co-teaching @xcite . in each epoch, the co-teaching process first performs co-distillation, where two peer task models are used to select "reliable" training examples to train the third model. the selected samples with pseudo labels are then added to the target task figure 1 : training flow of uxla. after training the base task models θ (1) , θ (2) , and θ (3) on source labeled data ds (warmup), we use two of them (θ (j) , θ (k) ) to pseudo-label and co-distill the unlabeled target language data (d t ). a pretrained lm (gen-lm) is used to generate new vicinal samples for both source and target languages, which are also pseudo-labeled and co-distilled using the two task models (θ (j) , θ (k) ) to generate ds and dt. the third model θ (l) is then progressively trained on these datasets: {ds, d t } in epoch 1, dt in epoch 2, and all in epoch 3. ## experiments we consider three tasks in the zero-resource crosslingual transfer setting. we assume labeled training data only in english, and transfer the trained model to a target language. for all experiments, we report the mean score of the three models that use different seeds. ## analysis in this section, we analyze uxla by dissecting it and measuring the contribution of its each of the components. for this, we use the xner task and analyze the model based on the results in ## related work recent years have witnessed significant progress in learning multilingual pretrained models. notably, mbert @xcite extends (english) bert by jointly training on 102 languages. xlm @xcite extends mbert with a conditional lm and a translation lm (using parallel data) objectives. @xcite train the largest multilingual language model xlm-r with roberta @xcite . @xcite dredze (2019), keung et al. (2019), and pires et al. (2019) evaluate zero-shot cross-lingual transferability of mbert on several tasks and attribute its generalization capability to shared subword units. pires et al. (2019) also found structural similarity (e.g., word order) to be another important factor for successful crosslingual transfer. k et al. (2020), however, show that the shared subword has a minimal contribution; instead, the structural similarity between languages is more crucial for effective transfer. older data augmentation approaches relied on distributional clusters @xcite . a number of recent methods have been proposed using contextualized lms @xcite @xcite @xcite . these methods rely on labels to perform label-constrained augmentation, thus not directly comparable with ours. also, there are fundamental differences in the way we use the pretrained lm. unlike them our lm augmentation is purely unsupervised and we do not perform any fine-tuning of the pretrained vicinity model. this disjoint characteristic gives our framework the flexibility to replace θ lm even with a better monolingual lm for a specific target language, which in turn makes uxla extendable to utilize stronger lms that may come in the future. in a concurrent work @xcite , we propose a contextualized lm based data augmentation for neural machine translation and show its advantages over traditional back-translation gaining improved performance in low-resource scenarios. ## conclusion we propose a novel data augmentation framework, uxla, for zero-resource cross-lingual task adaptation. it performs simultaneous self-training with data augmentation and unsupervised sample selection. with extensive experiments on three different cross-lingual tasks spanning many language pairs, we have demonstrated the effectiveness of uxla. for the zero-resource xner task, uxla sets a new sota for all the tested languages. for both xnli and paws-x tasks, with only 5% labeled data in the source, uxla gets comparable results to the baseline that uses 100% labeled data. figures 3(c ) and 3(d) present the per-sample loss (i.e., mean loss per sentence w.r.t. the pseudo labels) distribution in histogram without and with confidence penalty, respectively. here, accurate-2 refers to the sentences which have at most two wrong ner labels, and sentences containing more than two errors are referred to as noisy samples. it shows that without confidence penalty, there are many noisy samples with a small loss which is not desired. in addition to that, the figures also suggest that the confidence penalty helps to separate the clean samples from the noisy ones either by clustering or by model confidence. figures 4(a ) and 4(b) present the loss distribution in a scatter plot by sorting the sentences based finally, figures 4(c ) and 4(d) show the length distribution of all vs. the selected sentences (by distillation by model confidence) without and with confidence penalty. @xcite shows that cross-lingual ner inference is heavily dependent on the length distribution of the samples. in general, the performance of the lower length samples is more accurate. however, if we only select the lower length samples we will easily overfit. from these plots, we observe that the confidence penalty also helps to perform a better distillation as more sentences are selected (by the distillation procedure) from the lower length distribution, while still covering the entire lengths. this shows that using the confidence penalty in training, model becomes more robust. in summary, comparing the figures 3(c-d ) -4(cd), we can conclude that training without confidence penalty can make the model more prone to over-fitting, resulting in more noisy pseudo labels. training with confidence penalty not only improves pseudo labeling accuracy but also helps the distillation methods to perform better noise filtering.