Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training (2021)

Abstract: Pre-trained multilingual language encoders, such as multilingual BERT and XLM-R, show great potential for zero-shot cross-lingual transfer. However, these multilingual encoders do not precisely align words and phrases across languages. Especially, learning alignments in the multilingual embedding space usually requires sentence-level or word-level parallel corpora, which are expensive to be obtained for low-resource languages. An alternative is to make the multilingual encoders more robust; when fine-tuning the encoder using downstream task, we train the encoder to tolerate noise in the contextual embedding spaces such that even if the representations of different languages are not aligned well, the model can still achieve good performance on zero-shot cross-lingual transfer. In this work, we propose a learning strategy for training robust models by drawing connections between adversarial examples and the failure cases of zero-shot cross-lingual transfer. We adopt two widely used robust training methods, adversarial training and randomized smoothing, to train the desired robust model. The experimental results demonstrate that robust training improves zero-shot cross-lingual transfer on text classification tasks. The improvement is more significant in the generalized cross-lingual transfer setting, where the pair of input sentences belong to two different languages.

## introduction zero-shot cross-lingual transfer learning aims to learn models with data available in one or more source languages and use them in other target languages for which there is no data (zero-resource) available. the zero-shot cross-lingual transfer has a great practical value for low-resource languages since it reduces the requirement of labeled data to learn models for downstream tasks, e.g., text classification @xcite and question answering @xcite . recently, pre-trained multilingual language encoders, such as multilingual bert @xcite and xlm-r @xcite , demonstrate promising performance on zero-shot crosslingual transfer learning for a wide range of downstream tasks @xcite . these language encoders learn a shared multilingual contextual embedding space; they are able to represent word pairs in parallel sentences with similar contextual representations. however, the multilingual encoders fail to capture this similarity when the source and target languages are less similar at levels of morphology, syntax, and semantics @xcite . prior studies @xcite @xcite have shown that aligning the representations of different languages in the multilingual embedding space plays an important role for zero-shot cross-lingual transfer learning. as illustrated in figure 1a , words with similar meanings (e.g. this, ceci, and 这) have similar representations in the contextual multilingual embedding space, even though these words are in different languages. this alignment helps models transfer the learned knowledge from source languages to target languages. therefore, several works focus on improving the quality of alignments in the multilingual embedding space @xcite @xcite . nevertheless, learning such alignments usually requires sentence-level or word-level parallel corpora, which are expensive to be obtained for low-resource languages. in addition, because the meanings of words in different languages are usually not exactly matched, learn a perfect alignment could be impossible. in this work, we start from another point of view to improve zero-shot cross-lingual transfer performance. we aim to make the multilingual encoders robust such that they can tolerate a certain amount of noise in the input embeddings. more specifically, as shown in figure 1b , we target to construct robust regions (orange circles) for embeddings in the multilingual embedding space. during training, the robust model is expected to output similar predictions for embeddings in the same robust region. therefore, as long as similar words in different languages fall into the same robust region, even if they are not perfectly aligned, the model can still have similar predictions for them. to learn the robust model, we first draw connections between adversarial examples @xcite @xcite and the failure cases of zero-shot cross-lingual transfer, and then study two widely used robust training methods to learn the robust model: (1) adversarial training @xcite and (2) randomized smoothing @xcite . both of them can make the model robust against perturbations in the input embeddings by modifying the training objective when fine-tuning model for the downstream task. for randomized smoothing, we also adopt the data augmentation approach @xcite to learn the robust model. we perform experiments on two cross-lingual text classification tasks, paraphrase identification and natural language inference 1 . the experimental results demonstrate that robust training indeed improves the performance of zero-shot cross-lingual transfer on the classification benchmarks: paws-x @xcite and xnli @xcite . on average the cross-lingual transfer performance improves by 2.1 and 1.6 points on paws-x and xnli, respectively. in addition, we show that robust training remarkably improves generalized 1 our code is available at https://github.com/ uclanlp/robust-xlt cross-lingual transfer @xcite . in this setting, the pair of input sentences in the text classification tasks belong to two different languages, e.g., paraphrase prediction for a pair of sentences in english and korean. embedding space alignments. learning to align embedding spaces have always been an important research topic to improve multilinguality. early works focus on word embedding spaces @xcite @xcite . recently, many approaches are proposed to align contextual word embedding spaces, such as learning rotation projections @xcite @xcite and fine-tuning pre-trained multilingual language models @xcite @xcite @xcite @xcite . however, most of them require additional supervision signals, such as parallel sentence pairs @xcite @xcite , bilingual dictionary @xcite @xcite , or both @xcite . these additional supervised corpora are usually expensive for low-resource languages. embedding misalignment handling. instead of directly aligning the representations, there is a line of research making the model be aware of the embedding misalignment issues by considering additional syntactic features, such as part-of-speech @xcite and dependency parse trees @xcite @xcite @xcite , and other syntactic features @xcite . however, those syntactic features require large human efforts to obtained. robust training. recently, adversarial attacks are presented to check the robustness of nlp models, such as character manipulation @xcite , word replacements @xcite @xcite , and syntactic rearrangements @xcite . to against those attacks, various robust training methods are proposed. for example, @xcite trains a robust model by data augmentation with generated adversarial examples. other works @xcite @xcite consider adversarial training, which includes the adversarial accuracy to the training objective. a few studies propose transformations on inputs before feeding them to models @xcite . randomized smoothing @xcite is presented to make models robust against noise in input representations. another line of research aims at providing theoretical guarantee of robustness, including interval bound propagation methods @xcite and verification methods @xcite . most of those robust training methods focus on defending adversarial attacks, while we propose to apply robust training methods to improve the zero-shot cross-lingual transfer performance. ## zero-shot cross-lingual transfer with robust training in this work, we focus on zero-shot cross-lingual transfer for text classification tasks. our goal is to learn a classifier f from a set of training examples in source languages x @xmath0, y i )} n @xmath1. at test time, we directly use the classifier f to conduct inference on a set of test examples in target languages x @xmath2. we expect that the classifier f can transfer the learned knowledge from the source languages to the target languages. ## experiments we conduct experiments to verify that robust training indeed improves the performance of zero-shot cross-lingual transfer. ## conclusion in this work, we propose a robust model by drawing connections between adversarial examples and the failure cases of zero-shot cross-lingual transfer. we adopt two robust training methods, adversarial training and randomized smoothing, to train the desired robust model. the experimental results demonstrate that robust training improves zero-shot cross-lingual transfer on text classification tasks. in addition, the improvement is more significant in the generalized cross-lingual transfer setting.