Consistency Regularization for Cross-Lingual Fine-Tuning (2021)

Abstract: Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation. In addition, we employ model consistency to regularize the models trained with two augmented versions of the same training set. Experimental results on the XTREME benchmark show that our method significantly improves cross-lingual fine-tuning across various tasks, including text classification, question answering, and sequence labeling.

## introduction pre-trained cross-lingual language models (conneau and @xcite @xcite have shown great transferability across languages. by fine-tuning on labeled data in a source language, the models can generalize to other target languages, even without any additional training. such generalization ability reduces the required annotation efforts, which is prohibitively expensive for low-resource languages. recent work has demonstrated that data augmentation is helpful for cross-lingual transfer, e.g., translating source language training data into target languages @xcite , and generating codeswitch data by randomly replacing input words in the source language with translated words in target languages @xcite . by populating the dataset, their fine-tuning still treats training instances independently, without considering the inherent correlations between the original input and its augmented example. in contrast, we propose to utilize consistency regularization to better leverage data augmentation for cross-lingual fine-tuning. intuitively, for a semantic-preserving augmentation strategy, the predicted result of the original input should be similar to its augmented one. for example, the classification predictions of an english sentence and its translation tend to remain consistent. in this work, we introduce a cross-lingual finetuning method xtune that is enhanced by consistency regularization and data augmentation. first, example consistency regularization enforces the model predictions to be more consistent for semantic-preserving augmentations. the regularizer penalizes the model sensitivity to different surface forms of the same example (e.g., texts written in different languages), which implicitly encourages cross-lingual transferability. second, we introduce model consistency to regularize the models trained with various augmentation strategies. specifically, given two augmented versions of the same training set, we encourage the models trained on these two datasets to make consistent predictions for the same example. the method enforces the corpus-level consistency between the distributions learned by two models. under the proposed fine-tuning framework, we study four strategies of data augmentation, i.e., subword sampling @xcite , code-switch substitution @xcite , gaussian noise @xcite , and machine translation. we evaluate xtune on the xtreme benchmark @xcite , including three different tasks on seven datasets. experimental results show that our method outperforms conventional fine-tuning with data augmentation. we also demonstrate that xtune is flexible to be plugged in various tasks, such as classification, span extraction, and sequence labeling. we summarize our contributions as follows: ## related work cross-lingual transfer besides learning crosslingual word embeddings @xcite @xcite @xcite , most recent work of cross-lingual transfer is based on pre-trained cross-lingual language models @xcite @xcite . these models generate multilingual contextualized word representations for different languages with a shared encoder and show promising cross-lingual transferability. cross-lingual data augmentation machine translation has been successfully applied to the cross-lingual scenario as data augmentation. a common way to use machine translation is to finetune models on both source language training data and translated data in all target languages. furthermore, @xcite proposed to replace a segment of source language input text with its translation in another language. however, it is usually impossible to map the labels in source language data into target language translations for token-level tasks. zhang et al. (2019) used code-mixing to perform the syntactic transfer in cross-lingual dependency parsing. fei et al. (2020) constructed pseudo translated target corpora from the gold-standard annotations of the source languages for cross-lingual semantic role labeling. fang et al. (2020) proposed an additional kullback-leibler divergence self-teaching loss for model training, based on autogenerated soft pseudo-labels for translated text in the target language. besides, qin et al. (2020) finetuned models on multilingual code-switch data, which achieves considerable improvements. consistency regularization one strand of work in consistency regularization focused on regularizing model predictions to be invariant to small perturbations on image data. the small perturbations can be random noise @xcite , adversarial noise @xcite and various data augmentation approaches @xcite @xcite . similar ideas are used in the natural language processing area. both adversarial noise @xcite @xcite and sampled gaussian noise @xcite ) are adopted to augment input word embeddings. another strand of work focused on consistency under different model parameters @xcite , which is complementary to the first strand. we focus on the cross-lingual setting, where consistency regularization has not been fully explored. ## methods conventional cross-lingual fine-tuning trains a pretrained language model on the source language and directly evaluates it on other languages, which is also known as the setting of zero-shot cross-lingual fine-tuning. specifically, given a training corpus d in the source language (typically in english), and a model f (•; θ) that predicts task-specific probability distributions, we define the loss of cross-lingual fine-tuning as: ## conclusion in this work, we present a cross-lingual fine-tuning framework xtune to make better use of data augmentation. we propose two consistency regularization methods that encourage the model to make consistent predictions for an example and its semantically equivalent data augmentation. we explore four types of cross-lingual data augmentation strategies. we show that both example and model consistency regularization considerably boost the performance compared to directly fine-tuning on data augmentations. meanwhile, model consistency regularization enables semi-supervised training on the unlabeled target language translations. xtune combines the two regularization methods, and the experiments show that it can improve the performance by a large margin on the xtreme benchmark. variable xnli paws-x pos ner xquad mlqa tydiqa