Effective Fine-Tuning Methods for Cross-lingual Adaptation (2021)

Abstract: Large scale multilingual pre-trained language models have shown promising results in zero- and few-shot cross-lingual tasks. However, recent studies have shown their lack of generalizability when the languages are structurally dissimilar. In this work, we propose a novel fine-tuning method based on co-training that aims to learn more generalized semantic equivalences as a complementary to multilingual language modeling using the unlabeled data in the target language. We also propose an adaption method based on contrastive learning to better capture the semantic relationship in the parallel data, when a few translation pairs are available. To show our method’s effectiveness, we conduct extensive experiments on cross-lingual inference and review classification tasks across various languages. We report significant gains compared to directly fine-tuning multilingual pre-trained models and other semi-supervised alternatives.

## introduction self-supervised pre-trained models @xcite @xcite have revolutionized natural language processing (nlp). such pre-training with language modeling objectives provides a useful initial point for model parameters that adapt well to new tasks with supervised fine-tuning. building on the success of monolingual pre-trained language models (lm) such as bert @xcite and roberta @xcite , multilingual models like mbert @xcite , xlm @xcite , xlm-r @xcite have pushed the state-of-the-art on cross-lingual tasks by pre-training large transformer @xcite models jointly on many languages. the multilingual pre-trained lms support zeroshot transfer from a source language to target lan-guages, meaning that fine-tuning the pre-trained lm on the source-language labeled data such as english, could transfer well to other languages. recent research @xcite @xcite has shown that the transfer capability of these multilingual lms mainly relies on the structural similarity between the source and target languages. when the target language is structurally dissimilar to the source, the transfer ability is shown to be low in the zero-shot setting. since the multilingual lms are generally trained with a self-supervised masked language modeling (mlm) objective without considering parallel information or semantic equivalences, they cannot capture well semantic similarity across languages as reflected by their low tatoeba score @xcite . this could also potentially harm their zero-shot transfer performance on the tasks as @xcite show that injecting cross-lingual signals by replacing masked tokens with semantically similar words from other languages improves mbert's multilinguality and zero-shot cross-lingual inference (xnli) results. concurrently, the multilingual embedding models such as laser @xcite and labse @xcite use parallel data to learn language invariant sentence representation by encoding texts from different languages into a shared embedding space. these models can capture semantic similarity well as often shown by their high tatoeba scores and their success in tasks that involve cross-lingual similarity such as crosslingual retrieval and bitext mining. however, it has been shown that these models generally lag behind the multilingual lms on zero-shot cross-lingual classification tasks like xnli @xcite . we hypothesize source information might be necessary to achieve better zero-shot transfer as shown empirically by @xcite with intermediate task fine-tuning in the source language (english). in this work, we argue that the multilingual em-bedding models and multilingual lms are complementary to each other -the task adaptability of the multilingual lms can be complemented by the semantic awareness of the sentence embedding models and vice versa. for this, we first propose a co-training @xcite framework that facilitates the multilingual lms and sentence embedding models to learn from each other by using unlabeled data from target languages. secondly, we aim to improve the multilingual lm fine-tuning when there are small amounts of parallel pairs within the training datasets for tasks. compared to the existing translate-train methods @xcite ) that simply use translation as a data-augmentation method to generate labeled data in the target language, we propose a novel language adaptation approach based on contrastive learning that aligns the parallel data to model the semantic relationship between the translation pairs for effective fine-tuning. we performed extensive experiments on xnli @xcite and multilingual amazon review corpus (marc) @xcite datasets. the experimental results demonstrate that our approach outperforms previous methods for various classification tasks across different languages by a good margin. in particular, on xnli, our proposed co-training method improves over the original mbert and xlm-r by 2.3% and 1.7% on average for zero-shot cross-lingual transfer. on marc, our approach gets on average 8% and 1.1% gains for mbert and xlm-r, respectively. ## related works since the introduction of the transformer network @xcite , it has become a common model of choice for language representation learning. pre-trained transformer-based models such as mbert @xcite have proven effective in learning cross-lingual information. mbert was pre-trained on raw wikipedia texts in 104 languages using masked language modeling (mlm) and next sentence prediction (nsp) tasks with no explicit cross-lingual objective. xlm-r @xcite improves over mbert by training longer with more data from commoncrawl, and without the nsp objective. meanwhile, several studies examine what makes these pre-trained language models multilingual, and why it works well for cross-lingual transfer. @xcite hypothesize that the cross-lingual capability of mbert benefits from having a shared (sub-word) vocabulary for all languages, which helps to bind the languages by mapping the token representations into a shared space. k et al. (2020) point out that the contribution from shared subwords is minimal. on the other hand, the structural similarity (e.g., word order, word frequency, etc.) is more important for effective cross-lingual transfer. another line of work on multilingual pretraining focuses on generating multilingual sentence embeddings such that semantically similar sentences across different languages will be closer in a shared vector space. laser @xcite uses an encoder-decoder architecture @xcite . it trains on large parallel data to learn multilingual fixed-length sentence embedding for 93 languages on a translation task. multilingual universal encoder (muse) @xcite uses a dual-encoder architecture that is trained on one billion crawled question-answering pair with a translation ranking task: given a sentence from the source language and a group of candidate text from target languages, the model needs to recognize the corresponding translation of the source-language text from the candidates. labse @xcite ) is based on the bert architecture using the same translation ranking task with muse but is trained on a much larger dataset of six billion translation pairs. some researchers also tried to introduce crosslingual alignment from parallel data as an auxiliary objective of the original mlm in the pre-training. @xcite align mbert embeddings in a post-hoc manner. they first apply a statistical word aligner to align tokens in the parallel sentences. then, mbert is tuned via minimizing the meansquared error between the embedding of the english words and the corresponding words in other languages. @xcite tried to minimize the vector-space distance between a source language sentence and its translation during the pre-training. the problem with these kinds of methods is that they either need to pre-train a new model from scratch @xcite , or need to do a second round of pre-training on top of the original multilingual lm @xcite . from both the computation and data perspective, the cost is very high. in another line of work, researchers use data augmentation to solve language adaptation problem in cross-lingual tasks. for example, @xcite use xlm-r's mask language model to augment the data with vicinal samples. @xcite proposed a labeled sequence translation method to translate source-language ner training data to target languages and train a generation-based multilingual data augmentation method. these methods are orthogonal to our methods. summary current work on multilingual pretraining either does not consider sentence-level cross-lingual alignment in the pre-training (making them sacrifice transfer capability to structurally dissimilar language), or they only consider alignment signals, which makes them expensive to train. in contrast, in our work we focus on utilizing the parallel information in the fine-tuning phase with minimum costs to avoid building a pre-trained model from scratch or using large amounts of parallel data. our approach can also naturally co-operate with different multilingual models based on different pre-training objectives. ## methods in this section, we describe our co-training framework for cross-lingual transfer ( §3.1) followed by the contrastive language adaptation ( §3.2). ## experiments to show the effectiveness of our model, we evaluate our proposed methods on two cross-lingual classification tasks as we describe below. ## conclusion & future work in this paper, we have proposed an effective finetuning method to improve cross-lingual transfer capability of multilingual pre-trained lms. in contrast to previous work, our proposed co-training framework can make multilingual pre-trained lms learn cross-lingual semantic relationships from the multilingual embedding model. moreover, we propose a novel language adaptation approach based on contrastive learning. when there exist translation pairs within the training dataset, our language adaption approach can better model the semantic relationship across languages on translation pairs for effective fine-tuning. extensive experiments have been conducted on the xnli and amazon multilingual review dataset, which show that our method outperforms previous methods on both zero-shot transfer and few-shot transfer. for future studies, we will investigate the data selection policies for the co-training methods. in some cases, the distribution of the labeled data could be different from that of the unlabeled data. it may yield a sampling bias in the training iterations of co-training that shifts towards the unlabeled set, thus hurting the model performance. a more robust