Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020)

Abstract: Multilingual pretrained language models (such as multilingual BERT) have achieved impressive results for cross-lingual transfer. However, due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. In this work, we present two approaches to improve zero-shot cross-lingual classification, by transferring the knowledge from monolingual pretrained models to multilingual ones. Experimental results on two cross-lingual classification benchmarks show that our methods outperform vanilla multilingual fine-tuning.

## introduction supervised text classification heavily relies on manually annotated training data, while the data are usually only available in rich-resource languages, such as english. it requires great effort to make the resources available in other languages. various methods have been proposed to build cross-lingual classification models by exploiting machine translation systems @xcite @xcite , and learning multilingual embeddings @xcite @xcite . recently, multilingual pretrained language models have shown surprising cross-lingual effectiveness on a wide range of downstream tasks @xcite @xcite . even without using any parallel corpora, the pretrained models can still perform zero-shot cross-lingual classification @xcite @xcite . that is, these models can be fine-tuned in a source language, and then directly evaluated in other target languages. despite the effectiveness of cross-lingual transfer, the multilingual pretrained language models have their own drawbacks. due to the constant number of model parameters, the model capacity of the richresource languages decreases if we adds languages for pre-training. the curse of multilinguality results in that the multilingual models usually perform worse than their monolingual competitors on downstream tasks @xcite . the observations motivate us to leverage monolingual pretrained models to improve multilingual models for cross-lingual classification. in this paper, we propose a multilingual finetuning method (monox) based on the teacherstudent framework, where a multilingual student model learns end task skills from a monolingual teacher. intuitively, monolingual pretrained models are used to provide supervision of downstream tasks, while multilingual models are employed for knowledge transfer across languages. we conduct experiments on two widely used cross-lingual classification datasets, where our methods outperform baseline models on zero-shot cross-lingual classification. moreover, we show that the monolingual teacher model can help the student multilingual model for both the source language and target languages, even though the student model is only trained in the source language. ## background: multilingual fine-tuning we use multilingual bert @xcite for multilingual pretrained language models. the pretrained model uses the bert-style transformer @xcite architecture, and follows the similar fine-tuning procedure as bert for text classification, which is illustrated in figure 1(a) . to be specific, the first input token of the models is always a special classification token [cls]. during fine-tuning, the final hidden state of the special token is used as the sentence representation. in order to output predictions, an additional softmax classifier is built on top of the sentence representation. denoting d as the training data in the source language, the pretrained models are fine-tuned with standard cross-entropy loss: ## methods as shown in figure 1(b) , we first fine-tune the monolingual pretrained model in the source language. then we transfer task knowledge to the multilingual pretrained model by soft (section 3.1) or hard (section 3.2) labels. we describe two variants of our proposed method (monox) as follows. ## conclusion in this work, we investigated whether a monolingual pretrained model can help cross-lingual classification. our results have shown that, with a roberta model pretrained in english, we can boost the classification performance of a pretrained multilingual bert in other languages. for future work, we will explore whether mono-lingual pretrained models can help other crosslingual nlp tasks, such as natural language generation @xcite .