Preserving Cross-Linguality of Pre-trained Models via Continual Learning (2021)

Abstract: Recently, fine-tuning pre-trained language models (e.g., multilingual BERT) to downstream cross-lingual tasks has shown promising results. However, the fine-tuning process inevitably changes the parameters of the pre-trained model and weakens its cross-lingual ability, which leads to sub-optimal performance. To alleviate this problem, we leverage continual learning to preserve the original cross-lingual ability of the pre-trained model when we fine-tune it to downstream tasks. The experimental result shows that our fine-tuning methods can better preserve the cross-lingual ability of the pre-trained model in a sentence retrieval task. Our methods also achieve better performance than other fine-tuning baselines on the zero-shot cross-lingual part-of-speech tagging and named entity recognition tasks.

## introduction recently, multilingual language models @xcite , pretrained on extensive monolingual or bilingual resources across numerous languages, have been shown to enjoy surprising cross-lingual adaptation abilities, and fine-tuning them to downstream crosslingual tasks has achieved promising results @xcite . taking this further, better pre-trained language models have been proposed to improve the cross-lingual performance, such as using larger amounts of pre-trained data with larger pre-trained models @xcite , and utilizing more tasks in the pre-training stage @xcite . however, we observe that multilingual bert (mbert) @xcite , a pre-trained language model, forgets the masked language model (mlm) task that has been learned and partially loses the cross-lingual ability (from a cross-lingual in this paper, we consider a new direction to improve the cross-lingual performance, which is to preserve the cross-lingual ability of pre-trained multilingual models in the fine-tuning stage. motivated by the continual learning @xcite @xcite that aims to learn a new task without forgetting the previous learned tasks, we adopt a continual learning framework to constrain the parameter learning in the pre-trained multilingual model when we fine-tune it to downstream tasks in the source language. specifically, based on the results in figure 1 , we aim to maintain the cross-linguality of pre-trained multilingual models by utilizing mlm and xsr tasks to constrain the parameter learning in the fine-tuning stage. experiments show that our methods help pretrained models better preserve the cross-lingual ability. additionally, our methods surpass other fine-tuning baselines on the strong multilingual model mbert and xlmr @xcite on zero-shot cross-lingual part-of-speech tagging (pos) and named entity recognition (ner) tasks. ## related work cross-lingual methods, which alleviate the need for obtaining large amounts of annotated data in target languages, have been applied to multiple nlp tasks, such as task-oriented dialogue systems @xcite , part-of-speech tagging @xcite @xcite , named entity recognition @xcite @xcite , abstractive summarization @xcite , and dependency parsing @xcite . recently, multilingual language models @xcite @xcite , pre-trained on a largescale data corpus across a great many languages, have significantly improved the cross-lingual performance. however, the corresponding fine-tuning techniques have been less studied. @xcite investigated the effectiveness of fine-tuning mbert by freezing its partial bottom layers, and @xcite further analyzed the fine-tuning of mbert. ## methodology in this section, we first describe the gradient episodic memory (gem) @xcite , a continual learning framework, which we adopt to constrain the fine-tuning process. then, we introduce how we fine-tune the pre-trained multilingual model with gem. ## results & analysis does gem preserve the cross-lingual ability? from in addition, although gem w/ xsr aggravates the catastrophic forgetting in the mlm task, it is able to significantly improve the xsr performance due to the usage of the xsr supervision. furthermore, incorporating both the mlm and xsr tasks can better preserve the performance in both tasks. gem vs. mtf from in addition, we observe that adding xsr objec3 the results of xlmr are included in the appendix. ## conclusion in this paper, we propose to preserve the crosslinguality of pre-trained language models in the fine-tuning stage. to do so, we adopt a continual learning framework, gem, to constrain the parameter learning in pre-trained multilingual models based on the mlm and xsr tasks when we finetune them to downstream tasks. experiments on the mlm and xsr tasks illustrate that our methods can better preserve the cross-lingual ability of pretrained models. furthermore, our methods achieve better performance than fine-tuning baselines for the strong multilingual models mbert and xlmr on the zero-shot cross-lingual pos and ner tasks.