Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast (2023)

Abstract: Existing research has shown that a multilingual pre-trained language model fine-tuned with one (source) language also performs well on downstream tasks for non-source languages, even though no fine-tuning is done on these languages. However, there is a clear gap between the performance of the source language and that of the non-source languages. This paper analyzes the fine-tuning process, discovers when the performance gap changes and identifies which network weights affect the overall performance most. Additionally, the paper seeks to answer to what extent the gap can be reduced by reducing forgetting. Based on the analysis results, a method named Fine-tuning slow and fast with four training policies is proposed to address these issues. Experimental results show the proposed method outperforms baselines by a clear margin.

## introduction multilingual pre-trained language models (lms), such as mbert @xcite and xlm-r @xcite have shown strong zero-shot cross-lingual transfer capabilities. such a model f is usually pre-trained with unlabeled corpora d in multiple languages s to enable the model to learn cross-lingual knowledge h pre cross . to adapt to a downstream task, the pre-trained lm f is typically fine-tuned with a supervised dataset d ŝ of the downstream task t in one source language ŝ ∈ s due to data scarcity in other languages. when the fine-tuned model f is applied to the test set of the same task in the source language ŝ, it achieves strong performance p ŝ. interestingly, when f is applied to non-source languages, it can also achieve good performance @xcite . we denote the average performance on the test sets of other languages than ŝ as p s/ŝ . however, the * work done during internship at microsoft research asia † corresponding author figure 1 : the performance gap p ŝ -p s/ŝ every hundred updates on the xnli dataset. 'original performance gap' means that we directly fine-tune the model, and 'fine-tuning slow'/'fine-tuning fast'/'our method' means that we use the fine-tuning slow algorithm/finetuning fast algorithm/the combination of both algorithms respectively to fine-tune the model. ## method this section proposes a method to avoid the forgetting of cross-lingual knowledge h pre cross (stability) and to encourage the forgetting of task knowledge for the pre-training task h pre task and learn new task's knowledge h new task (plasticity). the core is to set different learning rate policies for the weights of the model based on both the layer's location and the training phase. ## experiment we now use three downstream tasks: named entity recognition (ner), question answering (qa), and natural language inference (nli), to experimentally evaluate the performance of our proposed fine-tuning slow and fast algorithm under the zero-shot and few-shot settings. ## conclusion this paper first analyzed when the performance gap arises and where the important cross-lingual knowl-edge is and reduced the lower bound of the performance gap by avoiding forgetting cross-lingual knowledge. based on our analysis, a novel method is proposed to control the forgetting effect in finetuning a multi-lingual pre-trained model. we verify its effectiveness over multiple datasets and settings.