Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models (2022)

Abstract: Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks. Previously, for zero-shot cross-lingual evaluation, pre-trained models are only fine-tuned on English data and tested on a variety of target languages. In this paper, we do cross-lingualevaluation on various NLU tasks (sentence classification, sequence labeling, question answering) using prompt-tuning and compare it with fine-tuning. The results show that prompt tuning achieves much better cross-lingual transfer than fine-tuning across datasets, with only 0.1% to 0.3% tuned parameters. Additionally, we demonstrate through the analysis that prompt tuning can have better cross-lingual transfer-ability of representations on downstream tasks with better aligned decision boundaries.

## introduction large multilingual language models @xcite @xcite show surprisingly impressive zero-shot crosslingual transfer on nlp tasks, even though they are only trained from monolingual corpora. recently, large-scale benchmarks such as xtreme @xcite and xglue @xcite ) are introduced for cross-lingual evaluation. in a cross-lingual transfer setting, models are only fine-tuned on the task-specific annotations in one language and evaluated in other languages. during fine-tuning, pre-trained language models are used for initialization and the entire model parameters are tuned on downstream tasks. while fine-tuning obtains strong performance, it is inefficient. also as shown in @xcite , the cross-lingual transfer gap between the performance on the english test set and all other languages is large even with the best baseline xlm-r @xcite . recently, prompt tuning, where only a small amount of additional parameters (i.e. prompts) is added and tuned, but the original model is kept frozen. much fewer parameters or no parameters are tuned and thus the training is a lot more efficient. prompt tuning still performs worse than fine-tuning in lots of nlp tasks @xcite @xcite . more recently, @xcite ; @xcite ; @xcite indicate prompt tuning is competitive with fine tuning on some of the nlu tasks. language model capacity (e.g., 10 billion parameters) is a key ingredient for these approaches to succeed. more recently, @xcite shows prompt tuning can also be comparable on several hard monolingual sequence labeling tasks such as extractive question answers. in this paper, we aim to investigate the effect of prompt tuning in cross-lingual tasks.we freeze the entire multilingual language model and tune task prompts on the english training set for downstream tasks (sentence classification, structure prediction, question answering). even with medium size multilingual language model (less than 1 billion parameters), prompt tuning achieves much higher performance than fine-tuning on various nlu tasks. according to the analysis results, prompt tuning does fewer changes to sentence representations than fine-tuning and keeps good cross-lingual sentence representations. we also find that the decision boundaries of different language sentence representations after prompt tuning on english data are almost aligned well. however, these decision boundaries of different languages after fine-tuning are a large difference. these aligned decision boundaries can lead to stronger cross-lingual transfer. this work sheds light on the strong cross-lingual ability of prompt tuning. our results suggest prompt tuning is better than fine-tuning on crosslingual transfer. our contributions are summarized as follows: we show that prompt tuning can per-form much better as compared to fine-tuning for cross-lingual transfer; we also show prompt tuning works better in the case of the cross-lingual transfer due to the relative small robust changes it brings to the originally learned representations. we investigate prompt tuning on cross-lingual understanding on a pre-trained multilingual language model. the framework is shown in figure 1 . our setting is similar to li and liang (2021); @xcite . the continuous prompts are added as prefix tokens and tuned during learning. in the implementation, the prompts are operated as past keys and values in each transformer layer. each transformer layer has separated prompts. these continuous prompts are optimized, but multilingual language model parameters are frozen. we perform experiments on four datasets included in xtreme: cross-lingual natural language inference (xnli; @xcite , cross-lingual adversarial dataset for paraphrase identification (paws-x; @xcite , part-of-speech tagging on the universal dependencies (ud-pos; @xcite , cross-lingual question answering on xquad @xcite and tydiqa-goldp @xcite . three categories of downstream tasks are included: (1) sentence classification); (2) structure prediction; (3) question answering. ## results tuned parameter sizes comparison for the prompt tuning test results in overall results cross-lingual transfer gap according to the above result, on average, prompt tuning achieves better performance than fine tuning. ## analysis in order to perform some analysis on prompt tuning and fine tuning, we select 1000 samples for each language (en, de, es, fr, ja, ko, zh ) from paws-x @xcite dataset. for each english language sample in our selections, there is a human translated sample from the other six languages. 2 figure 2 shows t-sne visualization of sample representations from frozen multilingual language model xlm-r. samples' representations are clustered well respect to languages, however, there is weak correlation with labels. ## conclusion in this work, we compared prompt tuning and fine tuning on cross-lingual understanding with multilingual languages models, finding that prompt tuning achieves a better performance. this suggest that it is promissing to use prompt tuning on cross-lingual transfer.