Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More (2024)

Abstract: Soft Prompt Tuning (SPT) is a parameter-efficient method for adapting pre-trained language models (PLMs) to specific tasks by inserting learnable embeddings, or soft prompts, at the input layer of the PLM, without modifying its parameters. This paper investigates the potential of SPT for cross-lingual transfer. Unlike previous studies on SPT for cross-lingual transfer that often fine-tune both the soft prompt and the model parameters, we adhere to the original intent of SPT by keeping the model parameters frozen and only training the soft prompt. This does not only reduce the computational cost and storage overhead of full-model fine-tuning, but we also demonstrate that this very parameter efficiency intrinsic to SPT can enhance cross-lingual transfer performance to linguistically distant languages. Moreover, we explore how different factors related to the prompt, such as the length or its reparameterization, affect cross-lingual transfer performance.

## introduction fine-tuning pre-trained language models (plms) on task-specific labeled data requires large amounts of computational resources and may cause catastrophic forgetting of the pre-trained knowledge @xcite . in multilingual settings, this may lead to poor cross-lingual transfer performance @xcite . to address these challenges, @xcite introduced soft prompt tuning (spt), a method that inserts learnable embeddings, or soft prompts, at the plm's input layer. the plm then makes predictions using the output of its pre-trained language modeling head. the key advantage of spt lies in its ability to leverage the pre-existing knowledge within plms while reducing the reliance on extensive task-specific fine-tuning. spt has been shown to achieve remarkable results in various monolingual downstream tasks, especially in few-shot settings. motivated by this success, some recent works have also explored the use of spt for cross-lingual transfer, where the goal is to leverage a multilingual language model (mllm) to transfer knowledge from a high-resource to a low-resource language. however, these works have not fully exploited the potential of spt. some have appended a newly initialized classifier to the model @xcite , hindering the suitability of spt for few-shot learning. others have fine-tuned the entire model along with the prompt @xcite , which reduces the computational efficiency of spt. this is especially problematic given the growing size of state-of-the-art language models. therefore, we explore the impact on spt's cross-lingual transfer performance when adhering to the original methodology of @xcite , which involves fine-tuning only the soft prompt while keeping all model parameters frozen. specifically, this paper contributes to the field of cross-lingual spt by: in this study, we conduct experiments on a topic classification dataset in 52 different languages and using 4 different models in few-shot settings. we believe that our findings can improve the existing methods that aim to enhance cross-lingual spt, particularly in the context of current state-of-the-art models with billions of parameters where parameter efficiency is crucial. ## experimental setup besides adhering to the original setup of spt, enabling parameter-efficient and data-efficient training, our study also sets itself apart in its objectives from the existing literature. rather than simply demonstrating superior cross-lingual transfer performance of spt over fine-tuning, our research aims to show that the minimal impact on the mllm's representation space not only generally enhances transfer performance but is particularly effective for linguistically distant languages. we provide more specific details on our experimental setup in appendix a. ## results we provide the full results across all models and languages in appendix d. the results reveal that model freezing not only boosts cross-lingual transfer performance (figure 2 ) but additionally is a step towards closing the transfer gap between linguistically distant and similar languages. this can be seen in in other words, by fine-tuning fewer parameters, cross-lingual transfer, especially to linguistically distant languages, is enhanced. this improvement over full-model fine-tuning may be attributed to the reduced impact on the mllm's representation space during fine-tuning @xcite . ing spt also provides parameter efficiency as finetuning is restricted to a number of soft prompt tokens, resulting in only a few thousand parameters in total. this is less than 0.01% of the parameters fine-tuned in previous studies @xcite . for illustration, the storage requirement for a copy of the xglm-1.7b model is approximately 3.2 gb, whereas a prompt needs less than 100kb. with respect to training duration, our observations indicate that the time required for training only the soft prompt is less than half compared to when training all model parameters. this benefit becomes even more pronounced when considering the increasing sizes of state-of-the-art models. ## discussion previous works on spt for cross-lingual transfer in few-shot settings suffers from two major drawbacks: 1) fine-tuning all model parameters along with the prompt reduces the computational efficiency of spt; 2) a bias towards target languages that are linguistically closer to the source language. our study tackles these issues by showing that by simply keeping model parameters frozen during spt, we can make progress in addressing both these challenges. through our experiments, which covered a wider and more diverse range of languages than prior work on cross-lingual spt, we observed intriguing effects of non-linguistic variables (such as model freezing, prompt length, and reparameterization) on the transfer performance for individual languages. additionally, our results reveal languagespecific differences that invite further inquiry into the possibility of tailoring prompts to the target language (e.g., applying prompt reparameterization or not depending on the linguistic distance between the target language family and the source language) rather than using a single prompt for universal transfer across languages. we believe that our findings will benefit future work on crosslingual spt and potentially improve the existing techniques @xcite , becoming more valuable as we adopt larger state-of-the-art models with billion-and trillion-scale parameters @xcite . most target languages, particularly for those that are linguistically more distant. furthermore, we found that shorter prompts enhance spt's crosslingual transfer performance, and that some target language families benefit from reparameterization while others are adversely affected by it.