Soft Language Prompts for Language Transfer (2025)

Abstract: Cross-lingual knowledge transfer, especially between high- and low-resource languages, remains challenging in natural language processing (NLP). This study offers insights for improving cross-lingual NLP applications through the combination of parameter-efficient fine-tuning methods. We systematically explore strategies for enhancing cross-lingual transfer through the incorporation of language-specific and task-specific adapters and soft prompts. We present a detailed investigation of various combinations of these methods, exploring their efficiency across 16 languages, focusing on 10 mid- and low-resource languages. We further present to our knowledge the first use of soft prompts for language transfer, a technique we call soft language prompts. Our findings demonstrate that in contrast to claims of previous work, a combination of language and task adapters does not always work best; instead, combining a soft language prompt with a task adapter outperforms most configurations in many cases.

## introduction many multilingual large language models (llms) have been developed in recent years, demonstrating promising performance on various nlp tasks across multiple languages @xcite . these models are pre-trained on extensive corpora of unlabelled data in numerous languages, allowing an adaptation to linguistic characteristics and nuances. in addition, llms are often further trained on downstream tasks in a selected subset of languages @xcite . however, only few llms focus on lowresource languages @xcite @xcite . as the number of covered languages in the model increases, the issue of the curse of multilinguality arises. this problem occurs when the llm's capacity is limited, causing languages with (3) evaluating of cross-lingual transfer capabilities on 10 mid-and low-resource languages. an alternative to language-specific tuning is cross-lingual transfer, where researchers investigate the knowledge transfer between high and lowresource languages. in cross-lingual transfer methods, an llm is trained on a downstream task in one language, most often high-resource, and evaluated in other languages @xcite . however, training only task-specific representations does not always capture the nuances of languages on which the llm has not been trained or has been trained only to a small extent. therefore, incorporating language-specific features can enhance knowledge transfer across languages. previous work has primarily investigated language and task representations by training language and task-specific adapters @xcite or by employing language arithmetics @xcite . nonetheless, other approaches that involve incorporating additional parameters to the model for language representation have not been thoroughly explored. this opens the opportunity to explore a combina-tion of language and task representations using other methods and their impact on cross-lingual knowledge transfer. to explore the utilization of language and task representations, we evaluate various configurations by combining two parameter-efficient fine-tuning (peft) methods that incorporate additional parameters into the llm, namely adapters and prompttuning. leveraging these additional language-and task-specific parameters increases the capacity of an mt0-base model and improves performance for cross-lingual transfer. we evaluate the performance of each configuration by training on six highresource languages and evaluating its effectiveness on 10 mid-and low-resource languages on four selected tasks foot_0 . our main contributions are: ## related work adapters and soft prompts. peft methods are designed to address the problem of the increasing number of trainable parameters in llms @xcite @xcite @xcite . these methods reduce the number of trained parameters and incorporate new parameters commonly used to train llms on other tasks. adapters @xcite and @xcite represent two peft methods for adapting llms to different nlp domains. adapters incorporate new parameters into the transformer architecture by including down-and up-projection layers along with residual connection, while prompt-tuning introduced trainable soft-prompts prepended to input embeddings to condition the llm's generation. limitations of multilingual llms. one major limitation of llms is catastrophic forgetting, which occurs when training the llm on a new task, causing it to partially or entirely forget previously learned knowledge for other tasks @xcite @xcite . this forgetting extends beyond task-specific knowledge to language-specific knowledge if the model is fine-tuned on a subset of the original languages @xcite . another challenge with multilingual llms is associated with the number of languages on which these llms have been pre-trained @xcite . previous research has shown that as the number of languages covered by llms increases, their performance on various nlp tasks degrades @xcite . additionally, low-resource languages are often underrepresented during pre-training, resulting in poor performance in these languages @xcite . ## methodology we propose a comprehensive study on combinations of language and task representations using adapters and soft prompts. we evaluate for the first time the capabilities of soft language prompts in a systematic manner and evaluate the performance of diverse combinations of soft prompts and adapters for cross-lingual transfer. our pipeline, illustrated in figure 1 , consists of three steps. firstly, we train language adapters and soft prompts in 16 languages on unlabelled data. secondly, along with trained language representation, we incorporate task representation into the llm's architecture and train the task representation on four tasks in six high-resource languages. finally, we evaluate the cross-lingual transfer on 10 mid-and low-resource languages and four selected tasks. in the following sections, we first give details on methods that we investigate for representing language (section 3.1) and task information (section 3.2). we then explain the combinations of soft prompts and adapters we evaluate (section 3.3). ## experiments in the following section, we describe the process of selection of models (section 4.1), languages (section 4.2) and tasks (section 4.3). ## results and analyis this section summarizes our observations based on the experiments that were conducted. we provide the overall findings in section 5.1 and then the results for particular tasks, especially question answering (section 5.2), named entity recognition (section 5.3), natural language inference (section 5.4) and check-worthy claim detection tasks (section 5.5). ## discussion based on our experiments, we summarize our observations below. prompt tuning performs better with fine-tuned models. in our preliminary model selection experiments, we found that prompt tuning does not improve the performance of pre-trained llms (e.g., mt5) trained only on unlabelled data for downstream tasks. however, prompt-tuning can enhance the performance of already fine-tuned llms on any labelled data, even if the specific tasks were not part of the prior fine-tuning. this was confirmed in our experiments with ner and check-worthy claim detection, where fine-tuned llms delivered superior results despite no previous task-specific training on these tasks. soft language prompts with task adapters perform best in many cases. our approach of combining soft language prompts with task adapters demonstrated better performance in many cases, compared to the approach of combined language and task adapters, which has been shown to be very effective in previous work. specifically, the combination of soft language prompts and task adapters is most effective on the classification tasks, achieving superior results most often. for languages with a different script (e.g., spanish and telugu), these differences were over 20%. language representations are unable to capture linguistic characteristics using small number of unlabelled data. language representations have several limitations that led to configurations without language representations performing consistently better on cross-lingual transfer to highly low-resource languages, such as telugu, urdu, and malayalam. we postulate that the reason is the small number of wikipedia articles on which the language representations were trained, rendering them unable to adequately capture sufficient linguistic characteristics. ## conclusion our study provides a comprehensive evaluation of various configurations of adapters and soft prompts for cross-lingual transfer in mid-and low-resource languages. with the systematic evaluation of task adapters, soft task prompts, and combinations of language and task representations, we identified configurations that positively affect llm's performance across different tasks and languages. our findings demonstrated that the combination of soft language prompts and task adapters emerged as an effective alternative for transferring knowledge between languages. furthermore, our findings provide valuable insights for the utilization of a combination of peft methods for cross-lingual transfer, while highlighting the need to incorporate language-specific knowledge.