Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024)

Abstract: Multilingual pre-trained language models (mPLMs) have demonstrated notable effectiveness in zero-shot cross-lingual transfer tasks. Specifically, they can be fine-tuned solely on tasks in the source language and subsequently applied to tasks in the target language. However, for low-resource languages unseen during pre-training, relying solely on zero-shot language transfer often yields sub-optimal results. One common strategy is to continue training PLMs using masked language modeling objectives on the target language. Nonetheless, this approach can be inefficient due to the need to adjust all parameters for language adaptation. In this paper, we propose a more efficient solution: soft-prompt tuning for language adaptation. Our experiments demonstrate that with carefully designed prompts, soft-prompt tuning enables mPLMs to achieve effective zero-shot cross-lingual transfer to downstream tasks in previously unseen languages. Notably, we found that prompt tuning outperforms continuously trained baselines on two text classification benchmarks, encompassing 20 low-resource languages while utilizing a mere 0.28% of the tuned parameters. These results underscore the superior adaptability of mPLMs to previously unseen languages afforded by soft-prompt tuning compared to traditional fine-tuning methods.

## introduction the issue of gathering sufficient annotated data for downstream tasks becomes particularly challenging for low-resource languages, leading to extensive research on zero-shot cross-lingual transfer. this basic approach involves fine-tuning a model using annotated data in a source language and evaluating its performance directly on data in a target language. multilingual pre-trained language 1 the source code is publicly available at https://github.com/miulab/unseenadapt . models (mplms) have demonstrated remarkable success in zero-shot cross-lingual transfer across various nlp tasks @xcite @xcite . despite significant progress, it is impractical for mplms to cover all languages globally due to two primary reasons. first, including additional languages increases the vocabulary size, posing challenges in managing and maintaining. second, there exists a trade-off between the number of languages covered and the model's capacity @xcite . consequently, when mplms encounter an unseen target language, the performance of zero-shot crosslingual transfer often falls short of expectations. therefore, the task of adapting mplms to unseen languages has emerged as a crucial concern. a naive approach involves continuing to train plms using the masked language modeling (mlm) objective on unlabeled text in the unseen target language, aiming to leverage the language-specific capabilities @xcite . however, with the increasing number of parameters in plms, tuning the entire model becomes more resource-intensive and inefficient. additionally, in the case of lowresource unseen languages, the limited availability of data for continued training may compromise the model's generalizability if full parameters are tuned. prompting has emerged as a solution to avoid the overhead associated with fine-tuning by leveraging natural language prompts to query pre-trained source seen unseen @xcite target target fine-tuning 89.01 79.2 42.58 prompt-tuning 88.94 79.9 @xcite building upon prior research and empirical findings, our objective is to adapt mplms to previously unseen low-resource languages and achieve effective cross-lingual transfer. in this challenging scenario, the model lacks prior exposure to the target language, necessitating the development of novel strategies to accomplish our objective. at the same time, the adaptation process needs to be parameterefficient, as tuning all parameters becomes impractical with the increasing scale of mplms. in this paper, we investigate the effectiveness of soft-prompt tuning for adapting mplms to unseen languages. specifically, we keep all parameters in the mplm frozen and solely focus on tuning the prefix soft prompts within the overall framework. our results demonstrate that incorporating soft prompts significantly enhances the mplm's ability to generalize to new languages, leading to a superior zero-shot cross-lingual performance on downstream tasks compared to fine-tuning. figure 1 illustrates a simplified example of the zero-shot cross-lingual transfer process in our experiments. in summary, our contributions can be summarized in 3-fold: adapter applying adapters @xcite is one of the representative strategies to achieve parameter-efficient fine-tuning. when performing adapter tuning, the original model's weight is untouched and only the newly added adapter layers are tuned. @xcite utilize adapter to achieve transfer learning to multiple downstream tasks and attain near fine-tuning performance while having significant trainable-parameter reduction. @xcite introduced mad-x, a modular framework designed to perform parameterefficient cross-lingual transfer through the use of adapters. their framework trains a language adapter (la) for each source and target language using mlm, and a task adapter (ta) for the task from the source languages, incorporating the corresponding la during training. during inference, their method achieves impressive task performance on the target language by stacking the task adapter with the target language adapter, showcasing its great potential for zero-shot cross-lingual transfer. prompt tuning prompt tuning emerged as a strategy to leverage knowledge from pre-trained models and avoid the overhead associated with fine-tuning. for instance, @xcite use trainable continuous prompts as input tokens for the pre-trained model and show that prompt-tuning becomes more competitive with scale. similarly, @xcite incorporate prefix embeddings into each layer of the pre-trained model, resulting in more direct impacts on model predictions and achieving performance on par with fine-tuning across various model scales. @xcite demonstrated the potential of prompt-tuning for zero-shot cross-lingual transfer. however, their experiments primarily concentrated on cross-lingual transfer among languages that were already included in the training data of the mplms. in contrast, our objective is to adapt mplms to previously unseen and low-resource languages, which poses a more challenging scenario. ## soft-prompt language adaptation in this paper, our focus lies in investigating the effectiveness of soft-prompt tuning in adapting mplms to previously unseen low-resource target languages. we aim to evaluate its performance in zero-shot cross-lingual transfer across downstream tasks. the experimental procedure comprises two stages, as illustrated in figure 2 . ## conclusion in this paper, our objective is to adapt multilingual pre-trained language models(mplms) to previously unseen languages and enhance their crosslingual transferability. initially, we demonstrate the necessity of adapting mplms to new languages by comparing their performance in cross-lingual transfer between known and unknown languages. subsequently, we propose using soft-prompt tuning to accomplish efficient language adaptation and effective zero-shot cross-lingual transfer to downstream tasks. our primary findings indicate that employing soft-prompt tuning for language adaptation can yield comparable performance to baseline methods such as fine-tuning and adapter-based approaches, while utilizing significantly fewer tunable parameters. furthermore, we conduct various experiments to delve deeper into soft-prompt-based language adaptation, examining factors such as data and prompt settings. our experiments reveal that the soft prompts in lower layers function as a language-dependent component while tuning only the soft prompts in upper layers for downstream tasks leads to improved results. additionally, softprompt-based language adaptation demonstrates consistent performance even with limited amounts of unlabeled target language data and few-shot downstream data. these findings collectively affirm the superior efficiency of soft-prompt-based language adaptation, both in terms of the number of trainable parameters and the volume of data.