Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation (2025)

Abstract: With the release of new large language models (LLMs) like Llama and Mistral, zero-shot cross-lingual transfer has become increasingly feasible due to their multilingual pretraining and strong generalization capabilities. However, adapting these decoder-only LLMs to new tasks across languages remains challenging. While parameter-efficient fine-tuning (PeFT) techniques like Low-Rank Adaptation (LoRA) are widely used, prefix-based techniques such as soft prompt tuning, prefix tuning, and Llama Adapter are less explored, especially for zero-shot transfer in decoder-only models. We present a comprehensive study of three prefix-based methods for zero-shot cross-lingual transfer from English to 35+ high- and low-resource languages. Our analysis further explores transfer across linguistic families and scripts, as well as the impact of scaling model sizes from 1B to 24B. With Llama 3.1 8B, prefix methods outperform LoRA-baselines by up to 6% on the Belebele benchmark. Similar improvements were observed with Mistral v0.3 7B as well. Despite using only 1.23M learning parameters with prefix tuning, we achieve consistent improvements across diverse benchmarks. These findings highlight the potential of prefix-based techniques as an effective and scalable alternative to LoRA, particularly in low-resource multilingual settings.

## introduction large language models (llms) exhibit strong multilingual and zero-shot generalization abilities due to exposure to diverse pretraining data. nonetheless, cross-lingual transfer remains challenging given the linguistic diversity and complexity of adapting large models efficiently without significant computational overhead. to address the high computational and memory costs of full model finetuning, recent advances in parameter-efficient finetuning @xcite techniques focus on updating only a small subset of model parameters while keeping the majority of the pretrained weights frozen. this design significantly reduces the adaptation cost and makes large-scale models more practical for multilingual and domain-specific applications. methods such as low-rank adaptation (lora) @xcite and instruction-tuned adapters have shown promising results in efficiently tailoring models to new tasks without requiring extensive resources. among the various peft techniques, prefix-based approaches like soft prompting @xcite and prefix tuning @xcite are particularly compelling because they introduce learnable components either at the input or within the transformer stack, enabling flexible task adaptation without altering the underlying architecture of the model. while these prefix-based techniques have been shown to be effective in monolingual scenarios and task-specific settings, their potential in facilitating zero-shot cross-lingual transfer is under-explored. this is especially relevant for decoder-only llms, which are increasingly being deployed in multilingual environments. unlike encoder-decoder models that have been more thoroughly studied for transfer across languages, decoder-only models present unique challenges due to their reliance on autoregressive decoding. understanding whether prefix-based peft methods can enhance zero-shot cross-lingual performance in such models has not been previously studied in detail. in this work, we provide the first systematic study of prefix-based peft methods for zero-shot cross-lingual transfer in decoder-only llms. our contributions can be summarized as follows: together, our findings position prefix-based adaptation as a lightweight yet powerful strategy for cross-lingual and reasoning-oriented applications, particularly in resource-constrained multilingual settings. ## related work cross-lingual transfer is a key challenge in multilingual nlp. it is traditionally tackled through full fine-tuning of multilingual models. however, with large decoder-only llms like llama and mistral, full fine-tuning is costly, leading to peft approaches. lora @xcite introduces lowrank trainable matrices into frozen weights to reduce training overhead. alternatively, prefix-based methods either add learnable tokens at the input layer @xcite or to attention keys and values at each layer (li and liang, 2021), enabling efficient task adaptation. soft prompt tuning has been extensively studied for cross-lingual transfer in encoder and encoderdecoder models, particularly in classification tasks. for instance, @xcite demonstrated that soft prompts can generalize better across languages with fewer parameters, following the "less is more" principle. similarly, @xcite utilized multilingual verbalizers and contrastive label smoothing to further enhance cross-lingual classification. recent work such as @xcite introduced language-specific soft prompts specifically designed for transfer learning, showing that combining language-specific and taskspecific prompts improves generalization. however, these prior works predominantly used multi-lingual encoder-only and encoder-decoder models, and appended prefix tokens only to the input. as soft prompts have several limitations in effectively adapting models to new tasks, prefix tuning emerged as a promising approach. cross-lingual alignment through prompt-based pretraining, as proposed by @xcite , further improved intent classification and slot-filling performance but it is not a zero-shot setting (as in our work). a recent variant of prefix tuning is llama adapter @xcite that introduced zero-initialized attention mechanisms for efficient prefix training and achieved strong instruction-following capabilities; however, they did not evaluate on any multilingual benchmarks. a related line of work has focused on extending prefix tuning to instance-specific adaptation based on the input prompt for improved model performance @xcite @xcite . few comparative studies have examined parameter-efficient tuning for multilingual settings, and most have been restricted to encoder-only models or small decoder-only models with only a few million parameters. for instance, (zhao and schütze, 2021) systematically compared discrete prompting, soft prompting, and fine-tuning on the few-shot multilingual nli task using xlm-roberta-base. similarly, @xcite compared prompt tuning with fine-tuning across diverse nlu tasks on xlm-r and mbert. @xcite evaluate prefix tuning on the encoder-only xlm-r model and showed its effectiveness over full fine-tuning in zero-shot cross-lingual transfer. @xcite investigated a decoder-based multilingual model (xglm), but their analysis was limited to a single small model. they showed that prompt tuning can sometimes surpass fine-tuning, particularly for low-resource languages, although performance remained highly sensitive to the underlying tokenization scheme. our work significantly extends their analysis to large decoder-only llms and presents a comprehensive comparison of multiple prefix-based methods, including soft prompts, prefix tuning, and llama adapter. ## methodology low-rank adaptation (lora). lora @xcite is a parametric fine-tuning technique that has become one of the most popular approaches to enable cross-lingual transfer in llms. it introduces trainable low-rank matrices, typically in the query, key, and value projections, while keeping the base model frozen. these learned matrices are added to the original weights during inference. unlike prefix-based methods, lora directly modifies the model parameters. a standard cross-lingual transfer setup involves fine-tuning the model using lora on task-specific english data and evaluating it on the target language of interest. formally, let @xmath0.g., w q , w k , w v ). instead of updating w directly, lora parameterizes the weight update as a product of two low-rank matrices: given an input hidden state @xmath1, the output of the adapted projection layer is computed as: here, only a and b are trainable, while w remains frozen. this formulation enables efficient fine-tuning by reducing the number of trainable parameters and allowing task-specific adaptation without updating the full weight matrices. prefix tuning. given an llm, prefix tuning (li and liang, 2021) introduces a set of learnable prefix tokens to all layers of the transformer. in our implementation, we only append the learnable prefixes to the final l layers of the transformer. the main intuition is that these prefix tokens act as additional context vectors that the model can attend to. these vectors guide the model toward task-specific behavior, while the pretrained parameters of the llm remain frozen. formally, let p @xmath2, where k is the number of prefix tokens and d is the embedding dimension. we consider the computation for the (m + 1)-th token, denoted by t @xmath3.the layer's input hidden states (including the current token) are represented as h @xmath4. each attention head operates on these hidden states using projection matrices w q , w k , w @xmath5. the query vector corresponding to the current token t l is computed using the frozen projection matrix w q : the keys and values corresponding to the input sequence (h l ) are also computed using the frozen projection matrices w k and w v : the key idea of prefix tuning is the concatenation of the learnable prefix parameters with keys and values derived from the input. k l and v l are expanded matrices encompassing both the learned prefixes and the input sequence. the attention scores are obtained by comparing the query q l against the concatenated keys k l : the attention distribution is computed by applying the softmax function, which weights the contributions of both the prefix and the input tokens: finally, as is typically done in transformer models, the attended output representation at layer l is computed as a weighted sum of the concatenated values v l , followed by an output projection: llama adapter. the llama adapter @xcite builds upon the principles of prefix tuning but introduces an important modification to stabilize training in large-scale llms. specifically, it replaces the standard attention mechanism with a zero-initialized variant. this mitigates instabilities that often arise from randomly initialized prefix tokens in the early stages of fine-tuning. to further enhance stability, a learnable gating mechanism is introduced, allowing the model to gradually scale the influence of prefix tokens during optimization. the gated attention scores are given by: soft prompts. soft prompts @xcite involve prepending learnable continuous embeddings to the input, serving a similar goal as manual prompts. however, instead of manually selecting discrete prompts, soft prompting optimizes a continuous set of embeddings that serve as the prompt. this allows the model to learn how to best steer its behavior through gradient-based updates to the soft prompts. let @xmath6, where k denotes the number of prompt tokens and d is the hidden dimension. given an input sequence t , the modified input t is obtained by prepending the soft prompts: with s updated via gradient-based optimization during fine-tuning. unlike prefix tuning, which injects key-value pairs at every transformer layer, soft prompting only modifies the input embeddings. ## analysis and ablations comparison with lora fine-tuning. tables 1 and 2 ( and tables 14 and 15 in the appendix) shows the performance of llama 3.1 and mistral v0.3 models across various tuning strategies, including lora, soft prompt tuning, prefix tuning, and llama adapters on the xnli and xquad datasets. to ensure fair comparisons, the number of trainable parameters in lora was matched with those of the prompt-based methods by setting @xmath7. the results show that prefixbased methods consistently outperform lora on both llama 3.1 8b and mistral v0.3 7b with english as the source language. this highlights the ability of prefix-based tuning for effective multilingual adaptation, even with as little as 1.23m model parameters being trained. we observe consistent improvements from prefix tuning across all benchmarks. using llama 3.1 (8b), prefix tuning achieves up to 28% higher accuracy on xnli, 13% higher f1 on xquad, and 18% higher accuracy on belebele compared to the base model. moreover, it provides additional gains of up to 4-6% over lora, as shown in tables 1 , 2 , 4a, and 4b. similar trends are observed for mistral, with consistent improvements reported in tables , ## conclusion we show that prefix-based adaptation methods are a practical and efficient mechanism for cross-lingual transfer in decoder-only llms. methods like soft prompting, prefix-tuning, and llama adapters introduce learnable prefixes at different layers, while using relatively small numbers of trainable parameters. this leads to highly efficient, task-specific cross-lingual learning. crucially, this performance was achieved using only english training data. we hypothesize this success stems from learning language-agnostic behaviors. by adding context vectors while keeping the base model frozen, these methods preserve the llm's inherent multilingual capabilities. in contrast, methods that alter full model weights (e.g., full fine-tuning and lora) suffer from catastrophic forgetting when adapted monolingually, degrading performance in unseen languages. these findings advocate for prefix-based adaptation as a robust strategy for zero-shot cross-lingual transfer.