Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLM s for In-Context Learning in Low-Resource Languages (2025)

Abstract: LLMs are typically trained in high-resource languages, and tasks in lower-resourced languages tend to underperform the higher-resource language counterparts for in-context learning. Despite the large body of work on prompting settings, it is still unclear how LLMs should be adapted cross-lingually specifically for in-context learning in the low-resource target languages. We perform a comprehensive study spanning five diverse target languages, three base LLMs, and seven downstream tasks spanning over 4,100 GPU training hours (9,900+ TFLOPs) across various adaptation techniques: few-shot prompting, translate-test, fine-tuning, embedding re-initialization, and instruction fine-tuning. Our results show that the few-shot prompting and translate-test settings tend to heavily outperform the gradient-based adaptation methods. To better understand this discrepancy, we design a novel metric, Valid Output Recall (VOR), and analyze model outputs to empirically attribute the degradation of these trained models to catastrophic forgetting. To the extent of our knowledge, this is the largest study done on in-context learning for low-resource languages with respect to train compute and number of adaptation techniques considered. We make all our datasets and trained models available for public use.

## introduction large language models (llms) have been at the forefront of the advancements in natural language processing (nlp), evidenced by state-ofthe-art results on numerous benchmarks @xcite . llms are pretrained with large corpora of english text data, so the best llms are primarily monolingual and english-based, leaving other languages behind. performance for tasks in non-english languages 1 https://huggingface.co/collections/christoukmaji/toukmaji-flanigan-gem25 figure 1 : we report vor scores (valid output recall, the proportion of model outputs that follow the in-context labeling scheme) vs input perplexity for each adaptation method and task, averaged across target languages, and random seeds. prompting-based methods (prompt, translate) demonstrate lower perplexity and higher vor than gradient-based methods, suggesting that gradient-based methods suffer from catastrophic forgetting, degrading both linguistic ability and instructionfollowing alignment after training. trained models lose the ability to learn in-context post-training while simultaneously performing worse on the target language. there have been several approaches to help make llms more multilingual. one approach we perform a systematic evaluation of crosslingual transfer approaches specifically for incontext learning to identify patterns for optimal transfer settings. to the extent of our knowledge, this is the largest study (with respect to tflops and gpu training hours) on cross-lingual transfer for in-context learning in low-resource languages spanning three base llms, five low-resource tar-get languages, five adaptation methods, and seven nlp tasks. our results show that the prompt and translate settings tend to heavily outperform the gradient-based adaptation methods. to better understand this discrepancy, we design valid output recall (vor), a novel metric, and analyze model outputs to empirically attribute the degradation of these trained models to catastrophic forgetting. ## related work the work of @xcite is the most similar to ours. this work evaluates multilingual adaptation of llms for in-context learning with an emphasized study on vocabulary expansion and embedding re-initialization strategies. this study finds that that vocabulary expansion and embedding reinitialization can help bridge the gap between the performance of english and non-english languages in llms. our work differs from this in that embedding re-initialization is just one of the adaptation methods that we evaluate in our study. @xcite perform a study that evaluates on a subset of our adaptation methods -namely, translate-test and few-shot prompting. the study finds that the translate-test adaptation method outperforms few-shot prompting in most languages and tasks. this work differs from ours in that the study only considers prompt-based adaptation methods and no gradient-based approaches, like ours does. @xcite conduct an analysis on few-shot prompting across many language models, tasks, and languages, but do not consider any other adaptation methods. others have benchmarked adaptation methods, but they differ from our study in that our main focus is on several low-resource languages @xcite @xcite there is a multitude of work in cross-lingual transfer, but only the papers above have a similar emphasis of benchmarking adaptation methods for in-context learning. other primary lines of work in cross-lingual transfer from monolingual llms include 1) performing further training with the target language @xcite @xcite @xcite , 2) modifying the embedding matrix and vocabulary to better fit the target language @xcite @xcite @xcite @xcite @xcite @xcite or 3) instruction-tuning with the target language @xcite or cross-lingually @xcite @xcite . these generalized cross-lingual transfer techniques are evaluated in our study. ## methods we use three different multi-billion parameter base llms -llama 2 7b @xcite , @xcite , and phi-2 (javaheripi and bubec, 2023) -for in-context learning in five diverse low-resource languages. we opt to use these models since they are primarily-monolingual, opensource, and are capable of in-context learning. we use english as our source language and evaluate on a set of five diverse low-resource target languages: hausa (hau), kinyarwanda (kin), luganda (lug), burmese (bur), and thai (tha). ## experiments we evaluate the following five methods for adapting an llm trained in a source language for prompting with a target language. ## results the results in figure 2 display few-shot prompting and translate-test adaptation methods surprisingly tend to heavily outperform the gradient-based adaptation methods. below, we provide an empirical analysis of the llms' outputs, and show this disparity can be attributed to catastrophic forgetting @xcite , which can occur in llms during continued training @xcite . in order to determine whether the performance degradation is attributed to insufficient knowledge of the target language or to task forgetting, we design valid output recall (vor), a metric to quantify an llm's ability to instruction-follow labels incontext. vor is the proportion of llm outputs of a test set that follow the same labeling scheme that was instructed and provided in-context. for example, in a binary-classification task instance where an llm is instructed to output a label @xmath0, 1} for test instance i in a test dataset with size n and the llm output ŷi , then v @xmath1 the vor is compared with the perplexity of the inputs. intuitively, this isolates the evaluation of an llm's task alignment and instruction-following ability from its linguistic ability. in figure 1 , we observe that gradient-based adaptation methods have both higher perplexities and lower vor compared to few-shot prompting. these results empirically verify that the trained models are losing the ability to learn incontext post-training while simultaneously performing worse on the target language. this suggests that the gradient-based methods in our training environment suffer from catastrophic forgetting 2 https://fasttext.cc/docs/en/pretrained-vectors.html since both linguistic knowledge and task alignment deteriorate. ## conclusion this work provides the largest comprehensive study on adapting primarily-english llms to lowresource languages for in-context learning. five adaptation methods are evaluated across three base llms using five diverse target languages on seven downstream tasks. few-shot prompting and translate-test worked the best in nearly all cases, but there is no trend between which of the two works best. we design a novel metric, valid output recall (vor), and provide an empirical analysis on llm outputs to show that models adapted with gradient-based methods degraded due to catastrophic forgetting.