P rompt R efine: Enhancing Few-Shot Performance on Low-Resource I ndic Languages with Example Selection from related Example Banks (2025)

Abstract: Large Language Models (LLMs) have recently demonstrated impressive few-shot learning capabilities through in-context learning (ICL). However, ICL performance is highly dependent on the choice of few-shot demonstrations, making the selection of the most optimal examples a persistent research challenge. This issue is further amplified in low-resource Indic languages, where the scarcity of ground-truth data complicates the selection process. In this work, we propose PromptRefine, a novel Alternating Minimization approach for example selection that improves ICL performance on low-resource Indic languages. PromptRefine leverages auxiliary example banks from related high-resource Indic languages and employs multi-task learning techniques to align language-specific retrievers, enabling effective cross-language retrieval. Additionally, we incorporate diversity in the selected examples to enhance generalization and reduce bias. Through comprehensive evaluations on four text generation tasks—Cross-Lingual Question Answering, Multilingual Question Answering, Machine Translation, and Cross-Lingual Summarization using state-of-the-art LLMs such as LLAMA-3.1-8B, LLAMA-2-7B, Qwen-2-7B, and Qwen-2.5-7B, we demonstrate that PromptRefine significantly outperforms existing frameworks for retrieving examples.

## introduction large language models (llms) have recently made remarkable progress, demonstrating humanlevel performance across a wide range of tasks @xcite . however, despite these advancements, most llms, such as llama-3 @xcite , llama-2 @xcite , and qwen @xcite , are predominantly pre-trained on english texts, leading to significant performance disparities when applied to low-resource, non-english languages @xcite . the scarcity of groundtruth paired data in many low-resource languages makes text generation particularly challenging, as fine-tuning llms becomes infeasible in such settings. this issue is especially pronounced in lesserknown indic languages such as tibetan which has only around 5000 wikipedia articles compared to 6.6m+ wikipedia articles in english, despite having approximately 6m speakers. in this work, we focus on downstream generation tasks with lowresource indic languages, a critical challenge for making llms more widely accessible. when downstream tasks have limited labeled data, few-shot learning or in-context learning (icl) has emerged as a powerful and practical approach for text generation @xcite @xcite @xcite . icl operates by providing the llm with a prompt that consists of task-specific instructions and a set of input-output examples (demonstrations) to guide the model's output generation for a specific input query. while icl is computationally efficient (it requires no parameter updates), it faces two critical challenges when labeled data is scarce, particularly for low-resource indic languages. proposed approach. in this work, we focus on an icl approach for enhancing llm performance in low-resource indic languages. specifically, our goal is to identify an optimal subset of guiding examples to be inserted into the prompt, without modifying the llm parameters. rather, we train retrievers to extract a near optimal set of examples (demonstrations) based on the input query. to this end, we propose a novel trainable approach promptrefine that combines examples from auxiliary example banks associated with the related task. while promptrefine is broadly applicable across different language families and even lowresource task clusters, in this study we focus on indic languages @xcite , to enhance performance specifically on low-resource indic languages through improved example selection. given a specific input text generation task in a low-resource indic language, we use the following set of key ideas to enhance model performance (see algorithm 1): 1) first, we utilize example banks from closely-related relatively high-resource foot_0 indic languages to provide relevant guidance, improving llm performance on low-resource tasks. therefore, the first step in our algorithmic approach given an input low-resource task is to select relevant high-resource languages (see algorithm 2) with associated example banks. 2) second, merging these example banks is non-trivial, as each language-specific retriever is trained on a unique representation space. to address this, we adapt data-scarce multi-task learning techniques @xcite to align the individual retrievers into a shared representation space, enabling cross-language retrieval. 3) third, we incorporate diversity in the selected examples to improve generalization and reduce bias during generation. to empirically validate the effectiveness of promptrefine, we evaluate its performance across four distinct text generation tasks on a subset of languages outlined in @xcite : cross-lingual question answering (xorqa-in), multilingual question-answering (xquad-in), ma-chine translation (flores-in), and cross-lingual summarization (crosssum-in), using recent llms such as llama-3.1-8b @xcite , llama-2-7b @xcite , qwen-2-7b @xcite , and qwen-2.5-7b @xcite . our results demonstrate that promptre-fine significantly improves generation performance in all tasks compared to baseline approaches for example selection. specifically, using llama-3.1-8b as the llm, promptrefine achieves a token-f1 improvement of +16.07 and +8.26 over zero-shot prompting and the current state-ofthe-art retriever @xcite , respectively, in the cross-lingual qa task. similarly, using qwen-2-7b, we observe an improvement in chrf1 of up to +7.77 on the machine translation task compared to the baseline of selecting semantically similar examples. to ensure a comprehensive evaluation, we also test promptrefine on proprietary llms such as @xcite , where promptrefine outperforms baseline retrievers on translation task, aligning with our previous findings. ## related works in-context learning (icl). first introduced in brown (2020), icl has emerged as a powerful approach that enables large language models (llms) to "learn by analogy" by providing a few input-output examples as demonstrations, without requiring any update to model parameters. in recent years, a plethora of studies have provided insights on the underlying mechanism of icl. @xcite suggested that, by conditioning on a prompt, the task of predicting the next word becomes linearly separable, while @xcite observed that for icl, the model infers a shared latent concept between the provided examples. a study pointed out that models do not rely as heavily on the provided input-output mappings as previously thought, indicating more nuanced learning dynamics in icl @xcite . @xcite ; @xcite ; @xcite showed that the in-context learning ability of llms can be improved through self-supervised or supervised training. a group of studies have also explored to understand the factors affecting icl @xcite @xcite @xcite ) and the underlying working mechanism of icl @xcite @xcite example selection for icl. despite the tremendous success, the performance of icl is sensitive to specific settings, including the prompt template @xcite @xcite , the selection @xcite @xcite and order of demonstration examples @xcite , and other factors @xcite ). existing literature on example selection can be broadly categorized into two major groups: (1) unsupervised methods depending on pre-defined metrics. @xcite proposed selecting the closest neighbors as demonstrations, while @xcite observed that electing diverse demonstrations improves compositional generalization in icl. @xcite ; @xcite ; li and qiu (2023) explored using the output distributions of language models to select few-shot examples. (2) on the other hand, the other group of studies @xcite @xcite proposed fine-tuning a retriever model to select few-shot demonstrations. recent works have also explored using reinforcement learning approaches @xcite and chainof-thought reasoning @xcite for example selection. in this study, we propose an alternating minimization approach for selecting the optimal set of in-context examples to enhance llm performance on low-resource indic languages. ## proposed framework problem setup. despite the growing focus on evaluating the multilingual capabilities of large language models (llms), there remains a substantial performance gap between high-resource languages and those with limited web resources @xcite . in this study, we aim to address this gap by enhancing the few-shot performance of low-resource indic languages. to low/mid-resource languages: bodo, odia, santali, rajasthani, manipuri, awadhi, marwari, and maithili. auxiliary high-resource languages: bengali, hindi, marathi, gujarati, kannada, malayalam, tamil, telugu, urdu. additionally, we assume the availability of data from relatively high-resource indic languages, which we refer to as the auxiliary dataset. this assumption is justified by the fact that these languages have relatively ample web-text resources @xcite . ## discussion note, ablation studies on threshold parameter δ for selecting related languages and number of icl examples are deferred to appendix c. why is choosing a closely related auxiliary example bank important? in section 4.1.1, we discussed the approach of selecting a closely related high-resource indic language as an auxiliary example bank for a target low-resource language. to highlight importance of selecting the right set of related languages, we present ablation studies in this section. we evaluate three setups: (1) our method (alg. 2) of selecting the most closely related language as the auxiliary example bank, (2) selecting a random language, and (3) selecting the most unrelated language. we show the few-shot performance on cross-lingual qa task in figure 1 , with retrievers trained under each setup. our findings show that: (1) as expected, using the most closely related language as the auxiliary example bank yields the best performance, as it provides relevant guidance to the llm, and (2) selecting a random or unrelated language results in little to no improvement, with performance remaining close to the relevancebased epr baseline @xcite that only uses target-language specific data. ## conlusion the pre-training data for state-of-the-art llms @xcite predominantly comprises english text, resulting in suboptimal performance on low-resource indic languages @xcite . to address this, we propose a novel alternating minimization approach promptrefine for example selection that improves icl performance on low-resource indic languages. our approach follows a threestep framework: (1) identifying closely related high-resource indic languages and utilizing their example banks, (2) iteratively refining retriever embeddings, and (3) employing diversity-based finetuning to rank subsets of in-context examples for a given input query. comprehensive testing across various tasks and llms demonstrates that promptrefine significantly enhances text generation quality on low-resource indic languages. ## limitations in section 5.1, we demonstrated the effectiveness of our proposed framework across diverse tasks. however, promptrefine has the following limitations: