Democratizing LLM s for Low-Resource Languages by Leveraging their E nglish Dominant Abilities with Linguistically-Diverse Prompts (2024)

Abstract: Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars. However, in low-resource languages, obtaining such hand-picked exemplars can still be challenging, where unsupervised techniques may be necessary. Moreover, competent generative capabilities of LLMs are observed only in high-resource languages, while their performances among under-represented languages fall behind due to pre-training data imbalance. To elicit LLMs’ ability onto low-resource languages without any supervised data, we propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. These prompts are then used to create intra-lingual exemplars to perform tasks in the target languages. Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages. We also show that fine-tuning a 7B model on data generated from our method helps it perform competitively with a 175B model. In non-English translation tasks, our method even outperforms supervised prompting by up to 3 chrF++ in many low-resource languages. When evaluated on zero-shot multilingual summarization, our method surpasses other English-pivoting baselines by up to 4 ROUGE-L and is also favored by GPT-4.

## introduction recent scaling effort in foundation large language models @xcite @xcite with massive pre-training data has enabled them to learn a broad range of natural language tasks through few-shot in-context learning, where a few inputoutput exemplars are concatenated to the test input to prompt the model to predict the output and no gradient update of the model is performed. while most llms are pre-trained with multilingual corpora in addition to the gigantic english corpus, and have been shown to demonstrate impressive abilities in other languages @xcite @xcite , they only excel in high-resource languages, such as french. further, they may still require pivoting the inputs into english, that is, performing tasks in english first before reverting the response back to native outputs @xcite . improving llms' abilities in extremely low-resource languages can be even more challenging, particularly where the data coverage is less than 0.0001% @xcite or none at all @xcite . we also found that the models may confusedly respond in a wrong language or struggle with low-resource non-latin scripts due to overly fragmented tokenization, where words are broken into many byte-level tokens. in this work, we propose linguistically-diverse prompting (ldp), a technique that promotes an llm to perform generative tasks in low-resource languages by demonstrating few-shot exemplars in a diverse set of high-resource languages. this method works in both unsupervised setup with foundation base llms @xcite and pseudo-zero-shot setup instruction-tuned counterparts @xcite @xcite , by synthetically creating few-shot examples from zeroshot prompting. an example of ldp for unsupervised translation task is shown in figure 1 , where we gather a small set of synthetic @xmath0. then, we concatenate them as inputoutput few-shot prompts to illicit the llm to produce translation in low-resource languages. meanwhile, section 3, along with figure 2 , explains ldp in other generalized adoptions in many other tasks. our method is based on the following empirical observations of llms: (i) in-context exemplars may play a larger role in helping the model locate the task in its pre-trained knowledge @xcite , (ii) llms possess dominant abilities in english while they may lag behind in other lower-resource languages @xcite @xcite . our method is shown to perform on par with supervised prompting in unsupervised translation tasks between english and 13 indic and 21 african low-resource languages, with @xcite and instructgpt (text-davinci-003) @xcite models. furthermore, adapting our method to @xmath1. in multilingual summarization tasks @xcite , our zero-shot ldp method outperforms both basic prompting and other english-pivoting methods by up to 4 rouge-l and is generally favored by gpt-4-eval @xcite . with gpt-3.5, our method considerably improve performance of zeroshot question answering xquad @xcite and tydiqa @xcite tasks in 7 languages. our method can even enable llama-2 base @xcite to perform conversational instruction following tasks and improve the chat model in sea-bench @xcite for 2 languages that were not instruction-tuned. ## related work large language models (llms) display outstanding capabilities because they are pre-trained on massive amounts of internet text data @xcite @xcite . without any gradient update, base llms are able to perform in-context learning by simply observing a list of high-quality input-output exemplars @xcite . this technique works across many tasks that involve language understanding, reason-ing and generation @xcite @xcite . much research has been conducted to understand in-context learning. some suggest that the models secretly perform gradient descent on the exemplars @xcite , while others demonstrate that most of the knowledge is learned during pre-training, and the exemplars are only to provide evidence for the model to locate the intended task via a bayesian inference process @xcite @xcite . most llms are trained with multilingual corpora @xcite , even if these make up a tiny fraction of the large english corpora @xcite . despite that, llms still exhibit strong multilingual capabilities with high-resource languages like french, german and chinese, often with the help of englishpivoting using supervised translation systems @xcite or prompting the model to firstly generate intermediate english context @xcite . @xcite is one of the llms trained with the most number of languages, whose roots corpus consists of data from 46 languages @xcite . the roots corpus includes 34 indic and african languages regarded as low-resource, with each language having a pre-training coverage of less than 1% in hindi for the indic group, to 2e -5 % in tumbuka for the african group, as shown in figure 5 in the appendix. more recent multilingual models, like seallm or aya, were also introduced with better instruction-following abilities @xcite . ad-hoc multilingual pre-training approaches, like word-pair mining, have been explored @xcite . meanwhile, other works inspect the evaluation suites @xcite and the tokenization process for multilingual llms @xcite . exclusively only for unsupervised translation tasks, our linguistically-diverse prompting (ldp) strategy is also an englishpivoting method, but it is different from other crosslingual counterparts @xcite in that while others only pivot inputs to english intermediates, we use in-context pairs between english and a diverse set of high-resource languages to promote the intended task in the target low-resource language. for other intra-lingual tasks like instruction following, where input and output are both expected to be in the same language, our ldp helps prevent english-tuned models @xcite from responding with english answer given a non-english query. part of our work also intersects with unsupervised multilingual machine translation (umt), where back-translation is proven to be effective @xcite @xcite @xcite , along with other methods @xcite . english-pivoting is also prominent in the realm of machine translation, where training models on high-resource en↔x bitext improves lower-resource en↔y tasks @xcite . analyses of machine translation using llms have also been done. @xcite show that gpt models can perform competitively alongside the best mt models. @xcite focus on optimizing supervised exemplars selection strategies, while @xcite discover that using specific coherent prompts for each input helps improve performance. nonetheless, such studies only consider supervised instructiontuned models @xcite , which may risk test-set contamination @xcite . thus, there is still limited research involving low-resource languages in completely zero-shot setups. as such, since lowresource languages may not enjoy the privilege of having large unlabeled data to conduct searching, only random selection is used in this study, while optimal exemplar selection is out of scope. ## experiments in this section, we evaluate our method in various translation (sections 4.1 and 4.2), summarization (4.3), question answering (4.4) and instructionfollowing (4.5) across different settings and languages. we also conduct extensive analyses to provide further insights into our method (4.6). ## conclusion we introduce linguistically-diverse prompting (ldp), which is designed to use synthetic highquality in-context exemplars from high-resource languages to prompt llms to perform generative tasks in low-resource languages. our unsupervised approach achieves on par with supervised few-shot learning while using zero supervision in english to and from 34 low-resource indic and african translation tasks, even outperforming supervised prompting in non-english-centric directions. our method also outperforms other english-pivoting techniques in multilingual summarization.