LLM s Are Few-Shot In-Context Low-Resource Language Learners (2024)

Abstract: In-context learning (ICL) empowers large language models (LLMs) to perform diverse tasks in underrepresented languages using only short in-context information, offering a crucial avenue for narrowing the gap between high-resource and low-resource languages.Nonetheless, there is only a handful of works explored ICL for low-resource languages with most of them focusing on relatively high-resource languages, such as French and Spanish. In this work, we extensively study ICL and its cross-lingual variation (X-ICL) on 25 low-resource and 7 relatively higher-resource languages.Our study not only assesses the effectiveness of ICL with LLMs in low-resource languages but also identifies the shortcomings of in-context label alignment, and introduces a more effective alternative: query alignment. Moreover, we provide valuable insights into various facets of ICL for low-resource languages.Our study concludes the significance of few-shot in-context information on enhancing the low-resource understanding quality of LLMs through semantically relevant information by closing the language gap in the target language and aligning the semantics between the targeted low-resource and the high-resource language that the model is proficient in. Our work highlights the importance of advancing ICL research, particularly for low-resource languages.

## introduction large language models (llms) have displayed remarkable generalization capability in various tasks @xcite @xcite @xcite @xcite @xcite @xcite . nonethe-such as french, spanish, chinese, and japanese. in this work, we expand upon the concept of cross-lingual semantic similarity and in-context alignment, specifically focusing on low-resource languages. our hypothesis posits that their effectiveness may be compromised in low-resource languages due to the weak representation of the labels and sentences for the target languages. to test our hypothesis, we explore cross-lingual incontext learning (x-icl) covering 25 low-resource languages from various language families and compare them with the performance of 7 relatively higher resource languages, including french (fra), spanish (spa), german (deu), italian (ita), portuguese (por), arabic (arb), and hindi (hin). our result suggests that the x-icl performance decays correlate to the size of pre-training data of the target languages, which aligns with our hypothesis. moreover, to our surprise, contrary to the results reported in @xcite , we found that in-context label alignment does not work for all the languages under study and introduced an alternative alignment method namely in-context query alignment, which significantly improves the alignment quality compared to the in-context label alignment. to this end, we explore alternatives for x-icl approaches covering variations of in-context alignment information, prompt, label encoding, and strategy for selecting in-context learning exemplars. we extensively analyze all these factors and their effect on the downstream task performance of all the languages under study. our results and analysis highlight the following key takeaways: ## methods figure 1 presents the general framework of x-icl, comprising: 1) cross-lingual in-context alignment ( §3.1), 2) cross-lingual alignment formatting and 3) label configuration as parts of cross-lingual prompting ( §3.2), as well as 4) cross-lingual retrieval ( §3.3). we assess variations of these x-icl aspects to understand their effectiveness on different language resource levels. ## result and discussion the per-dataset results of our experiments are shown in appendix h. for brevity, we report the analysis mainly for xglm-7.5b, since we observe that the bloom-7b results are similar. we show the results for bloom in appendix g. ## conclusion we systematically investigate the application of x-icl with llms, focusing on low-resource languages. our comprehensive analysis sheds light on multiple facets of x-icl with llms. our examination of in-context alignment reveals the limitation of label alignment, thus we suggest a more effective alternative: query alignment. efforts to enhance x-icl via formatting consistency only exhibit a marginal impact on low-resource languages. our exploration of exemplar retrieval approaches underscores the significance of employing cross-lingual semantic similarity in x-icl. lastly, we analyze the effectiveness of x-icl in the context of lowresource languages. despite being outperformed by translate-test icl, x-icl remains relevant, especially when there is no mt model available for the target language-a circumstance prevalent in low-resource language scenarios. our exploration of icl and x-icl methods addresses the linguistic data gap in low-resource languages. in scenarios where no monolingual corpus or machine translation system exists, our work underscores the significance of icl and x-icl as a viable solution. by investigating the limitations of in-context label alignment and proposing a more effective in-context query alignment approach, we aim to enhance the applicability of icl and x-icl on low-resource languages. this research is motivated by the need to provide computational solutions for languages lacking adequate linguistic resources for lm training. our findings emphasize that icl and x-icl are useful in scenarios where alternative resources are absent, promoting linguistic diversity and inclusivity in the development of language technologies. all the datasets used in our experiments follow the license and term of use of the datasets.