The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis (2024)

Abstract: In-context learning is a popular inference strategy where large language models solve a task using only a few labeled demonstrations without needing any parameter updates. Although there have been extensive studies on English in-context learning, multilingual in-context learning remains under-explored, and we lack an in-depth understanding of the role of demonstrations in this context. To address this gap, we conduct a multidimensional analysis of multilingual in-context learning, experimenting with 5 models from different model families, 9 datasets covering classification and generation tasks, and 56 typologically diverse languages. Our results reveal that the effectiveness of demonstrations varies significantly across models, tasks, and languages. We also find that strong instruction-following models including Llama 2-Chat, GPT-3.5, and GPT-4 are largely insensitive to the quality of demonstrations. Instead, a carefully crafted template often eliminates the benefits of demonstrations for some tasks and languages altogether. These findings show that the importance of demonstrations might be overestimated. Our work highlights the need for granular evaluation across multiple axes towards a better understanding of in-context learning.

## introduction an intriguing property of large language models (llms) is their ability to perform in-context learning @xcite , i.e., solve a task conditioned on a few demonstrations at inference time, without updating the model parameters. it has been shown to be an efficient alternative to finetuning when adapting models to diverse tasks and domains @xcite @xcite . in light of the success of incontext learning, there has been increased interest in better understanding the factors that influence its success, such as demonstration selection @xcite @xcite , prompt design @xcite , and more generally on understanding how and why in-context learning works @xcite @xcite @xcite . however, most recent work on in-context learning predominantly focuses on english, and the exploration of multilingual in-context learning generally lags behind. this is problematic, as results that apply to english might not hold for other languages, especially those that are less represented in llm training data. while there have been a few studies on in-context learning that go beyond english, they either focus on benchmarking llms on multilingual tasks without in-depth exploration, e.g., mega @xcite and buffet @xcite , or zoom in on specific capabilities such as mathematical reasoning @xcite , machine translation @xcite , or code-switching @xcite . in this work, we take a multidimensional approach @xcite that unifies these strands of research and comprehensively evaluate the multilingual in-context learning abilities of llms. we focus on dissecting the actual impact of in-context demonstrations, which is crucial for understanding model behaviour. our research covers various models, tasks, and languages, and we seek to answer the following research questions: ## experimental setup models. we evaluate two types of llms: pretrained base models and chat models. our base models include xglm @xcite and llama 2 @xcite . our chat models are llama 2-chat, gpt-3.5 @xcite and @xcite . specifically, we use xglm-7.5b, llama-2-13b, and llama-2-13b-chat on huggingface @xcite , and we access gpt-3.5-turbo-16k and gpt-4-32k apis via microsoft azure. 3 tasks and datasets. we experiment on a diverse range of multilingual classification and generation tasks, using 9 datasets covering 56 languages in total. our dataset selection largely follows mega @xcite , but we add datasets for extremely under-represented african languages. our classification tasks include natural language inference (nli), paraphrase identification, commonsense reasoning and sentiment analysis, with the following datasets: xnli @xcite , indicxnli @xcite , paws-x @xcite , xcopa @xcite , xstorycloze @xcite and afrisenti @xcite . our generation tasks are extractive question answering (qa) and machine translation (mt), for which we use xquad @xcite , tydiqa-goldp @xcite , and mafand @xcite . see appendix a.1 for more details. in-context learning. for each test sample, we select @xmath0, 2, 4, 8} foot_2 different demonstrations, which are randomly sampled unless otherwise specified. all demonstrations are in the same language as the test sample, and all templates are in english. we employ appropriate task-specific templates for different model types. all templates and data splits are shown in appendix a.2. ## better templates further reduce the benefits of demonstrations in-context learning performance depends not only on the demonstrations, which we have varied so far, but also on how they are formatted using templates. previous work @xcite has shown that modifying the template changes task performance. this section thus seeks to examine the interplay between template choice and demonstrations. template design. in the zero-shot setting, we observe that chat models tend to generate verbose responses (e.g., "sure! i can help you with that") or explanations (e.g., "the reason is that ...") that pose a challenge for automatic evaluation. we observe a reduction in this behaviour with icl, which leads us to question whether demonstrations are merely a means to format model responses. to see if we can achieve the same effect with minor template engineering, we augment the original templates with instructions that focus on output formatting. we call these formatting-focused templates which are shown in in this section, we focus on xcopa, afrisenti, xquad, and tydiqa, as these are the classification and generation tasks that seem to benefit most from in-context demonstrations (see section 4). however, as figure 6 shows, the performance gap between zero-shot and in-context learning diminishes with formatting-focused templates. the gap reduction is more substantial for qa datasets (i.e., the generation tasks) than for xcopa and afrisenti (i.e., the classification tasks). we speculate that it is simpler for the model to generate label words for classification tasks with a pre-defined label space than to answer questions in a way that is easy to evaluate automatically. in the latter case, formatting-focused templates can teach output styling, largely eliminating the benefits of demonstrations. with our new formatting-focused templates, we revisit the impact of the input-label mapping discussed in section 5. as figure 7 shows the language-specific effects of formatting-focused templates on xquad (results for other tasks are in appendix d.1). for llama 2-chat, demonstrations remain essential even with a formatting-focused template for most languages, but not greek and hindi. gpt-3.5 and gpt-4 also show variance across languages. moreover, for most languages, zero-shot learning with minor template engineering can match and even exceed in-context learning performance, aligning with previous work on gpt-3 (reynolds and mcdonell, @xcite . the fact that we can achieve the same effects through template engineering or demonstrations reinforces our hypothesis that models are not actually learning tasks on the fly. instead, some combination of demonstrations and templates serves to activate prior knowledge of a task and encourage a consistent output format for automatic evaluation. ## discussion our systematic study provides strong evidence that the importance of in-context demonstrations on existing multilingual datasets might be overestimated, as it highly depends on the model, task, and language used. for strong instruction-following models, the effect of demonstrations is superficial and can be eliminated with minor template engineering. these findings open up new questions, which we discuss below. ## related work multilingual in-context learning. most multilingual in-context learning studies focus on benchmarking llms on diverse tasks and comparing them with smaller fine-tuned models @xcite @xcite . as these works focus on benchmark-ing, their analysis of the role of demonstrations is limited. ahuja et al. (2023) explore different prompting strategies by adjusting the language of templates and demonstrations. zhang et al. (2023) find that demonstrations sometimes do not contribute to or even degrade model performance on code-switching. zhu et al. (2023) look at machine translation and analyze the effects of template and demonstration selection with xglm. in the context of cross-lingual transfer, shi et al. (2022), tanwar et al. (2023), and agrawal et al. (2023) investigate demonstration selection for specific applications. in contrast, we take a much broader perspective and investigate the actual impact of demonstrations across a wide range of models, tasks and languages. english-centric demonstration analysis. most of the current demonstration analysis literature focuses on english: lu et al. (2022) analyze the sensitivity of icl to the order of demonstrations, min et al. (2022c) and yoo et al. (2022) explore whether the ground truth labels matter for classification tasks, and wei et al. (2023) investigate the sensitivity of various model families to different input-label mappings. similarly, pan et al. (2023) disentangle task recognition and task learning by manipulating the label space. beyond this, shi et al. (2023a) and wang et al. (2023a) modify the validity of chain-of-thought (cot) reasoning steps in demonstrations and explore the impact of this modification on mathematical reasoning. also focusing on cot, chen et al. (2023) investigate how varying the number of demonstrations affects performance. ## conclusion in this paper, we conduct an in-depth multidimensional analysis on the impact of demonstrations in multilingual in-context learning. we find that the use of demonstrations does not always provide benefits compared to zero-shot learning, and that there is a large variance in performance across models, datasets and languages. while the quality of demonstrations influences the performance of base llms on certain tasks, the impact is significantly reduced for llms tuned with alignment techniques. we also examine the interplay between demonstrations and templates, finding that a carefully crafted template can further decrease the benefits of demonstrations. our granular analysis contributes novel insights with nuance and paves the way for a more thoughtful multilingual icl evaluation.