Complementary Explanations for Effective In-Context Learning (2023)

Abstract: Large language models (LLMs) have exhibited remarkable capabilities in learning from expla- nations in prompts, but there has been limited understanding of exactly how these explana- tions function or why they are effective. This work aims to better understand the mechanisms by which explanations are used for in-context learning. We first study the impact of two dif- ferent factors on the performance of prompts with explanations: the computation trace (the way the solution is decomposed) and the natural language used to express the prompt. By per- turbing explanations on three controlled tasks, we show that both factors contribute to the ef- fectiveness of explanations. We further study how to form maximally effective sets of expla- nations for solving a given test query. We find that LLMs can benefit from the complemen- tarity of the explanation set: diverse reasoning skills shown by different exemplars can lead to better performance. Therefore, we propose a maximal marginal relevance-based exemplar selection approach for constructing exemplar sets that are both relevant as well as comple- mentary, which successfully improves the in- context learning performance across three real- world tasks on multiple LLMs.

## introduction large language models (llms) have achieved promising progress in learning from only a few exemplars in prompts via in-context learning (icl) @xcite . to scale to complex tasks, recent work in the past year has shown that llms can benefit from explanations in prompts, particularly for tasks involving multistep reasoning @xcite @xcite @xcite . however, while including explanations in prompts has been demonstrated to be useful, little has been shown regarding what particular features make them effective and how they function in icl. figure 1 : prompting opt @xcite with explanations where we perturb the computation traces or natural language. perturbing either traces or natural language will lead to performance degradation. our work aims to better understand the mechanisms by which explanations are used for icl. as shown in the example for a multi-step reasoning task in figure 1 , we view an explanation as a combination of a computation trace and natural language which glues together the states in the trace. we design a series of probing experiments that perturb the explanations (as shown in figure 2 ) and test llms' performance to understand the sensitivity of llms on these two factors. the results suggest that both factors contribute to making effective explanations, as llms see substantial performance degradation when prompted with defective explanations. nonetheless, incomplete explanations are still beneficial compared to no explanations at all (figure 1 ). this suggests that llms "faithfully" follow the reasoning process specified by the explanations to some extent, as opposed to naively following the template patterns while disregarding critical information @xcite . the observations from our probing experiments lead us to focus next on understanding what makes an effective set of exemplars with explanations for solving a given test query. we primarily focus on two aspects, the exemplar-exemplar interplay (how exemplars work together) and the query-exemplar interplay. on the former, we find that the complementarity of the exemplar set is beneficial, as llms can fuse different reasoning processes exhibited by individual exemplars in-context. for these studies, we probe llms with a mixture of two types of exemplars; each type only specifies a part of the reasoning process (see figure 3 for detailed instances). we also test how the relevance between the query and the exemplars impacts the performance. choosing the nearest neighbors (nn) of the query to prompt llms has been shown to be effective in the standard prompting setting @xcite . our experiments covering three similarity metrics show that this is also applicable in the setting that prompts llms with explanations. our analyses inspire us to rethink the exemplar selection process of using explanations for icl. the prominent nn-based paradigm only considers relevance @xcite @xcite , which could result in selecting mostly similar exemplars. we argue that complementarity should also be considered when constructing explanation-infused prompts. therefore, we propose an exemplar selection strategy based on the maximum marginal relevance (mmr) @xcite approach which selects exemplars that are both relevant as well as diverse. the underlying rationale is that a diverse set of exemplars is more likely to showcase complementary reasoning types that are required to illustrate the reasoning required in the query. we test our mmrbased strategy on three real-world datasets spanning multiple reasoning tasks. on a powerful llm, text-davinci-002, our mmr-based strategy is able to improve the accuracy over the baseline of using random exemplars by 4.0%, 3.9%, and 8.5% on gsm, ecqa, and e-snli, respectively. in summary, our main findings are: (1) we show that both the computation trace and natural language contribute to making effective explanations for icl. (2) we show that llms can benefit from exemplar sets that exhibit both complementarity and relevance to a given test query. (3) we propose an mmr-based exemplar selection strategy considering both complementarity and relevance and demonstrate that it is more effective than solely choosing the nearest neighbors. ## background in-context learning our study is focused on the usage of explanations in in-context learning (icl). let q be the test query to solve. the standard icl prompts a language model, m , with a set of exemplar input-output pairs, {(q 1 , a 1 )...(q m , a m )}, and predict an answer â for the query: in addition to just input-output pairs, we can also include explanations (in the style of scratchpad @xcite or chain-of-thought @xcite ) in prompts, which leads the llm to generate explanations for its predictions as well: the end task performance of icl is sensitive to the selected exemplars @xcite . while much prior work uses a fixed set of manually selected exemplars @xcite , there is also work devoted to studying how to select more effective exemplars from a pool of exemplars. given a test query q, the task is to select a set of m exemplars from a pool of n exemplars @xmath0, e 1 , a 1 )...(q n , e n , a n )} to construct a prompt for solving q. we note that this yields varying exemplar sets for different queries. datasets & large language models our analysis is based on model performance on various reasoning datasets. for probing experiments, we mainly use symbolic reasoning datasets, including 1) letter concatenation (let cat) @xcite which requires extracting the last letters of two words and then concatenating them, 2) coin flips @xcite which reasons about the states of a coin after two steps of operations (flipping or not flipping), and 3) grade school math (gsm) @xcite which focuses on grade school-level arithmetic reasoning problems expressed in natural language. to investigate the effectiveness of different exemplar selection strategies, we use two more textual reasoning datasets, namely ecqa @xcite ) and e-snli @xcite , in addition to gsm. the task of e-snli is to decide whether a premise entails a hypothesis. ecqa asks multiple-choice commonsense questions. these three tasks include human-annotated explanations and cover diverse reasoning abilities. our experiments cover an array of llms, including opt-175b @xcite , gpt-3 (davinci) @xcite , instructgpt (text-davinci-001) @xcite , and text-davinci-002. in addition, we also use gpt-3 codex models @xcite that are finetuned on a large scale of code snippets in our exemplar selection experiments, namely code-davinci-001 and code-davinci-002. though codex models primarily target code-related applications, we find that they are strong in textual reasoning tasks as well. ## do llms follow explanations? we first investigate what makes explanations effective for llms to learn from. we view an explanation as a computation trace (t ) that is transformed by a natural language function (l) which maps the trace to a complete natural language explanation. a computation trace t is the chain of intermediate steps (instantiated as tokens in explanations), s 1 , . . . , s n , that are used to derive the final answer. for instance, the trace for letcat is the two last letters of the two words and the concatenated two-letter tokens; the traces for gsm are the intermediate equations. these computation traces are wrapped by natural language function l to form the final explanation l(s 1 , . . . , s n ), which presumably makes the generation of these traces more "natural" with respect to language modeling. setup we choose the three symbolic reasoning datasets mentioned in section 2 for our probing experiments for two reasons. first, llms see substantial benefits from including explanations in prompts for these tasks. second, we can easily manipulate the traces in their explanations. the gold explanations are directly taken from or adapted from @xcite . more details on the gold explanations used for these tasks can be found in appendix a. we experiment on both llms trained with vanilla language modeling objectives (opt, davinci) and llms that are aligned with human letcat question: take the last letters of the words in "bill gates" and concatenate them. gold: the last letter of bill is l . the last letter of gates is s . concatenating l and s is ls . so the answer is ls. mask1: the last letter of bill is _. the last letter of gates is _. concatenating l and s is ls. so the answer is ls. mask2: the last letter of bill is l. the last letter of gates is s. concatenating _ and _ is _. so the answer is ls. incorrect: the last letter of bill is y. the last letter of gates is e. concatenating y and e is ye. so the answer is ye. no nl: bill, l. gates, s. l, s, ls. so the answer is ls. coinflip question: a coin is heads up. ka does not flip the coin. sal flips the coin. is the coin still heads up? gold: the coin started heads up. ka does not flip the coin, so it becomes heads up. sal flips the coin, so it becomes tails up. so the answer is no. mask1: the coin started heads up. ka does not flip the coin, so it becomes _ up. sal flips the coin, so it becomes tails up. so the answer is no. mask2: the coin started heads up. ka does not flip the coin, so it becomes heads up. sal flips the coin, so it becomes _ up. so the answer is no. incorrect: the coin started heads up. ka does not flip the coin, so it becomes tails up. sal flips the coin, so it becomes heads up. so the answer is yes. no nl: heads, heads, tails. so the answer is no. gsm question: leah had 32 chocolates and her sister had 42. if they ate 35, how many pieces do they have left in total? gold: leah had 32 chocolates and leah's sister had 42. that means there were originally 32+42=74 chocolates. 35 have been eaten. so in total they still have 74-35=39 chocolates. the answer is 39. mask1: leah had 32 chocolates and leah's sister had 42. that means there were originally 32+42=_ chocolates. 35 have been eaten. so in total they have _-35=39 chocolates. the answer is 39. mask2: leah had 32 chocolates and leah's sister had 42. that means there were originally _ chocolates. 35 have been eaten. so in total they have _ chocolates. the answer is 39. incorrect: leah had 32 chocolates and leah's sister had 42. that means there were originally 32+42=62 chocolates. 35 have been eaten. so in total they have 62-35=27 chocolates. the answer is 27. no nl: 32+42=74, 74-35=39. the answer is 39. expectations via different forms of instruction tuning (text-davinci-001, text-davinci-002). ## mmr for exemplar selection we have established that emplar-exemplar interplay together with the query-exemplar interplay impacts the performance of using explanations in icl. this leads us to rethink how to select good exemplars for a given query. based on our prior analysis on the effects of complementarity and relevance in section 4.1, we argue that a good set should consist of relevant exemplars that collaboratively cover the reasoning skills required for solving the query. q := [s(q, qi)]q @xmath1; ▷ the similarity between query and exemplars in t 4: t := {}; 5: while |t | < k do 6: q := equation( 1 ); ▷ get the next exemplar based on eq (1) 7: t.add(q) 8: the prominent paradigm, i.e., nn-based exemplar selection strategy, only considers the relevance between the exemplars and the query. yet, selecting nearest neighbors could result in mostly similar exemplar sets, which can possibly limit collaboration. we argue that complementarity should also be considered in the exemplar selection process, so that the selected set could have a higher chance to illustrate the required reasoning processes. in practice, it is tricky to decide whether the reasoning underlying a set of exemplars is complementary categorically. we therefore use diversity as a proxy, since a set of less similar exemplars is arguably more likely to exhibit complementarity. to that end, we propose a maximal-marginalrelevance @xcite (mmr) based exemplar selection strategy. the idea is to select exemplars that are relevant to the query while being diverse enough to be collaborative. suppose for the given query q, we have already selected a set of exemplars @xmath2, then we will pick up the next exemplar according to: ## related work the growing scale of pretrained language models has granted them the ability to learn a new task from a few examples via in-context learning @xcite . various approaches have been proposed to improve icl in recent years, including meta-tuning llms @xcite , calibration of icl @xcite , automatically determining the orders of exemplars @xcite , and alternative formulation of icl based on pmi @xcite or noisy-channel @xcite . more closely related to our work, prior research also contributes to better understanding icl as bayesian inference @xcite or experiments that study what makes in-context learning works @xcite . our work focuses on understanding the usage of explanations in icl, as opposed to standard prompting where the llms are presented with only input-output pairs. in particular, our work is connected to prior research on effective ways for selecting in-context exemplars @xcite @xcite @xcite . while past work primarily focuses on the effectiveness of using relevant examples in the standard prompting paradigm, we examine the benefits of the complementary exemplars when prompting with explanations. we also propose an mmr-based strategy, which is more effective than the nn-based approach on various llms across three tasks. lastly, including textual explanations in prompts has exhibited remarkable benefits for llms to learn various reasoning tasks. using scratchpad @xcite or chain-of-thought @xcite significantly boosts icl performance on multi-step reasoning tasks such as arithmetical reasoning and symbolic reasoning. using free-text rationales is also helpful for more unstructured tasks like qa and nli @xcite . while recent work largely aims to find better ways to prompt llms with explanations @xcite @xcite , we focus on analyzing the role of explanations in icl and what makes effective explanations. ## conclusion we have presented a series of studies on what makes effective explanations for in-context learning. we first investigated the impacts of compu-tation traces and natural language in explanations. through a set of probing experiments, we found that llms rely on both of them to effectively learn from explanations. we further examined the interplay among exemplars within prompts and the interplay between exemplars and the query. our analysis uncovered the benefits of constructing prompts by selecting complementary explanations that are relevant to the query. lastly, we proposed an mmrbased exemplar selection strategy, which successfully improved the end task performance across three important datasets.