More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLM s with In-Context Sampling (2024)

Abstract: While most existing works on LLM prompting techniques focus only on how to select a better set of data samples inside one single prompt input (In-Context Learning or ICL), why can not we design and leverage multiple prompts together to further improve the LLM’s performance? In this work, we propose In-Context Sampling (ICS), a low-resource LLM prompting technique to produce confident predictions by optimizing the construction of multiple ICL prompt inputs. Extensive experiments with three open-source LLMs (FlanT5-XL, Mistral-7B, and Mixtral-8x7B) on four NLI datasets (e-SNLI, Multi-NLI, ANLI, and Contract-NLI) and one QA dataset (CommonsenseQA) illustrate that ICS can consistently enhance LLMs’ performance. An in-depth evaluation with three data similarity-based ICS strategies suggests that these strategies can further elevate LLM’s performance, which sheds light on a new yet promising future research direction.

## introduction large language models (llms) with billions of parameters, such as flan-t5 @xcite , llama @xcite , and mistral @xcite , have demonstrated exceptional natural language interpretation capability in terms of understanding versatile prompt inputs 1 . in comparison with much smaller language models like bert @xcite and gpt @xcite , such llms can understand not only figure 1 : our proposed ics paradigm comprises three steps: 1) sample representative icl demonstration candidates, 2) augment different icl prompt inputs from the sampled candidates and acquire llm's prediction for each input correspondingly, and 3) determine and vote llm's most confident prediction. more complex and detailed task narratives but also a few task examples with annotations within the prompt inputs, namely few-shot in-context learning (icl) @xcite . as a prominent prompting strategy to exploit llms' task-solving capabilities, especially for un-seen tasks, icl inserts a few data examples as well as their corresponding annotations into the prompt input. the data examples, along with their annotations, serve as demonstrations foot_0 for the targeting task. the demonstrations are expected to facilitate llms' better understanding of the task narrative, the expected outputs, and potentially the underlying rationales needed for solving the task. several recent works investigate the influence of different icl setups, including the number, ordering, and combinations of demonstrations @xcite @xcite . however, there is no common ground for the best icl strategy yet. moreover, despite llms' superb natural language interpretation and generation capability, realworld tasks requiring extensive domain expertise remain challenging for llms (e.g., children's education and mental issue detection @xcite @xcite ), and thus, how to exploit llms' ability with icl for solving these tasks is an under-explored topic but holds great promise. we hypothesize that different icl demonstrations provide llms with distinct knowledge about the task, leading to disparate understanding and predictions for the same data. consequently, a research question emerges: can we augment multiple icl prompt inputs efficiently to facilitate more accurate and confident llm predictions? to address this question, we propose in-context sampling (ics), a low-resource methodology inspired by the query-by-committee strategy @xcite and the fewshot in-context learning approach. ics follows a three-step pipeline as shown in figure 1: 1. sample demonstration candidates; 2. augment icl prompt inputs and predictions; 3. vote the most confident label. we also propose three data similarity-based ics strategies inspired by established data sampling strategies for active learning @xcite . we believe ics can be a more reliable prompting paradigm than the traditional icl, better squeezing llm's task-solving capabilities and seamlessly supporting "plug-and-play" customizations. our evaluation of the ics paradigm comprises bi-fold. first, we benchmark the effectiveness of a baseline ics strategy with the traditional icl approach on three open-source llms (flan-t5-xl @xcite , mistral-7b @xcite , and mixtral-8x7b @xcite )) foot_1 over five datasets, including four natural language inference (nli) @xcite datasets as well as the commonsenseqa (cqa) dataset @xcite . among four nli datasets, three are general-domain nli tasks of increasing difficulty (e-snli @xcite , multi-nli @xcite , and anli @xcite ), and the last one is contract-nli @xcite , a domain-specific nli dataset for the real-world contract review task. we also investigate how different sample sizes and the number of icl prompt inputs affect model reliability in terms of performance enhancement. results indicate that ics can consistently improve prediction accuracy and robustness despite llms demonstrating different levels of icl capabilities. we further investigate the additional advantages provided by three different ics strategies through simulations with the best-performing setting from the previous experiment, compared with the random ics and traditional icl approaches on the aforementioned four datasets. despite being conceptually straightforward, all three types of databased strategies can effectively and consistently improve llm performance, leading to a broader research scope to exploit ics in the future. ## ics prompting paradigm given a natural language task instruction i and a datum to predict @xmath0, llms can take the in-context learning (icl) input format, denoted as: our hypothesis stands on the shoulder of the query-by-committee @xcite strategy that has been around for a long time. the original concept is to ask a committee of models to vote on whether the unlabeled data needs to be annotated, where the voting models focus on competing hypotheses. however, most existing works focused on measuring the disagreements among committee models @xcite and creating different committees with probabilistic and nonprobabilistic models @xcite @xcite @xcite . in this work, we present in-context sampling (ics), a low-resource paradigm for llms through effectively augmenting icl prompt inputs, as shown in figure 1 . we view the ics strategy as exploring efficient approaches to create committee icl prompt inputs and query llms for the most confident prediction. ics consists of three steps: ## evaluations the evaluation of our proposed ics paradigm comprises bi-fold. first, in section 4.1, we execute a benchmark experiment between the random ics strategy and traditional icl approach on five datasets with two llms to demonstrate the paradigm effectiveness. additionally, we attempt to identify a sample size and the amount of augmented icl combinations that strike a balance across three perspectives: 1) encompass sufficient diversity to represent the underlying data adequately, 2) possess robustness toward confident predictions, and 3) minimize annotation costs. subsequently, in section 4.2, we pick the best-performing parameters from the first experiment to compare the additional advantages of the three proposed ics strategies described above in section 3.1. we experiment on three generic nli tasks of increasing difficulties: e-snli @xcite , multi-nli @xcite , anli @xcite , a domain-specific contract-nli @xcite dataset, and the commonsenseqa @xcite dataset (dataset statistics in appendix b). we originally considered llama2 @xcite but eventually excluded it because our preliminary experiment, discussed in appendix d, shows that llama2 tends to output "neutral" regardless of the inputs on anli. we also conduct a small-scale ablation study with openai's closedomain gpt-3.5 in appendix e. we intended to manipulate and investigate two controlled variables of ics: the size of sampled demonstration candidates n, where @xmath1, 100, 250, 500}, and the number of augmented prompt inputs k for each data to be predicted, where @xmath2, 5, 10, 20}. we fix the number of demonstrations in each prompt input as three across all methodologies and experiments. the baseline is the vanilla icl approach with randomly chosen three examples, denoted as baseline in figure 2 and icl in diagrams from appendix c. we consider 500 annotations a reasonable budget cap for various real-world, low-resource scenarios. each setting is repeated and averaged over 10 trials to counter the sampling randomness. all the detailed experiment settings, including the task instruction narrative, are reported in appendix a. ## discussion limited performance with flan-t5 flan-t5 models have been fine-tuned on various downstream tasks, including nli. this fine-tuning could indeed influence the models' performance in incontext learning scenarios, potentially skewing the effectiveness of ics. additionally, we observe flan-t5-xl results in poor performance on contract-nli from figure 2 , despite it can perform adequately well on the other three generic-domain nli datasets. we conduct an ablation study with flan-t5-xl for icl to investigate the potential reasons and report in appendix f. given the ablation study results, we hypothesize several possible reasons: 1) flan-t5-xl falls short of properly interpreting long text sequences; 2) flan-t5-xl was not fine-tuned to elevate the ability to interpret icl demonstrations, and 3) flan-t5-xl lacks the necessary domain knowledge to solve the contract-nli task. ## conclusion this work presents in-context sampling (ics), a novel in-context learning paradigm for probing confident predictions by sampling demonstration candidates and augmenting different icl prompt inputs. our experiments show that even ics with the random strategy can lead to consistent accuracy improvement compared with the traditional icl approach, and further illustrate the additional helpfulness provided by three fundamental but effective data similarity-based sampling strategies with ics. our work lays the foundation for implementing icl-based applications to support non-expert users in the real world, as they do not know how to write a single perfect prompt to get their work done but often write multiple prompt inputs @xcite . our method aligns well with such user scenarios.