M onte C arlo Sampling for Analyzing In-Context Examples (2025)

Abstract: Prior works have shown that in-context learning is brittle to presentation factors such as the order, number, and choice of selected examples. However, ablation-based guidance on selecting the number of examples may ignore the interplay between different presentation factors. In this work we develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples. We find that previous guidance on how many in-context examples to select does not always generalize across different sets of selected examples and orderings, and whether one-shot settings outperform zero-shot settings is highly dependent on the selected example. Additionally, inspired by data valuation, we apply our sampling method to in-context example selection to select examples that perform well across different orderings. We find a negative result, that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.

## introduction in-context learning is an emergent ability of llms @xcite where an llm learns to perform an unseen task by seeing a number of demonstrations in the context window @xcite . while in-context learning has shown significant potential as a way to extract relevant information from an llm and align the model with user expectations, it has also exhibited brittleness to an assortment of factors. for example, model performance when learning in-context is sensitive to which examples are selected @xcite @xcite as well as their orderings @xcite @xcite @xcite . another important parameter, the number of examples, has received comparably little attention. prior works have suggested that one-shot settings outperform zero-shot settings even when a random label is used @xcite . additional ablations have guided this parameter by citing performance plateaus at set numbers of examples @xcite @xcite . however, it is unclear whether this guidance holds when accounting for other sensitive factors such as different orderings and selected examples. previous work on data valuation has shown the efficacy of applying monte carlo sampling to evaluate datum contributions under different permutations in fine-tuning settings @xcite . inspired by this, we develop a monte carlo sampling-based method to investigate the impact of number of examples while using permutations to account for order and selection of in-context examples. specifically, we utilize monte carlo sampling and analyze performance with the addition of each exemplar. we find that performance plateaus at previously suggested numbers of examples do not consistently generalize under different permutations. further, we find that one-shot performance may be more sensitive to the selected example than previously recognized @xcite and the guidance of one-shot outperforming zero-shot is dependent on the selected example. finally, we find that using monte carlo sampling to select in-context examples may increase robustness to effects from ordering and selected examples, but unexpectedly, does not lead to performance improvements over random sampling. ## related work in-context learning in many prior works investigating in-context learning sensitivity to ordering and selected examples (see appendix a for de-scription of in-context learning), a fixed number of examples are used @xcite @xcite , with common guidance from prior ablations stating performance plateaus around @xmath0. recent work has looked at the how the number of examples impacts performance on chain-of-thought reasoning benchmarks @xcite and suggested that fewer examples may be needed, yet it is otherwise unclear the effect of number of demonstrations on other tasks and prompting frameworks, particularly when controlling for order and selected examples. monte carlo sampling monte carlo sampling has been widely adopted in the data valuation literature to provide unbiased approximations of the shapley value @xcite , based on prior work on monte carlo methods for shapley value approximation @xcite @xcite (see appendix a for additional details). recent works have also applied principles from data valuation to incontext learning example selection and ordering @xcite @xcite . while in data valuation, monte carlo sampling methods are used to calculate the marginal contribution of each data point averaged over a number of permutations, our motivation differs. in our setting, we are motivated by the utility of monte carlo sampling to provide an unbiased estimate of the influence of number of examples on in-context learning performance by reducing influence of ordering and selected examples. ## method to study the effect of the number of examples, we aim to reduce the influence of ordering and selected examples as confounding factors. while averaging across trials in previous work helps reduce the influence of selected examples, it does not account for different orderings. additionally, prior work on ordering only addresses up to @xmath1. this motivates the use of monte carlo sampling with incremental exemplar additions as 1) the use of permutations reduces the influence of ordering, and 2) the averaging across multiple trials reduces the influence of selected examples. we explain this as well as our method in detail below. ## experiment setup detailed descriptions of our setup can be found in appendix b. ## conclusion in this work, we proposed a monte carlo-based sampling method to investigate previous guidance regarding number of examples to use and one-vs. zero-shot settings. we further investigated whether this method could be used to select in-context examples, finding a performance vs. robustness tradeoff.