In-Context Learning on a Budget: A Case Study in Token Classification (2025)

Abstract: Few shot in-context learning (ICL) typically assumes access to large annotated training sets. However, in many real world scenarios, such as domain adaptation, there is only a limited budget to annotate a small number of samples, with the goal of maximizing downstream performance. We study various methods for selecting samples to annotate within a predefined budget, focusing on token classification tasks, which are expensive to annotate and are relatively less studied in ICL setups. Across various tasks, models, and datasets, we observe that no method significantly outperforms the others, with most yielding similar results, including random sample selection for annotation. Moreover, we demonstrate that a relatively small annotated sample pool can achieve performance comparable to using the entire training set. We hope that future work adopts our realistic paradigm which takes annotation budget into account.

## introduction in-context learning (icl) has emerged as a highly efficient and robust method for various textual tasks. in this paradigm, a large language model (llm) is exposed to a small number of annotated samples, termed demonstration examples, which are provided as part of the prompt, before the sample which the model is required to annotate, which we will refer to as inference sample henceforth. while the reasons for icl's success are still contested @xcite , it has been observed that icl prompts commonly outperform zero-shot prompts, where no demonstration examples are provided @xcite . furthermore, a recent line of work has found that the choice of demonstration examples can lead to improved results over random demonstration selection. for example, @xcite found that choosing the nearest neighbors of the inference sample in the training set leads to improvements over random demonstration selection on 6 tasks, such as sentiment analysis or question answering. in all of these, the demonstration examples are chosen from large annotated training sets, ranging from 3.5k samples up to 78k samples. in this work, we address the following research question: how can we maximize icl performance on a given annotation budget? this question is particularly relevant for real-world domain adaptation settings, where a large pool of annotated samples is unavailable for selecting demonstration examples. instead, there are large sets of unannotated samples (e.g., raw text in the target domain), and a fixed budget to annotate only a small portion of them. as depicted in figure 1 , we define the task as pool selection, i.e., selecting a small pool of k examples out of a large corpus of raw texts. these samples are annotated and serve as the available pool for demonstration examples. we implement several methods for pool selection, e.g., clustering the train set and selecting a representative example from each cluster, and test them on three token classification tasks: named entity recognition, dependency parsing, and partof-speech tagging. we select these tasks as they are relatively understudied in the context of icl and are expensive to annotate due to the need for linguistic expertise and domain-specific knowledge @xcite . we evaluate several state-of-the-art llms on token classification benchmarks. we observe that none of the methods consistently outperforms the others, and, surprisingly, randomly selecting samples for the annotation pool performs comparably to more carefully designed approaches in certain scenarios. furthermore, we find that a relatively small pool (∼200 samples) allows llms to perform over 88% as when demonstrations are selected from the full training set. pool selection on a budget k few shot inference person org loc you are an ner classifier that identifies the following entities:.. statement made by @xmath0. 1 2 3 a large pool of raw train and test texts in the target domain pool selection strategy chooses a subset of the train texts collect annotations for the samples in the pool. instantiate prompts with few shot examples chosen from pool based on their distance from the inference sample. work in order to report more realistic icl performance, and explore new methods for sample selection for other tasks and domains. ## icl on a budget here we propose a conceptual framework for icl in a realistic domain adaptation setting (depicted in figure 1 ), where there are no apriori annotated datasets for the target task. instead, we assume that there is a large corpus of raw texts in the target domain, and a fixed budget for annotating a small portion of them, such that they can serve as potential demonstration examples during inference. intuitively, the goal of the annotation process is to maximize downstream icl performance. below we formalize the task of pool selection, and describe 4 selection strategies, which aim to maximize different aspects, e.g., coverage of the training set versus coverage of the test set. in the following section we evaluate these approaches for token classification tasks. ## evaluation we focus on token classification tasks, as they are both understudied in the context of icl, while traditionally expensive to annotate at scale. importantly, as is commonly done in active learning scenarios (e.g., @xcite , we use existing datasets to simulate the process shown in figure 1 , where instead of collecting manual annotations, we sample from the existing annotations. ## related work similar to pool selection, active learning @xcite also aims to select samples for annotation rather than assuming all samples are annotated. however, active learning operates during training and relies on access to an oracle or intermediate model results (e.g., confidence scores), whereas pool selection assumes no access to the model during training and only relies on observing the outputs of the model. recently, @xcite introduced a pool selection method as a method for improving downstream performance, which we evaluate as one of our approaches for annotation pool selection (cluster). our work is conceptually different in that it proposes a realistic paradigm under which to examine icl performance where there are no annotated samples. subsequently, we differ from them in that we study three different token level tasks, different pool selection methods, and particularly focus on the effect of the pool size on downstream performance. ## conclusion we proposed the framework of icl on a budget and studied different methods for pool selection, focusing on token classification tasks. we hope this work will inspire more realistic assumptions on the amount of labeled data used in different icl settings.