PICL e: Pseudo-annotations for In-Context Learning in Low-Resource Named Entity Detection (2025)

Abstract: In-context learning (ICL) enables Large Language Models (LLMs) to perform tasks using few demonstrations, facilitating task adaptation when labeled examples are hard to come by. However, ICL is sensitive to the choice of demonstrations, and it remains unclear which demonstration attributes enable in-context generalization. In this work, we conduct a perturbation study of in-context demonstrations for low-resource Named Entity Detection (NED). Our surprising finding is that in-context demonstrations with partially-correct annotated entity mentions can be as effective for task transfer as fully correct demonstrations. Based off our findings, we propose Pseudo-annotated In-Context Learning (PICLe), a framework for in-context learning with noisy, pseudo-annotated demonstrations. PICLe leverages LLMs to annotate large quantities of demonstrations in a zero-shot first pass. We then cluster these synthetic demonstrations, sample specific sets of in-context demonstrations from each cluster, and predict entity mentions using each set independently. Finally, we use self-verification to select the final set of entity mentions. We extensively evaluate PICLe on five biomedical NED datasets and show that, with zero human-annotation, PICLe outperforms ICL in low-resource settings where few gold examples can be used as in-context demonstrations.

## introduction with in-context learning (icl), large language models (llms) can be adapted to perform many tasks using few demonstrations @xcite @xcite . this emergent property of llms is particularly beneficial in tasks where limited supervision data is available for fine-tuning models, such as in specialized domains where only expensive expert annotations can be relied upon to produce quality data (e.g., biomedical, clinical, and legal domains, among many others), and in situations where inhouse proprietary datasets must be compiled with few available experts to perform the annotation. despite its promise in these settings, icl is highly sensitive to the choice of the demonstrations @xcite @xcite , and it remains unclear which characteristics of demonstrations are critical for successful task adaptation. consequently, prior work has explored which demonstration characteristics lead to successful task adaptation in icl @xcite @xcite , but these studies have focused on scalar-output tasks with a limited, predefined label space, such as classification. as a result, demonstration characteristics that optimize performance remain unclear for tasks that require structured, open-ended prediction such as named entity detection (ned). in ned, the goal is to identify all mentions of a specific type of entity within a given query, producing a structured output, with a label space effectively bounded only by the number of domain entities. in this work, we focus on ned given its high number of use cases @xcite @xcite , particularly in specialized domains where effective annotation is challenging, as (1) it requires considerable domain expertise, and (2) entities can change over time, introducing distribution shifts in supervised datasets. we conduct a thorough analysis of demonstration properties that impact in-context adaptation in ned. first, we analyze the importance of the context-label correspondence of in-context demonstrations by introducing noise through various perturbations that preserve different aspects of this mapping. second, we investigate the effects of partial correctness in demonstrations. in ned, partial correctness refers to annotations that differ from the complete list of entity mentions in the input query but still share some overlap with it. to explore this, we apply various perturbation schemes to produce demonstrations with different levels of correctness. we find that while a semantic mapping between the demonstration context and label is essential, even weak semantic mappings can provide sufficient signals for in-context task transfer. moreover, our analysis reveals that in-context learning is surprisingly resilient to partially correct annotations, provided that the demonstration labels include a large number of entities. following this analysis, we introduce pseudoannotated in-context learning (picle), a framework for in-context ned with pseudo-annotated demonstrations that requires no human labeling effort. first, we exploit a pool of unlabeled samples to obtain pseudo-annotations through zeroshot prediction from llms, followed by a selfverification step in which the model is prompted to verify the type of individual entities. then, the (noisy) pseudo-annotated samples are clustered, and demonstration sets are sampled from each cluster individually. these cluster-specific demonstrations are used to predict the entities mentioned in the test query. predictions from all clusters are consolidated to obtain the final set of entity mentions. in our evaluations with multiple llms across 5 biomedical entity detection datasets @xcite @xcite , we show that picle is as effective as, and on average outperforms, standard icl that uses gold-labeled demonstrations. contributions. in summary, 1. we conduct a perturbation study to identify demonstration attributes that enable icl in low-resource ned. we find that above a surprisingly low correctness threshold, partially correct annotations are as effective for icl as demonstrations with fully correct gold annotations. 2. we propose picle, a novel framework for incontext learning that uses pseudo-annotated demonstrations as in-context examples. we show that without human-annotation effort, picle competes and even outperforms icl with gold-labeled demonstrations in resourcescarce settings. ## related work what matters in in-context learning? in-context learning is remarkably effective for performing various nlp tasks with only a few task demonstrations appended to the prompt @xcite . however, despite a large body of work on designing novel in-context learning methods (e.g., @xcite @xcite , it is not yet fully understood what makes in-context learning effective, with multiple works demonstrating surprising variables, such as the impact of the demonstration order @xcite , the term frequencies of test examples in pretraining data @xcite , and basic output calibration @xcite @xcite . consequently, recent works explore how demonstration components might be separately responsible for in-context transfer. @xcite show that in-context demonstrations serve to show the label space of demonstrations, the distribution of their input text, and their overall format. however, yoo et al. ( 2022 ) perform quantifiable analysis on the impact of ground-truth label demonstrations on a larger set of tasks and datasets and find that ground-truth labels have substantial impacts on icl performance. @xcite continue this line of work and show that the degree to which the label mapping influences task transfer depends on the scale of the model, and that smaller models are more capable of ignoring misaligned label mappings. @xcite show similar results for cot reasoning, finding that cot is also possible without valid demonstrations, and that demonstrations that are relevant to the query and have the correct order of reasoning steps are more important for effective transfer. however, these works focus on classification tasks, which lack the concept of partial correctness; a label is either fully correct or entirely incorrect. in token-level tasks like ned, however, the list of annotated entities can be partially correct. we show that partially correct demonstrations can perform as effectively as fully correct ones-a result not addressed by these prior works. furthermore, contrary to @xcite 's findings for classification tasks, we show that icl demonstrations with fully incorrect labels are not effective in ned. ## experimental setup the task of named entity detection (ned) requires detecting all mentions of entities in a text. we formulate the task such that the language model is given a passage of text as part of a prompt and must predict the list of entities that are mentioned in the passage. optionally, in few-shot settings (i.e., in-context learning), the prompt also contains several demonstrations, each including an example passage and a corresponding list of mentioned entities in the passage. ## in-context ned with pseudo-annotated demonstrations in this section, we propose picle, a framework for pseudo-annotating unlabeled samples that can be leveraged for in-context learning. this framework consists of two stages (figure 3 ). in the first stage, we start with a set of unlabeled samples and prompt the model in a zero-shot pass to extract entities in each sample. then, we improve the quality of these pseudo-annotations by prompting the model to verify each predicted entity (i.e., selfverification; @xcite , and filter entities that are not of the correct entity type. we use kmeans clustering to group the pseudo-annotated samples into k clusters based off the embedding of their text and pseudo-annotations. 7 each cluster is used as an individual pool of demonstrations for the downstream ned task. in the second stage, we prompt the model k times, each time choosing the demonstrations from one cluster of pseudoannotated samples (a sampling method we refer to as sp-k-means, i.e., specialized k-means). then, for each entity in the k lists of predictions, we perform a self-verification step to verify if the entity has the correct type or not, and retain the extracted entities that have the correct entity type. in all of our experiments, we pseudo-annotate 1000 samples from the training set of the datasets with greedy decoding. bc5-chem bc5-disease bc2gm chemprot-chem chemprot-gene 0.2 0.4 0.6 0.8 micro f1 zero-shot knn from 10 gold samples knn from 50 gold samples knn from 100 gold samples picle (ours) figure 4: performance of picle, zero-shot, and 10-shot icl with gold demonstrations selected from 10, 50, 100 gold examples using mistral. the error bars show the variance across 5 seeds for sampling subsets of gold examples. all methods are followed by self-verification. picle performance we evaluate picle on the same biomedical ned datasets used for our analysis in section 4 and compare picle's performance with standard icl using gold demonstrations sampled from different demonstration pool sizes, representing various degrees of annotation scarcity. for baselines that use gold annotations as in-context examples, we initially sample demonstration pools of size n from the full training set of each dataset, which range in size from 4.5k to 12.5k examples ( our results in figure 4 show that across most datasets (with the exception of chemprot-chem), picle significantly outperforms the zero-shot baseline by an average of 10.7% (57.1% compared to 46.4%). furthermore, picle also matches or outperforms in-context learning with gold demon-strations in resource-scarce settings, even beating an in-context learning baseline that has access to 100 human-annotated demonstrations (57.1% vs. 52.8%). we note that the dataset with the highest performance, bc5chem (77.7% average f1 score for picle), contains entity annotations whose surface forms generally contain fewer tokens (see we also compare picle with a supervised baseline, fine-tuning a domain-specific language encoder, biomednlp-biomedbert-large, on various numbers of gold annotations (see ## conclusion in this work, we study the demonstration attributes that enable in-context generalization for named entity detection. we find that the context-label semantic correspondence is crucial for effective incontext ned, and without this correspondence, incontext examples hurt performance, pushing it below zero-shot ned. however, our analysis demonstrates that partially correct demonstration label sets are just as effective as gold label sets, provided a sufficient number of correct label mappings are found in the demonstration. based off these findings, we design an icl framework, picle, for named entity detection that leverages llms to produce pseudo-annotated examples that can be used for in-context transfer. our results on five biomedical ned datasets demonstrate that picle is more effective than zero-shot prediction and outperforms in-context learning with gold demonstrations when gold demonstrations are scarce.