Exploring LLM Annotation for Adaptation of Clinical Information Extraction Models under Data-sharing Restrictions (2025)

Abstract: In-hospital text data contains valuable clinical information, yet deploying fine-tuned small language models (SLMs) for information extraction remains challenging due to differences in formatting and vocabulary across institutions. Since access to the original in-hospital data (source domain) is often restricted, annotated data from the target hospital (target domain) is crucial for domain adaptation. However, clinical annotation is notoriously expensive and time-consuming, as it demands clinical and linguistic expertise. To address this issue, we leverage large language models (LLMs) to annotate the target domain data for the adaptation. We conduct experiments on four clinical information extraction tasks, including eight target domain data. Experimental results show that LLM-annotated data consistently enhances SLM performance and, with a larger number of annotated data, outperforms manual annotation in three out of four tasks.

## introduction in-hospital text data often contains valuable clinical information not captured by structured fields in electronic health records @xcite @xcite . fine-tuned small language models (slms) offer computationally efficient inference for extracting such information and have been shown to outperform prompt-based large language models (llms) @xcite . however, slms fine-tuned on in-hospital data (i.e., the source domain) often experience performance degradation when applied to data from a different hospital (i.e., the target domain) due to domain-specific vocabulary and formatting @xcite @xcite . additionally, patient for the adaptation, only the slm is available from the source domain due to data-sharing restrictions. the source slm struggles with target-specific formats, such as "<brand @xmath0.g., "well-butrin (bupropion)"). our approach adapts the source slms by (1) sampling target data based on slm's high uncertainty, (2) annotating them with an llm as an alternative to manual annotation, and (3) fine-tuning the source slm with the newly annotated data. to address this gap, we explore active learning with llm annotation in an sfda setting, inspired by recent studies @xcite @xcite . fig. 1 shows an overview of our approach. the objective of sfda is to adapt fine-tuned sourcedomain slms to unlabeled target-domain data. we formulate sfda as a three-step process: (1) sampling target-domain data with target-specific formats and vocabularies based on the slm's uncertainty, (2) annotating the selected samples using an llm as an alternative to manual annotation, and to evaluate the effectiveness of our approach, we conduct experiments on four clinical information extraction tasks encompassing eight source-target dataset pairs (summarized in tables 1 and 2 ). in a preliminary experiment, we apply llm zero-shot annotation in step (2) and observe a performance decline in clinical named entity recognition (ner). upon analyzing the quality of the resulting annotations, we attribute this decline to the performance gap between the llm and the fine-tuned source slm, aligning with recent findings @xcite . motivated by this, we introduce a novel llm annotation method termed slm-assisted llm annotation (sala). in this method, we guide an llm to correct the slm's prediction on the sampled data in step (1) instead of generating annotation from scratch. by doing so, we aim to maintain annotation quality that matches or exceeds the source slm performance across various tasks. we evaluate our approach against unadapted slms, as well as the best-performing formulations of selftraining and active learning derived from @xcite . our results demonstrate that active learning with llm annotation consistently enhances slm performance across all tasks and, with a larger number of annotated samples, outperforms active learning with human annotation in three out of four tasks. our contributions are summarized as follows: to achieve reliable human-free adaptation, we explore sfda with llm annotation, conducting an extensive evaluation across multiple clinical tasks. llm as active annotator: recent advancements have shown that large language models (llms) are effective annotators for active learning @xcite @xcite . these methods typically generate annotations with an llm and improve the performance of yet-to-be-fine-tuned slms. on the other hand, some research has shown that prompt-based llms often underperform compared to fine-tuned slms on specialized tasks such as clinical named entity recognition (ner) @xcite . it remains unknown whether llm annotation can enhance slms in clinical sfda, particularly considering the potential negative impact on already fine-tuned clinical slms. we answer this by evaluating llm annotation in the sfda setting and proposing a novel llm annotation method that integrates the source slm's prediction. results are averaged over three runs with different seeds. scores on the source domain for nd and ter are cited from @xcite . ## motivation: performance degradation our motivation is to address the performance degradation of source-domain fine-tuned slms when applied to target-domain data. in this section, we showcase the performance degradation on the target-domain datasets summarized in here, "lantus (insulin glargine)" is a single medication, with the format "<brand @xmath1." however, since such formatting is barely present in the source domain (beth), the source slm incorrectly identifies "lantus" and "insulin glargine" as separate medications. re: errors in re are often caused by variations in the vocabulary used to describe relationships between concepts. below is an example from beth: input: <@xmath2. in this example, "s/p" stands for "status post", indicating that the "diabetic ulcer" did not improve after "surgery". however, since this abbreviation is not used in the source domain (part), the slm fails to correctly interpret the relationship between the concepts and predicts "treatment improves". nd: in nd, formatting differences between the source and target-domains often lead to errors. below is an example from mimic: input: tobacco: no, <@xmath3: no ... the input lists the patient's social history, indicating the absence of alcohol use. the source slm fails in detecting the negation for "alcohol" since the source-domain data rarely includes this type of formatting. ter: the target datasets for ter are derived from non-clinical data, making vocabulary differences a frequent source of errors. input: ap-ny-12-05-98 0942est ... in this example, "ap" and "ny" represent "asociate press" and "new york", respectively. however, since such vocabularies are not present in the mayo clinical notes, the source slm fails to properly recognize the temporal expression "12-05-98" as a standard month-date-year format. the examples above highlight the challenges of applying the source slms to target domains. ## preliminary: zero-shot annotation to mitigate performance degradation, we leverage llms to annotate target-domain data within our approach (fig. 1 ). as a preliminary experiment, we evaluate the effectiveness of zero-shot annotation with the following specifications. active learning with zero-shot annotation: in step (1), we select target samples with the highest predictive entropy, following @xcite . in step (2), we prompt llms with one of the selected samples and a detailed task description at a time to generate an annotation. in step (3), we fine-tune the source slm with the generated annotations, mapping them to the slm's label space with a rule-based resolver. throughout this study, we use downloaded open-source llms to safeguard clinical data and prevent thirdparty sharing in real-world applications. specifically, we employ llama-3.3-70b foot_2 @xcite , representing a general-domain llm, and med42-70b foot_3 @xcite , represent-method # ner nd beth part i2b2 mimic source 96 51.5 39.2 85.1 80.9 zero-shot 96 52.4 67.5 89.2 82.3 zero-shot med 96 54.6 46.1 89.2 89.7 source 384 64.1 66.8 88.8 65.2 zero-shot 384 49.3 58.5 94.3 90.5 zero-shot med 384 49.9 44.2 94.5 92.8 fig. 2 illustrates the performance of the source slms adapted with zero-shot annotation for ner and nd. for the evaluation, we experimented with the number of selected samples from 96 to 384 instances. as the number of annotations increases, performance in ner decreases, whereas it improves in nd. this result suggests that the scalability of llm annotation can not be fully exploited with zero-shot annotation in tasks like ner. we hypothesize that the performance decline stems from a gap between the source slm's performance and zero-shot annotation. ## proposed: sala to improve the llm annotation in target domains, we introduce sala, slm-assisted llm annotation. inspired by recent studies @xcite , we incorporate the source slm prediction into the llm's in-context learning. as depicted in fig. 3 , we guide the llm to generate annotations based on three inputs: for the annotation, we prompt llms to correct errors in the low-confidence prediction while referencing the task description and high-confidence examples. this correction-based approach helps maintain annotation quality that matches or exceeds the source slm's performance across various tasks by constraining llm annotation to improve upon the slm prediction. in this section, we elaborate on how high-confidence examples are retrieved (sect. 6.1) and how low-confidence predictions are corrected (sect. 6.2). finally, we present the overall algorithm of sala to iteratively improve the annotation quality (sect. 6.3). notations: we denote the source slm as s, the llm as p, a target-domain sample as x @xmath4, and an annotation target as @xmath5. ## experiment we conduct an experiment comparing our approach against existing sfda baselines. with sala used for the llm annotation, the proposed method is denoted as "active + sala". our focus is on the performance of the slms, as computationally efficient inference is crucial for processing in-hospital text data. a comparison with llm zero-shot and few-shot inferences is provided in appendix a.3. ## discussion in this work, we introduced active learning with llm annotation in an sfda setting, and demonstrated that the proposed llm annotation method (sala) can enhance the performance of source slms across various tasks. in this section, we address remaining questions regarding the effectiveness of our approach, particularly in the utilization of slm assistance for llm annotation (sect. 8.1) and the sampling method for annotation targets ## conclusion in this paper, we explore active learning with llm annotation, adapting slms fine-tuned on the source domain to the target domain's specific formats and vocabularies. to improve annotation quality, we introduce a novel llm annotation method, sala, which incorporates the source slm prediction into the llm annotation through in-context learning. through experiments across four clinical information extraction tasks and eight target datasets, we demonstrate that the proposed approach consistently enhances the performance of source slms and outperforms active learning with human annotation in three out of four tasks with a larger annotation number. furthermore, we show that slm assistance improves the quality of llmgenerated annotations in challenging tasks such as clinical ner. these results highlight the potential of llm annotation as a scalable and effective alternative to human annotation in clinical sfda, where clinical and linguistic expertise is typically required.