Few-shot clinical entity recognition in E nglish, F rench and S panish: masked language models outperform generative model prompting (2024)

Abstract: Large language models (LLMs) have become the preferred solution for many natural language processing tasks. In low-resource environments such as specialized domains, their few-shot capabilities are expected to deliver high performance. Named Entity Recognition (NER) is a critical task in information extraction that is not covered in recent LLM benchmarks. There is a need for better understanding the performance of LLMs for NER in a variety of settings including languages other than English. This study aims to evaluate generative LLMs, employed through prompt engineering, for few-shot clinical NER. We compare 13 auto-regressive models using prompting and 16 masked models using fine-tuning on 14 NER datasets covering English, French and Spanish. While prompt-based auto-regressive models achieve competitive F1 for general NER, they are outperformed within the clinical domain by lighter biLSTM-CRF taggers based on masked models. Additionally, masked models exhibit lower environmental impact compared to auto-regressive models. Findings are consistent across the three languages studied, which suggests that LLM prompting is not yet suited for NER production in the clinical domain.

## introduction electronic health records (ehr) are rich sources of clinical information @xcite , which often appear in unstructured text only @xcite . efficiently extracting information from ehrs into a more structured form can help advance clinical research, public health surveillance and clinical decision support @xcite . named entity recognition (ner) is a critical primary step in information extraction that aims to identify and categorize mentions of relevant entities in text. in the context of clinical information extraction, these can be mentions of clinical entities such as disorders or drugs. extracting these entities can significantly enhance concept normalization @xcite @xcite as well as facilitate interpreting patient profiling and phenotyping @xcite . clinical ner is widely regarded as a challenging problem : clinical entities are often jargon or ambiguous, and clinical texts have a nonstandard phrasal structure @xcite . additionally, the sensitive nature of ehrs results in a lack of publicly available clinical corpora, which are often restrictively licensed and predominantly available in english. moreover, the annotation of clinical ner data demands substantial domain expertise, rendering such campaigns both costly and timeintensive @xcite @xcite . additionally, due to the diversity of clinical cases, data annotated for one biomedical application might not necessarily be helpful for another. consequently, there is a critical need for data-efficient clinical ner, specifically few-shot ner approaches. large language models (llms) -specifically, causal, generative models -have demonstrated significant promise in few-shot learning across a wide variety of tasks, including text classification, machine translation, and question answering @xcite . this supports our main research question: can few-shot learning performance of llms transfer to the task of named entity recognition? this study aims to evaluate generative llms, employed through prompt engineering, for fewshot clinical ner from the perspective of f1 performance and environmental impact. challenges of few-shot ner we identify methodological challenges with the evaluation of few-shot prompting for ner. first, there is no standard, widely adopted manner of prompting llms for ner tasks @xcite @xcite , resulting in significant challenges for reproducibility and variations in results that are difficult to interpret. second, many efforts towards "few-shot learning" with llms design prompts based on their performance on large heldout validation datasets @xcite @xcite , which is not consistent with a few-shot setup. in addition, in-context learning performance is shown to depend greatly on the prompt structure: a small change in task phrasing, the examples presented, the order of examples, or the tagging format can affect the performance @xcite @xcite . therefore, making these choices assuming large annotated validation dataset leads to performances that are shown @xcite to be over-optimistic and impossible to find in a real few-shot setting. third, @xcite show that the majority of recent studies employing prompt engineering in medical applications lack a non-prompt-related baseline, such as fine-tuned bert-like masked language models (mlms), which complicates the accurate assessment of the performance of these llms. finally, most of these studies are mainly concentrated on english, and based on gpt, which is mainly trained on english, @xcite @xcite @xcite , limiting the generalizability of evaluations to other languages. the contributions of this work are as follows: few-shot ner with generative llms recently, prompt construction has gained interest in the community @xcite . while most related work focused on studying prompt formulation and exploring better-performing prompt structures @xcite @xcite also known as "prompt engineering", other work proposed continuous optimization of the prompt through prompt tuning @xcite @xcite , usually reporting marginal improvements over baselines. there is no standard, widely adopted manner of building ner prompts @xcite . in fact, ner associates to each instance a set of spans, each of which having a type. this structured nature of the prediction make it hard to find an intuitive but efficient manner to prompt a language model for ner, that adapts well to all contexts. for instance, the main practice is to use separate prompts for different entity types @xcite @xcite . this choice seems well-suited when the task is interested in a handful of types of entities (typically 5-10). when interested in less entity types, a single prompt can be used for detecting all entities @xcite . on the other hand, if there is more entity types, it could be interesting to enumerate every possible span in the input sentence and let the model predict the entity type of the span, if any @xcite . this method, on the inverse, is impractical for long inputs. we identify three strategies for prompting llms. constrained prompting attempts to better formulate the ner task by constraining the generation to fill in specific hand-crafted templates, usually adapted to mlms @xcite @xcite . listing prompts consist in simply making the language model predict the entities in a list @xcite . tagging prompts were studied more recently by @xcite . they make the language model surround entity mentions with special tags. few-shot clinical ner mlm-based few-shot ner has also been explored in the biomedical domain @xcite . metric leaning @xcite and label encoding @xcite have been explored, as well as other approaches such as active learning @xcite , supervised pretraining @xcite and prompt-based learning @xcite @xcite . few studies have focused on llm-promptingbased few-shot clinical ner. in @xcite , gpt-3 and chatgpt are evaluated on the 2010 i2b2/va task @xcite on languages other than english, @xcite use instructgpt-3 is used to build multilingual training corpora to train smaller models, and in @xcite , chatgpt is evaluated on an ner challenge focused on extracting medical procedures in spanish. another interesting direction is partly finetuning @xcite a general-domain llm on clinical text @xcite @xcite , and prompting the resulting llm. ## conclusion this study assessed the performance of two types of large languages models, for few-shot entity recognition in three languages. our experiments show that few-shot learning performance is significantly lower in the clinical vs. general domain. while masked language models perform better than causal language models (higher f1, lower co2 emissions), llm prompting is not yet suited for effective information extraction.