On-the-fly Definition Augmentation of LLM s for Biomedical NER (2024)

Abstract: Despite their general capabilities, LLMs still struggle on biomedicalNER tasks, which are difficult due to the presence of specialized terminology and lack of training data. In this work we set out to improve LLM performance on biomedical NER in limited data settings via a new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly. During this process, to provide a test bed for knowledge augmentation, we perform a comprehensive exploration of prompting strategies. Our experiments show that definition augmentation is useful for both open source and closed LLMs.For example, it leads to a relative improvement of 15% (on average) in GPT-4 performance (F1) across all (six) of our test datasets. We conduct extensive ablations and analyses to demonstrate that our performance improvements stem from adding relevant definitional knowledge. We find that careful prompting strategies also improve LLM performance, allowing them to outperform fine-tuned language models in few-shot settings. To facilitate future research in this direction, we release our code at https://github.com/allenai/beacon.

## introduction despite the impressive zero-and few-shot capabilities of llms generally, their performance on named entity recognition (ner) over biomedical text remains underwhelming. for instance, @xcite observe that using @xcite with in-context learning performs worse than a smaller, fine-tuned pretrained language model given the same amount of data. despite significant real-world utility, several aspects make this task challenging even for state-of-theart llms. biomedical texts use specialized terminology that often requires domain expertise to interpret. in addition to complicating the task, this ## experimental framework models we evaluate sota llms over a set of biomedical ner datasets from the bigbio benchmark @xcite . we assess a variety of models including closed models available via apii.e., open ai's gpt-3.5 @xcite and gpt-4 (openai, 2023) and anthropic's claude 2 (anthropic, 2023)-and an open-source model (llama 2; @xcite . we list all models in ## icl for biomedical ner in this section we establish the baseline performance of llms in zero-and few-shot settings over all datasets. to contextualize these results, we also report on the performance of a smaller, fine-tuned model (flan-t5 xl; chung et al. 2022). ## augmenting prompts with definitions icl approaches rely on the parametric knowledge acquired by the models during pre-training. however, this internal knowledge can be incorrect, insufficient, or outdated. prior work has tried to address knowledge gaps in llms by augmenting prompts with relevant factual knowledge on-the-fly, improving performance on language understanding tasks like question answering @xcite . this motivates us to explore whether dynam-ically augmenting prompts with relevant knowledge improves icl performance for biomedical ner. in our work, we focus on a specific category of knowledge-definitions of biomedical concepts present in the input text. intuitively, generic llms may not be proficient with biomedical concepts; providing targeted information at test time may permit fast adaptation to this domain. we propose to operationalize this approach as follows. first, we curate a knowledge base of biomedical concept definitions and leverage an off-the-shelf entity linker to map occurrences of concepts to entries in the knowledge base ( §4.1). second, we perform inference with a sequence of prompts: we prompt models to extract entities as discussed in §3, and then craft follow-up prompts ## assessing the utility of definition knowledge we further assess the utility of concept definitions by conducting ablation experiments probing the following dimensions: (1) relevance of concept definitions; and (2) source of definition knowledge. we conduct all experiments in the single-turn zero-shot setting ( §4.2), with one closed model (gpt-4) and one open-source model (llama 2), on the two datasets with the largest gains in performance from concept definitions (cdr and ncbi for gpt-4; medm and chia for llama 2). ## conclusions in this work, we extensively evaluated the performance of icl approaches for biomedical ner with modern llms. we compared different combinations of input and output formats and characterized the main types of errors made by these models. we then proposed and evaluated a method for rapid adaptation of general llms to biomedical ner tasks by providing models with concept definitions from an external knowledge base dynamically. we perform inference with a sequence of prompts, allowing models to revise their predictions given definitions of key concepts in the input. the first prompt asks the model to extract entities from the input; subsequent prompts are augmented with definitions for all biomedical concepts including the entities identified in the first prompt, and ask the model to revise its predictions. our evaluation-conducted over 6 datasetsshowed consistent and often substantial improvements over baselines, especially in zero-shot settings. ablations confirm that the observed gains stem from the models' ability to capitalize on the concept definitions. in particular, we observe that without these definitions the models are unable to meaningfully improve their predictions. while we only considered datasets from a specialized domain (biomedicine), our ablations show that our approach can also be used with more general knowledge bases, such as wikidata. this provides some evidence for the potential utility of this approach in other domains. we leave a thorough exploration of this for future work.