Document-level Clinical Entity and Relation extraction via Knowledge Base-Guided Generation (2024)

Abstract: Generative pre-trained transformer (GPT) models have shown promise in clinical entity and relation extraction tasks because of their precise extraction and contextual understanding capability. In this work, we further leverage the Unified Medical Language System (UMLS) knowledge base to accurately identify medical concepts and improve clinical entity and relation extraction at the document level. Our framework selects UMLS concepts relevant to the text and combines them with prompts to guide language models in extracting entities. Our experiments demonstrate that this initial concept mapping and the inclusion of these mapped concepts in the prompts improves extraction results compared to few-shot extraction tasks on generic language models that do not leverage UMLS. Further, our results show that this approach is more effective than the standard Retrieval Augmented Generation (RAG) technique, where retrieved data is compared with prompt embeddings to generate results. Overall, we find that integrating UMLS concepts with GPT models significantly improves entity and relation identification, outperforming the baseline and RAG models. By combining the precise concept mapping capability of knowledge-based approaches like UMLS with the contextual understanding capability of GPT, our method highlights the potential of these approaches in specialized domains like healthcare.

## introduction generative pre-trained transformer (gpt) models have shown significant potential across various clinical tasks, including information extraction, summarization, and question-answering @xcite @xcite . generative models are able to generate contextually relevant text given a prompt. however, for real-world clinical use, in tasks that require high precision, it is equally important to understand the context and minimize the errors that come from gpt models. however, accuracy of these models is limited to their training data. while gpt models are great at capturing nuanced contextual information, they often fall short in accurately identifying all medical concepts, possibly due to limited or outdated domain-specific data @xcite . knowledge bases store domain-specific data. medical knowledge bases, such as, unified medical language system (umls) knowledge base @xcite , include comprehensive information about medical concepts. integrating knowledge bases with language models is an open research area with multiple works exploring different ways of integrating them with language models, such as bert @xcite . there are limited studies on the integration of medical knowledge bases, particularly umls, with most recent large language models (llms), such as gpt. to address this limitation, we introduce an approach for clinical entity extraction that leverages umls for knowledge augmentation. while gpt can identify nuanced contextual information, umls includes a comprehensive repository of domain-specific clinical concepts that gpt may not recognize, such as brand names for drugs, abbreviations, acronyms, and aliases @xcite . our contributions in this paper are summarized as follows: ## discussion and conclusion our study highlights the significance of merging the strengths of domain-specific knowledge bases, such as umls, with the contextual understanding capabilities of llms, such as gpt. our hybrid approach, integrating mapped umls concepts with gpt, shows improvement in the model's ability to identify specific entities not inherently within its training data. our results on entity and relation extraction task indicated that leveraging mapped umls concepts as additional guidance to the gpt model, helps create focused and unique prompts that significantly enhances gpt's performance. this approach proves more effective than the standard rag technique. in conclusion, the ability to generate tailored prompts based on umls concepts offers a promising avenue for improving accuracy and relevance of extracted entities, ultimately enhancing the utility of llms in biomedical text analysis tasks.