Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notes (2024)

Abstract: The automatic construction of knowledge graphs (KGs) is an important research area in medicine, with far-reaching applications spanning drug discovery and clinical trial design. These applications hinge on the accurate identification of interactions among medical and biological entities. In this study, we propose an end-to-end machine learning solution based on large language models (LLMs) that utilize electronic medical record notes to construct KGs. The entities used in the KG construction process are diseases, factors, treatments, as well as manifestations that coexist with the patient while experiencing the disease. Given the critical need for high-quality performance in medical applications, we embark on a comprehensive assessment of 12 LLMs of various architectures, evaluating their performance and safety attributes. To gauge the quantitative efficacy of our approach by assessing both precision and recall, we manually annotate a dataset provided by the Macula and Retina Institute. We also assess the qualitative performance of LLMs, such as the ability to generate structured outputs or the tendency to hallucinate. The results illustrate that in contrast to encoder-only and encoder-decoder, decoder-only LLMs require further investigation. Additionally, we provide guided prompt design to utilize such LLMs. The application of the proposed methodology is demonstrated on age-related macular degeneration.

## introduction there are several biomedical data corpora available that provide valuable knowledge, and one such source is pubmed @xcite . pubmed is a search engine that accesses med-line @xcite , which is a database of abstracts of medical publications and references. moreover, the widespread adoption of electronic medical records (emr) has brought various opportunities for medical knowledge discovery. knowledge graphs (kg) are often used for knowledge discovery, because graph-based abstraction offers numerous benefits when compared with traditional representations. they have been applied to various areas of healthcare, including identifying protein functions @xcite , drug repurposing @xcite , and detecting healthcare misinformation @xcite . another application may be a clinical trial design @xcite , during which identification of confounding variables is an important step. confounding variables may mask an actual association, or, more commonly falsely demonstrate an apparent association between the treatment and outcome when no real association between them exists. kgs are a powerful tool for organizing and representing knowledge in a graph structure, where nodes represent entities within a specific domain, while edges symbolize relationships between these entities. the type of relationships may vary depending on the domain, allowing for the use of directed or undirected graphs. for example, in @xcite , they employed a directed graph to encode causal relationships between diseases. other kgs may utilize both symmetric and asymmetric relationships. in our work, we specifically focus on using directed graphs to represent relationships between diseases and various factors, treatments, and manifestations that coexist with a patient while experiencing the disease (referred to as 'coexists_with'). recent advancements in large language models (llm) offer an opportunity to think about their ability to learn valuable representations from the knowl-edge encoded in medical corpora. effectively analyzing textual data and kg construction requires extensive domain knowledge and is often a timeconsuming process for medical experts. to address this challenge, we propose an end-to-end method for automatically constructing knowledge graphs from electronic medical record (emr) notes using llms, specifically through relation extraction. previous studies have suggested the utilization of specific llms for clinical relation extraction @xcite . however, due to the inherent safety-critical nature of healthcare, we conducted a comprehensive analysis of the performance and safety attributes of llms with varying architectures. to evaluate and assess their potential for medical applications and to address potential safety concerns, we introduced a manually annotated, private dataset and benchmarked the performance of 12 distinct llms. we have not performed an analysis on publicly available emr datasets, such as mimic-iii @xcite , because some of the models have used these datasets for training or fine-tuning. our analysis revealed that in contrast with encoder-only and encoder-decoder models, decoder-only models need further guidance to output in a structured manner, which is required for relation extraction to construct the kg. we, therefore, introduced a guided prompt design that helped to utilize some of such llms for our task and analyzed issues that are making others unsuitable. this rigorous assessment forms a critical foundation for the safe and effective deployment of llms in the healthcare domain. our work takes the form of the following contributions: one notable success in the construction of knowledge bases (kbs) from biomedical textual data is @xcite . semrep is a rule-based system that combines syntax and semantics with biomedical domain knowledge contained in the unified medical language system (umls) @xcite for semantic relation extraction. the range of predicates in semrep is diverse, including molecular interactions, disease etiology, and static relations. shalit et al. @xcite further improve the precision of semrep by adding three additional filtration steps. as one may observe, semrep utilizes various levels of language modeling. it has been experimentally demonstrated that llms intrinsically learn these levels of language specification, without explicit programming @xcite . in @xcite , bert-based models with probing are used to extract relations between biomedical entities. the authors observe that, although llms can extract biomedical knowledge, they are biased towards frequently occurring entities present in prompts. we do not argue about the bias of llms, but rather the complexity of extracting relations via probing. we propose providing larger context information than that which is solely present in the prompt. @xcite utilizes both structured and unstructured data from emr to construct knowledge graphs. the structured data includes icd-9 (international classification of diseases) diagnosis codes, while the unstructured data comprises various notes written by physicians and nurses to track a patient's course. on the other hand, @xcite employs 20 multimodal data resources to describe a disease with various relationships representing different biological scales. however, in this work, we solely concentrate on clinical notes for information extraction and kg construction. @xcite makes connection between llms and semantic reasoning to automatically generate a kg on the topic of sustainability. it further populates it with concrete instances using news articles from the internet. it experiments with rebel @xcite and chatgpt and shows that chatgpt (openai, 2023) is able to automatically create kgs from unstructured text, if reinforced with detailed instructions. the paper on few-shot clinical extraction using llms @xcite discusses the challenge of extracting important variables from clinical data and presents an approach that leverages large language models, specifically instructgpt @xcite , for zero-shot and few-shot information extraction from clinical text. the authors demonstrate the effectiveness of this approach in several nlp tasks that require structured outputs, such as span identification, token-level sequence classification, and relation extraction. to evaluate the performance of the system, the authors introduce new datasets based on a manual reannotation of the casi dataset @xcite . we argue that our setup is more complex as we do not consider clean, well-written, academic corpora such as pubmed @xcite and casi @xcite . the emr corpus contains a significant amount of grammatical errors ("there is some heme od .. ?"). practitioners use abbreviations and notations ("rto") not defined in the context, obfuscating the underlying information even further. our study benchmarks different llms of varying architectures and training procedures on this challenging dataset. ## dataset for this cohort study, data was obtained from the emr of the macula & retina institute, an independent health system in glendale, california, usa. the dataset included approximately 10,000 patient records of individuals with retina-related eye diseases who had visited the institute between 2008 and 2023. the study focused on extracting knowledge from the clinical notes, which are records of observations, plans, and other activities related to patient care. these notes contain a patient's medical history and reasoning and can be used to identify complex disease-related patterns such as potential treatments, causes, and symptoms. in total, the study analyzed 360,000 notes relating to 122 unique eye diseases. ## proposed method our proposed method constructs a kg of diseases and their factors, treatments, and manifestations that the patient exhibits while undergoing the disease. to achieve this, the system initially identifies disease-specific notes as described in subsection 4.1. next, for each category of medical entity, we design set of questions (subsection 4.5). we leverage an llm to answer a pre-designed set of questions, taking into consideration the aforementioned disease-specific notes as contexts as described in subsections 4.3 and 4.6. the list of llms that we experimented with are available in subsection 4.2. all the experiments are performed on 8xv100 (32gb vram) gpus which are widely accessible nowdays. the subsection 4.7 discuss postprocessing techniques utilized to get the final relations to construct the kg. ## results we now describe our experimental study over our techniques for constructing the kg. setup we construct a kg for age-related macular degeneration (amd), a progressive eye disease predominantly affecting older individuals with a high incidence rate. since kgs are typically too large to display directly, we provide their tabular representation instead. to reconstruct kgs from tables 3 and 4 , connect treatments listed in the treatment column to amd (the target entity) using arrows. similarly, connect factors from the factor column to amd, and establish connections to amd with undirected edges for entities in the coexists_with column. the evaluation is based on precision and recall, which represent the ratio of correctly extracted terms by the model to all terms extracted by the model, and the ratio of correctly extracted terms by the model to all actual terms available in the clinical notes. the same metrics have been calculated for each entity (treatment, factor, and coexists_with) separately. therefore, the ground truth for comparison has been the entity values available in the clinical notes. thus, we needed to review all clinical notes related to amd and extract all factors, treatments, and 'coexists_with' terms. you can find the explanation of these terms in subsection 4.5. the amd-related notes have been identified according to subsection 4.1 and preprocessed as described in subsection 3.1. these steps leave us with 320 clinical notes.we refer to the process of extracting terms as annotation. this annotation was carried out by two of the authors, a retina specialist, and a clinical research coordinator. to establish a consistent annotation schema, a set of examples was jointly annotated. following this, each annotator independently annotated the same set of examples, and the two sets of annotations were then combined via a joint manual adjudication process. as a result, we extracted 11 different treatments, 8 different factors, and 8 'coexists_with' terms from the clinical notes. precision and recall results quantitative results for decoder-only models using 0-shot, few-shot, and instruction-based prompting techniques are not available. these models did not produce structured outputs, rendering them unsuitable for our task. additional information can be found in decoder-only models. unlike other prompting techniques, guided instructionbased prompting (as described in subsection 4.4) has demonstrated significant improvements. this prompt design allowed us to utilize only three decoder-only models for this task, out of the seven we experimented with. these models include llama 2 @xcite , vicuna-33b @xcite , and wizardlm-70b @xcite . the other four did not produce structured outputs with this prompt design, similar to the results obtained with the other three prompting techniques. notably, wizardlm-70b achieves the highest recall for factors and treatments, demonstrating that the incorporation of additional guidance has enhanced the understanding of the task by some of the decoder-only models, resulting in more precise and accurate answers. we believe that further research is required to explore the potential of decoder-only models for challenging relation extraction tasks, and future investigations may enhance their reliability. see prediction examples in appendix e. decoder-only models here we describe the challenges that make some of these models (biogpt @xcite , opt @xcite , opt-iml-max @xcite , bloom (scao et al., 2022)) with any of the prompting techniques were unsuitable for clinical relation extraction, thus kg construction. some of the models are prone to "hallucinating", a term commonly used to refer to the models generating responses that are factually incorrect or nonsensical. see such examples in appendix e.2.1. furthermore, we observed cases where some models generated correct responses, but these responses did not originate from the given context. another concern was the generation of excessively verbose or repetitive responses. despite being contextually correct, the lengthy and redundant nature of these outputs complicated the postprocessing phase, making the integration of such responses into our kg construction pipeline impossible. see such examples in appendix e.2.2. qualitative example: amd we continue using amd as a qualitative example. amd is a progressive eye disease affecting the retina, specifically the macula. the risk factors for amd have been studied extensively and have widely been known to include age, race, smoking status, diet, and genetics @xcite . the exact reasons and mechanisms behind amd are not yet fully researched. there are multiple pathways and factors for drusen formation and amd progression, so it is hard to disentangle them. large and numerous drusen are associated with an increased risk of developing advanced amd @xcite . the pathophysiologic landscape of amd potentially involves degenerative transformations within several ocular components, including the outer retinal layers, the photoreceptors, retinal pigment epithelium (rpe) characterized by the loss of the ellipsoid zone (ez) and atrophic changes, accumulation of subretinal/submacular fluids, perturbations in bruch's membrane leading to choroidal neovascularization (cnvm), and areas of choriocapillaris nonperfusion resulting in macular atrophy and fibrosis @xcite . medical evaluators annotated drusen, genetics, cnvm, smoking, rpe irregularities, submacular/subretinal fluid, fibrosis, and loss of ez zone as risk factors for amd. the kg constructed with the utilization of flan-ul2 with guided instruction-based prompting that have relatively the best quantitative performance, is visually presented in notably, besides factors, the graph also highlights a spectrum of terms that are linked to potential treatments and symptoms associated with amd. among the treatment entities are aredother treatments indicated include pdt (photodynamic therapy), the utilization of amsler grid, supplementation of omega-3 fatty acids, and consumption of specific foods such as fish, spinach, and green leafy vegetables. the symptomatic aspects of amd encompass a range of visual impairments and clinical manifestations. patients afflicted with amd often experience poor visual acuity, metamorphopsia (distorted vision), and can be diagnosed with either dry or wet amd. additionally, the management of the condition often involves undergoing assessments such as foreseehome and macula risk genetic testing, which play a pivotal role in monitoring the progression and development of amd. each of these terms is identified as values to the 'coexists_with' entity within the graph. we also show the kg constructed by semmeddb @xcite in ## conclusion in this paper, we propose an end-to-end approach that harnesses llms for the automatic generation of kgs from emr notes. kgs hold significant value in numerous healthcare domains, including drug discovery and clinical trial design. the entities involved in the kg construction process encompass diseases, factors, treatments, and manifestations that co-occur with patients experiencing these diseases. through extensive evaluation across various llm architectures, we have demonstrated that encoder-decoder models outperform others in clinical relation extraction. additionally, we emphasize the need for additional investigation into the suitability of decoder-only models for medical applications, particularly given their critical safety implications. furthermore, we provide guided prompt design to utilize these models. we believe that an automated knowledge extraction method may deliver substantial benefits to the medical community and facilitate further research in the field.