Incorporating medical knowledge in BERT for clinical relation extraction (2021)

Abstract: In recent years pre-trained language models (PLM) such as BERT have proven to be very effective in diverse NLP tasks such as Information Extraction, Sentiment Analysis and Question Answering. Trained with massive general-domain text, these pre-trained language models capture rich syntactic, semantic and discourse information in the text. However, due to the differences between general and specific domain text (e.g., Wikipedia versus clinic notes), these models may not be ideal for domain-specific tasks (e.g., extracting clinical relations). Furthermore, it may require additional medical knowledge to understand clinical text properly. To solve these issues, in this research, we conduct a comprehensive examination of different techniques to add medical knowledge into a pre-trained BERT model for clinical relation extraction. Our best model outperforms the state-of-the-art systems on the benchmark i2b2/VA 2010 clinical relation extraction dataset.

## introduction in recent years pre-trained language models (plms) such as elmo @xcite , bert @xcite , xlnet @xcite , and gpt @xcite have become very popular as they can effectively boost the performance of diverse nlp tasks such as information extraction @xcite , sentiment analysis @xcite , question/answering @xcite and language entailment @xcite . these models are trained on large text corpora using self-supervised tasks such as masked language modeling (mlm) and next sentence prediction. as these models can learn meaningful context-sensitive text embeddings, they are frequently used to encode input text in many downstream text analysis tasks. however, plms trained on general-domain text (e.g., books, wikipedia and webdata) may not be ideal for domain-specific nlp applications (e.g., biomedical nlp). in this research, we explore how medical knowledge can be added to plms to facilitate clinical relation extraction. previously, significant effort has been made on adding domain knowledge into plms. based on the types of knowledge added, we can group the work into two categories: integrating domain text in plms @xcite @xcite and integrating domain-specific knowledge graphs into plms @xcite @xcite . in this study, we conduct a comprehensive investigation of these methods. we test their effectiveness in integrating knowledge from unified medical language system (umls) into bert for clinical relation extraction. here we focus on umls because it is one of the most widely used bio-medical knowledge sources for clinical nlp. among all the plms, we focus on bert since it often achieves the state of the art performance on diverse nlp tasks. the main contributions of our work include: • we conducted a comprehensive empirical analysis of the effectiveness of applying diverse knowledge integration techniques to combine medical knowledge encoded in umls with embeddings from pre-trained bert models for clinical relation extraction. in this section, we survey the representative work on topics that are most relevant to this research: (a) incorporating domain text in bert during model training and (b) combining knowledge graph infor-mation with bert. we also summarize the state-ofthe-art techniques for extracting clinical relations from text. ## methodology the main steps in this research include: (a) generating text embeddings using bert, (b) aligning the entities in text with the concepts in umls, (c) generating umls knowledge graph embeddings and (d) integrating umls knowledge with bert.