Enhancing Clinical BERT Embedding using a Biomedical Knowledge Base (2020)

Abstract: Domain knowledge is important for building Natural Language Processing (NLP) systems for low-resource settings, such as in the clinical domain. In this paper, a novel joint training method is introduced for adding knowledge base information from the Unified Medical Language System (UMLS) into language model pre-training for some clinical domain corpus. We show that in three different downstream clinical NLP tasks, our pre-trained language model outperforms the corresponding model with no knowledge base information and other state-of-the-art models. Specifically, in a natural language inference task applied to clinical texts, our knowledge base pre-training approach improves accuracy by up to 1.7%, whereas in clinical name entity recognition tasks, the F1-score improves by up to 1.0%. The pre-trained models are available at https://github.com/noc-lab/clinical-kb-bert .

## introduction natural language processing (nlp) is increasingly becoming an important tool in medical research. different nlp applications have been developed for assisting physicians, helping to increase the efficiency of performing a variety of tasks. for example, the clinical concept extraction task @xcite labels clinical findings, treatments, and tests contained in a clinical report, whereas the temporal relations extraction task @xcite identifies temporal expressions of the clinical concept. the need for these applications calls for better clinical natural language models. the recent advances in language model pre-training greatly boosted the performance of nlp models. compared with the traditional nlp-or word2vec-based techniques, the pre-trained language models capture the meaning of the words based on the context of each sentence. some pre-trained language models, including bert @xcite , albert @xcite , roberta @xcite , and xlnet @xcite , achieved state-of-the-art performance on most nlp tasks. these models can be further fine-tuned on a domain-specific corpus for specific tasks. in addition to a language model, knowledge base information can help to build a better nlp model. during language model pre-training, the representation of the words is learned from some given corpus. thus, the language model would not be able to produce a meaningful representation of a concept missing from the corpus. this problem is critical, especially in a low-resource setting like the clinical domain. on the other hand, a knowledge base contains a plethora of concepts and relations. leveraging such information has the potential to yield models that can generalize better without resorting to extremely large training corpora. for example, the unified medical language system (umls) provides 54 types of relations between biomedical concepts of 127 semantic types. it contains 27.7 million relations, which, as we show, can be leveraged to improve clinical nlp models for various downstream tasks. in this paper, we propose a novel method for fusing the knowledge base information into the language model pre-training stage. specifically, we use a negative sampling method to convert the relations in the knowledge base into a classification problem and introduce a new loss function during the pre-training that shapes the representation of the concept according to the relation graph in the knowledge base. we obtained two different pre-trained language models from biobert @xcite and albert @xcite and achieved state-of-the-art performance in two downstream nlp tasks. finally, an ablation study is performed for showing the effectiveness of our proposed method. ## related work pre-trained language model in the clinical domain. there are many previous works for pre-trained language models using biomedical domain corpora. for example, @xcite @xcite pre-trained different language models on the biomedical and clinical corpora and the models showed significant improvement on various downstream tasks. also, @xcite compared the performance of a model pre-trained with an open domain corpus to a model pre-trained with a clinical domain corpus, and concluded that the latter achieved superior performance. ## experimental results to evaluate our pre-trained clinical models, we selected two named entity recognition (ner) tasks: i2b2 2010 @xcite and i2b2 2012 @xcite , and one natural language inference (nli) task mednli @xcite . ## discussion the following ablation study is performed to verify if the knowledge base helps improve the performance of the pre-trained model. we pre-trained an albert model using mimic-iii without knowledge base information, and we reported the downstream model performances for different model configurations (see for the ner tasks, we follow the ner setting in @xcite . we have achieved a similar performance compared to models with additional bidirectional lstm layers @xcite . for the i2b2 2012, task, our albert model outperforms previous baselines. this suggests the effectiveness of the pre-trained language model with knowledge base information. ## conclusion we proposed a simple but effective way to include knowledge base information into clinical domainspecific language model pre-training. two pre-trained models, clinical kb-bert and clinical kb-albert, are obtained and are shown to achieve state-of-art results for two types of downstream tasks. an ablation study was performed to show the effectiveness of including the knowledge base. future work will evaluate the proposed models on additional clinical downstream tasks.