U mls BERT : Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the U nified M edical L anguage S ystem M etathesaurus (2021)

Abstract: Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have achieved state-of-the-art results in biomedical natural language processing tasks by focusing their pre-training process on domain-specific corpora. However, such models do not take into consideration structured expert domain knowledge from a knowledge base. We introduce UmlsBERT, a contextual embedding model that integrates domain knowledge during the pre-training process via a novel knowledge augmentation strategy. More specifically, the augmentation on UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus is performed in two ways: i) connecting words that have the same underlying ‘concept’ in UMLS and ii) leveraging semantic type knowledge in UMLS to create clinically meaningful input embeddings. By applying these two strategies, UmlsBERT can encode clinical domain knowledge into word embeddings and outperform existing domain-specific models on common named-entity recognition (NER) and clinical natural language inference tasks.

## introduction in recent years, the volume of data being collected in healthcare has grown considerably. a significant proportion of the data is in text form, which requires advanced natural language processing (nlp) models to process. this has led to the creation of high-performing, optimized nlp models focused on the biomedical domain. contextual word embedding models, such as elmo @xcite and bert @xcite have achieved state-of-the-art results in many nlp tasks. initially tested in a general domain, these models have also been successfully applied in the biomedical domain by pre-training them on biomedical corpora, leading to the best performances in a variety of biomedical nlp tasks @xcite , @xcite . however, current biomedical applications of transformer-based natural language understanding (nlu) models do not incorporate structured expert domain knowledge from a knowledge base into their embedding pre-training process. the unified medical language system (umls) @xcite metathesaurus is a compendium of many biomedical terminologies with the associated information, such as synonyms and categorical groupings. it allows for the connection of words that represent the same or similar 'concept'. for example, the words 'lungs' and 'pulmonary' share a similar meaning and thus can be mapped to the same concept unique identifier (cui) cui: c0024109. additionally, umls allows the grouping of concepts according to their semantic type @xcite . for example, 'skeleton' and 'skin' have the same 'body system' semantic type, and 'inflammation' and 'bleed' are in the 'pathologic function' semantic type. in this paper, we present and publicly release foot_0 a novel architecture for augmenting contextual embeddings with clinical domain knowledge. specifically: (i) we are the first, to the best of our knowledge, to propose the usage of domain (clinical) knowledge from a clinical metathesaurus (umls metathesaurus) in the pre-training phase of a bert-based model (umlsbert) in order to build 'semantically enriched' contextual representations that will benefit from both the contextual learning (bert architecture) and the domain knowledge (umls metathesaurus). @xcite we propose a new multi-label loss function for the pre-training of the masked language modelling (masked lm) task in the umlsbert that incorporates the connections between clinical words using the cui attribute of umls. (iii) we introduce a semantic type embedding that enriches the input embeddings process of the umlsbert by forcing the model to take into consideration the association between words that are of the same semantic type. (iv) finally, we demonstrate that umlsbert outperforms two popular clinical-based bert models (biobert and bio_clinicalbert) and a general domain bert model on different clinical named-entity recognition (ner) tasks and on one clinical natural language inference task. the rest of paper is organized as follows. related work is presented in section 2. the data that were used to pre-train and test the new umlsbert are described in section 3. the characteristics of the proposed umlsbert architecture for augmenting contextual embeddings with clinical knowledge are detailed in section 4. finally, the results of the down-stream tasks and the qualitative analysis are reported in section 5, and a conclusion and a plan for future work are presented in section 6. ## related work in @xcite , contextualized word embeddings were introduced in a bidirectional language model (elmo). this allowed the model to change the embedding of a word based on its imputed meaning, which was derived from the surrounding context. subsequently, @xcite proposed the bidirectional encoder representations from transformers (bert) which used bidirectional transformers @xcite to create context-dependent representations. for both models, pre-training is done on massive corpora and the context-sensitive embeddings can be used for downstream tasks. other approaches enhance the bert's performance by injecting external knowledge from a knowledge base. sense-bert @xcite is pre-trained to predict the supersenses (semantic class) of each word by incorporating lexical semantics (from the lexical database wordnet @xcite ) into the model's pre-training objective and by adding supersense information to the input embedding. in addition, glossbert @xcite focuses on improving word sense disambiguation by using context-gloss pairs on the sentence-pair classification task of a bert model. furthermore, there have been multiple attempts to improve the performance of contextual models in the biomedical domain. biobert is a bertbased model which was pre-trained on both general (bookscorpus and english wikipedia) and biomedical corpora (pubmed abstracts and pubmed cen-tral full-text articles) @xcite . the authors demonstrate that incorporating biomedical corpora in the pre-training process improves the performance of the model in downstream biomedical tasks. this is likely because medical corpora contains terms that are not usually found in a general domain corpus @xcite . finally, bio_clinicalbert @xcite further pre-trains biobert on clinical text from the mimic-iii v1.4 database @xcite . it is shown that the usage of clinical specific contextual embeddings can be beneficial for the performance of a model on different clinical nlp downstream tasks. ## data we use the multiparameter intelligent monitoring in intensive care iii (mimic-iii) dataset @xcite to pre-train the umlsbert model. mimic dataset consists of anonymized electronic medical records in english of over forty-thousand patients who were admitted to the intensive care units of the beth israel deaconess medical center (boston, ma, usa) between 2001 and 2012. in particular, umlsbert is trained on the no-teevents table, which contains 2,083,180 rows of clinical notes and test reports. ## results in this section, we present the results of an empirical evaluation of the umlbert model. in particular, we provide a comparison between different available bert models to show the efficiency of our proposed model on different clinical nlp tasks. in addition, we provide the results of an ablation test to exam the effect of the semantic type embeddings on the performance of the model. furthermore, we conduct a qualitative analysis of the embedding of each model in order to illustrate how medical knowledge improves the quality of medical embeddings. finally, we provide a visualized comparison of the embeddings of the words that are associated with semantic types between umls-bert and bio_clinicalbert. ## conclusion and future work this paper presents umlsbert, a novel bertbased architecture that incorporates domain (biomedical) knowledge in the pre-training process of a contextual word embeddings model. we demonstrate that umlsbert can learn the association of different clinical terms with similar meaning in the umls metathesaurus. umlsbert can also create more meaningful input embeddings by leveraging information from the semantic type of each (biomedical) word. finally, we confirm that these modifications can improve the model's performance as our umlsbert model outperforms other biomedical bert models in various downstream tasks. as for future work, we plan to address the limitations of this study including: (i) examining the effect of augmenting contextual embeddings with medical knowledge when more complicated layers are used atop of the output embedding of umls-bert. (ii) exploring the umls hierarchical associations between words that extend the concept connection that we investigated in this paper. (iii) testing our model in other datasets and biomedical tasks (e.g. relation extraction task @xcite ) to investigate further the strengths and weaknesses of our model.