Improving Biomedical Pretrained Language Models with Knowledge (2021)

Abstract: Pretrained language models have shown success in many natural language processing tasks. Many works explore to incorporate the knowledge into the language models. In the biomedical domain, experts have taken decades of effort on building large-scale knowledge bases. For example, UMLS contains millions of entities with their synonyms and defines hundreds of relations among entities. Leveraging this knowledge can benefit a variety of downstream tasks such as named entity recognition and relation extraction. To this end, we propose KeBioLM, a biomedical pretrained language model that explicitly leverages knowledge from the UMLS knowledge bases. Specifically, we extract entities from PubMed abstracts and link them to UMLS. We then train a knowledge-aware language model that firstly applies a text-only encoding layer to learn entity representation and then applies a text-entity fusion encoding to aggregate entity representation. In addition, we add two training objectives as entity detection and entity linking. Experiments on the named entity recognition and relation extraction tasks from the BLURB benchmark demonstrate the effectiveness of our approach. Further analysis on a collected probing dataset shows that our model has better ability to model medical knowledge.

## introduction large-scale pretrained language models (plms) are proved to be effective in many natural language processing (nlp) tasks @xcite . however, there are still many works that explore multiple strategies to improve the plms. firstly, in specialized domains (i.e biomedical domain), many works demonstrate that using indomain text (i.e. pubmed and mimic for biomedical domain) can further improve downstream tasks figure 1 : an example of the biomedical sentence. two entities "glycerin" and "inflammation" are linked to c0017861 (1,2,3-propanetriol) and c0011603 (dermatitis) respectively with a relation triplet (c0017861, may_prevent, c0011603) in umls. as an applied discipline that needs a lot of facts and evidence, the biomedical and clinical fields have accumulated data and knowledge from a very early age @xcite . one of the most representative work is unified medical language system (umls) @xcite ) that contains more than 4m entities with their synonyms and defines over 900 kinds of relations. figure 1 shows an example. there are two entities "glycerin" and "inflammation" that can be linked to c0017861 (1,2,3-propanetriol) and c0011603 (dermatitis) respectively with a may_prevent relation in umls. as the most important facts in biomedical text, entities and relations can provide information for better text understanding @xcite . to this end, we propose to improve biomedical plms with explicit knowledge modeling. firstly, we process the pubmed text to link entities to the knowledge base. we apply an entity recognition and linking tool scispacy @xcite to annotate 660m entities in 3.5m documents. secondly, we implement a knowledge enhanced language model based on @xcite , which performs a text-only encoding and a text-entity fusion encoding. text-only encoding is responsible for bridging text and entities. text-entity fusion encoding fuses information from tokens and knowledge from entities. finally, two objectives as entity extraction and linking are added to learn better entity representations. to be noticed, we initialize the entity embeddings with transe @xcite , which leverages not only entity but also relation information of the knowledge graph. we conduct experiments on the named entity recognition (ner) and relation extraction (re) tasks in the blurb benchmark dataset. results show that our kebiolm outperforms the previous work with average scores of 87.1 and 81.2 on 5 ner datasets and 3 re datasets respectively. furthermore, our kebiolm also achieves better performance in a probing task that requires models to fill the masked entity in umls triplets. we summary our contributions as follows foot_0 : ## approach in this paper, we assume to access an entity set @xmath0, ..., e t }. for a sentence @xmath1, ..., x n }, we assume some spans @xmath2, ..., x j ) can be grounded to one or more entities in e. we further assume the disjuncture of these spans. in this paper, we use umls to set the entity set. ## experiments in this section, we first introduce the pretraining details of kebiolm. then we introduce the blurb datasets for evaluating our approach. finally, we introduce a probing dataset based on umls triplets for evaluating knowledge modeling. ## conclusions in this paper, we propose to improve biomedical pretrained language models with knowledge. we query & answer cui scibert kebiolm omalizumab may treat [mask] migraine asthma c0004096: asthma the disease severe allergic asthma phentolamine may diagnose [mask] depression pheochromocytoma c0031511: phaeochromocytoma the serotonin syndrome renovascular hypertension [mask] is noun form of hepatic it liver c0023884: liver the form of hepatic hepatic only [mask] may prevent tetanus it vaccination c0305062: tetanus toxoid bcg vaccination prophylactic tetanus vaccination