I nfusing D isease K nowledge into BERT for H ealth Q uestion A nswering, M edical I nference and D isease N ame R ecognition (2020)

Abstract: Knowledge of a disease includes information of various aspects of the disease, such as signs and symptoms, diagnosis and treatment. This disease knowledge is critical for many health-related and biomedical tasks, including consumer health question answering, medical language inference and disease name recognition. While pre-trained language models like BERT have shown success in capturing syntactic, semantic, and world knowledge from text, we find they can be further complemented by specific information like knowledge of symptoms, diagnoses, treatments, and other disease aspects. Hence, we integrate BERT with disease knowledge for improving these important tasks. Specifically, we propose a new disease knowledge infusion training procedure and evaluate it on a suite of BERT models including BERT, BioBERT, SciBERT, ClinicalBERT, BlueBERT, and ALBERT. Experiments over the three tasks show that these models can be enhanced in nearly all cases, demonstrating the viability of disease knowledge infusion. For example, accuracy of BioBERT on consumer health question answering is improved from 68.29% to 72.09%, while new SOTA results are observed in two datasets. We make our data and code freely available.

## introduction human disease is "a disorder of structure or function in a human that produces specific signs or symptoms" (oxford-english-dictionary, 2020). disease is one of the fundamental biological entities in biomedical research and consequently it is frequently searched for in the scientific literature @xcite and on the internet @xcite . knowledge of a disease includes information about various aspects of the disease, like the signs https://github.com/heyunh2015/diseasebert ## proposed method: disease knowledge infusion training in this section, we propose a new training task: disease knowledge infusion training. our goal is to integrate bert-like pre-trained language models with disease knowledge to achieve better performance on a variety of medical domain tasks including answering health questions, medical language inference, and disease name recognition. our approach is guided by three questions: which diseases and aspects should we focus on? how do we infuse disease knowledge into bert-like models? what is the objective function of this training task? ## experiments in this section, we examine disease knowledge infusion into six bert variants over three diseaserelated tasks: health question answering, medical language inference, and disease name recognition. reproducibility: the code and data in this paper is released. a model is firstly initialized with the pre-trained parameters from bert or its variants and then is further trained by disease knowledge infusion to capture the disease knowledge. we use a widely used pytorch implementation of bert and adam as the optimizer. we empirically set learning rate as 1e-5, batch size as 16 and as 10. because mesh (5,853 disease terms) is chosen as the disease vocabulary in our experiments, as a smaller vocabulary compared with others like umls (540,000 disease terms), we obtain a relatively small dataset of 14,617 passages. hence, the training of disease knowledge infusion is as fast as fine-tuning bert over downstream datasets, which takes 2-4 epochs to enhance bert for a better performance on downstream tasks, which will be discussed in section 4.5. the training is performed on one single nvidia v100 gpu and it takes about 10 minutes to complete one training epoch using bert-base architecture. the reproducibility for fine-tuning over downstream tasks will be detailed in section 4.2. ## conclusions in this paper, we propose a new disease infusion training procedure to augment bert-like pre-trained language models with disease knowledge. we conduct this training procedure on a suite of bert models and evaluate them over diseaserelated tasks. experimental results show that these models can be enhanced by this disease infusion method in nearly all cases.