Medical Knowledge-enriched Textual Entailment Framework (2020)

Abstract: One of the cardinal tasks in achieving robust medical question answering systems is textual entailment. The existing approaches make use of an ensemble of pre-trained language models or data augmentation, often to clock higher numbers on the validation metrics. However, two major shortcomings impede higher success in identifying entailment: (1) understanding the focus/intent of the question and (2) ability to utilize the real-world background knowledge to capture the con-text beyond the sentence. In this paper, we present a novel Medical Knowledge-Enriched Textual Entailment framework that allows the model to acquire a semantic and global representation of the input medical text with the help of a relevant domain-specific knowledge graph. We evaluate our framework on the benchmark MEDIQA-RQE dataset and manifest that the use of knowledge-enriched dual-encoding mechanism help in achieving an absolute improvement of 8.27% over SOTA language models.

## introduction the entailment task is similar with natural language inference (nli), involves identifying the semantic similarity between two natural language texts, premise (p ) and hypothesis (h). the nli task's effectiveness is crucial for developing a robust natural language understanding system that functions at a human level @xcite @xcite . recent literature suggests the use of contemporary language models (lms) @xcite , often ensembled, to achieve better performance @xcite @xcite on the nli task. however, our qualitative interpretation of the dataset and results suggests that lms fails when it comes to textual entailment (te), despite being the on-demand language model. the limitations belong to two major categories: does the medical textual entailment task benefit from the external domain knowledge to distinguish semantically identical medical sentences to recognize entailment? premise: what is the possibility of atypical pneumonia occurring again less than a month after treatment? hypothesis: what are the possible treatments for atypical pneumonia? contributions: (i) proposed medical-te framework, by utilizing domain-specific medical kgs to encode the global and semantic information of the premise and hypothesis, (ii) exploited the capabilities of semantic units in a graph network to encode medical-entities centered factual information for recognizing textual entailment, and (iii) evaluated the effectiveness of the proposed method over state-of-the-art language models and knowledge-infused baseline methods on the benchmark mediqa-rqe dataset. related work : the development of annotated te and nli medical datasets @xcite @xcite and a variety of pre-trained language models has led to a rise of extensive ongoing research in this field. majority of the systems developed for the te task adopts the multi-task learning (mtl) framework @xcite @xcite @xcite , ensemble method @xcite , and transfer learning @xcite for achieving better accuracy. @xcite employed the mtl approach @xcite @xcite in te task to learn from the auxiliary tasks of question answering (qa) and nli. the best performing system at medqa 2019-rqe shared task @xcite utilized the mtl approach to learn from intermediate nli task. further they used knowledge distillation approach to condense the information obtained from various models and transfer it into an single model. few of the works @xcite have explored the usage of the background knowledge or medical kgs @xcite in extracting information for the entailment task. however, the consideration of building a vocabulary graph from the textual and later enriching them with information from the medical kgs is still an unexplored territory. ## proposed approach the overall architecture of the proposed sem-kgn is illustrated in fig. -1. the rest of the section elaborates sem-kgn in detail. ## experimental results and analysis dataset and metrics: we used widely adopted benchmark entailment dataset, mediqa-rqe created by @xcite , released in the bionlp 2019 shared task. the dataset is derived from consumer health questions (chqs) and frequently asked questions (faqs) from the u.s. national library of medicine and national institute of health respectively. the training and validation set consists of total 8890 chq and faq pairs with the entail label of 4784 instances. the test set consist of 230 pairs with 115 entail labels. we used official evaluation metrics (accuracy) to evaluate our model. additionally, we also provided the precision, recall, and f1-scores for the evaluation. implementation details: we have chosen models' hyper-parameters empirically on the validation set. the base-uncased version of bert foot_2 of hidden size 768 with a max sequence length of 200 (160 for p and 40 for h) is used in all experiments reported in the paper. the size of dictionary to create the dictionary matrix was 30000. the dimension of the threshold of npmi is set to 0.3 to obtain meaningful relation between words. the last layer's hidden size of the graph-expanded knowledge encoder is set to 16. we use the adam optimiser (kingma and ba, 2014) for parameters update after every epoch of training. we set the standard batch size of 16, and trained for 5 epochs with a dropout rate of 0.2 and 2e -5 learning rate in all the experimental results reported in this work. baseline models: to show the effectiveness of sem-kgn, we adopted following competitive baseline models: sample examples and error type premise hypothesis true label biobert+ ki sem-kgn example 1 can you mail me patient information about glaucoma, i was recently diagnosed and want to learn all i can about the disease. how is glaucoma diagnosed ? non-entailed non-entailed non-entailed example 2 i was writing to inquire about more information regarding the diagnosis of oi. we have family members who are in the process of waiting for genetic testing to come back but are under allegations of child abuse. is there any information that may be helpful to us? how to diagnose osteogenesis imperfecta ? entailed non-entailed entailed error type 1: complex and long question hello, my dad, 68 years old, has gastritis, it did ache occasionally over the last several years. the other day, he went to hospital to have medical check-up with endoscopic ultrasonography, and found gist with about 1cm in size. dr. told him that he may consider surgery or not, it is up to him. what are we supposed to do? how is an endoscopic ultrasound performed ? non-entailed entailed non-entailed error type 2: ambiguous question can you please send me as much information as possible on "hypothyroidism". i was recently diagnosed with the disease and i am struggling to figure out what it is and how i got it. how is hypothyroidism diagnosed ? non-entailed entailed non-entailed the first entry shows the effectiveness of sem-kgn in being able to understand the focus of the premise and the hypothesis which is achieved by the dual encoding mechanism. the second entry in the table affirms the importance of domain-specific kgs in assimilating medical information. ## conclusion in this paper, we proposed a framework sem-kgn to recognize medical textual entailment. our framework utilized the local context from bert based document encoder and global context by expanding the vocabulary graph with the medical entities obtained from the medical knowledge-bases. we present an efficient aggregator scheme to fuse the multiple encoding. the proposed sem-kgn framework outperformed the competent pre-trained language model and knowledge-graph enabled language model architectures with fair margin. in future, we plan to explore multi-domain knowledge-graph and efficient graph embedding based techniques for medical textual entailment.