Entity-Enriched Neural Models for Clinical Question Answering (2020)

Abstract: We explore state-of-the-art neural models for question answering on electronic medical records and improve their ability to generalize better on previously unseen (paraphrased) questions at test time. We enable this by learning to predict logical forms as an auxiliary task along with the main task of answer span detection. The predicted logical forms also serve as a rationale for the answer. Further, we also incorporate medical entity information in these models via the ERNIE architecture. We train our models on the large-scale emrQA dataset and observe that our multi-task entity-enriched models generalize to paraphrased questions ~5% better than the baseline BERT model.

## introduction the field of question answering (qa) has seen significant progress with several resources, models and benchmark datasets. pre-trained neural language encoders like bert @xcite and its variants @xcite have achieved near-human or even better performance on popular open-domain qa tasks such as squad 2.0 @xcite . while there has been some progress in biomedical qa on medical literature @xcite , existing models have not been similarly adapted to clinical domain on electronic medical records (emrs). community-shared large-scale datasets like em-rqa @xcite allow us to apply stateof-the-art models, establish benchmarks, innovate and adapt them to clinical domain-specific needs. emrqa enables question answering from electronic medical records (emrs) where a question is asked by a physician against a patient's medical record context: the patient had an elective termination of her pregnancy on @xcite . the work-up for the extent of the patient's disease included mri scan of the cervical and thoracic spine which revealed multiple metastatic lesions in the vertebral bodies; a t 3 lesion extending from the body to the right neural for amina with foraminal obstruc tion. an abdominal and pelvic ct scan with iv contrast revealed bilateral pulmo nary nodules and bilateral pleural effusions, extensive liver metastases, narrowing of the intra hepatic ivc and distention of the azygous system suggestive of ivc obstructi on by liver metastases. question: how was the patient's extensive liver metastases diagnosed? paraphrase: what diagnosis was used for the patient's extensive liver metastases? logical form: {labevent (x) @xcite or procedureevent (x) [@xmath0, re @xmath1: an abdominal and pelvic ct scan with iv contrast (clinical notes). thus, we adapt these models for emr qa while focusing on model generalization via the following. (1) learning to predict the logical form (a structured semantic representation that captures the answering needs corresponding to a natural language question) along with the answer and (2) incorporating medical entity embeddings into models for emr qa. we now examine the motivation behind these. a physician interacting with a qa system on emrs may ask the same question in several different ways; a physician may frame a question as: "is the patient allergic to penicillin?" whereas the other could frame it as "does penicillin cause any allergic reactions to the patient?". since paraphrasing is a common form of generalization in natural language processing (nlp) @xcite , a qa model should be able to generalize well to such paraphrased question variants that may not be seen during training (and avoid simply memorizing the questions). however, current state-of-the-art models do not consider the use of meta-information such as the semantic parse or logical form of the questions in unstructured qa. in order to give the model the ability to understand the semantic information about answering needs of a question, we frame our problem in a multitask learning setting where the primary task is extractive qa and the auxiliary task is the logical form prediction of the question. fine-tuning on medical copora (mimic-iii, pubmed @xcite ) helps models like bert align their representations according to medical vocabulary (since they are previously trained on open-domain corpora such as wikitext @xcite ). however, another challenge for developing emr qa models is that different physicians can use different medical terminology to express the same entity; e.g., "heart attack" vs. "myocardial infarction". mapping these phrases to the same umls semantic type foot_0 as disease or syndrome (dsyn) provides common information between such medical terminologies. incorporating such entity information about tokens in the context and question can further improve the performance of qa models for the clinical domain. our contributions are as follows: ## problem formulation we formulate the emr qa problem as a reading comprehension task. given a natural language question (asked by a physician) and a context, where the context is a set of contiguous sentences from a patient's emr (unstructured clinical notes), the task is to predict the answer span from the given context. along with the (question, context, answer) triplet, also available as input are clinical entities extracted from the question and context. also available as input is the, logical form (lf) that is a structured representation that captures answering needs of the question through entities, attributes and relations required to be in the answer @xcite . a question may have multiple paraphrases where all paraphrases map to the same lf (and the same answer, fig. 1 ). ## methodology in this section, we briefly describe bert @xcite , ernie @xcite and our proposed model. ## datasets we used emrqa foot_1 and made foot_2 datasets for our experiments. we provide a brief summary of each dataset and the methodology followed to split these datasets into train and test sets. lf: medicationevent (|medication|) [@xmath2: a logical form (lf) and its respective question templates (paraphrases). the emrqa corpus has over 1m + question, logical form, and answer/evidence triplets, an example of a context, question, its logical form and a paraphrase is shown in fig 1 . the evidences are the sentences from the clinical note that are relevant to a particular question. there are total 30 logical forms in the emrqa dataset foot_3 . made made 1.0 @xcite dataset was hosted as an adverse drug reactions (adrs) and medication extraction challenge from emrs. this dataset was converted into a qa dataset by following the same procedure as enumerated in the literature of emrqa @xcite . made qa dataset is smaller than emrqa, as emrqa consists of multiple datasets taken from i2b2 @xcite whereas made only has specific relations and entity mentions to that of adrs and medications. this resulted in a clinical qa dataset which has different properties as compared to emrqa. made also has lesser number of logical forms (8 lfs) as compared to emrqa because of fewer entities and relations. the 8 lfs for made are provided in appendix b. ## experiments in this section, we briefly discuss the experimental settings, clinical entity extraction method, implementation details of our proposed model and evaluation metrics for our experiments. ## results and discussion in this section, we compare the results of all the models that we introduced in § 3. with the help of different experiments, we try to analyse whether the induced entity and logical form information does clinical entity information improve models' performance? across all settings, the f1score of cernie improves by ∼ 2-5% over bert and ∼ 0.75 -3% over cbert. the exact match performance improved by ∼ 3 -4.5 over bert and 1.5 -3.25% over cbert. also, as expected, the performance in sentence setting (-s) improved relatively more than it did in paragraph-setting. the entity-enriched tokens help in identifying the tokens which are required by the question. for example, in fig. 3 , the token 'infiltrative' in the question as well as the context get highlighted with the help of the identified entity 'topp' (therapeutic or preventive procedure) and then relevant tokens in the context, chest x ray, get highlighted with the relevant entity 'diap' (diagnostic procedure). this information aids the model in narrowing down its focus to highlighted diagnostic procedures in the context for answer extraction. question: how was diffuse infiltrative topp diagnosed fndg ? context: earlier that day, pt had a chest x ray diap which showed diffuse infiltrative topp process concerning for ards. answer: chest x ray figure 3 : an example of a question, context, their extracted entities and expected answer. ## related work in the general domain, bert-based models are on the top of different leader boards across various tasks, including qa tasks @xcite . the authors of @xcite applied bert to the ms-marco passage retrieval qa task and observed improvement over state of the art results. @xcite further extended the work by combining bert with re-ranking of predictions for queries that will be issued for each document. however, bert-based models have not been adapted to answering physician questions on emrs. in case of domain-specific qa, logical forms or semantic parse are typically used to integrate the domain knowledge associated with a kb-based (knowledge base) structured qa datasets, where a model is learnt for mapping a natural language question to a lf. geoquery (zelle and mooney, 1996), and atis @xcite , are the oldest known manually generated question-lf annotations on closed-domain databases. qald @xcite , free 917 @xcite , simplequestions @xcite contain hundreds of hand-crafted questions and their corresponding database queries. prior work has also used lfs as a way to generate questions via crowdsourcing @xcite . webquestions @xcite ) contains thousands of questions from google search where the lfs are learned as latent representations in helping answer questions from freebase. prior work has not investigated the utility of logical forms in unstructured qa, especially as a means to generalize the qa model across different paraphrases of a question. there have been efforts on using multi-task learning for efficient question answering, such as the authors of @xcite tried to learn multiple tasks together resulting in an overall boost in the performance of the model on squad @xcite . similarly, the authors of @xcite also utilised the information across different tasks which lie at the intersection of vision and natural language processing to improve the performance of their model across all tasks. the authors of @xcite utilised weak supervision to the model while predicting the answer but not much work has been done to incorporate the logical form of the question for unstructured question answering in a multi-task setting. hence, we decided to explore this direction and incorporate the structured semantic information of the questions for extractive question answering. ## conclusion the proposed entity-enriched qa models trained with an auxiliary task improve over the state-of-theart models by about 3 -6% across the large-scale clinical qa dataset, emrqa @xcite (as well as made @xcite ). we also show that multitask learning for logical forms along with the answer results in better generalizing over unseen paraphrases for emr qa. the predicted logical forms also serve as an accompanying justification to the answer and help in adding credibility to the predicted answer for the physician.