Clinical Reading Comprehension: A Thorough Analysis of the emr QA Dataset (2020)

Abstract: Machine reading comprehension has made great progress in recent years owing to large-scale annotated datasets. In the clinical domain, however, creating such datasets is quite difficult due to the domain expertise required for annotation. Recently, Pampari et al. (EMNLP’18) tackled this issue by using expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes. In this paper, we provide an in-depth analysis of this dataset and the clinical reading comprehension (CliniRC) task. From our qualitative analysis, we find that (i) emrQA answers are often incomplete, and (ii) emrQA questions are often answerable without using domain knowledge. From our quantitative experiments, surprising results include that (iii) using a small sampled subset (5%-20%), we can obtain roughly equal performance compared to the model trained on the entire dataset, (iv) this performance is close to human expert’s performance, and (v) BERT models do not beat the best performing base model. Following our analysis of the emrQA, we further explore two desired aspects of CliniRC systems: the ability to utilize clinical domain knowledge and to generalize to unseen questions and contexts. We argue that both should be considered when creating future datasets.

## introduction medical professionals often query over clinical notes in electronic medical records (emrs) to find information that can support their decision making @xcite @xcite . one way to facilitate such information seeking activities is to build a natural language question answering (qa) system that can extract precise answers from clinical notes @xcite @xcite @xcite . 1 our code is available at https://github.com/ xiangyue9607/clinirc . context: ... for htn control, pt was given hctz and lopressor which sufficiently controlled his bp. pt was sent home on hctz 25mg daily and atenolol 50mg daily. ... additional comments: 1.) take hydrochlorothiazide 25mg daily and atenolol 50mg daily for your blood pressure. you should also take aspirin 81mg daily. question: what was the dosage prescribed of hydrochlorothiazide? answer: additional comments: 1.) take hydrochlorothiazide 25mg daily and atenolol 50mg daily for your @xcite question: why has the patient been prescribed hctz? answer: for htn control, pt was given hctz and lopressor which sufficiently machine reading comprehension (rc) aims to automatically answer questions based on a given document or text corpus and has drawn wide attention in recent years. many neural models @xcite @xcite @xcite have achieved very promising results on this task, owing to large-scale qa datasets @xcite @xcite @xcite . unfortunately, clinical reading comprehension (clinirc) has not observed as much progress due to the lack of such qa datasets. in order to create qa pairs on clinical texts, annotators must have considerable medical expertise and data handling must be specifically designed to address ethical issues and privacy concerns. due to these requirements, using crowdsourcing like in the open domain to create large-scale clinical qa datasets becomes highly impractical @xcite . recently, @xcite found a smart way to tackle this issue and created emrqa, the first large-scale qa dataset on clinical texts. instead of relying on crowdsourcing, emrqa was semiautomatically generated based on annotated question templates and existing annotations from the n2c2 (previously called i2b2) challenge datasets 2 . example qa pairs from the dataset are shown in figure 1 . in this paper, we aim to gain a deep understanding of the clinirc task and conduct a thorough analysis of the emrqa dataset. we first explore the dataset directly by carrying out a meticulous qualitative analysis on randomly-sampled qa pairs and we find that: 1) many answers in the emrqa dataset are incomplete and hence are hard to read and ineffective for training ( §3.1). 2) many questions are simple: more than 96% of the examples contain the same key phrases in both questions and answers. though @xcite claims that 39% of the questions may need knowledge to answer, our error analysis suggests only a very small portion of the errors (2%) made by a state-of-theart reader might be due to missing external domain knowledge ( §3.2). following our qualitative analysis of the emrqa dataset, we conduct a comprehensive quantitative analysis based on state-of-the-art readers and bert models (bert-base @xcite as well as its biomedical and clinical versions: biobert @xcite and clinicalbert @xcite ) to understand how different systems behave on the emrqa dataset. surprising results include: 1) using a small sampled subset (5%-20%), we can obtain roughly equal performance compared to the model trained on the entire dataset, suggesting that many examples in the dataset are redundant ( §4.1). after completing our analysis of the dataset, we explore two potential needs for systems doing clinirc: 1) the need to represent and use clinical domain knowledge effectively ( §5.1) and 2) the need to generalize to unseen questions and contexts ( §5.2). eral types of clinical questions that require domain knowledge and can frequently appear in the real clinical setting. we also carry out an experiment showing that adding knowledge explicitly yields around 5% increase in f1 over the base model when tested on samples that we created by altering the original questions to involve semantic relations. to study generalizability, we ask medical experts to create new questions based on the unseen clinical notes from mimic-iii @xcite , a freely accessible critical care database. we find that the performance of the best model trained on emrqa drops by 40% under this new setting, showing how critical it is for us to develop more robust and generalizable models for the clinirc task. in summary, given our analysis of the emrqa dataset and the task in general, we conclude that future work still needs to create better datasets to advance clinirc. such datasets should be not only large-scale, but also less noisy, more diverse, and allow researchers to directly evaluate a system's ability to encode domain knowledge and to generalize to new questions and contexts. ## overview of the emrqa dataset similar to the open-domain reading comprehension task, the clinical reading comprehension (clinirc) task is defined as follows: definition 2.1. given a patient's clinical note (context) @xmath0, ..., c n } and a question @xmath1, ..., t m }, the clinirc task aims to extract a continuous span @xmath2, c i+1 , ..., c i+k }(1 ≤ @xmath3, where c i , t j are tokens. the emrqa dataset @xcite was semi-automatically generated from expertannotated question templates and existing i2b2 annotations. more specifically, clinical question templates were first created by human experts. then, manual annotations from the medication information extraction, relation learning, and coreference ## in-depth qualitative analysis in this section, we carry out an in-depth analysis of the emrqa dataset. we aim to examine (1) the quality and (2) level of difficulty for the generated qa pairs in the emrqa dataset. ## comprehensive quantitative analysis in this section, we conduct comprehensive experiments on the emrqa dataset with state-of-theart readers and recently dominating bert models. full experimental settings are described in appendix a. ## desiderata in real-world clinirc following our analysis of the emrqa dataset, we further study two aspects of clinical reading comprehension systems that we believe are crucial for their real-world applicability: the need to encode clinical domain knowledge and to generalize to unseen questions and documents. ## related work we present a brief overview of open-domain, biomedical and clinical question answering tasks, which are most related to our work: question answering (qa) aims to automatically answer questions asked by humans based on external sources, such as web @xcite , knowledge base @xcite and free text @xcite . as an important type of qa, reading comprehension intends to answer a question after reading the passage @xcite . recently, the release of large-scale rc datasets, such as @xcite , stanford question-answering dataset (squad) @xcite makes it possible to solve rc tasks by building deep neural models @xcite @xcite . more recently, contextualized word representations and pretrained language models, such as elmo @xcite , gpt @xcite , bert @xcite , have been demonstrated to be very useful in various nlp tasks including rc. by seeing diverse contexts in large corpora, these pretrained language models can capture the rich semantic meaning and produce more accurate and precise representations for words given different contexts. even a simple classifier or score function built upon these pretrained contextualized word representations perform well in extracting answer spans @xcite . biomedical and clinical qa. due to the lack of large-scale annotated biomedical or clinical data, qa and rc systems in these domains are often rule-based and heuristic feature-based @xcite @xcite . in recent years, bioasq challenges @xcite proposed the biomedical semantic qa task, where the participants need to respond to each test question with relevant articles, snippets and exact answers. šuster and daelemans (2018) use summary points of clinical case reports to build a large-scale cloze-style dataset (clicr), which is similar to the style of cnn & daily mail dataset. @xcite presents pubmedqa, which extracts question-style titles and their corresponding abstracts as the questions and contexts respectively. a few qa pairs are annotated by human experts and most of them are annotated based a simple heuristic rule with "yes/no/maybe". due to the great power of contextualized word representations, pretrained language models also have been introduced to biomedical and clinical domain, e.g., bioelmo @xcite , biobert @xcite , and clinicalbert @xcite . they adopt similar architectures of the original models but pretrained on the medical and clinical corpus, such as pubmed articles and mimic-iii @xcite clinical notes. ## conclusion we study the clinical reading comprehension (clinirc) task with the recently created emrqa dataset. our qualitative and quantitative analysis as well as exploration of the two desired aspects of clinirc systems show that future clinical qa datasets should not only be large-scale but also less noisy and more diverse. moreover, questions that involve complex relations and are across different domains should be included, and then more advanced external knowledge incorporation methods as well as domain adaptation methods can be carefully designed and systematically evaluated.