emr KBQA : A Clinical Knowledge-Base Question Answering Dataset (2021)

Abstract: We present emrKBQA, a dataset for answering physician questions from a structured patient record. It consists of questions, logical forms and answers. The questions and logical forms are generated based on real-world physician questions and are slot-filled and answered from patients in the MIMIC-III KB through a semi-automated process. This community-shared release consists of over 940000 question, logical form and answer triplets with 389 types of questions and ~7.5 paraphrases per question type. We perform experiments to validate the quality of the dataset and set benchmarks for question to logical form learning that helps answer questions on this dataset.

## introduction the last decade has seen widespread adoption of electronic health records (ehrs) across hospitals and clinics in the us @xcite . physicians often seek answers to questions from a patient's ehr to support clinical decision-making @xcite . it is not too hard to imagine a future where a physician interacts with an ehr system and asks it complex questions and expects precise answers, with adequate context, from a patient's record @xcite . central to such a world is a medical question answering system that processes natural language questions asked by physicians and finds answers to the questions in structured and unstructured sources in the patient's record. however, the longitudinal, domain specific nature of patient records along with privacy concerns makes it difficult to develop large-scale annotated datasets for training machine learning models. this motivated @xcite to develop the first community-shared patient qa dataset, emrqa, using a semi-automated process and create a large- ## related work the question answering (qa) problem is usually defined over unstructured texts or structured knowledge bases (kb qa). in case of kb qa, questions are usually mapped to logical forms (or a query language using sql, sparql, etc.) @xcite ) that are then used to retrieve the answer. in the medical domain, there is limited prior work on answering patient-specific questions over structured clinical data. @xcite introduce target logical form definitions and present a rule based method for converting natural language questions over structured data in the ehr into logical forms. they work with a dataset of 446 questions collected during clinician icu visits and propose an approach using question decomposition, concept recognition and normalization, and rule based semantic parsing. however, the questions and logical forms were not publicly released. in contrast, we present a large-scale communityshared dataset of over 900k generated questions from 52 unique question templates, logical forms and answers. more recently, @xcite create a new large-scale question-sql pair dataset (mimic-sql) on the mimic-iii dataset, again using the generation process as in @xcite . they propose a deep learning based translate-edit model for question-to-sql generation that adapts the widely used sequence-to-sequence model to directly generate the sql query for a given question, and also performs edits using an attentivecopying mechanism. the questions in the dataset are always asked over a patient-cohort such as "how many patients had the diagnosis icd9 code 53190?". however, the questions in emrqa are specific to a patient. this makes a big difference as the corpus for answering is smaller (limited to the patient's record, which may include several admissions), the answers may be viewed in conjunction with answers from the unstructured record, the type of questions asked varies, and redundancy and variability in answers to the same question may affect model performance. @xcite construct an ehr qa dataset from mimic-iii where the question-answer pairs are represented in sql (table-based) and sparql (graph-based). here again, the questions are defined over patient cohorts; e.g., "what number of married patients suffered from other convulsions?", making it inherently different from the emrkbqa task. they construct a knowledge graph by relating tables in the database and explore both table-based and graph-based qa (using sparql). emrkbqa maps questions to logical forms based on a schema of entities and relations. the tables and columns in the kb are mapped to the entities and attributes in the schema. logical forms capturing the information need expressed in the question are then instantiated from this schema. thus, emrkbqa instantiates logical forms from a relational schema (representing entities and relations typically found in the ehr) and facilitates a query language/ resource independent way of representing questions and answering them beyond just individual tables in the kb. kb-based qa datasets (question semantic parsing) use annotated question and logical form pairs for supervision where the logical forms (that can be then easily be mapped to any query language) are used to retrieve answers from a database @xcite @xcite . emrkbqa provides a dataset that can be used to train models to retrieve answers to natural language questions (by mapping them to logical forms) from the structured part of the ehr. the logical forms are instantiated from a schema that captures domain entities, attributes and relations proposed in emrqa @xcite . we demonstrate the value of the dataset by answering natural language questions posed by physicians as follows. we first train state of the art sequence models for semantic parsing to map questions to (query-language agnostic) logical forms. we then map the learned logical forms to the desired query language (sql) using a deterministic process. questions. emrkbqa contains natural lan- to also insert instances of just month and day, or date without time when slot-filling (along with using the original format). doing so ensures that the questions are more likely to be naturally asked. as in this example, the questions are patientspecific and the expected answer is in the structured part of the patient record. each question template is also assigned one or more question types, which is a new field (not in emrqa) to further categorize question templates in emrkbqa. question type can take one or more of the following values: a side-effect of the generation process (slotfilling) is that all yesno questions have a yes answer. we counter this by also generating questions where the answer will be no. we do this by slotfilling |problem|, |test|, |medication|, |treatment| based on the question and using top 50 most frequently occurring entities in appropriate tables (based on the entity type). some of these questions are now bound to have no as the answer when applied to our patient set. the types of questions are a consequence of the questions provided by the physicians who were polled for the initial question set. this was independent of any underlying data and simply based on what they would want to know about their own patients. while several other questions may be answerable on any underlying kb (like mimic), we wanted the question set to reflect what an actual physician may want to know from a patient record. logical forms. logical forms are a structured representation that capture the information need expressed in the question through entities, relations and attributes and are generated as a by-product of the emrqa generation process. they provide a human-comprehensible symbolic representation, linking questions to answers, and help build interpretable models critical to the medical domain @xcite . they are formally defined by @xcite in em-rqa. they encapsulate how we are answering a question (since that can be subjective). they are instantiated from a schema representing entities and relations found in the ehr. we use the same events and attributes from emrqa logical forms used in emrkbqa mapped to mimic-iii schema vitalevent • test • vitalname • date • result • status procedureevent • test • procedurename • date • result • status • abnormalresultflag labevent • test • labname • date • result • status • abnormalresultflag labevents, d_labitems d_labitems.label labevents.charttime labevents.value & labevents.valueuom labevents.flag not available medicationevent • medication/treatment • medicationname • startdate • enddate • strength • route • formulation • dosage prescriptions prescriptions.drug prescriptions.startdate prescriptions.enddate prescriptions.route prescriptions.prod_strength prescriptions.form_unit_disp not available procedureevent • treatment • procedurename • date • status procedureevents_mv, d_items (@xmath0, "peritoneal dialysis", or "ventilation") d_items.label procedureevents_mv.starttime not available emrqa question template entities emrqa logical form attributes mimic-iii data tables and fields smokingquitevent • quitdate • method smokinguseevent • istobaccouser • yearsofuse • packperday chartevents, d_items (@xmath1.label: chartevents.value not available not available conditionevent • problem • conditionname • diagnosisdate • status diagnoses_icd, d_icd_diagnoses, admissions d_icd_diagnoses.long_title prior to admissions.dischtime not available chartevents, admissions (@xmath2.value prior to admissions.admittime not available or figure 3: mapping between emrkbqa schema entities, attributes and tables (yellow boxes) and columns in mimic (shown in blue). see mimic schema for a description of mimic table and column names(johnson et al., 2016) in our example, the logical form for question template "is the patient on |medication|?" would be annotated as "medicationevent |medication|", where |medication| would be slot-filled with medication names from the kb. the logical form helps identify appropriate tables, entities and values required from the kb. structured data typically factually records lab values, vitals, conditions on admission, and medications but rarely records relations between these en-tities. in case of emrkbqa, none of the questions that involve resolving relations to answer a question in emrqa are answerable from structured data in mimic. however, answering questions about schema entities and attributes requires querying and combining information from multiple related tables in mimic. while logical forms are an outcome of the process used to generate emrqa, they are not essential to answering questions over unstructured data like clinical notes. the more traditional use of logical forms is in answering natural language questions from a structured kb. it is easier to convert a question to logical form than to sql (which is longer and more complex for most questions, often including multiple nested queries and joins). they provide a query-language agnostic intermediate representation that captures information need expressed in the question using a representation that is perhaps more annotator friendly. moreover, since logical forms are defined over a schema that captures domain-specific entities and relations, they are independent of the underlying database type or query language. question paraphrase groups. question paraphrases are different ways of asking the same thing. the emrkbqa dataset is paraphrase rich with an average of 7.5 paraphrases per question. in emrk-bqa, questions that map to the same logical form and share the same question type are considered paraphrases. the dataset has 52 question template groups where each group maps to the same logical form. this is because the answer to a question may vary based on question type even if they map to the same logical form. e.g., consider the questions in ## task definition and models each instance in emrkbqa consists of the follwing elements -question, question paraphrase group, question type, logical form, answer -defined in section 3. our goal is to build a model that when presented with a test question on the kb, provides an answer. we achieve this by first modeling the question to logical form learning problem as a semantic parsing task. here, given an input natural language question, we predict its logical form. next, we map the predicted logical form to a sql query in a deterministic manner to retrieve the answer from the kb. the answer is the set of cell values from the underlying kb that answer the question. we detail these two steps in the following sections. ## discussion advantages of emrkbqa. emrkbqa is the first large-scale community shared patient-specific qa dataset for answering physician questions from structured patient records. it follows a semiautomated process similar to emrqa (which releases qa pairs on clinical notes), where logical forms are the only expert-provided input. these logical forms lend credibility to the dataset as they capture entities, attributes, and relations required to answer a question and enable slot filling and answer generation. some highlights of emrkbqa are (1) question quality. unlike emrqa, emrk-bqa slot-fills entities with discretion by filtering out certain entities based on their attributes (like certain diagnoses based on icd codes, medications based on drug type). this results in more realistic realization of question instances. (2) question diversity. the dataset is rich in paraphrases (paraphrase groups have been updated from emrqa) (3) dataset difficulty. we provide paraphrase-level splits that helps train models that can generalize to unseen paraphrases of the train questions at test time. this is useful in practical settings. as described in the error analysis, in learning to map questions to logical forms, the challenges include recognizing the correct entity spans and types from the question, learning to predict long logical forms, and generating multiple attributes and constraints in the logical form. (4) logical forms generated from the same schema as emrqa, allowing the schema to be a unifying factor across structured and unstructured qa. this allows for future updates in a uniform manner. ## conclusion we create a new large-scale dataset, emrkbqa, for answering patient-specific physician questions from structured patient records. this communityshared release is created in a semi-automated manner and consists of over 900k question-logical formanswer triples, 389 question types (templates), with ≈7.5 paraphrases per question type. we benchmark the dataset and quantify its usefulness in answering questions by training models for semantic parsing of questions to logical forms.