Learning to Ask Like a Physician (2022)

Abstract: Existing question answering (QA) datasets derived from electronic health records (EHR) are artificially generated and consequently fail to capture realistic physician information needs. We present Discharge Summary Clinical Questions (DiSCQ), a newly curated question dataset composed of 2,000+ questions paired with the snippets of text (triggers) that prompted each question. The questions are generated by medical experts from 100+ MIMIC-III discharge summaries. We analyze this dataset to characterize the types of information sought by medical experts. We also train baseline models for trigger detection and question generation (QG), paired with unsupervised answer retrieval over EHRs. Our baseline model is able to generate high quality questions in over 62% of cases when prompted with human selected triggers. We release this dataset (and all code to reproduce baseline model results) to facilitate further research into realistic clinical QA and QG: https://github.com/elehman16/discq .

## introduction physicians often query electronic health records (ehr) to make fully informed decisions about patient care @xcite . @xcite found that it takes an average of 8.3 minutes to answer a single question, even when physicians are trained to retrieve information from an ehr platform. natural language technologies such as automatic question answering (qa) may partially address this problem. there have been several dataset collection efforts that aim to facilitate the training and evaluation of clinical qa models @xcite @xcite . however, template-based @xcite and other kinds of automated generation @xcite ## discq dataset we work with 10 medical experts of varying skill levels, ranging from senior medical students to practicing mds, to construct a dataset of 2029 questions over 100+ discharge summaries from mimic-iii @xcite . ## task setup we consider the task of generating questions that are relevant to a patient's care, given a discharge summary and a trigger. afterwards, we attempt to find answers to these generated questions (figure 2 ). we also examine model performance for when the trigger is not provided and must instead be predicted. the task of generating questions without triggers can be viewed similarly to answer-agnostic question generation. we take a similar approach to @xcite , in which we implement a pipeline system that first selects key phrases from the passage and then generates questions about the selected key phrases. while a majority of past works attempt to ensure that the generated question is answerable @xcite @xcite , we do not impose this constraint. in fact, we argue that the ability to generate unanswerable questions is necessary for realworld applications, as a question answering system should be able to identify such questions. these questions can be used as hard-negatives to train and calibrate qa systems. ## models pre-trained transformers have become ubiquitous in many natural language processing tasks @xcite @xcite , including natural language generation @xcite . additionally, large-scale transformers have demonstrated the importance of parameter count for both upstream @xcite and downstream tasks, especially in lowresource settings @xcite . as these results were mainly shown in nonclinical general domains, we find it important to evaluate both medium-sized and large models. we formulate trigger detection as a tagging problem, for which we fine-tune clinicalbert @xcite . for question generation, we experiment with both bart (406m parameters) @xcite and t0 (11b parameters) @xcite . question generation is formulated as a conditional generation problem and modelled via a sequence-to-sequence approach. during evaluation, we use greedy sampling to produce generated text. reducing context size due to memory constraints and the limited sequence length of pretrained models, we only select the part of the discharge summary containing the trigger. this is done in two possible ways: (1) extracting the sentence 8 with the trigger or multiple sentences if a trigger spans across sentence boundaries or (2) extracting a chunk of size 512 containing the trigger in it. to check if this context is actually used by the models we also fine-tune bart without extra discharge summary context (trigger text only). handling multiple questions 41% of the discq examples have multiple questions per trigger. sometimes the questions depend on each other: for this reason, we train and evaluate models in two different setups: split questions (by the ?-symbol) and combined questions. while the split-questions format might be more comparable to pre-existing work, the combined-questions setting likely models more realistic behavior of medical professionals. prompting schick and schütze (2021) demonstrate that adding natural language instructions to the model input can significantly improve model quality. the area of prompting has recently gained widespread popularity @xcite and has had particular success in low-supervision scenarios @xcite . t0 @xcite is a fine-tuned t5 @xcite ) model trained on 64 datasets and prompts from the public pool of prompts @xcite . given a trigger and some context from the discharge summary, we finetune t0++ and bart with the following prompt: "{context}after reading the above emr, what question do you have about "{trigger}"? question:". ## results we split 2029 questions into train (70%), validation (10%) and test (20%) sets 9 and fine-tune the models as described in section 5. to evaluate trigger detection, we use token-level precision, recall and f1 score. for automated evaluation of question generation we use rouge-l @xcite , meteor @xcite and bertscore @xcite metrics. to monitor the diversity of generated questions, we measure the fraction of unique questions on the evaluation set. as the question generation task has high variability of plausible generations, the utility of automatic metrics is debatable due to poor correlation with human evaluation @xcite @xcite @xcite . for this reason, we additionally perform human evaluation (section 7). 9 we use a document level split. ## human evaluation human evaluation is still the most reliable way to compare generative models for diverse tasks like question generation. common categories for question generation to consider are grammar, difficulty, answerability and fluency @xcite @xcite . however, not all of these categories are relevant to clinical question generation. we evaluate questions generated using our pipeline, as well as gold standard questions on the following four categories (binary scale): understandability can an individual familiar with medical/clinical language understand the information needs expressed, even if the question is not a complete sentence or contains grammar/spelling errors? nontriviality is the question unanswerable with respect to the sentence it was triggered/generated from? a question that would be considered trivial would be "did the patient have a fever?" if the context presented was "the patient had a fever". relevancy to trigger is the trigger or the sentence containing the trigger related to the question? clinical meaningfulness will the answer to this question be helpful for further treatment of this patient or understanding the patient's current condition? or alternatively, is it reasonable that a medical professional would ask this question given the provided context? annotations were divided evenly between medical experts. each question is scored independently by two different annotators. however, due to time constraints, there are no discussions between annotators about their decisions. we also ensure that annotators did not receive discharge summaries that they had seen previously. lastly, it is important to note that annotations were assigned blindly. annotators were informed that they would be scoring both human and machine generated questions, but were not informed about (1) where the question was generated from (i.e., human or machine) and ( 2 ) the proportion of human:machine generated questions. we score questions using the tree presented in figure 5 . in cases in which the question is both understandable and nontrivial, we additionally ask medical experts to determine whether or not the proposed answer fully answers, partially answers or is irrelevant to the question. results can be seen in ## discussion we evaluate performance of both the best bart and t0 model with respect to rouge-l score. we select 400 questions generated from each model, half of which are generated with gold triggers and the other half with predicted triggers, as described in section 6.1. two medical experts score each question. due to the subjective nature of the task, we find moderate agreement between annotators with respect to scoring questions (κ = 0.46) and scoring answer sufficiency (κ = 0.47). we use the "satisfies all" column (i.e., satisfies all four human evaluation categories) to calculate agreement between questions. results show that the t0 model prompted with gold triggers successfully generates a high-quality question 62.5% of the time ( context generated question trigger type question type pt reports that he noticed a right neck mass last october size, outline (asymmetry), color, elevation, evolving? sign/symptom sign/symptom she was also significantly tachypneic were there interventions done to address this? sign/symptom treatment according to dr. <name>, she has had stable deficits for many years without any flare-like episodes. how is her vision now? assessment sign/symptom her bicarb began to drop and she developed an anion gap acidosis confusion? confusion? agitation? hand tremors? bounding pulses? problem sign/symptom as human evaluation demonstrates, despite low automatic metric scores, both bart and t0 achieve reasonable success in generating coherent, relevant and clinically interesting questions. to evaluate if the automated metrics can capture the quality of generated questions, we calculate the spearman's rank correlation coefficient between human evaluation and automatic metrics. we find extremely low and statistically insignificant correlation for rouge-l (-0.09), meteor (-0.04) and bertscore (-0.04). this is unsurprising, as these automatic metrics are not designed to capture the categories we examine during human evaluation. we also score the answers selected by our clin-icalbert model trained on emrqa (section 6.3). interestingly, we find that of the answers the model successfully recovers, 44% are extracted from the remainder of the discharge summary used to gen-erate the question. the remaining 56% come from nursing notes, radiology/ecg reports and previous discharge summaries. however, for a majority of the questions, we are unable to recover a sufficient answer ( ## conclusion we present discharge summary clinical questions (discq), a new human-generated clinical question dataset composed of 2000+ questions paired with the snippets of text that prompted each question. we train baseline models for trigger detection and question generation. we find that despite poor performance on automatic metrics, we are