P anini QA : Enhancing Patient Education Through Interactive Question Answering (2023)

Abstract: A patient portal allows discharged patients to access their personalized discharge instructions in electronic health records (EHRs). However, many patients have difficulty understanding or memorizing their discharge instructions (Zhao et al., 2017). In this paper, we present PaniniQA, a patient-centric interactive question answering system designed to help patients understand their discharge instructions. PaniniQA first identifies important clinical content from patients’ discharge instructions and then formulates patient-specific educational questions. In addition, PaniniQA is also equipped with answer verification functionality to provide timely feedback to correct patients’ misunderstandings. Our comprehensive automatic & human evaluation results demonstrate our PaniniQA is capable of improving patients’ mastery of their medical instructions through effective interactions.1

## introduction limited patient understanding of their medical conditions can lead to poor self-care at home. upon hospital discharge, physicians often provide discharge instructions to aid in patients' recovery and disease self-management @xcite . however, some patients may have difficulty understanding and memorizing instructions due to low health literacy, limited memory, or an absence of supervision. for example, research shows that patients only retain a minimal amount of information from discharge instructions, with an immediate forgetting rate of up to 80% @xcite . further, when instructions are misinterpreted by patients, there is often a lack of corrective intervention. limitations in a patient's understanding of their medical conditions hinder their prospects of recovery. it is imperative to investigate new methods of patient education to enhance health outcomes. in this study, we explore a novel method inspired by dialogic reading @xcite to educate patients through interactive questionanswering. dialogic reading actively involves patients in the learning process by following the p.e.e.r. sequence: prompt, evaluate, expand, and repeat, which enables patients to engage in a meaningful dialogue, further strengthening their understanding and retention of the material. as illustrated in figure 1 , our dialog agent asks questions about key aspects of discharge instructions and encourages patients to read and understand the instructions to provide accurate answers thoroughly. crafting questions that effectively meet educational objectives is challenging @xcite . a suitable question should be based on the patient's discharge instruction and aim to improve their understanding of health conditions, such as ''what was the probable cause of your chest pain?''. conversely, the question ''how does cardiac catheterization help treat a heart attack?'' illustrated in figure 1 , may exceed the education scope, as it is unanswerable or requires knowledge beyond the provided discharge instruction. such questions are considered unsuitable for patient education. we introduce new question-generation methods that draw on the advancements of large language models (llms) @xcite @xcite . utilizing openai's gpt-3.5 model, we generate informative questions from discharge instructions. further, we combine llms with medical event and relation extraction to constrain the model, producing questions that target salient medical events identified in the discharge instructions. we create a new dataset with expert-annotated medical events and relations for discharge instructions from the mimic-iii @xcite database. while earlier efforts have annotated events that physicians would discuss during patient handoff @xcite , our focus is on identifying pairs of medical events with correlational or causal relationships. by pos-ing questions about one event, we guide patients toward the other as potential answers. our system further incorporates an answer verification module to provide instant patient feedback. when patients give correct answers, the bot confirms them, reinforcing their understanding. if answers are incorrect or partially correct, the bot clarifies misunderstandings and provides additional information. extensive automatic and human evaluations demonstrate the efficacy of our question-generation methods and show that paniniqa holds great promise for promoting patient education. to summarize, our research contributions are as follows. we explore a new way of educating patients regarding their health conditions through interactive question-answering. our approach aligns with the p.e.e.r. dialogic reading theory that promotes patients' active participation in comprehending medical events. we compare questions generated using openai's gpt-3.5 model, our enhanced method with medical event extraction, and human-written questions tailored for patient education. we meticulously evaluated all questions, answers, and patients' educational outcomes. through comprehensive human evaluations, we demonstrate that paniniqa holds promise for patient education. future work includes controlling the difficulty of questions, prioritizing questions given patients' health literacy, and enabling interactive learning of medical concepts. ## related work there is a growing need to improve patients' understanding regarding their hospital experiences @xcite @xcite . lack of understanding can result in non-adherence to discharge instructions and readmission to the hospital due to poor selfcare at home. previous research has attempted to generate hospital course summaries for patients using lay language @xcite @xcite @xcite . this paper goes a step further by utilizing interactive question answering to communicate essential medical events from discharge instructions to patients, thus enhancing their understanding and retention of the material. our proposed method differs from existing clinical question-answering studies in several aspects. most clinical qas are designed to satisfy individuals' information needs, with questions modeled after those that can be asked by physicians @xcite @xcite . these systems focus on improving the accuracy of their answers @xcite @xcite . in contrast, our goal is to educate patients and prompt them with questions that will enhance patients' understanding of their doctors' visits. a successful qa system should be comprehensive and exhaustive, asking all relevant questions and prioritizing them based on the patient's medical history and health literacy. successful patient education requires effective questioning @xcite . particularly, question generation has been studied using template-based @xcite @xcite and neural seq2seq models @xcite @xcite @xcite . instruction-tuned llms have demonstrated exceptional abilities in conversing with humans @xcite @xcite @xcite . however, most research has been conducted using commoncrawl, wikipedia, and other generic texts. considering the factuality issues of neural language models @xcite , question generation in the medical domain remains challenging. learning through conversation can improve education outcomes @xcite @xcite @xcite . dialogic reading @xcite @xcite has demonstrated that engaging children in a guided conversation with parents while reading storybooks can significantly enhance their learning outcomes. while engaging physicians in highquality conversations may not always be feasible, the use of question answering facilitated by a chatbot could be a valuable means of helping patients acquire a deeper understanding of their health conditions. ## question answering in the gpt era llms such as chatgpt have led to significant advancements in generative ai @xcite @xcite @xcite . fine-tuning neural models on specific tasks often yields superior results. furthermore, llms acquire emergent abilities through instruction tuning and reinforcement learning using human feedback @xcite . this allows them to generalize to new tasks effectively. common human-llm interactions include (a) zeroshot prompting, where users provide a prompt for the llm to complete, and (b) in-context learning, where users give task examples and ask the llm to solve a new case, potentially involving a multi-step reasoning process @xcite . in this study, we focus on zero-shot prompting to assess the llm's ability to comprehend discharge instructions. llms possess vast world knowledge, and their performance on knowledge-intensive tasks correlates with training data and model size @xcite . however, it remains unclear whether llms have enough domain knowledge to facilitate patient education. for example, gpt-3, with its 175 billion parameters, is trained on general data sources such as common crawl, webtext2, books, and wikipedia @xcite . yet, the model still generates factually inconsistent errors within their output. our study presents an initial evaluation of gpt models' potential in interactive patient education. following the p.e.e.r. framework of dialogic reading, we employ gpt models to perform the following tasks: question generation. we use openai's gpt-3.5 model (text-davinci-003) to generate informative questions from a discharge instruction. the questions aim at helping patients understand crucial medical events. our prompt is ''generate n questions to help the patient understand crucial medical events in the above discharge instruction.'' similar to a teacher designing exam questions, we anticipate the gpt model to produce a set of questions all at once rather than incrementally. the questions must collectively cover the salient events identified in the discharge instruction while minimizing redundancy. answer verification. useful feedback is essential for improving patient comprehension of the material. to perform this task, we prompt the gpt model with ''as a physician, your goal in the conversation is to help your patient better understand the discharge instructions before they leave the hospital.'' utilizing openai's api, we also provide the original discharge instruction, interaction history, and current question-answer pair as key-value pairs for the model. we then instruct the model to ''verify if the patient's answer is correct, incorrect, or partially correct, and generate a suitable response to improve the patient's comprehension of this question.'' we empirically compared two gpt models, text-davinci-003 and gpt-3.5-turbo (chatgpt), and selected chatgpt for answer verification as it is optimized for chat and generally produces higher quality responses. ## extracting salient medical events in this section, we present our question-answering system that emphasizes identifying salient medical events and their relations. we generate targeted questions using them and apply the same answer verification module described previously. a typical discharge instruction includes visit recap, which recaps a patient's clinical visit, including symptoms, diagnoses, treatments, and test results. patients are expected to understand the relationships among these medical events, such as how the treatment ercp relates to cholangitis as illustrated in ## data annotation we seek to annotate discharge instructions from the mimic-iii database (v1.4) @xcite with key medical events that are important for patients to understand. mimic-iii is a publicly available repository of de-identified health records of over 40,000 patients collected from the beth israel deaconess medical center in massachusetts. our aim is to identify text snippets in discharge instructions that correspond to significant medical events, including symptoms, diseases, test results, and treatments. we annotate not only individual events but also their relationships. they are organized into a hierarchy as outlined in the schema shown in we recruited five medical experts to create a sizable dataset. they are md students at umass chan medical school and have a high level of language proficiency. each expert is given 150 discharge notes to annotate. it is possible to skip some notes due to low text quality. annotators were also given detailed instructions and examples. we developed a web-based interface to facilitate the annotation process, which has been iteratively improved to meet the needs of this study. due to budget constraints, we assign one annotator to each discharge note. in total, we completed 458 discharge notes with medical event annotations. our annotation consists of two phases. in the first phase, an expert selects text snippets from the discharge instruction corresponding to medical events that the patient needs to understand. each a key distinction between our work and earlier dataset curation efforts @xcite @xcite is that the ear-lier efforts aim to annotate questions that physicians would ask during patient hand-off, which may be informal and unanswerable based on the discharge instruction. in contrast, our focus is on annotating salient medical events that are essential to patient's understanding of their medical conditions. we split our annotated data into train / validation / test splits, which contain 338 / 60 / 60 discharge instructions, respectively. for relation identification, we use the event pairs from the human-annotated relations as positive relations and all other medical event pairs of compliant types (e.g., the event pair types in ## evaluating information extraction to improve llms' ability to generate educationally effective questions for patient education, we designed an information extraction (ie) module (medical event/relation identification) to guide question generation. we report automatic evaluation results for different ie methods in this section. ## evaluating patient education in order to comprehensively evaluate patient education outcome, we conducted human evaluations from both the patient's and the physician's perspectives, as well as a gpt-4 powered automatic evaluation. these evaluations focus on two main aspects: (1) the generated question's quality of different models (gpt, gpt+ie, and human ground-truth); (2) the preference of different designs of the interaction experience (none of support, raising questions only, and raising questions and verifying answers). ## limitations and ethical considerations this study offers valuable insights, but with a few limitations we would like to note. biases. large language models trained on vast amounts of text data can pick up biases present in data. for example, they may prefer certain questions related to aspirin or even associate certain health conditions with specific groups of people. they may also perpetuate misinformation and provide incorrect information. in addition, people who participated in our evaluation have different levels of language proficiency and medical background. these biases may be mitigated by enhancing model alignment with each individual's background and health literacy level. ## conclusion in this study, we present paniniqa, a patientcentric interactive question answering system designed to help patients understand and memorize their discharge instructions. paniniqa generates educational questions from discharge instructions after identifying salient medical events and event relations. llms with prompting is promising for question-answer generation, but sometimes hallucinating. extensive evaluations highlight the importance of providing answer feedback.