EHDC hat: A Knowledge-Grounded, Empathy-Enhanced Language Model for Healthcare Interactions (2024)

Abstract: Large Language Models (LLMs) excel at a range of tasks but often struggle with issues like hallucination and inadequate empathy support. To address hallucinations, we ground our dialogues in medical knowledge sourced from external repositories such as Disease Ontology and DrugBank. To improve empathy support, we develop the Empathetic Healthcare Dialogues dataset, which utilizes multiple dialogue strategies in each response. This dataset is then used to fine-tune an LLM, and we introduce a lightweight, adaptable method called Strategy Combination Guidance to enhance the emotional support capabilities of the fine-tuned model, named EHDChat. Our evaluations show that EHDChat significantly outperforms existing models in providing emotional support and medical accuracy, demonstrating the effectiveness of our approach in enhancing empathetic and informed AI interactions in healthcare.

## introduction the integration of technology into healthcare continues to foster innovations aimed at enhancing patient outcomes. the emergence of large language models (llms) has significantly advanced their ability to provide responses to user queries concerning medications, dispense general health and wellness advice, and offer support for mental health issues @xcite @xcite . however, the effectiveness of these llms is contingent upon the quality of the training data they receive. one limitation is the tendency of llm to hallucinate and generate erroneous or fabricated medical knowledge which may have serious consequences. another limitation is their frequent disregard for emotional cues, which can result in interactions that feel insincere and disconnected from the user's emotional state. this issue highlights a gap in the ability of llms to engage in genuinely empathetic doctor, i've been feeling really lightheaded and i've noticed these frequent headaches lately. i'm worried it could be something serious. it's been going on for about a week now, and i've also been feeling more tired than usual. could it be related to my hypertension? lightheadedness and fatigue are sometimes associated with high blood pressure. have you been monitoring your blood pressure at home? it's essential for us to check your current blood pressure levels to assess the situation. i should first validate his/her emotions and then ask details about his/her symptoms. i should answer his/her question and and then gather more information about the situation. communication, which is essential for addressing the nuanced needs of patients @xcite . to address this gap, we need high-quality training data that can provide accurate responses and reflects human emotions. existing emotional support datasets such as esconv @xcite , extes @xcite , and reddit mental health forums @xcite offer valuable insights into empathetic interactions. however, these resources often lack comprehensive medical knowledge, limiting their applicability in healthcare contexts. conversely, medical datasets like meddialog @xcite , coviddialog @xcite and cmtmedqa @xcite provide extensive healthcare information but frequently lack the depth of genuine empathetic interactions. we develop an empathetic healthcare dialogue (ehd) dataset foot_0 to train smaller models in generating human-like empathetic responses within the healthcare domain. to construct this dataset, we leverage llms, known to have gathered a wide array of information from vast human corpora and specialized databases. however, recognizing the potential for llm hallucination, we ground our dialogues in external verified medical knowledge repositories, thereby enhancing the reliability and the depth of conversations across a wide range of healthcare topics and concerns. we also devise a method to incorporate conversational strategies such that each response can utilize multiple strategies. we fine-tune an llm using the ehd dataset and introduce a lightweight, plug-andplay method, called strategy combination guidance (scg) to guide the model in selecting a good combination of strategies to improve response quality. we further integrate retrieval-augmented generation (rag) technology @xcite , which provides relevant context for more accurate and contextually appropriate responses, resulting in our ehdchat. quantitative experiment results show that ehd-chat is far superior in providing emotional support and giving accurate medical information. human evaluation reveals a preference for the responses generated by ehdchat. case studies also illustrate that employing a variety of conversational strategies leads to more natural dialog. ## related work research on medical dialogue systems has primarily focused on developing question-answering systems @xcite @xcite . chatdoctor @xcite utilizes chat-gpt to generate synthetic medical conversation data, thereby allowing the fine-tuning of a llama model that is able to simulate dialogues between doctors and patients. doctorglm @xcite adapted this approach to develop a system that simulates doctor-patient conversations in chinese. these dialogue systems generally do not provide adequate emotional support due to the scarcity of datasets that include effective emotional support conversations @xcite . early efforts to curate emotional support datasets primarily focused on transcribing therapist sessions @xcite or compiling emotional question-answer pairs from online platforms @xcite @xcite . however, these datasets often lack proper design and consist mainly of single-turn responses. esconv @xcite incorporates conversation strategies and utilize crowdsourcing to create high-quality and multi-turn emotional support datasets. building upon esconv, extes @xcite employs llms to synthesize additional scenarios and dialogues that incorporate more emotional support strategies, further expanding the available data for training these specialized dialogue systems. current medical dialogue datasets lack emotional content, while empathetic datasets lack medical knowledge. our work aims to create a new dataset that combines empathetic dialogues with accurate medical information. this fills a gap in existing resources for doctor-patient conversations, providing a more complete dataset for training ai in healthcare communication. ## ehd dataset our goal is to generate a dataset comprising of a wide range of synthetic, multi-turn dialogues between doctors and patients that are not only emotionally supportive, but also clinically informative. we produce dialogues via the self-chat technique @xcite and reference-based dialogue generation methods @xcite @xcite . previous research @xcite has shown that emotional support strategies can boost empathy levels @xcite . to enhance the empathetic nature of the dialogues, we incorporate a mix of known emotional support strategies, drawing on research that demonstrates the increased effectiveness of combining two strategies within a single response @xcite . figure 2 shows the prompt template used for dialogue generation which includes basic instructions, a medical knowledge reference, dialogue strategies and an example dialogue. for medical knowledge on diseases, we utilize the disease ontology (schriml, 2018) as our reference source. as the information available in the disease ontology is limited to basic descriptions of diseases, we leverage the external links stored in disease ontology to directly extract data from the corresponding web pages. for knowledge on your task is to create a medical empathetic dialogue between doctor (ai) and patient (user) based on the provided medical reference. the generated dialogue should also draw from the dialogue example and incorporate multiple dialogue strategies in each doctor's response. here is a comprehensive list of all possible strategies for responding in conversations for emotional support: 1. reflective statements: repeat or rephrase what the person has expressed to show that you're actively listening. 2. clarification: seek clarification to ensure a clear understanding of the person's emotions and experiences. (all other strategies)... refer to the provided example to see how multiple strategies are used in each doctor's response and to understand the required return format: {example} the generated dialogue should focus on the provided medical knowledge. the patient should inquire about this specific medical knowledge, possibly starting with symptoms or expressing concerns about the disease or medicine. the doctor's response can refer the given medical content. this medical knowledge is as follows: {medical reference} limit patient responses to 40 words or less and doctor responses to 100 words or less to maintain clarity in communication. ensure a profound exploration by structuring the dialogue with a minimum of 10 rounds, guaranteeing an indepth and thorough conversation (all other settings)... drugs, we utilize the public dataset called drug-bank @xcite to extract the description, effect and dosage of drugs. in total, we compile 4,223 entries related to disease knowledge and 3,229 entries on drug knowledge. we randomly select 100 dialogues from extes @xcite , and retain the first sentence of each dialogue as the initial prompt. then we instruct gpt-4 using the prompt in figure 2 to generate responses that incorporate two emotional support strategies per response. the medical reference mentioned in the prompt is selected from a predetermined set of commonly encountered diseases or drugs information. figure 3 shows an example of the dialogue generated. the conversation is focused on a specific disease scabies and each response employs at least 2 strategy combinations, highlighted in red. after generation, we evaluate the accuracy of the medical content and the correct integration of multiple emotional support strategies in each response. dialogues identified as medically inaccurate or lacking in emotional support strategies are removed. the remaining generated dialogues form the example dialogues in our prompt to generate additional dialogues, with each dialogue centered around a single medical concept. ## ehdchat building on the ehd dataset, we develop an empathetic healthcare dialogue system called ehd-chat that is able to provide accurate responses to medical queries while demonstrating empathy and understanding of the user's emotional states. this system is specifically designed to handle the dual challenges of medical accuracy and emotional empathy, essential in healthcare settings. we randomly partition the ehd dataset into a training set comprising of 7,175 medical terms across 31,918 dialogues, and a test set containing the remaining 300 medical terms in 1,385 dialogues. for each dialogue in the training set, we generate training instances where the target output is the next response, and the preceding conversation provides the context. these instances are used to fine-tune the llama-1 (7b) model @xcite , leveraging its capabilities to handle complex dialogue scenarios. in order to generate more precise and effective responses, we introduce a strategy combination guidance (scg) module to optimize the selection of emotional support strategy combinations for each response. we achieve this by taking 5,000 dialogue rounds in the ehd dataset and employ llm to determine potential strategy combinations for each round. for each strategy combination, we prompt the llm to generate a corresponding response. the most effective strategy combination is then selected based on which generated response achieves the highest gptscore. figure 4 shows the strategy combination selection process. we use these instances to further refine the fine-tuned llama-1 model, culminating in the development of the proposed ehdchat @xcite . note that scg is a lightweight, plug-and-play module that can be applied to any existing emphathetic healthcare dialogue systems to guide their response generation process. ## experiments we carry out a series of experiments to evaluate the performance of our proposed ehdchat in terms of providing emotional support as well as the accuracy of the shared medical knowledge during the conversation. we implement the following baselines for comparison: we fine-tuned the models on two a100 80g gpus with a batch size of 16, using a 3% warmup phase and cosine learning scheduler. ## case studies finally, we present case studies to show the emotional support capabilities and medical knowledge correctness of ehdchat compared to exteschat, esconvchat, and chatdoctor. figure 5 shows the responses generated by the various models given the same dialogue input. we observe that exteschat focuses solely on emotional support (highlighted in yellow), failing to address patient query about his diagnosis of myocardial infarction. on the other hand, chatdoctor's response exclusively focuses on the impact of myocardial infarction on health (highlighted in green), neglecting the patient's emotional state. in contrast, ehdchat employ a combination of dialogue strategies (emotional validation+share information) to first acknowledge the patient's emotion before addressing his medical query, leading to a more balanced response. figure 6 demonstrates the strategy transitions used in ehdchat and how the transitions are able to provide better emotional support and answer medical queries raised by users. we see that ehd-chat is able to employ a variety of conversational strategy and the dialogues are more natural offering both emotional support and answering the medical queries raised by users. in contrast, escon-vchat repeatedly responses with "i understand..", while exteschat primarily offers emotional support with occasional superficial medical advice, and chatdoctor's responses only focus on addressing the medical queries. ## conclusion we have described a new approach for generating empathetic multi-turn dialogues for the healthcare domain. by allowing a single response to contain multiple conversation strategies, we are able to enhance the level of emotional support to the user. our approach incorporates verified medical references and employs a robust verification method to reduce errors and ensure the authenticity of the information in the generated dialogues. our analysis confirms that the ehd dataset is both empathetic and medically accurate. our ehdchat, trained on the ehd dataset, has shown improvement both in terms of emotional support and medical proficiency, surpassing state-of-the-art chatbots. despite its strengths, ehdchat may face challenges when dealing with complex medical issues, underscoring the urgent need for further research to reduce inaccuracies and enhance the reliability of chatbots in critical applications. ehdchat (emotional validation + shre information): i understand this must be overwhelming for you. a myocardial infarction occurs when a blockage in a coronary artery prevents blood flow to the heart muscle, leading to tissue damage. we'll need to assess the extent of the blockage and discuss treatment options to manage your condition effectively. exteschat: hi there, i'm here to listen and support you. it's completely normal to feel scared and overwhelmed after receiving such news. i'll do my best to provide you with the information you need. chatdoctor: 1. the blockage is due to cholesterol deposits in the walls of your coronary artery which are responsible for supplying blood to the heart muscle. 2. this causes . ... patient: doctor, i was just diagnosed with a myocardial infarction, and i'm really scared. they mentioned something about blockages in my arteries. can you explain what that means for my health and what i can expect next? esconvchat: i am sorry to hear that. i can understand why you are scared. figure 5: case study to show the responses generated by the various models given the same dialogue input. ehdchat (reflective statements + clarification): i hear you're experiencing chest pain, arm pain, sweating, and dizziness. these are classic symptoms of a heart attack. have you noticed any shortness of breath or nausea along with these symptoms? exteschat: hi there! i'm here to listen and help. it sounds like you're experiencing a lot of pain and discomfort. can you tell me more about what's been going on? chatdoctor: 1) yes, it is possible for the symptoms you are describing to be due to a heart attack. 2) however, there are other causes of similar symptoms such as panic attacks or anxiety. 3) the best ... esconvchat: i'm sorry to hear that. i'm sure you are feeling anxious about it. user: hey doctor, i've been having this really bad chest pain that's spreading to my left arm. i'm sweating buckets and feeling dizzy. is this a heart attack? ehdchat (emotional validation + collaborative planning): it's understandable to feel scared. we can confirm a heart attack by checking your ecg for specific changes. it's crucial to act promptly. let's schedule an ecg to assess your heart's electrical activity and determine the best course of action. exteschat: i understand your fear. it's completely normal to feel scared in such situations. have you tried any self-care practices or relaxation techniques to manage your anxiety? chatdoctor: 1) yes, there are ways to confirm whether you are having a heart attack. 2) you should go to an emergency room immediately if you have these symptoms. 3) if you cannot get to an er right away ... esconvchat: i understand that. i'm sure you are scared. user: i'm so scared. i've read about heart attacks and their severity. is there any way to confirm if this is really happening to me? ehdchat (share information + offer hope): risk factors for heart attacks include high cholesterol, high blood pressure, diabetes, smoking, and a family history of heart issues. understanding these factors can help us manage your risk effectively. with the right steps, we can work towards preventing future heart attacks. exteschat: it's important to remember that heart attacks can happen to anyone, regardless of their age or gender. however, certain factors like high cholesterol, smoking, and diabetes can increase your risk. it's possible that you may have one or more of these factors. chatdoctor: 1) yes, there are certain factors that increase your risk of having a heart attack. these include smoking, high blood pressure, diabetes, high cholesterol, family history of early heart disease, stress, obesity, physical inactivity, etc. ...