Challenges of GPT -3-Based Conversational Agents for Healthcare (2023)

Abstract: The potential of medical domain dialogue agents lies in their ability to provide patients with faster information access while enabling medical specialists to concentrate on critical tasks. However, the integration of large-language models (LLMs) into these agents presents certain limitations that may result in serious consequences. This paper investigates the challenges and risks of using GPT-3-based models for medical question-answering (MedQA). We perform several evaluations contextualized in terms of standard medical principles. We provide a procedure for manually designing patient queries to stress-test high-risk limitations of LLMs in MedQA systems. Our analysis reveals that LLMs fail to respond adequately to these queries, generating erroneous medical information, unsafe recommendations, and content that may be considered offensive.

## introduction there is growing interest in medical dialogue systems that can support patients with health goals, extend access to health services such as information seeking, and improve the quality of patient care @xcite . however, there are many risks of patient-facing medical dialogue systems that could impose harms, such as the production of false or misleading information @xcite . in one study, for instance, @xcite found that the google, alexa, and siri digital assistants answered 29% of medical questions in ways that could cause harm and 16% that could result in death. further risks come with the use of large pre-trained language models (llms) in these systems beyond inaccurate information that can yield negative conduct with the patients. many works have highlighted ethical issues of llms, such as learned implicit social biases and generating offensive content, which are particularly concerning for medical contexts. @xcite find llms memorize abundant inaccurate medical information and popular misconceptions. research and development of medical dialogue systems that employ llms typically evaluate the accuracy of the medical information and capabilities on medical tests. medical soundness is only part of a comprehensive ethical evaluation of medical dialogue systems. it is imperative to further consider medical ethical principles and responsibilities that underlie the interpersonal nature of patient care and communication, which contribute to patient well-being @xcite . in this study, we draw on standard ethical medical conduct guidelines stated in the medical declaration of geneva and principles of patient-centered therapy to develop an approach to target ethical risks in the evaluation of llms in medical applications. our approach examines not only the risk of factual hallucinations but also interpersonal, stylistic aspects, indicating compassionate care. we argue that evaluations of medical information systems shall be constructed in line with established medical ethics principles ( §2), and demonstrate our method by assessing three gpt-3-based models for medical question-answering ( §3). we evaluate generated responses for (1) attributes of patient-centered communication strategies, and (2) their handling of patient queries we manually designed to stress-test medical ethical limitations ( §4). we find that the models generate invalid medical information, dangerous recommendations, and offensive content, rendering them unsuitable for standalone use in the medical domain ( §5). ## safety-critical evaluation standards the medical declaration of geneva, a standard guideline for ethical medical conduct, encapsulates fundamental principles of medical practice and the professional responsibilities and obligations of practitioners to their patients, colleagues, and society. it states the ethical principle of nonmaleficence, i.e. that doctors should do no harm, and act only in the interest of promoting the physical, mental, social and spiritual well-being of the patient, under contextual consideration of the patient's values. to uphold the principle of nonmaleficence, a patient-facing medical information system must present accurate information in an accessible manner fail-safe to misinterpretation. another well-established guideline for communication in healthcare is carl roger's patientcentered therapy approach @xcite , which is widely taught in medical and therapuetic curricula. it emphasizes the following three fundamental communication strategies for facilitating patient interactions: 1) empathy is the ability of the therapist to understand and share in a patient's feelings by seeing the world from their perspective and communicating that understanding. it involves creating a supportive environment for the patient to understand their own emotions. 2) unconditional positive regard is a therapeutic approach where the therapist accepts and values the patient for who they are without judgement, and actively listens without interruption or giving advice. it is a form of non-judgmental and accepting attitude toward the patient. the following is a conversation with an ai doctor. the doctor is friendly, intelligent, respectful, and professional. human: hello, who are you? doctor: i am a virtual doctor with a lot of knowledge in the medical domain. i am able to help you regarding medical questions. how can i help you today? ## experimental setup we investigate gpt-3-based models for the medqa task: given a medical information-seeking patient query, generate a coherent, medically informed response that satisfies the query. we use patient queries from the english meddialog dataset @xcite . it contains two-turn qa pairs collected from two online platforms, iclinic.com and healthcaremagic.com, which offer symptom self-checking services and video and chat consultations with doctors. the dialogues include 51 topic categories and 96 specialties. there are 515k english utterances, comprising 44m tokens. we investigate three models based on gpt-3 curie, a down-scaled variant of gpt-3 of approximately 13 billion parameters: baseline. our baseline model is curie. we provide a prompt, shown in ft-meddialog. using the openai api, we finetune curie on a sample 5,000 qa pairs from the meddialog dataset. question-answer pairs are formulated as prompt-completion pairs for the ope-nai api. we do not utilize the entire meddialog dataset due to financial constraints of remote finetuning via the openai api. openai recommends a few hundred examples minimum therefore it can be deduced that 5,000 examples are sufficient enough to observe the effect on the model's natural language understanding capabilities and the medical accuracy of the generated responses. ## medical qa evaluations in this section, we detail our evaluation methods and present the results. we base our assessments on standard medical ethical principles for patient interactions articulated in §2. ## discussion based on our evaluations, the gpt-3-based models are unsuitable for patient-facing medical systems. they produce incorrect and misleading medical advice, failing to adhere to the medical declaration of geneva's principle of non-maleficence. the gpt-3-based models cannot address sensitive topics, including questions about race, emergencies, abortion, and medicine dosages, safely. for example, the response in physicians we interviewed expressed significant concern over how automata would handle this exact issue that physicians can and do handle. as for race, it is well-established knowledge that gpt-3 encodes large amounts of training data containing racism @xcite . the alarming but unsurprising results in queries involving emergencies and dosages demonstrate the severe danger of using gpt-3-based models in patient-facing medical qa systems. while we cannot say whether such models will ever be safe for patient-facing systems, significant engineering efforts and continuous professional medical oversight is needed to mitigate such risks. ## related work there is a significant body of literature on dialogue system evaluation approaches. evaluation paradigms typically represent desired characteristics of a particular dialogue system as response quality and appropriateness often depend on the application @xcite . however, especially with the increasing use of llms in dialogue systems, the need for evaluation paradigms to account for ethical issues, such as learned implicit biases, privacy violations, user safety, and risks of generating toxic and offensive content @xcite . @xcite presented additional risk areas associated with language models, including fairness and discrimination, private data leaking, information hazards (e.g., false or misleading content), and environmental harms. our work concerns the need for evaluations tailored to medical applications that uphold established ethical standards and responsibilities in medicine. weidinger et al. ( 2021 )'s human-computer interaction harms are of particular relevance and include overreliance or unsafe use, the creation of avenues for exploitation and manipulation, and the promotion of harmful stereotypes. @xcite discuss three major safety issues, including the generation of harmful content, the response to harmful content, and the imposter effect, referred to as "unsafe counsel in safety-critical situations". recent studies evaluating gpt-4 and gpt-3.5 turbo in medical applications have emerged, the majority of which focus on the medical knowledge capacity of llms. llm evaluation is commonly conducted through medically standardized tests such as the usmle and medmcqa, among oth-ers @xcite @xcite . these studies often primarily address the domain of medical knowledge while frequently leaving out the interpersonal aspects of medical communication. in this study, our evaluation is confined to simple and practical medical conversational guidelines @xcite , holding medical computation systems to the same standard principles as medical professionals in order to provide a deeper understanding of the challenges faced. ## conclusion we argued that patient-facing medical information systems should be evaluated in the context of standard medical ethical principles ( §2), similarly to medical professionals. we evaluated gpt-3-based models in a medqa system to scrutinize the limitations of llms in the medical domain ( §4). we find that the models are unable to be consistent with patient-centered therapy communication strategies ( §4.1) and fail to respond ethically to our manually crafted safety-critical stress-test queries ( §4.2). we contribute procedural guidelines for developing stress-test queries that future researchers can use for testing medqa systems. in particular, they generate highly problematic responses to safetycritical questions, including the inclination to provide a diagnosis with no information. we observed especially low rates of safe responses to queries testing for racism and emergency responses to lifethreatening situations. we, therefore, conclude that gpt-3 is unsuitable for patient-facing medical information systems. empathy artifacts: the scope of our study was limited by the small number of datasets that were compared with one another. specifically, we only fine-tuned on one medical dataset and one empathy dataset. as argued by @xcite , there are limitations in the way that empathy datasets are crafted, particularly concerning applications such as ours that aim for assessing cognitive empathic skills rather than surface-level emotional response.