Don’t be my Doctor! Recognizing Healthcare Advice in Large Language Models (2024)

Abstract: Large language models (LLMs) have seen increasing popularity in daily use, with their widespread adoption by many corporations as virtual assistants, chatbots, predictors, and many more. Their growing influence raises the need for safeguards and guardrails to ensure that the outputs from LLMs do not mislead or harm users. This is especially true for highly regulated domains such as healthcare, where misleading advice may influence users to unknowingly commit malpractice. Despite this vulnerability, the majority of guardrail benchmarking datasets do not focus enough on medical advice specifically. In this paper, we present the HeAL benchmark (HEalth Advice in LLMs), a health-advice benchmark dataset that has been manually curated and annotated to evaluate LLMs’ capability in recognizing health-advice - which we use to safeguard LLMs deployed in industrial settings. We use HeAL to assess several models and report a detailed analysis of the findings.

## introduction large language models (llms) have impressive capabilities in natural language understanding and generation @xcite , and are becoming an integral part of our society. however, these models are typically trained on massive large-scale datasets, such as common crawl 2 , and if developed without proper governance, they can readily generate outputs that are not only inaccurate but potentially harmful. therefore, it is crucial to establish safeguards to ensure their responsible use @xcite , especially in heavily regulated industries, such as healthcare, law, and finance, that deal with critical decision-making @xcite . the research community has been focusing on a number of risks involving llms, from bias, dataset poisoning, lack of explainability, hallucinations, non-repeatability, sexually explicit content, hatebased content, privacy violation, and many others @xcite @xcite . the challenge that we are focusing on in this work is the ability of llms to provide answers that can be misconstrued as direct advice in the healthcare domain. health-related information is widespread on the web in various formats and writing styles, such as personal blogs, social media, hospital websites, etc. some of these sources may contain personal discussions about treatment history or diagnoses, leading to diverse and often unreliable training data for llms. this creates a potential for users to be misled into taking harmful actions. to avoid this, and the not-sosubtle risk of lawsuits, llms need to be carefully designed to differentiate between informative responses and actionable advice. this distinction can be quite subtle, making it a complex classification task for llm developers. the end goal is to ensure users are empowered with knowledge but not directed down a path that might have unintended consequences. to the best of our knowledge, recognizing advice in the context of safeguarding llms remains an under-explored area. this task is challenging due to several prominent factors. first, there is a limited amount of annotated data available to train ai models for this specific task. second, synthetically generating such data is inherently difficult, as capturing the nuances of human advice requires complex scenarios and contexts, not to mention also considering implicit expressions of advice. even web-crawled data, a potential source of training examples, needs meticulous verification to avoid misleading or irrelevant information. our work seeks to address this data scarcity issue by constructing a new benchmark dataset heal for health-advice identification. the motivation behind developing heal is that we desire an evaluation benchmark that is more representative of real-world deployment use cases. llm outputs are typically conversational, but existing benchmarks @xcite ) contain text from purely academic and medical sources. as a result, while the same content may be present, the style of language and text is drastically different in existing benchmarks from what deployed models would see. heal addresses this gap by combining academic sources with a large portion of conversational-style sources such as user forums, which is closer to the language style that would be seen in the real world. we construct our benchmark using "focused crawling", a simple and versatile methodology that can be adapted for data acquisition for other subtle classification tasks within the llm domain. note that all examples in our dataset are manually annotated. our heal dataset is a step in the direction of better evaluation benchmarks for health-advice detector models that would be deployed in the wild. better validation strategies ensure that only models with proper performance are exposed. by promoting the development of robust health-advice detection models, we aim at intercepting potential health-advice from llms: this is extremely important in real-world scenarios since potentially incorrect health-advice can lead to disastrous consequences. the contributions of this work are as follows: (1) we introduce a methodology for "focused crawling" that provides guidelines for gathering task relevant data. focused crawling works by first targeting relevant web sources, then extracting relevant content via seed keywords, and then ensuring sample correctness using human annotation. ## related work the safety of large language models (llms) has recently gained significant attention across the general public, industry, and the research community, with a proliferation of studies on the subject. it is now generally accepted that llms need a layer of "guardrails" to address several risks arising from the automatic generation of text, including bias, potential for unsafe actions, dataset poisoning, lack of explainability, hallucinations, non-repeatability, privacy, fairness, verifiable accountability @xcite as well as the ethical repercussions of llms' security threats, including prompt injection, jailbreaking, personal identifiable information (pii) exposure, sexually explicit content, and hate-based content @xcite . general solutions to safeguard llms fall into two categories: (i) external models or filters to prevent harmful outputs @xcite and/or (ii) specific safety training in the fine-tuning phase @xcite . other approaches focus more on input prompts, e.g. torchopera @xcite , which exploits vector databases, rule-based wrappers, and other specialized mechanisms to adjust unsafe or incorrect content. in terms of evaluating the effectiveness of llms' safeguards, @xcite propose the safety and over-defensiveness evaluation (sode) benchmark, a collection of safe and unsafe prompts and evaluation methods for systematic evaluation and analysis over 'safety @xcite there is a consensus on the need for ethical frameworks and auditing systems as well as for evaluations tailored to specific domains @xcite , especially in high-stakes domains such as law, medicine, and finance. one recent work @xcite evaluates the effectiveness of safeguards to prevent llms from being misused to generate health disinformation, and assesses various llms' generation of health disinformation. available health-specific approaches have been @xcite focuses on effective and safe patient interactions, using current conversation data and historical patient information. polaris @xcite on the other hand has a human-in-the-loop component -performed by nurses -to increase safety and reduce hallucinations. @xcite explore the sensitivity of llms to variations in user input, i.e how different descriptions of the same symptoms can lead to different diagnoses. the focus of our work is also on the medical domain, but -in contrast with state-of-the-art -we are specifically concerned with the detection of llm outputs that contain explicit medical advice. when it comes to the medical domain, there are several literature surveys on safeguarding llms. they explore training techniques, clinical validation, ethical considerations, data privacy, regulatory frameworks @xcite , accuracy, bias, patient confidentiality, responsibility @xcite , fairness, non-maleficence, transparency, the risk of producing harmful or convincing but inaccurate content @xcite , inaccurate medical advice, patient privacy violations, the creation of falsified documents or images @xcite , and generally the challenges associated with the use of llms in the context of diagnostic medicine @xcite . @xcite depict guidelines on integrating llms into healthcare and medical practices, while others propose performance metrics to evaluate llms in the biomedical domain @xcite . none of these studies, however, specifically address the recognition of medical advice in the output of llms. our purpose is similar to that of @xcite , which advocates the need for concrete criteria to determine the appropriateness of advice, although they address the legal domain. advice detection in the medical domain has already been explored in the literature @xcite , but focusing on academic medical text (i.e. scholarly articles from pubmed) @xcite or text extracted from professional websites @xcite , while assessing the performance of classical classification models (such as bert-base or tf-idf) trained on such data. unlike prior work, heal encompasses a wider variety of data sources, of which a significant component contains conversational-style text. this is crucial because our benchmark more closely resembles the conversational-style of output that real llms produce. additionally, we also conduct experiments on a wider range of relevant and popular language models, from bert-based models up to gpt-4o. ## recognizing advice in llms' responses our work is focused on understanding (i) how well llms can self-regulate against providing direct health advice to users, (ii) if external methods and filters should be added as safeguards, and (iii) how effective available benchmarks are on assessing the accuracy of health advice identification in llm outputs. our methodology involves, (i) using a variety of available llms and retrieving predictions by prompting the models in a zero-shot manner (section 5.1), (ii) fine-tuning bert-based models (section 5.2) and (iii) benchmarking all models against our newly generated heal benchmark (section 4.1). ## datasets we construct our health advice benchmark (heal) with data from four sources: webmd, mayo clinic, everyday health, and reddit. we provide details on the data creation process in section 4.1. the fine-tuning of our bert classifiers relies on additional publicly available datasets (see section 4.2). ## experiments and discussion we evaluate a variety of transformer-based models, ranging from 110m to over 175b parameters, on our gold standard benchmark. the evaluation and results are detailed below. ## conclusion in this work, we introduced the heal benchmark, which evaluates how well models can detect healthadvice in an industrial deployment setting. we drew our data from a variety of sources covering a wide range of distributions, from more formal, academic-like sources, to those that are more conversational (and more likely to occur during deployment). we benchmark a variety of models that users might encounter, from bert all the way up to gpt-4o, and note that there remains room for improvement for all of the models. additionally, we also conduct an error analysis for all the models, identifying what types of samples all models struggle on, and what individual models may frequently misclassify. future directions should focus on techniques/algorithms to improve the bert fine-tuning process, or methods that can provide insight on how to combat common failure modes.