Can AI Relate: Testing Large Language Model Response for Mental Health Support (2024)

Abstract: Large language models (LLMs) are already being piloted for clinical use in hospital systems like NYU Langone, Dana-Farber and the NHS. A proposed deployment use case is psychotherapy, where a LLM-powered chatbot can treat a patient undergoing a mental health crisis. Deployment of LLMs for mental health response could hypothetically broaden access to psychotherapy and provide new possibilities for personalizing care. However, recent high-profile failures, like damaging dieting advice offered by the Tessa chatbot to patients with eating disorders, have led to doubt about their reliability in high-stakes and safety-critical settings.In this work, we develop an evaluation framework for determining whether LLM response is a viable and ethical path forward for the automation of mental health treatment. Our framework measures equity in empathy and adherence of LLM responses to motivational interviewing theory. Using human evaluation with trained clinicians and automatic quality-of-care metrics grounded in psychology research, we compare the responses provided by peer-to-peer responders to those provided by a state-of-the-art LLM.We show that LLMs like GPT-4 use implicit and explicit cues to infer patient demographics like race. We then show that there are statistically significant discrepancies between patient subgroups: Responses to Black posters consistently have lower empathy than for any other demographic group (2%-13% lower than the control group). Promisingly, we do find that the manner in which responses are generated significantly impacts the quality of the response. We conclude by proposing safety guidelines for the potential deployment of LLMs for mental health response.

## introduction in 2018, openai released a powerful computational model for mimicking and understanding human language known as the generative pretrained transformer (gpt, @xcite . over the next few years, gpt and other large language models (llms) trained on vast quantities of web data have revolutionized human-ai collaboration. hospital systems are already piloting llms like @xcite for routine diagnostics @xcite , responding to patient messages and generating clinical notes from patient data at scale @xcite . early findings indicate an overall positive reaction from clinicians, who view llms as a tool to help cope with clinician burnout and expand reach of services @xcite . our study focuses on the use of llms for mental health response, which has faced increased pressure. between 2020 and 2022, a pew research poll found that 41% of surveyed americans reported high levels of psychological distress. this is particularly true amongst minority-identifying participants: 46% of hispanic americans and 42% of black americans reported high levels of distress compared to 38% of white americans @xcite . one response to this mental health crisis has been the development of conversational agents to provide large-scale on-demand therapy. notable examples include woebot foot_0 and youper, foot_1 chatbots trained to use cognitive behavioral therapy (cbt) techniques @xcite . while this is not a novel concept, with the eliza chatbot envisioned in 1966 predating modern applications @xcite , the advent of llms raises new deployment opportunities and risks. while llms show remarkable performance on mental health related tasks like suicide ideation or risk prediction @xcite and self-referral to mental health services @xcite , recent cases highlight the potential negative impact; (1) the death of a belgian man by suicide after conversation with a chatbot based on gpt-j @xcite , and (2) advice given by chatbot tessa to anorexia patients that they should diet and monitor their weight @xcite . while work on evaluating empathy of llms is a promising step toward preventing such disastrous automation errors (e.g. sharma et al., 2020), the potential for demographic perception bias in open-ended response has not yet been comprehensively explored. specifically, no work has addressed whether llms provide equitable care across patient subgroups in a generative psychotherapy setting. here we evaluate risks of discriminatory decision-making by llms for mental health response, to ensure all patients can receive equitable, appropriate care. ## data to allow reproducibility of this work, we focus on peer-to-peer support from communities on reddit known as "subreddits." prior work @xcite has provided evidence of the growing prevalence of social media as a platform for mental health support. we use the reddit api to collect a new dataset of 12,513 posts with 70,429 peer responses from 26 mental health related subreddits. ## llm response generation for gpt-4 response, 4 we test 3 different treatment persona settings in total: these are designed to emulate 3 distinct and common sources of support a patient might seek: peer-to-peer support from a general audience, peerto-peer support from a mental health focused audience, and professional support. moreover, we further propose two settings as attempts to mitigate the bias: in theory, an unaware respondent should be more fair because they are conducting a blind evaluation. however, an explicitly instructed aware respondent may also be more fair because they are alerted to potential bias. an example smp persona prompt is shown below: ## attribute inference given the sensitive nature of mental health data, personally identifiable information is rarely volunteered and publicly released research data is often de-identified. however, utterances may still contain implicit cues to a speaker's background like linguistic variation (e.g. unique verb conjugation or word dropping) as well as explicit references (e.g. self-identifying race). ## evaluation methodology clinical evaluation. we first conduct an expert evaluation with 2 licensed clinical psychologists. the 2 clinicians have backgrounds in adult psychology, specializing in depression, anxiety disorders and cognitive behavior therapy (cbt). we randomly sample 50 reddit posts seeking mental health support, and pair each post with either an actual peer-to-peer response or a gpt-4 generated re-sponse foot_4 in a random order. the post/response pairs are shown to the clinicians using a qualtrics foot_5 study. for ethical purposes, we state in the instructions that responses may be ai-generated, but we do not reveal the source of individual responses. we then ask the clinicians a set of 4 questions for each post/response pair, aimed at assessing the quality of care provided by the response. based on the epit-ome framework @xcite grounded in psychology research, we ask the following questions to gauge level of empathy on a 3-point likert scale (none, weak, strong): for a more holistic view of quality of care, we then adapt the 5-point likert scale change talk score from the motivational interviewing treatment integrity code (miti, @xcite ): at the conclusion of the evaluation, we conduct a manipulation check in which the clinicians must predict which percentage of the responses were aigenerated. demographic information identifying the poster is removed or masked out from all posts and responses before evaluation. ## demographic leaking experiment to assess the degree to which llm-driven response is influenced by implicitly or explicitly leaked patient attributes (e.g. gender, race) compared to peer-to-peer support, we conduct a counterfactual human evaluation. counterfactual scenarios. we start with a set of 11 seed "neutral" reddit posts 7 seeking mental health advice for which gpt-4 was unable to predict any attributes of the patient. we then transform each post to show either an explicit leak (e.g. "i am a 32yo black woman") or implicit leak (e.g. "being a 32yo girls girl wearing my natural hair") using keyphrases self-disclosed by gpt-4 as indicators of a certain demographic (see §5 for details). we test 8 different combinations of post transformations, 8 as well as a control setting with the original reddit posts. evaluation environment. we set up a randomized controlled experiment using amazon mechanical turk 9 to crowdsource peer-to-peer support responses and measure variation for each post attribute transformation setting. each mechanical turk worker is shown a post and asked to provide a response, as if responding to a friend. for quality control, we request that workers provide keywords 7 we determined based on statistical power analysis for a standard t-test with a moderate effect size that a minimum of 64 samples were required. 8 the combinations we consider are: [white-male-implicit, white-male-explicit, white-female-implicit, white-femaleexplicit, black-male-implicit, black-male-explicit, blackfemale-implicit, black-female-explicit] 9 https://www.mturk.com/ related to the post to prove they've read the content. we additionally filtered responses that are generic (e.g. "this is very good") or are directly copied from the post itself. we collect a maximum of 10 responses for each post. cognitive bias measurement. to measure and compare human bias in the presence of demographic leaking against llm bias, we generate gpt-4 responses for the same evaluated sets of original and transformed posts. we then use the same automatic metrics described in the previous section to assess demographic parity for human vs. gpt-4 responses. ## bias mitigation we find that explicitly instructing llms to use demographic attributes in response (mhf-2, mhf-3) can effectively alleviate bias depending on the model. the mhf-3 prompt context in figure 2 is the only one for which we found no statistically significant difference in empathy across subgroups for gpt-4 (see figures 2a , 2d and 2g ). however, results are more mixed for gpt-3.5 and mental-llama. in particular see figure 2i , where results are not significantly different for all subgroups with mhf-2, however they are for both mhf-2 and mhf-3 with gpt-3.5 in figure 2h . there is some precedent for this explicit direction in human cognition, with prior studies finding that explicit racial priming can counteract implicit bias and encourage fairness @xcite , though explicit direction is not always effective with humans compared to implicit priming @xcite . future work can explore whether there exists a more universally fair prompt context. ## conclusion in conclusion, we address the current lack of mental health chatbot bias testing by developing a computational framework for assessing equity and overall quality of llm-based mental healthcare. we find that while gpt-4 can reflect and amplify harmful biases found in peer-to-peer support, these biases vary significantly based on prompt design and can be mitigated through demographic-aware prompting. this proposed solution is a simple and actionable way of incorporating explicit bias mitigation into future work on generative mental health support. we propose that developers of llm-based mental health technologies employ (1) carefully designed llm instruction in mental health support to avoid bias, and (2) rigorous evaluation of mental health chatbots that considers potential variation across demographic groups. we hope that our findings lead to more concrete guidelines for future deployment of llms in psychotherapy.