Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments (2024)

Abstract: The shortage of clinical workforce presents significant challenges in mental healthcare, limiting access to formal diagnostics and services. We aim to tackle this shortage by integrating a customized large language model (LLM) into the workflow, thus promoting equity in mental healthcare for the general population. Although LLMs have showcased their capability in clinical decision-making, their adaptation to severe conditions like Post-traumatic Stress Disorder (PTSD) remains largely unexplored. Therefore, we collect 411 clinician-administered diagnostic interviews and devise a novel approach to obtain high-quality data. Moreover, we build a comprehensive framework to automate PTSD diagnostic assessments based on interview contents by leveraging two state-of-the-art LLMs, GPT-4 and Llama-2, with potential for broader clinical diagnoses. Our results illustrate strong promise for LLMs, tested on our dataset, to aid clinicians in diagnostic validation. To the best of our knowledge, this is the first AI system that fully automates assessments for mental illness based on clinician-administered interviews.

## introduction mental health has become a vital element of overall well-being. the prevalence of mental illness poses, however, a critical challenge to healthcare, underscoring the urgent need for an increased capacity of mental health services. only 29% of people with psychosis receive formal care, leaving a significant portion completely untreated (who: world health organization (2021)). aside from obstacles such as high costs, limited awareness, and stigma surrounding mental health, the shortage of the mental health workforce has been a major factor exacerbating this gap. according to who, the average ratio of mental health workers per 100,000 population was 13, making it difficult for people to access reliable and readily administrated mental health diagnostics, as well as subsequent support and interventions. the emergence of large language models (llms) has suggested innovative solutions to this challenge. several studies have explored llm applications in mental health for condition detection @xcite , support and counseling @xcite as well as clinical decision-making @xcite , and shown the feasibility for llms to enhance the workforce of mental healthcare @xcite . by harnessing llms' ability to interpret languages that involve high expertise, it is possible to mitigate the service gap in the healthcare ecosystem through the automation of condition detection and diagnosis without the need of training so many professionals, which is both costly and time-consuming. despite these advancements, notable limitations persist in the current research on automatic diagnosis for mental health. most studies have focused on prevalent conditions like stress @xcite and depression @xcite , with scant attention to less common but more severe conditions like post-traumatic stress disorder (ptsd). moreover, while prior studies have leveraged data from social media, clinical notes, and electronic health records, very few have utilized clinical interviews, and even in those cases, they rely on basic self-administered scales estimated in dialogues between computers and patients @xcite . no work has employed diagnostic interviews between real clinicians and patients that are systematically conducted, resulting in a dearth of practical research on the automatic diagnosis of mental illness. in this paper, we present an llm-based system that listens to hour-long conversations between clinicians and patients and performs diagnostic assessments for ptsd. our final model is evaluated by clinicians specialized in ptsd, suggesting a great potential for llms while highlighting certain limitations (section 6). our primary contributions are: foot_0 to the best of our knowledge, this is the inaugural system designed to conduct diagnostic assessments on mental health while interpreting real-world interviews administered by clinicians. we believe that this work will foster clinical collaboration between human experts and artificial intelligence, thus promoting equitable access to appropriate care for all populations affected by mental illness. ## related work pre-trained language models have been widely applied in many healthcare tasks @xcite @xcite @xcite . the emergence of llms has introduced new capabilities and innovations in healthcare to this domain @xcite . this section introduces the related research of llms and their applications in healthcare, particularly in mental health. ## ptsd interview data this study utilizes data from diagnostic interviews administered as part of a larger study on risk and resiliency to the ptsd development in a population seeking medical care @xcite . participants were recruited from waiting rooms in primary care, gynecology and obstetrics, and diabetes medical clinics at a publicly funded, safety-net hospital. data were collected from 2012 to 2023, and inclusion criteria were ages between 18 and 65 with the capacity to provide informed consent. the parent study was conducted according to the latest version of the declaration of helsinki (world medical association, 2013), and consent from the participants was obtained after explaining the procedures. the informed consent was approved by our institutional review board and research oversight committee. ## data processing every video is converted into an mp3 audio file and transcribed by two automatic speech recognizers, whose results are aligned to produce a high-quality transcript. the transcript is segmented into multiple sections based on the relevant questions, and each question is paired with its assessment result. ## error analysis a thorough error analysis is conducted by proportionally sampling 100+ examples per variable type. six types of major errors are identified ( ## conclusion in this study, we undertook the task of automating ptsd diagnostics using 411 clinician-administered interviews. to ensure the data quality, we develop an end-to-end pipeline streamlining transcription, alignment, segmentation, and assessment pairing. we also construct a pioneering framework for this task by leveraging two state-of-the-art llms. our findings reveal the substantial potential of llms in assisting clinicians with diagnostic validation and decision-making processes. our error analysis suggests future directions for improvement, such as incorporating external information or commonsense knowledge to engineer more comprehensive instructions. we envision that this framework holds promise for addressing a broader spectrum of mental health conditions and offers novel insights into llm applications within the mental health domain. we plan to collect more data and train a custom llm to better preserve patients' privacy, and develop a dialogue system to conduct the interviews.