Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis (2024)

Abstract: In this study, we introduce ANGST, a novel, first of its kind benchmark for depression-anxiety comorbidity classification from social media posts. Unlike contemporary datasets that often oversimplify the intricate interplay between different mental health disorders by treating them as isolated conditions, ANGST enables multi-label classification, allowing each post to be simultaneously identified as indicating depression and/or anxiety. Comprising 2876 meticulously annotated posts by expert psychologists and an additional 7667 silver-labeled posts, ANGST posits a more representative sample of online mental health discourse. Moreover, we benchmark ANGST using various state-of-the-art language models, ranging from Mental-BERT to GPT-4. Our results provide significant insights into the capabilities and limitations of these models in complex diagnostic scenarios. While GPT-4 generally outperforms other models, none achieve an F1 score exceeding 72% in multi-class comorbid classification, underscoring the ongoing challenges in applying language models to mental health diagnostics.

## introduction with the advancement of web technologies, social media platforms such as reddit @xcite , twitter @xcite , and reachout @xcite , become popular hubs for mental health support and information exchange. they offer an anonymous, safe space that fosters a sense of community and empowerment that begets in-depth mental health * equal contribution discussions @xcite . consequently, there is a growing body of research on "digital psychiatry" @xcite that analyzes the mental health discourse and language usage on these platforms to enhance the discovery, understanding, and detection of mental health concerns. however, despite significant efforts, there exist numerous concerns regarding the curation of these resources. ## related work corpora related to mental health disorders. since the past decade, social media platforms have been actively used for compiling datasets for various mental health issues @xcite @xcite . for instance, clpsych15 @xcite , rsdd @xcite , depression reddit @xcite , and dreaddit @xcite are commonly used corpora for depression analysis. similarly, datd @xcite and the anxiety on reddit corpus (shen and rudzicz, 2017) are popular benchmarks for anxiety issues. umd @xcite and t-sid @xcite are compiled exclusively for identifying suicide ideation. the clpsych16 dataset @xcite ) supports multi-label classification, covering depression, control, and ptsd categories. additionally, the cams @xcite and sad @xcite datasets offer more granular classifications, categorizing the causes of depression into six and nine categories, respectively. on a broader scale, smhd @xcite , swmh @xcite encompass a more comprehensive coverage of several mental health disorders. @xcite aggregated multiple mental health-related datasets into a unified benchmark named imhi. more recently, @xcite proposed psyeval, a suite of mental health-related tasks designed to evaluate the performance of large language models. ## yet another mental health dataset? in this section, we conduct a comprehensive crosssectional analysis of angst. we scrutinize it across various facets and juxtapose it with existing mental health corpora, thereby underlining its unique characteristics and distinguishing features. specifically, we compare angst against sdcnl @xcite , depression reddit @xcite dreaddit @xcite , and datd @xcite . we employ the corpora comparison strategy as outlined by @xcite . ## conclusion in this study, we propose angst a novel dataset aimed at bridging critical gaps in the comorbid diagnosis of depression and anxiety from social media posts. by embracing a multi-label classification approach, angst promotes research in early detection and understanding of comorbid depression and anxiety conditions, which are often co-occurring yet underserved by existing datasets. our benchmarking experiments with state-of-theart language models, including both discriminative and generative methods, highlight the superior performance of gpt-4 in both binary and multilabel classification tasks, reflecting its advanced capabilities in symptom identification and context understanding. conversely, llms such as gpt-3.5 and llama underperform relative to domainspecific plms like mental-xlm, underscoring the inherent challenges and limitations of these models in handling the complexity of mental health diagnosis from text. however, despite these advancements, all models demonstrated a tendency to misclassify complex cases, indicating a need for improved methods that can differentiate the subtleties of mental health language and symptom expressions, especially in settings where disorders co-occur. in the future, we aim to explore hybrid models that combine the strengths of both discriminative and generative approaches to improve the accuracy and reliability of mental health assessments based on social media data, ultimately contributing to the development of more effective tools for early diagnosis and intervention, thus improving outcomes for individuals with mental health needs.