Leveraging LLM s for Translating and Classifying Mental Health Data (2024)

Abstract: Large language models (LLMs) are increasingly used in medical fields. In mental health support, the early identification of linguistic markers associated with mental health conditions can provide valuable support to mental health professionals, and reduce long waiting times for patients.Despite the benefits of LLMs for mental health support, there is limited research on their application in mental health systems for languages other than English. Our study addresses this gap by focusing on the detection of depression severity in Greek through user-generated posts which are automatically translated from English. Our results show that GPT3.5-turbo is not very successful in identifying the severity of depression in English, and it has a varying performance in Greek as well. Our study underscores the necessity for further research, especially in languages with less resources.Also, careful implementation is necessary to ensure that LLMs are used effectively in mental health platforms, and human supervision remains crucial to avoid misdiagnosis.

## introduction mental health issues (e.g., depression, anxiety, and post-traumatic stress disorder (ptsd)) are prevalent worldwide and pose significant challenges to public health @xcite . traditional methods for diagnosing mental health conditions often rely on self-reported surveys, clinical interviews, and standardised assessments conducted by trained professionals @xcite . while these methods are effective, they are also resource-intensive, time-consuming, and may not always be accessible to individuals in need, particularly for speakers of languages beyond english. in this context, the application of llms to detect mental health symptoms from textual data offers a compelling alternative. these models can analyse large volumes of text data (e.g., social media posts, forum discussions, and personal narratives) quickly to identify linguistic markers associated with mental health conditions @xcite . this capability opens up new avenues for early detection and intervention, providing valuable support to mental health professionals and potentially reaching out to the patients whose symptoms may be overlooked and/or save time (e.g., long waiting times). despite the potential benefits, the performance of llms in multilingual mental health symptom detection remains underexplored. previous studies have primarily focused on english-language datasets, leaving a gap in our understanding of how these models perform in other linguistic contexts @xcite . hence, our work raises the following research questions: to address these research questions, first, we assess a state-of-the-art multilingual llm when predicting the severity of mental health in english user-generated posts. then, we automatically translate these posts from english to greek, a language for which there are no resources for this task @xcite , and re-assess the performance of the llm. our research not only contributes to the development of more robust and inclusive aidriven mental health diagnostic tools but also emphasises the importance of culturally and linguistically sensitive approaches in mental health care beyond english. the contribution of this work lies into the evaluation of the predictive power of a popular llm in detecting the severity of depression across english and greek. το παραπάνω κείµενο κατηγοριοποιείται στο επίπεδο σοβαρότητας κατάθλιψης 3. δεν το ξέρω αυτό. νομίζω ότι είμαι υπεύθυνος για όλα αυτά. για τους δράστες, για την κακοποίηση, για τη δυσλειτουργία, για την κατάθλιψη μου, τις αυτοκτονικές μου σκέψεις, την αθλιότητα της ζωής μου. όλα αυτά. είμαι υπεύθυνος. i don't know that. i think i'm responsible for all of it. for the perpetrators, for the abuse, for the dysfunction, for my depression, my suicidal ideation, my pathetic life. all of it. i'm responsible. predicted:3, ground truth:3 translate with llm to greek validate ground truth translation predict if indication of mental health condition or level tweet or reddit post in english llm llm prediction result figure 1: an illustration of our proposed methodology. ## related work llms have remarkable accuracy in detecting mental health symptoms by leveraging their ability to understand context and semantics at a deeper level. examples include biobert @xcite , and clinicalbert @xcite , which are pretrained on biomedical corpora or clinical notes. in contrast, models like mentalbert @xcite , and disorbert (aragón et al., 2023) are pre-trained on mental health-related social media data. additionally, research by @xcite showed that nlp can effectively assess depression and ptsd from clinical notes, further validating the utility of these models in a healthcare setting. although the details about training and evaluation are not always transparent, the multilingual capabilities of llms enable these models to understand and generate text in various languages. a recent example is the xlm-r model, which has been trained on a vast amount of multilingual data and shows strong performance across multiple languages. according to @xcite , their model xlm-r outperforms previous models on a wide range of tasks, demonstrating that leveraging large-scale multilingual data can lead to improvements in cross-lingual understanding. despite these advancements, significant challenges remain in achieving truly equitable performance across all languages and handling culturally specific contexts accurately @xcite . languages with limited digital text data still pose a considerable challenge for llms, often resulting in lower performance and less reliable outputs. addressing this issue requires more inclusive data collection practices and further research into transfer learning techniques that can better utilise limited resources @xcite . additionally, capturing cultural nuances and contextspecific meanings is a complex task, as language is deeply intertwined with cultural and societal norms. efforts to improve these aspects include developing more sophisticated algorithms and incorporating diverse and representative datasets @xcite , ensuring that the benefits of multilingual llms are accessible to a broader range of users globally. more recently, a plethora of social network datasets targeting mental health, have been available @xcite . the authors gathered social media posts from reddit and twitter regarding depression, ptsd, schizophrenia, and eating disorders. moreover, multiple models were finetuned on small-sized publicly available annotated mental health datasets by the authors to use them for labelling their introduced mentalhelp dataset. nevertheless, the dataset includes only posts in english, and thus its use is restrictive, disallowing further research for multilingual scenarios. ## proposed methodology our methodology leverages llms, in order to translate english social media posts to another language (greek), and then to predict mental health conditions accordingly. specifically, we translate the social media posts to greek via an llm, and we feed the resulting translations to a prompt that asks the llm to predict specific severity levels of mental health conditions. we assess the llm by comparing the predicted classes in both languages against the ground truth labels. we note that although our study focuses on greek, our method is applicable to other language pairs as well. an illustration of the proposed approach for evaluating llms for multilingual detection of mental health conditions is shown in figure 1 . ## experiments we select the depseverity dataset of @xcite , which consists of posts from the social media platform reddit, regarding different levels of depression. the posts (in english) are already labelled in terms of four levels of severity: minimal, we use gpt3.5-turbo @xcite ) through its api to translate the posts and predict the labels. the temperature parameter is set to 0, so the outcome is reproducible, regarding translations and predictions. we approach the task with text classification, comparing the predicted classes with the ground-truth ones, reporting precision, recall and f1. we experiment with english as the source and greek as the target language. the prompt we used to predict the severity levels is shown in preliminary prompting before exposing our llm to any posts, definitions, or instructions, either for the translation or the classification task, we asked how it would classify posts to different levels of depression severity. the response of llm was that it would initially try to identify language patterns associated with depression, such as: more specifically, it would try to adapt the four depression severity levels to fit the context of social media posts, as follows. we can infer that the llm expects posts with very generic indications of negative signs. classification in the source language initially, we experimented with the data in their source language (english), to set the baseline performance. that is, no translation step has been performed at this stage. as we observe in classification in the target language in greek, the worst results are observed for the mild level (@xmath3.06), similarly to english (@xmath4.07). overall, a drop in scores is observed across all classes except for the moderate level, where results improve (@xmath5.23; from 0.17). also, although the scores for the two edges remain relatively high, the score for the lowest severity dropped in greek (@xmath6.14). ## conclusion in our study, we focused on the ability of an llm to predict the severity of depression in user-generated posts in english (source language) and in greek (target language) when the posts are machinetranslated by the same llm. our findings show that there is room for improvement in the source language (english) and that the edge classes are easier to handle. in the target language (greek), results dropped for all but the moderate level, for which results increased considerably. considering the varying performance of the llm across the two languages, there is a need for utmost precautions not to rely on llms solely for translation in any healthcare setting. as stated by @xcite , diagnosis of mental health should never be left alone to automatic systems, and it should never replace the diagnosis by human professionals, to avoid possible errors and/or misdiagnoses. our approach, however, does not aim to assist the patients. by contrast, it is potentially useful to train professionals in the mental healthcare domain, which can be vital for languages other than english.