Explainable Depression Detection Using Large Language Models on Social Media Data (2024)

Abstract: Due to the rapid growth of user interaction on different social media platforms, publicly available social media data has increased substantially. The sheer amount of data and level of personal information being shared on such platforms has made analyzing textual information to predict mental disorders such as depression a reliable preliminary step when it comes to psychometrics. In this study, we first proposed a system to search for texts that are related to depression symptoms from the Beck’s Depression Inventory (BDI) questionnaire, and providing a ranking for further investigation in a second step. Then, in this second step, we address the even more challenging task of automatic depression level detection, using writings and voluntary answers provided by users on Reddit. Several Large Language Models (LLMs) were applied in experiments. Our proposed system based on LLMs can generate both predictions and explanations for each question. By combining two LLMs for different questions, we achieved better performance on three of four metrics compared to the state-of-the-art and remained competitive on the one remaining metric. In addition, our system is explainable on two levels: first, knowing the answers to the BDI questions provides clues about the possible symptoms that could lead to a clinical diagnosis of depression; second, our system can explain the predicted answer for each question.

## introduction being one of the leading global public health issues, depression is common, costly, debilitating, and associated with an increased risk of suicide @xcite . since depression has become a prevalent mental health issue, early detection of symptoms could greatly improve the chances of proper treatment. traditional methods of detection, usually human-led, are expensive to conduct and might be individually biased. in this study, we propose a method to analyze and select social media writings to help identify potential symptoms of depression. then, we propose an explainable method that uses the selected writings to automatically fill in the beck's depression inventory (bdi) questionnaire @xcite for the social media user (see figure a1 for the full questionnaire). the questionnaire then provides the level of depression of the user based on all the answers. the main contributions of this paper are: ## related work to develop computational methods for depression detection using textual information, analyzing word usage became a natural starting point. through statistical investigation, researchers found that negative emotion, cause, sensory, and the first person singular words were more commonly used when describing activities such as breakup @xcite . linguistic inquiry and word count (liwc), a computerized text analysis tool, was developed to assess word usage in psychologically meaningful categories (tausczik and pennebaker, 2010). the tool was built by creating dictionaries from domain knowledge, with the words categorized into different groups. in addition to social and semantic features, linguistic n-gram features extracted from social media data were used by @xcite for estimating the degree of depression. @xcite further considered using demographic data such as age and gender as features, for classifying depressive symptoms based on social media (twitter data) on a population level. term frequency-inverse document frequency (tf-idf), which is a classic method for weighting words, was used to prepare features for predicting mental illness from social media @xcite . deep learning methods also attracted researchers working on the subject. @xcite proposed a method using a neural network model to identify the risk of self-harm or depression, using data from social media twitter and reddit. researchers participated in clef erisk 2017 @xcite focused on classifying users into binary targets: at risk or non-risk of depression. husseini orabi et al. (2018) explored the effectiveness of convolutional neural networks (cnns) and recurrent neural networks (rnns), on detecting signs of depression using unstructured text data extracted from twitter, released for the shared task on computational linguistics and clinical psychology (clpsych) 2015 @xcite . as part of clef erisk 2019, there was a proposed task about using the bdi questionnaire to automatically predict the depression level of social media users based on their social media writings, and the task continued in 2020 and 2021 @xcite . importantly, data was provided for the tasks. this led to somewhat explainable depression predictors, by indicating possible symptoms (such as lack of sleep, loss of appetite, and reduced physical activity). see figure a1 for the full set of questions from the bdi questionnaire and their possible answers. in general, the performance of the systems that participated in this shared task was poor. deep learning was used by a few of the participants in the task. for example, while participating in the shared task task 3 at clef erisk 2021 @xcite ## evaluation the same evaluation metrics were applied for the task in erisk 2019, 2020,and 2021 @xcite . the four metrics used for evaluation are: the ahr is the hit rate averaged over all the users. the hit rate measures the number of answers systems automatically fill in that are exactly the same as the actual answers provided by the users. the acr is the closeness rate averaged over all the users. it takes into account that the answers represent an ordinal scale, rather than merely separate options. to get the closeness rate, first compute the absolute difference between the automatically filled answer and the actual answer, then transform the calculated absolute difference into an effectiveness score as follows: the adodl is the difference between the system's and actual overall depression levels averaged over all users. the difference between overall depression levels (dodl) is obtained by first calculating the overall depression levels for the system-filled and actual questionnaire, then computing the absolute difference between the two overall scores. the dodl is normalized as follows: the dchr measures the closeness of the depression estimation achieved over all users according to the established depression categories introduced in figure 1 . it calculates the fraction of cases where the automated questionnaire led to a category that is identical to the user's actual depression category. ## results and discussion the experimental results of our systems using llms are shown in we can learn that the usage of usesim for user writing selection is helpful, and it is generally better to have more writings kept so that the model could have more information about the user, and the writings would be more focused on the specific question. in our experiments, the usage of top-5 dataset leads to a better performance than using the top-1 dataset. when using the top-5 dataset, the model neuralchat-7b-v3-1 performed better than sus-chat-34b on the metrics ahr, adodl, and dchr. this is surprising since the neural-chat-7b-v3-1 is much smaller than sus-chat-34b in terms of size/number of parameters. the reason could be the language focus and the application of the dpo algorithm. the llama-2-13b-chat model did not perform well on any of the metrics. through experimenting on the training data, the neural-chat-7b-v3-1's answers on questions 4, 8, 9, 11, @xcite @xcite @xcite although the performance on some metrics is still not outstanding, our systems scored over 85% on adodl, which is an improvement considering that adodl is the most critical metric for measuring depression at the population level @xcite . many runs scored over 50 on dchr, meaning that they predicted correctly for more than half of the test subjects on predicting their depression category. our experiments proved that llms have learned knowledge about various depression-related symptoms, and they can make better inferences than supervised deep learning techniques, with zeroshot learning (no training) and properly designed prompts. ## conclusion and future work in the study, we designed a system that searches for relevant sentences in numerous user writings, and applied it to provide cleaner data for a depression estimation system based on llms that automatically answers the questions from the bdi questionnaire. the resulting depression detection system has good performance on several metrics, and could sufficiently explain its answers to every question on the questionnaire for every user, without training on labeled data. however, due to the randomness of cyberspace, users' speeches on social media platforms cannot fully, objectively, accurately, and consistently describe their status of various depression-related symptoms. in the future, it would be good to collect larger high-quality datasets, so that we can run more experiments to calibrate our system and verify its effectiveness. also, with more computing resources and more powerful llms, much more user writings could be given to the model rather than filtered out, and it is expected that this would improve the performance. since our system does not need a large amount of training data, only a small set of labeled examples to design prompts, this is a promising avenue for automatically answering other types of mental health questionnaires, such as phq-9, anxiety questionnaires, etc.