Delving into the Depths: Evaluating Depression Severity through BDI -biased Summaries (2024)

Abstract: Depression is a global concern suffered by millions of people, significantly impacting their thoughts and behavior. Over the years, heightened awareness, spurred by health campaigns and other initiatives, has driven the study of this disorder using data collected from social media platforms. In our research, we aim to gauge the severity of symptoms related to depression among social media users. The ultimate goal is to estimate the user’s responses to a well-known standardized psychological questionnaire, the Beck Depression Inventory-II (BDI). This is a 21-question multiple-choice self-report inventory that covers multiple topics about how the subject has been feeling. Mining users’ social media interactions and understanding psychological states represents a challenging goal. To that end, we present here an approach based on search and summarization that extracts multiple BDI-biased summaries from the thread of users’ publications. We also leverage a robust large language model to estimate the potential answer for each BDI item. Our method involves several steps. First, we employ a search strategy based on sentence similarity to obtain pertinent extracts related to each topic in the BDI questionnaire. Next, we compile summaries of the content of these groups of extracts. Last, we exploit chatGPT to respond to the 21 BDI questions, using the summaries as contextual information in the prompt. Our model has undergone rigorous evaluation across various depression datasets, yielding encouraging results. The experimental report includes a comparison against an assessment done by expert humans and competes favorably with state-of-the-art methods.

## introduction nowadays, numerous individuals in the world suffer from diverse mental conditions that disrupt their cognition and conduct and, ultimately, represent a detriment to their quality of life @xcite . as an illustration, depression stands out as one of the most prevalent mental disorders, positioning itself as a primary catalyst for suicidal tendencies @xcite . a major depressive disorder is a significant medical condition that has adverse effects on emotions, thoughts, and behaviors. depression induces feelings of sadness and a diminished interest in previously enjoyable activities. this condition can result in various emotional and physical challenges, impacting one's ability to perform effectively both in the workplace and at home @xcite . currently, only approximately 20% of those afflicted receive necessary early intervention, with a significant proportion of mental health expenditures allocated to the maintenance of psychiatric institutions as opposed to activities encompassing detection, prevention, and recovery @xcite . given these circumstances, there exists an urgent need to design effective approaches for the early detection of depression, aiming to avoid harm to individuals suffering from this condition. the ubiquity of social media data has paved the way for data-driven research in the field of mental health analysis @xcite . a significant portion of individuals conduct the bulk of their social interactions within the digital realm crafted by social media platforms such as facebook, twitter, reddit, and instagram. nowadays, researchers have access to extensive corpora of online dialogues on diverse topics. this wealth of data holds particular significance in medicine, where progress in our understanding of mental health could directly contribute to life-saving quality-of-life measures and improvements. exploiting public interactions offers a valuable avenue for comprehending depression, thereby amplifying the potential to identify individuals displaying depressive indicators and facilitating professional intervention @xcite . diverse techniques rooted in natural language processing (nlp), text classification (tc), and information retrieval (ir) have been employed to discern signs of depression, with a particular focus on linguistic and sentiment analysis @xcite . however, most of the existing studies have been confined to distinguishing between a depression group and a control group (two-class classification) and provide no further explanation or explicit standardized signs that health professionals can analyze. furthermore, conventional strategies have demonstrated their effectiveness in detecting depressive individuals based on their textual interactions @xcite , but they heavily rely on the intricate process of feature engineering (e.g., by extracting optimal user attributes that reflect the subject's feelings and psychological state). however, the nlp landscape has radically evolved in recent years, with the ascent of large language models (llms). new models, such as chatgpt, have gained immense popularity due to their capacity to deliver zero-shot and few-shot predictions across diverse tasks foot_0 . this ability stems from the llms' augmented scale, with a huge number of parameters that inherently empower them to encapsulate the subtleties inherent in massive amounts of textual data. this becomes particularly pivotal when confronting linguistic data, given the inherent variance in word significance dependent on the context. to properly exploit current llms to support bdi-based screening, the parametric knowledge of the llm, which provides a sophisticated understanding of human language, needs to be enriched with userspecific interactions related to standardized depression symptoms. this is precisely the main goal of our research. more specifically, this study designs effective search strategies to mine bdi-biased summaries from the users' posting history and proposes the utilization of llms for quantifying levels of depression. our approach can be regarded as a retrievalthen-read method @xcite that augments the llm knowledge with personalized bdi-biased summaries built for each category of the bdi-ii questionnaire. bdi @xcite is a recognized psychological instrument designed to assess the manifestation of 21 depressive symptoms, such as sadness, pessimism, or loss of energy. we can summarize our contributions as follows: the examination of public mental health via social media has experienced significant growth in recent years @xcite @xcite . recent research has focused on depressive symptom detection to enhance mental health models, highlighting their potential to enhance performance, general applicability, and interpretability @xcite . for instance, in @xcite , the authors introduced methods for identifying depression that incorporate various levels of constraints based on the symptoms outlined in the phq9 questionnaire, a tool used by clinicians for screening depression. their experiments, conducted across three social media datasets, revealed that their model can adapt to unfamiliar data, surpassing a conventional bert-based approach. another study @xcite ) presented an approach for automatically gauging the severity of depression in social media users. this research team tackled the task of quantifying the intensity of depression indicators and explored using neural language models to capture different facets of a user's writings. they presented two alternative methodologies to assess the sensitivity of symptoms in terms of the user's willingness to openly discuss them. the first method relies on global language patterns from the user's posts, while the second method seeks direct mentions of symptom-related concerns. both techniques led to automatic estimates of the overall bdi-ii score. furthermore, in @xcite , an efficient semantic pipeline was introduced for evaluating depression severity in individuals based on their social media content. the authors selected a sample of user sentences to create semantic rankings. the approach was supported by a reference index of training sentences that correspond to depressive symptoms and severity levels. subsequently, they employed the sentences derived from these rankings as evidence for predicting the severity of symptoms in users. in a different direction, zhang et al. (2022) introduced a method for screening risky posts guided by psychiatric scales. this method identified posts that exhibit risk factors associated with the dimensions outlined in clinical depression scales, providing a basis for a comprehensible diagnosis. to enhance the transparency of predictions, this team proposed a hierarchical attentional network integrated with bert, known as han-bert. in recent years, with the proliferation of large language models (llms), there has been a response to the limitations observed in psychological knowledge by developing specialized language models that offer improved accuracy in providing psychological advice @xcite . such endeavors have sparked our interest in exploring the potential of llms to respond to questionnaires related to depression symptoms and compare them with the assessment done by an expert psychologist. our approach can be seen as a novel application of retrieve-then-read methods for llms @xcite , where the parametric llm model is conditioned by personalized summaries for each user. ## proposed approach the objective of this research consists of estimating the level of depression from a thread of users' posts @xcite . additionally, we contrast our estimates with the answers provided by an expert psychologist, who is also presented with user-level evidence mined from social media. to that end, we summarize the post history of each user, and our model estimates the response to each bdi item based on the evidence found in bdi-specific summaries. the approach consists of three main steps: ## evaluation one noteworthy observation is that both variants of chatgpt obtained better results than the other automatic models when it came to fine-grained metrics that compute the effectiveness over individual questions (ahr, acr). these results highlight how close the answers given by these two models are to the answers provided by the users. the chatgpt models tend to yield high values in the acr metric. this is an important outcome since acr focuses on the closeness between the real and automated responses, and a system with high acr might have some potential to understand the feelings of the individual about the bdi symptom and develop psychological screening tools accordingly. on the other hand, bert-squad excelled in terms of global metrics (adodl, dchr) that focus on the divergence between the overall estimates of depression. it's noteworthy that chatgpt version 3.5 consistently excels in producing responses that closely align with user input, while, version 4 tends to perform better when evaluated using broader global metrics, possibly owing to its enhanced capacity for generalizing information. still, there is much room for improvement in accurately predicting human responses. this is partly due to limited data availability, as there are many bdi topics that are not discussed or disclosed on social media. in any case, it is important to note that some automatic systems were on par with (or superior to) the assessment is done by human experts. in fact, the best automatic systems yielded equivalent performance to the expert psychologist in the fine-grained metrics (ahr and acr) and better performance in the overall depression estimates (adodl and dchr). in any case, the overall predictions (as reflected by dhcr) do not match those of the real surveys and this suggests that some bdi symptoms are difficult to grasp. regarding the time required, the expert took approximately 30 to 42 hours for each dataset (approximately 35 minutes per user). instead, the llms took around 2-3 minutes to answer each user's questions. this signifies a substantial reduction in time, showcasing a pivotal advantage of automated methods that can facilitate the screening processes. by optimizing the extraction and analysis through computational tools, health professionals have the opportunity to allocate their saved time to the most confusing cases or just to review the output of the llms. last, we have done an additional comparison between the predictions generated by the chatgpt 3.5 model and those of the domain expert. this comparison allows us to understand the degree of similarity between the respective responses. to that end, we employed cohen's kappa score. the purpose is to provide insights into the model's performance by examining its alignment with human expertise across the entire range of users. these scores consistently hover around 0.28 for the 2019 and 2020 datasets and 0.0648 for 2021. this value, although modest, signifies a fair level of agreement between our model's predictions and those of the expert. in the 2021 dataset, we observe a low level of agreement; however, it is noteworthy that even in this collection the automated systems consistently outperformed the experts in predicting symptoms. these agreement levels underscore the model's capability to generate responses that align with expert judgments, demonstrating its reliability and effectiveness in providing valuable insights. while the agreement is not high, the model's performance is promising, considering the inherent complexity of the task at hand. these findings reinforce the model's potential to assist decision-making in the mental health domain. comparison against erisk participants: to put these results in perspective, nonetheless, this also opens up opportunities for refining our model's architecture and incorporating additional techniques to bridge the gap between its current performance and that of the most effective erisk systems. ## analysis and discussion it is important to assess the extent to which the bdi topics have relevant sentences and the individual impact of bdi questions on the overall depression score. to that end, we analyze here the presence of relevant sentences for each bdi topic and plot it against the average rating in the ground truth (see figure 4 ). the blue bars represent the proportion of users that had at least one relevant sentence for the corresponding topic (i.e. a non-empty summary). for instance, in 2019, for the topic of 'sadness', only 75% of the users had at least one related sentence. the red bars represent the average severity score provided by the users. certain themes, such as pessimism and selfdislike, are prominent (consistently provide relevant sentences for the majority of users) and tend to receive higher severity scores compared to other bdi symptoms. this suggests a correspondence between the real feelings of these users and their social media activity (i.e., they tend to disclose thoughts about these symptoms). other topics, such as loss of energy or punishment, have fewer relevant sentences (less than half of the users have at least one relevant sentence for these topics). interestingly, in the case of energy loss, users provided high severity estimates, but the model could not find much evidence. this highlights a significant barrier in screening depression symptoms. if the model cannot find pertinent information on these topics then it can hardly supply a reliable estimate. in those cases, we assumed a rating of 0 and, thus, the models might be underestimating the state of the individual. in the future, it will be interesting to study other alternatives, such as estimating the overall depression scores based on partially filled questionnaires or estimating the missing bdi symptoms based on the most similar symptoms. ## conclusions and future work in this study, we address a critical global concern, the prevalence of depression. we are committed to inducing a positive impact on automated methods for depression screening. to that end, we need a deeper understanding of depression symptoms and more evidence of how the symptoms reflect on social media. we have presented a comprehensive approach that involves extracting bdi-biased summaries from users' publications and exploiting different large language models to estimate the responses of those users to the beck depression inventory. our evaluation across various depression datasets yielded promising results, showcasing our method's potential to contribute to the understanding and assessment of depression. some of the proposed variants compete favorably with stateof-the-art methods and expert human evaluations. this work represents a valuable step forward in leveraging the power of data to address mental health challenges on a broader scale. in future work, we want to explore the application of other lexical resources that are even more specialized for the task of extraction of relevant sentences, as well as the usage of clinical data to train more specialized language models. furthermore, the primary focus of our work revolves around leveraging these summaries. specifically, our interest lies in the potential application of this tool to extract valuable linguistic indica-tors. this application could be useful in enhancing psychologists' understanding of how depression manifests in social media contexts. by delving into linguistic patterns and cues from user-generated content we could offer valuable insights that contribute to the refinement of psychologists' working knowledge. we also are interested in expanding this study to different languages, since most of the work related to mental disorders has focused on english. finally, this study represents a preliminary exploration but we believe that the ability to model user behavior through social media analysis offers promising prospects for the development of future wellness-oriented technologies. this innovative technology has the potential to function as a preemptive warning system, conducting extensive analyses and delivering pertinent information concerning mental health without compromising user privacy. for example, we could design local, regional, or national estimates of the prevalence of multiple depression symptoms, allowing authorities to make informed decisions about professional assistance, emotional support campaigns, and so forth. under this context, users should always retain autonomy in choosing to have access to certain recommendations or preemptive measures, empowering them to make informed decisions about their well-being.