Semantic Similarity Models for Depression Severity Estimation (2023)

Abstract: Depressive disorders constitute a severe public health issue worldwide. However, public health systems have limited capacity for case detection and diagnosis. In this regard, the widespread use of social media has opened up a way to access public information on a large scale. Computational methods can serve as support tools for rapid screening by exploiting this user-generated social media content. This paper presents an efficient semantic pipeline to study depression severity in individuals based on their social media writings. We select test user sentences for producing semantic rankings over an index of representative training sentences corresponding to depressive symptoms and severity levels. Then, we use the sentences from those results as evidence for predicting symptoms severity. For that, we explore different aggregation methods to answer one of four Beck Depression Inventory (BDI-II) options per symptom. We evaluate our methods on two Reddit-based benchmarks, achieving improvement over state of the art in terms of measuring depression level.

## introduction around two-thirds of all cases of depression remain undiagnosed according to conservative estimates @xcite . to help with this problem, governments and agencies have launched programs to raise awareness of mental health in their citizens @xcite . in this context, detecting and receiving appropriate treatment in the early stages of these diseases is essential to reduce their impact and case escalation @xcite . however, the insufficient resources of the public health systems severely limit their capacity for case detection and diagnosis. as an alternative to public health systems, social platforms are a promising channel to assess risks in an unobtrusive manner @xcite , where people tend to consider these platforms as comfortable media to express their feelings and concerns @xcite . exploiting this type of user-generated content, nlp techniques have shown promising results in terms of identifying depressive patterns and linguistic markers @xcite . due to the growing popularity of the mental health detection models, the community also produced diverse datasets @xcite , with the early risk prediction on the internet (erisk) @xcite , and the computational linguistics and clinical psychology (clpsych) @xcite being the two most popular benchmarks in the field. they produce task definitions, datasets and evaluation methodologies to encourage research in this domain. depression identification from social media posts faces challenges considering their integration into clinical settings @xcite . previous studies formulated this task as a binary classification problem (i.e., depressed vs control users) @xcite . despite achieving promising results under this setting, ignoring different levels of depression limits the capacity to prioritize users with higher risks @xcite . moreover, most existing approaches focused on the use of engineered features, which may be more difficult to interpret than other clinical markers foot_0 , such as the integration of recognized depressive symptoms @xcite . similarly, the blackbox nature of deep learning models also limits the ability to understand their decisions, especially by domain experts like clinicians. in this paper, we perform a fine-grained analysis of depression severity using semantic features to detect the presence of symptom markers. our methods adhere to accepted clinical protocols by automatically completing the bdi-ii @xcite , a questionnaire used to measure depression. the bdi-ii includes 21 recognized symptoms, such as sadness, fatigue or sleep issues. each symptom has four alternative responses scaled in severity from 0 to 3. using a sentence-based pipeline, we build 21 different symptom-classifiers for estimating the user responses to the symptoms. for this purpose, we employ erisk collections related to @xcite @xcite . in our pipeline, we explore selection algorithms to filter relevant sentences to each bdi-ii symptom from training users. once filtered, we index these training sentences with the user responses as labels (0 -3) as examples of how people with different severities speak about the symptom. then, to predict test users responses, we select their relevant sentences, which serve as queries to produce a semantic ranking over the indexed training sentences. finally, we construct two aggregation methods based on the ranking results to estimate the symptoms severity. the main contributions of this work are: 1) we present a semantic retrieval pipeline to perform a fine-grained classification of the severity of depressive symptoms. following the symptoms covered by the bdi-ii, our methods also consider different depression severity levels, distinguishing between lower and higher risks. 2) we propose a data selection process using a range of unsupervised and semi-supervised selection strategies to filter relevant sentences for the symptoms. 3) experiments using different variants of our pipeline achieved remarkable results in two erisk collections, outperforming state of the art considering depression severity in both datasets. ## related work extensive research investigated the use of engineered features to identify linguistic markers and patterns related to mental disorders from different social platforms @xcite @xcite . for instance, the liwc writing analysis tool @xcite , equipped with psychological categories, revealed remarkable differences in writing style between depression and control groups @xcite . other studies used depression and emotional lexicons to determine depression markers @xcite , whereas @xcite examined additional distinctive features, leveraging profile metadata (e.g. posting hours or posts length) and social activity to examine the mental state of individuals. the recent advances in contextualized embeddings significantly impacted many nlp-related tasks, including depression detection in social media. these deep learning models have consistently outperformed engineered features on diverse datasets @xcite . however, they lack the interpretability clinicians require to rely on the results from automated screening methods @xcite . to enhance interpretability, the works proposed to the erisk depression estimation shared task based their efforts on predicting bdi-ii symptoms responses @xcite @xcite . in this study, we work with the erisk collections and follow the same evaluation methodology. in contrast to these approaches, our methods highlight the user posts that lead to every symptom decision, which may be helpful for further inspection of model predictions. besides the works presented to erisk, two recent studies explored the use of depressive symptoms to screen social media posts. @xcite aggregated symptoms from different questionnaires into a bert-base model to calculate symptom risk at post level. @xcite experimented with various methods using symptom markers to detect depression, demonstrating their potential to improve the generalization and interpretability of their methods. in this case, authors considered the symptoms from the ph9q questionnaire @xcite to define manual pattern-based strategies and train symptom-classifiers at post level. both approaches formulated their methods with a binary classification setting, while our approach considers different severity levels. we also differ in that we pre-compute dense representations of training posts, rather than relying on pre-trained language models, which may be slow for many practical cases @xcite . our approach only needs a few post encodings and cosine similarity calculations, improving the efficiency of our solutions. ## method problem definition: we aim to estimate the depression severity level for users based on the writing history (wh) of their social media posts. we define depression severity levels following the clinical classification schema of the bdi-ii score @xcite test queries q ... aggregation methods for decisions based on all rankings ,..., ) = d sleep q energy 1 r q sleep q energy q sad ? ? ? r sleep ,..., ) = d sad r k r sad 1 aggr( ,..., ) = d energy r r energy 1 s sad energy k sleep 1 k ... ... ... ... ... ... aggr( aggr( from test user energy sleep sad ... ... ... index with silver sentences for symptom: loss of energy silver sentences ranking for: sentence rank r energy ag 1 energy ... ... (ag , 3) i (ag , 2) n 3 ... ... 2 q energy 1 silver label ... ... et @xcite . the score is the sum of the option responses to the 21 symptoms covered by this questionnaire, and it is associated with four depression levels. instead of relying on a unique classifier to calculate that score, we build 21 different symptomclassifiers (i.e., one for each bdi-ii symptom). we categorize the symptoms into one of its four possible response options. therefore, we formulate each classifier as a multi-class classification problem. our approach relies on two main components: 1) a semantic retrieval pipeline ( §3.1) and 2) silver sentences selection ( §3.2). the symptomclassifiers follow a semantic retrieval pipeline to predict every symptom decision. this pipeline searches for semantic similarities over an index of silver sentences for a specific symptom s, denoted as ag s . these silver sentences are considered relevant to s, and each one has a label corresponding to the symptom options 3 , defined as o. formally, ag s is the set containing the pairs of the silver sentences ag i and their corresponding label o i for the symptom s, where ag @xmath0, o i )}, 3 throughout the rest of the paper, we will refer to these severity options as the labels of the symptoms. ## experimental settings we evaluate the performance of our methods in the erisk2020 and 2021 collections. in erisk2020, we use 2019 as training data. in erisk2021, we use the 2019 and 2020 collections as training. the competing methods used the same collection splits, while some of them also considered external datasets ( §4.2). in our experiments, we study the two components of our approach: i) the performance of the semantic retrieval pipeline ( §3.1) and ii) the effectiveness of the sentence selection strategies ( §3.2). for this reason, our methods consist of combinations of these components. we consider three hyperparameters: 1) the value k of the number of test queries, q @xmath1, ..., q s k }. ## results and discussion the comparison is based on the use of questionnaire and symptom level metrics ( §4.2). questionnaire level: our approach achieves the best dchr, which considers the percentage of times that the system estimates the severity level of the users correctly. most of our variants outperform all prior work in this metric, with the accum recall-aug dataset correctly estimating at least 50% of the depression levels for both collections. in more detail, it improves 13 and 10 points over the best previous results for erisk2020 and 2021, respectively. a similar phenomenon occurs in the rest of the questionnaire metrics. in the error metric, rmse, our results also show less estimation error in the bdi-ii score. symptom level: although in erisk2020, our ahr figures are close to the best baselines, that is not the case in 2021. ahr computes the ratio of option responses estimated correctly. the explanation is that we tuned the model hyperparameters for the dchr metric since clinicians believe that assessing overall depression levels is more valuable than focusing on specific symptoms @xcite . tuning for ahr may produce worse overall results because the model could be failing to a greater amount in the non-correct answers, resulting in higher overall error. to illustrate that effect, we produced an oracle to obtain the best hyperparameters for each symptom-classifier, maximizing ahr using the accum voting-sbert variant. with this oracle, we achieved an ahr of 41.77 and 37.32 for erisk2020 and 2021, which improves all baselines. however, the oracle obtained worse results in dchr (24.29 and 36.25). this is because tuning each individual symptom-classifier would require much more training data. we may improve the results for some symptoms with enough data but produce predictions with higher errors (e.g., 0 vs 3) for symptoms with few training samples. finally, with respect to the sentence selection strategies, we can observe that using the options descriptions as queries (bm25 and sbert) performs worse than the augmented dataset (aug dataset). this emphasizes the importance of a precise candidate selection. moreover, despite the distribution of depression levels varies in both collections (see ## conclusions we present an effective semantic pipeline to estimate depression severity in individuals from their social media data. we address this challenge as a multi-class classification task, where we distinguish between depression severity levels. the proposed methods base their decisions on the presence of clinical symptoms collected by the bdi-ii questionnaire. with this aim, we introduce two data selection strategies to screen out candidate sentences, both unsupervised and semi-supervised. for the latter, we also propose an annotation schema to obtain relevant training samples. our approaches achieve state-of-the-art performance in two different reddit benchmark collections in terms of measuring the depression level of individuals. additionally, we illustrate how our semantic retrieval pipeline provides strong interpretability of the symptom decisions, highlighting the most relevant sentences by semantic similarities.