IJS @ LT - EDI : Ensemble Approaches to Detect Signs of Depression from Social Media Text (2023)

Abstract: This paper presents our ensembling solutions for detecting signs of depression in social media text, as part of the Shared Task at LT-EDI@RANLP 2023. By leveraging social media posts in English, the task involves the development of a system to accurately classify them as presenting signs of depression of one of three levels: “severe”, “moderate”, and “not depressed”. We verify the hypothesis that combining contextual information from a language model with local domain-specific features can improve the classifier’s performance. We do so by evaluating: (1) two global classifiers (support vector machine and logistic regression); (2) contextual information from language models; and (3) the ensembling results.

## introduction 1 in the diagnostic and statistical manual of mental disorders, fifth edition (dsm-5), depressive disorders (which include major depressive disorder) are defined by "the presence of sad, empty, or irritable mood, accompanied by somatic and cognitive changes that significantly affect the individual's capacity to function" (american psychiatric @xcite . importantly, depression differs from regular feelings and mood changes, as it has an important impact on the lives of those who suffer from it. an extreme example of this is the high comorbidity between major depressive disorder and suicidal behavior @xcite , which is the cause of death of more than 700,000 people every year @xcite . depression is worryingly common: according to the world health organization, around 5 percent of 1 trigger warning: the paper contains mentions of suicidal behavior and ideation. depsign-lt-edi@ranlp-2023 (detecting signs of depression from social media text) @xcite is a shared task hosted by the language technology for equality, diversity, inclusion workshop 3 . its aim is to detect signs of depression from social media posts. more specifically, the goal is to produce a system that, given social media posts in english, classifies them as presenting signs of depression belonging to one of the following classes: "severe", "moderate", or "not depression". our contributions are threefold. we evaluate (1) the standalone contextual information from language models; (2) the machine learning-based classifier with global information; and (3) the ensembling of the two best classifiers. our work brings valuable insights for detecting signs of depression. the paper is organized as follows. section 2 provides an overview of related works. section 3 discusses the dataset used in this research, while section 4 explains the process of developing our solution. section 5 shows the evaluation of the experiments, while the error analysis is presented in section 6. finally, section 7 concludes the paper with our ideas for future work. ## related work aspects such as one's personality, emotional state, ideology, and mental health are shown to be reflected in one's language-not only in the semantics but also in the syntax used @xcite . having depression as the focus opens two main paths: the analysis of which kind of language is used by individuals suffering from depression (approach a), and the detection of depression through language analysis (approach b). furthermore, any of the two can contribute to the other (the first to the second (approach c) and the second to the first (approach d)). the first path showed, for example, that people suffering from depression use first-person singular pronouns more frequently (in spoken language: @xcite ; in written text: @xcite . furthermore, as one would expect, the sentiment is more negative in individuals suffering from depression's language (babu and kanaga, 2022). these and other aspects have been employed as features to detect depression from text data (approach c) @xcite @xcite . depression detection is becoming a trending topic in shared tasks. an example is the depression and ptsd on twitter task organized in the context of the workshop on computational linguistics and clinical psychology 4 in 2015 (coppersmith et al., 4 clpsych.org 2015). the early risk detection on the internet workshop (erisk) has been hosting shared tasks about depression, along with shared tasks about other mental-health-related issues, since 2017 foot_2 . the workshop on language technology for equality, diversity, and inclusion (lt-edi) foot_3 , organized by the association for computational linguistics foot_4 since 2021, has been including the shared task "detecting signs of depression from social media text" since 2022 @xcite . in the occasion of the 2022 edition (lt-edi-acl2022) @xcite , the first and second place went, respectively, to (poświata and perełkiewicz, 2022) and @xcite . the latter employed vader and sentence embeddings from pre-trained models to generate sentiment scores. then, the authors adopted three different methods: a) gradient boosting models, b) pretrained models, and c) contrastive pre-trained models. they finally ensembled the three approaches, obtaining the classifier which ranked second in the competition. the winning approach (poświata and perełkiewicz, 2022) included three main parts. a) in the first, transformer-based language models were fine-tuned on the train set. b) in the second, a corpus based on reddit posts on mental health, depression, and suicide was created. a transformer-based language model was pre-trained on the corpus, resulting in deproberta (poświata and perełkiewicz, 2022). deproberta was then fine-tuned on the train set. c) in the third step, the best between the developed models were ensembled, obtaining the best-performing classifier of last year's depsign-lt-edi shared task. ## dataset we used the dataset provided by the organizers of the shared task. the dataset contains 7,201 instances for training, 3,245 instances for validation, and 499 for evaluation. each sample is composed of three columns: pid, text, and label. ## methods our proposed solution involves four main steps (global classifiers, contextual classifiers, ensembling, and post-processing), as presented in the following subsections. by global classifiers, we mean classifiers that take non-contextual features into account, i.e., features that apply to the entire document, without considering the relationships between words or phrases. contrarily, contextual classifiers do take such relationships into account. ## results ## error analysis while previous studies @xcite showed the final ensemble to be the best-performing approach, this is not the case in our study. in fact, roberta shows an overall higher performance than any other contextual classifier, any global classifier, and any ensemble. one factor contributing to the fact that the best-performing ensemble, roberta + svm, has such a low performance, is that the class imbalance was not addressed properly when training the global classifiers. besides, we noticed inconsistency among the training, development, and test sets. while the training and development sets contain only latin alphabet characters, the test set contains special non-latin characters (i.e., chinese characters) as shown in ## conclusion in this paper, we presented a framework to detect signs of depression from social media text, as proposed by the lt-edi shared task. our procedure involved 4-steps: (1) the extraction of global information from language models, (2) the extraction of local information from svm and lg classifiers, (3) average ensembling, and (4) post-processing. the results demonstrate the effectiveness of our framework. our bert-based model ranked 7 th /31 while our roberta model could have achieved the 2 nd /31 in the lt-edi competition. however, our ensemble approaches showed lower performance than our roberta classifier. in our future work, we intend to apply upsampling to imbalanced datasets like the one provided for depsign-lt-edi@ranlp-2023 and improve our feature engineering. furthermore, depression detection could be followed by the development of interventional systems to support change in the context of mental health @xcite @xcite .