NLP _ CHRISTINE @ LT - EDI -2023: R o BERT a & D e BERT a Fine-tuning for Detecting Signs of Depression from Social Media Text (2023)

Abstract: The paper describes the system for the 4th Shared task on “Detecting Signs of Depression from Social Media Text” at LT-EDI@RANLP 2023, which aimed to identify signs of depression on English social media texts. The solution comprised data cleaning and pre-processing, the use of additional data, a method to deal with data imbalance as well as fine-tuning of two transformer-based pre-trained language models, RoBERTa-Large and DeBERTa-V3-Large. Four model architectures were developed by leveraging different word embedding pooling methods, namely a RoBERTa-Large bidirectional GRU model using GRU pooling and three DeBERTa models using CLS pooling, mean pooling and max pooling, respectively. Although ensemble learning of DeBERTa’s pooling methods through majority voting was employed for better performance, the RoBERTa bidirectional GRU model managed to receive the 8th place out of 31 submissions with 0.42 Macro-F1 score.

## introduction depression is a severe mental disorder that can significantly impact an individual's thoughts, emotions, behavior, and daily routine. @xcite . this condition is classified into three levels, namely mild, moderate, and severe, based on the number of symptoms present. these symptoms may include feelings of sadness, hopelessness, irritability, guilt, insomnia, fatigue, loss of appetite, and disinterest in activities. mild depression typically manifests with 5-6 symptoms, while moderate depression involves 7-8 symptoms, and severe depression includes 9 or more symptoms, which may include hallucinations, delusions, suicidal thoughts, or even attempts @xcite . depression is a widespread issue, affecting approximately 280 million individuals worldwide @xcite . as social media continues to serve as a platform for individuals to express their emotions @xcite , the identification of symptoms of depression through automated means holds the potential for prompt intervention, psychological aid, and the avoidance of adverse outcomes. the 4 th shared task on detecting signs of depression from social media text at lt-edi@ranlp 2023 @xcite challenged participants to develop text classification systems that can classify english social media posts into three classes, namely not depression, moderate and severe depression. this paper presents the system developed for this competition, with the code available on the provided github link. 1 the structure of this paper is as follows: firstly, section 2 presents a discussion of the previous related work followed by the presentation of the data analysis in section 3, and an overview of the developed system in section 4. in section 5, an outline of the experimental setup is provided, while section 6 presents the results and error analysis. finally, the paper concludes with section 7, which discusses future work. ## related work previous work on detecting signs of depression from social media text was conducted at lt-edi@ranlp 2022. the majority of participating teams used transformer-based language models, such as bert @xcite , @xcite , while several teams used traditional machine learning methods like logistic regression @xcite , support vector machines, random forest, and xgboost classifiers @xcite . the top-ranking team, opi, experimented with bert, roberta, and xlnet, trained roberta-large from scratch on depression corpora (deproberta), fine-tuned it and created an ensemble model resulting in attaining a 0.583 macro-f1 score @xcite . the nycu twd team, which came in second place by achieving a 0.552 macro-f1 score, experimented with gradient boosting, pre-trained transformer language models, vader, supervised contrastive learning, and ensemble learning @xcite . ## conclusion and future work the lt-edi@ranlp 2023 shared task 4 entailed the development of a system aimed at addressing data imbalance, cleaning, pre-processing, and finetuning pre-trained language models to accurately identify depression in english social media posts. two pre-trained language models, roberta-large and deberta-v3-large, were employed and finetuned for this purpose. among the four pooling methods tested, the roberta-large bidirectional gru model demonstrated the best performance. this model effectively identified posts exhibiting signs of depression, particularly at moderate levels. however, it struggled with detecting nondepressive posts and may occasionally mistake severe depression for moderate depression. to further enhance the models' performance, future efforts should focus on incorporating more non-depressive texts into the training data and experimenting with the multi-layer structure of pretrained transformer models as well as various hyperparameters. overall, this system has the potential to serve as a valuable tool for early detection of depression, enabling prompt intervention and support for individuals who may be experiencing mental health problems.