KADO @ LT - EDI - ACL 2022: BERT -based Ensembles for Detecting Signs of Depression from Social Media Text (2022)

Abstract: Depression is a common and serious mental illness that early detection can improve the patient’s symptoms and make depression easier to treat. This paper mainly introduces the relevant content of the task “Detecting Signs of Depression from Social Media Text at DepSign-LT-EDI@ACL-2022”. The goal of DepSign is to classify the signs of depression into three labels namely “not depressed”, “moderately depressed”, and “severely depressed” based on social media’s posts. In this paper, we propose a predictive ensemble model that utilizes the fine-tuned contextualized word embedding, ALBERT, DistilBERT, RoBERTa, and BERT base model. We show that our model outperforms the baseline models in all considered metrics and achieves an F1 score of 54% and accuracy of 61%, ranking 5th on the leader-board for the DepSign task.

## introduction in our current society, depression is a common but serious mental disorder that involves sadness and lack of interest in all day-to-day activities (ghd; @xcite . depression can negatively affect different aspects of a person's life and can cause a person to suffer severely and function poorly at work, in the family, or in society in general and at its worst, depression can lead to suicide. based on the data provided by world health organization, over 700,000 people die due to suicide every year (who). therefore, early diagnosis of this problem is very important and is a challenge for individual and public health @xcite . because of the complex nature of any mental disorder, it is very difficult to diagnose a patient's mental illness by traditional approaches. however, due to the integration of social media into people's daily lives, evidence has been presented to diagnose depressive symptoms using data provided by users. the study of social media, especially in the field of public health, is rapidly growing. on social media platforms such as facebook, twitter, instagram, and others, people can freely interact with each other and share their thoughts, feelings, ideas, emotions, activities, etc and express themselves through the content they post on these platforms. this leads to a large amount of data that contains valuable information about people's interests, moods, and behaviors. hence many researchers claim that social media analysis is a very helpful source in various contexts especially in mental health understanding @xcite ). ## related work there have been many studies on the prediction of social media mental disorders in which the data were collected directly from user surveys using some well-known questionnaires or from public posts using keywords, related phrases, or regular expression @xcite . several approaches to study mental health have been proposed through the analysis of user behavior on social media. mental health has been studied on different social media platforms such as twitter, instagram, flickr, and facebook. in @xcite , using a deep neural network, an analysis was performed to diagnose depression on the twitter database. @xcite , has also analyzed twitter social media text for public health prediction. binary and ternary classifications are two types of classification problems here. in the first one, sentiments are classified into two polarities or classes: positive and negative @xcite , and in the ternary classification, the sentiments are classified into three classes as positive, negative and neutral @xcite which in this case, more classification error is expected than binary classification. for more accurate classification, the data can be classified into several subclasses. in @xcite for example, having a level of depression from 1-55% is considered as non-depressed and above level of 55% is considered as depressed. the defined subclasses were normal, mild depression, borderline depression, moderate depression, and severe depression. in this article, we specifically focus our efforts on this kind of classification task. our goal is to distinguish between the normal users, users with mild depression, and those with severe depression. ## transfer learning typically, models are trained from scratch with random initialization of network parameters. but in another approach, the model is first pre-trained for a general task and then tuned to a specific task, which allows the model to be trained faster with less training data. originally, transfer learning is known for fine-tuning the deep learning models taught on the imagenet data-set @xcite . recently, several techniques and architectures of transfer learning have been emerged, which has significantly improved most nlp tasks. transfer learning can be used in applications where there is not sufficient training data for that task. the first phase of the transfer learning strategy is generally referred to as semi-supervised training in which the network is first trained as a language model on a comprehensive and large data set and then followed by supervised training that is trained by the desired labeled training data set. ## methodology in this work, an ensembling strategy was used to fuse the results of several bert models. given that, each of these pre-trained models has its strengths and weaknesses as different classification methods. as a result, ensembling them can improve the result. each constituent model is trained on a pretty same training data and the same loss function is used for parameter estimation of each model. our experiments showed that each of these models makes different errors. so, for this problem, we have used the majority voting mechanism to make the final prediction to use the strength of each model (see in figure 2 ). before fine-tuning each of the pretrained models, the proper number of epochs must be known. using a validation data-set that is held back from training, we identify overfitting by looking at validation metrics like loss and accuracy and define correct number of epochs for training each model. whenever the loss value in the validation set increases, it means that network training should be stopped by this number of epochs. from then on, given the data-set for this task, we train the model and tune it to the predefined number of epochs to perform well on unseen data points.