scube MSEC @ LT - EDI - ACL 2022: Detection of Depression using Transformer Models (2022)

Abstract: Social media platforms play a major role in our day-to-day life and are considered as a virtual friend by many users, who use the social media to share their feelings all day. Many a time, the content which is shared by users on social media replicate their internal life. Nowadays people love to share their daily life incidents like happy or unhappy moments and their feelings in social media and it makes them feel complete and it has become a habit for many users. Social media provides a new chance to identify the feelings of a person through their posts. The aim of the shared task is to develop a model in which the system is capable of analyzing the grammatical markers related to onset and permanent symptoms of depression. We as a team participated in the shared task Detecting Signs of Depression from Social Media Text at LT-EDI 2022- ACL 2022 and we have proposed a model which predicts depression from English social media posts using the data set shared for the task. The prediction is done based on the labels Moderate, Severe and Not Depressed. We have implemented this using different transformer models like DistilBERT, RoBERTa and ALBERT by which we were able to achieve a Macro F1 score of 0.337, 0.457 and 0.387 respectively. Our code is publicly available in the github

## introduction in the digital world, the usage of social media has become most common among the people. social media is used without any limits to share happiness, joy, sadness, loneliness and all other personal emotions. the contents shared by the people reflect the mental state of the person and can act as an indicator of their depression level @xcite . depression is a serious mental illness that negatively affects how you feel, the way you think and how you act. fortunately, it is also treatable. depression causes feelings of sadness and/or a loss of interest in activities you once enjoyed. it can lead to a variety of emotional and physical problems and can decrease your ability to function at work and at home. sometimes, social media could be the reason for the depression. it is necessary to measure the level of depression from the social media text to treat them and to avoid the negative consequences. detecting levels of depression is a challenging task since it involves the mindset of the people which can change periodically. our aim is to detect levels of depression with the use of deep learning transformer models to achieve the best results @xcite . the shared task detecting signs of depression from social media text was a part of lt-edi 2022-acl 2022 @xcite . the task is based on english comments. the task was a classification problem based on the labels "not depressed", "moderate" and "severe". for example, "my life gets worse every year : that's what it feels like anyway...." fall under the category moderate, "words can't describe how bad i feel right now : i just want to fall asleep forever." fall under the category severe and "is anybody else hoping the coronavirus shuts everybody down?" fall under the category not depressed. for the shared task, a model based on transformers was first proposed which was trained using the training data set provided for the corresponding task followed by validation of the trained model using the evaluation data set. then the model was tested using the testing data set to predict the category of the text based on which the evaluation of the shared task was done. ## related works identifying depression from social media posts involves detecting whether the user associated with the posts could be identified for depression and this could be represented as a text classification prob- learning technique approaches used limitation traditional approach statistical, data driven, rule based requires specific features like and lexicon based approaches syntactic markers, psycho-linguistic features and temporal dependencies machine learning approach svm, rf, dt, nb, knn, lr requires proper fine tuning of parameters and does not show significant impact on the precision deep learning approach nn, rnn, lstm handling of heterogeneous and transformer models and feature vector representation associated with the performance early detection of different emotions of people including depression from their social media posts had been done using a hybrid model which is a combination of two machine learning algorithms namely support vector machine and naive bayes algorithm @xcite . the performance measures of the model had been analyzed by fine tuning the parameters associated with the algorithms. detection of depression from bengali posts and commentaries had been implemented and evaluated using different machine learning algorithms like support vector machine, random forest, decision tree, k-nearest neighbors, naive bayes (multinomial naive bayes) and logistic regression. the results had shown that the same precision had been achieved by all the algorithms @xcite . social media posts of high school students, college students and working professionals had also been considered in specific for identifying mental health using the above mentioned machine learning algorithms (narayanrao and lalitha surya kumari, 2020). use of deep learning models had been depicted for the prediction of mental disorders such as depression. a multi-task hierarchical neural network with topic attention had been used for identifying health issues from social media posts. bidirectional gated recurrent units had been used to analyze the hierarchical relationship @xcite among documents, sentences and words based on which attention weights are enhanced for words. the posts with unstructured text data that display depression had been identified more effectively by deep learning models than by using supervised learning methods @xcite . the role of sentiment analysis in identifying depression had been shown which had improved the performance of the model by using different deep learning techniques for the process of classification @xcite . better performance had been achieved when the heterogeneous and feature vector representation associated with social media posts had been handled and transformer based models @xcite had been utilized for classification of depression and suicidal posts. depression and associated negative emotions had been identified from sina weibo, using deep learning methods @xcite . ## data set we have used "distilbert-base-cased" model for implementing the classification task of identifying depression from social media text which comprises of 6-layer, 768-hidden layers and also 12-heads, 65m parameters. it is a smaller version than bert which is incredibly less expensive and quicker to train than bert. ## experimental setup we have used the virtual gpu (tesla k80) provided by google colab for implementing different transformer models.the processing time was found to be 5.43 min, 15.46 min, 5.48 min for distilbert, ## error analysis the model roberta had resulted in an f1 score of 0.457 which is low compared to 0.583 which is the f1 score obtained by the topper of the leader board. this shows that more false positive and false negative classification has occurred in our proposed model. the data set provided is highly imbalanced in nature which could also be considered as a reason for the poor performance of the model. the data set could be converted to a balanced data set by using different up-sampling and down-sampling techniques. the classification report generated during the evaluation of the model is shown in ## conclusion as social media platforms play a crucial role in today's world and the posts shared replicate the internal mental state of the user, the task of identifying depression from social media posts have become an important research area. the methodology associated with our submission used three different transformer models to implement the above said task namely distilbert, albert and roberta of which the roberta model had shown a better performance with a f1 score of 0.457. this score is not an optimal value and shows the availability of scope to fine tune the transformer models for improving the performance of the model. the process can be more effectively done when depression markers are identified and the context based informations of the posts are considered while developing models to identify depression from social media texts.