Findings of the Shared Task on Detecting Signs of Depression from Social Media (2022)

Abstract: Social media is considered as a platform whereusers express themselves. The rise of social me-dia as one of humanity’s most important publiccommunication platforms presents a potentialprospect for early identification and manage-ment of mental illness. Depression is one suchillness that can lead to a variety of emotionaland physical problems. It is necessary to mea-sure the level of depression from the socialmedia text to treat them and to avoid the nega-tive consequences. Detecting levels of depres-sion is a challenging task since it involves themindset of the people which can change period-ically. The aim of the DepSign-LT-EDI@ACL-2022 shared task is to classify the social me-dia text into three levels of depression namely“Not Depressed”, “Moderately Depressed”, and“Severely Depressed”. This overview presentsa description on the task, the data set, method-ologies used and an analysis on the results ofthe submissions. The models that were submit-ted as a part of the shared task had used a va-riety of technologies from traditional machinelearning algorithms to deep learning models. It could be observed from the result that thetransformer based models have outperformedthe other models. Among the 31 teams whohad submitted their results for the shared task,the best macro F1-score of 0.583 was obtainedusing transformer based model.

## introduction according to the world health organization (who) @xcite , depression is a common disorder across the world that has an impact on the affected person's mood and feelings. this mental health disorder affects a large part of society every year with varying symptoms like lack of interest, insomnia, thought of death etc. the symptoms may vary according to the intensity of disorder. if the intensity is too extreme and left untreated, it may lead to serious consequences @xcite . thus, early prediction of depression is inevitable. on the other hand, in this digital era, social media applications have become a great platform to look over one's mood and feelings @xcite . thus, social media data can be used to detect depression. erisk@clef @xcite served as a root for this task, which aims to detect depression from the social media data. in addition to this, many custom data sets @xcite , arose that aims to detect mental illness from various social media platforms like twitter @xcite @xcite , @xcite , @xcite , etc. among these social media platforms, reddit possesses more textual data and thus, postings from reddit were analysed to detect the level of depression. also, many research works were based only on the detection of the presence of depression rather than detecting the level of depression. thus, this shared task aims to detect the level of depression from reddit postings. ## task description depsign-lt-edi@acl-2022 foot_0 aims to detect the signs of depression of a person from their social media postings wherein people share their feelings and emotions. given social media postings in english, the system should detect the signs of depression into three levels of depression namely "not depressed", "moderately depressed" and "severely depressed". ## data description for detecting the level of depression from social media data, postings from reddit were scraped and labeled into three class labels namely "not de-pressed", "moderately depressed", and "severely depressed". the guidelines and details of annotation are explained in @xcite . the data set is a "tab separated valued" data that has been distributed in three splits namely train set, evaluation set and test data. the details of data distribution are shown in ## methodology the number of teams that participated in the shared task was 31 by which we received a total of 68 submissions. the details of methodologies used by the submissions are explained in this section. team name features extraction classifier additional data set used (if any) 1 nycu_twd (wei-yao et al., 2022), vishwaas, ai901@cnu , mucs (asha et al., 2022), depressionone (suman and radhika, 2022), kucst (manex and amann, 2022), kecsaiyans, kec_deepsign_acl2022 word embeddings machine learning classifiers -2 iiserb (tanmay, 2022) cbow, doc2vec machine learning classifiers erisk-clef 3 bfcai, iiserb (tanmay, 2022) tf-idf machine learning classifiers -4 e8@ijs (ilija et al., 2022) -autobot -5 uagd -keras neural network -6 unibuc_nlp glove cnn -7 ga custom word embeddings cnn -8 ssn (adarsh and betina, 2022) -transfer learning -9 ssn, niksss -recurrent neural networks -10 niksss, kecsaiyans, kec_deepsign_acl2022 -long short term memory -11 umuteam (josé antonio and rafael, 2022) linguistic features, sentence embeddings transformers -12 e8@ijs (ilija et al., 2022) combination of textual (sentence-transformers, distilbert, lsa) features and knowledge-graph transformers -13 opi (rafał and michał, 2022) -transformers reddit mental health data set 14 arguably, bert 4ever @xcite , kado @xcite , ssn, ablimet, vishwaas , sclab@cnu, ai901@cnu , beast, mucs @xcite , depressionone @xcite , ssn_mlrg3 @xcite , niksss, scubemsec @xcite , ssn_mlrg1 @xcite , racai -transformers -15 titowak -transformers sentiment140, suicide detection 16 deepblues @xcite , filipn (filip and györgy, 2022) -domain specific transformers - ## evaluation the evaluation was done using all the performance metrics of sklearn. the submitted runs were ranked using macro f1 score, since the data set is unbalanced. the rank list of the teams were tabulated in from the table, it is clear that the system designed by the opi team (roberta large with additional data set) achieved a best macro f1score of 0.583. the best accuracy score of 0.671 was achieved by two teams namely titowak (roberta with additional data set) and racai (xlm-roberta). the best macro precision and recall scores of 0.591 and 0.586 were attained by team beast. regarding the weighted scores of f1, precision and recall, the system designed by team uagd (keras neural network) performed better than the other systems. ## analysis and discussion out of the 31 teams that participated, 27 teams used deep learning models to detect the level of depression. the methodologies of the teams are summarized in ## conclusion depression is a common mental illness that has an impact on a person's mood and feelings which may lead to serious consequences if not noticed and treated at an early stage. thus, detecting depression at an early stage is a very predominant need. in this shared task depsign-lt-edi@acl-2022, the reddit postings were used to detect levels of depression using three labels namely "not depressed", "moderately depressed", and "severely depressed". a total of 31 teams submitted their results and most of the systems were built using transformers and its variants. the systems were evaluated using macro-averaged f1-score. the best performing system of team opi has used an ensemble method of roberta and attained an f1 score of 0.583.