OPI @ LT - EDI - ACL 2022: Detecting Signs of Depression from Social Media Text using R o BERT a Pre-trained Language Models (2022)

Abstract: This paper presents our winning solution for the Shared Task on Detecting Signs of Depression from Social Media Text at LT-EDI-ACL2022. The task was to create a system that, given social media posts in English, should detect the level of depression as ‘not depressed’, ‘moderately depressed’ or ‘severely depressed’. We based our solution on transformer-based language models. We fine-tuned selected models: BERT, RoBERTa, XLNet, of which the best results were obtained for RoBERTa. Then, using the prepared corpus, we trained our own language model called DepRoBERTa (RoBERTa for Depression Detection). Fine-tuning of this model improved the results. The third solution was to use the ensemble averaging, which turned out to be the best solution. It achieved a macro-averaged F1-score of 0.583. The source code of prepared solution is available at https://github.com/rafalposwiata/depression-detection-lt-edi-2022 .

## introduction depression (major depressive disorder) is a common and serious medical illness that, according to world health organization (who), already affects about 322 million people worldwide @xcite . the main symptoms of depression include: feeling sad or having a depressed mood, loss of interest or pleasure, feeling worthless or guilty, insomnia or hypersomnia, thoughts of death and suicidal ideation or suicide attempts @xcite . when diagnosed and treated quickly, it can greatly improve quality of live and in some cases even save it. such rapid detection of depression signs is possible, for example, based on the social media posts of the individual @xcite . following this assumption, @xcite organized at lt-edi-acl2022 the shared task on detecting signs of depression from social media text. the task was to create a system that, given social media posts in english, should classify the level of depression as 'not depressed', 'moderately depressed' or 'severely depressed'. in this paper we present our solution for this competition. the paper is organized as follows. section 2 describes related work with particular emphasis on issues of depression detection in social media. section 3 presents the dataset and its modification. the process of developing our solution is explained in section 4. the next section shows performed experiments, the results, along with the error analysis. finally, section 6 concludes this paper. ## related work de @xcite authored one of the first papers on detecting depression based on social media posts. in their work, they collected a group of twitter foot_0 users diagnosed with depression whose one-year posts were used to create a statistical classifier to estimate the risk of depression. @xcite prepared the dataset in a similar way but for japanese users, and then trained a support vector machines (svm) classifier to estimate the presence of active depression. @xcite created a dataset based on reddit foot_1 posts in which users were assigned to one group: depressed or control. then, among other things, they analyzed their posts using the linguistic inquiry and wordcount tool (liwc) @xcite . @xcite also used reddit as a data source and with other datasets they verified how training data can affect the quality of a svm-based model to identify depression. @xcite use different types of approaches to text encoding (the liwc dictionary, latent dirichlet allocation (lda) topics or n-grams) to explore the users' linguistic usage in the depressive posts. arora and arora (2019 @xcite ; @xcite . for text-based classification, the last few years have been primarily a time of deep learning and large pre-trained transformer-based language models @xcite . this kind of solutions achieve state-of-the-art results for numerous classification tasks @xcite @xcite . ## dataset the dataset used in the competition consists of english posts from reddit, where each was annotated with one of the labels: not depression, moderate and severe @xcite . the first label indicates a case where no signs of depression were identified. the other two labels show that symptoms in the post indicate moderate or severe depression respectively. example texts with labels from the dataset are presented in ## our solution we organized the work on our solution into three steps, which will be presented in the following subsections. ## conclusion in this paper, we presented a solution to the shared task on detecting signs of depression from social media text at lt-edi-acl2022. the use of ensemble averaging previously fine-tuned language models proved to be the best. as part of this work, in addition to the models designed for this competition, we also prepared a new pre-train language model, deproberta. in the future it can be used for other depression detection tasks. we plan to pretrain it further on a larger corpus of texts expressing depression, as an extension of this work. the code of our solution and prepared models are available online at https://github.com/rafalposwiata/depressiondetection-lt-edi-2022.