D eep L earning B rasil@ LT - EDI -2023: Exploring Deep Learning Techniques for Detecting Depression in Social Media Text (2023)

Abstract: In this paper, we delineate the strategy employed by our team, DeepLearningBrasil, which secured us the first place in the shared task DepSign-LT-EDI@RANLP-2023 with the advantage of 2.4%. The task was to classify social media texts into three distinct levels of depression - “not depressed,” “moderately depressed,” and “severely depressed.” Leveraging the power of the RoBERTa and DeBERTa models, we further pre-trained them on a collected Reddit dataset, specifically curated from mental health-related Reddit’s communities (Subreddits), leading to an enhanced understanding of nuanced mental health discourse. To address lengthy textual data, we introduced truncation techniques that retained the essence of the content by focusing on its beginnings and endings. Our model was robust against unbalanced data by incorporating sample weights into the loss. Cross-validation and ensemble techniques were then employed to combine our k-fold trained models, delivering an optimal solution. The accompanying code is made available for transparency and further development.

## introduction depression, a prevalent mental disorder, profoundly affects individuals' mood and emotions, impacting approximately 300 million people worldwide, representing 4.4% of the global population @xcite . early detection of depression plays a vital role in preventing serious consequences and providing timely support. with the rise of social media platforms, people often express their thoughts and experiences, making these platforms potential sources for detecting mental health issues. in light of this, the depsign-lt-edi@ranlp-2023 shared task @xcite was organized to identify signs of depression by analyzing social media posts. building on the success of the previous task in 2022 @xcite , where transformer-based models were used to detect different levels of depression, this paper presents the winning approach developed by deeplearn-ingbrasil for the depsign-lt-edi@ranlp-2023 shared task. our solution combines state-of-the-art transformer models, namely roberta and deberta v3 @xcite , trained on a comprehensive reddit dataset collected from mental health-related subreddits. to address the challenge of lengthy texts, we employed truncation techniques that capture the essential information by focusing on the beginnings and endings of the posts. additionally, we address the issue of imbalanced data distribution by incorporating sample weights into the loss function. through cross-validation and ensemble techniques, we combined multiple models trained on different folds, resulting in an optimized solution. for transparency and further development, the code for our solution is available on our github repository foot_0 . the remaining sections of this paper are organized as follows: section 2 provides an overview of previous research on detecting depression through social media analysis and on employed techniques; section 3 details the task, including its dataset composition and distribution among the different classes; section 4 explains the methodology and techniques employed in our approach; section 5 presents the results of our experiments; section 6 shows the results of our final submission, comparing between different approaches. in section 7, we discuss the results obtained from comparing roberta and deberta models, as well as the effectiveness of ensemble methods and ordinal classification losses. depsign-lt-edi@ranlp-2023 shared task, and the previous edition followed this trend, whose aim is to detect different levels of depression using reddit posts @xcite . the depsign competitions were created based on the erisk -early detection of depression tasks hosted in 2017 and 2018 @xcite . given depressed and non-depressed users, the dataset was composed of a collection of posts for each user (in chronological order); the objective was to detect risk signals as soon as possible @xcite . in text classification benchmarks, transformer models are the current state-of-art, such as bert, roberta, and deberta v3 @xcite @xcite . transformer models leverage contextual linguistic knowledge from massive corpora training in a masked-language task. this initial step called pre-training. then, the pretrained architecture can be finetuned for a specific task, transferring the initial knowledge acquired in the pre-training step. the winner's of depsign-lt-edi@acl-2022 shared task employed various classification techniques, mostly around language models with the transformer architecture. poświata and perełkiewicz, 2022 achieved first place by using techniques such as further pretraining a roberta model on depression corpora and used ensemble techniques by taking the average of the probabilities outputs of multiple models. @xcite achieved second place by employing a weighted ensemble of gradient boosting model, lightgbm and xgboost, and fine-tuned roberta, electra, and deberta v3 models. singh and motlicek, 2022 employed voting ensembles from xlnet, bert, and roberta. another strategy used to deal with unbalanced data used by haque et al., 2021, which aims to detect child depression from the cross-sectional survey, was to use sample class weight on the classification loss. for dealing with long texts, previous work by sun et al., 2019 explored truncation methods on bert models for classifying long articles from imdb and chinese sogou news datasets. they hypothesized that the key information in an article is typically found at the beginning and end. we explore this technique as explained in 5.3. ## task depsign-lt-edi@ranlp-2023 shared task was organized to identify signs of depression by analyzing posts on social networks. task follows the previous edition (depsign-lt-edi@acl-2022). given a posting in english on social media, the task is to detect the signs of depression in that posting, classifying it into three levels of depression: 'not depressed', 'moderately depressed', and 'severely depressed'. the main evaluation metric is macro-f1 @xcite . data was gathered from reddit post archives from subreddits where the people discuss mental health, such as r/mental health and r/depression. after the post collection, the data was cleaned, removing non-ascii characters and emoticons and annotated from two domain experts following guidelines @xcite . annotators were oriented to label "not depressed", examples that reflect momentary feelings, ask for medications or ask for help for other people's conditions. secondly, examples labelled as "moderate" reflect change in feelings or shows hope for life or do not indicate the feeling completely immersed in any situations. thirdly, examples considered as "severe depression" were the ones containing more than one disorder conditions or explained about history of suicide attempts @xcite . from the annotation guidelines and the training data examples, they notice that simply expressing as sad does not indicate depression. therefore, the task domain is challenging, since depression is a clinical condition that is not as easy to detect as negative feelings such as sadness, which could easily contribute to false positive examples. the task data also contains internet slang, whereas traditional models are not trained on these terms. we pretrain the models as mentioned in section 5.2 in other to leverage these domain problems. the task data are composed of all the examples from 2022 tasks plus newly annotated samples, summing up to 10,446 examples in training and development sets. following the 2022 task, the label distribution is unbalanced, the most representative class being 'moderate', followed by 'not depression' and the least representative, severe. the imbalance is similar in the training and development splits, as shown in ## methodology during the development of our solution, we employ various techniques to address different challenges associated with the task, including unbalanced data, domain-specific characteristics, and lengthy sentences in the datasets. due to time constraints during the competition, we approach each aspect sequentially and evaluate the performance of different techniques using our evaluation methodology. we adopt an iterative process, where techniques that demonstrated superior performance were retained and carried forward to subsequent experiments. for our solution, we adopt the fine-tuning approach proposed by @xcite to train a bidirectional transformer model for text classification. in line with this approach, we pass the final features of the [cls] token through a pooling layer, and subsequently through a classifier output layer, the default loss function used in our implementation is cross entropy. we opt for a two-step validation methodology. first, to fine-tune our models' hyperparameters, we perform a grid search operation by training on the training set and evaluating with the macro-f1 metric on the development set of the task data. the hyperparameters we tuned included the learning rate, the dropout rate of the task layer, the number of warm-up steps, and weight decay. ## experiments the experiments were performed on a machine using 2 nvidia tesla v100 gpus. we experiment different strategies on transformers in sequentially in order to save time. the base model for our experiments is the roberta large @xcite , unless cited otherwise. ## final submission and results our best performing model, mentalberta, trained with loss sample weights and 50% head + 50% tail truncation, achieved a macro f1 score of 0.618 on the cross-validation data. the models resulting from the experiments were also included in the final submission ensemble. we selected the top 9 models and used each model's 4 folds to compose part of the final ensemble submissions. our three final submissions were: the best results were obtained from the best-model4mean submission, with a score of 0.470. this secured us the 1st place in the depsign-lt-edi@ranlp2023 shared task. it is worth noting that ensembling more than 4 models resulted in a degradation of the final score. ## conclusion in this paper, we have described our approach and techniques that led our team, deeplearning-brasil, to secure the 1st place in the depsign-lt-edi@ranlp2023 shared task. our objective was to classify social media texts into three levels of depression. leveraging the power of roberta and deberta models, we pre-trained them on a curated reddit dataset from mental health-related communities. to address the challenge of lengthy texts, we employed truncation methods that focused on the beginnings and endings of the content. in dealing with unbalanced data, we used techniques such as undersampling, oversampling, loss of sample weights, and data augmentation to mitigate the impact of imbalanced class distributions. ensemble techniques were employed to combine the strengths of multiple models. our experiments showed that the choice of ensemble method varied depending on the fold and dataset characteristics. overall, our winning approach highlights the importance of effective pre-training, addressing unbalanced data, domain adaptation, and strategic ensemble techniques. we will make available pretraining data foot_1 and our mentalberta model foot_2 .