iicteam@ LT - EDI -2023: Leveraging pre-trained Transformers for Fine-Grained Depression Level Detection in Social Media (2023)

Abstract: Depression is a prevalent mental illness characterized by feelings of sadness and a lack of interest in daily activities. Early detection of depression is crucial to prevent severe consequences, making it essential to observe and treat the condition at its onset. At ACL-2022, the DepSign-LT-EDI project aimed to identify signs of depression in individuals based on their social media posts, where people often share their emotions and feelings. Using social media postings in English, the system categorized depression signs into three labels: “not depressed,” “moderately depressed,” and “severely depressed.” To achieve this, our team has applied MentalRoBERTa, a model trained on big data of mental health. The test results indicated a macro F1-score of 0.439, ranking the fourth in the shared task.

## introduction depression is a significant mental health condition that affects individuals worldwide. it is characterized by persistent feelings of sadness, lack of interest in daily activities, and a range of emotional and physical symptoms. the world health organization (who) estimates that more than 300 million people of all ages suffer from depression, making it a significant public health concern @xcite . early detection and intervention play a crucial role in effectively addressing depression and reducing its impact on individuals' lives @xcite . recognizing the importance of early detection, researchers have turned to artificial intelligence (ai) techniques to develop automated systems for the detection of depression. social media platforms have emerged as a valuable source of data for understanding individuals' mental health, as people often express their thoughts, emotions, and experiences in their online posts. by leveraging the vast amount of user-generated content on social media, researchers aim to detect signs of depression in a timely manner and provide appropriate support (d'alfonso, 2020). in this research, we focus on leveraging pre-trained learning models, specifically mentalroberta transformers, for fine-grained depression detection in social media. pre-trained transformers have shown remarkable success in various natural language processing tasks, and we seek to harness their capabilities to accurately identify and classify depression-related patterns in social media text @xcite . one of the challenges in depression detection is the presence of imbalanced datasets, where instances of non-depressive posts and moderate level of depression are significantly outnumbered by severe depression-related posts. to address this issue, we employ text augmentation techniques to artificially increase the number of depressive instances in the dataset, thereby enhancing the model's ability to learn from the imbalanced data distribution. the primary contribution of our research lies in developing a robust model for fine-grained depression detection by leveraging pre-trained mentalroberta transformers and employing text augmentation techniques to handle imbalanced datasets. by detecting depression at a fine-grained level, we aim to provide more nuanced insights into individuals' mental health states and facilitate targeted interventions and support. in the following sections, we will describe the related work in the field of depression detection from social media and discuss the methodolo-gies and experiments conducted in our research. the results obtained will demonstrate the effectiveness and contributions of our proposed approach for leveraging pre-trained learning in fine-grained depression detection. ultimately the conclusion and future works will be drawn. ## literature surveys depression is a significant mental health issue that affects millions of people worldwide. early detection and intervention are crucial for providing timely support to individuals suffering from depression. with the rise of social media platforms, researchers have begun exploring the potential of leveraging pre-trained learning models for fine-grained depression detection in social media posts. in a recent study, @xcite employed multi-modal data and proposed a novel method that combines topic modeling using transformers and a deep 1d convolutional neural network (cnn) for acoustic feature modeling. the results demonstrated that the deep 1d cnn and transformer models achieved state-of-the-art performance for audio and text modalities, respectively. furthermore, the multi-modal results are comparable to the state-of-the-art depression detection systems. furthermore, (martnez-castano et al., 2020) investigated early detection of signs of selfharm and measuring the severity of the signs of depression. their approach focused on utilizing bert-based classifiers trained specifically for each task. the results demonstrated that this approach yielded excellent performance across various measures. the study suggested that trained bert-based classifiers can accurately identify social media users at risk of selfharming, with a precision rate of up to 91.3 %. recent study also performed depression detection in low-resource language like thai, and found out xlm-roberta based on transformers has performed the best with 79.12% accuracy @xcite . a study by @xcite focused on leveraging the application of temporal deep learning models with a transformer architecture to predict future diagnosis of depression using electronic health record (ehr) data. the model demonstrated improved precision-recall area under the curve (prauc) for depression prediction, achieving a prauc increase from 0.70 to 0.76 compared to the best baseline model. @xcite investigated the detection of signs of depression in social media text using transformer models like distilbert, roberta, and albert. the prediction process involved assigning three labels to the data: moderate, severe, and not depressed. the evaluation of their models revealed macro f1 scores of 0.337, 0.457, and 0.387 for distil-bert, roberta, and albert, respectively. one notable study by @xcite focused on utilizing a hybrid deep learning model called roberta-bilstm to extract features from sequences of depression text. the model combines the strengths of sequence models and transformer models while mitigating the limitations of sequence models. by utilizing the robustly optimized bert approach, the model maps words into a meaningful word embedding space and effectively captures long-distance contextual semantics using the bidirectional long short-term memory model. experimental results demonstrated that this model holds promise for improving the accuracy and robustness of depression detection, aiding in the timely identification and treatment of individuals experiencing depression. another relevant study by @xcite demonstrated that transformer ensembles outperformed individual transformerbased classifiers in detecting depression. this finding underscores the significance of leveraging ensemble models to improve the accuracy and robustness of depression detection from social media posts. by harnessing the power of transfer learning, we can effectively apply knowledge gained from one dataset to enhance the performance on a different dataset, expanding the applicability of our models. these previous studies collectively illustrated the effectiveness and versatility of leveraging pre-trained learning for fine-grained depression detection in social media. by fine-tuning pretrained transformers, researchers have been able to capture intricate linguistic patterns and emotional expressions indicative of depression. this approach holds immense promise for developing accurate and scalable tools for early detection and intervention in individuals at risk of depression. 3 methods the training data has been provided by competition organizers, further divided into training sets and validation sets 80:20 as demonstrated in figure 1 . after splitting dataset, text augmentation techniques have been applied because the shared training data is imbalanced. for the classification, mentalroberta has been trained and fine-tuned on training sets. furthermore, this fine-tuned mentalroberta has been utilized to assess testing performance with test sets. ## results and discussion the training loss is presented at different steps of the training process in figure 3 . at step 500, the training loss is 0.8063, indicating that the model's predictions have a relatively higher deviation from the actual target values. as the training progresses, the training loss decreases, indicating that the model is improving its performance and fitting the training data better. at step 2500, the training loss is 0.5932, showing a further reduction in the loss value. monitoring the training loss helps assess the convergence and performance of the model during training. lower training loss values generally indicate a better fit to the training data. however, it is important to note that the training loss alone may not fully reflect the model's performance on unseen data or in real-world scenarios. evaluation on separate validation or test datasets is necessary to assess the model's generalization ability and overall performance. the training performance results show that the model generated accuracy 69.96 %, f1-score 69.94 %. then we evaluated the model and using the 499 samples of the test set, the results shows that a macro f1-score of 0.439, as the fourth-ranked participant in the shared task. ## conclusion the depsign-lt-edi project focused on the detection of depression signs in individuals based on their social media postings. by utilizing the mentalroberta model trained on mental health data, the model classified the signs of depression into three labels: "not depressed," "moderately depressed," and "severely depressed." the result obtained from the evaluation of the system showcased a macro f1-score of 0.439, positioning the system as the fourthranked participant. another area for future investigation involves incorporating multimodal analysis. by integrating textual analysis with other modalities such as images, videos, and audio, a more holistic understanding of individuals' mental health states can be achieved. this multimodal approach has the potential to improve the accuracy and reliability of depression detection systems. furthermore, there is room for refining the classification system to capture finer levels of depression severity. developing models that can distinguish between different levels of depression, ranging from mild to moderate and severe, would enable a more nuanced understanding of individuals' mental well-being. this, in turn, could facilitate more targeted interventions and personalized support. it is also crucial to consider ethical considerations in future research endeavors. addressing privacy concerns, obtaining informed consent, and mitigating potential biases in depression detection from social media are essential. striving for transparency and interpretability in the developed models while ensuring data protection and respecting individuals' autonomy is of utmost importance. by exploring these future directions, the field of depression detection from social media can continue to advance. this progress would lead to the development of more accurate and effective tools for early intervention, support, and treatment for individuals experiencing depression.