IIITDWD @ LT - EDI -2023 Unveiling Depression: Using pre-trained language models for Harnessing Domain-Specific Features and Context Information (2023)

Abstract: Depression has become a common health problem impacting millions of individuals globally. Workplace stress and an unhealthy lifestyle have increased in recent years, leading to an increase in the number of people experiencing depressive symptoms. The spread of the epidemic has further exacerbated the problem. Early detection and precise prediction of depression are critical for early intervention and support for individuals at risk. However, due to the social stigma associated with the illness, many people are afraid to consult healthcare specialists, making early detection practically impossible. As a result, alternative strategies for depression prediction are being investigated, one of which is analyzing users’ social media posting behaviour. The organizers of LT-EDI@RANLP carried out a shared Task to encourage research in this area. Our team participated in the shared task and secured 21st rank with a macro F1 score 0f 0.36. This article provides a summary of the model presented in the shared task.

## introduction depression is a common mental health illness affecting millions worldwide, causing significant suffering and even terrible outcomes such as suicide. despite its frequency and negative impact, depression frequently remains unrecognized and untreated, particularly among young adults. it is essential to understand the wide-ranging harmful effects of this invisible killer to address the worldwide mental health crisis. over 280 million people worldwide suffer from depression, and the number is rising, according to world health organisation (who) foot_0 . the effects of depression are disastrous for both mental and physical health. it limits a person's ability to succeed in life, including work, relationships, and personal fulfilment. additionally, depression ranks as the second leading cause of teenage mortality, highlighting the urgent need for improved detection and treatment strategies foot_1 . traditional diagnostic procedures, which rely on patient self-reports, remarks from family or friends, and mental state examinations, frequently encounter major problems. there are many people who are depressed who do not receive the appropriate treatment because of underdiagnosis, undertreatment, cultural stigma, and inaccurate assessments. the rise of social media platforms like facebook, twitter, and whatsapp in recent years has provided new avenues for understanding mental health conditions like depression @xcite @xcite . more and more people are using these platforms to express their thoughts, feelings, and everyday experiences, which tells us a lot about their mental health. by leveraging user behaviours, language patterns, and social connections, researchers are exploring the potential of social media as a tool for diagnosing and forecasting depression. the covid-19 epidemic has underlined the necessity of technology-based interventions in mental healthcare. with the adoption of social distancing measures and lockdowns, people have resorted to social media for communication, self-expression, and support. the pandemic's impact on mental health, limited resources, and overworked healthcare systems have highlighted the need for creative and scalable methods for depression detection and treatment. overall, depression remains a substantial global concern, demanding novel techniques for identification and treatment. the combination of social media and behavioural factors offers a promising path for forecasting depression levels. the advancement of technology, including artificial intelligence and machine learning techniques, presents an intriguing potential for leveraging the massive volumes of data available on social media networks. we can use these technologies to create strong, personalized systems that help healthcare professionals, researchers, and individuals identify and treat depression more effectively @xcite @xcite . several methods for handling social media data to determine users' depression conditions have been presented. however, most of these approaches have relied on handcrafted features with shallow machine learning-based models @xcite @xcite . these approaches often require domain expertise to identify features, resulting in biased feature values. furthermore, the handcrafted feature extraction method is laborious and time-consuming, resulting in a longer training time. in addition, many of these models struggle to generalize successfully with new information. researchers have recently tried to alleviate these constraints by employing pre-trained transformer models @xcite @xcite . however, to the best of our knowledge, none of these models have successfully linked domain knowledge with linguistic patterns. to overcome this gap, our proposed work performed experiments with pubmed bert @xcite trained on clinical data, to harness domain knowledge. these studies were carried out as part of the lt-edi@ranlp joint task on detecting signs of depression from social media text. notably, our proposed model finished in the 21st position among the participating teams. the remaining part of paper is arranged as follows: section 2 addresses the recent literature. further model building details are discussed in section 3. finally, section 4 provides insights into the model results. furthermore, its implications on society and future research directions are provided in the last section. ## background study depression detection addresses the interdisciplinary topic of clinical psychology and social media data mining. several studies have been proposed to analyze social media users' behaviour through their content. the results from these studies conclude that individuals with depression tend to use more negative verbal content when interacting with friends or posting on social media. most of these models are conducted using either machine learning-based or deep learning-based methods. this section will provide insights into some selected works from the past. @xcite conducted an experiment involving n-gram features representation, such as tf-idf, linguistic features, and lda topics. the study utilized logistic regression (lr), support vector machine (svm), multi-layer perceptron (mlp), random forest (rf), and adaboost to train the models. the experimental results indicated that svm achieved an accuracy of 81%. however, the highest performance was achieved using the multi-layer perceptron, with an accuracy of 91% for depression and non-depression classification. several studies have also focused on syntactic and semantic features for detecting depressive comments from social media data. liu and shu extracted syntactic and semantic-based features to train several supervised learning models, such as naive bayes (nb), k-nearest neighbors (knn), logistic regression, and support vector machine, as base learners. later, a simple logistic regression model was used to stack the outcomes of the base learners @xcite . ## model building in this section, we outline the model submitted for the shared task of identifying depression levels in social media data. the proposed model comprises three primary steps: data cleaning and preprocessing, feature extraction, and classification. this section will thoroughly explain each of these stages. the architecture of the model is illustrated in fig 1 . ## result and discussion the proposed model was trained to identify class labels such as 'not depression,' 'moderate,' and 'severe.' the comparative results of the model are summarized in the proposed model was tested on both balanced data after smoothing and the original text to assess the impact of text smoothing on its performance. as shown in in evaluating the model performance for the shared task lt-edi @ranlp2023, the organizers utilized the macro-f1 score. among the proposed methods, the combination of pubmed bert with svm on balanced data achieved a higher macro-f1 score and was therefore chosen for submission in the shared task. our proposed model received the 21st rank among the participating teams, with a macro-f1 score of 0.36 on unseen data. ## conclusion and future enhancements the study presents the model submitted to the lt-edi@ranlp 2023 shared task, which aims to detect signs of depression in a social media text. the proposed model experimented with two approaches: svm and bilstm as classifiers. the study concludes that pubmed bert embeddings combined with svm classifier on a balanced dataset achieve more uniformly distributed weighted f1 scores across all the labels. the proposed model secured the 21st rank in the competition. however, the model's performance could be further improved by developing a more robust algorithm capable of capturing domain-specific information and contextual details from the input text.