T ech SSN 1 at LT - EDI -2023: Depression Detection and Classification using BERT Model for Social Media Texts (2023)

Abstract: Depression is a severe mental health disorder characterized by persistent feelings of sadness and anxiety, a decline in cognitive functioning resulting in drastic changes in a human’s psychological and physical well-being. However, depression is curable completely when treated at a suitable time and treatment resulting in the rejuvenation of an individual. The objective of this paper is to devise a technique for detecting signs of depression from English social media comments as well as classifying them based on their intensity into severe, moderate, and not depressed categories. The paper illustrates three approaches that are developed when working toward the problem. Of these approaches, the BERT model proved to be the most suitable model with an F1 macro score of 0.407, which gave us the 11th rank overall.

## introduction depression has emerged as a significant mental health issue in recent years, affecting millions of individuals worldwide. simultaneously, the use of social media platforms has skyrocketed, becoming an integral part of people's daily lives. @xcite talks about the clinical causes of depression and the various treatments for it. social media platforms provide individuals with an outlet to express their emotions and share personal experiences. @xcite examines the relationship between depression and stress over time including effects of childhood and lifetime stress exposure. people often turn to these platforms as a means of seeking support, and validation, or simply as an avenue to connect with others who may be going through similar struggles. de @xcite conducted an analysis on various factors such as social engagement, emotion, language, and linguistic styles, as well as mentions of antidepressant medication. the purpose of this analysis was to develop a statistical classifier that can estimate the likelihood of experiencing depression. the primary objective of the depsign-ltedi task @xcite is to identify indications of depression in individuals based on their social media posts, and analyze english-language social media content and classify the signs of depression into three categories, moderate, severe, and not depressed. ## dataset the dataset consists of the posting id, text data, and the corresponding label @xcite . the dataset provides three labels representing the various degrees of depression namely severe, moderate, and no depression. ## depression detection system we have employed three different models in the three test runs , in this respective order: bert: we employed the 'bert-base-uncased' pre-trained model, known as bert (bidirectional encoder representations from transformers), as the foundation for this depression analysis task. to tailor bert to the specific given depression dataset, fine-tuning of the bert model is done through training using the labeled train data. additionally, to ensure compatibility with the model, a label encoder is utilized to convert the target labels into numerical values. word2vec and svc: in this approach, word2vec, a technique that generates word embeddings -distributed numerical representations of word features is used for text-vector representation. these embeddings capture the contextual meaning of words within vocabulary and enable the model to identify semantic relationships among words that share similar meanings. to leverage the power of these word vectors, a classification model support vector classifier (svc) is employed to train on and predict using these word embeddings. tfidf-linearsvc: linearsvc is a machine learning algorithm implemented in python's scikitlearn library, based on the support vector machine (svm) algorithm. linearsvc aims to find a linear decision boundary that maximizes the margin between different classes. the model learns a set of weights for each feature and combines them linearly to make predictions. to vectorize the given texts, tfidfvectorizer is used, which converts text into numerical feature vectors. subsequently, lin-earsvc model is fitted to these vectors. ## methodology the codes utilized in this research paper are available within the github repository github repository. this repository contains a collection of jupyter notebook files that correspond to the methodologies and techniques detailed in the paper. the methodology employed in this paper consists of three approaches. the first method makes use of the bidirectional encoder representations from transformers (bert) "bert-base-uncased" pre-trained model. the bert model was tailored to the dataset provided. the dataset was tokenized, and converted into the input features with the assistance of the bert tokenizer. no pre-processing steps were undertaken in this model. following this, the input features and labels were converted to tensors, and the tensordataset was created. the model was trained for three epochs using the optimizer adamw, which has a learning rate of 2e-5. finally, cross-validation was performed with the help of the development data for fine tuning the trained model and tested on the test data. figure 1 shows the work flow of the system. the second approach involved the usage of word2vec and support vector classification (svc). word embedding is a method in which words are converted into an arithmetic representation termed a vector and each word in a sentence can be represented through this vector. these vecparameters score number of epochs 1 batch size 16 learning rate 2e-5 maximum sequence length 512 figure 2 shows the working of this system. the linearsvc and tf-idf vectorizer are used in the final strategy. similar to the word2vec model, the term frequency-inverse document frequency text vectorizer combines the concepts of term frequency (tf) and document frequency (df). no pre-processing steps were undertaken in this model. the term frequency is an indication of the frequency count of words in a document and gives an idea about the importance of a given word. the quantity of documents that use a particular term is known as the document frequency. linearsvc is a machine learning algorithm implemented in python's scikit-learn library, based on the support vector machine (svm) algorithm that aims to find a linear decision boundary that maximizes the margin between different classes. figure 3 depicts the steps in developing this model. ## conclusion in conclusion, an analysis was experimented with the provided social media texts using three models: bert, word2vec with svc, and tf-idf with linearsvc. among these models, bert demonstrated the highest accuracy in predicting the target labels. it is worth noting that accurately analyzing and predicting signs of depression is challenging for any model, as it may struggle to comprehend the deeper layers of a sentence and grasp the nuanced tone of the text. improving the accuracy of the model can be achieved by ensuring a more balanced distribution of the dataset, where the number of cases for all three target labels is nearly equal. this would help prevent biases and provide a more comprehensive understanding of different degrees of depression. moving forward, our aim is to advance our knowledge in the field of natural language processing (nlp) by exploring various models and implementing them in different use cases to analyze signs of depression from social media texts effectively.in addition,we have planned to work on the various variants of bert models for better understanding, such as distilbert. by undertaking such endeavors, we can enhance our understanding and contribute to the development of more robust and accurate nlp techniques in detecting mental health conditions. models with vader and contrastive learning for detecting signs of depression from social media. in proceedings of the second workshop on language technology for equality, diversity and inclusion, pages 136-139, 2022.