Sentiment-guided Transformer with Severity-aware Contrastive Learning for Depression Detection on Social Media (2023)

Abstract: Early identification of depression is beneficial to public health surveillance and disease treatment. There are many models that mainly treat the detection as a binary classification task, such as detecting whether a user is depressed. However, identifying users’ depression severity levels from posts on social media is more clinically useful for future prevention and treatment. Existing severity detection methods mainly model the semantic information of posts while ignoring the relevant sentiment information, which can reflect the user’s state of mind and could be helpful for severity detection. In addition, they treat all severity levels equally, making the model difficult to distinguish between closely-labeled categories. We propose a sentiment-guided Transformer model, which efficiently fuses social media posts’ semantic information with sentiment information. Furthermore, we also utilize a supervised severity-aware contrastive learning framework to enable the model to better distinguish between different severity levels. The experimental results show that our model achieves superior performance on two public datasets, while further analysis proves the effectiveness of all proposed modules.

## introduction according to the world health organization (who) foot_0 , depression is a common mental disorder and a leading cause of disability worldwide. it is estimated that around 350 million people suffer from depression and over 70% of people do not receive timely treatment @xcite . therefore, if depression is detected at an early stage, it could be useful for disease intervention and treatment. in addition, with social media becoming increasingly popular, people often share their feelings and mental states through social media platforms such as twitter and reddit, making social media posts critical for mental illness detection, including depression detection @xcite @xcite . recent advances in natural language processing (nlp) have played an increasingly essential role in supporting the analysis of user-generated contents from social media, including mental illness surveillance @xcite @xcite . there are many methods leveraging nlp technologies for automated depression detection. however, existing depression detection methods mainly treat detection as a binary classification task due to the limitation of existing annotated datasets. according to the beck's depression inventory (bdi) @xcite , not all cases of depression are the same; depression could be classified as: minimal, mild, moderate and severe (see figure 1 ). for our methods to be translational, detecting the severity level of depression is expected to be more important for further prevention and treatment @xcite . for instance, evidence-based psychological therapies could be appropriate for mild or moderate depression @xcite , while users with severe level might be provided more resource-intensive interventions @xcite . previous psychological studies @xcite have examined the correlation between sentiment and depression, illustrating that sentiment can reflect the user's state of mind and help to identify depressed individuals @xcite . for example, a depressed user may post a comment "today was a really bad day. i had no energy and thought about suicide all day" with a negative sentiment, while non-depressed individuals tend to express a positive or neutral attitude, such as "idk why i'm doing this. i guess i just am. maybe it will help someone else". however, existing sever-i don't know. was this okay? *** i hate him? i really don't k*** what to make of the situation. ity detection methods mainly focus on modeling the semantic information of posts while ignoring the relevant sentiment information. additionally, traditional classification models with cross-entropy loss treat all severity levels equally, making it challenging to distinguish between closely-labeled categories. for example, 'moderate' and 'severe' are semantically closer in severity level than 'mild' and 'severe', making them more difficult to distinguish. to tackle the above challenges, we propose a sentiment-guided transformer model with severity-aware contrastive learning for depression severity detection. we first leverage post encoders mentalroberta and sentilare @xcite to obtain the semantic as well as sentiment hidden features and then design a sentiment-guided transformer model to better fuse these two features. for the second challenge, we implement a supervised severity-aware contrastive learning framework, which helps the model adaptively differentiate the weights between different negative samples based on the label information. we validate our method on two depression severity detection datasets. the experimental results show that our model consistently outperforms other baselines, such as depressiongcn @xcite and mentalroberta @xcite . our main contributions can be summarized as follows: ## related work with the fast-growing numbers of social media users, using user-generated posts to detect depression manually is no longer practical. automated nlp and text mining technologies provide new opportunities for mental health analysis @xcite . early researchers @xcite extracted linguistic features (e.g., linguistic inquiry and word count (liwc), bagof-words) and statistical features (e.g., n-gram, term frequency-inverse document frequency (tf-idf)) from posts, then employed them in traditional machine learning methods such as svm @xcite , logistic regression @xcite and random forest @xcite to detect depression. recently, deep learning-based models have been used for depression detection, allowing models to automatically capture valuable features without feature engineering. some works have shown that cnn @xcite and rnn @xcite , including long short term memory (lstm) @xcite and gated recurrent unit (gru) @xcite are effective. furthermore, plms such as bert @xcite , roberta @xcite , @xcite are also widely utilized for many mental illness detection tasks due to strong context modelling ability. in particular, @xcite trained the domain-specific language model men-talroberta for mental healthcare, which introduces the domain-related knowledge and outperforms basic plms on several mental health datasets. the depression severity identification task is becoming increasingly attractive due to the practical clinical implications, so some studies have emerged @xcite . although the severity level is measured on an ordinal scale, current methods treat it as a traditional multi-classification task. @xcite first reformulate depression identification as an ordinal classification problem via ordinal loss and employ graph convolutional neural network (gcn) to capture contextual information, which achieves state-of-the-art performance on depression severity datasets. sentiment analysis is one of the key technologies in nlp, which aims to analyse people's sentiments or opinions expressed in texts (usually classified as positive, negative and neutral). sentiment analysis is widely applied to reviews, social media, newswires, and medical informatics @xcite @xcite . recently, plms-based models have achieved competitive performances in sentiment analysis. for example, @xcite propose sentibert, a variant of bert model that effectively captures compositional sentiment semantics. @xcite present sentilare with a context sentiment mechanism and knowledge integration, which facilitates sentiment understanding. contrastive learning is a recent technique for enhancing the performance of models by learning different representations of contrasting samples. the main purpose is to generate negative pairs using data augmentation and minimise the contrastive loss of the positive pairs. similar methods are transferred to nlp tasks, like consert @xcite for better unsupervised sentence representation learning. supervised contrastive learning can leverage label information for representation learning in a supervised setting @xcite , which improves the ability of the model to differentiate between samples with different labels. recent work also uses supervised contrastive learning for depression detection tasks @xcite , illustrating its effectiveness. ## conclusion in this paper, we propose a depression severity detection model based on sentiment-guided transformer and severity-aware contrastive learning. the sentiment-guided transformer can efficiently fuse semantic and sentiment information from user's posts. then we use a severity-aware contrastive learning framework, which can fully leverage label information for capturing severityspecific features and helping the model distinguish closely-labeled categories. the experimental results show that our model performs better than recent models and achieves superior performance on two publicly available datasets. further analysis determines the effectiveness of all proposed modules.