NYCU _ TWD @ LT - EDI - ACL 2022: Ensemble Models with VADER and Contrastive Learning for Detecting Signs of Depression from Social Media (2022)

Abstract: This paper presents a state-of-the-art solution to the LT-EDI-ACL 2022 Task 4: Detecting Signs of Depression from Social Media Text . The goal of this task is to detect the severity levels of depression of people from social media posts, where people often share their feelings on a daily basis. To detect the signs of depression, we propose a framework with pre-trained language models using rich information instead of training from scratch, gradient boosting and deep learning models for modeling various aspects, and supervised contrastive learning for the generalization ability. Moreover, ensemble techniques are also employed in consideration of the different advantages of each method. Experiments show that our framework achieves a 2nd prize ranking with a macro F1-score of 0.552, showing the effectiveness and robustness of our approach.

## introduction social media enable people to communicate and acquire information regardless of distance due to the rapid growth of the internet. besides, people can express their emotions about posts, news, and discussions on social media through texts and videos, which has thus attracted researchers who are interested in analyzing the emotional behavior of user comments. for instance, @xcite introduced a speech act classification twitter dataset and presented an attention mechanism to incorporate intra-modal and inter-modal information. audib-ert, which adopts the multimodal nature of the human voice, was proposed to screen depression @xcite . , and @xcite proposed a social-contagion based framework based on meta-learning for early detection of depression. in this challenge hosted by lt-edi 1 , given social media posts in english, the goal is to detect the † equal contributions. 1 https://sites.google.com/view/lt-edi-2022/home signs of depression and classify them into three labels, namely not depression, moderate, and severe. to tackle the shared task, we propose a framework with three methods for modeling the given texts. specifically, the sentence embedding is produced by pre-trained models, and the vad score (positive, neutral, negative, and compound) is generated by vader @xcite . then, our first method utilized sentence embedding and vad scores in gradient boosting models using smote @xcite to mitigate the imbalance issue. the second method used a multi-layer perceptron (mlp) to fine-tune the pre-trained models. in addition, the third method further incorporated vad embedding with mlp to classify the signs of depression. furthermore, the third method adopted supervised contrastive learning @xcite in both sentence embedding and vad embedding to enhance the capability of generalization. afterwards, we used ensemble techniques, which have been used for substantially improving model performance @xcite , to consider the advantage of each method for boosting the performance. we use the dataset provided by @xcite to detect the signs of depression from social media text. the dataset contains 8,891 posts for training, 4,496 posts for validation, and 3,245 posts for evaluation, while each sample is composed of three columns: pid, text, and label. in summary, our main results and observations are described as follows: social media are among the platforms used to express one's emotions. they can therefore be viewed as an environment to study and discover user feelings. recently, there have been several approaches to detecting signs of depression to eliminate the negative impact of emotions. for instance, toto et al. (2021) introduced a framework with transfer learning to the multi-modality of textual context and audio characteristics of the human voice. zogan et al. (2021) proposed depressionnet by summarizing history posts as a summary of the user and applying different modalities to infer user behavior, which motivated us to include vad scores as the additional post feature in this challenge. ## method figure 1 illustrates the pipeline of our framework. given the input text, we first generate sentiment features (i.e., vad scores) by vader and sentence embeddings from pre-trained models. then, we adopt three methods to model various aspects of the text, and apply ensemble techniques for integrating these predictions. specifically, we use an unsupervised sentiment prediction, vader, to assign sentiment scores to each sentence for measuring the sentiment effect of the word. ## conclusion in this paper, we introduce a framework for the detecting signs of depression from social media text challenge which incorporates three different methods, namely gradient boosting models, pretrained models, and contrastive pre-trained models. furthermore, ensemble techniques are adopted to enable our model's ability to integrate the strengths of each model. the experimental results demonstrate the effectiveness of our framework and verify the different capabilities of each method. thus, ensembling three approaches achieves better performance on both the validation set and the testing set, resulting in a second ranking and achieving a competitive performance.