D eep B lues@ LT - EDI - ACL 2022: Depression level detection modelling through domain specific BERT and short text Depression classifiers (2022)

Abstract: We discuss a variety of approaches to build a robust Depression level detection model from longer social media posts (i.e., Reddit Depression forum posts) using a mental health text pre-trained BERT model. Further, we report our experimental results based on a strategy to select excerpts from long text and then fine-tune the BERT model to combat the issue of memory constraints while processing such texts. We show that, with domain specific BERT, we can achieve reasonable accuracy with fixed text size (in this case 200 tokens) for this task. In addition we can use short text classifiers to extract relevant text from the long text and achieve slightly better accuracy, albeit, trading off with the processing time for extracting such excerpts.

## introduction depression has been found to be a major cause behind at least 800,000 deaths committed through suicide each year worldwide foot_0 . moreover, it has been found in earlier research that depressed individuals show help seeking behavior through their social media posts @xcite . so analyzing social media posts for depression detection is an important research area @xcite . in our work, we analyze reddit social media posts to identify whether a particular post exhibits either of three levels of depression, including (1) no depression, (2) moderate depression, and (3) severe depression, as a part of a shared task challenge @xcite . we use a state-of-the-art transformer-based model called bert, which was pre-trained on mental health related social media data. further, we compare our model with two variations of the same, with other models trained on (1) relevant excerpt extracted reddit posts and (2) a subset of depressive sentences in reddit posts calculated with the help of short text classifiers. in the next sections, we elaborate on each of our strategies. ## depression level detection through fine-tuning mental bert (mbert) bert @xcite , which stands for bidirectional encoder representations from transformers, has been found to be very effective in different downstream nlp tasks such as, text classification @xcite and depressive post detection @xcite . here we use a mental health pre-trained bert model, called mental bert (mbert), which was pre-trained on several mental health forums under reddit @xcite . further, we fine-tune this model on the provided training dataset @xcite for this shared task. since fine tuning bert based models on longer text requires significant memory resources, we limit our text data to the first 200 tokens, which covers around 70% of the total samples provided. before feeding input to our model, we convert all texts to lower case, and use an uncased version of the mbert model for fine tuning. we also experiment with further enhancement of our classifier by fine tuning it through a selection of 200 "relevant" tokens from constituent depressive sentences for a post from the training sample, which are longer than 200 tokens and also depression-indicative. in addition, we investigate whether the distribution of constituent depressive sentences in each posts also have some predictive power for this task. ## extracting relevant excerpts for fine-tuning mental bert (re-mbert) to extract relevant excerpts, we use a majority voting classifier (mvc) which is built using four depressive short text or tweet classifiers. three of these classifiers use different pre-trained word and sentence embeddings and represent each sentence through either averaged embedding of all the constituent words of that sentence, or the sentence embedding of the sentence itself. the left classifier uses zero-shot modelling for classifying each sentence for signs of depression. description of these classifiers including the datasets they were trained on, have been previously described @xcite . since we cannot extract more than 200 tokens for each of our posts and, within those 200 tokens, we may not have all the relevant tokens which are important for this task, we plan to extract relevant (or depressive) constituent sentences or excerpts from the posts which have more than 200 tokens and which are labeled as either carrying signs of "moderate" or "severe" depression in the training set. to do this, we parse each post by exploiting punctuation, i.e. ".", "?" and "!" to find its constituent sentences. we then feed those sentences to mvc, where the above mentioned four short text classifiers vote to indicate whether the constituent sentences of a post are depressionindicative or not; we only take a sentence as a representative text for depression if at-least three of those four classifiers agree. we apply the same short text pre-processing as we did while training our short text classifiers to clean each sentences within our posts. in this cleaning process, with the help of a python library named "ekphrasis" @xcite , we re-contract word contractions, replace elongated words in their original form, convert all to lower case and remove nonwords, so our cleaned sentence is mostly regular words separated by spaces. finally after fine-tuning our mbert classifier (see figure 1 ), we infer the labels from the provided test set and use extracted excerpts only for the posts having greater than 200 tokens (see figure 2 ). in summary, in our training set, we only extract excerpts when a post is depression-indicative and longer than 200 tokens. if no excerpts are extracted or the post is less than 200 tokens long or it is not depression-indicative, then we use the cleaned version of the original post and feed it to our mbert classifier. after this procedure is completed, the total posts left beyond 200 tokens were 1667 which is more than a 50% reduction than the original number of posts having more than 200 tokens (i.e., 4018). all the posts from the original training set beyond 200 tokens and with depression indication are now pre-processed so that those now contain only depressive excerpts which is important for our classification. our assumption with the posts with less than or equal 200 posts is that, they have more depressive sentence density than their longer counter-parts. train + dev set samples from shared task samples with token @xmath0, if no excerpts are available then use whole reddit post train set to train mbert on 200 tokens samples indicating moderate/severe depression yes no held-out set test set samples from shared task samples with token @xmath1, if no excerpts are available then use whole reddit post infer from learnt mbert on 200 tokens ## experimental setup we mix the training and development set provided in the shared task data, which is a total of 13,387 samples, and then split it into a training set of size: 12,589, and validation set of size: 128, and held-out set of size: 670 samples. for the mbert classifier we use uncased mental bert foot_1 . we use maximum token @xmath2, number of @xmath3, training and test batch @xmath4. we employ a nvidia-geforce-rtx 3070 gpu with 8 gb of integrated memory and 32gb of ram. for sprop, we use a mlp classifier foot_2 with default settings and max iteration value of 300, during label inference time we take the argmax of the output label probabilities using predict_proba() function. although bert-based modelling takes around 30 minutes for training and testing, excerpt extraction for creating re-mbert takes a number of days to complete. in the next section we report and analyze the performance of our top models, i.e. mbert and re-mbert. ## result analysis we report both label based accuracy for our heldout set over all samples (see tables 1 and 2 ) and overall accuracy scores (i.e., avg. precision, recall, weighted-f1 and macro-f1 across all labels over all samples, see ## conclusion we have described a few strategies for the depression level detection shared task from reddit posts. we use state-of-the-art mental health data pre-trained bert model (mbert) and further fine-tune it with the shared task data and achieve 7th position in terms of macro-f1 score and 3rd position in terms of weighted f1 score compared to 30 other participating teams. we also present a strategy (re-mbert) to consider while training mbert in a resource constrained environment through a subset of relevant sentence selection for longer posts. our strategy shows some improve-ments in both training and test set which is stimulating and encouraging.