Predicting Moments of Mood Changes Overtime from Imbalanced Social Media Data (2022)

Abstract: Social media data have been used in research for many years to understand users’ mental health. In this paper, using user-generated content we aim to achieve two goals: the first is detecting moments of mood change over time using timelines of users from Reddit. The second is predicting the degree of suicide risk as a user-level classification task. We used different approaches to address longitudinal modelling as well as the problem of the severely imbalanced dataset. Using BERT with undersampling techniques performed the best among other LSTM and basic random forests models for the first task. For the second task, extracting some features related to suicide from posts’ text contributed to the overall performance improvement. Specifically, a number of suicide-related words in a post as a feature improved the accuracy by 17%.

## introduction social media platforms are widely used nowadays. the nature of these platforms allows people to be open and express themselves and share daily details about their activities and thoughts. as a result, social media data have been used in research for many years to understand users' mental health. a number of techniques have been proposed in the recent literature on monitoring mental health state over time. for example, a study by @xcite was conducted to investigate suicidal risks from twitter. the authors used a time-aware transformer model with a pre-collected data set for suicide ideation and applied their model on 34,306 tweets from 32,558 users. the main goal was to classify if the person is at risk based on their sequence of tweets. another study was conducted for detecting mood change by @xcite . they proposed a predictive model to determine if a post is associated with a moment of cognitive change. in this paper, we explain our approach to the clpsych @xcite shared task, which consists of two subtasks, as follows: subtask a: subtask a tries to capture those moments when a user's mood deviates from their baseline mood based on a user's postings throughout a specific time period -this is a post-level sequential classification task. the full task description can be found in @xcite . subtask b: a user-level classification task on predicting the degree of suicide risk. an individual/user is considered to belong to one of four categories: no, low, medium or severe risk based on their posts on reddit "r/suicidewatch". the full task description can be found in @xcite . ## dataset data used for this shared task was pulled from reddit. this well-known social media platform contains communities known as "subreddits", each of which covers a different topic. for subtask a, subreddits relating to mental health were used in this task. a total of 186 users were included in this study, with 256 timelines and a total of 6205 posts. the average time span for each user is 2 months. data annotation was carried out by four annotators with multiple training rounds and mediation. timelines were manually checked to ensure that they contain content indicating mood. each post was labelled with one of three labels: is for switches i.e (mood shifts from positive to negative, or vice versa), ie for escalations -gradual mood progression from negative (positive) to very negative (very positive), and 0 for no change. subtask a data can be found on @xcite @xcite . the data for this task was severely imbalanced. the values distribution were 79% for 0, 15% for ie, and only 6% labelled as is. for subtask b, four clinical experts annotated the user based on data from the suicidewatch subreddit to one relative suicide risk severity. subtask b data can be found on @xcite .each user was labelled with one of four labels: "none", "low", "moderate", "severe" representing their suicidal risk level. the classes "low" and "none" were merged together to address the class sparsity issue. the resulting class set is composed of the "severe", "moderate" or "low" classes for 127 users with the frequencies of 48%, 43% and 9% respectively. all authors have signed a data user agreement (dua) and non-disclosure agreement (nda) to have access to the dataset. ## methods in this section, we will describe the methods we developed to address these two shared subtasks. ## results and evaluation results from the all models in both tasks on the blind test set are shown in ## discussion for subtask a, as shown in when it comes to all participants in this year's clpsych shared task, our bert model ranked the third best performing model for post-level metrics evaluation. this emphasizes the feasibility and usefulness of the undersampling technique used. for subtask b, compared to baseline models, rf3 was the only model able to predict the class "low". a possible explanation is that using the suicidal words count feature helped in identifying "low" suicidal risk users. on the other hand, results for "moderate" and "severe" classes were less compared to baseline models, this might be because we did not normalize the number of suicidal words to the number of posts per user and thus the model was inflated for users with more posts. it is essential that the limitations of this study are considered in future studies. firstly, the suicidal words list is collected from different sources and from analysis and manual inspection of the dataset. it could be expanded and validated to include more suicide related words. another limitation is that we did not fine-tune any embedding model to our dataset (except for bert). we used general pretrained embeddings such as glove. also, we aimed to try oversampling techniques to address data imbalance but we could not achieve that due to time constraints. ## conclusions and future work in this paper, we presented our system description for clpsych shared task @xcite . the task consists of two subtasks. subtask a aims to detect moments of mood change for posts in a timeline. for this, first we undersample the dataset to address the severe imbalance in dataset by filtering out the posts with positive sentiment irrelevant to mood changes. bert without explicit modelling of the post sequence outperforms other models. subtask b aims to classify a user to correspondent suicide risk-level. for this task, we extracted additional features and performed random forests. the proposed model succeeded in detecting the least represented class. in future, we aim to perform oversampling using gpt-3 to balance the dataset. we also aim to expand the suicidal words list and extract additional features from the text that could enhance obtained results.