Emotionally-Informed Models for Detecting Moments of Change and Suicide Risk Levels in Longitudinal Social Media Data (2022)

Abstract: In this shared task, we focus on detecting mental health signals in Reddit users’ posts through two main challenges: A) capturing mood changes (anomalies) from the longitudinal set of posts (called timelines), and B) assessing the users’ suicide risk-levels. Our approaches leverage emotion recognition on linguistic content by computing emotion/sentiment scores using pre-trained BERTs on users’ posts and feeding them to machine learning models, including XGBoost, Bi-LSTM, and logistic regression. For Task-A, we detect longitudinal anomalies using a sequence-to-sequence (seq2seq) autoencoder and capture regions of mood deviations. For Task-B, our two models utilize the BERT emotion/sentiment scores. The first computes emotion bandwidths and merges them with n-gram features, and employs logistic regression to detect users’ suicide risk levels. The second model predicts suicide risk on the timeline level using a Bi-LSTM on Task-A results and sentiment scores. Our results outperformed most participating teams and ranked in the top three in Task-A. In Task-B, our methods surpass all others and return the best macro and micro F1 scores.

## introduction tracking and identifying moments of change in a user's social media longitudinal data could be a possible identifier of their mental health deterioration and be especially useful for those with suicidal ideation @xcite . in this 2022 clpsych shared task, the goal is to tackle two challenges. task-a aims to identify mood shifts and gradual mood progressions from users' timelines, where each timeline has a list of longitudinal posts from a close time range. meantime, task-b aims to detect suicide risk levels of the users. we were allowed to provide three submissions for task-a and two for task-b. the second task-b submission was expected to use the results from task-a. the dataset of this shared task is a mixture of three separate datasets: umd from 2019 clpsych @xcite , e-risk with some additional data @xcite , and a new collection called reddit-new @xcite . the dataset has 255 timelines: 204 in training/51 in the unlabeled test set. our team (called wresearch for "women in research") decided to use emotionally-informed features for their ability to capture mood changes. in task-a, we combine a seq2seq autoencoder and machine learning (ml) models to capture moments of change in a user's timeline. meanwhile, in task-b, we were partially influenced by the 2021 clpsych results, which showed that merging longterm posts of a user could capture long-term suicidal ideation @xcite . we used the post-level features extracted in task-a to compute user-level emotionbandwidth features and concatenated them with statistical n-gram features to detect suicidal risk levels. additionally, we experimented with a timelinelevel prediction model using bi-lstm. the success of our results compared to the other teams and the baselines suggest that our emotionallyinformed models are advantageous for dealing with the tasks at hand. ## methods the training set in this challenge includes data on users with three suicide risk levels (severe/moderate/low). a user can have multiple timelines, where a timeline is a chronologically ordered sequence of posts. each post is labeled as is for switches in mood (sudden mood shifts from positive to negative, or vice versa), ie for mood escalations (gradual mood changes from neutral or positive to a higher positive, or neutral, or negative to a higher negative), or o to represent the baseline (neutral) mood @xcite . in the implementations, for machine learning models, scikit-learn (version 1.0.2) @xcite , for deep learning models, pytorch (version 1.11.0+cu102) and keras (version 2.7.0) libraries @xcite are used. ## results in tables 1 , 2 , and 3, we present the test set results of task-a obtained from three different evaluation techniques. each table summarizes the results obtained on the three submissions: seq2seq + one of the selected classifiers (i.e., 1=lr, 2=xgboost, and 3=the ensemble method). the shared task provided two baselines from the mood change study @xcite . the first baseline (b1 in the tables) uses tf-idf features with lr. the second baseline (b2) uses bert trained with talklife website posts, treats each post as an instance (i.e., completely ignoring the timeline sequence), and is trained using the alpha-weighted focal loss. we also include the best (max) and worst (min) values for each metric obtained by competing submissions to allow better readability of the results. we add an asterisk (*) next to the results when the best performance is achieved by our models. ## discussion in comparison to the submissions of other teams that participated in this shared task @xcite , our models achieved the top three ## conclusion in this shared task, we tackled two problems: capturing mood changes from timelines of posts of reddit users and detecting their suicide risk levels. the results reveal that our methods performed the highest macro and micro f1 scores in suicide risklevel detection and performed in the top three in mood-change detection. our models can inspire future research for accurately detecting abrupt mood changes among social media users. these models also might shed light on users' suicide risk levels, thus enabling early mental-health intervention to prevent suicidal events.