Towards Capturing Changes in Mood and Identifying Suicidality Risk (2022)

Abstract: This paper describes our systems for CLPsych?s 2022 Shared Task. Subtask A involves capturing moments of change in an individual?s mood over time, while Subtask B asked us to identify the suicidality risk of a user. We explore multiple machine learning and deep learning methods for the same, taking real-life applicability into account while considering the design of the architecture. Our team achieved top results in different categories for both subtasks. Task A was evaluated on a post-level (using macro averaged F1) and on a window-based timeline level (using macro-averaged precision and recall). We scored a post-level F1 of 0.520 and ranked second with a timeline-level recall of 0.646. Task B was a user-level task where we also came in second with a micro F1 of 0.520 and scored third place on the leaderboard with a macro F1 of 0.380.

## introduction globally, close to 800,000 people die by suicide each year @xcite . suicide is the fourth leading cause of death among @xcite . though suicide is such a dire issue, a myriad of obstacles such as social stigma, apprehensions about privacy, financial concerns, etc., prevent many from seeking professional help. over the last couple of years, there has been an influx of suicide and mental health posts on social media, especially from the young users -social media's primary consumers. anonymous social media platforms such as mental health blogs or reddit forums have become increasingly popular as they can share their personal stories without judgment. people who face similar issues can share their experiences, give advice, motivate and persuade them to seek counsel from professionals. therefore, social media has become a valuable source of linguistic cues for work to identify mental health problems from textual data @xcite @xcite . a challenge in the area of mental illness detection and suicide risk identification on social media is the importance of focusing on the individual and detecting the critical point where intervention is necessary from a batch of posts. this shared task @xcite breaks up the problem into two problem statements: subtask a: given a user's posts over a certain period in time, this task aims to capture those subperiods during which a user's mood deviates from their baseline mood. this is defined as a postlevel sequential classification task @xcite . it encourages us to identify moments of change in the individual's mood over a timeline of about two months. a moment of change (moc) is defined as a post/sequence of posts in a timeline indicating that the user's behavior or mental health status is shifted. this is represented in the form of the following labels: is (indicating a switch in the user's mood), ie (indicating an escalation of the user's mood) and, o (refers to all other cases). subtask b: this is a user-level classification problem to predict the degree of suicide risk on reddit. a user is considered to belong to one of four categories: no risk (or "none"), low, moderate, and severe risk based on their posts on r/suicidewatch @xcite . we present several approaches to tackle both subtasks, keeping in mind real-life application and the temporal aspect of this problem. in the first subtask, we observed that the post-level classification would be influenced by its context. hence, due to the longitudinal nature of the task, we use a transformer-based lstm architecture. post-level representations are generated using sentence transformer models and passed through an lstm layer to consider historical context before developing the final output label. our second task considers the need for detection mechanisms to continuously monitor suicide risk with the introduction of new posts to a user's history. we, therefore, first evaluate on a post-level using finetuned transformer models. we then adopt a majority voting strategy to assign the final label to the user. our models outperform the baseline and rank in the top 3 submitted models for both subtasks across various categories. ## data the dataset contains 255 timelines taken from users who have posted on mental health-related subreddits and /r/suicidewatch. each timeline consists of 10 to 122 posts each. the data given for this task is taken from 3 separate datasets. the e-risk dataset @xcite is primarily used for subtask a, while the reddit datasets, such as the umd dataset, are used for both subtasks @xcite . the dataset was split into a train and test dataset, with the training dataset having 149 users, 204 timelines, and 5143 posts, and the test dataset having 36 users, 51 timelines, and 1052 posts. of the 149 users in the training dataset, 61 were labeled as 'severe', 55 as 'moderate', 11 as 'low', and 22 remained unlabelled. ## baseline experiments we experiment with various popular machine learning, text classification algorithms on a post-level. majority voting is then applied for the task b experiments to generate the final label. count vectors and tf-idf vectors for different levels of input tokens (words, n-grams, etc.) served as the primary features for most of our baseline models. we used scikit-learn @xcite and keras @xcite libraries to develop and evaluate the models. ## results and analysis the comparison between the baseline models and our main models can be found in results on the unseen test set for our submitted models can be found in our moc-lstm model scores a post-level f1 of 0.520 and ranks second with a timeline-level recall of 0.646. our finetuned-roberta model ranks second with a micro f1 of 0.520 and scored third place with a macro f1 of 0.380. ## conclusion in this shared task, we have worked towards detecting moments of change and the suicidality risk of a user based on their post history. our moc-lstm model allows us to determine post-level information and timeline level classification, enabling us to better understand mental health by identifying the specific moments of change in emotions. the finetuned roberta model works to identify atrisk users based on their post timeline to better care for them. in the future, we plan to consider multi-modal approaches for different social media platforms. this helps give a better picture of the user's mental health since people use different media for different purposes. we also plan to build more efficient models that work on longer timelines to provide these warnings in a real-time platformagnostic manner and help identify at-risk users.