Weakly-Supervised Methods for Suicide Risk Assessment: Role of Related Domains (2021)

Abstract: Social media has become a valuable resource for the study of suicidal ideation and the assessment of suicide risk. Among social media platforms, Reddit has emerged as the most promising one due to its anonymity and its focus on topic-based communities (subreddits) that can be indicative of someone’s state of mind or interest regarding mental health disorders such as r/SuicideWatch, r/Anxiety, r/depression. A challenge for previous work on suicide risk assessment has been the small amount of labeled data. We propose an empirical investigation into several classes of weakly-supervised approaches, and show that using pseudo-labeling based on related issues around mental health (e.g., anxiety, depression) helps improve model performance for suicide risk assessment.

## introduction suicide has been identified as one of the leading causes of deaths and approximately 1.5% of people die by suicide every year @xcite . despite years o clinical research on suicide, researoners have concluded that suicide cannot be predicted using the standard clinical practice of asking patients about their suicidal thoughts @xcite . recently, @xcite and @xcite discuss the opportunities of using social media combined with natural language processing (nlp) techniques to complement traditional clinical records and help in suicide risk analysis and early suicide intervention. to facilitate further research on automatic suicide risk assessment, @xcite proposed a shared task, where they collected user data from r/suicidewatch subreddit and annotated it with user-level suicide risk: no-risk, low-risk, mediumrisk and high-risk. by comparing the results of the participating teams in this shared task, @xcite conclude that one of the major challenges comes from the insufficient data for intermediate suicide risk levels (i.e., low risk and medium risk) rather than extreme risk levels (i.e., no risk and high risk). @xcite find that using a dual bert-lstm-attention model to separately extract information from both suicidewatch and non-suicidewatch posts together with feature engineering that includes emotion features, personality scores, user's anxiety and depression scores are important for model performance. in this paper, instead of feature engineering or complex model architectures, we explore whether weakly supervised methods and data augmentation techniques based on clinical psychology research can help improve model performance. we explore several weakly-supervised methods, and show that a simple approach based on insights from clinical psychology research (o'connor and nock, 2014) obtains the best performance. this model uses pseudo-labeling (pl) on data from the subreddits r/anxiety and r/depression, which are considered important risk factors for suicide. we also present a potential application of our model for studying the suicide risk among people who use drugs, opening the door for using nlp methods to deepen our understanding between opioid use disorder (oud) and mental health. the code for this paper can be found at https://github.com/yangalan123/ wm-sra . ## methods we focus on task a from the clpsych 2019 shared task "predicting the degree of suicide risk in reddit posts" @xcite . the goal of the task is to predict the user-level suicide risk category based on their posts in the r/suicidewatch subreddit. specifically, a user u i is associated with a col-lection of n(i) posts c @xmath0,1 , x i,2 , . . . , x i,n(i) }, where each post x i,k (1 ≤ @xmath1, k) sentences x i,@xmath2,1 , s ik,2 , . . . , s ik,m(i,k) ]. we need to predict y @xmath3, b, c, d} using c i , where a, b, c, d represent no-risk, low-risk, medium-risk and highrisk, respectively. in the original dataset, there are 496 users in the training set and 125 users in the test sets. we further split 100 users from the training set to create the validation set. the sizes for the train/valid/test sets are 746, 173, and 186 respectively. data pre-processing following the advice in @xcite , we replace all human names and urls in the reddit posts with special tokens " person " and " url ", respectively. we also remove punctuation and stop words besides lowercasing. due to the limitation of gpu memory, we split those large posts to be passages with no more than 128 words foot_0 and make sure that the split point is not in the middle of the sentence foot_1 . such passages are treated as separate posts. model architecture our architecture is a bert @xcite model. we also experimented with other state-of-the-art pre-trained language models (plms), including roberta @xcite and xlnet @xcite , but found bert to work the best and thus consider it as our baseline architecture (more details can be found in appendix a). each post x i,k is fed into bert @xcite and we get post embedding e i,@xmath4,k ). then we do simple mean-pooling to obtain the user embedding ## experiments and results we implement our bert model based on huggingface transformer @xcite . due to the limitation of gpu memory, we only use the base version.we split 20% of original training data to be the validation set and fix the split for all models. the model selection is made by early stopping and we train all models for 20 epochs with the batch size 32. for users with too many posts and words, we only sample 100 passages for them. task-adaptive pre-training after applying task-adaptive pre-training on bert, we see small performance gains over bert (i.e., from 0.427 to 0.432). that might be because even we use the whole corpus provided by the shared task, it is still not large enough. multi-view learning word-mask strategy improves over the bert baseline. compared with the adaptive pre-training results on bert, which also do word-level masking but only trained on language modeling, we can see that mvl provides a more efficient way to utilize a small training corpus and bring 3.1% gain on macro-f1. however, all the other mvl approaches hurt the performance when compared to the bert baseline. this might be because the proposed sentence-level perturbation strategy can seriously break the semantics of each post and thus influence the overall performance, and random sampling over sentences hurts most. furthermore, we wanted to test the role of the clinical psychology aspect of our pseudo-labeling approach. does the gain come from the meaningful domains (anxiety and depression) or just by adding additional data? to answer this, we use additional data provided by task c of the shared task that contains posts from random subreddits (e.g., sports). we do two experiments: 1) assign low-risk to all such users and 2) assign the gold labels provided by the task via crowdsourcing. we add the same size as for the other pseudo-label experiment (8% of training data). the results (exp 10 & 11 in ## error analysis in this section, we take a closer look at the prediction results of our best model (clinical psychol-ogy inspired pseudo labeling using r/depression as medium risk) by looking at the confusion matrix and sampled error cases. we plot the confusion matrices for the baseline model (exp 1 in we can also see that the main problem of our best model, is still the confusion between "b" (low-risk) and "c" (medium-risk). in addition, the problem of wrongly predicting the examples belonging to intermediate classes to high-risk ones still exists. by manual investigation, we find that both problems require expertise in mental health to make the subtle distinctions. for example, the following text comes from a low-risk example foot_4 that is wrongly predicted as high-risk by our best model: " sadness has taken me. . . i am sad , lonely , and i have no interest in living anymore. . . i didnt want to die. . . my mind is diseased , unable to take happiness. . . i have no interest in forming any more. . . . i dont think ill do it. . . " it can be seen that there are many negative or even desperate expressions (marked as red) in this examples, mixed with some short signals (marked as blue) possibly indicating a person considered at low-risk. the model can be fooled by the massive negative expressions and make the wrong predictions if the model is not aware of the true intent of the person. therefore, reliable intent identification that could consider user posts across time and other information would be a powerful tool to help the model prevent mistakes like this. ## application: predicting suicide risk of people who use drugs in order to further verify the effectiveness of our model in real-world applications, we create a simulation scenario: we apply our best model (exp 7) over the data that is collected for 612 users who post on both r/opiates and r/suicidewatch. r/opiates is a subreddit where people discuss topics around opioid usage (e.g., drug doses, withdrawal anguish, daily experiences, harm reduction). this community members could often be at a high suicide risk @xcite . we apply our model over their 1, 176 posts on r/suicidewatch and find that our model predicts that 15.52% of them are no-risk, while 84.48% of them are of low-risk, medium-risk and high-risk. the results on sampled 2, 863 r/opiate posts are 30.56% for no-risk and 69.44% for at least some risk. the predicted outputs are highly aligned with reported results using crowdsourcing annotation of suicidal or not-suicidal by @xcite and show the effectiveness of our model in this simulated scenario. 6 we hope this will open the door of using nlp methods to investigate the link between suicidal ideation and fatal overdoses among people who use drugs. ## conclusions we investigated a series of weakly-supervised methods and find that pseudo-labeling on data related to risk factors for suicide (depression, anxiety) can help improve model performance. this provides an alternative way to use theoretically-grounded models (e.g., compared to feature engineering). we also show a potential use case of this work for understanding suicidal ideation among users who use drugs (e.g., opiates).