Determining a Person’s Suicide Risk by Voting on the Short-Term History of Tweets for the CLP sych 2021 Shared Task (2021)

Abstract: In this shared task, we accept the challenge of constructing models to identify Twitter users who attempted suicide based on their tweets 30 and 182 days before the adverse event’s occurrence. We explore multiple machine learning and deep learning methods to identify a person’s suicide risk based on the short-term history of their tweets. Taking the real-life applicability of the model into account, we make the design choice of classifying on the tweet level. By voting the tweet-level suicide risk scores through an ensemble of classifiers, we predict the suicidal users 30-days before the event with an 81.8% true-positives rate. Meanwhile, the tweet-level voting falls short on the six-month-long data as the number of tweets with weak suicidal ideation levels weakens the overall suicidal signals in the long term.

## introduction suicide is amongst the most pressing public health issues facing today's society, stressing the need for rapid and effective detection tools. as people are increasingly self-expressing their distress on social media, an unprecedented volume of data is currently available to detect a person's suicide risk @xcite @xcite . in this shared task, we aim to construct tools to identify suicidal twitter users (who attempted suicide) based on their tweets collected from spans of 30-days (subtask 1) and six months (subtask 2) before the adverse event's occurrence date @xcite . the small number of users in the labeled collections of subtask 1 (57 suicidal/57 control) and subtask 2 (82 suicidal/82 control) and the scarcity of tweets for some users pose these tasks as small-dataset classification challenges. on that note, @xcite reported high performance with deep learning (dl) methods on these collections after enriching them with additional data (418 suicidal/418 control). when formulating the strategy to attack the challenge, we were motivated by the real-life applicability of the methods. some social media domains already started implementing auto-detection tools to prevent suicide @xcite . these tools continuously monitor the presence of suicide risk in new posts. therefore, we chose to train the models at the tweet level. next, we develop a majority voting scheme over the classified tweets to report an overall suicide risk score for a user. we employ simple machine learning (ml) methods and create an ensemble. we also experiment with dl methods to assess whether complexity would improve the results. since successful ml applications thrive on feature engineering @xcite , we conduct feature selection to evaluate and determine the best feature sets for the models. our experiments suggest that majority voting (mv) over tweet-level classification scores is a viable approach for the short-term prediction of suicide risk. we observe that dl methods require plentiful resources despite the small size of the datasets. simple ml methods with feature selection return satisfactory results, and the performance further improves by the ensemble classifier. we also observe that the mv approach falls short on the six-month-long data regardless of the applied model. yet this limitation provides the invaluable insight that suicidal ideation signals are more significant when the date of the suicidal event is closer, which stresses the need for more complex, noise immune models for longer time-spanning data. in this context, we consider a noise-immune model as a suicidal ideation detection model that is not affected by tweets lacking suicidal ideation. experimentation framework: before designing the experiments, we face a critical choice: should we merge all tweets per user, or should we perform the assessment per tweet and then aggregate the scores? to answer this, we consider a real-life risk assessment system. the system should provide a score every time someone posts a tweet. some social media domains already implement these systems @xcite . hence, we select to train the models to classify tweets, then apply majority voting (mv) per user to compute a risk score based on the tweet scores. our framework is described in figure 1 . experiments with standard ml methods: before ml experiments, we initially explore a simple approach that constructs graphs from training sets and computes how well the given texts match the graphs @xcite . however, tweets proved to be unfit for the method due to low word counts. as most ml methods depend on learning from features, we select n-gram features where @xmath0. for bigrams (@xmath1, we apply a sliding window over concurrent words using the nltk library @xcite . next, we eliminate infrequent n-grams from the training set to reduce uninformative features (occurring @xmath2, ≤10 tweets in 182-days training sets). subsequently, we scale the features by row-normalizing them with the root of the sum of the square (i.e. variation) of the feature values. among the popular ml methods in suicide literature is logistic regression (lr) @xcite @xcite . we select the "liblinear" solver with default settings for being recommended for small datasets @xcite . to cover diverse mathematical frameworks and assumptions, we also include two naive bayes methods (gaussian (gnb) and multinomial (mnb) with default settings) @xcite . we also experiment with k-nearest neighbors with different distance (uniform, weighted) and neighborhood (@xmath3, 5, 8}) settings, but we eliminate it for low within-dataset results. similarly, ensemblelearning methods (adaboost, xgboost, random forest) also return underwhelming performance despite the parameter tuning, and thus, were eliminated. additionally, we evaluate support vector machines (svm) for their popularity in suicide research @xcite @xcite . svm with rbf kernel proves to be successful but requires costly parameter tuning, while linear svm (lsvm) shows success on withindataset evaluations with less cost. consequently, we select lsvm of sklearn (default settings) for the shared task @xcite , which returns only binary classification results. to convert them to probabilities, we apply probability calibration with logistic regression (calibratedclassifiercv). feature selection: following the ml method selections, we evaluate the effect of feature selection on ml performance. to compute feature importance scores, we also use the lr. for each selected number of features, we gather top suicidal and control features. next, we train and evaluate the ml methods in a leave-one-out (loo) framework using those features. the feature selection results of the selected ml methods for two subtasks are in figure 2 . we select the best ml models from these plots. experiments with ensemble: ensemble classifiers previously showed success in ml challenges @xcite . since every classifier renders predicted probabilities for every data point, we build an ensemble classifier to optimize the results of four selected ml methods (lr, gnb, mnb, lsvm). we adopt a weighting ensemble method where the weight of each classifier is set proportional to its performance @xcite . we call this method weighted ensemble (wens). experiments with dl: to measure whether results would improve with complexity, we also evaluate shallow dl methods. we use the pre-trained transformer model bert-base-uncased @xcite to catch the linguistics features of the tweets. the embeddings are then fed to a dl recurrent units-based architecture to learn text sequence orders. we experiment with two types of recurrent neural networks (rnns): long short term memory (lstm) @xcite , and gated recurrent unit (gru) known for overcoming vanishing and exploding gradient problems faced by vanilla rnns during training @xcite . after assessing various configurations of both architectures, we settle on a multi-layer bi-directional gru with the following characteristics: embedding dimen-@xmath4, number of @xmath5, batch @xmath6. we call this model gru-bert. we include a drop-out to regularise learning and a fully connected layer with a sigmoid activation to produce the classification for each tweet. finally, we include the same majority voting framework to infer the classification on the user level. we use pytorch @xcite and scikit-learn @xcite ) libraries for implementation. ## results before training each classifier, we employ the best performing top features from the figure 2 , where every classifier has its most fitting top features for each subtask. next, we construct a loo crossvalidation framework for within-dataset evaluations. 1 it is important to note that, in each step of the loo, we choose new user ids for evaluation and completely exclude all of their tweets from the training sets to evade ml methods potentially learning the way a person drafts tweets. that means the within-dataset loo results of a subtask are reported for all users of the labeled set. moreover, the labeled datasets have more users than the unlabeled test sets per subtask (e.g. 57 vs. 11 suicidal users in subtask1). ergo, we expect a high magnitudinal difference between the within-dataset and the test results. the within-dataset evaluation results of the selected methods are in ## discussion in subtask 1, the test set results show that feature selection can considerably enhance the performance of ml models compared to the baseline. we also find that the ensemble classifier is comparably better than the baseline in this subtask. meanwhile, though the baseline of clpsych2021 is the same as our lr, our additional mv and feature selection together enable lr to substantially outperform the baseline. these successes of simple ml methods indicate that a collection of tweets from within the 30-days of a suicidal event is good enough to capture the existence of suicidal ideation, which is an important finding for future real-life suicide prevention applications. in contrast to the observations from subtask 1, our test results on subtask 2 are unsatisfactory. yet, they provide the valuable insight that suicidal signals are more significant in the short-term, and older tweets lacking suicidal ideation generate noise. this insight suggests the need to account for a time-domain aspect. to investigate the viability of this claim, we experiment with a simple time-decay coefficient in the mv framework and evaluate it through lr on the test set. we multiply each vote by the coefficient 2 notwithstanding, on both subtasks, the shallow dl methods we experimented with perform poorly. these results could be attributed to overfitting on the small dataset and noise sensitivity for the larger time-spanning dataset. additionally, regardless of the dataset size, these methods proved to be computationally expensive. as within-dataset experiments using simple ml methods outperformed these expensive shallow dl methods, we excluded the latter from the test set evaluation. future work on dl will include deeper, more complex, and noise immune methods that could integrate convolutional neural networks (cnn), deeper lstm or gru layers, and experiments with various word embedding models. if we compare our findings with those in coppersmith et al. (2018) , we observe different results in terms of short-term versus long-term dataset classifications. we attribute these different outcomes to the fact that the original study optimizes the design for detecting trait-level (relevant to risk for any point in time) suicide risk when we endeavor to identify suicidal ideation at the state level (immediate risk presence). this design choice, along with tweet-level classification, enabled our model to recognize suicidal nuances in short-term tweets. meanwhile, we were unable to detect any suicidal ideation through manual inspection (reading and interpreting the tweets) over most of these tweets due to their noisy and ambiguous nature. ## conclusion in this shared task, we investigate various models for identifying suicide risk based on user's tweets. inspired by real-life applications, we focus on assessing suicide risk on the tweet level. experimental results reveal that the ensemble classifier can identify suicidal users from 30-days tweets with a high performance rate, demonstrating the power of majority voting over tweet-level classifications for short-term suicide risk detection. meanwhile, we construe from the underwhelming results on the six-month dataset that these models were more sensitive to the signals relevant to short term risk than those relevant to long term risk. in future work, we will incorporate a temporal aspect to improve the noise immunity of our models, and we will continue experimenting with more complex models.