Using Psychologically-Informed Priors for Suicide Prediction in the CLP sych 2021 Shared Task (2021)

Abstract: This paper describes our approach to the CLPsych 2021 Shared Task, in which we aimed to predict suicide attempts based on Twitter feed data. We addressed this challenge by emphasizing reliance on prior domain knowledge. We engineered novel theory-driven features, and integrated prior knowledge with empirical evidence in a principled manner using Bayesian modeling. While this theory-guided approach increases bias and lowers accuracy on the training set, it was successful in preventing over-fitting. The models provided reasonable classification accuracy on unseen test data (0.68<=AUC<= 0.84). Our approach may be particularly useful in prediction tasks trained on a relatively small data set.

## introduction suicide is a troubling public health issue @xcite , with an estimated prevalence of over 800,000 cases per year worldwide @xcite . suicide rates have been climbing steadily over the past two decades @xcite @xcite , especially in high-income countries @xcite . research has identified many risk factors linked to suicide @xcite , and suicide attempts @xcite . despite these advances, directing these insights into real-life risk identification and suicide prevention remains challenging @xcite . early identification is crucial, as direct, brief, and acute interventions are helpful in preventing suicide attempts @xcite . for the sake of early detection, there are increasing attempts to try and find warning signs in publicly-available social media data. as part of this effort, the 2021 computational linguistics and clinical psychology workshop (clpysch), have provided access to de-identified twitter feeds of individuals who have made suicide attempts (as well as others who have not), with the task of predicting suicide attempts based on tweets up to 30 days (subtask 1) or 182 days (subtask 2) before such attempts. machine-learning algorithms and natural language processing ("nlp") methods have proven highly useful on many prediction problems. current approaches typically rely on inductive algorithms that learn regularities in the data. when data are noisy (as is the case in human behavior), the ability to generalize predictions often depends on the size of the training set. given the sensitive nature of suicide-related data, labeled data on this matter are scarce. this relative scarcity of training examples (e.g., 114/164 individuals in the current task) presents a difficult prediction problem, and increased risk of model over-fitting. in light of the unique properties of this problem, we reasoned that an emphasis on domain knowledge (rather than on algorithmic solution) is warranted, and may help reduce over-fitting. therefore, we adopted the following principles for the prediction task: 1. we used logistic regression rather than potentially more complex models that are often more prone to over-fitting (e.g., dnn, svm, rf). 2. we engineered and evaluated many theory-driven features, based on our domain expertise in psychology (e.g., @xcite . 3. we integrated prior knowledge and the empirical evidence in a principled manner. using bayesian modeling, we incorporated empirical priors from past findings in psychology literature. when we lacked specific priors for a feature of interest, we regularized our parameters using general, domain-level empirical priors (van zwet and gelman, 2020), derived from a meta-analysis of replication studies in psychology @xcite . ## methodology participants in the shared task were given a training set which consisted of 2485 tweets from 114 individuals, 57 having attempted suicide and 57 controls, in the 30-day set, and 15928 tweets from 164 individuals, 82 in each group, in the 182-day set. ## discussion we trained simple classification models, based on psychological features, to determine which individuals may attempt suicide. we used psychologicallyinformed and weakly-informed bayesian models as well as regularized regression models. our models yielded moderately successful predictions on subtask 1, and considerably better predictions on subtask 2 (0.791 ≤ au @xmath0.844, comparable to cohen's d of 1.145 -1.430). in this task, the informed bayesian model (m1) was more successful than the weakly-informed (m2). the data-driven regularized regression models (m3) were slightly less accurate in subtask 1 than the informed model (m1), and slightly more accurate in subtask 2, perhaps due to the fact that subtask 2 included more data than subtask 1. in addition, in both tasks the bayesian models (m1, m2) were particularly successful in avoiding false positive prediction outcomes. admittedly, in the case of suicide detection, it may be prudent to "err on the side of caution", to avoid missing patients in need of care. however, language-based screening on social media tends to be targeted more for broad risk-detection @xcite . in the case of early risk detection it may also be valid to avoid false alarms in order to reduce unwarranted alarm, especially given the potential for suicidal suggestibility. our theory-driven features, as well as the informed bayesian models, were reliant on domain knowledge to help overcome the problem posed by working with small data sets. indeed, incorporating knowledge gained from previous research seemed to have aided in forming a generalized model that did not exhibit over-fitting. another benefit of this approach lies in model interpretability and in its conduciveness to cumulative scientific discovery. we relied on prior empirical findings, and produced updated empirical priors-in light of the task datawhich are simple to interpret and share with others (refer to the majority of previous work in suicide prediction was done by using proxies to suicidal behavior such as clinical risk assessment and suicidal ideation, (see @xcite @xcite . thanks to the clpsych workshop, and the access to valuable data directly indicative of suicidal behavior, we were able to present similar prediction accuracies on actual suicide attempts. the findings derived from this data show great promise for the use of nlp in suicide prevention. ## conclusion our current work provides a synthesis between classic scientific and novel data-driven paradigms. future research is needed to further explore how psychological knowledge and data science methods can be combined to aid in the gradual accumulation of scientific knowledge, and produce actionable predictions that may help save lives.