A Risk-Averse Mechanism for Suicidality Assessment on Social Media (2022)

Abstract: Recent studies have shown that social media has increasingly become a platform for users to express suicidal thoughts outside traditional clinical settings. With advances in Natural Language Processing strategies, it is now possible to design automated systems to assess suicide risk. However, such systems may generate uncertain predictions, leading to severe consequences. We hence reformulate suicide risk assessment as a selective prioritized prediction problem over the Columbia Suicide Severity Risk Scale (C-SSRS). We propose SASI, a risk-averse and self-aware transformer-based hierarchical attention classifier, augmented to refrain from making uncertain predictions. We show that SASI is able to refrain from 83% of incorrect predictions on real-world Reddit data. Furthermore, we discuss the qualitative, practical, and ethical aspects of SASI for suicide risk assessment as a human-in-the-loop framework.

## introduction suicide is a global phenomenon responsible for 1.3% of deaths worldwide @xcite . while it is the leading cause of death among 14-35 year olds in the us @xcite , suicide rates have increased by 13% in japan between july to september 2020 @xcite . it hence becomes critical to extend clinical and psychiatric care, which relies heavily on identifying those at risk. while 80% of patients do not undergo clinical treatment, 60% of those who succumbed to suicide denied having suicidal thoughts to mental health experts @xcite . however, studies show eight out of ten people shared suicidal thoughts on social media @xcite . the advent of natural language processing (nlp) shows promise for suicide risk assessment based on online user behavior @xcite figure 1 : end-to-end pipeline for suicide risk assessment. when sasi assesses the posts, it returns the predicted risk level along with a certainty score. with a human-in-the-loop framework, these predictions can be sorted into various risk levels. sasi assigns high priority to uncertain predictions, for an immediate review by mental health experts. @xcite , with automatic risk assessment algorithms outperforming traditional clinical methods @xcite . numerous deep learning methods already exist, which include leveraging suiciderelated word-embeddings @xcite , social graphs @xcite @xcite and historical context @xcite . however, mental health is a safety-critical realm, where technological failure could lead to severe harm to users on social media @xcite . one such case was covered by register (2020), wherein a medical bot suggested a mock patient kill themselves, demonstrating that unintended harmful behavior can emerge from ai systems @xcite . despite the significant power of traditional nlp methods, such models are inherently designed to make a prediction even when not confident. this poses a challenge when working with critical tasks like suicide risk assessment, for which it may be hard to make a prediction due to various reasons such as task hardness or contained ambiguity. such a system may associate a lower risk level to a user who needs urgent help. a resulting delayed response from mental health experts may lead to adverse consequences. we hence need systems that assign high priority to uncertain predictions, for immediate review and response. contributions: we reformulate suicide risk assessment as a prioritized prediction task which factors in uncertainty, and propose sasi: a risk-averse mechanism for suicidality assessment on social media. sasi is risk-averse in the sense that it is self-aware, as it incorporates a selection function to measure uncertainty. based on a set threshold value, sasi refrains from making a prediction when it is uncertain. we show that sasi can act as a tool to efficiently prioritize users who need immediate attention. through a human-in-the-loop framework that involves a domain expert, sasi assigns high priority to uncertain predictions to avoid critical failure (figure 1 ). we demonstrate the effectiveness of sasi using a real-world gold standard reddit dataset. through a series of experiments, we show sasi refrains from making 83% of incorrect predictions. we further demonstrate its effectiveness through a qualitative study and discuss the ethical implications. ## conclusion with a motivation to provide a robust solution to fine-grained suicide risk assessment on social media, we present sasi, a framework that integrates the concept of selective prioritization to existing deep learning based risk-assessment techniques. sasi is self-aware, wherein it refrains from making a prediction when uncertain, and instead assigns high priority to such data samples for immediate review by mental health experts. we demonstrated the effectiveness of sasi through quantitative evaluations on real-world data, wherein sasi avoided high-risk situations by refraining from making 83% of incorrect predictions. through a qualitative analysis, we described how sasi can be used as a part of a human-in-the-loop framework, facilitating efficient responses from mental health experts. we further acknowledge that the studied data may be susceptible to demographic, expert annotator, and medium-specific biases @xcite . while the essence of our work is to aid in the early detection of at-risk users and early intervention, any interventions must be well-thought, failing which may lead to counter-helpful outcomes, such as users moving to fringe platforms, making it harder to provide assistance @xcite . care should be taken to not to create stigma, and interventions must hence be carefully planned by consulting relevant stakeholders, such as clinicians, designers, and researchers @xcite , to maintain social media as a safe space for individuals looking to express themselves @xcite . it is also essential that clinicians and human moderators are not overburdened @xcite . for instance, "alarm fatigue" is when alarms are so excessive, many of which are false positives, that healthcare providers become desensitized from alarms @xcite .