Utilizing Large Language Models to Identify Evidence of Suicidality Risk through Analysis of Emotionally Charged Posts (2024)

Abstract: This paper presents our contribution to the CLPsych 2024 shared task, focusing on the use of open-source large language models (LLMs) for suicide risk assessment through the analysis of social media posts. We achieved first place (out of 15 participating teams) in the task of providing summarized evidence of a user’s suicide risk. Our approach is based on Retrieval Augmented Generation (RAG), where we retrieve the top-k (k=5) posts with the highest emotional charge and provide the level of three different negative emotions (sadness, fear, anger) for each post during the generation phase.

## introduction while healthcare systems are crucial in identifying suicide risk, the limited time available to clinicians often hinders a comprehensive assessment of all risk factors @xcite . expressions of suicidal thoughts are among the most significant warning signs. however, the standard practice of clinicians inquiring about these thoughts has not been reliably effective in predicting and preventing suicide @xcite . it was revealed that the majority of patients who commit suicide had not reported suicidal thoughts to their healthcare providers @xcite . the clpsych 2024 shared task @xcite addresses the significant challenge of generating supporting evidence for clinical assessments, with a specific focus on suicide risk assessment using open-source large language models (llms). this task concentrates on analyzing linguistic content from social media posts to substantiate the assigned suicide risk levels of individuals @xcite . by examining users' posting activities on online forums, the goal is to extract, in an unsupervised manner, evidence within these posts that supports the pre-assigned risk levels. ## related work the clpsych shared tasks 2019 @xcite and 2022 @xcite were mainly focused on suicide risk prediction and mood swing detection, which was predominantly considered a multi-class classification problem. the top approaches in the previous shared task have predominantly utilized transformer-based models and multitask learning, yet the capabilities of prompting-based approaches in this context remains largely unexplored. there is a growing interest in the responsible use of llms in healthcare, including in psychotherapy and mental health assessment @xcite . the recent advancements have seen a significant increase in the zeroshot classification abilities of llms, alongside a deepened understanding of mental health issues @xcite . these models are increasingly recognized for their effectiveness in extracting information, especially in identifying mental health crises. they have demonstrated the ability to generate explainable findings and exhibit reasoning capabilities, which significantly enhances their utility in mental health assessments @xcite . this evolution in the capabilities of llms sets a new precedent for our approach and underscores the potential of these models in contributing to mental health assessments. ## data and tasks we use the reddit suicidality dataset provided by the organizers of the 2019/2024 clpsych workshop @xcite @xcite . our team's utilization of this data and our participation in the associated tasks adhere to the ethical review standards outlined by the organizers. the dataset comprises posts from 125 reddit users on various subreddits where each user has at least one post in r/suicidewatch. all users are categorized by experts into four risk levels: no risk, low risk, moderate risk, and severe risk. the distribution of the length of posts is shown in figure 1 . the participants of the shared task were asked to contribute to the following two methods of extracting evidence of suicidality from the users' posts: system: you are a suicide prevention therapist expert. you are performing psychological analysis of suicidality risks of the author of the following reddit post: title: {posttitle} content: {postcontent} it is estimated that the author has at least {risklevel} risk of suicidality. extract all relevant evidence (snippets) of at least {risklevel} suicidality exclusively from the post's content. formatting: strictly provide each snippet and only the snippets delimited by a semicolon(';') • task a -highlighting suicidal evidence: ## results the official results of task b are shown in our submission (uzh_clyp) achieved first place out of the 42 submitted runs according to the official scores. the official score of the shared task attempts to measure agreement between the modelgenerated summary and the human expert analysis @xcite using a model trained on natural language inference @xcite . this agreement is measured with two scores: the mean consistency, defined as the average sentence-level probability of consistency, 1 -p(contradiction), and the max contradiction, which represents the average maximum probability of a contradiction occurring, max(p (contradiction)). our submission performed best within task b among all submissions. it is worth noting that the top teams achieve promising performance, with only minor differences between them. however, another interesting insight from this generally high performance is that it demonstrates the ability of today's llms to generate analyses that align closely with the assessments of human experts. ## conclusion the system presented in this paper demonstrates the potential of using open-source llms to identify evidence of suicidality utilizing emotionally charged social media posts. our main innovation is the combination of rag with emotional regression of posts. this technique was found to be effective, as evidenced by the first-place performance in shared task b and the insights from our ablation experiments. our results highlight the ability of current llms to accurately summarize evidence of a user's suicide risk from online posts that closely align with human expert assessments. in conclusion, this study highlights the potential of llms in healthcare, particularly for mental health assessments. while the approach shows promise, especially in suicide risk analysis from social media posts, it also poses challenges, such as the risk of inaccurate content generation. future research should aim to enhance the accuracy of these models and consider the ethical implications of applying ai in sensitive health contexts. this research opens up new possibilities for the application of llms in mental health services, suggesting a path towards integrating them with traditional healthcare methods for more effective outcomes.