P sy GUARD : An Automated System for Suicide Detection and Risk Assessment in Psychological Counseling (2024)

Abstract: As awareness of mental health issues grows, online counseling support services are becoming increasingly prevalent worldwide. Detecting whether users express suicidal ideation in text-based counseling services is crucial for identifying and prioritizing at-risk individuals. However, the lack of domain-specific systems to facilitate fine-grained suicide detection and corresponding risk assessment in online counseling poses a significant challenge for automated crisis intervention aimed at suicide prevention. In this paper, we propose PsyGUARD, an automated system for detecting suicide ideation and assessing risk in psychological counseling. To achieve this, we first develop a detailed taxonomy for detecting suicide ideation based on foundational theories. We then curate a large-scale, high-quality dataset called PsySUICIDE for suicide detection. To evaluate the capabilities of automated systems in fine-grained suicide detection, we establish a range of baselines. Subsequently, to assist automated services in providing safe, helpful, and tailored responses for further assessment, we propose to build a suite of risk assessment frameworks. Our study not only provides an insightful analysis of the effectiveness of automated risk assessment systems based on fine-grained suicide detection but also highlights their potential to improve mental health services on online counseling platforms. Code, data, and models are available at https://github.com/qiuhuachuan/PsyGUARD.

## introduction it is well-documented that suicide is a major public health problem worldwide @xcite @xcite . each suicide death represents a catastrophic tragedy and is reported to directly or indirectly affect many 在过去的几个月里，我一直有自杀倾向，最近还出现了自 杀行为。 ( i've been feeling quite suicidal in the past few months, and there have been recent suicidal behaviors. ) individuals, including relatives, friends, and society @xcite @xcite . online counseling services are available in many countries, allowing for confidential and immediate help to those in need free of time and space, and therefore play a critical role in suicide prevention @xcite , which can effectively halt user suicidal ideation @xcite . however, the lack of domain-specific systems to enhance chinese fine-grained suicide detection and corresponding risk assessment in online counseling poses a significant challenge for automated crisis intervention with the purpose of suicide prevention. motivation indeed, many researchers have been working on the development of automated systems for suicide detection that can be used in real-life production @xcite @xcite @xcite . despite advancements in such automated detection systems, existing studies are mainly confined to suicide detection, often ignoring fine-grained suicidal actions or thoughts and corresponding approaches for risk assessment, as illustrated in figure 1 . automated suicide detection and risk assessment systems can help scale support services to reach a larger population, especially considering the increasing prevalence of online counseling and mental health support platforms. further, by providing automated support and intervention, individuals may feel more comfortable seeking help online, thus reducing the stigma associated with mental health issues and suicide prevention @xcite . empirical evidence @xcite indicates that individuals are willing to interact with human or ai counselors, with many having disclosed their suicidal thoughts, plans, and actions, underscoring the importance of automated systems for suicide detection and risk assessment. the deficiencies in existing systems significantly limit automated systems' capability to ensure safe, helpful, customized services in providing mental health support, which motivates us to carry out the work presented in this paper. challenges the lack of fine-grained suicide detection datasets in psychological counseling is a major challenge. currently, numerous studies have made significant progress in detecting suicidal ideation, but they primarily focus on social media platforms @xcite @xcite @xcite rather than on counseling conversations. therefore, using such datasets directly for risk detection in online counseling may not be suitable due to a gap in user expressions, such as emojis, urls, images, or special marks. additionally, challenges posed by datasets collected from electronic health records @xcite or mental health records also include gaps in data format. furthermore, most studies that primarily focus on binary suicidal ideation detection @xcite @xcite face challenges in considering the granularity of suicide ideation categories in the real world. the lack of a comprehensive suite of risk assessments for corresponding suicide categories is another challenge. in addition to users mentioning that they have attempted suicide, simply identifying fine-grained suicide categories is not sufficient to conclude whether a user will actually commit suicide. risk assessment can directly guide how to intervene in a crisis situation. therefore, suicide classification is the initial step in suicide preven-tion, and further risk assessment is required, which is largely overlooked by current studies. our approach in this paper, to our knowledge, we are the first to propose to study an automated system for suicide detection and risk assessment in psychological counseling. our paper is organized into five main parts. section 2 ( §2) describes the existing works related to ours. section 3 ( §3) demonstrates the detailed process of taxonomy construction. section 4 ( §4) elaborates on rigorous data collection. section 5 ( §5) constructs extensive baselines, and section 6 ( §6) provides a simple yet effective framework for risk assessment prior to crisis intervention. our contributions we believe our work offers a new perspective on building an automated system for suicide detection and risk assessment in psychological counseling within the research community. our contributions can be summarized as follows: ## taxonomy construction to build an automated system for suicide detection and risk assessment in psychological counseling, we first propose to develop a novel taxonomy for categorizing the level of suicide based on suicide actions and thoughts. in collaboration with experts 1 in psychological counseling, we have adapted and refined existing suicidal taxonomies, whether the textual content contains suicidal ideation based on the user's perspective? ## data collection to validate the feasibility of our proposed finegrained taxonomy in real-world settings and address the limitations that current datasets are typically collected from a single source, as well as to automate risk assessment further, we collect a large-scale user input corpus from several media person is a doctoral student majoring in computer science and is the first author of this paper. next, we will describe the process of constructing psysuicide, including raw data collection, development of an annotation platform, initial annotator training, iterative human annotation, disagreement adjudication, and quality control. finally, we will present the statistics of the psysuicide dataset, as illustrated in ## automated system for suicide detection we conduct our experiments using pre-trained language models (lms) and large language models (llms). all experiments in this paper are performed on nvidia a100 8 × 80g gpus. ## 2 figure 3 : our automated risk assessment framework when interacting with a user who has active suicidal ideation. the top panel presents the flow chart of our automated system for suicide detection and risk assessment. the bottom panel presents a concrete example for illustration. the text on a pink background will be adaptively replaced by the detected suicidal category. the text on a gray background is a dialogue session between a user and a counselor, where responses are generated by the llm. the text on a yellow background is a risk assessment result completed using the prompting method. corresponding chinese version, see figure 11 . the model we use is gpt-4o (gpt-4o-2024-05-13). for other cases, please refer to appendix j. ## automated system for risk assessment to further utilize suicidal ideation detection, we propose a framework for risk assessment. once our automated suicide detection system identifies a user expressing a suicide attempt, our system will first recommend the user to the free 24-hour helpline and inform the professional counselor for crisis intervention and further referral. for the other four types of suicidal ideation and five types of non-suicidal categories with different risk levels, we use an automated risk assessment framework to interact with users, as shown in figure 3 . moreover, nine types of screening questions @xcite for risk assessment are shown in figure 12 in appendix j. ## conclusion in sum, we present a novel and theoretically grounded fine-grained taxonomy for detecting suicidal ideation, merging risk levels with categories of suicidal actions and thoughts. we address gaps by introducing the psysuicide dataset, which is manually annotated with experts and rigorous quality control. further, we develop various baselines based on pre-trained lms and llms and create an llm-based risk assessment framework for users during text-based online counseling. our work provides an insightful analysis of the effectiveness of automated risk assessment systems and their potential advantages in improving mental health services in online counseling platforms.