Event Detection for Suicide Understanding (2022)

Abstract: Suicide is a serious problem in every society. Understanding life events of a potential patient is essential for successful suicide-risk assessment and prevention. In this work, we focus on the Event Detection (ED) task to identify event trigger words of suicide-related events in public posts of discussion forums. In particular, we introduce SuicideED: a new dataset for the ED task that features seven suicidal event types to comprehensively capture suicide actions and ideation, and general risk and protective factors. Our experiments with current state-of-the-art ED systems suggest that this domain poses meaningful challenges as there is significant room for improvement of ED models. We will release SuicideED to support future research in this important area.

## introduction suicide is a serious and growing problem in our society 1 . the most common procedure for suicide risk assessment is for clinicians to set up clinical interviews with potential patients that will provide rating scales based on a list of preset questions @xcite . however, interviews and similar activities require the willingness of potential patients to participate. given the associated mental states, such participation can be challenging to obtain for patients with high suicidal risks. in the meantime, people are increasingly spending more of their time on social networks, sharing inner thoughts and daily activities. this collection of social posts might draw a comprehensive picture of the patient's life that can be used to support the diagnosis of suicidal conditions. in fact, moderators of some social networks (e.g., reddit, reachout) use social posts to monitor suicide and apply immediate intervention if necessary. to assist with the processing of the large amount of posts, there have been a few methods and tools 1 https://www.nimh.nih.gov/health/statistics/suicide for automatically analyzing online posts to detect suicidal intent @xcite @xcite . however, these studies mainly focus on assessing the patients' susceptibility to suicide and fail to consider contributory life events that cause/lead to such conditions. in this paper, we argue that recognizing suicide-related events is also critical to suicide understanding, identification and prevention, and natural language processing (nlp) methods are necessary to support automatic identification of such events from the vast and growing number of social media posts. this work aims to advance the ultimate goal of creating nlp methods for suicide understanding by exploring the novel task of event detection (ed) for suicide-related events. ed is an important task in information extraction (ie) whose purpose is to identify event trigger words/mentions text data @xcite . take, for instance, the following paragraph: i don't date anyone and never will. it s a reason why i have no friends and never will. i wanna be funny and have a personality and be desirable but i 'm not that. i know its depression that causes it. adapted to our interest in suicide-related events, an ed system should be able to recognize "date" and "have" as trigger words for deteriorated personal relationship events (i.e., risk factors); "wanna", "have", and "desirable" as triggers for protective factor events, and "depression" as a trigger for a health-related risk factor event. the vast majority of advanced methods for ed are based on training deep neural networks on large labeled corpora @xcite . as such, to facilitate research in ed for suicide prevention, a key requirement is to have a benchmark dataset to standardize the development and evaluation of ed models. unfortunately, a large amount of existing suiciderelated datasets are protected due to sensitive pri-vacy concerns and, thus, fail to support the larger research community @xcite @xcite . moreover, these existing datasets are created to detect potential suicidal attempts based on text classification @xcite @xcite , which does not provide event trigger annotations of suicide-related events for ed. to overcome such challenges, this paper introduces suicideed, a new dataset for suicidal event detection that is manually annotated for seven distinct event types to comprehensively characterize suicide-related events regarding suicidal actions, thoughts, and risk and protective factors. to enable data sharing, our dataset is based on public posts from reddit where personal information is not presented to avoid privacy issues. the suicideed dataset is challenging as it involves informal texts, and requires event factuality and affected entity reasoning. our experiments show that the performance of current state-of-the-art ed models on suicideed lags behind their performance on other general-purpose ed datasets, thus calling for more research effort for suicide-specific ed. to facilitate future research in this area, suicideed is released publicly foot_0 for the research community. ## ontology design an important, previously unexplored, question is what constitutes relevant events that can provide useful insights for clinicians to better understand and recognize suicide-related incidents. accordingly, we consult specialized literature related to suicidal-behavior identification and treatment @xcite @xcite to define the event categories for our dataset. as such, we design the event types to be exclusive to avoid type overlapping, and sufficiently comprehensive to cover relevant/impactful suicide-related events in the data. eventually, we select the following seven event types that capture suicide-related actions, thoughts, and risk/protective factors. the first two event types are concerned with statements to indicate suicidal attempts or intentions. in particular, the action event type is dedicated to the direct expressions for actual suicidal attempts/actions, e.g., "i've started cutting myself again" foot_1 . in contrast, the second event type, ideation, represents suicidal inner thoughts, feelings, or desires, where no real action present, e.g., "i'm going to kill myself soon". these two types directly integrate factuality differentiation into the event types to better address the uniqueness of the data where hypothetical events are prevalent and understanding the factuality of events is critical to suicide intervention and prevention. the second group of event types focuses on external events that increase a subject's susceptibility to suicidal behaviours, i.e., risk factors (rf) @xcite . given the diverse nature of rf, four event types are proposed. rf-life events include mentions of a death of a close/loved entity, e.g., "my dog just died". rf-relationship concerns events related to social isolation, family breakdowns, or any mention of deteriorated interpersonal relationship, e.g., "my dad kicked me out of the house". events for rf-health cover mentions of physical diseases, mental illness, and behaviors that directly affect the subject's health, e.g., "i feel depressed". finally, rf-other incorporates every other rf event that cannot be assigned to life, relationship, or health issues but still qualify as rf, including financial issues, chronic abuse, and general quality-of-life problems. the final type, protective, captures events that drive an individual towards a better mentalhealth state, involving a broad range of positive activities, such as receiving effective medication or being motivated by social connections, e.g., "the medication seems to be helping". a detailed description and representative examples for each event type are presented in appendix b. ## data collection and annotation the documents for suicideed are collected from publicly available posts from reddit.com. in particular, we focus on three subreddits (subgroups) that contain a high percentage of suicide-related posts: r/suicidewatch, r/depression, and r/mentalhealth. each original post is considered as a separate document and only posts with more than 50 words are considered to increase the probability of an event being present. given the event types described in section 2, an annotation job posting is created on upwork.com and seven freelance annotators with previous experience in mental health and psychology, such as physicians and psychology graduates, are recruited. they are provided with a comprehensive guideline document 4 with thorough annotation instructions and numerous detailed examples for training. the annotators are instructed to select a single word for each event trigger (i.e., the most important) that clearly evokes the event, following the practices of prior ed work @xcite . overall, we annotate 2,300 documents for the seven event types from which the proportions of documents taken from the subreddits r/suicidewatch, r/depression, and r/mentalhealth are 50%, 30%, and 20%, respectively. we select 20% of the documents to be used for co-annotation, leading to a fleiss' kappa score of 0.8 (i.e., close to the almost perfect agreement range of [0.81 -1.0]). the remaining 80% of documents are distributed among the annotators for individual annotation. to facilitate and standardize future research, we divide suicideed into three different portions for training, test, and development purposes. ## dataset challenges compared to existing, general purpose, ed datasets, e.g., ace-05 @xcite , maven @xcite , and cyseced (trong et al., 2020), our suicideed dataset features at least three unique challenges for ed models. first, as its documents are obtained from reddit posts, suicideed involves texts where infor-4 https://github.com/nlp-uoregon/suicideed/guidelines.pdf mal words (e.g., "wanna", "gonna") are prevalent, sentences might not follow well-structured grammar rules, and first-person point of view is the main writing style. this is in contrast to existing ed datasets where documents are often retrieved from news outlets or reports with formal and wellstructured texts. second, in addition to the relevance to suicide, the event types in suicideed sometimes require models to simultaneously consider event factuality to accurately determine the types. this is clear for differentiating action and ideation where the key distinction concerns event factuality. another example involves the potential confusions between protective and rf where different event factuality might lead to different event types for the similar expressions. for instance, in the sentence "i have a lot of friends", the event trigger "have", belongs to the protective type given that it reveals a positive environment for the subject. on the contrary, in the sentence "i had a lot of friends", the trigger "had" should be considered as a rf-relationship type as it might instead imply current deteriorated social connections. furthermore, the event type determination in suicideed also necessitates appropriate identification of the entity that should be considered for the effect of an event. for instance, in the sentence "my sister killed herself.", the trigger word "killed" should have the action type if the entity of consideration is "sister". however, if we consider the event from the point of view of the poster/speaker, "killed" should be a rf-life event. in suicideed, the annotators are instructed to take the first person point of view (i.e., the poster) in the annotation decision. as such, ed models are expected to learn this feature from the data to achieve good performance. finally, some triggers in general-purpose datasets such as ace05 can be type-indicative to a great extent. for instance, the trigger word "bought" is almost certainly indicative for the transaction:transfer-ownership type. in contrast, figure 1 illustrates the ambiguity of the event triggers in suicideed by presenting the label distribution of the top 5 most frequent event trigger words. as can be seen, there is no particular dominant label for even the most frequent words. hence, a ed model must effectively capture surrounding context of triggers to perform classification. ## experiments to assess the complexity of the ed task in sui-cideed, we evaluate the performance of the following state-of-the-art ed models: cnn: a convolutional neural network for ed (nguyen and grishman, 2015); dmbert: a dynamic multipooling model based on bert @xcite ; berted: a bert-based model augmented with multi-layer perception @xcite ; bertgcn: a graph convolutional network (gcn) based on dependency trees (nguyen and grishman, 2018); gatedgcn: a gcn model using bert and trigger-aware gating mechanism @xcite ; and eegcn: a gcn model that exploits syntactic structure and typed dependency information @xcite . all of these models leverage the pre-trained bert model to obtain representation vectors. the hyperparameters of the models are fine-tuned over the development data. additionally, we further finetune the pre-trained bert model over unlabeled reddit posts from the same three subreddits (i.e., about 40k posts) using masked language modeling @xcite . we report the model performance when the finetuned bert replaces the original pre-trained bert to explore the effectiveness of domain customization of bert for the informal texts in reddit. ## related work suicide detection and prevention using nlp methods has caught the attention of many researchers. due to the privacy restrictions associated with clinical databases, researchers have used publiclyavailable data from social media with manual annotations of recognizable signals of mental health issues @xcite . the majority of methods, however, focus on detecting suicidal attempts or assessing suicide propensity of users based on social media posts @xcite @xcite . as such, these prior works have only explored the setting of overall text classification which fails to explore fine-grained analysis/classification at word level required to reveal suicide-related events as this paper does. prior research efforts for ed, in general-purpose settings, have introduced various methodologies to address such task, including feature engineering @xcite @xcite and deep learning @xcite @xcite @xcite models. however, such prior work mainly utilizes the ed datasets with general event types and formal texts, i.e., ace-05 @xcite , that might not be helpful for particular domains with unique requirements such as the one addressed in this work. recently, there have been some effort on creating new datasets for ed in more specific domains, including biomedical texts @xcite , literary texts @xcite , cybersecurity texts @xcite , fine-grained event types (le and nguyen, 2021), and historical texts @xcite . however, none of existing ed datasets explore suicide-related events in social media texts. ## conclusion we present suicideed, the first dataset focused on the event detection task for suicide-related events. suicideed is manually annotated for seven event types and provides enough training examples to develop large-scale deep learning models. we perform extensive evaluations of state-of-the-art ed models that demonstrate the challenges entailed by this difficult domain and call for further efforts to improve performance. in the future, we plan to extend suicideed to annotate event arguments and other event properties to better support event analysis and understanding for suicide.