Event Detection for Suicide Understanding
Abstract
Suicide is a serious problem in every society.
Understanding life events of a potential patient is essential for successful suicide-risk assessment and prevention.
In this work, we focus on the Event Detection (ED) task to identify event trigger words of suicide-related events in public posts of discussion forums.
In particular, we introduce SuicideED: a new dataset for the ED task that features seven suicidal event types to comprehensively capture suicide actions and ideation, and general risk and protective factors.
Our experiments with current state-of-the-art ED systems suggest that this domain poses meaningful challenges as there is signiﬁcant room for improvement of ED models.
We publicly release SuicideED to support future research in this important area.
Introduction
Suicide is a serious and growing problem in our society 1 .
The most common procedure for suicide risk assessment is for clinicians to set up clinical interviews with potential patients that will provide rating scales based on a list of preset questions (Ross et al., 2012).
However, interviews and similar activities require the willingness of potential patients to participate.
Given the associated mental states, such participation can be challenging to obtain for patients with high suicidal risks.
In the meantime, people are increasingly spending more of their time on social networks, sharing inner thoughts and daily activities.
This collection of social posts might draw a comprehensive picture of the patient’s life that can be used to support the diagnosis of suicidal conditions.
In fact, moderators of some social networks (e.g., Reddit, Reachout) use social posts to monitor suicide and apply immediate intervention if necessary.
To assist with the processing of the large amount of posts, there have been a few methods and tools for automatically analyzing online posts to detect suicidal intent (Ji et al., 2018; Shing et al., 2018;
Coppersmith et al., 2015; Milne et al., 2016).
However, these studies mainly focus on assessing the patients’ susceptibility to suicide and fail to consider contributory life events that cause/lead to such conditions.
In this paper, we argue that recognizing suicide-related events is also critical to suicide understanding, identiﬁcation and prevention, and natural language processing (NLP) methods are necessary to support automatic identiﬁcation of such events from the vast and growing number of social media posts.
This work aims to advance the ultimate goal of creating NLP methods for suicide understanding by exploring the novel task of Event Detection (ED) for suicide-related events.
ED is an important task in Information Extraction (IE) whose purpose is to identify event trigger words/mentions text data(Ahn, 2006; Ji and Grishman, 2008).
Take, for instance, the following paragraph:
I don’t date anyone and never will.
It s a reason why I have no friends and never will.
I wanna be funny and have a personality and be desirable but I ’m not that.
I know its depression that causes it.
Adapted to our interest in suicide-related events, an ED system should be able to recognize “ date ” and “ have ” as trigger words for deteriorated personal relationship events (i.e., risk factors); “ wanna ”, “ have ”, and “ desirable ” as triggers for protective factor events, and “ depression ” as a trigger for a health-related risk factor event.
The vast majority of advanced methods for ED are based on training deep neural networks on large labeled corpora (Nguyen and Grishman, 2015; Chen et al., 2015).
As such, to facilitate research in ED for suicide prevention, a key requirement is to have a benchmark dataset to standardize the development and evaluation of ED models.
Unfortunately, a large amount of existing suiciderelated datasets are protected due to sensitive pri- vacy concerns and, thus, fail to support the larger research community (Coppersmith et al., 2015;
Vioules et al., 2018; Bhat and Goldman-Mellor, 2017).
Moreover, these existing datasets are created to detect potential suicidal attempts based on text classiﬁcation (Vioules et al., 2018;
Bhat and Goldman-Mellor, 2017; Shing et al., 2018), which does not provide event trigger annotations of suicide-related events for ED.
To overcome such challenges, this paper introduces SuicideED, a new dataset for suicidal event detection that is manually annotated for seven distinct event types to comprehensively characterize suicide-related events regarding suicidal actions, thoughts, and risk and protective factors.
To enable data sharing, our dataset is based on public posts from Reddit where personal information is not presented to avoid privacy issues.
The SuicideED dataset is challenging as it involves informal texts, and requires event factuality and affected entity reasoning.
Our experiments show that the performance of current state-of-the-art ED models on SuicideED lags behind their performance on other general-purpose ED datasets, thus calling for more research effort for suicide-speciﬁc ED.
To facilitate future research in this area, SuicideED is released publicly 2 for the research community.
Ontology Design
An important, previously unexplored, question is what constitutes relevant events that can provide useful insights for clinicians to better understand and recognize suicide-related incidents.
Accordingly, we consult specialized literature related to suicidal-behavior identiﬁcation and treatment (Gutierrez, 2006; de Ruiter and Nicholls, 2011; O’Connor et al., 2013) to deﬁne the event categories for our dataset.
As such, we design the event types to be exclusive to avoid type overlapping, and sufﬁciently comprehensive to cover relevant/impactful suicide-related events in the data.
Eventually, we select the following seven event types that capture suicide-related actions, thoughts, and risk/protective factors.
The ﬁrst two event types are concerned with statements to indicate suicidal attempts or intentions.
In particular, the ACTION event type is dedicated to the direct expressions for actual suicidal attempts/actions, e.g., “ I’ve started cutting myself again ” 3 .
In contrast, the second event type,
IDEATION , represents suicidal inner thoughts, feelings, or desires, where no real action present, e.g., “ I’m going to kill myself soon ”.
These two types directly integrate factuality differentiation into the event types to better address the uniqueness of the data where hypothetical events are prevalent and understanding the factuality of events is critical to suicide intervention and prevention.
The second group of event types focuses on external events that increase a subject’s susceptibility to suicidal behaviours, i.e., risk factors (RF) (Gutierrez, 2006).
Given the diverse nature of RF, four event types are proposed.
RF-LIFE events include mentions of a death of a close/loved entity, e.g., “ My dog just died ”.
RF-RELATIONSHIP
concerns events related to social isolation, family breakdowns, or any mention of deteriorated interpersonal relationship, e.g., “ My dad kicked me out of the house ”.
Events for RF-HEALTH cover mentions of physical diseases, mental illness, and behaviors that directly affect the subject’s health, e.g., “ I feel depressed ”.
Finally, RF-OTHER incorporates every other RF event that cannot be assigned to life, relationship, or health issues but still qualify as RF, including ﬁnancial issues, chronic abuse, and general quality-of-life problems.
The ﬁnal type, PROTECTIVE , captures events that drive an individual towards a better mentalhealth state, involving a broad range of positive activities, such as receiving effective medication or being motivated by social connections, e.g., “ The medication seems to be helping ”.
A detailed description and representative examples for each event type are presented in Appendix B.
The documents for SuicideED are collected from publicly available posts from reddit.com . In particular, we focus on three subreddits (subgroups) that contain a high percentage of suicide-related posts: r/SuicideWatch , r/depression , and r/mentalhealth . Each original post is considered as a separate document and only posts with more than 50 words are considered to increase the probability of an event being present.
Given the event types described in section 2, an annotation job posting is created on upwork.com and seven freelance annotators with previous experience in mental health and psychology, such Train
[Table]
[Table] [Table]
as physicians and psychology graduates, are recruited. They are provided with a comprehensive guideline document 4 with thorough annotation instructions and numerous detailed examples for training.
The annotators are instructed to select a single word for each event trigger (i.e., the most important) that clearly evokes the event, following the practices of prior ED work (Nguyen and Grishman, 2015).
Overall, we annotate 2,300 documents for the seven event types from which the proportions of documents taken from the subreddits r/SuicideWatch , r/depression , and r/mentalhealth are 50%, 30%, and 20%, respectively.
We select 20% of the documents to be used for co-annotation, leading to a Fleiss’ Kappa score of 0 .
8 (i.e., close to the almost perfect agreement range of [0 . 81 − 1 . 0] ).
The remaining 80% of documents are distributed among the annotators for individual annotation.
To facilitate and standardize future research, we divide SuicideED into three different portions for training, test, and development purposes.
Table 1 presents some statistics for the different data portions while Table 2 shows the event type distribution.
Dataset Challenges
Compared to existing, general purpose, ED datasets, e.g., ACE-05 (Walker et al., 2006), MAVEN (Wang et al., 2020), and CySecED (Trong et al., 2020), our SuicideED dataset features at least three unique challenges for ED models.
First, as its documents are obtained from Reddit posts, SuicideED involves texts where infor- mal words (e.g., “ wanna ”, “ gonna ”) are prevalent, sentences might not follow well-structured grammar rules, and ﬁrst-person point of view is the main writing style.
This is in contrast to existing ED datasets where documents are often retrieved from news outlets or reports with formal and wellstructured texts.
Second, in addition to the relevance to suicide, the event types in SuicideED sometimes require models to simultaneously consider event factuality to accurately determine the types.
This is clear for differentiating ACTION and IDEATION where the key distinction concerns event factuality.
Another example involves the potential confusions between PROTECTIVE and RF where different event factuality might lead to different event types for the similar expressions.
For instance, in the sentence “ I have a lot of friends ”, the event trigger “ have ”, belongs to the PROTECTIVE type given that it reveals a positive environment for the subject.
On the contrary, in the sentence “ I had a lot of friends ”, the trigger “ had ” should be considered as a RF-RELATIONSHIP type as it might instead imply current deteriorated social connections.
Furthermore, the event type determination in SuicideED also necessitates appropriate identiﬁcation of the entity that should be considered for the effect of an event.
For instance, in the sentence “ My sister killed herself. ”, the trigger word “ killed ” should have the ACTION type if the entity of consideration is “ sister ”.
However, if we consider the event from the point of view of the poster/speaker, “ killed ” should be a RF-LIFE event.
In SuicideED, the annotators are instructed to take the ﬁrst person point of view (i.e., the poster) in the annotation decision.
As such, ED models are expected to learn this feature from the data to achieve good performance.
Finally, some triggers in general-purpose datasets such as ACE05 can be type-indicative to a great extent.
For instance, the trigger word “bought” is almost certainly indicative for the
Transaction:Transfer-Ownership type.
In contrast, Figure 1 illustrates the ambiguity of the event triggers in SuicideED by presenting the label distribution of the top 5 most frequent event trigger words.
As can be seen, there is no particular dominant label for even the most frequent words.
Hence, a ED model must effectively capture surrounding context of triggers to perform classiﬁcation.
do get had think wish
Percentage ACTION
IDEATION
PROTECTIVE
RF-HEALTH
RF-LIFE
RF-RELATION
RF
[Figure]
OTHER
Non
[Figure]
Eventive
Experiments
To assess the complexity of the ED task in SuicideED, we evaluate the performance of the following state-of-the-art ED models: CNN : a convolutional neural network for ED (Nguyen and
Grishman, 2015); DMBERT : a dynamic multipooling model based on BERT (Wang et al., 2019); BERTED : a BERT-based model augmented with multi-layer perception (Yang et al., 2019);
BERTGCN : a graph convolutional network (GCN) based on dependency trees (Nguyen and Grishman, 2018); GatedGCN : a GCN model using BERT and trigger-aware gating mechanism (Lai et al., 2020); and EEGCN : a GCN model that exploits syntactic structure and typed dependency information (Cui et al., 2020).
All of these models leverage the pre-trained BERT model to obtain representation vectors.
The hyperparameters of the models are ﬁne-tuned over the development data. Additionally, we further ﬁnetune the pre-trained
BERT model over unlabeled Reddit posts from the same three subreddits (i.e., about 40K posts) using masked language modeling (Devlin et al., 2019).
We report the model performance when the ﬁnetuned BERT replaces the original pre-trained BERT to explore the effectiveness of domain customization of BERT for the informal texts in Reddit.
[Table]
Table 3 presents the performance of the models on the SuicideED test set.
Our ﬁrst observation is that ﬁne-tuning BERT over Reddit posts can successfully improve the performance of all ED models. This improvement, though, is less pronounced for the more recent and advanced ED models, i.e., GatedGCN and EEGCN.
Second, the performance of the graph-based models (e.g., GatedGCN and EEGCN) is signiﬁcantly better than those for non-graph-based models (i.e., CNN, DMBERT, and BERTED).
As such, despite the informal nature of texts that can hinder the performance for dependency parsing, dependency trees are still helpful for the representation learning of ED models in SuicideED.
Finally and most importantly, we ﬁnd that the performance of existing ED models on SuicideED is substantially below than the typical performance of such models general purpose ED datasets (e.g., 77.6% on ACE-05 with GatedGCN and EEGCN) (Lai et al., 2020).
These results then suggests the unique challenges of SuicideED for ED models and highlight the need for further, domain-speciﬁc research to improve ED for suicide-related events.
Related Work
Suicide detection and prevention using NLP methods has caught the attention of many researchers.
Due to the privacy restrictions associated with clinical databases, researchers have used publiclyavailable data from social media with manual annotations of recognizable signals of mental health issues (Coppersmith et al., 2015; Shing et al., 2018).
The majority of methods, however, focus on detecting suicidal attempts or assessing suicide propensity of users based on social media posts (Coppersmith et al., 2015;
Bhat and Goldman-Mellor, 2017; Shing et al., 2018; Zirikly et al., 2019).
As such, these prior works have only explored the setting of overall text classiﬁcation which fails to explore ﬁne-grained analysis/classiﬁcation at word level required to reveal suicide-related events as this paper does.
Prior research efforts for ED, in general-purpose settings, have introduced various methodologies to address such task, including feature engineering (Ahn, 2006; Ji and Grishman, 2008; Li et al., 2013) and deep learning (Nguyen and Grishman, 2015;
Chen et al., 2015; Wang et al., 2019; Cui et al., 2020; Ngo Trung et al., 2021; Pouran Ben Veyseh et al., 2021b,a) models.
However, such prior work mainly utilizes the ED datasets with general event types and formal texts, i.e., ACE-05 (Walker et al., 2006), that might not be helpful for particular domains with unique requirements such as the one addressed in this work.
Recently, there have been some effort on creating new datasets for ED in more speciﬁc domains, including biomedical texts (Kim et al., 2009), literary texts (Sims et al., 2019), cybersecurity texts (Satyapanich et al., 2020;
Trong et al., 2020), ﬁne-grained event types (Le and Nguyen, 2021), and historical texts (Lai et al., 2021).
However, none of existing ED datasets explore suicide-related events in social media texts.
Conclusion
We present SuicideED, the ﬁrst dataset focused on the event detection task for suicide-related events.
SuicideED is manually annotated for seven event types and provides enough training examples to develop large-scale deep learning models.
We perform extensive evaluations of state-of-the-art ED models that demonstrate the challenges entailed by this difﬁcult domain and call for further efforts to improve performance.
In the future, we plan to extend SuicideED to annotate event arguments and other event properties to better support event analysis and understanding for suicide.
Statement of Ethics and Human Subject Research
Working with sensitive data such as mental health information from human subjects requires taking special care.
This becomes particularly relevant in this case as our main objective is to provide a dataset for general public use.
Benton et al.
(2017) discuss, however, that research with human subjects information is exempted from the required full Institutional Review Board (IRB) review if the data is already available from public sources or if the identity of the subjects cannot be recovered.
By design, Reddit is a platform where users remain anonymous and make their posts available to the general public.
Nonetheless, additional privacy measures were taken by removing any username mentions from the documents as they can sometimes include identiﬁable information.
Furthermore, unlike previous works where the main objective is to assess suicidal risk at the user level (Coppersmith et al., 2015; Bhat and Goldman-Mellor, 2017; Shing et al., 2018; Zirikly et al., 2019), this work focuses on sentence-level ED.
As such, our dataset does not include any user-level information that could be used to identify individual subjects.
Hence, this work is considered exempt from review by our University’s IRB as the documents used are already publicly available and the original posters are impossible to identify.
Minimizing impact on the annotators : All prospective annotators were informed beforehand about the nature of the related text material and were made aware of its potential impact on their mental health.
All chosen annotators had background knowledge/training on the subject at hand and were either clinicians or psychology graduates.
Any candidates who reported suffering from, or having a history of, mental health-related issues were not considered out of concern for their health.
Acknowledgement
This research has been supported by the Army Research Ofﬁce (ARO) grant W911NF-21-1and the NSF grant CNS-1747798 to the IUCRC Center for Big Learning.
This research is also based upon work supported by the Ofﬁce of the Director of National Intelligence (ODNI),
Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No.
201919051600006 under the Better Extraction from Text
Towards Enhanced Retrieval (BETTER) Program.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies, either expressed or implied, of ARO, ODNI, IARPA, the Department of Defense, or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
This document does not contain technology or technical data controlled under either the
U.S. International Trafﬁc in Arms Regulations or the U.S. Export Administration Regulations.
We thank the anonymous reviewers and Tracy King for their helpful feedback.
A Topic Modeling
To better understand the topics related to suicide in the SuicideED dataset, we run a topic modeling analysis using Latent Dirichlet Allocation (LDA) (Blei et al., 2003) over the documents in the dataset.
We extract ten topics from the analysis and present their words in Table 4.
English stop-words, the least ( p < 0 .
01 ), and most frequent words ( p > 0 . 2 ) were removed in the analysis.
Interestingly, it can be observed that posts can be summarized into 3 main categories: school (2, 8), work (5, 9, 10), and family (3, 4, 6, 7), which somehow reﬂects the sources of mental issues. Words
[Table]
hate, thoughts, point, stop, care, say, [Table] , living
[Table]
wish, hate, worse, try, year, shit, school, thoughts world, parents, suicidal, right, bad, point, person tell, love, tired, days, person, doing, death, mom pain, real, tried, need, maybe, work, hurt, tired, talk
[Table]
care, told, tired, said, parents, bad, need, leave, right
[Table]
care, matter
[Table]
days
[Table]
actually
[Table]
feels
[Table]
B Annotation Guideline
Table 5 and 6 present a detailed description of event types and examples for each event type in our SuicideED dataset.
Type
Description
Examples
A C
T I
ON
This category includes any event of an individual engaging in actions that bring them closer to dying by suicide.
These include any previous suicide attempts, preparatory acts towards a future attempt, or self-inﬂicted violence.
When annotating this type of event, it is important that an actual action takes place and that it goes beyond verbalization or intent.
As such, sentences containing these events mainly talk about the past or ongoing situations.
A previous suicide attempt is a self-inﬂicted, potentially injurious behavior with an intent to die as a result.
I tried to kill myself last night.
A preparatory act consists of any acts of preparation toward making a suicide attempt.
Must be beyond verbalization or thought such as assembling a method (e.g. buying a gun, collecting pills) or preparing for death (e.g. writing a suicide note and a will).
Just looked online for the quickest way
I left a note for my parents
Self-inﬂicted violence includes self-directed, harmful behaviors that do not have a clear intent to die as a result.
I’ve started cutting myself again
I D
E A T
I ON
These events focus on expressing thoughts and feelings but no actual action is present.
These, however, are not related to actions such as preparatory acts and, instead, refer to verbalizations of inner feelings/desires.
It includes passive thoughts about wanting to be dead:
I wish I was dead
And, active thoughts about killing oneself.
I am going to kill myself soon
P R
O
TE
C T
I
V E
These events are related to capacities, qualities, environmental and personal resources that increase resilience; drive an individual toward growth, stability, health, and/or an increase in coping with different life events.
For this category, please annotate any sentence that showcases a positive impact on an individual.
These can be verbalizations of self-worth and willingness to get better, access to medical resources, positive personal relationships, positive cultural beliefs, etc.
Access to effective behavioral health care and medication:
My therapist says that I should talk more.
The medication seems to be helping .
Connectedness to individuals, family, community, and social institutions:
At least my friends are there for me
Life skills (including problem-solving skills and coping skills, ability to adapt to change):
I’ve always been good at helping people
Self-esteem and a sense of purpose or meaning in life:
My life is much better than many people.
Expressing a willingness to improve:
I really want to get better.
I wanna be funny and outgoing.
Cultural, religious, or personal beliefs that discourage suicide:
I know God disapproves of what I’m thinking.
Type
Description
Examples
R F - L I
F E
This risk factor event is easy to identify as a loss of life of a both human an non-human entities.
The loss of life might be explicitly or implicitly expressed.
[Table]
expressed:
[Table] [Table]
[Table] [Table]
R F - R
EL
A T I
ON
S H
I
P
These include events such as social isolation, family breakdowns, divorce, etc.
Include in this category all events that show a loss of connection with other people.
These can be both verbalization of feelings of isolation or actual incidents of loss of an interpersonal relationship such as a break-up or argument with another individual.
Social isolation:
I don’t have anyone to talk to.
Family breakdown:
My dad just kicked me out of the house.
Divorce:
After my divorce , I started drinking
R F - H
E A
LT
H
These include events such as social isolation, family breakdowns, divorce, etc.
Include in this category all events that show a loss of connection with other people.
These can be both verbalization of feelings of isolation or actual incidents of loss of an interpersonal relationship such as a break-up or argument with another individual.
Mental disease/disorder such as depression,
PTSD:
Can’t deal with my depression right now.
I’ve been diagnosed with BPD.
Chronic or long-term disease, pain, and disability:
I’m just giving into my eating disorder.
Recently, my diabetes has been acting up.
Misuse and abuse of alcohol or other drugs:
I’ve been drinking a lot lately
R F - O
T H
E R
These events include all other risk factors that do not fall into the LIFE, RELATIONSHIP, or
HEALTH categories.
As such, these can be events of very diverse natures such as ﬁnancial issues, chronic abuse, discrimination, or general quality of life problems.
Financial hardship:
Can’t afford to pay rent anymore...
[Table]