S c AN : Suicide Attempt and Ideation Events Dataset (2022)

Abstract: Suicide is an important public health concern and one of the leading causes of death worldwide. Suicidal behaviors, including suicide attempts (SA) and suicide ideations (SI), are leading risk factors for death by suicide. Information related to patients’ previous and current SA and SI are frequently documented in the electronic health record (EHR) notes. Accurate detection of such documentation may help improve surveillance and predictions of patients’ suicidal behaviors and alert medical professionals for suicide prevention efforts. In this study, we first built Suicide Attempt and Ideation Events (ScAN) dataset, a subset of the publicly available MIMIC III dataset spanning over 12k+ EHR notes with 19k+ annotated SA and SI events information. The annotations also contain attributes such as method of suicide attempt. We also provide a strong baseline model ScANER (Suicide Attempt and Ideation Events Retriever), a multi-task RoBERTa-based model with a retrieval module to extract all the relevant suicidal behavioral evidences from EHR notes of an hospital-stay and, and a prediction module to identify the type of suicidal behavior (SA and SI) concluded during the patient’s stay at the hospital. ScANER achieved a macro-weighted F1-score of 0.83 for identifying suicidal behavioral evidences and a macro F1-score of 0.78 and 0.60 for classification of SA and SI for the patient’s hospital-stay, respectively. ScAN and ScANER are publicly available.

## introduction for decades, suicide has been one of the leading causes of death @xcite . the suicide rate in the united states increased from 10.5 per 100, 000 in 1999 to 14.2 in 2018, a 35% increase @xcite . globally, 740, 000 people context: [*name*] is a 22 year old male with a history of angry and impulsive behavior who is transferred from an outside hospital s/p tylenol overdose. annotations: two instances of suicide attempts are annotated in the paragraph. 1. [*name*] ... overdose: annotated for suicide attempt and is assigned 'unsure' category as there is no definite documentation that it is a suicide attempt. 2. on sat ... broken: this is also annotated for suicide attempt and is assigned the category: 't71' (asphyxiation, hanging). commit suicide each year. the rates of suicidal behaviors, suicide attempt (sa) and suicide ideation (si), are much higher @xcite . a prior study shows that a large proportion of suicide victims sought care well before their death @xcite . suicidal behaviors, including sa and si are recorded by clinicians in electronic health records (ehrs). this knowledge can in turn help clinicians assess risk of suicide and make prevention efforts @xcite . the diagnostic icd codes include suicidality codes for both sa and si. however a study has shown that icd codes can only capture 3% si events, while 97% of si events are described in notes @xcite . in addition, of patients described with sa in their ehr notes, only 19% had the corresponding icd codes @xcite . therefore, it is important to develop natural language process-ing (nlp) approaches to capture such important suicidality information. researchers have developed nlp approaches to detect sa and si from ehr notes @xcite @xcite . these studies either used rule-based approaches @xcite @xcite or built the sa and si identification models on a small set @xcite or private set @xcite @xcite of ehr notes. it is also difficult to compare the results of those studies as they varied in ehr data, data curation, as well as nlp models, which were not made available to the public. in this study, we present scan: suicide attempt and ideation events dataset, a publicly available ehr dataset that is a subset of the mimic iii data @xcite . scan contains 19, 690 expert-annotated sa and si events with their attributes (e.g., methods for sa) over 12, 759 ehr notes. specifically, experts annotated suicidality evidence or sentences relevant to sa and si events during a patient's stay at the healthcare facility, an example of sa annotations is shown in fig 1 . the evidences were put together to assess whether the patient has an sa or si event. we also present scaner (suicide attempt and ideation events retriever), a roberta-based nlp model that is built on a multi-task learning framework for retrieving evidences from the ehrs and then predicting a patient's sa or si event using the complete set of ehr notes from the hospital stay using a multi-head attention model. we focus on the prediction of sa and si using all the ehr notes during a patient's stay because for the whole duration, multiple ehr notes and note types are generated, including admission notes, nursing notes, and discharge summary notes. suicidal information are described in multiple notes throughout the stay. for example, a patient was admitted to the hospital with opioid overdose. it was documented initially in the admission note as an sa, but later dismissed as an accident after physician's evaluation. in another example, an opioid overdose admission was first documented as an accident on admission, but later documented to be an sa event after clinical assessment. both scan and scaner capture sa and si information at the hospital-stay level. scaner is able to retrieve suicidal evidences from ehr notes with a macro-weighted f1-score of 0.83 and is able to predict sa and si with a macro f1-score of 0.78 and 0.60, respectively. our annotation guidelines, scan, and scaner system will be made publicly available, making scan a benchmark ehr dataset for sa and si events detection. we will release the training and evaluations splits used in this study for benchmarking new models. ## related works efforts on detecting sa and si within ehrs have been explored in recent years. most work used rule-based or traditional machine learning-based approaches. in one study, experts created handcrafted rules from mentions of suicidality (both sa and si) and then used the rules to identify suicidality as positive, negative, or unknown in a document @xcite . the rule-based approaches are limited by their scalability. in another study, structured and unstructured ehrs were used to classify at the hospital-stay level as sa, si, or no mention of suicidal behavior @xcite . the training data consisted of only 112 sa, 49 si and 322 unrelated examples. in contrast, scan comprises of 697 hospital-stays with more than 19,000 suicidal event examples over 12,759 clinical notes. only traditional machine learning models such as random forest @xcite were explored. in contrast, scaner was built on the state-of-the-art self-attention based model. hybrid approaches have also been developed to identify sa at the hospital-stay level @xcite . in that study, a post-processing heuristic rule-based filter (e.g., removing negated events) was applied to the machine-learning-based classifier (a svm @xcite classifier) to reduce false positives. training and evaluation were done also on relatively small datasets (500 for training and 500 for evaluation). finally, weakly supervised approaches have been developed to identify si from ehrs @xcite . in that study, authors used icd codes to identify 200 patients with si and then obtained ehr notes of those patients (6, 588). this ehr note dataset was then used as the 'current' si training data. the remaining 400 patients were labelled as 'potential' si and their 12, 227 ehrs were also labelled the same. authors used multiple statistical machine learning models and one deep learning model: convolutional neural network. (bhat and goldman-mellor, 2017) also used feed-forward neural networks to predict suicide attempts over 500k unique patients but the ehr data for this study is not publicly available. @xcite surveyed multiple studies where the researchers worked on private datasets @xcite for suicide attempt and ideation prediction. whereas in our study, in contrast to using the icd codes which has considerable errors, domain experts chart-reviewed a large, publicly available set of ehrs for si and sa, along with their attributes (e.g., positive or negative sa, si and the type of self-harm such as asphyxiation and overdose). ## dataset in this section, we introduce scan (suicide attempt and ideation events dataset) and describe it's data collection and annotation process. we also discuss some examples from scan along with basic dataset statistics. ## methodology in this section, we introduce scaner (suicide attempt and ideation events retreiver): a strong baseline model for our dataset. scaner consists of two sub-modules: (1) an evidence retriever module that extracts the evidences related to sa and ## results and discussion since the labels for both the retriever and prediction task are imbalanced, we used macro-weighted precision, recall, and f1-score to evaluate the overall performance of our models. macro-weighted metrics provide better model insights across all labels. the retriever module achieves an overall f1score of 0.63 for sa prediction and 0.64 for si prediction at paragraph-level. the performance for positive sa and si evidence is much higher than the performance for neg_unsure sa and negative si. we looked at the confusion matrices for sa and si paragraph-level prediction and found that largely scaner made mistakes between positive and neg_unsure labels for sa prediction and between positive and negative labels for si prediction (refer appendix c). the poor performance in sa for neg_unsure evidence prediction is mainly due to data sparsity where the neg_unsure cases are only 1743; in contrast, the positive cases are 4-fold higher. similarly, for si the positive cases are 1.4 times higher than the negative cases. ## conclusion in this paper, we introduce scan: a publicly available suicide attempt (sa) and ideation (si) events dataset that consists of 12, 759 ehr notes with 19, 960 unique evidence annotations for suicidal behavior. to our knowledge, this is the largest and publicly available dataset for sa and si, an important resource for suicidal behaviors research. we also provide a strong roberta baseline model for the dataset: scaner (sa and si retriever) which consists of two sub-modules: (a.) an evidence retriever module that extracts all the relevant evidence paragraphs from the patient's notes and (b.) a prediction module that evaluates the extracted evidence paragraphs and predicts the sa and si event label for the patient's stay at the hospital. scan and scaner could help extract suicidal behavior