Self-Adapted Utterance Selection for Suicidal Ideation Detection in Lifeline Conversations (2023)

Abstract: This paper investigates a crucial aspect of mental health by exploring the detection of suicidal ideation in spoken phone conversations between callers and counselors at a suicide prevention hotline. These conversations can be lengthy, noisy, and cover a broad range of topics, making it challenging for NLP models to accurately identify the caller’s suicidal ideation. To address these difficulties, we introduce a novel, self-adaptive approach that identifies the most critical utterances that the NLP model can more easily distinguish. The experiments use real-world Lifeline transcriptions, expertly labeled, and show that our approach outperforms the baseline models in overall performance with an F-score of 66.01%. In detecting the most dangerous cases, our approach achieves a significantly higher F-score of 65.94% compared to the baseline models, an improvement of 8.9%. The selected utterances can also provide valuable insights for suicide prevention research. Furthermore, our approach demonstrates its versatility by showing its effectiveness in sentiment analysis, making it a valuable tool for NLP applications beyond the healthcare domain.

## introduction the suicide prevention hotline provides a free service for the people with mental issues to have the support on a 24/7 basis. on the other end of the call, a group of counselors who are trained volunteers are waiting for calls online in shifts. as mental health problems are becoming more and more prevalent in society, the difficulty of finding a sufficient number of counselors forms a common challenge that the suicide prevention hotline of many countries in the world is facing. the overwhelmed counselors and the busy phone line make the individuals in urgent need of help unable to get the opportunity to have the counseling. in order to alleviate of demand of suicide counseling, the intelligent conversation system forms a potential direction to study for aiding the hotline service. fully automated chatbots as counselors providing counseling services are still impractical because the current natural language processing (nlp) technology has not yet been able to construct a reliable chatbot like a well-trained counselor to talk to these people who are mentally unstable and at high risk of suicide. in this work, we approach the suicide counseling aiding from a more practical aspect. instead of creating a chatbot to replace the human counselors, our goal is to propose a model for suicidal ideation detection @xcite . during the counseling, our model can be used for monitoring the conversation and identifying the suicide attempt of the caller. once it is discovered that the caller is at a high risk of suicide or self-harm, more experienced counselors and other resources can immediately intervene at this time to prevent further harm. text-based suicidal ideation detection is a hot topic in the nlp area. in addition to medical records, clinical notes @xcite , and suicide notes @xcite , previous work on textual suicidal ideation detection mainly focused on the social media data, such as tweets @xcite @xcite , microblogs @xcite , or online forums @xcite @xcite . broadly monitoring the suicidality of users on the social media is no doubt a new way for public health professionals to prevent suicides. however, social media-based suicidal ideation detection can cover only a part of cases as not everyone uses the social media and the users with suicidal ideation may not share their troubles publicly. the scope of this work is entirely different from those based on social media data. we focus on the lifeline conversations between callers in trouble and the well-trained counselors. unlike the social media data, which are posted by the users unilaterally, the conversational data are made of the interaction between two or even more parties (e.g., the caller's family member). in the case of suicide prevention hotlines, a conversation is typically made by a caller and a counselor. the well-trained counselor will try to guide the caller to express her or his emotions and distress, revealing rich information for suicide risk assessment. the data used in this work were collected from the real world recordings provided by taiwan lifeline international. 1 psychologists were invited for assessing and labeling the suicidal ideation of the caller in each conversation. to the best of our knowledge, this is a pioneering work of conversation-based suicidal ideation detection. unlike social media posts, the conversational data is usually longer and noisier. in our real world data, a conversation contains 515 utterances and 7,981 tokens in average, causing a barrier to models analyzing the content. in the long conversation, not all utterances provide the cue for suicide risk assessment. instead of analyzing the whole conversation, we focus on key interactions between the caller and the counselor. however, extracting the important utterances from the hotline conversation can be still challenging due to the lack of sufficient training data. for this reason, we propose a novel self-adapted approach to utterance selection. our idea is to focus on a number of discriminative utterances that provide useful information for nlp models to do the classification. to accomplish this goal, a distant-supervised model for suicidal ideation detection at the utterance level is further built for generating the training data for the utterance selection model. our approach to detecting suicidal ideation in phone-call conversations is distinct from reinforcement learning methods that are used to select important features or instances for training the main model. instead, our distant-supervision methodology leverages information from the conversation level to the utterance level based on a key observa-1 http://www.life1995.org.tw/ tion that the risk of suicidal ideation can often be determined by only a few utterances. our approach also differs from previous hierarchical and multigrained approaches @xcite ) that seek to distill fine-grained information from every smaller piece of the entire input. in this work, we consider the suicidal ideation labels at the conversation level as the distant label and train a model to identify the discriminative utterances that lead to assessing the suicide risk of the caller. due to the lack of annotation at the utterance level, we introduce a novel relabeling mechanism that is unique to our distant-supervision method. the result is a final model that can efficiently perform suicidal ideation detection on short, condensed inputs. our approach is evaluated on the real-world data labeled by psychologists, showing that the selfadapted utterance selection successfully improves the performance in suicidal ideation detection. we also explore our approach in another domain. experimental results confirm that our approach is also effective in sentiment analysis, a typical nlp task. the contributions of this work are threefold: ## dataset we obtained the phone call recordings from taiwan lifeline international, and the audio recordings were transcribed by the experts with a psychology background. the transcriptions contain both counselors and callers' utterances, which are conversation-based forms. the transcriptions were further annotated by psychologists for assessing the caller's suicide risk level into five grades, including no suicidal ideation, occasional, frequent, ongoing suicide plan, and ongoing suicide attempt. the outlier conversations, such as not mention to suicide, cannot identify, statement not in person, and other situation will be removed. a high kappa value of 0.89 was measured, confirming a strong inter-rater agreement. due to the data sparsity, with experts' approval, we further merge the two weak ideation grades (i.e., no suicidal ideation and occasional) into low risk and the two strong ideation grades (i.e., ongoing suicide plan and ongoing suicide attempt) into high risk, resulting in three suicide risk levels. ## methodology given a conversation @xmath0, x 2 , ..., x n ), where x i is either the prompt made by the counselor or the response made by the caller repeatedly, an intuitive way to predict the level of the caller's suicidal ideation can be defined in equation 1 . as mentioned in section 1, x can be very long and noisy, resulting in a poorer performance. thus, our idea is to predict y given a short and condensed version of x. that is, we aim to approximate p (y|x) by p c (y|x), where x ⊆ x and p c (•) makes the prediction based on the subset of x. in section 4.1, we present a novel self-adapted utterance selection approach that finds the utterances providing useful information that is easier for the underlying backbone model to capture. section 4.2 shows the distant-supervised learning employed to train our model for utterance selection. section 4.3 describes the combination of our framework, and section 4.4 presents our model in the inference stage. figure 1(a) and figure 1(b) illustrate the details of our model in the training and the inference stages, respectively. figure 1(c) shows the dataflow of our framework in evaluation. ## experiments we employ five-fold cross validation, with a development set split 10% from the training set. significance testing is conducted using mcnemar's test. the value of τ is set at 0.5, and the adamw optimizer is utilized with a total of 40 epochs. the baseline models include the vanilla bert/longformer models in the end-to-end supervised learning, the multi-turn gru model, the jlpc model proposed by previous work for depression detection in interviews @xcite , the lda-based utterance selection method, reinforcement learning (rl) model, and the liwcbased methods applying gradient boosting decision tree (gbdt) and multilayer perception (mlp) respectively, which are widely-applied in psychological tasks. in the experiments, the input length with bert is 512 tokens, and with the longformer model is 2,048 tokens. the vanilla baseline is fed with the concatenation of all utterances formed as @xcite x counselor @xcite x caller @xcite . the gru model handles each turn in the conversation with the bert/longformer encoder separately and combines the information from the tokens with the gru layer. jlpc model employs the latent prompt categorization on the utterances of counselors and also makes the prediction on first hundred pairs content. the lda model performs the topic modeling for the conversation and clusters utterances into several groups. in brief, it is equivalent to let lda model replace p u (•) and p s (•) models to do the utterance selection. we select the utterances from each cluster to form the input to the bert/longformer model and report the best performance by one of them. our novel distant supervised approach is also compared with reinforcement learning @xcite . we simplify our model to p s (•) and p c (•) as the framework of reinforcement learning. the p s (•) as the agent is a binary classification to select the utterances according to the reward. the reward is defined as the difference from the loss of p c (•) between two continuous epochs. different from the models based on transformer, the liwc-based model represent the conversation as a 79-dimensional bag-of-word vector, where the value of each dimension indicates the occurrences of the words belonging to the corresponding chinese version of liwc (linguistic inquiry and word count) category. 2 for each conversation, we perform word segmentation and obtain the distribution of liwc categories. the mlp network is constructed for making the prediction as one of the baselines. gbdt classifiers which combine several weak learners and calculate with the gradient boosting decision tree algorithm are also adopted to accomplish the classification. ## results and discussion ## conclusion suicidal ideation detection is one of the core interests in the domain of intelligent healthcare. this work explores the task on a new kind of data, the lifeline conversations, and presents a novel selfadapted approach that condenses the long and noisy conversation for alleviating the model to identify the suicidal risk level of the caller. experimental results on real-world data show our approach is not only significantly superior to existing models but also capable of applying to other domains. our approach is also effective in the scenario of early detection, a practical issue for suicide prevention. with the advancement of large language models such as chatgpt, the development of intelligent mental health chatbots is becoming a reality in the near future. our detection model can be integrated into these chatbots to trigger an emergency response if a high suicide risk level is detected.