Integrating Supervised Extractive and Generative Language Models for Suicide Risk Evidence Summarization (2024)

Abstract: We propose a method that integrates supervised extractive and generative language models for providing supporting evidence of suicide risk in the CLPsych 2024 shared task. Our approach comprises three steps. Initially, we construct a BERT-based model for estimating sentence-level suicide risk and negative sentiment. Next, we precisely identify high suicide risk sentences by emphasizing elevated probabilities of both suicide risk and negative sentiment. Finally, we integrate generative summaries using the MentaLLaMa framework and extractive summaries from identified high suicide risk sentences and a specialized dictionary of suicidal risk words. SophiaADS, our team, achieved 1st place for highlight extraction and ranked 10th for summary generation, both based on recall and consistency metrics, respectively.

## introduction identifying suicide risk from online discussions is crucial problem. the 2018 and 2019 shared task at clpsych posed the task of predicting the level of suicide risk annotated by experts from reddit posts @xcite . in the 2024 shared task @xcite , the task is further developed to provide supporting evidence about an individual's suicide risk level on the basis of their linguistic content. there are two related subtasks. first subtask is to provide highlights; relevant evidence spans supporting the expert-assigned risk level. second subtask is to provide evidence summaries which synthesizes the identified evidence into insights that are consistent with human-written summaries. two main approaches for text summarization exist: extractive and generative. the extractive approach focuses on selecting significant portions of the original text, often using techniques like sentence extraction and machine learning-based sentence ranking @xcite . in contrast, the generative approach involves creating coherent summaries by understanding the context and meaning of the input text, employing advanced neural network architectures such as transformer models pretrained for language understanding and generation @xcite . the extractive approach excels in selecting crucial sentences based on supervised learning and explicit extraction criteria. in contrast, the generative approach is advantageous for understanding context and generating summaries without the need for explicit guidance. while the extractive approach struggles with the holistic contextual consideration, the generative approach faces challenges in reliably extracting desired evidence through prompt engineering. consequently, when clear criteria are present, the extractive approach is preferable; however, for generating contextually comprehensive summaries, the generative approach is more suitable. given the dual requirements of this year's shared task -identifying high suicide risk sentences and comprehensively considering various aspects of the entire post, including emotions, cognitions, behavior, and motivation -we propose an integrated method combining both extractive and generative approaches. our contributions include: (1) developing a bert (devlin et al., 2019) based model for sentence-level suicide risk and negative sentiment estimation. (2) identifying high suicide risk sentences precisely by focusing on elevated probabilities of both suicide risk and negative integrating supervised extractive and generative language models for suicide risk evidence summarization rika tanaka * yusuke fukazawa * graduate degree program of applied data sciences sophia university graduate school tokyo, japan sentiment. (3) integrating generative summaries using the mentallama (yang et al., 2023) framework and extractive summaries from identified high-risk sentences and a specialized suicidal risk words dictionary. the following sections detail our proposed method, results, and conclusion. figure 1: overall process of proposed model. ## proposed method the proposed method, outlined in fig. 1 , comprises two parts: highlight extraction and summary generation. to identify sentences indicating suicide risk, we employ a supervised extractive approach, leveraging bert's finetuning capabilities for enhanced contextual understanding. our model, fine-tuned on bert, estimates suicide risk and negative sentiment at the sentence level. for summary generation, we combine extractive and generative approaches. extractive summaries are crafted using patterns derived from high suicide risk sentences and associated keywords. generative summaries are produced using mentallama. the overall summary is an integration of both approaches. in the following, we detailed sentence level suicide risk classification, sentiment classification, highlight extraction, and summary generation. ## results organizers assess submitted highlights based on recall and precision, with recall measuring gold highlight prediction using bert-score semantic similarity @xcite . precision gauges the quality of predicted supporting evidence. summarized evidence is evaluated for consistency, indicating the absence of contradiction by calculating the probability of it conflicting with the gold summary. further details can be found in the paper @xcite . two highlight submission patterns were employed -one using only suicide risk classification and the other combining suicide risk classification, sentiment classification, and mentallama. in the summary generation subtask, it achieved the 10th rank with a consistency metric of 0.944. the lower score is attributed to two reasons: insufficient attention to consistency when integrating multiple summaries and the absence of prompt engineering to incorporate shared task background, relevant aspects, and evaluation metrics into the prompts, despite using simple prompts.