Extracting and Summarizing Evidence of Suicidal Ideation in Social Media Contents Using Large Language Models (2024)

Abstract: This paper explores the use of Large Language Models (LLMs) in analyzing social media content for mental health monitoring, specifically focusing on detecting and summarizing evidence of suicidal ideation. We utilized LLMs Mixtral7bx8 and Tulu-2-DPO-70B, applying diverse prompting strategies for effective content extraction and summarization. Our methodology included detailed analysis through Few-shot and Zero-shot learning, evaluating the ability of Chain-of-Thought and Direct prompting strategies. The study achieved notable success in the CLPsych 2024 shared task (ranked top for the evidence extraction task and second for the summarization task), demonstrating the potential of LLMs in mental health interventions and setting a precedent for future research in digital mental health monitoring.

## introduction large language models (llms) such as gpt (generative pre-trained transformer) @xcite have become cornerstones in the field of natural language processing domain. their ability to process and generate human-like text, learned from extensive datasets, empowers them to recognize and interpret complex language patterns on various reasoning tasks, such as arithmetic, commonsense, and symbolic reasoning @xcite . one of the critical abilities of llms is text span extraction from unstructured data, such as social media posts (e.g. reddit, twitter, facebook, etc.) @xcite @xcite . this process involves identifying and extracting specific segments of text that contain relevant information or unique characteristics. in the mental health context, this capability becomes indispensable for spotting signs of mental illness, such as depression, anxiety, and particularly suicidal thoughts or tendencies in online conversations. ## clpsych 2024 shared task and dataset the clpsych 2024 shared task @xcite centers on employing large language models (llms) to identify supporting evidence of an individual's suicide risk level from their social media posts. the challenge requires using an llm to extract and coherently present evidence from posts that align with the pre-assigned risk levels of low, moderate, or severe suicide risk @xcite . this task aims to utilize the generative capabilities of llms in producing supportive evidence for clinical assessments. for this shared task, we were provided umd suicidality dataset @xcite . this dataset includes social media posts from users on the reddit platform, specifically from the r/suicidewatch subreddit. these posts have been annotated with suicide risk levels by experts and crowdsource workers, categorizing them into no, low, moderate, or severe risk. participants are tasked with using llms to identify and extract text spans from these posts that support the assigned risk levels. this dataset provides a unique opportunity to explore the application of llms in mental health analysis, particularly in assessing and understanding suicide risk from online interactions. ## prompting strategies in this section, we explore various prompting strategies for text span extraction and summarization in the realm of mental health analysis, utilizing large language models (llms). focusing specifically on two llms, @xcite and tulu-2-dpo-70b @xcite , this part of the study examines how diverse prompting techniques can enhance the extraction and summarization of relevant information from large datasets in the context of mental health. ## result analysis in this section, we conduct a comprehensive analysis comparing zero-shot and few-shot prompting strategies, assessing their effectiveness in enabling in-context learning for text span extraction tasks. additionally, we examine the performance of direct prompting and chain-of-thought prompting strategies, highlighting their impact on precision and coherence in text span extraction. we also investigate how the inclusion of meta-information such as sentiment, emotion, and suicide risk levels influences the llms' ability to identify and summarize mental health-related content. this evaluation provides insights into the strengths and weaknesses of these strategies in the evidence extraction and summarization tasks. ## conclusion this study demonstrates the significant potential of large language models (llms) in mental health analysis, particularly in identifying and summarizing suicidal ideation from social media content. through the clpsych 2024 shared task, we successfully applied advanced llms using both fewshot and zero-shot prompting strategies, achieving notable performance in evidence extraction and summarization tasks. our findings emphasize the efficacy of llms in handling complex mental health data and highlight the impact of chain-of-thought (cot) prompting on evidence extraction. the study also highlights the importance of incorporating meta-information to enhance the evidence summarization tasks. our success in the clpsych 2024 shared task highlights the practical application of llms in mental health interventions, paving the way for future advancements in digital mental health monitoring.