Overview of the CLP sych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts (2024)

Abstract: We present the overview of the CLPsych 2024 Shared Task, focusing on leveraging open source Large Language Models (LLMs) for identifying textual evidence that supports the suicidal risk level of individuals on Reddit. In particular, given a Reddit user, their pre- determined suicide risk level (‘Low’, ‘Mod- erate’ or ‘High’) and all of their posts in the r/SuicideWatch subreddit, we frame the task of identifying relevant pieces of text in their posts supporting their suicidal classification in two ways: (a) on the basis of evidence highlighting (extracting sub-phrases of the posts) and (b) on the basis of generating a summary of such evidence. We annotate a sample of 125 users and introduce evaluation metrics based on (a) BERTScore and (b) natural language inference for the two sub-tasks, respectively. Finally, we provide an overview of the system submissions and summarise the key findings.

## introduction recent statistics on mental health related problems during and after the covid-19 pandemic are striking. in the us, almost 50% of adults aged 18-44 reported a mental illness in 2023, foot_0 whereas similar rates of the eu population had experienced emotional or psychosocial problems between june 2022-23. 2 partially due to the limited accessibility of support services, individuals often seek support in online social media by sharing their thoughts and concerns and engaging in discussions with their peers. research at the intersection of natural language processing (nlp) and mental health has focused on exploiting such user generated content in order to automatically detect vulnerable users @xcite @xcite or monitor their well-being over time @xcite . however, in real-world scenarios, detection is only part of the need: downstream evaluation and intervention would be facilitated by an understanding of why a user's text led them to be flagged @xcite . large language models (llms) @xcite @xcite are currently dominating the field of nlp. work at the intersection of nlp and mental health has leveraged such models for classification @xcite , data augmentation @xcite or reasoning @xcite , among others. recent research explores the language understanding and mental health reasoning capabilities of llms using instruction fine-tuning and chain-of-thought prompting (cot) @xcite . instead of direct phrase extraction, llms are instructed to provide step-by-step reasoning, leveraging inherent knowledge to generate humanlike language @xcite . such approaches pose the risk of incorrect predictions and flawed reasoning, especially in complex conversations @xcite . this year's clpsych shared task focused on leveraging open source llms for the purpose of finding evidence in online posts that supports the level of suicidal risk of their author. in particular, we define two sub-tasks (thereafter 'tasks') on the basis of (a) highlighting and (b) summarising such supporting evidence. working with the umd reddit suicidality dataset @xcite , we present the process of defining the task (section 3), selecting and annotating a subset of 125 reddit users (section 4), introducing our evaluation metrics (section 5) and summarising the approaches and the best-performing system of each team (section 7). in this overview paper we make the following contributions: ## task definition and instructions we define two tasks aimed at leveraging llms in order to find evidence within text that has been shared by particular online social media users supporting their pre-assigned suicide risk level ('low', 'moderate' or 'high'). the distinction between the two tasks is based on the way that this evidence is expected to be provided. task a for our first task, participants were asked to provide the evidence supporting the pre-defined suicide risk level of a user by highlighting relevant phrases within the text posted by the user. each user could have multiple posts in the dataset; task a was defined at the post (document) leveli.e., highlighting relevant phrases within each post made by a particular user. task b our second task required generating a summary of evidence supporting a user's assigned risk level, across multiple posts made by the user. as opposed to task a, task b was performed at the user level -i.e., generating a single summary per user. summaries were limited to 300 tokens. no ground truth data were provided to the teams, except for a single example of a user with a preassigned suicide risk level for whom we shared the expected highlights (task a) and summary (task b), as annotated by our experts (see section 4.2). compared to the latest clpsych shared tasks, where the expected outputs were a class label either at the user level @xcite or on a longitudinal basis @xcite , this year's edition was considerably more open-ended. we therefore provided a list of 'aspects to consider' to the teams, which were compiled on the basis of our internal annotation instructions. these aspects, which were based on literature on suicidal risk (see section 4.2), included, but were not limited to, the following: from feeling sad to experiencing unbearable psychological pain, the self-disclosed emotions of the user could play an important role in the risk level assigned to the individual. ## conclusion this work presented the overview of the clpsych shared task 2024, focusing on leveraging open source llms to find supporting textual evidence for the suicide risk level of an online user, based on their online posts. we defined two tasks for finding such evidence -based on (a) text highlighting of relevant spans at the post-level and (b) summarising the evidence at the user-level. we generated a dataset of 125 social media users to facilitate evaluation and introduced the associated evaluation metrics for measuring system performance. lastly, we have summarised the approaches taken by 13 teams and provided an overview of their results, their commonalities and novel aspects of their work.