X in H ai@ CLP sych 2024 Shared Task: Prompting Healthcare-oriented LLM s for Evidence Highlighting in Posts with Suicide Risk (2024)

Abstract: In this article, we introduce a new method for analyzing and summarizing posts from r/SuicideWatch on Reddit, overcoming the limitations of current techniques in processing complex mental health discussions online. Existing methods often struggle to accurately identify and contextualize subtle expressions of mental health problems, leading to inadequate support and intervention strategies. Our approach combines the open-source Large Language Model (LLM), fine-tuned with health-oriented knowledge, to effectively process Reddit posts. We also design prompts that focus on suicide-related statements, extracting key statements, and generating concise summaries that capture the core aspects of the discussions. The preliminary results indicate that our method improves the understanding of online suicide-related posts compared to existing methodologies.

## introduction suicide prevention is a key aspect of psychological research that addresses a critical need in mental health care. there have been different approaches to the research on suicide prevention. the suicide risk level prediction task uses machine learning algorithms to assess and predict suicide risk levels in individuals, offering a significant advancement in the field of mental health and preventive care. the recent evolution of large language models (llms) @xcite ) has brought about a paradigm shift in computer-based language understanding, profoundly improving the capacity to uncover latent meanings and intricacies within the language. in this paper, we explore an innovative approach that synergizes traditional natural language processing (nlp) techniques with the advanced capabilities of llm. our study explores the potential 2 task the clpsych 2024 shared task @xcite aims at utilizing llms for finding supporting evidence about an individual's suicide risk level. ## system our system is a pipeline built using a healthcareoriented llm, which accepts reddit posts from users at risk of suicide and prompts for formatted output to extract evidence accordingly. the overview of the system is shown in figure 1 . the implementation details and source code of our system are available in our online repository 1 . ## results figure 3 in the appendix and highlights metrics include recall, gauging the extent to which relevant evidence was captured. the precision metric assesses the accuracy of the evidence extracted. additionally, weighted recall provides insight into the length appropriateness of the evidence. lastly, the harmonic mean of precision and recall offers a balanced view of both metrics. summarized evidence is evaluated based on consistency, reflecting the absence of contradictions in the summaries compared to expert-written narratives. the contradiction metric further refines this analysis by penalizing any contradicting information, acknowledging the complexity inherent in texts that encompass both risk and protective factors. the table captures the essence of our comparative study, where phrase-level extractions (v3phrase and v4-phrase) were generally more precise-adept at identifying pivotal information, as reflected by their precision scores. sentence-level extractions (v2-sentence), while offering comprehensive insights, often included additional context that did not always contribute to the assessment's focus, as evidenced by the weighted recall scores. this distinction underscores the importance of selecting the appropriate extraction level depending on the desired balance between detail and breadth in evidence gathering. ## conclusion in conclusion, we constructed a prompt-based evidence highlighting and summarization system for the suicide risk evaluation task utilizing the healthcare-oriented llm xinhai. we utilized gpt-4 to further enhance our prompt design and conducted experiments at phrase-level and sentencelevel highlighting.