Archetypes and Entropy: Theory-Driven Extraction of Evidence for Suicide Risk (2024)

Abstract: Research on psychological risk factors for suicide has developed for decades. However, combining explainable theory with modern data-driven language model approaches is non-trivial. In this study, we propose and evaluate methods for identifying language patterns aligned with theories of suicide risk by combining theory-driven suicidal archetypes with language model-based and relative entropy-based approaches. Archetypes are based on prototypical statements that evince risk of suicidality while relative entropy considers the ratio of how unusual both a risk-familiar and unfamiliar model find the statements. While both approaches independently performed similarly, we find that combining the two significantly improved the performance in the shared task evaluations, yielding our combined system submission with a BERTScore Recall of 0.906. Consistent with the literature, we find that titles are highly informative as suicide risk evidence, despite the brevity. We conclude that a combination of theory- and data-driven methods are needed in the mental health space and can outperform more modern prompt-based methods.

## introduction with the advent of large language models (llms) @xcite , studies exploring their potential for estimating suicide risk from social media data have proliferated @xcite @xcite . such studies, however, chiefly emphasize predictive accuracy over explainability and interpretability @xcite , limiting both their clinical applicability and their utility in testing theories of suicide. our team, swell, takes a psychological theory-informed approach to produce evidential explanations and summaries for the assigned suicide risk score of reddit users. input: "things would be better if i was not here" despite substantial effort dedicated to extracting explanatory rationale for llm answers for math, physics, and even theory of mind @xcite @xcite , there has been limited work in building similar explanatory pipelines for mental health diagnostics. the clpsych-2024 shared task asked teams to provide evidences and summaries for suicide risk from social media posts @xcite . our main contributions include three novel methods for suicide risk evidence extraction based on (1) theory-based archetype representations of suicidality including with llama2-chat @xcite , (2) an llm-based relative entropy method, and (3) a hybrid combination of entropy with archetypes. additionally, we provide (4) a method for prompt-based explanation summaries, and (5) associations of theory-based archetypes with trained expert annotations. further, we release the code associated with our submissions. 1 2 background conceptualizing suicide risk. one of the most prominent theoretical conceptualizations of suicide is joiners' interpersonal theory of suicide (ipts) @xcite which is comprised of 3 factors that jointly characterize suicide risk: 1) acquired capability, a person's increased tolerance for physical pain and fear of death, which can develop over time through suicidal ideation and repeated exposure to painful or fear-inducing experiences @xcite ; 2) perceived burdensomeness, an individual's belief that their existence or presence is a burden on others @xcite ; 3) thwarted belongingness, the perception or experience of not belonging to, or feeling disconnected from, meaningful social relationships despite efforts to form connections @xcite . prior work suggests that suicide becomes possible when an individual experiences high levels of all 3 constructs @xcite . ## results we discuss the result of our experiments with the methods described in §4. we report the results of our official submissions in appendix a. 3 archetypes capture relevant highlights. entropy combined with archetypes further improves recall. we find that entropy-based methods have a high recall owing to better coverage of highlights signaling suicide risk, however, this comes at a small cost of precision, as seen in we further compare the performance boost afforded by using entropy-dgpt2 x archetypes against better modeling of user-level context using entropy-hart, selecting top 40% of sentences from both the methods as the suicide risk evidence. title of a post is highly informative. in llama2-chat is consistent with summarization. summaries generated by llama2-chat (13b) had high consistency and low contradiction scores across all submissions. this may have resulted from (1) the model's ability to precisely identify the suicide risk from appropriate psychological dimensions inferred from the span(s), and (2) a prompt carefully crafted to consider the important psychological dimensions to provide the summary. while these summarized explanations are more convincing for high-risk users, we also find that the model is extremely sensitive to the inputs. for example, posts with very few spans from low-risk users were still surmised to exhibit a "heightened risk of suicide." ## conclusion we combined theory-driven archetypes with datadriven language models to extract evidence from users' social media posts that support the assigned suicide risk levels. we found that scores derived for joiner's constructs show a significant correlation with assigned suicide risk. combining the relative entropy scores with joiner construct scores improves upon relative entropy alone, which is demonstrated in the experimental results on both the shared task test set as well as our set of internal annotations. these rigorous data-driven methods grounded in theory also outperformed extensive prompting of instruction-following llms. still, archetypes alone yield the highest precision in both evaluations. this demonstrates the importance of theoretically derived constructs in language modeling approaches to build explainable approaches for mental health diagnosis.