Cheap Ways of Extracting Clinical Markers from Texts (2024)

Abstract: This paper describes the Unibuc Archaeology team work for CLPsych’s 2024 Shared Task that involved finding evidence within the text supporting the assigned suicide risk level. Two types of evidence were required: highlights (extracting relevant spans within the text) and summaries (aggregating evidence into a synthesis). Our work focuses on evaluating Large Language Models (LLM) as opposed to an alternative method that is much more memory and resource efficient. The first approach employs an LLM that is used for generating the summaries and is guided to provide sequences of text indicating suicidal tendencies through a processing chain for highlights. The second approach involves implementing a good old-fashioned machine learning tf-idf with a logistic regression classifier, whose representative features we use to extract relevant highlights.

## introduction suicidal-themed messages on social media platforms can represent an indicator of suffering and mental health issues. according to @xcite , 6% of individuals aged 18-25 responded affirmatively to the survey questions on suicide ideation. interdisciplinary work on psychology and computational linguistics @xcite uses statistical models to identify various risks based on the content of social media posts or based on multi-modal characteristics such as time of post, user gender and class @xcite . gaining awareness of the risk of suicide is essential, as it allows state organizations to offer support to those in need, and consequently, preventive measures can be taken, potentially saving the lives of those contemplating suicide. therefore, it may be beneficial from multiple perspectives to develop methods through which the presence of suicidal thoughts can be determined on the basis of text posts on social networks. however, as @xcite suggests, relying solely on algorithmic methods can introduce biases, risks, and, ultimately, case-by-case analyses must be carried out by experts. in this paper, as part of the shared task of the 2024 workshop on computational linguistics and clinical psychology @xcite , we address the identification of suicidal evidence in users' posts on reddit by extracting phrases, expressions, key-words, and various types of summaries that can explain such labels. the shared task has been framed from the perspective of large language models (llms) with a suggestive title in this sense: "utilising llms for finding supporting evidence about an individual's suicide risk level". although llms are the current standard in natural language processing @xcite , deploying such models at scale can be prohibitively expensive, while the pre-trainig can often be resource-and data-intensive, making such models available only for well-resourced languages and large research laboratories. we address this task from the perspective of finding solutions for fast inference, and propose two variants: 1) to create a straightforward and cheap (as in time-efficient) pipeline for training and identifying suicidal evidence and 2) to use prompting with quantized llms @xcite executed locally on cpu. the former is based on traditional machine learning classification techniques consisting of a tf-idf vectorizer over word ngrams paired with a feature importance selection process from a linear logistic regression classifier. our results in the shared task show that a machine learning pipeline can achieve competitive evaluation scores (top 3 recall) by leveraging the risk assement annotations from the provided dataset @xcite . however, our best-performing model is a combination of llms used to generate good-quality summarizations and machine learning to detect highlights. ## data analysis the annotated data provided for the shared task participants is identical to the previous edition clpsych 2019 shared task: predicting the degree of suicide risk in reddit posts @xcite and here we include a brief summary of its subdivisions: task a: users on r/suicidewatch reddit annotated based on their risk level across multiple posts using crowdsourced annotations. expert: user posts annotated by experts of different specialties. tasks b and c of annotations that we did not use in this work. all data annotations contain suicide risk categories (corbitt-hall et al., 2016) marked with letters signifying different degrees: (a) no risk, (b) low, (c) moderate, and (d) severe risk. expert data is of higher quality, it consists of 332 posts, the majority (49%) are labeled medium risk, followed by 28% high risk and 23% low risk. the 2024 shared task @xcite evaluation data (not released to participants) contains additional annotations of suicide risk evidence (highlights and summaries) for 125 users of the expert subset. our work only uses task a and the expert subsets. ## good old-fashioned machine learning (goml) the first approach, which also obtained the highest recall amongst submissions, is based on the following steps. tokenization is done using a regular expression of the form r'\b[^\d\w]+\b' and we employ a range of n-grams between 2 and 4 words. we crossvalidate several models on different subsamples of risk annotations labeled as follows: 1.1 test -a model trained solely on task a test set (186 posts), 1.2 taska a model trained on the entire task a, and 1.3 a+e a model trained on both expert and taska data. ## language models for efficient text generation, we use a 4-bit quantized model (q4_k_m) together with llama-cpp foot_1 and langchain @xcite libraries. we use openhermes 2.5 based on mistral @xcite that has been fine-tuned on code. according to the authors foot_2 training on a good ratio of code instruction of around 7-14% of the total dataset boosted several noncode benchmarks, including truthfulqa, agieval, and gpt4all suite. the language models approach can be summarized in the following steps: to extract summaries, we run the model only once with the following prompt: as a psychologist and expert therapist, summarize the content by identifying any indications of suicidal thoughts. provide evidence from the text to support your analysis. \n \n post body: {content_body}\n \n analysis:. when using goml with option 4.2, the content body consists in the concatenation of important sentences instead of the post bodies. we found that the model tends to hallucinate and copy paste content from the text, unless the word analysis is explicitly mentioned at the end. ## results and discussion our three official submissions for the shared task in this order are: recall is computed as the average of the maximal semantic similarity between each gold highlight and all predicted highlights based on bertscore @xcite . a point of critique that we can raise here is that introducing duplicate highlights of different sizes will generate a better overall recall score. in practice, such a system could potentially slow down an expert looking for indicators of suicide. for example, our submission "llm duplicates" from for this downstream task of identifying highlights, we did not observe significant improvements in performance when training the logistic regression classifier with more data, nor did we observe a degradation of performance when training on the smallest amount of samples consisting only of the test set of task a. this is encouraging for potential extensions of the goml methodology to less-resourced languages. the generated summaries are evaluated by taking the probability scores (from an external nli tool) of having a summary that contradicts the gold sentence as a premise. in terms of consistency and contradiction while goml performs competitively to more resource-intensive approaches in detecting highlights, the same cannot be said about summaries. our test 1 model that used textrank for extractive summarization obtained one of the worst contradiction and consistency scores in the entire competition. its main advantage remains that it can run the entire machine learning pipeline to train the classifier and generate all the evidence (highlights and summaries) for the 125 users in less than 60 seconds. in contrast, our quantized llm on cpu runs in 3.5 hours for the same set of users. to be consistent with our comparisons, in all of our approaches, we have only used a cpu server with 7 cores and 64 gb of memory to compute the results. given the surprising efficacy of the traditional machine learning model, we ask whether sentences containing important features have specific linguistic characteristics. sentences are divided into two categories: important if they contain important features for classification and with the label other otherwise. our statistical analyses visible also in figure 2 indicate that important sentences are generally more likely to have pronouns, verbs, and adjectives. in terms of mean value, pronouns and verbs are statistically different at a p-@xmath0.05 in important sentences more often than in the rest. similarly, mean sentence lengths are statistically larger in important sentences than in the other ones. adverbs show no difference between the two classes, and adjectives and nouns obtain a pvalue of 0.6 after 100,000 permutations. given the nature of permutation tests, this is equivalent to saying that there is a 6% chance of observing a difference in means for adjectives and nouns due to chance. our brief analyses show that important sentences have different (statistically significant) linguistic patterns that can distinguish them from the rest. we believe that this could be one of the reasons behind the good evaluation scores and the suitability of the goml approach to extract highlights from this particular dataset. ## conclusions to conclude, our results show that a classifier paired with a machine learning explainability method can be a useful tool for identifying important sentences, phrases, and highlights that are representative of a given class. this is encouraging for languages where current llms do not perform as well or where the amount of data and compute resources is limited. additionally, our experiments show that noisy generated output containing duplicates achieves better recall, leading to the conclusion that relying on a single metric can be detrimental to this task. we believe that ultimately expert human judgments would be the best measure for evaluating and selecting the most useful systems based on multiple criteria. in general, when investigating the output of llm-based approaches, we could observe better quality in terms of the generated text and langchain reasoning. our work shows that these results can be further improved by combining llms with good old-fashioned machine learning methods.