From Conversation to Automation: Leveraging LLM s for Problem-Solving Therapy Analysis (2025)

Abstract: Problem-Solving Therapy (PST) is a structured psychological approach that helps individuals manage stress and resolve personal issues by guiding them through problem identification, solution brainstorming, decision-making, and outcome evaluation. As mental health care increasingly adopts technologies like chatbots and large language models (LLMs), it is important to thoroughly understand how each session of PST is conducted before attempting to automate it. We developed a comprehensive framework for PST annotation using established PST Core Strategies and a set of novel Facilitative Strategies to analyze a corpus of real-world therapy transcripts to determine which strategies are most prevalent. Using various LLMs and transformer-based models, we found that GPT-4o outperformed all models, achieving the highest accuracy (0.76) in identifying all strategies. To gain deeper insights, we examined how strategies are applied by analyzing Therapeutic Dynamics (autonomy, self-disclosure, and metaphor), and linguistic patterns within our labeled data. Our research highlights LLMs’ potential to automate therapy dialogue analysis, offering a scalable tool for mental health interventions. Our framework enhances PST by improving accessibility, effectiveness, and personalized support for therapists.

## introduction in recent years, modern stressors like covid-19, global conflicts, and climate disasters have led to rising mental health conditions, especially among adults @xcite . in response, the world health organization (who) calls for urgent transformation in mental health care @xcite . problem-solving therapy (pst) helps manage these stressors by enhancing problemsolving skills, reducing psychological distress, and strengthening coping mechanisms @xcite @xcite . it seems we're alike in trying to do it all, thinking it's best if we handle everything. learning to delegate is tough but freeing when we get it right. have you found success in delegating? yes, i have. it was a relief discussing it with my family and seeing it work out. given your worries and frustrations, have you thought of ways to help yourself feel less burdened by the situation? i've accepted what needs to happen, like needing outside help. it's about moving forward without letting it bother me. figure 1 : a real-life example of therapist and client exchanges in a problem-solving therapy session. as rapidly evolving trends continue to shape the critical area of mental health, systematic methods for analyzing pst interactions remain limited @xcite . this gap is especially relevant with the rise of large language model (llm)-powered mental health chatbots @xcite . before widespread adoption, it is crucial to assess how pst strategies are applied in practice, as therapists' approaches vary based on client needs and contexts @xcite . pst follows a structured process, identifying problems, brainstorming solutions, decision-making, and evaluation, to strengthen clients' problem-solving skills and coping abilities @xcite . while new technologies hold promise, their adaptability and responsiveness must be ensured for effective pst automation @xcite . previous work leveraged llms for analyzing therapist-client interactions @xcite . for instance, @xcite llms, particularly chatgpt, to identify utterancelevel features in therapist-client conversations to classify counselor's utterances. in pst, such research in automated annotations is still lacking due to limited data and costly domain expert annotation. automating the annotation of various cogni-tive therapy techniques is crucial for developing efficient models, like llms, that can navigate the complexity of therapeutic dialogues and identify effective interventions. this capability could enable real-time suggestions to therapists or direct interventions in automated systems. our work addresses this gap by using llms to automate pst annotation, using both closed-and open-source models to categorize therapist utterances. we analyzed 240 real-world anonymized pst sessions (68,306 dialogue exchanges), applying llms to annotate strategies based on existing and adapted pst and facilitative strategy codebooks. our results show that llms efficiently and accurately annotate pst strategies. we also analyzed how these strategies evolve across sessions finding that therapists progressively shift from exploratory strategies, such as defining problems and brainstorming solutions, to strategies focused on implementation and evaluation. additionally, we analyze therapeutic dynamics (autonomy, selfdisclosure, metaphor) and linguistic properties in all utterances to understand how these strategies were applied. our results show that therapists adapt their language to session goals, using more non-directive language and metaphors in the early stages and transitioning to directive language and concrete communication as therapy progresses. this approach enhances therapist-client interaction analysis, supports scalable and adaptive mental health interventions, and enables real-time identification of effective therapeutic techniques. our framework advances nlp and mental health care, showing llms' potential to transform pst research. finally, integrating facilitative strategies refines the pst framework, offering deeper insights into targeted therapeutic interventions. all codes used in this study are available in our github repository. foot_0 2 related work ## conclusion this study highlights the potential of llms like gpt-4o for effective annotation of therapeutic di-alogues. gpt-4o achieved a notable f1 score of 0.76, with a low entropy value of 0.035, demonstrating its efficiency and reliability in categorizing pst strategies. we fine-tuned transformer-based models on a sample of 5,000 therapist utterances annotated by gpt-4o and showed that modernbert exceeded the performance of our second-best llm, gpt-4, showcasing the effectiveness of using llmgenerated annotations to fine-tune smaller, domainspecific models. we also expanded the pst codebook to include a new facilitators dimension, enriching our analysis of therapist-client interactions. we also analyzed therapeutic dynamics (autonomy, self-disclosure, and metaphor) and linguistic patterns in our data to better understand 'how' pst strategies are used and in what contexts. our findings showed that all models performed better at identifying 'ps core' strategies, indicating a strong capability in recognizing structured therapeutic elements. our research offers valuable insights into enhancing technology integration in mental health care, aiming to make therapeutic interventions both more accessible and more effective.