ERD : A Framework for Improving LLM Reasoning for Cognitive Distortion Classification (2024)

Abstract: Improving the accessibility of psychotherapy with the aid of Large Language Models (LLMs) is garnering a significant attention in recent years. Recognizing cognitive distortions from the interviewee’s utterances can be an essential part of psychotherapy, especially for cognitive behavioral therapy. In this paper, we propose ERD, which improves LLM-based cognitive distortion classification performance with the aid of additional modules of (1) extracting the parts related to cognitive distortion, and (2) debating the reasoning steps by multiple agents. Our experimental results on a public dataset show that ERD improves the multi-class F1 score as well as binary specificity score. Regarding the latter score, it turns out that our method is effective in debiasing the baseline method which has high false positive rate, especially when the summary of multi-agent debate is provided to LLMs.

## introduction large language models (llms) are dominating the research areas in machine learning and artificial intelligence, broadening its usage in various applications @xcite @xcite @xcite . especially in the medical domain, palm @xcite and its variants, such as med-palm @xcite , are equipped with medical data and instructions to answer the questions from clinical field @xcite . in addition, conversational ai assistant chatbots are devised to support patients with mental health issues @xcite @xcite @xcite @xcite . recognizing the fact that individuals with mental disorders hesitate to seek in-person medical con-* equal contribution † corresponding authors sultations @xcite , previous studies @xcite @xcite attempt to enhance the accessibility and quality of psychotherapy through the use of llms with chain-of-thought (cot) reasoning @xcite . these models aim to detect the user's personality and interpret their mental state in order to generate more empathetic responses. for example, diagnosis-of-thought (dot) uses llms to classify cognitive distortions from utterances, which is a crucial part of cognitive behavior therapy (cbt) @xcite . while the dot method holds promise, one key challenge that remains an open issue is the tendency of the model to overdiagnose cognitive distortions, incorrectly inferring irrational thought patterns even when the user's statements are benign. in addition, the distortion classification performance of dot in multi-class setup is close to that of random guessing, which limits its usage in practice. in this paper, we tackle these issues by proposing a new framework for classifying cognitive distortions from the user utterances, by introducing modules for debiasing the overdiagnosing tendency of existing methods and for improving the performance on classifying distortion types inferred from the utterances. our main contributions can be summarized as below: ## erd we propose extraction-reasoning-debate (erd), a framework for classifying distortions in a given user speech, as shown in fig. 1 . the prompts we used can be found in figure 3 in appendix. below we elaborate each step in our framework. ## conclusion we introduce erd, a framework using llms to estimate the cognitive distortion contained in the user utterances through three steps: extracting distorted parts within the utterances, reasoning the estimation of the corresponding distortion classes, and debating the initial estimation using multiple agents. compared with existing baselines only having the reasoning step, including the extraction 4 295 and debating steps improve the distortion classification performance by 9% and improve the distortion assessment specificity by over 25%. such improvements is crucial to cognitive behavior therapy since erd is more adept at correctly identifying cases without distortions, avoiding the pitfall of over-diagnosing cognitive distortions. furthermore, experimental results reveal that we can control the behavior of erd with various prompting options. 7 298 figure 4 : the output of erd for a given speech, when the debate step is not activated. llm estimates that the speech contains cognitive distortion (type "labeling") even though it does not.