Can You Share Your Story? Modeling Clients’ Metacognition and Openness for LLM Therapist Evaluation (2025)

Abstract: Understanding clients’ thoughts and beliefs is fundamental in counseling, yet current evaluations of LLM therapists often fail to assess this ability. Existing evaluation methods rely on client simulators that clearly disclose internal states to the therapist, making it difficult to determine whether an LLM therapist can uncover unexpressed perspectives. To address this limitation, we introduce MindVoyager, a novel evaluation framework featuring a controllable and realistic client simulator which dynamically adapts itself based on the ongoing counseling session, offering a more realistic and challenging evaluation environment. We further introduce evaluation metrics that assess the exploration ability of LLM therapists by measuring their thorough understanding of client’s beliefs and thoughts.

## introduction recent advances in large language models (llms) @xcite have sparked interest in their potential applications for mental health and well-being @xcite . one promising area is the development of llm therapists, which are dialogue agents designed to conduct counseling sessions with clients interactively and provide therapeutic support and guidance @xcite @xcite . while there are various psychological counseling paradigms, the most fundamental and foundational stage in all paradigms is the exploration stage, where the therapist and client collaboratively uncover the client's experiences, emotions, and thought patterns @xcite meier and davis, * corresponding author figure 1 : llm-based user simulator vs. real-world user: user simulators are more prone to act overly compliant and self-aware in therapeutic sessions. to address these evaluation challenges, a realistic client simulator is needed which can emulate realistic client behaviors. specifically, there are two notable differences between real-world clients and simulated clients: (1) openness -willingness to share personal information and consider alternative perspectives. clients with high openness often disclose experiences and actively engage the exploration stage, while those with low openness tend to be guarded and reluctant to share. (2) metacognition -awareness of their own thoughts, emotions, and cognitive processes, as well as their ability to recognize throughout the exploration stage. a client with high metacognitive abilities can articulate themselves, whereas a client with low metacognition may struggle to recognize or verbalize their emotional states, leading to more stilted or guarded interactions. these two factors determine the difficulty of the experimentation stage, thus, client simulators aiming for real-world scenarios must incorporate these traits to effectively evaluate llm therapists. however, llms are typically optimized for fluent, coherent text generation, which can produce overly cooperative responses by default. as a result, simulating a realistically resistant or unaware client (i.e., client with low openness and low metacognition) is non-trivial. llms often fill in gaps with more detail than a reluctant, self-unaware client might provide, or they may inadvertently shift to cooperative, high-disclosure styles once the conversation becomes complex. hence, prompt engineering alone is insufficient for simulation of challenging clients. motivated by these limitations, we propose a novel evaluation framework that utilizes a controllable and realistic client simulator for assessing the exploration ability of llm therapists. our framework features a client simulator with adjustable levels of openness and metacognition. to closely emulate authentic client behaviors, our client simulator dynamically adjusts openness throughout the session in response to the rapport established with the llm therapist. also, depending on the level of metacognition, the speed at which the client simulator can identify their negative thoughts and core beliefs varies. with our framework, we evaluate whether llm therapists can help clients recognize and articulate their own experiences and thoughts. specifically, llm therapists conduct counseling sessions with our client simulator and are evaluated by comparing their understanding of clients -specifically core beliefs, intermediate beliefs, and potential early childhood memoriesand the ground-truth information. by focusing on the llm's capacity to explore thoughts and experiences, we aim to provide a robust measure of llm therapists' potential as a therapeutic tool. our contributions are two-fold: (1) we are the first to introduce a client simulator that dynamically adjusts the openness and metacognition throughout the session based on the quality of the counseling session, offering a more realistic and challenging evaluation environment. (2) we propose a novel evaluation framework mindvoyager that assesses the exploration ability of llm therapists by comprehensively comparing their understanding on multiple aspects of the client, ensuring the necessity of the dynamic user simulator. ## preliminaries most existing methods for evaluating llm therapists have adopted a self-play based approach, which utilizes llms as client simulators and simulates psychological counseling sessions by instructing llm therapists and client simulators @xcite . while the client simulator plays a vital role in the evaluation methods, the simulation ability of llms has been overlooked. this raises concerns about self-play based evaluation methods and also llm therapists evaluated by using naive client simulators. to this end, in this section, we analyze critical differences between llm-based client simulators and real-world clients. ## mindvoyager we propose mindvoyager, a framework for assessing the exploration ability of llm therapists utilizing a client simulator with adjustable metacognition and openness. ## conclusion in this work, we introduce a framework mindvoy-ager to evaluate the exploration ability of llm therapists by using a client simulator with dynamic openness and metacognition. mindvoyager consists of three phases.