Eeyore: Realistic Depression Simulation via Expert-in-the-Loop Supervised and Preference Optimization (2025)

Abstract: Large Language Models (LLMs) have been previously explored for mental healthcare training and therapy client simulation, but they still fall short in authentically capturing diverse client traits and psychological conditions. We introduce Eeyore , an 8B model optimized for realistic depression simulation through a structured alignment framework, incorporating expert input at every stage.First, we systematically curate real-world depression-related conversations, extracting depressive traits to guide data filtering and psychological profile construction, and use this dataset to instruction-tune Eeyore for profile adherence. Next, to further enhance realism, Eeyore undergoes iterative preference optimization—first leveraging model-generated preferences and then calibrating with a small set of expert-annotated preferences.Throughout the entire pipeline, we actively collaborate with domain experts, developing interactive interfaces to validate trait extraction and iteratively refine structured psychological profiles for clinically meaningful role-play customization.Despite its smaller model size, the Eeyore depression simulation outperforms GPT-4o with SOTA prompting strategies, both in linguistic authenticity and profile adherence.

## introduction psychological science, like other scientific domains such as chemistry, physics, medicine, and neuroscience @xcite @xcite , has increasingly recognized the transformative power of large language models (llms) to advance the field @xcite . recent studies have shown that llms can support psychology in areas like measurement @xcite , experimentation @xcite , and clinical practice @xcite . in particular, leveraging the role-playing capabilities of 1 eeyore and all annotated data are open-sourced at https: //github.com/michigannlp/eeyore . llms to simulate therapy-related roles, for example, a client with ongoing depression, has shown promise in helping novice counselors or psychiatrists practice their clinical skills @xcite . however, despite their promise, existing llmdriven simulations face limitations that hinder their adoption in professional clinical training. current approaches rely heavily on prompt engineering @xcite @xcite @xcite , which cannot overcome the inherent biases and structural constraints of general-purpose llms @xcite . recent studies have raised concerns about the validity of using llms for clinical training, particularly regarding their inability to authentically represent patient experiences and their tendency to generate overly sanitized or misleading responses @xcite @xcite @xcite . these concerns highlight the need for a structured approach that moves beyond generic prompting strategies. in this work, we develop a structured alignment framework to optimize llms for capturing the language, cognitive patterns, and experiential traits of individuals with depression in clinical training scenarios. as outlined in figure 1 , our framework integrates three specialized alignment endeavors in a sequential pipeline, incorporating expert feedback at each stage. the three key innovations in our framework are: language-specific alignment. as noted by @xcite , biases in training data can undermine the authenticity of simulated patient interactions. general-purpose llms (e.g., gpt-4) are not optimized on specialized data, which creates an inherent ceiling on simulating depressive speech patterns (e.g., self-harm ideation, or cognitive distortions), even with care-figure 1 : the alignment pipeline for optimizing llms to simulate individuals with depression in clinical training. expert involvement is highlighted in green. icons by @xcite . profile-guided role-playing via instruction-tuning. depression manifests uniquely in each individual, and clinical training requires exposure to varied cases of depression for customized practice. to achieve this, we structure each conversation in our dataset with a corresponding psychological profile that encodes important information about depressive traits. these profiles undergo iterative refinement through expert critiques to ensure clinical accuracy and relevance. we instructiontune an llm using system prompts that specify the client's profile and conversation context, allowing it to role-play with consistency and realism across different depressive manifestations. this approach lets practitioners engage with a broad spectrum of depressive profiles, mirroring real-world variations in symptoms and experiences. ## methodology our framework, illustrated in figure 1 , consists of three stages: (1) language-specific alignment, where we curate a dataset of depression-related conversations with structured psychological profiles; (2) profile-guided role-playing, where we instruction-tune the model for realistic profile adherence; and (3) iterative preference optimization, where we refine the model via model-generated and expert-annotated preferences. ## experiment we evaluate eeyore within both human and automatic evaluation, comparing its performance to state-of-the-art baselines for patient simulation in mental health support. all evaluations are conducted in an online testing setting, ensuring realtime interaction between evaluators and chatbots. ## conclusion we introduced eeyore, a model optimized for realistic depression simulation through a structured alignment framework. expert involvement is central to our pipeline, guiding data curation, profile refinement, and preference optimization to align the model with clinical expectations. evaluations demonstrated that eeyore outperforms state-of-theart baselines in linguistic authenticity and profile adherence. our work highlights the importance of structured optimization and expert collaboration in llm-based patient simulation.