How Real Are Synthetic Therapy Conversations? Evaluating Fidelity in Prolonged Exposure Dialogues (2025)

Abstract: Synthetic data adoption in healthcare is driven by privacy concerns, data access limitations, and high annotation costs. We explore synthetic Prolonged Exposure (PE) therapy conversations for PTSD as a scalable alternative for training clinical models. We systematically compare real and synthetic dialogues using linguistic, structural, and protocol-specific metrics like turn-taking and treatment fidelity. We introduce and evaluate PE-specific metrics, offering a novel framework for assessing clinical fidelity beyond surface fluency. Our findings show that while synthetic data successfully mitigates data scarcity and protects privacy, capturing the most subtle therapeutic dynamics remains a complex challenge. Synthetic dialogues successfully replicate key linguistic features of real conversations, for instance, achieving a similar Readability Score (89.2 vs. 88.1), while showing differences in some key fidelity markers like distress monitoring. This comparison highlights the need for fidelity-aware metrics that go beyond surface fluency to identify clinically significant nuances. Our model-agnostic framework is a critical tool for developers and clinicians to benchmark generative model fidelity before deployment in sensitive applications. Our findings help clarify where synthetic data can effectively complement real-world datasets, while also identifying areas for future refinement.

## introduction training machine learning models in sensitive domains like healthcare remains a challenge @xcite . access to real clinical conversations, which are crucial for modeling tasks like mental health diagnostics and therapeutic dialogue understanding, is severely limited by high annotation costs, patient privacy concerns @xcite @xcite , and ethical constraints on data sharing. synthetic data has emerged as a promising alternative, offering scalability and privacy preservation while reducing dependence on real-world annotations @xcite . in trauma-focused mental health care, particularly prolonged exposure (pe) therapy for ptsd, large language models (llms) can generate synthetic therapy dialogues at scale. however, questions remain about whether these dialogues capture more than surface-level fluency, specifically the subtle dynamics of therapeutic fidelity such as emotional pacing, avoidance management, and protocol adherence @xcite . to address this, we develop and validate methods to measure clinically relevant fidelity in generated dialogues, moving beyond standard metrics like coherence or perplexity. prior work shows that while synthetic pe sessions can convincingly mimic real sessions in tone and structure @xcite , they may still commit fidelity lapses, such as premature reflection or reinforcement of avoidance, which often go unnoticed by both automatic metrics and nonclinical annotators @xcite @xcite . for instance, a therapist saying "that's a really powerful insight" mid-exposure may appear empathic but violates pe protocol by derailing trauma processing (see fig. 1 ). without clinical expertise, both humans and automated metrics tend to overestimate fidelity. we introduce a fidelity-focused lens that evaluates dialogues on multiple dimensions: linguistic coherence, adherence to pe protocols, and the therapist's navigation of key clinical interactions (e.g., managing avoidance, suds (subjective units of distress) monitoring). this framework integrates automated scoring and expert clinical assessment for a more rigorous evaluation of synthetic dialogue quality. we present the first large-scale, multidimensional comparison of real and synthetic pe therapy sessions, analyzing their linguistic, structural, and clinical fidelity characteristics. our findings demonstrate the current strengths and limitations of synthetic dialogues and inform future improvements in generation and evaluation methods. our study contributes to bridging the real-synthetic data gap and provides a roadmap for advancing synthetic dialogue in sensitive healthcare domains. ## motivation developing synthetic clinical dialogues is essential to overcome persistent barriers in mental health ai. in the context of prolonged exposure (pe) therapy, four challenges stand out: (1) data scarcity, due to the high cost and effort of annotation; (2) privacy constraints, which limit access to sensitive patient narratives; (3) lack of diversity, with datasets often failing to capture varied trauma types and demographics; and (4) evaluation inconsistency, as fidelity assessment lacks standardized benchmarks. this work directly addresses these issues by generating and evaluating synthetic pe dialogues that preserve protocol fidelity, enable scalable annotation, and support more equitable and robust training data for mental health applications. ## related work while synthetic clinical conversation datasets have advanced evaluations for general counseling and cbt, work specific to prolonged exposure (pe) and structured trauma-focused therapy is limited. most studies focus on (1) synthetic dataset generation, (2) evaluation metrics beyond lexical similarity, and (3) human-in-the-loop validation, but generally lack pe-specific considerations. ## methodology we analyzed 400 prolonged exposure therapy conversations, consisting of 200 real-world and 200 synthetic. to demonstrate our evaluation framework, this case study uses 200 synthetic conversations from the publicly available thousand voices of trauma dataset @xcite . the data was generated using claude sonnet 3.5 @xcite with pe-specific prompting frameworks adapted from clinical guidelines. this dataset includes diverse therapist-client interactions across trauma types, therapy phases, and demographics. the real-world pe sessions were collected under irb-approved protocols, with participant consent, and cannot be released due to privacy constraints. each session (1-1.5 hours) was transcribed using amazon healthscribe (services, 2023), manually verified, and reformatted by merging consecutive speaker turns for readability. each conversation followed a validated therapy fidelity checklist (see all evaluation code, pre-processing pipelines, and metric definitions will be released to support reproducibility and external validation. ## discussion this paper evaluated the fidelity of synthetic pe therapy conversations by comparing them to real interactions using linguistic, structural, and statistical analyses. we discuss the findings from four key perspectives: system-level metrics and correlation ( ## future work future nlp research should focus on developing generative models with improved capabilities for tracking long-range dependencies and emotional dynamics (e.g., for emotional habituation, suds progression), and creating automated metrics or classifiers trained with clinical expert annotations to better assess therapeutic process alignment. ## conclusions this study evaluated synthetic therapeutic dialogues, showing that while they replicate structural features like turn-taking, they fall short in utterance length and conversational variability. statistical analyses confirm these gaps, revealing that surface-level fluency can mask clinically meaningful fidelity lapses. we contribute a fidelity-aware evaluation framework tailored to prolonged exposure (pe) therapy, along with metrics highlighting that current generative models, despite their fluency, lack the nuance needed for high-stakes therapeutic contexts. the metrics and methods in this framework are model-agnostic and can be readily applied by the research community to benchmark and improve dialogues generated from any large language model. our findings underscore the risk of overestimating quality through non-expert or automated evaluations. for nlp, this work emphasizes the need for clinically grounded benchmarks, richer linguistic modeling, and methods like expertin-the-loop validation and time-aware metrics to improve fidelity in domain-specific generation.