Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions (2025)

Abstract: We present an end-to-end framework for generating synthetic users for evaluating interactive agents designed to encourage positive behavior changes, such as in health and lifestyle coaching. The synthetic users are grounded in health and lifestyle conditions, specifically sleep and diabetes management in this study, to ensure realistic interactions with the health coaching agent. Synthetic users are created in two stages: first, structured data are generated grounded in real-world health and lifestyle factors in addition to basic demographics and behavioral attributes; second, full profiles of the synthetic users are developed conditioned on the structured data. Interactions between synthetic users and the coaching agent are simulated using generative agent-based models such as Concordia, or directly by prompting a language model. Using two independently-developed agents for sleep and diabetes coaching as case studies, the validity of this framework is demonstrated by analyzing the coaching agent’s understanding of the synthetic users’ needs and challenges. Finally, through multiple blinded evaluations of user-coach interactions by human experts, we demonstrate that our synthetic users with health and behavioral attributes more accurately portray real human users with the same attributes, compared to generic synthetic users not grounded in such attributes. The proposed framework lays the foundation for efficient development of conversational agents through extensive, realistic, and grounded simulated interactions.

## introduction personalization is a key capability for any interactive agents aiming to encourage positive behavior changes, for instance in health and lifestyle @xcite . the efficacy of such agents must be assessed via interactions with users; however, collecting and evaluating diverse, long-term human interactions with agents is costly and time-consuming, and the development of such agents can be significantly accelerated with realistic simulation. we address the challenge of realistic user simulation in this context by proposing an end-to-end framework for generating cohorts of synthetic users grounded in health, lifestyle, and behavior, and having them interact with coaching agents. the framework is generally applicable across a range of health and lifestyle domains; we validate it in the domains of sleep and diabetes coaching. synthetic users generated with large language models (llms) have been widely studied in recent years @xcite @xcite @xcite , often for general purpose alignment and personalization @xcite , and have been designed to reflect demographic information about the population at large. we focus on a more targeted application, where the synthetic users are employed for interaction with foundation models or agents specialized in health, as done in recent work @xcite @xcite . we provide an overview of such work in section 2. we demonstrate an end-to-end-framework for designing a cohort of synthetic users exhibiting health conditions consistent with their specific demographics and behavioral factors, in order for their interaction with a coaching agent to realistically reflect their needs and challenges. we construct the synthetic users by first generating a sample of natural language "vignettes", which are in turn generated based on real demographics, health, and behavioral data (such as the big five markers; @xcite . our framework also allows for optionally including additional information: domain-specific rubrics such as "barriers" and "goals" as used by human coaches; challenges to action derived from the com-b behavioural model @xcite ; or rich backstories for the synthetic users, generated consistently with the structured attributes from real data. finally, based on this cohort of vignettes, realistic user-coach simulation can be performed by direct llm calls or by using generative agentbased models such as the concordia @xcite open-source system. more details will be discussed in section 3. we demonstrate the efficacy of our methodology for generating health-grounded synthetic personas in two separate domains, sleep coaching (section 4) and diabetes coaching (section 5), using two independently developed sleep coaching and diabetes coaching agents. we verify that our synthetic users indeed communicate behaviors, health conditions, and barriers consistent with their assigned attributes, by inspecting the internal state-of-user model of the coaching agent for each synthetic user, by a holistic evaluation of the user-coach interactions by trained human experts, and by a comparative evaluation of our health-grounded synthetic users against generic synthetic users, again by trained human experts. ## background in this section, we briefly survey the development and goals of llm-based synthetic personas in general, and then provide further details on prior literature on the design and application of synthetic personas for health. we next describe software systems for simulating agent interactions. finally, we discuss some challenges in using llm-derived synthetic users for agent development and beyond. ## methods while llms trained on a large data corpus (e.g., the entire internet) partially represent the general human population, the distributions of subpopulations and health conditions represented in a single model are skewed and poorly understood @xcite . furthermore, the instruction fine-tuning step (e.g., reinforcement learning with human feedback) typically employed by highperformance llms further impacts the distribution of generated outputs. in this work we propose an end-to-end framework for generating synthetic users grounded in reality, by starting from human user data including demographics, health & lifestyle, and behavioral & psychological characteristics, to accurately represent a desired population (figure 1 ). this real cohort can be sampled either uniformly, or with oversampling or undersampling explicitly designed by researchers (e.g., to better surface underrepresented subgroups or conditions). given this sampled cohort, we optionally add additional health conditions conditioned on already specified attributes of the synthetic users, either by using the llm or via a rule-based algorithm, and also optionally add rich backstories of the synthetic user, conditioned on existing attributes @xcite . this combination of background information grounded in real data constitutes a "vignette" of the synthetic user, all based in natural language. in the following sections we explore how these vignettes can be used to generate interactions between synthetic users and coaching agents, in two separate health coaching scenarios-sleep coaching and diabetes coaching-with two independently developed coaching agents. ## experiments: sleep coaching sleep is vital for health and well-being; insufficient sleep and untreated sleep disorders can have a detrimental effect on cognitive function, mood, mental health, and cardiovascular, cerebrovascular, and metabolic health @xcite . sleep disorders are highly prevalent: according to the 2020 national health interview survey (nhis) (adjaye-gbewonyo et al., 2022), 14.5% of us adults had trouble falling asleep most days over the 30-day study period, and over a quarter of adults do not meet the minimum recommended sleep duration per night. ## experiments: diabetes coaching diabetes mellitus, commonly referred to as diabetes, is a highly prevalent chronic disease, affecting 15% of all us adults in 2021 (cdc, 2024). understanding lifestyle barriers is crucial for effectively addressing the challenges this condition presents @xcite . the barriers significantly impact an individual's ability to adhere to recommended behaviors and achieve optimal health outcomes. many barriers are intricately driven by demographic, socioeconomic, and clinical circumstances, collectively manifesting into specific challenges. by accurately representing these interconnections, we can create synthetic user profiles that faithfully reflect the real-world complexities encountered by individuals with cardiometabolic conditions, thereby enhancing the realism and applicability of our synthetic users interacting with a diabetes coaching agent (figure 1 ). ## conclusion we designed an end-to-end framework for generating synthetic users for evaluating coaching agents grounded in health, lifestyle, behavioral, and psychological attributes complementing basic demographics from human data. the additional sampling process was employed to ensure that the distribution of these attributes could be explicitly controlled (i.e., to ensure applicability to less represented sub-populations). realistic backstories and additional health conditions (where not available in human user data) were conditionally generated from the grounded data. the final vignettes generated by this framework were used in a generative agent-based model framework to effectively simulate interactions between the synthetic users and the given coaching agent. we evaluated our methods in two independent, highly prevalent health coaching use cases of sleep coaching and diabetes management. efficient development of autonomous agents is significantly accelerated by evaluation metrics that can be computed without complete dependence on human users, since human evaluation is costly and time-consuming. while there is a large body of literature on the design of general-purpose synthetic users in this setting (see section 2), we have highlighted the importance of grounding synthetic users' health and behavioral attributes on a real human dataset, with concrete demonstrations and evaluations of our end-to-end pipeline in two independently developed coaching agents. although some of our demonstrations relied on access to the coaching agents' internal user model, the simulated interactions generated by our framework can be evaluated in a way that is agnostic to the design of the agent, so the framework can be applied to any conversational agents. the extensive, realistic, and grounded simulated interactions produced by our framework lay the foundation for efficient development of coaching agents, potentially through a fine-tuning or reinforcement learning loop, as will be pursued in future research.