Evaluating LLM -Generated Versus Human-Authored Responses in Role-Play Dialogues (2025)

Abstract: Evaluating large language models (LLMs) in long-form, knowledge-grounded role-play dialogues remains challenging. This study compares LLM-generated and human-authored responses in multi-turn professional training simulations through human evaluation (N = 38) and automated LLM-as-a-judge assessment. Human evaluation revealed significant degradation in LLM-generated response quality across turns, particularly in naturalness, context maintenance and overall quality, while human-authored responses progressively improved. In line with this finding, participants also indicated a consistent preference for human-authored dialogue. These human judgements were validated by our automated LLM-as-a-judge evaluation, where GEMINI 2.0 FLASH achieved strong alignment with human evaluators on both zero-shot pairwise preference and stochastic 6-shot construct ratings, confirming the widening quality gap between LLM and human responses over time. Our work contributes a multi-turn benchmark exposing LLM degradation in knowledge-grounded role-play dialogues and provides a validated hybrid evaluation framework to guide the reliable integration of LLMs in training simulations.

## introduction the rapid advancement of large language models (llms) has led to their increasing application in role-play dialogue systems across diverse domains, such as healthcare @xcite and education @xcite . a key application of these systems is professional skills training in reallife scenarios. for instance, @xcite integrated an llm to role-play as a virtual patient, generating responses with realistic medical symptoms for clinical training. similarly, in education, llm-based pedagogical agents enhance students' collaborative learning @xcite . such training simulations require that llms not only generate contextually appropriate responses but also remain grounded in given character profiles and domain-specific knowledge. furthermore, these interactions are often guided by pedagogical goals for an immersive and effective training experience @xcite . however, despite the growing deployment of llms in these applications, their evaluation remains a significant challenge @xcite . a critical gap exists in the availability of high-quality open-source datasets and benchmarks designed for knowledge-grounded role-play dialogue settings @xcite , which are essential for guiding simulation optimisation @xcite . most available benchmarks focus either on open-domain conversations, which prioritise generating engaging, open-ended dialogue @xcite , or on task-oriented conversations, where an agent assists a user with a concrete task, such as booking a hotel @xcite . neither paradigm fully addresses the demands of our targeted dialogue systems. moreover, the inadequacy of current evaluation methods is further intensified by their tendency to disregard the multi-turn nature of dialogues. the prevailing benchmark, mt-bench @xcite , predominantly evaluates llms on a coarsegrained level using minimal turn-taking, such as two-turn dialogues. such a methodology fails to account for performance degradation over longer interactions, a phenomenon demonstrated by @xcite , who found that even advanced llms perform significantly worse in multi-turn contexts. accordingly, a comprehensive multi-turn evaluation is essential to gain an understanding of how llms function in long-form dialogue systems. in this paper, we aim to address these gaps by conducting a multi-turn comparative analysis of llm-generated responses and human-authored responses within knowledge-grounded training simulations. by systematically analysing the quality of llm-generated responses throughout these inter-actions, we aim to determine if llms can capture the nuances of real-life conversations and maintain comparable performance to humans over time. this motivates our research question: ## experiment 1: human evaluation the main goal of this experiment was to address our main research question, the comparative quality of llm-generated and human-authored responses in a multi-turn, knowledge-grounded role-play. to guide our investigation, we formulated two subresearch questions (sub-rqs): subrq1: how do the quality perceptions of llmgenerated and human-authored responses change as a dialogue progresses over turns? subrq2: what key factors most strongly influence participants' perceptions of response quality? ## experiment 2: automated evaluation experiment 1 provided strong evidence that the quality of llm-generated responses degraded over the course of a long-form dialogue and that humanauthored responses were consistently preferred over llm-generated ones. to investigate whether these findings generalise across broader conversational contexts, we designed a second experiment using an llm-as-a-judge for automated evaluation. the experiment was structured in two phases. first, we conducted a validation phase, where we systematically evaluated how well llms mimic the human judgements from experiment 1. second, after identifying the evaluation method that reached the highest agreement with human judgements, we proceeded to the generalisation phase. in this phase, we applied the validated method to assess responses across three additional conversational scenarios, each designed for a different communication skill: motivational interviewing, selling, and consulting. each selected scenario spans over 30 turns to ensure sufficient length for robust multi-turn analysis. this two-phase approach was implemented for two distinct evaluation tasks: construct rating and pairwise preference. ## discussion and conclusions answering our research questions led to three principal findings. first, we observed significant degradation in the perceived quality of llm-generated responses relative to human-authored responses as knowledge-grounded role-play dialogues progress. this trend was validated by both human evaluation and our automated llm-as-a-judge assessments and was most prominent in the human-rated dimensions of overall quality, natural, and context maintenance. second, multi-turn analysis revealed opposing trajectories and confirmed that llm performance degrades sharply over turns, while human-authored responses improved throughout the conversation (subrq1). this aligns with prior work on context degradation in extended interactions @xcite but expands it to knowledge-grounded pedagogical settings. third, participants' quality perceptions were most strongly driven by interesting, maintains context, and natural (subrq2). experts in our focus group highlighted additional nuances: character consistency, pedagogical nudging and conciseness. these factors resonate with the design principles of educational role-play systems @xcite but have not been quantitatively assessed. to scale our investigation, we validated the llmas-a-judge approach @xcite for both construct rating and pairwise preference tasks. using gemini 2.0 flash with a random few-shot prompting strategy, we achieved high alignment with human judgements, providing a viable method for automated evaluation of long-form dialogues. the consistent preference for human-authored responses in pairwise evaluations further reinforces the reliability of llm judges for comparative assessments, as suggested by @xcite . this evaluation framework also provides a methodology for benchmarking specialised judging models, such as prometheus-2 @xcite and judgelm @xcite . notably, however, automated evaluation revealed that llms struggle to evaluate output of their kind, as seen in the lower correlations for llm-generated responses in zero-shot settings. this calls for caution when choosing llm judges for evaluating llm performance. overall, our findings highlight both the potential and current limitations of llms in trainingoriented role-play dialogues. despite our validated evaluation pipeline that enables scalable assessment, llms still struggle to sustain high-quality, context-sensitive responses across extended interactions. this long-context degradation remains a significant barrier. until such challenges are addressed, human authors continue to be the gold standard for crafting engaging and pedagogical role-play scenarios.