The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogs (2025)

Abstract: Large Language Model (LLM)-based agents have significantly impacted Task-Oriented Dialog Systems (TODS) but continue to face notable performance challenges, especially in zero-shot scenarios. While prior work has noted this performance gap, the behavioral factors driving the performance gap remain under-explored. This study proposes a comprehensive evaluation framework to quantify the behavior gap between AI agents and human experts, focusing on discrepancies in dialog acts, tool usage, and knowledge utilization. Our findings reveal that this behavior gap is a critical factor negatively impacting the performance of LLM agents. Notably, as task complexity increases, the behavior gap widens (correlation: 0.963), leading to a degradation of agent performance on complex task-oriented dialogs. For the most complex task in our study, even the GPT-4o-based agent exhibits low alignment with human behavior, with low F1 scores for dialog acts (0.464), excessive and often misaligned tool usage with a F1 score of 0.139, and ineffective usage of external knowledge. Reducing such behavior gaps leads to significant performance improvement (24.3% on average). This study highlights the importance of comprehensive behavioral evaluations and improved alignment strategies to enhance the effectiveness of LLM-based TODS in handling complex tasks.

## introduction in recent years, advancements in artificial intelligence, particularly in large language models (llms) @xcite @xcite @xcite @xcite @xcite , and llmbased agents @xcite @xcite , have garnered significant attention for task-oriented dialog systems (tods). while llm agents have the promise of eliminating complex modular designs in traditional tods @xcite , recent studies, however, have shown that llm agents, particularly in zero-shot scenarios, struggle to perform optimally @xcite @xcite . though several studies have identified this performance gap, there is limited work investigating the behavioral causes. by considering three specific dialog acts, @xcite found that llms generate language with significantly less grounding compared to humans. however, this is not necessarily the only source of the behavior gap-the discrepancies between llm agents and human experts. a comprehensive study of the behavior gap is necessary. this involves not only identifying the dimensions along which these discrepancies occur but also understanding the factors that may affect this gap, such as task complexity and the choice of llms. additionally, it is important to quantify the impact of these discrepancies on agent performance. in this study, we propose a comprehensive evaluation framework to measure the behavior gap across three key behavioral dimensions. one dimension is dialog acts, traditionally used to represent an utterance at the level of illocutionary force @xcite @xcite ; the second dimension is tool usage, and the third is external knowledge usage, both included due to their crucial role in modern llm agents @xcite @xcite . our framework operates in a teacher-forcing setting @xcite to avoid dependence on user simulators, which may introduce additional discrepancies, and could be implementation-specific. our analysis reveals significant behavioral gaps between llm agents and human experts across three tasks of varying complexity levels. specifi-cally, llms exhibit misalignment in dialog acts, excessive but often incorrect tool usage, and inefficient representation of external knowledge. these gaps become especially pronounced as we move from relatively simple slot-filling tasks to more complex ones, particularly in the product customer support (pcs) dataset. pcs is a new tod dataset studied in this work that captures real-world customer-human expert interactions. furthermore, our analysis reveals that the behavior gap negatively impacts llm agent performance, and reducing this gap leads to statistically significant improvements. in summary, our contributions are as follows: behavior and performance evaluation fed @xcite provides an evaluation framework for assessing fine-grained dialog quality using dialogpt @xcite . however, since dialogpt is trained on reddit data, its applicability to tods is limited. recent advancements have explored llm-as-a-judge approaches @xcite as alternative evaluation methods for dialog and other complex language generation tasks that are traditionally difficult to assess. despite these efforts, few studies have focused on the behavioral analysis of llm agents. one notable study @xcite found that llms exhibit less conversational grounding compared to humans. however, conversational grounding is only one of several factors contributing to the behavior gap. we build on these works by introducing a comprehensive behaviorcentric analysis that spans dialog acts, tool selection, and knowledge grounding, providing deeper insight into how specific behavioral gaps influence overall tod agent performance. ## method in this work, we introduce a comprehensive framework designed to systematically analyze the behavior gap between llm-based agents and human experts. this framework includes: using this framework, we evaluated the behavior and performance of llm agents (section 3.2) across three tasks of varying complexity (section 3.1). ## discussion in this study, we identify a clear behavior gap between llm agents and human experts-a gap that widens as tasks complexity increases. our analysis reveals three key behavior gaps: llm agents often adopt different dialog strategies, use tools more frequently but less effectively, and rely on copying retrieved knowledge rather than synthesizing it into meaningful insights. these gaps negatively impact agent performance, but aligning llm behavior more closely with human strategies mitigates this effect. our findings emphasize the importance of enhancing llm agents' multi-turn dialog capabilities. our observation that injecting human dialog acts or human-selected tools into prompts improves performance is consistent with prior research on the significance of selecting effective dialog strategies @xcite @xcite . beyond ensuring smooth dialog flow, addressing gaps in external knowledge usage is crucial, as it enables the agent to efficiently and effectively leverage external knowledge to provide actionable insights. this requirement is particularly crucial for more complex tasks @xcite @xcite , as our findings indicate a correlation between task complexity and behavior gap. collectively, these insights emphasize that training models to grasp not only lexical tokens but also dialog strategies is beneficial. ## limitations the accuracy of our analysis depends on the reliability of the llm-based classifiers used in our framework. while these classifiers were validated on benchmark datasets with ground-truth annotations, applying them to domains that significantly differ from those benchmarks may require further validation to ensure consistent performance. moreover, our analysis is confined to turn-level comparisons due to the teacher-forcing evaluation setup. while this setting provides controlled, interpretable diagnostics, it limits our ability to assess full dialog-level strategies and dynamics. extending the framework to evaluate dialog-level behavior remains an avenue for future work. lastly, our study does not include recently released reasoning llms such as gpt-o1 or deepseek-r1 @xcite , which may exhibit distinct behavioral patterns. understanding how these newer models behave within our framework is an open question for future exploration.