Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation (2024)

Abstract: Most task-oriented dialogue (TOD) benchmarks assume users that know exactly how to use the system by constraining the user behaviors within the system’s capabilities via strict user goals, namely “user familiarity” bias. This data bias deepens when it combines with data-driven TOD systems, as it is impossible to fathom the effect of it with existing static evaluations. Hence, we conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios. In particular, we compare users with 1) detailed goal instructions that conform to the system boundaries (closed-goal) and 2) vague goal instructions that are often unsupported but realistic (open-goal). Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system, in which 92% of the dialogues had significant issues. Moreover, we conduct a thorough analysis to identify distinctive features between the two settings through error annotation. From this, we discover a novel “pretending” behavior, in which the system pretends to handle the user requests even though they are beyond the system’s capabilities. We discuss its characteristics and toxicity while showing recent large language models can also suffer from this behavior.

## introduction task-oriented dialogue (tod) systems aim to accomplish specific user goals by comprehending their requests and making appropriate api calls or database (db) searches @xcite . tod systems typically use a pipeline approach, connecting separate modules such as intent detection, dialogue state tracking, policy management, and natural language generation, often requiring complex rules or heuristics. end-to-end (e2e) tod systems figure 1 : contrastive dialogues according to user familiarity with the system. users with high familiarity converse with the system within the predefined scenario since they already know the system well. however, users with low familiarity are more likely to talk about a broad range of topics beyond the system's capacities leading to the erroneous response of the system. however, despite the numerous studies on tod systems and the great successes of large language models, we argue that there is a huge gap between the tod studies and deployable tod systems. among the many reasons hindering end-to-end systems from being widely adopted in the industry, the instability of such systems makes it harder to match the robustness of conventional pipeline systems. we hypothesize that the major source of this instability lies in the naive assumption about the users during tod data collection. we call this the user familiarity bias, as illustrated in figure 1 . for instance, during wizard-of-oz style data collection @xcite , the user-role workers are provided with detailed instructions on the goal they need to achieve which conforms with the system capabilities @xcite . hence, as the user behaviors are strictly constrained, this process simulates users who know exactly how to use the system @xcite . other datasets based on user simulation, such as m2m and sgd @xcite , include the same user familiarity bias, as they simulate users based on predefined user goals and rules specifying how to converse. on the other hand, real users in the wild often have fairly creative or vague goals way beyond the system coverage, and this user-side bias prevents us from modeling such realistic conversations. in this paper, we conduct an interactive user study on one of the most-used wizard-of-oz-styled tod benchmarks, multiwoz @xcite , to investigate the impact of user familiarity. the main objective of this study is to determine whether the familiarity of the user with the tod system's capabilities influences the successful completion of a conversational task. to do this, we first divide the users into two groups: closed-goal and open-goal. the former user group is provided with detailed user goal instructions that are from the multiwoz, while the latter is given only a portion of the instructions along with some realistic goals that are not supported by the system, thereby simulating users who are not familiar with tod systems. based on each goal type they are assigned to, the users converse with a recent e2e tod system, galaxy @xcite , which is trained on diverse tod datasets including multiwoz. our user study reveals that 92% of the dialogues in the open-goal setting has significant issues that often lead to failure in achieving the user goals. moreover, we find that various inconveniences caused by the tod system force users to quit the conversation regardless of the goal types. we thoroughly analyze the resulting conversations to identify the impact of user familiarity by annotating erroneous turns. in particular, we figure out six prevalent errors in both goal settings. as expected, open-goal dialogues contain more problematic turns, and the open-goal setup causes more irrelevant and unfaithful responses. in addition, we identify unfaithful responses as "pretending" behaviors of the system that primarily arises in the open-goal setting. this is a phenomenon similar to hallucination, in which the dialogue system pretends to handle the user's requests even though they are beyond the system's boundaries, but more potentially harmful because it is almost impossible for users to verify the reliability of the information during conversation since the hallucinated pieces of information are usually service-specific. we believe this issue is relatively underexplored as we witness most previous works focused on the closed-goal setting, and our qualitative analysis of the open-goal dialogues demonstrates that such pretending behaviors are prevalent and crucial. finally, we conduct case studies to check whether recent large language models with strong zero-shot performance can mitigate each conversational error. we show that large language models are proficient to handle errors within given context, but preventing pretending problem highly depends on the system design, not only on language models' performance. our contributions are threefold: (1) interactive user study that breaks away from the closed-goal assumption; (2) examination of the characteristics of erroneous situations in both dialogue-and turnlevels; and (3) demonstration of the "pretending" problem of the tod systems, especially as observed in an open-goal assumption, where the agent deceives users as if it handled their exceptional requests. ## interactive user study in this section, we explain the experimental setups of our interactive user studies on the current stateof-the-art data-driven tod model. our focus lies on creating realistic scenarios, breaking away from evaluation solely based on tod benchmarks. in particular, we are curious about the influence of user familiarity on the tod system. we describe the details of the study in the following sections. ## conclusion in this work, we demonstrate user familiarity bias in current tod benchmarks, which the recent tod research community has overlooked. to effectively unveil the bias, we contrast two user groups with different user goals via an interactive user study. against the closed-goal within the constrained scenarios, we introduce a control user group by assigning unconstrained scenarios to the participants, namely open-goal. users in the two groups converse with the academically-discussed tod chatbot following the given closed or open-goals. our study reveals the tod system exposed to the user familiarity bias significantly fails to converse with the users with open-goals. we identify prevalent error types by analyzing the resulting conversations. furthermore, we highlight the pretending behaviors of the tod system with its characteristics and toxicity, which are not easily solved by simply utilizing large language models.