Effects of Naturalistic Variation in Goal-Oriented Dialog (2020)

Abstract: Existing benchmarks used to evaluate the performance of end-to-end neural dialog systems lack a key component: natural variation present in human conversations. Most datasets are constructed through crowdsourcing, where the crowd workers follow a fixed template of instructions while enacting the role of a user/agent. This results in straight-forward, somewhat routine, and mostly trouble-free conversations, as crowd workers do not think to represent the full range of actions that occur naturally with real users. In this work, we investigate the impact of naturalistic variation on two goal-oriented datasets: bAbI dialog task and Stanford Multi-Domain Dataset (SMD). We also propose new and more effective testbeds for both datasets, by introducing naturalistic variation by the user. We observe that there is a significant drop in performance (more than 60% in Ent. F1 on SMD and 85% in per-dialog accuracy on bAbI task) of recent state-of-the-art end-to-end neural methods such as BossNet and GLMP on both datasets.

## introduction end-to-end dialog systems that learn from humanto-human conversations have huge potential for various goal-oriented dialog tasks such as hotel, restaurant and flight reservations. recent work @xcite has shown that it is possible to train dialog models in an end-toend manner and achieve satisfactory results. there are several benchmarks @xcite @xcite to evaluate the performance of neural models for goal-oriented dialog. however, these benchmarks assume a world of a "perfect" user who always provides precise, concise, and correct utterances. these goal-oriented datasets are largely collected by crowdsourcing, where a crowdsource worker enacts the part of a real user by following a set template of instructions provided for the task. this method results in a dataset where most of the user utterances are straight-forward, stick to the goal and tend to leave out the variation commonly found in naturally occurring conversational data. for example, in making a restaurant reservation, a user may perform the following actions: a) check on the customer care agent's welfare, b) comment on the weather in the opening of the conversation, c) ask about business hours or about whether the restaurant accepts reservations as a preliminary question to the reservation request and d) paraphrase his/her prior request with more details. each of these actions is natural variation present in human-to-human conversations. although some templates ask the crowd workers to paraphrase their request, they never ask workers to simulate the full range of naturalistic variation @xcite . this naturalistic variation has been thoroughly documented in the conversation analysis literature @xcite , and further adapted for designing automated conversational agents @xcite . the core reason for this omission is that natu-ralistic variation is often confused with "chit chat" @xcite . @xcite writes, in common usage, "chit chat" means inconsequential talk. but much talk that may appear on the surface to be inconsequential in fact serves a variety of functions in managing the conversation itself. in this work, we focus on the full range of activities observed in naturally occurring conversations, referred to as "natural variation". our goal in this work is three-fold: recently, few approaches have been explored to study the behavior of neural dialog systems in the presence of synthetically introduced perturbations to the dialog history. @xcite created the babi+ dataset, an extension of babi dialog task-1, by introducing variations like hesitations, restarts and corrections. zhao and eskenazi (2018) created simdial, which simulates spoken language phenomena, e.g. self-repair and hesitation. @xcite introduce utterance-level and wordlevel perturbations on various benchmarks. however, such variations have been largely artificial and do not reflect the "natural variation" commonly found in naturally occuring conversational data. @xcite show that often models do not generalize well to examples from new annotators at test time who did not contribute to training data, which reinforces our choice of introducing natural variation in the test set for evaluation. ## datasets we study and observe issues in multiple goaloriented dialog benchmarks. in this work, we focus on two multi-turn goal-oriented datasets: babi dialog task and smd for evaluating the impact of natural variation. we provide details on issues in the following datasets: smd, camrest676 @xcite , frames @xcite and air-dialogue @xcite in the appendix. ## naturalistic variation in order to better approximate natural variation in our datasets, we utilize the natural conversation framework (ncf) @xcite . the ncf is a framework for designing conversational agents that more closely emulate natural conversation than most of today's chatbots and voice assistants. the ncf is organized into two kinds of patterns: conversational activities and conversation management. the conversational activity patterns (denoted by a) handle the main business of conversation, i.e. the user request and the services provided by agent. on the other hand, conversation management patterns help the user and agent to manage the conversation itself. conversation management occurs on two levels: sequence level (denoted b) and conversation level (denoted c). after studying the 100 patterns in ncf, we identified a subset of the 32 patterns which are most commonly found in goal-oriented natural conversations and use them in our work. we excluded the remaining 68 patterns that covered other types of conversations e.g. quiz, question-answer jokes, voice-based assistant setting, etc. we provide details of these 32 patterns in the appendix. for each pattern p from the 32 ncf patterns, we identify conversations in the test set where p could have been present in the conversation if the crowd worker was not limited to a given template. we define rules and heuristics based on the annotations e.g. dialog acts, slot information, etc. captured by the crowdsource worker from the user utterance. after introducing additional user utterances and agent responses per ncf pattern, we perform manual review of 20% of updated dialogs randomly to ensure that incorporating the pattern does not make the dialog incoherent @xcite . after manual review, we select a subset of 9 patterns and incorporate them in the two datasets. pattern smd babi (a) open request screening 64 54 (a) open request user detail request -143 (b) example request 23 -(b) misunderstanding report 35 314 (b) other correction 24 522 (b) sequence closer (not helped) 6 811 (b) sequence closer (repaired) 139 189 (c) capability expansion 151 811 (c) recipient correction 100 - although the naturalistic variation increases the complexity of the dialog, the added utterances do not increase the complexity of the goal, in other words, they do not introduce new topics or courses of action, they merely expand the existing ones. ## experiments as the baselines for our experiments. we use the best performing hyper-parameters reported by both models for each dataset. the test results reported (in for evaluation of the synthetic babi dialog task-5, we use per-response and per-dialog accuracy @xcite . for smd, we use a) bleu @xcite and b) entity f1 @xcite scores. we evaluate bossnet and glmp models on both the original and the updated test set. we do not evaluate the models on their ability to generate the newly added system responses as part of the naturalistic variation, but only on the system responses originally present in the test set. ## results from we observe that the drop in performance on babi is much less than smd. this is because babi is a synthetic dataset with a small set of fixed agent responses. since the models are evaluated only on the agent responses present in the original test set, additional user and agent utterances for incorporating natural variation do not affect performance too much. on the other hand, smd is a real-world dataset of human-to-human conversations collected by crowdsourcing and we observe a much higher drop across both bleu and ent f1 scores. we perform ablation experiments to study the impact of each pattern (presented in our results clearly show that naturalistic variation present during deployment affects model performance and will result in lower than expected performance for a given dialog system in production. ## conclusion this work studies the dangers of using crowdsourced data, without templates for the natural range of activities in conversation, such as the natural conversation framework @xcite , to train end-to-end dialog systems. we highlight the impact on the performance of state-ofthe-art models on our new and effective testbeds for babi dialog task-5 and smd datasets, which have naturalistic variation. we believe this opens up a new and promising research direction for devising improved strategies for crowdsourcing goaloriented datasets, as well as improved models that can better handle interactions with real users. tiancheng zhao and maxine eskenazi. 2018. zeroshot dialog generation with cross-domain latent actions. in proceedings of the 19th annual sigdial meeting on discourse and dialogue, pages 1-10.