Is M ulti WOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator (2022)

Abstract: Task-Oriented Dialogue (TOD) systems are drawing more and more attention in recent studies.Current methods focus on constructing pre-trained models or fine-tuning strategies while the evaluation of TOD is limited by a policy mismatch problem.That is, during evaluation, the user utterances are from the annotated dataset while these utterances should interact with previous responses which can have many alternatives besides annotated texts.Therefore, in this work, we propose an interactive evaluation framework for TOD. We first build a goal-oriented user simulator based on pre-trained models and then use the user simulator to interact with the dialogue system to generate dialogues.Besides, we introduce a sentence-level and a session-level score to measure the sentence fluency and session coherence in the interactive evaluation. Experimental results show that RL-based TOD systems trained by our proposed user simulator can achieve nearly 98% inform and success rates in the interactive evaluation of MultiWOZ dataset and the proposed scores measure the response quality besides the inform and success rates.We are hoping that our work will encourage simulator-based interactive evaluations in the TOD task.

## introduction building intelligent dialogue systems has become a trend in natural language process applications especially with the help of powerful pre-trained models. specifically, task-oriented dialogue (tod) systems @xcite are to help users with scenarios such as booking hotels or flights. these tod systems @xcite @xcite usually first recognize user's intents and then generate corresponding responses based on an external database containing booking information. therefore, the key factor in tod is the interaction between users and dialogue systems. however, in current tod system evaluation process, traditional evaluation process uses annotated user utterances in multi-turn dialogue sessions no matter what responses the dialogue system generated, as illustrated in figure 1 . while in realworld dialogues, the user utterances are coherent with responses from the other speaker (which is the service provider). therefore, in tod evaluation, using annotated utterances without interaction with the dialogue system will cause a policy mismatch, which would weaken the soundness of the evaluation results. the mismatch might hurt the evaluation process since some responses may be correct and coherent but use a different policy with the annotated responses. also, incoherent dialogue histories will affect the response generation. with current state-of-the-art models achieving similar performance, it is natural to consider that the bottleneck in the performance of current tod systems is not the model capability but the evaluation strategy. since incorporating human interactions during evaluation is costly, a feasible method is to build an automatic interactive evaluation framework that can solve the policy mismatch problem. in this paper, we propose a complete interactive evaluation framework to evaluate the tod system. we first build a strong dialogue user simulator based on pre-trained models, and we use the proposed simulator to deploy interactive evaluations. in simulator learning, we introduce a goalguided user utterance generation model based on sequence-to-sequence pre-trained models. then we use reinforcement learning to train both the user simulator and the dialogue system to boost interaction performance. in interactive evaluations, we use the simulator to generate user utterances based on interactions with the generated responses instead of using static user utterances that have been annotated in advance. therefore, during evaluation, user utterances will respond to the generated responses, which can avoid the mismatch between stale user utterances and generated responses. further, in interactive evaluations, the quality of the generated texts cannot be measured since traditional bleu cannot be calculated without oracle texts. to better evaluate the performance of dialogue systems, we introduce two automatic scores to evaluate the response quality at both sentence-level and sessionlevel. the sentence-level score is to evaluate the sentence fluency and the session-level score is to evaluate the coherence between turns in a dialogue session. also, these proposed scores can be used in traditional evaluation methods as well as the annotated dataset as a meta-evaluation to explore the importance of using user simulators to construct interactive evaluations. we construct experiments on multiwoz dataset @xcite based on pre-trained models and use our proposed simulator and scores to run interactive evaluations. experimental results show that interactive evaluations can achieve over 98% inform and success rates, indicating that the bottleneck of tod performance is the lack of proper evaluation methods. the proposed scores show that our proposed simulator can help achieve promising evaluation results in the interactive evaluation framework. also, we explore the performance of rl-based models and we also use proposed scores to find that rl methods might hurt response quality to achieve high success rates. therefore, we can summarize our contributions: (a) we construct an evaluation framework that avoids policy mismatch problems in tod. ## interactive evaluation framework in our proposed interactive evaluation framework, we first build a goal-state guided user simulator to model user policies and generate high-quality user utterances. then we construct the interactive evaluation framework and introduce two scores to evaluate the interactive inference results. ## conclusion in this paper, we focus on the evaluation of current end-to-end tod systems. we construct an automatic interactive evaluation framework with a strong user simulator. besides, we obtain extremely high interactive evaluation performance on multi-woz dataset by jointly training our user simulator and dialogue system. through the interactive evaluation framework, we can conclude several hints for future studies on tod systems: (1) current tod needs more challenging and complicated dataset and scenarios; (2) interactive evaluation process should be considered in proper evaluation of tod systems; (3) reinforcement learning used in training user simulators and dialogue systems requires more careful design to consider both task success rate and quality of generated texts.