Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation (2020)

Abstract: There is a growing interest in developing goal-oriented dialog systems which serve users in accomplishing complex tasks through multi-turn conversations. Although many methods are devised to evaluate and improve the performance of individual dialog components, there is a lack of comprehensive empirical study on how different components contribute to the overall performance of a dialog system. In this paper, we perform a system-wise evaluation and present an empirical analysis on different types of dialog systems which are composed of different modules in different settings. Our results show that (1) a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels, (2) component-wise, single-turn evaluation results are not always consistent with the overall performance of a dialog system, and (3) despite the discrepancy between simulators and human users, simulated evaluation is still a valid alternative to the costly human evaluation especially in the early stage of development.

## introduction many approaches and architectures have been proposed to develop goal-oriented dialog systems to help users accomplish various tasks @xcite . unlike open-domain dialog systems, which are designed to mimic human conversations rather than complete specific tasks and are often implemented as end-to-end systems, a goal-oriented dialog system has access to an external database on which to inquire about information to accomplish tasks for users. goal-oriented dialog systems can be grouped into three classes based on their architectures, as illustrated in fig. 1 . the first class is the pipeline (or modular) systems which typically consist of the four components: natural language understanding (nlu) @xcite , dialog state tracker (dst) @xcite , dialog policy @xcite , and natural language generation (nlg) @xcite . ## related work developers have been facing many problems when evaluating a goal-oriented dialog system. a range of well-defined automatic metrics have been designed for different components in the system, e.g., joint goal accuracy in dst and task success rate in policy optimization introduced in as human evaluation by asking crowd-sourcing workers to interact with a dialog system is much expensive @xcite and prone to be affected by subjective factors @xcite , researchers have tried to realize automatic evaluation of dialog systems. simulated evaluation @xcite is widely used in recent works @xcite @xcite and platforms @xcite @xcite , where the system interacts with a user simulator which mimics human behaviors. such evaluation can be conducted at the dialog act or natural language level. the advantages of using simulated evaluation are that it can support multi-turn language interaction in a full end-to-end fashion and generate dialogs unseen in the original corpus. ## conclusion and discussion in this paper, we have presented the system-wise evaluation result and empirical analysis to estimate the practicality of goal-oriented dialog systems with a number of configurations and approaches. though our experiments are only conducted on multiwoz, we believe that such results can be generalized to all goal-oriented scenarios in dialog systems. we have the following observations: hotel restaurant info reqt book info book @xmath0:30 @xmath1: i would prefer it in the centre area . actually i need a moderately priced restaurant . are there any fitting that description ? this restaurant should serve british food too . u: i would prefer it in the centre area . actually i need a moderately priced restaurant . are there any fitting that description ? this restaurant should serve british food too . d: restaurant-food:british, restaurantpricerange:moderate, restaurant-area:centre d: restaurant-food:british, restaurantpricerange:moderate, restaurant-area:centre s: i have 4 options for you ! how about the copper kettle ? it is located in the centre . s: i have 4 options for you ! it is in the centre area . that is a british restaurant . the copper kettle is just your thing . it is in the centre area . u: i need a table for 8 people . can you try to book it at 16:30 . we will be there sunday night . u: it will be for 8 people . can you try for 16:30 ? i am also looking for a restaurant to book for sunday. d: restaurant-people:8, restaurant-day:sunday, restaurant-time:16:30 d: restaurant-people:8, restaurant-day:sunday, restaurant-time:16:30 s: your reference number is 00000013 . s: all set . your reference number is 00000013 . u: i would like to keep it in the moderate range , please . oh , i need parking. u: hello , i would like to find a hotel that includes free parking . i also need a moderate place to stay . d: hotel-parking:yes, hotel-pricerange:moderate d: hotel-parking:yes s: i have 14 different options for you ! how about hamilton lodge ? fits your request perfectly . it has free wifi . s: reference number is : 00000013 . u: that sounds fine , i just need the postcode though . u: what is the postcode ? d: d: s: the postal code for that hotel is cb41da . s: thank you for using our services . u: i need a room starting on sunday . there is going to be 8 of us. how about only 2 nights . u: that sounds fine , i just need the postcode though . d: hotel-day:sunday, hotel-people:8, hotel-stay:2 d: s: here is the booking information : booking was successful . reference number is : 00000001. s: glad to have been of help . thank you for using the service . enjoy the rest of your day ! u: thank you very much , goodbye . u: i just need the postcode . d: d: s: you are very welcome. goodbye. s: ok , thank you . have a good day .