M ulti WOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation (2022)

Abstract: The MultiWOZ 2.0 dataset has greatly stimulated the research of task-oriented dialogue systems. However, its state annotations contain substantial noise, which hinders a proper evaluation of model performance. To address this issue, massive efforts were devoted to correcting the annotations. Three improved versions (i.e., MultiWOZ 2.1-2.3) have then been released. Nonetheless, there are still plenty of incorrect and inconsistent annotations. This work introduces MultiWOZ 2.4, which refines the annotations in the validation set and test set of MultiWOZ 2.1. The annotations in the training set remain unchanged (same as MultiWOZ 2.1) to elicit robust and noise-resilient model training. We benchmark eight state-of-the-art dialogue state tracking models on MultiWOZ 2.4. All of them demonstrate much higher performance than on MultiWOZ 2.1.

## introduction in recent years, tremendous advances have been made in the research of task-oriented dialogue systems, attributed to a number of publicly available dialogue datasets like dstc2 @xcite , frames @xcite , woz @xcite , m2m @xcite , multiwoz 2.0 @xcite , sgd @xcite , crosswoz @xcite , ri-sawoz @xcite , and treedst @xcite . among them, multiwoz 2.0 is the first large-scale dataset spanning multiple domains and thus has attracted the most attention. however, substantial noise has been found in the dialogue state annotations of multiwoz 2.0 @xcite . to remedy this issue, @xcite fixed 32% of dialogue state annotations across 40% of the dialogue turns, resulting in an improved version multiwoz 2.1. despite the significant improvement in annotation quality, multiwoz 2.1 1 multiwoz 2.4 is released to the public at https:// github.com/smartyfh/multiwoz2.4 . massive efforts have been made to further improve the annotation quality of multiwoz 2.1, resulting in multiwoz 2.2 @xcite ) and multiwoz 2.3 @xcite . however, they both have some limitations. more concretely, mul-tiwoz 2.2 allows the presence of multiple values in the dialogue state. but it does not cover all the value variants. this incompleteness brings about serious inconsistencies. multiwoz 2.3 focuses on dialogue act annotations. the noise on dialogue state annotations has not been fully resolved. in this work, we introduce multiwoz 2.4, an updated version on top of multiwoz 2.1, to improve dialogue state tracking evaluation. specifically, we identify incorrect and inconsistent annotations in the validation set and test set, and fix them meticulously. this refinement results in changes to the state annotations of more than 41% of turns over 65% of dialogues. since our main purpose is to improve the correctness and fairness of model evaluation, the annotations in the training set remain unchanged. even so, our empirical study shows that much better performance can be achieved on multiwoz 2.4 than on all the previous versions. error type conversation example multiwoz 2.1 multiwoz 2.4 (i) context mismatch usr: hello, i would like to book a taxi from restaurant 2 two to the museum of classical archaeology. taxi-@xmath0: i am looking for a restaurant that serves portuguese food. rest.-@xmath1.-@xmath2: i need a place to dine in the centre of town. rest.-@xmath3.-@xmath4: please recommend one and book it for 6 people. sys: i would recommend express by holiday inn cambridge. from what day should i book? usr: starting saturday. i need 5 nights for 6 people. hotel-book @xmath5: i am planning a trip in cambridge. hotel-@xmath6: i recommend charlie chan. would you like a table? usr: yes. monday, 8 people, 10:30. rest.-@xmath7.-@xmath8: something classy nearby for dinner, preferably italian or indian cuisine? rest.-@xmath9.-@xmath10: i need a train leaving after 10:00. train-@xmath11:15 train-@xmath12:00 (vi) unnecessary annotation usr: i am looking for a museum. sys: the broughton house gallery is a museum in the centre. usr: that sounds good. could i get their phone number? attraction-@xmath13, a noisy training set motivates us to design robust and noise-resilient training mechanisms, e.g., data augmentation @xcite and noisy label learning @xcite . considering that collecting noise-free large multi-domain dialogue datasets is costly and labor-intensive, we believe that training robust dialogue state tracking models from noisy training data will be of great interest to both industry and academia. ## annotation refinement in multiwoz 2.0 & 2.1, the dialogue state is represented as a series of slot-value pairs. for example, attraction-@xmath14. considering that multiwoz 2.1 has significantly improved the annotation quality of multiwoz 2.0, we choose to continue the refinement on the basis of multiwoz 2.1. another choice is to perform the refinement on top of multiwoz 2.2. however, as mentioned earlier, multiwoz 2.2 allows each slot to have multiple value variants. this relaxation increases the difficulty of annotating. it is challenging to include all the value variants. new value variants may also emerge as time goes by. even worse, some value variants are ambiguous and invalid. for instance, "peking" can be a shared variant of "peking university" and "peking restaurant". hence, it is an ambiguous value variant. besides, the benchmark evaluation on multiwoz 2.2 shows no evident performance improvements over multiwoz 2.1 @xcite . in light of these, multiwoz 2.1 is a better basis for our refinement. ## human evaluation we also perform a human evaluation on the quality of the refined annotations. we randomly sampled 50 dialogues from the test set and recruited 5 computer science students to compare our refinement against the annotations in multiwoz 2.1. specifically, the raters were asked to assign a score to each turn of the sampled dialogues based on the following criteria: 1) -2: a score of -2 means that both the refined annotation and original annotation are not completely correct; 2) -1: a score of -1 means that the original annotation is correct while the refined annotation is problematic; 3) 0: a score of 0 means that both the refined annotation and original annotation are correct, that is, no changes have been made to the original annotation; 4) 1: a score of 1 means that the refined annotation is correct while the original annotation is invalid. we obtain an average score of 0.1653, meaning that our refined annotations are more accurate. we further employ fleiss' kappa @xcite to measure the level of agreement among different raters. we obtain κ = 0.9226, which indicates an almost perfect agreement across the five raters. we illustrate the score distributions of different raters in figure 3 . from this figure, we can intuitively observe that there is a high level of agreement among the five raters. figure 3 also shows that in most cases, the refined annotation and the original annotation are both correct, meaning that there is no need to make any changes to the original annotation. this is desirable, as our refinement is based on multiwoz 2.1 which has already fixed lots of annotation errors. around 20% of annotations in multiwoz 2.4 are deemed to be more accurate than multiwoz 2.1, while only about 1% of annotations in multiwoz 2.1 are evaluated as better. this verifies again that our refinement has higher quality. we further inspected the annotations in multi-woz 2.1 that are assessed to be more appropriate. we found that these annotations are mainly related to the slot hotel-type. this slot has four candidate values {"hotel", "guest house", "none", "dontcare"}, which are relatively confusing because the term "hotel" is also one candidate value. in practice, when a user says "i am looking for a hotel with 4 stars", the user may actually mean that "i am looking for a place to stay with 4 stars". however, by convention, the term "hotel" is used more often, even though the user does not mean that the hotel type must be "hotel". in our refinement procedure, we chose to annotate this slot based on the whole dialogue session to understand the true user intention (i.e., hotel @xmath15. this ambiguous slot tells us that it is crucial to develop appropriate slots and candidate values that will not cause any confusions to the annotators. ## caveats and lessons learned although we have tried our best to correct as many annotations in the validation set and test set as possible, it is unlikely that we have fixed all the annotation errors. in fact, there are several challenges we faced during the refinement process that are particularly difficult to overcome. firstly, as dis-cussed earlier, the candidate values of some slots are confusing, which makes it really challenging to choose the most appropriate value. secondly, in some scenarios, the user intention can have different interpretations. for example, the user utterance "the hotel does not need to have internet though" can mean that the user does not need internet at all (hotel-@xmath16. thirdly, some slots may have multiple values. sometimes these values should even be ordered according to users' preferences. when there are too many values (more than two), it is also questionable if the corresponding slot should be annotated. suppose that the system recommended 10 museums to the user and the user asked "does any of them have zero entrance fee?", should the slot attraction-name be annotated? further, the dialogue state can be regarded as a structured representation of the complex user intentions. due to the complexity of the language itself, some information will be inevitably lost when transforming unstructured user utterances into structured state representations. in this regard, dialogue state annotating is in essence a challenging task. given these challenges, it is necessary to define unambiguous slots and unconfusable candidate values to facilitate state annotating. it is also important to provide annotators with full instructions for each slot so that they can make consistent annotations. ## conclusion we introduce multiwoz 2.4, an updated version of multiwoz 2.1, by rectifying (almost) all the annotation errors in the validation set and test set. we keep the annotations in the training set as is to encourage robust and noise-resilient model training. we further benchmark eight state-of-the-art dialogue state tracking models on multiwoz 2.4 to facilitate future research. all the benchmark models have demonstrated much better performance on multiwoz 2.4 than on multiwoz 2.1. multiwoz 2.4 can also be applied to train better overall dialogue systems, e.g., by utilizing data augmentation techniques to generate high-quality training data based on the clean validation set.