M ulti WOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines (2020)

Abstract: MultiWOZ is a well-known task-oriented dialogue dataset containing over 10,000 annotated dialogues spanning 8 domains. It is extensively used as a benchmark for dialogue state tracking. However, recent works have reported presence of substantial noise in the dialogue state annotations. MultiWOZ 2.1 identified and fixed many of these erroneous annotations and user utterances, resulting in an improved version of this dataset. This work introduces MultiWOZ 2.2, which is a yet another improved version of this dataset. Firstly, we identify and fix dialogue state annotation errors across 17.3% of the utterances on top of MultiWOZ 2.1. Secondly, we redefine the ontology by disallowing vocabularies of slots with a large number of possible values (e.g., restaurant name, time of booking). In addition, we introduce slot span annotations for these slots to standardize them across recent models, which previously used custom string matching heuristics to generate them. We also benchmark a few state of the art dialogue state tracking models on the corrected dataset to facilitate comparison for future work. In the end, we discuss best practices for dialogue data collection that can help avoid annotation errors.

## introduction task-oriented dialogue systems have become very popular in the recent years. such systems assist the users in accomplishing different tasks by helping them interact with apis using natural language. dialogue systems consist of multiple modules which work together to facilitate such interactions. most architectures have a natural language understanding and dialogue state tracking module to generate a structured representation of user's preferences from the dialogue history. this structured representation is used to make api calls and as a signal for other modules. then, the dialogue policy module determines the next actions to be taken by the dialogue system. this is followed by the natural language generation module, which converts the generated actions to a natural language utterance, which is surfaced to the user. recently, data-driven techniques have achieved state-of-the-art performance for the different dialogue systems modules @xcite @xcite . however, collecting high quality annotated dialogue datasets remains a challenge for the researchers because of the extensive annotation required for training the modules mentioned above. many public datasets like dstc2 @xcite , woz @xcite , simulateddialogue @xcite , multi-woz @xcite , taskmaster @xcite , sgd @xcite , etc. have been very useful to facilitate research in this area. among these datasets, multiwoz is the most widely used benchmark for dialogue state tracking. it contains over 10,000 dialogues spanning 8 domains, namely -restaurant, hotel, attraction, taxi, train, hospital, bus, and police. since its inception, the multiwoz dataset has undergone a few updates. @xcite introduced user dialogue actions providing a structured semantic representation for user utterances. @xcite fixed 32% of dialogue state annotations across 40% of the turns and introduced slot descriptions, culminating in multiwoz 2.1, a new version of the dataset. despite the large scale of corrections introduced in multiwoz 2.1, there are still many unaddressed annotation errors @xcite . furthermore, several approaches to dialogue state tracking use span annotations identifying the locations in the user and system utterances where slot values have been mentioned, to make the system efficient and generalizable to new slot values @xcite @xcite @xcite @xcite . because of the absence of these span annotations in multiwoz, these approaches resort to generating them using custom string matching heuristics, making their comparison difficult. to address these limitations, we introduce mul-tiwoz 2.2 foot_0 , an updated version of the multiwoz dataset. our contributions are threefold. the paper is organized as follows. first we describe the different types of annotation errors and inconsistencies we observed in multiwoz 2.1 (section 2). then, we outline the redefinition of ontology (section 3), followed by the description of correction procedure (section 4) and new annotations we introduce (section 5). finally, in section 6, we present the performance of a few recent dialogue state tracking models on multiwoz 2.2. ## annotation errors the multiwoz dataset was collected using a wizard-of-oz setup @xcite . in this setup, two crowd-workers are paired together, one acting as a user and the other as the dialogue agent. each dialogue is driven by a unique set of instructions specifying the user goal, which are shared with the crowd-worker playing the role of the user. after every user turn, the crowd-worker playing the role of the dialogue agent (wizard) annotates the updated dialogue state. after updating the state, the tool shows the set of entities matching the dialogue state to the wizard, who then uses it to generate a response which is sent to the user. remaining annotations such as the system actions are collected using a second annotation task. the wizard-of-oz setup is widely considered to produce natural conversations, as there is no turn level intervention guiding the flow of the dialogue. however, because of its heavy reliance on humans for generating the correct annotations, the procedure is prone to noisy annotations. we identified two major classes of errors outlined below, which were not corrected in multiwoz 2.1. ## ontology issues although multiwoz 2.0 provides a predefined ontology which is claimed to enumerate all slots and the possible values for every slot, it has been reported to be incomplete. as a result, many researchers have built their own ontology to achieve a better performance @xcite . to fix the problem of incompleteness, mul-tiwoz 2.1 rebuilt the ontology by listing all values present in dialogue states across the dataset, but it still has some unaddressed issues. first, for some slots, multiple values sharing the same semantics are listed. some examples are "8pm" and "20:00", "a and b guesthouse" and "a and b guest house", "cheap|moderate" and "moderate|cheap" for the slots restaurant-booktime, hotel-semi-name and hotel-semi-pricerange respectively. we find that 51% of the values for the slot hotel-name are not semantically unique, and similar figures for the restaurant-name and attraction-name slots. such duplicate values make evaluation hard since multiwoz 2.1 only assumes one correct value for each slot in the dialogue state. second, we observe multiple slot values in the ontology that can't be associated with any entities in the database. values like "free" for slot attraction-name; "cam", "dif", and "no" for slot restaurant-name are some examples. such values could be introduced in the ontology because of typographical errors in the utterances or annotation errors. our investigation shows that 21.0% of the slot values in the ontology can't be directly mapped back to the values in the database through exact string matching. we also observed a few logical expressions like "cheap|moderate", "not(hamilton lodge)" etc. in the ontology. we believe that these expressions, although semantically correct, add noise during training. the ontology should either omit such expressions altogether or include all possible expressions to enable generalization to cases not observed in the training data. ## correction procedure to avoid the issues described above, we advocate the definition of ontology prior to data collection. this not only serves as a guideline for annotators, but also prevents annotation inconsistencies in the dataset and corruption of the ontology from typographical and annotation errors. this section describes our definition of the new ontology, which we call schema, followed by the corrections made to the state and action annotations. lastly, we also show the statistics of our modifications. ## additional annotations besides the span annotations, we also add active user intents and requested slots for every user turn. predicting active user intents and requested slots are two new sub-tasks that can be used to evaluate model performance and facilitate dialogue state tracking. prediction of active intents or apis is also essential for efficiency in large-scale dialogue systems which support hundreds of apis. based on the action and state annotations, we define a single search intent for the attraction, bus, hotel, and police domains and a single booking intent for taxi domain, whereas for the restaurant, hotel, and train domains, both search and booking intents are defined. ## dialogue state tracking benchmarks recent data-driven dialogue state tracking models that achieve state-of-the-art performance mainly adopt two classes of methods: span-based and candidate-based. span-based methods extract values from dialogue history and are suitable for tracking states of non-categorical slots, while candidatebased methods that perform classification on predefined candidate lists to extract values are bettersuited for categorical slots. to test models' performance on both categorical and non-categorical slots, we selected three dialogue state tracking models that use a mixture of both methods to benchmark the performance on the updated dataset: sgdbaseline @xcite , trade @xcite , and ds-dst @xcite . trade considers each slot as a mixture of categorical and non-categorical slot. it uses a pointer generator architecture to either generate the slot value from a pre-defined vocabulary or tokens in the dialogue history. on the contrary, sgdbaseline has separate tracking strategies for categorical and non-categorical slots. it first uses a shared pretrained bert @xcite to encode a context embedding for each user turn, a slot embedding for each slot, and a slot value embedding for each slot value in the candidate list of the categorical slots. then, it utilizes linear networks to perform classification for the categorical slot and to find start and end span indices for noncategorical slots. ds-dst is a recently proposed model achieving state-of-the-art performance on multiwoz 2.1 using pre-trained bert. the main difference between ds-dst and sgd-baseline is that the context embedding used in ds-dst is conditioned on the domain-slot information while it is not in sgd-baseline. we use joint goal accuracy as our metric to evaluate the models' performance. the joint goal accuracy is defined as the average accuracy of predicting all the slot values for a turn correctly. the performance of different models is shown in and sgd-baseline demonstrate considerably higher performance on non-categorical slots than categorical slots. we infer that it may be caused by the corrections ensuring that the value in the dialogue state is also present in the dialogue history for all non-categorical slots. ## discussion the wizard-of-oz paradigm is a very powerful technique to collect natural dialogues. however, the process of annotating these dialogues is prone to noise. in this section, we discuss some of the best practices to follow during task-oriented dialogue data collection so as to minimize annotation errors. it is important to define an ontology or schema before data collection, listing the interface of all the domains and apis. the schema should identify categorical slots, which have a fixed set of possible values, and the annotation interface should enforce the correctness of these slots. in particular, the interface should only allow the annotator to pick one of the values specified in the schema. for non-categorical slots, the interface should only allow values which have been mentioned in the dialogue history, and display an error otherwise. these simple checks help avoid typographical errors and value paraphrasing issues, discussed in section 2. the annotation task can be followed by simple validation checks to identify erroneous annotations, which can be fixed by a follow-up crowd-sourcing task. for instance, listing the set of all possible values for every slot in the dataset helped us quickly identify instances listing "thursday" as the value for a time slot or "no" as the name of a hotel. we also observed a few annotations utilizing logical expressions to represent the dialogue state. for instance, some dialogue state annotations utilize string "@xmath0, and "cinema|entertainment|museum|theatre" to mean that all values separated by "|"are acceptable. however, such values are disproportionately rare in the dataset (<1% of dialogues), thus making it difficult for models to handle such cases. it brings into question how to define a more expressive representation which can support such complex annotations and how we should design the model capable of handling such cases. we hope that as the state tracking technology advances, there will be more focus on this direction. on the other hand, it is important to ensure that such complex constraints are proportionately represented in the dataset if the system is intended to support them. ## conclusion multiwoz 2.1 @xcite is an improved version of the multiwoz 2.0 dataset, which is extensively used as a benchmark for dialogue state tracking. we identify annotation errors, inconsistencies and ontology related issues which were left unaddressed in multiwoz 2.1, and publish a corrected version -multiwoz 2.2. we added a new schema, standardized slot values, corrected annotation errors and standardized span annotations. furthermore, we annotated active intents and requested slots for each user turn, and added missing user and system actions besides fixing existing ones. we benchmark a few state-of-the-art models on the new dataset: experimental results show that the models' performance is similar between multiwoz 2.1 and multiwoz 2.2. we hope the cleaned dataset helps make fairer comparisons among models and facilitate research in this field.