Annotation Inconsistency and Entity Bias in M ulti WOZ (2021)

Abstract: MultiWOZ (Budzianowski et al., 2018) is one of the most popular multi-domain taskoriented dialog datasets, containing 10K+ annotated dialogs covering eight domains. It has been widely accepted as a benchmark for various dialog tasks, e.g., dialog state tracking (DST), natural language generation (NLG) and end-to-end (E2E) dialog modeling. In this work, we identify an overlooked issue with dialog state annotation inconsistencies in the dataset, where a slot type is tagged inconsistently across similar dialogs leading to confusion for DST modeling. We propose an automated correction for this issue, which is present in 70% of the dialogs. Additionally, we notice that there is significant entity bias in the dataset (e.g., “cambridge” appears in 50% of the destination cities in the train domain). The entity bias can potentially lead to named entity memorization in generative models, which may go unnoticed as the test set suffers from a similar entity bias as well. We release a new test set with all entities replaced with unseen entities. Finally, we benchmark joint goal accuracy (JGA) of the state-of-theart DST baselines on these modified versions of the data. Our experiments show that the annotation inconsistency corrections lead to 7-10% improvement in JGA. On the other hand, we observe a 29% drop in JGA when models are evaluated on the new test set with unseen entities.

## introduction commercial virtual assistants are used by millions via devices such as amazon alexa, google assistant, apple siri, and facebook portal. modeling such conversations requires access to high quality and large task-oriented dialog datasets. many researchers have devoted great efforts to creating such datasets and multiple task-oriented dialog datasets, e.g., @xcite , multiwoz @xcite , taskmaster @xcite , schema-guided dialog @xcite with fine-grained dialog state annotation have been released in the recent years. among task-oriented dialog datasets, multi-woz @xcite has gained the most popularity. the dataset contains 10k+ dialogs and covers eight domains: attraction, bus, hospital, hotel, restaurant, taxi, train and police. each dialog can cover one or multiple domains. the inclusion of detailed annotations, e.g., task goal, dialog state, and dialog acts for both user side and system side, renders multiwoz a universal benchmark for many dialog tasks, such as dialog state tracking @xcite @xcite , dialog policy optimization (yang @xcite and end-to-end dialog modeling @xcite @xcite . several recent papers, such as simpletod @xcite , trade @xcite , marco @xcite , evaluate their models solely on the mul-tiwoz dataset, which makes their findings highly dependent on the quality of this dataset. over the last couple of years, several sources of errors have been identified and corrected on multi-woz. @xcite pre-processed the dataset by normalizing the text and annotations. @xcite further corrected the dialog state annotations on over 40% dialog turns and proposed multiwoz 2.1. recently, @xcite identified some more error types, fixed annotations from nearly 30% dialogs and added span annotations for both user and system utterances, leading to the multi-woz 2.2. concurrent with this work, @xcite released multiwoz 2.3 further looking at slot value count num. cambridge 8,086 london liverpool street 760 leicester 746 stansted airport 711 stevenage 710 ely 695 norwich 692 bishops stortford 667 broxbourne 634 peterborough 630 birmingham new street 624 london kings cross 609 kings lynn 574 total 16,138 annotation consistency by exploring co-references. while most of the previous works focus on correcting the annotation errors and inconsistencies within a dialog, where annotation contradicts the dialog context, we noticed another overlooked source of confusion for dialog state modeling, namely annotation inconsistency across different dialogs. we first show that the dialogs have been annotated inconsistently with respect to the slot type 'name'. figure 1 shows two dialogs in the attraction domain with similar context. in one dialog the attraction name is annotated while not in the other one. this inconsistency leads to a fundamental confusion for dialog state modeling whether to predict the attraction name or not in similar scenarios. in section 3, we dive deeper into this problem and propose an automated correction for this problem. we further found a second source of potential issue, entity bias, where the distribution of the slot value in the dataset is highly imbalanced. in figure 2 , we observed that "cambridge" appears as the train destination in 50% of the dialogs in train domain while there are 13 destinations. as a result, a dialog system trained on this imbalanced data might be more likely to generate "cambridge" as the slot value even though "cambridge" might not even be mentioned in dialog history. in section 4, we discuss this problem in more detail and suggest a new test set with all entities replaced with ones never seen during training. finally, in section 5, we benchmark the state-of-the-art dialog state tracking models on these new versions of data and conclude with our findings. our contributions in this paper can be summarized as follows: to solve those kinds of errors, eric et al. (2019) adopted both manual corrections and automated corrections. after asking the human annotator to go over each dialog turn-by-turn, they also wrote scripts to canonicalize slot values to match the entities from the database. besides, they also kept multiple slot values for over 250 turns in the case that multiple values are included in the dialog context. in addition to correcting dialog states, multiwoz 2.1 also corrects typos within dialog utterances for better research exploration of copy-based dialog models. as a result, over 40% of turns (around 30% of dialog state annotations) are corrected. finally, multiwoz 2.1 also adds slot description for exploring few-shot learning and dialog act based on the pipeline from @xcite . ## annotation inconsistency multiwoz is collected following the wizard-of-oz setup @xcite , where each dialog is conducted by two crowd-workers. one crowd-worker plays the role of a human user and another one plays the role of the dialog system. the user would be assigned wh a task goal, which describes the information of the task object and is required to conduct the itconversation sticking to the task goal. the dialog system is required not only to respond to the user and complete the task, but also to take down the task-related information in the form of the slot values. since the slots is annotated by the crowd-worker, and different dialogs employ different crowd-workers, the strategies they decide whether to take the information down are also different. especially for the information provided by the dialog system, some crowd-workers decide to take it down while some do not. in figure 1 , we show two dialogs with similar context. both of the users ask for an architectural attraction in the centre area, and both of the dialog systems response with a result, all saints church. then, both users acknowledge the result and ask for more information on the result. however, in the left dialog, the attraction name, all saints church, is not annotated as one of the slot values, whereas the right dialog includes it in the dialog state. the source of this discrepancy may be that the annotator in the left dialog thinks the dialog system already knows this information and only information provided by the user should be annotated. on the other hand, the annotator in the right dialog might notice that the user has acknowledged the result, attraction name, and asked questions based on this result, hence it should be included in the dialog state. having said that, the system cannot answer such follow-up questions without the annotation of the attraction name in the dialog state. more examples of the inconsistency from other slot types or domains are listed in figure 10 in appendix. after exploring all the dialogs from the test set, we manually examined and corrected each test dialog turn by adding the missing annotations. we first checked whether there are any missing annotations from the user utterance. for example, in the sixth turn of dialog "mul0690.json", user is asking for "a moderate hotel with free wifi and parking". however, the token "moderate" from "pricerange" type is not included in the annotations. so, we added the slot "hotel pricerange moderate" to this turn. on the system side, for each dialog turn, we identified a possible slot value in the system response which is not included in the annotations, e.g., "all saints church" in the left dialog ("mul1088.json) in figure 1 , which is determined to be a possible slot value from "attraction" domain and slot type "name" based on the database and ontology file. then, we examined those dialogs with slot annotations of "attraction" domain and slot type "name". if we could find such dialogs with similar dialog context and containing annotations of the same domain and slot type, we complemented the annotations by adding the missing slot value. in figure 4 , we illustrate the fraction of added annotations that come from the user utterance vs. system side. each row corresponds to a slot type from a certain domain. as can be seen, the majority of the added utterances are from the system side, which confirms our original hypothesis: annotators often have no disagreement to take down information from the user utterance. however, they have different opinions about whether to annotate slots based on system responses. for the training and validation sets, we write regular expressions that match the test set corrections and apply the scripts to automatically correct the annotations based on the database and ontology file, and modify the dialogs automatically. on average, about 20% of the dialogs involved slot modification for each slot type. the "name" and "type" slot types involve the most modification with around 40%. as we mainly focus on the missing slots extracted from system responses in these automated scripts, we ignore the slot types that can be solely modified by the user utterance, such as "book day" and "book people". as can be seen, this process resulted in modification of totally more than 70% of the dialogs . to verify the correctness of the automated correction method, we randomly sampled 100 modified dialogs and 100 unchanged dialogs, and check them manually. the verification result is shown in the ## entity bias as discussed previously, another issue that we observe with multiwoz is the entity bias (e.g., "cambridge" appears in train destination in the majority of dialogs -figure 2 ). besides the "traindestination" slot type, we further explore the similar bias problem in all other 30 slot types. for each slot type, we quantify the frequency at which each possible slot value appears in the training data. we quantify the entity bias using two metrics. for a vector @xmath0, . . . , r r ) of frequencies of r entities, we define the normalized shannon entropy as normalized shannon entropy is bounded between 0 and 1, where h 1 /h 0 = 0 implies a deterministic distribution with all weight on a single entity and 1 implies a perfectly uniform distribution. since shannon entropy does not capture the tail of the distribution, we also report the normalized minentropy, which is given by min-entropy captures the normalized likelihood of the most frequently appearing entity in the list of all possibilities. for example, as in figure 2 , frequency of the entity "cambridge" is about 50% which is much higher than 7% (which would have been its frequency had all 13 possible entities were uniformly distributed). the entity bias for all 30 slots in the dataset is depicted in on the other hand, the slot types involving day and people seem to be nicely balanced, such as "hotel-day" and "hotel-people". this might be because those values are actually uniformly made up, while values of other slot types like "type", "food" are real values and indeed follow certain real-world distributions. these entity biases are potentially amplified by the learning models, which would lead to biased generation. in figure 5 , we show one such case from simpletod @xcite where in the current turn, the user is providing the information of "day" in the "train" domain. the dialog state tracking model successfully extracts the token "wednesday" and updates dialog states in the red rectangle. however, the model also adds the dialog state "train destination cambridge", while "cambridge" has never been mentioned in the dialog history, which is potentially explained by the severe entity bias present in the "train-destination." different from the annotation inconsistency problem, we do not make any modification to the training dataset based on our observation with respect to the entity bias. strictly speaking, bias cannot be considered as a source of error in the dataset, and it needs to be tackled via better modeling efforts. although the entity bias hurts the prediction accuracy of low-frequency slots and results in generating extra high-frequency slots, it also reflects certain real-world facts/biases as the dialogs are conducted by humans. this usually helps the learning task with limited training data, e.g., dialog domain adaptation @xcite . while we keep the bias in the original dataset intact, we propose a new test set with all entities replaced with new ones unseen in the training data to facilitate the identification of whether models capitalize on such biases. for each slot type from each domain in the multiwoz, we find a similar slot type in the schema-guided dataset @xcite . for the slot values belonging to those slot type, we replace them with unseen values from the schema-guided dataset. examples of dialog with replaced entities, along with predicted slots by our benchmark model is shown in figure 6 and figure 11 (appendix). ## benchmarking state-of-the-art models to verify our corrections of the dialog state annotations, we benchmark state-of-the-art dialog state tracking (dst) models on our modified dataset. traditionally, for dst task, the slot value is predicted by selecting from pre-defined candidates or extracting from dialog context. we adopt trade @xcite as a representative of the mixture of these two methods. more recent works focus more on fine-tuning pre-trained model, which purely generates slot values based on dialog history. we choose simpletod @xcite and fine-tuned bart @xcite as benchmark models for dst as well. trade @xcite integrates grubased @xcite encoder-decoder model and pointer-generator @xcite to learn to copy slot values either from the dialog context or from the pre-defined value candidates. simpletod (hosseini-asl et al., 2020) builds a dst model by fine-tuning gpt2 @xcite , a large pre-trained language model. it combines all the condition information, including dialog history, previous dialog states and user utterance into a single sequence as input and let the language model learn to generate a sequence, containing dialog states and system response. here we only feed in dialog states as ground truth output during training step, so that the trained model is specially designed for dst. dst-bart builds a dst model by fine-tuning bart @xcite on the dst task in multiwoz. bart is a denoising autoencoder pretrained with corrupted text, making it more robust to noisy data. it consists of a bidirectional encoder and a left-to-right autoregressive decoder. ## concluding remarks multiwoz is a well-annotated task-oriented dialog dataset, and widely used to evaluate dialog-related tasks. previous works like multiwoz 2.1 and mul-tiwoz 2.2 have carefully identified and corrected several errors in the dataset, especially for the dialog state annotations. building on multiwoz 2.2, we identified annotation inconsistency across different dialogs as a source of confusion for training dialog state tracking models. we proposed a correction and released a new version of the data with corrections. we also identified named entity bias as another source of issue, and released a new test set with all named entities replaced with unseen ones. finally, we benchmarked a few state-of-the-art dialog state tracking models on the new versions of the data, showing 5-10% performance improvement on the new corrected data, and 29% performance drop when evaluation is done on the new test set with replaced entities. we hope the better understanding of multiwoz helps us gain more insights into dialog evaluation on this dataset. while we corrected some errors in this work, we observe a few remaining problems in multiwoz. first, there are some cases where the annotation contradicts with the database. for example, in dialog "mul2523.json", the user is asking about "autumn house", which is annotated as type of "guest house" in the database. however, the dialog state annotation labels the hotel type as "hotel". this disagreement might not hurt the training of dialog state tracking models, but affects the now popular end-to-end dialog models, which are trained on multiwoz. there still some annotation errors. for example, in dialog "mul0072.json", the "monday" has never been mentioned, while the dialog states include the slot "hotel day monday".