Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances (2022)

Abstract: The primary purpose of dialogue state tracking(DST), a critical component of an end-toend conversational system, is to build a model that responds well to real-world situations. Although we often change our minds from time to time during ordinary conversations, current benchmark datasets do not adequately reflect such occurrences and instead consist of over-simplified conversations, in which no one changes their mind during a conversation. As the main question inspiring the present study, “Are current benchmark datasets sufficiently diverse to handle casual conversations in which one changes their mind after a certain topic is over?” We found that the answer is “No” because DST models cannot refer to previous user preferences when template-based turnback utterances are injected into the dataset. Even in the the simplest mind-changing (turnback) scenario, the performance of DST models significantly degenerated. However, we found that this performance degeneration can be recovered when the turnback scenarios are explicitly designed in the training set, implying that the problem is not with the DST models but rather with the construction of the benchmark dataset.

## introduction the dialogue state tracking (dst) module is a part of a task-oriented dialogue system, the main role of which is to extract essential information of user preferences from various conversational situations. based on the given information from the previous module, the dst module finds appropriate slotvalue pairs to understand the current conversational situations, and these pairs are then delivered to the next module to continue the conversation. hence, building an accurate dst model is a key success factor of the overall task-oriented dialogue system not only because it can convince users that the system perfectly understands what they are talking about, but also because appropriate responses can be generated based on the result of the dst model. as in other natural language processing (nlp) tasks, two main components are mandatory to build a good dst model: (1) well-structured machine learning models and (2) sufficiently large datasets that contain various real-world conversational situations with fewer biases for training the model. since the introduction of transformer and bert @xcite , various breakthrough model structures have been designed for dst, such as sumbt and som-dst @xcite , and have shown an excellent performance. with respect to dst-specific datasets, by contrast, some benchmark datasets, such as woz @xcite and multiwoz @xcite , have been introduced; however, their sizes and coverage are not yet satisfactory owing to the relatively high labeling cost. for example, the multiwoz only consists of approximately 10,000 dialogues from some different domains, which is significantly smaller than other nlp datasets such as squad or imdb @xcite . whereas the multiwoz has been used as a standard benchmark dataset for dst, there has been an increasing number of recent studies reporting the concerns regarding the inherent limitations of this dataset. first, newer versions of multiwoz have been proposed to address certain issues such as annotation errors, typos, standardization, annotation consistency, and other factors @xcite @xcite . @xcite pointed out an entity bias issue, i.e., only a small number of values in the ontology account for the majority of labels. for example, a large number of 'traindestination' slots take the value 'cambridge' in the @xcite . in addition, with coco @xcite , an overestimation of the held-out accuracy was pointed out by showing that the training and evaluation sets of the multiwoz i would like to find some expensive mexican cuisine. chiquito restaurant bar is an expensive restaurant in the south. would you like me to book it? sure that would be great. okay, is there a particular day and time that would work for you? yes i would like it made for wednesday for 7 people at 18:30 please. booked! your i would like to find some expensive mexican cuisine. chiquito restaurant bar is an expensive restaurant in the south. would you like me to book it? sure that would be great. that's it for now. thanks. thanks. oh, can you find cheaper restaurant? it would be better. restaurant -food (mexican) restaurant -pricerange (expensive) have a similar distribution, and controllable counterfactual goals were proposed that do not change the original dialogue flow but generate a new dialogue with different responses. although previous studies have raised inherent problems in the multiwoz, most have tended to focus on correcting the annotation inconsistency or entity biases, which enforces the dialogue in the dataset to be more idealistic. however, in realworld conversations, the dialogue flow between two speakers is not always as fluent as those in the multiwoz, e.g., one can occasionally change one's mind during a conversation. for example, figure 1a shows a sample dialogue in the multi-woz. no slot that appears once appears again in the subsequent dialogue turns. as the main hypothesis motivating this study, real conversations do not always continue as shown in figure 1a , but often continue as shown in figure 1b . individuals change their mind during a conversation, and thus some slot-value pairs (same slot but different values) repeatedly appear in an entire dialogue. this hypothesis has led us to raise the main question of this paper: "can the current benchmark dataset handle a situation in which users change their mind after a certain amount of turn?" our assumption is that the turnback situation of a user will hamper the robust evaluation of dst models because such models do not have a chance to learn the situation in which the values of specific slots are changed during the conversation. to experimentally verify our assumption, we investigate how dst models handle additional turnback dialogues by injecting template-based utterances under different scenarios on the multiwoz. it is common for users to change their decisions in various ways in the real world, and thus we define four turnback situations as follows: the remaining states are more complicated variants of the simplest versions by modifying the number of repetitions or slots. there are some ways to generate turnback utterances such as manually annotating dialogues or generating with the help of language models @xcite . in this study, we injected turnback utterances at the end of the existing dialogue using pre-defined templates for two reasons. first, locating turnback utterances at the end of the dialogue is a better way to verify the ability handling long-range contexts for the model. second, template-based-generated utterances explicitly mention the information of domain, slot, and value in a raw text, which can play a role as the minimal form of turnback scenarios. we found even these simple and explicit forms of turnback utterances are sufficient to disclose the problem. in this paper, we evaluate the performance of turnback situations with trade, sumbt, and transformer-dst @xcite @xcite . the results show that existing models cannot detect changing user preferences when injecting turnback utterances in the test set; the same trends are also shown in all variants of turnback scenarios. we further determined that including turnback utterances appropriately during the training phase can make a model robust because the model performance rebounds. to summarize, the main contributions of this paper can be summarized as follows: ## method to test whether the model trained with the current dst dataset can track the change in value of the turnback situation, we assume four turn-back scenarios and inject these turnback utterances at the end of every dialogue, as represented as figure 2 . in other words, each data containing dialogue of t turns can be formulated as x @xmath0, u usr 1 ), . . . , (u sys t , u usr t )}, and we then append an extra template-generated turn with one of the aforementioned turnback situations at the end of the existing data, resulting in x @xmath1, u usr 1 ), . . . , (u sys k , u usr k )}, where @xmath2. figure 3 shows examples of a turnback used in each dataset. note that we used different templates for different datasets to avoid an overlap across the datasets. whenever applying a template-based utterance generation, the arbitrary template of each phase is selected at each turn of dialogue. as the main purpose of this paper is to investigate whether the model can follow the user's mindchanging utterances, we designed the simplest form of turnback utterances: injecting them to the last turn and generating utterances using templates. the former is to assume the mind-changing within the longest history in a single dialogue, and the latter is to show that models cannot track changing values when even the most informative turnback utterances are explicitly provided. accordingly, we defined four variants of turnback situations as follows: single turnback users change the value of a particular slot only once, as shown in figure 2a . basically, a single turnback utterance is constructed using the last turn of the dialogue because it contains accumulated belief states that appeared throughout the dialogue. figure a1 shows the process of generating a single turnback utterance and skipping the process when there is no belief stated return turnback users change the value of a particular slot but return to the original value again, as shown in figure 2b . this means that the final belief state after injecting a return turnback utterance is the same as the belief state of the original dataset. in this case, the first turnback utterance can be generated like a single turnback process, and the second turnback utterance is then generated identically by simply replacing the changed value with the original value. dual-value turnback users sequentially change the value of a particular slot twice, as shown in figure 2c . dual value turnback utterances can be generated in the same way as return turnback utterances, but can be generalized to a triple or quadruple value turnback if there are more than two available values in the slot on the ontology. dual-slot turnback users first change the value of a particular slot and then also change the value of a different slot, as represented in figure 2d . this can be generated simply by applying a single turnback twice; however, there must be more than two total belief states to apply this scenario. ## conclusion a dst model should focus on properly reacting to unpredictable scenarios from a human speaker. from this perspective, using realistic benchmark datasets for the model is crucial. to validate recent dst models trained on the commonly used dst benchmark dataset, we first designed a templatebased (but enough to verify the hypothesis) data injection method to create a turnback situation and modified the test dataset by appending one of four trunback scenarios to the end of the dialogue. our experiment showed that the current model trained using the existing benchmark cannot track the changing values well when users change their decisions. we also conducted additional experiment to investigate whether the model performance can be recovered if the turnback utterances are properly included in the training dataset. experimental results showed that the joint goal accuracy was improved for all turnback scenarios when the models were trained on the dataset with turnback utterances. the ablation study shows that moderately including the turnback utterances can manage a broader range of turnback proportions. our experimental results emphasize that constructing a right benchmark dataset is as important as developing an advanced model structure in nlp tasks. despite the meaningful results, we argue that the turnback utterance is just one of many situations that can happen in a real-world conversation. if more diverse realistic dialogue scenarios are reflected in the dst benchmark dataset, the bias of models trained on it can be significantly reduced.