How to Stop an Avalanche? J o D e M : Joint Decision Making through Compare and Contrast for Dialog State Tracking (2022)

Abstract: Dialog state tracking (DST) is a core component in task-oriented dialog systems. Existing state-of-the-art DST model incorporates insight and intuition from the human experience into design of supplementary labels, which greatly assisted the training process of turn-by-turn DST model. Though the turn-by-turn scheme and supplementary labels enabled satisfactory performance on the task, most of the DST models of this fashion label or process the raw dialogue data on the premise that the last turn dialogue state is always correct, which is usually not the case. In this paper, we address the negative impact resulted from the premise above as the avalanche phenomenon. After that, we propose JoDeM, a state-of-the-art DST model which can tackle the Avalanche phenomenon with two mechanisms. First mechanism is a jointly decision making method to extract key information from the dialogue. Second mechanism is a compare and contrast dialogue update technique to prevent error accumulation. Example study and graph analysis are presented to support our claim about the harmfulness of avalanche phenomenon. We also conduct quantitative and qualitative experiments on the high quality MultiWOZ2.3 corpus dataset to demonstrate that the proposed model not only outperforms the existing state-of-the-art methods, but also proves the validity of solving avalanche degradation problem.

## introduction goal-oriented dialog (god) systems, or taskoriented dialogue (tod) systems have recently attracted growing attention and significant progress has been made @xcite @xcite . well-known commercial dialogue systems include the apple siri, amazon alexa, or microsoft cortana. in a complete god system, dialog state tracking (dst) * corresponding author. since the blossom of the application of pretrained language model, the accuracy of dst models has increased tremendously. especially, turn-byturn schematic dst models @xcite with insightful design of auxiliary labels and data structure have dominated the field, where most of the among-the-best works are of this genre @xcite . however, this type of models all suffer from a major flaw, the avalanche phenomenon. the avalanche phenomenon is the result of wrong premise during the labeling process which will only occur in the dst models with turn-by-turn scheme. as oppose to the trending turn-by-turn scheme, early multi-domain dst methods follow a dialog history scheme. model of this scheme takes the whole or window-sized dialogue history as input. it predicts slot value without explicitly discriminating over turns of utterances. despite the benefits of making prediction based on a more comprehensive and complete data at once, dialog history scheme has several drawbacks. the length of dialogues is often too long for pre-trained language model to process. more essentially, processing an entire dialogue at once violates the instant update nature of dst. aligning with the need of instant update, turn-by-turn scheme was proposed @xcite . models of this scheme input the dialogue state generated from the previous turn and the most recent turn utterance and output the up-dated dialogue state. the advantages of turn-byturn scheme resulted in great performance boost, in which most of the among-the-best works are of this scheme. on top of the choice of better scheme, to achieve a superior performance, these state-ofthe-art dst models made the best use of auxiliary labels. basic input of turn-by-turn schematic dst models are the current turn utterance and last turn dialogue state, where the basic output is the updated current turn dialogue state. using only basic output as golden training label inevitably leads to a sub-optimal result due to complexity of dst task. mainstream dst systems typically incorporate supplemental labels to guide the model towards better performance. for example, @xcite obtain key information from the dialogue span directly labels the starting and ending index of the key phrase for dst models to learn span detection. however, the high utilization of supplementary labels in turn-by-turn schematic models have induced a new obstacle in developing a more robust and high quality dst system. in practice, a training instance, which is a turn of dialog in an entire dialogue for turn-by-turn systems, are randomly shuffled along with instance from other dialogues. for convenience and effective training, supplementary labels are made under the assumption that the input previous turn dialogue state is correct. while in a considerable amount of cases, models have to make prediction under incorrect last turn dialog state. in those cases, the supplementary labels will also be incorrect themselves because they are also made under the false assumption. these facts add up to a poor robustness against noisy input, making the final performance way lower than expectation. to reflect this kind of characteristic where errors induce more errors, we name this phenomenon the avalanche phenomenon. although solutions and strategies on similar error accumulation phenomenon are widely explored in auto-regressive characteristic tasks @xcite , there are significant differences, resulting in different approaches to tackle the issue. detailed comparison and analysis can be found in the discussion part of the appendix. in this paper, we propose jodem: joint decision making dst system with a compare and contrast mechanism. as mentioned, there are two major issues that directly contribute to the exis-figure 1 : an example of the avalanche phenomenon creating a deficit between the reality and the ideal on the joint accuracy of a dst systems. the jodem model contains eight modules that divides the whole dst process into three stages. the first stage contains a utterance encoder. the second stage contains four parallel modules, namely, a domain update, a slot gate, a slot type, and a span detection module. the third stage contains a dialogue state update module. as shown in the figure, first, we use bert as the pre-trained language module to embed turn utterance. then, a parallel decision making procedure is adopted by the four modules to extract key information from the embedded utterance. at last, the dialogue state update module designed to address the avalanche phenomenon is applied to output the updated dialogue state. after introducing related work and the details of jodem, we conduct multiple standard and customized evaluation and analysis in this paper to show that not only jodem achieved a state-of-theart performance, but also the reason why it achieved such robustness against the avalanche phenomenon. in short, our contribution is twofold: ## related work depending on the inputs, existing dst models are categorized to history-based and turn-by-turn based @xcite . the former scheme takes the whole or window-sized dialogue history as input to recurrent neural networks or networks @xcite . for example, hjst considers the full dialogue history using a hierarchical rnn @xcite . works such as @xcite treats the entire dialogue as a concatenated sequence while using bi-lstm or rnn as an encoder. there are also works inputting the whole history or window-sized dialogue history into bert such as @xcite . in order to overcome the limitations of historybased scheme mentioned in the introduction, turnby-turn dst systems was developed. typically, model of this scheme takes the previous turn dialogue state and the current turn utterance as input to generate new dialogue state @xcite @xcite . basic label of the dst task is the correct dialogue state at each turn, which is often insufficient for the model to learn from effectively. the most common example of supplementary label is the starting index and ending index of the value phrase utilized in the span-based models @xcite @xcite . @xcite , a turn-by-turn model designed a set of operation-based labels to guide the updating process of dialogue state. heck et al. 2020 defined three copy strategy and labeled the original dialogue state tracking process with more refined information. these attempts have made significant result on the performance by incorporating human knowledge to the training process by applying supplementary labels. however, these labels are created under the assumption that the last dialogue state at every turn is flawless, while in reality it is usually not the case. the gap between ideal and reality creates a major drawback on the performance and robustness. in our jodem model, we not only design our supplementary label base on fine intuition, but also address the drawback resulted from the avalanche phenomenon. ## jodem: joint decision making through compare and contrast the proposed jodem model in figure 2 consists of eight components that are located in three different stages of the dst process. the first stage is the utterance encoder that encodes the basic inputs, i.e., system and user utterance into vector embedding. to the second stage, which is the joint decision making stage. in this stage, key information is extracted from the utterance embedding by the following component, domain update, slot gate, type prediction, span detection and co-ref classification. at the last stage, compare and contrast mechanism is applied by the dialogue state update component to update the dialogue state according to the key information from the second stage and the previous turn dialog state. before formally getting into the detail of the jodem model, we first layout the necessary mathematical notations and proper definition for the dst problem. we define a complete dialogue as @xmath0, u 1 ) , ..., (s t , u t )}, which has t sets, or turns of system and user utterance that are in a sequential order. the dialogue states of an entire dialogue which is a set of dialogue state from all t turns is defined as @xmath1, ..., ds t }, where ds i is the dialogue state of the ith turn. each turn's dialogue state is a set which takes multiple triplets of format (domain, slot, value) as its elements. to complete a dst task is equivalent to the following statement: for any turn t, given the turn utterance (s t , u t ) and the last turn dialogue state ds t-1 as input, we should output ds t , which contains the correct set of triplets (domain, slot, value). ## conclusion we proposed a novel, robust dst model jo-dem to address the rarely discussed problem, the avalanche phenomenon. we showed that the trending topnotch dst systems all suffer from the avalanche phenomenon with quantitative results and evidence. by multiple control experiments, we demonstrated how the overall structure and different techniques served the performance and robustness of the jodem model. we achieved a state-of-the-art performance on joint accuracy and the criterion we design for measuring the impact of the avalanche phenomenon. finally, through the success of jodem, we show that the avalanche phenomenon is worth solving and that there is more potential in this perspective for the dst task.