Efficient Dialogue State Tracking by Selectively Overwriting Memory (2020)

Abstract: Recent works in dialogue state tracking (DST) focus on an open vocabulary-based setting to resolve scalability and generalization issues of the predefined ontology-based approaches. However, they are inefficient in that they predict the dialogue state at every turn from scratch. Here, we consider dialogue state as an explicit fixed-sized memory and propose a selectively overwriting mechanism for more efficient DST. This mechanism consists of two steps: (1) predicting state operation on each of the memory slots, and (2) overwriting the memory with new values, of which only a few are generated according to the predicted state operations. Our method decomposes DST into two sub-tasks and guides the decoder to focus only on one of the tasks, thus reducing the burden of the decoder. This enhances the effectiveness of training and DST performance. Our SOM-DST (Selectively Overwriting Memory for Dialogue State Tracking) model achieves state-of-the-art joint goal accuracy with 51.72% in MultiWOZ 2.0 and 53.01% in MultiWOZ 2.1 in an open vocabulary-based DST setting. In addition, we analyze the accuracy gaps between the current and the ground truth-given situations and suggest that it is a promising direction to improve state operation prediction to boost the DST performance.

## introduction building robust task-oriented dialogue systems has gained increasing popularity in both the research and industry communities @xcite . dialogue state tracking (dst), one of the essential tasks in task-oriented dialogue systems @xcite , is keeping track of user goals or intentions throughout a dialogue in the form of a set of slot-value pairs, i.e., dialogue state. because the 1 the code is available at github.com/clovaai/som-dst. figure 1 : an example of how som-dst performs dialogue state tracking at a specific dialogue turn (in this case, fifth). the shaded part is the input to the model, and "dialogue state at turn 5" at the right-bottom part is the output of the model. here, update operation needs to be performed on the 10th and 11th slot. dst at this turn is challenging since the model requires reasoning over the long-past conversation. however, som-dst can still robustly perform dst because the previous dialogue state is directly utilized like a memory. traditional neural dst approaches assume that all candidate slot-value pairs are given in advance, i.e., they perform predefined ontology-based dst @xcite @xcite . most previous works that take this approach perform dst by scoring all possible slot-value pairs in the ontology and selecting the value with the highest score as the predicted value of a slot. such an approach has been widely applied to datasets like dstc2 and woz2.0, which have a relatively small ontology size. @xcite although this approach simplifies the task, it has inherent limitations: (1) it is often difficult to obtain the ontology in advance, especially in a real scenario @xcite , (2) predefined ontologybased dst cannot handle previously unseen slot values, and (3) the approach does not scale large since it has to go over all slot-value candidates at every turn to predict the current dialogue state. indeed, recent dst datasets often have a large size of ontology; e.g., the total number of slot-value candidates in multiwoz 2.1 is 4510, while the numbers are much smaller in dstc2 and woz2.0 as 212 and 99, respectively @xcite . to address these issues, recent methods employ an approach that either directly generates or extracts a value from the dialogue context for every slot, allowing open vocabulary-based dst @xcite @xcite . while this formulation is relatively more scalable and robust to handling unseen slot values, many of the previous works do not efficiently perform dst since they predict the dialogue state from scratch at every dialogue turn. in this work, we focus on an open vocabularybased setting and propose som-dst (selectively overwriting memory for dialogue state tracking). regarding dialogue state as a memory that can be selectively overwritten (figure 1 ), som-dst decomposes dst into two sub-tasks: (1) state operation prediction, which decides the types of the operations to be performed on each of the memory slots, and (2) slot value generation, which generates the values to be newly written on a subset of the memory slots (figure 2 ). this decomposition allows our model to efficiently generate the values of only a minimal subset of the slots, while many of the previous works generate or extract the values of all slots at every dialogue turn. moreover, this decomposition reduces the difficulty of dst in an open-vocabulary based setting by clearly separating the roles of the encoder and the decoder. our encoder, i.e., state operation predictor, can focus on selecting the slots to pass to the decoder so that the decoder, i.e., slot value generator, can focus only on generating the values of those selected slots. to the best of our knowledge, our work is the first to propose such a selectively overwritable memorylike perspective and a discrete two-step approach on dst. ## previous open vocabulary-based dst many works on recent task-oriented dialogue datasets with a large scale ontology, such as mul-tiwoz 2.0 and multiwoz 2.1, solve dst in an open vocabulary-based setting @xcite @xcite . @xcite show the potential of applying the encoder-decoder framework @xcite to open vocabulary-based dst. however, their method is not computationally efficient because it performs autoregressive generation of the values for all slots at every dialogue turn. @xcite tackle the drawback of the model of @xcite , that their model generates the values of all slots at every dialogue turn, by using a hierarchical decoder. in addition, they come up with a new notion dubbed inference time complexity (itc) to compare the efficiency of different dst models. itc is calculated using the number of slots j and the number of corresponding slot values m . foot_0 following their work, we also calculate itc in appendix b for comparison. @xcite introduce another work that tackles the efficiency issue. to maximize the computational efficiency, they use a non-autoregressive decoder to generate the slot values of the current dialogue state at once. they encode the slot type information together with the dialogue context and the delexicalized dialogue context. they do not use the previous turn dialogue state as the input. @xcite process the dialogue context in both domain-level and slot-level. they make the final representation to generate the values using a late fusion approach. they show that there is a performance gain when the model is jointly trained with response generation. however, they still generate the values of every slot at each turn, like @xcite . @xcite formulate dst as a reading comprehension task and propose a model named dst reader that extracts the values of the slots from the input. they introduce and show the importance of the concept of a slot carryover module, i.e., a component that makes a binary decision whether to carry the value of a slot from the previous turn dialogue state over to the current turn dialogue state. the definition and use of discrete operations in our work is inspired by their work. @xcite target the issue of illformatted strings that generative models suffer from. in order to avoid this issue, they take a hybrid approach. for the slots they categorize as picklistbased slots, they use a predefined ontology-based approach as in the work of @xcite ; for the slots they categorize as span-based slots, they use a span extraction-based method like dst-reader @xcite . however, their hybrid model shows lower performance than when they use only the picklist-based approach. although their solely picklist-based model achieves state-of-the-art joint accuracy in multiwoz 2.1, it is done in a prede-fined ontology-based setting, and thus cannot avoid the scalability and generalization issues of predefined ontology-based dst. ## selectively overwriting memory for dialogue state tracking figure 2 illustrates the overview of som-dst. to describe the proposed som-dst, we formally define the problem setting in our work. ## analysis 6.1 choice of state operations we also investigate the performance when only three operations are used or two more state operations, yes and no, are used. yes and no represent the cases where yes or no should be filled as the slot value, respectively. the performance drops in all of the cases. ## conclusion we propose som-dst, an open vocabulary-based dialogue state tracker that regards dialogue state as an explicit memory that can be selectively overwritten. som-dst decomposes dialogue state tracking into state operation prediction and slot value generation. this setup makes the generation process efficient because the values of only a minimal subset of the slots are generated at each dialogue turn. som-dst achieves state-of-the-art joint goal accuracy on both multiwoz 2.0 and multiwoz 2.1 datasets in an open vocabulary-based setting. som-dst effectively makes use of the explicit dialogue state and discrete operations to perform relatively robust dst even in complicated conversations. further analysis shows that improving state operation prediction has the potential to increase the overall dst performance dramatically. from this result, we propose that tackling dst with our proposed problem definition is a promising future research direction. inference time complexity (itc) proposed by @xcite defines the efficiency of a dst model using j, the number of slots, and m , the number of values of a slot. going a step further from their work, we report itc of the models in the best case and the worst case for relatively more precise comparison.