EDC : Effective and Efficient Dialog Comprehension For Dialog State Tracking (2024)

Abstract: In Task-Oriented Dialog (TOD) systems, Dialog State Tracking (DST) structurally extracts information from user and system utterances, which can be further used for querying databases and forming responses to users. The two major categories of DST methods, sequential and independent methods, face trade-offs between accuracy and efficiency. To resolve this issue, we propose Effective and Efficient Dialog Comprehension (EDC), an alternative DST approach that leverages the tree structure of the dialog state. EDC predicts domains, slot names and slot values of the dialog state step-by-step for better accuracy, and efficiently encodes dialog contexts with causal attention patterns. We evaluate EDC on several popular TOD datasets and EDC is able to achieve state-of-the-art Joint Goal Accuracy (JGA). We also show theoretically and empirically that EDC is more efficient than model designs used by previous works.

## introduction task-oriented dialog (tod) systems are essential for building intelligent virtual assistants and chatbots, and aim to fulfill users' goals by understanding their intentions, performing corresponding actions and responding with results to users. in a tod system, dialog state tracking (dst) extracts information such as user intention and system suggestions from dialog history and represents the known information of the dialog in a structural way, which can be further used for querying entities in databases and generating responses to users. structural dialog state usually consists of one or multiple mappings for different domains, each describing user requirements or system suggestions of one type of entity or service. within a domain, a slot identifies a single property and is represented by the pair of a slot name and a slot value in the domain's corresponding mapping. existing dst methods can be categorized into two types: sequential and independent. both operate on partial dialog history, the concatenation of past user and system utterances up to a certain number of interaction rounds, and henceforth referred to as context. sequential methods predict dialog state in one go by generating a single sequence with present domains and slots. independent methods predict dialog state by separately determining the existence and value of all possible domain-slot combinations. they may be further divided into generative and classification-based methods: generative independent methods predict slot value by sequence-to-sequence generation, while classification-based independent methods remodel slot value prediction as a multiple-choice problem, and select slot value from a list of candidates. between both categories of dst methods, one issue we discovered is the trade-off between accuracy and efficiency. sequential methods are efficient because generation only happens once and non-present domains and slots do not need to be predicted. however, when the dialog state becomes more complex, the target sequence becomes longer and different to generate precisely. independent methods are more precise because they only need to deal with one slot at a time, but are inefficient when dealing with a large amount of domains and slots. by leveraging the hierarchical structure of dialog state, we argue a third approach is possible for dst: viewing dialog state as a three-level tree of present domains, slot names and slot values, and predicting this tree level-by-level. in this way, both effectiveness and efficiency can be achieved: length of each individual target sequence is reduced, and ability to predict only present domains and slots are preserved. in this paper, we propose effective and efficient dialog comprehension (edc), a multi-pass dst method utilizing tree prediction concept mentioned above. edc is based on existing pre-trained lan-guage models, and also learns to predict user and system actions as auxiliary tasks in conjunction to dialog states. in addition, edc also efficiently encodes dialog contexts of all rounds in the same dialog once with causal attention patterns. we evaluate edc on multiwoz 2.2 @xcite and m2m @xcite datasets. edc is able to achieve state-of-the-art joint goal accuracy (jga) among sequential methods, and also outperforms most independent methods. we then perform experiments on ablation settings and alternative designs to justify the design of our method and model. finally, we show both theoretically and empirically that edc is more efficient than traditional model designs during both training and prediction. ## edc: effective and efficient dialog comprehension in this section, we introduce the detailed design of edc. first, we formulate the dst problem and show how dst can be converted into a tree prediction problem. then, we propose an efficient dialog context encoding technique based on the incremental property of dialog context. next, we explain how the dialog state data can be decoded from our model step-by-step. finally, we present the auxiliary tasks edc learns to perform and the optimization objective. ## experiments in this section, we present various experimental results of edc. we start with our training datasets and settings. then, we compare the metrics of edc to those of previous works as well as ablation settings and alternative designs. next, we analyze the efficiency of edc by computing and comparing its theoretical and empirical computational complexity. finally, we categorize edc's dst prediction errors on selected samples. ## (36%) 5 (20%) 4 (16%) anno. ## error analysis finally, we analyze edc's dst prediction errors on multiwoz 2.2. we randomly collect 25 round predictions with unique errors, where unique is defined as at least one new error of a slot is introduced in the current round, as opposed to slot prediction errors carried over from previous rounds. we divide all unique errors into three types: errors can also be classified by their origin: examples of these error categories can be found in appendix section c. ## conclusion we proposed effective and efficient dialog comprehension (edc), an alternative dst method that leverages the tree structure of dialog state, and efficiently encodes dialog contexts with causal attention patterns. edc achieves state-of-the-art dst performance, surpassing all sequential and most independent methods, which being efficient to train both theoretically and empirically. in the future, we plan to extend edc into a full multi-task tod system that can also generate responses to users, in addition to its current capabilities of dst and user and system action prediction. we are also interested in fine-tuning large language models (llm) to perform tod tasks, and bringing structural information awareness to open domain dialog systems powered by llms.