Knowledge-Aware Graph-Enhanced GPT -2 for Dialogue State Tracking (2021)

Abstract: Dialogue State Tracking is central to multi-domain task-oriented dialogue systems, responsible for extracting information from user utterances. We present a novel hybrid architecture that augments GPT-2 with representations derived from Graph Attention Networks in such a way to allow causal, sequential prediction of slot values. The model architecture captures inter-slot relationships and dependencies across domains that otherwise can be lost in sequential prediction. We report improvements in state tracking performance in MultiWOZ 2.0 against a strong GPT-2 baseline and investigate a simplified sparse training scenario in which DST models are trained only on session-level annotations but evaluated at the turn level. We further report detailed analyses to demonstrate the effectiveness of graph models in DST by showing that the proposed graph modules capture inter-slot dependencies and improve the predictions of values that are common to multiple domains.

## introduction this paper investigates two aspects of dialogue state tracking (dst) for multi-domain taskoriented dialogue @xcite . we present a novel hybrid architecture that augments gpt-2 @xcite with dialogue act representations derived from graph attention networks (gats) @xcite in such a way that allows causal, sequential prediction of slot values while explicitly modelling the relationships between slots and values across domains. our approach uses gats to improve predictions of values that are shared across domain-slots and that might otherwise be treated independently. as a related line of work, we investigate a form of sparsely supervised dst training and find that our hybrid architecture offers improved robustness with weak supervision. dst can be improved by modelling the relationship between slots and values across domains. this has been explored recently by @xcite who suggest three types of relationships between domain-slots pairs that can be modelled explicitly: (1) pairs that share the same candidate set, such @xmath0; (2) pairs whose candidate values are subsets, as could happen @xmath1; and (3) correlated values between domain-slot pairs, such as when the 'star' level of a booked hotel correlates with the price range of a reserved restaurant. graph neural networks (gnns) have been proposed to captures the interactions among slots and values and to improve dst performance @xcite @xcite . these relationships can be represented as edges in graph-based models, where domains, slots, and values are nodes in the graphs. however previous work has not explored quantitatively or in depth how graph models utilize the relationships they model. @xcite and @xcite provide example cases where the predictions of correlated values were potentially enhanced by their model, while @xcite and @xcite present ablation studies showing marginal improvements brought by their graph modules. @xcite and @xcite further show joint accuracies over different dialogue turns, but there is more that can be said about how gats can improve dst. one of the aims of this paper is to more deeply analyze how graph models can lead to improved dst on top of an already good gpt-2 baseline system. graph models may also compensate for some potential drawbacks associated with using generative models for dst. as a well-known generative model, gpt-2 offers powerful, left-to-right generation incorporating a causal attention mechanism. we note that hosseini-asl et al. (2020) have demonstrated that gpt-2 can identify slot values as a prediction task, with variable length token sequences produced sequentially with interspersed special tokens indicating slot boundaries. the ability to easily generate token sequences of arbitrary lengths is a valuable feature of the model, although it may come at the expense of modelling power relative to models with non-causal attention mechanisms, such as bert @xcite . in particular, gpt-2's causality requires that the prediction of later slot values can depend explicitly on previously predicted slot values, but that the reverse is not possible. this can lead to decreased performance in predicting slot values that occur early on. we find that augmenting gpt-2 prediction with representations derived from gats allows some sharing of information between slots prior to prediction to improve this gpt-2 limitation. capturing the relationships of slot values across domains also offers the opportunity to make better use of limited training data, particularly in sparsely supervised and weakly supervised scenarios @xcite . in a 'last turn' annotation scenario, annotations are available only for the final turn of a task-oriented dialogue. this is unlike the fullyannotated multiwoz setting, which offers turnlevel annotations throughout the entire dialogue session. as an annotation option, generating summary annotations at the completion of a recorded session is an attractive alternative to creating a detailed, turn-by-turn annotation of the entire dialogue @xcite . if it is possible to use only these session-level annotations to train a dst system that still achieves acceptable tracking performance, the chore of creating new annotated dst datasets could be made much easier. the challenges in using this summary data are significant, however. using only the final-turn annotations in multiwoz 2.0 reduces the training set to 14.3% of its original size (in annotated turns). we summarize the contributions of our work as follows: while we do show dst accuracy improvements over a strong gpt-2 baseline, we emphasise that our aim is mainly to investigate and improve prediction of domain-slot values using relationships that otherwise are left unmodelled by the baseline. ## related work statistical dst prioritises general and extensible systems based on machine-learning architectures @xcite @xcite . systems must be able to predict slot values from domain-specific lists such as list of hotel names as well as from more open-ended categories such as days, prices, and times. recent trends are to combine several strategies to deal differently with the two types of values @xcite @xcite . for example, @xcite combine a span predictor for non-enumerable slot values and a cosine similarity matching that exploits a bert model to extract representations for enumerable slot values, with a dual-strategy model jointly handling both types of slot values; @xcite use both a span predictor and a candidate classifier and combine their predictions with gating functions. our work is based on gpt-2 and we note that generative models such as gpt-2 are less widely used in dst tasks, possibly because they raise additional challenges for information aggregation subject to the causality, as discussed in sec. 1. however these recent results show that these models can yield competitive dst accuracy @xcite . previous work has addressed sharing information between slots either by explicitly copying values @xcite or by sharing embeddings @xcite @xcite . beyond copying and sharing, as we note in sec. 1 @xcite developed a graph attention network, and chen et al., 2020 also developed a schema-guided multi-domain approach embedding slot relations in edges of graph neural networks. @xcite enhanced a strong base model som-dst @xcite with a schema graph to exploit relations among domain-slots. gcdst @xcite uses a state graph to transfer domain-slot features and hard-copy states directly from historical states. ## graph neural networks in this section we review graph attention networks (gats) @xcite as will be used in this paper. a weighted undirected graph at each dialogue turn t is defined as @xmath2, e) with a node set v consisting of n nodes {v i }, and an edge set e containing all edges between nodes. we define an n × n binary symmetric adjacency matrix s, where [s] @xmath3, v j ) / ∈ e and 1 otherwise. associated with each node v i are feature vectors x @xmath4. these are gathered into matrices x of dimension n × f , where f is the input feature size. note that sx is mathematically equivalent to passing the features of each graph node to its neighbours. in this way s k @xmath5. as illustrated in fig. 1 , @xmath6, while @xmath7. a gat layer transforms an input @xmath8. each k-hop gat layer consists of p attention heads a (p) which incorporate @xmath9, ..., k -1 rounds of feature aggregation (as shown in fig. 1 ) across the graph as gats are formed as a cascade of l gat layers g ℓ , each with its own multi-headed graph attention mechanisms a p ℓ . at time t, the gat transforms a set of input features note that in this paper, we set output dimension @xmath10. ## dialogue state tracking with gpt-2 and graph neural networks we take a three-step approach to incorporating gnns into gpt-2 for dialogue state tracking (see fig 2 ). at each turn we first present gpt-2 with the dialogue history to generate features for all possible domain-slots and values in the ontology. these features are then fed into a gat which captures relationships amongst domain-slots and values. the features produced at the output layer of the gat are then incorporated into a second application of gpt-2 which performs the actual prediction of the dialogue state values. ## experiments we report dialogue state tracking performance on multiwoz 2.0 @xcite with its multi-domain goal-oriented dialogue conversations and annotations. for direct comparison to the previous literature, we use the same preprocessing as @xcite and @xcite . two metrics are used for evaluating the model performance: slot accuracy measures the ratio of successful slot value predictions among all the slots of each dialogue turn in ground-truth. joint goal accuracy compares the predicted belief state to the ground truth at every dialogue turn, and the output is considered correct only if all the predicted slot values exactly match the ground truth values. ## dst performance we first compare our model with baseline systems. as shown in in the sparsely-supervised scenario, the performance of the baseline gpt-2 model drops to 48.07% joint accuracy (l0p0k0-nograph, the sparsely supervised scenario further shows the value of augmenting gpt-2 with representations derived from gats ( however, multiple gat layers (@xmath11, @xmath12, models 7,8,9, we summarize our findings as below: in the following sections (sec. 6.1, 6.2, and 6.3), we investigate how graph modules improve the performance of the base fine-tuned gpt-2 model. ## conclusion we presented a novel hybrid architecture that augments gpt-2 with representations derived from graph attention networks in such a way to allow causal, sequential prediction of slot values. our analysis shows that these graph-enhanced models mitigate some of the issues that arise in prediction with left-to-right generative models. we also demonstrate that our model can exploit dependencies among domain-slot values, improving accuracy for systems trained with weak supervision.