Conversational Semantic Parsing for Dialog State Tracking (2020)

Abstract: We consider a new perspective on dialog state tracking (DST), the task of estimating a user’s goal through the course of a dialog. By formulating DST as a semantic parsing task over hierarchical representations, we can incorporate semantic compositionality, cross-domain knowledge sharing and co-reference. We present TreeDST, a dataset of 27k conversations annotated with tree-structured dialog states and system acts. We describe an encoder-decoder framework for DST with hierarchical representations, which leads to ~20% improvement over state-of-the-art DST approaches that operate on a flat meaning space of slot-value pairs.

## introduction task-based dialog systems, for example digital personal assistants, provide a linguistic user interface for all kinds of applications: from searching a database, booking a hotel, checking the weather to sending a text message. in order to understand the user, the system must be able to both parse the meaning of an utterance and relate it to the context of the conversation so far. while lacking the richness of a conversation between two humans, the dynamics of human-machine interaction can still be complex: the user may change their mind, correct a misunderstanding or refer back to previously-mentioned information. language understanding for task-based dialog is often termed "dialog state tracking" (dst) @xcite , the mental model being that the intent of the user is a partially-observed state that must be re-estimated at every turn given new information. the dialog state is typically modelled as a set of independent slots, and a standard dst system will maintain a distribution over values for each slot. in contrast, language understanding for other nlp applications is often formulated as semantic parsing, which is the task of converting a single-turn utterance to a graphstructured meaning representation. such meaning representations include logical forms, database queries and other programming languages. these two perspectives on language understanding-dst and semantic parsinghave complementary strengths and weaknesses. dst targets a fuller range of conversational dynamics but typically uses a simple and limiting meaning representation. semantic parsing embraces a compositional view of meaning. by basing meaning on a space of combinable, reusable parts, compositionality can make the nlu problem space more tractable (repeated concepts must only be learned once) and more expressive (it becomes possible to represent nested intents). at the same time, most semantic parsing research treats a sentence as an isolated observation, detached from conversational context. this work unifies the two perspectives by reformulating dst as conversational semantic parsing. as in dst, the task is to track a user's goal as it accumulates over the course of a conversation. the goal is represented using a structured formalism like those used in semantic parsing. specifically, we adopt a hierarchical representation which captures domains, verbs, operators and slots within a rooted graph grounded to an ontology. the structured dialog state is capable of tracking nested intents and representing compositions in a single graph (turn 5 turn utterance and annotation 1 hi can you book me a flight to paris please. user.flight.book.object.equals .destination.equals.location.equals.paris sure, when and where will you depart? system.prompt.flight.book.object.equals .source .departuredatetime 2 tomorrow from london user.flight.book.object.equals .destination.equals.location.equals.paris .source.equals.location.equals.london .departuredatetime.equals .date.equals.definedvalue.equals.tomorrow i found 5 flights for you. the earliest one departs at 10 am with a cost of £105. would you like it? system .prompt.flight.book.object.equals .departuredatetime.equals.time .inform.flight.find .count.equals.5 .object.equals .departuredatetime.equals.time.equals .hour.equals.10 .meridiem.equals.am .price.equals.105 3 do i have any calendar event on that day? user.calendarevent.checkexistence.object.equals .datetimerange.equals .date.equals.definedvalue.equals.tomorrow no you don't have any event. system.inform.calendarevent.find.notexisted 4 i would book the 10 am flight for me please user.flight.book.object.equals .destination.equals.location.equals.paris .source.equals.location.equals.london .departuredatetime.equals .date.equals.definedvalue.equals.tomorrow .time.equals .hour.equals.10 .meridiem.equals.am here is your booking information. please confirm. system.offer.flight.book 5 direction to my next meeting. user.navigation.find.object.equals .destination.equals.reference .calendarevent.object.equals .listoffset.equals.1 here is the direction to your meeting at the glasshouse. system .navigation.inform.find .calendarevent.inform.find.object.equals .location.equals.glasshouse by @xcite and @xcite , we have collected a large dataset of task-oriented dialogs annotated with hierarchical meaning representations. each dialog was generated through a two-step process. first, a generative dialog simulator produces a meaningful conversational flow and a template-based utterance for each turn in the conversation. then the utterances are paraphrased by human annotators to render more realistic and natural conversations. the resulting dataset, which we call treedst, covers 27k conversations across 10 domains. conversations in treedst are non-linear: they contain glitches which represent system failures and uncooperative user behaviors such as underand over-specifying slot information. there are also use cases not addressed in existing slot-filling datasets, including compositional intents and multiintent utterances. the second contribution of this work is a conversational semantic parser that tackles dst as a graph generation problem. at each turn, the model encodes the current user utterance and representations of dialog history, based upon which the decoder generates the updated dialog state with a mixture of generation and copy mechanism. in order to track practical conversations with intent switching and resumption @xcite , we adopt a stack @xcite to represent multiple tasks in the dialog history, and a parent-pointer decoder to speed up decoding. we conducted controlled experiments to factor out the impact of hierarchical representations from model architectures. overall our approach leads to 20% improvement over stateof-the-art dst approaches that operate on a flat meaning space. on the other hand, single-turn semantic parsers have long used structured meaning representations to address compositionality @xcite @xcite @xcite . solutions range from chart-based constituency parsers @xcite to more recent neural sequence-to-sequence models @xcite . the general challenge of scaling semantic parsing to dst is that dialog state, as an accumulation of conversation history, requires expensive context-dependent annotation. it is also unclear how utterance semantics can be aggregated and maintained in a structured way. in this work we provide a solution to unify dst with semantic parsing. ## dialog state tracking the objective in the conversational semantic parsing task is to predict the updated dialog state at each turn given the current user input and dialog history p (y u t |x u t , @xmath0, @xmath1. we tackle the problem with an encoder-decoder model: at turn t, the model encodes the current user utterance x u t and dialog history, conditioned on which the decoder predicts the target dialog state y u t . we call this model the tree encoder-decoder, or ted. ## experiments setup we split the treedst data into train (19,808), test (3,739) and development (3,733) sets. for evaluation, we measure turn-level dialog state exact match accuracy averaged over all turns in the test set. we evaluate the proposed model with its "vanilla" decoder (ted-vanilla) and its parent-pointer variant (ted-pp). in both cases, the utterance encoder has 2 layers of 500 dimensions; the system act encoder and dialog state encoder have 2 layers of 200 dimensions; and the decoder has 2 lstm layers of 500 dimensions. dimensions of word and tree node embeddings are 200 and 50 respectively. training uses a batch size of 50 and adam optimizer @xcite . validation is performed every 2 epochs and the training stops when the validation error does not decrease in four consecutive evaluations. the hyper-parameters were selected empirically based on an additional dataset that does not overlap with treedst. baselines in order to factor out the contribution of meaning representations from model changes in decoders accuracy ted-vanilla 0.622 ted-pp 0.622 ted-flat 0.535 comer @xcite 0.509 trade @xcite 0.513 we also introduce as baselines two state-ofthe-art slot-filling dst models based on encoderdecoders: they include comer @xcite which encodes the previous system response transcription and the previous user dialog state and decodes slot values; and trade @xcite which encodes all utterances in the history. since both ted-flat and the two baselines are trained with flattened slot-value representations, we can compare various models in this setup. treedst flattening to flatten treedst we collapse each path from the domain to leaf nodes into a single slot. verb nodes in the path are excluded to avoid slot explosion. take the following tree as an example: the operator equals is not shown in the slot names to make the names more concise. ## conclusion this work reformulates dialog state tracking as a conversational semantic parsing task to overcome the limitations of slot filling. dialog states are represented as rooted relational graphs to encode compositionality, and encourage knowledge sharing across different domains, verbs, slot types and dialog participators. we demonstrated how a dialog dataset with structured labels for both user and system utterances can be collected with the aid of a generative dialog simulator. we then proposed a conversational semantic parser that performs dst with an encoder-decoder model and a stackbased memory. a parent-pointer decoder is further proposed to speed up tree prediction. experimental results show that our dst solution outperforms slot-filling-based trackers by a large margin.