Spoken Language Understanding for Task-oriented Dialogue Systems with Augmented Memory Networks (2021)

Abstract: Spoken language understanding, usually including intent detection and slot filling, is a core component to build a spoken dialog system. Recent research shows promising results by jointly learning of those two tasks based on the fact that slot filling and intent detection are sharing semantic knowledge. Furthermore, attention mechanism boosts joint learning to achieve state-of-the-art results. However, current joint learning models ignore the following important facts: 1. Long-term slot context is not traced effectively, which is crucial for future slot filling. 2. Slot tagging and intent detection could be mutually rewarding, but bi-directional interaction between slot filling and intent detection remains seldom explored. In this paper, we propose a novel approach to model long-term slot context and to fully utilize the semantic correlation between slots and intents. We adopt a key-value memory network to model slot context dynamically and to track more important slot tags decoded before, which are then fed into our decoder for slot tagging. Furthermore, gated memory information is utilized to perform intent detection, mutually improving both tasks through global optimization. Experiments on benchmark ATIS and Snips datasets show that our model achieves state-of-the-art performance and outperforms other methods, especially for the slot filling task.

## introduction task-oriented dialogue systems have attracted significant attention, which have been greatly advanced by deep learning techniques. traditionally, these dialog systems have been built as a pipeline, with modules including spoken language understanding (slu), dialog state tracking, action selection and language generation. among these problems, slu, including intention detection and slot filling @xcite , is a key yet challenging problem to parse users' utterances into semantic frames in order to capture a conversation's core meaning. traditionally, intention detection is treated as a classification problem, whereas slot filling is usually defined as sequence labeling problem, where in-out-begin (iob) format is applied for representing slot tags as illustrated in given an utterance, slu determines users' intention and maps it into predefined semantic slots. the input is a sequence of words, and the output is a sequence of predefined slot ids. a specific intent is assigned for the whole sentence. in the traditional pipeline approach, intent detection and slot filling are implemented separately. however, separate modeling of those two tasks is insufficient to take full advantage of all supervised signals, as they share semantic knowledge. for example, if the intent of an utterance is "find_a_flight", it is more likely to contain slots "de-parture_city" and "arrival_city" rather than "restau-rant_name". another drawback of the pipeline method is that errors made in upper stream modules may propagate and be amplified in downstream components, which however could possibly be eased in joint model @xcite . recently, joint model for intent detection and slot filling has been proposed and achieved promising results @xcite @xcite . though achieving promising performance, their models suffer from two major issues: 1) modeling of slot context. though the latent memory of rnns can model history information, they are inherently unstable over long time sequences because the memories are the rnn hidden states. @xcite observes that rnns tend to focus more on short-term memories and forcefully compress historical records into one hidden state vector. thus, simple rnns cannot preserve long-term slot context of the conversation, which is crucial to future slot tagging. 2) bi-directional interaction between slot filling and intent detection. the majority of joint modeling work has studied how to utilize intent information to improve slot filling performance. however, the beneficial impact of slot information on intent detection is mostly ignored. in fact, slots and intents are closely correlative, thus mutually reinforcing each other. in this paper, we propose a new framework to jointly model intent detection and slot filling in order to achieve a deeper level of semantic modeling. specifically, our model is distinguished from previous work primarily in two ways. ## related works since intent detection can be treated as an utterance classification problem, different classification methods, such as support vector machines (svm) and rnns @xcite , have proposed to solve it. on the other hand, for slot filling, hidden markov models (hmm) and conditional random fields (crf) @xcite @xcite were used to solve slot filling problem. later rnn based methods had become popular. for example, @xcite ; @xcite employed rnns for sequence labeling in order to perform slot filling. alternatively, intent detection and slot filling can be done jointly to overcome the error propagation. @xcite first proposed joint work using rnns for learning the correlation between intent and slots. hakkani-tür et al. (2016) adopted a rnn for slot filling and the last hidden state of the rnn was used to predict the utterance intent. @xcite introduced an attention-based rnn encoder decoder model to jointly perform intent detection and slot filling. an attention weighted sum of all encoded hidden states was used to predict the utterance intent. all those models outperform the pipeline models via mutual enhancement between two tasks. most recently, some work tries to model the intent information for slot filling explicitly in the joint model. @xcite ; @xcite proposed the gate mechanism to explore incorporating the intent information for slot filling. however, as the sequence becomes longer, it is risky to simply rely on the gate function to sequentially summarize and compress all slots and context information in a single vector @xcite . @xcite proposed the bi-model to consider the cross-impact between the intent and slots and achieve state-ofthe-art results. @xcite proposed a hierarchical capsule neural network to model the hierarchical relationship among word, slot, and intent in an utterance. @xcite introduces a sf-id network to establish the interrelated mechanism for slot filling and intent detection tasks. compared with their work, our method explicitly models the long-term slot context knowledge which is beneficial to both slot filling and intent detection. memory network provides a principled approach for modeling long-range dependency which has advanced many nlp tasks such as machine transla-tion @xcite and question answering @xcite . the initial framework of memory networks was proposed by @xcite . following the idea, @xcite proposed an end-to-end memory augmented model that significantly reduced the requirement of supervision during training. key-value memory network @xcite encoded prior knowledge by introducing a key memory structure which storeed facts to address to the relevant memory value. none of them is to model slot context information dynamically especially in single turn conversational systems. in this paper, we demonstrate how memory networks can be used to model long-term slot context knowledge and the interaction between intent detection and slot filling. ## proposed model memory networks show promising results on learning long-range dependency, but they are insensitive to represent temporal dependencies between memories @xcite . rnns tend to be opposite. thus, it makes sense for us to combine those networks together to model long-term slot context information. in this section, we present a specific key-value dynamic memory module to collect and remember slot clues in the dialog context. then context memory is used to enhance the encoder-decoder based model to perform slot filling and intent detection. as illustrated in figure 1 , our proposed model is composed of an encoder-decoder, and a key-value memory module including key-memory, value-memory, a memory read unit, and a memory write unit. given a single-turn dialog, the encoder transforms a word in user utterances into a dense vector by using a shared self-attentive encoder. then the memory network encodes longterm slot context information by incorporating historical slot tags through memory attention and write operations of the memory network. the slot decoder integrates short-term hidden state of self-attention encoder and the long-term slot context generated by attentively reading the value-memory to generate slot tagging at each timestamp. later, intent decoder performs token level intent detection, which is seen as a coarse-grained intent detection result. finally, a fine-grained intent detection is produced by gating memory modules. both intent detection and slot filling are optimized simultaneously via a joint learning scheme. ## conclusion in this paper, we propose a joint model to perform spoken language understanding with an augmented key-value memory to model slot context in order to capture long-term slot information. in addition, we adopt a gating mechanism to incorporate slot context information for intent classification to improve intent detection performance. reciprocally, joint optimization promotes slot filling performance further by memory sharing between those two tasks. experiments on two public datasets show the effectiveness of our proposed model and achieve stateof-the-arts results.