Infusing Context and Knowledge Awareness in Multi-turn Dialog Understanding (2023)

Abstract: In multi-turn dialog understanding, semantic frames are constructed by detecting intents and slots within each user utterance. However, recent works lack the capability of modeling multi-turn dynamics within a dialog in natural language understanding (NLU), instead leaving them for updating dialog states only. Moreover, humans usually associate relevant background knowledge with the current dialog contexts to better illustrate slot semantics revealed from word connotations, where previous works have explored such possibility mostly in knowledge-grounded response generation. In this paper, we propose to amend the research gap by equipping a BERT-based NLU framework with knowledge and context awareness. We first encode dialog contexts with a unidirectional context-aware transformer encoder and select relevant inter-word knowledge with the current word and previous history based on a knowledge attention mechanism. Experimental results in two complicated multi-turn dialog datasets have demonstrated significant improvements of our proposed framework. Attention visualization also demonstrates how our modules leverage knowledge across the utterance.

## introduction in conventional task oriented dialog systems, natural language understanding (nlu) modules aim to transform utterances into meaningful semantic representations for dialog management @xcite . it mainly detects associated dialog acts or intents and extracts key slot information as so-called 'semantic frames' @xcite , shown in speaker utterance 1. user is there something that's maybe a good intelligent comedy? act & slots: request (genre: comedy) knowledge: (intelligent; related to; well_informed) (comedy; related to; comic) (comedy; is a; drama) 2. system whiskey tango foxtrot is the only adult comedy i see playing in your area. would you like to try that? act & slots: inform (movie: whiskey tango foxtrot) inform (genre: adult comedy) inform (distance limits: in your area) confirm_question knowledge: publicly available models like bert or xlnet provide universal contextualized representations that could be adapted for learning task-oriented contexts. however, it may not give full play to its value when tagging some rare words like foxtrot together with tango as movie in however, there are mainly three challenges lying in the way of such integration: (1) heterogeneous information fusion: the vector space of kg entities is inconsistent with that of the pretrained models. (2) knowledge noise: overwhelming knowledge for models may adversely cause redundant noises for more ambiguity. many works in knowledge grounded dialog generation has applied term-level denoising @xcite or filtering techniques @xcite to refine the adopted knowledge for better semantic considerations. (3) inter-token knowledge sharing: @xcite predicts a slot for a given word along with its own associated knowledge. however, real sentences may contain phrases where knowledge between words should be shared to probably enrich the entire utterance semantics. to overcome these challenges and ground knowledge in contextual nlu, which is less explored in the research community, we propose a context and knowledge awareness nlu framework (cka-nlu) to effectively incorporate relevant knowledge and dialog history in dialog understanding. the key ingredients lie in how we can efficiently integrate relevant knowledge and previous history for understanding. we first introduce a context attention module to retrieve context-aware representations. different from previous works of determining a given word's slot based on its own knowledge, our objectives require models to aggregate both previous dialog contexts and all intrasentence knowledge facts together to formulate context-attended knowledge vectors in the same space. such vectors are a weighted combination of all knowledge facts based on the aggregated information until the current turn. we use attention masks and filtering to remove adversarial effects from redundant knowledge noises. finally we adopt these context-attended vectors for nlu tasks with rnn decoders. experiment results have shown superior performances of our methods that beat all competitive baselines. our contributions are as follows: 1. we propose a novel cka-nlu framework that incorporates inter-word knowledge with intersentence contexts to fill the void of relevant knowledge exploration for important nlu tasks. 2. we demonstrate the benefits of adopting knowledge for token-level slot filling and dialog history for sentence-level intent detection. 3. experimental and attention visualization results show that our model achieves superior performances over several competitive baselines and demonstrates how our model adopts the knowledge. ## problem formulation for each utterance x @xmath0, w n 2 , . . . , w n t } in a task-oriented dialog x with n utterances, given the domain ontology of a dialog act set a and a slot set s, we aim to find one or more acts {a n i } foot_0 and a sequence of slot tags {s n 1 , s n 2 , . . . , s n t } to construct a semantic frame. namely, we hope to maximize the joint log likelihood of a and s in eq 1 given a parametrized model θ, its context c @xmath1, . . . , x n-1 } and associated knowledge k @xmath2, x n ) for the current utterance x n . we deem k g as an external large knowledge base with knowledge represented as triples (head h, relation r, tail t) and ϕ(•) helps to extract related knowledge pairs for x n ( §3.2.1). it will be critical to match correct knowledge based on current dialog history and the utterance for better dialog understanding. ## conclusion in this paper, we propose a novel bert-based knowledge-augmented network to effectively incorporate dialog history and external knowledge in the joint nlu tasks. compared to recent works which consider only intra-word knowledge, we instead raise the knowledge awareness by selecting all relevant knowledge triples in an utterance with the current dialog contexts. we found that our framework is verified to be effective in two complex multi-turn dialog datasets where contexts and knowledge are crucial in dialog act detection and slot filling respectively. the visualization shows that our models adopt some key knowledge in particular words and learn to grasp useful information for better interpretability. these context-attended knowledge vectors could be easily applied to downstream dialog state tracking or management tasks.