Semantic Representation for Dialogue Modeling (2021)

Abstract: Although neural models have achieved competitive results in dialogue systems, they have shown limited ability in representing core semantics, such as ignoring important entities. To this end, we exploit Abstract Meaning Representation (AMR) to help dialogue modeling. Compared with the textual input, AMR explicitly provides core semantic knowledge and reduces data sparsity. We develop an algorithm to construct dialogue-level AMR graphs from sentence-level AMRs and explore two ways to incorporate AMRs into dialogue systems. Experimental results on both dialogue understanding and response generation tasks show the superiority of our model. To our knowledge, we are the first to leverage a formal semantic representation into neural dialogue modeling.

## introduction dialogue systems have received increasing research attention @xcite @xcite , with much recent work focusing on social chats @xcite and task-oriented dialogues @xcite . there are two salient subtasks in dialogue modeling, namely dialogue understanding @xcite @xcite and response generation @xcite . the former refers to understanding of semantic and discourse details in a dialogue history, and the latter concerns making a fluent, novel and coherent utterance. the current state-of-the-art methods employ neural networks and end-to-end training @xcite for dialogue modeling. for instance, sequence-to-sequence models have been used to encode a dialogue history, before directly synthesizing the next utterance @xcite @xcite dialogue history: … speaker-1 : recently, i've been obsessed with horror films. speaker-2 : oh, how can you be infatuated with horror films? they're so scary . speaker-1 : yeah, you are right i used to not watch horror films, but after seeing silence of the lamb with mike last month, i fell in love with them. speaker-2 : it's amazing. but if i were you, i wouldn't have the courage to watch the first one. speaker-1 : but it's really exciting . ## constructing dialogue amrs figure 2 illustrates our method for constructing a dialogue-level amr graph from multiple utterancelevel amrs. given a dialogue consisting multiple utterances, we adopt a pretrained amr parser @xcite to obtain an amr graph for each utterance. for utterances containing multiple sentences, we parse them into multiple amr graphs, and mark them belonging to the same utterance. we construct each dialogue amr graph by making connections between utterance amrs. in particular, we take three strategies according to speaker, identical concept and co-reference information. speaker we add a dummy node and connect it to all root nodes of utterance amrs. we add speaker tags (e.g., speaker1 and speaker2) to the edges to distinguish different speakers. the dummy node ensures that all utterance amrs are connected so that information can be exchanged during graph encoding. besides, it serves as the global root node to represent the whole dialogue. identical concept there can be identical mentions in different utterances (e.g. "possible" in the first and the forth utterances in figure 2 ), resulting in repeated concept nodes in utterance amrs. we connect nodes corresponding to the same nonpronoun concepts by edges labeled with same foot_2 . this type of connection can further enhance crosssentence information exchange. raw utterance texts (b) utterance amr graphs (c) dialogue amr graph amr parsing graph merge s1 s2 could i have my bill, please? s2 certainly, sir. ## baseline system we adopt a standard transformer @xcite for dialogue history encoding. typically, a transformer encoder consists of l layers, taking a sequence of tokens (i.e., dialogue history) @xmath0, w 2 , ..., w n }, where w i is the i-th token and n is the sequence length, as input and produces vectorized word representations {h l 1 , h l 2 , ..., h l n } iteratively, @xmath1, ..., l]. overall, a transformer encoder can be written as: and emb denotes a function that maps a sequence of tokens into the corresponding embeddings. each transformer layer consists of two sub-layers: a self-attention sub-layer and a position-wise feed forward network. the former calculates a set of attention scores: the position-wise feed-forward (ffn) layer consists of two linear transformations: ## proposed model our model takes a dialogue history s and the corresponding dialogue amr as input. formally, 5 similar to the encoder, there is also multi-head attention, a position-wise feed-forward layer and residual connections, which we omit in the equations. an amr is a directed acyclic graph @xmath2, e , where v denotes a set of nodes (i.e. amr concepts) and e (i.e. amr relations) denotes a set of labeled edges. an edge can be further represented by a triple n i , r ij , n j , meaning that the edge is from node n i to n j with label r ij . we consider two main ways of making use of dialogue-level amrs. the first method (figure 3(a) ) uses amr semantic relations to enrich a textual representation of the dialogue history. we project amr nodes onto the corresponding tokens, extending transformer by encoding semantic relations between words. for the second approach, we separately encode an amr and its sentence, and use either feature fusion (figure 3(b) ) or dual attention (figure 3(c )) to incorporate their embeddings. ## dialogue understanding experiments we evaluate our model on dialogre @xcite , which contains totally 1,788 dialogues, 10,168 relational triples and 36 relation types in total. on average, a dialogue in dialogre contains 4.5 relational triples and 12.9 turns. we report experimental results on both original (v1) and updated (v2) english version. 7 ## response generation experiments we conduct experiments on the dailydialog benchmark @xcite , which contains 13,119 daily multi-turn conversations. on average, the number of turns for each dialogue is 7.9, and each utterance has 14.6 tokens. ## analysis this section contains analysis concerning the effects of graph features, dialogue length and model robustness. we use dual model for experiments since it gives slightly better results than hier. ## related work semantic parsing for dialogue some previous work builds domain-specified semantic schema for task-oriented dialogues. for example, in the pegasus @xcite system, a sentence is first transformed into a semantic frame and then used for travel planing. @xcite use semantic features to help a dialogue system perform certain database operations. @xcite represent task-oriented conversations as semantic trees where intents and slots are tree nodes. they solve intent classification and slot-filling task via semantic parsing. @xcite design a rooted semantic graph that integrates domains, verbs, operators and slots in order to perform dialogue state tracking. all these structures are designed for specified task only. in contrast, we investigate a general semantic representation for the modeling of everyday conversations. ## conclusion we investigated the feasibility of using amrs for dialogue modeling, describing an algorithm to construct dialogue-level amrs automatically and exploiting two ways to incorporate amrs into neural dialogue systems. experiments on two benchmarks show advantages of using amr semantic representations model on both dialogue understanding and dialogue response generation.