On Incorporating Structural Information to improve Dialogue Response Generation (2020)

Abstract: We consider the task of generating dialogue responses from background knowledge comprising of domain specific resources. Specifically, given a conversation around a movie, the task is to generate the next response based on background knowledge about the movie such as the plot, review, Reddit comments etc. This requires capturing structural, sequential and semantic information from the conversation context and the background resources. We propose a new architecture that uses the ability of BERT to capture deep contextualized representations in conjunction with explicit structure and sequence information. More specifically, we use (i) Graph Convolutional Networks (GCNs) to capture structural information, (ii) LSTMs to capture sequential information and (iii) BERT for the deep contextualized representations that capture semantic information. We analyze the proposed architecture extensively. To this end, we propose a plug-and-play Semantics-Sequences-Structures (SSS) framework which allows us to effectively combine such linguistic information. Through a series of experiments we make some interesting observations. First, we observe that the popular adaptation of the GCN model for NLP tasks where structural information (GCNs) was added on top of sequential information (LSTMs) performs poorly on our task. This leads us to explore interesting ways of combining semantic and structural information to improve the performance. Second, we observe that while BERT already outperforms other deep contextualized representations such as ELMo, it still benefits from the additional structural information explicitly added using GCNs. This is a bit surprising given the recent claims that BERT already captures structural information. Lastly, the proposed SSS framework gives an improvement of 7.95% on BLUE score over the baseline.

## introduction neural conversation systems that treat dialogue response generation as a sequence generation task @xcite often produce generic and incoherent responses @xcite . the primary reason for this is that, unlike humans, such systems do not have any access to background knowledge about the topic of conversation. for example, while chatting about movies, we use our background knowledge about the movie in the form of plot details, reviews, and comments that we might have read. to enrich such neural conversation systems, some recent works @xcite @xcite incorporate external knowledge in the form of documents which are relevant to the current conversation. for example, @xcite released a dataset containing conversations about movies where every alternate utterance is extracted from a background document about the movie. this background document contains plot details, reviews, and reddit comments about the movie. the focus thus shifts from sequence generation to identifying relevant snippets from the background document and modifying them suitably to form an appropriate response given the current conversational context. intuitively, any model for this task should exploit semantic, structural and sequential information from the conversation context and the background document. for illustration, consider the chat shown in figure 1 from the holl-e movie conversations dataset @xcite . in this example, speaker 1 nudges speaker 2 to talk about how james's wife was irritated because of his career. the right response to this conversation comes from the line beginning at "his wife mae . . . ". however, to generate this response, it is essential to understand that (i) his refers to james from the previous sentence; (ii) quit boxing is a contigu-source doc: ... at this point james braddock (russel crowe) was a light heavyweight boxer, who was forced to retired from the ring after breaking his hand in his last fight. his wife mae had prayed for years that he would quit boxing, before becoming permanently injured. ... conversation: speaker 1(n): yes very true, this is a real rags to riches story. russell crowe was excellent as usual. speaker 2(r): russell crowe owns the character of james bradock, the unlikely hero who makes the most of his second chance. he's a good fighter turned hack. speaker 1(n): totally! oh by the way do you remember his wife ... how she wished he would stop speaker 2(p): his wife mae had prayed for years that he would quit boxing, before becoming permanently injured. the text in bold in the first block is the background document which is used to generate the last utterance in this conversation. n, p, and r correspond to the type of background knowledge used: none, plot, and review as per the dataset definitions. for simplicity, we show only a few of the edges for the background knowledge at the bottom. the edge in blue corresponds to the co-reference edge, the edges in green are dependency edges and the edge in red is the entity edge. to capture such multi-faceted information from the document and the conversation context we propose a new architecture that combines bert with explicit sequence and structure information. we start with the deep contextualized word representations learnt by bert which capture distributional semantics. we then enrich these representations with sequential information by allowing the words to interact with each other by passing them through a bidirectional lstm as is the standard practice in many nlp tasks. lastly, we add explicit structural information in the form of dependency graphs, co-reference graphs, and entity co-occurrence graphs. to allow interactions between words related through such structures, we use gcns which essentially aggregate information from the neighborhood of a word in the graph. of course, combining bert with lstms in itself is not new and has been tried in the original work @xcite for the task of named entity recognition. similarly, @xcite combine lstms with gcns for the task of machine translation. to the best of our knowledge, this is the first work that combines bert with explicit structural information. we investigate several interesting questions in the context of dialogue response generation. for example, 1. are bert-based models best suited for this task? to systematically investigate such questions we propose a simple plug-and-play semantics-sequences-structures (sss) framework which allows us to combine different semantic representations (glove @xcite , bert @xcite , elmo @xcite ) with different structural priors (dependency graphs, co-reference graphs, etc.). it also allows us to use different ways of combining structural and sequential information, e.g., lstm first followed by gcn or vice versa, or both in parallel. using this framework we perform a series of experiments on the holl-e dataset and make some interesting observations. first, we observe that the conventional adaptation of gcns for nlp tasks, where contextualized embeddings obtained through lstms are fed as input to a gcn, exhibits poor performance. to overcome this, we propose some simple alternatives and show that they lead to better performance. second, we observe that while bert performs better than glove and elmo, it still benefits from explicit structural information captured by gcns. we find this interesting because some recent works @xcite @xcite suggest that bert captures syntactic information, but our results suggest that there is still more information to be captured by adding explicit structural priors. third, we observe that certain graph structures are more useful for this task than others. lastly, our best model which uses a specific combination of semantic, sequential, and structural information improves over the baseline by 7.95% on the bleu score. ## related work there is an active interest in using external knowledge to improve the informativeness of responses for goal-oriented as well as chit-chat conversations @xcite @xcite . even the teams participating in the annual alexa prize competition @xcite have benefited by using several knowledge resources. this external knowledge can be in the form of knowledge graphs or unstructured texts such as documents. many nlp systems including conversation systems use rnns as their basic building block which typically captures n-gram or sequential information. adding structural information through treebased structures @xcite or graph-based structures @xcite on top of this has shown improved results on several tasks. for example, gcns have been used to improve neural machine translation @xcite by exploiting the semantic structure of the source sentence. similarly, gcns have been used with dependency graphs to incorporate structural information for semantic role labelling @xcite , neural machine translation @xcite and entity relation information in question answering @xcite and temporal information for neural dating of documents @xcite . there have been advances in learning deep contextualized word representations @xcite with a hope that such representations will implicitly learn structural and relational information with the interaction between words at multiple layers @xcite . these recent developments have led to many interesting questions about the best way of exploiting rich information from sentences and documents. we try to answer some of these questions in the context of background aware dialogue response generation. ## background in this section, we provide a background on how gcns have been leveraged in nlp to incorporate different linguistic structures. the syntactic-gcn proposed in @xcite is a gcn @xcite variant which can model multiple edge types and edge directions. it can also dynamically determine the importance of an edge. they only work with one graph structure at a time with the most popular structure being the dependency graph of a sentence. for convenience, we refer to syntactic-gcns as gcns from here on. let g denote a graph defined on a text sequence (sentence, passage or document) with nodes as words and edges representing a directed relation between words. let n denote a dictionary of list of neighbors with n (v) referring to the neighbors of a specific node v, including itself (self-loop). let dir(u, v) ∈ {in, out, self } denote the direction of the edge, (u, v). let l be the set of different edge types and let l(u, v) ∈ l denote the label of the edge, (u, v). the (k + 1)-hop representation of a node v is computed as in summary, a gcn computes new representation of a node u by aggregating information from it's neighborhood n (v). when @xmath0, the aggregation happens only from immediate neighbors, i.e., 1 hop neighbors. as the value of k increases the aggregation implicitly happens from a larger neighborhood. ## proposed model given a document d and a conversational context q the task is to generate the response @xmath1, y 2 , ...., y m . this can be modeled as the problem of finding a y that maximizes the probability p (y|d, q) which can be further decomposed as as has become a standard practice in most nlg tasks, we model the above probability using a neural network comprising of an encoder, a decoder, an attention mechanism, and a copy mechanism. the copy mechanism essentially helps to directly copy words from the document d instead of predicting them from the vocabulary. our main contribution is in improving the document encoder where we use a plug-and-play framework to combine semantic, structural, and sequential information from different sources. this enriched document encoder could be coupled with any existing model. in this work, we couple it with the popular get to the point (gttp) model @xcite as used by the authors of the holl-e dataset. in other words, we use the same attention mechanism, decoder, and copy mechanism as gttp but augment it with an enriched document encoder. below, we first describe the document encoder and then very briefly describe the other components of the model. we also refer the reader to the supplementary material for more details. ## experimental setup in this section, we briefly describe the dataset and task setup followed by the pre-processing steps we carried to obtain different linguistic graph structures on this dataset. we then describe the different baseline models. our code is available at: https://github.com/nikitacs16/horovod_ gcn_pointer_generator ## results and discussion in