Dialogue over Context and Structured Knowledge using a Neural Network Model with External Memories (2020)

Abstract: The Differentiable Neural Computer (DNC), a neural network model with an addressable external memory, can solve algorithmic and question answering tasks. There are various improved versions of DNC, such as rsDNC and DNC-DMS. However, how to integrate structured knowledge into these DNC models remains a challenging research question. We incorporate an architecture for knowledge into such DNC models, i.e. DNC, rsDNC and DNC-DMS, to improve the ability to generate correct responses using both contextual information and structured knowledge. Our improved rsDNC model improves the mean accuracy by approximately 20% to the original rsDNC on tasks requiring knowledge in the dialog bAbI tasks. In addition, our improved rsDNC and DNC-DMS models also yield better performance than their original models in the Movie Dialog dataset.

## introduction recently, deep neural networks have made significant progress in complex pattern matching of various tasks such as computer vision and natural language processing. however, these models are limited in their ability to represent data structures such as graphs and trees, to use variables, and to handle representations over long sequences. recurrent neural networks (rnns) can capture the long-range dependencies of sequential data and are also known as turing-complete @xcite and therefore are capable of simulating arbitrary procedures, if properly wired. however, rnns struggled with the vanishing gradients problem @xcite . the long short-term memory (lstm) architecture addressed this problem by taking gating mechanisms into rnn architectures and calculating the gradients by element-wise multiplication with the gate value at every time-step @xcite . lstms became quite successful and helped to outperform traditional models because of the sequence to sequence model @xcite and attention mechanisms @xcite . yet lstm based models have not reached a real solution to the problems mentioned as the limitations of deep neural networks. on the other hand, in the von neumann architecture, programs can be run by three fundamental mechanisms: a processing unit that performs simple arithmetic operations and logic ones, a control unit that takes control flow instructions such as sequential execution, conditional branch and loop, and a memory unit that data and instructions are written to and read from during computation @xcite . this architecture can represent complex data structures and learn to perform algorithmic tasks. it separates computation by a processor (i.e. a processing unit and a control one) and memory, whereas neural networks mix computation and memory into the network weights. to improve the performance of standard neural networks, @xcite proposed a neural network model with an addressable external memory called neural turing machine (ntm). the whole architecture of ntm is differentiable, therefore can be trained end-to-end including how to access to the memory. further, @xcite improved the memory access mechanism and proposed differentiable neural computer (dnc). it solved algorithmic tasks over structured data such as traversal and shortest-path tasks of a route map and an inference task of a family tree. in the experiment on question answering with premises, input sequences were written to the memory and necessary information to infer the answer was read from the memory, hence representing variables. dnc was also able to learn long sequences by the dynamic memory access mechanism. there are various improved versions of dnc, such as rsdnc @xcite and dnc-dms @xcite . however, how to integrate structured knowledge into these dnc models remains a challenging research question. this paper investigates how to incorporate structured knowledge into dnc models. we extend single-memory unit dnc models to a multiple-memory architecture that leverages both contextual and structured knowledge information. we add an extra memory unit that encodes knowledge from knowledge bases. in contrast with rnns, the memory-augmentation of dnc allows an explicit storage and manipulation of complex data structures over a long time-scale @xcite . our main contributions are as follows: the whole paper is organized as follows. section 2 briefly introduces the dnc, rsdnc and dnc-dms models. we describe our proposed model in section 3 and our experiments and detailed analysis in section 4. section 5 introduces related works. finally, we conclude this paper and explore the future work in section 6. ## differentiable neural computer the differentiable neural computer (dnc) is a neural network coupled to an external memory matrix @xmath0, as shown in figure 1 . it uses attention mechanisms to define weightings over n locations of the memory matrix m that represent which locations should be read or written mainly. for the read operation, the read vector r is computed as a weighted sum over the memory locations by applying a read weighting w r over memory m : @xmath1, •]w r [i] for the write operation, the memory m is modified by using a write weighting w w to first erase with an erase vector e, then add a write vector v: the weightings are defined by the following three attention mechanisms: the whole system is differentiable and can be learned with gradient descent. recently, variations of the dnc model have been proposed, such as the robust and scalable dnc (rsdnc) @xcite and the dnc-dms @xcite . robust and scalable dnc : focusing on qa tasks, @xcite extended the dnc to be more robust and scalable (rsdnc), with the following four improvements: (1) using only the content-based memory unit to reduce memory consumption and training time, (2) applying layer normalization to lower the variance in performance between different runs, (3) using bypass dropout to make the memory unit's effect stronger, and (4) introducing a bidirectional architecture to encode input sequences in a more informative way. schmidhuber (2019) tackled three problems of vanilla dnc and proposed an improved model called dnc-dms. first, the lack of key-value separation makes the contentbased address distribution flat and prevents the model from accessing specific parts of a memory. by masking improper parts of both lookup key and memory content, the key-value separation can be controlled dynamically. second, memory de-allocation mechanisms do not affect memory content which is crucial to content-based addressing and result in memory aliasing. thus, dnc-dms proposed to erase memory content completely. lastly, chaining multiple reads with the temporal linkage matrix exponentially blurs the address distribution. exponentiation and renormalization of the distribution reduced the effect of exponential blurring and improved sharpness of the link distribution. ## proposed method we expand three models, dnc, rsdnc, and dnc-dms, by adding an extra memory architecture to store structured knowledge. therefore, our proposed model consists of a control unit and two memory units. one memory unit stores contextual information in the dialogue and we call it "context memory". the other memory unit stores knowledge information and we call it "knowledge memory". the differences from the original dnc models are to introduce the knowledge memory and add the operation for it. figure 2 shows the overview of our proposed model based on dnc. the procedures at every time-step t are described as follows: the write operation to the knowledge memory is not performed. the read vectors r c t and r k t are appended to the controller input at the next time-step. the read and write operations to two memories are performed by repeating the above procedures. to build the knowledge memory unit used in our models, we first run the original dnc model on a knowledge base (kb). we then use the pretrained memory unit as the knowledge memory unit in our proposed models. the process to build this knowledge memory is described in the next section. knowledge memory building we built a knowledge memory unit with a knowledge base (kb). facts in the kb have a resource description framework (rdf) foot_0 triple structure "(subject, relation, object)". for example, information such as "washington, d.c. is the capital of the u.s." is expressed as (the u.s., capital, washington, d.c.). we applied the original dnc model, which has a single memory, to learn kb facts by giving the model all three components of a triple and any two of a triple and then making the model learn to return the other of a triple. for instance, when inputs for the model are '' the u.s.'', '' capital'', '' washington, d.c.'', '' the u.s.'', and ''capital'', the output is '' washington, d.c.''. the model was trained using all triples of the kb and produced a memory unit which stores the whole kb. we used this pre-trained memory unit as the knowledge memory unit in our proposed models. we trained the dnc model using memory dimensions of 512 × 128 because the results of our proposed models were better than when we also trained using memory dimensions of 256 × 64 which were the same as context memory dimensions in our proposed method. whereas the context memory just stores each dialogue content in the dataset, the knowledge memory stores the whole content of the kb and thus it is reasonable that the knowledge memory needs to be larger than the context memory. we evaluated the model with the accuracy and used transe @xcite ) for kb's word embeddings. ## experiments we evaluated our approach on two dialogue datasets, the (6) dialog babi tasks @xcite and the movie dialog dataset @xcite . both datasets require context compre-hension and knowledge background, and provide dialogue data on a specific domain and rdf triple data to answer questions in the dialogue. ## conclusion we added knowledge memory architecture to three dnc models, vanilla dnc, rsdnc, and dnc-dms, and experimentally analyzed the effect of our addition on dialogue tasks that require background knowledge. our proposed models, dnc+km, rs-dnc+km, and dnc-md+km outperformed their original models on full dialog tasks in the (6) dialog babi tasks dataset. in particular, each model obtained an improvement of approximately 14%, 20%, and 7%, respectively on tasks which require kb facts. in the movie dialog dataset, our rs-dnc+km and dnc-dms+km performed better than their original models. in future work we will investigate the behavior of the knowledge memory in detail and study how to build and use the knowledge memory more effectively in the whole architecture. we will also conduct experiments with models that are different from dnc models such as key-value memory networks @xcite to compare with our models.