A NALYSIS OF M EMORY O RGANIZATION FOR D YNAMIC N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
An increasing number of neural memory networks have been developed, leading to the need for a systematic approach to analyze and compare their underlying memory capabilities
Thus , in this paper, we propose a taxonomy for four popular dynamic models: vanilla recurrent neural network, long short-term memory, neural stack and neural RAM and their variants.
Based on this taxonomy, we create a framework to analyze memory organization and then compare these network architectures. This analysis elucidates how different mapping functions capture the information in the past of the input and helps to open the dynamic neural network black box from the perspective of memory usage.
Four representative tasks that would ﬁt optimally the characteristics of each memory network are carefully selected to show each network’s expressive power. We also discuss how to use this taxonomy to help users select the most parsimonious type of memory network for a speciﬁc task.
Two natural language processing applications are used to evaluate the methodology in a realistic setting
I NTRODUCTION
Memory is crucial for both cognitive science and machine learning in applications that require the use of past information such as sequence learning problems.
It is fair to say that recurrent neural networks are the most popular memory networks nowadays. Elman and Jordan proposed the ﬁrst classic version of the recurrent neural network (RNN) Elman (1990)Jordan (1986).
which is often referred to as vanilla RNNs (vRNNs) currently.
However, since the memory of vRNN is internal memory encoded in hidden states, vRNN is difﬁcult to implement in practice: if the temporal dependency relationship of the time series is complex and long, the number of hidden units has to be tremendously increased to fully capture useful past information. The long range dependence problem is one common example of this difﬁcultyTrinh et al. (2018)Belletti et al. (2018)Bengio et al.
Many dynamic neural networks with new memory structures have recently emerged as improvements of the vRNN architecture.
(The terms recurrent, dynamic, and memory neural network are used interchangeably in this paper.)
Some of these networks adopt internal memory; some adopt external memory; some adopt logic gates; while others adopt an attention mechanism.
As expected, all these methods have advantages for some speciﬁc tasks, but selecting or designing an optimal memory model for a new task is very difﬁcult unless we have a clear understanding of how their memory helps solve problem at hand.
This is not an easy issue because ‘memory’ is a very abstract concept and the speciﬁc memory requirements for a speciﬁc task are implicit, which means that quantitatively conceptualizing and analyzing memory is a very hard problem.
Cognitive scientists have deﬁned many different types of memory, which shows the richness of the concept. In engineering, there are only a few engineering quantiﬁers of memory such as memory depth and memory resolution, but they are not enough for the ever-growing applications of machine learning.
Thus, the main goal of this paper is to investigate the characteristics of memory implemented by each architecture and what information these networks can extract from the input time series.
We illustrate the role and characteristics of mapping functions and describe how memory is implemented of each memory network. We also classify popular memory networks into four types and ﬁnd the essential relationships amongst these classes.
The results are summarized in a rigorous taxonomy, which serves as a simple way to quantify the capabilities of extracting and using past information of each class. A secondary goal is to employ the knowledge gained from the different characteristics of these memory structures to help users select the most parsimonious type of memory network given the problem at hand.
We achieve this goal by connecting models’ relative expressive power to the memory requirements of different tasks and use four synthetic tasks to exemplify each task type.
Moreover, two natural language processing applications are used to evaluate the methodology in a realistic setting.
R ELATED W ORK
Among the abundant recurrent network literature, minimal attention has been given to understanding and analysis of their memory characteristics.
Omlin & Giles (1996) discussed how vRNNs behaves like deterministic ﬁnite-state automata.
Gers & Schmidhuber (2001)Rodriguez (2001)Schmidhuber et al.
compared the performance of long short-term memory (LSTM) Hochreiter & Schmidhuber (1997) and vRNN on some context-free/sensitive language.
Collins et al.
(2016) studied the capacity of recurrent nets and how difﬁcult they are to train.
Karpathy et al.
(2015) visualized longterm interactions and representations learned by recurrent networks.
Greff et al.
(2017) empirically studied the importance of various computational components of LSTM.
Jozefowicz et al.
(2015) evaluated a variety of recurrent neural network architectures and attempted to ﬁnd the best one.
Chung et al.
compared gated recurrent unit (GRU) networks Cho et al. (2014) to LSTM.
Yogatama et al.
(2018) tested and compared performances of sequential, random access, and stack memory architectures on language modeling dataset.
These works assessed network performance based on the output error. By contrast, our work focuses on how these networks encode information to solve a problem.  A M EMORY N ETWORK T AXONOMY
In this section, we organize these popular dynamical networks according to their underlying memory structures.
We classify them into four classes as shown in Fig.1, i.e., vRNN
⊆
LSTM ⊆ neural stack ⊆ neural RAM. Some classes are named after a typical model. For example, the vRNN class also includes IRNN Le et al.
GRU Cho
Gers & Schmidhuber (2000). The neural stack class includes the architecture in Sun (1993); Sun et al. Joulin & Mikolov (2015); Grefenstette et al. (2015), and
the neural RAM class includes NTM Graves et al.
DNC Graves et al.
(2016), enhanced LSTM Graves (2013) and Weston et al.
vRNN’s internal memory has a lot of limitations since all the useful information is compressed in the hidden states that are continuously changing with the input samples: the past contents and their time information are mixed together, i.e., a certain event can not be extracted without interference, the hidden state has to be updated at every time steps, the long range dependency problem makes it hard to train, etc.
These problems make vRNN unpractical. Compared to vRNN, the type of memory of LSTM, neural stack and neural RAM is external. LSTM introduced an external memory slot and the corresponding gate mechanism to address the long range dependency problem.
With this external memory slot, a useful event in the past can be extracted without interference.
However, if there are multiple useful events, they have to be compressed as a compounded event and accessed as a whole.
The speciﬁc content of each event and their temporal order can not be captured.
Neural stack and neural RAM addressed this problem by introducing multiple-slot external memory, where multiple events can be stored and their temporal order information is also retained.
The difference between neural RAM and neural stack is the contents of neural RAM can be accessed randomly.
The different memory organizations give these networks different expressive power.
Actually one class of network architectures can be seen as a special case of another class as shown in Fig.1.
The memory types are summarized in Table.1. In next section, we will analyze their underlying structures in details and showing their essential relationships. [Table] LSTM memory of a single compounded event at a
[Table]
Neural memory of multiple events, information of each event should be external memory stack used sequentially, only one event is accessible at each time step Neural memory of multiple events, all are accessible at each external memory RAM time step, no restriction on how many times they are used  M EMORY S TRUCTURE A NALYSIS In this section, we analyze the memory structures of four popular RNNs: vRNN, LSTM, neural stack and neural RAM.
Attention is paid to how their underlying memory organizations lead to different features and expressive power.
4. V RNN
[Figure]
The vRNN network Jordan (1986) is composed of three layers: input, hidden recurrent and output.
In addition to the feedforward connections, a feedback connection from the hidden layer to itself exists.
The architecture is shown in Fig.2(a).
The dynamics of the hidden layer can be written as
[Equation]
where x t , h t and o t are the input, hidden state and output vector at time t .
We use w and b to represent the weight and bias, and f ( x ) is the nonlinear activation function.
A vRNN induces memory by encoding the past information in its hidden state units h t , which are continuously changing and forced to be used every time; thus, the memory of the vRNN is called state memory or internal memory.
Fig.2(b) shows the state transition diagram of a vRNN, where s 0 , s 1 , ..., s 4 represent the state at times t 0 , t 1 , ..., t 4 , respectively.
The arrows show the dependency of the variables.
Here, state s 1 is decided by only s 0 , s 2 is decided by only s 1 and so on.
(All the memory visualization ﬁgures in this paper ignore the current input.)
As the number of hidden units is limited in practice, a compromise between memory depth and memory resolution always exists in the vRNN De Vries & Principe (1992).
For long memory depth sequences, a vRNN requires a very large number of hidden units to achieve acceptable accuracy.
If the sequences are composed of symbols or discrete numbers, the limitation of vRNN is even easier to see: the process can be viewed from a Markov transition model perspective.
Speciﬁcally, the vRNN attempts to learn a ﬁrst-order Markov transition model (with transition probability 1), where the current state is decided by only current input and the state in the previous step.
Thus, since the state space is not very large (the number of states is less than the size of input symbols’ alphabet) for ﬁrst-order Markov sequences, the vRNN always performs well.
However, for higher-order Markov sequences or sequences that do not satisfy the Markov property, the vRNN still attempts to build a ﬁrst-order Markov state model, which results in a very large state space (several old states must be combined into a new state).
The compromise between memory depth and memory resolution (which are related to the number and temporal resolution of the states) would make vRNN not suitable for these kinds of sequences. Hence, vRNN is only suitable for lower-order Markov chains.
4. LSTM
LSTM was proposed to address the vanishing gradient problem of vRNN.
It was the ﬁrst memory network that had to address the problem of how to store and retrieve information from an external source to improve performance (through gating networks).
In this section, we analyze how LSTM provides greater ﬂexibility from the perspective of memory usage.
[Figure] In contrast to vRNN, in the classic LSTM shown in Fig.3(a), the feedback connection of the hidden layer must pass through external memory m t ,
[Equation]
where h t (or c t ) is the network state.
The external memory m t is a combination of m t − 1 and the current state c t .
If g i,t = 0 and g f,t = 1 for several successive time steps, the content saved in the external memory m t would constitute the long-term memory of the system.
The external memory m t adds greater ﬂexibility to the state transition diagram.
As shown in Fig.3(b), the current state s t (represented by hidden state h t ) depends on either the previous state s t − 1 or the external memory m t − 1 (if forget gate g f,t = 0 and input gate g i,t = 1 , s t depends on s t − 1 ; if g f,t = 1 and g i,t = 0 , s t depends on m t − 1 ; if 0 < g f,t < 1 and 0 < g i,t < 1 , s t depends on both m t − 1 and s t − 1 .
The detailed calculations of these gates are provided in Appendix A.1).
For example, s 1 depends on the previous state s 0 , as illustrated by the blue arrows; s 7 and s 8 depend on the long-term memory M 00 , as illustrated by the yellow arrows; and s 9 depends on both the previous state s 8 and the long-term memory M 00 .
The introduced external memory circumvents the compromise between the memory depth and memory resolution that is always present in the state memory in vRNN.
For instance, for a 10th-order binary Markov sequence whose state dependence relationship is s t = f
( s t − 1 , s t − 10 ) , a vRNN has to learn a state space with 2 10 states (10 states are combined into a new state); however, LSTM needs to learn a state model with only 2 states and an external memory storing the state information from 10 steps in the past.
By constructing this short path between long-term memory and the current state, LSTM performs much better than vRNN for sequences that skip intermediate values of time dependencies. This long-term memory can also be seen as the event extracted from the input time series. The relationship of LSTM and vRNN is shown in Theorem1.
Theorem 1. LSTM degrades to a vRNN if i) all three gates are constants, i.e., g o = 0 , g i = 1 and g f = 0 ; ii) weight w hc and bias b c are set to constants, i.e., w hc = I and b c = 0 ; iii) the activation function t ( x ) is a linear activation function, i.e., t 1 ( x ) = x.
Proof.
See Appendix B.1.
Although LSTM is more effective than vRNN, some limitations still exist.
For example, if there is no skip in the time dependence, i.e., s t = f
( s t − 1 , s t − 2 , ..., s t − 10 ) , LSTM and vRNN have the same expressive power.
Therefore, the statement that "LSTM is always better than vRNN" is not correct.
Another drawback of LSTM is the transient storage of long-term memory.
In other words, if the long-term memory is updated, the old value is erased.
For example, in Fig.3(b), at time t 9 , when M is updated to M01, M00 is erased.
Thus, the future states no longer have access to memory M00.
Therefore, this architecture is useful when the previous states do not need to be reutilized after they are updated.
4. N EURAL S TACK
A neural stack is a neural network that uses a stack as its external memory.
The stack is controlled by either a feedforward network or a vRNN.
One property of a stack is that only the topmost content of the stack can be read or written.
Writing to the stack is implemented via three operations: push, adding an element to the top of the stack; pop, removing the topmost element of the stack; no-operation, keeping the stack unchanged.
A diagram of a neural stack network is shown in Fig.4(a) (here, we use the architecture in Joulin & Mikolov (2015)).
Elements in the stack are updated as follows:
[Equation]
[Figure]
At time t 2 , instead of replacing M 01 with a new state M 10 , a new belt M 1 is created to save M 10 .
In this way, both M 01 and Mare kept. Similarly, at time t 5 , M 30 is saved in another belt M 3 . At time t 5 , the content in the stack is M 01 , M 11 , and M 30 , where M 30 is the topmost element.
s t ( i ) is the content of the stack at time t in position i .
s t (0) is the topmost content at time t , c is the candidate content to be pushed onto the stack, and d pusht , d popt and d no − op t are the push, pop and no-operation signals.
All operations have to be implemented by continuous functions over a continuous domain to train the network with backpropagation through time.
The calculation details of the stack contents and the corresponding operators are given in Appendix A.2.
Since recurrence is introduced by stack memory, the dynamics of the model are
[Equation]
where r t is the vector read at time t ,
[Equation]
t (0) .
[Equation]
Although the neural stack architecture looks very different from that of vRNN and LSTM, some underlying similarities exist from a memory organization perspective.
Fig.4(b) shows the memory space for the neural stack.
In contrast to LSTM, a neural stack can store more than one useful event in its external memory bank.
( In LSTM, if there is only one useful event, it can be stored as it is, but if there are multiple useful events, they have to be compressed into one compounded event and can only be accessed as a whole.)
For example, at time t 0 , M 00 is saved in memory belt M 0 , and at time t 2 , M 10 is saved in belt M 1 .
The black arrow on the left of the memory content points to the top of the stack in each time step.
All these contents can be addressed as needed.
For example, M 10 is used again at time t 4 after popping out M 20 in belt M 2 .
All the useful information of the input is retained with this external memory.
In contrast to state memory, the previous content is not altered but is stored in its original or transformed form.
As the past content and operations are separate, we can efﬁciently select the useful content from this structured memory rather than using the mixture of all previous content.
LSTM can be seen as a special case of a neural stack as proved in Theorem 2.
In a neural stack, if all the contents in the stack below the topmost element are never addressed again, a single memory belt is sufﬁcient.
In this case, the neural stack degrades to LSTM , as shown in Fig.4(b) in the green dashed box.
The stack operators (push, pop, no-operation) in the neural stack have the same function as the input and forget gates in LSTM : deciding how to revise the memory contents.
The problem in LSTM is that the previous memory is erased after the update, which also occurs continuously with the vRNN state.
Hence, both learning models have difﬁculties in performing some simple memorization tasks, such as reversing a sequence.
However, the external memory bank in a neural stack can help to solve this problem via the online storage and extraction of more than one content.
Theorem 2.
The neural stack can degrade to LSTM if the pop signal is zero, i.e., d popt = 0 .
The d pusht in a neural stack works as the input gate in LSTM, and d no − op t in a neural stack works as the forget gate in LSTM.
Proof.
See Appendix B.2. Although the neural stack can return to previous memory, two constraints exist.
First, the neural stack cannot jump to any memory position: the previous memory must be addressed and updated sequentially.
For example, as shown in the second line of Fig.4(b), if we want to return to the memory in belt M 1 , we ﬁrst have to pass the memory in belt M 2 .
Second, the memory content is forgotten after it is popped out of the stack.
For example, at time t 4 , the memory in belt M 2 is popped out, so in the subsequent time steps, the content in belt M 2 can no longer be accessed and updated.
From the state transition analysis above, we can conclude that the stack neural network is the ideal choice for tasks where the previous memory must be addressed sequentially (ﬁrst in last out).
4. N EURAL RAM
Dynamic neural networks with external random access memory have recently been studied.
In these networks, all the contents in the memory bank can be randomly accessed.
The neural Turing machine Graves et al.
(2014) (NTM), whose network architecture is shown in Fig.5(a), is one example.
The challenge with this network is that all the memory addresses are discrete in nature.
To learn, read and write addresses by error backpropagation, the addresses have to be extended to continuous domain.
[Figure]
One solution to this difﬁculty is to read from and write to all the memory slots with different strengths.
These strengths represent the probability that each slot is be read from and written to.
Speciﬁcally, the reading vector at time step t is
[Equation]
where m is a memory bank with M memory slots, w rt ( i ) is the normalized reading weight for the i th slot at time t , which satisﬁes (cid:80)
[Equation]
In the writing process, each memory slot is updated as
[Equation]
where w rt ( i ) is the writing weight, and e t ( i ) is the erasing weight for memory slot i at time t .
The detailed calculations of these weights are presented in Appendix A.3.
The dynamics of the hidden layer are
[Equation]
Fig.5(b) shows the memory structure of neural RAM.
The RAM network can be seen as an improvement of the neural stack in the sense that all the contents in the memory bank can be read from and written to multiple times.
Moreover, no requirements exist for the order of storing, updating and accessing the memory elements.
For example, in Fig.5(b), at time t 0 , memory M 00 is stored in belt M 0 ; at time t 1 , the system control can directly jump to belt M 2 to store M 20 .
Additionally, the reading and writing slots can be different.
For example, at t 1 , the network writes to belt M 2 and reads from M 0 .
The black arrows on the left of the contents in external memory represent the reading contents.
This neural RAM network can degrade to a neural stack if the memory accessing order is restricted.
Their relationship is stated in Theorem 3.
Similarly, the network can degrade to LSTM if only one memory belt is used.
The above analysis indicates that neural RAM is the most powerful network among the models discussed in this paper.
Theorem 3.
Neural RAM can degrade to a neural stack if
i) all the reading weights except that of the topmost memory slot are set to zero, w r t ( i ) =
0 , if i (cid:54) = 0 ; ii) only the writing weight for the topmost memory slot is learned, and all others are copied from it, w rt ( i ) =
w rt (0) , if i (cid:54) = 0 ; iii) in the writing process, instead of learning all the contents to be written to the stack, as in Eq.(11), only the content of M0 is learned as c t (0) =
t ( w (cid:48)
T hc h t − 1 + b c ) + γ m t − 1 (1) , and all other contents are calculated as c t ( i )
= m t − 1 ( i − 1)
+ γ m t − 1 ( i + 1) , if i (cid:54) = 0 .
iv) only the writing and erasing weights for the topmost element are learned, and all others are copied from the values of the topmost element, w rt ( i )
=
w rt (0) , e t ( i ) = e t (0) .
Proof.
See Appendix B.3.
M APPING FROM NETWORK TYPES TO TASK TYPES
From the analysis above, we know that the proposed taxonomy for the four most conventional memory architectures appears as a simple way to quantify the past information: state memory, memory of one compounded event, memory of multiple events with restricted access order, memory of multiple events with randomly access order.
In this section, we will discuss how to utilize this proposed taxonomy to help practitioners select the right model.
The proposed taxonomy resembles the hierarchical organization of automata: vRNN ⇔ Finite state machine, neural stack ⇔
Deterministic pushdown automaton, neural RAM ⇔ Turing machine.
Hence, if our task is sequence recognition or classiﬁcation, the recognizable sequences for each network can be illustrated by the Chomsky hierarchy.
However, these networks can perform additional sequence learning tasks, such as prediction.
In this case, sequences do not need to satisfy the restricted grammars.
For example, no need exists for the input sequences to always start from a start state and return to an accepted state.
Hence, to ensure that our taxonomy ﬁts these more general sequences, we have to analyze the memory requirements of the speciﬁc task ﬁrst and match it with these four types of memory networks.
Four representative tasks that would ﬁt optimally the characteristics of each memory organization are discussed here: counting, counting with interference, reversing and repeat copying. Therefore, practitioners can compare their own problem with these four tasks and get some hints to select the right model.
We will analyze the memory requirements of each task individually.
Counting In the counting task, the input sequences are composed of a ’s, b ’s and c ’s. The output sequence is the cumulative count of the number of a ’s. For instance, when the input sequence is aaabcaa , the output sequence would be 1233345.
For this type of sequence, a state variable is required to remember the number of a ’s. A state transition occurs when an a is encountered.
In this problem, the state space is not very large, and a ﬁrst-order Markov state model provides a sufﬁcient description.
Hence, the counting task can be completed as long as the network has one feedback loop.
In this paper, "task can be completed" means the output error is almost zero.
Counting with interference In the counting with interference task, the input sequences are the same as those of the counting task.
The goal is still to count the number of a ’s, but if a b or c is encountered, the output should be b or c .
For example, if the input is aabbaca , the output sequence is 12 bb 3 c 4 .
For this type of problem, an external memory cache is required because the hidden layer’s output (internal memory) is overwritten when a b or c is encountered.
If we want to recall the number of a ’s, the value must be stored in external memory for future use, and an input gate must be used to protect the external memory when inputting b and c ( g i = 1 when the input is a , and g i = 0 when the input is b or c ).
Thus, LSTM, neural stack and neural RAM are capable of solving this problem.
However, in a vRNN, since the only memory is the state memory and the output is forced to be a function of this state memory, the interference of b and c prevent a vRNN from accomplishing this task.
Reversing The third task is sequence reversing.
For example, if the input sequence is abacdeδxxxxxx , the output sequence should be xxxxxxxedcaba.
δ is the delimiter symbol, and x represents any symbol.
When δ is encountered in the input sequence, regardless of the subsequent symbols, the output is the input symbols before δ in reverse order.
For this task, all the useful past information should be stored and then retrieved in reverse order.
Hence, the memory must have the ability to store more than one element, and the reading order is related to the writing order.
Since vRNN does not have a memory bank and LSTM’s memory is forgotten after being updated, these two networks fail in this task.
By contrast, the neural stack and neural RAM can both store more than one content and can solve this task, which satisfy the “ﬁrst in last out” principle.
Repeat copying The most challenging task is repeat copying, where the output sequence is the input sequence repeated several times.
For example, if the input sequence is adbcεxxxxxxxxxxxxxx, the output should be xxxxxxxadbcadbcadbc.
That is, when the repeating number symbol ε is encountered, the output is the previous input sequence repeated ε times.
For this type of task, more than one past content must be stored and retrieved multiple times, in this case, 3 times.
Since all the saved information in a neural stack is forgotten after being popped out, a neural stack cannot perform this task; thus, only a neural RAM can complete this task.
This classiﬁcation of tasks is meaningful since it can guide users in the right direction.
If we select the wrong type of network, an error and/or speed penalty will occur regardless of how we adjust the hyperparameters.
As shown in our experiment, for sequence reversing (i.e., the third type of task), neural stack and neural RAM with 6 hidden neurons converge to near zero error, but for vRNN and LSTM, even if we set the number of hidden neurons to 1000, the output will always ﬂuctuate around a nonzero value.
E XPERIMENTS
We test the performance of the four networks on the synthetic tasks described in last section to illustrate the impact of different memory organizations.
We also visualize how each network encodes information to solve a problem in AppendixD. Then, we use two natural language processing applications to elaborate how to employ the knowledge gained from the different characteristics of the memory structures to help users to select the appropriate type of network.
The details of parameters settings are given in AppendixC.
6. S YNTHETIC TASKS
The learning curves for the four tasks when using different networks are shown in Fig.6.
The performance is measured in terms of MSE for ﬁrst two tasks and output entropy for the last two tasks.
We use the same number of units in all the architectures to ensure a fair comparison.
The results indicate that for counting, all four networks can achieve an almost zero error; for counting with interference, all the networks except vRNN can complete the task; for sequence reversing, neural stack and neural RAM are suitable networks; and for repeat copying, neural RAM is the only network that can solve the problem.
We also apply different parameter settings, for instance, varying the number of hidden units from 5 to 1000. The performances are the same as shown in Fig.6, except a different nonzero error value is observed when the network is not capable of accomplishing the task.
6. N ATURAL LANGUAGE PROCESSING
Synthetic problems could be designed to exemplify the expressive power of the different memory networks.
For real-world problems, this task is more complex because the memory requirements of the problem may be difﬁcult to determine and the problem may be a blend of classes.
In this case, all the networks may be able to solve the problem to a certain extent.
Hence, in this section, we illustrate how to select the minimum network resources to accomplish the task with relatively good performance.
Sentiment analysis The ﬁrst experiment is sentiment analysis to infer the emotional tone of a text as negative or positive.
The sentiment of a text is mainly based on the occurrence and reoccurrence of some discriminating words.
Speciﬁcally, if there are a lot of positive words, such as ‘good, love’ in the text, the sentiment of the text is always positive, and vice versa.
Hence, external memory whose value is affected by these discriminating words is useful to judge the emotional tone of text. Thus
LSTM, neural stack and neural RAM should perform better than vRNN.
[Table] [Table] Since the goal is to classify the emotional tone as either 1 or 0, the speciﬁc contents of the text are not important and thus do not need to be all stored.
For example, as long as it is a positive word, whether it is ‘love’‘good’ or ‘like’ is not very important.
Hence, multi-slot memory which can store multiple contents does not have more advantages than the single-slot memory.
We test the network performance on the IMDb movie review dataset Pennington et al.
The results are presented in Table 2, which shows that vRNN performs worst.
LSTM, neural stack, and neural RAM have similar performances; thus, our analysis is veriﬁed.
Question answering The second experiment investigates the performance of these four networks on three question answering tasks from the bAbI datasetWeston et al.
The target is to answer a question after reading a short story.
For example, for the story “Mary got the milk there.
John moved to the bedroom.
Sandra went back to the kitchen.
Mary travelled to the hallway.”, and the question “Where is the milk?”, the expected answer is “hallway”.
For this problem, to give the correct answer, the network must memorize the facts that Mary got the milk and traveled to the hallway.
Additionally, since the does not know the question before reading the story, all the potentially useful facts must be stored.
Thus, an external memory bank in which any content can be visited is useful.
According to our memory capability analysis, the neural RAM should perform the best in this scenario.
The results in Table 3 conﬁrm that neural RAM achieves the best performance. From the results we can see that, neural stack’s performance is worse than LSTM. We know that both LSTM and neural stack are not suitable for this task.
If we apply LSTM or neural stack to the problem, they will try their best to ﬁnd their approximate solutions.
If we apply LSTM to this task, LSTM would compress all the useful information in its external memory, although the right answer is mixed with other information, the output can at least get some information from it.
However, if we apply neural stack to this task, the push signal sometimes is very large ( d push ≈ 1 , d pop ≈ 0 , d no − op ≈ 0 ), which means that the right answer will then be pushed down in the stack which cannot be accessed when needed. Hence, we think although the neural stack attempts to ﬁnd the right answer, it is more likely to be stuck in local points due to its complicated operation. But we have to be aware that it is not due to the memory limitation of stack itself, a better implementation of the three operators may help improve neural’s stack performance for this kind of problem.  C ONCLUSION
In this paper, we analyze the memory structures of several recurrent networks and propose a taxonomy.
We use four synthetic tasks and two natural language processing problems to illustrate utility of the taxonomy.
Although we show differences in performance in the experiments, it is too early to say that we presented all the tools necessary to select the parsimonious memory architecture for a given application.
Because the user has to analyze the requirements of the application, which may not be trivial, additionally work is needed to create guidelines to assist practitioners.
A N ETWORK ARCHITECTURE COMPONENTS A. LSTM
The three gates are in LSTM are calculated as follows:
[Equation]
[Equation] [Equation] where w hg i , w hg f , w hg o are K h × 1 weights , w hg i , w hg f , w hg o are K
i × 1 weights and b g i ,
b g i and b g i are bias.
These three gates give ﬂexibility to operate on memories.
A. N EURAL S TACK
Neural network interacts with the stack memory by d pusht , d popt , d no − op t , c t and r t .
According to
Sun (1993) the domain of the operations is relaxed to any real value in [0 , 1] .
This extension adds a continuous amplitude dimension to the operations.
For example, if the push signal d push = 1 , the current vector will be pushed into the stack as it is, if d push = 0 . 8 , the current vector is ﬁrst multiplied by 0.8 and then pushed into the stack.
d
[Equation]
, d
[Equation]
, d no
[Equation]
op t and c t are decided by the hidden layer outputs and the corresponding weights,
[Equation]
where w hd is the K h × 3 weights and b op is the 3 × 1 bias.
[Equation]
where w hc is the K h × N weights and b op is the N × 1 bias.
Here we assume all the elements saved in the stack are N × 1 vectors.
A. N EURAL RAM
In the neural RAM, the read weighting w rt ( i ) is learned as,
[Equation]
(17) w ha is the K h
× M weight and b a is M × 1 bias, [Equation]
rt [Equation] [Equation] [Equation]
[Equation]
The nonlinear activation function f is usually set as softmax function.
The write weighting is learned as,
[Equation]
(18) where w hb is K h × M weight, w xb are K
i × M weight, b b is M × 1 bias.
And the erase weighting is learned as,
[Equation]
(19) where e t = [ e t (1) , e t (2) , ..., e t ( M )] T , w he is K h
× M weights, w xe is K
i × M weight, and b e are M × 1 bias.
In practice, instead of learning the read and write head from scratch, some methods were proposed to simplify the learning process.
For example in Neural Turing machine Graves et al.
(2014), e t ( i ) is coupled with write weight w wt ( i ) , e t ( i ) = 1 − w wt ( i ) .
And reading weight w rt and writing weight w wt are obtained by content-addressing and location-addressing mechanisms.
The content-addressing mechanism gives the weights w rt ( i ) (or w wt ( i ) ) by checking the similarity of the key d with all the contents in the memory, the normalized version is,
[Equation]
where α is the parameter to control the precision of the focus, K is a similarity measure.
Then, the weights will be further adapted by the location-addressing mechanism.
For example, the weights obtained by content addressing can ﬁrstly blend with the previous weight and then shifted for several steps,
[Equation]
g t is the gate to balance the previous weight and current weight, n is the shifting steps, [ i − n ] M means the circular shift for M entities.
Since the shifting operation is not differentiable, the method in Graves et al.
(2014) should be utilized as an approximation.
Another example is Graves et al.
(2016) which improves the performance even more.
To be speciﬁc, for reading, a matrix to remember the order of memory locations they are written to can be introduced.
With this matrix, the read weight is a combination of the content-lookup and the iterations through the memory location in the order they are written to.
And for writing, a usage vector is introduced, which guides the network to write more likely to the unused memory.
With this modiﬁcation, the neural RAM gets ﬂexibility similar to working memory of human cognition which makes it more suitable to intelligent prediction.
With these modiﬁcations, the training time for the neural RAM is also reduced.
B T HEOREM PROOFS B. P ROOF OF T HEOREM
Compared to vRNN, LSTM introduces an external memory and the gate operation mechanism.
So if we set the output gate g o = 0 , input gate g i = 1 and the forget gate g f = 0 instead of learning from the sequences, the dynamics of LSTM is degraded to vRNN as follows, [Equation]
[Equation]
Here (23) is due to g o = 0 , (24) is {+due to g i = 
+}
and g f = 0 , (26) is because the weight w hc and bias b c are set as constants and the activation function t ( x ) is set as linear activation function,
[Equation]
hc
[Equation]
Since Eq.(22) is the dynamic of LSTM and Eq.(26) is the dynamic of vRNN, the argument that vRNN is a special case of LSTM is proved.
B. P ROOF OF T HEOREM
The dynamic of LSTM is,
[Equation]
And the dynamic of neural stack is,
[Equation]
According to Eq.(27) and Eq.(28), the dynamics of the neural stack have similar form as LSTM except for the reading vector, i.e., the reading vector is r t =
g o s t (0) .
If we set the pop signal as zero, d popt = 0 , and no operation on the stack contents except for the topmost elements is available, then
[Equation]
Since the d pusht , d no − [Equation] t are calculated in the same way the input gate g i,t and forget gate g f,t are calculated in LSTM as shown in Eq.(14) to Eq.(15), the read vector would be,
[Equation]
In this manner, this is exactly how the LSTM organizes its memory,
[Equation]
d pusht can be seen as the input gate and d no − op t can be seen as the forget gate.
Hence, it is proved that
LSTM can be seen as a special case of neural stack.
B. P ROOF OF T HEOREM
Neural RAM is more powerful than neural stack because it has access to all the contents in the memory bank.
If we restrict the read and write vector, neural RAM is degraded to neural stack. To be speciﬁc, for the read head w rt , all the read weights except the topmost are set to zeros,
[Equation]
here w (cid:48) ha is K h × 1 vector
and b a is the scalar.
Eq.(29) is a special case of Eq.(17).
[Equation]
all others are calculated as,
[Equation]
[-Here (29) is due to g o = 0 , (30) is due to g i = 
 and g f = 0 , (32) is because-]And in the writing process, Eq.(30), Eq.(31) can be seen as a special case of Eq.(11) because h t − 1 in (11) is a function of m t − 1 . Substitute (30)(31) into the memory update equation of neural RAM (12), we get,
[Equation]
Finally since Eq.(3) and Eq.(3) can be seen as a special case of Eq.(18) and Eq.(19), the neural stack can be treated as a special case of neural RAM. Compared the memory writing operation of neural stack (7) and neural RAM (32), we can see that, w wt (0) , γw wt (0) and e t (0) works as the push, pop and no-operate operations respectively. Compared the memory reading operation of neural stack and neural RAM, we can see that we can see that, w rt (0) in neural RAM (29) works as the output gate in neural stack (9).
C E XPERIMENT PARAMETERS SETTING C. COUNTING AND COUNTING WITH INTERFERENCE
In the experiment, the activation function is Relu in vRNN.
In LSTM, the external memory’s content is initialized as zero.
In the neural stack, the push, pop and no-op operations are initialized as random numbers with mean 0 and variance 1.
At ﬁrst, there is only one content in the stack which is initialized as zero.
The depth of the stack can increase to any number as required.
In neural RAM, memory depth is set as M = 3 .
In LSTM, neural stack and neural RAM, memory width is set as N = 3 , the nonlinear activation functions for all the gates are sigmoid functions and others are tanh.
The number of input neurons, hidden neurons and output neurons are 3.
All the weights are initialized as random numbers with mean 0 and variance 1, all the bias are initialized as 0.1.
For counting task, the model is trained with the synthetic sequences up to length 20.
When the input is a , the ﬁrst elements in the output vector would add one, otherwise, the output vector is unchanged.
After encoding, the input and output vectors are, time step input sequence output sequence [1 0 0]
For counting with interference task, after encoding, the input and output vectors are, [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
C. REVERSING AND REPEAT COPYING
Some network settings are different from the ﬁrst two experiments.
In vRNN, the activation function in the hidden layer is sigmoid function since we use entropy instead of mean square error as the cost function.
In neural RAM, the word size and memory depth are set as 16.
The length of read and write vectors are also set as 16.
The number of input neurons, hidden neurons and output neurons are 6, 64, 6.
The model is trained with sequences up to length 20.
In repeat copying experiment, the training sequences are composed of a starting symbol (cid:15) , some symbols in set { a, b, c, d, e } followed by a repeating number symbol δ and some random symbols.
(cid:15), a, b, c, d, e are one-hot encoded with on value 1 and off value 0; δ is encoded with on value n and off value 0, n is the repeating number.
C. SENTIMENT ANALYSIS
In our experiments, the number of input neurons, hidden neurons and output neurons are 50, 64,for all the four network architectures.
After encoding all the words into vectors, they are fed into the network one by one.
Here we use a pretrained model: GloVe Pennington et al.
(2014) to create our word vector.
The matrix contains 400,000 word vectors, each with a dimensionality of 50.
The matrix is created in a way that words having similar deﬁnitions or context reside in the relatively same position in the vector space.
The decision of the tone of the paragraph will be made at the end of the paragraph.
The output is [1, 0] for the positive text and [0, 1] for the negative text.
The dataset adopted here is the lmdb movie review dataMaas et al.
(2011) which has 12500 positive reviews and 12500 negative reviews.
Here we use 11500 reviews for training and 1000 data for testing.
In our experiments, the nonlinear activation functions for all the gates are sigmoid.
The activation functions at the output layer are sigmoid and others are tanh.
In neural RAM, the word size and memory depth are set as 64.
The number of read and write head are 4 and 1.
The results are average of 20 runs with random initializations.
C. QUESTION ANSWERING
For each task, we use the 10,000 questions to train and report the error rates on the test set.
The experimental settings for LSTM and neural RAM are the same as Graves et al.
(2016) and the results for these two networks are from Graves et al.
In vRNN and neural stack, the nonlinear activation functions for all the gates are sigmoid.
The activation functions at the output layer are sigmoid and others are tanh.
The number of input neurons, hidden neurons and output neurons are 150, 64, 150.
The memory width for neural stack is 64.
D MEMORY WORKING PROCESS VISUALIZATION FOR THE SYNTHETIC TASKS D. COUNTING
Fig.7 to Fig.10 show the details of the memory contents after the models are well-trained.
They are tested on a input sequence bbacacbabababcc .
The little checks on the left side of each image mark the memories which affect the output results.
In Fig.7, when a is received, the ﬁrst element of the hidden layer is increased by 1.
In Fig.8, when a is received, the ﬁrst element of M 0 is decreased by around 0.15.
As long as there is at least one element ( h 0 in vRNN and the ﬁrst element of M 0 in
LSTM) in the memory learning the pattern, after multiplying with the weight vector, the output of the network can give the expected values.
Fig.9 shows how the neural stack uses its memory.
Although the neural stack has the potential to use unbounded number of stack contents, it only uses the topmost content ( M 0 ) here, i.e. the push and no-operation cooperate to learn the pattern.
The contents pushed into M 1 to M 5 are never used again.
Fig.10 shows the memory contents of the neural RAM and the corresponding operations of it.
From Fig.10(a), we can see that M 0 , M 1 and M 2 cooperate to learn the pattern.
When a is received, the values in M 0 , M 1 and M 2 are increased.
The read weights for
M 0 , M 1 and M 2 are all around 0.3 as shown in Fig.10(b).
From this experiment we can see that
M 1 and M 2 are redundant here.
D. COUNTING WITH INTERFERENCE
The experiment settings are same as "counting" task.
Fig.11 to Fig.13 shows the memory usage of LSTM, neural stack and neural RAM.
Fig.11 shows that when a is received, the third element of the memory content would increase by around 0.2.
Fig.12 and Fig.13 also show the similar incremental patterns of neural stack and neural RAM.
An notable difference between Fig.11-Fig.
and Fig.8-Fig.9 is the usage of the memory.
When dealing with counting task, the output gates are always 1, however, when dealing with counting with interference task, the output gates are 0 when inputting b and c , this helps to cut off the interference from the memory. Similarly, the read vector
[Figure] [Figure] are always around [0.3, 0.3, 0.3] in Fig.10, however, in Fig.13, the read vector’s elements are almost zeros when encountering b and c .
The read vector here works as the output gate in LSTM and neural stack.
It also shows why neural RAM does not need an output gate.
D. REVERSING
Fig.14 shows how neural stack utilizes its stack memory to solve this problem.
Since each memory bank’s word size is 16, here we only use colors instead of the speciﬁc numbers to show the values of contents in memory.
Different from the ﬁrst two tasks, the function of the stack is ﬁnally exploited.
In the ﬁrst half sequence, the input symbols are encoded as 16-elements vectors and pushed into the stack.
In the second half of the sequence, the contents in the stack are popped out sequentially.
It should be noticed that as long as the contents are popped out, they can not be revisited anymore.
Different from neural stack, the contents in neural stack are never wiped as shown in Fig.15.
The contents in the memory bank are only wiped if they are useless in the future or they have to be wiped [Figure] and corresponding read and write to make space for new stuffs.
Another feature of the memory bank for neural RAM is the memory banks are not used in order such as M0, M1, M2...
In this example, the memory banks are used in the order M0, M2, M7, M13....
But as long as the network knows the writing order, the task can be accomplished.
Fig15(b)(c) shows the reading and writing weights, we can see that the second half of the reading weights is the mirror of the ﬁrst half of the sequence of the writing weights, which means the network learns to reverse.
D. REPEAT COPYING
Since neural RAM is the only network that can handle this problem, Fig.16 shows how the neural
RAM solves this problem.
From the writing weights we can see that, the starting symbol is saved in
M0, and symbols needed to be repeated are save in M2, M4, M6, M9.
After t=4, the network would read from M2/M5, M4,M6,M9.
At the beginning of every loop, the network reads from both M and M5 probably because the repeating time symbol δ is saved in M5.
The value in M5 can tell the
[Figure]