Original: L EARNING W HAT TO R EMEMBER : L ONG - TERM E PISODIC M EMORY N ETWORKS FOR L EARNING FROM S TREAMING D ATA
Revised: L EARNING W HAT TO R EMEMBER : L ONG - TERM E PISODIC M EMORY N ETWORKS FOR L EARNING FROM S TREAMING D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Current generation of memory-augmented neural networks has limited scalability as they cannot efﬁciently process data that are too large to ﬁt in the external memory storage.
Revised: Current generation of memory-augmented neural networks has limited scalability as they cannot efﬁciently process data that are too large to ﬁt in the external memory storage.

Original: One example of this is lifelong learning scenario where the model receives unlimited length of data stream as an input which contains vast majority of uninformative entries.
Revised: One example of this is lifelong learning scenario where the model receives unlimited length of data stream as an input which contains vast majority of uninformative entries.

Original: We tackle this problem by proposing a memory network ﬁt for long-term lifelong learning scenario, which we refer to as Long-term
Revised: We tackle this problem by proposing a memory network ﬁt for long-term lifelong learning scenario, which we refer to as Long-term

Original: Episodic Memory Networks (LEMN), that features a RNN-based retention agent that learns to replace less important memory entries based on the retention probability generated on each entry that is learned to identify data instances of generic importance relative to other memory entries, as well as its historical importance.
Revised: Episodic Memory Networks (LEMN), that features a RNN-based retention agent that learns to replace less important memory entries based on the retention probability generated on each entry that is learned to identify data instances of generic importance relative to other memory entries, as well as its historical importance.

Original: Such learning of retention agent allows our long-term episodic memory network to retain memory entries of generic importance for a given task.
Revised: Such learning of retention agent allows our long-term episodic memory network to retain memory entries of generic importance for a given task.

Original: We validate our model on a path-ﬁnding task as well as synthetic and real question answering tasks, on which our model achieves signiﬁcant improvements over the memoryaugmented networks with rule-based memory scheduling as well as an RL-based baseline that does not consider relative or historical importance of the memory.
Revised: We validate our model on a path-ﬁnding task as well as synthetic and real question answering tasks, on which our model achieves signiﬁcant improvements over the memoryaugmented networks with rule-based memory scheduling as well as an RL-based baseline that does not consider relative or historical importance of the memory.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In a real world machine learning problem, it is common to process a tremendous amount of sequential data (e.g. dialogues, videos, and medical records).
Revised: In a real world machine learning problem, it is common to process a tremendous amount of sequential data (e.g. dialogues, videos, and medical records).

Original: To model such sequential data, researchers often use recurrent neural networks (RNNs) because of its capability to process sequentially dependent data.
Revised: To model such sequential data, researchers often use recurrent neural networks (RNNs) because of its capability to process sequentially dependent data.

Original: Recently, popular variants of RNNs such as Long Short-Term Memory (LSTM, Hochreiter & Schmidhuber (1997)) and Gated Recurrent Unit (GRU, Chung et al.
Revised: Recently, popular variants of RNNs such as Long Short-Term Memory (LSTM, Hochreiter & Schmidhuber (1997)) and Gated Recurrent Unit (GRU, Chung et al.

Original: (2014)) have achieved impressive performances on a number of tasks, such as machine translation, image annotation, and question answering.
Revised: (2014)) have achieved impressive performances on a number of tasks, such as machine translation, image annotation, and question answering.

Original: Nevertheless, despite such impressive achievements of the recurrent neural networks, it is still challenging to model long sequential data such as dialogues since RNNs have a limited memory capacity.
Revised: Nevertheless, despite such impressive achievements of the recurrent neural networks, it is still challenging to model long sequential data such as dialogues since RNNs have a limited memory capacity.

Original: The only memory that LSTM or RNN contains is a ﬁxed dimensional vector.
Revised: The only memory that LSTM or RNN contains is a ﬁxed dimensional vector.

Original: The LSTM or RNN performs memorization by adding to, and erasing, this single vector.
Revised: The LSTM or RNN performs memorization by adding to, and erasing, this single vector.

Original: This limited memory capacity can be problematic when dealing with a long sequence.
Revised: This limited memory capacity can be problematic when dealing with a long sequence.

Original: To overcome this scalability problem, researchers have proposed neural network architectures that contain external memory module, including Neural Turing machine (NTM, Graves et al.
Revised: To overcome this scalability problem, researchers have proposed neural network architectures that contain external memory module, including Neural Turing machine (NTM, Graves et al.

Original: and End-to-End Memory Networks (Sukhbaatar et al. (2015)).
Revised: and End-to-End Memory Networks (Sukhbaatar et al. (2015)).

Original: These memory-augmented networks have mechanisms to write representation to the memory cells and read from them.
Revised: These memory-augmented networks have mechanisms to write representation to the memory cells and read from them.

Original: The addressing of the memory cells can be done either by an index to perform sequential read/write or by contents to perform a content-based access.
Revised: The addressing of the memory cells can be done either by an index to perform sequential read/write or by contents to perform a content-based access.

Original: The memory-augmented networks have been shown to be useful for memory-oriented tasks such as copying, pathﬁnding, and question answering.
Revised: The memory-augmented networks have been shown to be useful for memory-oriented tasks such as copying, pathﬁnding, and question answering.

Original: Yet, they are mostly targeting toy problems, using the external memory as a working memory in the human memory mechanism, and need to manage a relatively small number of memory entries.
Revised: Yet, they are mostly targeting toy problems, using the external memory as a working memory in the human memory mechanism, and need to manage a relatively small number of memory entries.

Original: Thus, there has been less focus on how to efﬁciently manage the memory.
Revised: Thus, there has been less focus on how to efﬁciently manage the memory.

Original: However, when the network needs to process data that are too large to ﬁt into the memory, we need to carefully manage the memory cell such that the external memory module contains the most informative pieces of data.
Revised: However, when the network needs to process data that are too large to ﬁt into the memory, we need to carefully manage the memory cell such that the external memory module contains the most informative pieces of data.

Original: In other words, we should be able to determine which memory cell is unlikely to be used in the future so it can be replaced with the incoming data when memory is full.
Revised: In other words, we should be able to determine which memory cell is unlikely to be used in the future so it can be replaced with the incoming data when memory is full.

Original: To this end, we consider this memory management problem as a learning problem and train an agent by reinforcement learning, such that it learns to keep the most important information in memory and evict unimportant ones.
Revised: To this end, we consider this memory management problem as a learning problem and train an agent by reinforcement learning, such that it learns to keep the most important information in memory and evict unimportant ones.

Original: How can we then select which memory entries are most important?
Revised: How can we then select which memory entries are most important?

Original: To achieve this goal, we propose to train the memory module itself using reinforcement learning to delete the most uninformative memory entry in order to maximize its reward on a future task.
Revised: To achieve this goal, we propose to train the memory module itself using reinforcement learning to delete the most uninformative memory entry in order to maximize its reward on a future task.

Original: However, this is a challenging task since for most of the time, this scheduling should be performed without knowing which task will arrive when.
Revised: However, this is a challenging task since for most of the time, this scheduling should be performed without knowing which task will arrive when.

Original: Thus, deciding which memory to keep and which to erase should be done by considering relative generic importance of the memory entries.
Revised: Thus, deciding which memory to keep and which to erase should be done by considering relative generic importance of the memory entries.

Original: To tackle this challenge, we implemented the memory retention mechanism using a spatio-temporal recurrent neural network, that can learn relative importance among the memory entries as well as their historic importance.
Revised: To tackle this challenge, we implemented the memory retention mechanism using a spatio-temporal recurrent neural network, that can learn relative importance among the memory entries as well as their historic importance.

Original: We refer to this memory-augmented networks with spatio-temporal retention mechanism as Long Term Episodic
Revised: We refer to this memory-augmented networks with spatio-temporal retention mechanism as Long Term Episodic

Original: Memory Networks (LEMN).
Revised: Memory Networks (LEMN).

Original: LEMN can perform selective memorization to keep a compact set of important pieces of data that will be useful for future tasks in lifelong learning scenarios.
Revised: LEMN can perform selective memorization to keep a compact set of important pieces of data that will be useful for future tasks in lifelong learning scenarios.

Original: We validate our proposed retention mechanism against naive scheduling method as well as RL-based scheduling on three different tasks, namely path-ﬁnding, episodic question answering and long question answering, against which it signiﬁcantly outperforms.
Revised: We validate our proposed retention mechanism against naive scheduling method as well as RL-based scheduling on three different tasks, namely path-ﬁnding, episodic question answering and long question answering, against which it signiﬁcantly outperforms.

Original: Our contribution is twofold:
Revised: Our contribution is twofold:

Original: • We consider a novel task of learning from streaming data, where the size of the memory is signiﬁcantly smaller than the length of the data stream.
Revised: • We consider a novel task of learning from streaming data, where the size of the memory is signiﬁcantly smaller than the length of the data stream.

Original: • To implement a retention mechanism, we propose a retention agent that can be integrated with existing external memory neural networks, which is trained with reinforcement learning to keep memory cells of general importance.
Revised: • To implement a retention mechanism, we propose a retention agent that can be integrated with existing external memory neural networks, which is trained with reinforcement learning to keep memory cells of general importance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Memory-augmented networks Graves et al.
Revised: Memory-augmented networks Graves et al.

Original: (2014) propose Neural Turing Machine (NTM), which has a memory and a controller that reads from and writes to memory.
Revised: (2014) propose Neural Turing Machine (NTM), which has a memory and a controller that reads from and writes to memory.

Original: The controller composed of read heads and write heads uses soft-attention mechanism to access memory for differentiable memory access.
Revised: The controller composed of read heads and write heads uses soft-attention mechanism to access memory for differentiable memory access.

Original: The NTM has two addressing mechanisms: content-based addressing and locationbased addressing.
Revised: The NTM has two addressing mechanisms: content-based addressing and locationbased addressing.

Original: Location-based addressing allows a data sequence to be stored in consecutive memory cells to preserve its sequential order.
Revised: Location-based addressing allows a data sequence to be stored in consecutive memory cells to preserve its sequential order.

Original: However, once the write head accesses a distant memory cells, the sequential order of information in consecutive memory cells is no longer preserved.
Revised: However, once the write head accesses a distant memory cells, the sequential order of information in consecutive memory cells is no longer preserved.

Original: Graves et al.
Revised: Graves et al.

Original: (2016) propose Differentiable Neural Computer that extends the NTM to address the issue by introducing a temporal link matrix.
Revised: (2016) propose Differentiable Neural Computer that extends the NTM to address the issue by introducing a temporal link matrix.

Original: As it is costly to write into memory while preserving its sequential order, memory-augmented networks with write mechanism are mostly used in cases where it is not necessary to track which sequential order the memory has been written (Santoro et al., 2016;
Revised: As it is costly to write into memory while preserving its sequential order, memory-augmented networks with write mechanism are mostly used in cases where it is not necessary to track which sequential order the memory has been written (Santoro et al., 2016;

Original: Vinyals et al., 2016; Kaiser et al., 2017; Kim et al., 2018).
Revised: Vinyals et al., 2016; Kaiser et al., 2017; Kim et al., 2018).

Original: Unlike NTM, End-to-End Memory
Revised: Unlike NTM, End-to-End Memory

Original: Networks (MemN2N) (Sukhbaatar et al., 2015) and Dynamic Memory Networks (DMN+) (Xiong et al., 2016) do not have write mechanism but store all the inputs into memory.
Revised: Networks (MemN2N) (Sukhbaatar et al., 2015) and Dynamic Memory Networks (DMN+) (Xiong et al., 2016) do not have write mechanism but store all the inputs into memory.

Original: For this reason, the sequential order of the inputs are preserved at no extra cost; thus they are more suitable for episodic question answering problems such as bAbI tasks (Weston et al., 2015).
Revised: For this reason, the sequential order of the inputs are preserved at no extra cost; thus they are more suitable for episodic question answering problems such as bAbI tasks (Weston et al., 2015).

Original: In addition, they have sophisticated read mechanisms that allow to reason through multiple hops (or passes in DMN+).
Revised: In addition, they have sophisticated read mechanisms that allow to reason through multiple hops (or passes in DMN+).

Original: Seo et al.
Revised: Seo et al.

Original: (2016) propose more advanced soft-attention based read mechanism, Bi-Directional Attention
Revised: (2016) propose more advanced soft-attention based read mechanism, Bi-Directional Attention

Original: Flow (BiDAF), which obtains impressive performance on a difﬁcult reading comprehension dataset,
Revised: Flow (BiDAF), which obtains impressive performance on a difﬁcult reading comprehension dataset,

Original: Stanford Question Answering Dataset (SQuAD, Rajpurkar et al.
Revised: Stanford Question Answering Dataset (SQuAD, Rajpurkar et al.

Original: Oh et al.
Revised: Oh et al.

Original: (2016) extend
Revised: (2016) extend

Original: MemN2N to train deep reinforcement learning agents, Memory Q-Network (MQN) and Feedback
Revised: MemN2N to train deep reinforcement learning agents, Memory Q-Network (MQN) and Feedback

Original: Recurrent Memory Q-Network (FRMQN), to solve 3D Mazes.
Revised: Recurrent Memory Q-Network (FRMQN), to solve 3D Mazes.

Original: However, such neural networks have the limitation that the size of external memory should be large enough to store all the data.
Revised: However, such neural networks have the limitation that the size of external memory should be large enough to store all the data.

Original: Memory Retention Policy Most of the existing approaches (MemN2N, DMN+, BiDAF and FRMQN) do not consider the case where memory becomes full, and simply truncate the sequence of data to the size of memory if it is longer than the memory size.
Revised: Memory Retention Policy Most of the existing approaches (MemN2N, DMN+, BiDAF and FRMQN) do not consider the case where memory becomes full, and simply truncate the sequence of data to the size of memory if it is longer than the memory size.

Original: Many of mutable external memory neural networks employ least-recently-used (LRU) based memory retention policy, which overwrites new data into the least used memory cell.
Revised: Many of mutable external memory neural networks employ least-recently-used (LRU) based memory retention policy, which overwrites new data into the least used memory cell.

Original: This policy, although more reasonable than FIFO, is still limited as it is a hand-crafted rule and does not consider actual long-term dependencies between a memory entry and the task.
Revised: This policy, although more reasonable than FIFO, is still limited as it is a hand-crafted rule and does not consider actual long-term dependencies between a memory entry and the task.

Original: This is a critical limitation in lifelong learning setting, since some of the memory entries written in the long past and have not been accessed for long may still be necessary to respond to queries that arrive in the far future.
Revised: This is a critical limitation in lifelong learning setting, since some of the memory entries written in the long past and have not been accessed for long may still be necessary to respond to queries that arrive in the far future.

Original: Our work, on the other hand, is able to learn such long-term dependencies.
Revised: Our work, on the other hand, is able to learn such long-term dependencies.

Original: DNTM (G¨ulc¸ehre et al., 2016) learns where to overwrite using reinforcement learning as done in our work.
Revised: DNTM (G¨ulc¸ehre et al., 2016) learns where to overwrite using reinforcement learning as done in our work.

Original: Yet, it only considers the pairwise relationships between the current data instance and each individual memory entries, while our model learns both relative importance and historic importance of a memory entry using a spatio-temporal RNN architecture.
Revised: Yet, it only considers the pairwise relationships between the current data instance and each individual memory entries, while our model learns both relative importance and historic importance of a memory entry using a spatio-temporal RNN architecture.

Original:  L EARNING W HAT TO R EMEMBER FROM S TREAMING D ATA
Revised:  L EARNING W HAT TO R EMEMBER FROM S TREAMING D ATA

Original: We consider the problem of learning from a long data stream that contains large portion of unimportant, noisy data (e.g. routine greetings in dialogs) with limited memory.
Revised: We consider the problem of learning from a long data stream that contains large portion of unimportant, noisy data (e.g. routine greetings in dialogs) with limited memory.

Original: Formally, an agent A takes as input a streaming data (e.g. sentences or images)
Revised: Formally, an agent A takes as input a streaming data (e.g. sentences or images)

Original: X
Revised: X

Original: = { x 1 , · · · , x T } and manages an external memory M =
Revised: = { x 1 , · · · , x T } and manages an external memory M =

Original: [ m 1 , · · · , m N ] , m i
Revised: [ m 1 , · · · , m N ] , m i

Original: ∈ R d where T (cid:29) N .
Revised: ∈ R d where T (cid:29) N .

Original: Thus, the agent should decide which memory cell to evict to store incoming data.
Revised: Thus, the agent should decide which memory cell to evict to store incoming data.

Original: To this end, the agent should learn the relative importance of memory entries for a future task.
Revised: To this end, the agent should learn the relative importance of memory entries for a future task.

Original: In traditional reinforcement learning scheme, we can formulate this problem as learning the policy π ( a t | s t ) to maximize a return R , where action a t is a memory cell to erase, and state s t =
Revised: In traditional reinforcement learning scheme, we can formulate this problem as learning the policy π ( a t | s t ) to maximize a return R , where action a t is a memory cell to erase, and state s t =

Original: [ M t ; x t ] , where M t is the current memory and x t is the input at time t .
Revised: [ M t ; x t ] , where M t is the current memory and x t is the input at time t .

Original: The agent appends x t to the memory M t until it reaches the maximum size N .
Revised: The agent appends x t to the memory M t until it reaches the maximum size N .

Original: If the memory is full, the agent erases one memory cell based on the policy π ( m i | M t , x t ) to append x t .
Revised: If the memory is full, the agent erases one memory cell based on the policy π ( m i | M t , x t ) to append x t .

Original: Thus, we can consider 1 − π ( m i | M t , x t ) as the retention value of each memory.
Revised: Thus, we can consider 1 − π ( m i | M t , x t ) as the retention value of each memory.

Original: At arbitrary time step t f , it encounters a task T (e.q. question answering) with a reward R T , whose reward is deﬁned by the speciﬁc task.
Revised: At arbitrary time step t f , it encounters a task T (e.q. question answering) with a reward R T , whose reward is deﬁned by the speciﬁc task.

Original: For QA task, the reward will be 1 if it generated a correct answer and −otherwise.
Revised: For QA task, the reward will be 1 if it generated a correct answer and −otherwise.

Original: The instance x t which arrive at timestep t can be in any data format, and is transformed into a memory vector representation
Revised: The instance x t which arrive at timestep t can be in any data format, and is transformed into a memory vector representation

Original: e t ∈ R
Revised: e t ∈ R

Original: d to be stored in memory.
Revised: d to be stored in memory.

Original: [Figure]
Revised: [Figure]

Original: 3. M EMORY -R ETENTION M ECHANISMS
Revised: 3. M EMORY -R ETENTION M ECHANISMS

Original: In this section, we describe three different memory-retention mechanisms in detail.
Revised: In this section, we describe three different memory-retention mechanisms in detail.

Original: We ﬁrst encode input x t and each memory cell m t,i to a vector representation as follows:
Revised: We ﬁrst encode input x t and each memory cell m t,i to a vector representation as follows:

Original: [Equation]
Revised: [Equation]

Original: where c t , e t,i ∈ R d , ψ ( · ) can be an input embedding similar to RNN controller in (G¨ulc¸ehre et al., 2016) or position encoding in (Sukhbaatar et al., 2015) and ϕ ( · ) can be a memory encoding layer of the base external memory neural network.
Revised: where c t , e t,i ∈ R d , ψ ( · ) can be an input embedding similar to RNN controller in (G¨ulc¸ehre et al., 2016) or position encoding in (Sukhbaatar et al., 2015) and ϕ ( · ) can be a memory encoding layer of the base external memory neural network.

Original: For example, MemN2N (Sukhbaatar et al., 2015) uses a bag of words and embedding matrices to convert to memory representation.
Revised: For example, MemN2N (Sukhbaatar et al., 2015) uses a bag of words and embedding matrices to convert to memory representation.

Original: Input-Matching LEMN
Revised: Input-Matching LEMN

Original: This is the simplest RL-based retention mechanism that is similar to
Revised: This is the simplest RL-based retention mechanism that is similar to

Original: DNTM from G¨ulc¸ehre et al.
Revised: DNTM from G¨ulc¸ehre et al.

Original: and utilizes the memory usage information, which we refer to as Input-Matching LEMN (IM-LEMN, Figure 1a).
Revised: and utilizes the memory usage information, which we refer to as Input-Matching LEMN (IM-LEMN, Figure 1a).

Original: The usage of each memory representation m i is computed as the learned similarity between it and the current input x t .
Revised: The usage of each memory representation m i is computed as the learned similarity between it and the current input x t .

Original: As in G¨ulc¸ehre et al.
Revised: As in G¨ulc¸ehre et al.

Original: (2016), we least recently used (LRU) addressing by computing the exponential moving average v t of the logit z t , the LRU factor γ t , and the policy as follows:
Revised: (2016), we least recently used (LRU) addressing by computing the exponential moving average v t of the logit z t , the LRU factor γ t , and the policy as follows:

Original: [Equation]
Revised: [Equation]

Original: where W γ ∈ R d , b γ ∈ R , softmax ( z i ) is exp ( z i ) / (cid:80) j exp ( z j ) , σ ( z ) is 1 / (1 + exp ( z )) .
Revised: where W γ ∈ R d , b γ ∈ R , softmax ( z i ) is exp ( z i ) / (cid:80) j exp ( z j ) , σ ( z ) is 1 / (1 + exp ( z )) .

Original: This model estimates the policy based on the similarities between input x t and memory cells m t,i .
Revised: This model estimates the policy based on the similarities between input x t and memory cells m t,i .

Original: Spatial LEMN
Revised: Spatial LEMN

Original: A major drawback of IM-LEMN is that the score of each memory depends only on the input x t .
Revised: A major drawback of IM-LEMN is that the score of each memory depends only on the input x t .

Original: In other words, the score is computed independently between memory cell and input but does not consider its relative importance to other data instances in the memory.
Revised: In other words, the score is computed independently between memory cell and input but does not consider its relative importance to other data instances in the memory.

Original: To overcome this limitation, we propose Spatial LEMN (S-LEMN, Figure 1b) which computes the relative importance of memory cells to its neighbors and other memory cells using a bidirectional GRU as follows:
Revised: To overcome this limitation, we propose Spatial LEMN (S-LEMN, Figure 1b) which computes the relative importance of memory cells to its neighbors and other memory cells using a bidirectional GRU as follows:

Original: [Equation]
Revised: [Equation]

Original: where W f ∈ R 2 d × d , b f ∈ R d , GRU θ is a Gated Recurrent Unit parameterized by θ , [ −→ e t,i , ←− e t,i ] is a concatenation of features, ReLU is a rectiﬁed linear unit.
Revised: where W f ∈ R 2 d × d , b f ∈ R d , GRU θ is a Gated Recurrent Unit parameterized by θ , [ −→ e t,i , ←− e t,i ] is a concatenation of features, ReLU is a rectiﬁed linear unit.

Original: We use multi-layer perceptron (MLP) with scalar output to estimate the policy as follows:
Revised: We use multi-layer perceptron (MLP) with scalar output to estimate the policy as follows:

Original: [Equation]
Revised: [Equation]

Original: where W h ∈ R d × d/ 4 , b h ∈ R d/ 4 , W g ∈ R d/ 4 , b g ∈ R .
Revised: where W h ∈ R d × d/ 4 , b h ∈ R d/ 4 , W g ∈ R d/ 4 , b g ∈ R .

Original: Thus, it can compute the general importance of memory cell itself and the relation between its neighbors in contrast to IM-LEMN.
Revised: Thus, it can compute the general importance of memory cell itself and the relation between its neighbors in contrast to IM-LEMN.

Original: Spatio-Temporal LEMN
Revised: Spatio-Temporal LEMN

Original: In episodic tasks, the importance of memory also changes over time and tasks.
Revised: In episodic tasks, the importance of memory also changes over time and tasks.

Original: Hence, we propose Spatio-Temporal LEMN (ST-LEMN, Figure 1c) that uses a GRU over time to consider the historical importance as well as relative importance of each memory entry.
Revised: Hence, we propose Spatio-Temporal LEMN (ST-LEMN, Figure 1c) that uses a GRU over time to consider the historical importance as well as relative importance of each memory entry.

Original: We simply replace the hidden layer in Equation (1) with a GRU over time as follows:
Revised: We simply replace the hidden layer in Equation (1) with a GRU over time as follows:

Original: [Equation]
Revised: [Equation]

Original: Memory Update Using aforementioned memory-retention mechanism, the agent samples the memory cell index i from the probability distribution π ( m i | M t , x t ) .
Revised: Memory Update Using aforementioned memory-retention mechanism, the agent samples the memory cell index i from the probability distribution π ( m i | M t , x t ) .

Original: Then it erases the i th memory cell and appends the input x t as follows:
Revised: Then it erases the i th memory cell and appends the input x t as follows:

Original: [Equation]
Revised: [Equation]

Original: No Operation (NOP) As done in G¨ulc¸ehre et al.
Revised: No Operation (NOP) As done in G¨ulc¸ehre et al.

Original: (2016), for IM-LEMN we add a NOP memory cell at the end of memory to consider the case where the input is not written to the memory.
Revised: (2016), for IM-LEMN we add a NOP memory cell at the end of memory to consider the case where the input is not written to the memory.

Original: For SLEMN and ST-LEMN, we append x t to the place of NOP such that it could be selected for removal.
Revised: For SLEMN and ST-LEMN, we append x t to the place of NOP such that it could be selected for removal.

Original: We integrate these retention mechanisms into base memory-augmented neural networks to enable to efﬁciently maintain its external memory.
Revised: We integrate these retention mechanisms into base memory-augmented neural networks to enable to efﬁciently maintain its external memory.

Original: The details of the base networks are given in the Experiment section.
Revised: The details of the base networks are given in the Experiment section.

Original: We use Asynchronous Advantage Actor-Critic (A3C, Mnih et al.
Revised: We use Asynchronous Advantage Actor-Critic (A3C, Mnih et al.

Original: (2016)) with
Revised: (2016)) with

Original: Generalized Advantage Estimation (GAE, Schulman et al. (2015)) to train all models.
Revised: Generalized Advantage Estimation (GAE, Schulman et al. (2015)) to train all models.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We evaluate the proposed retention agent on three different tasks in following subsections
Revised: We evaluate the proposed retention agent on three different tasks in following subsections

Original: . 4. M AZE
Revised: . 4. M AZE

Original: Oh et al.
Revised: Oh et al.

Original: (2016) proposed a task for memory-based deep reinforcement learning (RL) agents, where the agent should navigate through a 3D maze and enter the correct exit, and demonstrated that the agents with external memory outperform the agents without external memory.
Revised: (2016) proposed a task for memory-based deep reinforcement learning (RL) agents, where the agent should navigate through a 3D maze and enter the correct exit, and demonstrated that the agents with external memory outperform the agents without external memory.

Original: However, they assumed that the agent had a sufﬁciently large memory to store every observed cell, although the actual amount of information was not as much as the size of the memory.
Revised: However, they assumed that the agent had a sufﬁciently large memory to store every observed cell, although the actual amount of information was not as much as the size of the memory.

Original: In this experiment, we compare the navigator agent with and without ST-LEMN.
Revised: In this experiment, we compare the navigator agent with and without ST-LEMN.

Original: We followed the same experimental setup as used in Oh et al.
Revised: We followed the same experimental setup as used in Oh et al.

Original: except that we use A3C algorithm (Mnih et al.
Revised: except that we use A3C algorithm (Mnih et al.

Original: with GAE (Schulman et al.
Revised: with GAE (Schulman et al.

Original: (2015)) instead of Q-Learning algorithm
Revised: (2015)) instead of Q-Learning algorithm

Original: (Mnih et al. (2015)) to train the agent, and used only three actions - Move forward, Look Left, Look Right - for expedited training.
Revised: (Mnih et al. (2015)) to train the agent, and used only three actions - Move forward, Look Left, Look Right - for expedited training.

Original: We use
Revised: We use

Original: MQN and FRMQN as base networks to see the effect of the model without recurrent connections and with recurrent connections.
Revised: MQN and FRMQN as base networks to see the effect of the model without recurrent connections and with recurrent connections.

Original: We compared the performance of these base models with learned retention and FIFO scheduling under two different environments.
Revised: We compared the performance of these base models with learned retention and FIFO scheduling under two different environments.

Original: [Figure]
Revised: [Figure]

Original: IMaze This environment contains an I-shaped maze, where the agent should reach the correct exit based on the color of the initial cell (See Figure 2a).
Revised: IMaze This environment contains an I-shaped maze, where the agent should reach the correct exit based on the color of the initial cell (See Figure 2a).

Original: We train the models on mazes with varying lengths of corridors N = { 5 , 7 , 9 } , and validate on mazes with corridors with 15 different lengths
Revised: We train the models on mazes with varying lengths of corridors N = { 5 , 7 , 9 } , and validate on mazes with corridors with 15 different lengths

Original: N = { 4 , 6 , 8 , 10 , 15 , 20 , 25 , 30 , 35 , 40 , 100 , 200 } .
Revised: N = { 4 , 6 , 8 , 10 , 15 , 20 , 25 , 30 , 35 , 40 , 100 , 200 } .

Original: We set the memory size to 4 for all compared models.
Revised: We set the memory size to 4 for all compared models.

Original: We observe that the agent with ST-LEMN successfully retains the indicator information in its memory while passing through the long corridor by removing useless frames that describe the corridor (Figure 3a and 3b).
Revised: We observe that the agent with ST-LEMN successfully retains the indicator information in its memory while passing through the long corridor by removing useless frames that describe the corridor (Figure 3a and 3b).

Original: At the end of corridor, agent with ST-LEMN retrieves stored indicator frame to decide which way to go.
Revised: At the end of corridor, agent with ST-LEMN retrieves stored indicator frame to decide which way to go.

Original: Agents with FIFO scheduling do not retain the indicator information and always exits at the same goal.
Revised: Agents with FIFO scheduling do not retain the indicator information and always exits at the same goal.

Original: FRMQN, which is an agent that has recurrent connection between time in its context embedding can complete task correctly on mazes with short corridor even with FIFO scheduling.
Revised: FRMQN, which is an agent that has recurrent connection between time in its context embedding can complete task correctly on mazes with short corridor even with FIFO scheduling.

Original: Yet, it fails on mazes with long corridors since it is difﬁcult to learn long-term dependencies without explicitly storing the cell in the memory.
Revised: Yet, it fails on mazes with long corridors since it is difﬁcult to learn long-term dependencies without explicitly storing the cell in the memory.

Original: In contrast, we observed that the agents with ST-LEMN can solve the maze with small ﬁxed-sized memories regardless of the length of maze, by learning the long-term importance of input data instances (Figure 2c).
Revised: In contrast, we observed that the agents with ST-LEMN can solve the maze with small ﬁxed-sized memories regardless of the length of maze, by learning the long-term importance of input data instances (Figure 2c).

Original: Random Maze: Single Goal with Indicator We also test with the randomly generated maze as in Oh et al.
Revised: Random Maze: Single Goal with Indicator We also test with the randomly generated maze as in Oh et al.

Original: (Figure 2b), testing for the Single Goal with Indicator (SingInd) task, where the agent should reach the correct goal based on the indicator that can be observed at the starting
Revised: (Figure 2b), testing for the Single Goal with Indicator (SingInd) task, where the agent should reach the correct goal based on the indicator that can be observed at the starting

Original: [Figure]
Revised: [Figure]

Original: position.
Revised: position.

Original: As shown in Table 1 the retention agent has signiﬁcantly higher success rate compared to the agent with FIFO scheduling, as it retains the indicator frame while it navigates through the random maze.
Revised: As shown in Table 1 the retention agent has signiﬁcantly higher success rate compared to the agent with FIFO scheduling, as it retains the indicator frame while it navigates through the random maze.

Original: As shown in 3c, 3d the retention agent retains the indicator discovered at the start of maze in its memory and retrieves its information when it needs to make a decision
Revised: As shown in 3c, 3d the retention agent retains the indicator discovered at the start of maze in its memory and retrieves its information when it needs to make a decision

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. S YNTHETIC E PISODIC Q UESTION A NSWERING
Revised: 4. S YNTHETIC E PISODIC Q UESTION A NSWERING

Original: Dataset Weston et al.
Revised: Dataset Weston et al.

Original: (2015) created a synthetic dataset for episodic question answering, called bAbI, that consists of 20 tasks.
Revised: (2015) created a synthetic dataset for episodic question answering, called bAbI, that consists of 20 tasks.

Original: Among tasks, we evaluate memory-retention mechanisms on two supporting facts task (task 2, Figure 4a).
Revised: Among tasks, we evaluate memory-retention mechanisms on two supporting facts task (task 2, Figure 4a).

Original: Additionally, We generate noisy and large version of the two supporting facts task from open-sourced template Weston et al.
Revised: Additionally, We generate noisy and large version of the two supporting facts task from open-sourced template Weston et al.

Original: Each task equally has ﬁve questions in an episode, where all questions share context sentences given in the episode.
Revised: Each task equally has ﬁve questions in an episode, where all questions share context sentences given in the episode.

Original: In the noisy two supporting fact task, which we refer to as Noisy task (Figure 4b), we inject noise sentences into original two supporting facts task to investigate the ability of retention mechanism to ﬁlter out the noise.
Revised: In the noisy two supporting fact task, which we refer to as Noisy task (Figure 4b), we inject noise sentences into original two supporting facts task to investigate the ability of retention mechanism to ﬁlter out the noise.

Original: We organize this synthetic dataset as follows: 60% of dataset have no noise sentence; 10% of dataset have approximately 30% noise sentences; 10% of dataset have approximately 45% noise sentences; 10% of dataset have approximately 60% noise sentences.
Revised: We organize this synthetic dataset as follows: 60% of dataset have no noise sentence; 10% of dataset have approximately 30% noise sentences; 10% of dataset have approximately 45% noise sentences; 10% of dataset have approximately 60% noise sentences.

Original: Totally, the length of each episode is ﬁxed to 45 (5 questions + 40 facts).
Revised: Totally, the length of each episode is ﬁxed to 45 (5 questions + 40 facts).

Original: Questions are placed after every 8 facts.
Revised: Questions are placed after every 8 facts.

Original: In large and noisy two supporting facts task, which is called Large task (Figure 4c), composition of tasks is similar to Noisy
Revised: In large and noisy two supporting facts task, which is called Large task (Figure 4c), composition of tasks is similar to Noisy

Original: but the length of episode and the position of questions vary.
Revised: but the length of episode and the position of questions vary.

Original: The length of each episode is randomly chosen between 20 and 80.
Revised: The length of each episode is randomly chosen between 20 and 80.

Original: Questions can be placed anywhere in episode.
Revised: Questions can be placed anywhere in episode.

Original: [Figure]
Revised: [Figure]

Original: Training Detail
Revised: Training Detail

Original: We use MemN2N(Sukhbaatar et al.
Revised: We use MemN2N(Sukhbaatar et al.

Original: as base networks.
Revised: as base networks.

Original: Particularly, we use base MemN2N with position encoding representation, 3 hops and adjacent weight tying.
Revised: Particularly, we use base MemN2N with position encoding representation, 3 hops and adjacent weight tying.

Original: We set the dimension of memory representations to d = 20 .
Revised: We set the dimension of memory representations to d = 20 .

Original: We compare FIFO, IM-LEMN, S-LEMN and
Revised: We compare FIFO, IM-LEMN, S-LEMN and

Original: ST-LEMN on the two supporting facts task, and two modiﬁed tasks.
Revised: ST-LEMN on the two supporting facts task, and two modiﬁed tasks.

Original: IM-LEMN on this task uses an average of attention logits from each hop as z t in Equation (3.1).
Revised: IM-LEMN on this task uses an average of attention logits from each hop as z t in Equation (3.1).

Original: Also, it embeds input using GRU similar to a GRU controller in G¨ulc¸ehre et al.
Revised: Also, it embeds input using GRU similar to a GRU controller in G¨ulc¸ehre et al.

Original: S-LEMN and ST-LEMN use the value memory of the ﬁrst hop in the base MemN2N as a memory representation e t,i in Equation (3.1).
Revised: S-LEMN and ST-LEMN use the value memory of the ﬁrst hop in the base MemN2N as a memory representation e t,i in Equation (3.1).

Original: We constrain the size of memory as 5 or 10 to validate the scheduling performance of our retention mechanism.
Revised: We constrain the size of memory as 5 or 10 to validate the scheduling performance of our retention mechanism.

Original: Since we jointly train the agent for QA task and memory retention, for stable training we pretrain
Revised: Since we jointly train the agent for QA task and memory retention, for stable training we pretrain

Original: MemN2N with FIFO mechanism for 50k steps and then go on with training other mechanisms.
Revised: MemN2N with FIFO mechanism for 50k steps and then go on with training other mechanisms.

Original: We train our models using ADAM optimizer (Kingma & Ba (2014)) with the learning rate of 0.001 for 200k steps on the Original and Noisy tasks, and for 400k steps on the Large task.
Revised: We train our models using ADAM optimizer (Kingma & Ba (2014)) with the learning rate of 0.001 for 200k steps on the Original and Noisy tasks, and for 400k steps on the Large task.

Original: Results Table 2 shows the results on synthetic episodic question answering tasks.
Revised: Results Table 2 shows the results on synthetic episodic question answering tasks.

Original: Sukhbaatar et al.
Revised: Sukhbaatar et al.

Original: (2015) reports the lowest error by multiple agents to cope with large variance in performance, and we follow this evaluation measure.
Revised: (2015) reports the lowest error by multiple agents to cope with large variance in performance, and we follow this evaluation measure.

Original: For the Original and Noisy tasks, we measure error rate using best performing parameter among three takes of training, and among ﬁve takes for the Large task.
Revised: For the Original and Noisy tasks, we measure error rate using best performing parameter among three takes of training, and among ﬁve takes for the Large task.

Original: As shown in table 2, our suggested memory retention mechanism signiﬁcantly outperforms naive policy and policy inspired by LRU in all cases.
Revised: As shown in table 2, our suggested memory retention mechanism signiﬁcantly outperforms naive policy and policy inspired by LRU in all cases.

Original: For more detailed analysis, we present two sampled data from IM-LEMN and ST-LEMN in Figure 5.
Revised: For more detailed analysis, we present two sampled data from IM-LEMN and ST-LEMN in Figure 5.

Original: IM-LEMN’s selection policy seems to largely depend on the current context, and therefore it deletes not only noise sentences but also informative sentences as well.
Revised: IM-LEMN’s selection policy seems to largely depend on the current context, and therefore it deletes not only noise sentences but also informative sentences as well.

Original: Compared to IM-LEMN, memory of ST-LEMN does not have any noise sentences in its memory but evenly contains informative facts about location and object acquisition.
Revised: Compared to IM-LEMN, memory of ST-LEMN does not have any noise sentences in its memory but evenly contains informative facts about location and object acquisition.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4. T RIVIA QA
Revised: 4. T RIVIA QA

Original: Dataset TriviaQA(Joshi
Revised: Dataset TriviaQA(Joshi

Original: et al., 2017) is a question-answering dataset over 950K question-answer pairs in 662K evidence documents collected manually.
Revised: et al., 2017) is a question-answering dataset over 950K question-answer pairs in 662K evidence documents collected manually.

Original: The problem is difﬁcult to solve by conventional models as it requires multiple reasoning with large number of sentences.
Revised: The problem is difﬁcult to solve by conventional models as it requires multiple reasoning with large number of sentences.

Original: The average number of words per document is 2895, which is infeasible to handle using existing reading comprehension models.
Revised: The average number of words per document is 2895, which is infeasible to handle using existing reading comprehension models.

Original: We limit the number of words per document to 800 and use only questions that could be spanned within 400 words out of the 800 for training for expedited training.
Revised: We limit the number of words per document to 800 and use only questions that could be spanned within 400 words out of the 800 for training for expedited training.

Original: Also, we only use the ﬁrst spanned word as labels since TriviaQA provides only the answer word and not their correct indices in the document.
Revised: Also, we only use the ﬁrst spanned word as labels since TriviaQA provides only the answer word and not their correct indices in the document.

Original: For a test set, we use all words in a document for each question-answer pair.
Revised: For a test set, we use all words in a document for each question-answer pair.

Original: Since the dataset does not provide labels for the test set, we use a validation set for test which contains both distant supervision set whose labels are collected automatically and veriﬁed set evaluated by the annotator.
Revised: Since the dataset does not provide labels for the test set, we use a validation set for test which contains both distant supervision set whose labels are collected automatically and veriﬁed set evaluated by the annotator.

Original: We evaluate our work only on the Wikipedia dataset, since previous work (Joshi et al., 2017), (Pan et al., 2017), (Yu et al., 2018) report similar results on both datasets.
Revised: We evaluate our work only on the Wikipedia dataset, since previous work (Joshi et al., 2017), (Pan et al., 2017), (Yu et al., 2018) report similar results on both datasets.

Original: [Figure]
Revised: [Figure]

Original: Base network As base network, we use BiDAF(Seo et al., 2016), which is one of the state-ofthe-art reading comprehension model that predicts the indices for the exact location of the answer in the given document and modiﬁed the word-level intermediate representations to sentence-level representations using RNN to handle an entire sentence at a time.
Revised: Base network As base network, we use BiDAF(Seo et al., 2016), which is one of the state-ofthe-art reading comprehension model that predicts the indices for the exact location of the answer in the given document and modiﬁed the word-level intermediate representations to sentence-level representations using RNN to handle an entire sentence at a time.

Original: We set the memory size N = 10 , and the word length per memory slot to 20.
Revised: We set the memory size N = 10 , and the word length per memory slot to 20.

Original: We trained our memory-augmented BiDAF using
Revised: We trained our memory-augmented BiDAF using

Original: ADAM optimizer (Kingma & Ba, 2014) with the initial learning rate of 0.0001.
Revised: ADAM optimizer (Kingma & Ba, 2014) with the initial learning rate of 0.0001.

Original: Results Table 3 shows the results on TriviaQA dataset.
Revised: Results Table 3 shows the results on TriviaQA dataset.

Original: In (Joshi et al., 2017), they selected the highest score for ExactMatch and F1 score among multiple documents that could ﬁnd the answer to a question.
Revised: In (Joshi et al., 2017), they selected the highest score for ExactMatch and F1 score among multiple documents that could ﬁnd the answer to a question.

Original: Our model outperforms all baselines because it has the ability to correctly identify informative sentences that are required to answer a given question.
Revised: Our model outperforms all baselines because it has the ability to correctly identify informative sentences that are required to answer a given question.

Original: As shown in Figure 6, when new context information arrives at the model, our model determines which memory information is the most unnecessary based on the predicted the retention value.
Revised: As shown in Figure 6, when new context information arrives at the model, our model determines which memory information is the most unnecessary based on the predicted the retention value.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We considered the problem of learning from streaming data, where the size of the data is too large to ﬁt into the memory of a memory-augmented network.
Revised: We considered the problem of learning from streaming data, where the size of the data is too large to ﬁt into the memory of a memory-augmented network.

Original: To solve the problem of retaining important data instances, we proposed Long-term Episodic Memory Network (LEMN), which is able to remember data instances of long-term generic importance.
Revised: To solve the problem of retaining important data instances, we proposed Long-term Episodic Memory Network (LEMN), which is able to remember data instances of long-term generic importance.

Original: Using reinforcement learning, LEMN learns to decide which memory entry to replace when the memory becomes full, based on both relative importance between memory entries and their historical importance.
Revised: Using reinforcement learning, LEMN learns to decide which memory entry to replace when the memory becomes full, based on both relative importance between memory entries and their historical importance.

Original: We validated our LEMN on three different tasks, namely path ﬁnding, episodic question answering and long question answering against rule-based memory scheduling methods as well as an RL-agent trained without consideration of relative and historic importance of memory entries, against which it signiﬁcantly outperforms.
Revised: We validated our LEMN on three different tasks, namely path ﬁnding, episodic question answering and long question answering against rule-based memory scheduling methods as well as an RL-agent trained without consideration of relative and historic importance of memory entries, against which it signiﬁcantly outperforms.

Original: Further analysis of LEMN shows that such good performance comes from its ability to retain instances of long-term importance.
Revised: Further analysis of LEMN shows that such good performance comes from its ability to retain instances of long-term importance.

Original: As future work, we plan to apply our model to dialogue generation task for conversational agents.
Revised: As future work, we plan to apply our model to dialogue generation task for conversational agents.

Revised: A S HUFFLING M EMORY E NTRIES

Revised: We evaluate the effect of perturbing a sequential order of memory cells on an episodic question answering dataset.

Revised: We perturb the sequential order of the memory of our models (S-LEMN and

Revised: ST-LEMN) when estimating the policy π ( a t

Revised: | s t ) in training phase.

Revised: We compare the models with the memory shufﬂed (Shufﬂed S-LEMN and Shufﬂed ST-LEMN) and ones with the sequential order preserved (S-LEMN and ST-LEMN) in Table 4.

Revised: We observe that perturbing a sequential order results in the performance degeneration.

Revised: This result shows that our models learn the relation between ordered memory cells to perform the episodic task where a sequential order is obviously important.

Revised: [Table]