Original: N EURAL M AP : S TRUCTURED M EMORY FOR D EEP R EINFORCEMENT L EARNING
Revised: N EURAL M AP : S TRUCTURED M EMORY FOR D EEP R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A critical component to enabling intelligent reasoning in partially observable environments is memory.
Revised: A critical component to enabling intelligent reasoning in partially observable environments is memory.

Original: Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past k frames or an LSTM layer.
Revised: Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past k frames or an LSTM layer.

Original: More recent work (Oh et al., 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past k frames.
Revised: More recent work (Oh et al., 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past k frames.

Original: But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last k frames.
Revised: But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last k frames.

Original: In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with.
Revised: In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with.

Original: This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags.
Revised: This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags.

Original: We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to environments that were not seen during training.
Revised: We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to environments that were not seen during training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Memory is a crucial aspect of an intelligent agent’s ability to plan and reason in partially observable environments.
Revised: Memory is a crucial aspect of an intelligent agent’s ability to plan and reason in partially observable environments.

Original: Without memory, agents must act reﬂexively according only to their immediate percepts and cannot execute plans that occur over an extended time interval.
Revised: Without memory, agents must act reﬂexively according only to their immediate percepts and cannot execute plans that occur over an extended time interval.

Original: Recently, Deep Reinforcement Learning agents have been capable of solving many challenging tasks such as Atari Arcade
Revised: Recently, Deep Reinforcement Learning agents have been capable of solving many challenging tasks such as Atari Arcade

Original: Games (Mnih et al., 2015), robot control (Levine et al., 2016) and 3D games such as Doom (Lample & Chaplot, 2016), but successful behaviours in these tasks have often only been based on a relatively short-term temporal context or even just a single frame.
Revised: Games (Mnih et al., 2015), robot control (Levine et al., 2016) and 3D games such as Doom (Lample & Chaplot, 2016), but successful behaviours in these tasks have often only been based on a relatively short-term temporal context or even just a single frame.

Original: On the other hand, many tasks require long-term planning, such as a robot gathering objects or an agent searching a level to ﬁnd a key in a role-playing game.
Revised: On the other hand, many tasks require long-term planning, such as a robot gathering objects or an agent searching a level to ﬁnd a key in a role-playing game.

Original: Neural networks that utilized external memories have recently had an explosion in variety, which can be distinguished along two main axes: memories with write operators and those without.
Revised: Neural networks that utilized external memories have recently had an explosion in variety, which can be distinguished along two main axes: memories with write operators and those without.

Original: Writeless external memory systems, often referred to as “Memory Networks” (Sukhbaatar et al., 2015;
Revised: Writeless external memory systems, often referred to as “Memory Networks” (Sukhbaatar et al., 2015;

Original: Oh et al., 2016), typically ﬁx which memories are stored.
Revised: Oh et al., 2016), typically ﬁx which memories are stored.

Original: For example, at each time step, the memory network would store the past M states seen in an environment.
Revised: For example, at each time step, the memory network would store the past M states seen in an environment.

Original: What is learnt by the network is therefore how to access or read from this ﬁxed memory pool, rather than what contents to store within it.
Revised: What is learnt by the network is therefore how to access or read from this ﬁxed memory pool, rather than what contents to store within it.

Original: The memory network approach has been successful in language modeling, question answering (Sukhbaatar et al., 2015) and was shown to be a sucessful memory for deep reinforcement learning agents in complex 3D environments (Oh et al., 2016).
Revised: The memory network approach has been successful in language modeling, question answering (Sukhbaatar et al., 2015) and was shown to be a sucessful memory for deep reinforcement learning agents in complex 3D environments (Oh et al., 2016).

Original: By side-steping the difﬁculty involved in learning what information is salient enough to store in memory, the memory network introduces two main disadvantages.
Revised: By side-steping the difﬁculty involved in learning what information is salient enough to store in memory, the memory network introduces two main disadvantages.

Original: The ﬁrst disadvantage is that a potentially signiﬁcant amount of redundant information could be stored.
Revised: The ﬁrst disadvantage is that a potentially signiﬁcant amount of redundant information could be stored.

Original: The second disadvantage is that a domain expert must choose what to store in the memory, e.g. for the DRL agent, the expert must set M to a value that is larger than the time horizon of the currently considered task.
Revised: The second disadvantage is that a domain expert must choose what to store in the memory, e.g. for the DRL agent, the expert must set M to a value that is larger than the time horizon of the currently considered task.

Original: On the other hand, external neural memories having write operations are potentially far more efﬁcient, since they can learn to store salient information for unbounded time steps and ignore any other useless information, without explicitly needing any a priori knowledge on what to store.
Revised: On the other hand, external neural memories having write operations are potentially far more efﬁcient, since they can learn to store salient information for unbounded time steps and ignore any other useless information, without explicitly needing any a priori knowledge on what to store.

Original: One prominent research direction within write-based architectures has been neural memories based on the types of memory structures that are found in computers, such as tapes, RAM, and GPUs.
Revised: One prominent research direction within write-based architectures has been neural memories based on the types of memory structures that are found in computers, such as tapes, RAM, and GPUs.

Original: In contrast to typical recurrent neural networks, these neural computer emulators have far more structured memories which follow many of the same design paradigms that digital computers have traditionally utilized.
Revised: In contrast to typical recurrent neural networks, these neural computer emulators have far more structured memories which follow many of the same design paradigms that digital computers have traditionally utilized.

Original: One such model, the Differentiable Neural Computer (DNC) (Graves et al., 2016) and its predecessor the Neural Turing Machine (NTM) (Graves et al., 2014), structure the architecture to explicitly separate memory from computation.
Revised: One such model, the Differentiable Neural Computer (DNC) (Graves et al., 2016) and its predecessor the Neural Turing Machine (NTM) (Graves et al., 2014), structure the architecture to explicitly separate memory from computation.

Original: The DNC has a recurrent neural controller that can access an external memory resource by executing differentiable read and write operations.
Revised: The DNC has a recurrent neural controller that can access an external memory resource by executing differentiable read and write operations.

Original: This allows the DNC to act and memorize in a structured manner resembling a computer processor, where read and write operations are sequential and data is store distinctly from computation.
Revised: This allows the DNC to act and memorize in a structured manner resembling a computer processor, where read and write operations are sequential and data is store distinctly from computation.

Original: The DNC has been used sucessfully to solve complicated algorithmic tasks, such as ﬁnding shortest paths in a graph or querying a database for entity relations.
Revised: The DNC has been used sucessfully to solve complicated algorithmic tasks, such as ﬁnding shortest paths in a graph or querying a database for entity relations.

Original: Building off these previous external memories, we introduce a new architecture called the Neural
Revised: Building off these previous external memories, we introduce a new architecture called the Neural

Original: Map, a structured memory designed speciﬁcally for reinforcement learning agents in 3D environments.
Revised: Map, a structured memory designed speciﬁcally for reinforcement learning agents in 3D environments.

Original: The Neural Map architecture overcomes some of the shortcomings of the previously mentioned neural memories.
Revised: The Neural Map architecture overcomes some of the shortcomings of the previously mentioned neural memories.

Original: First, it uses an adaptable write operation and so its size and computational cost does not grow with the time horizon of the environment as it does with memory networks.
Revised: First, it uses an adaptable write operation and so its size and computational cost does not grow with the time horizon of the environment as it does with memory networks.

Original: Second, we impose a particular inductive bias on the write operation so that it is 1) well suited to 3D environments where navigation is a core component of sucessful behaviours, and 2) uses a sparse write operation that prevents frequent overwriting of memory locations that can occur with
Revised: Second, we impose a particular inductive bias on the write operation so that it is 1) well suited to 3D environments where navigation is a core component of sucessful behaviours, and 2) uses a sparse write operation that prevents frequent overwriting of memory locations that can occur with

Original: NTMs and DNCs.
Revised: NTMs and DNCs.

Original: To accomplish this, we structure a DNC-style external memory in the form of a 2-dimensional map, where each position in the map is a distinct memory.
Revised: To accomplish this, we structure a DNC-style external memory in the form of a 2-dimensional map, where each position in the map is a distinct memory.

Original: To demonstrate the effectiveness of the neural map, we run it on a variety of 2D partially-observable maze-based environments and test it against LSTM and memory network policies.
Revised: To demonstrate the effectiveness of the neural map, we run it on a variety of 2D partially-observable maze-based environments and test it against LSTM and memory network policies.

Original: Finally, to establish its scalability, we run a Neural Map agent on a set of challenging 3D maze environments based on the video game Doom.
Revised: Finally, to establish its scalability, we run a Neural Map agent on a set of challenging 3D maze environments based on the video game Doom.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: A Markov Decision Process (MDP) is deﬁned as a tuple ( S , A , T , γ, R ) where S is a ﬁnite set of states, A is a ﬁnite set of actions, T ( s (cid:48) | s, a ) is the transition probability of arriving in state s (cid:48)
Revised: A Markov Decision Process (MDP) is deﬁned as a tuple ( S , A , T , γ, R ) where S is a ﬁnite set of states, A is a ﬁnite set of actions, T ( s (cid:48) | s, a ) is the transition probability of arriving in state s (cid:48)

Original: when executing action a in initial state s , γ is a discount factor, and R ( s, a, s (cid:48) ) is the reward function of executing action a in state s and ending up at state s (cid:48) .
Revised: when executing action a in initial state s , γ is a discount factor, and R ( s, a, s (cid:48) ) is the reward function of executing action a in state s and ending up at state s (cid:48) .

Original: We deﬁne a policy π ( ·| s ) as a mapping from a state s to a distribution over actions, where π ( a i | s ) denotes the probability of action a i given that we are in state s .
Revised: We deﬁne a policy π ( ·| s ) as a mapping from a state s to a distribution over actions, where π ( a i | s ) denotes the probability of action a i given that we are in state s .

Original: The value of a policy V π ( s ) is the expected discounted cumulative reward when starting from state s and sampling actions according to π , i.e.: V π ( s ) =
Revised: The value of a policy V π ( s ) is the expected discounted cumulative reward when starting from state s and sampling actions according to π , i.e.: V π ( s ) =

Original: E π [ (cid:80) ∞ t =

Revised: E π [ (cid:80) ∞ t =

Original: γ t R t | s 

Revised: γ t R t | s

Original: = s ] .
Revised: = s ] .

Original: An optimal value function, denoted V ∗ ( s ) , is the maximum value we can get from state s according to any policy, i.e. V ∗ ( s ) =
Revised: An optimal value function, denoted V ∗ ( s ) , is the maximum value we can get from state s according to any policy, i.e. V ∗ ( s ) =

Original: max π V π ( s ) .
Revised: max π V π ( s ) .

Original: An optimal policy π ∗ is deﬁned as a policy which achieves optimal value at each state, i.e. V π ∗ ( s )
Revised: An optimal policy π ∗ is deﬁned as a policy which achieves optimal value at each state, i.e. V π ∗ ( s )

Original: =
Revised: =

Original: V ∗ ( s ) .
Revised: V ∗ ( s ) .

Original: An optimal policy is guaranteed to exist (Sutton &
Revised: An optimal policy is guaranteed to exist (Sutton &

Original: Barto, 1998).
Revised: Barto, 1998).

Original: The REINFORCE algorithm (Williams, 1992) iteratively updates a given policy π in the direction of the optimal policy.
Revised: The REINFORCE algorithm (Williams, 1992) iteratively updates a given policy π in the direction of the optimal policy.

Original: This update direction is deﬁned by ∇ π log π ( a t
Revised: This update direction is deﬁned by ∇ π log π ( a t

Original: | s t ) G t with G t = (cid:80) ∞ k =0 γ k R t + k being the future cumulated reward for a particular episode rollout.
Revised: | s t ) G t with G t = (cid:80) ∞ k =0 γ k R t + k being the future cumulated reward for a particular episode rollout.

Original: The variance of this update is typically high but can be reduced by using a “baseline” b t ( s t ) , which is a function of the current state.
Revised: The variance of this update is typically high but can be reduced by using a “baseline” b t ( s t ) , which is a function of the current state.

Original: Therefore the baseline-augmented update equation is ∇ π log π ( a t | s t )( G t − b t ( s t )) .
Revised: Therefore the baseline-augmented update equation is ∇ π log π ( a t | s t )( G t − b t ( s t )) .

Original: The typically used baseline is the value function, b t ( s t ) =
Revised: The typically used baseline is the value function, b t ( s t ) =

Original: V π ( s t ) .
Revised: V π ( s t ) .

Original: This combination of
Revised: This combination of

Original: REINFORCE with value function baseline is commonly termed the “Actor-Critic” algorithm.
Revised: REINFORCE with value function baseline is commonly termed the “Actor-Critic” algorithm.

Original: In this paper, we utilize Advantage Actor-Critic (A2C) (Mnih et al., 2016) with Generalized Advantage Estimation (Schulman et al., 2015), which can be seen as a specialization of the actor-critic framework when using deep networks to parameterize the policy and value function.
Revised: In this paper, we utilize Advantage Actor-Critic (A2C) (Mnih et al., 2016) with Generalized Advantage Estimation (Schulman et al., 2015), which can be seen as a specialization of the actor-critic framework when using deep networks to parameterize the policy and value function.

Original: The policy is a function of the state, parameterized as a deep neural network: π ( a | s ) =
Revised: The policy is a function of the state, parameterized as a deep neural network: π ( a | s ) =

Original: f θ ( s, a ) , where f is a deep neural network with parameter vector θ .
Revised: f θ ( s, a ) , where f is a deep neural network with parameter vector θ .

Original:  N EURAL M AP
Revised:  N EURAL M AP

Original: In this section, we will describe the details of the neural map.
Revised: In this section, we will describe the details of the neural map.

Original: We assume we want our agent to act within some 2- or 3-dimensional environment.
Revised: We assume we want our agent to act within some 2- or 3-dimensional environment.

Original: The neural map is the agent’s internal memory storage that can be read from and written to during interaction with its environment, but where the write operator is selectively limited to affect only the part of the neural map that represents the area where the agent is currently located.
Revised: The neural map is the agent’s internal memory storage that can be read from and written to during interaction with its environment, but where the write operator is selectively limited to affect only the part of the neural map that represents the area where the agent is currently located.

Original: For this paper, we assume for simplicity that we are dealing with a 2-dimensional map.
Revised: For this paper, we assume for simplicity that we are dealing with a 2-dimensional map.

Original: This can easily be extended to 3-dimensional or even higher-dimensional maps (i.e. a 4D map with a 3D sub-map for each cardinal direction the agent can face).
Revised: This can easily be extended to 3-dimensional or even higher-dimensional maps (i.e. a 4D map with a 3D sub-map for each cardinal direction the agent can face).

Original: Let the agent’s position be ( x, y ) with x ∈ R and y ∈ R and let the neural map M be a C × H × W feature block, where C is the feature dimension, H is the vertical extent of the map and W is the horizontal extent.
Revised: Let the agent’s position be ( x, y ) with x ∈ R and y ∈ R and let the neural map M be a C × H × W feature block, where C is the feature dimension, H is the vertical extent of the map and W is the horizontal extent.

Original: Assume there exists some coordinate normalization function ψ ( x, y ) such that every unique ( x, y ) can be mapped into ( x (cid:48) , y (cid:48) ) , where x (cid:48) ∈ { 0 , . . .
Revised: Assume there exists some coordinate normalization function ψ ( x, y ) such that every unique ( x, y ) can be mapped into ( x (cid:48) , y (cid:48) ) , where x (cid:48) ∈ { 0 , . . .

Original: , W − 1 } and y (cid:48) ∈ { 0 , . .
Revised: , W − 1 } and y (cid:48) ∈ { 0 , . .

Original: , H − 1 } .
Revised: , H − 1 } .

Original: For ease of notation, suppose in the sequel that all coordinates have been normalized by ψ into neural map space.
Revised: For ease of notation, suppose in the sequel that all coordinates have been normalized by ψ into neural map space.

Original: Let s t be the current state embedding, M t be the current neural map, and ( x t , y t ) be the current position of the agent.
Revised: Let s t be the current state embedding, M t be the current neural map, and ( x t , y t ) be the current position of the agent.

Original: The Neural Map is deﬁned by the following set of equations:
Revised: The Neural Map is deﬁned by the following set of equations:

Original: [Equation]
Revised: [Equation]

Original: read ( M t ) , c t = context ( M t , s t , r t ) ,
Revised: read ( M t ) , c t = context ( M t , s t , r t ) ,

Original: [Equation]
Revised: [Equation]

Original: = write ( s t , r t , c t , M ( x t
Revised: = write ( s t , r t , c t , M ( x t

Original: [Equation]
Revised: [Equation]

Original: t ) ,
Revised: t ) ,

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: where w ( x t ,y t ) t represents the feature at position ( x t , y t ) at time t , [ x 1 , . .
Revised: where w ( x t ,y t ) t represents the feature at position ( x t , y t ) at time t , [ x 1 , . .

Original: , x k ] represents a concatenation operation, and o t is the output of the neural map at time t which is then processed by another deep network f to get the policy outputs π t ( a | s ) .
Revised: , x k ] represents a concatenation operation, and o t is the output of the neural map at time t which is then processed by another deep network f to get the policy outputs π t ( a | s ) .

Original: We will now separately describe each of the above operations in more detail:
Revised: We will now separately describe each of the above operations in more detail:

Original: Global Read Operation: The read operation passes the current neural map M t through a deep convolutional network and produces a C -dimensional feature vector r t .
Revised: Global Read Operation: The read operation passes the current neural map M t through a deep convolutional network and produces a C -dimensional feature vector r t .

Original: The global read vector r t summarizes information about the entire map.
Revised: The global read vector r t summarizes information about the entire map.

Original: Context Read Operation:
Revised: Context Read Operation:

Original: The context operation performs context-based addressing to check whether certain features are stored in the map.
Revised: The context operation performs context-based addressing to check whether certain features are stored in the map.

Original: It takes as input the current state embedding s t and the current global read vector r t and ﬁrst produces a query vector q t .
Revised: It takes as input the current state embedding s t and the current global read vector r t and ﬁrst produces a query vector q t .

Original: The inner product of the query vector and each feature M ( x,y ) t in the neural map is then taken to get scores a ( x,y ) t at all positions ( x, y ) .
Revised: The inner product of the query vector and each feature M ( x,y ) t in the neural map is then taken to get scores a ( x,y ) t at all positions ( x, y ) .

Original: The scores are then normalized to get a probability distribution α ( x,y ) t over every position in the map, also known as “soft attention” (Bahdanau et al., 2015).
Revised: The scores are then normalized to get a probability distribution α ( x,y ) t over every position in the map, also known as “soft attention” (Bahdanau et al., 2015).

Original: This probability distribution is used to compute a weighted average c t over all features M ( x,y ) t .
Revised: This probability distribution is used to compute a weighted average c t over all features M ( x,y ) t .

Original: To summarize:
Revised: To summarize:

Original: [Equation]
Revised: [Equation]

Original: where W is a weight matrix.
Revised: where W is a weight matrix.

Original: The context read operation allows the neural map to operate as an associative memory: the agent provides some possibly incomplete memory (the query vector q t ) and the operation will return the completed memory that most closely matches q t .
Revised: The context read operation allows the neural map to operate as an associative memory: the agent provides some possibly incomplete memory (the query vector q t ) and the operation will return the completed memory that most closely matches q t .

Original: So, for example, the agent can query whether it has seen something similar to a particular landmark that is currently within its view.
Revised: So, for example, the agent can query whether it has seen something similar to a particular landmark that is currently within its view.

Original: Local Write Operation:
Revised: Local Write Operation:

Original: Given the agent’s current position ( x t , y t ) at time t , the write operation takes as input the current state embedding s t , the global read output r t , the context read vector c t and the current feature at position ( x t , y t ) in the neural map M ( x t ,y t ) t and produces, using a deep neural network f w , a new C-dimensional vector w ( x t ,y t ) t + .
Revised: Given the agent’s current position ( x t , y t ) at time t , the write operation takes as input the current state embedding s t , the global read output r t , the context read vector c t and the current feature at position ( x t , y t ) in the neural map M ( x t ,y t ) t and produces, using a deep neural network f w , a new C-dimensional vector w ( x t ,y t ) t + .

Original: This vector functions as the new local write candidate vector at the current position ( x t , y t ) : w ( x t ,y t ) t + = f w ([ s t , r t , c t , M ( x t ,y t ) t ])
Revised: This vector functions as the new local write candidate vector at the current position ( x t , y t ) : w ( x t ,y t ) t + = f w ([ s t , r t , c t , M ( x t ,y t ) t ])

Original: GRU-based Local Write Operation As previously deﬁned, the write operation simply replaces the vector at the agent’s current position with a new feature produced by a deep network.
Revised: GRU-based Local Write Operation As previously deﬁned, the write operation simply replaces the vector at the agent’s current position with a new feature produced by a deep network.

Original: Instead of this hard rewrite of the current position’s feature vector, we can use a gated write operation based on the recurrent update equations of the Gated Recurrent Unit (GRU) (Chung et al., 2014).
Revised: Instead of this hard rewrite of the current position’s feature vector, we can use a gated write operation based on the recurrent update equations of the Gated Recurrent Unit (GRU) (Chung et al., 2014).

Original: Gated write operations have a long history in unstructured recurrent networks and they have shown a superior ability to maintain information over long time lags versus ungated networks.
Revised: Gated write operations have a long history in unstructured recurrent networks and they have shown a superior ability to maintain information over long time lags versus ungated networks.

Original: The GRU-based write operation is deﬁned as:
Revised: The GRU-based write operation is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where x (cid:12) y is the Hadamard product between vectors x and y , σ ( · ) is the sigmoid activation function and W ∗ and U ∗ are weight matrices.
Revised: where x (cid:12) y is the Hadamard product between vectors x and y , σ ( · ) is the sigmoid activation function and W ∗ and U ∗ are weight matrices.

Original: Using GRU terminology, r ( x t ,y t ) t
Revised: Using GRU terminology, r ( x t ,y t ) t

Original: + is the reset gate, ˆ w ( x t ,y t ) t
Revised: + is the reset gate, ˆ w ( x t ,y t ) t

Original: + is the candidate activation and z ( x t ,y t ) t
Revised: + is the candidate activation and z ( x t ,y t ) t

Original: + is the update gate.
Revised: + is the update gate.

Original: By making use of the reset and update gates, the GRU-based update can modulate how much the new write vector should differ from the currently stored feature.
Revised: By making use of the reset and update gates, the GRU-based update can modulate how much the new write vector should differ from the currently stored feature.

Original: Map Update Operation: The update operation creates the neural map for the next time step.
Revised: Map Update Operation: The update operation creates the neural map for the next time step.

Original: The new neural map M t +1 is equal to the old neural map M t , except at the current agent position ( x t , y t ) , where the current write candidate vector w ( x t ,y t ) t + is stored:
Revised: The new neural map M t +1 is equal to the old neural map M t , except at the current agent position ( x t , y t ) , where the current write candidate vector w ( x t ,y t ) t + is stored:

Original: [Equation]
Revised: [Equation]

Original: , for
Revised: , for

Original: [Equation]
Revised: [Equation]

Original: a, b
Revised: a, b

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original:  E GO - CENTRIC N EURAL M AP
Revised:  E GO - CENTRIC N EURAL M AP

Original: A major disadvantage of the neural map as previously described is that it requires some oracle to provide the current ( x, y ) position of the agent.
Revised: A major disadvantage of the neural map as previously described is that it requires some oracle to provide the current ( x, y ) position of the agent.

Original: This is a difﬁcult problem in and of itself, and, despite being well studied, it is far from solved.
Revised: This is a difﬁcult problem in and of itself, and, despite being well studied, it is far from solved.

Original: The alternative to using absolute positions within the map is to use relative positions.
Revised: The alternative to using absolute positions within the map is to use relative positions.

Original: That is, whenever the agent moves between time steps with some velocity ( u, v ) , the map is counter-transformed by ( − u, − v ) , i.e. each feature in the map is shifted in the H and W dimensions.
Revised: That is, whenever the agent moves between time steps with some velocity ( u, v ) , the map is counter-transformed by ( − u, − v ) , i.e. each feature in the map is shifted in the H and W dimensions.

Original: This will mean that the map will be ego-centric, i.e. the agent’s position will stay stationary in the center of the neural map while the world as deﬁned by the map moves around them.
Revised: This will mean that the map will be ego-centric, i.e. the agent’s position will stay stationary in the center of the neural map while the world as deﬁned by the map moves around them.

Original: Therefore in this setup we only need some way of extracting the agent’s velocity, which is typically a simpler task in real environments (for example, animals have inner ears and robots have accelerometers).
Revised: Therefore in this setup we only need some way of extracting the agent’s velocity, which is typically a simpler task in real environments (for example, animals have inner ears and robots have accelerometers).

Original: Here we assume that there is some function ξ ( u (cid:48) , v (cid:48) ) that discretizes the agent velocities ( u (cid:48) , v (cid:48) )
Revised: Here we assume that there is some function ξ ( u (cid:48) , v (cid:48) ) that discretizes the agent velocities ( u (cid:48) , v (cid:48) )

Original: so that they represent valid velocities within the neural map ( u, v ) .
Revised: so that they represent valid velocities within the neural map ( u, v ) .

Original: In the sequel, we assume that all velocies have been properly normalized by ξ into neural map space.
Revised: In the sequel, we assume that all velocies have been properly normalized by ξ into neural map space.

Original: Let ( pw, ph ) be the center position of the neural map.
Revised: Let ( pw, ph ) be the center position of the neural map.

Original: The updated ego-centric neural map operations are shown below:
Revised: The updated ego-centric neural map operations are shown below:

Original: [Equation]
Revised: [Equation]

Original: Where M t is the current neural map M t reverse transformed by the current velocity ( u t , v t ) so that the agents map position remains in the center ( pw, ph ) .
Revised: Where M t is the current neural map M t reverse transformed by the current velocity ( u t , v t ) so that the agents map position remains in the center ( pw, ph ) .

Original: Counter Transform Operation: The CounterTransform operation transforms the current neural map M t by the inverse of the agent’s current velocity ( u t , v t ) .
Revised: Counter Transform Operation: The CounterTransform operation transforms the current neural map M t by the inverse of the agent’s current velocity ( u t , v t ) .

Original: Written formally:
Revised: Written formally:

Original: [Equation]
Revised: [Equation]

Original: else
Revised: else

Original: [Equation]
Revised: [Equation]

Original: While here we only deal with reverse translation, it is possible to handle rotations as well if the agent can measure it’s angular velocity.
Revised: While here we only deal with reverse translation, it is possible to handle rotations as well if the agent can measure it’s angular velocity.

Original: [Figure]
Revised: [Figure]

Original: Map Egoupdate Operation:
Revised: Map Egoupdate Operation:

Original: The egoupdate operation is functionally equivalent to the update operation except only the center position ( pw, ph ) is ever written to:
Revised: The egoupdate operation is functionally equivalent to the update operation except only the center position ( pw, ph ) is ever written to:

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To demonstrate the effectiveness of the Neural Map, we run it on 2D and 3D maze-based environments where memory is crucial to optimal behaviour.
Revised: To demonstrate the effectiveness of the Neural Map, we run it on 2D and 3D maze-based environments where memory is crucial to optimal behaviour.

Original: We compare to previous memory-based DRL agents, namely a simple LSTM-based agent which consists of a single pre-output LSTM layer as well as MemNN (Oh et al., 2016) agents.
Revised: We compare to previous memory-based DRL agents, namely a simple LSTM-based agent which consists of a single pre-output LSTM layer as well as MemNN (Oh et al., 2016) agents.

Original: 5. 2D G OAL -S EARCH E NVIRONMENT
Revised: 5. 2D G OAL -S EARCH E NVIRONMENT

Original: The “Goal-Search” environment is adapted from Oh et al.
Revised: The “Goal-Search” environment is adapted from Oh et al.

Original: Here the agent starts in a ﬁxed starting position within some randomly generated maze with two randomly positioned goal states.
Revised: Here the agent starts in a ﬁxed starting position within some randomly generated maze with two randomly positioned goal states.

Original: It then observes an indicator at a ﬁxed position near the starting state (i.e. the green tile at the top of the maze in Fig. 1a).
Revised: It then observes an indicator at a ﬁxed position near the starting state (i.e. the green tile at the top of the maze in Fig. 1a).

Original: This indicator will tell the agent which of the two goals it needs to go to (blue indicator → teal goal, green indicator → red goal).
Revised: This indicator will tell the agent which of the two goals it needs to go to (blue indicator → teal goal, green indicator → red goal).

Original: If the agent goes to the correct goal, it gains a positive reward while if it goes to the incorrect goal it gains a negative reward.
Revised: If the agent goes to the correct goal, it gains a positive reward while if it goes to the incorrect goal it gains a negative reward.

Original: Therefore the agent needs to remember the indicator as it searches for the correct goal state.
Revised: Therefore the agent needs to remember the indicator as it searches for the correct goal state.

Original: In depth details of the 2D environment are given in Appendix B.
Revised: In depth details of the 2D environment are given in Appendix B.

Original: The mazes during training are generated using a random generator.
Revised: The mazes during training are generated using a random generator.

Original: A held-out set of 1000 random mazes is kept for testing.
Revised: A held-out set of 1000 random mazes is kept for testing.

Original: This test set therefore represents maze geometries that have never been seen during training, and measure the agent’s ability to generalize to new environments.
Revised: This test set therefore represents maze geometries that have never been seen during training, and measure the agent’s ability to generalize to new environments.

Original: The ﬁrst baseline agent we evaluate is a recurrent network with 128 LSTM units.
Revised: The ﬁrst baseline agent we evaluate is a recurrent network with 128 LSTM units.

Original: The other baseline is the MQN, which is a memory-network-based architecture that performs attention over the past
Revised: The other baseline is the MQN, which is a memory-network-based architecture that performs attention over the past

Original: K states it has seen (Oh et al., 2016).
Revised: K states it has seen (Oh et al., 2016).

Original: Both LSTM and MQN models receive a one-hot encoding of the agent’s current location, previous velocity, and current orientation at each time step, in order to make the comparison to the ﬁxed-frame Neural Map fair.
Revised: Both LSTM and MQN models receive a one-hot encoding of the agent’s current location, previous velocity, and current orientation at each time step, in order to make the comparison to the ﬁxed-frame Neural Map fair.

Original: We test these baselines against several
Revised: We test these baselines against several

Original: Neural Map architectures, with each architecture having a different design choice.
Revised: Neural Map architectures, with each architecture having a different design choice.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: The results are reported in Table 1.
Revised: The results are reported in Table 1.

Original: During testing, we extend the maximum episode length from 100 to 500 steps so that the agent is given more time to solve the maze.
Revised: During testing, we extend the maximum episode length from 100 to 500 steps so that the agent is given more time to solve the maze.

Original: The brackets next to the model name represent the Neural Map dimensions of that particular model.
Revised: The brackets next to the model name represent the Neural Map dimensions of that particular model.

Original: From the results we can see that the Neural Map architectures solve the most mazes in both the training and test distributions compared to both LSTM and MQN baselines.
Revised: From the results we can see that the Neural Map architectures solve the most mazes in both the training and test distributions compared to both LSTM and MQN baselines.

Original: The results also demonstrate the effect of certain design decisions.
Revised: The results also demonstrate the effect of certain design decisions.

Original: One thing that can be observed is that using GRU updates adds several percentage points to the success rate (“Neural Map (15x15)”
Revised: One thing that can be observed is that using GRU updates adds several percentage points to the success rate (“Neural Map (15x15)”

Original: v.s. “Neural Map + GRU (15x15)”).
Revised: v.s. “Neural Map + GRU (15x15)”).

Original: We also tried downsampled Neural Maps, such that a pixel in the memory map represents several discrete locations in the environment.
Revised: We also tried downsampled Neural Maps, such that a pixel in the memory map represents several discrete locations in the environment.

Original: The Neural Map seems quite robust to this downsampling, with a downsampling of around 3 (6x6 v.s. 15x15) doing just a few percentage points worse, and still beating all baseline models.
Revised: The Neural Map seems quite robust to this downsampling, with a downsampling of around 3 (6x6 v.s. 15x15) doing just a few percentage points worse, and still beating all baseline models.

Original: The 6x6 model has approximately the same number of memory cells as “MQN-32”, but its performance is much better, showing the beneﬁt of having learnable write operations.
Revised: The 6x6 model has approximately the same number of memory cells as “MQN-32”, but its performance is much better, showing the beneﬁt of having learnable write operations.

Original: For the egocentric model, in order to cover the entire map we set the pixels to be 2x smaller in each direction, so each pixel is only a quarter of a pixel in the ﬁxed-frame map.
Revised: For the egocentric model, in order to cover the entire map we set the pixels to be 2x smaller in each direction, so each pixel is only a quarter of a pixel in the ﬁxed-frame map.

Original: Even with this coarser representation, the egocentric model did similarly to the ﬁxed frame one.
Revised: Even with this coarser representation, the egocentric model did similarly to the ﬁxed frame one.

Original: We demonstrate an example of what the Neural Map learned to address using its context operator in Appendix E.
Revised: We demonstrate an example of what the Neural Map learned to address using its context operator in Appendix E.

Original: Finally, we tried adding the one-hot position encoding as a state input to the Neural Map, as is done for the baselines.
Revised: Finally, we tried adding the one-hot position encoding as a state input to the Neural Map, as is done for the baselines.

Original: We can see that there is a small improvement, but it is largely marginal, with the
Revised: We can see that there is a small improvement, but it is largely marginal, with the

Original: Neural Map doing a decent job of learning how to represent its own position without needing to be told explicitly.
Revised: Neural Map doing a decent job of learning how to represent its own position without needing to be told explicitly.

Original: One interesting thing that we observed is that having the one-hot position encoding as an input to the egocentric map decreased performance, perhaps because it is difﬁcult for the network to learn a mapping between ﬁxed and egocentric frames.
Revised: One interesting thing that we observed is that having the one-hot position encoding as an input to the egocentric map decreased performance, perhaps because it is difﬁcult for the network to learn a mapping between ﬁxed and egocentric frames.

Original: Note that sometimes the percentage results are lower for the training distribution.
Revised: Note that sometimes the percentage results are lower for the training distribution.

Original: This is mainly because the training set encompases almost all random mazes except the ﬁxed 1000 of the test set, thus making it likely that the agent sees each training map only once.
Revised: This is mainly because the training set encompases almost all random mazes except the ﬁxed 1000 of the test set, thus making it likely that the agent sees each training map only once.

Original: Beyond train/test splits, the results are further separated by maze size.
Revised: Beyond train/test splits, the results are further separated by maze size.

Original: This information reveals that the memory networks are hardest hit by increasing maze size with sometimes a 20% drop in success on 13-15 v.s. 7-11.
Revised: This information reveals that the memory networks are hardest hit by increasing maze size with sometimes a 20% drop in success on 13-15 v.s. 7-11.

Original: This is perhaps unsurprising given the inherent ﬁxed time horizon of memory netwoks, and further reveals the beneﬁt of using write-based memories.
Revised: This is perhaps unsurprising given the inherent ﬁxed time horizon of memory netwoks, and further reveals the beneﬁt of using write-based memories.

Original: 5. 3D D OOM E NVIRONMENT D ESCRIPTION
Revised: 5. 3D D OOM E NVIRONMENT D ESCRIPTION

Original: To demonstrate that our method can work in much more complicated 3D environments with longer time lags, we implemented three 3D maze environments using the ViZDoom (Kempka et al., 2016)
Revised: To demonstrate that our method can work in much more complicated 3D environments with longer time lags, we implemented three 3D maze environments using the ViZDoom (Kempka et al., 2016)

Original: API and a random maze generator.
Revised: API and a random maze generator.

Original: Examples of all three environments are given in Figure 2.
Revised: Examples of all three environments are given in Figure 2.

Original: [Figure]
Revised: [Figure]

Original: Indicator Maze: The ﬁrst environment is a recreation of the 2D indicator maze task, where an indicator is positioned in view of the player’s starting state which is either a torch of red or green color.
Revised: Indicator Maze: The ﬁrst environment is a recreation of the 2D indicator maze task, where an indicator is positioned in view of the player’s starting state which is either a torch of red or green color.

Original: The goals are corresponding red/green towers that are randomly positioned throughout the maze that the player must locate.
Revised: The goals are corresponding red/green towers that are randomly positioned throughout the maze that the player must locate.

Original: Repeating Maze: The second environment is a variant of this indicator maze but whenever the player enters a goal state, it is teleported back to the beginning of the maze without terminating the episode (i.e. it retains its memory of the current maze).
Revised: Repeating Maze: The second environment is a variant of this indicator maze but whenever the player enters a goal state, it is teleported back to the beginning of the maze without terminating the episode (i.e. it retains its memory of the current maze).

Original: It gains a positive reward if it reaches the correct goal and a negative reward if it reaches the incorrect goal.
Revised: It gains a positive reward if it reaches the correct goal and a negative reward if it reaches the incorrect goal.

Original: After the ﬁrst goal is reached, the correct indicator color is no longer displayed within the maze and a red indicator is displayed afterwards instead (regardless if the correct goal is green).
Revised: After the ﬁrst goal is reached, the correct indicator color is no longer displayed within the maze and a red indicator is displayed afterwards instead (regardless if the correct goal is green).

Original: An episode ends after a predetermined number of steps which depends on the maze size.
Revised: An episode ends after a predetermined number of steps which depends on the maze size.

Original: The goal is therefore to ﬁnd a path to the correct goal, and then optimize that path so that it can reach it as many times as possible.
Revised: The goal is therefore to ﬁnd a path to the correct goal, and then optimize that path so that it can reach it as many times as possible.

Original: Minotaur Maze: The third environment has the agent start in a ﬁxed starting position next to the green tower, while the red tower is randomly placed somewhere in the maze.
Revised: Minotaur Maze: The third environment has the agent start in a ﬁxed starting position next to the green tower, while the red tower is randomly placed somewhere in the maze.

Original: The agent receives a small positive reward if it reaches the red tower, and a larger positive reward if after reaching the red tower it returns to the green tower.
Revised: The agent receives a small positive reward if it reaches the red tower, and a larger positive reward if after reaching the red tower it returns to the green tower.

Original: Therefore the agent must efﬁciently navigate to the red goal while accurately remember its entire path it so that it can backtrack to the start.
Revised: Therefore the agent must efﬁciently navigate to the red goal while accurately remember its entire path it so that it can backtrack to the start.

Original: All three environments used a RGB+D image of size 100x60 as input.
Revised: All three environments used a RGB+D image of size 100x60 as input.

Original: We generate maze geometries randomly at train time but make sure to exclude a test set of 10 mazes for each size [4, 5, 6, 7, 8] ( total).
Revised: We generate maze geometries randomly at train time but make sure to exclude a test set of 10 mazes for each size [4, 5, 6, 7, 8] ( total).

Original: For these environments, we tested out four architectures (see Appendix C for more details on both environments and architectures):
Revised: For these environments, we tested out four architectures (see Appendix C for more details on both environments and architectures):

Original: Neural Map with Controller LSTM:
Revised: Neural Map with Controller LSTM:

Original: Standard Neural Map with ﬁxed frame addressing and GRU updates.
Revised: Standard Neural Map with ﬁxed frame addressing and GRU updates.

Original: We combine the neural map design with an LSTM that aggregates past state, read and context vectors and produces the query vector for the next time step’s context read operation.
Revised: We combine the neural map design with an LSTM that aggregates past state, read and context vectors and produces the query vector for the next time step’s context read operation.

Original: See Appendix A for the modiﬁed Neural Map equations.
Revised: See Appendix A for the modiﬁed Neural Map equations.

Original: Ego Neural Map with Controller LSTM:
Revised: Ego Neural Map with Controller LSTM:

Original: Same as previous but with ego-centric addressing.
Revised: Same as previous but with ego-centric addressing.

Original: The other difference is that the Ego Neural Map does not receive any positional input unlike the othermodels, only receiving frame-by-frame ego-motion (quantized to a coarse grid).
Revised: The other difference is that the Ego Neural Map does not receive any positional input unlike the othermodels, only receiving frame-by-frame ego-motion (quantized to a coarse grid).

Original: LSTM:
Revised: LSTM:

Original: Single pre-output 256-dimensional LSTM layer.
Revised: Single pre-output 256-dimensional LSTM layer.

Original: FRMQN (Oh et al., 2016): Memory network with LSTM feedback.
Revised: FRMQN (Oh et al., 2016): Memory network with LSTM feedback.

Original: This design uses an LSTM to make recurrent context queries to the memory network database.
Revised: This design uses an LSTM to make recurrent context queries to the memory network database.

Original: In addition, for the memory network baselines we did not set a ﬁxed k but instead let it access any state from its entire episode.
Revised: In addition, for the memory network baselines we did not set a ﬁxed k but instead let it access any state from its entire episode.

Original: This means no information is lost to the memory network
Revised: This means no information is lost to the memory network

Original: it only
Revised: it only

Original: needs to process its
Revised: needs to process its

Original: history
Revised: history

Original: [Table]
Revised: [Table]

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: We can see that the Neural Map architectures work better than the baseline models, even though the memory network has access to its entire episode history at every time step.
Revised: We can see that the Neural Map architectures work better than the baseline models, even though the memory network has access to its entire episode history at every time step.

Original: The ego-centric Neural Map beats the ﬁxed frame map at Indicator, and gets similar performance on both Repeating and Minotaur environments, showing the ability of the Neural Map to function effectively even without global position information.
Revised: The ego-centric Neural Map beats the ﬁxed frame map at Indicator, and gets similar performance on both Repeating and Minotaur environments, showing the ability of the Neural Map to function effectively even without global position information.

Original: It is possible that having a ﬁxed frame makes path optimization easier, which would explain the larger rewards that the ﬁxed-frame model got in the Repeating task.
Revised: It is possible that having a ﬁxed frame makes path optimization easier, which would explain the larger rewards that the ﬁxed-frame model got in the Repeating task.

Original: We also investigated whether the neural map is robust to localization noise, which would be the case in a real world setting where we do not have access to a localization oracle and must instead rely on an error-prone odometry or SLAM-type algorithm to do localization.
Revised: We also investigated whether the neural map is robust to localization noise, which would be the case in a real world setting where we do not have access to a localization oracle and must instead rely on an error-prone odometry or SLAM-type algorithm to do localization.

Original: These results are presented in Appendix D.
Revised: These results are presented in Appendix D.

Original: For the baselines, we can see that FRMQN has difﬁculty learning on Repeating, only reaching the goal on average once.
Revised: For the baselines, we can see that FRMQN has difﬁculty learning on Repeating, only reaching the goal on average once.

Original: This could be because the indicator is only shown before the ﬁrst goal is reached and so afterwards it needs to remember increasingly longer time horizons.
Revised: This could be because the indicator is only shown before the ﬁrst goal is reached and so afterwards it needs to remember increasingly longer time horizons.

Original: Furthermore, because the red indicator is always shown after the ﬁrst goal is reached, it might be difﬁcult for the model to learn to do retrieval since the original correct indicator must be indexed by time and not image similarity.
Revised: Furthermore, because the red indicator is always shown after the ﬁrst goal is reached, it might be difﬁcult for the model to learn to do retrieval since the original correct indicator must be indexed by time and not image similarity.

Original: The FRMQN also has difﬁculty on Minotaur, probably due to needing to remember and organize a lot of spatial information (i.e. what actions were taken along the path).
Revised: The FRMQN also has difﬁculty on Minotaur, probably due to needing to remember and organize a lot of spatial information (i.e. what actions were taken along the path).

Original: For Indicator, the FRMQN does similarly to the LSTM.
Revised: For Indicator, the FRMQN does similarly to the LSTM.

Original: We can see that the spatial structure of the Neural Map aids in optimizing the path in Repeating, averaging 12 goal reaches even in the largest maze size.
Revised: We can see that the spatial structure of the Neural Map aids in optimizing the path in Repeating, averaging 12 goal reaches even in the largest maze size.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Other than the straightforward architectures of combining an LSTM with Deep Reinforcement
Revised: Other than the straightforward architectures of combining an LSTM with Deep Reinforcement

Original: Learning (DRL) (Mnih et al., 2016; Hausknecht & Stone, 2015), there has also been work on using more advanced external memory systems with DRL agents to handle partial observability.
Revised: Learning (DRL) (Mnih et al., 2016; Hausknecht & Stone, 2015), there has also been work on using more advanced external memory systems with DRL agents to handle partial observability.

Original: Oh et al.
Revised: Oh et al.

Original: (2016) used a memory network (MemNN) to solve maze-based environments similar to the ones presented in this paper.
Revised: (2016) used a memory network (MemNN) to solve maze-based environments similar to the ones presented in this paper.

Original: MemNN keeps the last M states in memory and encodes them into (key, value) feature pairs.
Revised: MemNN keeps the last M states in memory and encodes them into (key, value) feature pairs.

Original: It then queries this memory using a soft attention mechanism similar to the context operation of the Neural Map, except in the Neural Map the key/value features were written by the agent and aren’t just a stored representation of the last M frames seen.
Revised: It then queries this memory using a soft attention mechanism similar to the context operation of the Neural Map, except in the Neural Map the key/value features were written by the agent and aren’t just a stored representation of the last M frames seen.

Original: Oh et al.
Revised: Oh et al.

Original: (2016) tested a few variants of this basic model, including ones which combined both LSTM and memory-network style memories.
Revised: (2016) tested a few variants of this basic model, including ones which combined both LSTM and memory-network style memories.

Original: In contrast to memory networks, another research direction is to design recurrent architectures that mimic computer memory systems.
Revised: In contrast to memory networks, another research direction is to design recurrent architectures that mimic computer memory systems.

Original: These architectures explicitly separate computation and memory in a way anagolous to a modern digital computer, in which some neural controller (akin to a CPU) interacts with an external memory (RAM).
Revised: These architectures explicitly separate computation and memory in a way anagolous to a modern digital computer, in which some neural controller (akin to a CPU) interacts with an external memory (RAM).

Original: One recent model is similar to the Neural Map, called the
Revised: One recent model is similar to the Neural Map, called the

Original: Differentiable Neural Computer (DNC) (Graves et al., 2016), which combines a recurrent controller with an external memory system that allows several types of read/write access.
Revised: Differentiable Neural Computer (DNC) (Graves et al., 2016), which combines a recurrent controller with an external memory system that allows several types of read/write access.

Original: In addition to deﬁn- ing an unconstrained write operator (in contrast to the neural map’s write location being ﬁxed), the
Revised: In addition to deﬁn- ing an unconstrained write operator (in contrast to the neural map’s write location being ﬁxed), the

Original: DNC has a selective read operation that reads out the memory either by content or in the order that it was written.
Revised: DNC has a selective read operation that reads out the memory either by content or in the order that it was written.

Original: While the DNC is more specialized to solving algorithmic problems, the Neural Map can be seen as an extension of this Neural Computer framework to 3D environments, with a speciﬁc inductive bias on its write operator that allows sparse writes.
Revised: While the DNC is more specialized to solving algorithmic problems, the Neural Map can be seen as an extension of this Neural Computer framework to 3D environments, with a speciﬁc inductive bias on its write operator that allows sparse writes.

Original: Recently work has also been done toward sparsifying the read and write operations of the DNC (Rae et al., 2016).
Revised: Recently work has also been done toward sparsifying the read and write operations of the DNC (Rae et al., 2016).

Original: This work was not focused on 3D environments and did not make any use of task-speciﬁc biases like agent location, but instead used more general biases like “Least-Recently-Used” memory addresses to force sparsity.
Revised: This work was not focused on 3D environments and did not make any use of task-speciﬁc biases like agent location, but instead used more general biases like “Least-Recently-Used” memory addresses to force sparsity.

Original: Gupta et al.
Revised: Gupta et al.

Original: (2017) designed a similar 2D map structured memory, with the aim to do robot navigation in 3D environments.
Revised: (2017) designed a similar 2D map structured memory, with the aim to do robot navigation in 3D environments.

Original: These environments were based off image scans of real ofﬁce buildings, and they were preprocessed into a grid-world by quantizing the possible positions and orientations the agent could assume.
Revised: These environments were based off image scans of real ofﬁce buildings, and they were preprocessed into a grid-world by quantizing the possible positions and orientations the agent could assume.

Original: In contrast to our paper, which presents the Neural Map more as a general memory architecture for DRL agents, Gupta et al.
Revised: In contrast to our paper, which presents the Neural Map more as a general memory architecture for DRL agents, Gupta et al.

Original: (2017) focuses mainly on solving the task of robot navigation.
Revised: (2017) focuses mainly on solving the task of robot navigation.

Original: More concretely, the task in these environments was to navigate to a goal state, with the goal position either stated semantically (ﬁnd a chair) or stated in terms of the position relative to the robot’s coordinate frame.
Revised: More concretely, the task in these environments was to navigate to a goal state, with the goal position either stated semantically (ﬁnd a chair) or stated in terms of the position relative to the robot’s coordinate frame.

Original: Owing to this focus on navigation, they force their internal map representation (e.g. M t ) to be a prediction of free space around the robot.
Revised: Owing to this focus on navigation, they force their internal map representation (e.g. M t ) to be a prediction of free space around the robot.

Original: Another key difference was that their formulation lacked a context addressing operation.
Revised: Another key difference was that their formulation lacked a context addressing operation.

Original: Finally, their method used DAGGER (Ross et al., 2011), an imitation learning algorithm, to train their agent.
Revised: Finally, their method used DAGGER (Ross et al., 2011), an imitation learning algorithm, to train their agent.

Original: Since Doom actions affect translational/rotational accelerations, training using imitation learning is more difﬁcult since a search algorithm cannot be used as supervision.
Revised: Since Doom actions affect translational/rotational accelerations, training using imitation learning is more difﬁcult since a search algorithm cannot be used as supervision.

Original: An interesting addition they made was the use of a multi-scale map representation and a Value Iteration network (Tamar et al., 2016) to do better path planning.
Revised: An interesting addition they made was the use of a multi-scale map representation and a Value Iteration network (Tamar et al., 2016) to do better path planning.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we developed a neural memory architecture that organizes the spatial structure of its memory in the form of a 2D map, and allows sparse writes to this memory where the memory address of the write is in a correspondence to the agent’s current position in the environment.
Revised: In this paper we developed a neural memory architecture that organizes the spatial structure of its memory in the form of a 2D map, and allows sparse writes to this memory where the memory address of the write is in a correspondence to the agent’s current position in the environment.

Original: We showed its ability to learn, using a reinforcement signal, how to behave within challenging 2D and 3D maze tasks that required storing information over long time steps.
Revised: We showed its ability to learn, using a reinforcement signal, how to behave within challenging 2D and 3D maze tasks that required storing information over long time steps.

Original: The results demonstrated that our architecture surpassed baseline memories used in previous work.
Revised: The results demonstrated that our architecture surpassed baseline memories used in previous work.

Original: Additionally, we showed the beneﬁt of certain design decisions made in our architecture: using GRU updates instead of hard writes, demonstrating that the ego-centric viewpoint does not diminish performance and that the
Revised: Additionally, we showed the beneﬁt of certain design decisions made in our architecture: using GRU updates instead of hard writes, demonstrating that the ego-centric viewpoint does not diminish performance and that the

Original: Neural Map is robust to downsampling its memory.
Revised: Neural Map is robust to downsampling its memory.

Original: Finally, to show that our method can scale up to more difﬁcult 3D environments, we implemented several new maze environments in Doom.
Revised: Finally, to show that our method can scale up to more difﬁcult 3D environments, we implemented several new maze environments in Doom.

Original: Using a hybrid Neural Map + LSTM model, we were able to solve most of the scenarios at a performance higher than previous DRL memory-based architectures.
Revised: Using a hybrid Neural Map + LSTM model, we were able to solve most of the scenarios at a performance higher than previous DRL memory-based architectures.

Original: Furthermore, we demonstrated the ability of the Neural Map to be robust to a certain level of drift noise in its localization estimate.
Revised: Furthermore, we demonstrated the ability of the Neural Map to be robust to a certain level of drift noise in its localization estimate.

Original: A C ONTROLLER (E GO -)N EURAL M AP
Revised: A C ONTROLLER (E GO -)N EURAL M AP

Original: Here we describe the modiﬁcation to the Neural Map we utilized for the 3D maze tasks.
Revised: Here we describe the modiﬁcation to the Neural Map we utilized for the 3D maze tasks.

Original: We include an extra state h that represents the hidden and cell state of an LSTM.
Revised: We include an extra state h that represents the hidden and cell state of an LSTM.

Original: The Neural Map equations are therefore:
Revised: The Neural Map equations are therefore:

Original: [Equation]
Revised: [Equation]

Original: B 2D E NVIRONMENT D ETAILS
Revised: B 2D E NVIRONMENT D ETAILS

Original: The state input for the 2D environment is a 5 × 15 × 3 subsample of the complete maze so that the agent is able to see 15 pixel forward and 3 pixels on the side (center pixel + one pixel on each side of the agent) which is depicted in Fig.1b.
Revised: The state input for the 2D environment is a 5 × 15 × 3 subsample of the complete maze so that the agent is able to see 15 pixel forward and 3 pixels on the side (center pixel + one pixel on each side of the agent) which is depicted in Fig.1b.

Original: This view is obscured so the agent is prevented from seeing the identity of anything behind walls.
Revised: This view is obscured so the agent is prevented from seeing the identity of anything behind walls.

Original: The 5 binary channels in the observation represent object identities: channel 1 represents presence of walls, 2 represents the green indicator, 3 the blue indicator, 4 the red goal, and 5 the teal goal.
Revised: The 5 binary channels in the observation represent object identities: channel 1 represents presence of walls, 2 represents the green indicator, 3 the blue indicator, 4 the red goal, and 5 the teal goal.

Original: The LSTM and MemNN networks were given auxiliary information such as the one-hot encoding of the current agent’s true position in the maze.
Revised: The LSTM and MemNN networks were given auxiliary information such as the one-hot encoding of the current agent’s true position in the maze.

Original: Neural Map variants were not given position information in an auxiliary state unless speciﬁed by the “+ Pos” modiﬁer in Table 1.
Revised: Neural Map variants were not given position information in an auxiliary state unless speciﬁed by the “+ Pos” modiﬁer in Table 1.

Original: Actions in the environment included moving forward and turning left/right 90 degrees.
Revised: Actions in the environment included moving forward and turning left/right 90 degrees.

Original: For optimization, all architectures used the RMSprop optimization algorithm with gradients thresholded to norm 20 for LSTM, 100 for
Revised: For optimization, all architectures used the RMSprop optimization algorithm with gradients thresholded to norm 20 for LSTM, 100 for

Original: We used an auxiliary weighted entropy loss on the Synchronous Actor-Critic with weight 0.01.
Revised: We used an auxiliary weighted entropy loss on the Synchronous Actor-Critic with weight 0.01.

Original: The learning rates for LSTM models was 0.0025, 0.005 for Neural Map variants, and 0.001 for memory networks.
Revised: The learning rates for LSTM models was 0.0025, 0.005 for Neural Map variants, and 0.001 for memory networks.

Original: We obtained the hyperparameters during a limited hyperparameter sweep on a simpler version of the environment.
Revised: We obtained the hyperparameters during a limited hyperparameter sweep on a simpler version of the environment.

Original: We used A2C with number of time steps equal to 5.
Revised: We used A2C with number of time steps equal to 5.

Original: We trained formillion updates.
Revised: We trained formillion updates.

Original: The mazes were generated using an algorithm based on Depth-First Search to form a fully-connected maze.
Revised: The mazes were generated using an algorithm based on Depth-First Search to form a fully-connected maze.

Original: Afterwards each wall was deleted with probability p .
Revised: Afterwards each wall was deleted with probability p .

Original: p was sampled uniformly from between [0 , 0 .
Revised: p was sampled uniformly from between [0 , 0 .

Original: 75] at maze generation time.
Revised: 75] at maze generation time.

Original: Each episode, a random maze width was sampled uniformly from between [7 , 15] .
Revised: Each episode, a random maze width was sampled uniformly from between [7 , 15] .

Original: (Ego-)Neural Map Agent Details: For the global read operation, we used a convolutional network with 3 layers of 8 channels and kernel size 3.
Revised: (Ego-)Neural Map Agent Details: For the global read operation, we used a convolutional network with 3 layers of 8 channels and kernel size 3.

Original: The strides were set to 1 on the ﬁrst layer, and 2 to the second and third layers.
Revised: The strides were set to 1 on the ﬁrst layer, and 2 to the second and third layers.

Original: Padding was 1 on the ﬁrst layer and 0 on other layers.
Revised: Padding was 1 on the ﬁrst layer and 0 on other layers.

Original: The 3 convolutional layers were then followed by a fully-connected layer of dimension 

Revised: The 3 convolutional layers were then followed by a fully-connected layer of dimension

Original: and then another fullyconnected layer of size 32.
Revised: and then another fullyconnected layer of size 32.

Original: All activations were relu except the 32 dimension layer, which was set to tanh.
Revised: All activations were relu except the 32 dimension layer, which was set to tanh.

Original: Positions were normalized so that the largest map size used all positions, while smaller mazes used a subset of the map.
Revised: Positions were normalized so that the largest map size used all positions, while smaller mazes used a subset of the map.

Original: LSTM Agent Details: The LSTM agent had a single 128-dimension LSTM layer on top of the state embedding.
Revised: LSTM Agent Details: The LSTM agent had a single 128-dimension LSTM layer on top of the state embedding.

Original: The auxiliary state information was ﬁrst processed into a 256-dimension embedding.
Revised: The auxiliary state information was ﬁrst processed into a 256-dimension embedding.

Original: MQN Agent Details: Each past history state was 512-dimensional (256-dimensional key + 256dimensional value feature).
Revised: MQN Agent Details: Each past history state was 512-dimensional (256-dimensional key + 256dimensional value feature).

Original: The input state was processed by a convolutional network withchannels, ﬁlter size 3, stride 1 and padding 1.
Revised: The input state was processed by a convolutional network withchannels, ﬁlter size 3, stride 1 and padding 1.

Original: C 3D E NVIRONMENT D ETAILS
Revised: C 3D E NVIRONMENT D ETAILS

Original: The state input for all 3D environments was a 100x60 RGB+D image.
Revised: The state input for all 3D environments was a 100x60 RGB+D image.

Original: This was passed through a convolution network that was the same for all architectures.
Revised: This was passed through a convolution network that was the same for all architectures.

Original: It ﬁrst consists of a 2D convolution with 32 channels of ﬁlter size 8 with stride 4.
Revised: It ﬁrst consists of a 2D convolution with 32 channels of ﬁlter size 8 with stride 4.

Original: This was then passed to another 2D convolution with 64 channels of ﬁlter size 4 and stride 2.
Revised: This was then passed to another 2D convolution with 64 channels of ﬁlter size 4 and stride 2.

Original: Finally, the result was passed through a fully-connected layer with 512 features, which represented the current frame embedding.
Revised: Finally, the result was passed through a fully-connected layer with 512 features, which represented the current frame embedding.

Original: The frame embedding was then augmented with some auxiliary information about the map, which in the case of Neural Map,
Revised: The frame embedding was then augmented with some auxiliary information about the map, which in the case of Neural Map,

Original: FRMQN and LSTM architectures was 1) a one-hot encoding of the current time step, 2) the current orientation (North/East/West/South), 3) the 2D velocity (change in x/y position in a top-down 2D quantized grid of possible environment positions), and 4) a one-hot encoding of the agent’s current quantized position.
Revised: FRMQN and LSTM architectures was 1) a one-hot encoding of the current time step, 2) the current orientation (North/East/West/South), 3) the 2D velocity (change in x/y position in a top-down 2D quantized grid of possible environment positions), and 4) a one-hot encoding of the agent’s current quantized position.

Original: For Ego Neural Map, only 1, 2, 3 are used (i.e. it has no input of the agent’s current true position given by an oracle).
Revised: For Ego Neural Map, only 1, 2, 3 are used (i.e. it has no input of the agent’s current true position given by an oracle).

Original: The 512 frame encoding is concatenated with the auxiliary state information to form the complete state embedding.
Revised: The 512 frame encoding is concatenated with the auxiliary state information to form the complete state embedding.

Original: Actions in the maze consist of moving forward and turning left or right.
Revised: Actions in the maze consist of moving forward and turning left or right.

Original: For optimization, all architectures used the Adam optimization algorithm with gradients thresholded to a norm of 40.
Revised: For optimization, all architectures used the Adam optimization algorithm with gradients thresholded to a norm of 40.

Original: We used an auxiliary weighted entropy loss on the Synchronous Actor-Critic with weight 0.01.
Revised: We used an auxiliary weighted entropy loss on the Synchronous Actor-Critic with weight 0.01.

Original: The learning rate for LSTM models was 0.0005, while for other architectures it was set to 0.00075.
Revised: The learning rate for LSTM models was 0.0005, while for other architectures it was set to 0.00075.

Original: The hyperparameters were obtained during a limited hyperparameter sweep on a simpler version of the Indicator Maze environment.
Revised: The hyperparameters were obtained during a limited hyperparameter sweep on a simpler version of the Indicator Maze environment.

Original: We trained A2C with a number of steps equal to the episode length (no truncated backprop).
Revised: We trained A2C with a number of steps equal to the episode length (no truncated backprop).

Original: We trained the agent for 3000 steps, where each step consisted of a gradient obtained from 100 full episodes.
Revised: We trained the agent for 3000 steps, where each step consisted of a gradient obtained from 100 full episodes.

Original: The effective batch size was thus upper bounded by 500 ∗ 100 = 50000 .
Revised: The effective batch size was thus upper bounded by 500 ∗ 100 = 50000 .

Original: We used multithreading to calculate the batch gradients efﬁciently.
Revised: We used multithreading to calculate the batch gradients efﬁciently.

Original: Each update step took on the order of 1-3 minutes depending on the number of threads available, meaning each agent took on the order of a week to train.
Revised: Each update step took on the order of 1-3 minutes depending on the number of threads available, meaning each agent took on the order of a week to train.

Original: The mazes were generated using an algorithm based on Depth-First Search.
Revised: The mazes were generated using an algorithm based on Depth-First Search.

Original: Once the completed fully-connected maze was generated, random walls were deleted with a probability p .
Revised: Once the completed fully-connected maze was generated, random walls were deleted with a probability p .

Original: This p probability was chosen at maze creation time and was sampled uniformly from between 0.0 and 0.6.
Revised: This p probability was chosen at maze creation time and was sampled uniformly from between 0.0 and 0.6.

Original: Mazes were between size 4 to size 8.
Revised: Mazes were between size 4 to size 8.

Original: The size of a maze represents how many “cells” there are in the maze, where a cell is an area which can potentially have walls on each side.
Revised: The size of a maze represents how many “cells” there are in the maze, where a cell is an area which can potentially have walls on each side.

Original: A set of 50 test maze geometries were sampled to act as a test set, and were made sure to never be sampled during training.
Revised: A set of 50 test maze geometries were sampled to act as a test set, and were made sure to never be sampled during training.

Original: These 50 test mazes were generated with p = 0 , so they represent the most difﬁcult mazes seen during training due to their higher degree of partial-observability.
Revised: These 50 test mazes were generated with p = 0 , so they represent the most difﬁcult mazes seen during training due to their higher degree of partial-observability.

Original: Goal locations were sampled uniformly at random when the mazes are generated.
Revised: Goal locations were sampled uniformly at random when the mazes are generated.

Original: Indicator Maze Details: The indicator mazes used a curriculum approach to accelerate learning.
Revised: Indicator Maze Details: The indicator mazes used a curriculum approach to accelerate learning.

Original: In 50% of the episodes sampled, only one goal existed in the environment and the indicator color matched the single goal color, making entering an incorrect goal impossible.
Revised: In 50% of the episodes sampled, only one goal existed in the environment and the indicator color matched the single goal color, making entering an incorrect goal impossible.

Original: This curriculum prevented the agents from learning to always enter a single color goal, which happened often when learning on only double goal environments.
Revised: This curriculum prevented the agents from learning to always enter a single color goal, which happened often when learning on only double goal environments.

Original: The test environments only used double goals.
Revised: The test environments only used double goals.

Original: A reward of +1 was given to correct goal entry, and a negative reward of -1 was given to incorrect goal entry or episode terminating after a maximum number of time steps.
Revised: A reward of +1 was given to correct goal entry, and a negative reward of -1 was given to incorrect goal entry or episode terminating after a maximum number of time steps.

Original: These time steps depended on the maze size and were [150, 250, 300, 400, 500] for maze sizes [4, 5, 6, 7, 8].
Revised: These time steps depended on the maze size and were [150, 250, 300, 400, 500] for maze sizes [4, 5, 6, 7, 8].

Original: At test time, the maximum time steps were extended to [300, 500, 600, 800, 1000].
Revised: At test time, the maximum time steps were extended to [300, 500, 600, 800, 1000].

Original: Repeating Maze Details We included the same curriculum as the indicator maze.
Revised: Repeating Maze Details We included the same curriculum as the indicator maze.

Original: Rewards were the same as the indicator maze.
Revised: Rewards were the same as the indicator maze.

Original: After the ﬁrst time the agent reaches a goal, the red torch is shown at each subsequent episode (regardless of what the correct indicator was).
Revised: After the ﬁrst time the agent reaches a goal, the red torch is shown at each subsequent episode (regardless of what the correct indicator was).

Original: Maximum time steps were again [150, 250, 300, 400, 500] for maze sizes [4, 5, 6, 7, 8].
Revised: Maximum time steps were again [150, 250, 300, 400, 500] for maze sizes [4, 5, 6, 7, 8].

Original: At test time, the maximum time steps were extended to [300, 500, 600, 800, 1000].
Revised: At test time, the maximum time steps were extended to [300, 500, 600, 800, 1000].

Original: Minotaur Maze: For minotaur maze, a reward of +0.5 was given when reaching the randomly located goal and a reward of +1.0 was given when returning to the initial position.
Revised: Minotaur Maze Details: For minotaur maze, a reward of +0.5 was given when reaching the randomly located goal and a reward of +1.0 was given when returning to the initial position.

Original: Episodes are terminated once the agent completes the return path, otherwise a negative reward of -1 is given if the agent exceeds the maximum number of steps.
Revised: Episodes are terminated once the agent completes the return path, otherwise a negative reward of -1 is given if the agent exceeds the maximum number of steps.

Original: The maximum time steps were again [150, 250, 300, 400, 500] for maze sizes [4, 5, 6, 7, 8].
Revised: The maximum time steps were again [150, 250, 300, 400, 500] for maze sizes [4, 5, 6, 7, 8].

Original: At test time, the maximum time steps were extended to [300, 500, 600, 800, 1000].
Revised: At test time, the maximum time steps were extended to [300, 500, 600, 800, 1000].

Original: [Table]
Revised: [Table]

Original: (Ego-)Neural Map Agent Details: For the global read operation, we used a convolutional network with 3 layers of 8 channels and kernel size 3.
Revised: (Ego-)Neural Map Agent Details: For the global read operation, we used a convolutional network with 3 layers of 8 channels and kernel size 3.

Original: The strides were set to 1 on the ﬁrst layer, and 2 to the second and third layers.
Revised: The strides were set to 1 on the ﬁrst layer, and 2 to the second and third layers.

Original: Padding was 1 on the ﬁrst layer and 0 on other layers.
Revised: Padding was 1 on the ﬁrst layer and 0 on other layers.

Original: The 3 convolutional layers were then followed by a fully-connected layer of dimension 

Revised: The 3 convolutional layers were then followed by a fully-connected layer of dimension

Original: and then another fullyconnected layer of size 32.
Revised: and then another fullyconnected layer of size 32.

Original: All activations were relu except the 32 dimension layer, which was set to tanh.
Revised: All activations were relu except the 32 dimension layer, which was set to tanh.

Original: The Neural Map itself was size 32x15x15.
Revised: The Neural Map itself was size 32x15x15.

Original: Positions were normalized so that the largest map size used all 15x15 positions, while smaller mazes used a subset of the map.
Revised: Positions were normalized so that the largest map size used all 15x15 positions, while smaller mazes used a subset of the map.

Original: Additionally, during writing we split the 32 channels of the map into 8 channels per orientation (so if the agent is facing north, it writes only to the ﬁrst 8 dimensions, if south, the next 8, and so on).
Revised: Additionally, during writing we split the 32 channels of the map into 8 channels per orientation (so if the agent is facing north, it writes only to the ﬁrst 8 dimensions, if south, the next 8, and so on).

Original: LSTM Agent Details: The LSTM agent had a single 256-dimension LSTM layer on top of the state embedding.
Revised: LSTM Agent Details: The LSTM agent had a single 256-dimension LSTM layer on top of the state embedding.

Original: FRMQN Agent Details: Each past history state was 64-dimensional (32-dimensional key + 32dimensional value feature).
Revised: FRMQN Agent Details: Each past history state was 64-dimensional (32-dimensional key + 32dimensional value feature).

Original: Due to large matrix multiplies from storing the entire episode history, having larger feature sizes causes the attention operation to start becoming prohibitively expensive.
Revised: Due to large matrix multiplies from storing the entire episode history, having larger feature sizes causes the attention operation to start becoming prohibitively expensive.

Original: D N EURAL M AP WITH D RIFT N OISE M ODEL
Revised: D N EURAL M AP WITH D RIFT N OISE M ODEL

Original: We did an additional experiment on the Neural Map that featured drift noise to simulate the effects of the agent using a local visual odometry model that had small error in predicting each frame-by-frame transformation.
Revised: We did an additional experiment on the Neural Map that featured drift noise to simulate the effects of the agent using a local visual odometry model that had small error in predicting each frame-by-frame transformation.

Original: This is meant to represent a more realistic scenario (e.g. robotic navigation) where perfect localization is not feasible but a relatively accurate estimate can be provided, demonstrating the robustness of the architecture to noise.
Revised: This is meant to represent a more realistic scenario (e.g. robotic navigation) where perfect localization is not feasible but a relatively accurate estimate can be provided, demonstrating the robustness of the architecture to noise.

Original: For example, we could assume the Neural Map is run in parallel with a SLAM algorithm which provides an estimate of the agent’s current position.
Revised: For example, we could assume the Neural Map is run in parallel with a SLAM algorithm which provides an estimate of the agent’s current position.

Original: To model this noise, we add a zero-mean gaussian random variable to the oracle position with a variance that depends on the current time-step.
Revised: To model this noise, we add a zero-mean gaussian random variable to the oracle position with a variance that depends on the current time-step.

Original: In more detail, the noise-corrupted positions (ˆ x, ˆ y ) in an W × W size map provided to the Neural Map are:
Revised: In more detail, the noise-corrupted positions (ˆ x, ˆ y ) in an W × W size map provided to the Neural Map are:

Original: [Equation]
Revised: [Equation]

Original: This simulates the effect of an odometry algorithm which has independent zero-mean gaussian error with equal variance.
Revised: This simulates the effect of an odometry algorithm which has independent zero-mean gaussian error with equal variance.

Original: This error compounds over time causing the variance to grow with the time step.
Revised: This error compounds over time causing the variance to grow with the time step.

Original: We evaluate the Neural Map with noise σ = 1 / 100 on smaller versions of the 3D Doom maze tasks (maze sizes [4, 5, 6]) and compare it to the version with perfect odometry.
Revised: We evaluate the Neural Map with noise σ = 1 / 100 on smaller versions of the 3D Doom maze tasks (maze sizes [4, 5, 6]) and compare it to the version with perfect odometry.

Original: We train for 1500 steps of 100 episodes each step.
Revised: We train for 1500 steps of 100 episodes each step.

Original: Results are shown in Table 3.
Revised: Results are shown in Table 3.

Original: We can see that adding a small amount of error at each time step does not largely affect the results of the memory and can even beneﬁt it, with some noticeable improvements on Indicator and Repeating tasks.
Revised: We can see that adding a small amount of error at each time step does not largely affect the results of the memory and can even beneﬁt it, with some noticeable improvements on Indicator and Repeating tasks.

Original: It’s possible that the noise acts as a regularizer to speed up learning.
Revised: It’s possible that the noise acts as a regularizer to speed up learning.

Original: For the Minotaur task, since positional information is important because the agent must remember the entire path taken, adding noise causes slight decrease in reward in mazes of size 4 and 5, but otherwise performance is very similar.
Revised: For the Minotaur task, since positional information is important because the agent must remember the entire path taken, adding noise causes slight decrease in reward in mazes of size 4 and 5, but otherwise performance is very similar.

Original: Therefore this means that the
Revised: Therefore this means that the

Original: Neural Map is likely to work in the case where a localization oracle is not available and instead only error-prone odometry is.
Revised: Neural Map is likely to work in the case where a localization oracle is not available and instead only error-prone odometry is.

Revised: We also plot some example trajectories to compare the effect of noise.

Revised: We can see that the noise causes some slight aliasing in the position, which increases as time passes.

Revised: The positions are quantized to a 15 × 15 grid.

Revised: 14       groundtruthnoisy       14       groundtruthnoisy       14       groundtruthnoisy

Original: E S AMPLES OF C ONTEXT R EAD D ISTRIBUTION 
Revised: E S AMPLES OF C ONTEXT R EAD D ISTRIBUTION E. 2D E NVIRONMENT

Original: To provide some insight into what the Neural Map learns, we show samples of the probability distribution given by the context read operation in a 2D maze example.
Revised: To provide some insight into what the Neural Map learns, we show samples of the probability distribution given by the context read operation in a 2D maze example.

Original: We ran it on an example maze shown in Figure 3.
Revised: We ran it on an example maze shown in Figure 4.

Original: In this ﬁgure, the top row of images are the agent observations, the center row are the fully observable mazes and the bottom row are the probability distributions over locations from the context operation, e.g. the α ( x,y ) t values deﬁned by Eq. 2.
Revised: In this ﬁgure, the top row of images are the agent observations, the center row are the fully observable mazes and the bottom row are the probability distributions over locations from the context operation, e.g. the α ( x,y ) t values deﬁned by Eq. 2.

Original: In this maze, the indicator is blue, which indicates that the teal goal should be visited.
Revised: In this maze, the indicator is blue, which indicates that the teal goal should be visited.

Original: We can see that once the agent sees the incorrect red goal, the context distribution faintly focuses on the map location where the agent had observed the indicator.
Revised: We can see that once the agent sees the incorrect red goal, the context distribution faintly focuses on the map location where the agent had observed the indicator.

Original: On the other hand, when the agent ﬁrst observes the correct teal goal, the location where the agent observed the indicator lights up brightly.
Revised: On the other hand, when the agent ﬁrst observes the correct teal goal, the location where the agent observed the indicator lights up brightly.

Original: This means that the agent is using its context retrieval operation to keep track of the landmark (the indicator) that it has previously seen. 
Revised: This means that the agent is using its context retrieval operation to keep track of the landmark (the indicator) that it has previously seen. E. 3D E NVIRONMENT

Revised: We draw some examples of the context addressing probability distribution in the 3D Doom environment in Figure 5 (allocentric) and Figure 6 (egocentric).

Revised: We can see that the Neural Map learns to use its context addressing operator to retrieve the indicator torch identity, until it sees the correct corresponding tower.

Revised: Once it sees the correct tower there is a shift in how the agent uses the map and the probability map seems to invert, addressing the parts of the map that were unexplored.

Revised: This effect is consistent in both allocentric and egocentric variants.

Revised: This might be because the Neural

Revised: Map variant used on Doom had an internal LSTM which could enable it to remember the indicator identity for the short amount of time it took to walk up to the goal.

Revised: Indicator Prediction To determine whether the Neural Map was accurately storing the indicator identity within its memory, we train a logistic regression model on memory vectors sampled over 75 episodes.

Revised: We then attempt to predict the indicator on a held-out set of 25 episodes by taking the max prediction over all positions of the memory at the end of the episode.

Revised: We can see that a simple logistic regression is capable of recovering the indicator in 100% of the episodes, showing that the indicator identity can be easily extracted from, e.g., the context operator.

Revised: [Table]

Revised: F B ACKTRACKING

Revised: We also explored whether the allocentric and egocentric Neural Maps were capable of using their memories in order to do backtracking, i.e. re-visiting unexplored areas of the maze.

Revised: To measure this, we developed a variant of the Indicator Maze where the goal states were removed.

Revised: We want to measure how much of the maze is explored by the agent under this setting where there are no terminal states.

Revised: To measure how much of the maze was explored, we quantized the 50 test mazes into 11 discrete positions and counted how many of the quantized positions the agent visited.

Revised: We report results below in Table

Revised: [Table]