Original: I MPLEMENTING I NDUCTIVE BIAS FOR DIFFERENT NAVIGATION TASKS THROUGH DIVERSE RNN ATTRRACTORS
Revised: I MPLEMENTING I NDUCTIVE BIAS FOR DIFFERENT NAVIGATION TASKS THROUGH DIVERSE RNN ATTRRACTORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map.
Revised: Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map.

Original: The precise form of this representation is often considered to be a metric representation of space
Revised: The precise form of this representation is often considered to be a metric representation of space

Original: An internal representation, however, is judged by its contribution to performance on a given task and may thus vary between different types of navigation tasks.
Revised: An internal representation, however, is judged by its contribution to performance on a given task and may thus vary between different types of navigation tasks.

Original: Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment.
Revised: Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment.

Original: To focus on internal representations, we split learning into a task-agnostic pre-training stage that modiﬁes internal connectivity and a task-speciﬁc Q learning stage that controls the network’s output.
Revised: To focus on internal representations, we split learning into a task-agnostic pre-training stage that modiﬁes internal connectivity and a task-speciﬁc Q learning stage that controls the network’s output.

Original: We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state.
Revised: We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state.

Original: These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities.
Revised: These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities.

Original: By combining two types of networks in a modular structure, we could get better performance for both regularities.
Revised: By combining two types of networks in a modular structure, we could get better performance for both regularities.

Original: Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes – which can be shaped by pre-training and analyzed using dynamical systems methods.
Revised: Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes – which can be shaped by pre-training and analyzed using dynamical systems methods.

Original: Furthermore, we demonstrate that non-metric representations are useful for navigation tasks and their combination with metric representation leads to ﬂexibile multiple-task learning
Revised: Furthermore, we demonstrate that non-metric representations are useful for navigation tasks and their combination with metric representation leads to ﬂexibile multiple-task learning

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Spatial navigation is an important task that requires a correct internal representation of the world, and thus its mechanistic underpinnings have attracted the attention of scientists for a long time (O’Keefe & Nadel, 1978).
Revised: Spatial navigation is an important task that requires a correct internal representation of the world, and thus its mechanistic underpinnings have attracted the attention of scientists for a long time (O’Keefe & Nadel, 1978).

Original: A standard tool for navigation is a euclidean map, and this naturally leads to the hypothesis that our internal model is such a map.
Revised: A standard tool for navigation is a euclidean map, and this naturally leads to the hypothesis that our internal model is such a map.

Original: Artiﬁcial navigation also relies on
Revised: Artiﬁcial navigation also relies on

Original: SLAM (Simultaneous localization and mapping) which is based on maps (Kanitscheider & Fiete, 2017a).
Revised: SLAM (Simultaneous localization and mapping) which is based on maps (Kanitscheider & Fiete, 2017a).

Original: On the other hand, both from an ecological view and from a pure machine learning perspective, navigation is ﬁrstly about reward acquisition, while exploiting the statistical regularities of the environment.
Revised: On the other hand, both from an ecological view and from a pure machine learning perspective, navigation is ﬁrstly about reward acquisition, while exploiting the statistical regularities of the environment.

Original: Different tasks and environments lead to different statistical regularities.
Revised: Different tasks and environments lead to different statistical regularities.

Original: Thus it is unclear which internal representations are optimal for reward acquisition.
Revised: Thus it is unclear which internal representations are optimal for reward acquisition.

Original: We take a functional approach to this question by training recurrent neural networks for navigation tasks with various types of statistical regularities.
Revised: We take a functional approach to this question by training recurrent neural networks for navigation tasks with various types of statistical regularities.

Original: Because we are interested in internal representations, we opt for a two-phase learning scheme instead of end-to-end learning.
Revised: Because we are interested in internal representations, we opt for a two-phase learning scheme instead of end-to-end learning.

Original: Inspired by the biological phenomena of evolution and development, we ﬁrst pre-train the networks to emphasize several aspects of their internal representation.
Revised: Inspired by the biological phenomena of evolution and development, we ﬁrst pre-train the networks to emphasize several aspects of their internal representation.

Original: Following pre-training, we use Q-learning to modify the network’s readout weights for speciﬁc tasks while maintaining its internal connectivity.
Revised: Following pre-training, we use Q-learning to modify the network’s readout weights for speciﬁc tasks while maintaining its internal connectivity.

Original: We evaluate the performance for different networks on a battery of simple navigation tasks with different statistical regularities and show that the internal representations of the networks manifest in differential performance according to the nature of tasks.
Revised: We evaluate the performance for different networks on a battery of simple navigation tasks with different statistical regularities and show that the internal representations of the networks manifest in differential performance according to the nature of tasks.

Original: The link between task performance and network structure is understood by probing networks’ dynamics, exposing a low-dimensional manifold of slow dynamics in phase space, which is clustered into three major categories: continuous attractor, discrete attractors, and unstructured chaotic dynamics.
Revised: The link between task performance and network structure is understood by probing networks’ dynamics, exposing a low-dimensional manifold of slow dynamics in phase space, which is clustered into three major categories: continuous attractor, discrete attractors, and unstructured chaotic dynamics.

Original: The different network attractors encode different priors, or inductive bias, for speciﬁc tasks which corresponds to metric or topology invariances in the tasks.
Revised: The different network attractors encode different priors, or inductive bias, for speciﬁc tasks which corresponds to metric or topology invariances in the tasks.

Original: By combining networks with different inductive biases we could build a modular system with improved multiple-task learning.
Revised: By combining networks with different inductive biases we could build a modular system with improved multiple-task learning.

Original: Overall we offer a paradigm which shows how dynamics of recurrent networks implement different priors for environments.
Revised: Overall we offer a paradigm which shows how dynamics of recurrent networks implement different priors for environments.

Original: Pre-training, which is agnostic to speciﬁc tasks, could lead to dramatic difference in the network’s dynamical landscape and affect reinforcement learning of different navigation tasks.
Revised: Pre-training, which is agnostic to speciﬁc tasks, could lead to dramatic difference in the network’s dynamical landscape and affect reinforcement learning of different navigation tasks.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Several recent papers used a functional approach for navigation (Cueva & Wei, 2018; Kanitscheider & Fiete, 2017b; Banino et al., 2018).
Revised: Several recent papers used a functional approach for navigation (Cueva & Wei, 2018; Kanitscheider & Fiete, 2017b; Banino et al., 2018).

Original: These works, however, consider the position as the desired output, by assuming that it is the relevant representation for navigation.
Revised: These works, however, consider the position as the desired output, by assuming that it is the relevant representation for navigation.

Original: These works successfully show that the recurrent network agent could solve the neural SLAM problem and that this could result in units of the network exhibiting similar response proﬁles to those found in neurophysiological experiments (place and grid cells).
Revised: These works successfully show that the recurrent network agent could solve the neural SLAM problem and that this could result in units of the network exhibiting similar response proﬁles to those found in neurophysiological experiments (place and grid cells).

Original: In our case, the desired behavior was to obtain the reward, and not to report the current position.
Revised: In our case, the desired behavior was to obtain the reward, and not to report the current position.

Original: Another recent approach did deﬁne reward acquisition as the goal, by applying deep RL directly to navigation problems in an end to end manner (Mirowski et al., 2016).
Revised: Another recent approach did deﬁne reward acquisition as the goal, by applying deep RL directly to navigation problems in an end to end manner (Mirowski et al., 2016).

Original: The navigation tasks relied on rich visual cues, that allowed evaluation in a state of the art setting.
Revised: The navigation tasks relied on rich visual cues, that allowed evaluation in a state of the art setting.

Original: This richness, however, can hinder the greater mechanistic insights that can be obtained from the systematic analysis of toy problems – and accordingly, the focus of these works is on performance.
Revised: This richness, however, can hinder the greater mechanistic insights that can be obtained from the systematic analysis of toy problems – and accordingly, the focus of these works is on performance.

Original: Our work is also related to recent works in neuroscience that highlight the richness of neural representations for navigation, beyond Euclidian spatial maps (Hardcastle et al., 2017; Wirth et al., 2017).
Revised: Our work is also related to recent works in neuroscience that highlight the richness of neural representations for navigation, beyond Euclidian spatial maps (Hardcastle et al., 2017; Wirth et al., 2017).

Original: Our pre-training is similar to unsupervised, followed by supervised training (Erhan et al., 2010).
Revised: Our pre-training is similar to unsupervised, followed by supervised training (Erhan et al., 2010).

Original: In the past few years, end-to-end learning is a more dominant approach (Graves et al., 2014; Mnih et al., 2013) .
Revised: In the past few years, end-to-end learning is a more dominant approach (Graves et al., 2014; Mnih et al., 2013) .

Original: We highlight the ability of a pre-training framework to manipulate network dynamics and the resulting internal representations and study their effect as inductive bias.
Revised: We highlight the ability of a pre-training framework to manipulate network dynamics and the resulting internal representations and study their effect as inductive bias.

Original:  R ESULTS 3. T ASK D EFINITION
Revised:  R ESULTS 3. T ASK D EFINITION

Original: Navigation can be described as taking advantage of spatial regularities of the environment to achieve goals.
Revised: Navigation can be described as taking advantage of spatial regularities of the environment to achieve goals.

Original: This view naturally leads to considering a cognitive map as an internal model of the environment, but leaves open the question of precisely which type of map is to be expected.
Revised: This view naturally leads to considering a cognitive map as an internal model of the environment, but leaves open the question of precisely which type of map is to be expected.

Original: To answer this question, we systematically study both a space of networks – emphasizing different internal models – and a space of tasks – emphasizing different spatial regularities.
Revised: To answer this question, we systematically study both a space of networks – emphasizing different internal models – and a space of tasks – emphasizing different spatial regularities.

Original: To allow a systematic approach, we design a toy navigation problem, inspired by the Morris water maze (Morris, 1981).
Revised: To allow a systematic approach, we design a toy navigation problem, inspired by the Morris water maze (Morris, 1981).

Original: An agent is placed in a random position in a discretized square arena (size 15), and has to locate the reward location (yellow square, Fig 1A), while only receiving input (empty/wall/reward) from the 8 neighboring positions.
Revised: An agent is placed in a random position in a discretized square arena (size 15), and has to locate the reward location (yellow square, Fig 1A), while only receiving input (empty/wall/reward) from the 8 neighboring positions.

Original: The reward is placed in one of two possible locations in the room according to an external context signal, and the agent can move in one of the four cardinal directions.
Revised: The reward is placed in one of two possible locations in the room according to an external context signal, and the agent can move in one of the four cardinal directions.

Original: At every trial, the agent is placed in a random position in the arena, and the network’s internal state is randomly initialized as well.
Revised: At every trial, the agent is placed in a random position in the arena, and the network’s internal state is randomly initialized as well.

Original: The platform location is constant across trials for each context (see Methods).
Revised: The platform location is constant across trials for each context (see Methods).

Original: The agent is controlled by a RNN that receives the proximal sensory input, as well as a feedback of its own chosen action (Fig. 1B).
Revised: The agent is controlled by a RNN that receives the proximal sensory input, as well as a feedback of its own chosen action (Fig. 1B).

Original: The network’s output is a value for each of the 4 possible actions, the maximum of which is chosen to update the agent’s position.
Revised: The network’s output is a value for each of the 4 possible actions, the maximum of which is chosen to update the agent’s position.

Original: We use a vanilla RNN (see
Revised: We use a vanilla RNN (see

Original: Appendix for LSTM units) described by:
Revised: Appendix for LSTM units) described by:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where h t is the activity of neurons in the networks(512 neurons as default), W is connectivity matrix, τ is a timescale of update.
Revised: where h t is the activity of neurons in the networks(512 neurons as default), W is connectivity matrix, τ is a timescale of update.

Original: The sensory input f ( z t ) is fed through connections matrix W s , and action feedback is fed through W a .
Revised: The sensory input f ( z t ) is fed through connections matrix W s , and action feedback is fed through W a .

Original: The context signal C t is fed through matrix W c .
Revised: The context signal C t is fed through matrix W c .

Original: The network outputs a Q function, which is computed by a linear transformation of its hidden state.
Revised: The network outputs a Q function, which is computed by a linear transformation of its hidden state.

Original: Beyond the basic setting (Fig.
Revised: Beyond the basic setting (Fig.

Original: 1A), we design several variants of the task to emphasize different statistical regularities (Fig. 1C).
Revised: 1A), we design several variants of the task to emphasize different statistical regularities (Fig. 1C).

Original: In all cases, the agent begins from a random position and has to reach the context-dependent reward location in the shortest time using only proximal input.
Revised: In all cases, the agent begins from a random position and has to reach the context-dependent reward location in the shortest time using only proximal input.

Original: The ”Hole” variant introduces a random placement of obstacles (different numbers and positions) in each trial.
Revised: The ”Hole” variant introduces a random placement of obstacles (different numbers and positions) in each trial.

Original: The ”Bar” variant introduces a horizontal or vertical bar in random positions in each trial.
Revised: The ”Bar” variant introduces a horizontal or vertical bar in random positions in each trial.

Original: The various ”Scale” tasks stretch the arena in the horizontal or vertical direction while maintaining the relative position of the rewards.
Revised: The various ”Scale” tasks stretch the arena in the horizontal or vertical direction while maintaining the relative position of the rewards.

Original: The ”Implicit context” task is similar to the basic setting, but the external context input is eliminated, and instead, the color of the walls indicates the reward position.
Revised: The ”Implicit context” task is similar to the basic setting, but the external context input is eliminated, and instead, the color of the walls indicates the reward position.

Original: For all these tasks, the agent needs to ﬁnd a strategy that tackles the uncertain elements to achieve the goals.
Revised: For all these tasks, the agent needs to ﬁnd a strategy that tackles the uncertain elements to achieve the goals.

Original: Despite the simple setting of the game, the tasks are not trivial due to identical visual inputs in most of the locations and various uncertain elements adding to the task difﬁculty.
Revised: Despite the simple setting of the game, the tasks are not trivial due to identical visual inputs in most of the locations and various uncertain elements adding to the task difﬁculty.

Original: 3. T RAINING FRAMEWORK
Revised: 3. T RAINING FRAMEWORK

Original: We aim to understand the interaction between internal representation and the statistical regularities of the various tasks.
Revised: We aim to understand the interaction between internal representation and the statistical regularities of the various tasks.

Original: In principle, this could be accomplished by end-to-end reinforcement learning of many tasks, using various hyper-parameters to allow different solutions to the same task.
Revised: In principle, this could be accomplished by end-to-end reinforcement learning of many tasks, using various hyper-parameters to allow different solutions to the same task.

Original: We opted for a different approach - both due to computation efﬁciency (see Appendix III) and due to biological motivations.
Revised: We opted for a different approach - both due to computation efﬁciency (see Appendix III) and due to biological motivations.

Original: A biological agent acquires navigation ability during evolution and development, which shapes its elementary cognitive ability such as spatial or object memory.
Revised: A biological agent acquires navigation ability during evolution and development, which shapes its elementary cognitive ability such as spatial or object memory.

Original: This shaping provides a scaffold upon which the animal could adapt and learn quickly to perform diverse tasks during life.
Revised: This shaping provides a scaffold upon which the animal could adapt and learn quickly to perform diverse tasks during life.

Original: Similarly, we divide learning into two phases, a pre-training phase that is task agnostic and a Q learning phase that is task-speciﬁc (Fig. 2A).
Revised: Similarly, we divide learning into two phases, a pre-training phase that is task agnostic and a Q learning phase that is task-speciﬁc (Fig. 2A).

Original: During pre-training we modify the network’s internal and input connectivity, while Q learning only modiﬁes the output.
Revised: During pre-training we modify the network’s internal and input connectivity, while Q learning only modiﬁes the output.

Original: Pre-training is implemented in an environment similar to the basic task, with an arena size chosen randomly between 10 to 20.
Revised: Pre-training is implemented in an environment similar to the basic task, with an arena size chosen randomly between 10 to 20.

Original: The agent’s actions are externally determined as a correlated random walk, instead of being internally generated by the agent.
Revised: The agent’s actions are externally determined as a correlated random walk, instead of being internally generated by the agent.

Original: Inspired by neurophysiological ﬁndings, we emphasize two different aspects of internal representation - landmark memory (Identity of the last encountered wall) and position encoding (O’Keefe & Nadel, 1978).
Revised: Inspired by neurophysiological ﬁndings, we emphasize two different aspects of internal representation - landmark memory (Identity of the last encountered wall) and position encoding (O’Keefe & Nadel, 1978).

Original: We thus pre-train the internal connectivity to generate an ensemble of networks with various hyperparameters that control the relative importance of these two aspects, as well as which parts of the connectivity W, W a , W i are modiﬁed.
Revised: We thus pre-train the internal connectivity to generate an ensemble of networks with various hyperparameters that control the relative importance of these two aspects, as well as which parts of the connectivity W, W a , W i are modiﬁed.

Original: We term networks emphasizing the two aspects respectively MemNet and PosNet, and call the naive random network RandNet (Fig.
Revised: We term networks emphasizing the two aspects respectively MemNet and PosNet, and call the naive random network RandNet (Fig.

Original: 2A).
Revised: 2A).

Original: This is done by stochastic gradient descent on the following objective function:
Revised: This is done by stochastic gradient descent on the following objective function:

Original: [Equation]
Revised: [Equation]

Original: with z = ( x, y ) for position, I for landmark memory (identity of the last wall encountered), A for action.
Revised: with z = ( x, y ) for position, I for landmark memory (identity of the last wall encountered), A for action.

Original: The term on action serves as a regularizer.
Revised: The term on action serves as a regularizer.

Original: The three probability distributions are estimated from hidden states of the RNN, given by:
Revised: The three probability distributions are estimated from hidden states of the RNN, given by:

Original: [Equation]
Revised: [Equation]

Original: where W m , W p , W a are readout matrices from hidden states and [ h t − 1 , h t ] denotes the concatenation of last and current hidden states.
Revised: where W m , W p , W a are readout matrices from hidden states and [ h t − 1 , h t ] denotes the concatenation of last and current hidden states.

Original: Tables 1,2,3 in the Appendix show the hyperparameter choices for all networks.
Revised: Tables 1,2,3 in the Appendix show the hyperparameter choices for all networks.

Original: The ratio between α and β controls the tradeoff between position and memory.
Revised: The ratio between α and β controls the tradeoff between position and memory.

Original: The exact values of the hyperparameters were found through trial and error.
Revised: The exact values of the hyperparameters were found through trial and error.

Original: Having obtained this ensemble of networks, we use a Q-learning algorithm with TD-lambda update for the network’s outputs, which are Q values.
Revised: Having obtained this ensemble of networks, we use a Q-learning algorithm with TD-lambda update for the network’s outputs, which are Q values.

Original: We utilize the fact that only the readout matrix W o is trained to use a recursive least square method which allows a fast update of weights for different tasks (Sussillo & Abbott, 2009).
Revised: We utilize the fact that only the readout matrix W o is trained to use a recursive least square method which allows a fast update of weights for different tasks (Sussillo & Abbott, 2009).

Original: This choice leads to a much better convergence speed when compared to stochastic gradient descent.
Revised: This choice leads to a much better convergence speed when compared to stochastic gradient descent.

Original: The update rule used is:
Revised: The update rule used is:

Original: [Equation]
Revised: [Equation]

Original: where H is a matrix of hidden states over 120 time steps, αI is a regularizer and λ controls forgetting rate of past data.
Revised: where H is a matrix of hidden states over 120 time steps, αI is a regularizer and λ controls forgetting rate of past data.

Original: We then analyze the test performance of all networks on all tasks (Figure 2B and Table 3 in appendix).
Revised: We then analyze the test performance of all networks on all tasks (Figure 2B and Table 3 in appendix).

Original: Figure 2B,C show that there are correlations between different tasks and between different networks.
Revised: Figure 2B,C show that there are correlations between different tasks and between different networks.

Original: We quantify this correlation structure by performing principal component analysis of the performance matrix.
Revised: We quantify this correlation structure by performing principal component analysis of the performance matrix.

Original: We ﬁnd that the ﬁrst two PCs in task space explain 79% of the variance.
Revised: We ﬁnd that the ﬁrst two PCs in task space explain 79% of the variance.

Original: The ﬁrst component corresponds to the difﬁculty (average performance) of each task, while the coefﬁcients of the second component are informative regarding the nature of the tasks (Fig.
Revised: The ﬁrst component corresponds to the difﬁculty (average performance) of each task, while the coefﬁcients of the second component are informative regarding the nature of the tasks (Fig.

Original: 2B, right):
Revised: 2B, right):

Original: Bar (-0.49), Hole(-0.25), Basic(-0.21), Implicit context (-0.12), ScaleX (0.04), ScaleY (0.31), Scale (0.74).
Revised: Bar (-0.49), Hole(-0.25), Basic(-0.21), Implicit context (-0.12), ScaleX (0.04), ScaleY (0.31), Scale (0.74).

Original: We speculate these numbers characterize the importance of two different invariances inherent in the tasks.
Revised: We speculate these numbers characterize the importance of two different invariances inherent in the tasks.

Original: Negative coefﬁcients correspond to metric invariance.
Revised: Negative coefﬁcients correspond to metric invariance.

Original: For example, when overcoming dynamic obstacles, the position remains invariant.
Revised: For example, when overcoming dynamic obstacles, the position remains invariant.

Original: This type of task was fundamental to establish metric cognitive maps in neuroscience (O’Keefe & Nadel, 1978).
Revised: This type of task was fundamental to establish metric cognitive maps in neuroscience (O’Keefe & Nadel, 1978).

Original: Positive coefﬁcients correspond to topological invariance, deﬁned as the relation between landmarks unaffected by the metric information.
Revised: Positive coefﬁcients correspond to topological invariance, deﬁned as the relation between landmarks unaffected by the metric information.

Original: Observing the behavior of networks for the extreme tasks of this axis indeed conﬁrms the speculation.
Revised: Observing the behavior of networks for the extreme tasks of this axis indeed conﬁrms the speculation.

Original: Fig.
Revised: Fig.

Original: 3A shows that the successful agent overcomes the scaling task by ﬁnding a set of actions that captures the relations between landmarks and reward, thus generalizing to larger size arenas.
Revised: 3A shows that the successful agent overcomes the scaling task by ﬁnding a set of actions that captures the relations between landmarks and reward, thus generalizing to larger size arenas.

Original: Fig3B shows that the successful agent in the bar task uses a very different strategy.
Revised: Fig3B shows that the successful agent in the bar task uses a very different strategy.

Original: An agent that captures the metric invariance could adjust trajectories and reach the reward each time when the obstacle is changed.
Revised: An agent that captures the metric invariance could adjust trajectories and reach the reward each time when the obstacle is changed.

Original: This ability is often related to the ability to use shortcuts (O’Keefe & Nadel, 1978).
Revised: This ability is often related to the ability to use shortcuts (O’Keefe & Nadel, 1978).

Original: The other tasks intepolate between the two extremes, due to the presence of both elements in the tasks.
Revised: The other tasks intepolate between the two extremes, due to the presence of both elements in the tasks.

Original: For instance, the implicit context task requires the agent to combine landmark memory (color of the wall) with position to locate the reward.
Revised: For instance, the implicit context task requires the agent to combine landmark memory (color of the wall) with position to locate the reward.

Original: We thus deﬁne metric and topological scores by using a weighted average of task performance using negative and positive coefﬁcients respectively.
Revised: We thus deﬁne metric and topological scores by using a weighted average of task performance using negative and positive coefﬁcients respectively.

Original: Fig.
Revised: Fig.

Original: 3C shows the various networks measured by the two scores.
Revised: 3C shows the various networks measured by the two scores.

Original: We see that random networks (blue) can achieve reasonable performance with some hyperparameter choices, but they are balanced with respect to the metric topological score.
Revised: We see that random networks (blue) can achieve reasonable performance with some hyperparameter choices, but they are balanced with respect to the metric topological score.

Original: On the other hand, PostNet networks are pushed to the metric side and MemNet networks to the topological side.
Revised: On the other hand, PostNet networks are pushed to the metric side and MemNet networks to the topological side.

Original: This result indicates that the inductive bias achieved via task agnostic pre-training is manifested in the performance of networks on various navigation tasks
Revised: This result indicates that the inductive bias achieved via task agnostic pre-training is manifested in the performance of networks on various navigation tasks

Original: 3. L INKING REPRESENTATION TO DYNAMICS
Revised: 3. L INKING REPRESENTATION TO DYNAMICS

Original: What are the underlying structures of different networks that encode the bias for different tasks?
Revised: What are the underlying structures of different networks that encode the bias for different tasks?

Original: We approach this question by noting that RNNs are nonlinear dynamical systems.
Revised: We approach this question by noting that RNNs are nonlinear dynamical systems.

Original: As such, it is informative to detect ﬁxed points and other special areas of phase space to better understand their dynamics.
Revised: As such, it is informative to detect ﬁxed points and other special areas of phase space to better understand their dynamics.

Original: For instance, a network that memorizes a binary value might be expected to contain two discrete ﬁxed points (Fig. 4A).
Revised: For instance, a network that memorizes a binary value might be expected to contain two discrete ﬁxed points (Fig. 4A).

Original: A network that integrates a continuous value might contain a line attractorKim et al.
Revised: A network that integrates a continuous value might contain a line attractorKim et al.

Original: (2017); Kakaria & de Bivort (2017), and , a network that integrates position might contain a plane attractor – a 2D manifold of ﬁxed points – because this would enable updating x and y coordinates with actions, and maintaining the current position in the absence of action (Burak &
Revised: (2017); Kakaria & de Bivort (2017), and , a network that integrates position might contain a plane attractor – a 2D manifold of ﬁxed points – because this would enable updating x and y coordinates with actions, and maintaining the current position in the absence of action (Burak &

Original: Fiete, 2009).
Revised: Fiete, 2009).

Original: Trained networks, however, often converge to approximate ﬁxed points (slow points)
Revised: Trained networks, however, often converge to approximate ﬁxed points (slow points)

Original: (Sussillo & Barak; Mante et al., 2013; Maheswaranathan et al., 2019), as they are not required to maintain the same position for an inﬁnite time.
Revised: (Sussillo & Barak; Mante et al., 2013; Maheswaranathan et al., 2019), as they are not required to maintain the same position for an inﬁnite time.

Original: We thus expect the relevant slow points to be somewhere between the actual trajectories and true ﬁxed points.
Revised: We thus expect the relevant slow points to be somewhere between the actual trajectories and true ﬁxed points.

Original: We detect these areas of phase space using adaptations of existing techniques (Appendix 5.3, (Sussillo & Barak)).
Revised: We detect these areas of phase space using adaptations of existing techniques (Appendix 5.3, (Sussillo & Barak)).

Original: Brieﬂy, we drive the agent to move in the environment, while recording its position and last seen landmark (wall).
Revised: Brieﬂy, we drive the agent to move in the environment, while recording its position and last seen landmark (wall).

Original: This results in a collection of hidden states.
Revised: This results in a collection of hidden states.

Original: Then, for each point in this collection, we relax the dynamics towards approximate ﬁxed points.
Revised: Then, for each point in this collection, we relax the dynamics towards approximate ﬁxed points.

Original: This procedure results in points with different hidden state velocities for the three networks (Fig.
Revised: This procedure results in points with different hidden state velocities for the three networks (Fig.

Original: 4B) – RandNet does not seem to have any slow points, while PosNet and MemNet do, with MemNet’s points being slower.
Revised: 4B) – RandNet does not seem to have any slow points, while PosNet and MemNet do, with MemNet’s points being slower.

Original: The resulting manifold of slow points for a typical PosNet is depicted in Figure 4C, along with the labels of position and stimulus from which relaxation began.
Revised: The resulting manifold of slow points for a typical PosNet is depicted in Figure 4C, along with the labels of position and stimulus from which relaxation began.

Original: It is apparent that pretraining has created in PosNet a smooth representation of position along this manifold.
Revised: It is apparent that pretraining has created in PosNet a smooth representation of position along this manifold.

Original: The MemNet manifold represents landmark memory as 4 distinct ﬁxed points without a spatial representation.
Revised: The MemNet manifold represents landmark memory as 4 distinct ﬁxed points without a spatial representation.

Original: Note that despite the dominance of position representation in
Revised: Note that despite the dominance of position representation in

Original: PosNet, landmark memory still modulates this representation (Fig 3A, M) - showing that pretraining did not result in a perfect plane attractor, but rather in an approximate collection of 4 plane attractors (Fig.
Revised: PosNet, landmark memory still modulates this representation (Fig 3A, M) - showing that pretraining did not result in a perfect plane attractor, but rather in an approximate collection of 4 plane attractors (Fig.

Original: 3D, MP).
Revised: 3D, MP).

Original: This conjunctive representation can also be appreciated by considering the decoding accuracy of trajectories conditioned on the number of wall encounters.
Revised: This conjunctive representation can also be appreciated by considering the decoding accuracy of trajectories conditioned on the number of wall encounters.

Original: As the agent encounters the wall, the decoding of position from the manifold improves, implying the ability to integrate path integration and landmark memory (Fig Appendix 8).
Revised: As the agent encounters the wall, the decoding of position from the manifold improves, implying the ability to integrate path integration and landmark memory (Fig Appendix 8).

Original: We thus see that the pre-training biases are implemented by distinct attractor landscapes, from which we could see both qualitative differences between networks and a trade-off between landmark memory and position encoding.
Revised: We thus see that the pre-training biases are implemented by distinct attractor landscapes, from which we could see both qualitative differences between networks and a trade-off between landmark memory and position encoding.

Original: The continuous attractors of PosNet correspond to a metric representation of space, albeit modulated by landmark memory.
Revised: The continuous attractors of PosNet correspond to a metric representation of space, albeit modulated by landmark memory.

Original: The discrete attractors of MemNet encode the landmark memory in a robust manner, while sacriﬁcing position encoding.
Revised: The discrete attractors of MemNet encode the landmark memory in a robust manner, while sacriﬁcing position encoding.

Original: The untrained RandNet, on the other hand, has no clear structure, and relies on a short transient memory of the last landmark.
Revised: The untrained RandNet, on the other hand, has no clear structure, and relies on a short transient memory of the last landmark.

Original: Coloring is according to either X,Y coordinates or the identity of the last wall encountered (landmark memory, M).
Revised: Coloring is according to either X,Y coordinates or the identity of the last wall encountered (landmark memory, M).

Original: Note how the position is smoothly encoded on the manifold for PosNet, and memory is encoded by four discrete points for MemNet.
Revised: Note how the position is smoothly encoded on the manifold for PosNet, and memory is encoded by four discrete points for MemNet.

Original: The memory panels show a ﬁt of a plane to the X,Y coordinates, conditioned upon a given landmark memory – showing that PosNet also has memory information, and not just position.
Revised: The memory panels show a ﬁt of a plane to the X,Y coordinates, conditioned upon a given landmark memory – showing that PosNet also has memory information, and not just position.

Original: The networks used are 1,13,20 from Table 3.
Revised: The networks used are 1,13,20 from Table 3.

Original: The above analysis was performed on three typical networks and is somewhat time-consuming.
Revised: The above analysis was performed on three typical networks and is somewhat time-consuming.

Original: In order to get a broader view of internal representations in all networks, we use a simple measure of the components of the representation.
Revised: In order to get a broader view of internal representations in all networks, we use a simple measure of the components of the representation.

Original: Speciﬁcally, we drove the agent to move in an arena of inﬁnite size that was empty except a single wall (of a different identity in each trial).
Revised: Speciﬁcally, we drove the agent to move in an arena of inﬁnite size that was empty except a single wall (of a different identity in each trial).

Original: We then used GLM (generalized linear model) to determine the variance explained by both position and the identity of the wall encountered from the network’s hidden state.
Revised: We then used GLM (generalized linear model) to determine the variance explained by both position and the identity of the wall encountered from the network’s hidden state.

Original: Figure 6A shows these two measures for all the networks.
Revised: Figure 6A shows these two measures for all the networks.

Original: The results echo those measured with the battery of 7 tasks (Fig.
Revised: The results echo those measured with the battery of 7 tasks (Fig.

Original: 3C), but are orders of magnitude faster to compute.
Revised: 3C), but are orders of magnitude faster to compute.

Original: Indeed, if we correlate these measures with performance on the different tasks, we see that they correspond to the metric-topological axis as deﬁned by PCA (Fig.
Revised: Indeed, if we correlate these measures with performance on the different tasks, we see that they correspond to the metric-topological axis as deﬁned by PCA (Fig.

Original: 5B, compare with Fig.
Revised: 5B, compare with Fig.

Original: 2B, right).
Revised: 2B, right).

Original: 3. A MODULAR SYSTEM THAT COMBINES ADVANTAGES OF BOTH DYNAMICS
Revised: 3. A MODULAR SYSTEM THAT COMBINES ADVANTAGES OF BOTH DYNAMICS

Original: Altogether, we showed a differential ability of networks to cope with different environmental regularities via inductive bias encoded in their dynamics.
Revised: Altogether, we showed a differential ability of networks to cope with different environmental regularities via inductive bias encoded in their dynamics.

Original: Considering the tradeoff is a fundamental property of single module RNN, it is natural to ask if we could combine advantages from both dynamics into a modular system.
Revised: Considering the tradeoff is a fundamental property of single module RNN, it is natural to ask if we could combine advantages from both dynamics into a modular system.

Original: Inspired by Park et al.
Revised: Inspired by Park et al.

Original: We design a hierarchical system composed of the representation layer on the bottom and selector layer on top.
Revised: We design a hierarchical system composed of the representation layer on the bottom and selector layer on top.

Original: The representation layer concatenates PosNet and MemNet modules together, each evaluating action values according to its own dynamics.
Revised: The representation layer concatenates PosNet and MemNet modules together, each evaluating action values according to its own dynamics.

Original: The second layer selects the more reliable module based on the combined representation by assigning a value (reliability) to each module.
Revised: The second layer selects the more reliable module based on the combined representation by assigning a value (reliability) to each module.

Original: The module with maximal reliability makes the ﬁnal decision.
Revised: The module with maximal reliability makes the ﬁnal decision.

Original: Thus, the control of the agent shifts between different dynamics according to current input or history.
Revised: Thus, the control of the agent shifts between different dynamics according to current input or history.

Original: The modular system signiﬁcantly shifts the metric-topological balance (Fig2C, Fig3B).
Revised: The modular system signiﬁcantly shifts the metric-topological balance (Fig2C, Fig3B).

Original: The reliability V is learned similarly as Q(Appendix 5.7).
Revised: The reliability V is learned similarly as Q(Appendix 5.7).

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Our work explores how internal representations for navigation tasks are implemented by the dynamics of recurrent neural networks.
Revised: Our work explores how internal representations for navigation tasks are implemented by the dynamics of recurrent neural networks.

Original: We show that pre-training networks in a task-agnostic manner can shape their dynamics into discrete ﬁxed points or into a low-D manifold of slow points.
Revised: We show that pre-training networks in a task-agnostic manner can shape their dynamics into discrete ﬁxed points or into a low-D manifold of slow points.

Original: These distinct dynamical objects correspond to landmark memory and spatial memory respectively.
Revised: These distinct dynamical objects correspond to landmark memory and spatial memory respectively.

Original: When performing Q learning for speciﬁc tasks, these dynamical objects serve as priors for the network’s representations and shift its performance on the various navigation tasks.
Revised: When performing Q learning for speciﬁc tasks, these dynamical objects serve as priors for the network’s representations and shift its performance on the various navigation tasks.

Original: Here we show that both plane attractors and discrete attractors are useful.
Revised: Here we show that both plane attractors and discrete attractors are useful.

Original: It would be interesting to see whether and how other dynamical objects can serve as inductive biases for other domains.
Revised: It would be interesting to see whether and how other dynamical objects can serve as inductive biases for other domains.

Original: In tasks outside of reinforcement learning, for instance, line attractors were shown to underlie network computations (Mante et al., 2013; Maheswaranathan et al., 2019).
Revised: In tasks outside of reinforcement learning, for instance, line attractors were shown to underlie network computations (Mante et al., 2013; Maheswaranathan et al., 2019).

Original: An agent that has to perform several navigation tasks will require both types of representations.
Revised: An agent that has to perform several navigation tasks will require both types of representations.

Original: A single recurrent network, however, has a trade-off between adapting to one type of task or to another.
Revised: A single recurrent network, however, has a trade-off between adapting to one type of task or to another.

Original: As for possible reasons for the tradeoff - the hidden states could represent the current position, stimulus memory, and other ”irrelevant” features.

Original: Accurate path integration implies an ability to marginalize over stimulus memory and other features to obtain a clean version of the position.

Original: In principle, accurate decoding of the position could be done even if the hidden state contains all the other information.

Original: However, perhaps a more ”natural” way is to allow the dynamics to suppress these aspects of the representation, and thus facilitate readout of the desired position information.

Original: Furthermore, the attractor landscape picture provides a  dynamical reason for the tradeoff.
Revised: The attractor landscape picture provides a possible dynamical reason for the tradeoff.

Original: Position requires a continuous attractor, whereas stimulus memory requires discrete attractors.
Revised: Position requires a continuous attractor, whereas stimulus memory requires discrete attractors.

Original: It is possible to have four separated plane attractors, but perhaps easier  to converge to one or the other.
Revised: While it is possible to have four separated plane attractors, it is perhaps easier for learning to converge to one or the other.

Original: It is natural to ask whether we could build a system capable of both. We showed that  a modular system composed of RNNs with different dynamics, each optimized for one task category, is more ﬂexible than any single-module network for multiple-task learning.
Revised: A different solution to learn multiple tasks is by considering multiple modules, each optimized for a different dynamical regime. We showed that such a modular system is able to learn multiple tasks, in a manner that is more ﬂexible than any single-module network we could train.

Original: Pre-training alters network connectivity.
Revised: Pre-training alters network connectivity.

Original: The resulting connectivity is expected to be between random networks (Luko ˇ sevi ˇ cius & Jaeger, 2009) and designed ones (Burak & Fiete, 2009).
Revised: The resulting connectivity is expected to be between random networks (Lukoˇseviˇcius & Jaeger, 2009) and designed ones (Burak & Fiete, 2009).

Original: It is perhaps surprising that even the untrained RandNet can perform some of the navigation tasks using only Qlearning of the readout (with appropriate hyperparameters, see Tables 2,3 and section 4 ”Linking dynamics to connectivity” in Appendix).
Revised: It is perhaps surprising that even the untrained RandNet can perform some of the navigation tasks using only Qlearning of the readout (with appropriate hyperparameters, see Tables 2,3 and section 4 ”Linking dynamics to connectivity” in Appendix).

Original: This is consistent with recent work showing that some architectures can perform various tasks without learning (Gaier & Ha, 2019).
Revised: This is consistent with recent work showing that some architectures can perform various tasks without learning (Gaier & Ha, 2019).

Original: Studying the connectivity changes due to pre-training may help understand the statistics from which to draw better random networks (Appendix section 4).
Revised: Studying the connectivity changes due to pre-training may help understand the statistics from which to draw better random networks (Appendix section 4).

Original: Apart from improving the understanding of representation and dynamics, it is interesting to consider the efﬁciency of our two-stage learning compared to standard approaches.
Revised: Apart from improving the understanding of representation and dynamics, it is interesting to consider the efﬁciency of our two-stage learning compared to standard approaches.

Original: We found that end-toend training is much slower, cannot learn topological tasks and has weaker transfer between tasks (See Appendix section 5.2).
Revised: We found that end-toend training is much slower, cannot learn topological tasks and has weaker transfer between tasks (See Appendix section 5.2).

Original: Thus it is interesting to explore whether this approach could be used to accelerate learning in other domains, similar to curriculum learning (Bengio et al., 2009). 
Revised: Thus it is interesting to explore whether this approach could be used to accelerate learning in other domains, similar to curriculum learning (Bengio et al., 2009). A CKNOWLEDGMENTS

Original:  A PPENDIX 5. P ERFORMANCE MEASURE FOR EACH TASK
Revised:  A PPENDIX 5. P ERFORMANCE MEASURE FOR EACH TASK

Original: When testing the agent on a task, we perform a trial for each possible initial position of the agent.
Revised: When testing the agent on a task, we perform a trial for each possible initial position of the agent.

Original: Note that the hidden state is randomly initialized in each trial, so this is not an exhaustive search of all possible trial types.
Revised: Note that the hidden state is randomly initialized in each trial, so this is not an exhaustive search of all possible trial types.

Original: We then measure the time it takes the agent to reach the target.
Revised: We then measure the time it takes the agent to reach the target.

Original: This time is normalized by an approximate optimal strategy – moving from the initial position to a corner of the arena (providing x y information), and then heading straight to the reward.
Revised: This time is normalized by an approximate optimal strategy – moving from the initial position to a corner of the arena (providing x y information), and then heading straight to the reward.

Original: If the agent fails to reach the target after 120 steps, the trial score is − 1 :
Revised: If the agent fails to reach the target after 120 steps, the trial score is − 1 :

Original: [Equation]
Revised: [Equation]

Original: 5. P ERFORMANCE COMPARED TO OTHER APPROACHES
Revised: 5. T WO - STAGE VERSUS END - TO - END LEARNING

Original: To check how effective our pre-training methods compared to other methods, we take different learning protocols and network architectures for comparison.
Revised: To check the effectiveness of our two-stage learning (pre-training followed by Q-Learning), we contrast it with an end-to-end approach. We considered two approached to training: a classic deep Q learning version Mnih et al. (2013) and a method adapted from RDPG Heess et al.

Original: We use an end to end approach with both vanilla RNN and LSTM(section 5.6) against our pre-training method for various tasks.
Revised: The naive Q learning is separated into a play phase and training phase. During the play phase the agent collects experience as s t , a t , h t r t into a replay buffer. The action value Q is computed through TD lambda method as Q ( h, a ) as target function.

Revised: During the training phase the agent samples the past experience through replay buffer perform gradient descent to minimize the difference between expected Q ( h, a ) and target Q.

Revised: We found that for most tasks the deep Q learning method was better than the adapted RDPG, and thus used it for our benchmarks.

Revised: We used both LSTM and vanilla RNN for these test.

Original: In the basic task, all networks achieve optimal performance, but our approach is signiﬁcantly more data-efﬁcient even with random networks (ﬁg. 6A ).
Revised: We found that, for the basic task, all networks achieve optimal performance, but our approach is signiﬁcantly more data-efﬁcient even with random networks (ﬁg. 6A).

Original: For all topological tasks, the end to end approach fails with both vanilla RNN and LSTM(ﬁg 6C table 3).
Revised: For all topological tasks, the end-to-end approach fails with both vanilla RNN and LSTM (ﬁg 6C table 3).

Original: The end to end approach performs relatively well in metric tasks except for  implicit context task(ﬁg 6B table 3), which arrives as similar performance as PosNet but with a much slower converging speed ( 10 5 trials vs 10 4 trials).
Revised: The end to end approach performs relatively well in metric tasks, except for the implicit context task (ﬁg 6B table 3), which converges to a similar performance as PosNet but with a much slower convergence speed ( 10 5 trials vs 10 4 trials).

Original: For the end-to-end approaches, a critical question is whether an internal representation emerges, which enables better performance in similar tasks.
Revised: For the end-to-end approaches, a critical question is whether an internal representation emerges, which enables better performance in similar tasks.

Original: For instance, do networks that were successfully end-to-end trained in the basic task develop a representation that facilitates learning the bar task?
Revised: For instance, do networks that were successfully end-to-end trained in the basic task develop a representation that facilitates learning the bar task?

Original: To answer this question, we use networks that were end-to-end trained on one task and use them as a basis for RLS Q-learning of a different task.
Revised: To answer this question, we use networks that were end-to-end trained on one task and use them as a basis for RLS Q-learning of a different task.

Original: This allows comparison with the pre-trained networks.
Revised: This allows comparison with the pre-trained networks.

Original: Figure 7 shows that pre-training provides a better substrate for subsequent Q-learning - even when considering generalization within metric tasks.
Revised: Figure 7 shows that pre-training provides a better substrate for subsequent Q-learning - even when considering generalization within metric tasks.

Original: For the implicit context task, the difference is even greater.
Revised: For the implicit context task, the difference is even greater.

Original: 5. E XPLORING THE LOW D NETWORK DYNAMICS
Revised: 5. E XPLORING THE LOW D NETWORK DYNAMICS

Original: Recurrent neural networks are nonlinear dynamical systems.
Revised: Recurrent neural networks are nonlinear dynamical systems.

Original: As such, they behave differently in different areas of phase space.
Revised: As such, they behave differently in different areas of phase space.

Original: It is often informative to locate ﬁxed points of the dynamics, and use their local dynamics as anchors to understand global dynamics.
Revised: It is often informative to locate ﬁxed points of the dynamics, and use their local dynamics as anchors to understand global dynamics.

Original: When considering trained RNNs, it is reasonable to expect approximate ﬁxed points rather than exact ones.
Revised: When considering trained RNNs, it is reasonable to expect approximate ﬁxed points rather than exact ones.

Original: This is because a ﬁxed point corresponds to maintaining the same hidden state for inﬁnite time, whereas a trained network is only exposed to a ﬁnite time.
Revised: This is because a ﬁxed point corresponds to maintaining the same hidden state for inﬁnite time, whereas a trained network is only exposed to a ﬁnite time.

Original: These slow points (Sussillo & Barak; Mante et al., 2013) can be detected in several manners (Sussillo & Barak; Katz & Reggia, 2017).
Revised: These slow points (Sussillo & Barak; Mante et al., 2013) can be detected in several manners (Sussillo & Barak; Katz & Reggia, 2017).

Original: For the case of stable ﬁxed points (attractors), it is also possible to simulate the dynamics until convergence.
Revised: For the case of stable ﬁxed points (attractors), it is also possible to simulate the dynamics until convergence.

Original: In our setting, we opt for the latter option.
Revised: In our setting, we opt for the latter option.

Original: Because the agent never stays in the same place, we relax the dynamics towards attractors by providing as action feedback the average of all 4 actions.
Revised: Because the agent never stays in the same place, we relax the dynamics towards attractors by providing as action feedback the average of all 4 actions.

Original: The relevant manifold (e.g. a plane attractor) might contain areas that are more stable than others (for instance a few true ﬁxed points), but we want to avoid detecting only these areas.
Revised: The relevant manifold (e.g. a plane attractor) might contain areas that are more stable than others (for instance a few true ﬁxed points), but we want to avoid detecting only these areas.

Original: We thus search for the relevant manifold in the following manner.
Revised: We thus search for the relevant manifold in the following manner.

Original: We drive the agent to move in the environment, while recording its position and last seen stimulus (wall).
Revised: We drive the agent to move in the environment, while recording its position and last seen stimulus (wall).

Original: This results in a collection of hidden states, labelled by position and stimulus that we term the m = 0 manifold.
Revised: This results in a collection of hidden states, labelled by position and stimulus that we term the m = 0 manifold.

Original: For each point on the manifold, we continue simulating the dynamics for m extra steps while providing as input the average of all 4 actions, resulting in further m (cid:54) = 0 manifolds.
Revised: For each point on the manifold, we continue simulating the dynamics for m extra steps while providing as input the average of all 4 actions, resulting in further m (cid:54) = 0 manifolds.

Original: If these states are the underlying scaffold for the dynamics, they should encode the position (or memory).
Revised: If these states are the underlying scaffold for the dynamics, they should encode the position (or memory).

Original: We therefore choose m by a cross-validation method – decoding new trajectories obtained in the basic task by using the k = 15 -nearest neighbors in each m -manifold.
Revised: We therefore choose m by a cross-validation method – decoding new trajectories obtained in the basic task by using the k = 15 -nearest neighbors in each m -manifold.

Original: The red curve in Figure 8A shows the resulting decoding accuracy for position using
Revised: The red curve in Figure 8A shows the resulting decoding accuracy for position using

Original: PosNet, where the accuracy starts to fall around m = 25 , indicating that further relaxation leads to irrelevant ﬁxed 
Revised: PosNet, where the accuracy starts to fall around m = 25 , indicating that further relaxation leads to irrelevant ﬁxed points.

Original: 5. A TTRACTOR LANDSCAPE FOR DIFFERENT RECURRENT ARCHITECTURES
Revised: 5. A TTRACTOR LANDSCAPE FOR DIFFERENT RECURRENT ARCHITECTURES

Original: We tested the qualitative shape of the slow manifolds that emerge from pre-training other unit types.
Revised: We tested the qualitative shape of the slow manifolds that emerge from pre-training other unit types.

Original: Speciﬁcally, we pre-trained an LSTM network using the same parameters as PosNet 1 and MemNet 1 (Table 1).
Revised: Speciﬁcally, we pre-trained an LSTM network using the same parameters as PosNet 1 and MemNet 1 (Table 1).

Original: Figures 9 show a qualitatively similar behvior to that described in the main text.
Revised: Figures 9 show a qualitatively similar behvior to that described in the main text.

Original: Note that MemNet has slow regions instead of discrete points, and we suspect discrete attractors might appear with longer relaxation times.
Revised: Note that MemNet has slow regions instead of discrete points, and we suspect discrete attractors might appear with longer relaxation times.

Original: The differences between the columns also demonstrate that slow points, revealed by relaxation, are generally helpful to analyze dynamics of different types of recurrent networks.
Revised: The differences between the columns also demonstrate that slow points, revealed by relaxation, are generally helpful to analyze dynamics of different types of recurrent networks.

Original: 5. P RE - TRAINING PROTOCOLS AND PERFORMANCE OF NETWORKS
Revised: 5. P RE - TRAINING PROTOCOLS AND PERFORMANCE OF NETWORKS

Original: As explained in the main text, pre-training emphasizes decoding of either landmark memory or the position of the agent.
Revised: As explained in the main text, pre-training emphasizes decoding of either landmark memory or the position of the agent.

Original: We used several variants of hyperparameters to pre-train the networks.
Revised: We used several variants of hyperparameters to pre-train the networks.

Original: Equation 17, which is written again for convenience, deﬁnes the relevant parameters:
Revised: Equation 17, which is written again for convenience, deﬁnes the relevant parameters:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The agent was driven to explore an empty arena (with walls) using random actions, with a probability p of changing action (direction) at any step.
Revised: The agent was driven to explore an empty arena (with walls) using random actions, with a probability p of changing action (direction) at any step.

Original: Table 1 shows the protocols (hyperparameters), Table 2 shows the random networks hyperparameters, and table 3 shows the performance of the resulting networks on all tasks.
Revised: Table 1 shows the protocols (hyperparameters), Table 2 shows the random networks hyperparameters, and table 3 shows the performance of the resulting networks on all tasks.

Original: For all pre-training protocols an l2 regularizer of 10 − 6 on internal weights
Revised: For all pre-training protocols an l2 regularizer of 10 − 6 on internal weights

Revised: [Table]

Original: Different hyper parameter for RandNets:
Revised: Different hyper parameter for RandNets:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  to  reservoir computing 
Revised: Number of neurons used 512, time constant τ is taken to be 2, the choice of hyper parameters is according to standard reservoir computing litterature (Jaeger, 2010; Luko ˇ sevi ˇ cius & Jaeger).

Original: Jaeger) weights are taken from a standard Normal distribution.
Revised: The weights are taken from a standard Normal distribution.

Original: It is crucial to choose an appropriate stan [Table] dard deviation for success of training (Appendix section 5), which is summarized in 2, each unit represents 1 / √ N
Revised: It is crucial to choose an appropriate standard deviation for success of training (Appendix section 5), which is summarized in 2, each unit represents 1 / √ N

Original: [Table]
Revised: [Table]

Original: 5. E ND TO END TRAINING PROTOCOL

Original: The end to end training has a classic deep Q learning versionMnih et al.

Original: (2013) and a method adapted from RDPG in Heess et al.

Original: The naive Q learning is separated into a play phase and training phase.

Original: During the play phase the agent collects experience as s t , a t , h t r t into a replay buffer.

Original: The action value Q is computed through TD lambda method as Q ( h, a ) as target function.

Original: During the training phase the agent samples the past experience through replay buffer perform gradient descent to minimize the difference between expected Q ( h, a ) and target Q. For most of tasks the class deep

Original: Q learning method is better than the adapted RDPG.

Original: Thus we take the ﬁrst method as our standard end to end training benchmark.

Original: 5. M ODULAR N ETWORK PROTOCOL
Revised: 5. M ODULAR N ETWORK PROTOCOL

Original: The results of modular network are obtained from combining the PosNet 1 and MemNet 25 from table .
Revised: The results of modular network are obtained from combining the PosNet 1 and MemNet 25 from table .

Original: Both the learning of Q function and V function is learned in the the same way as main results equ (7,8,9, 10).
Revised: Both the learning of Q function and V function is learned in the the same way as main results equ (7,8,9, 10).

Original: 5. L INKING DYNAMICS TO CONNECTIVITY
Revised: 5. L INKING DYNAMICS TO CONNECTIVITY

Original: Pretraining modiﬁed the internal connectivity of the networks.
Revised: Pretraining modiﬁed the internal connectivity of the networks.

Original: Here, we explore the link between connectivity and and dynamics.
Revised: Here, we explore the link between connectivity and and dynamics.

Original: We draw inspiration from two observations in the ﬁeld of reservoir computing (Lukoˇseviˇcius & Jaeger, 2009).
Revised: We draw inspiration from two observations in the ﬁeld of reservoir computing (Lukoˇseviˇcius & Jaeger, 2009).

Original: On the one hand, the norm of the internal connectivity has a large effect on network dynamics and performance, with an advantage to residing on the edge of chaos (Jaeger, 2010).
Revised: On the one hand, the norm of the internal connectivity has a large effect on network dynamics and performance, with an advantage to residing on the edge of chaos (Jaeger, 2010).

Original: On the other hand, restricting learning to the readout weights (which is then fed back to the network, Sussillo & Abbott (2009)) results in a low-rank perturbation to the connectivity, the possible contributions of which were recently explored (Mastrogiuseppe & Ostojic, 2018). 
Revised: On the other hand, restricting learning to the readout weights (which is then fed back to the network, Sussillo & Abbott (2009)) results in a low-rank perturbation to the connectivity, the possible contributions of which were recently explored (Mastrogiuseppe & Ostojic, 2018). We thus analyzed both aspects. Fig.

Revised: 10A shows the norms of several matrices as they evolve through pre-training, showing an opposite trend for PosNet and MemNet with respect to the internal

Original: [Table]
Revised: [Table]

Original: 14.RandNet 0.
Revised: 14.RandNet 0.

Original: 15.RandNet 0.
Revised: 15.RandNet 0.

Original: 16.RandNet 0.
Revised: 16.RandNet 0.

Original: 17.RandNet 0.
Revised: 17.RandNet 0.

Original: 18.RandNet 0.
Revised: 18.RandNet 0.

Original: 19.MemNet 0.
Revised: 19.MemNet 0.

Original: 20.MemNet 0.
Revised: 20.MemNet 0.

Original: 21.MemNet 0.
Revised: 21.MemNet 0.

Original: 22.MemNet 0.
Revised: 22.MemNet 0.

Original: 23.MemNet 0.
Revised: 23.MemNet 0.

Original: 24.MemNet 0.
Revised: 24.MemNet 0.

Original: 25.MemNet 0.
Revised: 25.MemNet 0.

Original: 26.MemNet 0.
Revised: 26.MemNet 0.

Original: 27.MemNet 0.
Revised: 27.MemNet 0.

Original: [Table]
Revised: [Table]

Original: We thus analyzed both aspects. Fig. 10A shows the norms of several matrices as they evolve through pre-training, showing an opposite trend for PosNet and MemNet with respect to the internal connectivity W .
Revised: connectivity W .

Original: To estimate the low-rank component, we performed singular value decomposition on the change to the internal connectivity induced by pre-training (Fig. 10B).
Revised: To estimate the low-rank component, we performed singular value decomposition on the change to the internal connectivity induced by pre-training (Fig. 10B).

Original: [Equation]
Revised: [Equation]

Original: The singular values of the actual change were compared to a shufﬂed version, revealing their lowrank structure (Fig.
Revised: The singular values of the actual change were compared to a shufﬂed version, revealing their lowrank structure (Fig.

Original: 10C,D).
Revised: 10C,D).

Original: Note that pretraining was not constrained to generate such a lowrank perturbation.
Revised: Note that pretraining was not constrained to generate such a lowrank perturbation.

Original: Furthermore, we show that the low-rank structure is partially correlated to the network’s inputs, possibly contributing to their effective ampliﬁcation through the dynamics (Fig.
Revised: Furthermore, we show that the low-rank structure is partially correlated to the network’s inputs, possibly contributing to their effective ampliﬁcation through the dynamics (Fig.

Original: 10E-H).
Revised: 10E-H).

Original: Because we detected both types of connectivity changes (norm and low-rank), we next sought to characterize their relative contributions to network dynamics, representation and behavior.
Revised: Because we detected both types of connectivity changes (norm and low-rank), we next sought to characterize their relative contributions to network dynamics, representation and behavior.

Original: In order to assess the effect of matrix norms, we generated a large number of scaled random matrices and use the GLM analyse in ﬁgure 5 to access its inﬂuence on dynamics.
Revised: In order to assess the effect of matrix norms, we generated a large number of scaled random matrices and use the GLM analyse in ﬁgure 5 to access its inﬂuence on dynamics.

Original: We see the trade-off between landmark memory and path integration is affected by norm (Fig. 10E).
Revised: We see the trade-off between landmark memory and path integration is affected by norm (Fig. 10E).

Original: But the actual numbers, however, are much lower for the scaled random matrices compared to the pretrained ones – indicating the importance of the low-rank component (Fig. 10F) .
Revised: But the actual numbers, however, are much lower for the scaled random matrices compared to the pretrained ones – indicating the importance of the low-rank component (Fig. 10F) .

Original: Indeed, when removing even only the leading 5 ranks from ∆ W , Network encoding and performance on all tasks approaches that of RandNet.
Revised: Indeed, when removing even only the leading 5 ranks from ∆ W , Network encoding and performance on all tasks approaches that of RandNet.

Original: 5. B EHAVIOR OF DIFFERENT NETWORKS
Revised: 5. B EHAVIOR OF DIFFERENT NETWORKS

Original: Different networks develop diverse strategies for metric and topological tasks.
Revised: Different networks develop diverse strategies for metric and topological tasks.

Original: In this section, we give examples of typical trajectories of PosNet, MemNet, RanNet in the basic, bar and scaling tasks
Revised: In this section, we give examples of typical trajectories of PosNet, MemNet, RanNet in the basic, bar and scaling tasks.