Original: H IPPOCAMPAL NEURONAL REPRESENTATIONS IN CONTINUAL LEARNING
Revised: H IPPOCAMPAL NEURONAL REPRESENTATIONS IN CONTINUAL LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The hippocampus has long been associated with spatial memory and goal-directed spatial navigation.
Revised: The hippocampus has long been associated with spatial memory and goal-directed spatial navigation.

Original: However, the region’s independent role in continual learning of navigational strategies has seldom been investigated.
Revised: However, the region’s independent role in continual learning of navigational strategies has seldom been investigated.

Original: Here we analyse populationlevel activity of hippocampal CA1 neurons in the context of continual learning of two different spatial navigation strategies.
Revised: Here we analyse populationlevel activity of hippocampal CA1 neurons in the context of continual learning of two different spatial navigation strategies.

Original: Demixed Principal Component Analysis (dPCA) is applied on neuronal recordings from 612 hippocampal CA1 neurons of rodents learning to perform allocentric and egocentric spatial tasks.
Revised: Demixed Principal Component Analysis (dPCA) is applied on neuronal recordings from 612 hippocampal CA1 neurons of rodents learning to perform allocentric and egocentric spatial tasks.

Original: The components uncovered using dPCA from the ﬁring activity reveal that hippocampal neurons encode relevant task variables such decisions, navigational strategies and reward location.
Revised: The components uncovered using dPCA from the ﬁring activity reveal that hippocampal neurons encode relevant task variables such decisions, navigational strategies and reward location.

Original: We compare this hippocampal features with standard reinforcement learning algorithms, highlighting similarities and differences.
Revised: We compare this hippocampal features with standard reinforcement learning algorithms, highlighting similarities and differences.

Original: Finally, we demonstrate that a standard deep reinforcement learning model achieves similar average performance when compared to animal learning, but fails to mimic animals during task switching.
Revised: Finally, we demonstrate that a standard deep reinforcement learning model achieves similar average performance when compared to animal learning, but fails to mimic animals during task switching.

Original: Overall, our results gives insights into how the hippocampus solves reinforced spatial continual learning, and puts forward a framework to explicitly compare biological and machine learning during spatial continual learning.
Revised: Overall, our results gives insights into how the hippocampus solves reinforced spatial continual learning, and puts forward a framework to explicitly compare biological and machine learning during spatial continual learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The hippocampus has been long known to play a crucial role in spatial navigation (O’keefe & Nadel (1978);
Revised: The hippocampus has been long known to play a crucial role in spatial navigation (O’keefe & Nadel (1978);

Original: Tolman (1948)).
Revised: Tolman (1948)).

Original: However, there is still an ongoing debate in the ﬁeld on how exactly the hippocampus is involved in the encoding and learning of multiple tasks, such as allocentric and egocentric spatial tasks (Ekstrom et al.
Revised: However, there is still an ongoing debate in the ﬁeld on how exactly the hippocampus is involved in the encoding and learning of multiple tasks, such as allocentric and egocentric spatial tasks (Ekstrom et al.

Original: (2014); Feigenbaum & Rolls (1991); Fidalgo & Martin (2016)).
Revised: (2014); Feigenbaum & Rolls (1991); Fidalgo & Martin (2016)).

Original: In particular, it is not clear how the hippocampal neural population may encode such tasks and how these representations relate to the ability of animals for continual spatial learning.
Revised: In particular, it is not clear how the hippocampal neural population may encode such tasks and how these representations relate to the ability of animals for continual spatial learning.

Original: In reinforcement learning (RL), despite the growing interest in the topic, how to solve the continual learning problem is still unresolved (Parisi et al.
Revised: In reinforcement learning (RL), despite the growing interest in the topic, how to solve the continual learning problem is still unresolved (Parisi et al.

Original: Previous theoretical work has explored how animals may rely on the hippocampus to learn to explore a given environment based by combining
Revised: Previous theoretical work has explored how animals may rely on the hippocampus to learn to explore a given environment based by combining

Original: RL algorithms and spatial coding (Foster et al.
Revised: RL algorithms and spatial coding (Foster et al.

Original: More recently, there has been growing interest in understanding how to solve continual learning in deep neural networks (Zenke et al.
Revised: More recently, there has been growing interest in understanding how to solve continual learning in deep neural networks (Zenke et al.

Original: (2017); Kirkpatrick et al.
Revised: (2017); Kirkpatrick et al.

Original: But how both biological and artiﬁcial neural networks encode and solve spatial tasks in a continual reinforcement learning setting is not known.
Revised: But how both biological and artiﬁcial neural networks encode and solve spatial tasks in a continual reinforcement learning setting is not known.

Original: Here, we ﬁrst focus on analysing neural data from hippocampus CA1 population-level activity to extract meaningful latent features.
Revised: Here, we ﬁrst focus on analysing neural data from hippocampus CA1 population-level activity to extract meaningful latent features.

Original: We ﬁnd representations that are directly related to how the neural population encodes not only multiple tasks, but also reward signals, such as reward prediction errors.
Revised: We ﬁnd representations that are directly related to how the neural population encodes not only multiple tasks, but also reward signals, such as reward prediction errors.

Original: Next, we contrast these experimental observations with standard reinforcement learning algorithms, namely temporal difference learning and Q-learning theory.
Revised: Next, we contrast these experimental observations with standard reinforcement learning algorithms, namely temporal difference learning and Q-learning theory.

Original: Finally, we show that deep Q networks can achieve an average performance similar to animals, but fails to achieve the same relearning speed.
Revised: Finally, we show that deep Q networks can achieve an average performance similar to animals, but fails to achieve the same relearning speed.

Original:  R ESULTS
Revised:  R ESULTS

Original: Our results are based on a dataset of neural activity from a total of 612 hippocampal CA1 neurons, recorded during a reward-based T-maze traversal task (Ciocchi et al.
Revised: Our results are based on a dataset of neural activity from a total of 612 hippocampal CA1 neurons, recorded during a reward-based T-maze traversal task (Ciocchi et al.

Original: For the behavioural
Revised: For the behavioural

Original: - -     4 Time ( in seconds ) - - -
Revised: - -     4 Time ( in seconds ) - - -

Original: N o r m .
Revised: N o r m .

Original: f i r i
Revised: f i r i

Original: ng r a t e [ - ]
Revised: ng r a t e [ - ]

Original: Trial-averaged firing rates of 30 neurons
Revised: Trial-averaged firing rates of 30 neurons

Original: A
Revised: A

Original: B
Revised: B

Original: Example of trial-averaged z-scored ﬁring rates of 30 randomly selected CA1 neurons out ofaround the reward point (vertical dashed line).
Revised: Example of trial-averaged z-scored ﬁring rates of 30 randomly selected CA1 neurons out ofaround the reward point (vertical dashed line).

Original: experiment ﬁve food-deprived rats were trained to get sugar pellet rewards located at the extremities of rewarded arms in a T-maze (Fig. 1Ai).
Revised: experiment ﬁve food-deprived rats were trained to get sugar pellet rewards located at the extremities of rewarded arms in a T-maze (Fig. 1Ai).

Original: Rats were trained to learn two behavioral rules in order to be rewarded.
Revised: Rats were trained to learn two behavioral rules in order to be rewarded.

Original: The ﬁrst is based on allocentric spatial references (Fig. 1Aii) and the second on egocentric spatial references (Fig. 1Aiii).
Revised: The ﬁrst is based on allocentric spatial references (Fig. 1Aii) and the second on egocentric spatial references (Fig. 1Aiii).

Original: We analyse a total of 6 seconds of neuronal activity per trial across several days (around 500 trials in total).
Revised: We analyse a total of 6 seconds of neuronal activity per trial across several days (around 500 trials in total).

Original: For illustration, trial-averaged and normalized ﬁring activity of 30 randomly selected neurons from the recorded CA1 ensemble is presented in
Revised: For illustration, trial-averaged and normalized ﬁring activity of 30 randomly selected neurons from the recorded CA1 ensemble is presented in

Original: Fig.
Revised: Fig.

Original: 1B.
Revised: 1B.

Original: From the raw ﬁring rates is it virtually impossible to uncover meaningful trends around the reward delivery zone (dotted vertical line).
Revised: From the raw ﬁring rates is it virtually impossible to uncover meaningful trends around the reward delivery zone (dotted vertical line).

Original: As a result, we decided to focus on population-level analysis of the neuronal activity.
Revised: As a result, we decided to focus on population-level analysis of the neuronal activity.

Original: Recently a task-speciﬁc linear dimensionality reduction method, Demixed Principal Component
Revised: Recently a task-speciﬁc linear dimensionality reduction method, Demixed Principal Component

Original: Analysis (dPCA), has been introduced.
Revised: Analysis (dPCA), has been introduced.

Original: This algorithm has previously been tested on populationlevel data from the prefrontal and orbifrontal cortices by Kobak et al.
Revised: This algorithm has previously been tested on populationlevel data from the prefrontal and orbifrontal cortices by Kobak et al.

Original: (Kobak et al.
Revised: (Kobak et al.

Original: (2016)), but not on data from hippocampal spatial tasks.
Revised: (2016)), but not on data from hippocampal spatial tasks.

Original: Next, we describe our results obtained using dPCA and how these may be related to reinforcement learning algorithms.
Revised: Next, we describe our results obtained using dPCA and how these may be related to reinforcement learning algorithms.

Original: 2. B EHAVIOUR - SPECIFIC COMPONENTS IN CA1 NEURONAL POPULATIONS
Revised: 2. B EHAVIOUR - SPECIFIC COMPONENTS IN CA1 NEURONAL POPULATIONS

Original: We use demixed PCA to extract meaningful features from the data.
Revised: We use demixed PCA to extract meaningful features from the data.

Original: Like PCA, dPCA
Revised: Like PCA, dPCA

Original: (Kobak et al.
Revised: (Kobak et al.

Original: (2016)) allows us to extract a lower dimensional representation of the data, but in contrast to PCA in dPCA this representation is constrained such that the components extracted encode behaviourally relevant features (e.g. spatial strategy or correct vs incorrect decisions; see Supplementary Material for more details on dPCA).
Revised: (2016)) allows us to extract a lower dimensional representation of the data, but in contrast to PCA in dPCA this representation is constrained such that the components extracted encode behaviourally relevant features (e.g. spatial strategy or correct vs incorrect decisions; see Supplementary Material for more details on dPCA).

Original: The dPCA components extracted from the hippocampal CA1 data explained above are given in Figure 2.
Revised: The dPCA components extracted from the hippocampal CA1 data explained above are given in Figure 2.

Original: Although individual neurons do not exhibit a clear tuning for decision (i.e. correct or incorrect trials) or strategy (i.e. allocentric or egocentric tasks; cf.
Revised: Although individual neurons do not exhibit a clear tuning for decision (i.e. correct or incorrect trials) or strategy (i.e. allocentric or egocentric tasks; cf.

Original: Fig.
Revised: Fig.

Original: 1B), at the population level we observe a clear separation of the neural representations for decisions and strategies (Fig. 2B).
Revised: 1B), at the population level we observe a clear separation of the neural representations for decisions and strategies (Fig. 2B).

Original: Interestingly, we observe a clear separation in population activity patterns for correct and incorrect decisions in the leading demixed decision component (Fig.
Revised: Interestingly, we observe a clear separation in population activity patterns for correct and incorrect decisions in the leading demixed decision component (Fig.

Original: 2B, ﬁrst row, component 3).
Revised: 2B, ﬁrst row, component 3).

Original: It is interesting to note that the separation between correct and incorrect decisions increases gradually over time.
Revised: It is interesting to note that the separation between correct and incorrect decisions increases gradually over time.

Original: This increased separation starts before the reward delivery and increases substantially afterwards, being at its maximum where we believe to be the moment during which animals consume the actual reward (i.e. about a second of crossing the reward sensor represented by a vertical dashed line).
Revised: This increased separation starts before the reward delivery and increases substantially afterwards, being at its maximum where we believe to be the moment during which animals consume the actual reward (i.e. about a second of crossing the reward sensor represented by a vertical dashed line).

Original: This reward-representation appears to be related but also different from standard reward prediction errors predicted by RL algorithms (see more below) and classical observations in dopaminergic neurons (Schultz et al.
Revised: This reward-representation appears to be related but also different from standard reward prediction errors predicted by RL algorithms (see more below) and classical observations in dopaminergic neurons (Schultz et al.

Original: ; Wirth et al.
Revised: ; Wirth et al.

Original: In terms of multi-task representation, we ﬁnd a clean separation at the population-level between allocentric and egocentric tasks (Fig.
Revised: In terms of multi-task representation, we ﬁnd a clean separation at the population-level between allocentric and egocentric tasks (Fig.

Original: 2B, second row, component 4, suggesting that the hippocampus is directly involved in the learning and representation of these two tasks.
Revised: 2B, second row, component 4, suggesting that the hippocampus is directly involved in the learning and representation of these two tasks.

Original: Temporal components capture the most variance in the data and their population activities reveal reward-speciﬁc or reward-approach-speciﬁc responses (Fig.
Revised: Temporal components capture the most variance in the data and their population activities reveal reward-speciﬁc or reward-approach-speciﬁc responses (Fig.

Original: 2B, third row).
Revised: 2B, third row).

Original: We interpret the ﬁrst temporal component (1) as being analogous to an eligibility trace, the second temporal component (2) as being encoding reward location and the third temporal component (5) as exhibiting traces of reward prediction error signals as seen in classical conditioning experiments (Schultz et al.
Revised: We interpret the ﬁrst temporal component (1) as being analogous to an eligibility trace, the second temporal component (2) as being encoding reward location and the third temporal component (5) as exhibiting traces of reward prediction error signals as seen in classical conditioning experiments (Schultz et al.

Original: Fig.
Revised: Fig.

Original: 2B, third row).
Revised: 2B, third row).

Original: Note that the ﬁrst decision component (3) may be also interpreted as some form of (weak) reward prediction error as discussed above.
Revised: Note that the ﬁrst decision component (3) may be also interpreted as some form of (weak) reward prediction error as discussed above.

Original: [Figure]
Revised: [Figure]

Original: These population features we observe should become better deﬁned over learning.
Revised: These population features we observe should become better deﬁned over learning.

Original: That is consistent with what we ﬁnd when comparing earlier trials with later trials (Fig. 3).
Revised: That is consistent with what we ﬁnd when comparing earlier trials with later trials (Fig. 3).

Original: For example, the separation between the two tasks (egocentric vs allocentric) increases in later trials.
Revised: For example, the separation between the two tasks (egocentric vs allocentric) increases in later trials.

Original: A similar trend is observed between correct and incorrect trials.
Revised: A similar trend is observed between correct and incorrect trials.

Original: Finally, the eligibility trace-like component also becomes more pronounced in later trials.
Revised: Finally, the eligibility trace-like component also becomes more pronounced in later trials.

Original: 2. C OMPARISON BETWEEN STANDARD RL ALGORITHMS AND HIPPOCAMPAL REPRESENTATIONS
Revised: 2. C OMPARISON BETWEEN STANDARD RL ALGORITHMS AND HIPPOCAMPAL REPRESENTATIONS

Original: Here we contrast the features observed in the hippocampus with ideas commonly found in RL theories.
Revised: Here we contrast the features observed in the hippocampus with ideas commonly found in RL theories.

Original: We ﬁrst use a simple temporal difference (TD) learning algorithm with eligibility traces in a 1D environment (see SM for more details).
Revised: We ﬁrst use a simple temporal difference (TD) learning algorithm with eligibility traces in a 1D environment (see SM for more details).

Original: Presence of eligibility traces postulated by TD models is similar to the ones we found in our experimental data (Fig.
Revised: Presence of eligibility traces postulated by TD models is similar to the ones we found in our experimental data (Fig.

Original: 4Ai,Bi).
Revised: 4Ai,Bi).

Original: Regarding evidence for TD
Revised: Regarding evidence for TD

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table][Figure] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: y
Revised: y

Original: T i m e
Revised: T i m e

Original: I n t e r a c t i on
Revised: I n t e r a c t i on

Original: D / S
Revised: D / S

Original: [Figure]
Revised: [Figure]

Original: errors, our results only loosely support standard TD errors, in that the 3rd component distinguishes between correct and incorrect trials (Fig. 4Aii,Bii).
Revised: errors, our results only loosely support standard TD errors, in that the 3rd component distinguishes between correct and incorrect trials (Fig. 4Aii,Bii).

Original: However, from Fig.
Revised: However, from Fig.

Original: 3 we can see that this component is initially stronger (i.e. in earlier trials) and becomes weaker with learning around the expected reward location (i.e. 1s after crossing the sensor), consistent with changes in TD errors over learning.
Revised: 3 we can see that this component is initially stronger (i.e. in earlier trials) and becomes weaker with learning around the expected reward location (i.e. 1s after crossing the sensor), consistent with changes in TD errors over learning.

Original: Next we used epsilon -greedy Q-learning in a grid-world setting to model the experimental task (see
Revised: Next we used epsilon -greedy Q-learning in a grid-world setting to model the experimental task (see

Original: SM for details)
Revised: SM for details)

Original: we observe a separation between the two tasks in the value function (i.e. allocentric and egocentric; Fig.
Revised: we observe a separation between the two tasks in the value function (i.e. allocentric and egocentric; Fig.

Original: 4C,D), which appears to be similar to the one observed experimentally.
Revised: 4C,D), which appears to be similar to the one observed experimentally.

Original: However, as expected, this is not sufﬁcient for the model to learn the two tasks in a continual learning setting as we highlight next.
Revised: However, as expected, this is not sufﬁcient for the model to learn the two tasks in a continual learning setting as we highlight next.

Original: Indeed, our Q-learning model fails to learn the navigational strategies consecutively (Fig.
Revised: Indeed, our Q-learning model fails to learn the navigational strategies consecutively (Fig.

Original: 5B)
Revised: 5B)

Original: in contrast to animals that can sustain a performance above chance level (Fig. 5A).
Revised: in contrast to animals that can sustain a performance above chance level (Fig. 5A).

Original: We show the performance of animal SC03 to demonstrate that animals can learn both allocentric and egocentric strategies as is noticeable from the mean performance for both tasks being above chance level.
Revised: We show the performance of animal SC03 to demonstrate that animals can learn both allocentric and egocentric strategies as is noticeable from the mean performance for both tasks being above chance level.

Original: The model, on the other hand, is unable to switch over and recall policies learned for the allocentric task at trials 212-216.
Revised: The model, on the other hand, is unable to switch over and recall policies learned for the allocentric task at trials 212-216.

Original: This is because, as expected, the policy information pertaining to the tasks is overridden as tasks or starting locations change over.
Revised: This is because, as expected, the policy information pertaining to the tasks is overridden as tasks or starting locations change over.

Original: This clearly highlights the continual forgetting in standard RL systems.
Revised: This clearly highlights the continual forgetting in standard RL systems.

Original: However, it should be noted that some forgetting can be observed in animals as well (Fig.
Revised: However, it should be noted that some forgetting can be observed in animals as well (Fig.

Original: 5A).
Revised: 5A).

Original: The incapability of standard Tabular Q-learning algorithm to reach a performance similar to animals lead us to develop a deep Q-Network that may be able to better solve the two tasks.
Revised: The incapability of standard Tabular Q-learning algorithm to reach a performance similar to animals lead us to develop a deep Q-Network that may be able to better solve the two tasks.

Original: 2. C OMPARING DEEP Q- NETWORKS AND ANIMAL PERFORMANCE
Revised: 2. C OMPARING DEEP Q- NETWORKS AND ANIMAL PERFORMANCE

Original: We implement a deep Q network (DQN) with task-speciﬁc information provided to the neural network as input.
Revised: We implement a deep Q network (DQN) with task-speciﬁc information provided to the neural network as input.

Original: Such task-speciﬁc information to the DQN allows the network to learn the different rules of reaching reward locations and this method draws inspiration from the works of Anonymous et al.
Revised: Such task-speciﬁc information to the DQN allows the network to learn the different rules of reaching reward locations and this method draws inspiration from the works of Anonymous et al.

Original: and Serra et al.
Revised: and Serra et al.

Original: (Authors; Serr`a et al.
Revised: (Authors; Serr`a et al.

Original: In addition, there is evidence that speciﬁc neuromodulators (Acetylcholine) may provide such task-speciﬁc information.
Revised: In addition, there is evidence that speciﬁc neuromodulators (Acetylcholine) may provide such task-speciﬁc information.

Original: Nevertheless, unlike earlier works our DQN does not require a separate memory layer to store the task-speciﬁc information corresponding to the different tasks.
Revised: Nevertheless, unlike earlier works our DQN does not require a separate memory layer to store the task-speciﬁc information corresponding to the different tasks.

Original: Salient task-speciﬁc information for inclusion in the blocked cells of the input maze matrix are chosen with the help of a trial and error mechanism for training the network with different positive and negative values of information.
Revised: Salient task-speciﬁc information for inclusion in the blocked cells of the input maze matrix are chosen with the help of a trial and error mechanism for training the network with different positive and negative values of information.

Original: We used the ﬁrst 250 trials from our animal data (Fig. 5) to train and test the network ability to learn and remember the different tasks.
Revised: We used the ﬁrst 250 trials from our animal data (Fig. 5) to train and test the network ability to learn and remember the different tasks.

Original: We used Adam optimizer to train the network.
Revised: We used Adam optimizer to train the network.

Original: [Figure]
Revised: [Figure]

Original: y
Revised: y

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: T D e rr o r
Revised: T D e rr o r

Original: A v e r a g e v a l u e v a l u e a n d e li g .
Revised: A v e r a g e v a l u e v a l u e a n d e li g .

Original: t r a c e
Revised: t r a c e

Original: [Figure]
Revised: [Figure]

Original: The DQN consists of two hidden layers of 100 neuron units each.
Revised: The DQN consists of two hidden layers of 100 neuron units each.

Original: The input layer is of the size of the input maze, while the output layer contains four neurons with the Q-values for the four different actions that an agent can take to change its state in the maze.
Revised: The input layer is of the size of the input maze, while the output layer contains four neurons with the Q-values for the four different actions that an agent can take to change its state in the maze.

Original: Additionally, we found PRelu to be the most suitable activation function to solve this reinforced spatial navigation task involving allocentric and egocentric navigational strategies.
Revised: Additionally, we found PRelu to be the most suitable activation function to solve this reinforced spatial navigation task involving allocentric and egocentric navigational strategies.

Original: Interestingly, we observe that this network with task-speciﬁc memory achieves a performance for both allocentric and egocentric tasks comparable animals (animal SC03 is given as a typical example (Fig 5A).
Revised: Interestingly, we observe that this network with task-speciﬁc memory achieves a performance for both allocentric and egocentric tasks comparable animals (animal SC03 is given as a typical example (Fig 5A).

Original: In particular, the DQN learns the egocentric task more efﬁciently than the allocentric task as demonstrated by the higher mean performance (dashed blue) line in Fig.
Revised: In particular, the DQN learns the egocentric task more efﬁciently than the allocentric task as demonstrated by the higher mean performance (dashed blue) line in Fig.

Original: Even though the network struggles with the allocentric, it shows a mean performance above the chance level.
Revised: Even though the network struggles with the allocentric, it shows a mean performance above the chance level.

Original: It should also be noted that the DQN performs better in the egocentric task, while the animal ﬁnds it slightly more difﬁcult to learn the egocentric spatial navigation task rules.
Revised: It should also be noted that the DQN performs better in the egocentric task, while the animal ﬁnds it slightly more difﬁcult to learn the egocentric spatial navigation task rules.

Original: In other words, animal
Revised: In other words, animal

Original: SC03 performs marginally better in the allocentric tasks than in the egocentric tasks (Fig. 5A).
Revised: SC03 performs marginally better in the allocentric tasks than in the egocentric tasks (Fig. 5A).

Original: We compared our results from this DQN with task-speciﬁc information with a DQN without taskspeciﬁc information and this network performed signiﬁcantly better than the other one (results not shown), whose performance accuracies for both allocentric and egocentric tasks were below the chance level.
Revised: We compared our results from this DQN with task-speciﬁc information with a DQN without taskspeciﬁc information and this network performed signiﬁcantly better than the other one (results not shown), whose performance accuracies for both allocentric and egocentric tasks were below the chance level.

Original: Mean performance across all Q-learning models including deep Q-networks are compared in the following Fig.
Revised: Mean performance across all Q-learning models including deep Q-networks are compared in the following Fig.

Original: 7A against the mean performance of animal SC03 over learning the allocentric and egocentric tasks for 250 trials.
Revised: 7A against the mean performance of animal SC03 over learning the allocentric and egocentric tasks for 250 trials.

Original: To highlight the ability of the different systems for continual learning we plotted the re-learning performance of the Q-learning models and the animals after the networks/models and the animal has already been exposed to the two different task types (Fig.
Revised: To highlight the ability of the different systems for continual learning we plotted the re-learning performance of the Q-learning models and the animals after the networks/models and the animal has already been exposed to the two different task types (Fig.

Original: 7(B,C)).
Revised: 7(B,C)).

Original: A DQN that learns both tasks consecutively as the animal does in the behavioural experiment achieves a mean performance comparable to the animal (green and orange bars in Fig.
Revised: A DQN that learns both tasks consecutively as the animal does in the behavioural experiment achieves a mean performance comparable to the animal (green and orange bars in Fig.

Original: 7A).
Revised: 7A).

Original: A
Revised: A

Original: DQN learning only learning the egocentric task but with a reversal of reward locations (see SM
Revised: DQN learning only learning the egocentric task but with a reversal of reward locations (see SM

Original: Fig.
Revised: Fig.

Original: 8A(i,iv)) performs better (peach bar) than the animal.
Revised: 8A(i,iv)) performs better (peach bar) than the animal.

Original: On the other hand, a DQN learning only
Revised: On the other hand, a DQN learning only

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the allocentric task with reward location reversals fails to acquire a performance close to both the
Revised: the allocentric task with reward location reversals fails to acquire a performance close to both the

Original: DQN-Ego and the animal (peru colored bar).
Revised: DQN-Ego and the animal (peru colored bar).

Original: Tabular Q-learning model shows the worst mean performance amongst all the models as well against the animal’s mean performance.
Revised: Tabular Q-learning model shows the worst mean performance amongst all the models as well against the animal’s mean performance.

Original: This is expected because a standard Tabular Q-learning model is inefﬁcient in incorporating and sustaining complete information corresponding to the different strategies.
Revised: This is expected because a standard Tabular Q-learning model is inefﬁcient in incorporating and sustaining complete information corresponding to the different strategies.

Original: Additionally, we speculate that the DQN-Ego acquires a better performance than the DQN-Allo since the DQN-Ego encounters far less reward location change overs (dashed ‘Strategy Switch’ lines) when compared to the DQN-Allo model.
Revised: Additionally, we speculate that the DQN-Ego acquires a better performance than the DQN-Allo since the DQN-Ego encounters far less reward location change overs (dashed ‘Strategy Switch’ lines) when compared to the DQN-Allo model.

Original: Providing task-speciﬁc information in the DQN also shows a clear improvement in continual learning of tasks.
Revised: Providing task-speciﬁc information in the DQN also shows a clear improvement in continual learning of tasks.

Original: We postulate that the DQNs somehow learn to utilize this extra information (memory weights for tasks), perhaps internally segregating the activity patterns (even weights) corresponding to the task types.
Revised: We postulate that the DQNs somehow learn to utilize this extra information (memory weights for tasks), perhaps internally segregating the activity patterns (even weights) corresponding to the task types.

Original: In contrast, a Tabular Q-learning model without any extra information about the tasks struggles to reach comparable performance as the animal does.
Revised: In contrast, a Tabular Q-learning model without any extra information about the tasks struggles to reach comparable performance as the animal does.

Original: This also leads to the hypothesis that the hippocampus CA1 might also be using similar mechanism to encode the rules involved with the allocentric and egocentric navigational strategies and consequently, remember the reward locations as the task change over trials.
Revised: This also leads to the hypothesis that the hippocampus CA1 might also be using similar mechanism to encode the rules involved with the allocentric and egocentric navigational strategies and consequently, remember the reward locations as the task change over trials.

Original: [Figure]
Revised: [Figure]

Original: r f
Revised: r f

Original: [Figure]
Revised: [Figure]

Original: n c
Revised: n c

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: learning both tasks achieves
Revised: learning both tasks achieves

Original: Performance at re-learning the tasks in each Q-learning model and the animal is plotted against trials after a task type is switched to another in Fig.
Revised: Performance at re-learning the tasks in each Q-learning model and the animal is plotted against trials after a task type is switched to another in Fig.

Original: 7(B,C).
Revised: 7(B,C).

Original: The animal starts with a higher relearning performance value than any of the models at both allocentric and egocentric task switch points.
Revised: The animal starts with a higher relearning performance value than any of the models at both allocentric and egocentric task switch points.

Original: It also achieves rapidly relearns the switched-to task and this is observed from the orange line values above chance in Fig.
Revised: It also achieves rapidly relearns the switched-to task and this is observed from the orange line values above chance in Fig.

Original: 7B and Fig.
Revised: 7B and Fig.

Original: 7C. The DQN learning both tasks together shows a slow re-learning speed and fails to immediately re-learn the switched-to task.
Revised: 7C. The DQN learning both tasks together shows a slow re-learning speed and fails to immediately re-learn the switched-to task.

Original: However, this network on average attains a performance close to the animal as can be noticed from comparing the green and orange bars in Fig.
Revised: However, this network on average attains a performance close to the animal as can be noticed from comparing the green and orange bars in Fig.

Original: 7A for the switch to the egocentric task (brown line in Fig.
Revised: 7A for the switch to the egocentric task (brown line in Fig.

Original: 7B) than when the task is changed to allocentric rule (brown line in Fig.
Revised: 7B) than when the task is changed to allocentric rule (brown line in Fig.

Original: 7B).
Revised: 7B).

Original: Networks learning only egocentric (DQNEgo in Fig.
Revised: Networks learning only egocentric (DQNEgo in Fig.

Original: 7B) and allocentric (DQN-Allo in Fig.
Revised: 7B) and allocentric (DQN-Allo in Fig.

Original: 7C) have better re-learning speeds than the DQN learning both together.
Revised: 7C) have better re-learning speeds than the DQN learning both together.

Original: However, the re-learning performance for both DQN-Allo and DQN-Ego is inconsistent across trials.
Revised: However, the re-learning performance for both DQN-Allo and DQN-Ego is inconsistent across trials.

Original: Overall, standard deep RL models can achieve a performance comparable to animals, but fail to continually learn.
Revised: Overall, standard deep RL models can achieve a performance comparable to animals, but fail to continually learn.

Original: Although animals relearn faster, they are still not perfect relearners.
Revised: Although animals relearn faster, they are still not perfect relearners.

Original:  C ONCLUSION AND DISCUSSION
Revised:  C ONCLUSION AND DISCUSSION

Original: Overall, our results using dPCA show that the hippocampus encodes task variables such as decisions, rewards and strategies, which suggests that the hippocampus plays a key role in the ability for animals to perform continual spatial learning.
Revised: Overall, our results using dPCA show that the hippocampus encodes task variables such as decisions, rewards and strategies, which suggests that the hippocampus plays a key role in the ability for animals to perform continual spatial learning.

Original: We also demonstrate that some of the components we ﬁnd are consistent with theoretical models in reinforcement learning.
Revised: We also demonstrate that some of the components we ﬁnd are consistent with theoretical models in reinforcement learning.

Original: Our results suggest the existence of a non-standard form of TD learning implemented or used by the hippocampus.
Revised: Our results suggest the existence of a non-standard form of TD learning implemented or used by the hippocampus.

Original: This raises exciting possibilities for future research.
Revised: This raises exciting possibilities for future research.

Original: One avenue is to develop a model that gradually learns to separate the representations of the two tasks inspired by recent developments in continual learning where different groups of neurons may specialize in different tasks explaining the features and the ability for continual learning that we observe in experimental neuroscience (Zenke et al.
Revised: One avenue is to develop a model that gradually learns to separate the representations of the two tasks inspired by recent developments in continual learning where different groups of neurons may specialize in different tasks explaining the features and the ability for continual learning that we observe in experimental neuroscience (Zenke et al.

Original: (2017); Kirkpatrick et al.
Revised: (2017); Kirkpatrick et al.

Original: Our work also puts forward a framework that can be used to better compare how both machines and animals learn to solve continual learning problems.
Revised: Our work also puts forward a framework that can be used to better compare how both machines and animals learn to solve continual learning problems.

Original: A S UPPLEMENTARY M ATERIAL M ETHODS O VERVIEW A. D EMIXED P RINCIPAL C OMPONENT A NALYSIS
Revised: A S UPPLEMENTARY M ATERIAL M ETHODS O VERVIEW A. D EMIXED P RINCIPAL C OMPONENT A NALYSIS

Original: Demixed Principal Component Analysis (dPCA) (Kobak et al.
Revised: Demixed Principal Component Analysis (dPCA) (Kobak et al.

Original: not only reduces the dimensionality of the data without losing too much information but also facilitates visualization of neural responses tuning to experimental parameters like decisions, rewards and navigation rules.
Revised: not only reduces the dimensionality of the data without losing too much information but also facilitates visualization of neural responses tuning to experimental parameters like decisions, rewards and navigation rules.

Original: Another advantage of demixed PCA over standard Principal Component Analysis (PCA) methods is that it does not enforce orthogonality of the components and hence, enables successful demixing of population-level activity.
Revised: Another advantage of demixed PCA over standard Principal Component Analysis (PCA) methods is that it does not enforce orthogonality of the components and hence, enables successful demixing of population-level activity.

Original: We applied the recent version of the demixed PCA algorithm Kobak et al.
Revised: We applied the recent version of the demixed PCA algorithm Kobak et al.

Original: (2016) on trial-averaged neuronal ﬁring rates from the hippocampus CA1 of all ﬁve rats across all experimental days.
Revised: (2016) on trial-averaged neuronal ﬁring rates from the hippocampus CA1 of all ﬁve rats across all experimental days.

Original: The trial-averaged data was organised into a matrix with dimensions Neurons x Decision x Strategy X Time.
Revised: The trial-averaged data was organised into a matrix with dimensions Neurons x Decision x Strategy X Time.

Original: Note that ‘reward’ was not used as a demixing category (or task variable) in the algorithmic implementation since the inﬂuences of rewards could be detected from the ﬁring patterns in demixed decision and strategies components itself.
Revised: Note that ‘reward’ was not used as a demixing category (or task variable) in the algorithmic implementation since the inﬂuences of rewards could be detected from the ﬁring patterns in demixed decision and strategies components itself.

Original: Additionally, we used a regularisation term in dPCA’s cost function to prevent the model from overﬁtting.
Revised: Additionally, we used a regularisation term in dPCA’s cost function to prevent the model from overﬁtting.

Original: A. T ASK S IMULATION AND T EMPORAL D IFFERENCE L EARNING
Revised: A. T ASK S IMULATION AND T EMPORAL D IFFERENCE L EARNING

Original: The task is simulated in 5 x 5 grid (Fig 8), with rewards being placed only at the terminal states.
Revised: The task is simulated in 5 x 5 grid (Fig 8), with rewards being placed only at the terminal states.

Original: The simulations consider rule-switching and random start-location switching conditions as in the original experiment.
Revised: The simulations consider rule-switching and random start-location switching conditions as in the original experiment.

Original: However, the simulation design does not incorporate use of spatial cues for the allocentric task.
Revised: However, the simulation design does not incorporate use of spatial cues for the allocentric task.

Original: TD(0) and TD( λ ) algorithms are employed in the simulations to analyse the back propagation of estimated value and TD error from the rewarding states/time-points to the states/time-points coming before the reward states.
Revised: TD(0) and TD( λ ) algorithms are employed in the simulations to analyse the back propagation of estimated value and TD error from the rewarding states/time-points to the states/time-points coming before the reward states.

Original: We used two value functions at this stage for the two different strategies to see how each compares against the other.
Revised: We used two value functions at this stage for the two different strategies to see how each compares against the other.

Original: We also use eligibility traces to verify the notion of the hippocampus employing a decaying memory trace of any important spatiotemporal event.
Revised: We also use eligibility traces to verify the notion of the hippocampus employing a decaying memory trace of any important spatiotemporal event.

Original: Eligibility traces enable the model to distribute credit efﬁciently to past states that can predict future rewards.
Revised: Eligibility traces enable the model to distribute credit efﬁciently to past states that can predict future rewards.

Original: These traces have been shown to augment learning in reinforcement learning models (Sutton &
Revised: These traces have been shown to augment learning in reinforcement learning models (Sutton &

Original: Barto (1998)).
Revised: Barto (1998)).

Original: Furthermore, an (cid:15) greedy online Q-learning algorithm to model policy learning (what actions to take when and where) of animals in the allocentric and egocentric tasks is implemented.
Revised: Furthermore, an (cid:15) greedy online Q-learning algorithm to model policy learning (what actions to take when and where) of animals in the allocentric and egocentric tasks is implemented.

Original: We add to the (cid:15) -greedy approach the following Q-value update function in Equation 1 to ensure rapid convergence of the model.
Revised: We add to the (cid:15) -greedy approach the following Q-value update function in Equation 1 to ensure rapid convergence of the model.

Original: [Equation]
Revised: [Equation]

Original: Here, α is the learning rate, γ is discount factor and Q ( S t , A t ) is the Q-value of a state-action pair at time t.
Revised: Here, α is the learning rate, γ is discount factor and Q ( S t , A t ) is the Q-value of a state-action pair at time t.

Original: We think of this slight modiﬁcation in the algorithm as combining best of the two worlds (SARSA and ofﬂine Q-learning) and this variant as twice-as-greedy online Q-learning.
Revised: We think of this slight modiﬁcation in the algorithm as combining best of the two worlds (SARSA and ofﬂine Q-learning) and this variant as twice-as-greedy online Q-learning.

Original: Nevertheless, we also verify our results from this modiﬁed model with a standard SARSA model to observe any discrepancies in learning policies, which we do not ﬁnd.
Revised: Nevertheless, we also verify our results from this modiﬁed model with a standard SARSA model to observe any discrepancies in learning policies, which we do not ﬁnd.

Original: Additionally, this model does seem to converge slightly faster than a standard SARSA model(performance metrics not shown).
Revised: Additionally, this model does seem to converge slightly faster than a standard SARSA model(performance metrics not shown).

Original: A. S EPARATE DQN NETWORKS TO LEARN A LLOCENTRIC AND E GOCENTRIC S TRATEGIES
Revised: A. S EPARATE DQN NETWORKS TO LEARN A LLOCENTRIC AND E GOCENTRIC S TRATEGIES

Original: We use deep Q neural networks to simulate the behavioural experiment.
Revised: We use deep Q neural networks to simulate the behavioural experiment.

Original: Figs 9A and 9B show the results from using two deep Q-networks to learn the allocentric and egocentric tasks separately.
Revised: Figs 9A and 9B show the results from using two deep Q-networks to learn the allocentric and egocentric tasks separately.

Original: We include task-speciﬁc information into the input maze to assist the network in recognizing changing reward positions while following a single type of navigational rule.
Revised: We include task-speciﬁc information into the input maze to assist the network in recognizing changing reward positions while following a single type of navigational rule.

Original: It seems that Deep Q neural networks can learn more efﬁciently than the Tabular Q-learning algorithms.
Revised: It seems that Deep Q neural networks can learn more efﬁciently than the Tabular Q-learning algorithms.

Original: Overall performance accuracies of both the networks are above chance levels, which shows that the network is able to cope with changing start and reward positions within a particular type of strategy (allocentric or egocentric).
Revised: Overall performance accuracies of both the networks are above chance levels, which shows that the network is able to cope with changing start and reward positions within a particular type of strategy (allocentric or egocentric).

Original: However, moving averages in the allocentric task show the occurrence of catastrophic forgetting as the network struggles to cope with changing reward locations while starting from the same start positions.
Revised: However, moving averages in the allocentric task show the occurrence of catastrophic forgetting as the network struggles to cope with changing reward locations while starting from the same start positions.

Original: On the other hand, the DQN that learns only the egocentric task performs perfectly after 130 trials and is even able to deal with a change in reward locations at trial(dashed line).
Revised: On the other hand, the DQN that learns only the egocentric task performs perfectly after 130 trials and is even able to deal with a change in reward locations at trial(dashed line).

Original: Changing reward locations within a task type are also denoted as strategy switches in the ﬁgure.
Revised: Changing reward locations within a task type are also denoted as strategy switches in the ﬁgure.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]