E MERGENCE OF GRID - LIKE REPRESENTATIONS BY TRAINING RECURRENT NEURAL NETWORKS TO PERFORM SPATIAL LOCALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Decades of research on the neural code underlying spatial navigation have revealed a diverse set of neural response properties.
The Entorhinal Cortex (EC) of the mammalian brain contains a rich set of spatial correlates, including grid cells which encode space using tessellating patterns.
However, the mechanisms and functional signiﬁcance of these spatial representations remain largely mysterious.
As a new way to understand these neural representations, we trained recurrent neural networks (RNNs) to perform navigation tasks in 2D arenas based on velocity inputs.
Surprisingly, we ﬁnd that grid-like spatial response patterns emerge in trained networks, along with units that exhibit other spatial correlates, including border cells and band-like cells.
All these different functional types of neurons have been observed experimentally.
The order of the emergence of grid-like and border cells is also consistent with observations from developmental studies.
Together, our results suggest that grid cells, border cells and others as observed in EC may be a natural solution for representing space efﬁciently given the predominant recurrent connections in the neural circuits.
I NTRODUCTION
Understanding the neural code in the brain has long been driven by studying feed-forward architectures, starting from Hubel and Wiesel’s famous proposal on the origin of orientation selectivity in primary visual cortex (Hubel & Wiesel, 1962).
Inspired by the recent development in deep learning (Krizhevsky et al., 2012; LeCun et al., 2015; Hochreiter & Schmidhuber, 1997; Mnih et al., 2015), there has been a burst of interest in applying deep feed-forward models, in particular convolutional neural networks (CNN) (LeCun et al., 1998), to study the sensory system, which hierarchically extracts useful features from the sensory input (see e.g. , Yamins et al.
(2014); Kriegeskorte (2015);
Kietzmann et al.
(2017); Yamins & DiCarlo (2016)).
For more cognitive tasks, neural systems often need to maintain an internal representation of relevant variables without external stimuli- a process that requires more than feature extraction.
We will focus on spatial navigation, which typically requires the brain to maintain a representation of location and update it according to the animal’s movements and landmarks of the environment.
Physiological studies done in rodents and other mammals (including humans, non-human primates and bats) have revealed a variety of neural correlates of space in Hippocampus and Entorhinal Cortex (EC), including place cells (O’Keefe, 1976), grid cells (Fyhn et al., 2004; Hafting et al., 2005; Fyhn et al., 2008; Yartsev et al., 2011; Killian et al., 2012; Jacobs et al., 2013), along with border cells (Solstad et al., 2008), band-like cells (Krupic et al., 2012) and others (see Figure 1a).
The study of the neural underpinning of spatial cognition has provided an important window into how high-level cognitive functions are supported in the brain (Moser et al., 2008; Aronov et al., 2017).
How might the spatial navigation task be solved using a network of neurons?
Recurrent neural networks (RNNs) (Hochreiter & Schmidhuber, 1997; Graves et al., 2013; Oord et al., 2016; Theis & Bethge, 2015; Gregor et al., 2015) seem particularly useful for these tasks.
Indeed, recurrent-based continuous attractor networks have been one of the main types of models proposed for the formation of grid cells (McNaughton et al., 2006; Burak & Fiete, 2009; Couey et al., 2013) and place cells (Samsonovich &
McNaughton, 1997).
However, these models require hand-crafted and ﬁned tuned connectivity patterns, and the evidence of such speciﬁc 2D connectivity patterns has been largely absent.
Here we present a new model for understanding the representation of space in the neural system.
Speciﬁcally, we trained a RNN to perform spatial navigation tasks.
By leveraging the recent development in RNN training and knowledge of the navigation system in the brain, we show that training a RNN with biologically relevant constraints naturally gives rise to a variety of spatial response proﬁles as observed in Entorhinal Cortex (EC), including grid-like responses.
To our knowledge, this is the ﬁrst study to show how grid-like responses could emerge from training a RNN to perform navigation.
Our result implies that the neural representation in EC may be seen as a natural way for the brain to solve the navigation task efﬁciently.
More generally, it suggests that RNNs can be a powerful tool for understanding the neural mechanisms of certain high-level cognitive functions
[Figure]
100 recurrently connected units (or neurons) which receive two external inputs, representing the animal’s speed and heading direction.
The two outputs linearly weight the neurons in the RNN.
The goal of training is to make the responses of the two output neurons accurately represent the animal’s physical location.
c) Typical trajectory after training.
As shown, the output of the RNN can accurately, though not perfectly, track the animal’s location during navigation.
M ODEL
The overarching design goal for the network architecture, task, and training procedure was to ﬁnd the simplest model that still led to the rich spatial representations observed in EC.
2. M ODEL DESCRIPTION
Our network model consists of a set of recurrently connected units ( N = 100 ).
The dynamics of each unit in the network u i ( t ) is governed by the standard continuous-time RNN equation:
[Equation]
for i = 1 , . . .
, N .
The activity of each unit, u i ( t ) , is related to the activation of that unit, x i ( t ) , through a nonlinearity which in this study we take to be u
i ( t ) = tanh( x i ( t )) .
Each unit receives input from other units through the recurrent weight matrix W rec and also receives external input,
I ( t ) , that enters the network through the weight matrix W in .
Each unit has two sources of bias, b i which is learned and ξ i ( t ) which represents noise intrinsic to the network and is taken to be
Gaussian with zero mean and constant variance.
The network was simulated using the Euler method for T = 500 timesteps of duration τ/ 10 .
To perform a 2D navigation task with the RNN we linearly combine the ﬁring rates of units in the network.
The two linear readout neurons, y 1 ( t ) and y 2 ( t ) , are given by the following equation:
[Equation]
2. I NPUT TO THE NETWORK
The network inputs and outputs were inspired by simple spatial navigation tasks in open arena.
The inputs to the network were chosen to be speed and direction because cells tuned for speed and direction are observed experimentally and these are necessary for grid formation (Winter et al., 2015b;a).
Note that throughout the paper, we adopt the common assumption that the head direction of the animal coincides with the actual moving direction.
The outputs were the x- and y-coordinates of the integrated position.
The direction of the animal is modeled by modiﬁed Brownian motion to increase the probability of straight-runs, in order to be consistent with the typical rodent’s behavior in an open environment.
Special care is taken when the animal is close to the boundary.
The boundary of the environment will affect the statistics of the movement, as the animal cannot cross the boundary.
This fact was reﬂected in the model by re-sampling the angular input variable until the input angle did not lead the animal outside the boundary.
In the simulations shown below, the animal always starts from the center of the arena, but we veriﬁed that the results are insensitive to the starting locations.
2. T RAINING
We optimized the network parameters W rec , W in , b and W out to minimize the squared error in equation (3) between target x- and y-coordinates from a two dimensional navigation task (performed in rectangular, hexagonal, and triangular arenas) and the network outputs generated according to equation (2).
[Equation]
Parameters were updated with the Hessian-free algorithm (Martens & Sutskever, 2011) using minibatches of size M = 500 trials.
In addition to minimizing the error function in equation (3) we regularized the input and output weights according to equation (4) and the ﬁring rates of the units according to equation (5).
Overall, the training aims to minimize a loss function, that consists of the error of the animal, a metabolic cost, and a penalty for large network parameters.
[Equation]
The results are qualitatively insensitive to the initialization scheme used for the recurrent weight matrix W rec .
Simulations in the hexagonal environment were obtained by initializing the elements of W rec to be zero mean Gaussian random variables with variance 1 .
5 2 /N , and simulations in the square and triangular environments were initialized with an orthogonal W rec (Saxe et al., 2014).
We initialized the bias b and output weights W out to be zero.
The elements of W in were zero mean
Gaussian variables with variance 1 /N in .
[Figure]
R ESULTS
We run simulation experiments in arenas with different boundary shapes, including square, triangular and hexagonal.
Figure 1c shows a typical example of the model performance after training, which shows the network (red trace) can accurately track the animal’s actual path (black).
3. T UNING PROPERTIES OF THE MODEL NEURONS
We are mostly interested in what kind of representation the RNN has learned to solve this navigation task, and whether such a representation resembles the response properties of neurons in EC (Moser et al., 2008).
3.1. S PATIAL TUNING
To test whether the trained RNN developed such location-selective representations, we plot individual neurons’ mean activity level as a function of the animal’s location during spatial exploration.
Note that these average response proﬁles should not be confused with the linear ﬁlters typically shown in feedforward networks.
Surprisingly, we ﬁnd neurons in the trained RNN show a range of interesting spatial response proﬁles.
Examination of these response proﬁles suggests they can be classiﬁed into distinct functional types.
Importantly, as we will show, these distinct spatial response proﬁles can be mapped naturally to known physiology in EC.
The spatial responses of all units in trained networks from triangular and hexagonal arenas are shown in the Appendix.
Grid-like responses Most interestingly
, we ﬁnd some of the units in the RNN exhibit clear grid-like response (Figure 2a).
These ﬁring patterns typically exhibit multiple ﬁring ﬁelds, with each ﬁring ﬁeld exhibiting roughly circular symmetric or ellipse shape.
Furthermore, the ﬁring ﬁelds are highly structured, i.e., , when combined, are arranged on a regular lattice.
Furthermore, the structure of the response lattice depends on the shape of the boundary.
In particular, training the network to perform self-localization in a square environment tend to give rectangular grids.
In hexagonal environment and triangular environment, the grids are more close to hexagonal.
Experimentally, it is shown that (medial) EC contains so-called grid cells which exhibit multiple ﬁring ﬁelds that lie on a regular grid (Fyhn et al., 2004; Hafting et al., 2005).
The grid-like ﬁring patterns in our simulation are reminiscent of the grid cells in rodents and other mammals.
However, we also notice that the the grid-like model responses typically exhibit few periods, not as many as experimental data (see Figure 1a).
It is possible that using a larger network might reveal ﬁner grid-patterns in our model.
Nonetheless, it is surprising that the gird-like spatial representations can develop in our model, given there is no periodicity in the input.
Another potential concern is that, experimentally it is reported that the grids are often hexagonal (Hafting et al., 2005) even in square environment (see Figure 1a), though the grids are somewhat inﬂuenced by the shape of the environment.
However, the rats in these experiments presumable had spatial experience in other environment with various boundary shapes.
Experimentally, it would be interesting to see if grid cells would lie on a square lattice instead if the rats are raised in a single square environment-
a situation we are simulating here.
Border responses Many neurons in the RNN exhibit selectivity to the boundary (Figure 2c).
Typically, they only encode a portion of the boundary, e.g. one piece of wall in a square shaped environment.
Such properties are similar to the border cells discovered in rodent EC (Solstad et al., 2008;
Savelli et al., 2008; Lever et al., 2009).
Experimentally, border cells mainly ﬁre along one piece of wall, although some have been observed to ﬁre along multiple borders or along the whole boundary of the environment; interestingly, these multi-border responses were also observed in some RNN models.
Currently, it is unclear how the boundary-like response proﬁles emerge (Solstad et al., 2008;
Savelli et al., 2008; Lever et al., 2009).
Our model points to the possibility that the border cells may emerge without the presence of tactile cues.
Furthermore, it suggests that border cell formation may be related to the movement statistics of the animals, i.e. due to the asymmetry of the movement statistics along the boundary.
Band-like responses
Interestingly, some neurons in the RNN exhibit band-like responses (Figure 2b).
In most of our simulations, these bands tend to be parallel to one of the boundaries.
For some of the units, one of the bands overlaps the boundary, but for others, that’s not the case.
Experimentally, neurons with periodic-like ﬁring pattern have been recently reported in rodent EC.
In one study, it has been reported that a substantial portion of cells in EC exhibit band-like ﬁring characteristics (Krupic et al., 2012).
However, we note that based on the reported data in Krupic et al.
(2012), the band pattern is not as clear as in our model.
Spatially-stable but non-regular responses Besides the units described above
, most of the remaining units also exhibit stable spatial responses, but they do not belong to the above categories.
These response proﬁles can exhibit either one large irregular ﬁring ﬁeld; or multiple circular ﬁring ﬁelds, but these ﬁring ﬁelds do not show a regular pattern.
Experimentally these type of cells have also been observed.
In fact, it is recently reported that the non-grid spatial cells constitute a large portion of the neurons in Layer II and III of rodent EC(Diehl et al., 2017).
[Figure]
3.1. S PEED TUNING AND HEAD DIRECTION TUNING
Speed tuning We next ask how the neurons in the RNN are tuned to the inputs.
It turns out that many of the model neurons exhibit linear responses to the running speed of the animal, while some neurons show no selectivity to speed, as suggested by the near-ﬂat response functions.
Example response proﬁles are shown in Figure 3.
Interestingly, we observe that the model border cells tend to have almost zero speed-tuning (e.g., see Figure 3g,h).
Head direction tuning Furthermore, a substantial portion of the model neurons show direction tuning.
There are a diversity of direction tuning proﬁles, both in terms of the strength of the tuning and their preferred direction.
Example tuning curves are shown in Figure 3, and the direction tuning curves of a complete population are shown in the Appendix.
Interestingly, in general model neurons which show the strongest head direction tuning are only weakly spatially selective (see Figure 3a,b,c).
This suggests that there are a group of neurons which are mostly responsible for encoding the direction.
We also notice that neurons with more grid-like ﬁring can exhibit a variety of direction tuning strengths, from weak to strong (Figure 3d,e,f).
It is possible that the direction tuning of these cells comes from the spatially weakly tuned neurons.
Experimentally, the heading direction tuning in EC is well-known ( e.g. , Sargolini et al.
Both the grid and non-grid cells in EC exhibit head direction tuning (Sargolini et al., 2006).
Furthermore, the linear speed dependence of the model neurons is similar to the properties of speed cells reported recently in EC (Kropff et al., 2015).
Our result is also consistent with another recent study reporting that the majority of neurons in EC exhibit some amount of speed tuning (Hinman et al., 2016).
3.1. D EVELOPMENT OF THE TUNING PROPERTIES
We next investigate how the spatial response proﬁles evolve as learning/training progresses.
We report two main observations.
First, neurons that ﬁre selectively along the boundary typically emerge ﬁrst.
Second, the grid-like responses with ﬁner spatial tuning patterns only emerge later in training.
For visualization, we perform dimensionality reduction using the t-SNE algorithm (Maaten &
Hinton, 2008).
This algorithm embeds 100 model neurons during three phases of training (early, intermediate, and late) into a two-dimensional space according to the similarity of their temporal responses.
Here the similarity metric is taken to be ﬁring rate correlation.
In this 2D space as shown in Figure 4a, border cell representations appear early and stably persist through the end of training.
In contrast, grid-like cells typically undergo a substantial change in ﬁring pattern during training before settling into their ﬁnal grid-like representation (Figure 4b).
The developmental time line of the grid-like cells and border cells is consistent with developmental studies in rodents.
Experimentally, it is known that border cells emerge earlier in development, and they exist at about 2 weeks after the rat is born (Bjerknes et al., 2014).
The grid cells mature only at about 4 weeks after birth (Langston et al., 2010; Wills et al., 2010; Bjerknes et al., 2014).
Furthermore, our simulations suggest the reason why border cells emerge earlier in development is that computationally it may be easier to wire-up a network that gives rise to border cell responses.
3. T HE IMPORTANCE OF REGULARIZATION
We ﬁnd appropriate regularizations of the RNN to be crucial for the emergence of grid-like representations.
We only observed grid-like representations when the network was encouraged to store information while perturbed by noise.
This was accomplished by setting the speed input to zero, e.g. zero speed 90% of the time, and adding Gaussian noise to the network ( ξ i ( t ) in equation (1)); the precise method for setting the speed input to zero and the value of the noise variance is not crucial for our simulations to develop grid-like representations.
The cost function which aims to capture the penalization on the metabolic cost of the neural activity also acts as an important regularization.
Our simulations show that the grid-like representation did not emerge without this metabolic cost.
Instead, most of the units in the network exhibit border-like responses.
In Figure 5, we show typical simulation results for a square environment, with and without proper regularization.
Our results are consistent with the general notion on the importance of incorporating proper constraint for learning useful representations in neural networks (Bengio et al., 2013).
Furthermore, it suggests that, to learn a model with response properties similar to neural systems it may be necessary to incorporate the relevant constraints, e.g. , noise and metabolic cost.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
m en
[Figure]
on
[Figure] [Figure]
3. E RROR CORRECTION AROUND THE BOUNDARY
One natural question is whether the trained RNNs are able to perform localization when the path length exceeds the typical length during training (500 steps), in particular given that noise in the network would gradually accumulate, leading to a decrease in localization performance.
We test this by simulating paths of several orders of magnitude longer. Somewhat surprisingly, we ﬁnd the RNNs still perform well (Figure 6b). In fact, the squared error (averaged over every 10000 steps)
[Figure] appears to be stable. The spatial response proﬁles of individual units also remain stable.
This implies that the RNNs have acquired intrinsic error-correction mechanisms during training.
As shown earlier, during training some of the RNN units develop boundary-related ﬁring (Figure 2c), presumably by exploiting the change of input statistics around the boundary. We hypothesize that boundary interactions may enable error-correction through signals based on these boundary-related activities.
Indeed, we ﬁnd that boundary interactions can dramatically reduce the accumulated error (Figure 6a).
Figure 6a shows that, without boundary interactions, on average the squared error grows roughly linearly as expected, however, interactions with the boundaries substantially reduce the error, and more frequent boundary interactions can reduce the error further.
Error-correction on grid cells via boundary interactions has been proposed (Hardcastle et al., 2015), however, we emphasize that the model proposed here develops the grid-like responses, boundary responses and the error-correction mechanisms all within the same neural network, thus potentially providing a unifying account of a diverse set of phenomena.
D ISCUSSION
In this paper, we trained RNNs to perform path integration (dead-reckoning) in 2D arenas.
We found that after training RNNs with appropriate regularization, the model neurons exhibit a variety of spatial and velocity tuning proﬁles that match neurophysiology in EC.
What’s more, there is also similarity in terms of when these distinct neuron types emerge during training/development.
The EC has been long thought to be involved in path integration and localization of the animal’s location (Moser et al., 2008).
The general agreement between the different responses properties in our model and the neurophysiology provide strong evidence supporting the hypothesis that the neural population in EC may provide an efﬁcient code for representation self-locations based on the velocity input.
Recently, there has been increased interest in using complex neural network models to understand the neural code.
But the focus has been on using feed-forward architectures, in particular CNN (LeCun et al., 1998).
Given the abundant recurrent connections in the brain, it seems a particular fruitful avenue to take advantage of the recent development in RNN to help with neuroscience questions (Mante et al., 2013; Song et al., 2016; Miconi, 2017).
Here, we only show one instance fol- lowing this approach.
However, the insight from this work could be general, and potentially useful for other cognitive functions as well.
We note that there are a few recent studies which use place cells as the input to generate grid cells (Dordek et al., 2016; Stachenfeld et al., 2016), which are fundamentally different from our work.
In these feed-forward network models, the grid cells essentially perform dimensionality reduction based on the spatial input from place cells.
However, the main issue with these models is that, it is unclear how place cells acquire spatial tuning in the ﬁrst place.
To the contrary, our model takes the animal’s velocity as the input, and addresses the question of how the spatial tuning can be generated from such input, which are known to exist in EC (Sargolini et al., 2006; Kropff et al., 2015).
In another related study (Kanitscheider & Fiete, 2016), the authors train RNN with
LSTM units (Hochreiter & Schmidhuber, 1997) to perform different navigation tasks.
However, no grid-like spatial tuning patterns are reported.
Although our model shows a qualitative match to the neural responses observed in the EC, nonetheless it has several major limitations, with each offering interesting future research directions.
First, the learning rule we used seems to be biologically implausible.
We are interested in ﬁguring out how a more biologically plausible learning rule could give rise to a similar results (Miconi, 2017).
Second, the simulation results do not show a variety of spatial scales in grid-like cells.
Experimentally, it is known that grid cells have multiple spatial scales, that scale geometrically with a ratio 1 .
(Stensola et al., 2012).
We are investigating how to modify the model to get a hierarchy of spatial scales, perhaps by incorporating more neurons or modifying the regularization.
Finally, the dynamics of the trained network is not well-understood so far.
A better understanding would likely help identify the connectivity structure and dynamical rules that could support robust integration of the inputs.
A R ECTANGULAR ENVIRONMENT
[Figure]
B T RIANGULAR ENVIRONMENT
[Figure]
C H EXAGONAL ENVIRONMENT
[Figure]