L EARNING G RID C ELLS AS V ECTOR R EPRESENTATION OF S ELF -P OSITION C OUPLED WITH M ATRIX R EPRESENTATION OF S ELF -M OTION
A BSTRACT
This paper proposes a representational model for grid cells.
In this model, the 2D self-position of the agent is represented by a high-dimensional vector, and the 2D self-motion or displacement of the agent is represented by a matrix that transforms the vector.
Each component of the vector is a unit or a cell.
The model consists of the following three sub-models.
(1) Vector-matrix multiplication.
The movement from the current position to the next position is modeled by matrix-vector multiplication, i.e., the vector of the next position is obtained by multiplying the matrix of the motion to the vector of the current position.
(2) Magniﬁed local isometry.
The angle between two nearby vectors equals the Euclidean distance between the two corresponding positions multiplied by a magnifying factor.
Global adjacency kernel.
The inner product between two vectors measures the adjacency between the two corresponding positions, which is deﬁned by a kernel function of the Euclidean distance between the two positions.
Our representational model has explicit algebra and geometry.
It can learn hexagon patterns of grid cells, and it is capable of error correction, path integral and path planning.
I NTRODUCTION
Imagine you are walking in your living room in the dark at night without any visual cues.
Purely based on your self-motion, you know where you are while you are walking simply by integrating your self-motion.
This is called path integral (Hafting et al.
; Fiete et al.
; McNaughton et al.
You can also plan your path to the light switch or to the door.
This is called path planning (Fiete et al.
(2008); Erdem & Hasselmo (2012); Bush et al.
You need to thank your grid cells for performing such navigation tasks.
[Figure]
Figure 1(a) shows Dr. May-Britt Moser, who together with Dr. Edvard Moser, won the 2014 Nobel
Prize for Physiology or Medicine, for their discovery of the grid cells (Hafting et al.
; Fyhn et al.
; Yartsev et al.
; Killian et al.
; Jacobs et al.
; Doeller et al.
in 2005.
Their thesis advisor, Dr. John O’keefe, shared the prize for his discovery of the place cells (O’Keefe (1979)).
Both the place cells and grid cells are used for navigation.
The discoveries of these cells were made by recording the activities of the neurons of the rat when it moves within a square region.
See Figure 1(b).
Some neurons in the Hippocampus area are place cells.
Each place cell ﬁres when the rat moves to a particular location, and different place cells ﬁre at different locations.
The whole collection of place cells cover the whole square region.
The discovery of grid cells was much more surprising and unexpected.
The grid cells exist in the Entorhinal cortex.
Each grid cell ﬁres at multiple locations, and these locations form a regular hexagon grid.
See Figure 1(c).
The grid cells have been discovered across mammalian species, including human.
See Figure 1(d).
In this paper, we propose a representational model to explain the hexagon patterns of the grid cells, and to explain how the grid cells perform path integral and path planning.
We shall show that the grid cells are capable of error correction, which provides a justiﬁcation for the grid cells.
[Figure]
Figure 2 illustrates our model.
The 2D self-position of the agent is represented by a highdimensional vector, and the 2D self-motion or displacement of the agent is represented by a matrix that acts on the vector.
Each component of the vector is a unit or a cell.
In Figure 2, x denotes the 2D self-position, v ( x ) is the high-dimensional vector representation of the 2D x .
∆ x is the self-motion or one-step displacement.
M (∆ x ) is the matrix representation of ∆ x .
The model consists of the following three sub-models.
(1) Vector-matrix multiplication.
The movement from the current position x to the next position x + ∆ x is modeled by matrix-vector multiplication, i.e., the vector of the next position v ( x + ∆ x ) is obtained by multiplying the matrix of the motion, M (∆ x ) , to the vector of the current position v ( x ) .
(2) Magniﬁed local isometry.
The angle between two nearby vectors equals the Euclidean distance | ∆ x | between the two corresponding positions multiplied by a magnifying factor.
Global adjacency kernel.
The inner product between two vectors (cid:104) v ( x ) , v ( y ) (cid:105) measures the adjacency between the two corresponding positions, which is deﬁned by a kernel function f of the Euclidean distance | x − y | between the two positions x and y .
One additional feature is that the whole vector v ( x ) is partitioned into multiple sub-vectors, and each sub-vector is driven by an associated sub-matrix.
The whole system is like a multi-arm clock, with each arm rotating at a magniﬁed speed and spanning a 2D sub-manifold on a high-dimensional sphere.
Our experiments show that sub-models (1) and (2) are sufﬁcient for the emergence of the hexagon grid patterns of the grid cells.
Sub-model (2) makes the vector representation robust to noises or errors due to the magniﬁcation of the distance.
Sub-model (3) enables unique decoding of the position from the vector, because f is unimodal and peaked at 0, and it may serve as the link between the grid cells and place cells.
Together with sub-model (1), sub-model (3) enables path integral and path planning, because the adjacency (cid:104) v ( x ) , v ( y ) (cid:105) informs the Euclidean distance | x − y | .
All the three sub-models can be implemented by one-layer neural networks.
C ONTRIBUTIONS AND RELATED WORK
The following are the contributions of our work.
(1) We propose a representational model for grid cells, where the self-position is represented by a vector and the self-motion is represented by a matrix that acts on the vector.
(2) We show that our model can learn hexagon grid patterns.
(3) We show our model is capable of path integral, path planning, and error correction.
Many mathematical and computational models (Burak & Fiete (2009); Sreenivasan & Fiete (2011);
Blair et al.
(2007); de Almeida et al.
(2009)) have been proposed to explain the formation and function of grid cells.
Compared to previous computational models on grid cells, our model only makes very generic assumptions about the algebra and geometry of the representational scheme, without assuming Fourier plane waves or clock arithmetics.
Recently, deep learning models (Banino et al.
; Cueva & Wei (2018)) have been proposed to learn the grid-like units for navigation.
Our work was inspired by them.
Compared with these models, our model has explicit algebra and geometry.
In terms of algebra, our model has explicit matrix representation of self-motion, and the change of self-position is modeled by vector-matrix multiplication.
In terms of geometry, our model assumes that the vector rotates while the agent moves, and our model assumes magniﬁed local isometry and global adjacency kernel based on the angles between the vectors.
Expressing the adjacency kernel as the inner product between the vectors is related to the kernel trick (Cortes & Vapnik (1995)), random Fourier basis (Ng et al.
and spectral clustering (Ng et al.
R EPRESENTATIONAL MODEL OF GRID CELLS
Consider an agent navigating within a domain D = [0 , 1] × [0 , 1] (actually the shape does not matter, and it can be R 2 ).
We can discretize D into an N × N lattice.
N = 40 in our experiments.
Let x = ( x 1 , x 2 ) ∈
D be the self-position of the agent.
x is 2D. Suppose the agent wants to represent its self-position by a d -dimensional hidden vector v ( x ) .
We introduce the following three sub-models
. 3. S UB - MODEL 1 ABOUT MOTION ALGEBRA : VECTOR - MATRIX MULTIPLICATION
Suppose at a position x , the self-motion or one-step displacement is ∆ x , so that the agent moves to x + ∆ x after one step.
We assume that
[Equation]
where M (∆ x ) is a d × d matrix that depends on ∆ x .
While v ( x ) is the vector representation of the self-position x , M (∆ x ) is the matrix representation of the self-motion ∆ x .
We can illustrate the motion model by the following diagram:
[Equation]
See Figure 2(1).
Both v ( x ) and M (∆ x ) are to be learned.
We can discretize ∆ x , and learn a motion matrix M for each ∆ x .
We can also learn a parametric model for M .
To this end, we can further parametrize M =
I + ˜ M (∆ x ) such that each element of ˜ M (∆ x ) is a quadratic (or polynomial) function of ∆ x = (∆ x 1 , ∆ x 2 ) :
[Equation]
where the coefﬁcients β are to be learned.
The above may be considered a second-order Taylor expansion which is expected to be accurate for small ∆ x ∈ ∆ , where ∆ is the allowed collection of one-step displacements, e.g., ± 3 grid points in each direction on the 40 × 40 grid.
The motion model can be considered a linear recurrent neural network (RNN).
However, if we are to interpret M as the weight matrix, then the weight matrix is dynamic because it depends on the motion ∆ x .
One may implement it by discretizing ∆ x , so that we have a ﬁnite set of ∆ x , and thus a ﬁnite set of M (∆ x ) .
Then at each time step, the RNN switches between the ﬁnite set of motion matrices.
This is like the gearing operation of a multi-speed bicycle.
D
ISENTANGLED BLOCKS OR MODULES
For the sake of estimation accuracy and computational efﬁciency, we further assume that M (∆ x ) is block diagonal, i.e., we can divide v into K blocks of sub-vectors, v = ( v ( k ) , k
= 1 , ..., K ) , and v ( k ) ( x + ∆ x ) =
M ( k ) (∆ x ) v ( k ) ( x ) .
That is, the vector v consists of sub-vectors, each of which rotates in its own subspace, so that the dynamics of the sub-vectors are disentangled.
The assumption of disentangled blocks in our model is related to the modular organization of grid cells in neuroscience (Stensola et al.
(2012)), where a module refers to a region of grid cells where all cells share similar grid scales and orientations.
3. S UB - MODEL 2 ABOUT LOCAL GEOMETRY : MAGNIFIED LOCAL ISOMETRY
The above motion algebra alone is not sufﬁcient for learning the vector matrix representations, because a trivial solution is that all the v ( x ) are the same, and the matrix M (∆ x ) is always identity.
We need to properly displace the vectors v ( x ) .
So we shall model (cid:104) v ( x ) , v ( y ) (cid:105) both locally and globally.
Let d be the dimensionality of v ( k ) , i.e., the number of grid cells within the k -th block.
For the local geometry, we assume that for each block,
[Equation]
for all x and ∆ x such that α k | ∆ x | 2 ≤ c , i.e., ∆ x ∈
[Equation] [Equation]
k )
= {
[Equation]
x :
[Equation]
k | ∆ x | 2 ≤ c } .
In our experiments, we take c = 1 . 5 .
α k can be either designed or learned.
Based on sub-model (4), (cid:107) v ( k ) ( x ) (cid:107)
=
d for every x .
The inner product on the left hand side is d cos(∆ θ ) where ∆ θ is the angle between v ( k ) ( x ) and v ( k ) ( x + ∆ x ) .
1 − α k | ∆ x | 2 on the right hand side may be considered a second order Taylor expansion of a function f
( r ) such that f (0) = 1 , f (cid:48) (0)
= 0 , i.e., 0 is the maximum, and f (cid:48)(cid:48) (0)
=
− 2 α k .
It is also an approximation to cos( √ 2 α k | ∆ x | ) .
Let ω k = √ 2 α k , we have
Magniﬁed local isometry : ∆ θ = ω k | ∆ x | , (5) i.e., the angle between v ( x ) and v ( x + ∆ x ) magniﬁes the distance | ∆ x | by a factor of ω k uniformly for all x .
See Figure 2(2).
The factor ω k deﬁnes the metric of block k .
3. R OTATION AND PROJECTION
Since (cid:107) v ( k ) ( x ) (cid:107) is a constant for all x , M ( k ) (∆ x ) is an orthogonal matrix, and the self-motion is represented by a rotation in the d -dimensional space.
ω k | ∆ x | is the angular speed of rotation of the sub-vector k .
( v ( k ) ( x ) , ∀ x ) forms a 2D sub-manifold on the sphere in the d -dimensional space.
v ( k ) is like an arm of a clock except that the clock is not a 1D circle, but a 2D sub-manifold.
This 2D sub-manifold becomes a local codebook for the 2D positions within a local neighborhood.
For a vector v ( k ) , we can decode its position by projecting it onto the 2D sub-manifold, to get ˆ x = arg max x (cid:104) v ( k ) , v ( k ) ( x ) (cid:105) where the maximization is within the local neighborhood.
3. E RROR CORRECTION
The neurons are intrinsically noisy and error prone.
For ω k = √ 2 α k (cid:29) 1 , the magniﬁcation offers error correction because v ( k ) ( x ) and v ( k ) ( x + ∆ x ) are far apart, which is resistant to noises or corruptions.
That is, projection to the 2D sub-manifold codebook removes the noises.
Speciﬁcally, suppose we have a vector u = v ( k ) ( x ) + (cid:15) , where (cid:15) ∼ N(0 , s 2 I d ) , and I d is the d -dimensional identity matrix.
We can decode x from u based on the codebook by maximizing (cid:104) u, v ( k ) ( y ) (cid:105) =
(cid:104) v ( k ) ( x ) , v ( k ) ( y ) (cid:105) + (cid:104) (cid:15), v ( k ) ( y ) (cid:105) =
d (1 − α k | y − x | 2 )+ √ dsZ over y that is within a local neighborhood of x , where Z ∼ N(0 , 1) .
The optimal y will be close to x because if y deviates from x by ∆ x , then the ﬁrst term will drop by dα k | ∆ x | 2 , which cannot be made up by the second term √ dsZ due to noises, unless the noise level s is extremely large.
From the above analysis, we can also see that the larger d is, the more resistant the system is to noise, because the ﬁrst term is of order d and the second term is of order √ d .
In addition to additive noises, the system is also resistant to multiplicative noises including dropout errors, i.e., multiplicative Bernoulli 0/1 errors.
The dropout errors may occur due to noises, aging, or diseases.
They may also be related to the asynchronous nature of the neuron activities in computing.
3. H EXAGON GRID PATTERNS
While the magniﬁed local isometry enables error correction, it also causes global periodicity.
Because the angle between nearby v ( k ) ( x ) and v ( k ) ( x + ∆ x ) is magniﬁed, when we vary x , the vector v ( k ) ( x ) will rotate at a magniﬁed speed ω k | ∆ x | , so that it quickly rotates back to itself, like an arm of a clock.
Thus each unit of v ( k ) ( x ) is periodic with ω k determining the periodicity.
Our experiments show that sub-models (1) and (2) are sufﬁcient for the emergence of hexagon grid patterns.
In fact, we have the following analytical solution (see Appendix for a proof):
Theorem 1.
Let e ( x ) =
( e i (cid:104) a j ,x (cid:105) , j = 1 , 2 , 3) (cid:62) , and a 1 , a 2 , a 3 are three 2D vectors so that the angle between a i and a j is 2 π/ 3 for ∀
i (cid:54) =
j and | a j | = 2 √ α for ∀ j .
Let C be a random 3 × 3 complex matrix
such that C ∗ C
= I .
Then v ( x ) =
Ce ( x ) , M (∆ x ) =
C diag( e (∆ x ))
C ∗ satisfy equation 1 and equation 4 approximately for all x and small ∆ x . v ( x ) amounts to a 6 -dimensional real vector.
Since the angle between a i and a j is 2 π/ 3 for ∀
i (cid:54) =
j , patterns of v ( x ) over x have hexagon periodicity.
Moreover, the scale of the patterns is controlled by the length of a j , i.e., the scaling parameter α .
We want to emphasize that sub-models (1) and (2) are about local ∆ x , where we do not make any assumptions about global patterns, such as Fourier basis.
In contrast, the solution in the above theorem is global.
That is, our model assumes much less than the solution in the theorem.
Our experiments show that the hexagon patterns will emerge as long as the number of units is greater than or equal to 6.
3. S UB - MODEL 3 ABOUT GLOBAL GEOMETRY : ADJACENCY KERNEL
Because of the periodicity, each block ( v ( k ) ( x ) , ∀ x ) does not form a global codebook of 2D positions, i.e., there can be x (cid:54) =
y , but v ( k ) ( x ) = v ( k ) ( y ) , i.e., v ( k ) does not encode x uniquely.
We can combine multiple blocks to resolve the global ambiguity.
Speciﬁcally, let v ( x ) =
( v ( k ) ( x ) , k = 1 , ..., K ) be the whole vector, we assume the following global adjacency sub-model for the whole vector:
[Equation]
where ( v ( x ) , M (∆ x ) , α k , ∀ x, ∆ x, k ) are to be learned.
Recall d is the number of grid cells in each block, and K is the number of blocks.
f ( r ) is the adjacency kernel that decreases monotonically as the Euclidean distance r = | x − y | increases.
One example of f is the Gaussian kernel f ( r ) =
exp (cid:0) − r 2 / 2 σ 2 (cid:1) .
Another example is the exponential kernel f ( r ) =
exp ( − r/σ ) .
As a matter of normalization, we assume f (0) = 1 , which is the maximum of f ( r ) .
Since f (0) = 1 , (cid:107) v ( x ) (cid:107) 2 =
Kd for any x , and (cid:104) v ( x ) , v ( y ) (cid:105) =
( Kd )
cos θ , where θ is the angle between v ( x ) and v ( y ) , and we have
Global adjacency :
[Equation]
The angle between any two vectors is always less than π/ 2 .
See Figure 2(3).
By ﬁtting the multiple sub-vectors together, we still retain the error correction capacity due to magniﬁed local isometry, meanwhile we eliminate the ambiguity by letting (cid:104) v ( k ) ( x ) , v ( k ) ( y ) (cid:105) for different k cancel each other out by destructive interference as y moves away from x , so that we obtain unique decoding of positions.
Let C = { v ( x ) , x ∈ D } be the codebook sub-manifold, error correction of a vector u is obtained by projection onto C : arg max v ∈ C (cid:104) u, v (cid:105) .
The whole vector v is like a K -arm clock, with each v ( k ) being an arm rotating at a speed ω k | ∆ x | = √ 2 α k | ∆ x | .
3. L OCALIZATION AND HEAT MAP
( v ( x ) , ∀ x ) forms a global codebook for x .
It is a 2D sub-manifold on the sphere in the ( Kd ) - dimensional space.
For a vector v , we can decode its position by its projection on the codebook manifold.
Since f ( r ) is monotonically decreasing, h ( x ) = (cid:104) v, v ( x ) (cid:105) gives us the heat map to decode the position of v uniquely.
Let the decoded position be ˆ x , then ˆ x = arg max x (cid:104) v, v ( x ) (cid:105) .
We can obtain the one-hot representation δ ˆ x of ˆ x by non-maximum suppression on the heat map h ( x ) .
Let V =
( v ( x ) , ∀ x ) be the ( Kd )
× N 2 matrix (recall the domain D
= [0 , 1] 2 is discretized into an
N × N lattice), where each column is a v ( x ) .
We can write the heat map h ( x ) =
(cid:104) v, v ( x ) (cid:105) as a
N 2 -dimensional vector h
= V (cid:62) v , which serves to decode the position x encoded by v .
Conversely, for a one-hot representation of a position x , i.e., δ x , which is a one-hot N 2 -dimensional vector, we can encode it by v =
V δ x .
Both the encoder and decoder can be implemented by a linear neural network with connection weights V and V (cid:62) respectively, as illustrated by the following diagram:
Localization :
[Equation]
decoding
[Equation]
Note that in decoding v → h ( x ) → δ x and encoding δ x → v , we do not represent or operate on the 2D coordinate x explicitly, i.e., x itself is never explicitly represented, although we may use the notation x in the description of the experiments.
3. P ATH INTEGRAL
Path integral (also referred to as dead-reckoning) is the task of inferring the self-position based on self-motion (e.g., imagine walking in a dark room).
Speciﬁcally, the input to path integral is a previously determined initial position x 0 and motion sequences { ∆ x 1 , ..., ∆ x T } , and the output is the prediction of one’s current position x T .
We ﬁrst encode the initial position x 0 as v ( x 0 ) .
Then, by the motion model, the hidden vector v ( x T ) at time T can be predicted as:
[Equation]
We can then decode x T from v ( x T ) .
3. P ATH PLANNING
Our representation system can plan direct path from the starting position x 0 to the target position y by steepest ascent on the inner product (cid:104) v ( x ) , v ( y ) (cid:105) , i.e., let v 0 = v ( x 0 ) , the algorithm iterates
[Equation]
where ∆ is the set of allowable displacements ∆ x .
When a rat does path planning, even if it is not moving, its grid cells are expected to be active.
This may be explained by the above algorithm.
In path planning, the rat can also fantasize bigger step sizes that are beyond its physical capacity, by letting ∆ include physically impossible large steps.
In general, our representation scheme ( v ( x ) , M (∆ x ) , f ( | x − y | )) mirrors ( x, ∆ x, | x − y | ) .
Thus the learned representation is capable of implementing existing path planning algorithms in robotics (Siegwart et al.
(2011)) even though our system does not have explicit coordinates in 2D (i.e., the 2D coordinates x are never explicitly represented by two neurons).
3. P LACE CELLS AND SCALE
For each x , we may interpret (cid:104) v ( x ) , v ( y ) (cid:105) /
( Kd ) =
f ( | x − y | ) as a place cell whose response is f ( | x − y | ) if the agent is at y , or at least, we may consider f ( | x − y | ) as an internal input to the place cell based on self-motion, in addition to external visual cues.
For the Gaussian kernel f ( r ) =
exp( − r 2 / 2 σ 2 ) , the choice of σ determines the scale.
In path integral, for accurate decoding of the position x from the vector v in the presence of noises, we want σ to be small so that f ( r ) drops to zero quickly.
However, in path planning, for a vector v , we also want to know the Euclidean distance between its position x to a target position y , which may be far away from x .
The distance is | x − y | = f − 1 ( (cid:104) v, v ( y ) (cid:105) /
( Kd )) .
For accurate estimation of long distance in the presence of noises, we need σ to be large, so that the slope of f − 1 is not too big.
Perhaps we need multiple f ( r ) with different σ , and for each f ( r ) we have ( Kd ) f
( r ) = (cid:80)
Kk =1 γ k (cid:104) v ( k ) ( x ) , v ( k ) ( y ) (cid:105) , where different f ( r ) have different coefﬁcients ( γ k ) while sharing the same ( v ( k ) ( x )) .
We shall study this issue in future work.
3. G ROUP REPRESENTATION IN MOTOR CORTEX
Where does the self-motion ∆ x come from?
It comes from the movements of head and legs, i.e., each step of navigation involves a whole process of path integral and path planning of the movements of head and legs (as well as arms).
In general, we can use the same system we have developed for the movements of head, legs, arms, ﬁngers, etc.
Their movements form groups of actions.
In navigation, the movements belong to 2D Euclidean group.
In body movements, the movements belong to various Lie groups.
Our method can be used to learn the representational systems of these groups, and such representations may exist in motor cortex.
We leave this problem to future investigation.
L EARNING REPRESENTATION
The square domain D is discretized into a 40 × 40 lattice, and the agent is only allowed to move on the lattice.
We can learn ( v ( x ) , ∀ x ) and ( M (∆ x ) , ∀ ∆ x ) (or the β coefﬁcients that parametrize M ) by minimizing the following loss functions.
For sub-model (1) on vector-matrix multiplication, the loss function is
[Equation]
12) where x is sampled from the uniform distribution on D = [0 , 1] 2 , and ∆ x is sampled from the uniform distribution within a certain range ∆ 1 (3 grid points in each direction in our experiments).
The above motion loss is a single-step loss.
It can be generalized to multi-step loss
[Equation]
where (∆ x t , t = 1 , ..., T ) is a sequence of T steps of displacements, i.e., a simulated trajectory.
For sub-model (2) on magniﬁed local isometry, the loss function is
[Equation]
[Equation]
where for ﬁxed α k , ∆ x is sampled uniformly within the range ∆ 2 ( α k )
=
{ ∆ x : α k | ∆ x | 2 ≤ c } ( c = 1 . 5 in our experiments).
We can deﬁne L 2 = (cid:80)
Kk =1 L 2 ,k .
For sub-model (3) on global adjacency kernel, the loss function is
[Equation]
(15) where both x and y are sampled uniformly from D .
Let v ( x ) =
( v i ( x ) , i = 1 , ..., Kd ) , we also impose a regularization loss L 0 = (cid:80) Kdi =1 ( E x
[ v i ( x ) 2 ] − 1) 2 , to enforce uniform energy among the grid cell.
It also helps to break the symmetry caused by the fact that the loss function is invariant under the transformation v ( x ) →
Qv ( x ) , ∀ x , where Q is an arbitrary orthogonal matrix.
This loss is not crucial though, so we will make it implicit for the rest of the paper.
Fixing the magnitude of E x [ v i ( x )] within a certain range is biologically plausible, because v i ( x ) is a single cell.
For mathematical and computational convenience, we can also normalize v ( x ) so that (cid:107) v ( x ) (cid:107) 2 = 1 , (cid:104) v ( x ) , v ( y ) (cid:105) =
f ( | x − y | ) , (cid:104) v ( k ) ( x ) , v ( k ) ( x + ∆ x ) (cid:105) =
(1 − α k | ∆ x | 2 ) /K , and E x [ v i ( x ) 2 ]
= 1 / ( Kd ) .
When learning a single block, we can normalize v ( k ) ( x ) so that (cid:107) v ( k ) ( x ) (cid:107) 2 = 1 , (cid:104) v ( k ) ( x ) , v ( k ) ( x + ∆ x ) (cid:105) =
1 − α k | ∆ x | 2 and E x
[ v i ( x ) 2 ] = 1 /d .
The total loss function is a linear combination of the above losses, where the weights for combining the losses are chosen so that the weighted losses are of the same order of magnitude.
The loss function is minimized by Adam optimizer (Kingma & Ba (2014))
(lr = 0 . 03 ) for 6 ,iterations.
A batch of 30 , 000 examples, i.e., ( x, ∆ x t , t = 1 , ..., T )) for L 1 , ( x, ∆ x ) for L 2 , and ( x, y ) for L 3 are sampled at each learning iteration as the input to the loss function.
In the later stage of learning ( ≥ 4 , 000 iterations), (cid:107) v ( x ) (cid:107) = 1 is enforced by projected gradient descent, i.e., normalizing each v ( x ) after the gradient descent step.
E XPERIMENTS 5. L EARNING SINGLE BLOCKS : HEXAGON PATTERNS AND METRICS
[Figure]
We ﬁrst learn a single block with ﬁxed α k by minimizing L 1 + λ 2 L 2 ,k
(we shall drop the subscript k in this subsection for simplicity).
Figure 3 shows the learned units over the 40 × 40 lattice of x .
Figure 3(a) shows the learned results with 6 units and different values of α .
The scale or metric of the lattice is controlled by α .
The units within a block have patterns with similar scale and arrangement, yet different phases.
Figure 3(b) shows the learned results with 100 units and α = 72 , indicating that the grid-like patterns are stable and easy to learn even when the number of units is large.
5. L EARNING MULTIPLE HEXAGON BLOCKS AND METRICS
[Figure]
We learn multiple blocks by minimizing L 1 + λ 2 L 2 + λ 3 L 3 .
Instead of manually assigning α k , we learn α k by gradient descent, simultaneously with v and M .
In Figure 4(a), we show the learned units v ( x ) over the 40 × 40 lattice of x and the learned metrics α k .
A Gaussian kernel with σ = 0 .
08 is used for the global adjacency measure f
( | x − y | ) .
Block size is set to 6 and each row shows the learned units belonging to the same block.
The scales of the ﬁring ﬁelds are controlled by the learned α k .
Figure 4(b) illustrates the combination of multiple blocks.
For the localization model, given a vector v , the heat map of a single block (cid:104) v ( k ) , v ( k ) ( x ) (cid:105) has periodic ﬁring ﬁelds and cannot determine a location uniquely.
However, ambiguity disappears by combing the heat maps of multiple blocks, which have ﬁring ﬁelds of multiple scales and phases that add up to a Gaussian kernel (cid:104) v, v ( x ) (cid:105) .
The Gaussian kernel informs the place cell, by which the location is determined uniquely.
For a motion ∆ x , every block rotates in its own subspace with motion matrix M ( k ) (∆ x ) , resulting in phase shifting in the heat map of each single block.
In the subsequent experiments, we shall learn the grid cells by minimizing L 1 + λ 3 L 3 for simplicity.
[Figure]
ATH
[Figure]
Figure 5(a) shows an example of path integral result (time duration ˜ T = 40 ), where we use single step motion loss L 1 ,T =1 in learning.
Gaussian kernel with σ = 0 .
08 is used as the adjacency measure in L 3 .
We ﬁnd single-step loss is sufﬁcient for performing path integral.
The mean square error remains small ( ∼ 1 . 2 grid) even after 400 steps of motions (ﬁgure 5(b)).
The error is averaged over 1 , 000 episodes.
The motion loss can be generalized to multi-step L 1 ,T , as shown by equation 13.
In Figure 5(b), we show that multi-step loss can improve the performance slightly.
In Figure 5(c) we compare the learned models with ﬁxed number of units (96) but different block sizes.
We also compare the performance of models using Gaussian kernel ( σ = 0 .
and exponential kernel ( σ = 0 .
3 ) as the adjacency measure in the localization model.
The result shows that models with Gaussian kernel and block size ≥ 3 , and with exponential kernel and block size ≥have performances comparable to the model learned without block-diagonal assumption (block size = 96 ).
5. P ATH PLANNING
For path planning, we assume continuous x and ∆ x .
First, we design the set of allowable motions ∆ .
For a small length r , we evenly divide [0 , 2 π ] into n directions { θ i , i = 1 , ..., n } , resulting in n candidate motions ∆ = { ∆ x i = ( r cos( θ i ) , r sin( θ i )) , i = 1 , ..., n } .
These n small motions serve as motion basis.
Larger motion can be further added to ∆ by estimating the motion matrix
M ( k ∆ x i ) =
M k (∆ x i ) .
The starting position and destination can also be any continuous values, where the encoding to the latent vector is approximated by bilinear interpolation of nearest neighbors on the lattice.
In the experiments, we choose r = 0 .
05 and n = 100 , and add another set of motions with length 0 .
025 to enable accurate planning.
The system is learned with exponential kernel ( σ = 0 .
3 ) as global adjacency to encourage connection of long distance.
Figure 6(a) shows planning examples with six settings of motion ranges ∆ .
Including larger motions accelerates the planning process so that it ﬁnishes with less steps.
We deﬁne one episode to be a success if the distance between the end position and the destination is less than 0.025.
We achieve a success rate of > 99% over 1 ,episodes for all the six settings.
[Figure]
The learned system can also perform planning with obstacles, where the global adjacency between the agent’s current position and an obstacle serves as a repulsive forces.
Speciﬁcally, suppose z is an obstacle to avoid, the agent can choose the motion ∆ x t at time t by
[Equation]
(16) where a and b are the scaling and annealing parameters.
Figure 6(b) shows the planning result with a dot obstacle laid on the direct path between the starting position and destination, with tuning of a and b .
We choose a = 0 .
5 and b = 6 in subsequent experiments.
Now suppose we have more complicated obstacles { z i } mi =1 .
They can be included by summing over the kernels of every obstacle { a (cid:104) v ( z i ) , M (∆ x ) v t (cid:105) b } mi =1 and choosing ∆ x t at time t by ∆ x t = arg max ∆ x ∈ ∆ (cid:2) (cid:104) v ( y ) , M (∆ x ) v t (cid:105) − (cid:80) mi =1 a (cid:104) v ( z i ) , M (∆ x ) v t (cid:105) b (cid:3) .
Figure 6(c) shows some examples, where the obstacles mimicking walls, large objects and simple mazes.
The above method is related to the potential ﬁeld method in robotics (Siegwart et al.
D ISCUSSION : ROTATIONIST - CONNECTIONIST MODEL ?
In terms of general modeling methodology, a typical recurrent network is of the form v t = tanh( W ( v t − 1 , δ t )) , where v t is the latent vector, δ t is the input change or action.
v t − 1 and δ t are concatenated and linearly mixed by W , followed by coordinate-wise tanh non-linearity.
We replace it by a model of the form v t =
M ( δ t ) v t − 1 , where M ( δ t ) is a matrix that is non-linear in δ t .
M ( δ t ) is a matrix representation of δ t .
v t can be interpreted as neuron activities.
We can discretize the value of δ t into a ﬁnite set { δ } , and each M ( δ ) can be stored as synaptic connection weights that drive the neuron activities.
In prediction, the input δ t activates M ( δ t ) .
In planning or control, all the { M ( δ ) } are activated, and among them the optimal δ is chosen.
The matrix representation of the above model is inspired by the group representation theory, where the group elements are represented by matrices acting on the vectors (Fulton & Harris (2013)).
It underlies much of modern mathematics and holds the key to the quantum theory (Zee (2016)).
Perhaps it also underlies the visual and motor cortex, where neurons form rotating sub-vectors driven by matrices representing groups of
P ROJECT PAGE http://www.stat.ucla.edu/˜ruiqigao/gridcell/main.html
A CKNOWLEDGEMENT
We thank the three reviewers for their insightful comments and suggestions.
Part of the work was done while Ruiqi Gao was an intern at Hikvision Research Institute during the summer of 2018.
She thanks Director Jane Chen for her help and guidance.
We also thank Jiayu Wu for her help with experiments.
The work is supported by DARPA XAI project N66001-17-2-4029; ARO project
W911NF1810296; ONR MURI project N00014-16-1-2007; and a Hikvision gift to UCLA.
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research
A P ROOF FOR SECTION 3.
Proof.
( a j , j = 1 , 2 , 3) forms a tight frame in the 2D space, in that for any vector x in 2D,
[Equation]
[Equation]
Since C ∗ C =
I , we have
[Equation]
Then we have
[Equation]
23) where | a j | = 2 √ α .
The self motion from v ( x ) to v ( x + ∆ x ) is
[Equation]
[Equation] [Equation]
[Equation]
(27) where D (∆ x ) =
diag( e i (cid:104) a j , ∆ x (cid:105) , j = 1 , 2 , 3) .
If K > 1 and block size = 6 , we can ﬁt the multiple blocks together by a Fourier expansion of the kernel function
[Equation]
B H EXAGON GRID PATTERNS AND METRICS B. S IMULATED INPUT DATA
We obtain input data for learning the model by simulating agent trajectories with the number of steps equal to ˜ T in the square domain D .
D is discretized into a 40 × 40 lattice and the agent is only allowed to move on the lattice.
The agent starts at a random location x 0 .
At each time step t , a small motion ∆ x t ( ≤ 3 grids in each direction) is randomly sampled with the restriction of not leading the agent outside the boundary, resulting in a simulated trajectory { x 0 + (cid:80) ti =1 ∆ x i } ˜
Tt =1 .
˜ T is set to 1 , 000 to obtain trajectories that are uniformly distributed over the whole area.
Although the trajectories used for training are restricted to the lattice and with small motions, in Section 5.
we show that the learned model can be easily generalized to handle continuous positions and large motions.
In training, pairs of locations ( x, y ) are randomly sampled from each trajectory as the input to the adjacency loss L 3 , while consecutive position sequences ( x, ∆ x t , t = 1 , ..., T ) are randomly sampled as the input to the motion loss L 1 , with length speciﬁed by T (which is usually much smaller than the whole length of the trajectory ˜ T ).
B. L EARNED SINGLE BLOCK UNITS WITH DIFFERENT BLOCK SIZES
In (Blair et al.
, the grid cell response is modeled by three cosine gratings with different orientations and phases.
In our model, we learn such patterns of different scales without inserting artiﬁcial assumptions.
[Figure]
Figure 7 displays the response maps of the learned single block units with different block sizes.
For block sizes 4 and 5, the learned maps show square lattice patterns.
For block sizes greater than or equal to 6, the learned maps show hexagon lattice patterns.
B. L EARNED MULTIPLE BLOCK UNITS AND METRICS B.3. W ITH DIFFERENT BLOCK SIZES
Figure 8 displays the response maps of the multiple block units with different block sizes.
The metrics of the multiple blocks are learned automatically
. B.3. W ITH DIFFERENT SHAPES OF AREA
Figure 9 displays the response maps of the multiple block units with different shapes of the area, such as circle and triangle
. B. Q UANTITATIVE ANALYSIS OF SPATIAL ACTIVITY
We assess the spatial activity of the learned units quantitatively using measures adopted from the neuroscience literature.
Speciﬁcally, we quantify the hexagonal regularity of the grid-like patterns using the gridness score (Langston et al.
; Sargolini et al.
The measure is derived from the spatial autocorrelogram of each unit’s response map.
A unit is classiﬁed as a grid cell if its gridness score is larger than 0 .
For those units that are classiﬁed as grid cells, grid scale and orientation can be further derived from the autocorrelogram following Sargolini et al.
Figure 10(a) summarizes the results.
76 out of 96 learned units are classiﬁed as grid cells.
Most units with large learned α k are classiﬁed as grid cells, while those units with small α k are not due to the lack of a full period of hexagonal patterns.
The grid scales and orientations vary among units, while α k Learnedblocks 5 .
.92 .(a)
Block size =
α k
Learnedblocks 4 .6 .10 .11 .
.84 .93 .(b)
Block size =
α k
Learnedblocks 2 .6 .
.66 .67 .70 .88 .(c)
Block size =
α k
Learnedblocks 1 .9 .
.77 .81 .97 .(d)
Block size =remaining similar within each block.
We show the histograms of the grid orientations and scales in Figure 10(b) and 10(c) respectively.
Moreover, we average the grid scales within each block and make the scatter plot of the averaged grid scales and the learned 1 / √ α k in ﬁgure 10(d).
Interestingly, the grid scale is nicely proportional to the learned 1 / √ α k .
B. A BLATION STUDIES
We conduct ablation studies to assess various assumptions in our model.
B.5. L OSS TERMS
We learn v and M with multiple blocks by minimizing L 1 + λ 2 L 2 + λ 3 L 3 , which consists of (1) a motion loss L 1 , (2) a local isometry loss L 2 , and (3) a global adjacency loss L 3 .
Figure 11 shows the learned units when using only some of the components.
Grid-like patterns do not emerge if using only the adjacency loss L 3 , or only the isometry loss L 2 .
If using only the motion loss, the motion equation is approximately satisﬁed at the beginning of training, since both v and M are initialized from small values that are close to 0 .
The system cannot be learned without an extra term to push v ( x ) apart from each other.
Figure 11(c) shows the learned units using the adjacency loss and the motion loss, leaving out the isometry loss.
Grid-like patterns still emerge (also some strip-like patterns), although less obvious than the ones learned using the full loss.
B.5. A SSUMPTIONS OF MOTION MATRIX
In another ablation study, we drop the quadratic parametrization by β coefﬁcients and the block diagonal assumption of the motion matrix.
A separate motion matrix M (∆ x ) is learned for each displacement ∆ x , and v ( x ) is assumed to be a single block.
We use either the local isometry loss Lor the global adjacency loss L 3 in addition to the motion loss L 1 .
The results are shown in Figure 12.
With the isometry loss, the setting is similar to the one of learning single block, except that the parametrization of M (∆ x ) is dropped.
As shown in ﬁgure 12(a), the learned units resemble the ones learned with parametrized M (∆ x ) , but when the block size is large, the grid-like patterns are less obvious.
With the adjacency loss, grid-like patterns do not emerge any more.
α k
Learnedblocks 3 .6 .10 .
.93 .(e)
Block size =α k
Learnedblocks 5 .7 .
.95 .(f)
Block size =
α k
Learnedblocks 5 .6 .
.76 .(g)
Block size =
α k
Learnedblocks 3 .8 .
[Figure] [Figure]
C M ODELING EGOCENTRIC MOTION
The model can be generalized to handle egocentric motion that involves head direction.
α k
Learnedblocks 5 .5 .10 .12 .
.72 .93 .94 .(a)
Circular area α k
Learnedblocks 4 .
[Figure] [Figure]
C OUPLING TWO GRID SYSTEMS
The egocentric motion consists of angular velocity in the change of head direction and the spatial velocity along the current head direction.
We couple two grid systems, one for head direction and the other for the spatial position.
For notational simplicity, we put the “hat” notation on top of v and
M notation to denote the vector and matrix for the head direction.
Speciﬁcally, the agent has a head direction θ , which may change over time.
The agent moves along its head direction with scalar motion.
We can discretize the range of θ , [0 , 2 π ] , into ˆ N equally spaced values { θ i , i = 1 , ..., ˆ N } , and introduce a vector representation of self-direction.
That is, the agent represents its self-direction by a ˆ d -dimensional hidden vector ˆ v ( θ ) .
C.1. M OTION SUB - MODEL
Suppose at a position x , the agent has a head direction θ .
The self-motion is decomposed into (1) a scalar motion δ along the head direction θ
and then (2) a head direction rotation ∆ θ .
The self-motion ∆ x = ( δ cos θ, δ sin θ ) .
We assume
[Equation] [Equation]
, (32) where ˆ M (∆ θ ) is a ˆ d ×
ˆ d matrix that depends on ∆ θ , and M (∆ θ, ˆ v ( θ )) is a d × d matrix that depends on δ and ˆ v ( θ ) .
ˆ M (∆ θ ) is the matrix representation of the head direction rotation ∆ θ , while
M ( δ, ˆ v ( θ )) is the matrix representation of scalar motion δ along the direction θ .
[Figure]
Autocorrelogramsofthelearnedunits’sspatialratemaps α k
Autocorrelogramsofthelearnedunits 3 .44 .
[Figure] [Figure]
We can model M ( δ, ˆ v ( θ )) by an attention (or selection) mechanism:
[Equation]
[Equation]
(34) where M ( i ) ( δ ) is the matrix representation of scalar motion δ given the head direction θ i .
The inner product (cid:104) ˆ v ( θ ) , ˆ v ( θ i ) (cid:105) , that informs the angular distance between θ and θ i , serves as the attention weight.
b is an annealing (inverse temperature) parameter.
If b → ∞ , p = ( p i , i = 1 , ..., ˆ N ) becomes a one-hot vector for selection.
We can further assume that ˆ M (∆ θ ) and M ( i ) ( δ ) are block diagonal, and learn a parametric model for each of them by the second order Taylor expansion at ∆ θ and δ respectively.
C.1. L OCALIZATION SUB - MODEL
For the localization sub-model, we deﬁne the adjacency measures of self-direction and self-position separately.
Let ˆ f ( | θ 1 − θ 2 | ) be the adjacency measure between two directions θ 1 and θ 2 .
We use von Mises kernel ˆ f ( r )
= exp((cos( r ) − 1) /σ 2 ) , where ˆ f (0) = 1 .
For adjacency measure between self-positions x and y , we keep it the same as described in section 3.7.
[Figure]
C.1. L OSS FUNCTION FOR LEARNING
For learning the system, we can ﬁrst learn ˆ v and ˆ M (or the coefﬁcients that parametrize ˆ M ) by minimizing
[Equation] [Equation]
and then we learn v and M (or the coefﬁcients that parametrize M ) as before.
C. L EARNED UNITS FOR SELF - DIRECTION AND SELF - POSITION
Figure 13 shows a result of learning such an egocentric motion model by displaying the response curves of the learned units in the head direction system, ˆ v ( θ ) , for θ ∈ [0 , 2 π ] , as well as the response maps of the learned multiple block units in the self-position system, v ( x ) , for x ∈ [0 , 1] 2 .
C. C LOCK AND TIMESTAMP
We may re-purpose the head direction system as a clock, by interpreting θ ∈ [0 , 2 π ] as the time on a clock, and ˆ v ( θ ) as a timestamp for events happening over time.
This may be related to the recent neuroscience observations in Tsao et al.
D E RRORS D. E RROR CORRECTION
Unlike commonly used embedding in machine learning, here we embed a 2D position into a highdimensional space, and the embedding is a highly distributed representation or population code.
The
[Figure]
advantage of such a redundant code lies in its tolerance to errors.
We show that the learned system is tolerant to various sources of errors.
Speciﬁcally, in both path integral and path planning tasks, at every time step t , we randomly add (1) Gaussian noises or (2) dropout masks to the hidden units and see if the system can still perform the tasks well.
We ﬁnd that the decoding-encoding process (DE) is important for error correction.
That is, at each time step t , given the noisy hidden vector v t , we decode it to x t = arg max x (cid:104) v t , v ( x ) (cid:105) and then re-encode it to the hidden vector v ( x t ) .
Actually the whole process can be accomplished by projection to the codebook sub-manifold without explicit decoding, by obtaining the vector: arg max v ∈ C (cid:104) v t , v (cid:105) , where C = { v ( x ) , x ∈ [0 , 1] 2 } .
Table 1 shows the error correction results tested on path integral and path planning tasks.
Each number is averaged over 1 , 000 episodes.
We compute the overall standard deviation of { v ( x ) } for all x and treat it as the reference standard deviation ( s ) for the Gaussian noise.
For dropout noise, we set a percentage to drop at each time step.
With the decoding-encoding process, the system is quite robust to the Gaussian noise and dropout error, and the system still works even if 70% units are silenced at each step.
D. N OISY SELF - MOTION INPUT
Besides adding noises to the hidden units, we also experiment with adding noises to the self-motion ∆ x , and compare the performance of path integral with the one performed in the original 2D coordinates.
Speciﬁcally, at each time step, we add Gaussian noises to the self-motion ∆ x .
For path integral, we compute the mean square error between the predicted locations and ground truth locations.
Besides, we also compute the predicted locations using the 2D coordinates with the noisy self-motions.
Its mean square error serves as the reference error.
Table 2 shows the result, indicating that the error of the learned system is close to the error in the original 2D coordinates, i.e., our system does not blow up the noise.
[Figure]
α k
Learnedblocks 3 .6 .
.72 .80 .82 .85 .(b)
Path integral: MSE Path planning: success rate
Noise type 1 s 0 .
75 s 0 .
5 s 0 .
25 s 0 .
1 s 1 s 0 .
75 s 0 .
5 s 0 .
25 s 0 .
1 s
Gaussian (DE) 1.
Gaussian 6.
Noise type 70% 50% 30% 10% 5%
[Table]
Dropout (DE) 2.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Dropout
[Table]
E 3D ENVIRONMENT AND 1D TIME
The system can be generalized to 3D environments.
Speciﬁcally, we assume the agent navigates within a domain D
= [0 , 1] × [0 , 1] × [0 , 1] , which is discretized into a 40 × 40 × 40 lattice.
We learn a parametric model for motion matrix M in a residual form M =
I + ˜ M (∆ x ) , where ˜ M (∆ x ) is approximated by the second order Taylor expansion of ∆ x .
We ﬁrst learn a single block with ﬁxed α k by minimizing L 1 + λ 2 L 2 ,k .
A batch of 500,000 examples of ( x, y ) and ( x, ∆ x t , t = 1 , ..., T ) is sampled online at every iteration for training.
Figure 14 shows the learned units.
[Table] [Table] Next we learn multiple blocks by minimizing L 1 + λ 3 L 3 , and use the learned models to perform 3D path integral and path planning. For simplicity, we remove L 2 .
We use 8 blocks of units with block size 8 and choose Gaussian kernel ( σ = 0 . 08 ) as the adjacency measure for path integral and exponential kernel ( σ = 0 . 3 ) for path planning.
A batch of 200,000 examples is sampled online at every iteration for training.
E. 3D PATH INTEGRAL
Please see ﬁgure 15.
E. 3D PATH PLANNING
Please see ﬁgure 16.
E. 3D PATH PLANNING WITH OBSTACLES
Please see ﬁgure
. E. L EARNING IN 1D
The system can also be applied to 1D.
Inspired by Tsao et al.
(2018), the learned system in 1D may serve as a timestamp of events.
We assume domain D = [0 , 1] and discretize it into 100 time points.
A parametric M is learned by a residual form M
= I + ˜ M (∆ t ) , where each element of ˜ M (∆ t ) is parametrized as a function of (∆ t, ∆ t 2 ) .
16 blocks of hidden units with block size 6 are used.
Figure 18 visualizes the learned units over the 100 time points.
The response wave of every unit shows strong periodicity of a speciﬁc scale or frequency.
Within each block, the response waves of units have similar patterns, with different phases.