L EARNING G RID - LIKE U NITS WITH V ECTOR R EPRESENTATION OF S ELF -P OSITION AND M ATRIX R EPRESENTATION OF S ELF -M OTION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes a simple model for learning grid-like units for spatial awareness and navigation.
In this model, the self-position of the agent is represented by a vector, and the self-motion of the agent is represented by a block-diagonal matrix.
Each component of the vector is a unit (or a cell).
The model consists of the following two sub-models.
(1) Motion sub-model.
The movement from the current position to the next position is modeled by matrix-vector multiplication, i.e., multiplying the matrix representation of the motion to the current vector representation of the position in order to obtain the vector representation of the next position.
(2) Localization sub-model.
The adjacency between any two positions is a monotone decreasing function of their Euclidean distance, and the adjacency is modeled by the inner product between the vector representations of the two positions.
Both sub-models can be implemented by neural networks.
The motion sub-model is a recurrent network with dynamic weight matrix, and the localization sub-model is a feedforward network.
The model can be learned by minimizing a loss function that combines the loss functions of the two sub-models.
The learned units exhibit grid-like patterns (as well as stripe patterns) in all 1D, 2D and 3D environments.
The learned model can be used for path integral and path planning.
Moreover, the learned representation is capable of error correction.
I NTRODUCTION
Since the discovery of grid cells in the navigation system of the mammalian brain (Hafting et al.
(2005); Fyhn et al.
; Yartsev et al.
; Killian et al.
; Jacobs et al.
; Doeller et al.
(2010)), many mathematical and computational models (Burak & Fiete (2009); Sreenivasan & Fiete (2011);
Blair et al.
(2007); de Almeida et al.
(2009)) have been proposed to explain the formation and function of grid cells.
Recently, deep learning models (Banino et al.
; Cueva &
Wei (2018)) have been proposed to learn the grid-like units for navigation.
Such models consist of two sub-models.
One is the motion sub-model, which is in the form of a recurrent neural network, to capture the change of the latent vector along the trajectory of the agent.
The other is the localization sub-model, which is in the form of a feedforward neural network, to model the mapping from the latent vector to the position of the agent.
The models of grid-like units can be used for two important tasks in navigation.
One is path integral (Hafting et al.
; Fiete et al.
; McNaughton et al.
whose goal is to estimate the self-position of the agent given a sequence of it’s self-motions.
The other is path planning (Fiete et al.
(2008); Erdem & Hasselmo (2012); Bush et al.
whose goal is to plan a sequence of self-motions in order to reach a given target destination.
The model can be learned by supervised learning that minimizes the prediction error between the predicted position and the true position in path integral.
It can also be learned by reinforcement learning for path planning.
Our work is inspired by the recent deep learning models on grid cells.
However, instead of further generalizing and extending them, or making connections to the neuroscience of the grid cells, we seek to ﬁnd a simple and explicit model for learning grid-like units for spatial awareness and navigation.
Our motivation is that for such a basic system, we may only need a simple model that consists of one layer networks.
The motion sub-model is a one layer recurrent network with dynamic weight matrix, while the localization model is a one layer linear feedforward network.
Speciﬁcally, in our model, the self-position of the agent is represented by a vector of a certain dimensionality, where each component of the vector is a unit (or a cell).
The self-motion of the agent is represented by a matrix that depends on the self-motion or displacement.
When the agent moves from the current position to the next position, the vector representation undergoes a linear transformation, where the vector representation of the next position is obtained by multiplying the matrix of the self-motion to the vector representation of the current position.
For the sake of estimation accuracy and computational efﬁciency, we assume that the self-motion matrix is block diagonal.
That is, we partition the components of the vector into different blocks of sub-vectors, with each block consisting of a small number of units (e.g., 6 units), and each block undergoes its own linear transformation by a multiplication of a small matrix.
In other words, the dynamics of the blocks are disentangled.
In addition to the above motion sub-model in the form of vector-matrix multiplication, we have the localization sub-model which also embraces simplicity.
For any two positions, we deﬁne their adjacency as a kernel function of their Euclidean distance.
This kernel function is given to the learning algorithm.
It has the maximal value 1 at zero distance and decreases monotonically toas the distance increases.
In the localization sub-model, the adjacency between any two positions is given by the inner product between their vector representations.
This localization model enables the agent to decode the position of a vector for the purpose of the path integral.
Speciﬁcally, we compute the inner product between the given vector and the vectors of all the possible positions to obtain a heat map for all the possible positions, where the maximum is the inferred position.
This heat map may inform the place cells, together with visual cues of the environment.
The model also enables the agent to do path planning by a simple steepest ascent algorithm that sequentially maximizes the adjacency to the destination.
Since the adjacency is 1 at zero distance, the vector is of unit norm, and the self-motion is represented by a rotation of this unit vector on a high-dimensional sphere, where the arc length on the sphere informs the Euclidean distance in the original 2D space.
Both the motion and the localization sub-models can be viewed as simple neural networks, with the motion model being a one layer recurrent network with disentangled blocks and with dynamic weight matrix, while the localization model being a feedforward one layer linear network.
Thus, our model is a drastic simpliﬁcation of existing deep learning models of grid cells, without inserting any artiﬁcial assumptions such as Fourier basis or clock arithmetics, etc.
(Blair et al.
(2007); de Almeida et al.
We show that the learned units of the vector representation exhibit grid-like patterns as well as stripe patterns.
We also show that the learned model can do path integral and path planning.
Moreover, the learned representation is capable of error correction.
The following are the contributions of our work.
(1) We propose a novel model for learning grid-like units for navigation.
The model has explicit representations of self-position and self-motion in terms of vector and matrix.
(2) We show that the learned units exhibit periodic grid-like or strip patterns.
(3) We show that the learned model can be used for path integrals.
(4) We show that the learned model can perform path planning in the form of a simple steepest ascent algorithm.
(5) We show that the learned representation is tolerant of additive and drop out errors.
(6) We show that the model can be generalized to 1D and 3D environments, and learn meaningful grid-like patterns.
The learned model can also be used for path integral and path planning.
R ELATED WORK
Our work is inspired by recent deep learning models on grid cells (Banino et al.
; Cueva & Wei (2018)).
However, our model is much simpler and more explicit than existing models.
Our model only consists of one layer recurrent network and feedforward network, with highly interpretable sub-models of motion and localization.
It is a simple and natural model without adding artiﬁcial components such as Fourier basis.
Compared to previous computational models on grid cells, our model only assumes one layer networks, without making assumptions such as Fourier plane waves or clock arithmetics (Blair et al.
(2007); de Almeida et al.
As such, our goal is to ﬁnd a simple loss function to learn the system.
Similar to word2vec (Mikolov et al.
, the vector representation in our method may be called a place2vec.
In word2vec, the inner product of two vectors measures the similarity of the words.
In place2vec, the inner product of two vectors measures the adjacency of two positions.
Our model and the associated loss function is connected to several well-known machine learning methods.
The representation of the adjacency kernel between two positions by the inner product between their vector representations reminds us the kernel trick in support vector machine (Cortes & Vapnik (1995)), where the adjacency is the kernel function, and the vector is a high-dimensional representation of the 2D input.
The difference is that in kernel machines, the high-dimensional vector is implicit, while in our work, we aim to learn the vector representation explicitly.
The explicit representation of the kernel function in terms of inner product between vectors is related to symmetric matrix factorization (Lee & Seung (1999)), which is in turn related to spectral clustering (Ng et al.
and kernel k-means (Dhillon et al.
The motion sub-model is related to local linear embedding (Roweis & Saul (2000)), except that the vectors are related by matrix representations of self-motion.
In representation theory in mathematics (Fulton & Harris (2013)), the group elements are represented by matrices acting on the vectors.
We learn a special system of representation for representing self-position and self-motion.
M ODEL
Consider an agent navigating within a domain D = [0 , 1] × [0 , 1] .
We can discretize D into an
N × N lattice.
Let x = ( x 1 , x 2 ) ∈
D be the self-position of the agent.
Suppose the agent wants to represent its self-position by a d -dimensional hidden vector v ( x ) .
We introduce two sub-models.
3. M OTION SUB - MODEL
Suppose at a position x , the self-motion is ∆ x , so that the agent moves to x + ∆ x after one step.
We assume that
[Equation]
where M (∆ x ) is a d × d matrix that depends on ∆ x .
While v ( x ) is the vector representation of the self-position x , M (∆ x ) is the matrix representation of the self-motion ∆ x .
As we will show below, | v ( x ) | = 1 for all x , thus M (∆ x ) is a rotation matrix, and the self-motion is represented by a rotation in the d -dimensional sphere.
We can illustrate the motion model by the following diagram:
[Equation]
Both v ( x ) and M (∆ x ) are to be learned.
For the sake of estimation accuracy and computational efﬁciency, we further assume that M (∆ x ) is block diagonal, i.e., we can divide v = ( v ( k ) , k = 1 , ..., K ) into K blocks of sub-vectors, and v ( k ) ( x + ∆ x ) =
M ( k ) (∆ x ) v ( k ) ( x ) .
That is, the vector v consists of sub-vectors, each of which rotates in its own subspace.
We can learn a separate motion matrix M for each ∆ x within a limited range.
Because of the blockdiagonal form of M , we can afford to learn such motion matrices separately.
However, we can also learn a parametric model for M .
To this end, we can further parametrize M =
I + ˜ M (∆ x ) such that each element of ˜ M (∆ x ) is a quadratic (or polynomial) function of ∆ x = (∆ x 1 , ∆ x 2 ) :
[Equation]
where the coefﬁcients β are to be learned.
The above may be considered a second-order Taylor expansion which is expected to be accurate for small ∆ x .
The motion model can be considered a linear recurrent neural network (RNN).
However, if we are to interpret M as the weight matrix, then the weight matrix is dynamic because it depends on the mo- tion ∆ x .
One may implement it by discretizing ∆ x , so that we have a ﬁnite set of ∆ x , and thus a ﬁnite set of M (∆ x ) .
Then at each time step, the RNN switches between the ﬁnite set of motion matrices.
This is like the gearing operation of a multi-geared bicycle.
Another implementation is to write
M (∆ x ) v = v + B (1) (∆ x 1 v ) + B (2) (∆ x 2 v ) + B (11) (∆ x 21 v ) + B (22) (∆ x 22 v ) + B (12) (∆ x 1 ∆ x 2 v ) , where the B matrices consist of the corresponding β coefﬁcients.
We ﬁrst obtain new input vectors ∆ x 1 v , ∆ x 2 v , ..., ∆ x 1 ∆ x 2 v , and then treat B (1) , B (2) , ..., B (12) as the weight matrices operating on the new input vectors.
3. L OCALIZATION SUB - MODEL
Let A ( x, y ) be the adjacency measure between two positions x and y such that
A ( x, y ) =
f ( | x − y | ) , where f ( r ) decreases monotonically as the Euclidean distance r = | x − y | increases.
One example of f is the Gaussian kernel f ( r ) =
exp (cid:16) − r 2 2 σ 2 (cid:17) .
Another example is the exponential kernel f ( r ) =
exp (cid:0) − rσ (cid:1) .
As a matter of normalization, we assume f (0)
= 1 .
We assume the adjacency measure is given by the inner product between the hidden vectors
[Equation]
Since f (0) = 1 , | v ( x ) | = 1 for any x , and (cid:104) v ( x ) , v ( y ) (cid:105) =
cos α , where α is the angle between v ( x ) and v ( y ) .
Thus for a vector v , the distance between it’s position and x is f − 1 ( (cid:104) v, v ( x ) (cid:105) ) .
Since f ( r ) is monotonically decreasing, h ( x ) = (cid:104) v, v ( x ) (cid:105) gives us the heat map to decode the position of v .
Let the decoded position be ˆ x , then ˆ x = arg max x (cid:104) v, v ( x ) (cid:105) .
We can obtain the one-hot representation δ ˆ x of ˆ x by non-maximum suppression on the heat map h ( x ) .
The heat map h ( x ) can be combined with visual feature maps to infer the position if there are visual cues.
Let V =
( v ( x ) , ∀ x ) be the d × N 2 matrix, where each column is a v ( x ) .
We can write the heat map of a d -dimensional vector v by a N 2 -dimensional vector h
= V (cid:62) v , which serves to decode the position x encoded by v .
Conversely, for a one-hot representation of a position x , i.e., δ x , which is a one-hot N 2 -dimensional vector, we can encode it by v =
V δ x .
Both the encoder and decoder can be implemented by a linear neural network with connection weights V and V (cid:62) respectively, as illustrated by the following diagram:
Localization
[Equation]
Note that in decoding v → h ( x ) → δ x and encoding δ x → v , we do not represent or operate on the 2D coordinate x explicitly, i.e., x itself is never explicitly represented, although we may use the notation x in the description of the experiments.
L EARNING 4. L OSS FUNCTION
We can learn v and M (or the β coefﬁcients that parametrize M ) by minimizing
[Equation]
where λ > 0 is a tuning constant.
The ﬁrst term is the localization loss, and the second term is the motion loss.
In the ﬁrst expectation, we assume x and y are sampled uniformly from D .
In the second expectation, we assume that x is sampled uniformly from D , while ∆ x is sampled uniformly within a limited range.
The above loss function can be conveniently minimized by gradient descent.
We may re-parametrize
M (∆ x ) in a residual form M (∆ x ) = I + ˜ M (∆ x ) , and do gradient descent with respect to ˜ M .
In the case where ˜ M (∆ x ) is parameterized by a quadratic function of ∆ x = (∆ x 1 , ∆ x 2 ) , we can further back-propagate the gradient of ˜ M to the β coeffciients.
We impose regularization terms U ( v )
=
(cid:80) di
=1 (cid:0) E
x [ v i ( x ) 2 ] − 1 /d (cid:1) 2 to enforce uniform energy of the units for v = ( v i ( x ) , i = 1 , ..., d ) (cid:62) .
It also helps to break the symmetry caused by the fact that the loss function is invariant under the transformation v ( x ) →
Qv ( x ) , ∀ x , where Q is an arbitrary d × d orthogonal matrix.
The motion loss E x, ∆ x (cid:2) (cid:107) v ( x + ∆ x ) − M (∆ x ) v ( x ) (cid:107) 2 (cid:3) is a single-step loss.
It can be generalized to multi-step loss
[Equation]
where (∆ x t , t = 1 , ..., T ) is a sequence of T steps of displacements.
4. I MPLEMENTATION DETAILS
We obtain inputs for learning the model by simulating agent trajectories with duration ˜ T in the square domain D .
D is discretized into a 40 × 40 lattice and the agent is only allowed to move on the lattice.
The agent starts at a random location x 0 .
At each time step t , a small motion ∆ x t ( ≤grids in each direction) is randomly sampled with the restriction of not leading the agent outside the boundary, resulting in a simulated trajectory { x 0 + (cid:80) ti =1 ∆ x i } ˜
Tt =1 .
˜ T is set to 1 , 000 to obtain trajectories that are uniformly distributed over the whole area.
It is worth mentioning that although the trajectories used for training are restricted to the lattice and with small motions, in Section 5.
we show that the learned model can be easily generalized to handle continuous positions and large motions.
In training, pairs of locations ( x, y ) are randomly sampled from each trajectory as the input to the localization loss, while consecutive position sequences ( x, ∆ x t , t = 1 , ..., T ) are randomly sampled as the input to the motion loss term, with length speciﬁed by T (which is usually much smaller than the whole length of the trajectory ˜ T ).
The model is trained with Adam optimizer (Kingma & Ba (2014))
(lr = 0 . 03 ) for 6 , 000 iterations.
A batch of 30 , 000 examples (i.e., ( x, y ) and ( x, ∆ x t , t = 1 , ..., T )) is sampled from the simulated trajectories at each learning iteration as the input to the loss function.
λ is set to 0 .
1 to balance the magnitude of the two loss terms.
The weight for the regularization term is set to 5 , 000 .
In the later stage of learning ( ≥ 4 , 000 iterations), | v ( x ) | = 1 is enforced by projected gradient descent, i.e., normalizing each v ( x ) after the gradient descent step.
We use 16 blocks of units with block size 6 , resulting in 96 units in total.
E XPERIMENTS
First, we conduct experiments on simulation data to learn the units v ( x ) and motion matrices
M (∆ x ) parametrized by the β coefﬁcients.
Then, we show that the learned representation can perform path integral and path planning tasks.
Next, we show that the learned representation is capable of error correction.
Finally, we generalize the system to 1D and 3D environments.
Due to the limit of space, we put the 1D and 3D results in the Appendices B and F.
5. L EARNING SINGLE BLOCKS : HEXAGON PATTERNS AND METRICS
We begin with learning the system using a single block of units.
We derive a generic local adjacency by the second order Taylor expansion.
Then we show that learning with such a simple adjacency measure, grid-like patterns emerge.
When block size is larger than or equal to 6 , they are arranged in a very regular hexagon lattice.
The patterns remain stable and are easy to learn even when the block size is large.
Finally, we provide an analytical solution that helps understand the emergence of hexagon patterns.
5.1. L OCAL ADJACENCY AND METRIC
Consider a single block v ( k ) .
Assume
[Equation]
x − y | ) , (8) then A ( x, y ) = (cid:80)
Kk =
A k ( x, y ) , and f ( r ) =
(cid:80)
Kk =1 f k ( r ) .
For simplicity, let us assume we re-normalize v ( k ) ( x ) so that | v ( k ) ( x ) | 2 = 1 for all x .
Assume f k ( r ) is twice differentiable at the local maximum r = 0 , then f (cid:48) k
= 0 .
Let α k
=
− f (cid:48)(cid:48) k
(0) / 2 , by second order Taylor expansion, f k ( r ) ≈ 1 − α k r 2 for small r , i.e., locally, we have the adjacency for a single block:
[Equation]
Thus locally the angle or arc length ∠ ( v ( k ) ( x ) , v ( k [Equation] ( y ))
≈ √ 2 α k | x − y | , i.e., √ 2 α k is the metric of the block k .
α k can be either speciﬁed or learned.
5.1. L EARNING SINGLE BLOCKS
We learn the system using a single block of units and the generic local adjacency measure
[Equation]
Figure 1 shows the learned units over the 40 × 40 lattice of x .
Figure 1a shows the learned results with 6 units and different values of α .
The learned units have obvious grid-like patterns, which are arranged in a very regular hexagon lattice.
The scale or metric of the lattice is controlled by α .
The units within a block have patterns with similar scale and arrangement, and different phases.
Figure 1b shows the learned results with 100 units and α = 72 , indicating that the grid-like patterns are stable and easy to learn even when the number of units is large.
When block size is 4 or 5 , the learned patterns are arranged in a square lattice.
Please refer to the Appendix C for learned units with more choices of block size.
We would like to emphasize that the local adjacency (11) is merely a second order Taylor expansion, and it assumes much less than the analytical solution that we are going to introduce. [Figure]
= α =α =α =α = [Figure] =
[Figure]
5.1. A NALYTICAL SOLUTION
We provide an analytical solution.
While the analytical solution is about global patterns, our loss function is local in both localization loss and the motion loss.
The loss function is also very generic in terms of second order Taylor expansion and vector matrix multiplication.
We begin the analysis by considering a single block of units and using the generic local adjacency measure A ( x, y )
= 1 − α | x − y | 2 , which holds only when | x − y | is small.
We show that when a block of 6 units is used, there exists analytical solution v ( x ) and M (∆ x ) for the entire system.
The analytical solution has hexagon periodicity over the whole region, with the scale controlled by parameter α .
We refer to α as the scaling parameter.
Theorem 1.
Let e ( x ) =
( e i (cid:104) a j ,x (cid:105) , j = 1 , 2 , 3) (cid:62) , and a 1 , a 2 , a 3 are three 2D vectors so that the angle between a i and a j is 2 π/ 3 for ∀
i (cid:54) =
j and | a j | = 2 √ α for ∀ j .
Let C be a random 3 ×complex matrix
such that C ∗ C
= I .
Then v ( x ) =
Ce ( x ) , M (∆ x ) =
C diag( e (∆ x ))
C ∗ satisﬁes equation 1 ∀ x, ∆ x , and equation 8 ∀ x, y , approximately for small | x − y | .
Proof.
See Appendix A. v ( x ) amounts to a 6 -d real vector.
Under the condition | x − y | ≤ (cid:112) 1 . 5 /α , the adjacency measure
A ( x, y ) ≈ cos( √ 2 α | x − y | ) .
Since the angle between a i and a j is 2 π/ 3 for ∀
i (cid:54) =
j , patterns of v ( x ) over x have hexagon periodicity.
Moreover, the scale of the patterns is controlled by the length of a j , i.e., the scaling parameter α .
For multiple blocks v = ( v ( k ) , k = 1 , ..., K ) , the analytical solution exists for block size = 6 and a global translation invariant A ( x, y ) .
See Appendix A.
5. L EARNING MULTIPLE HEXAGON BLOCKS AND METRICS
Clear hexagon patterns emerge in learning single blocks with local adjacency.
However, a global adjacency is required for the agent to localize the position uniquely.
Inspired by the results in section 5.1, we propose to combine the global and local adjacencies together, and learn v and M by minimizing
[Equation] , where λ and γ are two tuning constants. A ( x, y ) is the global adjacency such as the Gaussian kernel, and A k ( x, y ) are the local adjacencies deﬁned by equation 11. We normalize A k ( x, y ) by K to ensure | v ( x ) | 2 = (cid:80) k | v ( k ) ( x ) | 2 = 1 .
Instead of manually assigning the scaling parameters α k , we learn α k by gradient descent with v and M simultaneously.
Since the local adjacency holds only when | x − y | is small, at each iteration, we sample ( x, y ) uniformly from D α k
= { ( x, y ) : | x − y | ≤ (cid:112) 1 . 5 /α k } for the k th expectation in the second term, where α k are the learned paramters at the current iteration.
In Figure 2a, we show the learned units v ( x ) over the 40 × 40 lattice of x and the learned metrics α k .
A Gaussian kernel with σ = 0 .
08 is used for the global adjacency measure.
Block size is set to 6 and each row shows the learned units belonging to the same block.
Interestingly, the learned units exhibit obvious grid-like patterns, with multiple ﬁring ﬁelds of roughly circle or ellipse shapes.
The scales of the ﬁring ﬁelds are controlled by the learned α k .
Furthermore, the ﬁring ﬁelds are arranged in a regular hexagon lattice.
Within each block, the units exhibit very similar patterns, with different phases.
In (Blair et al.
, the grid cell response is modeled by three cosine gratings with different orientations and phases.
In our model, we learn such patterns of different scales without inserting artiﬁcial assumptions.
We also try different choices of block sizes. When block size isor 5 , the learned patterns are arranged in square lattices. When block size is larger than or equal to 6 , the learned patterns are arrange in hexagon lattices. Please refer to Appendix D for more results.
[Figure]
Both localization and motion models can be interpreted as a combination of sub-blocks.
Figure 2b provides an illustration.
For the localization model, given a vector v , the heat map of a single block (cid:104) v ( k ) , v ( k ) ( x ) (cid:105) has periodic ﬁring ﬁelds and cannot determine a location uniquely.
However, ambiguities disappear by combing the heat maps of multiple blocks, which have ﬁring ﬁelds of multiple scales and phases and add up to a Gaussian kernel (cid:104) v, v ( x ) (cid:105) .
The Gaussian kernel informs the place cell, by which the location is determined uniquely.
For a motion ∆ x , every block rotates in its own subspace with motion matrix M ( k ) (∆ x ) , resulting in phase shifting in the heat map of each single block. In subsequent experiments, we only use global kernel to learn the grid cells for simplicity.
5. P ATH INTEGRAL
Path integral (also referred to as dead-reckoning) is the task of inferring the self-position based on self-motion (e.g., imagine walking in a dark room).
Speciﬁcally, the input to path integral is a previously determined initial position x 0 and known or estimated motion sequences { ∆ x 1 , ..., ∆ x T } , and the output is the prediction of one’s current position x T .
We test our learned system on this task.
We ﬁrst encode the initial position x 0 as v ( x 0 ) .
Then, by the motion model, the hidden vector v ( x T ) at time T can be predicted as:
[Equation]
That is, the learned system of ( v ( x ) , M (∆ x ) , ∀ x, ∆ x ) should satisfy the above condition.
Note that this does not imply (cid:81)
[Equation] Tt =1 ∆ x t (cid:17) .
The above equation is true only for a spatial system of ( v ( x ) , ∀ x ) .
Thus, ( v ( x ) , M (∆ x ) , ∀ x, ∆ x ) forms a whole system for spatial awareness.
By the localization model, the inferred position at time T is ˆ x T = arg max x (cid:104) v ( x T ) , v ( x ) (cid:105) .
[Figure]
Figure 3a shows an example of path integral result (time duration ˜ T = 40 ), where we use single step motion loss, unlike (Banino et al.
; Cueva & Wei (2018)), which train RNN or LSTM with long-term loss to ﬁt the path integral task.
Gaussian kernel with σ = 0 .
08 is used as the adjacency measure.
We ﬁnd single-step loss is sufﬁcient for performing path integral.
The mean square error remains small ( ∼ 1 . 2 grid) even after 400 steps of motions (ﬁgure 3b).
The error is average over 1 , 000 episodes.
The motion loss can be generalized to multi-step, as shown by equation 7.
In Figure 2b, we show that multi-step loss can improve the performance slightly.
A question is whether the assumed block-diagonal motion matrix M (∆ x ) affects the accuracy of path integral, since such assumption leads to much less parameters.
In Figure 3c we compare learned models with ﬁxed number of hidden units (96) but different block sizes.
We also compare the performance of models using Gaussian kernel ( σ = 0 .
and exponential kernel ( σ = 0 .
3 ) as the adjacency measure in the localization model
The result shows that models with Gaussian kernel and block size ≥ and with exponential kernel and block size ≥ 4 have comparative performances with the model learned without block-diagonal assumption (block size = 96 ).
5. P ATH PLANNING
Path planning is the process in which the agent plans a sequence of self-motions given a start position x 0 and a target destination y .
In the simplest scenario, where no obstacle is involved, the agent should be able to plan direct path between x 0 and y .
The learned system enables the agent to do path planning by steepest ascent according to the adjacency measure: (1) encoding x 0 and y to hidden vectors v ( x 0 ) and v ( y ) , and (2) sequentially maximizing the adjacency of the current hidden vector to v ( y ) .
Speciﬁcally, in the simplest scenario, suppose the hidden vector is v t at time t , the agent chooses a motion ∆ x t from a motion pool M by
[Equation]
) and then the hidden vector is updated by v t +1 =
M (∆ x t ) v t .
The planning process terminates if the adjacency (cid:104) v t , v ( y ) (cid:105) is close to 1 or the number of planning steps reaches a predeﬁned maximum value.
A critical question is how to choose the motion pool M .
To augment motion candidates, we no longer restrict the motion to be on the lattice.
Instead, for a small length r , we evenly divide [0 , 2 π ] into n directions { θ i , i = 1 , ..., n } , resulting in n candidate motions { ∆ x i =
( r ∗ cos( θ i ) , r ∗ sin( θ i )) , i = 1 , ..., n } .
These n small motions serve as motion basis.
Larger motion k ∆ x i can be further added to the pool by estimating the motion matrix M ( k ∆ x i ) =
M k (∆ x i ) .
The starting position and destination can also be any continuous values, where the encoding to the latent vector is approximated by bilinear interpolation of nearest neighbors on the lattice.
In the experiments, we choose r = 0 .
05 and n = 100 , and add another set of motion with length 0 .
025 to enable accurate planning.
The system is learned with exponential kernel ( σ = 0 .
3 ) for adjacency measure to encourage connection of long distance.
Figure 4a shows planning examples with six settings of motion ranges.
Including larger motions accelerates the planning process so that it ﬁnishes with less steps.
We deﬁne one episode to be a success if the distance between the end position and the destination is less than 0.025.
We achieve a success rate of > 99% over 1 ,episodes for all the six settings.
[Figure]
The learned system can also perform planning with obstacles.
Speciﬁcally, suppose z is an obstacle to avoid, the agent can choose the motion ∆ x t at time t by
[Equation]
(15) where a and b are the scaling and annealing parameters.
Figure 4 shows the planning result with a dot obstacle laid on the direct path between the starting position and destination, with tuning of a and b .
We choose a = 0 .
5 and b = 6 in subsequent experiments.
Now suppose we have more complicated obstacles { z i } mi =1 .
They can be included by summing over the kernels of every obstacle { a (cid:104) v ( z i ) , M (∆ x ) v t (cid:105) b } mi =1 and choosing ∆ x t at time t by
[Equation]
Figure 4c shows some examples, where the obstacles mimicking walls, large objects and simple mazes.
Please see more planning with obstacle results in the Appendix E.
The advantage of the proposed path planning method is its simplicity.
For simple environments, path planning can be accomplished by simple steepest ascent algorithm.
The method can be easily extended to moving target and moving obstacles.
There is no need for reinforcement learning or sophisticated optimal control.
5. E RROR CORRECTION
Unlike commonly used embedding in machine learning, here we embed a 2D position into a highdimensional space, and the embedding is a highly distributed representation or population code.
The advantage of such a redundant code is in it’s tolerance to errors.
We show that the learned system is tolerant to various errors of units.
Speciﬁcally, in both path integral and path planning tasks, at every time step t , we randomly add (1) Gaussian noises or (2) drop out masks to the hidden units and see if the system can still perform the tasks well.
We ﬁnd that the decoding-encoding process (DE) is important for error correction.
That is, at each time step t , given the noisy hidden vector v t , we decode it to x t = arg max x (cid:104) v t , v ( x ) (cid:105) and then re-encode it to the hidden vector v ( x t ) .
Table 1 shows the error correction results tested on path integral and path planning tasks.
Each number is averaged over 1 , 000 episodes.
We compute the overall standard deviation of { v ( x ) } for all x and treat it as the reference standard deviation ( s ) for the Gaussian noise.
For drop out noise, we set a percentage to drop at each time step.
With the decoding-encoding process, the system is quite robust to the Gaussian noise and drop out error, and the system still works even 70% units are silenced at each step.
[Table]
D ISCUSSION
The goal of this work is to ﬁnd a simple and natural model and the associated loss function to learn the grid-like units for navigation, in the hope of contributing to our understanding of the possible computational models of grid cells.
Our model consists of a localization sub-model and a motion sub-model.
We show that it is possible to learn grid-like units [Table] and the learned model is capable of path integral and path planning.
The vector representation is also robust to additive and drop out noises.
The model in this paper can be considered an internal system for spatial awareness.
This system may interact with the external sensory inputs such as visual cues.
For instance, the heat map can be combined with visual feature maps to infer the self-position and its vector representation.
Two key features that distinguish our model from existing models are: (1) The motion is encoded by the matrix that operates on vector representation of position.
This separates the representations of self-motion and self-position.
The inner product of two vectors, or equivalently, the arclength between the two unit vectors on the high-dimensional sphere, informs the Euclidean distance between the two underlying positions they encode.
This is crucial for path planning.
A P ROOF FOR SECTION 5.1.
Theorem 1.
Let e ( x ) =
( e i (cid:104) a j ,x (cid:105) , j = 1 , 2 , 3) (cid:62) , and a 1 , a 2 , a 3 are three 2D vectors so that the angle between a i and a j is 2 π/ 3 for ∀
i (cid:54) =
j and | a j | = 2 √ α for ∀ j .
Let C be any 3 × 3 complex matrix
such that C ∗ C
= I .
Then v ( x ) =
Ce ( x ) , M (∆ x ) =
C diag( e (∆ x ))
C ∗ satisﬁes equation∀ x, ∆ x , and equation 8 ∀ x, y , approximately for small | x − y | .
Proof.
( a j , j = 1 , 2 , 3) forms a tight frame in the 2D space, in that for any vector x in 2D,
[Equation]
[Equation]
Since C ∗ C =
I , we have
[Equation]
[Equation]
Then we have
[Equation] [Equation]
[Equation]
where ω = | a j | = 2 √ α .
The self motion from v ( x ) to v ( x + ∆ x ) is
[Equation]
where D (∆ x ) =
diag( e i (cid:104) a j , ∆ x (cid:105) , j = 1 , 2 , 3) .
If K > 1 and block size = 6 , for a translation invariance function A ( x, y ) , it can be approximated by Fourier expansion [Equation]
[Equation]
That is, we can ﬁt the multiple blocks together.
B L EARNING IN 1D, TIME 2 VEC OR TIMESTAMP
The system can also be applied to 1D.
Inspired by Tsao et al.
(2018), the learned system in 1D may serve as a time2vec for generating timestamps of events.
We assume domain D = [0 , 1] and discretize it into 100 time points.
A parametric M is learned by a residual form M
=
I + ˜ M (∆ t ) , where each element of ˜ M (∆ t ) is parametrized as a function of (∆ t, ∆ t 2 ) .
16 blocks of hidden units with block size 6 are used.
Figure 5 visualizes the learned units over the 100 time points.
The response wave of every unit shows strong periodicity of a speciﬁc scale or frequency, which is similar to the ﬁring rate of LEC single cell reported in Tsao et al.
Within each block, the response waves of units have similar patterns, with different phases.
The timestamp v ( t ) can be attached to the episode at time t .
We can recall the episode at time t based on v ( t ) .
The kernel A ( t, s ) =
(cid:104) v ( t ) , v ( s ) (cid:105) can be used to measure the temporal adjacency of events.
C L EARNED 2D SINGLE BLOCK UNITS WITH DIFFERENT BLOCK SIZES
[Figure]
D L EARNED 2D MULTIPLE BLOCK UNITS AND METRICS WITH DIFFERENT BLOCK SIZES
[Table]
(a
[Table]
{+[Table] [Table] [Figure] Learnedblocks 5 .6 .
+}
[Figure]
size =
α k
{+Learnedblocks 3 .8 .
+}
[Figure]
E 2D P ATH PLANNING WITH OBSTACLES
Please see ﬁgure 8 for more examples of 2D path planning with rectangular obstacles.
F L EARNING IN 3D ENVIRONMENTS
The system can be generalized to 3D environments.
Speciﬁcally, we assume the agent navigates within a domain D
= [0 , 1] × [0 , 1] × [0 , 1] , which is discretized into a 40 × 40 × 40 lattice.
We learn a parametric model for motion matrix M by a residual form M =
I + ˜ M (∆ x ) .
To estimate ˜ M (∆ x ) more precisely, we parametrize each element of ˜ M (∆ x ) as a cubic function of ∆ x = (∆ x 1 , ∆ x 2 , ∆ x 3 ) .
A batch of 200,000 examples of ( x, y ) and ( x, ∆ x t , t = 1 , ..., T ) is sampled online at every iteration for training.
We use 16 blocks of units with block size 6 and choose Gaussian kernel ( σ = 0 . 08 ) as the adjacency measure.
Figure 9 shows four learned 3D units, where the grid-like patterns emerge.
We also show that the learned system can perform path integral, simple path planning and planning with a cuboid obstacle tasks well.
In both 3D and 1D, the optimal block size is not 6, which may only be optimal for 2D. We shall explore different block sizes in 3D and 1D in future work.
F. 3D PATH INTEGRAL
Please see ﬁgure 10.
F. 3D PATH PLANNING
Please see ﬁgure 11.
F. 3D PATH PLANNING WITH OBSTACLES
Please see ﬁgure 12.