P REDICTION AND GENERALISATION OVER DIRECTED ACTIONS BY GRID CELLS
Anonymous authors
Paper under double-blind review
A BSTRACT
Knowing how the effects of directed actions generalise to new situations (e.g. moving North, South, East and West, or turning left, right, etc.) is key to rapid generalisation of action to new situations.
Markovian tasks can be characterised by a state space and a transition matrix and recent work has proposed that neural grid codes provide an efﬁcient representation of the state space, as eigenvectors of a transition matrix reﬂecting diffusion across states, that allows efﬁcient prediction of future state distributions.
Here we address the ability to predict the effects of directed actions across new tasks, irrespective of differences in local transition structure, to allow fast generalisation and form a "sense of direction".
We extend the eigenbasis prediction model, utilising tools from Fourier analysis, to prediction over arbitrary translation-invariant directed transition structures (i.e. displacement and diffusion), showing that a single set of eigenvectors can support predictions over arbitrary directed actions via action-speciﬁc eigenvalues.
We show the equivalence between the generalised prediction framework and traditional models of grid cells ﬁring driven by self-motion to perform path integration using oscillatory interference models (via Fourier components as velocity-controlled oscillators) or continuous attractor networks (via analysis of the update dynamics).
We thus provide a unifying framework for the role of the grid system in predictive planning, sense of direction and path integration.
The resulting model allows a single gridlike representation to support efﬁcient prediction over directed transitions in spatial and non-spatial tasks, supporting generalisable inference over directed actions across different tasks.
I NTRODUCTION
A "cognitive map" encodes relations between objects and supports ﬂexible planning (Tolman [40]), with hippocampal place cells and entorhinal cortical grid cells thought to instantiate such a map (O’Keefe and Dostrovsky [33]; Hafting et al.
Each place cell ﬁres when the animal is near a speciﬁc location, whereas each grid cell ﬁres periodically when the animal enters a number of locations arranged in a triangular grid across the environment.
Together, this system could support representation and ﬂexible planning in state spaces where common representational structure is preserved across states and tasks, affording generalisation and inference, e.g., in spatial navigation where Euclidean transition rules are ubiquitous (Whittington et al.
Recent work suggests that place cell ﬁring provides a local representation of state occupancy, while grid cells comprise an eigenbasis of place cell ﬁring covariance (Dordek et al.
[16]; Stachenfeld et al.
[39]; Sorscher et al.
[38]; Kropff and Treves [27]).
Accordingly, grid cell ﬁring patterns could be learned as eigenvectors of a symmetric (diffusive) transition matrix over state space, providing a basis set enabling prediction of occupancy distributions over future states.
The "intuitive planning" operates by replacing multiplication of state representations by the transition matrix with multiplication of each basis vector by the corresponding eigenvalue (Baram et al.
[2]; Corneil and Gerstner [14]).
Thus a distribution over state space represented as a weighted sum of eigenvectors can be updated by re-weighting each eigenvector by its eigenvalue to predict future state occupancy.
Fast prediction and inference given different actions across environments is vital and economical for the survival of animals, and intuitive planning, in its original form, supports such ability under a single transition structures, most often corresponding to symmetrical diffusion.
Here we assume a common translation-invariant transition structure underlying multiple speciﬁc tasks which break translation invariance in different ways (e.g., Euclidean spatial environments with periodic boundary conditions containing different patterns of obstacles) and generalise the prediction models with grid codes to incorporate directed transitions based on Fourier analysis.
We demonstrate the utility of the proposed theoretical framework by showing its ability for performing prediction and path integration in a number of simple yet illustrative examples.
We deﬁne a "sense of direction" quantity, i.e., which proportions of directed actions would best lead to the goal, based on the underlying transition structure ignoring local obstacles, and can be computed based on the "Fourier modes" cells under our formulation, which we propose as a computational role for the medial entorhinal grid cells.
We then show, on the theory-level, how different classes of computational models for grid cells could be uniﬁed, under the assumption that the grid cells ﬁring result from superpositions of inputs from neurons with Fourier-mode-like ﬁring patterns.
Hence demonstrating that the proposed model acts as a unifying theoretical framework for understanding grid cell ﬁring.
"I NTUITIVE PLANNING " WITH SYMMETRICAL TRANSITIONS
Intuitive planning represents the occupancy distribution over the state space as a weighted sum of the (real) eigenvectors of the (symmetric) transition matrix, so that the effect of one step of the transition dynamics on the distribution can be predicted by reweighting each of the eigenvectors by the corresponding eigenvalue.
And this generalises to calculating the cumulative effect of discounted future transitions (Baram et al.
Speciﬁcally, consider a transition matrix, T 2 R N ⇥ N , T ss 0 =
P ( s t +1 = s
| s t = s ) where s t encodes the state at time t and N is the number of states.
Then, T n is the n -step transition matrix, and has the same set of eigenvectors as T .
Speciﬁcally, the eigendecomposition of T and T n are:
[Equation]
where each column of the matrix Q is an eigenvector of T and
⇤ =
diag (  P ( T )) , where  P ( T ) is the set of eigenvalues of T .
Similarly, any polynomial in T , p ( T ) , shares the same set of eigenvectors as T and the set of eigenvalues  P ( p ( T ))
=
p (  P ( T )) .
Hence:
[Equation]
where w =
[Equation] [Equation]
[Equation]
The resolvent form (eq. 2) is an inﬁnite discounted summation of transitions, which under a policy and transition structure corresponding to diffusion, is equivalent to the successor representation (SR, Fig.
1E) with discounting factor  (Dayan [15]; Stachenfeld et al.
See Mahadevan and
Maggioni [30] for a related spectral approach using Fourier decomposition of T for estimating the value function.
The SR has been shown to be useful for navigation via gradient ascent of the future probability of occupying the target state, and has a linear relationship with the true underlying
Euclidean distances in spatial tasks (hence "intuitive planning", see Fig.
1 and Fig.
2D-E).
A : Example transition matrix; B : P ( s t +
= s
| s t = 5) ; C, D : same are shown for T 3 , showing predicted distribution over the next three time steps; E the resolvent form/SR (eq. 2) computed from the eigenbasis of the transition matrix; F : SR values for state 5 , which can used for navigation.
The eigenvectors of the diffusion transition matrix generally show grid-like patterns, suggesting a close relationship to grid cells.
However, intuitive planning cannot predict the effects of directed actions or drift (i.e., variable asymmetric transition structure), which is useful in itself and also corresponds to the presumed role of grid cells in path integration.
Moreover, predictions over different directed actions would require different eigendecompositions, hence incurring high computational costs that undermines its biological plausibility.
In Section 3 we unify the prediction and path integration approaches by exploiting translation invariant symmetries to generalise across actions, using a single common eigenbasis and cheaply calculated updates via action-dependent eigenvalues.
F LEXIBLE PLANNING WITH DIRECTED TRANSITIONS
Updating state representations to predict the consequences of arbitrary directed actions is an important ability of mobile animals, known as path integration and thought to depend on grid cells (McNaughton et al.
To generalise to directed transition structures, we consider the transition dynamics corresponding to translation (drift) and Gaussian diffusion with arbitrary variance (including 0 , equivalent to plain translation).
Our assumption that the transition structure is translation invariant (implying periodic boundary conditions), leads to circulant transition matrices.
Consider a 2D rectangular environment with length L and width W where each state is a node of the unit square grid, then the transition matrix can be represented by T 2 R LW ⇥ LW , with each row the vectorisation ( vec ( · ) ) of the matrix of transition probabilities starting from the speciﬁed location, i.e.,
T [ j, :] =
vec [ P ( s t +1 | s t = j )] , where T is constructed by considering the 2D state space as a 1D vector and concatenating the rows ( j = xL + y for ( x, y ) 2 [0 , W  1] ⇥ [0 , L  1] ), see Fig.
2A.
The transition matrix is circulant due to the translation invariance of the transition structure (see
Appendix Prop. A.1), and takes the following form:
[Equation]
The normalised eigenvectors of the circulant matrix T 2 R N ⇥ N ( N = LW ) are the vectors of powers of N th roots of unity (the Fourier modes):
[Equation]
where !
k = exp( 2 ⇡ iN k ) , for k = 0 , . . .
, N  1 , and i = p 1 .
Hence the matrix of eigenvectors (as the columns), F = ( q 0 , q 1 , . .
. , q N  1 ) , is just the (inverse) discrete Fourier transform matrix (Bracewell [4]), where F kj = !
kj for 0  k, j  N  1 .
The Fourier modes projected back onto the
L ⇥ W 2D spatial domain are plane waves, as shown in Fig.
2G, with wavevector determined by the value of k that speciﬁes the direction and spatial frequency of each plane wave (see Appendix B).
We can immediately compute the corresponding eigenvalues for the eigenvectors in eq.
4 (equivalent to taking the discrete Fourier transform (DFT) of the ﬁrst row (or column) of T , see Bracewell [4]):
[Equation]
where { T 0 , . .
, T N  1 } are the N unique elements that fully speciﬁes the circulant matrix T (eq. 3).
We can then utilise tools from Fourier analysis for efﬁcient updating of the eigenvalues whilst leaving the universal eigenbasis unaffected.
For a transition matrix T v corresponding to an arbitrary action (translation velocity) v = ( v x , v y ) , each row of T v is again a circulant, but shifted version of the corresponding row vector of the symmetric transition matrix corresponding to zero drift velocity, T 0 .
Speciﬁcally, the ﬁrst rows of the two matrices are related as follows:
[Equation]
Given the eigenvalues for T 0 , ⇤ 0 =
[Equation]
⇤ [Equation] N (via the DFT of the ﬁrst row of T 0 , eq. 5), we can immediately derive the eigenvalues of T v , ⇤ v , via a one-step update based on the
Fourier shift theorem (Bracewell [4]) without recomputing the eigendecomposition: [Equation] [Equation]
[Equation]
i , where  ( v ) = v x L + v y (8)
[Figure]
I n t u i t i v e d
i
[Figure]
t
[Figure]
Hence allows path integration by reweighting the common set of eigenvectors at each timestep by the updated eigenvalues corresponding to the current drift velocity (eq. 8).
Note that additionally, T can include diffusion, thus reweighting by the eigenvalues of the diffusive transition matrix also tracking of increasing uncertainty
Utilising the ﬁxed eigenbasis (eq. 4) and the respective eigenvalues (eq.
8) for arbitrary transition structures, we can make efﬁcient prediction for the distribution of future state occupancy with respect to arbitrary action (see Figs.
2B-C).
Adding translation to the translation-invariant transition matrix does not change the set of eigenvectors - allowing one set of eigenvectors (Fourier modes) to support prediction for actions in all directions (or plain diffusion), hence prediction of effects of directed actions can be efﬁciently generalised across environments.
We deﬁne a "sense of direction", as the angle of the transitions (or the linear combinations of the available actions in a non-spatial setting) that maximise the future probability of reaching the target state given an initial state, which is modelled by the SR matrix.
[Equation]
where  is the discounting factor, D j , j = 1 , · · · , LW are the eigenvalues for the symmetric diffusion transition matrix, k j , j = 1 , . .
. , LW are the wavevectors for the j -th Fourier components, s 0 , s G are the coordinates of the start and goal states, and v ✓ = ( v cos( ✓ ) , v sin( ✓ )) represents the velocity (with speed v and head direction ✓ ).
We see that the "sense of direction" supports generalisation of predictions of effects of actions across all environments with the same translation-invariant transition structure, i.e., such predicted effects ignore any differences in local structure.
See Appendix B for the derivation of eq. 9.
Note that here we assume that the goal state s G is known a priori, e.g., we consider a problem where the animal is navigating towards a previously visited location.
We propose that a computational role for the neural grid codes: generating a "sense of direction" (capturing the transition structure of the state space, ignoring the obstacles and boundaries) that reﬂects a global sense of orientation which allows generalisation to completely new environments, via utilising a Fourier basis for a larger toroidal pseudo space.
The proposed model can be applied to ﬂexible planning under arbitrary drift velocity as demonstrated in Fig.
3 (A-E).
An agent is trying to navigate towards a goal state in a windy grid world.
The navigation is performed by following the ascending "gradient" of the SR for occupancy of the target state (the resolvent metric, eq. 2).
The SR computed from the transition matrix including the effects of diffusion and wind (Fig.
3 B) based on our analysis (eq.
8) leads straight to the target (Fig. 3 C).
Given the analytical expression of the SR (eq. 2), we could efﬁciently adjust the SR matrix to accommodate local changes in the state space, e.g., insertion of a barrier, using the Woodbury inversion formula to update the parts of the SR matrix affected by the local obstacles (see Appendix A.3 for derivations); and again in this case, the agent correctly adjusts for the wind as well as taking the shortest path around the inserted wall (Fig.
3 D-E).
We note, however, that the proposed model is also able to solve tasks without periodic boundary conditions, by considering the original task state space, S 0 , being embedded into a larger, periodically bounded pseudo state space S p , at least twice as large in each dimension as S 0 (Fig. 3 F).
We again follow the previous procedures, utilising the Fourier modes, this time computed on S p , to perform predictions in S 0 (Fig.
3 F-G), and the performance is unaffected.
Note that under such formulation, the underlying transition structures can be applied to environments with both periodic and non-periodic boundary conditions - allowing sense of direction planning in either case.
We can also use our model for path integration (Fig.
3 H, see Section 4) in S 0 , by taking velocity inputs (from any path in the grid world) to update the state occupancy distribution (eq. 8).
We observe that the path integration performance is strongly sensitive with respect to self-motion noise, with perfect path integration when the uncertainty is low up to 1000 time steps (the discretisation of state space means that uncertainty below 0.075 has no effect ), and monotonically increasing path integration error when the uncertainty is higher, where the uncertainty represents the degree of diffusion caused by self-motion noise, in addition to translation.
The derived analytical expression for the sense of direction can be retrieved via a lookup table when the state space is small and discrete, whereas in large or continuous state spaces, it can be found either via optimisation algorithms (the problem is non-convex by its periodic nature, but there exists a unique global optimum), or by a neural network modelling the non-linear mapping between the quantity to be optimised over in eq.
9 and the sense of direction, see Bush et al.
[12] for neural network approaches to ﬁnding goal directions from grid representations.
To demonstrate the utility of state representation by eigenvectors weighted by action-dependent eigenvalues, we adapted a deep-Q network (DQN, Mnih et al.
[32]) to allow it to learn such representations (Fig. 4A), applied to the CartPole task (Barto et al.
The basic DQN takes 4 continuous state variables as input (cart position and velocity, pole angle and angular velocity) and learns to output Q values for actions 0 or 1 (cart movement left or right) via a temporal difference learning rule .
The gc-DQN has 2 additional networks: the ﬁrst takes as inputs the n low-frequency Fourier modes over the discretised state space (with N = 8 4 bins) and has n actions = 2 outputs to allow representation for each action given the input Fourier modes; the second takes as inputs the state variables and action and has 2 n outputs which serve as an additional action-dependent multiplier to the connection weights for the hidden layer of the ﬁrst network.
The outputs of the ﬁrst network and the outputs of the standard DQN are then fully connected to an output layer to learn the updated Q values.
We also tested versions of these networks with deep Dyna agents (Peng et al.
See Appendix.
E for details.
Preliminary results in ﬁg. 4(B) show that the gc-DQN and deep gc-Dyna-Q greatly accelerates learning comparing to the baseline agents, with relatively minor increase in the model complexity and computational costs.
The results support our hypothesis that Fourier eigenvectors and the possibility of weighting them by action-speciﬁc values aids prediction (in this case, of future value).
The focus of this paper is proposing a theoretical framework for state representation, prediction, planning and path integration via grid-like eigenvectors and action-dependent eigenvalues.
The neural network implementation above is only a preliminary attempt at combining this approach with deep learning (see also Mahadevan and Maggioni [30]).
A UNIFYING FRAMEWORK FOR MODELS OF GRID CELL FIRING
Our focus so far has been on proposing a ﬂexible and efﬁcient extension of the prediction models of grid cells (Baram et al.
[2]; Dordek et al.
[16]; Stachenfeld et al.
[39]) to arbitrary directed transitions.
[Figure]
e n
[Figure]
i
[Figure]
t
[Figure] [Figure]
I npu t : ( s t a t e , a c t i o n )
S A _ h
i
[Figure]
e n _
[Figure]
t
[Figure]
t : S t a t ea c t i o n m u t li p li e r s
I npu t : n
F o u r
i e r m o d e s
F o u r
i e r _
h
i dd e n _
[Figure]
u t
[Figure]
t : a c t i o n w e
i g h t s
[Figure]
t
[Figure]
c t i
[Figure]
Plots of average reward in each episode of the
CartPole task against the episode number.
See Appendix.
E for details of the simulation.
There exists many other computational models of grid cells that do not rely on inputs from the place cells, such as the continuous attractor network (CAN) models which models the emergence of grid-like patterns in a recurrently connected networks with pre-speciﬁed weights via performing path integration (Fuhs and Touretzky [19]; Burak and Fiete [6]; Corneil and Gerstner [14]), and the oscillatory interference (OI) models which models the formation of grid-like patterns via coincidence detection of velocity-dependent inputs during path integration (Welday et al.
[42]; Burgess et al.
[9]; Burgess [8]).
Here we build upon the previous work of unifying the normative and path integration models of grid cells ﬁring by Sorscher et al.
[38], extending their discussions and showed the equivalence of our generalised prediction framework with models of path integration by grid cells based on continuous attractor dynamics or oscillatory interference.
4. R ELATION TO CONTINUOUS ATTRACTOR NETWORK MODELS OF GRID CELLS
Several uniﬁed frameworks for grid cell models have been proposed, one of the most prominent by Sorscher et al.
[38], where they proved the equivalence between the maximisation of a spatial representation objective function under the prediction models and the pattern formation dynamics of the CANs under the models of path integration.
Their analysis, however, does not explicitly address the equivalence between the two classes of models when there is non-zero velocity inputs, corresponding to non-trivial asymmetric velocity-dependent connectivities in the CANs (Fuhs and
Touretzky [19]; Burak and Fiete [6]), which is the key element for performing path integration.
We explicitly address such equivalence utilising our framework based one Fourier analysis.
Assuming that a grid cell is formed via linear combination of selected Fourier modes (e.g., 6 Fourier modes at ⇡ 3 degrees increments with same frequencies):
[Equation]
where f j ’s are the selected Fourier modes with weights w j .
Inserting the single-cell representation into eqs.
(14, 15) from [38] we have the equations governing the dynamics of the grid cells under the normative and mechanistic path integration models respectively as following.
[Equation]
where  j is the corresponding eigenvalue of f j with respect to the (symmetric) transition matrix, T 0 ,  ( · ) is the Heaviside function, W is the recurrent connectivity matrix, b ( v ) is the velocity-dependent feedforward input to the grid cell under continuous attractor model which involves a constant baseline term and a velocity dependent term (Burak and Fiete [6]), ↵ is the multiplicative constant,  and µ are the multiplicative constant corresponding to the additive penalty terms in the (single-neuron)
Lagrangian under the normative models:
[Equation]
Now suppose the agent is moving with velocity v , which causes changes in the transition structure, given our previous formulation based on Fourier analysis (eq.
8) and using the ﬁrst-order Taylor expansion around
for approximating the exponential, eq. 11 and eq.
12 becomes equivalent, by identifying the corresponding terms ⌧ , W and b ( v ) , see Appendix B for detailed proof of equivalence.
The resulting expression for b ( v ) is directly related to the Fourier modes inputs, hence it correlates with the underlying grid codes with the baseline µ , hence is equivalent to the asymmetric velocitydependent parts of the recurrent connectivity matrix in the CAN.
We stress that the proposed model is on the theory-level, instead of the implementation level within which CAN models lies, hence we do not assume any speciﬁc neural network structure in the current analysis.
What we showed in this section is the equivalence between the differential equation governing the dynamics of updates in the CAN models and the gradients of the Lagrangian of the predictions model under non-zero velocities, based on the aforementioned assumption that the grid cell ﬁring is a linear combination of the Fourier modes (Sec. 3), hence showing that under non-stationary transition dynamics, the CAN models and the prediction models should also yield identical updates to the grid cells (up to scaling factors).
4. R ELATION TO OSCILLATORY INTERFERENCE MODELS OF GRID CELLS
For the oscillatory interference models of grid cells, path integration is achieved by tracking the phases of the velocity controlled oscillators (VCOs), which encode movement speed and direction by variations in burst ﬁring frequency (Burgess et al.
[9]; Burgess [8]).
The VCOs generate grid-like ﬁring patterns via coincidence detection by the grid cells (Hasselmo [23]; Welday et al.
The variation of frequency with velocity produces a phase code for displacement, enabling the modelled grid cells to perform path integration.
Namely, VCOs change their frequencies relative to the baseline according to the projection of current velocity, v ( t ) , onto the VCO’s "preferred direction", d :
[Equation]
[Equation]
where  is a positive constant, and f b ( t ) is the baseline frequency (the 4  11 Hz EEG theta rhythm).
It follows that the VCOs perform linear path integration since the relative phase between VCO’s frequency and baseline at time t ,  ab ( t ) =
a ( t )   b ( t ) , is proportional to the agent’s displacement in the preferred direction:
[Equation]
[Equation]
where x ( t ) is the agent’s location at time t .
The interference of VCOs whose preferred directions differ by multiples of ⇡ / 3 generate triangular grid-like patterns, provides an explanation of theta phase precession in grid cells in which ﬁring phase relative to the theta rhythm encodes distance travelled through the ﬁeld (Hafting et al.
[22]; Burgess [8]) and complements the attractor dynamics given by symmetrical connections between grid cells (Bush and Burgess [10]).
We note that the major challenge to the OI models posed in the neurophysiological community, the main experimental results against the OI models (that bats and humans do not have a reliable theta frequency oscillations) has recently been resolved: the required phase coding (theta phase precession) can occur relative to a variable baseline frequency (Bush and Burgess [11]) and has now been found in both bats and humans (Eliav et al.
[18]; Qasim et al.
We simulated the ﬁring of a grid cell synthesised from 6 Fourier modes inputs (Fig.
5 A-B), each ﬁring a spike at its complex phase in the current state, as a leaky integrate and ﬁre neuron performing coincidence detection using a real trajectory of a rat exploring a 50 cm ⇥ 50 cm box.
At each time step (corresponding to one theta cycle), the phase of each Fourier mode is updated according to eq.
8 given the current velocity, and ﬁres a spike at this phase if it is within the interval [  ⇡ / 4 , ⇡ / 4] (modelling modulation by the baseline theta oscillation).
The grid cell ﬁres a spike at the current location if the integrated inputs reach a threshold.
Note that we could also simulate a set of grid cells, with different offsets (depending on the initial phases of the Fourier modes) and different scales and orientations (depending on the choice of Fourier modes), such that the set of grid cells, like the
Fourier modes, comprises a basis for the state space and do so on the basis of path integration (for which environmental inputs are also required to prevent error accumulation [8; 7]).
Grid cells show "precession" in their ﬁring phase relative to theta as the animal moves through the ﬁring ﬁeld (signaling distance travelled within the ﬁeld)
(Hafting et al.
[22]; Jeewajee et al.
[24]; Climer et al.
[13]), which is captured by our model, similar to the OI model.
The Fourier modes whose wavevectors that are aligned with the current direction of translation advance in phase as the movement progresses.
Phase precession results from assuming that the Fourier modes aligned with movement direction are dominant inﬂuence on grid cell ﬁring (c.f. those aligned to the reverse direction).
The baseline "theta frequency" corresponds to the mean rate of change of phase of all
Fourier components and so could vary (for noise reduction, see Burgess and Burgess [7]; Burgess [8], without precluding phase coding (Eliav et al.
[18]; Bush and Burgess [11])).
By simulating random straight runs, we can see clear late-to-early phase precession (Fig.
5 C), analogous to that observed in grid cells (Hafting et al.
[22]) and the OI models (Burgess [8]).
We thus conclude that the OI model and our model perform path integration or prediction in the same way: the OI model changes the phase of each VCO relative to baseline corresponding to the component of translation along the VCO’s preferred direction, and this is exactly analogous to the complex phases of the Fourier modes being updated to reﬂect transitions along its wavevectors (multiplication by corresponding eigenvalues, eq.
8), and the phase changes with transitions perform path integration in the same way as the OI models.
D ISCUSSION
Understanding how different actions affect the agent’s state across environments is essential for generalisation.
Existing models are capable of such prediction for tasks with an invariant symmetric transition matrix.
We propose a computational role for the grid cell ﬁring: generating generalisable "sense of direction" (eq. 9) that captures the underlying transition structure across different environments (ignoring the speciﬁc obstacles in each environment), so that they can generalise prediction and path integration under directed transitions in new environments (see also Whittington et al.
Here we generalised the prediction model to provide a mathematical framework for understanding this function, and showed how it corresponds to other computational models of grid cells that are based on path integration, and provide a functional explanation for theta phase precession.
Having obtained the approximate transition structure of the environment, the proposed prediction model can perform the sense-of-direction planning in new environments (i.e., ﬁnding combinations of actions that most likely leads to the target state by ignoring local obstacles).
We make the prediction that there exists other (e.g., fronto-parietal) areas of the brain that are responsible for detecting and avoiding obstacles following the overall direction provided by the grid cells [17; 29].
The current model applies to translation-invariant transition structures, and use of the Fourier shift theorem to calculate eigenvalues also assumes a Euclidean state space.
We demonstrated ways of generalising planning to bounded or locally non-translation invariant transition structures in Section 3.
We note that machine learning methods based on a similar premise (creating a single representation to support planning via multiple different actions) might work even when the transitions are not strictly translation invariant (e.g., family trees, see Whittington et al.
A number of question and predictions can be raised from the proposed model.
If there exists a basis of neurons with Fourier-mode-like ﬁring patterns acting as inputs to cells in the entorhinal cortex, then the set of grid ﬁring patterns only comprises a small proportion of the entire set of ﬁring patterns that can be synthesised, how the composition corresponds to the heterogeneity of ﬁring patterns observed in the entorhinal cortex is yet to be understood, but experimental results have indicated the existence of periodic non-grid cells in the entorhinal cortex that also resemble combinations of small number of Fourier modes (Krupic et al.
We predict the use of the same set of grid cells (superpositions of Fourier modes inputs) for indicating direction to goal across different environments with the same global orientation – i.e. different boxes/mazes with the same distal orienting cue, hence supporting generalisability and faster learning.
We also predict the existence of topology-dependent grid cells (that exhibits distortions in the ﬁring patterns with respect to changes in the environmental topology) that complements the Euclidean-based grid cells proposed here for spatial navigation in dealing with local changes in the environmental topology.
Finally, the proposed model predicts future state occupancy from the transition matrix, future works could look at the reversed direction: vector navigation, i.e., inferring the translation between two locations given the phase codes for each (see
Bush et al.
[12]; corresponding to discriminative and generative models).
We showed that by giving standard DQN agents the ability to represent the current state as eigenvectors of the state space weighted by action-speciﬁc values, the resulting gc-DQN agent accelerates learning in the CartPole experiment (ﬁg. 4).
Future work will involve more rigorous study of the neural network implementation of the proposed grid cell model, which we hope will bring insights into the combination of our approach with deep learning.
Our approach offers potential generalisation to nonspatial tasks such as transitive inference (Von Fersen et al.
[41], see Appendix D) and, in combination with DQN agents, improves performance in the CartPole problem which is both non-spatial and contains actions that are not strictly translation-invariant.
A future direction in generalising the current model to non-spatial tasks will be to consider Fourier analysis on groups of operators utilising group-theoretic knowledge (Kondor and Trivedi [26]; Gao et al.
A CKNOWLEDGEMENTS
The authors thanks support from various funding resources.
The authors declare no competing ﬁnancial interests.