Deep neural networks have achieved impressive successes in fields ranging from object recognition to complex games such as Go 1,2 . Navigation, however, remains a substantial challenge for artificial agents, with deep neural networks trained by reinforcement learning 3-5 failing to rival the proficiency of mammalian spatial behaviour, which is underpinned by grid cells in the entorhinal cortex 6 . Grid cells are thought to provide a multi-scale periodic representation that functions as a metric for coding space 7,8 and is critical for integrating self-motion (path integration) 6,7,9 and planning direct trajectories to goals (vector-based navigation) 7,10,11 . Here we set out to leverage the computational functions of grid cells to develop a deep reinforcement learning agent with mammal-like navigational abilities. We first trained a recurrent network to perform path integration, leading to the emergence of representations resembling grid cells, as well as other entorhinal cell types 12 . We then showed that this representation provided an effective basis for an agent to locate goals in challenging, unfamiliar, and changeable environments-optimizing the primary objective of navigation through deep reinforcement learning. The performance of agents endowed with grid-like representations surpassed that of an expert human and comparison agents, with the metric quantities necessary for vector-based navigation derived from grid-like units within the network. Furthermore, grid-like representations enabled agents to conduct shortcut behaviours reminiscent of those performed by mammals. Our findings show that emergent grid-like representations furnish agents with a Euclidean spatial metric and associated vector operations, providing a foundation for proficient navigation. As such, our results support neuroscientific theories that see grid cells as critical for vector-based navigation 7,10,11 , demonstrating that the latter can be combined with path-based strategies to support navigation in challenging environments. The ability to self-localize in the environment and to update one's position on the basis of self-motion are core components of navigation 13 . We trained a deep neural network to path integrate within a square arena (2.2 m × 2.2 m), using simulated trajectories modelled on those of foraging rodents (see Methods). The network was required to update its estimate of location and head direction using translational and angular velocity signals, mirroring those available to the mammalian brain 12, 14, 15 (see Methods; Fig. 1a, b). Velocity was provided as input to a recurrent network with a long short-term memory (LSTM) architecture, which was trained using backpropagation through time (see Methods and Supplementary Discussion), allowing the network to dynamically combine current input signals with activity patterns reflecting past events (see Methods; Fig. 1a ). The LSTM projected to place and head direction units via a linear layer-units with activity defined as a simple linear function of their input (see Extended Data Fig. 1 for architecture) . Importantly, the linear layer was subject to regularization, in particular dropout 16 , such that 50% of the units were randomly silenced at each time step. The vector of activities in the place and head direction units, corresponding to the current position, was provided as a supervised training signal at each time step (see Methods; Extended Data Fig. 1 ). This form of supervision follows evidence that in mammals, place and head direction representations exist in close anatomical proximity to entorhinal grid cells 12 and emerge in rodent pups before the appearance of mature grid cells 17, 18 . Equally, in adult rodents, entorhinal grid cells are known to project to the hippocampus and appear to contribute to the neural activity of place cells 19 . As expected, the network was able to path integrate accurately in this setting involving foraging behaviour (mean error after 15 s trajectory was 16 cm versus 91 cm for an untrained network, effect size 2.83, 95% confidence interval (CI) 2.80-2.86; Fig. 1b, c) . Strikingly, individual units within the linear layer of the network developed stable spatial activity profiles similar to those of neurons within the entorhinal network 6, 12 (Fig. 1d , Extended Data Fig. 2 ). Specifically, 129 of the 512 linear layer units (25.2%) resembled grid cells, exhibiting significant hexagonal periodicity (gridness 20 ) versus a null distribution generated by a conservative field shuffling procedure (see Methods). The scale of the grid-patterns, measured from the spatial autocorrelograms of the activity maps 20 , varied between units (range 28 cm to 115 cm, mean 66 cm) and followed a multi-modal distribution, consistent with empirical results from rodent grid cells 21, 22 ( Fig. 1e) . To assess these clusters, we fit mixtures of Gaussians, finding the most parsimonious number by minimizing the Bayesian information criterion (BIC). The distribution was best fit by three Gaussians (means 47 cm, 70 cm, and 106 cm), indicating the presence of scale clusters with a ratio between neighbouring clusters of approximately 1.5, closely matching theoretical predictions 23 and also lying within the range reported for rodents 21, 22 (Fig. 1e , Extended Data Fig. 3 ). The linear layer also exhibited units resembling head direction cells (10.2%), border cells (8.7%), and a small number of place cells 12 as well as conjunctions of these representations 20, 24 (Fig. 1d , f, g, Extended Data Fig. 2 ). To ascertain how robust these representations were, we retrained the network 100 times, in each instance finding similar proportions of gridlike units (mean 23%, s.d. 2.8%, units with significant gridness scores) and other spatially modulated units (Extended Data Fig. 3 ). Conversely, grid-like representations did not emerge in networks without regularization (for example, dropout, see Methods; also see ref. 25, Extended Data Fig. 4) . Therefore, the use of regularization, including dropout (which has been viewed to be a parallel of noise in neural systems 16 ), was critical to the emergence of entorhinal-like representations. Notably, therefore, our results show that grid-like representations reminiscent of those found in the mammalian entorhinal cortex emerge in a generic network trained to path integrate, contrasting with previous approaches using pre-configured grid cells 26 (see Supplementary Discussion). © 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved. Furthermore, our results are consistent with the view that grid cells represent an efficient and robust basis for a location code updated by self-motion cues [6] [7] [8] [9] . Next, we sought to test the hypothesis that the emergent representations provide an effective basis function for goal-directed navigation in novel challenging and changeable environments, when trained through deep reinforcement learning. Entorhinal grid cells have been proposed to provide a Euclidean spatial metric and thus to support the calculation of goal-directed vectors, enabling animals to follow direct routes to a remembered goal, a process known as vector-based navigation 7, 10, 11 . Theoretically, the advantage of decomposing spatial location into a multi-scale periodic code, as provided by grid cells, is that the relative positions of two points can be retrieved by examining the difference in the code at the level of each scale-combining the modulus remainders to return the true vector 7, 11 (Fig. 2a). However, despite the obvious utility of such a framework, experimental evidence for the direct involvement of grid representations in goal-directed navigation is still lacking. To develop an agent with the potential for vector-based navigation, we incorporated the 'grid network' described above into a larger architecture that was trained using deep reinforcement learning (Fig. 2d,  Extended Data Fig. 5 ). As before, the grid network was trained using supervised learning but, to better approximate the information available to navigating mammals, it now received velocity signals perturbed with random noise as well as visual input. Experimental evidence suggests that place cell input to grid cells corrects for drift and anchors grids to environmental cues 21 . To parallel this, visual input was processed by a 'vision module' consisting of a convolutional network that produced place and head direction cell activity patterns that were provided as input to the grid network 5% of the time-akin to a moving animal making occasional, imperfect observations of salient environmental cues 27 (see Methods; Fig. 2b, c and Extended Data Fig. 5). The output of the linear layer of the grid network, corresponding to the agent's current location, was provided as input to the 'policy LSTM' , a second recurrent network that both controls the agent's actions and outputs a value function. Additionally, whenever the agent reached the goal, the 'goal grid code'-activity in the linear layer-was subsequently provided to the policy LSTM during navigation as an additional input. We first examined the navigational capacities of the agent in a simple setting inspired by the classic Morris water maze ( Fig. 2b, c ; 2.5 m × 2.5 m square arena; see Methods and Supplementary Results). Notably, the agent was still able to self-localize accurately in this more challenging setting, where ground truth information about location was not provided and velocity inputs were noisy (mean error after 15-s trajectory, 12 cm versus 88 cm for an untrained network; effect size, 2.82; 95% CI, 2.79-2.84; Fig. 2e) . Furthermore, the agent exhibited proficient goal-finding abilities, typically taking direct routes to the goal from arbitrary starting locations (Fig. 2h) . Performance exceeded that of a control place cell agent (Fig. 2f,  Methods), chosen because place cells provide a robust representation of self-location but are not thought to provide a substrate for longrange vector calculations 11 . We examined the units in the linear layer, again finding a heterogeneous population resembling those found in entorhinal cortex, including grid-like units (21.4%) as well as other spatial representations (Fig. 2g, Extended Data Fig. 6 )-paralleling the dependence of mammalian grid cells on self-motion information 15, 28 and spatial cues 6, 21 . We next turn to our central claim, that grid cells endow agents with the ability to perform vector-based navigation, enabling downstream regions to calculate goal-directed vectors by comparing current activity with that of a remembered goal 7, 10, 11 . In the agent, we expect these calculations to be performed by the policy LSTM, which receives the current activity pattern over the linear layer (termed 'current grid code; Fig. 2d and Extended Data Fig. 5 ) as well as that present the last time the agent reached the goal (termed 'goal grid code') and uses them to control movement. Hence we performed several manipulations, which yielded four lines of evidence in support of the vector-based navigation hypothesis (see Supplementary Results) . First, to demonstrate that the goal grid code provided sufficient information to enable the agent to navigate to an arbitrary location, we substituted it with a 'fake' goal grid code sampled randomly from a location in the environment (see Methods). The agent followed a direct path to the newly specified location, circling the absent goal (Fig. 2i) similar to rodents in probe trials of the Morris water maze (escape platform removed). Second, we demonstrated that withholding the goal grid code from the policy LSTM of the grid cell agent had a strikingly deleterious effect on performance (Extended Data Fig. 6c ). Third, we demonstrated that the policy LSTM of the grid cell agent contained representations of key components of vector-based navigation (Fig. 2j, k) , and that both Euclidean distance (difference in r = 0.17; 95% CI, 0.11-0.24) and allocentric goal direction (difference in r = 0.22; 95% CI, 0.18-0.26) were represented more strongly than in the place cell agent. Notably, a neural representation of goal distance has recently been reported in the mammalian hippocampus 29 . Finally, we provide evidence consistent with a prediction of the vector-based navigation hypothesis, namely that a targeted lesion (that is, silencing) to the most grid-like units within the goal grid code should have a greater adverse effect on performance and the representation of vector-based metrics (for example, Euclidean distance) than a sham lesion (that is, silencing of non-grid units; see Supplementary Results) . Having demonstrated the effectiveness of grid-like representations in optimizing one-shot goal learning in a simple square arena, we assessed the agent's performance in two challenging, procedurally generated multi-room environments, referred to as 'goal-driven ' and 'goal-doors' (see Methods). Notably, these environments are challenging for deep reinforcement learning agents with external memory (Extended Data  Fig. 7e , f, h, i and Supplementary Results). Again, the grid cell agent exhibited high levels of performance, was strikingly robust across a range of network hyperparameters (Extended Data Fig. 7a-c) , and reached the goal more frequently than either control agents or a human expert-a typical benchmark for the performance of deep Fig. 5) . PCP, place cell predictions; HDP, head direction cell predictions. e, Accuracy of self-location decoded from place cell units. f, Performance of grid cell agent and place cell agent (y axis shows reward obtained within a single episode, 10 points per goal arrival, grey band displays the 68% confidence interval based on 5,000 bootstrapped samples). g, As in Fig. 1 , the linear layer develops spatial representations similar to entorhinal cortex. Left to right, two grid cells, one border cell, and one head direction cell. h, On the first trial of an episode, the agent explores to find the goal and subsequently navigates directly to it. 'S' denotes the starting location. i, After successful navigation, the policy LSTM was supplied with a 'fake' goal grid-code, directing the agent to this location where no goal was present. j, k, Decoding of goal-directed metric codes (that is, Euclidean distance and direction) from the policy LSTM of grid cell and place cell agents. The bootstrapped distribution (1,000 samples) of correlation coefficients are each displayed with a violin plot overlaid on a Tukey boxplot. Letter reSeArCH reinforcement learning agents in game playing scenarios 2 ( Fig. 3e, f ; see Supplementary Results). Furthermore, when agents were tested without retraining in environments considerably larger than those seen previously, only the grid cell agent was able to generalize effectively ( Fig. 3g, h ; see Supplementary Results). Despite the complexity of the 'goal-driven' environment, we could still decode the key metric codes from the grid agent policy LSTM with high accuracy during the initial period of navigation, and decoding accuracy was substantially higher in the grid cell agent than in both the place cell and deep reinforcement learning control agents (Fig. 3j, k and Supplementary Results; see Extended Data Figs. 8, 9 for control agent architectures). Finally, a core feature of mammalian spatial behaviour is the ability to exploit novel shortcuts and traverse unvisited portions of space, a capacity thought to depend on vector-based navigation 9, 11 . Strikingly, the grid cell agent-but not comparison agents-robustly demonstrated these abilities in specifically designed neuroscience-inspired mazes, taking direct routes to the goal as soon as they became available (Fig. 4 , Extended Data Fig. 10 and Supplementary Results). Conventional simultaneous localization and mapping (SLAM) techniques typically require an accurate and complete map to be built, with the nature and position of the goal externally defined 30 . By contrast, the deep reinforcement learning approach described in this work has the ability to learn complex control policies end-to-end from a sparse reward, taking direct routes involving shortcuts to goals in an automatic fashion-abilities that exceed previous deep reinforcement learning approaches [3] [4] [5] , and that would have to be hand-coded in any SLAM system. d e * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *  Our work, in demonstrating that grid-like representations provide an effective basis for flexible navigation in challenging novel environments, supports theoretical models of grid cells in vector-based navigation that were previously lacking strong empirical support 7, 10, 11 . We also show that vector-based navigation can be effectively combined with a path-based barrier avoidance strategy to enable the exploitation of optimal routes in challenging multi-compartment environments. In sum, we argue that grid-like representations furnish agents with a Euclidean geometric framework-paralleling their proposed computational role in mammals as an early-developing Kantian-like spatial scaffold that serves to organize perceptual experience 17, 18 .
Both Cueva & Wei (2018) and REF showed that grid-like spatial response patterns emerge in trained networks for navigation tasks which demonstrate that grid cells are critical for vector-based navigation.
Vector-based navigation using grid-like representations in artificial agents
Nature
Nature
Computer Science Medicine