1 6 4 3 a r t I C l e S Learning to predict long-term reward is fundamental to the survival of many animals. Some species may go days, weeks, or even months before attaining primary reward, during which time aversive states must be endured. Evidence suggests that the brain has evolved multiple solutions to this reinforcement learning (RL) problem 1 . One solution is to learn a model or cognitive map of the environment 2 , which can then be used to generate long-term reward predictions through simulation of future states 1 . However, this solution is computationally intensive, especially in real-world environments where the space of future possibilities is virtually infinite. An alternative, model-free solution is to learn, from trial-and-error, a value function that maps states to long-term reward predictions 3 . However, dynamic environments can be problematic for this approach, because changes in the distribution of rewards necessitate complete relearning of the value function. Here we argue that the hippocampus supports a third solution: learning of a predictive map that represents each state in terms of its successor states 4 . This representation is sufficient for long-term reward prediction, is learnable using a simple, biologically plausible algorithm, and explains a wealth of data from studies of the hippocampus. Our primary focus is on understanding the computational function of hippocampal place cells, which respond selectively when an animal occupies a particular location in space 5 . A classic and still influential view of place cells is that they collectively furnish an explicit map of space 5 . This map can then be employed as the input to a modelbased 6,7 or model-free 8,9 RL system for computing the value of the animal's current state. In contrast, the predictive map theory views place cells as encoding predictions of future states, which can then be combined with reward predictions to compute values. This theory can account for why the firing of place cells is modulated by variables like obstacles, environment topology, and direction of travel. It also generalizes to hippocampal coding in nonspatial tasks. Beyond the hippocampus, we argue that entorhinal grid cells 10 , which fire periodically over space, encode a low-dimensional decomposition of the predictive map, useful for stabilizing the map and discovering subgoals. The successor representation An animal's optimal course of action will frequently depend on the location (or more generally, the 'state') that the animal is in. The hippocampus' purported role in representing location is therefore considered to be a very important one. The traditional view of state representation in the hippocampus is that the place cells index the current location by firing when the animal visits the encoded location and otherwise remain silent 5 . The main idea of the successor representation (SR) model, elaborated below, is that place cells do not encode place per se but rather a predictive representation of future states given the current state. Thus, two physically adjacent states that predict divergent future states will have dissimilar representations, and two states that predict similar future states will have similar representations. The SR emerges naturally from the definition of value (V) often used in RL. The value of a current state s is defined as the expected sum of the reward at each future state s t , multiplied by an exponentially decaying discount factor γ ∈ [0, 1] that downweights distal rewards: where s t is the state visited at time t (see Online Methods for formal mathematical detail). The value function can be decomposed into the inner product of the reward function with a predictive representation of the state (the SR) 4 , denoted by M: The SR encodes the expected discounted future occupancy of state s′ along a trajectory initiated in state s: A cognitive map has long been the dominant metaphor for hippocampal function, embracing the idea that place cells encode a geometric representation of space. However, evidence for predictive coding, reward sensitivity and policy dependence in place cells suggests that the representation is not purely spatial. We approach this puzzle from a reinforcement learning perspective: what kind of spatial representation is most useful for maximizing future reward? We show that the answer takes the form of a predictive representation. This representation captures many aspects of place cell responses that fall outside the traditional view of a cognitive map. Furthermore, we argue that entorhinal grid cells encode a low-dimensionality basis set for the predictive representation, useful for suppressing noise in predictions and extracting multiscale structure for hierarchical planning.
In REF , the authors propose to learn a model of the environment of an agent by encoding the prediction of its future states into cells, which correspond to a form of sensorimotor representations.
The hippocampus as a predictive map
Nature Neuroscience
Nature Neuroscience
Medicine Biology