Hippocampus Is What Happens while You’re Busy Making Other Plans
Authors: Kiyohito Iigaya, John P. O’Doherty
Journal: Neuron (2019)

In this issue of Neuron, Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar provide evidence to suggest that the human hippocampus, long known to support spatial memory, also plays a causal role in model-based planning. In this issue of Neuron, Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar provide evidence to suggest that the human hippocampus, long known to support spatial memory, also plays a causal role in model-based planning. Think of your commute in the morning. If it is to the same place that you worked for the past 20 years, your commute has become habitual and you are less likely to plan your exact path to your workplace before you leave. However, if on one day your route is unexpectedly blocked by roadwork, you may need to plan more explicitly about which alternative route to take. These two decision-making strategies are captured in machine-learning algorithms called model-free and model-based reinforcement learning (Daw et al., 2011Daw N.D. Gershman S.J. Seymour B. Dayan P. Dolan R.J. Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215Abstract Full Text Full Text PDF PubMed Scopus (964) Google Scholar). A model-free agent makes decisions without explicit prospection, solely relying on the overall success rates of past actions. In contrast, a model-based agent envisions how to get to a desired goal using an internal model of the world that is learned and stored in the agent’s memory (model-based planning). Previous studies have suggested that humans utilize both strategies (Daw et al., 2011Daw N.D. Gershman S.J. Seymour B. Dayan P. Dolan R.J. Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215Abstract Full Text Full Text PDF PubMed Scopus (964) Google Scholar). We currently have a very limited understanding of how the human brain performs model-based planning, though there are reasonable grounds to suspect that the hippocampus might be involved. In rodents, sequential activations of hippocampal place cells occur when animals pause and look about while navigating a maze as if they were planning, a behavior called vicarious trial and error (Redish, 2016Redish A.D. Vicarious trial and error.Nat. Rev. Neurosci. 2016; 17: 147-159Crossref PubMed Scopus (219) Google Scholar). A recent muscimol inactivation study showed the rodent hippocampus as a causal neural substrate for planning (Miller et al., 2017Miller K.J. Botvinick M.M. Brody C.D. Dorsal hippocampus contributes to model-based planning.Nat. Neurosci. 2017; 20: 1269-1276Crossref PubMed Scopus (92) Google Scholar). However, in humans, we lack causal evidence about the contribution of hippocampus to model-based planning. In this issue of Neuron, Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar addressed the lack of evidence in humans by administering two behavioral tasks to patients who had undergone unilateral anterior temporal lobectomy (ATL) for the treatment of intractable epilepsy. These patients are missing a part of their hippocampi on one side, as well as adjacent medial temporal lobe structures. In the first task, Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar tested whether the patients were impaired in model-based planning using a well-studied behavioral task (Daw et al., 2011Daw N.D. Gershman S.J. Seymour B. Dayan P. Dolan R.J. Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215Abstract Full Text Full Text PDF PubMed Scopus (964) Google Scholar). On each trial, participants chose from one of two spaceships, each of which had a unique probability to take the participants to one of the two possible planets. Upon arriving on a planet, participants were then asked to choose one alien from the two that were always present on each planet. The chosen alien would give participants a reward or no reward, whose probability changed slowly over time. Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar used computational modeling to dissect participants’ behavior, as model-based and model-free computational strategies are known to predict distinct behaviors in this task (Daw et al., 2011Daw N.D. Gershman S.J. Seymour B. Dayan P. Dolan R.J. Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215Abstract Full Text Full Text PDF PubMed Scopus (964) Google Scholar). A model-free agent makes decisions based on the overall reward rate of each spaceship. Such an agent is more likely to simply choose a previously rewarded spaceship, ignoring subsequent planets and aliens. In contrast, a model-based agent aims to reach the alien that is currently giving out the most rewards and would thus choose a spaceship that is most likely to take them to the desired planet on which that alien resides. Consequently, a model-based agent does not necessarily repeat the exact same choice on the next trial even if it was rewarded on the previous one (Daw et al., 2011Daw N.D. Gershman S.J. Seymour B. Dayan P. Dolan R.J. Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215Abstract Full Text Full Text PDF PubMed Scopus (964) Google Scholar). Consistent with previous studies (Daw et al., 2011Daw N.D. Gershman S.J. Seymour B. Dayan P. Dolan R.J. Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215Abstract Full Text Full Text PDF PubMed Scopus (964) Google Scholar), participants with intact hippocampi showed a mixture of model-based and model-free strategies in their choices (Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar). This was also the case in the ATL patients; however, those patients relied more heavily on a model-free strategy and less on a model-based one unlike the comparison group (Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar). Further analyses suggested that the effect was mainly driven by those patients who had right hippocampal lesions, as opposed to those with left hippocampal lesions. In fact, the size of the right hippocampal lesion across patients was negatively correlated with an individual’s tendency to use model-based planning overall. Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar also showed, using a virtual spatial navigation task, that boundary-based place memory is significantly impaired in the ATL patients, replicating the results of previous hippocampal lesion studies (e.g., Guderian et al., 2015Guderian S. Dzieciol A.M. Gadian D.G. Jentschke S. Doeller C.F. Burgess N. Mishkin M. Vargha-Khadem F. Hippocampal volume reduction in humans predicts impaired allocentric spatial memory in virtual-reality navigation.J. Neurosci. 2015; 35: 14123-14131Crossref Scopus (56) Google Scholar). They then compared task behavior between this spatial memory task and the spaceship choice task. Individuals with intact hippocampi showed a positive correlation between the spatial memory and the tendency to engage in model-based planning, while ATL patients did not (Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar). This suggests that the hippocampus might serve as a common substrate for both spatial memory and model-based planning (Figure 1). One caveat of the findings, however, is that it is challenging to localize the impairment to the hippocampus per se, because the ATL lesion encompasses multiple structures including parahippocampal cortex and amygdala. Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar did control for overall lesion size while still showing a correlation with hippocampal lesion extent, thereby partly mitigating this concern. Patients with epilepsy may also have abnormal tissue in the ATL that could complicate generalization of these conclusions to healthy brains. In spite of these limitations, human lesion studies, such as the present one, are highly valuable because they can illuminate causal roles for human deep brain structures in neural computations in the absence of the precise causal manipulations available to non-human animal researchers. One open question is whether the present findings can generalize to a context that is not spatial. The cover story of spaceships and planets might have encouraged participants to utilize a spatial planning strategy (e.g., by thinking about which planet to visit, though it is also debatable how the hippocampus plays a role in this). However, not all plans need to be spatial. In fact, a previous study showed that animals with hippocampal lesions can show intact, goal-directed (presumably also model-based) performance involving lever pressing, which likely does not depend strongly on spatial planning (Corbit and Balleine, 2000Corbit L.H. Balleine B.W. The role of the hippocampus in instrumental conditioning.J. Neurosci. 2000; 20: 4233-4239Crossref PubMed Google Scholar). Nevertheless, the human hippocampal complex has been suggested to represent even non-spatial information in the form of a cognitive map (Garvert et al., 2017Garvert M.M. Dolan R.J. Behrens T.E. A map of abstract relational knowledge in the human hippocampal-entorhinal cortex.eLife. 2017; 6: e17086Crossref PubMed Scopus (134) Google Scholar). Thus, the hippocampus could still play a role in model-based planning even in non-spatial situations. Another key open question is how the neurons in hippocampus perform model-based planning. First, how does the brain combine a map and a goal? In order to select the best path, one has to locate a goal that has the highest payoff. Studies of spatial navigation suggest that hippocampal-prefrontal cortex coupling may play a key role in the capacity for flexible navigation (Preston and Eichenbaum, 2013Preston A.R. Eichenbaum H. Interplay of hippocampus and prefrontal cortex in memory.Curr. Biol. 2013; 23: R764-R773Abstract Full Text Full Text PDF PubMed Scopus (765) Google Scholar). Second, how does the brain learn a map and a goal in the first place? Dopaminergic systems may couple with hippocampus to achieve this end (Shohamy and Wagner, 2008Shohamy D. Wagner A.D. Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events.Neuron. 2008; 60: 378-389Abstract Full Text Full Text PDF PubMed Scopus (347) Google Scholar). Lastly, the hippocampus has been suggested to play an essential role in another (third) reinforcement learning strategy called episodic control (Lengyel and Dayan, 2008Lengyel M. Dayan P. Hippocampal contributions to control: the third way.Adv. Neural Inf. Process. Syst. 2008; 20: 889-896Google Scholar), whereby decisions are guided by the memories of past episodes instead of a model. Further studies should address whether the hippocampus differentially contributes to multiple reinforcement learning strategies. In sum, Vikbladh et al., 2019Vikbladh O.M. Meager M.R. King J. Blackmon K. Devinsky O. Shohamy D. Burgess N. Daw N.D. Hippocampal contributions to model-based planning and spatial memory.Neuron. 2019; 102 (this issue): 683-693Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar show that hippocampus may be causally involved in model-based planning in humans. Further work will be needed to shed light on the computational mechanisms by which this process is mediated. Hippocampal Contributions to Model-Based Planning and Spatial MemoryVikbladh et al.NeuronMarch 11, 2019In BriefTesting patients with hippocampal damage, Vikbladh et al. demonstrate that model-based planning and place memory rely on a common hippocampal substrate. The study bridges the reinforcement learning and spatial memory literatures to clarify the scope of hippocampal contributions to behavior. Full-Text PDF Open Archive