Reinforcement Learning, Fast and Slow
Authors: Matthew Botvinick, Sam Ritter, Jane X. Wang, Zeb Kurth-Nelson, Charles Blundell, Demis Hassabis
Journal: Trends in Cognitive Sciences (2019)

Recent AI research has given rise to powerful techniques for deep reinforcement learning. In their combination of representation learning with reward-driven behavior, deep reinforcement learning would appear to have inherent interest for psychology and neuroscience. One reservation has been that deep reinforcement learning procedures demand large amounts of training data, suggesting that these algorithms may differ fundamentally from those underlying human learning. While this concern applies to the initial wave of deep RL techniques, subsequent AI work has established methods that allow deep RL systems to learn more quickly and efficiently. Two particularly interesting and promising techniques center, respectively, on episodic memory and meta-learning. Alongside their interest as AI techniques, deep RL methods leveraging episodic memory and meta-learning have direct and interesting implications for psychology and neuroscience. One subtle but critically important insight which these techniques bring into focus is the fundamental connection between fast and slow forms of learning. Deep reinforcement learning (RL) methods have driven impressive advances in artificial intelligence in recent years, exceeding human performance in domains ranging from Atari to Go to no-limit poker. This progress has drawn the attention of cognitive scientists interested in understanding human learning. However, the concern has been raised that deep RL may be too sample-inefficient – that is, it may simply be too slow – to provide a plausible model of how humans learn. In the present review, we counter this critique by describing recently developed techniques that allow deep RL to operate more nimbly, solving problems much more quickly than previous methods. Although these techniques were developed in an AI context, we propose that they may have rich implications for psychology and neuroscience. A key insight, arising from these AI methods, concerns the fundamental connection between fast RL and slower, more incremental forms of learning. Deep reinforcement learning (RL) methods have driven impressive advances in artificial intelligence in recent years, exceeding human performance in domains ranging from Atari to Go to no-limit poker. This progress has drawn the attention of cognitive scientists interested in understanding human learning. However, the concern has been raised that deep RL may be too sample-inefficient – that is, it may simply be too slow – to provide a plausible model of how humans learn. In the present review, we counter this critique by describing recently developed techniques that allow deep RL to operate more nimbly, solving problems much more quickly than previous methods. Although these techniques were developed in an AI context, we propose that they may have rich implications for psychology and neuroscience. A key insight, arising from these AI methods, concerns the fundamental connection between fast RL and slower, more incremental forms of learning. Over just the past few years, revolutionary advances have occurred in artificial intelligence (AI) research, where a resurgence in neural network or 'deep learning' methods [1LeCun Y. et al.Deep learning.Nature. 2015; 521: 436Crossref PubMed Scopus (42113) Google Scholar, 2Goodfellow I. et al.Deep Learning. Vol. 1. MIT Press, 2016Google Scholar] has fueled breakthroughs in image understanding [3Krizhevsky A. et al.Imagenet classification with deep convolutional neural networks.Adv. Neural Inf. Process. Syst. 2012; : 1097-1105Google Scholar, 4Eslami S.M.A. et al.Neural scene representation and rendering.Science. 2018; 360: 1204-1210Crossref PubMed Scopus (264) Google Scholar], natural language processing [5Bahdanau D. et al.Neural machine translation by jointly learning to align and translate.arXiv. 2014; : 1409.0473Google Scholar, 6Van Den Oord A. et al.Wavenet: a generative model for raw audio.arXiv. 2016; : 1609.03499Google Scholar], and many other areas. These developments have attracted growing interest from psychologists, psycholinguists, and neuroscientists, curious about whether developments in AI might suggest new hypotheses concerning human cognition and brain function [7Marblestone A.H. et al.Toward an integration of deep learning and neuroscience.Front. Comput. Neurosci. 2016; 10: 94Crossref PubMed Scopus (316) Google Scholar, 8Song H.F. et al.Reward-based training of recurrent neural networks for cognitive and value-based tasks.eLife. 2017; 6: e21492Crossref PubMed Scopus (1) Google Scholar, 9Yamins D.L.K. DiCarlo J.J. Using goal-driven deep learning models to understand sensory cortex.Nat. Neurosci. 2016; 19: 356Crossref PubMed Scopus (650) Google Scholar, 10Sussillo D. et al.A neural network that finds a naturalistic solution for the production of muscle activity.Nat. Neurosci. 2015; 18: 1025Crossref PubMed Scopus (229) Google Scholar, 11Khaligh-Razavi S.-M. Kriegeskorte N. Deep supervised, but not unsupervised, models may explain it cortical representation.PLoS Comput. Biol. 2014; 10: e1003915Crossref PubMed Scopus (554) Google Scholar]. One area of AI research that appears particularly inviting from this perspective is deep RL (Box 1). Deep RL marries neural network modeling (see Glossary) with reinforcement learning, a set of methods for learning from rewards and punishments rather than from more explicit instruction [12Sutton R.S. Barto A.G. Reinforcement Learning: An Introduction. MIT Press, 2018Google Scholar]. After decades as an aspirational rather than practical idea, deep RL has within the past 5 years exploded into one of the most intense areas of AI research, generating super-human performance in tasks from video games [13Mnih V. et al.Human-level control through deep reinforcement learning.Nature. 2015; 518: 529Crossref PubMed Scopus (13741) Google Scholar] to poker [14Moravčík M. et al.Deepstack: expert-level artificial intelligence in heads-up no-limit poker.Science. 2017; 356: 508-513Crossref PubMed Scopus (431) Google Scholar], multiplayer contests [15Jaderberg M. et al.Human-level performance in first-person multiplayer games with population-based deep reinforcement learning.arXiv. 2018; : 1807.01281Google Scholar], and complex board games, including go and chess [16Silver D. et al.Mastering the game of go with deep neural networks and tree search.Nature. 2016; 529: 484Crossref PubMed Scopus (8554) Google Scholar, 17Silver D. et al.Mastering chess and shogi by self-play with a general reinforcement learning algorithm.arXiv. 2017; : 1712.01815Google Scholar, 18Silver D. et al.Mastering the game of go without human knowledge.Nature. 2017; 550: 354Crossref PubMed Scopus (4476) Google Scholar, 19Silver D. et al.A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science. 2018; 362: 1140-1144Crossref PubMed Scopus (1270) Google Scholar].Box 1Deep Reinforcement LearningRL centers on the problem of learning a behavioral policy, a mapping from states or situations to actions, which maximizes cumulative long-term reward [12Sutton R.S. Barto A.G. Reinforcement Learning: An Introduction. MIT Press, 2018Google Scholar]. In simple settings, the policy can be represented as a look-up table, listing the appropriate action for any state. In richer environments, however, this kind of simple listing is infeasible, and the policy must instead be encoded implicitly as a parameterized function. Pioneering work in the 1990s showed how this function could be approximated using a multilayer (or deep) neural network ([78Tesauro G. Temporal difference learning and td-gammon.Commun. ACM. 1995; 38: 58-68Crossref Scopus (964) Google Scholar], L.J. Lin, PhD Thesis, Carnegie Melon University, 1993), allowing gradient-descent learning to discover rich, nonlinear mappings from perceptual inputs to actions (see panel A and below). However, technical challenges prevented the integration of deep neural networks with RL until 2015, when breakthrough work demonstrated how deep RL could be made to work in complex domains such as Atari video games [13Mnih V. et al.Human-level control through deep reinforcement learning.Nature. 2015; 518: 529Crossref PubMed Scopus (13741) Google Scholar] (see Figure IB and below). Since then, rapid progress has been made toward improving and scaling deep RL [79Hessel M. et al.Rainbow: combining improvements in deep reinforcement learning.arXiv. 2017; : 1710.02298Google Scholar], allowing its application to complex task domains such as Go [16Silver D. et al.Mastering the game of go with deep neural networks and tree search.Nature. 2016; 529: 484Crossref PubMed Scopus (8554) Google Scholar] and Capture the Flag [80Jaderberg M. et al.Population based training of neural networks.arXiv. 2017; : 1711.09846Google Scholar]. In many cases, the later advances have involved integrating deep RL with architectural and algorithmic complements, such as tree search [16Silver D. et al.Mastering the game of go with deep neural networks and tree search.Nature. 2016; 529: 484Crossref PubMed Scopus (8554) Google Scholar] or slot-based, episodic-like memory [52Graves A. et al.Hybrid computing using a neural network with dynamic external memory.Nature. 2016; 538: 471Crossref PubMed Scopus (801) Google Scholar] (see Figure IC and below). Other developments have focused on the goal of learning speed, allowing deep RL to make progress based on just a few observations, as reviewed in the main text.The figure illustrates the evolution of deep RL methods, starting in panel A with Tesauro's groundbreaking backgammon-playing system 'TD-gammon' [78Tesauro G. Temporal difference learning and td-gammon.Commun. ACM. 1995; 38: 58-68Crossref Scopus (964) Google Scholar]. This centered on a neural network which took as input a representation of the board and learned to output an estimate of the 'state value,' defined as expected cumulative future rewards, here equal simply to the estimated probability of eventually winning a game from the current position. Panel B shows the Atari-playing DQN network reported by Mnih and colleagues [13Mnih V. et al.Human-level control through deep reinforcement learning.Nature. 2015; 518: 529Crossref PubMed Scopus (13741) Google Scholar]. Here, a convolutional neural network (see [3Krizhevsky A. et al.Imagenet classification with deep convolutional neural networks.Adv. Neural Inf. Process. Syst. 2012; : 1097-1105Google Scholar]) takes screen pixels as input and learns to output joystick actions. Panel C shows a schematic representation of a state-of-the art deep RL system reported by Wayne and colleagues [51Wayne G. et al.Unsupervised predictive memory in a goal-directed agent.arXiv. 2018; : 1803.10760Google Scholar]. A full description of the detailed 'wiring' of this RL agent is beyond the scope of the present paper (but can be found in [51Wayne G. et al.Unsupervised predictive memory in a goal-directed agent.arXiv. 2018; : 1803.10760Google Scholar]). However, as the figure indicates, the architecture comprises multiple modules, including a neural network that leverages an episodic-like memory to predict upcoming events, which 'speaks' to a reinforcement-learning module that selects actions based on the predictor module's current state. The system learns, among other tasks, to perform goal-directed navigation in maze-like environments, as shown in Figure I. RL centers on the problem of learning a behavioral policy, a mapping from states or situations to actions, which maximizes cumulative long-term reward [12Sutton R.S. Barto A.G. Reinforcement Learning: An Introduction. MIT Press, 2018Google Scholar]. In simple settings, the policy can be represented as a look-up table, listing the appropriate action for any state. In richer environments, however, this kind of simple listing is infeasible, and the policy must instead be encoded implicitly as a parameterized function. Pioneering work in the 1990s showed how this function could be approximated using a multilayer (or deep) neural network ([78Tesauro G. Temporal difference learning and td-gammon.Commun. ACM. 1995; 38: 58-68Crossref Scopus (964) Google Scholar], L.J. Lin, PhD Thesis, Carnegie Melon University, 1993), allowing gradient-descent learning to discover rich, nonlinear mappings from perceptual inputs to actions (see panel A and below). However, technical challenges prevented the integration of deep neural networks with RL until 2015, when breakthrough work demonstrated how deep RL could be made to work in complex domains such as Atari video games [13Mnih V. et al.Human-level control through deep reinforcement learning.Nature. 2015; 518: 529Crossref PubMed Scopus (13741) Google Scholar] (see Figure IB and below). Since then, rapid progress has been made toward improving and scaling deep RL [79Hessel M. et al.Rainbow: combining improvements in deep reinforcement learning.arXiv. 2017; : 1710.02298Google Scholar], allowing its application to complex task domains such as Go [16Silver D. et al.Mastering the game of go with deep neural networks and tree search.Nature. 2016; 529: 484Crossref PubMed Scopus (8554) Google Scholar] and Capture the Flag [80Jaderberg M. et al.Population based training of neural networks.arXiv. 2017; : 1711.09846Google Scholar]. In many cases, the later advances have involved integrating deep RL with architectural and algorithmic complements, such as tree search [16Silver D. et al.Mastering the game of go with deep neural networks and tree search.Nature. 2016; 529: 484Crossref PubMed Scopus (8554) Google Scholar] or slot-based, episodic-like memory [52Graves A. et al.Hybrid computing using a neural network with dynamic external memory.Nature. 2016; 538: 471Crossref PubMed Scopus (801) Google Scholar] (see Figure IC and below). Other developments have focused on the goal of learning speed, allowing deep RL to make progress based on just a few observations, as reviewed in the main text. The figure illustrates the evolution of deep RL methods, starting in panel A with Tesauro's groundbreaking backgammon-playing system 'TD-gammon' [78Tesauro G. Temporal difference learning and td-gammon.Commun. ACM. 1995; 38: 58-68Crossref Scopus (964) Google Scholar]. This centered on a neural network which took as input a representation of the board and learned to output an estimate of the 'state value,' defined as expected cumulative future rewards, here equal simply to the estimated probability of eventually winning a game from the current position. Panel B shows the Atari-playing DQN network reported by Mnih and colleagues [13Mnih V. et al.Human-level control through deep reinforcement learning.Nature. 2015; 518: 529Crossref PubMed Scopus (13741) Google Scholar]. Here, a convolutional neural network (see [3Krizhevsky A. et al.Imagenet classification with deep convolutional neural networks.Adv. Neural Inf. Process. Syst. 2012; : 1097-1105Google Scholar]) takes screen pixels as input and learns to output joystick actions. Panel C shows a schematic representation of a state-of-the art deep RL system reported by Wayne and colleagues [51Wayne G. et al.Unsupervised predictive memory in a goal-directed agent.arXiv. 2018; : 1803.10760Google Scholar]. A full description of the detailed 'wiring' of this RL agent is beyond the scope of the present paper (but can be found in [51Wayne G. et al.Unsupervised predictive memory in a goal-directed agent.arXiv. 2018; : 1803.10760Google Scholar]). However, as the figure indicates, the architecture comprises multiple modules, including a neural network that leverages an episodic-like memory to predict upcoming events, which 'speaks' to a reinforcement-learning module that selects actions based on the predictor module's current state. The system learns, among other tasks, to perform goal-directed navigation in maze-like environments, as shown in Figure I. Beyond its inherent interest as an AI topic, deep RL would appear to hold special interest for psychology and neuroscience. The mechanisms that drive learning in deep RL were originally inspired by animal conditioning research [20Sutton R.S. Barto A.G. Toward a modern theory of adaptive networks: expectation and prediction.Psychol. Rev. 1981; 88: 135Crossref PubMed Scopus (924) Google Scholar] and are believed to relate closely to neural mechanisms for reward-based learning centering on dopamine [21Schultz W. et al.A neural substrate of prediction and reward.Science. 1997; 275: 1593-1599Crossref PubMed Scopus (5895) Google Scholar]. At the same time, deep RL leverages neural networks to learn powerful representations that support generalization and transfer, key abilities of biological brains. Given these connections, deep RL would appear to offer a rich source of ideas and hypotheses for researchers interested in human and animal learning, both at the behavioral and neuroscientific levels. And indeed, researchers have started to take notice [7Marblestone A.H. et al.Toward an integration of deep learning and neuroscience.Front. Comput. Neurosci. 2016; 10: 94Crossref PubMed Scopus (316) Google Scholar, 8Song H.F. et al.Reward-based training of recurrent neural networks for cognitive and value-based tasks.eLife. 2017; 6: e21492Crossref PubMed Scopus (1) Google Scholar]. At the same time, commentary on the first wave of deep RL research has also sounded a note of caution. On first blush it appears that deep RL systems learn in a fashion quite different from humans. The hallmark of this difference, it has been argued, lies in the sample efficiency of human learning versus deep RL. Sample efficiency refers to the amount of data required for a learning system to attain any chosen target level of performance. On this measure, the initial wave of deep RL systems indeed appear drastically different from human learners. To attain expert human-level performance on tasks such as Atari video games or chess, deep RL systems have required many orders of magnitude more training data than human experts themselves [22Tsividis P.A. et al.Human learning in Atari.2017 AAAI Spring Symposium Series. 2017; Google Scholar]. In short, deep RL, at least in its initial incarnation, appears much too slow to offer a plausible model for human learning. Or so the argument has gone [23Lake B.M. et al.Human-level concept learning through probabilistic program induction.Science. 2015; 350: 1332-1338Crossref PubMed Scopus (1414) Google Scholar, 24Marcus G. Deep learning: a critical appraisal.arXiv. 2018; : 1801.00631Google Scholar]. The critique is indeed applicable to the first wave of deep RL methods, reported beginning around 2013 (e.g., [25Mnih V. et al.Playing atari with deep reinforcement learning.arXiv. 2013; : 1312.5602Google Scholar]). However, even in the short time since then, important innovations have occurred in deep RL research, which show how the sample efficiency of deep RL can be dramatically increased. These methods mitigate the original demands made by deep RL for huge amounts of training data, effectively allowing deep RL to be fast. The emergence of these computational techniques revives deep RL as a candidate model of human learning and a source of insight for psychology and neuroscience. In the present review, we consider two key deep RL methods that mitigate the sample efficiency problem: episodic deep RL and meta-RL. We examine how these techniques enable fast deep RL and consider their potential implications for psychology and neuroscience. A key starting point for considering techniques for fast RL is to examine why initial methods for deep RL were in fact so slow. Here, we describe two of the primary sources of sample inefficiency. At the end of the paper, we will circle back to examine how the constellations of issues described by these two concepts are in fact connected. The first source of slowness in deep RL is the requirement for incremental parameter adjustment. Initial deep RL methods (which are still very widely used in AI research) employed gradient descent to sculpt the connectivity of a deep neural network mapping from perceptual inputs to action outputs (Box 1). As has been widely discussed not only in AI but also in psychology [26Kumaran D. et al.What learning systems do intelligent agents need? complementary learning systems theory updated.Trends Cogn. Sci. 2016; 20: 512-534Abstract Full Text Full Text PDF PubMed Scopus (278) Google Scholar], the adjustments made during this form of learning must be small, in order to maximize generalization [27Hardt M. et al.Train faster, generalize better: stability of stochastic gradient descent.arXiv. 2015; : 1509.01240Google Scholar] and avoid overwriting the effects of earlier learning (an effect sometimes referred to as 'catastrophic interference'). This demand for small step-sizes in learning is one source of slowness in the methods originally proposed for deep RL. A second source is weak inductive bias. A basic lesson of learning theory is that any learning procedure necessarily faces a bias–variance trade-off: the stronger the initial assumptions the learning procedure makes about the patterns to be learned (i.e., the stronger the initial inductive bias of the learning procedure) the less data will be required for learning to be accomplished (assuming the initial inductive bias matches what's in the data!). A learning procedure with weak inductive bias will be able to master a wider range of patterns (greater variance), but will in general be less sample-efficient [28Bishop C.M. Pattern Recognition and Machine Learning (information science and statistics). Springer-Verlag, 2006Google Scholar]. In effect, strong inductive bias is what allows fast learning. A learning system that only considers a narrow range of hypotheses when interpreting incoming data will, perforce, hone in on the correct hypothesis more rapidly than a system with weaker inductive biases (again, assuming the correct hypothesis falls within that narrow initial range). Importantly, generic neural networks are extremely low-bias learning systems; they have many parameters (connection weights) and are capable of using these to fit a wide range of data. As dictated by the bias–variance trade-off, this means that neural networks, in the generic form employed in the first deep RL models (Box 1) tend to be sample-inefficient, requiring large amounts of data to learn. Together, these two factors—incremental parameter adjustment and weak inductive bias—explain the slowness of first-generation deep RL models. However, subsequent research has made clear that both of these factors can be mitigated, allowing deep RL to proceed in a much more sample-efficient manner. In what follows, we consider two specific techniques, one of which confronts the incremental parameter adjustment problem, and the other of which confronts the problem of weak inductive bias. In addition to their implications within the AI field, both of these AI techniques bear suggestive links with psychology and neuroscience, as we shall detail. If incremental parameter adjustment is one source of slowness in deep RL, then one way to learn faster might be to avoid such incremental updating. Naively increasing the learning rate governing gradient descent optimization leads to the problem of catastrophic interference. However, recent research shows that there is another way to accomplish the same goal, which is to keep an explicit record of past events, and use this record directly as a point of reference in making new decisions. This idea, referred to as episodic RL [29Pritzel A. et al.Neural episodic control.arXiv. 2017; : 1703.01988Google Scholar, 30Gershman S.J. Daw N.D. Reinforcement learning and episodic memory in humans and animals: an integrative framework.Annu. Rev. Psychol. 2017; 68: 101-128Crossref PubMed Scopus (47) Google Scholar, 42Lengyel M. Dayan P. Hippocampal contributions to control: the third way.Adv. Neural Inf. Process. Syst. 2008; : 889-896Google Scholar], parallels 'non-parametric' approaches in machine learning [28Bishop C.M. Pattern Recognition and Machine Learning (information science and statistics). Springer-Verlag, 2006Google Scholar] and resembles 'instance-' or 'exemplar-based' theories of learning in psychology [31Logan G.D. Toward an instance theory of automatization.Psychol. Rev. 1988; 95: 492Crossref Scopus (2336) Google Scholar, 32Smith E.E. Medin D.L. Categories and Concepts. Vol. 9. Harvard University Press, 1981Google Scholar]. When a new situation is encountered and a decision must be made concerning what action to take, the procedure is to compare an internal representation of the current situation with stored representations of past situations. The action chosen is then the one associated with the highest value, based on the outcomes of the past situations that are most similar to the present. When the internal state representation is computed by a multilayer neural network, we refer to the resulting algorithm as 'episodic deep RL'. A more detailed explanation of the mechanics of episodic deep RL is presented in Box 2.Box 2Episodic Deep RLEpisodic RL algorithms estimate the value of actions and states using episodic memories [30Gershman S.J. Daw N.D. Reinforcement learning and episodic memory in humans and animals: an integrative framework.Annu. Rev. Psychol. 2017; 68: 101-128Crossref PubMed Scopus (47) Google Scholar, 43Bornstein A.M. et al.Reminders of past choices bias decisions for reward in humans.Nat. Commun. 2017; 8: 15958Crossref PubMed Scopus (106) Google Scholar, 44Bornstein A.M. Norman K.A. Reinstated episodic context guides sampling-based decisions for reward.Nat. Neurosci. 2017; 20: 997Crossref PubMed Scopus (82) Google Scholar]. Consider, for example, the episodic valuation algorithm depicted in Figure I, wherein the agent stores each encountered state along with the discounted sum of rewards obtained during the next n time steps. These two stored items comprise an episodic memory of the encountered state and the reward that followed. To estimate the value of a new state, the agent computes a sum of the stored discounted rewards, weighted by the similarity (sim.) between stored states and the new state. This algorithm can be extended to estimate action values by recording the actions taken along with the states and reward sums in the memory store, then querying the store to find only memories in which the to-be-evaluated action was taken. In fact, [81Blundell C. et al.Model-free episodic control.arXiv. 2016; : 1606.04460Google Scholar] used such an episodic RL algorithm to achieve strong performance on Atari games.The success of episodic RL depends on the state representations used to compute state similarity. In a follow-up to [81Blundell C. et al.Model-free episodic control.arXiv. 2016; : 1606.04460Google Scholar], Pritzel et al. [29Pritzel A. et al.Neural episodic control.arXiv. 2017; : 1703.01988Google Scholar] showed that performance could be improved by gradually shaping these state representations using gradient descent learning. These results demonstrated strong performance and state of the art data efficiency on the 57 games in the Atari Learning Environment [29Pritzel A. et al.Neural episodic control.arXiv. 2017; : 1703.01988Google Scholar], showcasing the benefits of combining slow (representation) learning and fast (value) learning. Episodic RL algorithms estimate the value of actions and states using episodic memories [30Gershman S.J. Daw N.D. Reinforcement learning and episodic memory in humans and animals: an integrative framework.Annu. Rev. Psychol. 2017; 68: 101-128Crossref PubMed Scopus (47) Google Scholar, 43Bornstein A.M. et al.Reminders of past choices bias decisions for reward in humans.Nat. Commun. 2017; 8: 15958Crossref PubMed Scopus (106) Google Scholar, 44Bornstein A.M. Norman K.A. Reinstated episodic context guides sampling-based decisions for reward.Nat. Neurosci. 2017; 20: 997Crossref PubMed Scopus (82) Google Scholar]. Consider, for example, the episodic valuation algorithm depicted in Figure I, wherein the agent stores each encountered state along with the discounted sum of rewards obtained during the next n time steps. These two stored items comprise an episodic memory of the encountered state and the reward that followed. To estimate the value of a new state, the agent computes a sum of the stored discounted rewards, weighted by the similarity (sim.) between stored states and the new state. This algorithm can be extended to estimate action values by recording the actions taken along with the states and reward sums in the memory store, then querying the store to find only memories in which the to-be-evaluated action was taken. In fact, [81Blundell C. et al.Model-free episodic control.arXiv. 2016; : 1606.04460Google Scholar] used such an episodic RL algorithm to achieve strong performance on Atari games. The success of episodic RL depends on the state representations used to compute state similarity. In a follow-up to [81Blundell C. et al.Model-free episodic control.arXiv. 2016; : 1606.04460Google Scholar], Pritzel et al. [29Pritzel A. et al.Neural episodic control.arXiv. 2017; : 1703.01988Google Scholar] showed that performance could be improved by gradually shaping these state representations using gradient descent learning. These results demonstrated strong performance and state of the art data efficiency on the 57 games in the Atari Learning Environment [29Pritzel A. et al.Neural episodic control.arXiv. 2017; : 1703.01988Google Scholar], showcasing the benefits of combining slow (representation) learning and fast (value) learning. In episodic deep RL, unlike the standard incremental approach, the information gained through each experienced event can be leveraged immediately to guide behavior. However, whereas episodic deep RL is able to go 'fast' where earlier methods for deep RL went 'slow,' there is a twist to this story: the fast learning of episodic