Original: Beyond Tabula Rasa: Reincarnating Reinforcement Learning
Revised: Beyond Tabula Rasa: Reincarnating Reinforcement Learning

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Learning tabula rasa , that is without any prior knowledge, is the prevalent workﬂowin reinforcement learning (RL) research.
Revised: Learning tabula rasa , that is without any previously learned knowledge, is the prevalent workﬂow in reinforcement learning (RL) research.

Original: However, RL systems, when applied tolarge-scale settings, rarely operate tabula rasa.
Revised: However, RL systems, when applied to large-scale settings, rarely operate tabula rasa.

Original: Such large-scale systems undergomultiple design or algorithmic changes during their development cycle and use adhoc approaches for incorporating these changes without re-training from scratch,which would have been prohibitively expensive.
Revised: Such large-scale systems undergo multiple design or algorithmic changes during their development cycle and use ad hoc approaches for incorporating these changes without re-training from scratch, which would have been prohibitively expensive.

Original: Additionally, the inefﬁciency ofdeep RL typically excludes researchers without access to industrial-scale resourcesfrom tackling computationally-demanding problems.
Revised: Additionally, the inefﬁciency of deep RL typically excludes researchers without access to industrialscale resources from tackling computationally-demanding problems.

Original: To address these issues, wepresent reincarnating RL as an alternative workﬂow or class of problem settings,where prior computational work ( e.g., learned policies) is reused or transferredbetween design iterations of an RL agent, or from one RL agent to another.
Revised: To address these issues, we present reincarnating RL as an alternative workﬂow or class of problem settings, where prior computational work ( e.g., learned policies) is reused or transferred between design iterations of an RL agent, or from one RL agent to another.

Original: As astep towards enabling reincarnating RL from any agent to any other agent, we focuson the speciﬁc setting of efﬁciently transferring an existing sub-optimal policyto a standalone value-based RL agent.
Revised: As a step towards enabling reincarnating RL from any agent to any other agent, we focus on the speciﬁc setting of efﬁciently transferring an existing sub-optimal policy to a standalone value-based RL agent.

Original: We ﬁnd that existing approaches fail inthis setting and propose a simple algorithm to address their limitations.
Revised: We ﬁnd that existing approaches fail in this setting and propose a simple algorithm to address their limitations.

Original: Equippedwith this algorithm, we demonstrate reincarnating RL’s gains over tabula rasa RLon Atari 2600 games, a challenging locomotion task, and the real-world problemof navigating stratospheric balloons.
Revised: Equipped with this algorithm, we demonstrate reincarnating RL’s gains over tabula rasa RL on Atari 2600 games, a challenging locomotion task, and the real-world problem of navigating stratospheric balloons.

Original: Overall, this work argues for an alternateapproach to RL research, which we believe could signiﬁcantly improve real-world RL adoption and help democratize it further.
Revised: Overall, this work argues for an alternate approach to RL research, which we believe could signiﬁcantly improve real-world RL adoption and help democratize it further.

Revised: Open-sourced code and trained agents at agarwl.github.io/reincarnating_rl .

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) is a general-purpose paradigm for making data-driven decisions.
Revised: Reinforcement learning (RL) is a general-purpose paradigm for making data-driven decisions.

Original: Dueto this generality, the prevailing trend in RL research is to learn systems that can operate efﬁcientlytabula rasa , that is without muchpreviously learned knowledge about the problem.
Revised: Due to this generality, the prevailing trend in RL research is to learn systems that can operate efﬁciently tabula rasa , that is without much learned knowledge including prior computational work such as existing datasets or learned policies. However, by throwing away prior computation, tabula rasa RL impedes the ability of deep neural networks to act as a “knowledge” sponge, which is one of the reasons behind their remarkable success.

Original: However,tabula rasa RL systems are typically the exception rather than the norm for solving large-scale RL problems [3, 11, 64, 73].
Revised: Furthermore, tabula rasa RL systems are typically the exception rather than the norm for solving large-scale RL problems [3, 4, 13, 54, 72, 82].

Original: Such large-scale RL systems often need to function for long periodsof time and continually experience new data; restarting them from scratch may require weeks ifnot months of computation, and there may be billions of data points to re-process – this makesthe tabula rasa approach impractical.
Revised: Such large-scale RL systems often need to function for long periods of time and continually experience new data; restarting them from scratch may require weeks if not months of computation, and there may be billions of data points to re-process – this makes the tabula rasa approach impractical.

Original: For example, the system that plays Dota 2 at a superhumanlevel [11] or the system that manipulates a robot hand for solving Rubik’s cube [3], underwent severalmonths of RL training with continual changes ( e.g., in model architecture, environment, etc ) duringtheir development; this necessitated building upon the previously trained system after such changesto circumvent re-training from scratch, which was done using ad hoc approaches.
Revised: For example, the system that plays Dota 2 at a human-like level [13] underwent several months of RL training with continual changes ( e.g., in model architecture, environment, etc ) during its development; this necessitated building upon the previously trained system after such changes to circumvent retraining from scratch, which was done using ad hoc approaches.

Original: Furthermore,tackling challenging problems with deep RL often incurs substantial computational and ﬁnancial cost: AlphaStar [73], which achieves grandmaster level in StarCraft, was trained using TPUs for more thana month and replicating it would cost several million dollars 1 .
Revised: Furthermore, tackling challenging problems with deep RL often incurs substantial computational and ﬁnancial cost: AlphaStar [82], which achieves grandmaster level in StarCraft, was trained using TPUs for more than a month and replicating it would cost several million dollars (Appendix A.1).

Original: As a result, the majority of the RLcommunity outside certain resource-rich labs is currently excluded from tackling such problems
Revised: As a result, the majority of the RL community outside certain resource-rich labs is currently excluded from tackling such problems.

Original: [Figure] To address both the computational and sample inefﬁciencies of tabula rasa RL, we present reincarnating RL (RRL) as an alternative research workﬂow or  class of problem settings.
Revised: To address both the computational and sample inefﬁciencies of tabula rasa RL we present reincarnating RL (RRL) as an alternative research workﬂow or a class of problems to focus on.

Original: RRL focuses onmaximally leveraging existing computational work, such as learned network weights and collecteddata, to accelerate training across design iterations of an RL agent or when moving from one agent toanother.
Revised: RRL seeks to maximally leverage existing computational work, such as learned network weights and collected data , to accelerate training across design iterations of an RL agent or when moving from one agent to another.

Original: In this workﬂow, agents need not be trained tabula rasa, except for initial forays into newproblems.
Revised: In RRL, agents need not be trained tabula rasa, except for initial forays into new problems.

Original: Different RRL problems can be instantiated depending on how the prior computationalwork is provided: logged datasets, learned policies, pretrained dynamics models, representations,etc (examples in Appendix A.6).
Revised: For example, imagine a researcher who has trained an agent A 1 for a long time ( e.g., weeks) but now this or another researcher wants to experiment with better architectures or RL algorithms. While the tabula rasa workﬂow requires re-training another agent from scratch, reincarnating RL provides the more viable option of transferring A 1 to another agent and training this agent further, or simply ﬁne-tuning A 1 (Figure 1).

Original:  RRL is an attempt to provide amore formal foundation for the priorad hoc strategies to confront the challenges of large-scale RL model development [ e.g., 3, 11].
Revised: As such, RRL can be viewed as an attempt to provide a formal foundation for the research workﬂow needed for real-world and large-scale RL models.

Original: Reincarnating RLalso suggests a different benchmarking paradigm that can democratize RL byallowing the broader community to tackle  problems that are currently infeasible with tabula rasa
Revised: Reincarnating RL can democratize research by allowing the broader community to tackle larger-scale and complex RL problems without requiring excessive computational resources.

Original: RL, and even continually improve and update existing trained agents, especially on problems whereimproving performance can lead to real-world impact ( e.g., balloon navigation [9], chip design [51],tokamak control [22]). For example, imagine a researcher who has trained a deep RL agent A 1for a long time ( e.g., weeks), but now this or another researcher wants to experiment with betteralgorithms or architectures.
Revised: As a consequence, RRL can also help avoid the risk of researchers overﬁtting to conclusions from small-scale RL problems. Furthermore, RRL can enable a benchmarking paradigm where researchers continually improve and update existing trained agents, especially on problems where improving performance has real-world impact ( e.g., balloon navigation [11], chip design [57], tokamak control [24]).

Original: While the tabula rasa workﬂow requires re-training another agent fromscratch, reincarnating RL provides the more viable option of transferring A 1 to another agent andtraining this agent further, or simply ﬁne-tuning A 1 (Figure 1).
Revised: Furthermore, a common real-world RL use case will likely be in scenarios where prior computational work is available ( e.g., existing deployed RL policies), making RRL important to study.

Original: However, beyond some large-scalereincarnation effortswith limited applicability (Section 3), theresearch community has not focusedmuch on studying reincarnating RL as a research problem, in part due to the prevalence of tabula rasa RL.
Revised: However, beyond some ad hoc large-scale reincarnation efforts (Section 3), the community has not focused much on studying reincarnating RL as a research problem in its own right.

Original: To this end, this work investigates this setting and its potential for accelerating RL research.
Revised: To this end, this work argues for developing general-purpose RRL approaches as opposed to ad hoc solutions.

Revised: Different RRL problems can be instantiated depending on how the prior computational work is provided: logged datasets, learned policies, pretrained models, representations, etc .

Original: As a step towards developing broadly applicable reincarnation approaches, we focus on the speciﬁcsetting of policy-to-value reincarnating RL (PVRL) for efﬁciently transferring a suboptimal teacherpolicy to a value-based RL student agent (Section 4).
Revised: As a step towards developing broadly applicable reincarnation approaches, we focus on the speciﬁc setting of policy-to-value reincarnating RL (PVRL) for efﬁciently transferring a suboptimal teacher policy to a value-based RL student agent (Section 4).

Original: Since it is undesirable to maintain dependencyon past teachers for successive reincarnations, we require a PVRL algorithm to “wean” off the teacherdependence as training progresses.
Revised: Since it is undesirable to maintain dependency on past teachers for successive reincarnations, we require a PVRL algorithm to “wean” off the teacher dependence as training progresses.

Original: We ﬁnd that prior approaches, when evaluated for PVRL onthe Arcade Learning Environment (ALE) [8], either result in small improvements over the tabularasa student or exhibit degradation when weaning off the teacher.
Revised: We ﬁnd that prior approaches, when evaluated for PVRL on the Arcade Learning Environment (ALE) [10], either result in small improvements over the tabula rasa student or exhibit degradation when weaning off the teacher.

Original: To address these limitations, weintroduce QDagger, which combines Dagger [60] with n -step Q-learning, and outperforms priorapproaches.
Revised: To address these limitations, we introduce QDagger, which combines Dagger [68] with n -step Q-learning, and outperforms prior approaches.

Original: Equipped with QDagger, we demonstrate the sample and compute-efﬁciency gains ofreincarnating RL over tabula rasa RL, on ALE and a humanoid locomotion task. Reincarnating RLalso makes it easier to make progress on a simulated real-world problem of navigating stratosphericballoons [9] in a computationally feasible manner (Section 5).
Revised: Equipped with QDagger, we demonstrate the sample and compute-efﬁciency gains of reincarnating RL over tabula rasa RL, on ALE, a humanoid locomotion task and the simulated real-world problem of navigating stratospheric balloons [11] (Section 5).

Original: We also discuss some considerations in reincarnating RL that require further investigation (Section 6). Finally, to improve the benchmarkingecosystem in reincarnating RL, we will open-source our code and trained agents.
Revised: Finally, we discuss some considerations in RRL as well as address reproducibility and generalizability concerns.

Original:  Preliminaries
Revised:  Preliminaries

Original: The goal in RL is to maximize the long-term discounted reward in an environment.
Revised: The goal in RL is to maximize the long-term discounted reward in an environment.

Original: We modelthe environment as an MDP, deﬁned as ( S , A , R, P, γ ) [59], with a state space S , an action space
Revised: We model the environment as an MDP, deﬁned as ( S , A , R, P, γ ) [66], with a state space S , an action space

Original: A , a stochastic reward function R ( s, a ) , transition dynamics P ( s (cid:48) | s, a ) and a discount factor γ ∈[0 , 1) .
Revised: A , a stochastic reward function R ( s, a ) , transition dynamics P ( s (cid:48) | s, a ) and a discount factor γ ∈ [0 , 1) .

Original: A policy π ( ·| s ) maps states to a distribution over actions.
Revised: A policy π ( ·| s ) maps states to a distribution over actions.

Original: The Q-value function Q π ( s, a )for a policy π ( ·| s ) is the expected sum of discounted rewards obtained by executing action aat state s and following π ( ·| s ) thereafter.
Revised: The Q-value function Q π ( s, a ) for a policy π ( ·| s ) is the expected sum of discounted rewards obtained by executing action a at state s and following π ( ·| s ) thereafter.

Original: Q π ( s, a ) is the ﬁxed point of Q ( s, a ) =
Revised: Q π ( s, a ) is the ﬁxed point of Q ( s, a ) =

Original: R ( s, a ) + 
Revised: R ( s, a ) + γE s (cid:48) ∼ P ( ·| s,a ) ,a (cid:48) ∼ π ( ·| s (cid:48) )

Original: [Equation] [ Q ( s [Equation] , a 
Revised: [ Q ( s (cid:48) , a (cid:48) )] .

Original: [Equation] DQN [52] builds on Q-learning [75] and parameterizes the Qvalue function, Q θ , with a neural net with parameters θ while following an (cid:15) -greedy policy withrespect to Q θ for data collection.
Revised: DQN [58] builds on Q-learning [84] and parameterizes the Qvalue function, Q θ , with a neural net with parameters θ while following an (cid:15) -greedy policy with respect to Q θ for data collection.

Original: DQN minimizes the temporal difference (TD) loss, L TD ( D S ) , ontransition tuples, ( s, a, r, s (cid:48) ) , sampled from an experience replay buffer D S collected during training:
Revised: DQN minimizes the temporal difference (TD) loss, L TD ( D S ) , on transition tuples, ( s, a, r, s (cid:48) ) , sampled from an experience replay buffer D S collected during training:

Original: [Equation]
Revised: [Equation]

Original: (1) where ¯ Q θ is a delayed copy of the same Q-network, referred to as the target network .
Revised: where ¯ Q θ is a delayed copy of the same Q-network, referred to as the target network .

Original: Modern valuebased RL agents, such as Rainbow [31], use n -step returns to further stabilize learning.
Revised: Modern valuebased RL agents, such as Rainbow [35], use n -step returns to further stabilize learning.

Original: Speciﬁcally,rather than training the Q-value estimate Q ( s t , a t ) on the basis of the single-step temporal differenceerror r t + γ max a (cid:48) Q ( s t +1 , a (cid:48) )
Revised: Speciﬁcally, rather than training the Q-value estimate Q ( s t , a t ) on the basis of the single-step temporal difference error r t + γ max a (cid:48) Q ( s t +1 , a (cid:48) )

Original: − Q ( s t , a t ) , an n -step target (cid:80) n − 1 k =

Revised: − Q ( s t , a t ) , an n -step target (cid:80) n − 1 k =

Original: γ k r t + k + γ n max a (cid:48) Q ( s t + n , a (cid:48) )
Revised: γ k r t + k + γ n max a (cid:48) Q ( s t + n , a (cid:48) )

Original: −
Revised: −

Original: Q ( s t , a t ) is used in the TD loss, with intermediate future rewards stored in the replay D .
Revised: Q ( s t , a t ) is used in the TD loss, with intermediate future rewards stored in the replay D .

Original:  Related work
Revised:  Related work

Original: Large-scale reincarnation efforts .
Revised: Prior ad hoc reincarnation efforts .

Original: Several high-proﬁle RL achievements employ a limited formof reincarnating RL.
Revised: While several high-proﬁle RL achievements have used reincarnation, it has typically been done in an ad-hoc way and has limited applicability.

Original: OpenAI Five [11], which can play Dota2 at a superhuman level, required 10months of large-scale RL training and went through continual changes in code and environment ( e.g.,expanding observation spaces) during development.
Revised: OpenAI Five [13], which can play Dota 2 at a human-like level, required 10 months of large-scale RL training and went through continual changes in code and environment ( e.g., expanding observation spaces) during development.

Original: To avoid restarting from scratch after suchchanges, OpenAI Five used “ surgery ” akin to Net2Net [15] style transformations to convert a trainedmodel to certain bigger architectures with custom weight initializations.
Revised: To avoid restarting from scratch after such changes, OpenAI Five used “ surgery ” akin to Net2Net [17] style transformations to convert a trained model to certain bigger architectures with custom weight initializations.

Original: AlphaStar [73] employspopulation-based training ( PBT ) [37], which periodically copies weights of the best performing valuebased agents and mutates hyperparameters during training.
Revised: AlphaStar [82] employs population-based training ( PBT ) [42], which periodically copies weights of the best performing value-based agents and mutates hyperparameters during training.

Original: Although PBT and surgery methods areefﬁcient, they have they can not be used for reincarnating RL when switching to arbitrary architectures( e.g., feed-forward to recurrent networks) or from one model class to another ( e.g., policy to a valuefunction).
Revised: Although PBT and surgery methods are efﬁcient, they have they can not be used for reincarnating RL when switching to arbitrary architectures ( e.g., feed-forward to recurrent networks) or from one model class to another ( e.g., policy to a value function).

Original: In this work, we focus on reincarnating RL methods that allow such architecture andalgorithm changes.

Original: Akkaya et al.
Revised: Akkaya et al.

Original: [3] trained RL policies for several months to manipulate a robothand for solving Rubik’s cube.
Revised: [4] trained RL policies for several months to manipulate a robot hand for solving Rubik’s cube.

Original: To do so, they “rarely trained experiments from scratch” but insteadinitialized new policies, with architectural changes, from previous trained policies using behaviorcloning via on-policy distillation [18, 58].
Revised: To do so, they “rarely trained experiments from scratch” but instead initialized new policies, with architectural changes, from previous trained policies using behavior cloning via on-policy distillation [20, 65].

Original: AlphaGo [64] also used behavior cloning on humanreplays for initializing the policy and ﬁne-tuning it further with RL.
Revised: AlphaGo [72] also used behavior cloning on human replays for initializing the policy and ﬁne-tuning it further with RL.

Original: However, behavior cloning isonly applicable for policy to policy transfer and is inadequate for the PVRL setting of transferringa policy to a value function [ e.g., 54, 71].
Revised: However, behavior cloning is only applicable for policy to policy transfer and is inadequate for the PVRL setting of transferring a policy to a value function [ e.g., 61, 80].

Revised: Contrary to such approaches, we apply reincarnation in settings where these approaches are not applicable including transferring a DQN agent to Impala-CNN Rainbow in ALE, and a distributed agent with

Revised: MLP architecture to a recurrent agent in BLE.

Original: Several prior works also ﬁne-tune existing agents withdeep RL for reducing training time, especially on real-world tasks such as chip ﬂoor-planning [51],robotic manipulation [38], aligning language models [5], and compiler optimization [70].
Revised: Several prior works also ﬁne-tune existing agents with deep RL for reducing training time, especially on real-world tasks such as chip ﬂoor-planning [57], robotic manipulation [43], aligning language models [6], and compiler optimization [79].

Original: In linewith these works, we ﬁnd that ﬁne-tuning a value-based agent can be an effective reincarnationstrategy (Figure 7).
Revised: In line with these works, we ﬁnd that ﬁne-tuning a value-based agent can be an effective reincarnation strategy (Figure 7).

Original: However, ﬁne-tuning is  constrained to use the same architecture as the agentbeing ﬁne-tuned.
Revised: However, ﬁne-tuning is often constrained to use the same architecture as the agent being ﬁne-tuned.

Original: Instead, we focus on reincarnating RL methods that do not have this limitation.
Revised: Instead, we focus on reincarnating RL methods that do not have this limitation.

Revised: Leveraging prior computation .

Revised: While areas such as ofﬂine RL, imitation learning, transfer in RL, continual RL etc focus on developing methods to leverage prior computation, such areas don’t strive to change how we do RL research by incorporating such methods as a part of our workﬂow.

Revised: For completeness, we contrast closely related approaches to PVRL, the RRL setting we study.

Original:  Leveraging existing agents .
Revised: - Leveraging existing agents .

Original: Existing policies have been previously used for improving data collection in RL [12, 14, 25, 43, 66, 76]; we evaluate one suchrecent approach, JSRL [71], which improvesexploration in goal-reaching RL tasks.
Revised: Existing policies have been previously used for improving data collection [14, 16, 28, 74, 86]; we evaluate one such approach, JSRL [80], which improves exploration in goal-reaching RL tasks.

Original: However, our PVRL experiments indicate that JSRL performspoorly on ALE.
Revised: However, our PVRL experiments indicate that JSRL performs poorly on ALE.

Original: Closely related to PVRL, Schmitt et al.
Revised: Schmitt et al.

Original: [63] propose kickstarting to speed-up actorcritic agents using an interactive teacher policy by combining on-policy distillation [18, 58] with RL.
Revised: [71] propose kickstarting to speed-up actor-critic agents using an interactive teacher policy by combining on-policy distillation [20, 65] with RL.

Original: Empirically, we ﬁnd that kickstarting is a strong baseline for PVRL, however it exhibits unstablebehavior without n -step returns and underperforms QDagger.
Revised: Empirically, we ﬁnd that kickstarting is a strong baseline for PVRL, however it exhibits unstable behavior without n -step returns and underperforms QDagger.

Original: PVRL also falls under the frameworkof agents teaching agents (ATA) [19] with RL-based students and teachers.
Revised: PVRL also falls under the framework of agents teaching agents (ATA) [21] with RL-based students and teachers.

Original: While ATA approaches,such as action advice [69], emphasize how and when to query the teacher or evaluating the utilityof teacher advice, PVRL focuses on sample-efﬁcient transfer and does not impose constraints onquerying the teacher.
Revised: While ATA approaches, such as action advice [78], emphasize how and when to query the teacher or evaluating the utility of teacher advice, PVRL focuses on sample-efﬁcient transfer and does not impose constraints on querying the teacher.

Original: PVRL is also different from prior work on accelerating RL using a heuristic ororacle value function [7, 17, 67], as PVRL only assumes access to a suboptimal policy.
Revised: PVRL is also different from prior work on accelerating RL using a heuristic or oracle value function [9, 19, 75], as PVRL only assumes access to a suboptimal policy.

Revised: Unlike PVRL methods that wean off the teacher, imitation-regularized RL methods [50, 59] stay close to the suboptimal teacher, which can limit the student’s performance with continued training (Figure 9).

Original:  Leveraging prior data .
Revised: - Leveraging prior data .

Original: Learning from demonstrations (LfD) [4, 27, 32, 36, 61] approaches focuson accelerating RL training using demonstrations.
Revised: Learning from demonstrations (LfD) [5, 30, 36, 40, 69] approaches focus on accelerating RL training using demonstrations.

Original: Such approaches typically assume access tooptimal or near-optimal trajectories, often obtained from human demonstrators, and aim to match thedemonstrator’s performance.
Revised: Such approaches typically assume access to optimal or near-optimal trajectories, often obtained from human demonstrators, and aim to match the demonstrator’s performance.

Original: Instead, PVRL focuses on leveraging a suboptimal teacher policy, whichcan be obtained from any trained RL agent, that we wean off during training.
Revised: Instead, PVRL focuses on leveraging a suboptimal teacher policy, which can be obtained from any trained RL agent, that we wean off during training.

Original: Empirically, we ﬁndthat DQfD [32], a well-known LfD approach to accelerate deep Q-learning, when applied to PVRL,exhibits severe performance degradation when weaning off the teacher.
Revised: Empirically, we ﬁnd that DQfD [36], a well-known LfD approach to accelerate deep Q-learning, when applied to PVRL, exhibits severe performance degradation when weaning off the teacher.

Original: Rehearsal approaches [53,57, 65] focus on improving exploration by replaying demonstrations during learning; we ﬁnd thatsuch approaches are ineffective for leveraging the teacher in PVRL.
Revised: Rehearsal approaches [60, 64, 73] focus on improving exploration by replaying demonstrations during learning; we ﬁnd that such approaches are ineffective for leveraging the teacher in PVRL.

Original: Ofﬂine RL [1, 44, 46] focuses onlearning solely from ﬁxed datasets while reincarnating RL focuses on leveraging prior information,which can also be presented as ofﬂine datasets, for speeding up further learning from environmentinteractions.
Revised: Ofﬂine RL [1, 49, 51] focuses on learning solely from ﬁxed datasets while reincarnating RL focuses on leveraging prior information, which can also be presented as ofﬂine datasets, for speeding up further learning from environment interactions.

Original: Recent work [40, 45, 48, 54] use ofﬂine RL to pretrain on prior data and then ﬁne-tuneonline.
Revised: Recent work [45, 50, 54, 61] use ofﬂine RL to pretrain on prior data and then ﬁne-tune online.

Original: We also evaluate this pretraining approach for PVRL and ﬁnd that it underperforms QDagger,which utilizes the interactive teacher policy in addition to the prior teacher collected data.
Revised: We also evaluate this pretraining approach for PVRL and ﬁnd that it underperforms QDagger, which utilizes the interactive teacher policy in addition to the prior teacher collected data.

Original:  Case Study: Policy to Value Reincarnating RL
Revised:  Case Study: Policy to Value Reincarnating RL

Original: While prior large-scale efforts have used a limited form of reincarnating RL (Section 3), it is unclearhow to design more broadly applicable RRL approaches.
Revised: While prior large-scale efforts have used a limited form of reincarnating RL (Section 3), it is unclear how to design more broadly applicable RRL approaches.

Original: To exemplify the challenges of designingsuch approaches, we focus on the RRL setting for accelerating training of a student agent givenaccess to a suboptimal teacher policy and some data from it.
Revised: To exemplify the challenges of designing such approaches, we focus on the RRL setting for accelerating training of a student agent given access to a suboptimal teacher policy and some data from it.

Original: While a policy-based student can beeasily reincarnated in this setting via behavior cloning [ e.g., 3, 51], we study the more challengingpolicy-to-value reincarnating RL ( PVRL ) setting for transferring a policy to a value-based studentagent.
Revised: While a policy-based student can be easily reincarnated in this setting via behavior cloning [ e.g., 4], we study the more challenging policy-to-value reincarnating RL ( PVRL ) setting for transferring a policy to a value-based student agent.

Original: While we can obtain a policy from any RL agent, we chose this setting because value-based
Revised: While we can obtain a policy from any RL agent, we chose this setting because value-based

Original: RL methods (Q-learning, actor-critic) can leverage off-policy data for better sample efﬁciency.
Revised: RL methods (Q-learning, actor-critic) can leverage off-policy data for better sample efﬁciency.

Original: To bebroadly useful for reincarnating agents, a PVRL algorithm should satisfy the following desiderata:
Revised: To be broadly useful for reincarnating agents, a PVRL algorithm should satisfy the following desiderata:

Original: • Teacher-agnostic .
Revised: • Teacher-agnostic .

Original: Reincarnating RL has limited utility if the student is constrained by the teacher’s architecture or learning algorithm.
Revised: Reincarnating RL has limited utility if the student is constrained by the teacher’s architecture or learning algorithm.

Original: Thus, we require the student to be teacher-agnostic.
Revised: Thus, we require the student to be teacher-agnostic.

Original: • Weaning .
Revised: • Weaning .

Original: It is undesirable to maintain dependency on past teachers when reincarnation may occur several times over the course of a project, or one project to another.
Revised: It is undesirable to maintain dependency on past teachers when reincarnation may occur several times over the course of a project, or one project to another.

Original: Thus, it is necessary
Revised: Thus

Original:  that the student’s dependence on the teacher policy can be weaned off, as training progresses.
Revised: , it is necessary that the student’s dependence on the teacher policy can be weaned off, as training progresses.

Original: • Sample-efﬁcient .
Revised: • Sample-efﬁcient .

Original: Naturally, reincarnating RL is only useful if it is computationally cheaper than training from scratch.
Revised: Naturally, reincarnating RL is only useful if it is computationally cheaper than training from scratch.

Original: Thus, it is desirable that the student can recover and possibly improve upon the teacher’s performance using fewer environment samples than training tabula rasa.
Revised: Thus, it is desirable that the student can recover and possibly improve upon the teacher’s performance using fewer environment samples than training tabula rasa.

Original: PVRL on Atari 2600 games .
Revised: PVRL on Atari 2600 games .

Original: Given the above desiderata for PVRL, we now empirically investigatewhether existing methods that leverage existing data or agents (see Section 3) sufﬁce for PVRL.
Revised: Given the above desiderata for PVRL, we now empirically investigate whether existing methods that leverage existing data or agents (see Section 3) sufﬁce for PVRL.

Original: The speciﬁc methods that we consider were chosen because they are simple to implement, and alsobecause they have been designed with closely related goals in mind.
Revised: The speciﬁc methods that we consider were chosen because they are simple to implement, and also because they have been designed with closely related goals in mind.

Original: Experimental setup .
Revised: Experimental setup .

Original: We conduct experiments on ALE with sticky actions [50].
Revised: We conduct experiments on ALE with sticky actions [56].

Original: To reduce thecomputational cost of our experiments, we use a subset of 10 commonly-used Atari 2600 games:
Revised: To reduce the computational cost of our experiments, we use a subset of 10 commonly-used Atari 2600 games:

Original: Asterix, Breakout, Space Invaders, Seaquest, Q ∗ Bert, Beam Rider, Enduro, Ms Pacman, Bowlingand River Raid.
Revised: Asterix, Breakout, Space Invaders, Seaquest, Q ∗ Bert, Beam Rider, Enduro, Ms Pacman, Bowling and River Raid.

Original: We obtain the teacher policy π T by running DQN [52] with Adam optimizer for400 million environment frames, requiring 7 days of training per run with Dopamine [13] on [-P
-] GPUs.
Revised: We obtain the teacher policy π T by running DQN [58] with Adam optimizer formillion environment frames, requiring 7 days of training per run with Dopamine [15] on [-P
-]P100 GPUs.

Original: We also assume access to a dataset D T that can be generated by the teacher (see Appendix A.7for results about dependence on D T ).
Revised: We also assume access to a dataset D T that can be generated by the teacher (see Appendix A.5 for results about dependence on D T ).

Original: For this work, D T is the ﬁnal replay buffer (1M transitions) 
Revised: For this work, D T is the ﬁnal replay buffer (1M transitions) logged

Original: logged by the teacher DQN, which is 100 times smaller than the data the teacher was trained on.
Revised: [Figure] by the teacher DQN, which is 100 times smaller than the data the teacher was trained on.

Original: For a challenging PVRL setting, we use DQN as the student since tabula rasa DQN requires a substantial amount of training to reach the teacher’s performance.
Revised: For a challenging PVRL setting, we use DQN as the student since tabula rasa DQN requires a substantial amount of training to reach the teacher’s performance.

Original: To emphasize sample-efﬁcient reincarnation,we train this student for only 10 million frames, a 40 times smaller sample budget than the teacher.
Revised: To emphasize sample-efﬁcient reincarnation, we train this student for only 10 million frames, a 40 times smaller sample budget than the teacher.

Original: Furthermore, we wean off the teacher at 6 million frames.
Revised: Furthermore, we wean off the teacher at 6 million frames.

Original: See Appendix A.4 for more details.
Revised: See Appendix A.3 for more details.

Original: Evaluation 
Revised: Evaluation .

Original: For reliable evaluation, we follow the recommendations of Agarwal et al.
Revised: For reliable evaluation, we follow the recommendations of Agarwal et al.

Original: [2] and reportthe Interquartile Mean (IQM) normalized scores with 95% bootstrap conﬁdence intervals (CIs) aggregated across 10 Atari games with 3 seeds each The
Revised: [2] and report the Interquartile Mean (IQM) normalized scores with 95% bootstrap conﬁdence intervals (CIs), aggregated across 10 Atari games with 3 seeds each.

Original:  normalization is done such that the random policy corresponds to a score of 0 and the teacher policy π T corresponds to a score of 1.
Revised: The normalization is done such that the random policy corresponds to a score of 0 and the teacher policy π T corresponds to a score of 1.

Original: This

Original:  differs from typically reported human-normalized scores, as we wanted to highlight the performance differences between the student and the teacher
Revised: This differs from typically reported human-normalized scores, as we wanted to highlight the performance differences between the student and the teacher.

Original: Next, we describe the approaches we investigate
Revised: Next, we describe the approaches we investigate.

Original: • Rehearsal : Since the student, in principle, can learn using any off-policy data, we can replay teacher data D T along with the student’s own interactions during training.
Revised: • Rehearsal : Since the student, in principle, can learn using any off-policy data, we can replay teacher data D T along with the student’s own interactions during training.

Original: Following Paine et al.
Revised: Following Paine et al.

Original: QDagger

Original: [Figure]

Original: z [Figure] [57] , the student minimizes the TD loss on mini-batches that contain ρ % of the samples from D T and the rest from the student’s replay D S (different ρ and n -step values in Figure A.12)
Revised: [64], the student minimizes the TD loss on mini-batches that contain ρ % of the samples from D T and the rest from the student’s replay D S (different ρ and n -step values in Figure A.12).

Original: • JSRL (Figure 3, left): JSRL [71] uses an interactive teacher policy as a “guide” to improveexploration and rolls in with the guide for a random number of environment steps.
Revised: • JSRL (Figure 3, left): JSRL [80] uses an interactive teacher policy as a “guide” to improve exploration and rolls in with the guide for a random number of environment steps.

Original: To evaluateJSRL, we vary the maximum number of roll-in steps, α , that can be taken by the teacher andsample a random number of roll-in steps between [0 , α ] every episode.
Revised: To evaluate JSRL, we vary the maximum number of roll-in steps, α , that can be taken by the teacher and sample a random number of roll-in steps between [0 , α ] every episode.

Original: As the student improves,we decay the steps taken by the teacher every iteration (1M frames) by a factor of β .• RL Pretraining :
Revised: As the student improves, we decay the steps taken by the teacher every iteration (1M frames) by a factor of β . • RL Pretraining :

Original: Given access to teacher data D T , we can pre-train the student using ofﬂineRL.
Revised: Given access to teacher data D T , we can pre-train the student using ofﬂine RL.

Original: To do so, we use CQL [41], a widely used ofﬂine RL algorithm, which jointly minimizesthe TD and behavior cloning on logged transitions in D T (Equation A.3).
Revised: To do so, we use CQL [46], a widely used ofﬂine RL algorithm, which jointly minimizes the TD and behavior cloning on logged transitions in D T (Equation A.3).

Original: Following pretraining,we ﬁne-tune the learned Q-network using TD loss on the student’s replay D S .•
Revised: Following pretraining, we ﬁne-tune the learned Q-network using TD loss on the student’s replay D S .

Original:  Kickstarting (Figure 3, right):
Revised: • Kickstarting (Figure 3, right):

Original: Akin to kickstarting [63], we jointly optimize the TD loss with anon-policy distillation loss on the student’s self-collected data in D S .
Revised: Akin to kickstarting [71], we jointly optimize the TD loss with an on-policy distillation loss on the student’s self-collected data in D S .

Original: The distillation loss uses thecross-entropy between teacher’s policy π T and the student policy π ( ·| s ) =
Revised: The distillation loss uses the cross-entropy between teacher’s policy π T and the student policy π ( ·| s ) =

Original: softmax( Q ( s, · ) /τ ) ,where τ corresponds to temperature.
Revised: softmax( Q ( s, · ) /τ ) , where τ corresponds to temperature.

Original: To wean off the teacher, we decay the distillation coefﬁcientas training progresses.
Revised: To wean off the teacher, we decay the distillation coefﬁcient as training progresses.

Original: Note that kickstarting does not pretrain on teacher data.
Revised: Note that kickstarting does not pretrain on teacher data.

Original: • DQfD (Figure 4, left): Following DQfD [31], we initially pretrain the student on teacher dataD T using a combination of TD loss with a large margin classiﬁcation loss to imitate the teacheractions (Equation A.4).
Revised: • DQfD (Figure 4, left): Following DQfD [35], we initially pretrain the student on teacher data D T using a combination of TD loss with a large margin classiﬁcation loss to imitate the teacher actions (Equation A.4).

Original: After pretraining, we train the student on its replay data D S , againusing a combination of TD and margin loss.
Revised: After pretraining, we train the student on its replay data D S , again using a combination of TD and margin loss.

Original: While DQfD minimizes the margin loss throughouttraining, we decay the margin loss coefﬁcient during the online phase, akin to kickstarting. 
Revised: While DQfD minimizes the margin loss throughout training, we decay the margin loss coefﬁcient during the online phase, akin to kickstarting. [Figure] [Figure]

Original: Results .
Revised: Results .

Original: Rehearsal, with best-performing teacher data ratio ( ρ = 1 / 16 ), is marginally better thantabula rasa DQN but signiﬁcantly underperforms the teacher (Figure 2, teal), which seems related tothe difﬁculty of standard value-based methods to learn from off-policy teacher data [56].
Revised: Rehearsal, with best-performing teacher data ratio ( ρ = 1 / 16 ), is marginally better than tabula rasa DQN but signiﬁcantly underperforms the teacher (Figure 2, teal), which seems related to the difﬁculty of standard value-based methods to learn from off-policy teacher data [63].

Original: JSRL doesnot improve performance compared to tabula rasa DQN and even hurts performance with a largenumber of teacher roll-in steps (Figure 3, left).
Revised: JSRL does not improve performance compared to tabula rasa DQN and even hurts performance with a large number of teacher roll-in steps (Figure 3, left).

Original: The ineffectiveness of JSRL on ALE is likely due tothe state-distribution mismatch between the student and the teacher, as the student may never visit thestates visited by the teacher and as a result, doesn’t learn to correct for its previous mistakes [14].
Revised: The ineffectiveness of JSRL on ALE is likely due to the state-distribution mismatch between the student and the teacher, as the student may never visit the states visited by the teacher and as a result, doesn’t learn to correct for its previous mistakes [16].

Original: Pretraining with ofﬂine RL on logged teacher data recovers around 50% of the teacher’s performanceand ﬁne-tuning this pretrained Q-function online marginally improves performance (Figure 2, pink).
Revised: Pretraining with ofﬂine RL on logged teacher data recovers around 50% of the teacher’s performance and ﬁne-tuning this pretrained Q-function online marginally improves performance (Figure 2, pink).

Original: However, ﬁne-tuning degrades performance with 1 -step returns, which is more pronounced withhigher values of CQL loss coefﬁcient (Figure A.13).
Revised: However, ﬁne-tuning degrades performance with 1 -step returns, which is more pronounced with higher values of CQL loss coefﬁcient (Figure A.13).

Original: We also ﬁnd that kickstarting exhibits performance degradation (Figure 3, right), which is severe with 1 -step returns, once we wean off the teacherpolicy.
Revised: We also ﬁnd that kickstarting exhibits performance degradation (Figure 3, right), which is severe with 1 -step returns, once we wean off the teacher policy.

Original: Akin to kickstarting, we again observe a severe performance collapse when weaning off thethe teacher dependence in DQfD (Figure 4, left), even when using n -step returns.
Revised: Akin to kickstarting, we again observe a severe performance collapse when weaning off the the teacher dependence in DQfD (Figure 4, left), even when using n -step returns.

Original: We hypothesizethat this performance degradation is caused by the inconsistency between Q-values trained using a[Figure] [Figure] combination of imitation learning and TD losses, as opposed to only minimizing the TD loss.
Revised: We hypothesize that this performance degradation is caused by the inconsistency between Q-values trained using a combination of imitation learning and TD losses, as opposed to only minimizing the TD loss.

Original: Wealso ﬁnd that using intermediate values of n -step returns, such as n = 3 (also used by Rainbow [31]),quickly recovers after the performance drop from weaning while larger n -step values impede learning,possibly due to stale target Q-values.
Revised: We also ﬁnd that using intermediate values of n -step returns, such as n = 3 (also used by Rainbow [35]), quickly recovers after the performance drop from weaning while larger n -step values impede learning, possibly due to stale target Q-values.

Original: These results reveal the sensitivity of prior methods in the
Revised: These results reveal the sensitivity of prior methods in the

Original: PVRL setting to speciﬁc hyperparameter choices ( n -step), indicating the need for developing stable
Revised: PVRL setting to speciﬁc hyperparameter choices ( n -step), indicating the need for developing stable

Original: PVRL methods that do not fail when weaning off the teacher.
Revised: PVRL methods that do not fail when weaning off the teacher.

Original: For practitioners, the takeaway is toconsider this hyperparameter sensitivity when weaning off the teacher for reincarnation.
Revised: For practitioners, the takeaway is to consider this hyperparameter sensitivity when weaning off the teacher for reincarnation.

Original: 4.1 QDagger: A simple PVRL baseline
Revised: 4.1 QDagger: A simple PVRL baseline

Original: To address the limitations of prior approaches, we propose QDagger, a simple method for PVRL thatcombines Dagger [60], an interactive imitation learning algorithm, with n -step Q-learning (Figure 4,right).
Revised: To address the limitations of prior approaches, we propose QDagger, a simple method for PVRL that combines Dagger [68], an interactive imitation learning algorithm, with n -step Q-learning (Figure 4, right).

Original: Speciﬁcally, we ﬁrst pre-train the student on teacher data D T by minimizing L QDagger ( D T ) ,which combines distillation loss with the TD loss, weighted by a constant λ .
Revised: Speciﬁcally, we ﬁrst pre-train the student on teacher data D T by minimizing L QDagger ( D T ) , which combines distillation loss with the TD loss, weighted by a constant λ .

Original: This pretraining phasehelps the student to mimic the teacher’s state distribution, akin to the behavior cloning phase in Dagger.
Revised: This pretraining phase helps the student to mimic the teacher’s state distribution, akin to the behavior cloning phase in Dagger.

Original: After pretraining, we minimize L QDagger ( D S ) on the student’s replay D S , akin to kickstarting, wherethe teacher “corrects” the mistakes on the states visited by the student.
Revised: After pretraining, we minimize L QDagger ( D S ) on the student’s replay D S , akin to kickstarting, where the teacher “corrects” the mistakes on the states visited by the student.

Original: As opposed to minimizingthe Dagger loss indeﬁnitely, QDagger decays the distillation loss coefﬁcient λ t ( λ 0 = λ ) as trainingprogresses, to satisfy the weaning desiderata for PVRL.
Revised: As opposed to minimizing the Dagger loss indeﬁnitely, QDagger decays the distillation loss coefﬁcient λ t ( λ 0 = λ ) as training progresses, to satisfy the weaning desiderata for PVRL.

Original: Weaning allows QDagger to deviate fromthe suboptimal teacher policy π T , as opposed to being perpetually constrained to stay close toπ T (Figure 9).
Revised: Weaning allows QDagger to deviate from the suboptimal teacher policy π T , as opposed to being perpetually constrained to stay close to π T (Figure 9).

Original: We ﬁnd that both decaying λ t linearly over training steps or using an afﬁne functionof the ratio of student and teacher performance worked well (Appendix A.4).
Revised: We ﬁnd that both decaying λ t linearly over training steps or using an afﬁne function of the ratio of student and teacher performance worked well (Appendix A.3).

Original: Assuming the studentpolicy π ( ·| s ) =
Revised: Assuming the student policy π ( ·| s ) =

Original: softmax( Q ( s, · ) /τ ) , the QDagger loss is given by:
Revised: softmax( Q ( s, · ) /τ ) , the QDagger loss is given by:

Original: [Equation]
Revised: [Equation]

Original: Figure 2 shows that QDagger outperforms prior methods and surpasses the teacher.
Revised: Figure 2 shows that QDagger outperforms prior methods and surpasses the teacher.

Original: We remark that
Revised: We remark that

Original: DQfD can be viewed as a QDagger ablation that uses a margin loss instead of a distillation loss, whilekickstarting as another ablation that does not pretrain on teacher data.
Revised: DQfD can be viewed as a QDagger ablation that uses a margin loss instead of a distillation loss, while kickstarting as another ablation that does not pretrain on teacher data.

Original: Equipped with QDagger, weshow how to incorporate PVRL into our workﬂow and demonstrate its beneﬁts over tabula rasa RL.[Figure] [Figure] R
Revised: Equipped with QDagger, we show how to incorporate PVRL into our workﬂow and demonstrate its beneﬁts over tabula rasa RL.

Original: [Figure]

Original: R e i n c a r n a t e d I Q N

Original: R e i n c a

Original: [Figure]

Original: n a

Original: [Figure]

Original: e

Original: [Figure]

Original: n e

Original: [Figure]

Original: u n e d

Original: P e

Original: [Figure]

Original: c

Original: [Figure]

Original:  Reincarnating RL as a research workﬂow
Revised:  Reincarnating RL as a research workﬂow

Original: Revisiting ALE .
Revised: Revisiting ALE .

Original: As Mnih et al.
Revised: As Mnih et al.

Original: [52]’s development of Nature DQN established the tabula rasa workﬂow on ALE, we demonstrate how iterating on ALE agents’ design can be signiﬁcantly acceleratedusing a reincarnating RL workﬂow, starting from Nature DQN, in Figure 1.
Revised: [58]’s development of Nature DQN established the tabula rasa workﬂow on ALE, we demonstrate how iterating on ALE agents’ design can be signiﬁcantly accelerated using a reincarnating RL workﬂow, starting from Nature DQN, in Figure 1.

Original: Although Nature DQNused RMSProp, Adam yields better performance than RMSProp [1, 55].
Revised: Although Nature DQN used RMSProp, Adam yields better performance than RMSProp [1, 62].

Original: While we can train another
Revised: While we can train another

Original: DQN agent from scratch with Adam, ﬁne-tuning Nature DQN with Adam and 3 -step returns, witha reduced learning rate ( Figure 7), matches the performance of this tabula rasa DQN trained for400M frames, using a 20 times smaller sample budget (Panel 2 in Figure 1).
Revised: DQN agent from scratch with Adam, ﬁne-tuning Nature DQN with Adam and 3 -step returns, with a reduced learning rate ( Figure 7), matches the performance of this tabula rasa DQN trained for 400M frames, using a 20 times smaller sample budget (Panel 2 in Figure 1).

Original: As such, on a P100 GPU,ﬁne-tuning only requires training for a few hours rather than a week needed for tabula rasa RL.
Revised: As such, on a P100 GPU, ﬁne-tuning only requires training for a few hours rather than a week needed for tabula rasa RL.

Original: Giventhis ﬁne-tuned DQN, ﬁne-tuning it further results in diminishing returns with additional frames dueto being constrained to use the 3-layer convolutional neural network (CNN) with the DQN algorithm.
Revised: Given this ﬁne-tuned DQN, ﬁne-tuning it further results in diminishing returns with additional frames due to being constrained to use the 3-layer convolutional neural network (CNN) with the DQN algorithm.

Original: Let us now consider how one might use a more general reincarnation approach to improve onﬁne-tuning, by leveraging architectural and algorithmic advances since DQN, without the samplecomplexity of training from scratch (Panel 3 in Figure 1).
Revised: Let us now consider how one might use a more general reincarnation approach to improve on ﬁne-tuning, by leveraging architectural and algorithmic advances since DQN, without the sample complexity of training from scratch (Panel 3 in Figure 1).

Original: Speciﬁcally, using QDagger to transfer theﬁne-tuned DQN, we reincarnate Impala-CNN Rainbow that combines Dopamine Rainbow [31], whichincorporates distributional RL [10], prioritized replay [62] and n -step returns, with an Impala-CNNarchitecture [24], a deep ResNet with 15 convolutional layers.
Revised: Speciﬁcally, using QDagger to transfer the ﬁne-tuned DQN, we reincarnate Impala-CNN Rainbow that combines Dopamine Rainbow [35], which incorporates distributional RL [12], prioritized replay [70] and n -step returns, with an Impala-CNN architecture [26], a deep ResNet with 15 convolutional layers.

Original: Tabula rasa Impala-CNN Rainbowoutperforms ﬁne-tuning DQN further within 25M frames.
Revised: Tabula rasa Impala-CNN Rainbow outperforms ﬁne-tuning DQN further within 25M frames.

Original: Reincarnated Impala-CNN Rainbowquickly outperforms its teacher policy within 5M frames and maintains superior performance over itstabula rasa counterpart throughout training for 50M frames.
Revised: Reincarnated Impala-CNN Rainbow quickly outperforms its teacher policy within 5M frames and maintains superior performance over its tabula rasa counterpart throughout training for 50M frames.

Original: To catch up with the performance of thisreincarnated agent’s performance, the tabula rasa Impala-CNN Rainbow requires additional trainingfor 50M frames (48 hours on a P100 GPU).
Revised: To catch up with the performance of this reincarnated agent’s performance, the tabula rasa Impala-CNN Rainbow requires additional training for 50M frames (48 hours on a P100 GPU).

Original: See Appendix A.5 for more training details.
Revised: See Appendix A.4 for more training details.

Original: Overall,these results indicate how past research on ALE could have been accelerated by incorporating areincarnating RL approach to designing agents, instead of always re-training agents from scratch.
Revised: Overall, these results indicate how past research on ALE could have been accelerated by incorporating a reincarnating RL approach to designing agents, instead of always re-training agents from scratch.

Original: Tackling a challenging control task .
Revised: Tackling a challenging control task .

Original: To show how reincarnating RL can enable faster experimentation, we apply PVRL on the humanoid:run locomotion task, one of the hardest control problemsin DMC [68] due to its large action space (21 degrees of freedom).
Revised: To show how reincarnating RL can enable faster experimentation, we apply PVRL on the humanoid:run locomotion task, one of the hardest control problems in DMC [77] due to its large action space (21 degrees of freedom).

Original: For this experiment, shown in
Revised: For this experiment, shown in

Original: Figure 5, we use actor-critic agents in Acme [33].
Revised: Figure 5, we use actor-critic agents in Acme [37].

Original: For the teacher policy, we use TD3 [26] trainedfor 10M environment steps and pick the best run.
Revised: For the teacher policy, we use TD3 [29] trained for 10M environment steps and pick the best run.

Original: We ﬁnd that ﬁne-tuning this TD3 agent degradesperformance after 15M environment steps (other learning rates in Appendix A.5), which may berelated to capacity loss in value-based RL with prolonged training [42, 49].
Revised: We ﬁnd that ﬁne-tuning this TD3 agent degrades performance after 15M environment steps (other learning rates in Appendix A.4), which may be related to capacity loss in value-based RL with prolonged training [47, 55].

Original: For reincarnation, weuse single-actor D4PG [6], a distributional RL variant of DDPG [47], with a larger policy and criticarchitecture than TD3.
Revised: For reincarnation, we use single-actor D4PG [8], a distributional RL variant of DDPG [53], with a larger policy and critic architecture than TD3.

Original: Reincarnated D4PG performs better than its tabula rasa counterpart for theﬁrst 10M environment interactions.
Revised: Reincarnated D4PG performs better than its tabula rasa counterpart for the ﬁrst 10M environment interactions.

Original: Both these agents converge to similar performance, which is[Figure] [Figure] likely a limitation of QDagger.
Revised: Both these agents converge to similar performance, which is likely a limitation of QDagger.

Original: This result also raises the question of whether better PVRL methodscan lead to reincarnated agents that outperform their tabula rasa counterpart throughout learning.
Revised: This result also raises the question of whether better PVRL methods can lead to reincarnated agents that outperform their tabula rasa counterpart throughout learning.

Original: Nevertheless, tabula rasa D4PG requires additional training for 10-12 hours on a V100 GPU to matchreincarnated D4PG’s performance, which might quickly add up to a substantial savings in computewhen running a large set of experiments ( e.g., architectural or hyperparameter sweeps).
Revised: Nevertheless, tabula rasa D4PG requires additional training for 10-12 hours on a V100 GPU to match reincarnated D4PG’s performance, which might quickly add up to a substantial savings in compute when running a large set of experiments ( e.g., architectural or hyperparameter sweeps).

Original: Balloon Learning Environment (BLE) [29].
Revised: Balloon Learning Environment (BLE) [33].

Original: One of the motivations of our work is to be able to usedeep RL in real-world tasks in a data and computationally efﬁcient manner.
Revised: One of the motivations of our work is to be able to use deep RL in real-world tasks in a data and computationally efﬁcient manner.

Original: To this end, the BLEprovides a high-ﬁdelity simulator for navigating stratospheric balloons using RL [9].
Revised: To this end, the BLE [Figure] [Figure] R e i n c a r n a t e d I Q N

Revised: R e

Revised: [Figure]

Revised: n c a

Revised: [Figure]

Revised: n a

Revised: [Figure]

Revised: e

Revised: [Figure]

Revised: n e

Revised: [Figure]

Revised: u n e d

Revised: P

Revised: [Figure]

Revised: c

Revised: [Figure]

Revised: provides a high-ﬁdelity simulator for navigating stratospheric balloons using RL [11].

Original: An agent in
Revised: An agent in

Original: BLE can choose from three actions to control the balloon: move up, down, or stay in place.
Revised: BLE can choose from three actions to control the balloon: move up, down, or stay in place.

Original: Theballoon can only move laterally by “surﬁng” the winds at its altitude; the winds change over timeand vary as the balloon changes position and altitude.
Revised: The balloon can only move laterally by “surﬁng” the winds at its altitude; the winds change over time and vary as the balloon changes position and altitude.

Original: Thus, the agent is interacting with a partiallyobservable and non-stationary system, rendering this environment quite challenging.
Revised: Thus, the agent is interacting with a partially observable and non-stationary system, rendering this environment quite challenging.

Original: For the teacher,we use the QR-DQN agent provided by BLE, called Perciatelli , trained using large-scale distributed
Revised: For the teacher, we use the QR-DQN agent provided by BLE, called Perciatelli , trained using large-scale distributed

Original: RL for 40 days on the production-level Loon simulator by Bellemare et al.
Revised: RL for 40 days on the production-level Loon simulator by Bellemare et al.

Original: [9] and further ﬁne-tunedin BLE.
Revised: [11] and further ﬁne-tuned in BLE.

Original: For our experiments, we train distributed RL agents using Acme with 64 actors for a budgetof 50,000 episodes on a single cloud TPU-v2, taking approximately 10-12 hours per run.
Revised: For our experiments, we train distributed RL agents using Acme with 64 actors for a budget of 50,000 episodes on a single cloud TPU-v2, taking approximately 10-12 hours per run.

Original: In Figure 6, we compare the ﬁnal performance of distributed agents trained tabula rasa (in pink),with reincarnation (in blue), and ﬁne-tuned (in yellow).
Revised: In Figure 6, we compare the ﬁnal performance of distributed agents trained tabula rasa (in pink), with reincarnation (in blue), and ﬁne-tuned (in yellow).

Original: We consider three agents, QR-DQN [21]with an MLP architecture (same as Perciatelli), IQN [20] with a Densenet architecture [35], and arecurrent agent R2D6 2 for addressing the partial observability in BLE.
Revised: We consider three agents, QR-DQN [23] with an MLP architecture (same as Perciatelli), IQN [22] with a Densenet architecture [39], and a recurrent agent R2D6 2 for addressing the partial observability in BLE.

Original: When trained tabula rasa, noneof these agents are able to match the teacher performance, with the teacher-lookalike QR-DQN agentperforming particularly poorly.
Revised: When trained tabula rasa, none of these agents are able to match the teacher performance, with the teacher-lookalike QR-DQN agent performing particularly poorly.

Original: As R2D6 and IQN have substantial architectural differences from theteacher, we utilize PVRL for transferring the teacher.
Revised: As R2D6 and IQN have substantial architectural differences from the teacher, we utilize PVRL for transferring the teacher.

Original: Reincarnation allows IQN to match and R2D6to surpass teacher, although both lag behind ﬁne-tuning the teacher.
Revised: Reincarnation allows IQN to match and R2D to surpass teacher, although both lag behind ﬁne-tuning the teacher.

Original: More details in Appendix A.5.2.
Revised: More details in Appendix A.4.2.

Original: When ﬁne-tuning, we are reloading the weights from Perciatelli, which was notably trained on abroader geographical region than BLE and whose training distribution can be considered a superset ofwhat is used by the other agents; this is likely the reason that ﬁne-tuning does remarkably well relativeto other agents in BLE.
Revised: When ﬁne-tuning, we are reloading the weights from Perciatelli, which was notably trained on a broader geographical region than BLE and whose training distribution can be considered a superset of what is used by the other agents; this is likely the reason that ﬁne-tuning does remarkably well relative to other agents in BLE.

Original: Efﬁciently transferring information in Perciatelli’s weights to another agentwithout the replay data from the Loon simulator presents an interesting challenge for future work.
Revised: Efﬁciently transferring information in Perciatelli’s weights to another agent without the replay data from the Loon simulator presents an interesting challenge for future work.

Original: Overall, the improved efﬁciency of reincarnating RL (ﬁne-tuning and PVRL) over tabula rasa RL, asevident on the BLE, could make deep RL more accessible to researchers without access to industrialscale resources as they can build upon prior computational work, such as model checkpoints, enablingthe possible reuse of months of prior computation ( e.g., Perciatelli).
Revised: Overall, the improved efﬁciency of reincarnating RL (ﬁne-tuning and PVRL) over tabula rasa RL, as evident on the BLE, could make deep RL more accessible to researchers without access to industrialscale resources as they can build upon prior computational work, such as model checkpoints, enabling the possible reuse of months of prior computation ( e.g., Perciatelli).

Original:  Considerations in Reincarnating RL
Revised:  Considerations in Reincarnating RL

Original: Reincarnation via ﬁne-tuning .
Revised: Reincarnation via ﬁne-tuning .

Original: Given access to model weights and replay of a value-based agent,a simple reincarnation strategy is to ﬁne-tune this agent.
Revised: Given access to model weights and replay of a value-based agent, a simple reincarnation strategy is to ﬁne-tune this agent.

Original: While naive ﬁne-tuning with the samelearning rate ( lr ) as the  original agent does not exhibit improvement, ﬁne-tuning  with a reduced lr, for only 1 million additional frames, results in 25% IQM improvement for DQN (Adam) and 50%
Revised: While naive ﬁne-tuning with the same learning rate ( lr ) as the nearly saturated original agent does not exhibit improvement, ﬁne-tuning [Figure] [Figure] with a reduced lr

Revised: [Figure]

Revised: for only 1 million additional frames

Original:  IQM improvement for Nature DQN trained with RMSProp (Figure 7).
Revised: [Figure] results in 25% IQM improvement for DQN (Adam) and 50% IQM improvement for Nature DQN trained with RMSProp (Figure 7).

Original: As reincarnating RL leveragesexisting computational work ( e.g., model checkpoints), it allows us to easily experiment with such hy-
Revised: As reincarnating RL leverages existing computational work ( e [Figure] g., model checkpoints)

Original: [Figure]
Revised: [Figure]

Revised: it allows us to easily experiment with such hyperparameter schedules

Original: [Figure]
Revised: [Figure]

Original: perparameter schedules, which can be expensive in the tabula rasa setting.
Revised: which can be expensive in the tabula rasa setting.

Original: Note that when ﬁne-tuning,one is forced to keep the same network architecture; in contrast, reincarnating RL grants ﬂexibility inarchitecture and algorithmic choices, which can surpass ﬁne-tuning performance (Figures 1 and 5).
Revised: Note that when ﬁne-tuning, one is forced to keep the same network architecture; in contrast, reincarnating RL grants ﬂexibility in architecture and algorithmic choices, which can surpass ﬁne-tuning performance (Figures 1 and 5).

Original: Difference with tabula rasa benchmarking .
Revised: Difference with tabula rasa benchmarking .

Original: Are student agents that are more data-efﬁcient whentrained from scratch also better for reincarnating RL?
Revised: Are student agents that are more data-efﬁcient when trained from scratch also better for reincarnating RL?

Original: In Figure 8, we answer this question in thenegative, indicating the possibility of developing better students for utilizing existing knowledge.
Revised: In Figure 8, we answer this question in the negative, indicating the possibility of developing better students for utilizing existing knowledge.

Original: Speciﬁcally, we compare Dopamine Rainbow [31] and DrQ [77], undertabula rasa and PVRL settings.
Revised: Speciﬁcally, we compare Dopamine Rainbow [35] and DrQ [87], under

Original: DrQ outperforms Rainbow in low-data regime when trained from scratch but underperforms Rainbowin the PVRL setting as well as when training longer from scratch.
Revised: DrQ outperforms Rainbow in low-data regime when trained from scratch but underperforms Rainbow in the PVRL setting as well as when training longer from scratch.

Original: Based on this, we speculate thatreincarnating RL comparisons might be more consistent with asymptotic tabula rasa comparisons.
Revised: Based on this, we speculate that reincarnating RL comparisons might be more consistent with asymptotic tabula rasa comparisons.

Original: Reincarnation vs. Distillation .
Revised: Reincarnation vs. Distillation .

Original: PVRL is different from imitation learning or imitation-regularized
Revised: PVRL is different from imitation learning or imitation-regularized

Original: RL as it focuses on using an existing policy only as a launchpad for further learning, as opposedto imitating or staying close to it.
Revised: RL as it focuses on using an existing policy only as a launchpad for further learning, as opposed to imitating or staying close to it.

Original: To contrast these settings, we run two ablations of QDagger forreincarnating Impala-CNN Rainbow given a DQN teacher policy: (1) Dagger [60], which only minimizes the on-policy distillation loss in QDagger, and (2) Dagger + QL, which uses a ﬁxed distillationloss coefﬁcient throughout training (as opposed to QDagger, which decays it; see Equation 2).
Revised: To contrast these settings, we run two ablations of QDagger for reincarnating Impala-CNN Rainbow given a DQN teacher policy: (1) Dagger [68], which only minimizes the on-policy distillation loss in QDagger, and (2) Dagger + QL, which uses a ﬁxed distillation loss coefﬁcient throughout training (as opposed to QDagger, which decays it; see Equation 2).

Original: Asshown in Figure 9, Dagger performs similarly to the teacher while Dagger + QL improves over theteacher but quickly saturates in performance.
Revised: As shown in Figure 9, Dagger performs similarly to the teacher while Dagger + QL improves over the teacher but quickly saturates in performance.

Original: On the contrary, QDagger substantially outperformsthese ablations and shows continual improvement with additional environment interactions.
Revised: On the contrary, QDagger substantially outperforms these ablations and shows continual improvement with additional environment interactions.

Original: Dependency on prior work .
Revised: Dependency on prior work .

Original: Findings in reincarnating RL are dependent on existing computationalwork ( e.g., teacher policies). This is similar to ML areas, such as NLP and computer vision, wherebuilding upon pretrained models is the dominant paradigm [ e.g., 16, 23, 30, 34].
Revised: While performance in reincarnating RL depend on prior computational work ( e.g., teacher policy in PVRL), this is analogous to how ﬁne-tuning results in NLP / computer vision depend on the pretrained models ( e.g., using BERT vs GPT-3).

Original: To investigate teacherdependence in PVRL, we reincarnate a ﬁxed student from three different DQN teachers (Figure 10).
Revised: To investigate teacher dependence in PVRL, we reincarnate a ﬁxed student from three different DQN teachers (Figure 10).

Original: As expected, we observe that a higher performing teacher results in a better performing student.
Revised: As expected, we observe that a higher performing teacher results in a better performing student.

Original: However, reincarnation from two policies  obtained from different agents, DQN(Adam) vs. a ﬁnetuned Nature DQN,that perform similarly results in different performance.
Revised: However, reincarnation from two policies with similar performance but obtained from different agents, DQN (Adam) vs. a ﬁne-tuned Nature DQN, results in different performance.

Original: This suggests that areincarnated student’s performance depends not only on the teacher’s performance but also on itsbehavior.
Revised: This suggests that a reincarnated student’s performance depends not only on the teacher’s performance but also on its behavior.

Original: Nevertheless, the ranking of PVRL algorithms remains consistent across these two teacherpolicies (Figure A.11).
Revised: Nevertheless, the ranking of PVRL algorithms remains consistent across these two teacher policies (Figure A.11).

Original: Please see Appendix A.2 for a broader discussion about reproducibility.
Revised: See Section 7 for a broader discussion about generalizability.

Revised:  Reproducibility, Comparisons and Generalizability in Reincarnating RL

Revised: Scientiﬁc Comparisons .

Revised: Fairly comparing reincarnation approaches entails using the exactly same computational work and workﬂow.

Original: Conclusion
Revised: For example, in the PVRL setting, the same teacher and data should be used when comparing different algorithms, as we do in Section 4. To enable this, it would be beneﬁcial if the researchers can release model checkpoints and the data generated (at least the ﬁnal replay buffers), in addition to open-source code for their trained RL agents. Indeed, to allow others to use the same reincarnation setup as our work, we have already open-sourced DQN (Adam) agent checkpoints and the ﬁnal replay buffer at gs://rl_checkpoints . [Figure]

Revised: [Figure]

Revised: Generalizability .

Revised: The generalizable ﬁndings in reincarnating RL would be about comparing algorithmic efﬁcacy given access to existing computational work on a task.

Revised: As such, the performance ranking of reincarnation algorithms is likely to remain consistent across different teachers.

Revised: In fact, we empirically veriﬁed this for the PVRL setting, where we ﬁnd that while using two different teacher policies, namely DQN(Adam) vs. a ﬁne-tuned Nature DQN, leads to different performance trends but the ranking of PVRL algorithms remain consistent: QDagger > Kickstarting >

Revised: Pretraining (see

Revised: Figure 2 and Figure A.11).

Revised: Practitioners can use the ﬁndings from reincarnating RL to try to improve on an existing deployed RL policy (as opposed to being restricted to running tabula rasa RL).

Revised: For example, this work developed QDagger using ALE and applied it to PVRL on other tasks with existing policies (Humanoid and BLE).

Revised: Reproducibility .

Original: This work shows that reincarnating RL is a more compute and data efﬁcient workﬂow than  tabula rasa
Revised: Reproducibility from scratch is challenging in RRL as it would require details of the generation of the prior computational work ( e.g., teacher policies), which may itself has been obtained via reincarnating RL. As reproducibility from scratch involves reproducing existing computational work, it could be more expensive than training tabula rasa, which beats the purpose of doing reincarnation. Furthermore, reproducibility from scratch is also difﬁcult in NLP and computed vision, where existing pretrained models ( e.g., , GPT-3) are rarely, if ever, reproduced / re-trained from scratch but almost always used as-is. Despite this difﬁculty, pretraining-and-ﬁne-tuning is a dominant paradigm in NLP and vision [ e.g., 18, 25, 34, 38], and we believe that a similar difﬁculty in

Original: RL.
Revised: RRL should not prevent researchers from investigating and studying this important class of problems. Instead, we expect that RRL research would build on open-sourced prior computational work. Akin to NLP and vision, where typically a small set of pretrained models are used in research, we believe that research on developing better reincarnating RL methods can also possibly converge to a small set of open-sourced models / data on a given benchmark, e.g., the agents and data we released on Atari or the 25 , 000 trained Atari agents released by Gogianu et al. [31], concurrent to this work.

Revised:  Conclusion

Revised: Our work shows that reincarnating RL is a more compute and data efﬁcient research workﬂow than tabula rasa RL and can help further democratize RL research.

Original: Nevertheless, our results also open several avenues for future work.
Revised: Nevertheless, our results also open several avenues for future work.

Original: Particularly, more research isneeded for enabling workﬂows that can incorporate knowledge provided in a form other than a policy,such as pretrained models  or representations, developing better methods for PVRL, and extending 
Revised: Particularly, more research is needed for enabling workﬂows that can incorporate knowledge provided in a form other than a policy, such as pretrained models [41, 76], representations [85] or LLMs [3], developing better methods for PVRL, and extending PVRL to learn from multiple suboptimal teachers [48, 52].

Original: PVRL to transfer a policy to model-based agents.
Revised: Furthermore, we believe that reincarnating RL would be especially important for continued progress in building embodied agents in open-ended domains [7, 27, 32]. Aligned with the spirit of work, Raffel [67] have called for collaboratively building and continually improving large pre-trained models in NLP and vision.

Original: We hope that this work motivates RL researchers torelease computational work ( e.g., model checkpoints), which would allow others to directly build ontheir work.
Revised: We hope that this work motivates RL researchers to release computational work ( e.g., model checkpoints), which would allow others to directly build on their work.

Original: In this regards, we have open-sourced our code and trained agents with their ﬁnal replay.
Revised: In this regards, we have open-sourced our code and trained agents with their ﬁnal replay.

Original: Concurrent to this work, Gogianu et al.

Original: [28] released 25 , 000 trained Atari agents, which we believewould further facilitate reincarnating RL.

Original: As Newton put it “If I have seen further it is by standingon the shoulders of giants”, we argue that reincarnating RL can substantially accelerate progress bybuilding on prior computational work, as opposed to always redoing this work from scratch.
Revised: As Newton put it “If I have seen further it is by standing on the shoulders of giants”, we argue that reincarnating RL can substantially accelerate progress by building on prior computational work, as opposed to always redoing this work from scratch.

Original: Societal Impacts
Revised: Societal Impacts

Original: In this work we present reincarnating RL an alternative workﬂow for doing RL research.

Original: Thisworkﬂow could positively impact society by reducing the computational burden on researchers and ismore environment friendly than tabula rasa RL.
Revised: Reincarnating RL could positively impact society by reducing the computational burden on researchers and is more environment friendly than tabula rasa RL.

Original: For example, reincarnating RL allow researchers totrain super-human Atari agents on a single GPU within a span of few hours as opposed to trainingfor a few days.
Revised: For example, reincarnating RL allow researchers to train super-human Atari agents on a single GPU within a span of few hours as opposed to training for a few days.

Original: Additionally, reincarnating RL is more accessible to the wider research community,as researchers without sufﬁcient compute resources can build on prior computational work fromresource-rich groups and even improve upon them using limited resources.
Revised: Additionally, reincarnating RL is more accessible to the wider research community, as researchers without sufﬁcient compute resources can build on prior computational work from resource-rich groups, and even improve upon them using limited resources.

Original: Furthermore, thisdemocratization could directly improve RL applicability for practical applications, as most businessesthat could beneﬁt from RL often cannot afford the expertise to design in-house solutions.
Revised: Furthermore, this democratization could directly improve RL applicability for practical applications, as most businesses that could beneﬁt from RL often cannot afford the expertise to design in-house solutions.

Original: However,this democratization could also make it easier to apply RL for potentially harmful applications.
Revised: However, this democratization could also make it easier to apply RL for potentially harmful applications.

Original: Additionally, reincarnating RL could carry forward the bias or undesirable traits from the previouslylearned systems.
Revised: Furthermore, reincarnating RL could carry forward the bias or undesirable traits from the previously learned systems.

Original: As such we urge practitioners to be mindful of how RL ﬁts into the wider sociotechnical context of its deployment.
Revised: As such, we urge practitioners to be mindful of how RL ﬁts into the wider socio-technical context of its deployment.

Original: Checklist
Revised: Acknowledgments

Revised: We would like to thank David Ha, Evgenii Nikishin, Karol Hausman, Bobak Shahriari, Richard Song, Alex Irpan, Andrey Kurenkov for their valuable feedback on this work. We thank Joshua Greaves for helping us set up RL agents for BLE.

Revised: We also acknowledge Ted Xiao, Dale Schuurmans, Aleksandra

Revised: Faust, George Tucker, Rebecca Roelofs, Eugene Brevdo, Pierluca D’Oro, Nathan Rahn, Adrien Ali

Revised: Taiga, Bogdan Mazoure, Jacob Buckman, Georg Ostrovski and Aviral Kumar for useful discussions.

Original: For all authors...
Revised: Rishabh Agarwal led the project from start-to-ﬁnish, deﬁned the scope of the work to focus on policy to value reincarnation, came up with a successful algorithm for PVRL, and performed the literature survey. He designed, implemented and ran most of the experiments on ALE, Humanoid-run and BLE, and wrote the paper. Max Schwarzer helped run DQfD experiments on ALE and as well as setting up some agents for the BLE codebase with Acme, was involved in project discussions and edited the paper.

Original: (a) Do the main claims made in  the abstract and introduction accurately reﬂect the paper’s
Revised: Work done as a student researcher at Google. Pablo Samuel Castro was involved in project discussions, helped in setting up the BLE environment and implemented the initial Acme agents, and helped with paper editing. Aaron Courville advised the project, helped with project direction and provided feedback on writing. Marc Bellemare advised the project, challenged Rishabh to come up with an experimental paradigm in which one continuously improves on an existing agent, and provided feedback on writing.

Original:  contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Section 5 (BLE and continuous control results) and reproducibility and evaluation concerns in Appendix A.
Revised: [Yes] Section 5 (BLE and continuous control results) and reproducibility and evaluation concerns in Sec(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Ap- pendix A.
Revised: [Yes] agarwl.github.io/reincarnating_rl .

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Appendix A.4 and A. (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] See Appendix A.3 and A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] 95% CIs (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] 95% CIs (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix A. 
Revised: [Yes] See Appendix A. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] Apache License, Version 2.
Revised: [Yes] Apache License, Version 2.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A AppendixA.1 AlphaStar cost estimation
Revised: A Appendix A.1 AlphaStar cost estimation

Original: We estimate the cost of AlphaStar [73] based on the following description in the paper: “In StarCraft,each player chooses one of three races — Terran, Protoss or Zerg — each with distinct mechanics.
Revised: We estimate the cost of AlphaStar [82] based on the following description in the paper: “In StarCraft, each player chooses one of three races — Terran, Protoss or Zerg — each with distinct mechanics.

Original: We trained the league using three main agents (one for each StarCraft race), three main exploiteragents (one for each race), and six league exploiter agents (two for each race).
Revised: We trained the league using three main agents (one for each StarCraft race), three main exploiter agents (one for each race), and six league exploiter agents (two for each race).

Original: Each agent was trainedusing 32 third-generation tensor processing units (TPUv3) over 44 days.”
Revised: Each agent was trained using 32 third-generation tensor processing units (TPUv3) over 44 days.”

Original: This corresponds to a total of 12 agents (= 3 main + 3 exploiter + 6 league) trained for a totalof 1056 TPU hours = 44 days * 24 hours/day on 32 TPU v3 chips.
Revised: This corresponds to a total of 12 agents (= 3 main + 3 exploiter + 6 league) trained for a total of 1056 TPU hours = 44 days * 24 hours/day on 32 TPU v3 chips.

Original: As per current pricing athttps://cloud.google.com/tpu/pricing , TPUv3 (v3-8) cost $8 per hour.
Revised: As per current pricing at https://cloud.google.com/tpu/pricing , TPUv3 (v3-8) cost $8 per hour.

Original: Based on this,we estimate the cost of replicating AlphaStar results for an independent researcher would be at1056 ∗ 12 ∗ 32 ∗ $8 = $3 , 244 , 032 .
Revised: Based on this, we estimate the cost of replicating AlphaStar results for an independent researcher would be at 1056 ∗ 12 ∗ 32 ∗ $8 = $3 , 244 , 032 .

Original: Please note that we are only considering the cost of replicationand not accounting for other costs such as hyperparameter tuning and evaluation.
Revised: Please note that we are only considering the cost of replication and not accounting for other costs such as hyperparameter tuning and evaluation.

Original: A.2 Reproducibility, Comparisons and Generalizability in Reincarnating RL

Original: Scientiﬁc Comparisons .

Original: Fairly comparing reincarnation approaches entails using the exactly samecomputational work and workﬂow.

Original: For example, in the PVRL setting, the same teacher and datashould be used when comparing different algorithms, as we do in Section 4.

Original: To enable this, it wouldbe beneﬁcial if the researchers can release model checkpoints and the data generated (at least theﬁnal replay buffers), in addition to open-source code for their trained RL agents.

Original: Indeed, to allowothers to use the same reincarnation setup as our work, we have already open-sourced DQN (Adam)agent checkpoints and the ﬁnal replay buffer.

Original: Generalizability .

Original: While performance in reincarnating RL in depend on prior computational work( e.g., teacher policy in PVRL), this is analogous to how ﬁne-tuning results in NLP / computer visiondepend on the pretrained models ( e.g., , using BERT vs GPT-3).

Original: We believe that generalizableﬁndings in reincarnating RL would be about comparing algorithmic efﬁcacy given access to existingcomputational work ( e.g., policies) on a task.

Original: As such, the performance ranking of reincarnation algorithms is likely to remain consistent acrossdifferent teachers.

Original: In fact, we empirically veriﬁed this for the PVRL setting, where we ﬁnd that whileusing two different teacher policies, namely DQN(Adam) vs. a ﬁne-tuned Nature DQN, leads todifferent performance trends but the ranking of PVRL algorithms remain consistent: QDagger >

Original: Kickstarting > Pretraining (see Figure 2 and Figure A.11).

Original: Practitioners can use the ﬁndings fromreincarnating RL to try to improve on an existing deployed RL policy (as opposed to being restrictedto running tabula rasa RL).

Original: For example, this work shows that QDagger is effective for training a

Original: QDagger

Original: Kickstarting

Original: Pretraining

Original: [Figure]

Original: with

Original: (Panel 2 in Figure 1)

Original: Teacher Data Ratio ( ρ ) 0.

Original: I Q M

Original: N o r m a li

Original: z e d

Original: S c

Original: o r e

Original: n = n = n = n = value-based agent given access to an existing policy on ALE and QDagger can be applied to newtasks where we do have access to an existing policy ( e.g., BLE with the Perciatelli teacher).

Original: Reproducibility .

Original: We acknowledge that reproducibility from scratch is challenging in the reincarnating RL setting as it would require details of the generation of the prior computational work ( e.g.,teacher policies), which may itself has been obtained via reincarnating RL.

Original: As reproducibility fromscratch involves reproducing existing computational work, it could be more expensive than trainingtabula rasa, which beats the purpose of doing reincarnation.

Original: Furthermore, “reproducibility fromscratch” is also difﬁcult in NLP and computed vision, where existing pretrained models ( e.g., , GPT-3)are rarely, if ever, reproduced / re-trained from scratch but almost always used as-is.

Original: Despite thedifﬁculty of reproducibility from scratch, pretraining-and-ﬁne-tuning is a dominant paradigm in NLPand vision, and we believe that a similar challenge in reincarnating RL should not prevent researchersfrom investigating and studying this important class of problems.

Original: Instead, we expect that research in reincarnating RL would build on open-sourced prior computational work.

Original: Akin to NLP and computer vision, where typically a small set of pretrained models areused in research, we believe that research on developing better reincarnating RL methods can alsopossibly converge to a small set of open-sourced models / data on a given benchmark, for example,the teacher DQN agents and data we released on Atari.

Original: As research progresses in reincarnating RL,we believe that standardized benchmarks would emerge (one relevant example is the development ofofﬂine deep RL benchmarks).

Original: Open-source checkpoints, data and code .

Original: We provide our code for ALE experiments in thesupplementary.

Original: Since the model checkpoints and replay data for ALE require a large amount ofmemory, we have released them in the public GCP bucket gs://rl_checkpoints 3 , which can be easilydownloaded using gsutil.

Original: A.3 Compute resources for PVRL experiments
Revised: A.2 Compute resources for PVRL experiments

Original: For experiments in section 4, we used a P100 GPU.
Revised: For experiments in section 4, we used a P100 GPU.

Original: For obtaining the teacher policy, the cost ofrunning the tabula rasa DQN for 400M frames for 10 games on 3 seeds each roughly amounts to7 days x 30 = 210 days of compute on a P100 GPU.
Revised: For obtaining the teacher policy, the cost of running the tabula rasa DQN for 400M frames for 10 games on 3 seeds each roughly amounts to 7 days x 30 = 210 days of compute on a P100 GPU.

Original: For each of the PVRL methods, we trainedon 10 games with 3 runs each for 10M frames, which roughly translates to 4-5 hours.
Revised: For each of the PVRL methods, we trained on 10 games with 3 runs each for 10M frames, which roughly translates to 4-5 hours.

Original: For ofﬂinepretraining, we train methods for 1 million gradient steps with a batch size of 32, which roughlyamounts to 6-7 hours (this could be further sped up by using large-batch sizes).
Revised: For ofﬂine pretraining, we train methods for 1 million gradient steps with a batch size of 32, which roughly amounts to 6-7 hours (this could be further sped up by using large-batch sizes).

Original: We list the number ofconﬁgurations we evaluated for each method below.
Revised: We list the number of conﬁgurations we evaluated for each method below.

Original: • Rehearsal: We tried 5 values of teacher data ratio ρ × 4 n -step values, amounting to a totalof20 conﬁgurations.• JSRL: We tried 4 values of teacher roll-in steps α × 2 values of decay parameter β × 2 valuesof n -step, amounting to 16 conﬁgurations.n = n = n = n =   10 Steps (x 100k) 0. I Q M
Revised: • Rehearsal: We tried 5 values of teacher data ratio ρ × 4 n -step values, amounting to a total of 20 conﬁgurations.

Original: N o r m a li z e d S c o r e
Revised: • JSRL:

Original: Offline    10 Env.

Original: Frames (x 1M)

Original: Online

Original: CQL Coefficient = 0.

Original: 10 Steps (x 100k) 0. I Q M N o r m a li z e d
Revised: We tried 4 values of teacher roll-in steps α × 2 values of decay parameter β × 2 values of n -step, amounting to 16 conﬁgurations.

Original: S c

Original: o r e

Original: Offline    10 Env.

Original: Frames (x 1M) Online CQL Coefficient = 1. • RL Pretraining: We tried 2 values of λ × 4 values of n -step, amounting to 8 conﬁgurations.
Revised: • RL Pretraining: We tried 2 values of λ × 4 values of n -step, amounting to 8 conﬁgurations.

Original: • Kickstarting: We report results for 4 values of n -step for a speciﬁc temperature and distillation loss coefﬁcient.
Revised: • Kickstarting: We report results for 4 values of n -step for a speciﬁc temperature and distillation loss coefﬁcient.

Original: For hyperparameter tuning, we evaluated 2 temperature values and 2 losscoefﬁcients with a speciﬁc n -step.
Revised: For hyperparameter tuning, we evaluated 2 temperature values and 2 loss coefﬁcients with a speciﬁc n -step.

Original: Overall, this corresponds to a total of 8 conﬁgurations.
Revised: Overall, this corresponds to a total of 8 conﬁgurations.

Original: • DQfD: We report results for 4 values of n -step × 2 values of margin loss coefﬁcients × 2 valuesof margin parameters, amounting to 16 conﬁgurations.
Revised: • DQfD: We report results for 4 values of n -step × 2 values of margin loss coefﬁcients × 2 values of margin parameters, amounting to 16 conﬁgurations.

Original: • QDagger: We report results for 4 n -step values for a speciﬁc temperature and distillation loss coefﬁcient.
Revised: QDagger: We report results for 4 n -step values for a speciﬁc temperature and distillation loss coefﬁcient.

Original: For hyperparameter tuning, we evaluated 2 different temperature and loss coefﬁcients with a speciﬁc n -step, akin to kickstarting.
Revised: For hyperparameter tuning, we evaluated 2 different temperature and loss coefﬁcients with a speciﬁc n -step, akin to kickstarting.

Original: Overall, this amounts to 8 conﬁgurations.
Revised: Overall, this amounts to 8 conﬁgurations.

Original: Based on the above, we evaluated a total of 60 (= 20+16+16+8 ) conﬁgurations without pretrainingwhile 32 conﬁgurations with pretraining.
Revised: Based on the above, we evaluated a total of 60 (= 20+16+16+8 ) conﬁgurations without pretraining while 32 conﬁgurations with pretraining.

Original: Each of these conﬁgurations was evaluated for 30 seeds.
Revised: Each of these conﬁgurations was evaluated for 30 seeds.

Original: This amounts to a total compute time of 300-375 days for runs without pretraining on ofﬂine datawhile 400-480 days of GPU compute for runs involving pretraining, resulting in a total compute timeof around 700 - 855 days on a P100 GPU.
Revised: This amounts to a total compute time of 300-375 days for runs without pretraining on ofﬂine data while 400-480 days of GPU compute for runs involving pretraining, resulting in a total compute time of around 700 - 855 days on a P100 GPU.

Original: A.4 PVRL: Experimental details
Revised: A.3 PVRL: Experimental details

Original: Atari 2600 Games : The subset of games in the paper includes games from the original Atari trainingset used by Bellemare et al.
Revised: Atari 2600 Games : The subset of games in the paper includes games from the original Atari training set used by Bellemare et al.

Original: [8] (Asterix, Beam Rider, Seaquest and Space Invaders) as well asvalidation games used by Mnih et al.
Revised: [10] (Asterix, Beam Rider, Seaquest and Space Invaders) as well as validation games used by Mnih et al.

Original: [52] (Breakout, Enduro, River Raid), except the games whichare nearly solved by DQN, such as Freeway and Pong.
Revised: [58] (Breakout, Enduro, River Raid), except the games which are nearly solved by DQN, such as Freeway and Pong.

Original: We do not use any hard exploration games asa DQN teacher does not provide a meaningful teacher policy for such games.
Revised: We do not use any hard exploration games as a DQN teacher does not provide a meaningful teacher policy for such games.

Original: The remaining threegames were chosen to test the student agent in environments with challenging characteristics such asrequiring planning as opposed to being reactive ( e.g., Ms Pacman, Q ∗ Bert), and sparse-reward gamesthat require long-term predictions (Bowling).
Revised: The remaining three games were chosen to test the student agent in environments with challenging characteristics such as requiring planning as opposed to being reactive ( e.g., Ms Pacman, Q ∗ Bert), and sparse-reward games that require long-term predictions (Bowling).

Original: Furthermore, most of these games can be signiﬁcantlyimproved over the teacher DQN performance, which is sub-human on half of the games (Ms Pacman,
Revised: Furthermore, most of these games can be signiﬁcantly improved over the teacher DQN performance, which is sub-human on half of the games (Ms Pacman,

Original: Q ∗ Bert, Bowling, Seaquest and Beam Rider).
Revised: Q ∗ Bert, Bowling, Seaquest and Beam Rider).

Original: Refer to Table A.2 for per-game teacher scores.
Revised: Refer to Table A.2 for per-game teacher scores.

Revised: [Figure]

Revised: [Figure]

Original: Common hyperparameters .
Revised: Common hyperparameters .

Original: We list the hyperparameters shared by all PVRL methods in Table A.1.
Revised: We list the hyperparameters shared by all PVRL methods in Table A.1.

Original: For all methods, we swept over n -step returns in { 1 , 3 , 5 , 10 } except for DQfD [32], which originallyused 10 -step returns, where we only tried 3 -step and 10 -step returns and JSRL for which we tried only 1 -step and 3 -step returns.
Revised: For all methods, we swept over n -step returns in { 1 , 3 , 5 , 10 } except for DQfD [36], which originally used 10 -step returns, where we only tried 3 -step and 10 -step returns and JSRL for which we tried only 1 -step and 3 -step returns.

Original: For PVRL methods without any pretraining phase, we use a learning rate of 6 .
Revised: For PVRL methods without any pretraining phase, we use a learning rate of 6 .

Original: 25 e − 5 (JSRL, Rehearsal), following the hyperparameter conﬁguration for Dopamine [13].
Revised: 25 e − 5 (JSRL, Rehearsal), following the hyperparameter conﬁguration for Dopamine [15].

Original: For methods that pretrain on ofﬂine data, we sweep over { 6 .
Revised: For methods that pretrain on ofﬂine data, we sweep over { 6 .

Original: e − 5 , 1 e − 5 } and found the learning rate of 1 e − 5 to perform better in our early experimentation and use it for our main results.
Revised: e − 5 , 1 e − 5 } and found the learning rate of 1 e − 5 to perform better in our early experimentation and use it for our main results.

Original: We discuss the method speciﬁc hyperparameters below.
Revised: We discuss the method speciﬁc hyperparameters below.

Original: For obtaining the teacher policy using value-based agent, we use the softmax( Q T ( s, · ) /τ ) over the teacher’s Q-function Q T and use the same temperature coefﬁcient τ for both the student and teacher policy.
Revised: For obtaining the teacher policy using value-based agent, we use the softmax( Q T ( s, · ) /τ ) over the teacher’s Q-function Q T and use the same temperature coefﬁcient τ for both the student and teacher policy.

Original: • Rehearsal : We tried 5 different values of the teacher data ratio ( ρ ) in{ 0 , 1 / 256 , 1 / 64 , 1 / 16 , 1 / 4 } .
Revised: • Rehearsal : We tried 5 different values of the teacher data ratio ( ρ ) in { 0 , 1 / 256 , 1 / 64 , 1 / 16 , 1 / 4 } .

Original: The loss, L Rehearsal , can be written as: L Rehearsal =
Revised: The loss, L Rehearsal , can be written as: L Rehearsal =

Original: [Table]

Original: [Table]

Original: Q ∗ bert [Table] River Raid [Table]
Revised: ρ L TD ( D T )

Original: Seaquest

Original: [Table] Space Invaders [Table] ρ L TD ( D T ) + (1 − ρ )
Revised: + (1 − ρ )

Original: L TD ( D S ) 
Revised: L TD ( D S ) .

Original: As can be seen from the results in Figure A.12, we ﬁnd a small value of teacher data ratio ( ρ = 1 / 16 ) with 3 -step returns to be the best performing conﬁguration
Revised: As can be seen from the results in Figure A.12, we ﬁnd a small value of teacher data ratio ( ρ = 1 / 16 ) with 3 -step returns to be the best performing conﬁguration.

Original: • JSRL .
Revised: • JSRL .

Original: As shown in Figure 4 (left), we swept over the maximum number of teacher roll in steps of α in { 0 , 100 , 1000 , 5000 } , and the decay parameter β , which governs how fast we decay the roll-in steps, in { 0 . 8 , 1 . 0 } .
Revised: As shown in Figure 4 (left), we swept over the maximum number of teacher roll in steps of α in { 0 , 100 , 1000 , 5000 } , and the decay parameter β , which governs how fast we decay the roll-in steps, in { 0 . 8 , 1 . 0 } .

Original: Note that β = 1 . 

Revised: Note that β = 1 .

Original: corresponds to JSRL-Random, which was found to be competitive in performance to JSRL [71].
Revised: corresponds to JSRL-Random, which was found to be competitive in performance to JSRL [80].

Original: [Figure]
Revised: n = n = n = n =   10 Steps (x 100k) 0.

Original: I Q M
Revised: I Q M

Original: N o r m a li z e d S c
Revised: N o r m a li z e d S c

Original: o r e
Revised: o r e

Original: Tabula Rasa Impala-CNN Rainbow
Revised: Offline    10 Env.

Original: [Figure] [Figure]
Revised: Frames (x 1M)

Original: Online
Revised: Online

Revised: CQL Coefficient = 0.

Revised: 10 Steps (x 100k) 0.

Original: [Figure]
Revised: I Q M N o r m a li z e d S c

Original: Rainbow
Revised: o r e Offline    10 Env. Frames (x 1M) Online

Revised: CQL Coefficient = 1.

Revised: [Table]

Revised: -network: ﬁlter size

Original: [Figure]
Revised: [Table]

Revised: × 8 , 4 × 4 , 3 × 3 Q -network: stride

Revised: [Table]

Revised: [Table]

Revised: Q -network: hidden units

Revised: [Table]

Original: • RL Pretraining : We use CQL, which optimizes the following loss:
Revised: • RL Pretraining : We use CQL, which optimizes the following loss:

Original: [Equation]
Revised: [Equation]

Original: The choice of CQL is motivated by its simplicity as well as recent ﬁndings that ofﬂine RLmethods that do not estimate the behavior policy are more suited for online ﬁne-tuning [54].
Revised: The choice of CQL is motivated by its simplicity as well as recent ﬁndings that ofﬂine RL methods that do not estimate the behavior policy are more suited for online ﬁne-tuning [61].

Original: We tried two different values of CQL coefﬁcient λ ( 0 . 3 and 1 . 0 ), as shown in Figure A.13,and report the results for the better performing coefﬁcient ( λ = 0 . 3 ) in the main paper. 
Revised: We tried two different values of CQL coefﬁcient λ ( 0 . 3 and 1 . 0 ), as shown in Figure A.13, and report the results for the better performing coefﬁcient ( λ = 0 . 3 ) in the main paper. [Table]

Original: • Kickstarting .
Revised: • Kickstarting .

Original: Kickstarting [63] uses the same loss as QDagger (Equation 2), but as dis- cussed in the main paper, kickstarting does not have any pretraining phase on ofﬂine data.
Revised: Kickstarting [71] uses the same loss as QDagger (Equation 2), but as discussed in the main paper, kickstarting does not have any pretraining phase on ofﬂine data.

Original: For the temperature hyperparameter τ for obtaining the policy π ( ·| s ) =
Revised: For the temperature hyperparameter τ for obtaining the policy π ( ·| s ) =

Original: softmax( Q ( s, · ) /τ ) , we tried 0 .
Revised: softmax( Q ( s, · ) /τ ) , we tried 0 .

Original: 1 and 1 . 0 .
Revised: 1 and 1 . 0 .

Original: Similarly, we swept over two initial values for distillation loss coefﬁcient ( λ 0 ), namely 1 . 0 and 3 . 0 .
Revised: Similarly, we swept over two initial values for distillation loss coefﬁcient ( λ 0 ), namely 1 . 0 and 3 . 0 .

Original: For experiments with the DQN student, we found the coefﬁcient 3 . 0 and temperature 0 .
Revised: For experiments with the DQN student, we found the coefﬁcient 3 . 0 and temperature 0 .

Original: 1 to perform the best.
Revised: 1 to perform the best.

Original: • DQfD .
Revised: • DQfD .

Original: Following Hester et al.
Revised: Following Hester et al.

Original: [32], DQfD uses the following loss for training:
Revised: [36], DQfD uses the following loss for training:

Original: [Equation]
Revised: [Equation]

Original: where η t corresponds to the margin loss coefﬁcient, a T ( s ) = argmax a π T ( a | s ) and f ( a T , a ) is a margin function that is 0 when a = a T and a positive margin m otherwise.
Revised: where η t corresponds to the margin loss coefﬁcient, a T ( s ) = argmax a π T ( a | s ) and f ( a T , a ) is a margin function that is 0 when a = a T and a positive margin m otherwise.

Original: We swept over the values { 1 . 0 , 3 . 0 } for both the margin parameter m and initial values η 0 of the margin loss coefﬁcient η t .
Revised: We swept over the values { 1 . 0 , 3 . 0 } for both the margin parameter m and initial values η 0 of the margin loss coefﬁcient η t .

Original: For the DQN student, we report the results for the better performing margin coefﬁcients in the main paper.
Revised: For the DQN student, we report the results for the better performing margin coefﬁcients in the main paper.

Original: • QDagger .
Revised: • QDagger .

Original: Akin to kickstarting, we swept over the values of temperature τ in { 0 . 1 , 1 . 0 } and distillation coefﬁcient λ 0 in { 
Revised: Akin to kickstarting, we swept over the values of temperature τ in { 0 . 1 , 1 . 0 } and distillation coefﬁcient λ 0 in { 1 . 0 , 3 . 0 } .

Original: For ALE experiments, we decay the distillationloss coefﬁcient every training iteration (1M environment frames) using the fraction ofexpected returns obtained by the student policy π compared to the teacher policy π T , thatis, λ t = 1 t<t 0 max(1 − G π /G πT , 0) .
Revised: For ALE experiments, we decay the distillation loss coefﬁcient every training iteration (1M environment frames) using the fraction of expected returns obtained by the student policy π compared to the teacher policy π T , that is, λ t = 1 t<t 0 max(1 − G π /G πT , 0) .

Original: For fair comparisons with other methods, we use thesame strategy for decaying the distillation loss coefﬁcient λ t for Kickstarting and the marginloss coefﬁcient η t for DQfD.
Revised: For fair comparisons with other methods, we use the same strategy for decaying the distillation loss coefﬁcient λ t for Kickstarting and the margin loss coefﬁcient η t for DQfD.

Original: A.5 Reincarnating 
Revised: A.4 Reincarnating RL as a workﬂow: Additional details

Original: RL as a workﬂow Additional details

Original: Revisting ALE .
Revised: Revisting ALE .

Original: We used the ﬁnal model checkpoints of Nature DQN [52] from Agarwal et al.
Revised: We used the ﬁnal model checkpoints of Nature DQN [58] from Agarwal et al.

Original: [1],which was trained for 200M frames using the hyperparameters in Dopamine [13].
Revised: [1], which was trained for 200M frames using the hyperparameters in Dopamine [15].

Original: The ﬁne-tuned
Revised: The ﬁne-tuned

Original: DQN (Adam) in panel 2 in Figure 1 uses 3 -step returns.
Revised: DQN (Adam) in panel 2 in Figure 1 uses 3 -step returns.

Original: For the tabula rasa Impala-CNN Rainbow,we use similar hyperparameters to Dopamine Rainbow except for learning rate ( lr ), for which weran a sweep over { 1 e − 4 , 1 e − 5 , 3 e − 5 } , shown in Figure A.14, and use the best performing lrof 3 e − 5 .For the reincarnated Impala-CNN Rainbow in Panel 3, we use a QDagger distillation [Figure] [Figure]
Revised: For the tabula rasa Impala-CNN Rainbow, we use similar hyperparameters to Dopamine Rainbow except for learning rate ( lr ), for which we ran a sweep over { 1 e − 4 , 1 e − 5 , 3 e − 5 } , shown in Figure A.14, and use the best performing lr of 3 e − 5 .

Original: a l u a t i o n E p

Original: i s o d e r e t u r n

Original: [Figure] All the different learning rates exhibit similar perfor- mance trends including severe degradation after prolonged training. [Figure] coefﬁcient of 1 . 0 and sweep over temperature parameter τ in { 0 . 1 , 1 . 0 } , as shown in Figure A.15.
Revised: For the reincarnated Impala-CNN Rainbow in Panel 3, we use a QDagger distillation coefﬁcient of 1 . 0 and sweep over temperature parameter τ in { 0 . 1 , 1 . 0 } , as shown in Figure A.15.

Original: Consistent with our other ﬁne-tuning results on ALE, in Panel 3, using a reduced lr of 3 e − 6 forﬁne-tuning the already ﬁne-tuned DQN agent results in better performance, compared to using an lrof 1 e − 5 , as used by ﬁne-tuned DQN.
Revised: Consistent with our other ﬁne-tuning results on ALE, in Panel 3, using a reduced lr of 3 e − 6 for ﬁne-tuning the already ﬁne-tuned DQN agent results in better performance, compared to using an lr of 1 e − 5 , as used by ﬁne-tuned DQN.

Original: A.5.1 Humanoid:Run
Revised: A.4.1 Humanoid:Run

Original: The purpose of humanoid:run experiments is to show the utility of reincarnation in a complexcontinuous control environment given access to a pretrained policy and some data from its ﬁnal replaybuffer.
Revised: The purpose of humanoid:run experiments is to show the utility of reincarnation in a complex continuous control environment given access to a pretrained policy and some data from its ﬁnal replay [Figure] [Figure] [Figure]

Revised: I Q M

Revised: N o r m a li z e d S c

Original:  Irrespective of performance from ﬁne-tuning performance TD3, we demonstrate the beneﬁtsof using reincarnation over tabula rasa D4PG.
Revised: o r e [Figure] buffer. Irrespective of performance from ﬁne-tuning performance TD3, we demonstrate the beneﬁts of using reincarnation over tabula rasa D4PG.

Original: For example, reincarnated D4PG achieves performanceobtained by SAC in 10M frames 4 in only half the number of samples ( i.e., 5M frames).
Revised: For example, reincarnated D4PG achieves performance obtained by SAC in 10M frames 3 in only half the number of samples ( i.e., 5M frames).

Original: For obtaining the pretrained policy, TD3 was chosen as (1) it’s a well-known method for continuouscontrol, and (2) we could use an off-the-shelf JAX implementation in Acme, which is competitive onhumanoid-run.
Revised: For obtaining the pretrained policy, TD3 was chosen as (1) it’s a well-known method for continuous control, and (2) we could use an off-the-shelf JAX implementation in Acme, which is competitive on humanoid-run.

Original: For tabula rasa TD3 [26], we used a learning rate ( lr ) of 3 e − 4 for both the policyand critic, which are represented using a MLP with 2 hidden layers of size (256, 256).
Revised: For tabula rasa TD3 [29], we used a learning rate ( lr ) of 3 e − 4 for both the policy and critic, which are represented using a MLP with 2 hidden layers of size (256, 256).

Original: For otherhyperparameters, we used the default values in Acme’s TD3 implementation.
Revised: For other hyperparameters, we used the default values in Acme’s TD3 implementation.

Original: For ﬁne-tuning TD3, we use the last 500K environment steps from the TD3’s replay buffer and usethe same hyperparameters as tabula rasa TD3 except for lr .
Revised: For ﬁne-tuning TD3, we use the last 500K environment steps from the TD3’s replay buffer and use the same hyperparameters as tabula rasa TD3 except for lr .

Original: Speciﬁcally, we swept over the lr in{ 1 e − 4 , 3 e − 4 , 3 e − 05 } , shown in Figure A.17, and ﬁnd that all lr s exhibit performance degradationwith prolonged training.
Revised: Speciﬁcally, we swept over the lr in { 1 e − 4 , 3 e − 4 , 3 e − 05 } , shown in Figure A.17, and ﬁnd that all lr s exhibit performance degradation with prolonged training.

Original: We hypothesize that this degradation is likely caused by network capacity loss with prolonged training in value-based RL methods [42, 49].
Revised: We hypothesize that this degradation is likely caused by network capacity loss with prolonged training in value-based RL methods [47, 55].

Original: We believe that this issue does notaffect our conclusions about the efﬁcacy of reincarnating RL over a tabula rasa workﬂow.
Revised: We believe that this issue does not affect our conclusions about the efﬁcacy of reincarnating RL over a tabula rasa workﬂow.

Original: Furtherinvestigation of this performance degradation is outside the scope of the present work.
Revised: Further investigation of this performance degradation is outside the scope of the present work.

Original: For the tabula rasa D4PG [6], we use MLP networks with 3 hidden layers of size (256 , 256 , 256) forthe policy and (512 , 512 , 256) for the critic.
Revised: For the tabula rasa D4PG [8], we use MLP networks with 3 hidden layers of size (256 , 256 , 256) for the policy and (512 , 512 , 256) for the critic.

Original: We used 3 e − 4 as the learning rate and a sigma value of0 . 2 that sets the variance of the Gaussian noise to the behavior policy.
Revised: We used 3 e − 4 as the learning rate and a sigma value of 0 . 2 that sets the variance of the Gaussian noise to the behavior policy.

Original: Other hyperparameters usethe default values for D4PG implementation in Acme.
Revised: Other hyperparameters use the default values for D4PG implementation in Acme.

Original: For reincarnating D4PG using QDagger, weminimize a distillation loss between the D4PG’s actor policy and the teacher policy from TD3 jointlywith the actor-critic losses.
Revised: For reincarnating D4PG using QDagger, we [Figure] [Figure] minimize a distillation loss between the D4PG’s actor policy and the teacher policy from TD3 jointly with the actor-critic losses.

Original: For QDagger speciﬁc hyperparameters, we pretrain using 200K gradientupdates as well as decay λ t to 0 over a period of 200K gradient updates during the online trainingphase.
Revised: For QDagger speciﬁc hyperparameters, we pretrain using 200K gradient updates as well as decay λ t to 0 over a period of 200K gradient updates during the online training phase.

Original: Additionally, we sweep over distillation coefﬁcient λ 0 in { 0 . 1 , 0 . 3 } , as shown in Figure A.18.
Revised: Additionally, we sweep over distillation coefﬁcient λ 0 in { 0 . 1 , 0 . 3 } , as shown in Figure A.18.

Original: Note that both TD3 and D4PG use 5 -step returns by default in Acme.
Revised: Note that both TD3 and D4PG use 5 -step returns by default in Acme.

Original: A.5.2 Balloon Learning Environment (BLE)
Revised: A.4.2 Balloon Learning Environment (BLE)

Original: Details about BLE can be found in [29].
Revised: Details about BLE can be found in [33].

Original: For self-containedness, we include important details below.
Revised: For self-containedness, we include important details below.

Original: Station Keeping .
Revised: Station Keeping .

Original: Stratospheric balloons are ﬁlled with a buoyant gas that allows them to ﬂoat forweeks or months at a time in the stratosphere, about twice as high as a passenger plane’s cruisingaltitude.
Revised: Stratospheric balloons are ﬁlled with a buoyant gas that allows them to ﬂoat for weeks or months at a time in the stratosphere, about twice as high as a passenger plane’s cruising altitude.

Original: Though there are many potential variations of stratospheric balloons, the kind emulated inthe BLE are equipped with solar panels and batteries, which allow them to adjust their altitude bycontrolling the weight of air in their ballast using an electric pump.
Revised: Though there are many potential variations of stratospheric balloons, the kind emulated in the BLE are equipped with solar panels and batteries, which allow them to adjust their altitude by controlling the weight of air in their ballast using an electric pump.

Original: However, they have no means topropel themselves laterally, which means that they are subject to wind patterns in the air around them.
Revised: However, they have no means to propel themselves laterally, which means that they are subject to wind patterns in the air around them.

Original: By changing its altitude, a stratospheric balloon can surf winds moving in different directions.
Revised: By changing its altitude, a stratospheric balloon can surf winds moving in different directions.

Original: BLE Evaluation .
Revised: BLE Evaluation .

Original: The goal of a BLE agent is to station-keep, i.e., to control a balloon to stay within50 km of a ﬁxed ground station by changing its altitude to catch winds that it ﬁnds favorable.
Revised: The goal of a BLE agent is to station-keep, i.e., to control a balloon to stay within 50 km of a ﬁxed ground station by changing its altitude to catch winds that it ﬁnds favorable.

Original: Wemeasure how successful an agent is at station-keeping by measuring the fraction of time the balloonis within the speciﬁed radius, denoted TWR50 ( i.e., the time within a radius of 50km).
Revised: We measure how successful an agent is at station-keeping by measuring the fraction of time the balloon is within the speciﬁed radius, denoted TWR50 ( i.e., the time within a radius of 50km).

Original: Environment Details .
Revised: Environment Details .

Original: The observation space in BLE is a 1099 dimensional array of continuous and boolean values.
Revised: The observation space in BLE is a 1099 dimensional array of continuous and boolean values.

Original: An agent in BLE can choose from three actions to control the balloon:move up, down, or stay in place.
Revised: An agent in BLE can choose from three actions to control the balloon: move up, down, or stay in place.

Original: The balloon can only move laterally by “surﬁng” the windsat its altitude; the winds change over time and vary as the balloon changes position and altitude
Revised: The balloon can only move laterally by “surﬁng” the winds at its altitude; the winds change over time and vary as the balloon changes position and altitude

Original: More details can be found at https://balloon-learning-environment.readthedocs.io/en/latest/environment.html .
Revised: More details can be found at https://balloon-learning-environment.readthedocs.io/ en/latest/environment.html .

Original: Perciatelli Teacher and Data .
Revised: Perciatelli Teacher and Data .

Original: Perciatelli is the name of the RL agent (QR-DQN with a MLParchitecture) trained by Bellemare et al.
Revised: Perciatelli is the name of the RL agent (QR-DQN with a MLP architecture) trained by Bellemare et al.

Original: [9] using a distributed RL setup for more than a month inthe production-level Loon simulator and released with the BLE environment after ﬁne-tuning for14M steps in BLE.
Revised: [11] using a distributed RL setup for more than a month in the production-level Loon simulator and released with the BLE environment after ﬁne-tuning for 14M steps in BLE.

Original: For the data, we use the ﬁnal replay buffer (of size 2M transitions) logged by the
Revised: For the data, we use the ﬁnal replay buffer (of size 2M transitions) logged by the

Original: Perciatelli agent during BLE ﬁne-tuning.
Revised: Perciatelli agent during BLE ﬁne-tuning.

Original: Architectures .
Revised: Architectures .

Original: The network architecture used by QR-DQN [21] and Perciatelli is a multilayerperceptron (MLP) with 7 hidden layers of size 600 each with ReLU activations and approximate thedistribution using 51 ﬁxed quantiles.
Revised: The network architecture used by QR-DQN [23] and Perciatelli is a multilayer perceptron (MLP) with 7 hidden layers of size 600 each with ReLU activations and approximate the distribution using 51 ﬁxed quantiles.

Original: For the DenseNet architecture [35] employed by IQN [20] and
Revised: For the DenseNet architecture [39] employed by IQN [22] and

Original: R2D6, we use 7 hidden layers of size 512 each (for TPU-efﬁciency), which contains signiﬁcantlymore parameters than the Perciatelli MLP.
Revised: R2D6, we use 7 hidden layers of size 512 each (for TPU-efﬁciency), which contains signiﬁcantly more parameters than the Perciatelli MLP.

Original: Additionally, R2D6 uses a LSTM layer of size 512 on topof the DenseNet encoder with QR-DQN loss, while IQN samples 128 quantiles for minimizing theimplicit quantile regression loss. 
Revised: Additionally, R2D6 uses a LSTM layer of size 512 on top of the DenseNet encoder with QR-DQN loss, while IQN samples 128 quantiles for minimizing the implicit quantile regression loss. [Figure] [Figure] c t e

Revised: [Figure]

Revised: t u r n

Original:  Hyperparameters .
Revised: [Figure] Hyperparameters .

Original: We set most hyperparameters for our distributed RL agents based on conﬁguration of the BLE Quantile agent, which can be found at agents/conﬁgs/quantile.gin.
Revised: We set most hyperparameters for our distributed RL agents based on conﬁguration of the BLE Quantile agent, which can be found at agents/conﬁgs/quantile.gin.

Original: For tabula rasaagents, we swept over the lr in { 2 e − 6 , 6 e − 6 , 1 e − 5 } and found 1 e − 5 to be the best performinglr .
Revised: For tabula rasa agents, we swept over the lr in { 2 e − 6 , 6 e − 6 , 1 e − 5 } and found 1 e − 5 to be the best performing lr .

Original: However, for ﬁne-tuning Perciatelli, we found that a lower lr of 1 e − 6 performs better.
Revised: However, for ﬁne-tuning Perciatelli, we found that a lower lr of 1 e − 6 performs better.

Original: Forreincarnated R2D6 and IQN, we use a lr of 1 e − 5 and 6 e − 6 respectively during the online phasewhile 2 × lr in the ofﬂine pretraining phase.
Revised: For reincarnated R2D6 and IQN, we use a lr of 1 e − 5 and 6 e − 6 respectively during the online phase while 2 × lr in the ofﬂine pretraining phase.

Original: We set distillation temperature τ to be equal to 1 . 0 andlinearly decay the QDagger distillation coefﬁcient λ t over a ﬁxed number of learner steps (1M for
Revised: We set distillation temperature τ to be equal to 1 . 0 and linearly decay the QDagger distillation coefﬁcient λ t over a ﬁxed number of learner steps (1M for

Original: R2D6 and 160000 for IQN).
Revised: R2D6 and 160000 for IQN).

Original: Furthermore, we swept over the initial value of λ t ( λ 0 ) in { 0 . 3 , 1 . 0 } andfound 1.0 to be better for R2D6 while 0.3 for IQN.
Revised: Furthermore, we swept over the initial value of λ t ( λ 0 ) in { 0 . 3 , 1 . 0 } and found 1.0 to be better for R2D6 while 0.3 for IQN.

Original: A.6 Reincarnating RL problems

Original: While there is a lot of closely related work to reincarnating RL (section 3), e.g., imitation learning +

Original: RL, ofﬂine RL, transfer in RL, etc , such papers typically don’t focus on incorporating such methods asa part of our research workﬂow, that is, how we do RL research itself.

Original: As such, different reincarnating

Original: RL (RRL) problems can be instantiated depending on how the prior computational work generatedfrom RL agents is provided.

Original: We list some examples below:

Original: • Learned policies: In this RRL setting, we only assume access to a suboptimal teacherpolicy and want to transfer this policy to another student agent using online environmentinteractions.

Original: Policy sub-optimality and interaction budget are important factors in this setting

Original: [Table]

Original: Our experiments in the PVRL setting show that this setting is challenging when the student

Original: is a deep Q-learning agent.

Original: • Logged datasets: Here, we only assumes access to a ﬁxed dataset for pretraining an agentand ﬁne-tune this agent with online interactions.

Original: Data quality and interaction budget areimportant considerations in this setting.

Original: For example, ofﬂine RL studies this RRL setting inthe limit where no environment interactions are allowed.• Learned policies and datasets: In this setting, we assume access to both a teacher policyas well as some data ( e

Original: g., , PVRL setting studied in this work).

Original: Note that PVRL setting isalso applicable given access to only a teacher policy, as the teacher policy can be used forcollecting an ofﬂine dataset.

Original: Learned representations: This corresponds to the setting where we are given access to learnedrepresentations, which we can use for efﬁciently learning another agent.

Original: For example, wecould try to distill representations learned by an existing agent to reincarnate another agent.• Learned dynamics models:

Original: Given access to existing dynamics model, we can speed up the

Original: [Table]

Original: learning of another agent by using the model instead of the true environment for generating

Original: environment interactions.

Original: [Table]

Original: A .7 Additional ablations for QDagger
Revised: A.5 Additional ablations for QDagger

Original: Dependence on ofﬂine replay data .
Revised: Dependence on ofﬂine replay data .

Original: In the PVRL setting on ALE, we assumed access to the lastreplay buffer (containing 1M transitions) of the teacher DQN (Adam) agent trained for 400M frames.
Revised: In the PVRL setting on ALE, we assumed access to the last replay buffer (containing 1M transitions) of the teacher DQN (Adam) agent trained for 400M frames.

Original: This assumption is aligned with maximally reusing existing computational work and often holdsin practice as last replay buffer of RL agents is typically logged on disk in addition to their policycheckpoints.
Revised: This assumption is aligned with maximally reusing existing computational work and often holds in practice as last replay buffer of RL agents is typically logged on disk in addition to their policy checkpoints.

Original: However, it is possible that we only have access to the teacher policy.
Revised: However, it is possible that we only have access to the teacher policy.

Original: In this scenario,even if we don’t have access to the teacher’s replay buffer, we can generate data using the teacherpolicy during the online RL phase (this would cost environment samples) and use this teacher[Figure] collected data for pretraining with QDagger loss (analogous to the behavior cloning phase in Dagger).[Figure]
Revised: In this scenario, even if we don’t have access to the teacher’s replay buffer, we can generate data using the teacher policy during the online RL phase (this would cost environment samples) and use this teacher collected data for pretraining with QDagger loss (analogous to the behavior cloning phase in Dagger).

Original: To verify this empirically, we ran a QDagger ablation on 4 games where we do not assume accessto the teacher’s replay buffer.
Revised: To verify this empirically, we ran a QDagger ablation on 4 games where we do not assume access to the teacher’s replay buffer.

Original: As shown in Figure A.19, we found that collecting the same amountof teacher data as the ofﬂine replay leads to comparable performance on 3/4 games (Asterix, Qbert,
Revised: As shown in Figure A.19, we found that collecting the same amount of teacher data as the ofﬂine replay leads to comparable performance on 3/4 games (Asterix, Qbert,

Original: Seaquest).
Revised: Seaquest).

Original: Somewhat surprisingly, collecting only 500K transitions from teacher results in comparable performance to using 1M transitions, except on Asterix where we see substantially lower performance.
Revised: Somewhat surprisingly, collecting only 500K transitions from teacher results in comparable performance to using 1M transitions, except on Asterix where we see substantially lower performance.

Original: The slightly better performance with ofﬂine replay could be related to its better diversity than teachercollected data.
Revised: The slightly better performance with ofﬂine replay could be related to its better diversity than teacher collected data.

Original: While we used (cid:15) -greedy exploration with the teacher policy, the ofﬂine replay data iscollected using possibly diverse policies, as the teacher agent was continually updated during training.
Revised: While we used (cid:15) -greedy exploration with the teacher policy, the ofﬂine replay data is collected using possibly diverse policies, as the teacher agent was continually updated during training.

Original: Effect of QDagger loss in online phase .
Revised: Effect of QDagger loss in online phase .

Original: In this work, we use QDagger loss during both the ofﬂinepretraining and online RL phase (Equation 2).
Revised: In this work, we use QDagger loss during both the ofﬂine pretraining and online RL phase (Equation 2).

Original: To evaluate the beneﬁt of QDagger loss in the onlinephase on ALE, we ran an ablation on 4 games where we use standard Q-learning instead of QDaggerloss during the online RL phase.
Revised: To evaluate the beneﬁt of QDagger loss in the online phase on ALE, we ran an ablation on 4 games where we use standard Q-learning instead of QDagger loss during the online RL phase.

Original: As shown in Figure A.20, we see that the online phase helps improveperformance on some games ( e.g., Space Invaders, Asterix) while not having much effect on others.
Revised: As shown in Figure A.20, we see that the online phase helps improve performance on some games ( e.g., Space Invaders, Asterix) while not having much effect on others.

Original: The impact of the QDagger loss in the online phase also depends on the ofﬂine replay buffer size.
Revised: The impact of the QDagger loss in the online phase also depends on the ofﬂine replay buffer size.

Original: For example, on the more complex BLE domain, we only had access to a small amount of replaydata relative to the training budget of the Perciatelli teacher.
Revised: For example, on the more complex BLE domain, we only had access to a small amount of replay data relative to the training budget of the Perciatelli teacher.

Original: With this teacher, using the QDaggerloss only during the ofﬂine phase resulted in much lower performance than using QDagger for bothofﬂine and online phase.[Figure]
Revised: With this teacher, using the QDagger loss only during the ofﬂine phase resulted in much lower performance than using QDagger for both ofﬂine and online phase.