Original: Ofﬂine-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
Revised: Ofﬂine-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble

Original: Abstract: Recent advance in deep ofﬂine reinforcement learning (RL) has madeit possible to train strong robotic agents from ofﬂine datasets.
Revised: Abstract: Recent advance in deep ofﬂine reinforcement learning (RL) has made it possible to train strong robotic agents from ofﬂine datasets.

Original: However, depending on the quality of the trained agents and the application being considered, itis often desirable to ﬁne-tune such agents via further online interactions.
Revised: However, depending on the quality of the trained agents and the application being considered, it is often desirable to ﬁne-tune such agents via further online interactions.

Original: In thispaper, we observe that state-action distribution shift may lead to severe bootstraperror during ﬁne-tuning, which destroys the good initial policy obtained via ofﬂine
Revised: In this paper, we observe that state-action distribution shift may lead to severe bootstrap error during ﬁne-tuning, which destroys the good initial policy obtained via ofﬂine

Original: RL.
Revised: RL.

Original: To address this issue, we ﬁrst propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policysamples from the ofﬂine dataset.
Revised: To address this issue, we ﬁrst propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples from the ofﬂine dataset.

Original: Furthermore, we leverage multiple Q-functionstrained pessimistically ofﬂine, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase.
Revised: Furthermore, we leverage multiple Q-functions trained pessimistically ofﬂine, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase.

Original: We show that theproposed method improves sample-efﬁciency and ﬁnal performance of the ﬁnetuned robotic agents on various locomotion and manipulation tasks.
Revised: We show that the proposed method improves sample-efﬁciency and ﬁnal performance of the ﬁnetuned robotic agents on various locomotion and manipulation tasks.

Revised: Our code is available at: https://github.com/shlee94/Off2OnRL .

Original:  Introduction
Revised:  Introduction

Original: Deep ofﬂine reinforcement learning (RL) [1] has the potential to train strong robotic agents withoutany further environment interaction by leveraging deep neural networks and huge ofﬂine datasets.
Revised: Deep ofﬂine reinforcement learning (RL) [1] has the potential to train strong robotic agents without any further environment interaction by leveraging deep neural networks and huge ofﬂine datasets.

Original: Accordingly, the research community has demonstrated that ofﬂine RL can train both simulated [2,3, 4, 5, 6, 7, 8, 9] and real [5, 10] robots that are often more performant than the behavior policy thatgenerated the ofﬂine dataset.
Revised: Accordingly, the research community has demonstrated that ofﬂine RL can train both simulated [2, 3, 4, 5, 6, 7, 8, 9] and real [5, 10] robots that are often more performant than the behavior policy that generated the ofﬂine dataset.

Original: However, thusly trained ofﬂine RL agents may be suboptimal, for (a)the dataset they were trained on may be suboptimal; and (b) environment in which they are deployedmay be different from the environment in which the dataset was generated.
Revised: However, thusly trained ofﬂine RL agents may be suboptimal, for (a) the dataset they were trained on may be suboptimal; and (b) environment in which they are deployed may be different from the environment in which the dataset was generated.

Original: This necessitates anonline ﬁne-tuning procedure, where the robot improves by gathering additional samples
Revised: This necessitates an online ﬁne-tuning procedure, where the robot improves by gathering additional samples.

Original: Off-policy RL algorithms are well-suited for ofﬂine-to-online RL, since they can leverage both ofﬂine and online samples.
Revised: Off-policy RL algorithms are well-suited for ofﬂine-to-online RL, since they can leverage both ofﬂine and online samples.

Original: Fine-tuning an ofﬂine RL agent using a conventional off-policy RL algorithm, however, is difﬁcult due to distribution shift, i.e., the robot may encounter unfamiliar stateaction regime that is not covered by the ofﬂine dataset.
Revised: Fine-tuning an ofﬂine RL agent using a conventional off-policy RL algorithm, however, is difﬁcult due to distribution shift, i.e., the robot may encounter unfamiliar stateaction regime that is not covered by the ofﬂine dataset.

Original: The Q-function cannot provide accuratevalue estimates for such out-of-distribution (OOD) online samples, and updates with such sampleslead to severe bootstrap error.
Revised: The Q-function cannot provide accurate value estimates for such out-of-distribution (OOD) online samples, and updates with such samples lead to severe bootstrap error.

Original: This leads to policy updates in an arbitrary direction, destroying thegood initial policy obtained by ofﬂine RL.
Revised: This leads to policy updates in an arbitrary direction, destroying the good initial policy obtained by ofﬂine RL.

Original: To address state-action distribution shift, we ﬁrst introduce a balanced replay scheme that enablesus to provide the robotic agent with near-on-policy samples from the ofﬂine dataset, in addition tosamples gathered online.
Revised: To address state-action distribution shift, we ﬁrst introduce a balanced replay scheme that enables us to provide the robotic agent with near-on-policy samples from the ofﬂine dataset, in addition to samples gathered online.

Original: Speciﬁcally, we train a network that measures the online-ness of availablesamples, then prioritize samples according to this measure.
Revised: Speciﬁcally, we train a network that measures the online-ness of available samples, then prioritize samples according to this measure.

Original: This adjusts the sampling distributionfor Q-learning to be closer to online samples, which enables timely value propagation and moreaccurate policy evaluation in the novel state-action regime.
Revised: This adjusts the sampling distribution for Q-learning to be closer to online samples, which enables timely value propagation and more accurate policy evaluation in the novel state-action regime.

Original: However, we ﬁnd that the above sampling scheme is not enough, for the Q-function may be overoptimistic about unseen actions at novel online states.
Revised: However, we ﬁnd that the above sampling scheme is not enough, for the Q-function may be overoptimistic about unseen actions at novel online states.

Original: This misleads the robot to prefer potentiallybad actions, and in turn, more severe distribution shift and bootstrap error.
Revised: This misleads the robot to prefer potentially bad actions, and in turn, more severe distribution shift and bootstrap error.

Original: We therefore proposea pessimistic Q-ensemble scheme.
Revised: We therefore propose a pessimistic Q-ensemble scheme.

Original: In particular, we ﬁrst observe that a speciﬁc class of ofﬂine RL algorithms that train pessimistic Q-functions [8, 9] make an excellent starting point for ofﬂine-toonline RL.
Revised: In particular, we ﬁrst observe that a speciﬁc class of ofﬂine RL algorithms that train pessimistic Q-functions [8, 9] make an excellent starting point for ofﬂine-toonline RL.

Original: When trained as such, the Q-function implicitly constrains the policy to stay near thebehavior policy during the initial ﬁne-tuning phase.
Revised: When trained as such, the Q-function implicitly constrains the policy to stay near the behavior policy during the initial ﬁne-tuning phase.

Original: Building on this observation, we leverage multiple pessimistic Q-functions, which guides the robotic agent with a more high-resolution pessimismand stabilizes ﬁne-tuning.
Revised: Building on this observation, we leverage multiple pessimistic Q-functions, which guides the robotic agent with a more high-resolution pessimism and stabilizes ﬁne-tuning.

Original: In our experiments, we demonstrate the strength of our method based on (1) MuJoCo [11] locomotion tasks from the D4RL [12] benchmark suite, and (2) vision-based robotic manipulation tasksfrom Singh et al.
Revised: In our experiments, we demonstrate the strength of our method based on (1) MuJoCo [11] locomotion tasks from the D4RL [12] benchmark suite, and (2) vision-based robotic manipulation tasks from Singh et al.

Original: We show that our method achieves stable training during ﬁne-tuning, whileoutperforming all baseline methods considered, both in terms of ﬁnal performance and sampleefﬁciency.
Revised: We show that our method achieves stable training during ﬁne-tuning, while outperforming all baseline methods considered, both in terms of ﬁnal performance and sampleefﬁciency.

Original: We provide a thorough analysis of each component of our method.
Revised: We provide a thorough analysis of each component of our method.

Original:  Background
Revised:  Background

Original: Reinforcement learning.
Revised: Reinforcement learning.

Original: We consider the standard RL framework, where an agent interacts withthe environment so as to maximize the expected total return.
Revised: We consider the standard RL framework, where an agent interacts with the environment so as to maximize the expected total return.

Original: More formally, at each timestep t , theagent observes a state s t , and performs an action a t according to its policy π .
Revised: More formally, at each timestep t , the agent observes a state s t , and performs an action a t according to its policy π .

Original: The environment rewards the agent with r t , then transitions to the next state s t +1 .
Revised: The environment rewards the agent with r t , then transitions to the next state s t +1 .

Original: The agent’s objective is to maximizethe expected return E π {+[ (cid:80) ∞ t =
+}
Revised: The agent’s objective is to maximize the expected return E π {+[ (cid:80) ∞ t =
+}

Original: [ (cid:80) ∞ t =0 γ t r t ] , where γ ∈ [0 , 1) is the discount factor.
Revised: γ t r t ] , where γ ∈ [0 , 1) is the discount factor.

Original: The unnormalized stationary state-action distribution under π is deﬁned as d π ( s, a ) :
Revised: The unnormalized stationary state-action distribution under π is deﬁned as d π ( s, a ) :

Original: = (cid:80) ∞ t =

Revised: = (cid:80) ∞ t =

Original: γ t d πt ( s, a ) , where d πt ( s, a )denotes the state-action distribution at timestep t of the Markov chain deﬁned by the ﬁxed policy π .
Revised: γ t d πt ( s, a ) , where d πt ( s, a ) denotes the state-action distribution at timestep t of the Markov chain deﬁned by the ﬁxed policy π .

Original: Soft actor-critic.
Revised: Soft actor-critic.

Original: We mainly consider off-policy RL algorithms, a class of algorithms that can, inprinciple, train an agent with samples generated by any behavior policy.
Revised: We mainly consider off-policy RL algorithms, a class of algorithms that can, in principle, train an agent with samples generated by any behavior policy.

Original: In particular, soft actor-critic[SAC; 13] is an off-policy actor-critic algorithm that learns a soft Q-function Q θ ( s, a ) parameterizedby θ and a stochastic policy π φ modeled as a Gaussian, parameterized by φ .
Revised: In particular, soft actor-critic [SAC; 13] is an off-policy actor-critic algorithm that learns a soft Q-function Q θ ( s, a ) parameterized by θ and a stochastic policy π φ modeled as a Gaussian, parameterized by φ .

Original: SAC alternates betweencritic and actor updates by minimizing the following objectives, respectively:
Revised: SAC alternates between critic and actor updates by minimizing the following objectives, respectively:

Original: [Equation]
Revised: [Equation]

Original: where B is the replay buffer, ¯ θ the delayed parameters, and α the temperature parameter.
Revised: where B is the replay buffer, ¯ θ the delayed parameters, and α the temperature parameter.

Original: Conservative Q-learning.
Revised: Conservative Q-learning.

Original: Ofﬂine RL algorithms are off-policy RL algorithms that utilize staticdatasets for training an agent.
Revised: Ofﬂine RL algorithms are off-policy RL algorithms that utilize static datasets for training an agent.

Original: In particular, conservative Q-learning [CQL; 9] pessimistically evaluates the current policy, and learns a lower bound (in expectation) of the ground-truth Q-function.
Revised: In particular, conservative Q-learning [CQL; 9] pessimistically evaluates the current policy, and learns a lower bound (in expectation) of the ground-truth Q-function.

Original: Tobe speciﬁc, policy evaluation step of CQL minimizes the following:
Revised: To be speciﬁc, policy evaluation step of CQL minimizes the following:

Original: [Equation]
Revised: [Equation]

Original: where ˆ π 
Revised: where ˆ π β ( a 0 | s

Original: [Equation][Table]
Revised: [Equation]

Original: is the empirical behavior policy,
Revised: is the empirical behavior policy,

Original: α 

Revised: α

Original: the trade-off factor,and B π the bellman operator.
Revised: the trade-off factor, and B π the bellman operator.

Original: The ﬁrst term is the usual Bellman backup, and the second term is theregularization term that decreases the Q-values for unseen actions, while increasing the Q-values forseen actions.
Revised: The ﬁrst term is the usual Bellman backup, and the second term is the regularization term that decreases the Q-values for unseen actions, while increasing the Q-values for seen actions.

Original: We argue that thusly trained pessimistic Q-function is beneﬁcial for ﬁne-tuning as well(see Figure 1c).
Revised: We argue that thusly trained pessimistic Q-function is beneﬁcial for ﬁne-tuning as well (see Figure 1c).

Original: Policy improvement step is the same as SAC deﬁned in (2).
Revised: Policy improvement step is the same as SAC deﬁned in (2).

Original:  Fine-tuning Ofﬂine RL Agent
Revised:  Fine-tuning Ofﬂine RL Agent

Original: In this section, we investigate the distribution shift problem in ofﬂine-to-online RL.
Revised: In this section, we investigate the distribution shift problem in ofﬂine-to-online RL.

Original: We ﬁrst explain why an agent being ﬁne-tuned can be susceptible to distribution shift, and why distributionshift is problematic.
Revised: We ﬁrst explain why an agent being ﬁne-tuned can be susceptible to distribution shift, and why distribution shift is problematic.

Original: Then, we demonstrate two important design choices that decide the effect ofdistribution shift on ﬁne-tuning: sample selection and choice of ofﬂine Q-function . 
Revised: Then, we demonstrate two important design choices that decide the effect of distribution shift on ﬁne-tuning: sample selection and choice of ofﬂine Q-function . [Figure]

Original: 3.1 Distribution Shift in Ofﬂine-to-Online RL
Revised: 3.1 Distribution Shift in Ofﬂine-to-Online RL

Original: In ofﬂine-to-online RL, there exists a distribution shift between d on ( s, a ) and d off ( s, a ), wherethe former denotes the state-action distribution of online samples in the online buffer B on , and
Revised: In ofﬂine-to-online RL [Figure] there exists a distribution shift between d on ( s, a ) and d off ( s, a )

Original: [Figure]
Revised: [Figure]

Original:  the latter that of ofﬂine samples in the ofﬂine buffer B off .
Revised: where the former denotes the state-action distribution of online samples in the online buffer B on , and the latter that of ofﬂine samples in the ofﬂine buffer B off .

Original: Figure 1a visualizes such distributionshift.
Revised: Figure 1a visualizes such distribution shift.

Original: Speciﬁcally, we trained a variational autoencoder [14] to reconstruct state-action pairs in thehalfcheetah-random dataset that contains uniform random policy rollouts in the halfcheetah environment.
Revised: Speciﬁcally, we trained a variational autoencoder [14] to reconstruct state-action pairs in the halfcheetah-random dataset that contains uniform random policy rollouts in the halfcheetah environment.

Original: Then, we compared the log-likelihood of (a) ofﬂine samples and (b) online samplescollected by a CQL agent trained on the same dataset.
Revised: Then, we compared the log-likelihood of (a) ofﬂine samples and (b) online samples collected by a CQL agent trained on the same dataset.

Original: There is a clear difference between ofﬂineand online state-action distributions.
Revised: There is a clear difference between ofﬂine and online state-action distributions.

Original: Such distribution shift is problematic, for the agent will enter the unseen state-action regime, where
Revised: Such distribution shift is problematic, for the agent will enter the unseen state-action regime, where

Original: Q-values (hence value estimates used for bootstrapping) can be very inaccurate.
Revised: Q-values (hence value estimates used for bootstrapping) can be very inaccurate.

Original: Updates in suchunseen regime results in erroneous policy evaluation and arbitrary policy updates, which destroysthe good initial policy obtained via ofﬂine RL.
Revised: Updates in such unseen regime results in erroneous policy evaluation and arbitrary policy updates, which destroys the good initial policy obtained via ofﬂine RL.

Original: Distribution shift can be especially severe in ofﬂineto-online RL, for the ofﬂine RL agent is often much more performant than the behavior policy (e.g.,
Revised: Distribution shift can be especially severe in ofﬂineto-online RL, for the ofﬂine RL agent is often much more performant than the behavior policy (e.g.,

Original: CQL can train a medium-level agent capable of running, using transitions generated by a randompolicy only).
Revised: CQL can train a medium-level agent capable of running, using transitions generated by a random policy only).

Original: Also, when the ofﬂine dataset is narrowly distributed, e.g., when it is generated bya single policy, the agent is more prone to distribution shift, for the agent easily deviates from thenarrow, seen distribution.
Revised: Also, when the ofﬂine dataset is narrowly distributed, e.g., when it is generated by a single policy, the agent is more prone to distribution shift, for the agent easily deviates from the narrow, seen distribution.

Original: 3.2 Sample Selection
Revised: 3.2 Sample Selection

Original: In light of the above discussion, we study how sample selection affects ﬁne-tuning.
Revised: In light of the above discussion, we study how sample selection affects ﬁne-tuning.

Original: We ﬁnd thatonline samples, which are essential for ﬁne-tuning, are also potentially dangerous OOD samplesdue to distribution shift.
Revised: We ﬁnd that online samples, which are essential for ﬁne-tuning, are also potentially dangerous OOD samples due to distribution shift.

Original: Meanwhile, ofﬂine samples are in-distribution and safe, but leads toslow ﬁne-tuning.
Revised: Meanwhile, ofﬂine samples are in-distribution and safe, but leads to slow ﬁne-tuning.

Original: As a concept experiment, we trained an agent ofﬂine via CQL (3, 2) on thehalfcheetah-medium dataset containing medium-level transitions, then ﬁne-tuned the agent via
Revised: As a concept experiment, we trained an agent ofﬂine via CQL (3, 2) on the halfcheetah-medium dataset containing medium-level transitions, then ﬁne-tuned the agent via

Original: SAC (1, 2).
Revised: SAC (1, 2).

Original: We see that using online samples exclusively for updates (denoted Online only in
Revised: We see that using online samples exclusively for updates (denoted Online only in

Original: Figure 1b) leads to unstable ﬁne-tuning, where the average return drops from about 4500 to below 3000.
Revised: Figure 1b) leads to unstable ﬁne-tuning, where the average return drops from about 4500 to below 3000.

Original: This demonstrates the harmful effect of distribution shift, where novel, OOD samplescollected online cause severe bootstrap error.
Revised: This demonstrates the harmful effect of distribution shift, where novel, OOD samples collected online cause severe bootstrap error.

Original: On the other hand, when using a single replay buffer for both ofﬂine and online samples then sampling uniformly at random (denoted Uniform in Figure 1b), the agent does not use enough onlinesamples for updates, especially when the ofﬂine dataset is large.
Revised: On the other hand, when using a single replay buffer for both ofﬂine and online samples then sampling uniformly at random (denoted Uniform in Figure 1b), the agent does not use enough online samples for updates, especially when the ofﬂine dataset is large.

Original: As a result, value propagation isslow, and as seen in Figure 1b, this scheme achieves initial stability at the cost of asymptotic performance.
Revised: As a result, value propagation is slow, and as seen in Figure 1b, this scheme achieves initial stability at the cost of asymptotic performance.

Original: This motivates a balanced replay scheme that modulates the trade-off between using onlinesamples (useful, but potentially dangerous), and ofﬂine samples (stable, but slow ﬁne-tuning).
Revised: This motivates a balanced replay scheme that modulates the trade-off between using online samples (useful, but potentially dangerous), and ofﬂine samples (stable, but slow ﬁne-tuning).

Original: 3.3 Choice of Ofﬂine Q-function
Revised: 3.3 Choice of Ofﬂine Q-function

Original: Another important design choice in ofﬂine-to-online RL is the ofﬂine training of Q-function.
Revised: Another important design choice in ofﬂine-to-online RL is the ofﬂine training of Q-function.

Original: In particular, we show that a pessimistically trained Q-function mitigates the effect of distribution shift, bystaying conservative about OOD actions in the initial training phase.
Revised: In particular, we show that a pessimistically trained Q-function mitigates the effect of distribution shift, by staying conservative about OOD actions in the initial training phase.

Original: As a concept experiment, we compared the ﬁne-tuning performance when using a pessimistically trained Q-function and whenusing a Q-function trained without any pessimistic regularization.
Revised: As a concept experiment, we compared the ﬁne-tuning performance when using a pessimistically trained Q-function and when using a Q-function trained without any pessimistic regularization.

Original: Speciﬁcally, for a given ofﬂinedataset, we ﬁrst trained a policy π φ and its pessimistic Q-function Q θ CQL via CQL (3).
Revised: Speciﬁcally, for a given ofﬂine dataset, we ﬁrst trained a policy π φ and its pessimistic Q-function Q θ CQL via CQL (3).

Original: Then wetrained a non-pessimistic Q-function Q θ FQE of the pre-trained ofﬂine policy π φ via Fitted Q Evaluation [FQE; 15], an off-policy policy evaluation method that trains a given policy’s Q-function.
Revised: Then we trained a non-pessimistic Q-function Q θ FQE of the pre-trained ofﬂine policy π φ via Fitted Q Evaluation [FQE; 15], an off-policy policy evaluation method that trains a given policy’s Q-function.

Original: Finally, we ﬁne-tuned { π φ , Q θ CQL } and { π φ , Q θ FQE } via SAC (1, 2).
Revised: Finally, we ﬁne-tuned { π φ , Q θ CQL } and { π φ , Q θ FQE } via SAC (1, 2).

Original: See Section C for more details.
Revised: See Section C for more details.

Original: As shown in Figure 1c, both pessimistic and non-pessimistic Q-functions show similar ﬁne-tuningperformance on the random dataset, which contains random policy rollouts with good action spacecoverage.
Revised: As shown in Figure 1c, both pessimistic and non-pessimistic Q-functions show similar ﬁne-tuning performance on the random dataset, which contains random policy rollouts with good action space coverage.

Original: However, when ﬁne-tuning an ofﬂine RL agent trained on the medium-expert dataset,which contains transitions obtained by a mixture of more selective and performant policies, nonpessimistic Q-function loses the good initial policy, reaching zero average return at one point.
Revised: However, when ﬁne-tuning an ofﬂine RL agent trained on the medium-expert dataset, which contains transitions obtained by a mixture of more selective and performant policies, nonpessimistic Q-function loses the good initial policy, reaching zero average return at one point.

Original: The reason is that Q FQE can be overly optimistic about OOD actions at novel states when bootstrapping from them.
Revised: The reason is that Q FQE can be overly optimistic about OOD actions at novel states when bootstrapping from them.

Original: In turn, the policy may prefer potentially bad actions, straying further away fromthe safe, seen trajectory.
Revised: In turn, the policy may prefer potentially bad actions, straying further away from the safe, seen trajectory.

Original: On the other hand, Q CQL remains pessimistic in the states encountered online initially, for (1) these states are incrementally different from seen states, and (2) Q-function willthus have similar pessimistic estimates due to generalization.
Revised: On the other hand, Q CQL remains pessimistic in the states encountered online initially, for (1) these states are incrementally different from seen states, and (2) Q-function will thus have similar pessimistic estimates due to generalization.

Original: This points to a strategy where weﬁrst train a pessimistic Q-function ofﬂine, then let it gradually lose the pessimism as the agent gainsaccess to a balanced mix of ofﬂine and online samples via balanced replay during ﬁne-tuning.
Revised: This points to a strategy where we ﬁrst train a pessimistic Q-function ofﬂine, then let it gradually lose the pessimism as the agent gains access to a balanced mix of ofﬂine and online samples via balanced replay during ﬁne-tuning.

Original: Furthermore, since a single agent’s Q-function may not be pessimistic enough, we may train multipleagents ofﬂine in parallel, then deploy online the ensemble agent equipped with a higher-resolutionpessimism of the Q-ensemble (see Section 4.2 for a more detailed explanation).
Revised: Furthermore, since a single agent’s Q-function may not be pessimistic enough, we may train multiple agents ofﬂine in parallel, then deploy online the ensemble agent equipped with a higher-resolution pessimism of the Q-ensemble (see Section 4.2 for a more detailed explanation).

Original:  Method
Revised:  Method

Original: [Figure]
Revised: [Figure]

Revised: We propose a simple yet effective framework that addresses the stateaction distribution shift described in

Revised: Section 3.

Revised: Our method comprises of two parts: (a) a balanced experience replay scheme, and (b) a pessimistic Qensemble scheme.

Original: 4.1 Balanced Experience Replay
Revised: 4.1 Balanced Experience Replay

Original: We introduce a balanced replay schemethat enables us to safely utilize onlinesamples by leveraging relevant, nearon-policy ofﬂine samples.
Revised: We introduce a balanced replay scheme that enables us to safely utilize online samples by leveraging relevant, nearon-policy ofﬂine samples.

Original: By doing so,we can widen the sampling distributionfor updates around the on-policy samples and enable timely value propagation.
Revised: By doing so, we can widen the sampling distribution for updates around the on-policy samples and enable timely value propagation.

Original: The challenge here is how to design a scheme that locates and retrieves such relevant, near-onpolicy samples from the ofﬂine dataset, which can often be huge.
Revised: The challenge here is how to design a scheme that locates and retrieves such relevant, near-onpolicy samples from the ofﬂine dataset, which can often be huge.

Original: To achieve this, we measure theonline-ness of all available samples, and prioritize the samples according to this measure.
Revised: To achieve this, we measure the online-ness of all available samples, and prioritize the samples according to this measure.

Original: In particular, when updating the agent, we propose to sample a transition ( s, a, s (cid:48) )
Revised: In particular, when updating the agent, we propose to sample a transition ( s, a, s (cid:48) )

Original: ∈ B off ∪B on witha probability proportional to the density ratio w ( s, a ) :
Revised: ∈ B off ∪B on with a probability proportional to the density ratio w ( s, a ) :

Original: = d on ( s, a ) /d off ( s, a ) of the given sample.
Revised: = d on ( s, a ) /d off ( s, a ) of the given sample.

Original: This way, we can retrieve a relevant, near-on-policy sample ( s, a, s (cid:48) )
Revised: This way, we can retrieve a relevant, near-on-policy sample ( s, a, s (cid:48) )

Original: ∈ B off by locating a transitionwith high density ratio w ( s, a ) .
Revised: ∈ B off by locating a transition with high density ratio w ( s, a ) .

Original: However, estimating the likelihoods d off ( s, a ) and d on ( s, a ) isdifﬁcult, since they can in principle be stationary distributions of complex policy mixtures 1 .
Revised: However, estimating the likelihoods d off ( s, a ) and d on ( s, a ) is difﬁcult, since they can in principle be stationary distributions of complex policy mixtures 2 .

Original: To avoidthis problem, we utilize a likelihood-free density ratio estimation method that estimates w ( s, a ) bytraining a network w ψ ( s, a ) parametrized by ψ , solely based on samples from B off and B on .
Revised: To avoid this problem, we utilize a likelihood-free density ratio estimation method that estimates w ( s, a ) by training a network w ψ ( s, a ) parametrized by ψ , solely based on samples from B off and B on .

Original: Training details.
Revised: Training details.

Original: Here we describe the training procedure for the density ratio estimator w ψ ( s, a ) indetail.
Revised: Here we describe the training procedure for the density ratio estimator w ψ ( s, a ) in detail.

Original: Following the idea of Sinha et al.
Revised: Following the idea of Sinha et al.

Original: [16], we use the variational representation of f-divergences[17].
Revised: [16], we use the variational representation of f-divergences [17].

Original: Let P and Q be probability measures deﬁned on some measurable space X , with P absolutelycontinuous w.r.t.
Revised: Let P and Q be probability measures deﬁned on some measurable space X , with P absolutely continuous w.r.t.

Original: Q , and f ( y ) :
Revised: Q , and f ( y ) :

Original: = y log 2 y y +

Revised: = y log 2 y y +

Original: + log 2 y +1 .
Revised: + log 2 y +1 .

Original: Then the Jensen-Shannon (JS) divergence is deﬁned as D JS ( P || Q ) =
Revised: Then the Jensen-Shannon (JS) divergence is deﬁned as D JS ( P || Q ) =

Original: (cid:82) X f ( dP ( x ) /dQ
Revised: (cid:82) X f ( dP ( x ) /dQ

Original: ( x )) dQ ( x ) .
Revised: ( x )) dQ ( x ) .

Original: We then estimate the density ratio dP/dQwith a parametric model w ψ ( x ) , by maximizing the lower bound of D JS
Revised: We then estimate the density ratio dP/dQ with a parametric model w ψ ( x ) , by maximizing the lower bound of D JS

Original: ( P || Q ) [17]:
Revised: ( P || Q ) [17]:

Original: [Equation]
Revised: [Equation]

Original: where w ψ ( x ) ≥ 0 is parametrized by a neural network whose outputs are forced to be non-negativevia activation functions, and f ∗ denotes convex conjugate.
Revised: where w ψ ( x ) ≥ 0 is parametrized by a neural network whose outputs are forced to be non-negative via activation functions, and f ∗ denotes convex conjugate.

Original: In particular, we obtain an estimatew ψ ( s, a ) of d on /d off by considering probability distributions P and Q with densities d on and d off ,respectively.
Revised: In particular, we obtain an estimate w ψ ( s, a ) of d on /d off by considering probability distributions P and Q with densities d on and d off , respectively.

Original: In practice, we sample from B on for the ﬁrst term in (4), and from B off for the latter.
Revised: In practice, we sample from B on for the ﬁrst term in (4), and from B off for the latter.

Original: For more stable density ratio estimates, we apply self-normalization [18] to the estimated densityratios over B off , similar to Sinha et al.
Revised: For more stable density ratio estimates, we apply self-normalization [18] to the estimated density ratios over B off , similar to Sinha et al.

Original: More details can be found in Section D.
Revised: More details can be found in Section D.

Original: 4.2 Pessimistic Q-Ensemble
Revised: 4.2 Pessimistic Q-Ensemble

Original: In order to mitigate distribution shift more effectively, we leverage multiple pessimistically trained
Revised: In order to mitigate distribution shift more effectively, we leverage multiple pessimistically trained

Original: Q-functions.
Revised: Q-functions.

Original: We consider an ensemble of N CQL agents pre-trained via update rules (2, 3), i.e.,{ Q θ i , π φ i } Ni =1 , where θ i and φ i denote the parameters of the i -th agent’s Q-function and policy,respectively.
Revised: We consider an ensemble of N CQL agents pre-trained via update rules (2, 3), i.e., { Q θ i , π φ i } Ni =1 , where θ i and φ i denote the parameters of the i -th agent’s Q-function and policy, respectively.

Original: Then we use the ensemble of actor-critic agents whose Q-function and policy aredeﬁned as follows:
Revised: Then we use the ensemble of actor-critic agents whose Q-function and policy are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: 5) where θ := { θ i } Ni =1 and φ := { φ i } Ni =1 .
Revised: where θ := { θ i } Ni =1 and φ := { φ i } Ni =1 .

Original: Note that the policy is simply modeled as Gaussian withmean and variance of the Gaussian mixture policy 1 N (cid:80) Ni =1 π φ i .
Revised: Note that the policy is simply modeled as Gaussian with mean and variance of the Gaussian mixture policy 1 N (cid:80) Ni =1 π φ i .

Original: In turn, θ and φ are updated viaupdate rules (1) and (2), respectively, during ﬁne-tuning.
Revised: In turn, θ and φ are updated via update rules (1) and (2), respectively, during ﬁne-tuning.

Original: By using a pessimistic Q-function, the agent remains pessimistic with regard to the unseen actions atstates encountered online during initial ﬁne-tuning.
Revised: By using a pessimistic Q-function, the agent remains pessimistic with regard to the unseen actions at states encountered online during initial ﬁne-tuning.

Original: This is because during early ﬁne-tuning, statesresemble those present in the ofﬂine dataset, and Q-function generalizes to these states.
Revised: This is because during early ﬁne-tuning, states resemble those present in the ofﬂine dataset, and Q-function generalizes to these states.

Original: As we showin our experiments, this protects the good initial policy from severe bootstrap error.
Revised: As we show in our experiments, this protects the good initial policy from severe bootstrap error.

Original: And by leveraging multiple pessimistically trained Q-functions, we obtain a more high-resolution pessimism aboutthe unseen data regime.
Revised: And by leveraging multiple pessimistically trained Q-functions, we obtain a more high-resolution pessimism about the unseen data regime.

Original: That is, when an individual Q-function may erroneously have high valuesfor unseen samples, Q-ensemble is more robust to these individual errors, and more reliably assignslower values to unseen samples.
Revised: That is, when an individual Q-function may erroneously have high values for unseen samples, Q-ensemble is more robust to these individual errors, and more reliably assigns lower values to unseen samples.

Original: Computational overhead of ensemble is discussed in Section B.
Revised: Computational overhead of ensemble is discussed in Section B.

Original:  Related work
Revised:  Related work

Original: Ofﬂine RL.
Revised: Ofﬂine RL.

Original: Ofﬂine RL algorithms aim to train RL agents exclusively with pre-collected datasets.
Revised: Ofﬂine RL algorithms aim to train RL agents exclusively with pre-collected datasets.

Original: To address the state-conditional action distribution shift, prior methods (a) explicitly constrain thepolicy to be closed to the behavior policy [2, 3, 4, 5, 19], or (b) train pessimistic value functions[7, 8, 9].
Revised: To address the state-conditional action distribution shift, prior methods (a) explicitly constrain the policy to be closed to the behavior policy [2, 3, 4, 5, 19], or (b) train pessimistic value functions [7, 8, 9].

Original: In particular, CQL [9] was used to learn various robotic manipulation tasks [10].
Revised: In particular, CQL [9] was used to learn various robotic manipulation tasks [10].

Original: We alsobuild on CQL, so as to leverage pessimism regarding data encountered online during ﬁne-tuning.
Revised: We also build on CQL, so as to leverage pessimism regarding data encountered online during ﬁne-tuning.

Original: Online RL with ofﬂine datasets.
Revised: Online RL with ofﬂine datasets.

Original: Several works have explored employing ofﬂine datasets for online
Revised: Several works have explored employing ofﬂine datasets for online

Original: RL to improve sample efﬁciency.
Revised: RL to improve sample efﬁciency.

Original: Some assume access to demonstration data [20, 21, 22, 23, 24],which is limited in that they assume optimality of the dataset.
Revised: Some assume access to demonstration data [20, 21, 22, 23, 24], which is limited in that they assume optimality of the dataset.

Original: To overcome this, Nair et al.
Revised: To overcome this, Nair et al.

Original: [25]proposed Advantage Weighted Actor Critic (AWAC), which performs regularized policy updates sothat the policy stays close to the observed data during both ofﬂine and online phases.
Revised: [25] proposed Advantage Weighted Actor Critic (AWAC), which performs regularized policy updates so that the policy stays close to the observed data during both ofﬂine and online phases.

Original: We insteadadvocate adopting pessimistic initialization , such that we may prevent overoptimism and bootstraperror in the initial online phase, and lift such pessimism once unnecessary, as more online samplesare gathered.
Revised: We instead advocate adopting pessimistic initialization , such that we may prevent overoptimism and bootstrap error in the initial online phase, and lift such pessimism once unnecessary, as more online samples are gathered.

Original: Some recent works extract behavior primitives from ofﬂine data, then learn to composethem online [26, 27, 28].
Revised: Some recent works extract behavior primitives from ofﬂine data, then learn to compose them online [26, 27, 28].

Original: It would be interesting to apply our method in these setups.
Revised: It would be interesting to apply our method in these setups.

Original: Experience replay.
Revised: Experience replay.

Original: The idea of retrieving important samples for RL was introduced in Schaul et al.[29], where they prioritize samples with high temporal-difference error.
Revised: The idea of retrieving important samples for RL was introduced in Schaul et al. [29], where they prioritize samples with high temporal-difference error.

Original: The work closest to ours is
Revised: The work closest to ours is

Original: Sinha et al.
Revised: Sinha et al.

Original: [16], which utilizes the density ratios between off-policy and near-on-policy state-actiondistributions as importance weights for policy evaluation.
Revised: [16], which utilizes the density ratios between off-policy and near-on-policy state-action distributions as importance weights for policy evaluation.

Original: Our approach differs in that we utilizedensity ratios for retrieving relevant samples from the ofﬂine dataset. 
Revised: Our approach differs in that we utilize density ratios for retrieving relevant samples from the ofﬂine dataset. [Figure]

Original: Ensemble methods.
Revised: Ensemble methods.

Original: In the context of model-free RL, ensemble methods have been studied foraddressing Q-function’s overestimation bias [30, 31, 32, 33, 34], for better exploration [35, 36, 37],or for reducing bootstrap error propagation [38].
Revised: In the context of model-free RL, ensemble methods have been studied for addressing Q-function’s overestimation bias [30, 31, 32, 33, 34], for better exploration [35, 36, 37], or for reducing bootstrap error propagation [38].

Original: The closest to our approach is Anschel et al.[32] that stabilizes Q-learning by using the average of previously learned Q-values as the target Qvalue.
Revised: The closest to our approach is Anschel et al. [32] that stabilizes Q-learning by using the average of previously learned Q-values as the target Qvalue.

Original: While prior works mostly focus on online RL and estimate the ground-truth Q-functions, weleverage an ensemble of pessimistically pre-trained Q-functions for safe ofﬂine-to-online RL.Ours AWAC BCQ
Revised: While prior works mostly focus on online RL and estimate the ground-truth Q-functions, we leverage an ensemble of pessimistically pre-trained Q-functions for safe ofﬂine-to-online RL.

Original: [Figure]

Original:  Experiments
Revised:  Experiments

Original: We designed our experiments to answer the following questions:
Revised: We designed our experiments to answer the following questions:

Original: • How does our method compare to existing ofﬂine-to-online RL methods and an online RLmethod that learns from scratch (see Figure 3)?
Revised: • How does our method compare to existing ofﬂine-to-online RL methods and an online RL method that learns from scratch (see Figure 3)?

Original: • Can our balanced replay scheme locate ofﬂine samples relevant to the current policy (see Figure 4a) and improve the ﬁne-tuning performance by utilizing these samples (see Figure 4c)?
Revised: Can our balanced replay scheme locate ofﬂine samples relevant to the current policy (see Figure 4a) and improve the ﬁne-tuning performance by utilizing these samples (see Figure 4c)?

Original: • Can our pessimistic Q-ensemble scheme discriminate unseen actions (see Figure 4b) and suc- cessfully stabilize the ﬁne-tuning procedure by mitigating distribution shift (see Figure 4d)?
Revised: • Can our pessimistic Q-ensemble scheme discriminate unseen actions (see Figure 4b) and successfully stabilize the ﬁne-tuning procedure by mitigating distribution shift (see Figure 4d)?

Original: • Does our method scale to vision-based robotic manipulation tasks (see Figure 5)?
Revised: • Does our method scale to vision-based robotic manipulation tasks (see Figure 5)?

Original: 6.1 Locomotion Tasks
Revised: 6.1 Locomotion Tasks

Original: Setup.
Revised: Setup.

Original: We consider MuJoCo [11] locomotion tasks, i.e., halfcheetah , hopper , and walker2d ,from the D4RL benchmark suite [12].
Revised: We consider MuJoCo [11] locomotion tasks, i.e., halfcheetah , hopper , and walker2d , from the D4RL benchmark suite [12].

Original: To demonstrate the applicability of our method on various suboptimal datasets, we use four dataset types: random , medium , medium-replay , andmedium-expert .
Revised: To demonstrate the applicability of our method on various suboptimal datasets, we use four dataset types: random , medium , medium-replay , and medium-expert .

Original: Speciﬁcally, random and medium datasets contain samples collected by a random policy and a medium-level policy, respectively.
Revised: Speciﬁcally, random and medium datasets contain samples collected by a random policy and a medium-level policy, respectively.

Original: medium-replay datasets contain all samplesencountered while training a medium-level agent from scratch, and medium-expert datasets contain samples collected by both medium-level and expert-level policies.
Revised: medium-replay datasets contain all samples encountered while training a medium-level agent from scratch, and medium-expert datasets contain samples collected by both medium-level and expert-level policies.

Original: For our method, we useensemble size N = 5 .
Revised: For our method, we use ensemble size N = 5 .

Original: More experimental details are provided in Section D.
Revised: More experimental details are provided in Section D.

Original: Comparative Evaluation.
Revised: Comparative Evaluation.

Original: We consider the methods outlined below as baselines for comparativeevaluation.
Revised: We consider the methods outlined below as baselines for comparative evaluation.

Original: For fair comparison, we applied ensemble to all baselines except SAC-ft, since theresults for SAC-ft with ensemble can be found in the ablation studies (see 
Revised: For fair comparison, we applied ensemble to all baselines except SAC-ft, since the results for SAC-ft with ensemble can be found in the ablation studies (see Figure 4c).

Original: Figure 4c).• Advantage Weighted Actor Critic [AWAC; 25]: an ofﬂine-to-online RL method that trains thepolicy to imitate actions with high advantage estimates.•
Revised: • Advantage Weighted Actor Critic [AWAC; 25]: an ofﬂine-to-online RL method that trains the policy to imitate actions with high advantage estimates. [Figure]

Original:  BCQ-ft: Batch-Constrained deep Q-learning [BCQ; 2], is an ofﬂine RL method that updatespolicy by modeling the behavior policy using a conditional VAE [39].
Revised: • BCQ-ft: Batch-Constrained deep Q-learning [BCQ; 2], is an ofﬂine RL method that updates policy by modeling the behavior policy using a conditional VAE [39].

Original: We extend BCQ to theonline ﬁne-tuning setup by applying the same update rules as ofﬂine training.• SAC-ft: Starting from a CQL agent trained via (3, 2), we ﬁne-tune the agent via SAC updates(1, 2).
Revised: We extend BCQ to the online ﬁne-tuning setup by applying the same update rules as ofﬂine training. • SAC-ft: Starting from a CQL agent trained via (3, 2), we ﬁne-tune the agent via SAC updates (1, 2).

Original: Justiﬁcation for excluding the CQL regularization term from (3) during ﬁne-tuning canbe found in Section E.1.• SAC: a SAC agent trained from scratch via (1, 2), i.e., the agent has no access to the ofﬂinedataset.
Revised: Justiﬁcation for excluding the CQL regularization term from (3) during ﬁne-tuning can be found in Section E.1. • SAC: a SAC agent trained from scratch via (1, 2), i.e., the agent has no access to the ofﬂine dataset.

Original: This baseline highlights the beneﬁt of ofﬂine-to-online RL, as opposed to fully onlineRL, in terms of sample efﬁciency.
Revised: This baseline highlights the beneﬁt of ofﬂine-to-online RL, as opposed to fully online RL, in terms of sample efﬁciency.

Original: [Figure] Figure 3 shows the performances of our method and baseline methods considered during the online
Revised: Figure 3 shows the performances of our method and baseline methods considered during the online

Original: RL phase.
Revised: RL phase.

Original: In most tasks, our method outperforms the baseline methods in terms of both sampleefﬁciency and ﬁnal performance.
Revised: In most tasks, our method outperforms the baseline methods in terms of both sampleefﬁciency and ﬁnal performance.

Original: In particular, our method signiﬁcantly outperforms SAC-ft, whichshows that balanced replay and pessimistic Q-ensemble are indeed essential.
Revised: In particular, our method signiﬁcantly outperforms SAC-ft, which shows that balanced replay and pessimistic Q-ensemble are indeed essential.

Original: We also emphasize that our method performs consistently well across all tasks, while the performances of AWAC and BCQ-ft are highly dependent on the quality of the ofﬂine dataset.
Revised: We also emphasize that our method performs consistently well across all tasks, while the performances of AWAC and BCQ-ft are highly dependent on the quality of the ofﬂine dataset.

Original: For example, we observe that AWAC and BCQ-ft show competitive performances in tasks where the datasetsare generated by high-quality policies, i.e., medium-expert tasks, but perform worse than SAC onrandom tasks.
Revised: For example, we observe that AWAC and BCQ-ft show competitive performances in tasks where the datasets are generated by high-quality policies, i.e., medium-expert tasks, but perform worse than SAC on random tasks.

Original: This is because AWAC and BCQ-ft employ the same regularized, pessimistic updaterule for ofﬂine and online setups alike, either explicitly (BCQ-ft) or implicitly (AWAC), which leadsto slow ﬁne-tuning.
Revised: This is because AWAC and BCQ-ft employ the same regularized, pessimistic update rule for ofﬂine and online setups alike, either explicitly (BCQ-ft) or implicitly (AWAC), which leads to slow ﬁne-tuning.

Original: Our method instead relies on pessimistic initialization , and hence enjoys muchfaster ﬁne-tuning, while not sacriﬁcing the initial training stability.
Revised: Our method instead relies on pessimistic initialization , and hence enjoys much faster ﬁne-tuning, while not sacriﬁcing the initial training stability.

Original: Balanced replay analysis.
Revised: Balanced replay analysis.

Original: To investigate the effectiveness of our balanced experience replayscheme for locating near-on-policy samples in the ofﬂine dataset, we report the ratios of ofﬂinesamples used for updates ﬁne-tuning proceeds.
Revised: To investigate the effectiveness of our balanced experience replay scheme for locating near-on-policy samples in the ofﬂine dataset, we report the ratios of ofﬂine samples used for updates ﬁne-tuning proceeds.

Original: Figure 4a shows that for the random task, ofﬂinesamples quickly become obsolete, as they quickly become irrelevant to the policy being ﬁne-tuned.
Revised: Figure 4a shows that for the random task, ofﬂine samples quickly become obsolete, as they quickly become irrelevant to the policy being ﬁne-tuned.

Original: However, for the medium-expert task, ofﬂine samples include useful expert-level transitions thatare relevant to the current policy, hence are replayed throughout the online training.
Revised: However, for the medium-expert task, ofﬂine samples include useful expert-level transitions that are relevant to the current policy, hence are replayed throughout the online training.

Original: This shows thatour balanced replay scheme is capable of utilizing ofﬂine samples only when appropriate.
Revised: This shows that our balanced replay scheme is capable of utilizing ofﬂine samples only when appropriate.

Original: Q-ensemble analysis.
Revised: Q-ensemble analysis.

Original: We quantitatively demonstrate that pessimistic Q-ensemble indeed providesmore discriminative value estimates, i.e., having distinguishably lower Q-values for unseen actionsthan for seen actions.
Revised: We quantitatively demonstrate that pessimistic Q-ensemble indeed provides more discriminative value estimates, i.e., having distinguishably lower Q-values for unseen actions than for seen actions.

Original: In particular, we consider a medium-expert dataset, where the ofﬂine datadistribution is narrow, and the near-optimal ofﬂine policy can be brittle.
Revised: In particular, we consider a medium-expert dataset, where the ofﬂine data distribution is narrow, and the near-optimal ofﬂine policy can be brittle.

Original: Let D real T :=
Revised: Let D real T :=

Original: { ( s i , a i ) }Ti =1be
Revised: { ( s i , a i ) }

Original:  the samples collected online up until timestep T .
Revised: Ti = be the samples collected online up until timestep T .

Original: We construct a “fake” dataset by replacing theactions in D real T with random actions, i.e., D fake T :=
Revised: We construct a “fake” dataset by replacing the actions in D real T with random actions, i.e., D fake T :=

Original: { ( s i , a unif ) } Ti =1 .
Revised: { ( s i , a unif ) } Ti =1 .

Original: Interpreting Q ( s, a ) as theconﬁdence value for classifying real and fake transitions, we measure the area under ROC (AUROC)curve values over the course of ﬁne-tuning.
Revised: Interpreting Q ( s, a ) as the conﬁdence value for classifying real and fake transitions, we measure the area under ROC (AUROC) curve values over the course of ﬁne-tuning.

Original: As seen in Figure 4b, Q-ensemble demonstrates superiordiscriminative ability, which leads to stable ﬁne-tuning. 
Revised: As seen in Figure 4b, Q-ensemble demonstrates superior discriminative ability, which leads to stable ﬁne-tuning. [Figure]

Original: Ablation studies.
Revised: Ablation studies.

Original: Figure 4c shows that balanced replay improves ﬁne-tuning performance by sampling near-on-policy transitions.
Revised: Figure 4c shows that balanced replay improves ﬁne-tuning performance by sampling near-on-policy transitions.

Original: On the other hand, two other na¨ıve sampling schemes – (a) Uniform, where ofﬂine and online samples are sampled uniformly from the same buffer, and (b) Onlineonly, where the ofﬂine agent is ﬁned-tuned using online samples exclusively – suffer from slowand unstable improvement, even with pessimistic Q-ensemble.
Revised: On the other hand, two other na¨ıve sampling schemes – (a) Uniform, where ofﬂine and online samples are sampled uniformly from the same buffer, and (b) Online only, where the ofﬂine agent is ﬁned-tuned using online samples exclusively – suffer from slow and unstable improvement, even with pessimistic Q-ensemble.

Original: This shows that balanced replay iscrucial for reducing the harmful effects of distribution shift.
Revised: This shows that balanced replay is crucial for reducing the harmful effects of distribution shift.

Original: [Figure] Also, Figure 4d shows that ﬁne-tuning performance improves as the ensemble size N increases,which shows that larger ensemble size provides higher-resolution pessimism, leading to more stablepolicy updates.
Revised: Also, Figure 4d shows that ﬁne-tuning performance improves as the ensemble size N increases, which shows that larger ensemble size provides higher-resolution pessimism, leading to more stable policy updates.

Original: Ablation studies for all tasks can be found in Section E.1.
Revised: Ablation studies for all tasks can be found in Section E.1.

Original: 6.2 Robotic Manipulation Tasks
Revised: 6.2 Robotic Manipulation Tasks

Original: Setup.
Revised: Setup.

Original: We consider three sparse-reward pixel-based manipulation tasks from Singh et al.
Revised: We consider three sparse-reward pixel-based manipulation tasks from Singh et al.

Original: [10]: (1)pick-place : pick an object and put it in the tray; (2) grasp-closed-drawer : grasp an objectin the initially closed bottom drawer; (3) grasp-blocked-drawer : grasp an object in the initiallyclosed bottom drawer, where the initially open top drawer blocks the handle for the bottom drawer.
Revised: [10]: (1) pick-place : pick an object and put it in the tray; (2) grasp-closed-drawer : grasp an object in the initially closed bottom drawer; (3) grasp-blocked-drawer : grasp an object in the initially closed bottom drawer, where the initially open top drawer blocks the handle for the bottom drawer.

Original: Episode lengths for the tasks are 40, 50, 80, respectively.
Revised: Episode lengths for the tasks are 40, 50, 80, respectively.

Original: The original dataset [10] for each task consists of scripted exploratory policy rollouts.
Revised: The original dataset [10] for each task consists of scripted exploratory policy rollouts.

Original: For example,for pick-place , the dataset contains scripted pick attempts and place attempts.
Revised: For example, for pick-place , the dataset contains scripted pick attempts and place attempts.

Original: However, it israrely the case that logged data ‘in the wild’ contains such structured, high-quality transitions only.
Revised: However, it is rarely the case that logged data ‘in the wild’ contains such structured, high-quality transitions only.

Original: We consider a more realistic setup where the dataset also includes undirected, exploratory samples– we replace a subset of the original dataset with uniform random policy rollouts.
Revised: We consider a more realistic setup where the dataset also includes undirected, exploratory samples – we replace a subset of the original dataset with uniform random policy rollouts.

Original: Note that randompolicy rollouts are common in robotic tasks [40, 41].
Revised: Note that random policy rollouts are common in robotic tasks [40, 41].

Original: We used ensemble size N = 4 for our method.
Revised: We used ensemble size N = 4 for our method.

Original: More details about the tasks and dataset construction are provided in Section D.2.
Revised: More details about the tasks and dataset construction are provided in Section D.2.

Original: Comparative Evaluation.
Revised: Comparative Evaluation.

Original: We compare our method with the method considered in Singh et al.[10], namely, CQL ﬁne-tuning with online samples only.
Revised: We compare our method with the method considered in Singh et al. [10], namely, CQL ﬁne-tuning with online samples only.

Original: CQL-ft fails to solve the task in some ofthe seeds, resulting in high variance as seen in Figure 5a.
Revised: CQL-ft fails to solve the task in some of the seeds, resulting in high variance as seen in Figure 5a.

Original: This is because CQL shows inconsistentofﬂine performance across random seeds due to such factors as difﬁculty of training on mixture data[12], instability of ofﬂine agents over stopping point of training [42], and sparsity of rewards.
Revised: This is because CQL shows inconsistent ofﬂine performance across random seeds due to such factors as difﬁculty of training on mixture data [12], instability of ofﬂine agents over stopping point of training [42], and sparsity of rewards.

Original: Withno access to (pseudo-)expert ofﬂine data and due to heavy regularization of CQL, such CQL agentshardly improve.
Revised: With no access to (pseudo-)expert ofﬂine data and due to heavy regularization of CQL, such CQL agents hardly improve.

Original: Meanwhile, our method consistently learns to perform the task within a reasonableamount of additional environment interaction (40K to 80K steps).
Revised: Meanwhile, our method consistently learns to perform the task within a reasonable amount of additional environment interaction (40K to 80K steps).

Original: Buffer analysis.
Revised: Buffer analysis.

Original: We analyze whether balanced replay scales to  robotic tasks based on image observations.
Revised: We analyze whether balanced replay scales to image-based robotic tasks.

Original: As seen in Figure 5b, without any privileged information, balanced replay automaticallyselects relevant ofﬂine samples for updates, while ﬁltering out task-irrelevant, random data as ﬁnetuning proceeds.
Revised: As seen in Figure 5b, without any privileged information, balanced replay automatically selects relevant ofﬂine samples for updates, while ﬁltering out task-irrelevant, random data as ﬁne-tuning proceeds.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we identify state-action distribution shift as the major obstacle in ofﬂine-to-online RL.
Revised: In this paper, we identify state-action distribution shift as the major obstacle in ofﬂine-to-online RL.

Original: To address this, we present a simple framework that incorporates (1) a balanced experience replayscheme, and (2) a pessimistic Q-ensemble scheme.
Revised: To address this, we present a simple framework that incorporates (1) a balanced experience replay scheme, and (2) a pessimistic Q-ensemble scheme.

Original: Our experiments show that the proposed methodperforms well across many continuous control robotic tasks, including locomotion and manipulationtasks.
Revised: Our experiments show that the proposed method performs well across many continuous control robotic tasks, including locomotion and manipulation tasks.

Original: We expect our method to enable more sample-efﬁcient training of robotic agents by leveragingofﬂine samples both for ofﬂine and online learning.
Revised: We expect our method to enable more sample-efﬁcient training of robotic agents by leveraging ofﬂine samples both for ofﬂine and online learning.

Original: We also believe our method could prove to beuseful for other relevant topics such as scalable RL [43] and RL safety [44].A Pseudocode We provide a pseudocode for the entire training procedure.
Revised: We also believe our method could prove to be useful for other relevant topics such as scalable RL [43] and RL safety [44].

Original: For conciseness of presentation, wehenceforth let τ =

Original: ( s, a, r, s (cid:48) ) denote a transition stored in the replay buffer.

Original: do 4: do 5:

Original: Sample a random minibatch { τ j } Bj =1 ∼ B off 6:

Original: Calculate L CQLcritic ( θ i ) and update θ i 7:

Original: Calculate L SACactor ( φ i ) and update φ i 8: end for 9: end for 10: //

Original: E NSEMBLE

Original: A GENT 11: Deﬁne Q-ensemble Q θ := 1 N (cid:80) Ni

Original: =1 Q θ i 12: Deﬁne π φ ( ·| s ) :

Original: = N (cid:18)

Original: N (cid:80) Ni

Original: =1 µ φ i ( s ) , 1 N (cid:80) Ni =1 ( σ 2 φ i ( s ) + µ 2 φ i ( s )) − µ 2 φ ( s ) (cid:19) environment interaction with π φ 11:

Original: Update B on ← B on ∪ { τ } , B priority ← B priority ∪ { ( τ, p 0 ) } 12: // U PDATE DENSITY RATIO ESTIMATION NETWORK 13:

Original: Sample { τ off i } Bi =1 ∼ B off , { τ on i } Bi =1 ∼ B on 14:

Original: Calculate L DR ( ψ ) and update ψ 15: // U PDATE AGENT AND PRIORITY VALUES 16:

Original: Sample a random minibatch { τ j } Bj =1 ∼ B priority 17:

Original: Calculate L SACcritic ( θ ) and update θ 18:

Original: Calculate L SACactor ( φ ) and update φ 19: for j = 1 , . . .

Original: , B do 20:

Original: Update priority of τ j to be

Original: [Equation]

Original: off

Original: [Equation]

Original: off [

Original: [Equation]

Original: [Equation]

Original: )] (see Training details for balanced replay in

Original: Section D for explanation) 21: [Equation] ← [Equation]
Revised: Acknowledgments

Original: 22: end for 23: end for

Original: B Computational Complexity

Original: Ensemble incurs additional computation overhead for training and inference. A na¨ıve implementation with ensemble size N leads to N × training and inference time. However, it is possible toparallelize the networks (which we did for MuJoCo locomotion tasks) for computational efﬁciency. C Training Details for Concept Experiments
Revised: This work was supported by Microsoft and Institute of Information & communications Technology

Original: VAE training.

Original: For generating the plot in Figure 1a, we ﬁrst split the halfcheetah-random ofﬂinedataset consisting of random halfcheetah transitions into training and validation datasets.

Original: Thenwe trained a variational autoencoder on the training dataset using the Adam optimizer [45] withlearning rate 1e − 3 .

Original: We estimated the log-likelihood of (a) online samples gathered by an ofﬂineagent trained via CQL (3, 2), and (b) ofﬂine samples in the validation dataset, using the importanceweight estimator [46].

Original: Sample selection analysis.

Original: For generating the plot in Figure 1b, we ﬁrst trained a CQL agent

Original: Q θ , π φ on the halfcheetah-medium dataset consisting of medium-quality halfcheetah transitions.

Original: Then, we ﬁne-tune the CQL agent with two different strategies: Uniform and Online only.

Original: First,

Original: Uniform refers to the strategy where we keep a single replay buffer, in which we store both ofﬂineand online samples, then sampling a minibatch from the replay buffer uniformly at random, i.e.,{ τ j } Bj =1 ∼ B off ∪ B on .

Original: Here, the sampling procedure is agnostic of whether the sample is ofﬂine oronline.

Original: On the other hand, Online only refers to the strategy that exploits online samples only, i.e.,{ τ j } Bj =1 ∼ B on .

Original: Choice of ofﬂine Q-function.

Original: For generating the plots in Figure 1c, we considered thehalfcheetah-random and halfcheetah-medium-expert ofﬂine datasets, each consisting ofrandom, and a mixture of medium and expert transitions, respectively.

Original: For a given ofﬂine dataset

Original: B off , we ﬁrst trained a CQL agent { Q θ CQL , π φ } via (3, 2).

Original: Then, with π φ and B off , we trained anon-pessimistic Q θ FQE via Fitted Q-Evaluation [FQE; 15].

Original: To elaborate, given a ﬁxed policy π φ ,

Original: FQE trains a Q-function starting from random initialization by performing policy evaluation (Bellman backup) until convergence.

Original: In practice, we used the Adam optimizer [45] with learning rate3e − 4 and batch size B = 256 , and trained for 250k iterations.

Original: In the online ﬁne-tuning phase, weﬁne-tuned { π φ , Q θ CQL } and { π φ , Q θ FQE } , respectively, via SAC update rules (1, 2).

Original: We applied thebalanced replay scheme introduced in Section 4.1, in order to facilitate ﬁne-tuning.

Original: D Experimental Setup Details

Original: [Figure]

Original: D.1 Locomotion

Original: [Figure]

Original: Task detail & ofﬂine dataset.

Original: For training ofﬂine RL agents, we used the publicly availabledatasets from the D4RL benchmark suite ( https://github.com/rail-berkeley/d4rl ) without anymodiﬁcation to the datasets.

Original: Speciﬁcally, we consider three MuJoCo locomotion tasks, namely, halfcheetah , hopper , and walker2d .

Original: The goal ofeach task is to move forward as fast as possible, while keeping the control cost minimal.

Original: For each task, we consider four types of datasets:

Original: • random : contains random policy rollouts;

Original: • medium : contains medium-level policy rollouts;

Original: • medium-replay : contains all samples seen while training a medium-level agent from scratch;• medium-expert : half of the dataset consists of medium-level policy rollouts and the other halfconsists of expert-level policy rollouts.

Original: Training details for ofﬂine RL.

Original: For training CQL agents, following Kumar et al.

Original: [9], we built CQLon top of the publicly available implementation of SAC ( https://github.com/vitchyr/rlkit ).

Original: As for network architecture, we use 2-layer multi-layer perceptrons (MLPs) for value and policynetworks (except for halfcheetah-medium-expert task where we found that 3-layer MLPs ismore effective for training CQL agents).

Original: For all locomotion experiments, following the setup in

Original: Kumar et al.

Original: [9], we trained ofﬂine RL agents for 1000 epochs without early stopping.

Original: Training details for online RL.

Original: For all locomotion experiments, we report the ﬁne-tuning performance during 250K timesteps for 4 random seeds.

Original: For our method, we used Adam optimizer [45]with policy learning rates chosen from { 3e − 4 , 3e − 5 , 5e − 6 } , and value learning rate of 3e − 4 .

Original: [-Weused ensemble size N = 
-]
Revised: [-Weused ensemble size N = 
-]

Original: We found that taking 5 × training steps (5000 as opposed to the usual1000) after collecting the ﬁrst 1000 online samples slightly improves the ﬁne-tuning performance.

Original: After that, we trained for 1000 training steps every time 1000 additional samples were collected.

Original: Training details for balanced replay.

Original: For training the density ratio estimation network w ψ ( s, a ) ,which was ﬁxed to be a 2-layer MLP for all our experiments, we used batch size 256 (i.e., 256ofﬂine samples and 256 online samples), and learning rate 3e − 4 for all locomotion experiments.

Original: When calculating the priority values, we applied self-normalization:

Original: [Equation]

Original: where x and P denote ( s, a ) and B off , respectively, and T is the temperature hyperparameter.

Original: Forall locomotion experiments, we used T = 5 . 0 .

Original: This stabilizes training by producing stable densityratios across random minibatches.

Original: Before ﬁne-tuning starts, we ﬁrst add the ofﬂine samples to the prioritized replay buffer with defaultpriority value 1 . 0 .

Original: Then, we set the default priority value to be high, in order to make sure onlinesamples are seen enough for RL updates.

Original: In particular, letting M denote the size of the ofﬂine buffer,we set the default priority value to be such that the initial 1000 online samples gathered would haveprobability ρ of being seen, where ρ is a hyperparameter, i.e., priority value of P 0 :=

Original: M 1000 · ρ 1 − ρ .

Original: We used ρ ∈ { 0 . 5 , 0 . 75 } .

Original: Note that after being seen once for an RL update, the given sample’s priority value is appropriatelyupdated to be ˜ w ψ ( s, a ) .

Original: Afterwards, the default priority value is updated to be the maximum priorityvalue seen during ﬁne-tuning, as in [29].

Original: See Section A for a pseudocode.

Original: Baselines.

Original: For AWAC [25], we use the publicly available implementation from the authors( https://github.com/vitchyr/rlkit/tree/master/examples/awac ) without any modiﬁcation to hyperparameters and architectures.

Original: For training BCQ [2], we use the publicly available implementation from https://github.com/rail-berkeley/d4rl_evaluations/tree/master/bcq .

Original: For the online setup (BCQ-ft), we used additional online samples for training.

Original: For

Original: SAC [13], we used the implementation from rlkit ( https://github.com/vitchyr/rlkit ) withdefault hyperparameters.

Original: [Figure]

Original: Task detail & ofﬂine dataset.

Original: We consider three sparse-reward pixel-based manipulation tasksfrom Singh et al.

Original: [10] (see Figure 7):

Original: • pick-place : pick an object and put it in the tray.

Original: • grasp-closed-drawer : grasp an object in the initially closed bottom drawer;

Original: • grasp-blocked-drawer : grasp an object in the initially closed bottom drawer, where the ini-

Original: tially open top drawer blocks the handle for the bottom drawer.

Original: Episode lengths for the tasks are 40, 50, 80, respectively.

Original: For each task, the ofﬂine dataset provided by the authors of Singh et al.

Original: [10] ( https://github.com/avisingh599/cog ) consists of highly structured transitions.

Original: For example, for pick-place ,the dataset contains grasp attempts from a randomized scripted policy and place attempts from a randomized scripted policy; for grasp-closed-drawer and grasp-blocked-drawer , dataset contains various behaviors such as opening and closing both drawers, grasping object in the scene, andplacing objects at random places.

Original: For a more detailed description, refer to Singh et al.

Original: Meanwhile, it is rarely the case that logged data ‘in the wild’ are generated by such structured, directedpolicies only.

Original: To consider a more realistic setup, we replace half of the original dataset with 100Kuniform random policy rollouts.

Original: We remark that uniform random policy rollouts are commonly usedin robotic tasks [40, 41].

Original: Training details for ofﬂine RL.

Original: We used the ofﬁcial implementation by the authors of Singh et al.[10] ( https://github.com/avisingh599/cog ) for training ofﬂine CQL agents on the modiﬁeddatasets, with default hyperparameters.

Original: Speciﬁcally, the value network is parametrized by three convolutional layers and two pooling layers in-between, followed by fully-connected layers of hiddensizes (1024, 512, 256) with ReLU activations.

Original: Kernels of size 3 × 3 with 16 channels are used forall three convolutional layers.

Original: Maxpooling is applied after each of the ﬁrst two convolutional layers,with stride 2.

Original: The policy network is identical to the value network, except it does not take actionas an input.

Original: Policy learning rate and value learning rate are 1e − 4 and 3e − 4 , respectively.

Original: For eachtask, we trained the agent for 500K train steps, then picked the best performing checkpoint in termsof average performance across seeds.

Original: Training details for online RL.

Original: For our main results, we report ﬁne-tuning performance during 1Kepisodes for 8 random seeds.

Original: For our method, we used Adam optimizer [45] with policy learningrate 3e − 5 and value learning rate 1e − 4 .

Original: Again, we applied 5 × training steps after collecting theﬁrst batch of samples.

Original: Architecture for density ratio estimation network w ψ is identical to the valuenetwork, except it has fully-connected layers with hidden sizes (256, 128).

Original: We used learning rate3e − 4 for w ψ and temperature T = 2 .

Original: 5 for balanced replay in all manipulation tasks.

Original: We usedensemble size N = 4 .

Original: Baseline.

Original: For CQL-online, we used the ofﬁcial implementation by the authors of Singh et al.

Original: [10]( https://github.com/avisingh599/cog ).

Original: Speciﬁcally, the same CQL regularization is appliedduring online ﬁne-tuning, while using online samples exclusively for updates.

Original: Learning rates are thesame as in ofﬂine training.

Original: E Additional Experiment

Original: E.1 Locomotion

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Effects of CQL Regularization on Fine-tuning.

Original: Here, we show that removing the regularizationterm from the CQL objective L CQLcritic is crucial for fast ﬁne-tuning.

Original: After training an ofﬂine RL agentvia CQL (3, 2), we ﬁne-tuned the agent via CQL objective (3) (denoted CQL Reg), and via SACobjective (1) (denoted No reg), respectively.

Original: For both methods, balanced replay was applied.

Original: Asshown in Figure 8, ﬁne-tuning with CQL regularization prevents the agent from improving via online interaction, for pessimistic updates during ﬁne-tuning (as opposed to pessimistic initialization )essentially results in failure to explore.

Original: Balanced replay analysis.

Original: We provide buffer analysis as in Figure 4a for all locomotion tasks.

Original: As seen in Figure 9, trends are similar across all environments, i.e., balanced replay automaticallydecides whether ofﬂine samples will be used or not depending their relevance to the current agent.

Original: Q-ensemble analysis.

Original: We provide further details and experiments for Q-ensemble analysis from Figure 4b. Letting D real T : = { ( s i , a i ) }
Revised: Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: Ti =1 be the samples collected online up until timestep T , weconstruct a corresponding fake dataset by replacing the actions in D real T by uniform random actions,i.e., D fake T :

Original: = { ( s i , a unif ) }

Original: Ti =1 .

Original: Note that to provide enough coverage of the action space, we sampled 50 random actions for each real state-action pair considered.

Original: Then, interpreting Q ( s, a ) as theconﬁdence value for classifying real and fake transitions, we measure the area under ROC (AUROC)curve values over the course of ﬁne-tuning.

Original: As seen in Figure 10, Q-ensemble demonstrates superiordiscriminative ability, which prevents distribution shift and leads to stable ﬁne-tuning.

Original: [Figure]

Original: [Figure]

Original: Ablation on balanced replay.

Original: We provide ablation study on balanced replay as in Figure 4c for alltasks (Figure 11).

Original: Balanced replay provides signiﬁcant performance gain compared to the other twosampling strategies: Online only, and Uniform.

Original: The beneﬁt is especially pronounced in the morecomplex environment of hopper and walker2d .

Original: Ablation on ensemble size.

Original: We provide learning curves for all tasks with varying ensemble size

Original: N ∈ { 1 , 2 , 5 } .

Original: As seen in Figure12, performance improves as N grows.

Original: However, due to thetrade-off between computational overhead and performance, we opted for N = 5 for our method.

Original: [Figure]

Original: [Figure]

Original: Dataset composition and performance.

Original: In our experiments, we observed that dataset compositionmay be related to the ﬁne-tuning performance in a counterintuitive way.

Original: In order to investigate this,we additionally ran our method on halfcheetah tasks for 1 million steps.

Original: As shown in Figure 13, ourmethod performed best on random dataset, reaching 15k average return, while reaching only 12k formedium and medium-replay datasets.

Original: We conjecture this is because the agent sees the most diverseset of samples when trained from scratch.

Original: This means that Q-function can extrapolate better, andthe agent can explore more efﬁciently as a result.

Original: While the medium-replay dataset also containsdiverse samples, it asymptotically performs worst, possibly because it contains signiﬁcantly less datacompared to the other datasets – random and medium contain 1 million transitions, respectively, andmedium-expert contains 2 million transitions.

Original: We note that failure to explore properly is not necessarily due to the pessimism in Q-function, butis rather a legacy of the dataset composition.

Original: To show this, we ﬁne-tuned an ensemble of FQEinitialized agents (neutral Q-function) with balanced replay (see Section 3.3 and Section C for details) then compared it with our method (pessimistic Q-function), as in Section 3.3.

Original: As shown in Figure 13, we observe that the two methods show more or less similar asymptotic performances with theexception of halfcheetah-random, where our method noticeably outperforms FQE-init.

Original: This showsthat initial pessimism does not harm the asymptotic online performance, while preventing initial performance degradation in many cases (meanwhile, FQE-init suffers from performance degradationfor medium and medium-expert tasks, a result in line with the observation from Section 3.3). Exploring different density ratio schemes. We also additionally experimented with density ratiow ( s, a ) : = d on ( s, a ) /d off ∪ on ( s, a ) instead of w ( s, a ) :
Revised: (No.2019-0-00075,

Original: = d on ( s, a ) /d off ( s, a ) for balanced replay.

Original: The hyperparameters were set identically to the original setup.

Original: As seen in Figure 14, we observedsimilar performance across most setups.

Original: However, the alternative scheme may have the potentialbeneﬁt of trending towards 1 . 0 asymptotically. [Figure] [Figure] [Figure]
Revised: Artiﬁcial Intelligence Graduate School Program(KAIST)).

Original: E.2 Manipulation

Original: Ablation studies.

Original: We provide ablation studies of our method for the robotic manipulation tasks(Figure 15).

Original: Starting from ofﬂine CQL agents, we consider:

Original: • SAC Online only: Fine-tune with SAC, with online samples only;

Original: [Figure]

Original: • SAC Balanced: Fine-tune with SAC, with balanced replay;

Original: [Figure]

Original: • SAC Ensemble Online only: Fine-tune an ensemble agent with SAC, with online samples only; • SAC Ensemble Balanced: Ours, i e., ﬁne-tune an ensemble agent with SAC and balanced replay.
Revised: We would like to thank anonymous reviewers for providing helpful feedbacks and suggestions in improving our paper

Original: Furthermore, we additionally experimented with CQL objectives instead of SAC objectives, andprovide the results (Figure 16).

Original: In most cases, both ensemble and balanced replay were essential for good performance.

Original: First, dueto reward sparsity, there were individual ofﬂine agents that, while producing meaningful trajectoriesthat resemble some trajectories in the ofﬂine dataset, were nonetheless not good enough to receivereward signals online.

Original: In this case, the ensemble agent becomes more robust than these individualagents, and was more likely to see enough reward signals for ﬁne-tuning.

Original: Second, even with ensemble, updating with online samples only resulted in severe distribution shift, and balanced replay wasessential to facilitate ﬁne-tuning.

Original: However, note that it only took our method about 200 episodes(around 8k to 16k environment steps) to recover the initial score, and that our method achieves bestasymptotic scores both with SAC and CQL objectives.

Original: Balanced replay analysis.

Original: We provide analysis for balanced replay in all manipulation tasks considered.

Original: As seen in Figure 17, without any privileged information, balanced replay automaticallyselects relevant ofﬂine samples for updates, while ﬁltering out task-irrelevant, random data as ﬁnetuning proceeds.