A DAPTIVE
B EHAVIOR
C LONING
R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Ofﬂine reinforcement learning, by learning from a ﬁxed dataset, makes it possible to learn agent behaviors without interacting with the environment.
However, depending on the quality of the ofﬂine dataset, such pre-trained agents may have limited performance and would further need to be ﬁne-tuned online by interacting with the environment.
During online ﬁne-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from ofﬂine to online data.
While constraints enforced by ofﬂine RL methods such as a behaviour cloning loss prevent this to an extent, these constraints also signiﬁcantly slow down online ﬁne-tuning by forcing the agent to stay close to the behavior policy.
We propose to adaptively weigh the behavior cloning loss during online ﬁnetuning based on the agent’s performance and training stability.
Moreover, we use a randomized ensemble of Q functions to further increase the sample efﬁciency of online ﬁne-tuning by performing a large number of learning updates.
Experiments show that the proposed method yields state-of-the-art ofﬂine-to-online reinforcement learning performance on the popular D4RL benchmark.
I NTRODUCTION
Ofﬂine or batch reinforcement learning (RL) deals with the training of RL agents from ﬁxed datasets generated by possibly unknown behavior policies, without any interactions with the environment.
This is important in problems like robotics, autonomous driving, and healthcare where data collection can be expensive or dangerous.
Ofﬂine RL has been challenging for model-free RL methods due to extrapolation error where the Q networks predict unrealistic values upon evaluations on outof-distribution state-action pairs (Fujimoto et al., 2019).
Recent methods overcome this issue by constraining the policy to stay close to the behavior policy that generated the ofﬂine data distribution (Fujimoto et al., 2019; Kumar et al., 2020; Kostrikov et al., 2021; Fujimoto & Gu, 2021), to demonstrate even better performance than the behavior policy on several simulated and real-world tasks (Siegel et al., 2020; Singh et al., 2020; Nair et al., 2020).
However, the performance of pre-trained policies will be limited by the quality of the ofﬂine dataset and it is often necessary or desirable to ﬁne-tune them by interacting with the environment.
Also, ofﬂine-to-online learning reduces the risks in online interaction as the ofﬂine pre-training results in reasonable policies that could be tested before deployment.
In practice, ofﬂine RL methods often fail during online ﬁne-tuning by interacting with the environment.
This ofﬂine-to-online RL setting is challenging due to: (i) the sudden distribution shift from ofﬂine data to online data.
This could lead to severe bootstrapping errors which completely distorts the pre-trained policy leading to a sudden performance drop from the very beginning of online ﬁne-tuning, and (ii) constraints enforced by ofﬂine RL methods on the policy to stay close to the behavior policy.
While these constraints help in dealing with the sudden distribution shift they signiﬁcantly slow down online ﬁne-tuning from newly collected samples.
We propose to adaptively weigh the ofﬂine RL constraints such as behavior cloning loss during online ﬁne-tuning.
This could prevent sudden performance collapses due to the distribution shift while also enabling sample-efﬁcient learning from the newly collected samples.
We propose to perform this adaptive weighing according to the agent’s performance and the training stability.
We start with TD3+BC, a simple ofﬂine RL algorithm recently proposed by Fujimoto & Gu (2021) which combines TD3 (Fujimoto et al., 2018) with a simple behavior cloning loss, weighted by an α hyperparameter.
We adaptively weigh this α hyperparameter using a control mechanism similar to the proportional–derivative (PD) controller.
The α value is decided based on two components: the difference between the current return and the target return (proportional term) as well as the change of return between current episode and the last episode (derivative term).
We demonstrate that these simple modiﬁcations lead to stable online ﬁne-tuning after ofﬂine pre-training on datasets of different quality.
We also use a randomized ensemble of Q functions (Chen et al., 2021) to further improve the sample-efﬁciency.
We attain state-of-the-art online ﬁne-tuning performance on locomotion tasks from the popular D4RL benchmark.
R ELATED W ORK
Ofﬂine RL .
Ofﬂine RL aims to learn a policy from pre-collected ﬁxed datasets without interacting with the environment (Lange et al., 2012; Agarwal et al., 2020;
Fujimoto et al., 2019; Kumar et al., 2019; Nachum et al., 2019; Siegel et al., 2020; Levine et al., 2020; Peng et al., 2019).
Off-policy RL algorithms allow for reuse of off-policy data (Konda & Tsitsiklis, 2000; Degris et al., 2012; Haarnoja et al., 2018;
Silver et al., 2014; Lillicrap et al., 2015; Fujimoto et al., 2018; Mnih et al., 2015)
but they typically fail when trained ofﬂine on a ﬁxed dataset, even if it’s collected by a policy trained using the same algorithm (Fujimoto et al., 2019; Kumar et al., 2019).
In actor-critic methods, this is due to extrapolation error of the critic network on out-of-distribution state-action pairs Levine et al.
Ofﬂine RL methods deal with this by constraining the policy to stay close to the behavioral policy that collected the ofﬂine dataset.
BRAC (Wu et al., 2019) achieves this by minimizing the
Kullback-Leibler divergence between the behavior policy and the learned policy.
BEAR (Kumar et al., 2019) minimizes the MMD distance between the two policies.
TD3+BC (Fujimoto & Gu, 2021) proposes a simple yet efﬁcient ofﬂine RL algorithm by adding an additional behavior cloning loss to the actor update.
Another class of ofﬂine RL methods learns conservative Q functions, which prevents the policy network from exploiting out-of-distribution actions and forces them to stay close to the behavior policy.
CQL (Kumar et al., 2020) changes the critic objective to also minimize the
Q function on unseen actions.
Fisher-BRC (Kostrikov et al., 2021) achieves conservative Q learning by constraining the gradient of the Q function on unseen data.
Model-based ofﬂine RL methods (Yu et al., 2020; Kidambi et al., 2020) train policies based on the data generated by ensembles of dynamics models learned from ofﬂine data, while constraining the policy to stay within samples where the dynamics model is certain.
In this paper, we focus on ofﬂine-to-online RL with the goal of stable and sample-efﬁcient online ﬁne-tuning from policies pre-trained on ofﬂine datasets of different quality.
Ofﬂine pre-training in RL .
Pre-training has been vastly investigated in the machine learning community from computer vision (Sharif Razavian et al., 2014; Donahue et al., 2014;
Yosinski et al., 2014) to natural language processing (Devlin et al., 2018; Turian et al., 2010).
Ofﬂine pre-training in RL could enable deployment of RL methods in domains where data collection can be expensive or dangerous.
(Silver et al., 2016; Gupta et al., 2019; Rajeswaran et al., 2017)
pre-train the policy network with imitation learning to speed up RL.
QT-opt (Kalashnikov et al., 2018) studies visionbased object manipulation using a diverse and large dataset collected by seven robots over several months and ﬁne-tune the policy with 27K samples of online data.
However, these methods pre-train using diverse, large, or expert datasets and it is also important to investigate the possibility of pretraining from ofﬂine datasets of different quality.
Yang & Nachum (2021); Ajay et al.
(2020) use ofﬂine pre-training to accelerate downstream tasks.
AWAC (Nair et al., 2020) and Balanced Replay
Lee et al.
(2021) are recent works that also focus on ofﬂine-to-online RL from datasets of different quality.
AWAC updates the policy network such that it is constrained during ofﬂine training while not too conservative during ﬁne-tuning.
Balanced Replay trains an additional neural network to prioritize samples in order to effectively use new data as well as near-on-policy samples in the ofﬂine dataset.
We compare with AWAC and Balanced Replay to attain state-of-the-art ofﬂine-to-online
RL performance on the popular D4RL benchmark.
Ensembles in RL .
Ensemble methods are widely used for better performance in RL (Faußer &
Schwenker, 2015; Osband et al., 2016; Chua et al., 2018; Janner et al., 2019).
In model-based RL,
PETS (Chua et al., 2018) and MBPO (Janner et al., 2019) use probabilistic ensembles to effectively model the dynamics of the environment.
In model-free RL, ensembles of Q functions have been shown to improve performance (Anschel et al., 2017; Lan et al., 2020).
REDQ (Chen et al., 2021) learns a randomized ensemble of Q functions to achieve similar sample efﬁciency as model-based methods without learning a dynamic model.
We utilize REDQ in this work for improved sampleefﬁciency during online ﬁne-tuning.
Speciﬁc to ofﬂine RL, REM (Agarwal et al., 2020) uses random convex combinations of multiple Q-value estimates to calculate the Q targets for effective ofﬂine RL on Atari games.
MOPO (Yu et al., 2020) uses probabilistic ensembles from PETS to learn policies from ofﬂine data using uncertainty estimates based on model disagreement.
MBOP (Argenson &
Dulac-Arnold, 2020) uses ensembles of dynamic models, Q functions, and policy networks to get better performance on locomotion tasks.
Balanced Replay (Lee et al., 2021) uses ensembles of pessimistic Q functions to mitigate instability caused by distribution shift in ofﬂine-to-online RL.
While ensembling of Q functions has been studied by several prior works (Lan et al., 2020; Chen et al., 2021), we combine it with behavioral cloning loss for the purpose of robust and sampleefﬁcient ofﬂine-to-online RL.
Adaptive balancing of multiple objectives in RL .
Ball et al.
(2020) train policies using learned dynamics models with the objective of visiting states that most likely lead to subsequent improvement in the dynamics model, using active online learning.
They adaptively weigh the maximization of cumulative rewards and minimization of model uncertainty using an online learning mechanism based on exponential weights algorithm.
In this paper, we focus on ofﬂine-to-online RL using model-free methods and propose to adaptively weigh the maximization of cumulative rewards and a behavioral cloning loss.
Exploration of other online learning algorithms such as exponential weights algorithm is a line of future work.
B ACKGROUND 3. R EINFORCEMENT L EARNING
Reinforcement learning (RL) deals with sequential decision making to maximize cumulative rewards.
RL problems are often formalized as Markov decision processes (MDPs).
An MDP consists of a set of states S , a set of actions A , a transition dynamics s t +1 ∼ p ( ·| s t , a t ) that represents the probability of transitioning to a state s t +1 by taking action a t in state s t at timestep t , a scalar reward function r t =
R ( s t , a t ) , and a discount factor γ ∈ [0 , 1] .
A policy function π of an RL agent is a mapping from states to actions and deﬁnes the behavior of the agent.
The value function V π ( s ) of a policy π is deﬁned as the expected cumulative rewards from state s : V π ( s ) =
E [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s 0 =
s ] , where the expectation is taken over state transitions s t +1 ∼ p ( ·| s t , a t ) and policy function a t ∼ π ( s t ) .
Similarly, the state-action value function Q π ( s , a ) is deﬁned as the expected cumulative rewards after taking action a in state s :
Q π ( s , a )
[Equation]
∞
[Equation]
γ
[Equation]
R
[Equation]
s
[Equation]
, a t ) | s 0 =
s , a 0 = a ] .
The goal of RL is to learn an optimal policy function π θ with parameters θ , that maximizes the expected cumulative rewards:
[Equation]
We use the TD3 algorithm for reinforcement learning (Fujimoto et al., 2018).
TD3 is an actor-critic method that alternatingly trains: (i) the critic network Q φ to estimate the Q π θ ( s , a ) values of the policy network π θ , and (ii) the policy network to produce actions that maximize the Q function: ∇ θ Q φ ( s , π θ ( s )) .
3. O FFLINE P RE - TRAINING
Ofﬂine reinforcement learning or batch reinforcement learning assumes that the agent is not able to interact with the environment but is given a ﬁxed dataset D of ( s , a , r, s (cid:48) )
tuples to learn from.
The data is assumed to be collected by an unknown behavioural policy (or a collection of policies).
The problem with using actor-critic methods for ofﬂine RL is extrapolation error due to the evaluation of the critic network on the next state and next action values Q ( s (cid:48) , a (cid:48) ) to compute the temporal difference error.
Here the next action a (cid:48) is sampled from the policy network a (cid:48) ∼ π θ ( s (cid:48) ) and this could lead to out-of-distribution evaluations of the critic network.
This is problematic as erroneous predictions of the critic on unfamiliar actions could be propagated to other critic predictions due to bootstrapping in temporal difference learning.
This will also lead to the policy network preferring actions with unrealistic value predictions.
This problem can be overcome either by constraining the policy network to stay close to the data distribution (Fujimoto & Gu, 2021) or by enforcing conservative estimates of the critic network on out-of-distribution samples (Kumar et al., 2020).
Fujimoto & Gu (2021) propose TD3+BC, a simple ofﬂine RL algorithm that regularizes policy learning in TD3 with a behavior cloning loss that constraints the policy actions to stay close to the actions in the ofﬂine dataset D .
This is achieved by adding a behavior cloning term to the policy loss:
[Equation]
where α is a weighing hyperparameter and
[Equation]
normalizes the Q values which help in balancing both losses.
The sum in the denominator is taken over a mini-batch and the gradients do not ﬂow through the critic term in the denominator.
O NLINE F INE - TUNING
RL agents trained from ofﬂine data tend to have limited performance and would further need to be ﬁne-tuned online by interacting with the environment.
During online ﬁne-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from ofﬂine data to online data.
Keeping the constrain used in ofﬂine pre-training, such as in Equation 1, could mitigate the collapse.
However, this will force the policy to stay close to the behavior policy (used to collect the dataset), thus leads to slow improvement.
In this section, we describe the two components of our online-tuning algorithm that enables stable and sample-efﬁcient online ﬁne-tuning.
4. A DAPTIVE W EIGHING OF B EHAVIOR C LONING L OSS
The most straightforward way to ﬁne-tune the pre-trained policy is by just removing the constrains used in ofﬂine pre-training.
For example, Balanced Replay (Lee et al., 2021) uses CQL (Kumar et al., 2020) during ofﬂine pre-training and uses SAC (Haarnoja et al., 2018) in ﬁne-tuning.
However, this strategy often leads to a performance collapse at the beginning of ﬁne-tuning, as shown in
Fig.
1 (with α = 0 ) and the TD3-ft in Fig.
In the TD3+BC algorithm we consider in this paper, an α hyperparameter is used to balance the RL objective and the behaviour cloning term which constrains the policy to stay close to the behavior policy (see Equation 1).
We use α ofﬂine and α online to distinguish the α hyperparameter value used during ofﬂine and online training respectively.
By default, we use α ofﬂine = 0 .
4 in all our experiments.
We use α online = 0 .
4 for TD3+BC
and we observe that this prevents sudden performance drops at the initial steps of online ﬁne-tuning, at the cost of very slow learning due to the strong behavior cloning constraint.
On the other hand, setting α online = 0 leads to sample-efﬁcient learning on some tasks at the cost of complete instability in other tasks.
This is due to the sudden distribution shift causing the policy network to change signiﬁcantly.
In Fig.
1, we present the inﬂuence of α online on the TD3+BC during ﬁne-tuning by trying different values of α online from [0 . 0 , 0 . 1 , 0 . 3] .
We can clearly see that using the behavior cloning loss with proper α online enables stable ﬁne-tuning.
However, the value of α online depends on the quality of the ofﬂine dataset and has signiﬁcant inﬂuence of the ﬁne-tuning performance.
For example, α online =ﬁts well on the Hopper-Random task while causes immediate collapse on Hopper-Medium and
Hopper-Medium-Expert tasks.
In our experiments, we found that when the ofﬂine dataset has narrow distribution or when the policy has already converged to a desired performance (comparable to the expert), it is usually beneﬁcial to maintain a higher α online .
When the data distribution is broader or when we still need to improve the agent by a large margin, a smaller α online works better.
During experiments, we can not ﬁnd a single α online that is suitable for all tasks and its value needs to be tuned carefully per tasks, which makes this method hard to be used in practice.
To solve this problem, we propose to adapt the weight of the behavior cloning loss according to two factors: (i) the difference between current episodic return and the target return, and (ii) the episodic
[Figure]
return between current episode and the last episode.
We adaptively change the α online hyperparameter as:
[Equation]
where we constrain α online between 0 and 0.4 (the value used during ofﬂine pre-training).
R current and
R last are normalized following the return normalization procedure used in D4RL.
R target is the target episodic return, which we set as 1 (corresponding to the expert policy) for all tasks.
K P controls how fast we decrease the α online according to current performance and K D determines how fast we increase the α online when the performance drops.
Intuitively, when the agent’s performance reaches the target episodic return, we try to maintain it during ﬁne-tuning.
But when the agent’s performance is low, we decrease the α online to allow the agent improving further.
The second term increases the α online when performance drops during training to mitigate performance collapse.
Equation 2 allows for adaptive weighing of the behavior cloning loss throughout online ﬁne-tuning.
This learning algorithm can automatically adjust the constraint enforced by the behavior cloning loss.
After ofﬂine pre-training the replay buffer is ﬁlled with ofﬂine samples and during online ﬁnetuning, they are slowly replaced by online samples.
Uniformly sampling mini-batches from this replay buffer for online ﬁne-tuning is inefﬁcient as it is dominated by ofﬂine samples.
After ofﬂine learning we simply remove 95% of random ofﬂine samples from the replay buffer to deal with this problem.
Our results show that the data down-sampling allows efﬁcient usage of novel data without destroying the training.
4. R ANDOMIZED E NSEMBLES OF C RITIC N ETWORKS
We propose to use an ensemble of Q functions to better deal with the distribution shift from ofﬂine pre-training and to improve the sample-efﬁciency of online ﬁne-tuning.
We use the Randomized
Ensembled Double Q-learning (REDQ) method proposed by Chen et al.
(2021) to learn an ensemble of critic networks.
The critic network is trained to satisfy the Bellman equation:
[Equation]
REDQ maintains an ensemble of N critic networks and randomly samples M networks for each critic update.
Given a mini-batch B of B transitions ( s , a , r, s (cid:48) ) , all critic networks in the ensemble are updated towards the same target:
[Equation] [Equation]
where M is a random subset of M critic networks and a (cid:48)
=
clip ( π θ ( s t +1 ) + (cid:15), a low , a high ) .
Here (cid:15) is Gaussian exploration noise with standard deviation σ policy and [ a low , a high ] is the action range.
REDQ updates the policy network to maximize the average predictions of the critic networks:
[Equation]
Algorithm
Initialize REDQ agent with critic parameters φ 1 , . .
. , φ N and policy parameters θ
Initialize target parameters θ (cid:48) ← θ and φ (cid:48)
i ← φ i , for i = 1 , . . .
, N
Initialize replay buffer R with ofﬂine data D for k = 0 to K do
Sample mini-batch B of B transitions ( s , a , r, s (cid:48) ) from R
Update critic parameters φ 1 , . .
. , φ N using Equation
Update actor parameters θ using Equation 4 with α = α ofﬂine
Update target networks θ (cid:48)
← τθ + (1 − τ ) θ (cid:48) and φ (cid:48)
i ← τφ i + (1 − τ ) φ (cid:48)
i end for
Randomly remove 95% of ofﬂine samples from R
Initialize α online =
α ofﬂine
Initialize R current and R last to store the return of current and previous episodes
Initialize environment for online ﬁne-tuning for every training episode do for t = 0 to T do
Act with exploration noise a t ∼ π θ ( s t ) + N (0 , σ expl )
Observe next state s t +1 and reward r t
Add ( s t , a t , r t , s t +1 ) to R for g = 0 to G do
Sample mini-batch B of B transitions ( s , a , r, s (cid:48) ) from R
Update critic parameters φ 1 , . .
. , φ N using Equation
Update actor parameters θ using Equation 4 with α = α online
Update target networks θ (cid:48)
← τθ + (1 − τ ) θ (cid:48) and φ (cid:48)
i ← τφ i + (1 − τ ) φ (cid:48) i end for end for
Set R last =
R current and R current =
(cid:80) Tt =
r t
Adapt α online based on R last and R current using Equationend for
We combine this REDQ policy update with a behaviour cloning loss (like in Equation 1) for robust learning (Fujimoto & Gu, 2021):
[Equation]
We show that this simple modiﬁcation of ensembling the critic networks (which can be run in parallel) improves ofﬂine-to-online learning.
We call this algorithm REDQ+AdaptiveBC.
Our algorithm is outlined as Algorithm 1.
E XPERIMENTS 5. O NLINE F INE - TUNING ON D4RL BENCHMARK
The goal of our experiments is to evaluate the stability and sample-efﬁciency of the proposed algorithm on online ﬁne-tuning after ofﬂine pre-training on datasets of different quality.
We evaluate our algorithm on online ﬁne-tuning after ofﬂine pre-training on the D4RL benchmark (Fu et al., 2020).
D4RL includes three locomotion tasks (halfcheetah, hopper, and walker) implemented in the
MuJoCo simulator (Todorov et al., 2012), wrapped in OpenAI Gym API (Brockman et al., 2016).
D4RL provides ﬁve different ofﬂine datasets for each task: Random, Medium, Medium-Replay,
Medium-Expert, and Expert.
The Random datasets are collected by random policies, Medium datasets are collected by an early-stopped soft actor-critic (SAC) (Haarnoja et al., 2018) agent with medium-level performance, Medium-Replay datasets consist of all samples in the replay buffer after training a medium-level agent, Medium-Expert datasets are mixed with expert demonstrations and sub-optimal demonstrations from a medium-level agent, and Expert datasets are expert demonstrations.
The “expert” in these datasets is a fully trained soft-actor critic agent.
We ignore the Expert
[Figure]
datasets in this paper as ofﬂine RL algorithms already achieve expert-level performance on these tasks and there is little to no beneﬁt in online ﬁne-tuning.
In Figure 2, we compare our REDQ+AdaptiveBC algorithm with two state-of-the-art ofﬂine-toonline RL algorithms (AWAC and Balanced Replay) and two baseline methods (TD3-ft and REDQ):
• Advantage Weighted Actor-Critic (AWAC)
(Nair et al., 2020) is an actor-critic method for ofﬂine-to-online RL that implicitly constraints the policy network to stay close to the behavior policy.
We produce the results for AWAC using code taken from https:// github.com/ikostrikov/jaxrl .
• Balanced Replay (Lee et al., 2021) is an ofﬂine-to-online RL method that prioritizes nearon-policy samples from the replay buffer.
This method also uses an ensemble of Q functions to prevent overestimation of Q values in the initial stages of online ﬁne-tuning.
We reproduced the results for this method using our own implementation.
For a fair comparison, we base our implementation on TD3+BC (instead of CQL originally used by Lee et al.
(2021)) while ensuring that we are able to reproduce the original results.
• TD3-ft is the standard TD3 algorithm (Fujimoto et al., 2018)
that was pre-trained ofﬂine using TD3+BC (Fujimoto & Gu, 2021).
• REDQ (scratch)
(Chen et al., 2021) is an RL method trained from scratch, without any access to the ofﬂine data.
This baseline emphasizes the importance of ofﬂine pre-training and online ﬁne-tuning.
We base our REDQ implementation on TD3 (instead of SAC used by Chen et al.
for compatibility with TD3+BC.
All methods (except AWAC) are implemented on top on TD3 and are run from the same codebase for a fair comparison.
For simplicity, we do not perform any state normalization like in the original
TD3+BC implementation (Fujimoto & Gu, 2021).
During ofﬂine pre-training, all algorithms are pre-trained on the ofﬂine dataset for one million gradient steps.
After pre-training, we ﬁne-tune the agents for 250,000 time steps by interacting with the environment.
We evaluate the agent every 5000 time steps and each evaluation consists of episodes.
We attain performance competitive to the state-of-the-art in this benchmark with our method stably improving the performance during online ﬁne-tuning.
We signiﬁcantly outperform all others methods in the HalfCheetah domain for three datasets.
We perform slightly worse than Balanced Replay on Walker2d-Medium and Walker2d-Medium-Replay but outperform it or get similar results in all other cases.
We need to mention that in both Walker2dMedium and Walker2d-Medium-Replay tasks, our method already reaches the target performance (predeﬁned following D4RL), and the α is increased automatically to maintain the performance and thus fail to improve further.
We signiﬁcantly outperform REDQ on all tasks, which demonstrates that we considerably beneﬁt from ofﬂine pre-training.
TD3-ft is able to improve from online ﬁnetuning but suffers from signiﬁcant performance drops due to the sudden distribution shift and the learning progress is slow due to the replay buffer being dominated by ofﬂine samples.
It should be noticed that, unlike other methods, our algorithm does not collapse immediately on all three
Medium-Expert tasks.
Both Balanced Replay and our method (REDQ+AdaptiveBC) use an ensemble of 10 Q networks, but in different ways.
Balanced Replay maintains a pair of ﬁve ensemble networks, average the predictions across each of the ﬁve networks and then takes the minimum of the averages as the ﬁnal prediction.
In our method, we simply consider the average of all 10 networks as the prediction but randomly sample a pair of Q networks to compute the critic targets (Equation 3).
We show that this simple modiﬁcation enables stable and sample-efﬁcient online ﬁne-tuning without the need for any complex sampling scheme from the replay buffer.
Similar to prior works (Fujimoto & Gu, 2021), we use feed-forward networks with two hidden layers as actor and critic networks for all the methods.
We use a batch size of 256 to train the network for all methods, except for AWAC where we use a larger batch size of 1024 (Nair et al., 2020).
During ofﬂine learning, we use α ofﬂine = 0 .
4 for all tasks, except Walker-Random where we use α ofﬂine = 100 since the dataset has a very narrow distribution.
We list all the hyperparameters used in our experiments in Table 1 in the Appendix.
5. A LGORITHMIC I NVESTIGATIONS
Adaptive Weighing of α online :
To evaluate whether the proposed method can correctly select a good α online for stable online ﬁne-tuning, in Figure 3, we compare the results obtained with the automatically tuned α online with manually tuned results.
To manually tune the α online , for each domain and each dataset, we do a grid search on α online over [0 . 0 , 0 . 1 , 0 . 2 , 0 . 3] and pick the best α online separately for each task.
We can see that with manually tuned α online , our method consistently outperformances other methods in Figure 2, except the Balanced Replay on Walker2d-Medium.
Our results with automatically tuned α online are slightly worse than manually tuned results on HalfCheetah tasks.
However, our method successfully ﬁnd the similar α online on Halfcheetah tasks as we manually selected after roughly 10episodes.
On the rest tasks, our automatically tuned results are competitive to the carefully picked results but saving lots of labor and computational resources.
Ofﬂine Dataset Downsampling : Balanced Replay (Lee et al., 2021) trains a neural network to estimate the priority of samples from ofﬂine data and online data.
In their methods, three replay buffer need to be maintained: ofﬂine dataset (0.1-2M samples), online dataset (0.25M samples) and a prioritized replay buffer (0.35M-2.25M samples) (Schaul et al., 2015), making it memory consuming (0.7M-4.5M samples).
Our method simply dowsamples the ofﬂine dataset by 95% , thus, our method only maintains one replay buffer to store online data but is preﬁlled with 0.05M ofﬂine data points, roughly saves 65% − 95% memory.
To demonstrate the effectiveness of dataset downsampling, we compare TD3 ft with and without dataset downsampling on three random datasets, shown in Figure 4.
Our results show that the downsampling procedure allows the agent effectively sampling the novel data encountered during ﬁne-tuning.
This is even important when the data quality of the ofﬂine data is not good enough, such as when the dataset is collected by a random policy.
[Figure] [Figure]
C ONCLUSION
We consider the problem of ofﬂine-to-online RL where an agent is ﬁrst pre-trained on ofﬂine data collected (by a possibly unknown behavior policy) and the agent is then ﬁne-tuned online by interacting with the environment.
This is desirable as pre-trained agents may have limited performance depending on the quality of the ofﬂine dataset.
Ofﬂine-to-online RL is challenging due to the sudden distribution shift from ofﬂine data to online data, and also the constraints enforced by ofﬂine
RL algorithms (such as a behavior cloning loss) during pre-training.
In this paper, we propose a simple mechanism to adaptively weight a behavior cloning loss during online ﬁne-tuning, based on agent performance and training stability.
We demonstrate that a randomized ensemble further helps to deal with these challenges to enable sample-efﬁcient online ﬁne-tuning performance.
We attain performance competitive to the state-of-the-art online ﬁne-tuning methods on locomotion tasks from the popular D4RL benchmark. A A BLATION ON ENSEMBLES In our experiments [Table]
we use ensembles to represent the critic network.
In [Table] we compare the training results with and without ensembles as well as the way to use ensembles. Our results show that using ensembles is not necessary to avoid performance collapse, however, it stabilizes training in most cases.
Also, the way to use ensembles matters.
We compare calculating target Q values with randomly sampled Q predictions and with minimal predictions.
Our results show that using minimal
Q predictions to calculate target Q values hurts the performance in most cases
[Table]
B H YPERPARAMETERS
All the hyperparameters and network architectures used in experiments are listed in Table 1.