A DAPTIVE Q- LEARNING FOR I NTERACTION -L IMITED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when an online interaction is costly.
Ofﬂine RL provides an alternative solution by directly learning from the logged dataset.
However, it usually yields unsatisfactory performance due to a pessimistic update scheme or/and the low quality of logged datasets.
Moreover, how to evaluate the policy under the ofﬂine setting is also a challenging problem.
In this paper, we propose a uniﬁed framework called Adaptive Q-learning for effectively taking advantage of ofﬂine and online learning.
Speciﬁcally, we explicitly consider the difference between the online and ofﬂine data and apply an adaptive update scheme accordingly, i.e., a pessimistic update strategy for the ofﬂine dataset and a greedy or no pessimistic update scheme for the online dataset.
When combining both, we can apply very limited online exploration steps to achieve expert performance even when the ofﬂine dataset is poor, e.g., random dataset.
Such a framework provides a uniﬁed way to mix the ofﬂine and online RL and gain the best of both worlds.
To understand our framework better, we then provide an initialization following our framework.
Extensive experiments are done to verify the effectiveness of our proposed method.
I NTRODUCTION
Conventional online reinforcement learning (RL) methods (Haarnoja et al., 2018; Fujimoto et al., 2018) usually learn from experiences generated by interactions with the online environment.
They are impractical in some real-world applications, e.g., dialog (Jaques et al., 2019) and education (Mandel et al., 2014), where interactions are costly.
Recently, ofﬂine RL (Levine et al., 2020) has aroused much attention.
It targets the above challenge by making the agent learn from an ofﬂine dataset collected by other policies in a purely data-driven manner.
The difference between online
RL and ofﬂine RL is shown in Figure 1.
Existing ofﬂine RL studies try to target the distribution mismatch or out-of-distribution actions issue by employing a pessimistic update scheme (Kumar et al., 2019; 2020) or in combination with imitation learning (Fujimoto et al., 2019).
However, when the dataset is ﬁxed, ofﬂine RL cannot learn the optimal policy (Kidambi et al., 2020), and even worse, when the dataset’s quality is poor, ofﬂine RL usually gains a relatively bad performance (Kumar et al., 2020; Fu et al., 2020; Levine et al., 2020).
On the other hand, it is challenging to evaluate the learned policy when learning totally from the ofﬂine dataset.
Even though some research topics, e.g., off-policy evaluation (Dann et al., 2014), study how to evaluate the learned policy without the interaction with the environment, it is still not ideal for the practical purpose.
Some recent works try to address the above issues by employing an ofﬂine-online setting.
Such methods (Lee et al., 2021; Nair et al., 2020) focus on pre-training a policy using the ofﬂine dataset and ﬁne-tuning the policy through further online interactions.
Even though their methods alleviate the above issues to some extent, their main bottleneck is that they do not consider the different characteristics of ofﬂine and online data.
For instance, pre-existing ofﬂine data can prevent agents from converging prematurely due to the potential diverse ofﬂine dataset, while online data can improve stability and accelerate convergence.
Generally, these different data are mixed and used by a pessimistic strategy to update the policy in their methods, which may be problematic since using a pessimistic strategy for online data may harm the policy performance (Nair et al., 2020).
Moreover, with sufﬁciently large and diverse ofﬂine data, a high-performing policy can be learned just using a pure online RL algorithm (Agarwal et al., 2020). And the online near-on-policy data also play a key role in improving the RL algorithm’s stability (Fujimoto et al., 2019).
Hence, we should take full advantage of both ofﬂine and online dataset.
To tackle the above problems, in this paper, we emphasize that online and ofﬂine RL should be coupled organically .
First, a separate updating strategy should be employed for online and ofﬂine data, respectively, considering their different characteristics.
To do so, we present a framework called adaptive Q-learning that integrates the advantage of ofﬂine learning and online learning effectively.
When learning from the ofﬂine dataset, we conduct a pessimistic update strategy.
In contrast, we use a greedy or non-pessimistic update strategy when learning from the online dataset.
Second, we design a novel replay buffer to distinguish the ofﬂine from online datasets in a simple way.
By utilizing such a novel framework and buffer design, the agent can achieve an expert policy using limited online interaction steps regardless of the quality of the ofﬂine dataset.
In the experiments, our proposed framework can achieve better performance by using only one ﬁfth number of interactions compared with the previous method (Nair et al., 2020).
Our contributions can be summarized as below:
• We propose a uniﬁed framework called Adaptive Q-learning that can effectively beneﬁt from both the ofﬂine dataset and limited number of online interaction data.
• Based on the general framework, we initialize a practical algorithm, called Greedyconservative Q-ensemble learning (GCQL) that builds on top of State-of-the-Art ofﬂine RL and online RL method.
• We empirically verify the effectiveness of our method by comprehensive experiments on the popular continuous control tasks MuJoCo (Todorov et al., 2012) with ofﬂine dataset coming from D4RL (Fu et al., 2020).
R ELATED W ORK
Online RL In general, online RL algorithms can be divided into two categories, i.e., on-policy and off-policy algorithms.
On-policy methods (Schulman et al., 2015; 2017) update the policy using data collected by its current behavior policy.
As ignoring the logged data collected by its history behaviour policies, they usually have a lower sample efﬁciency than the off-policy RL.
On the other hand, off-policy methods (Fujimoto et al., 2018; Chen et al., 2021) enable the policy to learn from experience collected by history behavior polices, however, they cannot learn well from history trajectories collected by other agents’ behavior policies (Fujimoto et al., 2019; Kumar et al., 2020).
Consequently, the need for huge online interaction makes online RL impractical for some real-world applications,such as dialog agents (Jaques et al., 2019) or education system (Mandel et al., 2014).
Ofﬂine RL Ofﬂine RL algorithms assume the online environment is unavailable and learn policies only from the pre-collected dataset.
As the value estimation error cannot be corrected using online interactions here, these methods tend to utilize a pessimistic updating strategy to relieve the distribution mismatch problem (Fujimoto et al., 2019; Kumar et al., 2019).
To implement such a strategy, model-free ofﬂine RL methods generally employ value or policy penalties to constrain the updated policy close to the data collecting policy (Wu et al., 2019; Kumar et al., 2020; Fujimoto et al., 2019;
He & Hou, 2020).
And model-based methods use predictive models to estimate uncertainties of states and then update the policy in a pessimistic way based on them (Kidambi et al., 2020; Yu et al., 2020).
Those ofﬂine RL methods cannot guarantee a good performance, especially when the data quality is poor (Kumar et al., 2020).
Besides, policy evaluation when the online environment is unavailable is also challenging.
Even though off-policy evaluation (OPE) methods (Dann et al., 2014) present alternative solutions, they are still far from perfect.
Above issues of online and ofﬂine RL motivate us to investigate the ofﬂine-online setting.
Ofﬂine-online RL Lee et al.
(2021) and Nair et al.
(2020) focus on the mixed setting where the agent is ﬁrst learned from the ofﬂine dataset, and then trained online.
Nair et al.
(2020) propose an advantage-weighted actor-critic (AWAC) method that restricts the policy to select actions close to those in the ofﬂine data by an implicit constraint.
When online interactions are available, such conservative constraint may have adverse effects on the performance.
Lee et al.
(2021) employ a balanced replay scheme to address the distribution shift issue.
It uses the ofﬂine data by only selecting near-on-policy samples.
Unlike these two works, our method utilizes all online and ofﬂine data, and explicitly considers the difference between them by adaptively applying non-conservative or conservative updating schemes, respectively .
Matsushima et al.
(2021) focuses on optimizing deployment efﬁciency, i.e., the number of distinct data-collection policies used during learning, by employing a behavior-regularized policy updating strategy.
Although in terms of deployment efﬁciency, their work is between online and ofﬂine RL, it ignores existing ofﬂine dataset, and dose not focusing on improving sample efﬁciency, while both are addressed in our paper.
Some works (Zhu et al., 2019; Vecerik et al., 2017; Rajeswaran et al., 2018; Kim et al., 2013) can also learn from online interactions and ofﬂine data.
However, they need expert demonstrations instead of any dataset, and this limits their applicability.
P RELIMINARIES
In RL, the interaction between the agent and environment is usually modelled using Markov decision process (MDP) ( S , A , p M , r, γ ) , with state space S (state s ∈ S ), action space A (action a ∈ A ).
At each discrete time step, the agent takes an action a based on the current state s , and the state changes into s (cid:48) according to the transition dynamics p M ( s (cid:48) | s, a ) , and the agent receives a reward r ( s, a, s (cid:48) )
∈ R .
The agent’s objective is to maximize the return, which is deﬁned as
R t = (cid:80) ∞
i =
t +1 γ
i r
( s i , a i , s i +1 ) , where t is the time step, and γ ∈ [0 , 1) is the discounted factor.
The mapping from s to a is denoted by the stochastic policy π : a ∼ π ( ·| s ) .
Policy can be stochastic or deterministic, and we use the stochastic from in this paper for generality.
Each policy π have a corresponding action-value function Q π ( s, a ) =
E π
[ R t | s, a ] , which is the expected return following the policy after taking action a in state s .
The policy π ’s action-value function can be updated by the Bellman operator T π :
[Equation]
Q-learning (Sutton & Barto, 2011) directly learns the optimal action-value function Q ∗ ( s, a )
=
max π Q π ( s, a ) , and such Q-function can be modelled using neural networks (Mnih et al., 2015).
In principle, off-policy methods, such as Q-learning, can utilize experiences collected by any policies, and thus they usually maintain a replay buffer B to store and repeatedly learn from experiences collected by behavior policies (Agarwal et al., 2020).
Such capability also enables off-policy methods to be used in the ofﬂine setting, by storing ofﬂine data into the buffer B , and not updating the buffer during learning since no further interactions are available here (Levine et al., 2020).
But this simple adjusting cannot guarantee the agent to have a reasonable performance, especially when the dataset is not diverse (Kumar et al., 2020; Agarwal et al., 2020; Fujimoto et al., 2019), and this is also the problem tackled in most ofﬂine RL works.
In this paper, we focus on the ofﬂine-online setting, where the agent is ﬁrst learned from the ofﬂine dataset, and then trained via online interactions .
And without additional remarks, online RL methods refer to off-policy algorithms in the rest of this paper.
We only use off-policy methods because they can make more use of ofﬂine data than on-policy ones for gaining high sample efﬁciency, and on-policy methods are not compatible with our proposed framework introduced next.
M ETHODOLOGY
In this section, we ﬁrst give an illustrative example to explain our motivation.
After that, we introduce the proposed general framework, namely Adaptive Q-learning, trying to couple online and ofﬂine RL in an organic way.
To implement the proposed framework, then we present the OnlineOfﬂine Replay Buffer, targeting retaining and distinguishing online and ofﬂine data simultaneously.
And ﬁnally we incorporate State-of-the-Art (SotA) online and ofﬂine RL algorithms into the framework, and introduce the proposed Greedy-Conservative Q-ensemble Learning algorithm in detail.
4. A N ILLUSTRATIVE EXAMPLE
We test the SotA ofﬂine RL method conservative Q-learning (CQL) (Kumar et al., 2020) and online RL method Randomized Ensembled Double Q-learning (REDQ)
(Chen et al., 2021) under the ofﬂine-online setting.
Here, we choose a widely used locomotion task Hopper (Todorov et al., 2012), and use the dataset hopper-medium-replay-v0 in the D4RL benchmark (Fu et al., 2020) as the ofﬂine dataset, which contains diverse experiences from different policies.
Speciﬁcally, we ﬁrst pre-train the agent with the ofﬂine dataset for 100 K steps.
Then, the agent is ﬁne-tuned online by alternately conducting the interaction and updating process, where the agent interacts with the environment for 1 K steps, and is updated for 10 K steps.
Such interleaving process ends until the total online interaction steps reach 90 K .
The ofﬂine-online setting in this task should be a favourable one for policy learning, because both diverse ofﬂine data and online interactions are available here.
Therefore, we expect following results: the initial ofﬂine training using the dataset will provide a relatively good but not perfect starting point for the agent, and then the agent can be further improved by the online alternating process since online interaction data can be obtained, and ﬁnally we may acquire a well-performed policy, even with normalized score close to or better than 100 (i.e., the performance of an expert).
Nonetheless, experiment results are shown in Figure 2, and they do not totally meet our expectation.
Speciﬁcally, starting points of two curves show that the ofﬂine algorithm CQL and online algorithm
REDQ can both obtain normalized scores greater than 0 (i.e., the performance of a random policy) but not very high after the initial ofﬂine training process.
This result is what we expected, because CQL is designed for the ofﬂine setting, and online off-policy methods, such as REDQ, can also learn from the diverse and large ofﬂine dataset even though it is ﬁxed (Agarwal et al., 2020).
However, although the starting scores (below 30 ) are far from the expert score and leave a large room for further promoting, both algorithms have troubles in the following online process.
REDQ suffers from severe instability issue and its performance drops signiﬁcantly during online learning, and in the end, the policy almost degenerates into a random one.
One the other hand, even though the CQL agent can keep stable during learning, its improvement is very limited.
These results indicate that pure online RL algorithm may be problematic for effectively handling ofﬂine data and online interaction data in a single training process, and pure ofﬂine RL algorithm cannot make good use of valuable online interaction data due to its conservative updating strategy.
Such observation motivates us to couple them in an organic way.
4. A DAPTIVE Q- LEARNING FRAMEWORK
In this subsection, we introduce our proposed framework.
The underlying idea is simple and can be described as follows.
When the agent learns from online and near-on-policy interaction data, we choose a more greedy or no-pessimistic updating strategy, since these data reﬂect the truth situation of the current policy.
By contrast, when data are sampled from the ofﬂine dataset, we tend to use a more pessimistic updating strategy.
Through such an adaptive way, we can make full use of both online and ofﬂine data, and explicitly consider their differences by separately applying suitable updating schemes.
Then, we give a formalization for above intuition based on Q-learning, and call this framework Adaptive Q-learning .
The updating function of this framework can be deﬁned by the following equation:
[Equation]
This function consists of two terms.
The ﬁrst term A ( Q ) stands for the greedy updating strategy, which is a regular updating function for Q-value in online RL algorithms, e.g., the bellman error.
The second term B ( Q ) stands for the pessimistic updating strategy, which is the value penalty, e.g., the Q-value regularizer (Wu et al., 2019).
Besides, a weight function W ( s, a ) is applied to the penalty term B ( Q ) .
This weight function is based on the sampled data type.
Speciﬁcally, when we use online and near-on-policy interaction data, W ( s, a ) will be a smaller value, and the updating relies more on the objective of online RL, leading to a relatively greedy strategy.
On the contrary, when we use ofﬂine data, W ( s, a ) will be a bigger value, and the updating strategy is relatively pessimistic.
Remark: Our behind intuition can also be easily formalized via the policy learning objective.
Such variant of our framework can be denoted by π k +1 ← arg max π k (cid:2)
A (cid:0) π k (cid:1) + W ( s, a ) B (cid:0) π k
(cid:1)(cid:3) ,
where
A ( π k ) is a objective for the policy in online RL and B ( π k ) is a policy penalty term.
We also provide a implementation for this variant based on TD3+BC algorithm (Fujimoto & Gu, 2021), and our framework can largely boost its performance within limited environment steps in most tasks (see
Appendix D for details).
4. OORB: O NLINE -O FFLINE R EPLAY B UFFER
Next, we introduce a simple but effective online-ofﬂine replay buffer (OORB) to distinguish between near-on-policy online interaction data, and the ofﬂine data.
OORB consists of two replay buffers.
One is the online buffer that collects the online interaction data.
Besides, to ensure the data in the online buffer is near-on-policy, we set it to be very small, and fresh online interaction data are stored into it by following a ﬁrst-in-ﬁrst-out rule. The other is the ofﬂine buffer consisting of the newly generated online interaction data and the ofﬂine dataset which may come from any policies.
Data are sampled from OORB following a Bernoulli distribution, which means that with a probability p , they are sampled from the online buffer, and with probability 1 − p , they are sampled from the ofﬂine buffer.
To beneﬁt from both online and ofﬂine data in a balanced way, we empirically set p to 0.5, and its effect on the ﬁnal performance is further tested via ablation studies in Section 5.4.
Results show that p = 0 .
5 works best overall, which conﬁrms our claim that ofﬂine data and online interaction data are both crucial for policy learning.
4. GCQL: G REEDY -C ONSERVATIVE Q- ENSEMBLE L EARNING
We then present a detailed implementation of our proposed framework, by incorporating SotA ofﬂine RL algorithm CQL and online RL algorithm REDQ, and we call our implemented algorithm
Greedy Conservative Q-ensemble Learning (GCQL).
Speciﬁcally, we use the updating function in REDQ as the ﬁrst term A ( Q ) in Equation 2, and use the conservative regularizer in CQL as the
1 Initialization : 2 Initialize policy π φ , ensemble Q functions Q
θ i , i ∈ N ,ofﬂine training steps t 3 Online exploration steps T on , ofﬂine update steps T off 4 Initialize T initial , sample possibility p , start sampling steps: T s 5 Initialize online buffer B on to empty, ofﬂine buffer B off ← ofﬂine dataset 6 Initialize online buffer size S on ←7 Initial ofﬂine learning : 8 Train the agent for T initial steps using the logged dataset 9 while True do t ←
Explore T on steps online
Store the T on steps experiences to both online buffer B on and ofﬂine buffer B off
S on ← S on + T on for t < T off do
Sample a random value p s ∼ U (0 , 1) if p s < p and S on > T s then
Sample a batch ( s, a ) from online buffer B on
Set the W ( s, a ) to end else
Sample a batch ( s, a ) from ofﬂine buffer B off
Set the W ( s, a ) to end
Update the Q functions by Equation
Update the policy by Equation t += end 28 end second term B ( Q ) , and the updating function can be presented by the following equation:
[Equation]
where the action ˙a is sampled from current policy, i.e., ˙a ∼ π φ ( ·| s ) , s ∼ D OORB and D OORB is the OORB replay buffer. ˆ B π ˆ Q k ( s , a ) is deﬁned by
[Equation]
We randomly select two Q functions from the ensemble Q functions and the M represents the selected Q functions’ index just following the REDQ’s setting.
The ˆ Q stands for a target Q function for stabilizing the learning process (Mnih et al., 2015).
The update function of policy is deﬁned by:
[Equation]
When sampled data is from the online replay buffer, we set W ( s, a ) to 0, otherwise 1, and this is why we use the term ”greedy-conservative” to describe our algorithm.
In another word, we greedy exploit the near-on-policy online data by the regular online RL scheme without any conservative regularizer.
On the contrary, we conservatively exploit the ofﬂine data by employing the ofﬂine RL
[Table]
regularizer. Formally, this strategy can be explained by Deﬁnition 6:
[Equation]
if ( s, a ) is sampled from online buffer otherwise (6)
Algorithm 1 summarizes our proposed method.
And we also explain the main steps in the Algorithm as follows.
Firstly, We ﬁrst learn from the existing ofﬂine data for T initial steps to leverage them.
To make good use of the ofﬂine data, we usually set a big value for T initial , e.g., 100 K .
Secondly, We begin the following interleaving learning steps.
We conduct the online exploration process for T on steps and store the new experiences to OORB.
We set the online exploration steps T on to a small value, e.g., 1 K .
In contrast, the ofﬂine update step T off is set to be larger than T on , e.g., 10 K .
We sample a batch from our OORB and update the policy and Q-functions.
If the sampled batch comes from the online buffer, then we set the weight value W ( s, a ) to 0, otherwise 1.
The above interleaving learning process is repeated till the end.
E XPERIMENTS
In this section, we design experiments to verify the effectiveness of our method from three perspectives: (1) the performance superiority compared with other baselines; (2) ablation studies to test the effect of each component used in our method.
(3) the inﬂuence of different hyper-parameters.
5. S ETTINGS
All experiments were done on the continuous control benchmark MuJoCo
(Todorov et al., 2012), and the ofﬂine dataset comes from the popular ofﬂine RL benchmark D4RL (Fu et al., 2020).
To make interaction limited, we set the number of online interaction steps for each iteration to a small value, i.e., 1 K .
To better exploit the ofﬂine dataset, we set the number of ofﬂine training steps to a large value, i.e., 100 K for T intial , and 10 K for T off .
The training process ends until the number of all online interaction steps reach 90 K , and the number of all ofﬂine updating steps reach 1 M .
For OORB, we set p = 0 . 5 as described in Section 4.3.
The size of the online buffer is set to 20 K , and the size of the ofﬂine buffer is set to 3 M .
For other hyper-parameters, we follow the default setting in baselines, except for the number of the ensemble Q, which is 5 in our experiments.
The above conﬁgurations keep same across all tasks.
As our main purpose is to present a new framework instead of gaining SotA performance, we do not ﬁne-tune these hyper-parameters for each task.
[Figure]
5. O VERALL PERFORMANCE
4 methods, i.e., GCQL (Ours), CQL, REDQ and AWAC, are tested on 3 tasks, i.e., Walker, Hopper and HalfCheetah, and each task has 5 different kinds of ofﬂine dataset, which are random-v0, medium-v0, expert-v0, medium-expert-v0, and medium-replay-v0.
As online interactions are available, we take the maximization over testing scores during the whole training process, and report the average of these max scores across ﬁve different seeds in Table 1.
Please note that the results of AWAC are directly taken from their paper (Nair et al., 2020).
As shown in Table 1, our method gains a better performance than the baselines in all tasks.
Particularly, for the medium-replay and random dataset, our GCQL outperforms the baselines by a large margin.
By contrast, CQL also achieves an expert performance on the high-quality dataset, e.g., expert and medium-expert tasks.
For these high-quality datasets, our method achieves a comparable or slightly better performance over CQL.
In terms of the online RL method REDQ, it fails on almost all tasks.
Even though we employ an online limited exploration process, conventional online RL cannot beneﬁt from such limited experiences.
On the contrary, such limited online experiences could bring catastrophic consequences, e.g., on the halfcheetah-medium-replay task, REDQ suffers a signiﬁcant instability issue.
On the other hand, when dataset quality is high, e.g., expert or medium-expert dataset, our method can leverage the beneﬁt of the ofﬂine RL that gains an expert performance with very limited interaction steps, such as around 20 K steps.
By contrast, when dataset quality is poor, e.g., the random or medium-replay dataset, only our method can learn effectively with less than 100 K online steps, which demonstrate that our method can take the advantage of the limited online experiences as much as possible.
In sum, our method can beneﬁt the most from the limited online experience while still maintain the ofﬂine learning’s ability.
Besides, We also include the corresponding learning curves in Appendix
A for a comprehensive understanding.
These learning curves demonstrate that our method also achieves a better ﬁnal average score at the last iteration in most cases.
5. A BLATION STUDIES
To investigate each component’s effect in our method, we conduct the following ablation studies.
To do this, we design four variants of our method.
GCQL WE : GCQL without the ensemble where we only use two Q functions same as the setting of CQL.
GCQL WO : GCQL without the online replay buffer where we only sample from the ofﬂine buffer.
GCQL WG : GCQL without the greedy strategy where we ﬁx the weighed W to 1. GCQL WC : GCQL without the conservative term where we ﬁx the W to 0.
We test all 5 kinds of ofﬂine dataset for the task Walker2d, and results averaged over three different random seeds are shown in Figure 3.
First, from Figure 3, it is easy to deduce that when the quality of data is high, i.e., including the expert dataset, almost all variants perform well except the GCQL WC.
The underlying reason is obvious as we ﬁrstly pre-train the policy by ofﬂine data.
If no such conservative restriction, then the pre-trained policy would suffer a serious distribution mismatch issue (Kumar et al.
and gain a poor pre-trained policy at last.
By contrast, with the conservative scheme, a relatively good pre-trained policy can be obtained.
That is why the starting point of GCQL WC is much lower than other variants.
Secondly, when the dataset is diverse, which means that the dataset is collected by different behavior policies and includes data from different distributions, such as the medium-replay dataset, the greedy scheme (including the ofﬂine-online replay buffer and the greedy update strategy) plays an important role while the ensemble feature seems have limited inﬂuence.
For instance, the performance declined signiﬁcantly for GCQL WO and GCQL WG on walker2d-medium-replay.
Thirdly, when the dataset is not diverse, such as the medium dataset which is collected by one medium policy, almost all curves grow slowly except GCQL WC.
This is caused by the characteristic of the dataset, because dataset diversity plays an important role for policy learning (Agarwal et al., 2020).
However, according to the learning curves, our method still has a clear performance improvement at the latter learning stage.
For the random dataset, all variants fail due to the poor quality of the dataset.
Instead, our method achieves a clear performance improvement under it, which indicates that every component is important in this case.
5. A NALYSIS ON HYPER - PARAMETERS
As we do not ﬁne-tune the hyper-parameters in our experiments, one may wonder how the hyperparameters affect the performance.
To this end, we conduct experiments to investigate their inﬂuence.
The detailed learning curves are shown in Appendix B.
As one main characteristic of our method is limited interaction, we ﬁx the online exploration step as 1 K for each iteration in the MuJoCo benchmark.
We then try settings with different initial update steps T initial , ofﬂine update steps T off , and sample possibility from online buffer p .
Speciﬁcally, T initial is tested with 2 e 5 and 5 e 4 ; T off is tested with 2 e 4 and 5 e 3 ; p is tested with 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 and 0 .
According to the evaluation results, we may conclude that the performance of our method is insensitive to the T initial and T off , especially when the dataset is not poor, e.g., datasets except for the random one.
By contrast, p has a bigger impact on the performance.
Particularly, methods with higher p performs better on the medium and medium-replay tasks, while the ones with lower p perform better on the other tasks.
That indicates that when dataset quality is very good or very bad, methods with more ofﬂine updating perform better.
On the other hand, only the variant of p = 0 .can achieve a clear performance improvement in both the medium and random datasets.
Overall, the default setting p = 0 . 5 is the most suitable setting, which indicates that taking the online and ofﬂine data equally important may be the best option in most cases.
C ONCLUSIONS
This paper ﬁrst discusses the disadvantages of online and ofﬂine RL, and the shortcomings of current ofﬂine-online RL methods.
Then, considering that ofﬂine and near-on-policy online datasets are both crucial for policy learning, we propose a uniﬁed framework that can adaptively and effectively take advantage of both ofﬂine and online data.
Furthermore, a practical algorithm based on the framework that greedily exploits the online experiences and conservatively exploits the ofﬂine experiences is presented.
We conduct comprehensive experiments to verify the effectiveness of our method.
In terms of the shortcomings, although our framework can take advantage of the ofﬂine and limited online dataset, the quality of the ofﬂine dataset still has a big impact on the performance.
For instance, our method is relatively slower when learning from the random dataset.
At last, we hope this work could contribute to bridging the gap between the practice and DRL research.
A L EARNING CURVES FOR ALL TASKS
Figure 4 indicate the whole learning curves for GCQL, REDQ and CQL.
B E XTRA EXPERIMENTS ON H YPER - PARAMETERS
Figure 5 and 6 shows the different hyper-parameters’ setting on the performance.
C E XTRA ABLATION STUDY ON H ALF C HEETAH
Figure 7 present extra ablation study on halfcheetah.
GCQL WG is the GCQL without the greedy update scheme but still with the online-ofﬂine two-level buffer, while GCQL WGO is the GCQL without the greedy scheme and online-ofﬂine replay buffer.
From Figure 7 c,d,e, it is clear that the greedy scheme plays an important role in boosting the performance when the dataset is not optimal or near-optimal.
On the other hand, from Figure 7 d, we can see the online-ofﬂine replay buffer is crucial for stabilizing the learning process where the GCQL WGO and CQL suffer serious stability issues which may be caused by extrapolation error (Fujimoto et al., 2019).
[Figure] [Figure]
[Figure]
D E XTRA EXPERIMENTS ON TD3+BC
Extra experiments on TD3+BC (Fujimoto & Gu, 2021) (without states normalization) for the suboptimal dataset, i e., random, medium, and medium-replay, as the ofﬂine RL method can per-
[Figure]
form very well on expert or near-on-expert datasets.
We simply apply our greedy-conservative learning framework and online-ofﬂine buffer to TD3+B without any ﬁne-tuning and call it: TD3BCGC.
The learning curves indicate that our greedy-conservative framework can still improve policy-penalty-based methods by a large margin in most tasks.
Moreover, it is very simple to apply our framework to TD3+BC where less than 20 lines codes are needed.
We modify the policy objective from π = argmax π E ( s,a ) ∼D (cid:2) λQ
( s, π ( s ))
− ( π ( s ) − a ) 2 (cid:3) to π = argmax π E
( s,a ) ∼D (cid:2) λQ
( s, π ( s ))
− W ( s, a )( π ( s ) − a ) 2 (cid:3) , where W ( s, a ) follows our setting introduced in section 4.4.
E E XTRA EXPERIMENTS ON OFF2ON
Here, we compared our method with OFF2ON (Lee et al., 2021) which also employs an online update scheme to ﬁne-tune a pre-trained agent.
As the author did not provide the pre-training code for CQL, we use the pre-trained CQL agents provided by OFF2ON’s author for online ﬁne-tuning.
All the hyper-parameters follow the default setting except the online interaction steps.
Figureindicated the whole learning curves.
From this ﬁgure, we can see GCQL has a comparable or better performance than OFF2ON.
It is worth noting that OFF2ON ﬁne-tuning their method, for instance, the critic’s neural network architecture is different from the original CQL paper.
In contrast, we did not ﬁne-tune these parameters, and all settings are the same for all tasks.
F E XTRA EXPERIMENTS ON OFFICIAL ONLINE REDQ
In this section, we conduct extra experiments on the online REDQ (Chen et al., 2021).
To guarantee the reproduction performance, we use the ofﬁcial code from the author, and all hyper-parameters are following its default setting, e.g., the number of Q is 10, and the utd-ratio is 20.
In contrast, the number of Q in GCQL is 5, and utd-ratio is 10.
REDQ-ONLINE updates its policy following a conventional online RL setting, i.e., learning from scratch without the ofﬂine pre-training.
From
Figure 10, it is clear that GCQL achieves a better or comparable performance than REDQ-ONLINE except for the walker2d-random dataset.
Speciﬁcally, when the dataset is good, e.g., including the expert dataset, GCQL can outperform the REDQ-ONLINE by a large margin.
On the other hand,
[Figure]
when the ofﬂine dataset is of poor quality, i.e., random-dataset, GCQL can still learn a comparable or better policy than REDQ-ONLINE in two of three tasks, i.e., the hopper-random and halfcheetahrandom.
[Figure] [Figure]