A DAPTIVE Q- LEARNING FOR I NTERACTION -L IMITED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when an online interaction is costly.
Ofﬂine RL provides an alternative solution by directly learning from the logged dataset.
However, it usually yields unsatisfactory performance due to a pessimistic update scheme or/and the low quality of logged datasets.
Moreover, how to evaluate the policy under the ofﬂine setting is also a challenging problem.
In this paper, we propose a uniﬁed framework called Adaptive Q-learning for effectively taking advantage of ofﬂine and online learning.
Speciﬁcally, we explicitly consider the difference between the online and ofﬂine data and apply an adaptive update scheme accordingly, i.e., a pessimistic update strategy for the ofﬂine dataset and a greedy or no pessimistic update scheme for the online dataset.
When combining both, we can apply very limited online exploration steps to achieve expert performance even when the ofﬂine dataset is poor, e.g., random dataset.
Such a framework provides a uniﬁed way to mix the ofﬂine and online RL and gain the best of both worlds.
To understand our framework better, we then provide an initialization following our framework.
Extensive experiments are done to verify the effectiveness of our proposed method.
I NTRODUCTION
Conventional online reinforcement learning (RL) methods (Haarnoja et al., 2018; Fujimoto et al., 2018) usually learn from experiences generated by interactions with the online environment.
They are impractical in some real-world applications, e.g., dialog (Jaques et al., 2019) and education (Mandel et al., 2014), where interactions are costly.
Recently, ofﬂine RL (Levine et al., 2020) has aroused much attention.
It targets the above challenge by making the agent learn from an ofﬂine dataset collected by other policies in a purely data-driven manner.
The difference between online
RL and ofﬂine RL is shown in Figure 1.
Existing ofﬂine RL studies try to target the distribution mismatch or out-of-distribution actions issue by employing a pessimistic update scheme (Kumar et al., 2019; 2020) or in combination with imitation learning (Fujimoto et al., 2019).
However, when the dataset is ﬁxed, ofﬂine RL cannot learn the optimal policy (Kidambi et al., 2020), and even worse, when the dataset’s quality is not diverse, ofﬂine RL usually gains a relatively poor performance (Kumar et al., 2020; Fu et al., 2020; Levine et al., 2020).
On the other hand, it is challenging to evaluate the learned policy when learning totally from the ofﬂine dataset.
Even though some research topics, e.g., off-policy evaluation (Dann et al., 2014), study how to evaluate the learned policy without the interaction with the environment, it is still not ideal for the practical purpose.
Some recent works try to address the above issues by employing an ofﬂine-online setting.
Such methods (Lee et al., 2021; Nair et al., 2020) focus on pre-training a policy using the ofﬂine dataset and ﬁne-tuning the policy through further online interactions.
Even though their methods alleviate the above issues to some extent, their main bottleneck is that they do not consider the different characteristics of ofﬂine and online data.
For instance, pre-existing ofﬂine data can prevent agents from converging prematurely due to the potential diverse ofﬂine dataset, while online data can improve stability and accelerate convergence.
Generally, these different data are mixed and used by a pessimistic strategy to update the policy in their methods, which may be problematic since using a pessimistic strategy for online data may harm the policy performance (Nair et al., 2020).
Moreover, with sufﬁciently large and diverse ofﬂine data, a high-performing policy can be learned just using a pure online RL algorithm, while the online data (near-on-policy) also play a key role in improving the online RL’s stability especially when dataset is not diverse (Fujimoto et al., 2019; Agarwal et al., 2020).
Hence, we should take full advantage of both ofﬂine and online dataset.
To tackle the above problems, in this paper, we emphasize that online and ofﬂine RL should be coupled organically .
First, a separate updating strategy should be employed for online and ofﬂine data, respectively, considering their different characteristics.
To do so, we present a framework called adaptive Q-learning that integrates the advantage of ofﬂine learning and online learning effectively.
When learning from the ofﬂine dataset, we conduct a pessimistic update strategy.
In contrast, we use a greedy or non-pessimistic update strategy when learning from the online dataset.
Second, we design a novel replay buffer to distinguish the ofﬂine from online datasets in a simple way.
By utilizing such a novel framework and buffer design, the agent can achieve an expert policy using limited online interaction steps regardless of the quality of the ofﬂine dataset.
In the experiments, our proposed framework can achieve better performance by using only one ﬁfth number of interactions compared with the previous method (Nair et al., 2020).
Our contributions can be summarized as below:
• We propose a uniﬁed framework called adaptive Q-learning with limited online interaction steps that can effectively beneﬁt from both the ofﬂine dataset and limited online data.
• Based on the general framework, we initialize a practical algorithm, called Greedyconservative Q-ensemble learning (GCQL) that builds on top of State-of-the-Art ofﬂine RL and online RL method.
• We empirically verify the effectiveness of our method by comprehensive experiments on the popular ofﬂine benchmark D4RL (Fu et al., 2020).
R ELATED W ORK
Online RL In general, online RL algorithms can be divided into two categories, i.e., on-policy and off-policy algorithms.
On-policy methods (Schulman et al., 2015; 2017) update the policy using data collected by its current behavior policy.
As ignoring the logged data collected by its history behaviour policies, they usually have a lower sample efﬁciency than the off-policy RL.
On the other hand, off-policy methods (Fujimoto et al., 2018; Chen et al., 2021) enable the policy to learn from experience collected by history behavior polices, however, they cannot learn well from history trajectories collected by other agents’ behavior policies (Fujimoto et al., 2019; Kumar et al., 2020).
Consequently, the need for huge online interaction makes online RL impractical for some real-world applications,such as dialog agents (Jaques et al., 2019) or education system (Mandel et al., 2014).
Ofﬂine RL Ofﬂine RL algorithms assume the online environment is unavailable and learn policies only from the pre-collected dataset.
As the value estimation error cannot be corrected using online interactions here, these methods tend to utilize a pessimistic updating strategy to relieve the distribution mismatch problem (Fujimoto et al., 2019; Kumar et al., 2019).
To implement such a strategy, model-free ofﬂine RL methods generally employ value or policy penalties to constrain the updated policy close to the data collecting policy (Wu et al., 2019; Kumar et al., 2020; Fujimoto et al., 2019;
He & Hou, 2020).
And model-based methods use predictive models to estimate uncertainties of states and then update the policy in a pessimistic way based on them (Kidambi et al., 2020; Yu et al., 2020).
Those ofﬂine RL methods cannot guarantee a good performance, especially when the data quality is not diverse (Kumar et al., 2020).
Besides, policy evaluation when the online environment is unavailable is also challenging.
Even though off-policy evaluation (OPE) methods (Dann et al., 2014) present alternative solutions, they are still far from perfect.
Above issues of online and ofﬂine RL motivate us to investigate the ofﬂine-online setting.
Ofﬂine-online RL Lee et al.
(2021) and Nair et al.
(2020) focus on the mixed setting where the agent is ﬁrst learned from the ofﬂine dataset, and then trained online.
Nair et al.
(2020) propose an advantage-weighted actor-critic (AWAC) method that restricts the policy to select actions close to those in the ofﬂine data by an implicit constraint.
When online interactions are available, such conservative constraint may have adverse effects on the performance.
Lee et al.
(2021) employ a balanced replay scheme to address the distribution shift issue.
It uses the ofﬂine data by only selecting near-on-policy samples.
Unlike these two works, our method utilizes all online and ofﬂine data, and explicitly considers the difference between them by adaptively applying non-conservative or conservative updating schemes, respectively .
Matsushima et al.
(2021) focuses on optimizing deployment efﬁciency, i.e., the number of distinct data-collection policies used during learning, by employing a behavior-regularized policy updating strategy.
Although in terms of deployment efﬁciency, their work is between online and ofﬂine RL, it ignores existing ofﬂine dataset, and dose not focusing on improving sample efﬁciency, while both are addressed in our paper.
Some works (Zhu et al., 2019; Vecerik et al., 2017; Rajeswaran et al., 2018; Kim et al., 2013) can also learn from online interactions and ofﬂine data. However, they need expert demonstrations instead of any dataset, and this limits their applicability.
P RELIMINARIES
In RL, the interaction between the agent and environment is usually modelled using Markov decision process (MDP) ( S , A , p M , r, γ ) , with state space S (state s ∈ S ), action space A (action a ∈ A ).
At each discrete time step, the agent takes an action a based on the current state s , and the state changes into s (cid:48) according to the transition dynamics p M ( s (cid:48) | s, a ) , and the agent receives a reward r ( s, a, s (cid:48) ) ∈ R .
The agent’s objective is to maximize the return, which is deﬁned as R t = (cid:80) ∞
i =
t +1 γ
i r
( s i , a i , s i +1 ) , where t is the time step, and γ ∈ [0 , 1) is the discounted factor.
The mapping from s to a is denoted by the stochastic policy π : a ∼ π ( ·| s ) . Policy can be stochastic or deterministic, and we use the stochastic from in this paper for generality.
Each policy π have a corresponding action-value function Q π ( s, a ) =
E π
[ R t | s, a ] , which is the expected return following the policy after taking action a in state s .
The policy π ’s action-value function can be updated by the Bellman operator T π :
[Equation]
Q-learning (Sutton & Barto, 2011) directly learns the optimal action-value function Q ∗ ( s, a ) =
max π Q π ( s, a ) , and such Q-function can be modelled using neural networks (Mnih et al., 2015).
In principle, off-policy methods, such as Q-learning, can utilize experiences collected by any policies, and thus they usually maintain a replay buffer B to store and repeatedly learn from experiences collected by behavior policies.
Such capability also enables off-policy methods to be used in the ofﬂine setting, by storing ofﬂine data into the buffer B , and not updating the buffer during learning since no further interactions are available here (Levine et al., 2020). But this simple adjusting cannot guarantee the agent to have a reasonable performance, especially when the dataset quality is not diverse (Kumar et al., 2020; Levine et al., 2020), and this is also the problem tackled in most ofﬂine
RL works. In this paper, we focus on the ofﬂine-online setting, where the agent is ﬁrst learned from the ofﬂine dataset, and then trained via online interactions . And without additional remarks, online RL methods refer to off-policy algorithms in the rest of this paper.
We only use off-policy methods because they can make more use of ofﬂine data than on-policy ones for gaining high sample efﬁciency, and on-policy methods are not compatible with our proposed framework introduced next.
M ETHODOLOGY
In this section, we ﬁrst give an illustrative example to explain our motivation.
After that, we introduce the proposed general framework, namely Adaptive Q-learning, trying to couple online and ofﬂine RL in an organic way.
To implement the proposed framework, then we present the OnlineOfﬂine Replay Buffer, targeting retaining and distinguishing online and ofﬂine data simultaneously.
And ﬁnally we incorporate State-of-the-Art (SotA) online and ofﬂine RL algorithms into the framework, and introduce the proposed Greedy-Conservative Q-ensemble Learning algorithm in detail.
4. A N ILLUSTRATIVE EXAMPLE
We test the SotA ofﬂine RL method conservative Q-learning (CQL) (Kumar et al., 2020) and online RL method Randomized Ensembled Double Q-learning (REDQ)
(Chen et al., 2021) under the ofﬂine-online setting. Here, we choose a widely used locomotion task Hopper (Todorov et al., 2012), and use the dataset hopper-medium-replay-v0 in the D4RL benchmark (Fu et al., 2020) as the ofﬂine dataset, which contains diverse experiences from different policies.
Speciﬁcally, we ﬁrst pre-train the agent with the ofﬂine dataset for 100 K steps. Then, the agent is ﬁne-tuned online by alternately conducting the interaction and updating process, where the agent interacts with the environment for 1 K steps, and is updated for 10 K steps.
Such alternating process ends until the total online interaction steps reach 90 K .
This ofﬂine-online setting should be a favourable one for policy learning, because both diverse ofﬂine data and online interactions are available here.
Therefore, we expect following results: the initial ofﬂine training will provide a relatively good starting point for the agent, and then the agent can be further improved by the online alternating process since online interaction data can be obtained, and ﬁnally we may acquire a well-performed policy, even with normalized score close to or better than 100 (i.e., the performance of an expert).
Nonetheless, experiment results are shown in Figure 2, and they do not totally meet our expectation.
Speciﬁcally, starting points of two curves show that the ofﬂine algorithm CQL and online algorithm
REDQ can both obtain normalized scores greater than 0 (i.e., the performance of a random policy) after the initial ofﬂine training process.
This result is what we expected, because CQL is designed for the ofﬂine setting, and online off-policy methods, such as REDQ, can also learn from the diverse and large ofﬂine dataset even though it is ﬁxed (Agarwal et al., 2020).
However, both algorithms have troubles in the following online process. REDQ suffers from severe instability issue and its performance drops signiﬁcantly during online learning, and in the end, the policy almost degenerates into a random one.
One the other hand, even though the CQL agent can keep stable during learning, its improvement is very limited.
These results indicate that pure online RL algorithm may be problematic for effectively handling ofﬂine data and online interaction data in a single training process, and pure ofﬂine RL algorithm cannot make good use of valuable online interaction data due to its conservative updating strategy. Such observation motivates us to couple them in an organic way.
4. A DAPTIVE Q- LEARNING FRAMEWORK
In this subsection, we introduce our proposed framework.
The underlying idea is simple and can be described as follows.
When the agent learns from online interaction data, we choose a more greedy or no-pessimistic updating strategy, since these data are near-on-policy.
By contrast, when data are sampled from the ofﬂine dataset, we tend to use a more pessimistic updating strategy.
Through such an adaptive way, we can make full use of both online and ofﬂine data, and explicitly consider their differences by separately applying suitable updating schemes.
Then, we give a formalization for above intuition based on Q-value and value penalty, and call this framework Adaptive Q-learning .
The learning objective of this framework can be deﬁned by the following equation:
[Equation]
This objective consists of two terms.
The ﬁrst term A ( Q ) stands for the greedy updating strategy, which is a regular objective for Q-value in online RL algorithms, e.g., the bellman error. The second term B ( Q ) stands for the pessimistic updating strategy, which is the value penalty, e.g., the Q-value regularizer (Wu et al., 2019). Besides, a weight function W ( s, a ) is applied to the penalty term B ( Q ) . This weight function is based on the sampled data type.
Speciﬁcally, when we use online interaction data, W ( s, a ) will be a smaller value, and the updating relies more on the objective of online RL, leading to a relatively greedy strategy.
On the contrary, when we use ofﬂine data, W ( s, a ) will be a bigger value, and the updating strategy is relatively pessimistic.
Remark: Our behind intuition can also be easily formalized via the policy learning objective.
Such variant of our framework can be denoted by π k +1 ← arg max π k (cid:2)
A (cid:0) π k (cid:1) + W ( s, a ) B (cid:0) π k (cid:1)(cid:3) , where A ( π k ) is a objective for the policy in onlien RL and B ( π k ) is a conservative policy penalty term.
We also provide implementation for this variant based on TD3+BC algorithm (Fujimoto & Gu, 2021), and our framework can largely boost its performance within limited environment steps (see
Appendix D for details).
4. OORB: O NLINE -O FFLINE R EPLAY B UFFER
Next, we introduce a simple but effective online-ofﬂine replay buffer (OORB) to distinguish online interaction data and ofﬂine data.
OORB consists of two replay buffers.
One is the online buffer that collects the online interaction data.
The other is the ofﬂine buffer consisting of the newly generated online interaction data and the ofﬂine dataset which may come from any policies.
To ensure the data in the online buffer is near-on-policy, we set it to be much smaller than the ofﬂine buffer.
Fresh online interaction data are stored into the online buffer by following a ﬁrst-in-ﬁrst-out rule.
Data are sampled from OORB following a Bernoulli distribution, which means that with a probability p , they are sampled from the online buffer, and with probability 1 − p , they are sampled from the ofﬂine buffer.
To beneﬁt from both online and ofﬂine data in a balanced way, we empirically set p to 0.5.
And to verify p ’s effect, we also conduct extra experiments to investigate the performance with different p in section 5.4.
The experiment results indicate that p = 0 .
5 is the most suitable option in most cases, conﬁrming our assumption: ofﬂine data and online data are both crucial for RL learning.
4. GCQL: G REEDY -C ONSERVATIVE Q- ENSEMBLE L EARNING
For understanding our framework better, we provide a practical algorithm.
According to equation 2, we need to deﬁne two parts in Q’s update objective.
The ﬁrst is the pessimistic regularization term used in an ofﬂine learning setting, and the second is a normal online Q objective.
For the ﬁrst term, we can employ a penalized value function, e.g., a divergence function between distributions over actions (Wu et al., 2019), or the conservative objective used in Kumar et al.
The second term is the normal Q-learning objective, such as the Bellman backup error.
We present an initialization on top of the state-of-the-art ofﬂine RL agent CQL (Kumar et al., 2020), and the online
RL agent REDQ (Chen et al., 2021), called Greedy conservative Q-ensemble learning (GCQL).
Speciﬁcally, we replace the Q target function of CQL with the random-ensemble Q functions of
1 Initialization : 2 Initialize policy π φ , ensemble Q functions Q
θ i , i ∈ N ,ofﬂine training steps t 3 Online exploration steps T on , ofﬂine update steps T off 4 Initialize T initial , sample possibility p , start sampling steps: T s 5 Initialize online buffer B on to empty, ofﬂine buffer B off ← ofﬂine dataset 6 Initialize online buffer size S on ←7 Initial ofﬂine learning : 8 Train the agent for T initial steps using the logged dataset 9 while True do t ←
Explore T on steps online
Store the T on steps experiences to both online buffer B on and ofﬂine buffer B off
S on ← S on + T on for t < T off do
Sample a random value p s ∼ U (0 , 1) if p s < p and S on > T s then
Sample a batch ( s, a ) from online buffer B on
Set the W ( s, a ) to end else
Sample a batch ( s, a ) from ofﬂine buffer B off
Set the W ( s, a ) to end
Update the Q functions by equation
Update the policy by equation t += end 28 end REDQ, i.e., the randomized min Q values, and keep the rest of the objective of CQL( H ). Formally, for each Q i , the updated objective can be deﬁned by following equation:
[Equation]
where the action ˙ a used in the ﬁrst left Q function is sampled from current policy, i.e., ˙ a ∼ π φ ( ·| s ) , s ∼ D OORB , D OORB is the OORB replay buffer, and ˆ B π ˆ Q k ( s , a ) is deﬁned by
[Equation]
We randomly select two Q functions from the ensemble Q functions and the M represents the selected Q functions’ index just following the REDQ’s setting.
The ˆ Q stands for a target Q function for stabilizing the learning process (Mnih et al., 2015).
The policy update is deﬁned by
[Equation]
When sampled data is from the online replay buffer, we set W ( s, a ) to 0, otherwise 1, that is why we called it greedy-conservative.
In another word, we greedy exploit the near-on-policy data by the normal online RL scheme without any conservative regularizer.
On the contrary, we conservatively exploit the ofﬂine data by employing the ofﬂine RL regularizer.
Formally, this strategy can be [Table] explained by following deﬁnition
[Equation]
sampled from buffer otherwise (6)
The practical algorithm is shown in Algorithm 1.
Here, we also explain the main steps in the
Algorithm as follows.
Firstly, We ﬁrst learn from the existing logged data for T initial steps to leverage ofﬂine dataset.
To make good use of the ofﬂine data, we usually set a big value for T initial , e.g., 100000 steps.
Secondly, We begin the following interleaving learning steps.
We conduct the online exploration process for T on steps and store the new experiences to OORB.
We set the online exploration steps T on to a small value, e.g., 1000.
In contrast, the ofﬂine update step T off is set to be larger than T on , e.g., 10000. We sample a batch from our OORB and update our policy and Q functions.
If the sampled batch comes from the online buffer, then we set the weight value W ( s, a ) to 0, otherwise 1.
Repeat the above interleaving learning steps.
E XPERIMENTS
In this section, we design experiments to verify the effectiveness of our method from three perspectives: 1.
the performance superiority compared with other baselines; 2.
ablation studies to explain the effectiveness of each component used in our method.
the impact of different hyper-parameters.
5. S ETTINGS
All experiments were done on the continuous control benchmark: Mujoco (Todorov et al., 2012).
The logged history dataset is the popular ofﬂine dataset D4RL (Fu et al., 2020).
To make interaction limited, we set the online interaction step for each iteration to a small value, i.e., 1000.
To better exploit the ofﬂine dataset, we set the ofﬂine training step to a large value, i.e., 100000 for T intial , and 10000 for T off .
1 million for entire ofﬂine-updates steps, and 90K for entire online exploration steps.
As we take the online and ofﬂine datasets equally important, we set the sampling possibility from the online buffer as 0.5.
We further set the size of the online buffer as 20000 and the size of the ofﬂine buffer as 3e6.
We follow the default setting of all the baselines, except for the number of the ensemble Q, which is 5 in our setting.
The above conﬁgurations are ﬁxed for all tasks.
As our main purpose is to present a new framework instead of gaining state-of-the-art performance, we do not ﬁne-tune these hyper-parameters. [Figure]
5. O VERALL PERFORMANCE
Table 1 shows the comparison results of Ours, CQL, AWAC, and REDQ.
The AWAC’s results are directly taken from their paper (Nair et al., 2020).
We test on the continuous control tasks from
D4RL, including random-v0, medium-v0, expert-v0, medium-expert-v0, and medium-replay-v0.
As the online exploration is available, we take the maximization over testing scores during the whole training process, and report the average of these max scores across different seeds in Table 1.
For a comprehensive understanding, we include the whole learning curves in Appendix A.
The reported results are the averaged test performance across ﬁve random seeds against the online interaction steps.
These learning curves demonstrate that our method also achieves a better ﬁnal average score at the last iteration in most cases.
As shown in Table 1, our method gains a better performance than the baselines in all tasks.
Particularly, for the medium-replay and random dataset, our GCQL outperforms the baselines by a large margin.
By contrast, CQL also achieves an expert performance on the high-quality dataset, e.g., expert and medium-expert tasks.
For these high-quality datasets, our method achieves a comparable or slightly better performance over CQL.
In terms of the online
RL method: REDQ, it fails on almost all tasks.
Even though we employ an online limited exploration process, conventional online RL cannot beneﬁt from such limited experiences.
On the contrary, such limited online experiences could bring catastrophic consequences, e.g., on the halfcheetah-mediumreplay task, REDQ suffers a signiﬁcant instability issue.
On the other hand, when dataset quality is high, e.g., expert or medium-expert dataset, our method can leverage the beneﬁt of the ofﬂine RL that gains an expert performance with very limited interaction steps, such as around 2e4 steps.
By contrast, when dataset quality is poor, e.g., the random or medium-replay dataset, only our method can learn effectively with less than 1e5 online steps, which demonstrate that our method can take the advantage of the limited online experiences as much as possible.
In sum, our method can beneﬁt the most from the limited online experience while still maintain the ofﬂine learning’s ability.
5. A BLATION STUDIES
To estimate each component’s contribution in our method, we conduct an ablation study on the walker2d.
To do this, we design four variants of our method.
GCQL WE : GCQL without the ensemble where we only use two Q functions same as the setting of CQL.
GCQL WO : GCQL without the online replay buffer where we only sample from the ofﬂine buffer.
GCQL WG : GCQL
These results are shown in ﬁgure 3, and we evaluate it on ﬁve datasets on walker2d across three random seeds.
Firstly, from ﬁgure 3, it is easy to deduce that when the quality of data is high, i.e., including the expert dataset, almost all variants perform well except the GCQL WC.
The underlying reason is obvious as we ﬁrstly pre-train the policy by ofﬂine data.
If no such conservative restriction, then the pre-trained policy would suffer a serious distribution mismatch issue (Kumar et al.
and gain a poor pre-trained policy at last.
By contrast, with the conservative scheme, the policy would obtain a relatively good pre-trained policy.
That is why the starting point of GCQL WC is much lower than other variants.
Secondly, when the dataset is diverse, i.e., the dataset is collected by different behavior policies and include data from different distributions, such as the medium-replay dataset, the greedy scheme (including the ofﬂine-online replay buffer and the greedy update strategy) plays an important role in such case while the ensemble feature seems have limited inﬂuence.
For instance, the performance declined signiﬁcantly for GCQL WO and GCQL WG on Walker2d-medium-replay.
Thirdly, when the dataset is not diverse, e.g., the medium dataset (data is collected by one medium policy), almost all tasks grow slowly except GCQL WC.
The underlying reason may be a lack of diversity, as Agarwal et al.
(2020) declaimed diverse dataset plays a key role in RL.
However, according to the learning curves, our method still has a clear performance improvement at the latter learning stage.
For the random dataset, all variants fail due to the poor quality of the dataset.
Instead, our method achieves a clear performance improvement under the random dataset, which shows the effectiveness of each component in this case.
5. A NALYSIS ON HYPER - PARAMETERS
As we do not ﬁne-tune the hyper-parameters in our experiments, one may wonder how the hyperparameters affect the performance.
To this end, we conduct experiments to discuss their inﬂuence.
The detailed learning curves are shown in Appendix B.
As one main characteristic of our method is limited interaction, we ﬁx the online exploration step as 1000 for each iteration in the Mujoco benchmark.
We then investigate the performance with the initial update steps T initial , the ofﬂine update steps T off , and the sample possibility from online buffer p .
Speciﬁcally, T initial is tested with 2e5 and 5e4; T off is tested with 2e4 and 5e3; p is tested with 0.3,0.4,0.5,0.6 and 0.7.
According to the evaluation results, we may conclude that our framework is insensitive to the T initial and T off , especially when the dataset is not poor, e.g., not the random dataset.
By contrast, p has a bigger impact on the performance.
Particularly, higher p performs better than the lower ones on the medium and medium-replay tasks, while the lower ones perform better on the other tasks.
That indicates that when dataset quality is very good or very bad, more ofﬂine update performs better.
On the contrary, more online update is preferred for the medium-quality dataset.
On the other hand, only the variant of p=0.5 can achieve a clear performance improvement in both the medium and random datasets.
Overall, the default setting p=0.5 is the most suitable setting across all tasks, which indicates that taking the online and ofﬂine data equally important may be the best option.
C ONCLUSIONS
This paper ﬁrst discusses the disadvantages of online RL and ofﬂine RL, and then the shortcomings of current ofﬂine-online methods.
After that, we propose a uniﬁed framework that can take advantage of online and ofﬂine learning effectively.
Furthermore, a practical initialization that greedily exploits the online experiences and conservatively exploits the ofﬂine experiences is presented.
We conduct comprehensive experiments to verify the effectiveness of our method.
In terms of the shortcomings, although our framework can take advantage of the ofﬂine and online dataset under the interaction-limited setting, the quality of the ofﬂine dataset still has a big impact on the performance.
For instance, the policy would converge to expert performance very soon when learning from the expert dataset.
By contrast, our method is relatively slower when learning from the random dataset.
Importantly, we also show that ofﬂine and online (near-on-policy) datasets are both crucial for RL learning.
Thus, through a uniﬁed way, we can gain the best from both sides.
At last, we hope this work could contribute to bridging the gap between the practice and DRL research.
A L EARNING CURVES FOR ALL TASKS
Figure 4 indicate the whole learning curves for GCQL, REDQ and CQL.
B E XTRA EXPERIMENTS ON H YPER - PARAMETERS
Figure 5 and 6 shows the different hyper-parameters’ setting on the performance.
C E XTRA ABLATION STUDY ON H ALF C HEETAH
Figure 7 present extra ablation study on halfcheetah.
GCQL WG is the GCQL without the greedy update scheme but still with the online-ofﬂine two-level buffer, while GCQL WGO is the GCQL without the greedy scheme and online-ofﬂine replay buffer.
From Figure 7 c,d,e, it is clear that the greedy scheme plays an important role in boosting the performance when the dataset is not optimal or near-optimal.
On the other hand, from Figure 7 d, we can see the online-ofﬂine replay buffer is crucial for stabilizing the learning process where the GCQL WGO and CQL suffer serious stability issues which may be caused by extrapolation error (Fujimoto et al., 2019). [Figure] [Figure] [Figure]
D E XTRA EXPERIMENTS ON TD3+BC
Extra experiments on TD3+BC (Fujimoto & Gu, 2021) (without states normalization) for the sub-optimal dataset, i.e., random, medium, and medium-replay, as the ofﬂine RL method
[Figure] can perform very well on expert or near-on-expert datasets.
We simply apply our greedyconservative learning framework and online-ofﬂine buffer to TD3+B without any ﬁne-tuning, namely, TD3BCGC.
The learning curves indicate that our greedy-conservative framework can still improve policy-penalty-based methods by a large margin in most tasks.
Moreover, it is very simple to apply our framework to TD3+BC where less than 20 lines codes are needed.
We modify the policy objective from π = argmax π E ( s,a ) ∼D (cid:2) λQ
( s, π ( s ))
− ( π ( s ) − a ) 2 (cid:3) to π = argmax π E
( s,a ) ∼D (cid:2) λQ
( s, π ( s ))
− W ( s, a )( π ( s ) − a ) 2 (cid:3) , where W ( s, a ) follows our setting introduced in section 4.4.
E E XTRA EXPERIMENTS ON OFF2ON
Here, we compared our method with OFF2ON (Lee et al., 2021) which also employs an online update scheme to ﬁne-tune a pre-trained agent.
As the author did not provide the pre-training code for CQL, we use the pre-trained CQL agents provided by OFF2ON’s author for online ﬁne-tuning.
All the hyper-parameters follow the default setting except the online interaction steps.
Figureindicated the whole learning curves.
From this ﬁgure, we can see GCQL has a comparable or better performance than OFF2ON.
It is worth noting that OFF2ON ﬁne-tuning their method, for instance, the critic’s neural network architecture is different from the original CQL paper.
In contrast, we did not ﬁne-tune these parameters, and all settings are the same for all tasks
. F E XTRA EXPERIMENTS ON OFFICIAL ONLINE REDQ
In this section, we conduct extra experiments on the online REDQ (Chen et al., 2021).
To guarantee the reproduction performance, we use the ofﬁcial code from the author, and all hyper-parameters are following its default setting, e.g., the number of Q is 10, and the utd-ratio is 20.
In contrast, the number of Q in GCQL is 5, and utd-ratio is 10.
REDQ-ONLINE updates its policy following a conventional online RL setting, i.e., learning from scratch without the ofﬂine pre-training. From
Figure 10, it is clear that GCQL achieves a better or comparable performance than REDQ-ONLINE except for the walker2d-random dataset.
Speciﬁcally, when the dataset is good, e.g., including the expert dataset, GCQL can outperform the REDQ-ONLINE by a large margin.
On the other hand,
[Figure]
when the ofﬂine dataset is of poor quality, i.e., random-dataset, GCQL can still learn a comparable or better policy than REDQ-ONLINE in two of three tasks, i.e., the hopper-random and halfcheetahrandom.
[Figure]
[Figure]