A DAPTIVE Q- LEARNING FOR I NTERACTION -L IMITED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when an online interaction is costly.
Ofﬂine RL provides an alternative solution by directly learning from the logged dataset.
However, it usually yields unsatisfactory performance due to a pessimistic update scheme or/and the low quality of logged datasets.
Moreover, how to evaluate the policy under the ofﬂine setting is also a challenging problem.
In this paper, we propose a uniﬁed framework called Adaptive Q-learning for effectively taking advantage of ofﬂine and online learning.
Speciﬁcally, we explicitly consider the difference between the online and ofﬂine data and apply an adaptive update scheme accordingly, i.e., a pessimistic update strategy for the ofﬂine dataset and a greedy or no pessimistic update scheme for the online dataset.
When combining both, we can apply very limited online exploration steps to achieve expert performance even when the ofﬂine dataset is poor, e.g., random dataset.
Such a framework provides a uniﬁed way to mix the ofﬂine and online RL and gain the best of both worlds.
To understand our framework better, we then provide an initialization following our framework.
Extensive experiments are done to verify the effectiveness of our proposed method.
I NTRODUCTION
Conventional online reinforcement learning (RL) methods (Haarnoja et al., 2018; Fujimoto et al., 2018) usually learn from experiences generated by interactions with the online environment.
They are impractical in some real-world applications, e.g., dialog (Jaques et al., 2019) and education (Mandel et al., 2014), where interactions are costly.
Recently, ofﬂine RL (Levine et al., 2020) has aroused much attention.
It targets the above challenge by making the agent learn from an ofﬂine dataset collected by other policies in a purely data-driven manner.
The difference between online
RL and ofﬂine RL is shown in Figure 1.
Existing ofﬂine RL studies try to target the distribution mismatch or out-of-distribution actions issue by employing a pessimistic update scheme (Kumar et al., 2019; 2020) or in combination with imitation learning (Fujimoto et al., 2019).
However, when the dataset is ﬁxed, ofﬂine RL cannot learn the optimal policy (Kidambi et al., 2020), and even worse, when the dataset’s quality is poor, ofﬂine
RL usually gains a relatively poor performance (Kumar et al., 2020; Fu et al., 2020; Levine et al., 2020).
On the other hand, it is challenging to evaluate the learned policy when learning totally from the ofﬂine dataset.
Even though some research topics, e.g., off-policy evaluation (Dann et al., 2014), study how to evaluate the learned policy without the interaction with the environment, it is still not ideal for the practical purpose.
Some recent works try to address the above issues by employing an ofﬂine-online setting.
Such methods (Lee et al., 2021; Nair et al., 2020) focus on pre-training a policy using the ofﬂine dataset and ﬁne-tuning the policy through further online interactions.
Even though their methods alleviate the above issues to some extent, their main bottleneck is that they do not consider the different characteristics of ofﬂine and online data.
For instance, pre-existing ofﬂine data can prevent agents from converging prematurely due to the potential diverse ofﬂine dataset, while online data can improve stability and accelerate convergence.
Generally, these different data are mixed and used by a pessimistic strategy to update the policy in their methods, which may be problematic since using a pessimistic strategy for online data may harm the policy performance (Nair et al., 2020).
Moreover, with sufﬁciently large and diverse ofﬂine data, a high-performing policy can be learned just using a pure online RL algorithm, while the online data (near-on-policy) also play a key role in improving the online RL’s stability especially when dataset is not diverse (Fujimoto et al., 2019; Agarwal et al., 2020).
Hence, we should take full advantage of both ofﬂine and online dataset.
To tackle the above problems, in this paper, we emphasize that online and ofﬂine RL should be coupled organically .
First, a separate updating strategy should be employed for online and ofﬂine data, respectively, considering their different characteristics.
To do so, we present a framework called adaptive Q-learning that integrates the advantage of ofﬂine learning and online learning effectively.
When learning from the ofﬂine dataset, we conduct a pessimistic update strategy.
In contrast, we use a greedy or non-pessimistic update strategy when learning from the online dataset.
Second, we design a novel replay buffer to distinguish the ofﬂine from online datasets in a simple way.
By utilizing such a novel framework and buffer design, the agent can achieve an expert policy using limited online interaction steps regardless of the quality of the ofﬂine dataset.
In the experiments, our proposed framework can achieve better performance by using only one ﬁfth number of interactions compared with the previous method (Nair et al., 2020).
Our contributions can be summarized as below:
• We propose a uniﬁed framework called adaptive Q-learning with limited online interaction steps that can effectively beneﬁt from both the ofﬂine dataset and limited online data.
• Based on the general framework, we initialize a practical algorithm, called Greedyconservative Q-ensemble learning (GCQL) that builds on top of state-of-the-art ofﬂine RL method CQL (Kumar et al., 2020) and online RL method REDQ (Chen et al., 2021)
• We empirically verify the effectiveness of our method by comprehensive experiments on the popular ofﬂine benchmark D4RL (Fu et al., 2020).
R ELATED W ORK
Online RL In general, online RL algorithms can be divided into two categories, i.e., on-policy and off-policy algorithms.
On-policy methods (Schulman et al., 2015; 2017) update the policy using data collected by its current behavior policy.
As ignoring the logged data collected by its history behaviour policies, they usually have a lower sample efﬁciency than the off-policy RL.
On the other hand, off-policy methods (Fujimoto et al., 2018; Chen et al., 2021) enable the policy to learn from experience collected by history behavior polices, however, they cannot learn well from history trajectories collected by other agents’ behavior policies (Fujimoto et al., 2019; Kumar et al., 2020).
Consequently, the need for huge online interaction makes online RL impractical for some real-world applications,such as dialog agents (Jaques et al., 2019) or education system (Mandel et al., 2014).
Ofﬂine RL Ofﬂine RL algorithms assume the online environment is unavailable and learn policies only from the pre-collected dataset.
As the value estimation error cannot be corrected using online interactions here, these methods tend to utilize a pessimistic updating strategy to relieve the distribution mismatch problem (Fujimoto et al., 2019; Kumar et al., 2019).
To implement such a strategy, model-free ofﬂine RL methods generally employ value or policy penalties to constrain the updated policy close to the data collecting policy (Wu et al., 2019; Kumar et al., 2020; Fujimoto et al., 2019;
He & Hou, 2020).
And model-based methods use predictive models to estimate uncertainties of states and then update the policy in a pessimistic way based on them (Kidambi et al., 2020; Yu et al., 2020).
Those ofﬂine RL methods cannot guarantee a good performance, especially when the data quality is poor (Kumar et al., 2020).
Besides, policy evaluation when the online environment is unavailable is also challenging.
Even though off-policy evaluation (OPE) methods (Dann et al., 2014) present alternative solutions, they are still far from perfect.
Above issues of online and ofﬂine RL motivate us to investigate the ofﬂine-online mixed setting.
Ofﬂine-online mixed RL Matsushima et al.
(2021) propose a deployment-efﬁciency setting that also focuses on a practical setting: how many times the data-collection policy has been changed.
They employ a KL-based trust-region optimization (Schulman et al., 2015) to update the policy.
However, it ignores the ofﬂine dataset and focuses on the deployment times of the behavior policy instead of the interaction steps.
Lee et al.
(2021) aims at ﬁne-tuning a pre-trained agent by online experiences and focus on the distribution shift issue.
They employ a balanced experience replay buffer and select the near-on-policy samples from ofﬂine data.
Even the samples are near-on-policy, they still apply a pessimistic strategy.
Nair et al.
(2020) propose an advantage-weighted actor-critic (AWAC) method that restricts the policy to stay close to the actions observed in the data by an implicit constraint.
When online interaction is available, such constraints may have adverse effects on the performance.
In contrast, our method explicitly considers the difference between ofﬂine and online learning and adaptive apply conservative or non-conservative update schemes.
Some works (Zhu et al., 2019; Vecerik et al., 2017; Rajeswaran et al., 2018; Kim et al., 2013) also can learn from online interaction and ofﬂine dataset, however, they need the expert demonstrations instead of any dataset which limit their applicability.
P RELIMINARIES
In reinforcement learning, the interaction with environment usually assumed to be a Markov decision process (MDP) ( S , A , p M , r, γ ) , with state space S , action space A , and transition dynamics p M ( s (cid:48) | s, a ) .
At each discrete time step, the agent receive a reward r ( s, a, s (cid:48) )
∈ R after conducting an action a in state s and arriving at next state s (cid:48) .
The agent’s objective is to obtain the maximum return which is deﬁned by R t = (cid:80) ∞
i =
t +1 γ
i r
( s i , a i , s i +1 ) .
The γ ∈ [0 , 1) is a discounted factor that down-weight the future rewards.
The policy can be deterministic, in which case it is usually denoted by µ : a = µ ( s ) or it may be stochastic, in which case it is usually denoted by π : a ∼ π ( ·| s ) .
Each policy π have a corresponding action-value function Q π ( s, a ) =
E π
[ R t | s, a ] , the expected return when following the policy after taking action a in state s .
The policy π ’s action-value function can be updated by the Bellman operator T π :
[Equation]
When applying a greedy optimal strategy, i.e., Q ∗ ( s, a ) = max π Q π ( s, a ) , it is known as the Qlearning that optimize the policy for maximizing the Q value (Sutton & Barto, 2011).
Such a policy and Q function can be approximated with neural networks (Mnih et al., 2015).
The above setting is typically off-policy or online RL as they need interaction with the environment to collect data.
In principle, off-policy RL algorithms maintain a replay buffer B storing the exploration experiences collected by behavior policy.
Ofﬂine RL algorithms are based on the off-policy RL techniques but without the online interaction assumption (Levine et al., 2020).
In another word, the replay buffer B size is ﬁxed, and no further interaction with the environment under the ofﬂine RL setting.
The data from the replay buffer can be collected by one or more behavior policies.
Ofﬂine RL provides a way to free RL agents from data collection.
The ofﬂine-online mixed setting focuses on pre-train a policy using the logged dataset and then ﬁne-tuning this pre-trained policy through future online interaction (Nair et al., 2020; Lee et al., 2021).
M ETHODOLOGY
In this section, we ﬁrst give an illustrative example to explain our motivation.
After that, we introduce the proposed general framework, namely Adaptive Q-learning, trying to couple online and ofﬂine RL in an organic way.
Then, we present details about the Online-Ofﬂine Replay Buffer, targeting retaining and distinguishing online and ofﬂine data simultaneously.
Finally, we implement the framework by incorporating SOTA online and ofﬂine RL algorithms and introduce the proposed
Greedy-Conservative Q-ensemble Learning in detail.
4. A N ILLUSTRATIVE EXAMPLE
We test the SOAT ofﬂine RL method CQL (Kumar et al., 2020) and off-policy RL method
REDQ (Chen et al., 2021) under the ofﬂine-online ﬁne-tuning setting using one of the D4RL dataset hopper-medium-replay-v0 that consists of diverse experiences.
Speciﬁcally, we ﬁrstly pre-train the agent with the ofﬂine dataset for 100000 steps, and then online explore 1000 steps.
After that, we do ofﬂine update 10000 steps.
We repeat such an interleaving process until the summed online exploration steps reach 90K.
The experiment results are showed in Figure 2.
We can see both CQL and REDQ have troubles during training under this setting, even though both online and ofﬂine data are available here, which should be beneﬁcial for policy learning.
Particularly, during the initial ofﬂine training, they are both capable of learning, i.e., the starting point’s performance is better than a random policy whose score is 0.
Such result veriﬁed Agarwal et al.
’s claim: when the dataset is diverse and large, online RL is capable of learning effectively from ofﬂine dataset.
However, when an online interaction is available, the online REDQ suffers severe instability issue, and performance drops signiﬁcantly.
In contrast, the CQL agent maintains a stable learning curve but with a very limited performance improvement due to its conservative strategy.
We expect ofﬂine RL or online RL can be capable of learning from the fresh on-policy data well, i.e., gain a score close to or better than the expert policy’s score: 100.
However, they only achieved a very limited performance improvement or failed to learn during the online interaction process.
Such observation indicates that current online or ofﬂine RL methods may be ineffective for beneﬁting from hybrid learning.
4. A DAPTIVE Q- LEARNING FRAMEWORK
Now, we introduce our proposed framework.
The underlying idea is simple.
When the agent learns from near-on-policy samples, we choose a more greedy or no-pessimistic strategy.
By contrast, when sampling from the ofﬂine dataset, we tend to a more pessimistic strategy (Jin et al., 2021).
Through such an adaptive strategy, we can make full use of both online and ofﬂine data.
Here, we take the value penalty as an example (Wu et al., 2019) to explain our framework.
The learning objective of adaptive Q learning consists of two parts.
One is the regular online Q objective presented by A ( Q ) ,e.g., the bellman backup error.
The other is the value penalty term presented by B ( Q ) , e.g., the Q value regularizer.
Then, an adaptive Q-learning objective can be deﬁned by the following equation.
[Equation]
The function W ( s, a ) weight the Q regularizer based on the sampled data.
Speciﬁcally, when the sampled data is near-on-policy, we rely on the online Q objective more, which means the W ( s, a ) will choose a small value, e.g., close to 0.
On the contrary, we will choose a bigger value for W ( s, a ) when the samples are from the ofﬂine buffer.
It is worth noting that we can easily adapt our framework to policy objective.
The equation 2 can be adjusted to π k +1 ← arg max π k W ( s, a ) B (cid:0) π k (cid:1) + A (cid:0) π k (cid:1) , where A ( π k ) is the normal policy objective and B ( π k ) is the policy penalty term.
We apply such variant to TD3+BC (Fujimoto &
Gu, 2021) which is a policy penalty method.
The results indicates that our framework boost its performance by a large margin with limited online interaction.
The details are presented in Appendix
D.
Now, the key problem becomes how to deﬁne near-on-policy samples.
To solve this issue, we introduce an online-ofﬂine replay buffer(OORB).
4. OORB: O NLINE -O FFLINE R EPLAY B UFFER
OORB aims to distinguish the limited online data from ofﬂine data.
A similar tool can be found in
Zheng et al.
(2020b;a); Zhang et al.
(2019) where they use a two-level replay buffer for boosting the learning of the off-policy agents.
However, there is a major difference between ours and theirs: their methods are episode-based, and they select the episode with high episode return, while ours focuses on deciding whether the experiences are near-on-policy or not.
In this sense, our OORB is much simpler and effective in our setting.
Speciﬁcally, the online-ofﬂine replay buffer consists of two replay buffers.
One is the online replay buffer that collects the online exploration experience.
The other is the ofﬂine replay buffer consisting of the newly generated online exploration data and the logged dataset collected by any policies.
To ensure the data in the online buffer is near-onpolicy, we set its size to be much smaller than the size of the ofﬂine buffer.
We store the fresh exploration experiences following a ﬁrst-in-ﬁrst-out rule.
When new experiences arrive, the earliest saved experiences will be overridden.
To sample from OORB, we follow a Bernoulli distribution, sampling data from the online buffer with a probability p , and the ofﬂine buffer with 1 − p .
To beneﬁt from both online and ofﬂine dataset in a balanced way , we set p as 0.5 in our experiments.
To verify other p ’s effect, we also conduct extra experiments to investigate the performance with different p in section 5.4.
The experiment results indicate that p = 0 .
5 is the most suitable option in most cases, conﬁrming our assumption: ofﬂine data and online data are both crucial for RL learning.
4. GCQL: G REEDY -C ONSERVATIVE Q- ENSEMBLE L EARNING
For understanding our framework better, we provide a practical algorithm.
According to equation 2, we need to deﬁne two parts in Q’s update objective.
The ﬁrst is the pessimistic regularization term used in an ofﬂine learning setting, and the second is a normal online Q objective.
For the ﬁrst term, we can employ a penalized value function, e.g., a divergence function between distributions over actions (Wu et al., 2019), or the conservative objective used in Kumar et al.
The second term is the normal Q-learning objective, such as the Bellman backup error.
We present an initialization on top of the state-of-the-art ofﬂine RL agent CQL (Kumar et al., 2020), and the online
RL agent REDQ (Chen et al., 2021), called Greedy conservative Q-ensemble learning (GCQL).
Speciﬁcally, we replace the Q target function of CQL with the random-ensemble Q functions of
REDQ, i.e., the randomized min Q values, and keep the rest of the objective of CQL( H ).
Formally, for each Q i , the updated objective can be deﬁned by following equation:
[Equation]
1 Initialization : 2 Initialize policy π φ , ensemble Q functions Q
θ i , i ∈ N ,ofﬂine training steps t 3 Online exploration steps T on , ofﬂine update steps T off 4 Initialize T initial , sample possibility p , start sampling steps: T s 5 Initialize online buffer B on to empty, ofﬂine buffer B off ← ofﬂine dataset 6 Initialize online buffer size S on ←7 Initial ofﬂine learning : 8 Train the agent for T initial steps using the logged dataset 9 while True do t ←
Explore T on steps online
Store the T on steps experiences to both online buffer B on and ofﬂine buffer B off
S on ← S on + T on for t < T off do
Sample a random value p s ∼ U (0 , 1) if p s < p and S on > T s then
Sample a batch ( s, a ) from online buffer B on
Set the W ( s, a ) to end else
Sample a batch ( s, a ) from ofﬂine buffer B off
Set the W ( s, a ) to end
Update the Q functions by equation
Update the policy by equation t += end 28 end where the action ˙ a used in the ﬁrst left Q function is sampled from current policy, i.e., ˙ a ∼ π φ ( ·| s ) , s ∼ D OORB , D OORB is the OORB replay buffer, and ˆ B π ˆ Q k ( s , a ) is deﬁned by [Equation]
[Equation]
We randomly select two Q functions from the ensemble Q functions and the M represents the selected Q functions’ index just following the REDQ’s setting.
The ˆ Q stands for a target Q function for stabilizing the learning process (Mnih et al., 2015).
The policy update is deﬁned by
[Equation]
When sampled data is from the online replay buffer, we set W ( s, a ) to 0, otherwise 1, that is why we called it greedy-conservative.
In another word, we greedy exploit the near-on-policy data by the normal online RL scheme without any conservative regularizer.
On the contrary, we conservatively exploit the ofﬂine data by employing the ofﬂine RL regularizer.
Formally, this strategy can be explained by following deﬁnition
[Equation]
The practical algorithm is shown in Algorithm 1.
Here, we also explain the main steps in the
Algorithm as follows.
Firstly, We ﬁrst learn from the existing logged data for T initial steps to leverage ofﬂine dataset.
To make good use of the ofﬂine data, we usually set a big value for T initial , e.g., 100000 steps.
Secondly, We begin the following interleaving learning steps.
We conduct the online exploration process for T on steps and store the new experiences to OORB.
We set the online
[Table]
to 0, otherwise 1.
Repeat the above interleaving learning steps.
E XPERIMENTS
In this section, we design experiments to verify the effectiveness of our method from three perspectives: 1.
the performance superiority compared with other baselines; 2.
ablation studies to explain the effectiveness of each component used in our method.
the impact of different hyper-parameters.
5. S ETTINGS
All experiments were done on the continuous control benchmark: Mujoco (Todorov et al., 2012).
The logged history dataset is the popular ofﬂine dataset D4RL (Fu et al., 2020).
To make interaction limited, we set the online interaction step for each iteration to a small value, i.e., 1000.
To better exploit the ofﬂine dataset, we set the ofﬂine training step to a large value, i.e., 100000 for T intial , and 10000 for T off .
1 million for entire ofﬂine-updates steps, and 90K for entire online exploration steps.
As we take the online and ofﬂine datasets equally important, we set the sampling possibility from the online buffer as 0.5.
We further set the size of the online buffer as 20000 and the size of the ofﬂine buffer as 3e6.
We follow the default setting of all the baselines, except for the number of the ensemble Q, which is 5 in our setting.
The above conﬁgurations are ﬁxed for all tasks.
As our main purpose is to present a new framework instead of gaining state-of-the-art performance, we do not ﬁne-tune these hyper-parameters.
5. O VERALL PERFORMANCE
Table 1 shows the comparison results of Ours, CQL, AWAC, and REDQ.
The AWAC’s results are directly taken from their paper (Nair et al., 2020).
We test on the continuous control tasks from
D4RL, including random-v0, medium-v0, expert-v0, medium-expert-v0, and medium-replay-v0.
As the online exploration is available, we take the maximization over testing scores during the whole training process, and report the average of these max scores across different seeds in Table 1. For a
[Figure] comprehensive understanding , we include the whole learning curves in Appendix A.
The reported results are the averaged test performance across ﬁve random seeds against the online interaction steps.
These learning curves demonstrate that our method also achieves a better ﬁnal average score at the last iteration in most cases.
As shown in Table 1, our method gains a better performance than the baselines in all tasks.
Particularly, for the medium-replay and random dataset, our GCQL outperforms the baselines by a large margin.
By contrast, CQL also achieves an expert performance on the high-quality dataset, e.g., expert and medium-expert tasks.
For these high-quality datasets, our method achieves a comparable or slightly better performance over CQL.
In terms of the online RL method: REDQ, it fails on almost all tasks.
Even though we employ an online limited exploration process, conventional online RL cannot beneﬁt from such limited experiences.
On the contrary, such limited online experiences could bring catastrophic consequences, e.g., on the halfcheetah-mediumreplay task, REDQ suffers a signiﬁcant instability issue.
On the other hand, when dataset quality is high, e.g., expert or medium-expert dataset, our method can leverage the beneﬁt of the ofﬂine RL that gains an expert performance with very limited interaction steps, such as around 2e4 steps.
By contrast, when dataset quality is poor, e.g., the random dataset or medium-replay.
Our method can also take the advantage of the limited online experiences as much as possible.
For instance, only our method can learn effectively with less than 1e5 online steps on random or medium-replay dataset.
In sum, our method can beneﬁt the most from the limited online experience while still maintain the ofﬂine learning’s ability.
5. A BLATION STUDIES
To estimate each component’s contribution in our method, we conduct an ablation study on the walker2d.
To do this, we design four variants of our method.
GCQL-we : GCQL without the ensemble where we only use two Q functions same as the setting of CQL.
GCQL-wo : GCQL without the online replay buffer where we only sample from the ofﬂine buffer.
GCQL-wg : GCQL without the greedy where we ﬁx the weighed W to 1.
GCQL-wc : GCQL without the conservative term where we ﬁx the W to 0.
These results are shown in ﬁgure 3, and we evaluate it on ﬁve datasets on walker2d across three random seeds.
Firstly, from ﬁgure 3, it is easy to deduce that when the quality of data is high, i.e., including the expert dataset, almost all variants perform well except the GCQL-wc.
The underlying reason is obvious as we ﬁrstly pre-train the policy by ofﬂine data.
If no such conservative restriction, then the pre-trained policy would suffer a serious distribution mismatch issue (Kumar et al.
and gain a poor pre-trained policy at last.
By contrast, with the conservative scheme, the policy would obtain a relatively good pre-trained policy.
That is why the starting point of GCQL-wc is much lower than other variants.
Secondly, when the dataset is diverse, i.e., the dataset is collected by different behavior policies and include data from different distributions, such as the medium-replay dataset, the greedy scheme (including the ofﬂine-online replay buffer and the greedy update strategy) plays an important role in such case while the ensemble feature seems have limited inﬂuence.
For instance, the performance declined signiﬁcantly for GCQL-wo and GCQL-wg on Walker2d-medium-replay.
Thirdly, when the dataset is not diverse, e.g., the medium dataset (data is collected by one medium policy), almost all tasks grow slowly except GCQL-wc.
The underlying reason may be a lack of diversity, as Agarwal et al.
(2020) declaimed diverse dataset plays a key role in RL.
However, according to the learning curves, our method still has a clear performance improvement at the latter learning stage.
For the random dataset, all variants fail due to the poor quality of the dataset.
Instead, our method achieves a clear performance improvement under the random dataset, which shows the effectiveness of each component in this case.
5. A NALYSIS ON HYPER - PARAMETERS
As we do not ﬁne-tune the hyper-parameters in our experiments, one may wonder how the hyperparameters affect the performance.
To this end, we conduct experiments to discuss their inﬂuence.
The detailed learning curves are shown in Appendix B.
As one main characteristic of our method is limited interaction, we ﬁx the online exploration step as 1000 for each iteration in the Mujoco benchmark.
We then investigate the performance with the initial update steps T initial , the ofﬂine update steps T off , and the sample possibility from online buffer p .
Speciﬁcally, the T initial was tested with 2e5 and 5e4, T off was tested with 2e4 and 5e3 and p was tested with 0.3,0.4,0.5,0.6 and 0.7.
According to the evaluation results, we may conclude that our framework is insensitive to the T initial and T off , especially when the dataset is not poor, e.g., not the random dataset.
By contrast, p has a bigger impact on the performance.
Particularly, higher p performs better than the lower ones on the medium and medium-replay tasks, while the lower ones perform better on the other tasks.
That indicates that when dataset quality is very good or very bad, more ofﬂine update performs better.
On the contrary, more online update is preferred for the medium-quality dataset.
On the other hand, only the variant of p=0.5 can achieve a clear performance improvement in both the medium and random datasets.
Overall, the default setting p=0.5 is the most suitable setting in terms of all tasks, which indicates that taking the online and ofﬂine data equally important may be the best option in most cases.
C ONCLUSIONS
This paper ﬁrst discusses the disadvantages of online RL and ofﬂine RL, and then the shortcomings of current ofﬂine-online mixed methods.
After that, we propose a uniﬁed framework that can take advantage of online and ofﬂine learning effectively.
Furthermore, a practical initialization that greedily exploits the online experiences and conservatively exploits the ofﬂine experiences is presented.
We conduct comprehensive experiments to verify the effectiveness of our method.
In terms of the shortcomings, although our framework can take advantage of the ofﬂine and online dataset under the interaction-limited setting, the quality of the ofﬂine dataset still has a big impact on the performance.
For instance, the policy would converge to expert performance very soon when learning from the expert dataset.
By contrast, our method is relatively slower when learning from the random dataset.
Importantly, we also show that ofﬂine and online (near-on-policy) datasets are both crucial for RL learning.
Thus, through a uniﬁed way, we can gain the best from both sides.
At last, we hope this work could contribute to bridging the gap between the practice and DRL research.
A L EARNING CURVES FOR ALL TASKS
Figure 4 indicate the whole learning curves for GCQL, REDQ and CQL.
B E XTRA EXPERIMENTS ON H YPER - PARAMETERS
Figure 5 and 6 shows the different hyper-parameters’ setting on the performance.
[Figure]
[Figure]
C E XTRA ABLATION STUDY ON H ALF C HEETAH
Figure 7 present extra ablation study on halfcheetah.
GCQL-wg is the GCQL without the greedy update scheme but still with the online-ofﬂine two-level buffer, while GCQL-wgo is the GCQL without the greedy scheme and online-ofﬂine replay buffer.
From Figure 7 c,d,e, it is clear that the greedy scheme plays an important role in boosting the performance when the dataset is not optimal or near-optimal.
On the other hand, from Figure 7 d, we can see the online-ofﬂine replay buffer is crucial for stabilizing the learning process where the GCQL-wgo and CQL suffer serious stability issues which may be caused by extrapolation error (Fujimoto et al., 2019).
D E XTRA EXPERIMENTS ON TD3+BC
Extra experiments on TD3+BC (Fujimoto & Gu, 2021) (without states normalization) for the sub-optimal dataset, i.e., random, medium, and medium-replay, as the ofﬂine
RL method can perform very well on expert or near-on-expert datasets.
The learning curves indicate that our greedy-conservative framework can still improve policy-penaltybased methods by a large margin in most tasks.
Moreover, it is very simple to apply our framework to TD3+BC where less than 20 lines codes are needed.
We modify the policy objective from π = argmax π E ( s,a ) ∼D (cid:2) λQ
( s, π ( s ))
− ( π ( s ) − a ) 2 (cid:3) to π = argmax π E
( s,a ) ∼D (cid:2) λQ
( s, π ( s ))
− W ( s, a )( π ( s ) − a ) 2 (cid:3) , where W ( s, a ) follows our setting introduced in section 4.4.
E E XTRA EXPERIMENTS ON OFF 2 ON
Here, we compared our method with off2on (Lee et al., 2021) which also employs an online update scheme to ﬁne-tune a pre-trained agent.
As the author did not provide the pre-training code for
[Figure] [Figure]
CQL, we use the pre-trained CQL agents provided by off2on’s author for online ﬁne-tuning.
All the hyper-parameters follow the default setting except the online interaction steps.
Figure 9 indicated the whole learning curves.
From this ﬁgure, we can see GCQL has a comparable or better performance
[Figure]
than off2on.
It is worth noting that off2on ﬁne-tuning their method, for instance, the critic’s neural network architecture is different from the original CQL paper.
In contrast, we did not ﬁne-tune these parameters, and all settings are the same for all tasks.
F E XTRA EXPERIMENTS ON OFFICIAL ONLINE REDQ
In this section, we conduct extra experiments on the online REDQ (Chen et al., 2021).
To guarantee the reproduction performance, we use the ofﬁcial code from the author, and all hyper-parameters are following its default setting, e.g., the number of Q is 10, and the utd-ratio is 20.
In contrast, the number of Q in GCQL is 5, and utd-ratio is 10.
REDQ-online updates its policy following a conventional online RL setting.
From Figure 10, it is clear that GCQL achieves a better or comparable performance than REDQ-online except for the walker2d-random dataset.
Speciﬁcally, when the dataset is good, e.g., including the expert dataset, GCQL can outperform the REDQ-online by a large margin.
On the other hand, when the ofﬂine dataset is of poor quality, i.e., random-dataset,
[Figure]
GCQL can still learn a comparable or better policy than REDQ-online in two of three tasks, i.e., the hopper-random and halfcheetah-random.
[Figure]