Uncertainty-Driven Pessimistic Q-Ensemble for Offline-to-Online Reinforcement Learning
Abstract
Re-using existing offline reinforcement learning (RL) agents is an emerging topic for reducing the dominant computational cost for exploration in many settings.
To effectively fine-tune the pre-trained offline policies, both offline samples and online interactions may be leveraged.
In this paper, we propose the idea of incorporating a pessimistic Q-ensemble and an uncertainty quantification technique to effectively fine-tune offline agents.
To stabilize online Q-function estimates during fine-tuning, the proposed method uses uncertainty estimation as a penalization for a replay buffer with a mixture of online interactions from the ensemble agent and offline samples from the behavioral policies.
In various robotic tasks on D4RL benchmark, we show that our method outperforms the state-of-the-art algorithms in terms of the average return and the sample efficiency.
Motivation
Deep offline reinforcement learning (RL) [9] trains deep neural networks from previously collected datasets to learn powerful robotic agents without additional environmental interactions.
Recently, offline RL methods [4, 6, 1, 11, 7] have performed better than the behavioral policies which produce the offline datasets.
Offline RL agents, however, may perform suboptimally 1) when the offline datasets follow possibly suboptimal behavioral policies; 2) when the behavioral policies are insufficiently explorative; or 3) when the datasets are too small.
This requires further online learning (fine-tuning) from additional online interactions generated by offline RL agents.
The goal of online fine-tuning of offline agents is to improve sample efficiency and asymptotic performance of online learning by efficiently leveraging offline datasets.
To fine-tune an offline RL agent, slightly modified off-policy RL methods may be used by leveraging both online and offline samples in training.
However, such modifications are typically difficult to make learning sample-efficient or asymptotically optimal due to distributional shift.
Fine-tuning an agent trained solely on an offline dataset may yield inaccurate value estimates due to online visitation of out-of-distribution (OOD) state-action samples.
The distributional shift between the offline dataset and observed online interactions causes large initial temporal difference errors, which in turn causes initial performance degradation and the agent to forget all information obtained from offline RL.
This leads to decreased sample efficiency by losing the benefit of having an initial good policy.
To address this problem, offline-to-online RL (Off2OnRL [8]) leverages a pessimistic Q-ensemble scheme.
Off2OnRL pessimistically trains multiple Q-functions to mitigate such bootstrapping errors caused by the distributional shift.
This constrains the learning policy to visit near the distribution of the behavioral policy to stabilize fine-tuning an offline RL agent.
Off2OnRL also introduced a prioritized buffer with balanced replay, which contains not only online interactions collected during fine-tuning but near-on-policy samples from the offline dataset.
This scheme enables accurate
Q-function estimation by training a network for prioritizing offline samples, which can bring the sampling distribution for Q-functions close to online samples.
The goals of online learning using an offline dataset are 1) to estimate a good initial policy that is not immediately forgotten during online learning; and 2) to improve sample efficiency.
However, we find that online fine-tuning incorporated with the offline dataset tends to be affected by uncertainty estimation since the buffer has online and offline samples generated from different state-action distributions.
It may suffer from high uncertainty when 1) the network for the balanced replay is not trained sufficiently; and 2) the offline dataset tends to be diverse around several trajectories produced by multiple policies.
In this paper, we introduce an uncertainty-driven pessimistic Q-ensemble (UPQ) for offline-to-online
RL.
We adopt a pessimistic ensemble of offline actor-critic agents to guide the learning policy with efficient pessimism.
To stabilize online Q-function estimates during online fine-tuning, the proposed method leverages uncertainty quantification as a penalization for a prioritized replay buffer where online samples from the ensemble agent and offline samples from the behavioral policy exist together.
In our experiments, we demonstrate that our method obtains high-performing policies with fewer online interactions, and hence outperforms the state-of-the-art algorithms in terms of the average return and the sample efficiency.
Methodology
We consider a Markov Decision Process (MDP) tuple G = ( S , A , r, P ) defined by the state space S , the action space A , the reward function r , and the transition distribution P .
The objective of an RL agent is to maximize the expected return E [ (cid:80) T − 1 i =
γ i r i ] , where γ ∈ [0 , 1) is the discount factor and
T is the episode horizon (total number of timesteps of an episode) that the agent optimizes over.
In our proposed method, we use N offline actor-critic agents { Q θ i , π ϕ i } i ∈ [ N ] for an ensemble from
Off2OnRL [8], where θ i and ϕ i are defined as the parameters of the i agent’s critic and actor, respectively.
We define the Q-function and the policy of the ensemble as follows:
[Equation]
where the parameters are defined as θ :
= { θ i } i ∈
[ N ] and ϕ :=
{ ϕ i } i ∈ [ N ] , respectively.
The defined π E ϕ follows a normal distribution with mean and variance of the Gaussian mixture 1 N (cid:80)
[-=
-]Ni =
1 π ϕ i for parameterization.
The modeled policy is the same as Off2OnRL [8].
Since the ensemble estimates the posterior distribution of its Q-functions, we use the standard deviation-based uncertainty quantification technique proposed in Pessimistic Bootstrapping for offline RL (PBRL [2]).
The uncertainty estimation at ( s ′ , a ′ ) of the target Q-functions is defined as follows:
[Equation]
where θ i − is the parameters of the i agent’s target Q-network and we denote the mean over the ensemble of the target Q-functions by Q Eθ − .
In policy evaluation, we use such uncertainty quantification as a penalization to the next Q-value for a mixture of online and offline samples from the prioritized replay buffer B proposed in Off2OnRL.
The Q E of the ensemble agent is updated through pessimistic
Q-function updates by fitting the following target for state-action pairs sampled from B :
Ensemble Agent
OfflineDataset
PrioritizedReplayBuffer
Prioritizedofflinesamples
Gradientupdate
Onlinesamples
CQL Agent
Actor
Q-net
CQL Agent
Actor
Q-net …
CQL Agent N
Actor
Q-net 𝑄 !
Samplingmini-batchtransitions (s, a, r, s’)
𝒰
Offlinedatacollection ( s !"
" ,a !"" )
Pretrainingoffline ℒ "#$%$& 𝜃
OnlineInteraction ℒ ’&%(# ∅ 𝜋 !
Offlineprocedure Onlineprocedure 𝒯𝑄 !
[Equation]
(4) where β is the tuning parameter for the uncertainty penalization.
To this end, the parameters of the
Q-network and the policy of the ensemble agent, θ and ϕ , are updated by minimizing the following objectives, respectively:
[Equation]
(6) where α is the parameter for temperature.
Figure 1 illustrates the overall workflow of the proposed method with uncertainty quantification.
The learning Q-function and policy are updated via update rules (5) and (6) by fine-tuning the pre-trained multiple ensemble agents.
Experimental Setup and Results
We use the D4RL benchmark [3], which includes datasets for data-driven deep reinforcement learning.
To demonstrate the performance of our method, three Mujoco [10] locomotion tasks (HalfCheetah,
Walker2d, and Hopper) are adopted with four types of datasets (random, medium, medium-replay, medium-expert) for comparative evaluation.
Specifically, the random dataset has collections by a random policy and the medium dataset contains samples from a medium-level policy trained via Soft Actor-Critic (SAC [5]).
The medium-replay dataset contains all samples observed during training a medium-level policy and the medium-expert dataset consists of samples recorded by both a medium-level agent and expert demonstrations.
We use a total of 12 task setups with the ’v2’ version datasets.
We compare the proposed method with several baseline algorithms including CQL [7] and Off2OnRL [8].
CQL provides a good baseline for the performance of an offline approach, which is also used for learning both Off2OnRL and our method.
The implementation of CQL and Off2OnRL are adopted from the official implementation at https://github.com/aviralkumar2907/CQL and https://github.com/shlee94/Off2OnRL , respectively. We fix most of the hyperparameters for [Table] a ndo
[Table]
[Figure] n [Figure] our method the same as the official Off2OnRL implementation.
We train N = 4 CQL agents forepochs with different seeds for all the locomotion tasks where we experiment.
Off2onRL and our method are trained for 200 epochs ( 200000 environmental steps) using the pre-trained CQL agents.
In our method, the parameter for uncertainty penalization β is varied from [0 . 1 , 0 . 01 , 0 .
Table 1 reports the performance of our method and baseline algorithms for each task.
UPQ performs similarly or better than the baseline methods in most of the tasks in terms of the average return and sample efficiency.
We find that UPQ has advantages in the tasks with non-optimal datasets marked as medium, medium-replay, and medium-expert.
The experiments show that UPQ performs the best with β ∈ [0 . 01 , 0 .
001] in many tasks.
Figure 2 illustrates the training curves for UPQ and baseline algorithms in all tasks.
We remark that the performance of UPQ shows more stable training curves in the early stage (especially 50000 − 100000 environmental steps) of fine-tuning in most of the tasks.
Conclusion
In this work, we propose UPQ with an uncertainty-driven pessimistic Q-ensemble and it is effective for improving offline-to-online RL methods.
To stabilize online Q-function estimates during fine-tuning, the proposed method leverages uncertainty quantification as a penalization for a prioritized replay buffer consisting of both online and offline samples.
Our experiments show that UPQ outperforms several baselines over robotic locomotion tasks in terms of the average return and the sample efficiency.
We expect that our method enables sample-efficient re-use of the existing offline agents and offline datasets.
We also expect to implement further improvements as future work.
Acknowledgments and Disclosure of Funding
This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government.
[22ZR1100, A Study of Hyper Connected Thinking Internet
Technology by autonomous connecting, controlling and evolving ways].