P ESSIMISTIC B OOTSTRAPPING FOR U NCERTAINTYD RIVEN O
FFLINE
R EINFORCEMENT L
EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Ofﬂine Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment.
Directly applying off-policy algorithms to ofﬂine RL usually fails due to the extrapolation error caused by the out-of-distribution (OOD) actions.
Previous methods tackle such problem by penalizing the Q-values of OOD actions or constraining the trained policy to be close to the behavior policy.
Nevertheless, such methods typically prevent the generalization of value functions beyond the ofﬂine data and also lack precise characterization of OOD data.
In this paper, we propose Pessimistic Bootstrapping for ofﬂine RL (PBRL), a purely uncertainty-driven ofﬂine algorithm without explicit policy constraints.
Speciﬁcally, PBRL conducts uncertainty quantiﬁcation via the disagreement of bootstrapped Q-functions, and performs pessimistic updates by penalizing the value function based on the estimated uncertainty.
To tackle the extrapolating error, we further propose a novel OOD sampling method.
We show that such OOD sampling and pessimistic bootstrapping yields provable uncertainty quantiﬁer in linear MDPs, thus providing the theoretical underpinning for PBRL.
Extensive experiments on D4RL benchmark show that PBRL has better performance compared to the state-of-the-art algorithms.
I NTRODUCTION
Deep Reinforcement Learning (DRL) (Sutton & Barto, 2018) achieves remarkable success in a variety of tasks.
However, in most successful applications, DRL requires millions of interactions with the environment.
In real-world applications such as navigation (Mirowski et al., 2018) and healthcare (Yu et al., 2019), acquiring a large number of samples by following a possibly suboptimal policy can be costly and dangerous.
Alternatively, practitioners seek to develop RL algorithms that learn a policy based solely on an ofﬂine dataset, where the dataset is typically available.
However, directly adopting online DRL algorithms to the ofﬂine setting is problematic.
On the one hand, policy evaluation becomes challenging since no interaction is allowed, which limits the usage of on-policy algorithms.
On the other hand, although it is possible to slightly modify the off-policy value-based algorithms and sample solely from the ofﬂine dataset in training, such modiﬁcation typically suffers from a signiﬁcant performance drop compared with their online learning counterpart (Levine et al., 2020).
An important reason for such performance drop is the so-called distributional shift .
Speciﬁcally, the ofﬂine dataset follows the visitation distribution of the behavior policies .
Thus, estimating the Q -functions of the corresponding greedy policy with the ofﬂine dataset is biased due to the difference in visitation distribution.
Such bias typically leads to a signiﬁcant extrapolation error for
DRL algorithms since the estimated Q -function tends to overestimate the out-of-distribution (OOD) actions (Fujimoto et al., 2019).
To tackle the distributional shift issue in ofﬂine RL, previous successful approaches typically fall into two categories, namely policy constraint (Kumar et al., 2019; Fujimoto & Gu, 2021) and conservatism (Kumar et al., 2020; Yu et al., 2020; 2021).
Policy constraint aims to restrict the learned policy to be close to the behavior policy, thus reducing the extrapolation error in policy evaluation.
Conservatism methods seek to penalize the Q -functions for OOD actions in policy evaluation and hinge on a gap-expanding property to regularize the OOD behavior.
Nevertheless, since policy constraints explicitly conﬁne the policy to be close to the behavior policy, it tends to be easily affected by the non-optimal behavior policy.
Meanwhile, although the conservatism algorithms such as CQL (Kumar et al., 2020) do not require policy constraints, CQL equally penalizes the OOD actions and lacks a precise characterization of the OOD data, which can lead to overly conservative value functions.
To obtain a more reﬁned characterization of the OOD data, uncertainty quantiﬁcation is shown to be effective when associated with the model-based approach (Yu et al., 2020; Kidambi et al., 2020), where the dynamics model can be learned in static data thus providing more stable uncertainty in policy evaluation.
Nevertheless, model-based methods need additional modules and may fail when the environment becomes high-dimensional and noisy.
In addition, the uncertainty quantiﬁcation in model-free RL is more challenging since the Q -function and uncertainty quantiﬁer need to be learned simultaneously (Yu et al., 2021).
To this end, we propose Pessimistic Bootstrapping for ofﬂine RL (PBRL), an uncertainty-driven model-free algorithm for ofﬂine RL.
To acquire reliable Q -function estimates and their corresponding uncertainty quantiﬁcation, two components of PBRL play a central role, namely, bootstrapping and OOD sampling.
Speciﬁcally, we adopt bootstrapped Q -functions (Osband et al., 2016) for uncertainty quantiﬁcation.
We then perform pessimistic Q -updates by using such quantiﬁcation as a penalization.
Nevertheless, solely adopting such penalization based on uncertainty quantiﬁcation is neither surprising nor effective.
Training the Q -functions based solely on the ofﬂine dataset does not regularize the OOD behavior of the Q -functions and suffers from the extrapolation error.
To this end, we propose a novel OOD sampling technique as a regularizer of the learned Q -functions.
Speciﬁcally, we introduce additional OOD datapoints into the training buffer.
The OOD datapoint consists of states sampled from the training buffer, the corresponding OOD actions sampled from the current policy, and the corresponding OOD target based on the estimated Q -function and uncertainty quantiﬁcation.
We highlight that having such OOD samples in the training buffer plays an important role in both the Q -function estimation and the uncertainty quantiﬁcation.
On the one hand, OOD sampling controls the OOD behavior in training, which guarantees the stability of the trained bootstrapped Q -functions.
On the other hand, a stable estimation of the bootstrapped Qfunctions further ensures that the uncertainty quantiﬁcation is reliable.
We further show that PBRL is provably efﬁcient under the linear MDP assumptions.
We highlight that PBRL exploits the OOD state-action pairs by casting a more reﬁned penalization over OOD data points, allowing PBRL to acquire better empirical performance than the policyconstraint and conservatism baselines.
As an example, if an action lies close to the support of ofﬂine data but is not contained in the ofﬂine dataset, the conservatism (Kumar et al., 2020) and policy constraint (Fujimoto et al., 2019) methods tend to avoid selecting it.
In contrast, PBRL tends to assign a small Q -penalty for such an action as the underlying epistemic uncertainty is small.
Hence, the agent trained with PBRL may consider such action if the corresponding value estimate is high, yielding better performance than the policy constraint and conservatism baselines.
Our experiments on the D4RL benchmark (Fu et al., 2020) show that PBRL provides reasonable uncertainty quantiﬁcation and yields better performance compared to the state-of-the-art algorithms.
P RELIMINARIES
We consider an episodic MDP deﬁned by the tuple ( S , A , T, r, P ) , where S is the state space, A is the action space, T ∈ N is the length of episodes, r is the reward function, and P is the transition distribution.
The goal of RL is to ﬁnd a policy π that maximizes the expected cumulative rewards
E (cid:2) (cid:80) T − 1 i =
γ t r i ] , where γ ∈ [0 , 1) is the discount factor in episodic settings.
The corresponding
Q -function of the optimal policy satisﬁes the following Bellman operator,
[Equation]
where θ is the parameters of Q -network.
In DRL, the Q -value is updated by minimizing the TDerror, namely E ( s,a,r,s (cid:48) )
[( Q −T Q ) 2 ] .
Empirically, the target T Q is typically calculated by a separate target-network parameterized by θ − without gradient propagation (Mnih et al., 2015).
In online RL, one typically samples the transitions ( s, a, r, s (cid:48) ) through iteratively interacting with the environment.
The Q -network is then trained by sampling from the collected transitions.
In contrast, in ofﬂine RL, the agent is not allowed to interact with the environment.
The experiences are sampled from an ofﬂine dataset D in = { ( s it , a it , r it , s it +1 ) } i ∈ [ m ] .
Naive off-policy methods such as Q -learning suffer from the distributional shift , which is caused by different visitation distribution of the behavior policy and the learned policy.
Speciﬁcally, the greedy action a (cid:48) chosen by the target
Q -network in s (cid:48) becomes OOD-action since ( s (cid:48) , a (cid:48) ) is not contained in D in .
The value functions evaluated on OOD actions have signiﬁcant extrapolation errors.
Such errors can be further ampliﬁed through propagation and potentially diverges training.
In what follows, we introduce PBRL, which tackles such a challenge by uncertainty quantiﬁcation and OOD sampling.
P ESSIMISTIC B OOTSTRAPPING FOR O FFLINE RL
In this section, we introduce the proposed PBRL algorithm.
First, we introduce the uncertainty quantiﬁer based on the bootstrapped Q -functions.
We then present the pessimistic update with OOD sampling in PBRL.
Finally, we discuss the theoretical connections between the OOD sampling and pessimism in PBRL and the provable efﬁcient ofﬂine RL algorithms.
3. U NCERTAINTY Q UANTIFICATION WITH B OOTSTRAPPING
In PBRL, we maintain K bootstrapped Q -functions in critic to quantify the epistemic uncertainty.
Formally, we denote Q k as the k -th Q -function in the ensemble.
Q k is updated by ﬁtting the following target,
[Equation]
where we denote (cid:98) T as the empirical Bellman operator, which estimates the expectation E
[ Q kθ − ( s (cid:48) , a (cid:48) ) | s, a ] empirically based on the ofﬂine dataset.
We adopt such an ensemble technique from Bootstrapped DQN (Osband et al., 2016), which is initially proposed for the online exploration task.
Intuitively, the ensemble forms an estimation of the posterior distribution of the estimated Qfunctions, which yields similar value on areas with rich data and diversely on areas with scarce data.
Indeed, the deviation among the bootstrapped Q -functions yields an epistemic uncertainty estimation, which we aim to utilize as a penalization in estimating the Q -functions.
Speciﬁcally, we introduce the following uncertainty quantiﬁcation U ( s, a ) based on the Q -functions { Q k } k ∈
[ K ] ,
[Equation]
Here we denote ¯ Q as the mean among the ensemble of Q -functions.
From the Bayesian perspective, such uncertainty quantiﬁcation yields an estimation of the standard deviation of the posterior of
Q -functions.
To better understand the effectiveness of such uncertainty quantiﬁcation, we illustrate with a simple prediction task.
We use 10 neural networks with identical architecture and different initialization as the ensemble.
We then train the ensemble with 60 datapoints in R 2 plane, where the covariate x is generated from the standard Gaussian distribution, and the response y is obtained by feeding x into a randomly generated neural network.
We plot the datapoints for training and the uncertainty quantiﬁcation in Fig. 1(a).
As shown in the ﬁgure, the uncertainty quantiﬁcation rises smoothly from the in-distribution datapoints to the OOD datapoints.
In ofﬂine RL, we perform regression ( s, a ) → (cid:98) T Q k ( s, a ) in D in to train the bootstrapped Qfunctions, which is similar to regress x
→ y in the illustrative task.
The uncertainty quantiﬁcation allows us to quantify the deviation of a datapoint from the ofﬂine dataset, which provides more reﬁned conservatism comparing to the previous methods (Kumar et al., 2020; Wu et al., 2019).
3. P ESSIMISTIC L EARNING
We now introduce the pessimistic value iteration based on the bootstrapped uncertainty quantiﬁcation.
The idea is to penalize the Q -functions based on the uncertainty quantiﬁcation.
To this end, we propose the following target for state-action pairs sampled from D
[Equation] [Equation]
where U θ − ( s (cid:48) , a (cid:48) ) is the uncertainty estimation at ( s (cid:48) , a (cid:48) ) based on the target network, and β in is a tuning parameter.
In addition, the empirical mean (cid:98) E s (cid:48) ∼ P ( ·| s,a ) ,a (cid:48) ∼ π ( ·| s ) is obtained by sampling the transition ( s, a, s (cid:48) ) from D in and further sampling a (cid:48) ∼ π ( · | s (cid:48) ) from the current policy π .
We denote (cid:98) T in Q kθ ( s, a ) as the in-distribution target of Q kθ ( s, a ) and write (cid:98) T in to distinguish the in-distribution target from that of the OOD target, which we introduce in the sequel.
[Figure]
Pessimistic Pessimistic
OfflineDataset
TargetQ-net
TargetQ-net TargetQ-net K … Actor MainQ-nets
Gradients
PBRLforofflinedata
PBRLforOODdata
MainQ
[Figure]
[Figure]
[Figure] [Figure] [Figure] We remark that there are two options to penalize the Q -functions through the operator (cid:98) T in .
In
Eq.
(4), we penalize the next- Q value Q kθ − ( s (cid:48) , a (cid:48) ) with the corresponding uncertainty U θ − ( s (cid:48) , a (cid:48) ) .
Alternatively, we can also penalize the immediate reward by ˆ r ( s, a ) :
= r ( s, a ) − U θ ( s, a ) , and use ˆ r
( s, a ) in place of r ( s, a ) for the target.
Nevertheless, since the datapoint ( s, a ) ∈ D in lies on rich-data areas, the penalization U θ ( s, a ) on the immediate reward is usually very small thus having less effect in training.
In PBRL, we penalize the uncertainty of ( s (cid:48) , a (cid:48) ) in the next- Q value.
Nevertheless, our empirical ﬁndings in §D suggest that solely penalizing the uncertainty for indistribution target is insufﬁcient to control the OOD performance of the ﬁtted Q -functions.
To enforce direct regularization over the OOD actions, we incorporate the OOD datapoints directly in training and sample OOD data of the form ( s ood , a ood ) ∈
D ood .
Speciﬁcally, we sample OOD states from the in-distribution dataset D in .
Correspondingly, we sample OOD actions a ood by following the current policy π ( · | s ood ) .
We highlight that such OOD sampling requires only the ofﬂine dataset D in and does not require additional generative models or access to the simulator.
It remains to design the target for OOD samples.
Since the transition P ( · | s ood , a ood ) and reward r ( s ood , a ood ) are unknown, the true target of OOD sample is inapplicable.
In PBRL, we propose a novel pseudo-target for the OOD datapoints,
[Equation]
which introduces additional uncertainty penalization U θ [Equation] s ood , a ood ) to enforce pessimistic Qfunction estimation, and β ood is a tuning parameter.
For OOD samples that are close to the indistribution data, such penalization is small and the OOD target is close to the Q -function estimation.
In contrast, for OOD samples that are distant away from the in-distribution dataset, a larger penalization is incorporated into the OOD target.
In our implementation, we introduce an additional truncation to stablize the early stage training as max { 0 , T ood Q kθ ( s ood , a ood ) } .
In addition, we remark that β ood is important to the empirical performance.
Speciﬁcally,
• at the beginning of training, both the Q -functions and the corresponding uncertainty quantiﬁcations are inaccurate.
We use a large β ood to enforce a strong regularization to OOD actions.
• We then gradually decrease β ood in the training process since the value estimation and uncertainty quantiﬁcation becomes more accurate in training.
A decreased β ood makes the uncertainty more critical to decide the scale of pessimism in (cid:98) T ood Q k .
After few epochs, β ood becomes a constant and the pessimistic update solely relies on the uncertainty.
Incorporating both the in-distribution target and OOD target, we conclude the loss function for critic in PBRL as follows,
[Equation]
where we iteratively minimize the regular TD-error for the ofﬂine data and the pseudo TD-error for the OOD data.
PBRL obtains a smooth and pessimistic value-function by reducing the extrapolation error caused by high-uncertain state-action pairs for both ( s, a ) and ( s ood , a ood ) .
Based on the pessimistic Q -functions, the corresponding policy is obtained by the following maximization problem,
[Equation]
where ϕ is the policy parameters.
Taking the minimum among ensemble Q -functions is also adopted in previous actor-critic methods like SAC (Haarnoja et al., 2018) and TD3 (Fujimoto et al., 2018).
The overall architecture of PBRL is illustrated in Fig. 1(b).
3. T HEORETICAL C ONNECTIONS TO LCB- PENALTY
In this section, we show the pessimistic target in PBRL aligns with the recent theoretical investigation to ofﬂine RL (Jin et al., 2021; Xie et al., 2021a).
From the theoretical perspective, an appropriate uncertainty quantiﬁcation is essential to the provable efﬁciency in ofﬂine RL.
Speciﬁcally, the ξ -uncertainty quantiﬁer plays a central role in the analysis of both online and ofﬂine RL (Jaksch et al., 2010; Azar et al., 2017; Wang et al., 2020a; Jin et al., 2020; 2021;
Xie et al., 2021a;b).
Deﬁnition 1 ( ξ -Uncertainty Quantiﬁer (Jin et al., 2021)) .
The set of penalization { Γ t } t ∈ [ T ] forms a ξ -Uncertainty Quantiﬁer if it holds with probability at least 1 − ξ that [Equation] s [Equation]
a ) |
[Equation]
Γ [Equation] s, a ) for all ( s, a ) ∈ S × A , where T is the Bellman equation and (cid:98) T is the empirical Bellman equation that estimates T based on the ofﬂine data.
In linear MDPs (Jin et al., 2020; Wang et al., 2020a; Jin et al., 2021) where the transition kernel and reward function are assumed to be linear to the state-action representation φ : S ×
A → R d , The following LCB-penalty (Abbasi-Yadkori et al., 2011; Jin et al., 2020) is known to be a ξ -uncertainty quantiﬁer for appropriately selected { β t } t ∈ [ T ] ,
[Equation]
where Λ t = (cid:80) mi =1 φ ( s it , a it ) φ ( s it , a it ) (cid:62) + λ · I accumulates the features of state-action pairs in D in and plays the role of a pseudo-count intuitively.
We remark that under such linear MDP assumptions, the penalty proposed in PBRL and Γ lcb ( s t , a t ) in linear MDPs is equivalent under a Bayesian perspective.
Speciﬁcally, we make the following claim.
Claim 1.
In linear MDPs, the proposed bootstrapped uncertainty β t · U ( s t , a t ) is an estimation to the LCB-penalty Γ lcb ( s t , a t ) in Eq.
(8) for an appropriately selected tuning parameter β t .
We refer to §A for a detailed explanation and proof.
Intuitively, the bootstrapped Q -functions estimates a non-parametric Q -posterior
(Osband et al., 2016; 2018a).
Correspondingly, the uncertainty quantiﬁer U ( s t , a t ) estimates the standard deviation of the Q -posterior, which scales with the LCBpenalty in linear MDPs.
As an example, we show that under the tabular setting, Γ lcb ( s t , a t ) is approximately proportional to the reciprocal pseudo-count of the corresponding state-action pair in the dataset (See Lemma 2 in §A).
In ofﬂine RL, such uncertainty quantiﬁcation measures how trustworthy the value estimations on state-action pairs are.
A low LCB-penalty (or high pseudo-count) indicates that the corresponding state-action pair aligns with the support of ofﬂine data.
Under the linear MDP or Bellman-consistent assumptions, penalizing the estimated value function based on the uncertainty quantiﬁcations is known to yield efﬁcient ofﬂine RL algorithm
(Jin et al., 2021; Xie et al., 2021a;b).
However, due to the large extrapolation error of neural networks, we ﬁnd that solely penalizing the value function of the in-distribution samples is insufﬁcient to regularize the ﬁtted value functions of OOD state-action pairs.
In contrast, a key to the success of linear MDP algorithms (Jin et al., 2020; Wang et al., 2020a; Jin et al., 2021) is the extrapolation ability through L 2 -regularization in the least-squares value iteration (LSVI), which guarantees that the linear parameterized value functions behave reasonably on OOD state-action pairs.
From a Bayesian perspective, such L 2 -regularization enforces a Gaussian prior on the estimated parameter of linear approximations, which regularizes the value function estimation on
OOD state-action pairs with limited data available.
Nevertheless, our empirical study in §D shows that L 2 -regularization is not sufﬁcient to regularize the OOD behavior of neural networks.
To this end, PBRL introduces a direct regularization over an OOD dataset.
From the theoretical perspective, we observe that adding OOD datapoint ( s ood , a ood , y ) into the ofﬂine dataset leads to an equivalent regularization to the L 2 -regularization under the linear MDP assumption.
Speciﬁcally, in linear MDPs, such additional OOD sampling yields a covariate matrix of the following form,
[Equation]
where the matrix Λ ood = (cid:80) ( s ood ,a ood ,y ) ∈D ood φ ( s ood , a ood ) φ ( s ood , a ood ) (cid:62) plays the role of the λ · I prior in LSVI.
It remains to design a proper target y in the OOD datapoint ( s ood , a ood , y ) .
The following theorem show that setting y =
T V h
+
( s ood , a ood ) leads to a valid ξ -uncertainty quantiﬁer under the linear MDP assumptions.
Theorem 1.
Let Λ ood (cid:23) λ · I .
Under the linear MDP assumption, for all the OOD datapoint ( s ood , a ood , y ) ∈
D ood , if we set y = T V t +1 ( s ood , a ood ) , it then holds for β t
= O (cid:0) T · √ d · log ( T/ξ ) (cid:1) that Γ lcb t ( s t , a t ) = β t [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
forms a valid
ξ
-uncertainty quantiﬁer
We refer to §A for a detailed discussion and the proof.
Theorem 1 shows that if we set y =
T V t +1 ( s ood , a ood ) , the bootstrapped uncertainty based on disagreement among ensembles is a valid ξ -uncertainty quantiﬁer.
However, such an OOD target is impossible to obtain in practice as it requires knowing the transition at the OOD datapoint ( s ood , a ood ) .
In practice, if TD error is sufﬁciently minimized, then Q ( s ood , a ood ) should well estimate the target T V t +1 .
Thus, in PBRL, we utilize
[Equation]
as the OOD target, where we introduce additional penalization Γ lcb ( s ood , a ood ) to enforce pessimism.
In addition, we remark that in theory, we require that the embeddings of the OOD sample are isotropic, in the sense that the eigenvalues of the corresponding covariate matrix Λ ood are lower bounded.
Such isotropic property can be guaranteed by randomly generating states and actions.
In practice, we ﬁnd that randomly generating states is more expensive than randomly generating actions.
Meanwhile, we observe that randomly generating actions alone are sufﬁcient to guarantee reasonable empirical performances since the generated OOD embeddings are sufﬁciently isotropic.
Thus, in our experiments, we randomly generate OOD actions according to our current policy and sample OOD states from the in-distribution dataset.
R ELATED W ORKS
Previous model-free ofﬂine RL algorithms typically rely on policy constraints to restrict the learned policy from producing the OOD actions.
In particular, previous works add behavior cloning (BC) loss in policy training (Fujimoto et al., 2019; Fujimoto & Gu, 2021; Ghasemipour et al., 2021), measure the divergence between the behavior policy and the learned policy (Kumar et al., 2019;
Wu et al., 2019; Kostrikov et al., 2021), apply advantage-weighted constraints to balance BC and advantages (Siegel et al., 2020; Wang et al., 2020b), penalize the prediction-error of a variational auto-encoder (Rezaeifar et al., 2021), and learn latent actions (or primitives) from the ofﬂine data (Zhou et al., 2020; Ajay et al., 2021).
Nevertheless, these methods may cause overly conservative value functions and are easily affected by the behavior policy (Nair et al., 2020; Lee et al., 2021b).
We remark that the OOD actions that align closely with the support of ofﬂine data could also be trustworthy.
CQL (Kumar et al., 2020) directly minimizes the Q -values of OOD samples and thus casts an implicit policy constraint.
Our method is related to CQL since both PBRL and CQL enforce conservatism in Q -learning.
In contrast, we conduct explicit uncertainty quantiﬁcation for OOD actions in pessimistic Q -learning, while CQL penalizes the Q -values of all OOD samples equally.
In contrast with model-free algorithms, model-based algorithms learn the dynamics model directly with supervised learning.
Similar to our work, MOPO (Yu et al., 2020) and MOReL (Kidambi et al., 2020) incorporate ensemble dynamics-model for uncertainty quantiﬁcation, and penalize the value function through pessimistic updates.
Other than the uncertainty quantiﬁcation, previous modelbased methods also attempt to constrain the learned policy through BC loss (Matsushima et al., , advantage-weighted prior (Cang et al., 2021), CQL-style penalty (Yu et al., 2021), and Riemannian submanifold (Tennenholtz et al., 2021).
Decision Transformer (Chen et al., 2021) builds a transformer-style dynamic model and casts the problem of ofﬂine RL as conditional sequence modeling.
The model-based methods suffer from additional computation costs and perform suboptimally in high-dimensional and stochastic environments (Chua et al., 2018; Janner et al., 2019).
In contrast, PBRL conducts model-free learning and is less affected by such challenges.
Our method is related to the previous online RL exploration algorithms based on uncertainty quantiﬁcation, including bootstrapped Q -networks (Bai et al., 2021; Lee et al., 2021a), ensemble dynamics (Sekar et al., 2020), Bayesian neural network (O’Donoghue et al., 2018;
Azizzadenesheli et al., 2018), and distributional value functions (Mavrin et al., 2019; Nikolov et al., 2019).
Uncertainty quantiﬁcation is more challenging in ofﬂine RL than its online counterpart due to the limited coverage of ofﬂine data and the distribution shift of the learned policy.
In model-based ofﬂine RL, framework, MOPO (Yu et al., 2020) and MOReL (Kidambi et al., 2020) incorporate ensemble dynamics-model for uncertainty quantiﬁcation.
BOPAH (Lee et al., 2020) combines uncertainty penalization and behavior-policy constraints, and adopts a gradient-based optimization for hyper-parameters.
In model-free ofﬂine RL, UWAC (Wu et al., 2021) adopts dropout-based uncertainty (Gal & Ghahramani, 2016) while relying on policy constraints in learning value functions.
In contrast, PBRL does not require additional policy constraints.
In addition, according to the previous study in image prediction with data shift (Ovadia et al., 2019), the bootstrapped uncertainty is shown to be more robust to data shift than the dropout-based approach
Our algorithm is inspired by the recent advances in the theory of both online RL and ofﬂine RL.
Previous works propose provably efﬁcient RL algorithms under the linear MDP assumption for both the online setting (Jin et al., 2020) and ofﬂine setting (Jin et al., 2021), which we follow for our analysis.
In addition, previous works also study the ofﬂine RL under the weaker Bellman completeness assumptions (Modi et al., 2021; Uehara et al., 2021; Xie et al., 2021a; Zanette et al., 2021).
Previous works also study the model-based RL under the kernelized nonlinear regulator (KNR) setting (Kakade et al., 2020; Mania et al., 2020), while we focus on model-free RL in this paper.
EDAC (An et al., 2021) is a concurrent work that uses the ensemble Q-network to ensure pessimism.
The main difference between EDAC and our proposed PBRL is the approach to handling OOD actions.
Speciﬁcally, EDAC calculates the gradients of each Q-function and diversiﬁes such gradients to ensure sufﬁcient penalization for OOD actions.
In contrast, PBRL penalizes the OOD actions through direct OOD sampling and the associated uncertainty quantiﬁcations.
Empirically, EDAC typically requires more Q -networks (typically between 10 to 50, according to Fig.
5 of An et al.
(2021)) than PBRL (6-10, see Fig. 4) to obtain reasonable performance.
E XPERIMENTS
In experiments, we further include an algorithm named PBRL-prior , a slight modiﬁcation to PBRL by using random prior functions (Osband et al., 2018b) that are originally proposed for Bootstrapped
DQN (Osband et al., 2016).
Speciﬁcally, each Q -function in PBRL-Prior contains a trainable network Q kθ and a prior network p k , where p k is randomly initialized and keeps ﬁxed in training.
The prediction of each Q -function is the sum of the trainable network and the ﬁxed prior as
Q kp = Q kθ + p k , where Q kθ and p k have the same network architecture.
The random prior function increases the diversity among ensemble members and improves the generalization of bootstrapped functions (Osband et al., 2018b).
We adopt SAC (Haarnoja et al., 2018) as the basic actor-critic architecture for both PBRL and PBRL-prior .
We refer to §B for the implementation details.
The code is available at https://github.com/review-anon/PBRL .
In D4RL benchmark (Fu et al., 2020) with various continuous-control tasks and datasets, we compare the baseline algorithms on the Gym and Adroit domains, which are more extensively studied in the previous research.
We compare PBRL and PBRL-Prior to several state-of-the-art algorithms, including (i) BEAR (Kumar et al., 2019) that enforces policy constraints through the MMD distance, (ii) UWAC (Wu et al., 2021) that improves BEAR through dropout uncertainty-weighted update, (iii) CQL (Kumar et al., 2020) that learns conservative value functions by minimizing Q -values of
OOD actions, (iv) MOPO (Yu et al., 2020) that quantiﬁes the uncertainty through ensemble dynamics in a model-based setting, and (v) TD3-BC (Fujimoto & Gu, 2021), which adopts adaptive BC constraint to regularize the policy in training.
[Table]
in Table 7 of Fujimoto & Gu (2021).
The scores for other baselines are obtained by re-training with the ‘v2’ dataset of D4RL (Fu et al., 2020).
Results in Gym domain. The Gym domain includes three environments (HalfCheetah, Hopper, and Walker2d) with ﬁve dataset types (random, medium, medium-replay, medium-expert, and expert), leading to a total of 15 problem setups.
We train all the baseline algorithms in the latest released ‘v2’ version dataset, which is also used in TD3-BC (Fujimoto & Gu, 2021) for evaluation.
For methods that are originally evaluated on the ‘v0’ dataset, we re-train these methods with their ofﬁcial implementations.
We refer to §B for the training details.
We train each method for one million time steps and report the ﬁnal evaluation performance through online interaction.
Table 1 reports the normalized score for each task and the corresponding average performance.
We ﬁnd CQL and TD3-BC perform the best among all baselines, and
PBRL outperforms the baselines in most of the tasks.
In addition, PBRL-Prior slightly outperforms PBRL and is more stable in training with a reduced variance among different seeds.
[Figure]
Compared to the baseline algorithms, PBRL has strong advantages in the non-optimal datasets, including medium, medium-replay, and mediumexpert.
Compared to the policy-constraint baselines, PBRL exploits the optimal trajectory covered in the dataset in a theoretically grounded way and is less affected by the behavior policy.
We give the average training curves in Fig.
The training speeds of PBRL and PBRL-Prior are slower than TD3-BC and CQL in the early stage, indicating that the uncertainty quantiﬁcation is inaccurate initially.
Nevertheless, PBRL and PBRL-Prior converge to better policies that outperform the baselines with the learning of uncertainty quantiﬁers.
Results in Adroit domain.
The adroit tasks are more challenging than the Gym domain in task complexity.
In addition, the use of human demonstration in the dataset also makes the task more challenging.
We refer the result of the experiments in §E. We observe that CQL and BC have the best average performance in all baselines, and PBRL outperforms these methods in most of the tasks.
Uncertainty quantiﬁcation.
To verify the effectiveness of the bootstrapped uncertainty quantiﬁcation, we record the uncertainty quantiﬁcation for different sets of state-action pairs in training.
In
[Figure]
our experiments, we consider sets that have the same states from the ofﬂine dataset but with different types of actions, including (i) a oﬄine , which is drawn from the ofﬂine in-distribution transition; (ii) a policy , which is selected by the training policy; (iii) a rand , which is uniformly sampled from the action space of the corresponding task; (iv) a noise1 = a oﬄine + N (0 , 0 . 1) , which adds a small
Gaussian noise onto the ofﬂine action to represent state-action pair that is close to in-distribution data; and (v) a noise2 =
a oﬄine + N (0 , 0 . 5) , which adds a large noise to represent the OOD action.
We compute the uncertainty and Q -value in ‘Walker2d’ task with two datasets (‘medium-replay’ and ‘medium’).
The results are shown in Fig.
We observe that (i) PBRL yields large uncertainties for ( s, a noise2 ) and ( s, a random ) , showing that the uncertainty quantiﬁcation is high on OOD samples.
(ii)
The ( s, a oﬄine ) pair has the smallest uncertainty in both settings as it is an in-distribution sample.
(iii)
The ( s, a noise1 ) pair has slightly higher uncertainty compared to ( s, a oﬄine ) , showing that the uncertainty quantiﬁcation rises smoothly from the in-distribution actions to the OOD actions.
(iv) The Q -function of the learned policy ( s, a policy ) is reasonable and does not deviate much from the in-distribution actions, which shows that the learned policy does not take actions with high uncertainty due to the penalty with uncertainty quantiﬁcation.
In addition, we observe that by incorporating the uncertainty quantiﬁcation and OOD sampling, the Q -functions obtained by PBRL does not suffer from the extrapolation error issue caused by distributional shift.
Ablation study.
In the following, we brieﬂy report the result of the ablation study.
We refer to §C and §D for the details.
(i) Number of bootstrapped- Q .
We attempt different numbers K of bootstrapped- Q in PBRL, and ﬁnd the performance to be reasonable for K ≥ 6 .
(ii) Penalization in (cid:98) T in .
We conduct experiments with the penalized reward discussed in §3.2 and ﬁnd the penalized reward does not improve the performance.
(iii) Factor β in in (cid:98) T in .
We conduct experiments with different β in from { 0 . 1 , 0 . 01 , 0 .
001 , 0 . 0001 } to study the sensitiveness.
Our experiments show that
PBRL performs the best for β in ∈ [0 . 0001 , 0 . 01] .
(iv) Factor β ood .
We use a large β ood at the initial training stage to enforce strong regularization over the OOD actions, and gradually decrease β ood in training.
We conduct experiments by setting β ood to different constants, and ﬁnd our decaying strategy generalizes better among tasks.
(v) The learning target of OOD actions .
We change the learning target of OOD actions to the most pessimistic zero target y =
and ﬁnd such a setting leads to overly pessimistic value functions with suboptimal performance.
(vi) Regularization types .
We conduct experiments with different regularization types, including our proposed OOD sampling,
L 2 -regularization, spectral normalization, pessimistic initialization, and no regularization.
We ﬁnd
OOD sampling the only reasonable regularization strategy and defer the complete report to §D.
C ONCLUSION
In this paper, we propose PBRL, an uncertainty-based model-free ofﬂine RL algorithm.
We propose bootstrapped uncertainty to guide the provably efﬁcient pessimism, and a novel OOD sampling technique to regularize the OOD actions in ofﬂine RL.
PBRL is closely related to the provable efﬁcient ofﬂine RL algorithm under the linear MDP assumption (Jin et al., 2021).
Our experiments show that PBRL outperforms several strong ofﬂine RL baselines in the D4RL environments, especially for suboptimal datasets.
PBRL exploits optimal trajectories contained in the suboptimal dataset and is less affected by the behavior policy.
Our illustrations suggest that incorporated with OOD sampling, PBRL produces reliable uncertainty quantiﬁcations and does not suffer from the extrapolation error.
R EPRODUCIBILITY S TATEMENT
The code of our work is available at the anonymous GitHub repository https://github.
com/review-anon/PBRL for the double-blind review.
The environments, datasets, and hyperparameters of our experiments are are given in the appendix.
All the lemmas and theorems of our work are proved and are discussed in details in our appendix.
The assumptions and settings for the theorems are also clariﬁed in the appendix.
A T HEORETICAL P ROOF A. B ACKGROUND OF LCB- PENALTY IN LINEAR MDP S
In this section, we introduce the provably efﬁcient LCB-penalty in linear MDPs (Abbasi-Yadkori et al., 2011; Jin et al., 2020; 2021).
We consider the setting of γ = 1 in the following.
In linear
MDPs, the feature map of the state-action pair takes the form of φ : S ×
A → R d , and the transition kernel and reward function are assumed to be linear in φ .
As a result, for any policy π , the stateaction value function is also linear in φ (Jin et al., 2020), that is,
[Equation]
The parameter w t can be solved in the closed-form by following the Least-Squares Value Iteration (LSVI) algorithm, which minimizes the following loss function,
[Equation]
(11) where V t +1 is the estimated value function in the ( t + 1) -th step, and r ( s i t , a i t )
+ V t +
( s i t +1 ) is the target of LSVI.
The explicit solution to (11) takes the form of (cid:98)
[Equation] [Equation]
m
[Equation]
i =
[Equation]
Here Λ t accumulate the state-action features from the training buffer.
Based on the solution of w t , the action-value function can be estimated by Q t ( s t , a t )
≈ (cid:98) w (cid:62) t φ ( s t , a t ) .
In addition, in ofﬂine RL with linear function assumption, the following LCB-penalty yields an uncertainty quantiﬁcation,
[Equation]
13) which measures the conﬁdence interval of the Q -functions with the given training data (AbbasiYadkori et al., 2011; Jin et al., 2020; 2021).
In ofﬂine RL, the pessimistic value function (cid:98) Q t ( s t , a t ) penalizes Q t by the uncertainty quantiﬁcation Γ lcb ( s t , a t ) as a penalty as follows,
[Equation]
) where w t is deﬁned in Eq.
Under the linear MDP setting, such pessimistic value iteration is known to be information-theoretically optimal (Jin et al., 2021).
In addition, exploration with Γ lcb ( s t , a t ) as a bonus is also provably efﬁcient in the online RL setting (Abbasi-Yadkori et al., 2011; Jin et al., 2020).
A. C ONNECTION BETWEEN THE BOOTSTRAPPED UNCERTAINTY AND Γ lcb
In the sequel, we considering a Bayesian linear regression perspective of LSVI in Eq.
According to the Bellman equation, the objective of LSVI is to approximate the Bellman target b it =
r ( s it , a it )
+ V t +
( s it +1 ) with the Q -function Q t , where V t +1 is the estimated value function in the ( t + 1) -th step.
In linear MDPs, We parameterize the Q -function by Q t ( s t , a t ) =
(cid:98) w (cid:62) t
φ ( s t , a t ) .
We further deﬁne the noise (cid:15) in this least-square problem as follows,
[Equation]
where w t is the underlying true parameter.
In the ofﬂine dataset with D in = { ( s it , a it , s it +1 ) } i ∈
[ m ] , we denote by (cid:98) w t the Bayesian posterior of w given the dataset D in .
In addition, we assume that we are given a Gaussian prior of the parameter w ∼ N (0 , I /λ ) as a non-informative prior.
The following Lemma establishes connections between bootstrapped uncertainty and the LCB-penalty Γ lcb .
Lemma 1 (Formal Version of Claim 1) .
We assume that (cid:15) follows the standard Gaussian distribution
N (0 , 1) given the state-action pair ( s it , a it ) .
It then holds for the posterior w t given D in that
Var
[Equation]
Proof.
The proof follows the standard analysis of Bayesian linear regression.
Under the assumption that (cid:15) ∼ N (0 , 1) , we obtain that
[Equation]
Recall that we have the prior distribution w ∼ N (0 , I /λ ) .
Our objective is to compute the posterior density (cid:98) w t = w | D in .
It holds from Bayes rule that
[Equation]
where p ( · ) denote the probability density function of the respective distributions.
Plugging (17) and the probability density function of Gaussian distribution into (18) yields
[Equation]
where we deﬁne
[Equation]
Then we obtain that (cid:98) w t = w | D in ∼ N ( µ t , Λ − 1 t ) .
It then holds for all ( s t , a t ) ∈ S
× A that
Var (cid:0) φ ( s it , a it ) (cid:62)
[Equation]
which concludes the proof.
In Lemma 1, we show that the standard deviation of the Q -posterior is equivalent to the LCBpenalty Var (cid:0) Q ( s t , a t ) (cid:1) =
φ ( s t , a t ) (cid:62) Λ − 1 t φ ( s t , a t ) introduced in §A.1.
Recall that our proposed bootstrapped uncertainty takes the form of
[Equation]
which is the standard deviation of the bootstrapped Q -functions.
Such bootstrapping serves as an estimation of the posterior of Q -functions (Osband et al., 2016).
Thus, our proposed uncertainty quantiﬁcation can be seen as an estimation of the LCB-penalty under the linear MDP assumptions.
In addition, under the tabular setting where the states and actions are ﬁnite, the LCB-penalty takes a simpler form, which we show in the following lemma.
Lemma 2.
In tabular MDPs, the bootstrapped uncertainty U ( s, a ) is approximately proportional to the reciprocal-count of ( s, a ) , that is,
[Equation]
Proof.
In tabular MDPs, we consider the joint state-action space d = |S| × |A| .
Then j -th stateaction pair can be encoded as a one-hot vector as φ ( s, a ) ∈ R d , where j ∈ [0 , d − 1] , thus is a special case of the linear MDP (Yang & Wang, 2019; Jin et al., 2020).
Speciﬁcally, we deﬁne φ ( s j , a j ) =
[Equation]
[Equation]
where the value of φ ( s j , a j ) is 1 at the j -th entry and 0 elsewhere.
Then the matrix Λ j = (cid:80) m i =
φ ( s i j , a i j ) φ ( s i j , a i j ) (cid:62) + λ · I is the sum of φ ( s j , a j ) φ ( s j , a j ) (cid:62) over ( s j , a j ) ∈ D in , which takes the form of
[Equation]
[Equation]
where the j -th diagonal element of Λ j is the corresponding counts for state-action ( s j , a j ) , i.e., n j = N s j ,a j .
It thus holds that
[Equation]
which concludes the proof.
A. R EGULARIZATION WITH OOD S AMPLING
In this section, we discuss how OOD sampling plays the role of regularization in RL, which regularizes the extrapolation behavior of the estimated Q -functions on OOD samples.
Similar to §A.2, we consider the setup of LSVI-UCB (Jin et al., 2020) under linear MDPs.
Speciﬁcally, we assume that the transition dynamics and reward function takes the form of
P t
[Equation]
(26) where the feature embedding φ : S ×A (cid:55)→ R d is known.
We further assume that the reward function r : S ×A (cid:55)→ [0 , 1] is bounded and the feature is bounded by (cid:107) φ (cid:107) 2 ≤ 1 .
Given the dataset D in , LSVI iteratively minimizes the least-square loss in Eq.
Recall that the explicit solution to Eq.
(11) takes the form of
[Equation]
We remark that for the regression problem in Eq.
(11), the L 2 -regularizer λ · (cid:107) w (cid:107) 22 enforces a Gaussian prior under the notion of Bayesian regression.
Such regularization ensures that the linear function approximation φ (cid:62) w it extrapolates well outside the region covered by the dataset D in .
However, as shown in §D, we observe that such L 2 -regularization is ineffective for ofﬂine DRL.
To this end
[Equation]
we propose OOD sampling in our proposed PBRL.
To demonstrate the effectiveness of
OOD sampling as a regularizer
[Equation]
we consider the following least-squares loss with OOD sampling and without the L 2 -regularizer,
[Equation]
The explicit solution of Eq.
(28) takes the form of
[Equation]
where
[Equation]
Hence, if we further set y = 0 for all ( s ood , a ood , y ) ∈
D ood , then (28) enforces a Gaussian prior with the covariance matrix Λ − 1 ood , where we deﬁne
[Equation]
Speciﬁcally, if we further enforce D ood = { ( s j , a j , 0) } j ∈ [ d ] with φ ( s j , a j ) =
λ · e j , where
e j ∈ R
d is the unit vector with the j -th entry equals one, it further holds that Λ ood = λ · I and Eq.
(28) is equivalent with Eq.
In addition, under the tabular setting, by following the same proof as in
Lemma 2, having such OOD samples in the training is equivalent with setting the count in Eq.
(25) to be
[Equation]
where N s j ,a j is the occurrence of ( s j , a j ) in the dataset and N ood s j ,a j is the occurrence of ( s j , a j ) in the OOD dataset.
However, to enforce such regularizer without affecting the estimation of value functions, we need to set the target y of the OOD samples to zero.
In practice, we ﬁnd such setup to be overly pessimistic.
Since the Q -network is smooth, such a strong regularizer enforces the Q -functions to be zero for state-action pairs from both the ofﬂine data and OOD data, as show in Fig.
12 and Fig.
13 of §C.
We remark that adopting a nonzero OOD target y does not hinder the effect of regularization as it still imposes the same prior in the covariate matrix (cid:101) Λ t .
However, adopting nonzero OOD target may introduce additional bias in the value function estimation and the corresponding uncertainty quantiﬁcation.
To maintain a consistent and pessimistic estimation of value functions, one needs to carefully design the nonzero OOD target y .
To this end, we recall the deﬁnition of a ξ -uncertainty quantiﬁer in Deﬁnition 1 as follows.
Deﬁnition 2 ( ξ -Uncertainty Quantiﬁer (Jin et al., 2021)) .
The set of penalization { Γ t } t ∈ [ T ] forms a ξ -Uncertainty Quantiﬁer if it holds with probability at least 1 − ξ that
[Equation]
for all ( s, a ) ∈ S × A , where T is the Bellman operator and (cid:98) T is the empirical Bellman operator that estimates T based on the data.
We remark that here we slightly abuse the notation T of Bellman operator and write T V ( s, a ) =
E [ r ( s, a )+ V ( s (cid:48) ) | s, a ] .
Under the linear MDP setup, the empirical estimation (cid:98) T V t +1 is obtained via ﬁtting the least-squares loss in Eq.
Thus, the empirical estimation (cid:98) T V t +1 takes the following explicit form,
[Equation]
where (cid:101) w t is the solution to the least-squares problem deﬁned in Eq.
We remark that such ξuncertainty quantiﬁer plays an important role in the theoretical analysis of RL algorithms, both for online RL and ofﬂine RL (Abbasi-Yadkori et al., 2011; Azar et al., 2017; Wang et al., 2020a; Jin et al., 2020; 2021; Xie et al., 2021a;b).
Our goal is therefore to design a proper OOD target y such that we can obtain ξ -uncertainty quantiﬁer based on the bootstrapped value functions.
Our design is motivated by the following theorem.
Theorem 2.
Let Λ ood (cid:23) λ · I .
For all the OOD datapoint ( s ood , a ood , y ) ∈
D ood , if we set y =
T V t +1 ( s ood , a ood ) , it then holds for β t
= O (cid:0) T · √ d · log ( T/ξ ) (cid:1)
that
[Equation]
forms a valid ξ -uncertainty quantiﬁer.
Proof.
Recall that we deﬁne the empirical Bellman operator (cid:98) T as follows,
[Equation]
It sufﬁces to upper bound the following difference between the empirical Bellman operator and
Bellman operator
[Equation]
Here we deﬁne w t as follows
[Equation]
where θ and ψ are deﬁned in Eq.
It then holds that
[Equation]
where we plug the solution of (cid:101) w t in Eq.
Meanwhile, by the deﬁnitions of (cid:101) Λ t and w t in Eq.
(30) and Eq.
(32), respectively, we have
[Equation]
Plugging (34) into (33) yields
[Equation]
where we deﬁne
[Equation]
Following standard analysis based on the concentration of self-normalized process (Abbasi-Yadkori et al., 2011; Azar et al., 2017; Wang et al., 2020a; Jin et al., 2020; 2021) and the fact that Λ ood (cid:23) λ · I , it holds that [Equation]
[Equation]
with probability at least 1 − ξ , where β t =
O (cid:0) T · √ d · log ( T/ξ ) (cid:1) .
Meanwhile, by setting y =
T V t +1 ( s ood , a ood ) , it holds that (ii) = 0 .
Thus, we obtain from (35) that
[Equation]
for all ( s, a ) ∈ S × A with probability at least 1 − ξ .
Theorem 2 allows us to further characterize the optimality gap of pessimistic value iteration.
In particular, the following corollary holds.
Corollary 1 (Jin et al.
Under the same conditions as Theorem 2, it holds that
[Equation]
Proof.
See e.g., Jin et al.
(2021) for a detailed proof.
We remark that the optimality gap in Corollary 1 is information-theoretically optimal under the linear
MDP setup with ﬁnite horizon (Jin et al., 2021).
Intuitively, for ofﬂine dataset with sufﬁciently good coverage on the optimal trajectories such as the experience from experts, such gap is small.
For dataset collected from random policy, such gap can be large.
Our experiments also support such intuition empirically, where the score obtained by training with the expert dataset is higher than that with the random dataset.
Theorem 2 shows that if we set y = T V t +1 ( s ood , a ood ) , then our estimation based on disagreement among ensembles is a valid ξ -uncertainty quantiﬁer.
However, such OOD target is impossible to obtain in practice as it requires knowing the transition at the OOD datapoint ( s ood , a ood ) .
In practice, if TD error is sufﬁciently minimized, then Q t +1 ( s, a ) should well estimate the target T V t +1 .
Thus, in practice, we utilize
[Equation]
as the OOD target, where we introduce additional penalization Γ t +1 ( s ood , a ood ) to enforce the pessimistic value estimation.
In addition, we remark that in theory, we require that the embeddings of the OOD sample are isotropic, in the sense that the eigenvalues of the corresponding covariate matrix Λ ood are lower bounded.
Such isotropic property can be guaranteed by randomly generating states and actions. In practice, we ﬁnd that randomly generating states is more expensive than randomly generating actions. Meanwhile, we observe that randomly generating actions alone are sufﬁcient to guarantee reasonable empirical performances since the generated OOD embeddings are sufﬁciently isotropic. Thus, in our experiments, we randomly generate OOD actions according to our current policy and sample OOD states from the in-distribution dataset.
B I MPLEMENTATION D ETAIL B. A LGORITHMIC D ESCRIPTION
1: Initialize: K bootstrapped Q -networks and target Q -networks with parameter θ and θ − , policy π with parameter ϕ , and hyper-parameters β in , β ood 2: Initialize: total training steps H , current frame h =3: while h < H do 4:
Sample mini-batch transitions ( s, a, r, s (cid:48) ) from the ofﬂine dataset D in 5: # Critic Training for ofﬂine data.
Calculate the bootstrapped uncertainty U θ − ( s (cid:48) , a (cid:48) ) through the target-networks.
Calculate the Q -target in Eq.
(4) and the resulting TD-loss | Q kθ ( s, a ) − (cid:98) T in Q k ( s, a ) | 2 .
8: # Critic Training for OOD data 9:
Perform OOD sampling and obtains N ood OOD actions a ood for each s .
Calculate the bootstrapped uncertainty U θ ( s, a ood ) for OOD actions through Q -networks.
Calculate the Q -target in Eq.
(5) and the pseudo TD-loss | Q kθ ( s, a ood ) − (cid:98) T ood Q k ( s, a ood ) | 2 .
Minimize | Q kθ ( s, a ) − (cid:98) T in Q k ( s, a ) | 2 + | Q kθ ( s, a ood ) − (cid:98) T ood Q k ( s, a ood ) | 2 to train θ by SGD.
13: # Actor Training 14:
Improve π ϕ by maximizing min k Q k ( s, a π ) − log π ϕ ( a π | s ) with entropy regularization.
Update the target Q -network via θ − ← (1 − τ ) θ − + τθ .
16: end while
B. H YPER - PARAMETERS
Most hyper-parameters of PBRL follow the SAC implementations in https://github.com/ rail-berkeley/rlkit .
We use the hyper-parameter settings in Table 2 for all the Gym domain tasks.
We use different settings of β in and β ood for the experiment for Adroit domain and ﬁx the other hyper-parameters the same as Table 2.
See §E for the setup of Adroit.
In addition, we use the same settings for discount factor, target network smoothing factor, learning rate, and optimizers as
CQL (Kumar et al., 2020).
[Table]
Baselines.
We conduct experiments on D4RL with the latest ‘v2’ datasets.
The dataset is released at http://rail.eecs.berkeley.edu/datasets/offline_rl/gym_mujoco_v2_
old/ .
We now introduce the implementations of baselines.
(i)
The implementation of CQL (Kumar et al., 2020) is adopted from the ofﬁcial implementation at https://github.com/ aviralkumar2907/CQL .
In our experiment, we remove the BC warm-up stage since we ﬁnd CQL performs better without warm-up for ‘v2’ dataset.
(ii) For BEAR (Kumar et al., 2019), UWAC (Wu et al., 2021) and MOPO (Yu et al., 2020), we adopt their ofﬁcial implementations at https://github.com/aviralkumar2907/BEAR , https://github.com/ apple/ml-uwac , and https://github.com/tianheyu927/mopo , respectively.
We adopt their default hyper-parameters in training.
(iii)
Since the original paper of TD3-BC (Fujimoto & Gu, 2021) reports the performance of Gym in ‘v2’ dataset in the appendix section, we directly cite the reported scores in Table 1.
The learning curve reported in Fig.
2 is trained by implementation released at https://github.com/sfujim/TD3_BC .
Computational cost comparison.
In the sequel, we compare the computational cost of PBRL against CQL.
We conduct such comparison based on the Halfcheetah-medium-v2 task.
We measure the number of parameters, GPU memory, and runtime per epoch (1K gradient steps) for both PBRL and CQL in the training.
We run experiments with a single A100 GPU.
We summarize the result in Table 3.
We observe that, similar to the other ensemble-based methods such as Bootstrapped
DQN (Osband et al., 2016), IDS (Nikolov et al., 2019), and Sunrize (Lee et al., 2021a), our method requires extra computation to handle the ensemble of Q -networks.
In addition, we remark that a large proportion of computation for CQL is due to the costs of logsumexp over multiple sampled actions (Fujimoto & Gu, 2021), which we do not require.
[Table]
B. R EMARK ON FORMULATIONS OF PESSIMISM IN ACTOR AND CRITIC
We use different formulations to enforce pessimism in actor and critic.
In the critic training, we use the penalized Q-function as in Eq.
(4) and Eq.
While in the actor training, we use the minimum of ensemble Q-function as in Eq.
According to the analysis in EDAC [6], using the minimum of ensemble Q-function min j =1 ,...,K Q j as the target is approximately equivalent to using ¯
Q − β 0 · Std( Q j ) with a ﬁxed β 0 as the target.
In contrast, in the critic training of PBRL, we tune different factors (i.e., β in and β ood ) for the in-distribution target and the OOD target, which yields better performance for the critic estimation.
In the actor training, since we already have pessimistic Q-functions learned by the critic, it is not necessary to enforce large penalties in the actor.
To see such a fact, we refer to the ablation study in Fig.
9, where utilizing the mean as the target achieves reasonable performances.
We remark that taking the minimum as the target avoids possible large values at certain state-action pairs, which may arise due to the numerical computation in ﬁtting neural networks.
As suggested by our ablation study, taking the minimum among the ensemble of Q-functions as the target achieves the best performance.
Thus, we use the minimum among the ensemble of Q-functions as the target in PBRL.
C A BLATION S TUDY
In this section, we present the ablation study of PBRL.
In training, we perform online interactions to evaluate the performance for every 1K gradient steps.
Since we run each method for 1M gradient steps totally, each method is evaluated 1000 times in training.
The experiments follow such evaluation criteria, and each curve is drawn by 1000 evaluated scores through online interaction.
Number of bootstrapped- Q .
We conduct experiments with different numbers of bootstrapped
Q -networks in PBRL, and present the performance comparison in Fig.
We observe that the performance of PBRL is improved with the increase of the bootstrapped networks.
We remark that since the training of ofﬂine RL is more challenging than online RL, it is better to use sufﬁcient bootstrapped Q -functions to obtain a reasonable estimation of the non-parametric Q -posterior.
We observe from Fig.
4 that using K = 6 , 8 , and 10 yields similar ﬁnal scores, and a larger K leads to more stable performance in the training process.
We adopt K = 10 for our implementation.
[Figure]
Uncertainty of in-distribution target.
We compare different kinds of uncertainty penalization in (cid:98) T in for in-distribution data.
(i) Penalizing the immediate reward only.
(ii) Penalizing the next- Q value only, which is adopted in PBRL.
(iii) Penalizing both the immediate reward and next- Q value.
We present the comparison in Fig.
We observe that the target- Q penalization performs the best, and adopt such penalization in the proposed PBRL algorithm.
[Figure]
Tuning parameter β in .
We conduct experiments with β in ∈ { 0 . 1 , 0 . 01 , 0 .
001 , 0 . 0001 } to study the sensitivity of PBRL to the tuning parameter β in for the in-distribution target.
We present the comparison in Fig.
We observe that in the ‘medium’ dataset generated by a single policy, the performance of PBRL is insensitive to β in .
One possible reason is that since the ‘medium’ dataset is generated by a single policy, the ofﬂine dataset tends to concentrate around a few trajectories and has low uncertainty.
Thus, the magnitude of β in has a limited effect on the penalty.
Nevertheless, in the ‘medium-replay’ dataset, since the data is generated by various policies, the uncertainty of ofﬂine data is larger than that of the ‘medium’ dataset (as shown in Fig. 3).
Correspondingly, the performance of PBRL is affected by the magnitude of β in .
Our experiment shows that PBRL performs the best for β in ∈ [0 . 0001 , 0 . 01] .
We adopt β in = 0 .
01 for our implementation.
[Figure]
Tuning parameter β ood .
We use a large β ood initially to enforce strong OOD regularization in the beginning of training, and then decrease β ood linearly while the training evolves.
We conduct experiments with constant settings of β ood ∈ { 0 . 01 , 0 .
We observe that in the ‘medium’ dataset, a large β ood = 1 . 0 performs the best since the samples are generated by a ﬁxed policy with a relatively concentrated in-distribution dataset.
Thus, the OOD samples tend to have high uncertainty and are less trustworthy.
In contrast, in the ‘medium-replay’ dataset, a small β ood ∈ { 0 . 1 , 0 . 01 } performs reasonably well since the mixed dataset has larger coverage of the state-action space and the uncertainty of OOD data is smaller than that of the ‘medium’ dataset.
Thus, adopting a smaller β ood for the ‘medium-replay’ dataset allows the agent to exploit the OOD actions and gain better performance.
To match all possible situations, we propose the decaying strategy.
Empirically, we ﬁnd such a decaying strategy performs well in both the ‘medium’ and the ‘medium-replay’ datasets.
[Figure]
We also record the Q -value for ( s, a ) ∼ D in in the training process.
As shown in Fig.
8, since both the in-distribution actions and OOD actions are represented by the same Q -network, a large constant β ood makes the Q-value for in-distribution actions overly pessimistic and leads to suboptimal performance.
It is desirable to use the decaying strategy for β ood in practice
[Figure]
Actor training.
We evaluate different actor training schemes in Eq.
(7), i
[Figure]
e., the actor follows the gradients of ‘min’ (in PBRL), ‘mean’ or ‘max’ value among K Q-functions.
The result in Fig.shows training the actor by maximizing the ‘min’ among ensemble- Q performs the best.
In the ‘medium-replay’ dataset, since the uncertainty estimation is difﬁcult in mixed data, taking ‘mean’ in actor can be unstable in training.
Taking ‘max’ in actor performs worse in both tasks due to overestimation of Q -functions.
[Figure]
Number of OOD samples.
The pessimism in critic training is implemented by sampling OOD actions and then performing uncertainty penalization, as shown in Eq.
In each training step, we perform OOD sampling and sample N ood actions from the learned policy.
We perform an ablation study with N ood ∈ { 0 , 2 , 5 , 10 } .
According to Fig 10, the performance is poor without OOD sampling (i.e., N ood = 0) .
We ﬁnd the performance becomes better with a small number of OOD actions (i.e., N ood = 2) .
Also, PBRL is robust to different settings of N ood .
OOD Target.
In §A.3, we show that setting the learning target of OOD samples to zero enforces a
Gaussian prior to Q -function under the linear MDP setting.
However, in DRL, we ﬁnd such a setup leads to overly pessimistic value function and performs poorly in practice, as shown in Fig.
Speciﬁcally, we observe that such a strong regularizer enforces the Q -functions to be close to zero for both in-distribution and OOD state-action pairs, as shown in Fig.
12 and Fig. 13.
In contrast,
[Figure]
our proposed PBRL target performs well and does not yield large extrapolation errors, as shown in
Fig.
12 and Fig.
[Figure] [Figure]
[Figure]
D R EGULARIZATION FOR PBRL
In this section, we compare different regularization methods that are popular in other Deep Learning (DL) literature to solve the extrapolation error in ofﬂine RL.
Speciﬁcally, we compare the following regularization methods.
• None .
We remove the OOD-sampling regularizer in PBRL and train bootstrapped Q -functions solely based on the ofﬂine dataset through (cid:98) T in .
• L 2 -regularizer.
We remove the OOD-sampling regularizer and use L 2 normalization instead.
As we discussed in §3.3, in linear MDPs, LSVI utilizes L 2 regularization to control the extrapolation behavior of Q -functions in OOD samples.
In DRL, we add the L 2 -norm of weights in the Q -network in loss functions to conduct L 2 -regularization.
We attempt two scale factors { 1 e − 2 , 1 e − 4 } in our experiments.
• Spectral Normalization (SN).
We remove the OOD-sampling regularizer and use SN instead.
SN constrains the Lipschitz constant of layers, which measures the smoothness of the neural network.
Recent research (Gogianu et al., 2021) shows that SN helps RL training when applied to speciﬁc layers of the Q -network.
In our experiment, we follow Gogianu et al.
and consider two cases, namely, applying SN in the output layer (denoted as SN[-1] ) and applying SN in both the output layer and the one before it (denoted as SN[-1,-2] ), respectively.
• Pessimistic Initialization (PI).
Optimistic initialization is simple and efﬁcient for RL exploration, which initializes the Q -function for all actions with a high value.
In online RL, such initialization encourages the agent to explore all actions in the interaction.
Motivated by this method, we attempt a pessimistic initialization to regulate the OOD behavior of the Q -network.
In our implementation, we draw the initial value of weights and a bias of the Q -networks from
the uniform distribution Unif(a , b) .
We try two settings in our experiment, namely, (i) PI-small that sets ( a, b ) to ( − 0 .
2 , 0) , and (ii) PI-large that sets ( a, b ) to ( − 1 . 0 , 0) .
In both settings, we remove the OOD sampling and use PI instead.
We illustrate (i) the normalized performance in the training process, (ii) the Q -value along the trajectory of training policy and (iii) the uncertainty quantiﬁcation along the trajectory of training policy.
We present the results in Fig.
14, Fig. 15, and Fig.
16, respectively.
In the sequel, we discuss the empirical results.
• We observe that OOD sampling is the only regularization method with reasonable performance.
Though L 2 -regularization and SN yields reasonable performance in supervised learning, they do not perform well in ofﬂine RL.
In the ‘medium-replay’ dataset, we observe that PI and SN can gain some score in the early stage of training.
Nevertheless, the performance drops quickly along with the training process.
We conjecture that both PI and SN have the potential to be effective with additional parameter tuning and algorithm design.
In conclusion, previous regularization methods for DL and RL is not sufﬁcient to handle the distribution shift issue in ofﬂine RL.
Combining such regularization techniques with the policy constraints and conservatism methods may lead to improved empirical performance, which we leave for future research.
[Figure] [Figure]
[Figure]
E E XPERIMENTS IN A NDROIT D OMAIN
In Adroit, the agent controls a 24 -DoF robotic hand to hammer a nail, open a door, twirl a pen, and move a ball, as shown in Fig. 17.
The Adroit domain includes three dataset types, namely, demonstration data from a human (‘human’), expert data from an RL policy (‘expert’), and ﬁfty-ﬁfty mixed data from human demonstrations and an imitation policy (‘cloned’).
The adroit tasks are more challenging than the Gym domain in task complexity.
In addition, the use of human demonstration in the dataset also makes the task more challenging.
We present the normalized scores in Table 4.
We set β in = 0 .
0001 and β ood = 0 .
The other hyper-parameters in Adroit follows Table 2.
In Table 4, the scores of BC, BEAR, and CQL are adopted from the D4RL benchmark (Fu et al., 2020).
We do not include the scores of MOPO (Yu et al., 2020) as it is not reported and we cannot ﬁnd suitable hyper-parameters to make it work in the Adroit domain.
We also attempt TD3-BC (Fujimoto et al., 2018) with different BC weights and fail in getting reasonable score for most of the tasks.
In addition, since UWAC (Wu et al., 2021) has a different evaluation process, we re-train
UWAC with the ofﬁcial implementation and report the scores in Table 4.
In Table 4, we add the average score without ‘expert’ dataset since the scores of ‘expert’ dataset dominate that of the rest of the datasets.
We ﬁnd CQL (Kumar et al., 2020) and BC have the best average performance among all baselines.
Meanwhile, we observe that PBRL slightly outperforms
CQL and BC.
We remark that the human demonstrations stored in ‘human’ and ‘cloned’ are inherently different from machine-generated ‘expert’ data since (i) the human trajectories do not follow the Markov property, (ii) human decision may be affected by unobserved states such as prior knowledge, distractors
[Table]
and the action history, and (iii) different people demonstrate differently as their solution policies.
Such characteristic makes the ‘human’ and ‘cloned’ dataset challenging for ofﬂine RL algorithms.
In addition, we remark that the recent study in robot learning from human demonstration also encounter such challenges (Mandlekar et al., 2021)
[Table] [Table]
F R ELIABLE E VALUATION FOR S TATISTICAL U NCERTAINTY
Recent research (Agarwal et al., 2021) proposes reliable evaluation principles to address the statistical uncertainty in RL.
Since the ordinary aggregate measures like mean can be easily dominated by a few outlier scores, Agarwal et al.
(2021) presents several efﬁcient and robust alternatives that are not unduly affected by outliers and have small uncertainty even with a handful of runs.
In this paper, we adopt these evaluation methods for each method in Gym domain with M task ∗ N seed runs.
• Stratiﬁed Bootstrap Conﬁdence Intervals.
The Conﬁdence Intervals (CIs) for a ﬁnite-sample score estimates the plausible values for the true score.
Bootstrap CIs with stratiﬁed sampling can be applied to small sample sizes and is better justiﬁed than sample standard deviations.
• Performance Proﬁles.
Performance proﬁles reveal performance variability through score distributions.
A score distribution shows the fraction of runs above a certain score and is given by ˆ F ( τ )
=
ˆ F ( τ ; x 1: M, 1: N ) =
1 M (cid:80) Mm =1 1 N (cid:80) Nn
=1 1 [ x m,n ≥ τ ] .
• Aggregate Metrics .
Based on bootstrap CIs, we can extract aggregate metrics from score distributions, including median, mean, interquartile mean (IQM), and optimality gap.
IQM discards the bottom and top 25% of the runs and calculates the mean score of the remaining 50% runs.
Optimality gap calculates the amount of runs that fail to meet a minimum score of η = 50 . 0 .
The result comparisons are give in Fig.
18 and Fig.
Speciﬁcally, Fig.
18 shows aggregate metrics based on 95% bootstrap CIs, and Fig.
19 shows performance proﬁles based on score distributions.
For both evaluations, our PBRL and PBRL-prior outperforms other methods with small variability.
[Figure] [Figure]