Why So Pessimistic? Estimating Uncertainties for Ofﬂine RL through Ensembles, and Why Their Independence Matters
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Motivated by the success of ensembles for uncertainty estimation in supervisedlearning, we take a renewed look at how ensembles of Q -functions can be leveragedas the primary source of pessimism for ofﬂine reinforcement learning (RL).
Webegin by identifying a critical ﬂaw in a popular algorithmic choice used by manyensemble-based RL algorithms, namely the use of shared pessimistic target valueswhen computing each ensemble member’s Bellman error.
Through theoreticalanalyses and construction of examples in toy MDPs, we demonstrate that sharedpessimistic targets can paradoxically lead to value estimates that are effectivelyoptimistic .
Given this result, we propose MSG, a practical ofﬂine RL algorithmthat trains an ensemble of Q -functions with independently computed targets basedon completely separate networks, and optimizes a policy with respect to the lowerconﬁdence bound of predicted action values.
Our experiments on the popular
D4RL and RL Unplugged ofﬂine RL benchmarks demonstrate that on challengingdomains such as antmazes, MSG with deep ensembles surpasses highly well-tunedstate-of-the-art methods by a wide margin.
Additionally, through ablations onbenchmarks domains, we verify the critical signiﬁcance of using independentlytrained Q -functions, and study the role of ensemble size.
Finally, as using separatenetworks per ensemble member can become computationally costly with largerneural network architectures, we investigate whether efﬁcient ensemble approximations developed for supervised learning can be similarly effective, and demonstratethat they do not match the performance and robustness of MSG with separatenetworks, highlighting the need for new efforts into efﬁcient uncertainty estimationdirected at RL.
Introduction
Ofﬂine reinforcement learning (RL), also referred to as batch RL [1], is a problem setting in whichone is provided a dataset of interactions with an environment in the form of a Markov decision process(MDP), and the goal is to learn an effective policy exclusively from this ﬁxed dataset.
Ofﬂine RLholds the promise of data-efﬁciency through data reuse, and improved safety due to minimizing theneed for policy rollouts.
As a result, ofﬂine RL has been the subject of signiﬁcant renewed interest inthe machine learning literature [2].
One common approach to ofﬂine RL in the model-free setting is to use approximate dynamicprogramming (ADP) to learn a Q -value function via iterative regression to backed-up target values .
The predominant algorithmic philosophy with most success in ADP-based ofﬂine RL is to encourageobtained policies to remain close to the support set of the available ofﬂine data.
A large variety ofmethods have been developed for enforcing such constraints, examples of which include regularizingpolicies with behavior cloning objectives [3, 4], performing updates only on actions observed inside[5, 6, 7, 8] or close to [9] the ofﬂine dataset, and regularizing value functions to underestimate thevalue of actions not seen in the dataset [10, 11, 12].
The need for such regularizers arises from inevitable inaccuracies in value estimation when functionapproximation, bootstrapping, and off-policy learning – i.e. The Deadly Triad [13] – are involved.
Inofﬂine RL in particular, such inaccuracies cannot be resolved through additional interactions with the
MDP.
Thus, remaining close to the ofﬂine dataset limits opportunities for catastrophic inaccuraciesto arise.
However, recent works have argued that the aforementioned constraints can be overlypessimistic, and instead opt for approaches that take into consideration the uncertainty about thevalue function [14, 15, 16], thus re-focusing the ofﬂine RL problem to that of deriving accurate lowerconﬁdence bounds (LCB) of Q -values.
In the empirical supervised learning literature, deep network ensembles (deﬁnition in Appendix K)and their more efﬁcient variants have been shown to be the most effective approaches for uncertaintyestimation, towards learning calibrated estimates and conﬁdence bounds with modern neural networkfunction approximators [17].
Motivated by this, in our work we take a renewed look into Q -ensembles,and study how to leverage them as the primary source of pessimism for ofﬂine RL.
In deep RL, a very popular algorithmic choice is to use an ensemble of Q -functions to obtain pessimistic value estimates and combat overestimation bias [18].
Speciﬁcally, in the policy evaluationprocedure, all Q -networks are updated towards a shared pessimistic temporal difference target .
Similarly in ofﬂine RL, in addition to the main ofﬂine RL objective that they propose, several existingmethods use such Q -ensembles [10, 3, 19, 20, 21, 22, 23, 8].
We begin by mathematically characterizing a critical ﬂaw in the aforementioned ensembling procedure.
Speciﬁcally, we demonstrate thatusing shared pessimistic targets can paradoxically lead to Q -estimates which are in fact optimistic !
We verify our ﬁnding by constructing pedagogical toy MDPs.
These results demonstrate that theformulation of using shared pessimistic targets is fundamentally ill-formed.
To resolve this problem, we propose Model Standard-deviation Gradients (MSG), an ensemble-basedofﬂine RL algorithm.
In MSG, each Q -network is trained independently , without sharing targets .
Crucially, ensembles trained with independent target values will always provide pessimistic valueestimates.
The pessimistic lower-conﬁdence bound (LCB) value estimate – computed as the meanminus standard deviation of the Q -value ensemble – is then used to update the policy being trained.
Evaluating MSG on the established D4RL [24] and RL Unplugged [25] benchmarks for ofﬂine
RL, we demonstrate that MSG matches, and in the more challenging domains such as antmazes,signiﬁcantly exceeds the prior state-of-the-art.
Additionally, through a series of ablation experimentson benchmark domains, we verify the signiﬁcance of our theoretical ﬁndings, study the role ofensemble size, and highlight the settings in which ensembles provide the most beneﬁt.
The use of ensembles will inevitably be a computational bottleneck when applying ofﬂine RL todomains requiring large neural network models.
Hence, as a ﬁnal analysis, we investigate whetherthe favorable performance of MSG can be obtained through the use of modern efﬁcient ensembleapproaches which have been successful in the supervised learning literature [26, 27, 28, 17].
Wedemonstrate that while efﬁcient ensembles are competitive with the state-of-the-art on simpler ofﬂine
RL benchmark domains, similar to many popular ofﬂine RL methods they fail on more challengingtasks, and cannot recover the performance and robustness of MSG using full ensembles with separateneural networks.
Our work highlights some of the unique and often overlooked challenges of ensemble-based uncertainty estimation in ofﬂine RL.
Given the strong performance of MSG, we hope our work motivatesincreased focus into efﬁcient and stable ensembling techniques directed at RL, and that it highlightsintriguing research questions for the community of neural network uncertainty estimation researcherswhom thus far have not employed sequential domains such as ofﬂine RL as a testbed for validatingmodern uncertainty estimation techniques.
Related Work
Uncertainty estimation is a core component of RL, since an agent only has a limited view intothe mechanics of the environment through its available experience data.
Traditionally, uncertaintyestimation has been key to developing proper exploration strategies such as upper conﬁdence bound(UCB) and Thompson sampling [29], in which an agent is encouraged to seek out paths whereits uncertainty is high.
Ofﬂine RL presents an alternative paradigm, where the agent must actconservatively and is thus encouraged to seek out paths where its uncertainty is low [14].
In eithercase, proper and accurate estimation of uncertainties is paramount.
To this end, much research hasbeen produced with the aim of devising provably correct uncertainty estimates [30, 31, 32], or at least bounds on uncertainty that are good enough for acting exploratorily [33] or conservatively [34].
However, these approaches require exceedingly simple environment structure, typically either a ﬁnitediscrete state and action space or linear spaces with linear dynamics and rewards.
While theoretical guarantees for uncertainty estimation are more limited in practical situations withdeep neural network function approximators, a number of works have been able to achieve practicalsuccess, for example using deep network analogues for count-based uncertainty [35], Bayesianuncertainty [36, 37], and bootstrapping [38, 39].
Many of these methods employ ensembles.
Infact, in continuous control RL, it is common to use an ensemble of two value functions and usetheir minimum for computing a target value during Bellman error minimization [18].
A numberof works in ofﬂine RL have extended this to propose backing up minimums or lower conﬁdencebound estimates over larger ensembles [3, 10, 19, 20, 22, 23, 21].
In our work, we continue to ﬁndthat ensembles are extremely useful for acting conservatively, but the manner in which ensemblesare used is critical.
Speciﬁcally our proposed MSG algorithm advocates for using independentlylearned ensembles, without sharing of target values, and this important design decision is supportedby empirical evidence.
The widespread success of ensembles for uncertainty estimation in RL echoes similar ﬁndings insupervised deep learning.
While there exist proposals for more technical approaches to uncertaintyestimation [40, 41, 42], ensembles have repeatedly been found to perform best empirically [26, 43].
Much of the active literature on ensembles in supervised learning is concerned with computationalefﬁciency, with various proposals for reducing the compute or memory footprint of training andinference on large ensembles [28, 44, 27].
While these approaches have been able to achieveimpressive results in supervised learning, our empirical results suggest that their performance sufferssigniﬁcantly in challenging ofﬂine RL settings compared to deep ensembles.
Pessimistic Q -Ensembles: Independent or Shared Targets?
In this section we identify a critical ﬂaw in how ensembles are commonly employed – in ofﬂineas well as online RL – for obtaining pessimistic value estimates [10, 3, 19, 20, 21, 22, 23, 8, 21],which can paradoxically lead to an optimism bonus!
We begin by mathematically characterizing thisproblem and presenting a simple ﬁx.
Subsequently, we leverage our results to construct pedagogicaltoy MDPs demonstrating the practical importance of the identiﬁed problem and solution.
3.1 Mathematical Characterization
[Equation] [Equation]
[Equation]
the trained ensem ble
We assume access to a dataset D composed of( s, a, r, s (cid:48) ) transition tuples from a Markov Decision Process (MDP) determined by a tuple
M = (cid:104)S , A , R , P , γ (cid:105) , corresponding to statespace, action space, reward function, transitionsdynamics, and discount, respectively.
As is standard in RL, we do not assume any knowledge of
R , P , other than that implicitly provided by thedataset D .
In this section, for clarity of exposition, we assume that the policies we consider aredeterministic , and that our MDPs do not haveterminal states.
We consider Q -value ensemble members givenby a parameterization Q
θ i , where i indexes intosome set Z , which is ﬁnite in practice but maybe inﬁnite or uncountable in theory.
We assume Z has an associated probability space allowing us tomake expectation E or variance V computations over the ensemble members.
Given a ﬁxed policy π ,a general dynamic programming based procedure for obtaining pessimistic value estimates is outlinedby the iterative regression described in the box above.
A key algorithmic choice in this recipe is where pessimism should be introduced.
This can bedone by either (a) pessimistically aggregating Q -values after training, i.e. inside Step 3, or (b)also incorporating pessimism during Step 2, by using a shared pessimistic target value y .
Throughour review of the ofﬂine RL (as well as online RL) literature, we have observed that the mostcommon approach is the latter, where the targets are pessimistic, shared, and identical acrossensemble members [10, 3, 19, 20, 21, 22, 23, 8].
Speciﬁcally, they are computed as, y i ( r, s (cid:48) , π ) =
PO( { r + γQ θ i ( s (cid:48) , π ( s (cid:48) )) , ∀ i ∈ Z } ) with PO being a desired pessimism operator aggregating the
TD target values of the ensemble members (e.g. “mean minus standard deviation", or “minimum").
In this section, our goal is to compare these two alternative approaches.
For our analysis, we will use“mean minus standard deviation" (a lower conﬁdence bound (LCB)) as our pessimism operator, anduse the notation Q LCB in place of Q pessimistic (deﬁned in the box above).
Under the LCB pessimismoperator we will have:
Independent Targets (Method 1):
[Equation]
Shared Targets (Method 2):
[Equation]
For both we have:
[Equation]
To characterize the form of Q LCB when using complex neural networks, we refer to the work on 155 inﬁnite-width neural networks, namely the Neural Tangent Kernel (NTK) [45].
We consider Q -valueensemble members, Q θ i , which all share the same inﬁnite-width neural network architecture (andthus the same NTK parameterization).
As noted in the algorithm box above, and as is the case indeep ensembles [43], the only difference amongst ensemble members Q
θ i is in their initial weightsθ i sampled from the neural network’s initial weight distribution.
Before presenting our results, we establish some notation relevant to the inﬁnite-width and NTKregime.
Let X , R, X (cid:48) denote data matrices containing ( s, a ) , r , and ( s (cid:48) , π ( s (cid:48) )) appearing in theofﬂine dataset D ; i.e., the k -th transition ( s, a, r, s (cid:48) ) in D is represented by the k -th rows in X , R, X (cid:48) .
Let A, B denote two data matrices, where similar to X , X (cid:48) , each row contains a state-action tuple( s, a ) ∈ S ×A .
The NTK, which governs the training dynamics of the inﬁnitely-wide neural network,is then given by the outer product of gradients of the neural network at initialization : ˆΘ (0) i ( A, B ) :=∇ θ Q θ i ( A ) · ∇ θ Q θ i ( B ) T | t =0 , where we overload notation Q
θ i
( A ) to represent the column vectorcontaining Q -values.
At inﬁnite-width in the NTK regime, ˆΘ (0) i ( A, B ) converges to a deterministickernel (i.e. does not depend on the random weight sample θ i ), and hence is the same for all ensemblemembers.
Thus, hereafter we will remove the index i from the notation of the NTK kernel andsimply write, ˆΘ (0)
( A, B ) .
With our notation in place, we deﬁne, C := ˆΘ (0) ( X (cid:48) , X ) · ˆΘ (0) ( X , X ) − 1 .
Intuitively, C is a | D | × | D | matrix where the element at column q , row p , captures a notion ofsimilarity between ( s, a ) in the q th row of X , and ( s (cid:48) , π ( s (cid:48) )) in the p th row of X (cid:48) .
We now have all the necessary machinery to characterize the form of Q LCB :
Theorem 3.1.
For a given ( s, a ) ∈ S×A , let Q (0)
θ i ( s, a ) denote Q
θ i ( s, a ) | t =
(value at initialization),with θ sampled from the initial weight distribution.
After t + 1 iterations of pessimistic policyevaluation, the LCB value estimate for ( s (cid:48) , π ( s (cid:48) ))
∈ X (cid:48) is given by,
Independent Targets (Method 1): (1)
[Equation]
Shared Targets (Method 2):
[Equation] [Equation]
where the square and square-root operations are applied element-wise.
1 Please refer to Appendix Ffor the proof.
As can be seen, the equations for the pessimistic LCB value estimates in both settings are similar,only differing in the third term.
The ﬁrst term is negligible and tends towards zero as the number ofiterations of policy evaluation increases.
The second term shared by both variants corresponds tothe expected result of the policy evaluation procedure without any pessimism (as before, we meanexpectation under θ sampled from the initial weight distribution).
Accordingly, the differing thirdterm in each variant exactly corresponds to the “pessimism” or “penalty” induced by that variant.
Considering the available ofﬂine RL dataset D as a restricted MDP in itself, we see that the useof Independent Targets (Method 1) leads to a pessimism term that performs “backups" along the trajectories that the policy would experience in this restricted MDP (using the geometric term1 + · · · + γ t C t ) before computing a variance estimate.
Meanwhile the use of Shared Targets (Method2) does the reverse – it ﬁrst computes a variance term and then performs the “backups".
While this difference may seem inconsequential, it becomes critical when one realizes that in Equation2 for Shared Targets (Method 2), the pessimism term (third term) may become positive, i.e. a negativepenalty , yielding an effectively optimistic LCB estimate.
Critically, with Independent Targets (Method1), this problem cannot occur .
3.2 Validating Theoretical Predictions
In this section we demonstrate that our analysisis not solely a theoretical result concerning theidiosyncracies of inﬁnite-width neural networks,but that it is rather straightforward to constructcombinations of an MDP, ofﬂine data, and a policy, that lead to the critical ﬂaw of an optimistic
LCB estimate.
Let d s , d a denote the dimensionality of stateand action vectors respectively.
We consider an
MDP whose initial state distribution is a spherical multivariate normal distribution N (0 , I) ,and whose transition function is given by
P ( s (cid:48) | s, a ) =
N (0 , I) .
Consider the procedurefor generating our ofﬂine data matrices, described in the box to the right.
This procedurereturns data matrices X , R, X (cid:48) by generating N episodes of length T , using a behavior policya ∼ N (0 , I) .
In this generation process, we set the policy we seek to pessimistically evaluate, π , toalways apply the behavior policy’s action in state s to the next state s (cid:48) .
To construct our examples, we consider the setting where we use linear models to represent Q θ i , withthe initial weight distribution being a spherical multivariate normal distribution, N (0 , I) .
With linearmodels, the equations for Q LCB takes an identical form to those in Theorem 3.1.
Given the described data generating process and our choice of linear function approximation, wecan compute the pessimism term for the Shared Targets (Method 2) (i.e. the third term in Theorem3.1, Equation 2).
We implement this computation in a simple Python script, which we include inthe supplementary material.
We choose, d s = 30 , d a = 30 , γ = 0 .
5 , N = 5 , T = 5 , and t = 1000( t is the exponent in the geometric term above).
We run this simulation 1000 times, each with adifferent random seed.
After ﬁltering simulation runs to ensure γ (cid:107) C (cid:107) <
1 (as discussed in an earlierfootnote), we observe that 221 of the simulation runs result in an optimistic LCB bonus, meaning thatin those experiments, the pessimism term was in fact positive for some ( s (cid:48) , π ( s (cid:48) ))
∈ X (cid:48) .
We havemade the python notebook implementing this experiment available in our supplementary material.
For further intriguing investigations in pedagogical toy MDPs regarding the structure of uncertainties,we strongly encourage the interested reader to refer to Appendix G.
Model Standard-deviation Gradients (MSG)
It is important to note that even if the pessimism term does not become positive for a particularcombination of MDPs, ofﬂine datasets, and policies, the fact that it can occur highlights that theformulation of Shared Targets is fundamentally ill-formed.
To resolve this problem we propose
Model Standard-deviation Gradients (MSG), an ofﬂine RL algorithm which leverages ensembles toapproximate the LCB using the approach of Independent Targets.
4.1 Policy Evaluation and Optimization in MSG
MSG follows an actor-critic setup.
At the beginning of training, we create an ensemble of N Qfunctions by taking N samples from the initial weight distribution.
During training, in each iteration,we ﬁrst perform policy evaluation by estimating the Q LCB for the current policy, and subsequentlyoptimize the policy through gradient ascent on Q LCB .
Policy Evaluation As motivated by our analysis in Section 3, we train the ensemble Q -functionsindependently using the standard least-squares Q-evaluation loss,
[Equation]
where θ i , ¯ θ i denote the parameters and target network parameters for the i th Q-function.
In each iteration, as is common practice, we do not update the Q -functions until convergence, andinstead update the networks using a single gradient step.
In practice, the expectation in L ( θ i ) isestimated by a minibatch, and the expectation in y i is estimated with a single action sample from thepolicy.
After every update to the Q-function parameters, their corresponding target parameters areupdated to be an exponential moving average of the parameters in the standard fashion.
Policy Optimization As in standard deep actor-critic algorithms, policy evaluation steps (learning
Q ) are interleaved with policy optimization steps (learning π ).
In MSG, we optimize the policythrough gradient ascent on Q LCB .
Speciﬁcally, our proposed policy optimization objective in MSGis,
[Equation]
where β ≤ 0 is a hyperparameter that determines the amount of pessimism.
4.2 The Trade-Off Between Trust and Pessimism
While our hope is to leverage the implicit generalization capabilities of neural networks to estimateproper LCBs beyond states and actions in the ﬁnite dataset D , neural network architectures canbe fundamentally biased, or we can simply be in a setting with insufﬁcient data coverage, suchthat the generalization capability of those networks is limited.
To this end, we augment the policyevaluation objective of MSG ( L ( θ i ) , equation 3) with a support constraint regularizer inspired by
CQL [11] 2 : H ( θ i ) =
E s ∼ D,a ∼ π ( s )
[ Q θ i ( s, a )] − E ( s,a ) ∼ D
[ Q θ i ( s, a )] .
This regularizer encouragesthe Q -functions to increase the values for actions seen in the dataset D , while decreasing the valuesof the actions of the current policy.
Practically, we estimate the latter expectation of H using thestates in the mini-batch, and we approximate the former expectation using a single sample from thepolicy.
We control the contribution of H ( θ i ) by weighting this term with weight parameter α .
Thefull critic loss is thus given by,
[Equation]
Empirically, as evidenced by our results in Appendix A.2, we have observed that such a regularizercan be necessary in two situations: 1)
The ﬁrst scenario is where the ofﬂine dataset only contains anarrow data distribution (e.g., imitation learning datasets only containing expert data).
We believethis is because the power of ensembles comes from predicting a value distribution for unseen ( s, a )based on the available training data.
Thus, if no data for sub-optimal actions is present, ensemblescannot make accurate predictions and increased pessimism via H becomes necessary.
The secondscenario is where environment dynamics can be chaotic (e.g. Gym [46] hopper and walker2d ).
In such domains it would be beneﬁcial to remain close to the observed data in the ofﬂine dataset.
Pseudo-code for our proposed MSG algorithm can be viewed in Algorithm Box 1.
Experiments
In this section we seek to empirically answer the following questions: 1) How well does MSGperform compared to current state-of-the-art in ofﬂine RL? 2)
Are the theoretical differences inensembling approaches (Section 3) practically relevant? 3)
When and how does ensemble size affectperfomance? 4)
Can we match the performance of MSG through efﬁcient ensemble approximationsdeveloped in the supervised learning literature?
[Figure] [Figure]
CQL
IQL
MSG ( N = 64 ) β α MSG ( N = [Table] [Table]
maze2d-
[Table] [Table] [Table] [Table]
antmaze-umaze-v
[Table]
[Table]
antmaze-umaze-diverse-v
[Table]
antmaze-medium-play-v
[Table]
[Table]
[Table]
5.1 Ofﬂine RL Benchmarks
D4RL Gym Domains We begin by evaluating MSG on the Gym domains ( halfcheetah ,hopper , walker2d ) of the D4RL ofﬂine RL benchmark [24], using the medium , medium-replay ,medium-expert [Table] and expert data settings.
Our results presented in Appendix A.2 (summarized in
Figure 4) demonstrates that MSG is competitive with well-tuned state-of-the-art methods CQL [11]and F-BRC [12].
D4RL Antmaze Domains Due to the narrow range of behaviors in Gym environments, ofﬂinedatasets for these domains tend to be very similar to imitation learning datasets.
As a result, manyprior ofﬂine RL approaches that perform well on D4RL Gym fail on harder tasks that require stitchingtrajectories through dynamic programming (c.f.
An example of such tasks are the D4RLantmaze settings, in particular those in the antmaze-medium and antmaze-large environments.
The data for antmaze tasks consists of many episodes of an Ant agent [46] running along arbitrarypaths in a maze.
The agent is tasked with using this data to learn a point-to-point navigation policyfrom one corner of the maze to the opposite corner, where rewards are given by a sparse signal thatis 1 when near the desired end location in the maze – at which point the episode is terminated –and 0 otherwise.
The undirected, extremely sparse reward nature of antmaze tasks make them verychallenging, especially for the large maze sizes.
Table 1 and Appendix B.2 present our results.
To the best of our knowledge, the antmaze domainsare considered unsolved, with few prior works reporting non-zero results on the large mazes [11, 47].
As can be seen, MSG obtains results that far exceed the prior state-of-the-art results reported by [47].
While some works that use specialized hierarchical approaches have reported strong results aswell [48], it is notable that MSG is able to solve these challenging tasks with standard architecturesand training procedures, and this shows the power that ensembling can provide – as long as theensembling is performed properly!
RL Unplugged
In addition to the D4RL benchmark, we evaluate MSG on the RL Unpluggedbenchmark [25].
Our results are presented in Figure 1.
We compare to results for Behavioral Cloning
[Figure]
(BC) and two state-of-the-art methods in these domains, Critic-Regularized Regression (CRR)
[7]and MuZero Unplugged [49].
Due to computational constraints when using deep ensembles, weuse the same network architectures as we used for D4RL experiments.
The networks we use areapproximately 160 -th the size of those used by the BC, CRR, and MuZero Unplugged baselinesin terms of number of parameters.
We observe that MSG is on par with or exceeds the currentstate-of-the-art on all tasks with the exception of humanoid.run , which appears to require the largerarchitectures used by the baseline methods.
Experimental details can be found in Appendix C.
Benchmark Conclusion Prior work has demonstrated that many ofﬂine RL approaches that perform well on Gym domains, fail to succeed on much more challenging domains [47].
Our resultsdemonstrate that through uncertainty estimation with deep ensembles, MSG is able to very signiﬁcantly outperform prior work on very challenging benchmark domains such as the D4RL antmazes.
5.2 Ensemble Ablations
[Figure]
Independence in Ensembles Ablation
In
Section 3, through theoretical arguments and toyexperiments we demonstrated the importance oftraining using “Independent" ensembles.
Here,we seek to validate the signiﬁcance of our theoretical ﬁndings using ofﬂine RL benchmarks, bycomparing Independent targets (as in MSG),to Shared-LCB and Shared-Min targets.
Ourresults are presented in Appendices A.3 and B.3,with a summary in Figures 2 and 4.
In the Gym domains (Appendix A.3), with ensemble size N = 4 , Shared-LCB signiﬁcantlyunderperforms MSG.
In fact, not using ensembles at all ( N = 1 ) outperforms Shared-LCB .
With ensemble size N = 4 , Shared-Min ison par with MSG.
When the ensemble size isincreased to N = 64 (Figure 7), we observethe performance of Shared-Min drops significantly on 7 / 12 D4RL Gym settings.
In constrast, the performance of MSG is stable anddoes not change.
In the challenging antmaze domains (Appendix B.3), for both ensemble sizes N = 4 and N = 64 ,
Shared-LCB and Shared-Min targets completely fail to solve the tasks, while for both ensemblesizes MSG exceeds the prior state-of-the-art (Table 1), IQL [47].
Independence in Ensembles Conclusion Our experiments corroborate the theoretical results in
Section 3, demonstrating that Independent targets are critical to the success of MSG.
These resultsare particularly striking when one considers that the implementations for MSG, Shared-LCB , and
Shared-Min differ by only 2 lines of code.
Ensemble Size Ablation An important ablation is to understand the role of ensemble size in MSG.
In the Gym domains, Figure 5 demonstrates that increasing the number of ensembles from 4 to 64does not result in a noticeable change in performance.
In the antmaze domains, we evaluate MSG under ensemble sizes { 1 , 4 , 16 , 64 } .
Figure 3 presentsour results.
Our key takeaways are as follows: • For the harder antmaze-large tasks, there is a clear upward trend as ensemble size increases.
• Using a small ensemble size (e.g. N = 4 ) isalready quite good, but more sensitive to hyperparameter choice especially on the harder tasks.
Very small ensemble sizes beneﬁt more from using α > 0 3 .
However, across the board, using α = 0is preferable to using too large of a value for α – with the exception of N = 1 which cannot takeadvantange of the beneﬁts of ensembling.
• When using lower values of β , lower values of α shouldbe used.
Ensemble Size Conclusion In domains such as D4RL Gym where ofﬂine datasets are qualitativelysimilar to imitation learning datasets, larger ensembles do not result in noticeable gains.
In domainssuch as D4RL antmaze which contain more data diversity, larger ensembles signiﬁcantly improve theperformance of agents.
5.3 Efﬁcient Ensembles
Thus far we have demonstrated the signiﬁcant performance gains attainable through MSG.
Animportant concern however, is that of parameter and computational efﬁciency: Deep ensembles of
Q -networks result in an N -fold increase in memory and compute usage, both in the policy evaluationand policy optimization phases of actor-critic training.
While this might not be a signiﬁcant problemin ofﬂine RL benchmark domains due to small model footprints 4 , it becomes a major bottleneckwith larger architectures such as those used in language and vision domains.
To this end, weevaluate whether recent advances in “Efﬁcient Ensemble" approaches from the supervised learningliterature transfer well to the problem of ofﬂine RL.
Speciﬁcally, the efﬁcient ensemble approaches weconsider are: Multi-Head Ensembles [26, 50, 51], MIMO Ensembles [27], and Batch Ensembles [28].
For a description of these efﬁcient ensembling approaches please refer to Appendix E. A runtimecomparison of different ensembling approaches can be viewed in Table 2.
D4RL Gym Domains Appendix A.4 presents our results in the D4RL Gym domains with ensemblesize N = 4 (summary in Figure 4).
Amongst the considered efﬁcient ensemble approaches, Batch
Ensembles [28] result in the best performance, which follows ﬁndings from the supervised learningliterature [17].
D4RL Antmaze Domains Appendix B.4 presents our results in the D4RL antmaze domains forboth ensemble sizes of N = 4 and N = 64 (summary in Figure 2).
As can be seen, compared to
MSG with deep ensembles (separate networks), the efﬁcient ensemble approaches we consider arevery unreliable, and fail for most hyperparameter choices.
Efﬁcient Ensembles Conclusion We believe the observations in this section very clearly motivatefuture work in developing efﬁcient uncertainty estimation approaches that are better suited to thedomain of reinforcement learning.
To facilitate this direction of research, in our codebase we haveincluded a complete boilerplate example of an ofﬂine RL agent, amenable to drop-in implementationof novel uncertainty-estimation techniques.
Discussion & Future Work
Our work has highlighted the signiﬁcant power of ensembling as a mechanism for uncertaintyestimation for ofﬂine RL.
In this work we took a renewed look into Q -ensembles, and studied how toleverage them as the primary source of pessimism for ofﬂine RL.
Through theoretical analyses andtoy constructions, we demonstrated a critical ﬂaw in the popular approach of using shared targetsfor obtaining pessimistic Q -values, and demonstrated that it can in fact lead to optimistic estimates.
Using a simple ﬁx, we developed a practical deep ofﬂine RL algorithm, MSG, which resulted in largeperformance gains on established ofﬂine RL benchmarks.
As demonstrated by our experimental results, an important outstanding direction is to study howwe can design improved efﬁcient ensemble approximations, as we have demonstrated that currentapproaches used in supervised learning are not nearly as effective as MSG with ensembles thatuse separate networks.
We hope that this work engenders new efforts from the community ofneural network uncertainty estimation researchers towards developing efﬁcient uncertainty estimationtechniques directed at reinforcement learning
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] The abstract and introduction summarize theorganization and contents of the paper.(b)
Did you describe the limitations of your work?
[Yes] The most important limitationof using ensembles is their N -fold increase in computational cost.
Section 5.3 framesthis problem and investigates a variety of approaches used in the supervised learningliterature.
We demonstrate that while “Efﬁcient Ensembles" perform well on thesimpler D4RL Gym domains, they fail on the D4RL antmaze domains.
We believe thissection very clearly motivates future work in developing efﬁcient uncertainty estimationapproaches that are better suited to the domain of reinforcement learning.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
Our workis focused on algorithmic advances in the ﬁeld of ofﬂine reinforcement learning (RL).
The potential positive and negative impacts of ofﬂine RL methods are directly functionof the domain to which they are applied.
In this work, we only apply our methods toopen
-sourced, simulated benchmark domains which do not have societal implications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Section 3.
sets up the stage for the introduction of Theorem 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] Please refer toAppendix F.
If
you ran experiments
(a) Did you include the code, data, and instructions needed to reproduce the
main
experi-
mental results (either in the supplemental material or as a
URL
[Yes] Please refer to
the supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
The
ofﬂine datasets
are provided by the benchmarks we used.
We
have detailed all experimental details, with pointers to details in the respective sections.
c)
Did you report error bars (e.g.,
with respect
to the random seed after running experiments multiple times)?
[Yes] In Table 1 we report mean and standard deviation acrossrandom seeds.
For the summarized bar plots in Figures 2 and 4 we only plot the mean.
This is because for each bar in this summary plot, we report the results for the besthyperparameter value (which for many settings, due to signiﬁcant computational cost,had 2 random seeds).
However, in this work we have shown in exact detail the densehyperparameter searches we have done, and furthermore, in we have reported the resultfor every single experiment individually (i.e. every combination of data setting, method,hyperparameter, and random seed) (Figure 3, Appendix A, Appendix B).
Reportingresults for each individual experiment demonstrate the trends, even in the settingswhere each hyperparameter was ran with 2 seed.
We have also included all of our
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No] We mentioned that all experiments
were done on Nvidia P100 GPU devices.
We did not include the total amount of
compute we used.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the D4RL [24]
and RL Unplugged [25] benchmarks.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include out codebase, python notebooks, as well as .pkl ﬁles of experimental
results.
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[No] D4RL [24] and RL Unplugged [25] are established and
open-sourced ofﬂine RL datasets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
The D4RL [24] and RL Unplugged [25] do
not contain such sensitive materials.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A
[Figure]
In this section we discuss results using the D4RL gym ( halfcheetah , hopper , walker2d )benchmark domains.
The dataset types that we consider are medium-v2 , medium-replay-v2 ,medium-expert-v2 , and expert-v2 .
A.1 Experimental Details
All policies and Q-functions are a 3 layer neural network with relu activations and hidden layer size256.
The policy output is a normal distribution that is squashed to [ − 1 , 1] using the tanh function.
All methods were trained for 3M steps.
CQL and MSG are trained with behavioral cloning (BC) forthe ﬁrst 50K steps.
F-BRC pretrains a behavioral cloning model for 1M steps.
MSG, CQL, and F-BRC, are tuned with an equal hyperparameter search budget of 12 hyperparameterchoices.
Each hyperparameter choice for each method is trained using 2 random seeds.
Each run isevaluated for 100 episodes.
We report results for all hyperparameters and all seeds.
For fairness ofcomparison, F-BRC is ran without adding a survival reward bonus.
MSG and CQL are implementedin our code, and for F-BRC we use the opensourced codebase.
Our reported CQL results appear to bebetter than or on par with values reported in prior works, which provided us with conﬁdence to useour own implementation.
We used the following values for hyperparameter search:
MSG β ∈ {− 4 .
− 8 . } , α ∈ { 0 ., 0 . 5 , 1 ., 2 ., 4 ., 8 . }CQL α ∈ { 0 ., 0 . 45 , 0 .
32 , 3 . 18 , 3 . 63 , 4 . 09 , 4 . 55 , 5 . }F-BRC λ ∈ { 0 ., 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 .
A.2 Baseline Comparison
We compare MSG with deep ensembles to CQL [11] and F-BRC[12], two methods that obtainstate-of-the-art results on the gym domains of the D4RL benchmark.
Figure 5 presents our results onthe gym domains.
The key takeaways of our results are as follows:
• On gym domains, MSG is generally competitive with current state-of-the-art
results.•
In the hopper and walker2d domains – where dynamics can be chaotic and staying closeto the data-support is beneﬁcial – CQL and F-BRC which rely on support constraints aremore reliable w.r.t.
the hyperparameter choice.
A.3 Ablations
We perform ablations w.r.t.
the key components of MSG.
Our key takeaways from the results in
Figures 6 and 7 for gym domains are as follows:
• Comparing MSG – which uses independent targets – to Shared-LCB and Shared-Min –which use shared targets – clearly demonstrates the signiﬁcance of our theoretical analysis in Section 3.
Despite differing in only 2 lines of code from MSG, Shared-LCB and Shared- Min very signiﬁcantly underpeform MSG.
In fact, using no ensembling at all ( N = 1) outperforms Shared-LCB and is on par with Shared-Min.
• In the gym domains of D4RL – which are close to being imitation learning datasets – usinglarge ensemble sizes does not result in noticeable gains in MSG.
A.4 Efﬁcient Ensembles
As discussed in Section 5.
[Figure]
we evaluate whether the performance of MSG with deep ensemblescan be matched using state-of-the-art “efﬁcient ensembles" from supervised learning literature.
Ourtakeaways from the results in Figure 8 for gym domains are as follows:
• From the efﬁcient ensembling approaches considered, Batch Ensembles tend to be the most
[Figure]
performant [28].
Interestingly, this follows the ﬁndings of [17] from the supervised learning
[Figure]
literature.
[Figure]
B D4RL Antmaze Benchmarks
[Figure]
In this section we discuss results using the D4RL antmaze benchmark domains.
We mainly focus ourexperiments on the antmaze-medium and antmaze-large environments, with the two availableofﬂine datasets of play-v0 and diverse-v0 .
B.1 Experimental Details
The rewards in the antmaze domains (which are either 0 .
or 1 . ) were transformedusing the following equation: 4( r − 0 . 5) .
This transformation has become common practice in prior works https://github.com/aviralkumar2907/CQL/blob/d67dbe9cf5d2b96e3b462b6146f249b3d6569796/d4rl/examples/cql_antmaze_new.py#L22 .
All policies and Q-functions are a 3 layer neural network with relu activations and hidden layer size256.
The policy output is a normal distribution that is squashed to [ − 1 , 1] using the tanh function.
All methods were trained for 2M steps.
CQL and MSG are trained with behavioral cloning (BC) forthe ﬁrst 50K steps.
F-BRC pretrains a behavioral cloning model for 1M steps.
MSG, CQL, and F-BRC, are tuned with an equal hyperparameter search budget of 8 hyperparameterchoices.
Each run is evaluated for 100 episodes.
We report results for all hyperparameters and allseeds.
For all MSG with deep ensembles experiments (that appear in Figure 3 also), and for the No
Ensembles ( N = 1 ) baseline we ran 5 random seeds per hyperparameter choice.
For all otherbaselines and ablations that appear in this section we ran 2 random seeds.
For fairness of comparison, F-BRC is ran without adding a survival reward bonus.
MSG and CQLare implemented in our code, and for F-BRC we use the opensourced codebase.
Our reported CQLresults appear to be better than or on par with values reported in prior works, which provided us withconﬁdence to use our own implementation.
We used the following values for hyperparameter search:
MSG β ∈ {− 4 .
− 8 . } , α ∈ { 0 ., 0 . 1 , 0 . 5 , 1 . }
CQL α ∈ { 0 ., 0 . 45 , 0 .
F-BRC λ ∈ { 0 ., 0 .
B.2 Baseline Comparison
We compare MSG with deep ensembles to CQL [11] and F-BRC[12].
Figure 11 presents our resultson the antmaze domains.
The key takeaways of our results are as follows:
• On the D4RL antmaze domains, MSG with deep ensembles far exceeds the results of prior
state of the art result Implicit Q-Learning (IQL) [47].
• CQL [11] and F-BRC [12] which obtain very strong results on the D4RL Gym domains
(Appendix A.2), completely fail on the D4RL antmaze domains
This mimics the results of
[47] that demonstrated many prior ofﬂine RL methods that perform strongly on D4RL
Gym
domains, fail on D4RL antmaze domains.
We believe this highlights the importance of using
more complex ofﬂine RL benchmarks that emphasize the stiching of diverse trajectories
through dynamic programming, as opposed to ofﬂine RL datasets for Gym domains that are
qualitatively very similar to imitation learning datasets.
[Table]
[Figure]
B.3 Ablations
We perform ablations w.r.t.
the key components of MSG.
Our key takeaways from the results in
Figures 12 and 13 for antmaze domains are as follows:
• Comparing MSG – which uses independent targets – to Shared-LCB and Shared-Min –
which use shared targets – clearly demonstrates the signiﬁcance of our theoretical analysis
in Section 3.
Despite differing in only 2 lines of code from MSG, Shared-LCB and Shared-
[Table]
Min very signiﬁcantly underpeform MSG.
In fact, using no ensembling at all ( N = 1)
[Table]
outperforms Shared-LCB and is on par with Shared-Min.
[Table]
• In the gym domains of D4RL – which are close to being imitation learning datasets – using
[Table]
large ensemble sizes does not result in noticeable gains in MSG.
[Table] [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure]
B.4 Efﬁcient Ensembles
As discussed in Section 5.3, we evaluate whether the performance of MSG with deep ensemblescan be matched using state-of-the-art “efﬁcient ensembles" from supervised learning literature.
Ourtakeaways from the results in Figures 14 and 15 for antmaze domains are as follows:
• Despite efﬁcient ensembles such as Batch Ensembles performing well on the D4RL
Gym
domains (Appendix A.4), they fail on the antmaze domains for both ensemble sizes of
N = 4 and N = 64 .
[Table]
[Figure]
[Figure]
[Table]
[Table]
[Table]
[Figure]
C RL Unplugged
C.1 DM Control Suite Tasks
The networks used in [25] for DM Control Suite Tasks are very large relative to the networks we usedin the D4RL benchmark; roughly the networks contain 60x more parameters.
Using a large ensemblesize with such architectures requires training using a large number of devices.
Furthermore, sincein our experiments with efﬁcient ensemble approximations we did not ﬁnd a suitable alternative todeep ensembles (section 5.3), we decided to use the same network architectures and N = 64 as inthe D4RL setting (enabling single-GPU training as before).
Our hyperparameter search procedure was similar to before, where we ﬁrst performed a coarse searchusing 2 random seeds and hyperparameters β ∈ {− 1 ., − 2 ., − 4 .
− 8 . } , α ∈ { 0 ., 0 . 5 } , and for the bestfound hyperparameter, ran ﬁnal experiments with 5 new random seeds.
Input: ofﬂine RL dataset D , Q -function and policy architectures, ensemble size N , MSG hyperparameters β, α
Create N Q -networks Q θ i , with parameters sampled from the initial weight distribution
Create target networks Q ¯ θ i , initialized as ¯ θ i ← θ i
Create the policy network π with parameters θ π
Function PolicyEvaluationStep( batch ) : for i = 1 , ..., N do /* Compute policy evaluation loss */
[Equation]
batch,
[Equation]
m )
[Equation] [Equation]
(cid:48)
[Equation]
/* Compute support constraint regularizer loss */ ∀
[Equation]
batch,
[Equation] [Equation]
m )
H
[Equation]
, a m ) /* Optimize the policy evaluation objective */
L ( θ i ) =
L ( θ i ) + α · H ( θ i ) θ i ← θ i − AdamUpdate (cid:16) L ( θ i ) , θ i (cid:17) ¯ θ i ← τ · ¯ θ i + (1 − τ ) · θ i
Function MSGPolicyOptimizationStep( batch ) : /* Compute LCB Q -values */ ∀ s m ∈ batch, a π,m ∼ π ( s m ) ∀ s m ∈ batch, Q LCB ( s m , a π,m )
= mean i
[ Q θ i ( s m , a π,m )] − β · std i
[ Q θ i ( s m , a π,m )] /
* Optimize the policy objective */
L ( θ π ) = 1 M (cid:80) m Q LCB ( s m , a π,m ) θ π ← θ π − AdamUpdate (cid:16) L ( θ π ) , θ π (cid:17)
Function Main() : /
* Optional initialization of the policy using behavioral cloning while training Q -functions */ for desired number of steps do batch ∼ D
PolicyEvaluationStep (batch)
Update π using behavioral cloning for desired number of steps do batch ∼ D
PolicyEvaluationStep (batch)
MSGPolicyOptimizationStep (batch)
E Efﬁcient Ensemble Approaches
Multi-Head [26, 50, 51] Multi-Head refers to ensembles that share a “trunk" network and haveseparate “head" networks for each ensemble member.
In this work, we modify the last layer of a
Q -network to output N predictions instead of a single Q -value, making the computational cost ofthis ensemble on par with a single network.
Multi-Input Multi-Output (MIMO) [27]
MIMO is an ensembling approach that approximatelyhas the same parameter and computational footprint as a single network.
The MIMO approachonly modiﬁes the input and output layers of a given network.
In MIMO, to compute predictions fora data-point x under an ensemble of size N , x is copied N times into x 1 , ..., x N , which are then concatenated and passed to the network.
The output of the network is a vector of size N , which issplit into N predictions y 1 , ..., y N representing the predictions of the ensemble members.
For addedclariﬁcation, we include Figure 16 depicting how a MIMO ensemble network functions.
For furtherdetails on how MIMO networks are trained, we refer the interested reader to the original work of[27].
Batch Ensembles [28] Batch Ensembles incorporate rank-1 modulations to the weights of fullyconnected layers.
More speciﬁcally, let W be the weight matrix of a given fully-connected layer,and let x be the input to the layer.
The output of the layer for ensemble member i is computed asσ ((( W T ( x ◦ r i ))
◦
s i )
+ b i ) , where ◦ is the element-wise product, parameters with superscript
i areseparate for each ensemble member, and σ is the activation function.
While Batch Ensembles isefﬁcient in terms of number of parameters, in our actor-critic setup its computational cost is similar todeep ensembles, since for policy updates we need to evaluate every ensemble member using separateforward passes.
F Theorem Proof
In this section, following our notation and iniﬁte-width networks setup described in Section 3.1, wepresent the proof for the following Theorem.
Theorem F.1.
For a given ( s, a ) ∈ S×A , let Q (0)
θ i ( s, a ) denote Q
θ i ( s, a ) | t =
(value at initialization),with θ sampled from the initial weight distribution.
After t + 1 iterations of pessimistic policyevaluation, the LCB value estimate for ( s (cid:48) , π ( s (cid:48) ))
∈ X (cid:48) is given by,
Independent Targets (Method 1):
[Equation]
Shared Targets (Method 2):
[Equation]
where the square and square-root operations are applied element-wise.
Proof.
First we will establish some additional notation.
Consider an inﬁnite-width NTK-parameterized [45] Q -function and let Q ( t ) ( s, a ) denote its valueprediction for state-action pair ( s, a ) after t iterations of pessimistic policy evaluation.
We will use thenotation θ ( t ) to refer to the parameters of this network after t iterations, but for simplicity of notation,when the context is clear we will omit the use of θ ( t ) .
Additionally, consider the linearization (Taylorexpansion) of Q about its initialization:
[Equation]
When performing pessimistic policy evaluation using linearized networks, we will use Q ( t ) lin ( s, a ) todenote the value prediction of Q lin after t iterations, and will use θ ( t ) lin to refer to its learned weights.
Note that θ (0)lin = 0 , and that ∀ ( s, a ) , Q (0) ( s, a ) =
Q (0)lin ( s, a ) .
In [52] (section 2.4) it is shown that when training an inﬁnitely wide neural network to performregression using mean squared error, subject to technical conditions on the learning rate used,the predictions of the trained network are equivalent to if we had trained the linearized networkinstead.
This means that after t iterations of our policy evaluation procedure, ∀ ( s, a ) , t, Q ( t ) lin ( s, a ) =
Q ( t ) ( s, a ) .
Hereafter we study the evolution of ensembles of linearized inﬁnite-width networks, Q lin ,across training iterations.
As a reminder, X , R, X (cid:48) denote data matrices containing ( s, a ) , r , and ( s (cid:48) , π ( s (cid:48) )) appearing in theofﬂine dataset D ; i.e., the k -th transition ( s, a, r, s (cid:48) ) in D is represented by the k -th rows in X , R, X (cid:48) .
Additionally, we deﬁned C := ˆΘ (0) ( X (cid:48) , X ) · ˆΘ (0) ( X , X ) − 1 .
Below, we will use notation suchas Q ( X ) to mean applying Q to each row of X and stacking the predictions into a vector.
ByˆΘ (0) (( s, a ) , X ) we mean to treat ( s, a ) as a row matrix and compute the 1 × | D | kernel matrix.
Derivation for Independent Targets When using Independent Targets, in iteration t + 1 eachnetwork uses its own temporal difference (TD) targets,
[Equation]
Using the equations in [52] (section 2.2, equations 9-10-11) we can derive,
[Equation]
[Equation]
Plugging in the expression for the targets Y ( t ) and recursively expanding the expressions above weobtain,
[Equation]
Per our earlier discussion above, from Equation 18 we can conclude that,
[Equation]
lin
[Equation]
We can now derive Q LCB for the Independent Targets setting, by computing the expectation andvariance of Q ( t +1)
( X (cid:48) ) with respect to the initial weight distribution.
Following [52] we obtain,
[Equation]
Var
[Equation]
We thus have,
[Equation]
where the square and square-root operators are applied element-wise.
Note that if γ (cid:107) C (cid:107) ≥ 1 , policyevaluation is liable to diverge.
Thus, in our discussions we avoid this degenerate case and assumeγ (cid:107)
Derivation for Shared LCB Targets When using Shared LCB Targets, in iteration t + 1 eachnetwork uses pessimistic temporal difference (TD) targets that are shared amongst ensemble members,
[Equation]
Using the equations in [52] (section 2.2, equations 9-10-11) we can derive,
[Equation] [Equation]
Noting that in the Shared-LCB setting Y ( t ) is not a random variable, we can now compute theexpectation and variance of Q ( t +1)
lin
( X (cid:48) ) with respect to the initial weight distribution,
[Equation]
[Equation]
Let A := (cid:113) Var[ Q ( t +1)
lin
( X (cid:48) )] .
Given the above equations
[Equation]
we can recursively compute the closedform for Y ( t ) as follows,
[Equation]
Based on this form
[Equation]
we can write,
[Equation]
[Equation]
[Equation]
Combining our derivations, and noting from our earlier discussion that ∀ ( s, a ) , t, Q ( t ) lin ( s, a ) =
Q ( t ) ( s, a ) , we have,
[Equation]
Thus, we have our results,
[Equation]
[Figure]
where the square and square-root operators are applied element-wise.
Note that if γ (cid:107) C (cid:107) ≥ 1 , policy
[Figure]
evaluation is liable to diverge.
Thus, in our discussions we avoid this degenerate case and assumeγ (cid:107) C (cid:107) < 1 , which makes the term O ( γ t (cid:107) C (cid:107) t ) negligible as t → ∞ .
[Figure]
The derivations of Equations 23 and ??
conclude our proof.
[Figure]
G.1 Qualitative Evaluation of Obtained Uncertainties
[Figure]
In this section we continue to study the implications of our theoretical results in Section 3 byconstructing a pedagogical toy MDP.
Our simple toy MDP allows us to follow the idealized setting ofthe presented theorem more closely, and allows for visualization of uncertainties obtained throughdifferent ensembling approaches.
Continuous Chain MDP
The MDP we consider has state-space S =
[ − 1 , 1] , action space A ∈ R ,deterministic transition dynamics s (cid:48)
=
s + a clipped to remain inside S , and reward functionr ( s, a ) =
[ s (cid:48) ∈ [0 .
Data Collection & Evaluation Policy
The ofﬂine dataset we generate consists of 40 episodes,each of length 30.
At the beginning of each episode we initialize at a random state s ∈ S .
In eachstep we take a random action sampled from Unif ( − 0 .
3 , 0 . 3) , and record all transitions ( s, a, r, s (cid:48) ) .
For evaluating the uncertainties obtained from different approaches, we create regions of missingdata by removing all transitions where s or s (cid:48) fall in the range [ − 0 .
The policy used forpolicy evaluation with the different ensembling approaches is ∀ s, π ( s ) = 0 .
Optimal Desired Form of Uncertainty Note that the evaluation policy π ( s ) = 0 .
1 is alwaysmoving towards the positive direction, and there is lack of data for states in the interval [ − 0 .
Hence, what we would expect is that in the region [0 . 33 , 1] there should not be a signiﬁcant amountof uncertainty, while in the region [ − 1 , 0 .
33] there should be signiﬁcantly more uncertainty aboutthe Q-values of π , because the policy will be passing through [ − 0 .
33] where there is no data.
Furthermore, as we move towards the negative axis, we would expect that in the region [ − 0 .
33]the uncertainty would gradually increase, while in the region [ − 1 , − 0 .
33] the uncertainty would notcontinue to increase.
Results We visualize and compare the uncertainties obtained when the targets in the pessimisticpolicy evaluation procedure are computed as:
• Independent Targets (as used in MSG): y i =
r + γ · Q θ i ( s (cid:48) , π ( s (cid:48) ))
[Table]
• Independent Double-Q Targets: y i =
r + γ · min (cid:104) Q 1 θ i ( s (cid:48) , π ( s (cid:48) )) , Q 2 θ i ( s (cid:48) , π ( s (cid:48) ))
(cid:105)
[Table]
• Shared Mean Targets: y = r + γ · mean (cid:104) Q
θ i ( s (cid:48) , π ( s (cid:48) ))
(cid:105)
[Table]
• Shared LCB Targets: y = r
+ γ · (cid:34) mean (cid:104) Q
θ i ( s (cid:48) , π ( s (cid:48) ))
(cid:105) − 2 · std (cid:104) Q
θ i ( s (cid:48) , π ( s (cid:48) ))
(cid:105)(cid:35)• Shared Min Targets: y = r + γ · min (cid:104) Q θ i ( s (cid:48) , π ( s (cid:48) ))
(cid:105)
Note that Independent Double-Q is still an independent ensemble, where each ensemble memberhas an architecture containing a min-pooling on top of two subnetworks.
In Figure 17 we plot the mean and two standard deviations of the Q -values predicted by the ensemblefor the policy we evaluated, π ( s ) = 0 .
1 (additional experimental details presented in Appendix G.3).
The ﬁrst striking observation is that, Independent targets (as used in MSG) effectively match ourdesired form of uncertainty: states that under the evaluation policy π ( s ) = 0 .
1 would lead to regionswith little data have wider uncertainties than states that do not.
A second observation is that, inthis toy construction, Shared LCB and Shared Min provide a seemingly good approximation tothe lower-bound of Independent predictions.
However, our theoretical results show that sharedtargets have critical failure cases.
Furthermore, our empirical results on challenging benchmarkdomains (Section 5.1) demonstrate that Shared LCB and Shared Min targets completely fail totrain successful policies, despite their implementation differing from Independent targets in only 2lines of code.
G.2 NTK vs. Maximal Parameterization
The toy experiment presented in section G.1 uses a single-hidden layer ﬁnite-width neural networkarchitecture with tanh activations.
The networks use the “standard weight parameterization" (i.e.the weight parameterization used in practice) as opposed to the NTK parameterization [53], and weoptimize the networks using the Adam optimizer [54].
While this setup is close to the practical settingand demonstrates the relevance of independent ensembles for the practical setting, an importantquestion posed by our reviewers is how close these results are to the theoretical predictions presentedin 3.1.
To answer this question, we present the following set of results.
Using the identical MDP and ofﬂine data as before, we implement 1 hidden layer neural networkswith erf non-lineartiy.
The networks are implemented using the Neural Tangents library [53], anduse the NTK parameterization.
The networks in the ensemble are optimized using full-batch gradientdescent with learning rate 1 for 500 steps of Fitted Q -Evaluation (FQE) [55], where in each FQEstep the networks are updated for 1000 gradient steps.
We vary the width of the networks from 32to 32768 in increments of a factor of 4 , plotting the mean and standard deviation of the networkpredictions.
The ensemble size is set to N = 16 , except for width 32768 where N = 4 .
We compare the results for ﬁnite-width networks to computing results for the inﬁnite-width settingin closed form (using Theorem 3.1).
Using the Neural Tangents library [53] we obtained the NTKfor the architecture described in the previous paragraph (1 hidden layer with erf non-linearity).
Wefound that the matrix inversion required in our equations results in numerical errors.
Hence, we makethe modiﬁcation
[Equation]
Figure 18 presents our results.
As the width of the networks grow larger, the shape of the uncertaintiesbecomes more similar to our closed-form equations (i.e. the variances become very small).
While wedo not have a rigorous explanation for why ﬁnite-width networks exhibit intuitively more desirablebehaviors, we present below a strong hypothesis backed by empirical evidence.
We believe rigorouslyanswering this question is an incredibly interesting avenue for future work.
Hypothesis: Inﬁnite-width networks in the NTK parameterization do not learn data-dependentfeatures [56].
[56] present a different approach for parameterizing inﬁnite-width networks calledthe “Maximal Parameterization", which enables iniﬁnite-width networks to learn data-dependentfeatures.
We perform the same experiment as above, by replacing the NTK-parameterized networkswith Maximal Parameterizations.
Figure 18 presents our empirical results for network widths from
32 to 32768.
Excitingly, we observe that with Maximial Parameterization, even our widest networksrecover the intuitively desired form of uncertainty described in section G.1!
The solutions of thesenetworks also appear much more accurate, particularly on the right hand side of the plot; we canobserve the correct stepped structure of the Q − values up until the interval [0 .
75 , 1] where the policyalways receives a reward of 1 .
Each step appears to be approximately 0 .
1 in width, which is size ofthe action of the policy being evaluated, ∀ s, π ( s ) = 0 .
G.3 Additional Implementation Details for Figure
To evaluate the quality of uncertainties obtained from different Q -function ensembling approaches,we create N = 64 Q -function networks, each being a one hidden layer neural network with hiddendimension 512 and tanh activation.
The initial weight distribution is a fan-in truncated normaldistribution with scale 10.0, and the initial bias distribution is fan-in truncated normal distributionwith scale 0.05.
We did not ﬁnd results with other activation functions and choices of initial weightand bias distribution to be qualitatively different.
We use discount γ = 0 .
99 and the networks areoptimized using the Adam [54] optimizer with learning rate 1e-4 .
In each iteration, we ﬁrst computethe TD targets using the desired approach (e.g. independent vs. shared targets) and then ﬁt the
Q -functions to their respective targets with 2000 steps of full batch gradient descent.
We train thenetworks for 1000 such iterations (for a total of 2000 × 1000 gradient steps).
Note that we do not usetarget networks.
Given the small size of networks and data, these experiments can be done within afew minutes using a single GPU in a Google Colaboratory notebook which we have included inthe supplementary material.
H Practical Workﬂow for Applying MSG to New Ofﬂine RL Datasets
Throughout this work, we have reported the results for every method, domain, hyperparameter choice,and random seed.
Based on our experience, in this section we outline a practical workﬂow forapplying MSG to new ofﬂine RL environments and datasets.
In addition to the the key take-aways from our discussion of Figure 3 in Section 5.2, here weinclude practical advice on hyperparameter tuning when applying MSG to a new domain.
Asidefrom the choice of neural network architectures, the main hyperparameters in MSG are β , i.e.the hyperparameter controlling the amount of pessimism in Q LCB (Equation 4), and α , i.e. thehyperparameter controlling the contribution of the CQL-style regularizer (Equation 5).
Intuitively,as described in Section 4.2, larger values of α become necessary when the ofﬂine dataset has anarrow distribution (e.g. imitation learning datasets), or in MDPs with more chaotic dynamics,where small deviations can be very costly and thus we must stay close to the provided data support.
Equipped with this intuition, for a new ofﬂine RL task and dataset, we would ﬁrst use low β values(e.g.
β ∈ {− 4 , − 8 } or lower), and search for an appropriate range of α (which may be 0 , as in ourreported antmaze-large results in Table 1).
For further hyperparameter tuning from this starting point,we would investigate if reducing pessimism by increasing β – and potentially modifying α – wouldlead to improved policies.
Generally, we ﬁnd MSG to be fairly robust, as evidenced by our results in
Appendices A and B, and Figure 3.
I Statistical Model
[Figure]
D
[Figure]
a) Q-function generative process: the graphical model above represents the induced distribution over Qfunctions when conditioning on a particular policy eval- uation algorithm, policy, ofﬂine dataset, and
[Table]
prior
[Table]
An interesting question posed by reviewers of our work was “[W]hatever formal reasoning systemwe’d like to use, what is the ideal answer, given access to arbitrary computational resources, so thatapproximations are unnecessary?
I.e., how do we quantify our uncertainty about the MDP and valuefunction before seeing data, and how do we quantify it after seeing data?"
It is important to begin by clarifying what is the mathematical object we are trying to obtainuncertainties over.
In this work, we do not quantify uncertainties about any aspects of the MDPitself (although this is an interesting question which comes up in model-based methods as wellas other settings such as Meta-Learning [57, 58]).
Our goal in this work is to directly estimate
Q π ( s, a ) =
r ( s, a ) + γ · E s (cid:48) ∼ MDP,a (cid:48)
∼ π
[ Q ( s (cid:48) , a (cid:48) )] , for a ∼ π ( s ) , and to obtain uncertainties about
Q π ( s, a ) .
Let Q ( s, a ) be a predictor S ×
A → R that needs to be evaluated on – and hopefully generalize wellto – ( s, a ) / ∈ D , where D is the ofﬂine dataset.
When we choose to represent Q ( s, a ) using neuralnetworks, Gaussian Processes, or K-nearest-neighbours, we are not just making approximations forcomputational reasons, but are actually choosing a function class which we believe will generalizewell to unseen ( s, a ) .
One practical example of learning Q -functions is to use Fitted Q -Evaluation (FQE)
[55] on theprovided data using gradient descent with a desired neural network architecutre.
Due to the randomweight initialization, this procedure induces a distribution on the Q -functions which is captured bythe probabilistic graphical model (PGM) in Figure 19a.
In other words, by conditionining on thepolicy, data, architecture, and policy evaluation algorithm, we are imposing a belief over Q -functions.
Note that this is effectively the same justiﬁcation as using ensembles in supervised deep learning, where ensembles are state-of-the-art for accuracy and calibration [17].
For the sake of theoreticalanalysis (Section 3), we studied this belief distribution under the inﬁnite-width NTK [45, 52] networksetting, in which case the distribution over Q -functions is a Gaussian Process.
The focus of this work is to ask the question: “Under this imposed belief, what should the policyupdate be?".
Our proposed answer is to optimize the policy with respect to the lower-conﬁdencebound of our beliefs: In the standard actor-critic setup, the policy optimization objective takes the formmax π E d ( s )
[ Q ( s, π ( s ))] , where d ( s ) is some distribution over states (e.g. the initial state distribution,a uniform distribution over states in the ofﬂine dataset D , etc.).
For the pessimistic ofﬂine RL settting,our proposed policy optimization objective takes the form max π LCB (cid:16) E d ( s )
[ Q ( s, π ( s ))] (cid:17) .
In MSG,for practical reasons, we convert this to max π E d ( s ) (cid:104) LCB (cid:16) Q ( s, π ( s )) (cid:17)(cid:105) (which is a lower-bound ofthe latter objective).
The graphical model in Figure 19b also highlights an example of interesting future directions:
Consider an ofﬂine RL setup where we keep track of the various policies generating the data andtheir Q -functions (approximated through Monte-Carlo estimation).
Then, we can impose a priordistribution over architectures, and using the available Q -functions, infer a posterior distribution overarchitectures (for a probabilistic interpretation, we could treat the output of a Q -function as the meanof a standard normal distribution).
Subsequently, given a new policy, we can learn its Q -functionunder the posterior neural network architecture distribution.
J Runtime Comparison Table
[Table]
K Deﬁnition of Deep Ensembles
In the deep learning literature, deep ensembles refers to the setting where an ensemble of neuralnetworks with identical network architectures are trained using the same data and objective functions, with the only difference in ensemble members being the random weight initialization of thenetworks [43, 17].
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
l k - E x p -
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Table] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]