EM A Q: E XPECTED -M AX Q-L EARNING O PERATOR FOR S IMPLE Y ET E FFECTIVE O FFLINE AND O NLINE RL
Anonymous authors
Paper under double-blind review
A BSTRACT
Off-policy reinforcement learning (RL) holds the promise of sample-efﬁcient learning of decision-making policies by leveraging past experience.
However, in the ofﬂine RL setting – where a ﬁxed collection of interactions are provided and no further interactions are allowed – it has been shown that standard off-policy RL methods can signiﬁcantly underperform.
Recently proposed methods often aim to address this shortcoming by constraining learned policies to remain close to the given dataset of interactions.
In this work, we closely investigate an important simpliﬁcation of BCQ (Fujimoto et al., 2018a) – a prior approach for ofﬂine RL – which removes a heuristic design choice and naturally restrict extracted policies to remain exactly within the support of a given behavior policy.
Importantly, in contrast to their original theoretical considerations, we derive this simpliﬁed algorithm through the introduction of a novel backup operator, Expected-Max
Q-Learning (EMaQ), which is more closely related to the resulting practical algorithm.
Speciﬁcally, in addition to the distribution support, EMaQ explicitly considers the number of samples and the proposal distribution, allowing us to derive new sub-optimality bounds which can serve as a novel measure of complexity for ofﬂine RL problems.
In the ofﬂine RL setting – the main focus of this work – EMaQ matches and outperforms prior state-of-the-art in the D4RL benchmarks (Fu et al., 2020a).
In the online RL setting, we demonstrate that EMaQ is competitive with Soft Actor Critic (SAC).
The key contributions of our empirical ﬁndings are demonstrating the importance of careful generative model design for estimating behavior policies, and an intuitive notion of complexity for ofﬂine RL problems.
With its simple interpretation and fewer moving parts, such as no explicit function approximator representing the policy, EMaQ serves as a strong yet easy to implement baseline for future work.
I NTRODUCTION
Leveraging past interactions in order to improve a decision-making process is the hallmark goal of off-policy reinforcement learning (RL) (Precup et al., 2001; Degris et al., 2012).
Effectively learning from past experiences can signiﬁcantly reduce the amount of online interaction required to learn a good policy, and is a particularly crucial ingredient in settings where interactions are costly or safety is of great importance, such as robotics Gu et al.
; Kalashnikov et al.
(2018a),
health Murphy et al.
, dialog agents (Jaques et al., 2019), and education Mandel et al.
In recent years, with neural networks taking a more central role in the RL literature, there have been signiﬁcant advances in developing off-policy RL algorithms for the function approximator setting, where policies and value functions are represented by neural networks (Mnih et al., 2015; Lillicrap et al., 2015; Gu et al., 2016b;a; Haarnoja et al., 2018; Fujimoto et al., 2018b).
Such algorithms, while off-policy in nature, are typically trained in an online setting where algorithm updates are interleaved with additional online interactions.
However, in purely ofﬂine RL settings, where a dataset of interactions are provided ahead of time and no additional interactions are allowed, the performance of these algorithms degrades drastically (Fujimoto et al., 2018a; Jaques et al., 2019).
A number of recent methods have been developed to address this shortcoming of off-policy RL algorithms.
A particular class of algorithms for ofﬂine RL that have enjoyed recent success are those based on dynamic programming and value estimation (Fujimoto et al., 2018a;
Jaques et al., 2019; Kumar et al., 2019; Wu et al., 2019; Levine et al., 2020).
Most proposed algorithms are designed with a key intuition that it is desirable to prevent policies from deviating too much from the provided collection of interactions.
By moving far from the actions taken in the ofﬂine data, any subsequently learned policies or value functions may not generalize well and lead to the belief that certain actions will lead to better outcomes than they actually would.
Furthermore, due to the dynamics of the MDP, taking out-of-distribution actions may lead to states not covered in the ofﬂine data, creating a snowball effect (Ross et al., 2011).
In order to prevent learned policies from straying from the ofﬂine data, various methods have been introduced for regularizing the policy towards a base behavior policy (e.g. through a divergence penalty (Jaques et al., 2019; Wu et al., 2019; Kumar et al., 2019) or clipping actions (Fujimoto et al., 2018a)).
Taking the above intuitions into consideration, in this work we investigate a simplifcation of the
BCQ algorithm (Fujimoto et al., 2018a)
(a notable prior work in ofﬂine RL), which removes a heuristic design choice and has the property that extracted policies remain exactly within the support of a given behavior policy.
In contrast to the theoretical considerations in the original work, we derive this simpliﬁed algorithm in a theoretical setup that more closely reﬂects the resulting algorithm.
We introduce the Expected-Max Q-Learning (EMaQ) operator, which interpolates between the standard Q-function evaluation and Q-learning backup operators.
The EMaQ operator makes explicit the relation between the proposal distribution and number of samples used, and leads to sub-optimality bounds which introduce a novel notion of complexity for ofﬂine RL problems.
In its practical implementation for the continuous control and function approximator setting, EMaQ has only two standard components (an estimate of the base behavior policy, and Q functions) and does not explicitly represent a policy, requiring ﬁtting one less function approximator than prior approaches (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019).
In online RL, EMaQ is competitive with Soft Actor Critic (SAC) (Haarnoja et al., 2018) and surpasses SAC in the deployment-efﬁcient setting (Matsushima et al., 2020).
In the ofﬂine RL setting – the main focus of this work – EMaQ matches and outperforms prior state-of-the-art in the D4RL (Fu et al., 2020a) benchmark tasks.
Through our explorations with EMaQ we make two intriguing ﬁndings.
First, due to the strong dependence of EMaQ on the quality of behavior policy used, our results demonstrate the signiﬁcant impact of careful considerations in modeling the behavior policy that generate the ofﬂine interaction datasets.
Second, relating to the introduced notion of complexity, in a diverse array of benchmark settings considered in this work we observe that surprisingly little modiﬁcation to a base behavior policy is necessary to obtain a performant policy.
As an example, we observe that while a HalfCheetah random policy obtains a return of 0 , a policy that at each state uniformly samples only 5 actions and chooses the one with the best value obtains a return of 2000 .
The simplicity, intuitive interpretation, and strong empirical performance of EMaQ make it a great test-bed for further examination and theoretical analyses, and an easy to implement yet strong baseline for future work in ofﬂine RL.
B ACKGROUND
Throughout this work, we represent Markov Decision Process (MDP) as M = (cid:104)S , A , r, P , γ (cid:105) , with state space S , action space A , reward function r : S×A→ R , transition dynamics P , and discount γ .
In ofﬂine RL, we assume access to a dataset of interactions with the MDP, which we will represent as collection of tuples D
= { ( s, a, s (cid:48) , r, t ) } N , where t is an indicator variable that is set to True when s (cid:48) is a terminal state.
We will use µ to represent the behavior policy used to collect D , and depending on the context, we will overload this notation and use µ to represent an estimate of the true behavior policy.
For a given policy π , we will use the notation d π ( s ) , d π ( s, a ) to represent the state-visitation and state-action visitation distributions respectively.
As alluded to above, a signiﬁcant challenge of ofﬂine RL methods is the problem of distribution shift.
At training-time, there is no distribution shift in states as a ﬁxed dataset D is used for training, and the policy and value functions are never evaluated on states outside of d µ ( s ) .
However, a very signiﬁcant challenge is the problem of distribution shift in actions .
Consider the Bellman backup for obtaining the Q-function of a given policy π ,
[Equation]
The target Q-values on the right hand side depend on action samples a (cid:48) ∼ π
( a (cid:48) | s (cid:48) ) .
If the sampled actions are outside the distribution of actions observed in D , the estimated Q-values can be erroneous leading to incorrect target values.
The effects of action distribution shift are further exacerbated in actor-critic algorithms; out of distribution (OOD) actions may incorrectly be assigned high values, in which case the policy will be updated to further sample OOD actions, leading to a hazardous loop.
An important approach – with particular recent interest – to mitigate the effects of both kinds of distributional shift is to devise methods for constraining learned policies to remain close to the behavior policy µ : d π ( s, a ) ≈ d µ ( s, a ) .
Below, we set the stage by reviewing a closely related prior work in ofﬂine RL.
Batch Constrained Q-Learning (BCQ)
In BCQ (Fujimoto et al., 2018a)
the aim is to constrain a
Q-Learning based algorithm such that it will be effective in the ofﬂine RL continuous control setting with function approximators.
To do so, the trained policy is parameterized as:
[Equation] [Equation]
where y ( s, a ) are target Q-values, Q ψ is learned with the objective in equation 4, µ ( a | s ) is an estimate of the base behavior policy (a generative model trained using the dataset D ), and ξ θ is an action perturbation model trained to modify actions towards more optimal ones.
Crucially , each component of the output of ξ θ is bounded to the range [ − Φ , Φ] .
The key intuition is that because a i are sampled from an estimate of the behavior policy, they should hopefully be within the distribution observed in D .
Thus, since the perturbation model is constrained by the hyperparameter Φ , the perturbed actions should not be too far from actions in the dataset.
This should mitigate errors in value estimates, which should in turn lead to better updates for the perturbation model.
E XPECTED -M AX Q-L EARNING
We make the observation that, in the BCQ algorithm, if we could obtain a good estimate µ ( a | s ) and sufﬁciently increased the number of samples N , there would be no need for the perturbation network ξ θ .
This simpliﬁcation would remove one additional function approximator and the associated hyperparameter Φ .
This is the driving intuition of our work, which we frame theoretically in a manner that encapsulates the key components: the behavior policy µ and number of samples N .
Below, we introduce the Expected-Max Q operator, illustrate its key properties for tabular MDPs, and obtain sub-optimality bounds which can serve as a novel measure of complexity of an ofﬂine RL problem for future theoretical work.
We then provide an extension to the ofﬂine RL setting with function approximators, and then discuss the generative model used to approximate the behavior policy.
3. E XPECTED -M AX Q O PERATOR
Let µ ( a | s ) be an arbitrary behavior policy, and let { a i } N ∼ µ ( a | s ) denote sampling N iid actions from µ ( a | s ) .
Let Q : S ×
A → R be an arbitrary function.
For a given choice of N , we deﬁne the
Expected-Max Q-Learning operator (EMaQ) T Nµ Q as follows:
[Equation]
Q-Evaluation for µ
[Equation]
This operator provides a natural interpolant between the on-policy backup for µ (Eq. 6) when
N = 1 , and the Q-learning backup (Eq. 7) as N → ∞ (if µ ( a | s ) has full support over A ).
We formalize these observations more precisely below when we articulate the key properties in the tabular MDP setting.
We discuss how this relates to existing modiﬁed backup operators in the related work.
3. D YNAMIC P ROGRAMMING P ROPERTIES IN THE T ABULAR MDP S ETTING
To understand any novel backup operator it is useful to ﬁrst characterize its key dynamic programming properties in the tabular MDP setting.
First, we establish that EMaQ retains essential contraction and ﬁxed-point existence properties, regardless of the choice of N ∈ N .
In the interest of space, all missing proofs can be found in Appendix A.
Theorem 3.1.
In the tabular setting, for any N ∈ N , T Nµ is a contraction operator in the L ∞ norm.
Hence, with repeated applications of the T Nµ , any initial Q function converges to a unique ﬁxed point.
Theorem 3.2.
Let Q Nµ denote the unique ﬁxed point achieved in Theorem 3.1, and let π Nµ ( a | s ) denote the policy that samples N actions from µ ( a | s ) , { a i } N , and chooses the action with the maximum Q Nµ .
Then Q Nµ is the Q-value function corresponding to π Nµ ( a | s ) .
Proof.
(Theorem 3.2)
Rearranging the terms in equation 5 we have,
[Equation]
Since by deﬁnition Q Nµ is the unique ﬁxed point of T Nµ , we have our result.
From these results we can then rigorously establish the interpolation properties of the EMaQ family.
Theorem 3.3.
Let π ∗ µ denote the optimal policy from the class of policies whose actions are restricted to lie within the support of the policy µ ( a | s ) .
Let Q ∗ µ denote the Q-value function corresponding to π ∗ µ .
Furthermore, let Q µ denote the Q-value function of the policy µ ( a | s ) .
Let µ ∗ ( s ) :
= (cid:82) Support ( π ∗ µ ( a | s ))
µ ( a | s ) denote the probability of optimal actions under µ ( a | s ) .
Under the assumption that inf s µ ∗ ( s ) > 0 and r ( s, a ) is bounded, we have that,
[Equation]
That is, Theorem 3.3 shows that, given a base behavior policy µ ( a | s ) , the choice of N makes
the
EMaQ operator interpolate between evaluating the Q-value of µ on the one hand, and learning the optimal Q-value function on the other (optimal subject to the support constraint discussed in
Theorem 3.3).
In the special case where µ ( a | s ) has full support over the action space A , EMaQ interpolates between the standard Q-Evaluation and Q-Learning operators in reinforcement learning.
Intuitively, as we increase N , the ﬁxed-points Q Nµ should correspond to increasingly better policies π Nµ ( a | s ) .
We show that this is indeed the case.
Theorem 3.4.
For all N, M ∈ N , where N > M , we have that ∀ s ∈ S , ∀ a ∈ Support ( µ ( ·| s )) ,
Q Nµ ( s, a ) ≥ Q Mµ ( s, a ) .
Hence, π Nµ ( a | s ) is at least as good of a policy as π Mµ ( a | s ) .
It is also valuable to obtain a sense of how suboptimal π Nµ ( a | s ) may be with respect to the optimal policy supported by the policy µ ( a | s ) .
Theorem 3.5.
For s ∈ S let,
[Equation]
The suboptimality of Q Nµ can be upperbounded as follows,
[Equation]
The same also holds when Q ∗ µ is replaced with Q N µ in the deﬁnition of ∆ .
3. A M EASURE OF C OMPLEXITY FOR O FFLINE RL
The bounds in equation 8 capture the main intuitions about the interplay between µ ( a | s ) and the choice of N .
If for each state, µ ( a | s ) places sufﬁcient mass over the optimal actions, π Nµ will be
Function
Function π test ( s ) : close to π ∗ µ .
The two variants of ∆( s, N ) , based on Q ∗ µ or Q Nµ , suggest an intriguing notion of difﬁculty for an ofﬂine RL problem.
If we could estimate either of these Q-value functions, then for a desired set of states (such as initial states) we could plot ∆( s, N ) as decreasing function of
N .
The rate at which this function decreases could serve as an intuitive notion of difﬁculty for a given ofﬂine ofﬂine RL problem which consists of an MDP and a given behavior policy.
While we leave theoretical investigations of this measure for future work, our empirical results in Sectiondemonstrate that the effective value of N may be surprisingly small.
3. O FFLINE RL S ETTING WITH F UNCTION A PPROXIMATORS
Typically, we are not provided with the policies that generated the provided trajectories.
Hence, as a ﬁrst step we ﬁt a generative model µ ( a | s ) to the ( s, a ) pairs in the ofﬂine dataset, representing the mixture of policies that generated this data (details below).
Having obtained µ ( a | s ) , we move on to the EMaQ training procedure.
Similar to prior works (Fujimoto et al., 2018a; Kumar et al., 2019;
Wu et al., 2019), we train K Q functions (represented by MLPs) and make use of an ensembling procedure to combat overestimation bias (Hasselt, 2010; Van Hasselt et al., 2016; Fujimoto et al., 2018b).
Letting D represent the ofﬂine dataset, the objective for the Q functions takes the following form:
[Equation]
where t is the indicator variable 1 [ s (cid:48) is terminal ] , and Q (cid:48)
ens represents the ensemble of target Q functions.
In short, we sample N actions from µ ( a (cid:48) | s (cid:48) ) and take the value of the best action to form the target.
The algorithm box describing the full training loop can be viewed in Algorithm 2.
Notably, we do not train an explicit neural network representing the policy.
At test-time, given a state s , we sample N actions from µ ( a | s ) and choose the action with the maximum value under the ensemble of Q functions (see Algorithm 1).
While TestEnsemble can differ from the Ensemble function used to compute target Q values 1 , in this work we used the same ensembling procedure with λ = 1 .
(with the exception of experiments in Section F).
3. I NTUITION FOR O FFLINE EM A Q AND M ODELING µ ( a | s )
The intuition for how ofﬂine EMaQ aims to address the problem of erroneous value estimates can be understood from attending to equation 10 and the form of the test-time policy (Algorithm 1).
In equation 10 we observe that target values for the Q functions are computed by sampling actions from the behavior policy estimate µ , and not a separately learned policy that may sample out of distribution actions, as in BCQ.
At test-time, our implicit policy is also formed by choosing amongst actions sampled from µ .
Hence, if µ accurately estimates the behavior policy well, we will never sample actions outside the support and will not need to evaluate the value of such actions.
In the practical setting where µ may have inaccuracies, the hyperparameter N acts as an implicit regularizer: Using very large values of N maximizes the chance of sampling actions that are out of distribution and have erroneous value estimates, while smaller N reduces the chance of this happening in every update iteration and therefore smoothens the incorrect values.
With the importance of a good behavior estimates accentuated in our proposed method, we pay closer attention to the choice of generative model used for representing µ .
Past works (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019) have typically used Variational Auto-Encoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) to represent the behavior distribution µ ( a | s ) .
Unfortunately, after training the aggregate posterior q agg ( z ) :=
E x
[ q ( z | x )] of a VAE does not typically align well with its prior, making it challenging to sample from in a manner that effectively covers the distribution it was trained on 2 .
We opt for using an autoregressive architecture based on MADE (Germain et al., 2015) as it allows for representing more expressive distributions and enables more accurate sampling.
Inspired by recent works (Metz et al., 2017; Van de Wiele et al., 2020), our generative model architecture also makes use of discretization in each action dimension.
Full details can be found in Appendix B.
R ELATED W ORK
Ofﬂine RL Many recent methods for ofﬂine RL (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019; Jaques et al., 2019), where no interactive data collection is allowed during training, mostly rely on constraining the learned policy to stay close to the data collection distribution.
Fujimoto et al.
(2018a)
clip the maximum deviation from actions sampled from a base behavior policy, while Kumar et al.
; Wu et al.
; Jaques et al.
(2019) incorporate additional distributional penalties (such as KL divergence or MMD) for regularizing learned policies to remain close to the base policy.
Our work is an instance of this family of approaches for ofﬂine RL; however, arguably our method is simpler as it does not involve learning an additional proposal-modifying policy Fujimoto et al.
(2018a), or modifying reward functions (Kumar et al., 2019;
Jaques et al., 2019).
Finding Maximizing Actions Na ¨ ıvely, EMaQ can also be seen as just performing approximate search for max a Q ( s, a ) in standard Q-learning operator, which has been studied in various prior works for Q-learning in large scale spaces (e.g. continuous).
NAF (Gu et al., 2016b) and
ICNN (Amos et al., 2017) directly constrain the function family of Q-functions such that the optimization can be closed-form or tractable.
QT-OPT (Kalashnikov et al., 2018b) makes use of two iterations of the Cross-Entropy Method (Rubinstein & Kroese, 2013), while CAQL (Ryu et al., 2019) uses Mixed-Integer Programming to ﬁnd the exact maximizing action while also introducing faster approximate alternatives.
In (Van de Wiele et al., 2020) – the most similar approach to our proposed method EMaQ – throughout training a mixture of uniform and learned proposal distributions are used to sample actions.
The sampled actions are then evaluated under the learned Q functions, and the top K maximizing actions are distilled back into the proposal distribution.
In contrast to our work, these works assume these are approximate maximization procedures and do not provide extensive analysis for the resulting TD operators.
Our theoretical analysis on the family of TD operators described by EMaQ can therefore provide new perspectives on some of these highly successful
Q-learning algorithms (Kalashnikov et al., 2018a; Van de Wiele et al., 2020) – particularly on how the proposal distribution affects convergence.
Modiﬁed Backup Operators Many prior works study modiﬁcations to standard backup operators to achieve different convergence properties for action-value functions or their induced optimal policies.
Ψ -learning (Rawlik et al., 2013) proposes a modiﬁed operator that corresponds to policy iterations with KL-constrained updates (Kakade, 2002; Peters et al., 2010; Schulman et al., 2015) where the action-value function converges to negative inﬁnity for all sub-optimal actions.
Similarly but distinctly, Fox et al. (2015); Jaques et al.
(2017); Haarnoja et al.
; Nachum et al.
(2017) study smoothed TD operators for a modiﬁed entropy- or KL-regularized RL objective.
Bellemare et al.
(2016) derives a family of consistent Bellman operators and shows that they lead to increasing action gaps (Farahmand, 2011) for more stable learning.
However, most of these operators have not been studied in ofﬂine learning.
Our work adds a novel family operators to this rich literature of operators for RL, and provides strong empirical validation on how simple modiﬁcations of operators can translate to effective ofﬂine RL with function approximations.
E XPERIMENTS
For all experiments we make use of the codebase of (Wu et al., 2019), which presents the BRAC off-policy algorithm and examines the importance of various factors in BCQ (Fujimoto et al., 2018a) and BEAR (Kumar et al., 2019) methods.
We implement EMaQ into this codebase.
We make use of the recently proposed D4RL (Fu et al., 2020b) datasets for bechmarking ofﬂine RL.
Online EMaQ
Despite obtaining strong online RL results competitive with and outperforming
SAC (Haarnoja et al., 2018)
(Figure 3), as the main focus of our work is for the ofﬂine setting, we have placed our online RL methodology and results in
Appendix F.
However, we emphasize that signiﬁcance of obtaining strong online RL performance with effectively the same algorithm as the ofﬂine setting should not be overlooked.
Most prior ofﬂine RL works have not considered how their methods might transfer to online or batched online setting, and recent work (Nair et al., 2020) has demonstrated the challenges of ﬁnetuning from a policy trained ofﬂine, in the online setting.
5. P RACTICAL E FFECT OF N AND THE C HOICE OF G ENERATIVE M ODEL µ
We begin by empirically evaluating key aspects of ofﬂine EMaQ, namely the effect of N , and choice of generative model used for representing the behavior estimate µ .
In prior approaches such as those described in the background section of this work, care must be taken in choosing the hyperparameter that dictates the extent to which learned policies can deviate from the base behavior policies; too small and we cannot improve upon the base policy, too large and the value of actions cannot be correctly estimated.
In EMaQ, at least in theory, choosing higher values of N should result in strictly better policies.
Additionally, there exists a concern that N may need to be impractically large.
Thus, we empirically investigate to what extent the monotonic trend holds in practice, and seek to understand what magnitudes of N result in good policies in practical benchmark domains.
Figure 1 presents our results with N ∈ { 5 , 10 , 25 , 50 , 100 , 200 , 400 } .
In the green plots, we observe that empirical results follow our intuitions: with increasing N the resultant policies become better.
In the medium-expert settings (i.e. orange plots), while for smaller values of N we observe strong performance, there appears to be a downward trend.
As discussed in Section 3.5, smaller values of N result in an implicit regularization.
Hence, the orange plots may indicate that even with the stronger choice of generative models in EMaQ, inaccuracies in value estimates may still exist, suggesting the need for future work that introduces better regularizers for the value functions than ensembling (Kumar et al., 2020).
Lastly, the red plots indicate settings where behavior is erratic.
Closer examination of training curves and our experiments with other off-policy methods (Figure 2) suggests that this may be due to the intrinsic nature of these environment and data settings.
The dashed horizontal lines in Figure 2 represent the performance of BEAR – which uses a VAE for representing µ – as reported in the D4RL (Fu et al., 2020b) benchmark paper (apples to apples comparison in Section 5.2).
Our results demonstrate that the combination of a strong generative model and EMaQ’s simply constrained backup operator can match and in many cases noticeably exceed results from prior algorithms and design choices.
Comparing Figure 2 to Figure 6 in the Appendix, we observe that our choice of generative model is crucial to the performance of EMaQ .
With a
VAE architecture as used in prior work, EMaQ’s performance is signiﬁcantly reduced, in most cases worse than prior reported results for BEAR, and never exhibits a monotonic trend as a function of
N .
This is despite the fact that when evaluating the performance of the behavior estimate µ under the two architecture choices results in almost identical results (the ﬁrst column of each sub-plot corresponding to µ ( a | s ) ).
We do not believe that autoregressive models are intrinsically better than
VAEs, but rather our results demonstrate the need for more careful attention on the choice of µ ( a | s ) .
Since EMaQ is closely tied to the choice of behavior model, it may be more effective for evaluating how well µ ( a | s ) represents the given ofﬂine dataset.
From a practical perspective, our results suggest that for a given domain, focusing efforts on building-in good inductive biases in the generative models and value functions might be sufﬁcient to obtain strong ofﬂine RL performance in many domains.
5. C OMPARISON ON D4RL O FFLINE RL B ENCHMARK
To evaluate EMaQ with respect to prior methods, we compare to two popular and closely related prior methods for ofﬂine RL, BCQ (Fujimoto et al., 2018a) and BEAR (Kumar et al., 2019).
As with the previous section, full experimental details can be found in Appendix G.1.
Figure 2 and
Table 1 present our empirical results.
Note that with our proposed autoregressive models, the results for BEAR are matched and in some cases noticeably above the values reported in the D4RL benchmark (Fu et al., 2020b) (green horizontal lines).
For easier interpretation, the plots are colored
[Table]
the same as in Figure 1.
Our key take-away is that despite its simplistic form, EMaQ is strongly competitive with prior state-of-the-art methods, and in the case of Table 1 outperforms prior approaches.
Despite this, there remain many domains in the D4RL benchmark on which none of the considered algorithms make any progress (Table 1), indicating that much algorithmic advances are still necessary for solving many of the considered domains.
A very eye-catching result in above ﬁgures is that in almost all settings of the standard Mujoco environments (Figures 1 and 2), just N = 5 signiﬁcantly improves upon µ ( a | s ) and in most settings matches or exceeds signiﬁcantly beyond previously reported results.
Concretely, this means that in the HalfCheetah-Random setting, if at each state we sample 5 actions uniformly random and choose the best one under the learned Q-value function, we convert a random policy with return 0 to a policy with return 2000.
In this way, EMaQ provides a quite intuitive and surprising measure of the complexity for ofﬂine RL problems.
This empirical observation also corroborates our discussion in Section 3.3, encouraging future theoretical investigations into ∆( s, N ) .
C ONCLUSION
In this work, we investigate a signiﬁcant simpliﬁcation of the BCQ (Fujimoto et al., 2018a) algorithm by removing the heuristic perturbation network.
By introducing the Expect-Max Q-Learning operator, we present a novel theoretical setup that takes into account the proposal distribution µ ( a | s ) and the number of action samples N , and hence more closely matches the resulting practical algorithm.
With fewer moving parts and one less function approximator, EMaQ matches and outperforms prior state-of-the-art in online and ofﬂine RL.
Our investigations with EMaQ demonstrate the signiﬁcance of careful considerations in the design of generative models used.
Furthermore, our theoretical and empirical ﬁndings bring into light novel notions of complexity for ofﬂine RL problems.
Given the simplicity, tractable theory, and state-of-the-art performance of EMaQ, we hope our work can serve as a foundation for future works on understanding and improving ofﬂine RL.
A P
[Equation]
All the provided proofs operate under the setting where µ ( a | s ) has full support over the action space.
When this assumption is not satisﬁed, the provided proofs can be transferred by assuming we are operating in a new MDP M µ as deﬁned below.
Given the MDP M = (cid:104)S , A , r, P , γ (cid:105) and µ ( a | s ) , let us deﬁne the new MDP M µ = (cid:104)S µ , A µ , r, P , γ (cid:105) , where S µ denotes the set of reachable states by µ , and A µ is A restricted to the support of µ ( a | s ) in each state in S µ .
A. C ONTRACTION M APPING
Theorem 3.1.
In the tabular setting, for any N ∈ N , T Nµ is a contraction operator in the L ∞ norm.
Hence, with repeated applications of the T Nµ , any initial Q function converges to a unique ﬁxed point.
Proof.
Let Q 1 and Q 2 be two arbitrary Q functions.
[Equation]
[Equation] [Equation]
where line 15 is due to the following: Let ˆ a = arg max { a i } N Q 1 ( s (cid:48) , a i )
[Equation]
A. L IMITING B EHAVIOR
Theorem 3.
[Equation]
Let π ∗ µ denote the optimal policy from the class of policies whose actions are restricted to lie within the support of the policy µ ( a | s ) .
Let Q ∗ µ denote the Q-value function corresponding to π ∗ µ .
Furthermore, let Q µ denote the Q-value function of the policy µ ( a | s ) .
Let µ ∗ ( s ) :
= (cid:82) Support ( π ∗ µ ( a | s ))
µ ( a | s ) denote the probability of optimal actions under µ ( a | s ) .
Under the assumption that inf s µ ∗ ( s ) > 0 and r ( s, a ) , we have that,
[Equation]
and
[Equation]
Let µ ∗ ( s ) :
= (cid:82) Support ( π ∗ µ ( a | s ))
µ ( a | s ) denote the probability of optimal actions under µ ( a | s ) .
To show lim N →∞ Q Nµ = Q ∗ µ , we also require the additional assumption that inf s µ ∗ ( s ) > 0 .
Proof.
Given that,
[Equation]
the unique ﬁxed-point of T 1 µ is the Q-value function of the policy µ ( a | s ) .
Hence Q 1 µ = Q µ .
The second part of this theorem will be proven as a Corollary to Theorem 3.
A. I NCREASINGLY B ETTER P OLICIES
Theorem 3.4.
For all N, M ∈ N , where N > M , we have that ∀ s ∈ S , ∀ a ∈ Support ( µ ( ·| s )) ,
Q Nµ ( s, a ) ≥ Q Mµ ( s, a ) .
Hence, π Nµ ( a | s ) is at least as good of a policy as π Mµ ( a | s ) .
Proof.
It is sufﬁcient to show that ∀ s, a, Q N +1 µ ( s, a ) ≥ Q Nµ ( s, a ) .
We will do so by induction.
Let
Q i denote the resulting function after applying T N +1 µ , i times, starting from Q Nµ .
Base Case
By deﬁnition Q 0 :
= Q Nµ .
Let s ∈ S , a ∈ A .
[Equation]
[Equation] [Equation]
Induction Step
Assume ∀ s, a, Q i ( s, a ) ≥ Q i − 1 ( s, a ) .
Q
[Equation]
Hence, by induction we have to ∀ i, j, i > j = ⇒ ∀ s, a, Q i ( s, a ) ≥ Q j ( s, a ) .
Since Q 0 = Q Nµ and lim i →∞ Q i = Q N +1 µ , we have than ∀ s, a, Q N +1 µ ( s, a ) ≥ Q Nµ ( s, a ) .
Thus π N +1 µ is a better policy than π Nµ , and by a simple induction argument, π Nµ is a better policy than π Mµ when N > M .
A. B OUNDS
Theorem 3.5.
For s ∈ S let,
[Equation]
The suboptimality of Q Nµ can be upperbounded as follows,
[Equation] [Equation]
The same also holds when Q ∗ µ is replaced with Q Nµ in the deﬁnition of ∆ .
Proof.
The two versions where ∆( s ) is deﬁned in terms of Q Nµ
Version with Q N µ
Let T QL denote the backup operation in Q -Learning.
Let ( T QL ) m =
T QL
[Equation]
T QL
[Equation]
... ◦ T QL
[Equation]
We know the following statements to be true:
[Equation]
Putting these together we have that,
[Equation]
Version with Q ∗ µ
Very similarly we have,
[Equation]
Corollary A.1.
Let V µ , Q µ , A µ denote the value, Q, and advantage functions of µ respectively.
When N = 1 we have that,
[Equation]
It is interesting how the sub-optimality can be upper-bounded in terms of a policy’s own advantage function.
Corollary A.2.
(Proof for second part of Theorem 3.3)
Proof.
We want to show lim N →∞ Q Nµ = Q ∗ .
More exactly, what we seek to show is the following,
[Equation]
or,
[Equation] [Equation]
Let (cid:15) > 0 .
Recall,
[Equation]
Let inf s µ ∗ ( s ) =
p > 0 .
Let the lower and upper bounds of rewards be (cid:96) and L , and let α = 11 − γ (cid:96) and β = 11 − γ L .
We have that,
[Equation]
Hence ∀ s ,
[Equation]
Thus, for large enough N we have that,
[Equation]
concluding the proof.
B A UTOREGRESSIVE G ENERATIVE M ODEL
The architecture for our autoregressive generative model is inspired by the works of (Metz et al., 2017; Van de Wiele et al., 2020; Germain et al., 2015).
Given a state-action pair from the dataset ( s, a ) , ﬁrst an MLP produces a d -dimensional embedding for s , which we will denote by h .
Below, we use the notation a i to denote the i th index of a , and a [: i ] to represent a slice from ﬁrst up to and not including the i th index, where indexing begins at 0.
We use a discretization in each action dimension.
Thus, we discretize the range of each action dimension into N uniformly sized bins, and represent a by the labels of the bins.
Let (cid:96) i denote the label of the i th action index.
Training We use separate MLPs per action dimension.
Each MLP takes in the d -dimensional state embedding and ground-truth actions before that index, and outputs N logits for the choice over bins.
The probability of a given index’s label is given by,
[Equation]
We use standard maximum-likelihood training (i.e. cross-entropy loss).
Sampling Given a state s , to sample an action we again embed the state, and sample the action indices one-by-one.
[Equation]
C A LGORITHM B OX
Algorithm 2:
Algorithm
Ofﬂine dataset D , Pretrain µ ( a | s ) on D
Initialize K Q functions with parameters θ i , and K target Q functions with parameters θ target i
Ensemble parameter λ , Exponential moving average parameter α
Function Ensemble( values ) : return λ · min( values ) + (1 − λ ) · max( values )
Function y target ( s, a, s (cid:48) , r, t ) : { a (cid:48) i } N ∼ µ ( a (cid:48) | s (cid:48) )
Qvalues ← [ ] for k ← 1 to N do /* Estimate the value of action a (cid:48) k
Qvalues.
append (cid:16) Ensemble (cid:0) [ Q targeti ( s (cid:48) , a (cid:48) k ) for all i ] (cid:1)(cid:17) return r + (1 − t ) · γ max( Qvalues ) while not converged do
Sample a batch { ( s m , a m , s (cid:48) m , r m , t m ) } M ∼ D for i = 1 , ..., K do
L ( θ i ) =
(cid:80) m (cid:16) Q i ( s m , a m ) − y target ( s m , a m , s (cid:48) m , r m , t m )
(cid:17)θ i ← θ i − AdamUpdate (cid:16) L ( θ i ) , θ i (cid:17) θ target i ← α · θ target i + (1 − α ) · θ i
D I NCONCLUSIVE E XPERIMENTS D. U PDATING THE P ROPOSAL D ISTRIBUTION
Akin to the work of (Van de Wiele et al., 2020), we considered maintaining a second proposal distribution ˜ µ that is updated to distill arg max { a i } N Q ( s, a ) , and sampling from the mixture of µ and ˜ µ .
In our experiments however, we did not observe noticeabel gains.
This may potentially be due to the relative simplicity of the Mujoco benchmark domains, and may become more important in more challenging domains with more uniformly distributed µ ( a | s ) .
E L AUNDRY L IST
• Autoregressive models are slow to generate samples from and EMaQ needs to take many samples, so it was slower to train than the alternative methods.
However, this may be addressed by better generative models and engineering effort.
F O NLINE RL
EMaQ is also applicable to online RL setting.
Combining strong ofﬂine RL methods with good exploration policies has the potential for producing highly sample-efﬁcient online RL algorithms.
Concretely, we refer to online RL as the setting where iteratively, a batch of M environment steps with an exploration policy are interleaved with M RL updates (Levine et al., 2020; Matsushima et al., 2020).
EMaQ is designed to remain within the support of the provided training distribution.
This however, is problematic for online RL which requires good exploration interleaved with RL updates.
To this end, ﬁrst, we modify our autoregressive proposal distribution µ ( a | s ) by dividing the logits of all softmaxes by τ > 1 .
This has the effect of smoothing the µ ( a | s ) distribution, and increasing the probability of sampling actions from the low-density regions and the boundaries of the support.
Given this online proposal distribution, a criteria is required by which to choose amongst sampled actions.
While there exists a rich literature on how to design effective RL exploration policies (Weng, 2020), in this work we used a simple UCB-style exploration criterion (Chen et al., 2017) as follows:
[Equation]
Given N sampled actions from the modiﬁed proposal distribution, we take the action with highest
Q explore.
We compare the online variant of EMaQ with entropy-constrained Soft Actor Critic (SAC) with automatic tuning of the temperature parameter (Haarnoja et al., 2018).
For EMaQ we swept the temperatures and used a ﬁxed bin size of 40, 8 Q-function ensembles and N = 200 .
For fairness of comparisons, we also ran SAC with similar sweeps over different collection batch sizes and number of Q-function ensembles.
In the fully online setting (trajectory batch size 1, Figure 3a), EMaQ is already competitive with SAC, and more excitingly, in the deployment-efﬁcient setting 3 (trajectory batch size 50K, Figure 3b), EMaQ can outperform SAC 4 .
Figures 4 and 5 present the results for all hyperparameter settings, for SAC and EMaQ, in the batch size 1 and batch size 50 K settings respectively.
In the fully online setting, EMaQ is already competitive with SAC, and more excitingly, in the deployment-efﬁcient setting, EMaQ can outperform SAC.
G O FFLINE RL E XPERIMENTAL D ETAILS
For each environment and data setting, we train an autoregressive model – as described above – on the provided data with 2 random seeds.
These generative models are then frozen, and used by the downstream algorithms (EMaQ, BEAR, and BCQ) as the base behavior policy ( µ ( a | s ) in EMaQ) 5 .
G.
C OMPARING O FFLINE RL M ETHODS
Following the bechmarking efforts of (Wu et al., 2019), the range of clipping factor considered for
BCQ was Φ ∈ { 0 . 005 , 0 . 015 , 0 . 05 , 0 . 15 , 0 . 5 } , and the range of target divergence value considered for BEAR was (cid:15) ∈ { 0 . 015 , 0 .
For both methods, the larger the value of the hyperparameter is, the more the learned policy is allowed to deviate from the µ ( a | s ) .
The rest of the hyperparameters use can be found in Table 2.
The autoregressive models have the following architecture sizes (refer to Appendix B for description of the models used).
The state embedding MLP consists of 2 hidden layers of dimension 750 with relu activations, followed by a linear embedding into a 750 dimensional state representation.
The individual MLP for each action dimension consist of 3 hidden layers of dimension 256 with relu activations.
Each action dimension is discretized into 40 equally sized bins.
[Figure]
G. EM A Q A BLATION E XPERIMENT
Hyperparameters are identical to those in Table 2, except batch size is 100 and number of updates is 500 K
. G. D ETAILS FOR T ABLE ?? E XPERIMENTS
Generative Model
The generative models used are almost identical to the description in Appendix
B, with a slight modiﬁcation that MLP i ( d, a [: i ]) is replace with MLP i ( d, Lin i ( a [: i ])) where Lin i is a linear transformation.
This change was not necessary for good performance; it was as architectural detail that we experimented with and did not revert prior generating Table ?? .
The model dimensions for each domain are shown in 3 in the following format (state embedding MLP hidden size, state embedding MLP number of layers, action MLP hidden size, action MLP number of layers, Ouput
[Figure]
[Table]
size of Lin i , number of bins for action discretization).
Increasing the number of discretization bins from 40 (value for standard Mujoco experiments) to 80 was the most important change.
Output dimension of state-embedding MLP is the same as the hidden size.
Hyperparameters Table 3 shows the hyperparameters used for the experiments in Table ?? .
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
H VAE R ESULTS H. I MPLEMENTATION
We also ran experiments with VAE parameterizations for µ ( a | s ) .
To be approximately matched in parameter count with our autoregressive models, the encoder and decoder both have 3 hidden layers of size 1024 with relu activations.
The dimension of the latent space was twice the number of action dimensions.
The decoder outputs a vector v which, and the decoder action distribution is deﬁned to be N ( Tanh ( v ) , I ) .
When sampling from the VAE, following prior work, samples from the VAE prior (spherical normal distribution) were clipped to the range [ − 0 .
5 , 0 . 5] and mean of the decoder distibution was used (i.e. the decoder distribution was not sampled from).
The KL divergence loss term was weighted by 0.5.
This VAE implementation was the one used in the benchmarking codebase of (Wu et al., 2019), so we did not modify it.
H. R ESULTS
As can be seen in Figure 6, EMaQ has a harder time improving upon µ ( a | s ) when using the VAE architecture described above.
However, as can be seen in Figure 7, BCQ and BEAR do show some variability as well when switching to the VAEs.
Since as an algorithm EMaQ is much more reliant on µ ( a | s ) , our hypothesis is that if it is true that the autoregressive models better captured the action distribution, letting EMaQ not make poor generalizations to out-of-distribution actions.
Figuresand 9 show autoregressive and VAE results side-by-side for easier comparison.
[Equation]
E XPERT S ETTING R ESULTS
In HalfCheetah, increasing N signiﬁcantly slows down the convergence rate of the training curves; while large N s continue to improve, we were unable to train them long enough for convergence.
In Walker, for EMaQ, BCQ, and most hyperparameter settings of BEAR, training curves have a prototypical shape of a hump, where performance improves up to a certain high value, and then continues to fall very low.
In Hopper, for higher values of N in EMaQ we observed that increasing batch size from 100 to 256 largely resolved the poor performance, but for consistency we did not alter Figure 1 with these values.
J C OMPARISON WITH S OFTMAX B ACKUP O PERATORS
We thank one of our ICLR 2021 reviewers for the motivation for this section.
For clarity of writing, we will write the forms for deterministic dynamics and remove the expectations over the next state.
An interesting connection to our proposed backup operators would be the following Softmax backup operator with similarities to EMaQ,
[Equation]
As suggested by our reviewer, the policy corresponding to soft ( a | s ) is a policy that aims to maximize Q-values, subject to a KL-constraint between itself and the policy µ ( a | s ) .
The looser the constraint, the larger the effective α and the farther the policy will be from µ .
One approach to
Monte Carlo estimation of the expectation on the right hand side could be to take samples using methods from the energy-based generative modelling literature.
An alternative approach which will more closely resembles EMaQ is to use self-normalized importance sampling,
[Equation]
In this form, the soft backup is similar to EMaQ, where instead of taking ths max Q-value over the N samples, we take an average over the N Q-values, weighted by the softmax probabilities in equation 73.
For a given N, the α = 0 would be equivalent to Q-evaluation of the policy µ ( a | s ) , and as α → ∞ , the soft backups approach EMaQ backups.
In Figure 10 we present empirical results with the soft backup operators, under a large range α ∈ { 1 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 } , in the Halfcheetah settings.
The EMaQ and soft-EMaQ were run with the same architectures, but were smaller than the ones used for the results in the main text.
We used the same checkpoints of the generative models as for the results in the main text.
The test-time policy for both approaches is the same, sampling N actions and taking the argmax action under the ensemble Q-value.
The only difference between the EMaQ and soft-EMaQ implementations was a one-line change to replace max with a softmax average of the Q-values.
Some interesting observations are the following: As anticipated, the soft EMaQ backups approach
EMaQ as the value of α is increased.
However, the necessary value of α to match the performance of
EMaQ can be quite large.
In the medium-expert setting, where ﬁgure 1 suggests challenges arising from the combination of large N s and function-approximators, we did not gain much advantage from soft backups, and only α ∈ { 8 , 16 , 32 } seem to have provided some mitigation of the problem for
N = 25 .
Since the soft backup introduces an additional hyperparameter that cannot be determined ahead of time, and does not seem to provide an advantage (at least in the limited Halfcheetah settings considered), from a practical perspective, we would prefer to use the regular EMaQ backup.
K Q UALITATIVE D IFFERENCES IN T RAINING C URVES
We have sometimes observed that the curves representing agent performance throughout training can be signiﬁcantly more stable under EMaQ in comparison to BEAR and BCQ.
A domain where the differences are particularly striking are the antmaze-umaze and antmaze-umaze-diverse domains.
In ﬁgure 4 we have included plots of agent performance during training under the variety of considered hyperparameters and random seeds.
It can be seen that in these two domains, initially the BCQ agents improves in performance close to the performance of EMaQ, and the drastically degrades with more training.
In constrast, EMaQ agents remain stable even after twice as many training iterations as BCQ, which may indicate the downside of the heuristic perturbation model for constraining actions. EMaQ
BCQ
antmaze-umaze antmaze-umaze-diverse
more stable
L L ARGER P LOTS FOR V ISIBILITY
Due to larger size of plots, each plot is shown on a separate page below.
For ablation results, see
Figure 11.
For MuJoCo results, see Figure 12.
F i gu r e 8 :
R e s u lt s w it
h
bo t h a u t o r e g r e ss
i v e a nd
VA E m od e l s i n on e p l o t f
o r ea s i e r c o m p a r i s on .
F i gu r e 9 :
R e s u lt s w it
h
bo t h a u t o r e g r e ss
i v e a nd
VA E m od e l s i n on e p l o t f
o r ea s i e r c o m p a r i s on .
F i gu r e 10 :
C o m p a r i s on o f
S o f tE M a Q w it h
E M a Q und e r a l a r g e r a ng e o f hyp e r p a r a m e t e r s i n t h e H a l f c h ee t a h do m a i n .
F i gu r e 11 :
R e s u l t s f o r e v a l u a ti ng
E M a Q on
D 4 R
L ( F u e t a l ., 2020b )
b e n c h m a r k do m a i n s , w it h
N ∈ {,,,, 100 , 200 , 400 } .
V a l u e s a bov e µ ( a | s ) r e p r e s e n t t
h e r e s u lt o f e v a l u a ti ng t
h e b a s e b e h a v
i o r po li c i e s .
H
o r i z on t a l g r ee n li n e s r e p r e s e n t t
h e r e po r t e d p e rf
o r m a n ce o f
B E A
R i n t h e
D 4 R
L b e n c h m a r k ( a pp l e s t
o a pp l e s c o m p a r
i s on s i n F i gu r e 2 ) .
R e f e r t o m a i n t e x t ( S ec ti on 5 . 1 )
f
o r d e s c
r
i p ti on o f c o l o r-
c od i ng .
F i gu r e 12 :
C o m p a r i s on o f E
M a Q ,
BC
Q , a nd
B E A
R on
D 4 R
L ( F u e t a l ., 2020b )
b e n c h m a r k do m a
i n s w h e n u s i ng ou r p r opo s e d a u t o r e g r e
ss
i v e µ ( a | s ) .
F o r bo t h
BC
Q a nd
B E A
R , fr o m l e f t t o r i gh t t h e a ll o w e
d d
e
v
i a ti on fr o m µ ( a | s ) i n c r ea s e s .
H
o r i z on t a l g r ee n li n e s r e p r e s e n t t
h e r e po r t e d p e rf
o r m a n ce o f
B E A
R i n t h e
D 4 R
L b e n c h m a r k .
C o l o r-
c od
i ng f o ll o w s
F i gu r e 1 .