D ENSITY E STIMATION F OR C ONSERVATIVE Q-L EARNING
A
BSTRACT
Batch Reinforcement Learning algorithms aim at learning the best policy from a batch of data without interacting with the environment.
Within this setting, one difﬁculty is to correctly assess the value of state-action pairs far from the data set.
Indeed, the lack of information may provoke an overestimation of the value function, leading to non-desirable behaviours.
A compromise between enhancing the performance of the behaviour policy and staying close to it must be found.
To alleviate this issue, most existing approaches introduce a regularization term to favor state-action pairs from the data set.
In this paper, we reﬁne this idea by estimating the density of these state-action pairs to distinguish neighbourhoods.
The resulting regularization guides the policy toward meaningful unseen regions, improving the learning process.
We hence introduce Density Conservative Q-Learning (DCQL), a sound batch RL algorithm that carefully penalizes the value function based on the information collected in the state-action space.
The performance of our approach is outlined on many classical benchmark in batch RL.
I NTRODUCTION
Transposing recent successes of Reinforcement Learning (RL) from domains such as recommendation systems (Rojanavasu et al., 2005; Zheng et al., 2018), video games (Mnih et al., 2013), go (Silver et al., 2017), to real-world systems is not possible without facing many challenges (DulacArnold et al., 2021).
One of those being that in many real-world applications, direct access to the system is limited or even forbidden.
A learning controller may incorrectly assess the implications of its actions and damage the system.
Batch (or ofﬂine)
Reinforcement Learning (Lange et al., 2012;
Levine et al., 2020) provides a framework to address those RL problems when no interaction with the system is allowed.
In place, the learner is given a data set collected under a (possibly unknown) behavioural policy and has to derive the most efﬁcient policy out of this data set.
One of the main issue in this setting is known as the value-function over-estimation problem (Fujimoto et al., 2019; Levine et al., 2020; Kumar et al., 2019a; Wu et al., 2019).
When the data set covers only a small subset of the state-action space, the agent typically wrongly extrapolates the value function related to out-of-distribution (OOD) pairs far from the data set.
This incorrect value estimate is then used as a target in the learning process, leading to highly increasing estimates which may lead to a disastrous learned policy.
This problem has been extensively studied in the traditional online Reinforcement Learning setting (Sutton & Barto, 1998).
However in this case, this issue is quite naturally alleviated: when a value function becomes high whether because the region is promising or because of an erroneous over-optimism, it will drive the agent to visit the related state-action pair.
Therefore, the agent will have the chance to directly check the consequences of such pairs and its estimation can be corrected if needed.
Incorporating uncertainty with Ensemble Learning (van Hasselt, 2010; van Hasselt et al., 2016; Anschel et al., 2017) can also enhance learning.
In the ofﬂine case, since no inspection can be done to investigate the accuracy of the estimates, those methods cannot be used, and hence classical deep off-policy methods may dramatically fail (Fujimoto et al., 2019; Levine et al., 2020).
Additional approaches must thus be investigated to build robust and efﬁcient agents.
An important family of state-of-the-art algorithms addressing this problem focus on constraining the learned policy to stay close to the data set either by minimizing its distance to the behavioural policy (Siegel et al., 2020a; Wu et al., 2019; Kumar et al., 2019b) or by penalizing unseen state-action pairs (Luo et al., 2019; Kumar et al., 2020; Yu et al., 2021).
In particular, Kumar et al.
(2020) propose to use a lower bound on the value function as a proxy during the policy optimization process.
This bound should be tight when state-action pairs are contained in the data set and loosened otherwise.
Following this direction, they proposed Conservative QLearning (CQL) that introduces a penalization on the value functions associated to OOD actions.
Nevertheless, given the lack of information regarding the behavioural policy , their regularization remains abrupt and might be problematic in practice.
First, the resulting lower bound is loosened with the recommended distributions.
Second, their approach does not consider neighbourhoods of actions and could therefore over-penalize interesting regions of the action space.
This problem is particularly relevant when the data set has been gathered with a behavioural policy with medium performances: actions close to the data set distribution leading to better results are very likely to appear in the optimization process and they must not be too heavily penalized.
In this paper, we tackle these problems by introducing a novel penalization based on an estimation of the probability density function of the data set instead of the behavioural policy.
Besides being easier to learn than the behavioural policy, this density is able to provide information on which regions are near the data set and thus safe to learn from, and which are the ones far from it and prone to over-estimation errors.
We show how to integrate this penalization to reﬁne CQL, leading to our proposed algorithm namely Density Conservative Q-Learning (D-CQL).
On top of instigating those relevant information, we show that this new regularization leads to a more appropriate lower bound on the value function.
We ﬁnally investigate empirically the relevance of our approach.
P RELIMINARIES AND M OTIVATIONS 2. N OTATIONS AND P OLICY I TERATION
The agent-environment framework is modeled as a Markov Decision Process (MDP) ( S , A , r, P, γ ) deﬁned by a state space S , an action space A , a transition kernel P : S ×
A → ∆( S ) , a reward function r : S ×
A → [ R min , R max ] and a discount factor γ ∈ ]0 , 1[ .
A policy π : S → ∆( A ) is a decision rule mapping a state over a distribution of actions.
The value of a policy is measured through the value function V π ( s ) =
E P [ (cid:80) t =
γ t r ( s t , a t ) | s 0 = s, a t ∼ π ( ·| s t )] and its associated
Q -value function Q π ( s, a ) =
E P [ (cid:80) t =
γ t r ( s t , a t ) | s 0 =
s, a
= a, a t ∼ π ( ·| s t ) ∀ t ≥ 1] .
The goal is to ﬁnd the policy maximizing these value functions.
Let r , Q be matrices associated to all state-action pairs and P π be the transition matrix induced by the policy π : P π ( s, s (cid:48) )
=
E a ∼ π ( ·| s )
[ P ( s (cid:48) | s, a )] .
Value functions can be learned by iterating the Bellman operator deﬁned for the Q -function as B π Q = r + γP π Q where P π Q ( s, a ) =
E s (cid:48) ∼ P ( ·| s,a ) ,a (cid:48) ∼ π ( ·| s )
[ Q ( s (cid:48) , a (cid:48) )] .
This operator is a γ -contraction (Puterman, 1994), hence having a unique ﬁxed point Q π .
Many algorithms rely on the Policy Iteration scheme (Lillicrap et al., 2016;
Fujimoto et al., 2018; Haarnoja et al., 2018), where the agent alternates between Policy Evaluation (PE) with the computation of Q π and Policy Improvement (PI) by maximizing the learned Q -values:
[Equation]
(policy evaluation
[Equation]
This iterative process converges towards the optimal policy (Sutton & Barto, 1998; Santos & Rust, 2004).
Since no knowledge is assumed on the environment, these steps are commonly solved using samples from a data set D
= { ( s i , a i , r i )
Ni =1 } .
The expectation under P ( ·| s, a ) is now estimated using samples and leads to the empirical Bellman operator ˆ B .
Actions that belong to the data set distribution given a state s are denoted as in-distribution (ID) and the ones far from it are out-ofdistribution (OOD).
We slightly abuse notations and consider D is also the distribution on S ×A×S induced by the data set and propose:
[Equation]
(approximate policy
[Equation]
(approximate policy improvement)
The Q -values and the policy π are commonly estimated using Neural Networks that are trained with gradient optimization methods.
In the approximate policy evaluation step, an important aspect is the expectation of actions from π ( ·| s (cid:48) ) in the bootstrapped target.
This is where OOD actions may appear: extrapolation errors emerge then are back-propagated to eventually lead to highly overoptimistic estimates.
Note that there are no OOD states in the Bellman update as the empirical operator only depends on a seen state s (cid:48) .
They might appear in model-based algorithms (Yu et al., 2021) but are, for now, out of the scope of model-free algorithms.
In this work, we focus on the batch setting where the agent cannot interact with the environment.
A ﬁxed data set has been gathered with an unknown behavioural policy π β .
No knowledge is assumed on π β as the data set can come from different sources: optimal control, human or a mixture of them (Fu et al., 2020; G¨ulc¸ehre et al., 2020).
A major challenge is to ﬁnd a good trade-off between staying close to the data set to avoid extrapolation errors and taking some liberty to overcome the suboptimality of the behaviour policy.
2. C ONSERVATIVE Q -L EARNING
CQL addresses the problem of Q -value overestimation exacerbated in batch RL (Fujimoto et al., 2019; Levine et al., 2020), caused by the back-propagation of a possibly incorrect bootstrapped target in the policy evaluation step.
The error associated to the the target may be important when the consequences of a given state-action pair are unknown, and could lead to too optimistic estimates.
In batch RL, the data set often describes a small subset of the state-action space thus OOD actions are likely to appear.
Besides, the agent may never get the chance to visit the state-action pairs related to high (learned) Q -values and cannot correct its wrong estimations.
Kumar et al.
(2020) propose to penalize the Q -values associated to actions not described by the data set while keeping intact actions from the data set.
This greatly stabilizes learning: the Q -estimates no longer take drastically high values, and it implicitly drives the agent to favor state-action pairs described in the data set.
This penalization translates into minimizing the Q -values on an arbitrary distribution µ ( e.g. uniform) and maximizing them on π β formally expressed as:
[Equation]
Assuming no sampling errors and that supp µ ⊂ supp π β , this update provides a lower-bound on the expected Q -values on the distribution µ : E a ∼ µ (cid:2) Q CQL ( s, a ) (cid:3) ≤ E a ∼ µ [ Q π ( s, a )] for all s ∈ S with α > 0 .
When µ is the current policy π , the lower bound is in the value function, that is
V CQL ( s ) ≤ V π ( s ) .
This property may be desirable on applications where we must check if the current value function is above a certain threshold such as Constrained Policy Optimization (Achiam et al., 2017) and Conservative Exploration (Garcelon et al., 2020).
However, for the common setting where µ is a uniform distribution over the action space, the lower bound is loosened to a uniform expectation, and the inequality in the value function no longer holds.
Another point of attention is the maximization term relying on the knowledge of the behavioural policy in equation 1.
In the general setting, it is out of reach as the data set may be gathered in various ways.
Kumar et al.
(2020) propose to use the empirical data set distribution ˆ π β ( a | s ) =
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Table]
, a sum of Dirac over the data set actions.
Not only does this further breaks the assumption supp µ ⊂ supp ˆ π β (as in practice supp π β (cid:54)⊂ supp ˆ π β ), it also might be problematic on the penalization itself: the distance between the selected actions and the data set distribution is no longer considered explicitly.
Actions not appearing in the data set will be equally penalized no matter their distance with D .
Thus, the regularization drives the agent to stay close to the behavioural policy.
We believe having a smoother regularization that considers which neighbourhoods of the data set are relevant is important to get the best agent as possible.
D ENSITY C ONSERVATIVE Q -L EARNING
In this section, we introduce Density-CQL (D-CQL), our algorithm circumventing these issues.
3. A NEW WEIGHTING SCHEME ON THE O UT -O F -D ISTRIBUTION ACTIONS
The penalization of CQL pushes the learned policy of the agent to stay very close to the behavioural policy as it penalizes any action that does not belong to the data set and thereby prevents the agent from investigating on potential relevant areas.
This problem is of high interest, for example, when the data set is gathered with a distinctly sub-optimal policy but more efﬁcient actions exist in the neighborhood of the sub-optimal ones.
The chances of ﬁnding a better policy would be reduced as any action that does not belong to the data set will be equally penalized.
Nevertheless, in view of the ﬁxed nature of the batch setting and the high sensibility of the Bellman update, we should be cautious with respect to which regions the agent should focus on.
Bearing these complications in mind, we propose to reﬁne the original regularization by appropriately weighting the actions according to how uncertain their consequences are given D .
To do so, we quantify the distance between the data set and the (sampled) penalized actions to soften the regularization.
We propose a regularization based on the joined density of state-action pairs in the data set, denoted ρ β : S ×
A → R + .
This density describes how likely a state-action pair belongs to the data set and, by extension, how uncertain its associated Q -function is.
Various techniques can be used to learn such a distribution.
In fact, the literature regarding density estimation is quite abundant (Bishop, 2007;
Uria et al., 2013; Kobyzev et al., 2020), even when dealing with high dimensional inputs such as images (Winkler et al., 2019; Papamakarios et al., 2019).
Our method can also beneﬁt from future developments in this area.
Using this density, we build a weighting scheme ζ ν ( a | s ) 1 based on how uncertain the consequences of an action a , given s , are w.r.t.
D .
It is integrated as follows:
[Equation]
The weighting scheme ζ ν ( a | s ) is able to control how much the Q -value of an action a chosen under µ ( ·| s ) should be penalized and therefore satisﬁes our goal.
Note that CQL corresponds to a particular version of D-CQL where ζ ν ( a | s )
=
1 − δ a D ( a ) , where δ a D denotes the Dirac over a D (the actions associated to state s in the data set).
The new regularization leads to a point-wise lower bound on the Q -values that is tight on in-distribution (ID) actions with a well-chosen ζ ν .
It thereby follows the classic Q -Learning procedure for ID actions, while only penalizing the Q -values associated to
OOD actions depending on their distance with respect to the data set distribution, controlled by ζ ν .
Theorem 1 For any µ satisfying supp µ ⊂ supp π β and assuming the absence of sampling errors, the Q -values obtained by iterating equation 2 are
[Equation]
and α > 0 leads to a point-wise lower bound on the true Q -values that becomes tight whenever ζ ν ( a | s ) = 0 .
The proof of this theorem follows closely the one in (Kumar et al., 2020, Theorem 3.1), but we still include it in Appendix A for completeness.
3. C HOICE OF THE WEIGHTING SCHEME
The weighting scheme should capture the uncertainty of an action a given a state s .
With a D the action present in the data set related to the state s , we propose the following indicator:
[Equation]
Notice that the weighting scheme contains a few tricks on top of the density information.
First, because the density may take high values in some areas, especially when dealing with narrow distributions, we normalize the weighting scheme by ρ β ( s, a D ) .
The state-action pair ( s, a D ) belongs to the data set, so its associated density is able to scale the weighting scheme well.
It also frees its dependency on the state s .
Second, considering the propensity of the Q -values to take extremely high values rapidly, the weighting scheme must be below 1.
Last but not least, making a too harsh distinction between ID and OOD actions may result in an unstable learning, hence the introduction of the hyper-parameter ν .
Hyper-parameter ν
We need ν = 0 for the lower bound of Theorem 1 to be tight.
Indeed, when ν = 0 , the Q -values of ID actions are not penalized as ζ ν = 0 .
It follows the intuition of penalizing only Q -values associated to OOD actions.
However, since OOD actions do not appear in the Bellman loss term of Equation (2), the minimum of their Q -values is unbounded which eventually affects the performances of the algorithm.
By setting ν > 0 , the regularization term will also include ID actions and this minimization will be counterbalanced by the Bellman loss that tends to maximize their Qvalues.
In other words, the lower ν , the greater the difference between OOD and ID Q -values to balance the objective.
In practice, it is common to have knowledge on the quality of the behavioural policy and on the system, which can help setting the right ν .
For instance, if we know that an expert has gathered the data set, we will want to stay close to it and set ν close to 0 .
Otherwise, if we know that the exploration has not been good and that there is no risk of harming the system, we can choose a bigger ν to allow the controller to consider uncertain actions.
In Appendix C.2, we conduct a detailed analysis of the effects of the OOD penalization with varying ν .
P RACTICAL ALGORITHM
Following the ideas expressed in (Kumar et al., 2020, Appendix A), we add an entropic regularization to Equation (2) that leads to the following update for Density Conservative Q-Learning (DCQL):
[Equation]
The log-sum-exp operator can be seen as a soft differential maximum operator.
Equation (5) only penalizes the highest Q -values weighted by ζ ν .
This expression exacerbate D-CQL’s behaviour described in Section 3.2 when ν is close to zero.
Same conclusion holds, the lower ν , the greater the difference between OOD and ID Q -values to balance the objective.
In practice, this operator is computed using Importance Sampling (IS) using M action samples from both the current policy and the uniform distribution U ( A ) .
We describe D-CQL in Algorithm 1.
Algorithm 1 D-CQL
Learn ρ β with behavioural cloning
Initialize Q θ and π Φ for k ∈ (1 , . . . )
do
Sample a batch B from D
For each state s ∈ B sample M actions from U ( A ) , and M actions from π Φ ( ·| s )
Update Q θ with gradient descent on Equation 5 using Importance Sampling
Update π Φ with gradient ascent on Equation policy improvement end for
R ELATED WORKS
Learning a policy from a ﬁxed data set has been of high interest in Reinforcement Learning.
A ﬁrst way to address this problem is using Behavioural Cloning.
When the data set is complete and comes from an expert policy, it may achieve great successes as shown in special cases of autonomous driving (Pomerleau, 1988) and ﬂying (Sammut et al., 1992).
However, when the agent ﬁnds itself in a situation not described by the data set, it may choose catastrophic actions and its performance can quickly degrade (Codevilla et al., 2019).
Imitation Learning (Hussein et al., 2017; Ho & Ermon, 2016) aims to counter this problematic but still requires an expert policy.
In real-world applications, the data set may come from sub-optimal policies and instead of reproducing the behavioural policy, the goal of Batch RL is to extract a better one.
Recently, many algorithms have been introduced, all relying on making sure the learned policy stays close to the behavioural one, avoiding the presence of Out-Of-Distribution (OOD) actions (Levine et al., 2020).
It can take several forms.
Many recent algorithms cope with the distributional shift by directly avoiding the selection of OOD actions by the policy, preventing the agent to learn from highly extrapolated values.
BCQ (Fujimoto et al., 2019) explicitly parametrize the policy to stay within a parametric ball around the behavioural policy.
Similarly, SPIBB (Laroche et al., 2019) restricts the support of the learned policy to remain in the support of the behavioural policy.
Other methods - such as BRAC (Kumar et al., 2019a) and
BEAR (Wu et al., 2019) - modify the RL objective with a measure of closeness between learned and behavioural policy.
AWR (Peng et al., 2019) or ABM (Siegel et al., 2020b) build a Trust Region around the behavioural policy.
Rather than a constraint between the policies, AlgaeDICE (Nachum et al., 2019) and OptiDICE (Lee et al., 2021) focused on the state-action stationary distributions which provided competitive results.
While successful on a variety of tasks, they do not explicitly counter the over-estimation problem that may prevent learning (Kostrikov et al., 2021).
Another line of work is to enhance the robustness of the agent while leaving the objective unchanged.
REM (Agarwal et al., 2020) chooses to not modify the objectives, but uses Ensemble Learning to build a mixture of targets stabilizing the updates.
In practice, it rarely competes with the above methods.
More related to our work is (Dadashi et al., 2021;
Rezaeifar et al., 2021) where a clever pessimistic bonus is introduced in the rewards acting as anti-exploration.
It represents the distance of the selected action with the one who would be selected by the behavioural policy.
The authors showed it acted similarly than a KL-penalty between the learned and behavioural policy, linking their work with Wu et al.
and
Kumar et al.
(2019a).
However, playing with the rewards might lead to a completely modiﬁed goal (Ng et al., 1999) and ﬁnding an appropriate reward modiﬁcation might be complicated (Harutyunyan et al., 2015).
We argue that modifying the Q -values lead to a better policy control as the whole objective is considered.
In this line of search, Kostrikov et al.
(2021) augmented the Q -values with the entropy of the behavioural policy and constrained them with as Fisher divergence term.
In contrast, rather than modifying the RL objective with an explicit penalty on the data set distance, we use the density as an uncertainty surrogate in the value function estimation.
Note our data set density estimation can be combined with most presented methods.
Especially, it could replace the ”anti-exploration” bonus or be introduced to improve the Behavioural Cloning step in (Wu et al., 2019) required to build the measures of closeness between policies.
E XPERIMENTS
In this section, we evaluate D-CQL on various settings.
Our algorithm was designed to take the most of data sets collected with a distinctly sub-optimal policy, where the trade-off between staying close to the behavioural policy and discovering better actions is culminating.
An ablation study on ν and on the density estimation technique used can be found in Appendix C.2.
6. D ATA SETS
Our algorithm is ﬁrst tested on four different data sets gathered on two classic control environments with continuous actions: Pendulum and MountainCarContinuous (Moore, 1990).
For each environment, two data sets were gathered with a bad and medium policy, where the policies have been created by running a Soft-Actor-Critic agent (Haarnoja et al., 2018) and early-stopping the training.
Pendulum is an inverted pendulum swing up problem, where the reward is a smooth function contained in [ − 16 , 0] .
The goal is to is to swing it up so it stays upright for 200 time-steps.
The expert policy induces an average return of − 150 , and the bad and medium policies respectively induce an average return of − 1000 and − 700 .
The data set possesses 20 thousand transitions.
MountainCarContinuous is also a classic environment in Reinforcement Learning.
A car must drive up a hill, but must ﬁrst drive back and forth to be able to climb it.
Unlike Pendulum, the reward is sporadic: it takes the value of +100 when the goal is reached, and − energy else.
The bad data set is composed of 1 success out of 100 trajectories, and the medium data set 4 successes out of 50 .
We have found both CQL and D-CQL easily solve this environment when the number of successes is higher.
The expert policy induces an average return of 95 .
Finally, we test our agents on the medium data sets from D4RL which is closer to real-world performances.
To further match a realistic setting we only consider a reduce proportion, X % , of the data sets.
Indeed, huge data collection might not be that realistic in practice ( e.g. low acquisition frequency) and learning on such huge data sets can be problematic w.r.t.
computation time.
6. E XPERIMENTAL PROTOCOL
Learning the density D-CQL relies on an accurate density estimation step.
Though any technique can be used, we propose to use a highly expressive method: Normalizing Flows (Rezende & Mohamed, 2015; Kobyzev et al., 2020).
This choice was fueled with their successes on high dimensional distributions (Papamakarios et al., 2019) and their elegance.
Indeed, they are known to generalize and scale well from complex distributions (Winkler et al., 2019) - including images (Kobyzev et al., 2020) - while keeping a tractable learning.
We optimize the model hyper-parameters with a grid search where each data set was split into a train and a test set.
The density estimator was then trained on the whole data set using the best selected hyper-parameters.
A comparison of different density estimation algorithms used for D-CQL can be found in Appendix C.2.
Learning the agents We compare our agents with CQL 2 , a state-of-the-art method that also penalizes OOD actions in the approximate policy evaluation step.
Experiments on the classic control data sets have been done with a grid search on all the hyper-parameters, including experiments with
CQL.
Considering the signiﬁcant length of the experiments on D4RL, the hyper-parameter search was done on 1 million (M) gradient steps.
Additional details are reported in Appendix C.1.
Metric We compare the agents’ performances using the normalized average return (NAR):
[Equation]
However, it is common knowledge that these methods are unstable and a decrease in performance during learning (Aviral Kumar, 2021; Kumar et al., 2021).
We thus propose to evaluate the best agents after 10 M gradient steps.
Thus, once the best hyper-parameters selected on 1 M gradient steps, we further continue the learning for 9 M steps for the best agent.
Regarding classic control data sets, we observed that 100 k updates were sufﬁcient to draw a relevant comparison.
We report the overall best normalized average return , i.e. the best normalized average return considering all epochs, and the best ﬁnal normalized average return in Table 1 and Table 2.
6. R ESULTS
Results on the classic control data sets are shown in Table 1.
We observe our reﬁned regularization improve the agent’s performance on both the overall and ﬁnal metrics on 3 out of 4 tasks.
The agent was able to take the most out of the neighbourhood of the data set and extract more efﬁcient policies.
Notice the greater variance on the MountainCarContinuous environment, explained by the sparsity of its reward function.
In any case, D-CQL was able to better take advantage of the information provided by the data set.
Regarding the more complex D4RL’s data sets, the results are reported in Table 2.
Our agent was able to beat CQL respectively 5 and 6 times out of 9 w.r.t.
the overall and ﬁnal metric.
On average, it D-CQL outperforms CQL by 3% and 7 . 5% on the overall and ﬁnal metric.
Overall, both Table 1 and Table 2 illustrate the fact that D-CQL outperforms or is comparable with
CQL.
Detailed plots can be found in Appendix C.3.
Furthermore note the decrease of performance for both methods w.r.t.
gradient steps emphasized by the differences between the best and ﬁnal
[Table]
[Table]
[Table]
performances.
While out of the scope of this paper
[Table]
it remains an open question in off-policy algorithms (Aviral Kumar, 2021; Kumar et al., 2021).
Last but not least, our experiments shown that even though the methods might prevent the over-estimation of the Q -values, agents’ performances have still non-negligible variances.
C ONCLUSION
In this paper we propose to introduce the use of the data set density ρ β which provides insights to guide the agent towards meaningful areas, inducing better performances in practice.
This work focused on model-free agents that do not consider OOD states.
However, the density ρ β also yields information on OOD states which can be integrated on model-based algorithms.
In addition, our experiments highlighted two other points of interests: the decreasing effect and the relatively high variance of the batch agents.
Despite some previous works aiming to cope with these problems, a complete understanding of these phenomenons is lacking which is crucial before batch
RL can be applied on real-world systems.
A CKNOWLEDGMENTS
We thank Christophe Prieur and Bogdan Robu from GIPSA Lab of Grenoble-Alpes University for the useful discussions we had regarding this work.
We also thank the anonymous reviewers for their feedback which improved the paper
A L OWER BOUND DETAILS
In this section, we re-state our main theorem and prove it.
An additional insight is provided to handle the sampling errors.
Theorem 2 For any µ satisfying supp µ ⊂ supp π β and assuming there are no sampling errors, then the Q -values obtained by iterating equation 2 are
[Equation]
α > 0 leads to a point-wise lower bound on the true Q -values that becomes tight whenever ζ ( a | s ) = 0 .
Proof
This proof is done assuming the true Bellman operator B has been used instead of ˆ B .
Setting the integrande of the derivative of 2 leads to
[Equation]
The interest of ζ is clearly exposed in this equation: the learned Q -values will be highly conservative when associated to OOD actions.
On the other hand, when ζ ( a | s ) ≈ 0 , the update leads to an application of the classic Bellman operator and drive ID Q -values to stay close to their real associated Q -value.
Let µ , π β and ζ the matrices associated to { µ ( a | s ) } , { π β ( a | s ) } and { ζ ( a | s ) } for all s ∈ D and a ∈ supp π β ( ·| s ) .
The operator associated to this update: T ζ : Q → B π Q k − α µ ζ π β deﬁned for any function Q on deﬁned on states belonging to D and actions belonging to supp π β ( ·| s ) is a γ -contraction.
Its ﬁxed point is
[Equation]
Remark 1 The sampling error can be handled using concentration inequalities and would lead to the following lower bound, that holds with probability 1 − δ :
[Equation]
We refer to Appendix C of Kumar et al.
(2020) for additional details.
B N ORMALIZING F LOWS
We now describe the used architecture to estimate ρ β based on Normalizing Flows.
They are based on the change of variable for a diffeomorphism f : X → Y and two densities deﬁned on X and Y
[Equation]
We then learn a diffeomorphism f that maps the unknown distribution on X to a known distribution on Y , for example a multivariate normal distribution so that the factor ρ Y f ( X ) is efﬁciently calculable.
In our case, let x =
( s, a ) and y its associated sample belonging to the Y distribution and let D their dimension.
Let’s assume there is only one differential mapping y =
f ( x ) , and let d < D .
We follow the architecture from RealNVP (Dinh et al., 2017) where f maps x to y as follows:
[Equation]
where s and t are functions deﬁned in R d
→ R m − d and (cid:12) is the Hadamard product.
This transformation results in a simple formula for the Jacobian determinant: | det (Jac f ) ( x ) | = exp (cid:16)(cid:80)
j s ( x 1: d ) j (cid:17) .
This mapping is then repeated N times where each input is randomly permuted before applying equation 11 to ensure each dimension is rightfully processed.
In all datasets, s and t are parametrized by a Neural Network with 1 layer.
The number of mappings
N as well as the number of units in each layer were selected thanks to grid search on respectively ( 5 , 10 , 20 ) and ( 64 , 128 , 256 ).
Y was chosen as a Multivariate Normal distribution N (0 , I ) .
C D-CQL EXPERIMENTS
For reproducible purposes, we now present additional details regarding the experiments.
CQL’s code was extracted from https://github.com/young-geng/CQL .
On all plots, the abscissa represents the number of epochs and the ordinate the cumulative average return over 5 seeds.
For the experiments on the classic control data sets, one epoch represents 100 gradient updates andupdates for the ones on the D4RL data sets.
C. H YPER - PARAMETERS
We ﬁrst present all the hyper-parameters used in the experiments.
For the experiments on the data sets of D4RL, most of the hyper-parameters - learning rate, size of the networks, optimizer, ...- were unchanged to have a proper comparison with CQL.
For the experiments on the classic control environments, the size of the hidden layers of the neural networks were minimized to 64 with the same number of layers.
The main hyper-parameters of the algorithms are α and ν , on which we performed a grid search for each experiment.
This grid search was done on ( 1 , 5 , 10 ) for α and on ( 0 . 0 , 0 . 1 , 0 .
for ν .
As mentioned in Section 6, the grid search for the classic control environments (Pendulum and MountainCarContinuous) was done on the complete experiments, that is on the 100 K gradient updates.
Regarding the experiments on the D4RL MuJoCo data sets, the grid search was performed on only 1 M gradient updates.
The best hyper-parameters were then chosen and we continued this experiment on 10 M updates.
C. A BLATION STUDY Study on ν
First, we present an ablation study on the hyper-parameter ν on the classic control environments.
For all environments, we set α to 5 and launched experiments with a varying ν .
Considering the length of the experiments on the D4RL data sets, we performed the ablation study on hopper-medium ( 1% and 5% ) and on walker2d-medium ( 1% and 5% ).
[Figure]
[Figure]
The impact on the hyper-parameter ν varies according to the environment.
Nevertheless, we observe the intuitive choice ν = 0 often never produce the best results which were generated with a higher ν .
We attribute this effect to the unbounded minimization of the Q-values towards minus inﬁnity discussed in Section 3.2.
This can be notably seen on pendulum-medium and hopper-medium- 1% , where the best ν is 0 .
This high number has to be put in perspective as ζ ν has been introduced as a multiplicative factor and is inside the soft differential maximum operator, both of which increase the importance of our weighting factor.
We also observe that having a distinction between ID and OOD actions in the penalization may not be useful in some environments.
For instance, data sets from the walker2d environment do not need to make such distinction as illustrated by the results of ν = 1 .
Forcing the Q -values to have a reasonable scale during learning seems to be enough for the Bellman update to work on this environment.
Study on the density estimation technique used We continue the ablation study by testing the robustness on the density estimation technique used.
We compare the Normalizing Flows RealNVP with two other methods: the auto-regressive algorithm RNADE (Uria et al., 2013) and a Mixture of
Gaussians (MOG).
To make fair comparisons, RNADE and MOG densities have also been created with a grid search on their hyper-parameters, that is on the number of their components, the number and size of the layers of the Neural Network(s) used.
We performed the study on data sets from the Pendulum environment, where we can see our algorithm is robust with respect to the density estimation technique used.
It remains to be conﬁrmed on higher dimensional environments.
[Figure]
C. C OMPARISON WITH CQL
We ﬁnally provide the plots for Table 1 and Table 2.
The hyper-parameters of the experiments are the one chosen from the ﬁnal normalized average return metric.
[Figure] [Figure]