E XPLOITING M INIMUM -V ARIANCE P OLICY E VALUATION FOR P OLICY O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Off-policy methods are the basis of a large number of effective Policy Optimization (PO) algorithms.
In this setting, Importance Sampling (IS) is typically employed as a what-if analysis tool, with the goal of estimating the performance of a target policy, given samples collected with a different behavioral policy.
However, in Monte Carlo simulation, IS represents a variance minimization approach.
In this ﬁeld, a suitable behavioral distribution is employed for sampling, allowing diminishing the variance of the estimator below the one achievable when sampling from the target distribution.
In this paper, we analyze IS in these two guises, showing the connections between the two objectives.
We illustrate that variance minimization can be used as a performance improvement tool, with the advantage, compared with direct off-policy learning, of implicitly enforcing a trust region.
We make use of these theoretical ﬁndings to build a PO algorithm, Policy Optimization via Optimal Policy Evaluation (PO 2 PE), that employs variance minimization as an inner loop.
Finally, we present empirical evaluations on continuous
RL benchmarks, with a particular focus on the robustness to small batch sizes.
I NTRODUCTION
Policy Optimization methods (PO, Deisenroth et al., 2013) have been widely exploited in Reinforcement Learning (RL, Sutton & Barto, 2018) with successful results in addressing, to name a few, continuous-control (e.g., Peters & Schaal, 2008; Lillicrap et al., 2016), robot manipulation (e.g., Gu et al., 2017; Chatzilygeroudis et al., 2020), and locomotion (e.g., Kohl & Stone, 2004; Duan et al., 2016).
Most of these algorithms employ the notion of trust region (Conn et al., 2000), introduced ante litteram in the RL literature by the safe RL approaches (Kakade & Langford, 2002; Pirotta et al., 2013), giving rise to a surge of effective algorithms, having TRPO (Schulman et al., 2015) as the progenitor.
The core of any RL algorithm, being value-based or policy-based, lies in the ability to employ the samples collected with the current (or behavioral ) policy to evaluate the performance of a candidate (or target ) policy (Sutton & Barto, 2018).
The skeleton rationale behind the usage of a trust region is to control the set of candidate policies whose performance can be accurately evaluated.
Intuition suggests that if the candidate policy is “sufﬁciently close” to the current one, this off-policy evaluation problem (Precup et al., 2000) will provide a good estimate for the performance of the candidate policy.
Formally, this idea has been studied in the ﬁeld of Importance Sampling (IS,
Owen, 2013) and the phenomenon is particularly evident looking at the IS estimator variance, which grows exponentially with the R´enyi divergence (R´enyi, 1961) between the behavioral and the target policy (Metelli et al., 2018; 2020).
In this off-policy learning (Off-PL) setting, IS is employed as a what-if analysis tool (Owen, 2013) and its role is passive , as samples have been already collected with the current behavioral policy.
In this sense, the trust region is an a-posteriori remedy for the limitations of off-policy evaluation, for controlling the uncertainty injected by the IS procedure.
However, IS originated in the Monte Carlo simulation community (Hesterberg, 1988; Hammersley, 2013) as an active tool for variance minimization (Off-VM).
While in Off-PL, the behavioral policy is ﬁxed and we look for the best target policy, whose performance we aim to estimate, here the roles are reversed.
Indeed, in Off-VM, the target policy is ﬁxed and we search for the behavioral policy (from which to collect samples) that yields an IS estimate with the minimum possible variance (Hammersley, 2013; Kahn & Marshall, 1953).
It might seem surprising, at ﬁrst, that sampling from a policy, other than the target one, can lead to an estimator with less variance (even zero in some cases) w.r.t.
the on-policy estimate.
In this role, IS has been previously employed in RL, mainly to address rare events (Frank et al., 2008; Ciosek & Whiteson, 2017) which naturally lead to high-variance estimates, when tackled on-policy.
The idea of explicitly using IS as a variance reduction technique, with the goal of ﬁnding an optimal behavioral policy, was proposed by (Hanna et al., 2017) for evaluation and subsequently combined with policy gradient learning (Hanna & Stone, 2018; Hanna et al., 2019).
However, in these works, the variance minimization (Off-VM) process and the off-policy learning (Off-PL) problem are treated separately.
The goal of this paper is to investigate the relation between variance minimization (Off-VM) and off-policy learning (Off-PL).
The core question we address can be summarized as: “ Can Off-VM be employed as a tool for Off-PL, overcoming the need for an explicit trust region? ”
Intuitively, given a target policy, when the reward function is positive, one way to reduce the variance of the IS estimator is to assign larger probability to the trajectories that have a large impact on the mean, i.e., those with high returns.
This provides a ﬁrst hint about the connection between the minimum-variance sampling policy and the performance improvement, i.e., between Off-VM and Off-PL.
Furthermore, it suggests that we could repeatedly apply the process of identifying the minimum-variance policy as a tool for policy improvement.
The interesting aspect of such an approach is that, by minimizing the variance, it implicitly controls the divergence between two consecutive policies.
In other words, it allows enforcing a trust region, without the need for divergence constraints or penalizations.
Outline of the Contributions In this paper, we provide theoretical, algorithmic, and experimental contributions.
After having introduced the background (Section 2), we present the problem of ﬁnding the minimum-variance behavioral distribution (Section 3).
Then, we study the properties of the Off-VM problem in two settings: unconstrained (Section 4) and constrained (Section 5).
First, we assume that there are no restrictions for choosing the behavioral distribution.
We show that the minimum-variance behavioral distribution, besides leading to the zero-variance estimator (Kahn &
Marshall, 1953), is guaranteed to yield a performance improvement, requiring the non-negativity of the reward only.
Furthermore, we prove that this approach allows controlling the divergence between two consecutive distributions, thus enforcing an implicit trust region.
Although this provides a valuable starting point, the minimum-variance distribution might be unrealizable given the environment transition model, i.e., there might be no policy inducing it.
For this reason, we move to the scenario in which the distributions are constrained in a suitable space.
In this setting, the zerovariance estimator could not be achievable.
Nevertheless, we prove that such a procedure can lead to a performance improvement and preserves the trust region enforcement.
Based on these theoretical results, we propose Policy Optimization via Optimal Policy Evaluation (PO 2 PE), a novel PO algorithm, that we particularize for parametric policy spaces (Section 6).
Finally, we provide numerical simulations on continuous-control benchmarks, in comparison with POIS (Metelli et al., 2018) and
TRPO (Schulman et al., 2015), with a particular focus on the robustness of PO 2 PE to small batch sizes (Section 7).
The proof of the results presented in the main paper are reported in Appendix A.
P RELIMINARIES
In this section, we provide the necessary background that will be employed in the paper.
Mathematical Notation Let X be a set, and let F X be a σ -algebra over X .
We denote with P p
X q the space of probability measures over p X , F X q .
Let P P P p X q , whenever needed, we assume that
P admits a density function p .
For a subset Y Ď R , we denote with B p X , Y q the space of measurable functions f : X Ñ Y .
Let P,Q P P p
X q be two probability measures such that P !
Q , i.e., P is absolutely continuous w.r.t.
Q , for every α Pr 0 , 8s , we deﬁne the α -R´enyi divergence as (R´enyi, 1961):
[Equation]
´ 1 log ş
X p p x q α q p x q 1 ´ α d x .
In the limit of α Ñ 1 , the R´enyi divergence reduces to the
KL-divergence D KL p P } Q q , while for α Ñ8 , it corresponds to esssup x „ Q t p p x q{ q p x qu .
Importance Sampling Let P,Q P P p X q with P !
Q and let f P B p X , R q .
Importance Sampling (IS, Owen, 2013) allows estimating the expectation of f under a target distribution P , i.e., E x „ P r
f p x qs having samples t x i u i Pr n s collected with a behavioral distribution Q : p µ P { Q “ 1 n ř i Pr n s p
p x i q q p x i q f p x i q . The IS estimator is unbiased (Owen, 2013), i.e., E x i „ Q r p µ P { Q s“ E x „ P r f p x qs , but it might suffer from large variance, due to the heavy-tailed behavior (Metelli et al., 2018).
The properties of p µ P { Q and its transformations have been extensively studied in the literature (e.g., Ionides, 2008; Thomas et al., 2015; Papini et al., 2019; Kuzborskij et al., 2021; Metelli et al., 2020).
Policy Optimization A Markov Decision Process (MDP, Puterman, 1994) is a 6-tuple M “ p S , A ,P,R,γ,D 0 q , where S is the state space, A is the action space, P : S
ˆ A Ñ P p S q is the transition model, R :
S
ˆ A Ñr 0 ,R max s is the reward function, γ Pr 0 , 1 s is the discount factor, and
D 0 P P p S q is the initial state distribution.
The agent’s behavior is modeled by a parametric policy π θ : S Ñ P p
A q belonging to a parametric policy space Π Θ “t π θ : θ P Θ Ď R d u .
The interaction between an agent and the MDP generates a trajectory τ “p s 0 ,a 0 ,s 1 ,a 1 ,...,s H ´ 1 ,a H ´ 1 ,s H q where H P N is the trajectory length and s 0 „ D 0 , a t „ π θ p¨| s t q , s t ` 1 „ P p¨| s t ,a t q for all t P t 0 ,...,H ´ 1 u .
Given a trajectory τ , the return is the discounted sum of the rewards R p τ q“ ř H ´t “ 0 γ t R p s t ,a t q .
For a policy π θ P Π Θ , we denote with p p¨| θ
q the induced trajectory distribution: p p τ | θ q“ D 0 p s 0 q ś H ´ 1 t “ 0 π θ p
a t | s t q P p s t ` 1 | s t ,a t q .
In the action-based (AB) setting, an agent aims at ﬁnding a parametrization fulﬁlling: θ ˚ P argmax θ P Θ t J p θ qu , where J p θ q“ E τ „ p p¨| θ q r R p τ qs is the expected return . π θ must be stochastic to ensure exploration. Instead, in the parameter-based (PB) setting, we consider a hyperpolicy ν ρ , belonging to a parametric hyperpolicy space N P “t ν ρ : ρ P P Ď R l u , from which we sample the parameters θ of the policy. In this case, the policy π θ can be deterministic since exploration is managed at the hyperpolicy level and the agent goal becomes to learn a hyperpolicy parametrization maximizing the expected return: ρ ˚ P argmax ρ P P t
J p ρ qu , where J p ρ q“ E θ „ ν ρ r J p θ qs .
In the paper, we keep the presentation as general as possible, introducing the results for arbitrary distributions.
Then, we will particularize for the parametric PO setting.
M INIMUM –V ARIANCE B EHAVIORAL D ISTRIBUTION
In this section, we revise Off-VM, i.e., the problem of ﬁnding a behavioral distribution Q P P p X q that induces an IS estimate p µ P { Q with minimum variance, knowing the (ﬁxed) target distribution
P P P p X q and function f P B p X , r 0 , 8qq .
Furthermore, we do not enforce any restriction on the possible forms of the behavioral distribution Q P P p X q .
The problem and the corresponding wellknown minimum-variance behavioral distribution Q ˚ are stated in the following (Kahn, 1950):
[Equation]
We observe that the IS estimator p µ P { Q ˚ is non-stochastic, equal to the quantity we aim to estimate, i.e., p µ P { Q ˚ “ E x „ P r f p x qs .
This suggests that the construction of Q ˚ is infeasible as it requires knowledge of E x „
P r f p x qs .
Since Q ˚ generates a non-stochastic estimator, it not only leads to zero-variance but, clearly, simultaneously minimizes the absolute central moments of any order.
A second, and most remarkable property, is that Q ˚ is a performance improvement w.r.t.
P , i.e., the expectation of f under Q ˚ is larger than the expectation of f under the target P
(Owen, 2013):
[Equation]
It is worth noting that the magnitude of the improvement is directly related to the reduction in variance Var x
„ P r f p x qs .
Equation (2) suggests an appealing connection between the problem of ﬁnding the minimum-variance behavioral distribution (Off-VM) and the problem of ﬁnding a target distribution that maximizes the expectation E x „
P r f p x qs (Off-PL).
In other words, we could employ
Off-VM as a performance improvement tool, by repeatedly solving the problem in Equation (1).
In the following two sections, we will delve into the properties of the repeated construction of the minimum-variance distribution as a performance improvement tool under two assumptions: (i) there are no restrictions in the choice of the behavioral distribution Q P P p X q (Section 4); (ii) the behavioral distribution must be chosen within a subset Q P Q Ď P p X q (Section 5).
In both cases, we will address the following three questions: (Q1) Does this procedure always generate a distribution that is a performance improvement ?
(Q2)
Does this procedure converge to a (global or local) maximum of f ?
(Q3)
Can we quantify the divergence between two consecutive distributions, i.e., does this procedure enforce a trust region ?
Before proceeding, let us map this general setting to PO. In the action-based (AB) setting, x is the trajectory τ , P and Q are trajectory distributions p p τ | θ q induced by policies π θ . Instead, in the parameter-based (PB) setting, x is the pair p θ ,τ q , P and Q are joint distributions ν ρ p θ q p p τ | θ q induced by hyperpolicies ν ρ .
In both cases, function f is the trajectory return R p τ q .
U NCONSTRAINED P ROBABILITY D ISTRIBUTION S PACE
In Section 3, we have seen that Q ˚ is a performance improvement w.r.t.
P .
We now generalize this construction, by composing function f with a non-negative monotonic strictly-increasing function h : r 0 , 8qÑr 0 , 8q .
The rationale behind this choice is that if h is strictly-increasing, then h
˝ f has the same maxima as f .
We start deﬁning the operator
I h
˝ f : P p X [Equation]
P p X
[Equation]
Thus, I h
˝ f takes as input a target distribution P P P p X q , a function h
˝ f P B p X , r 0 , 8qq , and outputs the minimum-variance behavioral distribution for the IS estimation of E x „ P r h p
f p x qqs , i.e.,
Q ˚ “ I h
˝ f r P s .
Intuitively, looking at Equation (3), by iterating the application of I h
˝
f , we will obtain distributions tending to assign larger probability mass to points x P X with high values of f p x q .
Concerning (Q1) , the following result, due to Ghosh et al.
(2020), generalizes Equation (2) showing that whenever h is increasing, we can prove that I h
˝ f r P s is a performance improvement w.r.t.
P .
Proposition 4.1 (Proposition 9 of Ghosh et al.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic increasing.
Then, I h
˝ f r P s is a performance improvement w.r.t.
P :
[Equation]
r [Equation] x „ P r h p f p x qqs ě 0 . Note that, since h is a monotonic increasing function, we have that Cov x „ P r h p f p x qq ,f p x qsě 0 (Cuadras, 2002).
The following sections tackle questions (Q2) and (Q3) .
4. C ONVERGENCE P ROPERTIES
We now address question (Q2) , analyzing the effect of repeatedly applying operator
I h
˝ f .
More formally, let us consider an initial distribution P P P p X q , and suppose to iterate the application of the operator I h ˝ f , generating the sequence of distributions p Q k q k P N , where Q 0 “ P and for every k P N ě
we have Q k “ I h
˝ f r Q k ´ 1 s“p
I h
˝ f q k r P s .
The following result shows that, under certain conditions, the operator I h
˝ f admits ﬁxed points and the sequence p Q k q k P N converges to a distribution Q 8 that assigns probability only to the global maxima of f , restricted to the support of P , i.e., supp p P q .
Theorem 4.2.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, the following statements hold:
(i) P is a ﬁxed point of I h
˝ f , i e., I h
˝ f r P s“ P a.s., if and only if Var x „ P r f p x qs“ 0 ; (ii) let X ˚ “ argmax x P supp p P q t f p x qu be the set of maxima of f restricted to the support of P .
If X ˚ is non-empty and measurable then, the repeated application of I h
˝
f converges to a distribution Q 8 “ lim k Ñ8 p
I h
˝ f q k r P s with support X ˚ .
In particular E x „ Q 8 r f p x qs“ max x P supp p
P q t f p x qu .
Some remarks are in order.
First, both properties are independent of the function h as long as it is non-negative and monotonic strictly-increasing.
This is expected since, h ˝ f admits the same set of global optima of f .
Second, as a corollary to point (i), any deterministic P is a ﬁxed point of I h
˝ f .
Finally, from point (ii), we deduce that if we select P that assigns non-zero probability to all points in X , i.e., supp p P q“ X , the iterated application of I h ˝ f converges to the distribution Q 8 such that
E x „ Q 8 r f p x qs“ max x P X t f p x qu , i.e., we are performing a global optimization of f .
4. I MPLICIT T RUST R EGION
The reader might wonder what are the advantages of casting the optimization of function f as such an iterative procedure.
The reason lies in question (Q3) .
We now prove that we are able to naturally control the divergence between two consecutive distributions Q k and Q k ` 1 “ I h
˝ f r Q k s with k P N , with the effect of enforcing an implicit trust region.
The following result shows how it is possible to obtain a bound on the α -R´enyi divergence between two consecutive distributions.
−     x f ( x )  40   k
E x ∼ Q k [ f ( x ) ]   10 −10 −10k
D K L ( Q k k Q k −) β
= 0 .β = 0 .β
=
.β
=
β =
β =β =
Theorem 4.3.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, for every α Pr 0 , 8s , it holds that:
[Equation]
In particular, for α “ 1 it holds that:
[Equation]
For α “ 2 , we obtain D 2 p
I h
[Equation] f r P s} P q“
[Equation]
E
[Equation]
p x qqs 2 . Thus, the divergence is large when the variance of h p f p x qq is.
The result is particularly remarkable as we are able to control the R ´ enyi divergences of any order α Pr 0 , 8s .
This is a relevant achievement since the trust regions commonly used, like KL-divergence (Schulman et al., 2015), are unable to control higherorder divergences that can still be inﬁnite.
We can also appreciate the role of the increasing function h that works as a regularizer with the effect of controlling the size of the trust region.
The following example shows that the faster h increases, the larger the induced trust region becomes.
Example 4.1.
We consider (a slight variation of) the one-dimensional Ackley function (Ackley, 2012): f p x q“´ 5 ` 20exp p´ 0 .
1414 | x |q` exp p 0 .
5 p cos p 2 πx q` 1 qq` e , shown in Figure 1 (left) and the class of increasing functions p h
˝ f
qp x q“ f p x q β where β ě 0 .
We consider an initial uniform distribution P “ Uni pr´ 5 , 5 sq .
In Figure 1, we plot the expectation of distribution Q k “p I h
˝
f q k r P s (center) and the KL-divergence between two consecutive distributions (right), as a function of the number of applications k , for the different β values.
We observe that convergence to the global optimum ( x ˚ “ 0 and f p x ˚ q“ 15 ) is faster for higher powers that also lead to larger trust regions.
C ONSTRAINED P ROBABILITY D ISTRIBUTION S PACE
The approach we have presented in Section 4 can be applied when there are no restrictions on the class of distributions that can be played, i.e., we can select Q in the whole space P
p X q .
However, in the action-based PO, we can intervene on the policy π θ factors only of the distribution p p τ | θ q“ D 0 p s 0 q ś H ´ 1 t “ 0 π θ p a t | s t q P p s t ` 1 | s t ,a t q , leading to a constrained setting. Similarly, in the parameterbased PO, we can act on the hyperpolicy ν ρ while keeping the trajectory distribution p p τ | θ q ﬁxed.
More in general, when considering a class of distributions Q Ď P p X q , even if P P Q , the distribution I h
˝ f r P s might not belong to Q .Furthermore, while I h
˝ f r P s minimizes all absolute central α -moments of the IS estimator, as it leads to a non-stochastic estimator (Section 3), there may exist different distributions in Q minimizing the different absolute central α -moments:
[Equation]
Apart from α “ 2 , where the problem in Equation (4) reduces to Equation (1), for general value of α Pr 0 , 8s , the optimization is not straightforward (e.g., Equation (4) is not differentiable for α Pp 0 , 2 q ).
The following result shows that performing a moment projection through the α -R´enyi divergence is a reasonable surrogate for minimizing the absolute central α -moments of Equation (4).
Proposition 5.1.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, for any α Pp 1 , 8q , it holds that:
[Equation]
 loooooooooooooooooooooooooomoooooooooooooooooooooooooon absolutecentral α -moment ď E x „ Q „ˆ p p x q q p x q h p f p x qq ˙ α  looooooooooooooomooooooooooooooon (non-central) α -moment “ e p α ´
q
D
α p
I h
˝ f r P s} Q q E x „ P r h p f p x qqs α .
Thus, having considered the subset of distributions Q Ď P p X q , whenever I h ˝ f r P sR Q , we replace it with the corresponding moment projection performed through the α -R´enyi divergence:
[Equation]
In the following, we shall address the questions (Q1) , (Q2) , and (Q3) for the constrained setting.
5. P ERFORMANCE I MPROVEMENT
In Proposition 4.1, we have seen that, whenever h is strictly-increasing, I h
˝ f r P s is a performance improvement w.r.t.
P , evaluated under function f (and also under the composition between f and any strictly-increasing function).
In this section, we address question (Q1) , showing that, when considering a subset of distributions Q Ď P p X q , the performance improvement cannot be in general guaranteed for f , but just for a speciﬁc monotonic transformation of f , depending on h and α .
Theorem 5.2.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Let Q Ď P p X q , Q P Q , and α Pr 0 , 8s , then, it holds that:
[Equation]
´
[Equation]
P
[Equation]
e p α ´ 1 q D α p
I h
˝ f r P s
[Equation]
¯
[Equation] In particular, for α “ 1 , it holds that (Ghosh et al., 2020, Proposition 6):
[Equation]
The theorem shows that by minimizing the α -moment of the transformed function h
˝ f , we are able to guarantee a performance improvement on the function p¨q α ˝ h ˝ f .
The result holds provided that
D α p
I
[Equation] r P [Equation] } Q
[Equation]
D α p
I h
˝ f r P s} P q , which is always guaranteed when P P Q and Q “ Q : , being
Q : deﬁned in Equation (5) as the minimizer of the second divergence term.
In particular, if we select h “p¨q 1 { α , the guarantee holds for the function f directly.
For all other choices, the performance improvement can be guaranteed for a monotonic transformation of f only.
5. C ONVERGENCE P ROPERTIES
We now turn to (Q2) .
By using Equation (5) as an iterate Q k ` 1 P argmin Q P Q t D α p
I h
˝ f r Q k s} Q qu to generate a sequence of distributions p Q k q k P N , we are not guaranteed to converge to any ﬁxedpoint distribution Q 8 , differently form the unconstrained setting (Theorem 4.2).
This is because the minimization might yield multiple solutions.
Nevertheless, we are able to provide guarantees on the ﬁnal divergence value and on the performance of the distributions Q k .
Theorem 5.3.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Let Q Ď P p X q and suppose that h ˝ f is bounded from above, then, the iterate
Q k ` 1 P argmin Q P Q t D α p
I h
˝ f r Q k s} Q qu (where possible ties are broken arbitrarily) satisﬁes:
(i) the sequence of divergences D α p
I h
˝ f r Q k s} Q k q is convergent; (ii) the sequence of expectations
E x „ Q k r h p f p x qq α s is non-decreasing in k P N and converges to a stationary point of E x „ Q r h p
f p x qq α s w.r.t.
Q P Q .
The convergence of the sequences D α p
I h
˝ f r Q k s} Q k q and E x „ Q k r h p f p x qq α s is derived by the performance improvement of Theorem 5.2.
Importantly, Theorem 5.3 shows the convergence to a stationary point of E x „ Q r h p
f
p x qq α s .
If Q is a parametric space Q Ξ “t Q ξ P P p X q : ξ P Ξ Ď R d u , then we are guaranteed to stop
when E x „ Q ξ r ∇ ξ log q ξ p x q h p f p x qq α s“ 0 , like for a general policy gradient method maximizing h p f p x qq α (Papini et al., 2018).
Compared to the result for the unconstrained distribution space (Theorem 4.2), we loose the convergence to a ﬁxed point.
This property can be recovered under the assumption that the iterate in Equation (5) admits a unique solution for every P .
In such a case, we will converge to a distribution Q 8 “ argmin Q P Q t D α p
I h
˝ f r Q s} Q qu .
5. I MPLICIT T RUST R EGION
In Theorem 4.3, we have proved that the α -R´enyi divergence between I h
˝ f r P s and P is bounded.
In this section, we answer (Q3) , wondering whether similar properties hold when we consider a limited set of distributions Q Ď P p X q .
The following result shows that, under a particular form of convexity (van Erven & Harremo¨es, 2014) of Q , we are able to control the trust region as well.
Theorem 5.4.
Let f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictly-increasing.
Let
Q Ď P p X q be a p 1 ´ α q -convex set
(van Erven & Harremo¨es, 2014, Deﬁnition 4), P P Q , Q : P argmin Q P Q t D α p
I h
˝ f r P s} Q qu , and α Pr 0 , 8s , then it holds that:
[Equation]
Therefore, we are always guaranteed that the trust region induced by Q : is tighter compared to the one induced by Q ˚ “ I h
˝ f r P s computed in Theorem 4.3, i.e., D α
[Equation]
P OLICY O PTIMIZATION VIA O PTIMAL P OLICY E VALUATION
In this section, we build a sample-based Off-PL algorithm, named Policy Optimization via Optimal Policy Evaluation (PO 2 PE), which uses Off-VM as an inner loop.
For generality, we consider a parametric distribution space Q Ξ “t Q ξ P P p X q : ξ P Ξ Ď R d u , a common setting met in PO.
Algorithm [Table]
do Optimization i do
Evaluation f j
M end ξ
`
8 end
The structure of PO 2 PE consists of two nested loops.
The outer loop ( Optimization ) acts on the target distribution q ξ i .
At the end of each outer iteration i P r I s , the target distribution q ξ i ` 1 is updated with the last behavioral distribution produced by the inner loop q ξ i,J ` 1 .
Instead, the inner loop ( Evaluation ) takes the target distribution provided by the outer loop q ξ i and provides a new behavioral distribution.
At each inner iteration j Pr J s , it collects samples D i,j with the current behavioral distribution q ξ i,j and employs them, together with all the samples collected so far p D i,k q k Pr j s , to compute the next behavioral distribution q ξ i,j ` 1 , with the goal of ﬁnding the behavioral distribution minimizing the absolute central α -moment of the IS estimator (Equation 4). As we shall see, the optimization is performed using samples and by resorting to a penalized objective.
Sample-based Optimization
The problem of ﬁnding the next behavioral distribution parameter ξ i,j ` 1 using the samples collected so far p D i,k q k Pr j s is an off-policy learning problem.
Let us deﬁne Φ i,j
“ 1 j ř k
Pr j s q ξ i,k as the mixture of the j behavioral distributions experienced so far in the inner loop.
Instead of directly estimating D α p
I h
˝ f r Q ξ i s} Q ξ qq , we refer to the (non-central) α -moment, which is connected to the original objective through Proposition 5.1.
Since we have samples coming from different behavioral distributions, we can use a multiple IS estimator Veach & Guibas (1995):
[Equation]
The (a) factor accounts that we are using samples collected with the mixture Φ i,j to estimate an expectation under q ξ , whereas the (b) factor is the actual variable we want to compute the expectation of, i.e., the α -moment.
It is simple to prove that the expectation of p d α is indeed the α -moment (Papini et al., 2019).
To minimize Equation (6), we employ a variance correction to mitigate the effect of ﬁnite samples (Metelli et al., 2018), theoretically grounded in the following result [Figure]
Theorem 6.1.
Let Q Ξ Ď P p X q be a set of parametric distributions and let ξ , ξ i P Ξ .
If } h ˝ f } 8 ď m , then, if samples are independent, for every δ Pr 0 , 1 s , with probability at least 1 ´ δ it holds that: E x „ ξ
[Figure] [Figure]
Some remarks are in order.
First, the integral within the square root is an upper bound to the variance of the α -moment estimator p
d α p
I h
˝ f r Q ξ i s} Q ξ ;Φ i,j q .
In particular, when ξ “ ξ i , we obtain the exponentiated R ´ enyi divergence, as illustrated in (Metelli et al., 2020).
When all involved distributions are Guassians, it is possible to provide a closed-form tight bound on this quantity (Appendix D).
Second, unlike the results available in the literature about concentration of IS estimator, without corrections or transformations, we are able to provide an exponential concentration inequality (dependence on delta of the form log p 1 { δ q ), instead of a polynomial concentration (dependence of the form 1 { δ ).
This is due to the fact that we are dealing with random variables that are bounded to zero from below and they allow applying stronger unilateral Bernstein’s concentration inequalities (Boucheron et al., 2009).
The reader might object that to optimize the proposed objective function, designed to enforce an implicit trust region, we are actually introducing an additional correction term.
This is necessary for theoretical purposes, but, as we shall see in the Section 7, the need for a penalization or constraint is signiﬁcantly less relevant than in existing approaches, like
TRPO (Schulman et al., 2015), or POIS (Metelli et al., 2018).
Sample Collection
In the action-based setting (AB-PO 2 PE), we sample n trajectories t τ l u l Pr n s independently with the policy π θ i,j and we build the dataset D i,j “tp τ l , R p τ l qqu l Pr n s .
Instead, in the parameter-based setting (PB-PO 2 PE), we sample independently n policy parameters t θ l u l Pr n s and for each of them we run policy π θ l once to generate trajectory τ l . The corresponding dataset is given by D i,j “tpp θ l ,τ l q , R p τ l qqu l Pr n s .
For the AB case, the correction in Theorem 6.1 is estimated from samples, as done for the R ´ enyi divergence in (Metelli et al., 2018), since it involves integrals between trajectory distributions, while the closed form exists for Gaussian distributions (Appendix D).
As noted in Section 5, PO corresponds to a constrained setting and, thus, we are in general unable to provide a performance improvement guarantee for every h (Theorem 5.2).
We show in Appendix C that performance improvement, independently on h , is ensured for deterministic environments and we show some (only theoretical) approaches to extend the guarantee to stochastic environments.
E XPERIMENTAL E VALUATION
In this section, we provide the experimental evaluation of PO 2 PE on continuous control tasks.
We ﬁrst compare the learning performance of PO 2 PE with POIS (Metelli et al., 2018) and TRPO (Schulman et al., 2015) on four benchmarks.
Then, deepen two relevant aspects of PO 2 PE: its robustness to small batch sizes and the effect of the function h .
All experiments are conducted with Gaussian policies, linear in the state, with ﬁxed variance.
The experimental details are reported in Appendix E.
Comparison with POIS and TRPO In Figure 2, we show the average return as a function of the number of collected episodes, with a batch size n “ 100 , using α “ 2 , h “ Id (identity function), and one inner iteration ( J “ 1 ).
In the Cartpole environment, we observe that the performance of
[Figure] [Figure]
β AB-PO 2 PE is slightly above that of AB-POIS and PB-PO 2 PE; while the fastest learning curve is shown by PB-POIS.
Instead, TRPO converges to a suboptimal policy that fails keeping the pole in the vertical position.
In the Inverted Double Pendulum experiment, the gap between AB-PO 2 PE and AB-POIS and TRPO is more evident. The PB versions outperform the AB ones with PO 2 PE slightly faster than POIS. In the Mountain Car domain, while AB-POIS, TRPO, and PB-PO 2 PE display a similar convergence speed, AB-PO 2 PE and PB-POIS reach the optimal performance faster.
Finally, in the Mujoco Swimmer domain (Todorov et al., 2012), AB-PO 2 PE and TRPO clearly outperform
AB-POIS, although the fastest learning curves are displayed by the PB versions of POIS and PO 2 PE.
Robustness to Small Batch Sizes Based on the previous results, we further investigate the properties of PO 2 PE in terms of variance control.
In the Cartpole domain, we test the robustness to the reduction of the batch size.
In Figure 3, we show the average return as a function of the number of collected episodes for batch sizes n Pt 11 , 50 u and different number of inner iterations J .
Also considering the n “ 100 case (Figure 2), we notice, as expected, that the variance of each setting increases overall as n decreases.
Nevertheless, PO 2 PE proves to be robust, always succeeding in reaching the optimal performance.
Differently, POIS suffers the reduced batch size, while TRPO always converging to the same suboptimal policy.
The desirable behavior of PO 2 PE is indeed an effect of the kind of objective function we employ that explicitly accounts for the variance of the estimator, trying to minimize it, and, as we have shown in the previous sections, it allows enforcing an implicit trust region.
Finally, a small number of inner iterations J is beneﬁcial for the stability.
Effect of the Function h
While previous experiments consider h as the identity function, we now investigate the effects of using h “p¨q β , i.e., a power function.
In Figure 4, we show the learning curves of the Inverted Double Pendulum for different values of β .
We notice that for β close to( 0 . 5 , 1 , 2 ) the curves are not very dissimilar, while for too extreme powers ( 0 . 1 and 4 ) the learning performance degrades.
This example shows an interesting phenomenon, i.e., even if we optimize a power of return, within certain limits, we are still able to converge to a (near-)optimal policy.
D ISCUSSION AND C ONCLUSIONS
In this paper, we have deepened the study of importance sampling beyond its usage as a passive tool for off-policy evaluation and learning.
We imported the role of IS as a variance reduction active tool, typical of the Monte Carlo simulation, to the off-policy learning setting.
We have illustrated that by minimizing the absolute central α -moment of the IS estimator we are able to guarantee the performance improvement for a monotonic transformation of the original objective and eventually converge, at least, to a stationary point.
Interestingly, this approach is able to naturally induce a trust region, mitigating the need for an explicit penalization or constraint.
The experimental evaluation conﬁrmed our theoretical ﬁndings.
PO 2 PE is able to outperform POIS and TRPO on several continuous control tasks.
Our algorithm has proved to be remarkably robust to the reduction of the batch size. This is a consequence of using the minimum-variance behavioral distribution that has the beneﬁt of inducing an implicit trust region.
We believe that this work contributes to shed light on an appealing facet of off-policy learning with possible new research opportunities.
Future works include an extension of the convergence analysis to the sample-based setting and an experimentation of PO 2 PE coupled with more complex policy architectures.
A P ROOFS AND D ERIVATIONS
In this appendix, we report the proofs and derivations, we have omitted in the main paper.
A. P ROOFS OF S ECTION
Proposition 4.1 (Proposition 9 of Ghosh et al.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic increasing.
Then, I h
˝ f r P s is a performance improvement w.r.t.
P :
[Equation]
Proof.
Let us consider the following derivation:
[Equation]
where we have exploited the deﬁnition of I
h
˝ f and the deﬁnition of covariance.
The result is obtained by recalling that h is increasing and the covariance between two increasing functions of the same random variable (i.e., h and the identity function) is non-negative (Cuadras, 2002).
Theorem 4.2.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, the following statements hold:
(i) P is a ﬁxed point of I h
˝ f , i.e., I h
˝ f r P s“ P a.s., if and only if Var x „ P r f p x qs“ 0 ; (ii) let X ˚ “ argmax x P supp p P q t f p x qu be the set of maxima of f restricted to the support of P .
If X ˚ is non-empty and measurable then, the repeated application of I h
˝
f converges to a distribution Q 8 “ lim k Ñ8 p
I h
˝ f q k r P s with support X ˚ .
In particular E x „ Q 8 r f p x qs“ max x P supp p
P q t f p x qu .
Proof.
We start with (i).
First of all, we observe that since h is monotonically strictly-increasing it holds that Var x „ P r f p x qs“ 0 if and only if Var x „ P r h p
f p x qqs“ 0 .
P is a ﬁxed point of I h
˝ f , i.e.,
P “ I h
˝ f r P
s a.s.
if and only if for all x P X it holds a.s.:
[Equation]
that occurs if and only if either p p x q“ 0 ( x R supp p P q ) or h p f p x qq“ E x „ P r h p f p x qqs .
( ñ )
Whenever p p x q is not zero, function h p f p x qq is a constant in supp p P q and, consequently, its variance under
P is zero.
( ð ) Suppose that Var x „ P r h p
f
p x qqs“ 0 , then h p f p x qq“ E x „ P r h p f p x qqs almost surely and, consequently p p x q h p f p x qq E x „ P r h p f p x qqs “ p p x q almost surely.
Let us now consider (ii).
First of all, we can easily observe that for every k P N :
[Equation]
Let f ˚ “ max x P supp p P q t f p x qu , consider the function g k p x q“ p p x q ´ f p x q f ˚ ¯ k and the limit:
[Equation]
otherwise .
Thus, we have:
[Equation]
Thus
[Equation]
the support of Q 8 is given by X ˚ .
Consequently, the expectation of f under Q 8 is given by:
[Equation]
Theorem 4.3.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, for every α Pr 0 , 8s , it holds that:
[Equation]
In particular [Equation] for α “ 1 it holds that:
[Equation]
Proof.
Let us consider the following derivation:
[Equation]
By observing that D α p
I h
˝ f r P s} P q“ 1 α ´ 1 log J , we obtain the result.
For α “ 1 , we provide an independent derivation:
[Equation]
where we exploited the deﬁnition of covariance in the last line
[Equation]
A. P ROOFS OF S ECTION
Proposition 5.1.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, for any α Pp 1 , 8q , it holds that:
[Equation]
Proof.
First of all, we observe that since E x „ Q ”
[Equation]
qq ı “ E x „
[Equation]
r h p f p x qqs , for α ě 1 , the absolute central α -moment is smaller or equal than the (non-central) α -moment.
Thus, for α ě 1 , we have:
[Equation]
By applying the deﬁnition of R´enyi divergences [Equation] we get the result.
Theorem 5.2.
Let P P P p X q , f P B p X , r 0 , 8qq [Equation] and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Let Q Ď P p X q , Q P Q
[Equation]
and α Pr [Equation]
8s
[Equation]
then, it holds that:
[Equation]
In particular, for α “
[Equation]
it holds that (Ghosh et al., 2020, Proposition 6):
[Equation]
h ˝ f r P [Equation] ˝ f
[Equation] P
[Equation]
Q qq
[Equation]
Proof.
Let us consider the following derivation:
[Equation]
¯
[Equation]
where line (7) derived from Lemma A.1.
The second inequality was provided in Propositionof (Ghosh et al., 2020).
Theorem 5.3.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Let Q Ď P p X q and suppose that h ˝ f is bounded from above, then, the iterate
Q k ` 1 P argmin Q P Q t D α p
I h ˝ f r Q k s} Q qu (where possible ties are broken arbitrarily) satisﬁes:
(i) the sequence of divergences D α p
I h
˝ f r Q k s} Q k q is convergent; (ii) the sequence of expectations
E x „ Q k r h p f p x qq α s is non-decreasing in k P N and converges to a stationary point of E x „ Q r h p
f p x qq α s w.r.t.
Q P Q .
Proof.
Let us consider the sequence of distributions p Q k q k P N , generated by the iterate in Equation (5), where possible ties are broken with an arbitrary (possibly with a tie-breaking rule T k different for every k ).
From Theorem 5.2, we have for every k P N :
[Equation]
where we simply exploited that Q k P argmin Q P Q t D α p
I h
˝ f r Q k s} Q qu .
Thus, E x „ Q k r h p f p x qq α s is a non-decreasing function of k .
Since h ˝ f is bounded, it must be that
lim k Ñ8 E x „ Q k r h p f p x qq α s“ µ 8 ă8 , that proves convergence.
Furthermore, being convergent, for k Ñ8 it must be that E x „ Q k r h p f p x qq α s“ E x [Equation]
[Equation] h p f p x qq α s and consequently D α p
I h
˝ f r Q k s} Q k q“ D α p
I h
˝ f r Q k s} Q k ` 1 q .
Therefore, even if the tie-braking rule prescribes to select Q k ` 1 ‰ Q k we could select Q k instead, since it lead to the same divergence value.
Consequently, being Q k a solution, we can assert that it is a stationary point of the function
D α p
I h
˝ f r Q k s}¨q (as well as Q k ` 1 ):
[Equation]
We observe that the latter expression is zero if and only if the gradient of E x „ Q r h p f p x qq α s w.r.t.
Q is zero.
Indeed:
[Equation]
Thus, the process converges to a stationary point of E x „ Q k r h p f
p x qq α s .
Theorem 5.4.
Let f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictly-increasing.
Let
Q Ď P p X q be a p 1 ´ α q -convex set
(van Erven & Harremo¨es, 2014, Deﬁnition 4), P P Q , Q : P argmin Q P Q t D α p
I h
˝ f r P s} Q qu , and α Pr 0 , 8s , then it holds that:
[Equation]
Proof.
The proof is a simple application of Lemma A.2, by taking Q Ð P , Q ˚ Ð Q : , and P Ð
I h
˝ f r P s .
A. P ROOFS OF S ECTION
Theorem 6.1.
Let Q Ξ Ď P p X q be a set of parametric distributions and let ξ , ξ i P Ξ .
If } h ˝ f } 8 ď m , then, if samples are independent, for every δ Pr 0 , 1 s , with probability at least 1 ´ δ it holds that:
[Equation]
(Obj) .
such a case function e p α ´ 1 qp¨q is decreasing in its argument.
Proof.
We start observing that each addendum of p d α p
I h
˝ f r Q ξ i s} Q ξ ;Φ i,j q is non negative.
Since all terms are i.i.d.
, we can apply unilateral Bernstein’s inequality (Maurer et al., 2003) that allows achieving an exponential concentration.
Thus, for every δ Pr 0 , 1 s , with probability at least 1 ´ δ it holds that:
[Equation]
Thus, it remains to provide a bound on the variance term.
We exploit the fact that h p f p x qqď m and that each addendum represents an i [Equation] i [Equation]
d. random variable:
[Equation]
A.
[Table]
Lemma A.1.
For every x ě 0 and α Pp 0 , 1 qYp 1 , 8q , it holds that:
[Equation]
Furthermore, for α “ 1 , it holds that:
[Equation]
Proof.
Consider the auxiliary function g α p x q“ x ´ 1 ´
[Equation]
´
[Equation]
˘ .
We are going to prove that the minimum of g α p x q is zero.
Suppose α ą 1 ,
then g α p 0 q“8 and g a p8q“8 .
Thus, the minimum must lie in between and since function g α is differentiable, we have:
[Equation]
Thus, we have g α p 1 q“ 0 .
Suppose now that α ă 1 , we have g α p 0 q
“ α 1 ´ α ą 0 and g α p8q“8 .
Thus, again, the minimum must lie in between and with the same calculations as before, we conclude g α p 1 q“ 0 .
The case α “ 1 is trivial.
Lemma A.2.
Let P P P p X q and let α Pp 0 , 8q .
Let Q Ď P p X q be an p α ´ 1 q -convex (van Erven &
Harremo¨es, 2014, Deﬁnition 4) subset of distributions.
Let Q ˚ P Q be the α -moment projection:
[Equation]
If Q ˚ exists, then for every Q P Q if holds that:
[Equation]
Proof.
The proof of the result is inspired to (van Erven & Harremo¨es, 2014, Theorem 14).
Let λ Pr 0 , 1 s and let us deﬁne Q λ as the p 1 ´ α, p 1 ´ λ,λ qq -mixture of Q ˚ and Q :
[Equation]
Let us ﬁrst observe that for λ “ 0 , we have Q
“ Q ˚ and Z 0 “
[Equation]
Since Q is p 1 ´ α q -convex and Q ˚ is the minimizer over Q , it holds that BB λ D α p P [Equation]
First of all, we compute: ż
[Equation]
The latter, for λ “ 0 , becomes: B B λ Z λ (cid:12)(cid:12) (cid:12) λ “
[Equation]
“ 1 1 ´ α
[Equation]
For calculation easiness, instead of directly operating on D α p P } Q λ q , we consider:
[Equation]
We now evaluate it at λ “ 0 :
B
B λ ż
[Equation]
For α ě 1 , we require BB λ
[Equation]
to obtain:
[Equation]
By applying both sides the log function and dividing by 1 α ´ 1 ą 0 we get the result.
Symmetrically, for α ă 1 , we require the converse BB λ ş
[Equation] “ 0 ď 0 .
Recalling that 1 α ´ 1 ă 0 , we obtain the desired result.
B O PTIMIZING M OMENTS OF f
In this appendix, we analyze the effect of optimizing a power of f instead of f .
Lemma B.1.
Let P P P p X q and f P B p X , r m,m s .
If α Pp 1 , 8q , it holds that:
[Equation]
In particular for α “ 2 , we have:
[Equation]
that is the Bhatia-Davis inequality for the variance.
Proof.
We explicitly consider the optimization problem, for α ě 1 and having denoted µ “
E x
[Equation]
P
[Equation]
p [Equation] [Equation] Since α ě 1 , the optimization problem corresponds to the maximization of a concave function subject to linear and box constraints.
It is simple to prove that the optimal solution must assign extreme values to function f .
Let p Pr 0 , 1 s , the linear and box constraints enforce:
[Equation]
From which, by substitution in the objective function, we have:
[Equation]
Thus, in general, optimizing moments of the function f , leads to different optimal policies compared to optimizing function f directly.
However, from the above results, we see that this discrepancy reduces when the expectation E x „ P r f p x qs approaches the extreme value m (and also m , but this is less interesting since we are maximizing).
The value m can be indeed achieved if we have no restrictions on the distribution space (Section 4).
C P ERFORMANCE I MPROVEMENT FOR THE MDP S ETTING
As already mentioned in Section 5, when we consider the MDP setting, we do not have the full control of the trajectory distribution p p τ | θ q“ D 0 p s 0 q ś H ´ 1 t “ 0 π θ p a t | s t q P p s t ` 1 | s t ,a t q as the factors involving the transition model P and the initial state distribution D 0 are out of the control of the policy and the policy itself π θ is limited due to the parametrization.
As a consequence, performing a step of optimization of the α -moment is able to provide improvement guarantees on J p θ q only by choosing the transformation function h “p¨q 1 α (Theorem 5.2).
In this appendix, we prove that under speciﬁc conditions we are able to provide guarantees on the improvement of J p θ q .
Moreover, we show some approaches, with a main theoretical interest, to extend the performance improvement guarantees to the case of stochastic MDPs.
They must not to be intended as implementation proposals, but rather as theoretical approaches to cope with this phenomenon.
C. A CTION - BASED PO
We start with the action-based PO setting.
Deterministic MDPs
If the MDP is deterministic (i.e., P and D 0 are deterministic), we denote the next state as s t ` 1 “ P p s t ,a t q .
Thus, the trajectory distribution is governed by the policy π θ only: p p τ | θ q“ ś H ´ 1 t “ 0 π θ p a t | s t q .
In such a case, provided that the policy space Θ is sufﬁciently powerful, by minimizing any of the α -moments, we are able to guarantee the performance improvement.
Indeed, in such a case, neglecting the limits of the parametrization Θ , we are in an unconstrained setting.
Stochastic MDPs For general stochastic MDPs, the trajectory density function depends on the transition model probabilities.
Thus, we need to design an estimator that get rids of these elements.
To this purpose, we denote with p p a | θ q as the probability of having observed a sequence of actions a “p a 0 ,a 1 ,...,a H ´ 1 q when playing policy π θ in the MDP.
If we take f “ E r R p τ q| a s , i.e., the expectation of the return conditioned to the sequence of actions, we are again in an unconstrained setting, neglecting the limits of Θ .
Therefore, we are able to guarantee the performance improvement on J p θ q .
Nevertheless, this requires to design an estimator of E r R p
τ q| a s , that might be a not easy task.
C. P ARAMETER - BASED PO
We now consider the parameter-based PO setting. Deterministic MDPs and Policies In the case of deterministic MDPs and deterministic policy π θ , we have that for every θ P Θ , the support of the trajectory distribution is made of one trajectory only. In such a case, ignoring the limits of the hyperpolicy parameter space P , we are in an unconstrained setting.
Stochastic MDPs or Policies
In the case of a stochastic MDP or a stochastic policy (or both), we have a trajectory distribution in which we do not have the possibility to intervene on the policy and transition model factors.
Thus, we need to consider an estimators that get rid of these stochastic elements.
If we take f “ E r R | θ s“ J p θ q , i.e., the expected return conditioned to a policy parametrization θ , we reduce to the unconstrained setting, with the corresponding performance improvement guarantee.
It is worth noting that, compared to the action-based setting, estimating J p θ q is notably simpler compared to the estimation of E r R p τ q| a s .
D C LOSED F ORM OF THE I NTEGRAL FOR G AUSSIANS
In this appendix, we derive a closed form for the integral involved in the computation of the bound of
Theorem 6.1 in the case that all involved distributions are Gaussians and for α “ 2 .
Let us introduce the notation:
[Equation]
We have to compute the following integral: ż
[Equation]
Let us start elaborating on the integrand function, denoting for properly sized vector x and matrix
S , } m } S “ x T Sx and | S | the determinant of S :
[Equation]
Now [Equation] we have to deal with the argument of the exponential:
[Equation]
We now proceed completing the square:
[Equation]
Thus
[Equation]
{+we have: ´ 
+}
[Equation]
Moreover, we observe that the following expression is the density of a k -variate normal distribution with mean M ´ 1 b and covariance matrix M ´ 1 :
[Equation]
Thus, its integral is 1.
Therefore, coming to the initial expression: ż
[Equation]
E E XPERIMENTAL D ETAILS
In this appendix, we report the experimental details and additional experimental results.
Infrastructure The experiments have been run on two machines:
• 2 x CPUs Intel(R)
Xeon(R) CPU E7-8880 v4 @
2.20GHz (22 cores, 44 thread, 55 MB cache) and 128 GB RAM; • 4 x Intel(R) Xeon(R) CPU E5-4610 v2 @
2.30GHz (8 cores, 16 thread, 16 MB cache) and 256 GB RAM.
Environments The environments are the rllab implementations (Duan et al., 2016), MIT license, https://github.com/rll/rllab .
The Swimmer environment belongs to the Mujoco suite (Todorov et al., 2012), MuJoCo Personal License, http://www.mujoco.org/ .
Algorithms The TRPO implementation is taken from baselines (Dhariwal et al., 2017), MIT licence, https://github.com/openai/baselines .
For POIS we use the original implementation (Metelli et al., 2018), MIT license, https://github.com/T3p/baselines .
Hyperparameters In order to properly compare the algorithms, a set of 20 seeds has been chosen.
A subset of 5 seeds, underlined, was used to test the performances during the tuning phase.
Once the optimal hyperparameters were found, the experiments were extended to the other 15 seeds.
In the following, we report the hyperparameter values for PO 2 PE.
The shift return refers to the need for making the return non-negative in order to perform the optimization of the α -moment in PO 2 PE.
This procedure is carried out independently at each algorithm iteration by subtracting the minimum return among the ones observed.
The variance init hyperparameter refers to the logarithm of the standard deviation.
All experiments have been carried out with
Gaussian policies linear with mean linear in the state variables and constant variance uniform over the state space.
Cartpole
• seeds: 0, 3, 11, 16, 19, 42, 66, 72, 84, 87, 90, 123, 222, 343, 404, 452, 542, 875, 943,• max iters:• policy: linear • policy init: zeros • capacity:• inner:• variance init: - • step size: 1 / gradient norm • penalization: True • delta: 0. • max ofﬂine iters:
[Table]
• seeds: 0, 3, 11, 16, 19, 42, 66, 72, 84, 87, 90, 123, 222, 343, 404, 452, 542, 875, 943,• max iters:• policy: linear • policy init: zeros • capacity:• inner:• variance init: - • step size: 2 / gradient norm • penalization: True • delta: 0.
• max ofﬂine iters:• shift return: True
[Table]
• seeds: 0, 3, 11, 16, 19, 42, 66, 72, 84, 87, 90, 123, 222, 343, 404, 452, 542, 875, 943,• max iters:• policy: linear • policy init: zeros • capacity:• inner:• variance init: - • step size: 2 / gradient norm • penalization: True • delta: 0. • max ofﬂine iters:
[Table]
• seeds: 0, 3, 11, 16, 19, 42, 66, 72, 84, 87, 90, 123, 222, 343, 404, 452, 542, 875, 943,• max iters:• policy: linear • policy init: zeros • capacity:• inner:• log-std init: -0. • step size: 1 / gradient norm • penalization: True • delta: 0. {+• max ofﬂine iters:• shift return: True 5× 
 Episodes+}
{+A v e r ag e r e t u r n σ =0 0 . 5× 
 Episodes σ = 0 .0 0 .+}
{+5× 
 Episodes σ = 0 .0+} 0 . {+5× 
 Episodes σ =+}
[Figure]
For POIS (both AB and PB) and TRPO, the same hyperparameter value have been used, except for the algorithm-speciﬁc ones that have been tuned with the same protocol discussed above ( δ KL P t 0 .
001 , 0 . 01 , 0 . 1 , 1 u ).
In particular, for POIS, we employ the line search procedure presented in the original paper for setting the step-size.
The following table summarizes the algorithm-speciﬁc hyperparameter values for the different algorithms and environments. Environment / Algorithm
PO 2 PE (delta)
AB-POIS (delta) TRPO (max kl) Cartpole 0. Mountain Car 0.
Inverted Double Pendulum 0.
Swimmer 0.
Environment / Algorithm
PB-POIS (delta)
PB-PO 2 PE (delta)
Cartpole 0.
Mountain Car 0.
Inverted Double Pendulum 0.
Swimmer 0.
E. N OISE R OBUSTNESS
As we have shown in Appendix C, using the trajectory return R p τ q as function f does no longer allow to provide performance improvement guarantees.
Nevertheless, we conjecture that the loss of this property is compensated by the variance reduction implicit in our approach.
In the direction of empirically showing this aspect, we tested the parameter-based version of PO 2 PE in the Inverted
Double Pendulum environment, with forced stochasticity in the environment.
Speciﬁcally, whenever an action is prescribed by the policy the actual action to be executed is obtained by adding while
Gaussian noise with standard deviation σ .
The results are shown in Figure 5.
We observe that our algorithm is overall competitive with PB-POIS and, in the case of σ “ 1 , signiﬁcantly outperforms
PB-POIS.
E. A BOUT R ETURN T RANSLATION
Our approach can be employed for non-negative functions f .
Since in the PO experimental evaluation we employ f “ R p τ q .
Under the assumption that the immediate reward is bounded
R p s,a qPr R min ,R max s for all p s,a qP S ˆ A , we can make the return function with a simple translation and preserving the optimality of policies:
[Equation]
where R min 1 ´ γ H 1 ´ γ is the minimum achievable return.
Of course, we can perform the translation even by using a constant c ě c min “´ R min 1 ´ γ H 1 ´ γ and still obtain a translated return that remains positive.
It is worth noting, from Theorem 4.3 that the size of the trust region is larger as the constant approaches the its minimum possible value.
For instance, we consider α “ 2 , f ě 0 , and we apply a further translation with c ě 0 .
From Theorem 4.3, we have:
[Equation]
Since E x „ P r f p x q 2 sě E x „ P r f p x qs 2 , we have that this expression is maximized with the smallest value of c , i.e., c “ 0 .