Policy Optimization via Optimal Policy Evaluation
Abstract
Off-policy methods are the basis of a large number of effective Policy Optimization (PO) algorithms.
In this setting, Importance Sampling (IS) is typically employed as a what-if analysis tool, with the goal of estimating the performance of a target policy, given samples collected with a different behavioral policy.
However, in Monte
Carlo simulation, IS represents a variance minimization approach.
In this ﬁeld, a suitable behavioral distribution is employed for sampling, allowing diminishing the variance of the estimator below the one achievable when sampling from the target distribution.
In this paper, we analyze IS in these two guises, showing the connections between the two objectives.
We illustrate that variance minimization can be used as a performance improvement tool, with the advantage, compared with direct off-policy learning, of implicitly enforcing a trust region.
We make use of these theoretical ﬁndings to build a PO algorithm, Policy Optimization via Optimal
Policy Evaluation (PO 2 PE), that employs variance minimization as an inner loop.
Finally, we present empirical evaluations on continuous RL benchmarks, with a particular focus on the robustness to small batch sizes.
Introduction
Policy Optimization methods [PO, 7] have been widely exploited in Reinforcement Learning [RL, 39] with successful results in addressing, to name a few, continuous-control [e.g., 33, 24], robot manipulation [e.g., 12, 3] and locomotion [e.g., 22, 9].
Most of these algorithms employ the notion of trust region [5], introduced ante litteram in the RL literature by the safe RL approaches [21, 34], giving rise to a surge of effective algorithms, having TRPO [38] as the progenitor.
The core of any RL algorithm, being value-based or policy-based, lies in the ability to employ the samples collected with the current (or behavioral ) policy to evaluate the performance of a candidate (or target ) policy [39].
The skeleton rationale behind the usage of a trust region is to control the set of candidate policies whose performance can be accurately evaluated.
Intuition suggests that if the candidate policy is “sufﬁciently close” to the current one, this off-policy evaluation problem [35] will provide a good estimate for the performance of the candidate policy.
Formally, this idea has been studied in the ﬁeld of Importance Sampling [IS, 30] and the phenomenon is particularly evident looking at the IS estimator variance, which grows exponentially with the Rényi divergence [37] between the behavioral and the target policy [27, 28].
In this off-policy learning (Off-PL) setting, IS is employed as a what-if analysis tool [30] and its role is passive , as samples have been already collected with the current behavioral policy.
In this sense, the trust region is an a-posteriori remedy for the limitations of off-policy evaluation, having the goal of controlling the uncertainty injected by the IS procedure.
However, IS originated in the Monte Carlo simulation community [17, 13] as an active tool for variance minimization (Off-VM).
While in Off-PL, the behavioral policy is ﬁxed and we look for the best target policy, whose performance we aim to estimate, here the roles are reversed.
Indeed, in OffVM, the target policy is ﬁxed and we search for the behavioral policy (from which to collect samples) that yields an IS estimate with the minimum possible variance [13, 19].
It might seem surprising, at ﬁrst, that sampling from a policy, other than the target one, can lead to an estimator with less variance
(even zero in some cases) w.r.t.
the on-policy estimate.
In this role, IS has been previously employed in RL, mainly to address rare events [10, 4] which naturally lead to high-variance estimates, when tackled on-policy.
The idea of explicitly using IS as a variance reduction technique, with the goal of ﬁnding an optimal behavioral policy, was proposed by [15] for evaluation and subsequently combined with policy gradient learning [14, 16].
However, in these works, the variance minimization (Off-VM) process and the off-policy learning (Off-PL) problem are treated separately.
The goal of this paper is to investigate the relation between variance minimization (Off-VM) and off-policy learning (Off-PL).
The core question we address can be summarized as: “ Can Off-VM be employed as a tool for Off-PL, overcoming the need for an explicit trust region? ”
Intuitively, given a target policy, when the reward function is positive, one way to reduce the variance of the IS estimator is to assign larger probability to the trajectories that have a large impact on the mean, i.e., those with high returns.
This provides a ﬁrst hint about the connection between the minimum-variance sampling policy and the performance improvement, i.e., between Off-VM and Off-PL.
Furthermore, it suggests that we could repeatedly apply the process of identifying the minimum-variance policy as a tool for policy improvement.
The interesting aspect of such an approach is that, by minimizing the variance, it implicitly controls the divergence between two consecutive policies.
In other words, it allows enforcing a trust region, without an explicit need for divergence constraints or penalizations.
Outline of the Contributions In this paper, we provide theoretical, algorithmic, and experimental contributions.
After having introduced the necessary background (Section 2), we present the problem of ﬁnding the minimum-variance behavioral distribution (Section 3).
Then, we study the properties of the Off-VM problem in two settings: unconstrained (Section 4) and constrained (Section 5).
First, we assume that there are no restrictions in the choice of the behavioral distribution.
We show that the minimum-variance behavioral distribution, besides leading to the well-known zero-variance estimator [19], is guaranteed to yield a performance improvement, requiring the non-negativity of the reward only.
Furthermore, we prove that this approach allows controlling the divergence between two consecutive distributions, thus enforcing an implicit trust region.
Although this provides a valuable starting point, the minimum-variance distribution might be unrealizable given the environment transition model, i.e., there might be no policy inducing it.
For this reason, we move to the scenario in which the available distributions are constrained in a suitable space.
In this setting, the zerovariance estimator could not be achievable.
Nevertheless, we prove that such a procedure can lead to a performance improvement and preserves the ability to enforce a trust region.
Based on these theoretical results, we propose Policy Optimization via Optimal Policy Evaluation (PO 2 PE), a novel PO algorithm, that we particularize for parametric policy spaces (Section 6).
Finally, we provide numerical simulations on continuous-control benchmarks, in comparison with POIS [27] and
TRPO [38], with a particular focus on the robustness of PO 2 PE to small batch sizes (Section 7).
The proof of the results presented in the main paper are reported in Appendix A.
Preliminaries
In this section, we report the necessary background that will be employed in the paper.
Mathematical Notation Let X be a set, and let F X be a σ -algebra over X .
We denote with P p
X q the space of probability measures over p X , F X q .
Let P P P p X q , whenever needed, we assume that P admits a density function p .
For a subset Y Ď R , we denote with B p X , Y q the space of measurable functions f : X Ñ Y .
Let P,Q P P p
X q be two probability measures such that P !
Q , i.e.,
P is absolutely continuous w.r.t.
Q , for every α Pr 0 , 8s , we deﬁne the α -Rényi divergence as [37]:
D α p P } Q q“ 1 α ´ 1 log ş
X p p x q α q p x q 1 ´ α d x .
In the limit of α Ñ 1 , the Rényi divergence reduces to the
KL-divergence D KL p P } Q q , while for α Ñ8 , it reduces to esssup x „ Q t p p x q{ q p x qu .
Importance Sampling Let P,Q P P p X q with P !
Q and let f P B p X , R q .
Importance Sampling [IS, 30] allows estimating the expectation of f under a target distribution P , i.e., E x „ P r
f p x qs having samples t x i u i Pr n s collected with a behavioral distribution Q , leading to the estimator:
[Equation]
The IS estimator is well-known to be unbiased [30], i.e., E x i „ Q r p µ P { Q s“ E x „ P r f p x qs , but it might suffer from large variance, due to the heavy-tailed behavior [27].
The properties of p µ P { Q and several of its transformations have been extensively studied in the literature [e.g., 18, 40, 32, 23, 28, 26, 29].
Policy Optimization A Markov Decision Process [MDP, 36] is a 6-tuple M “p S , A ,P,R,γ,D 0 q , where S is the state space,
A is the action space, P : S ˆ A Ñ P p S q is the transition model, R :
S ˆ
A Ñr 0 ,R max s is the reward function, γ Pr 0 , 1 s is the discount factor, and D 0 P P p S q is the initial state distribution.
The agent’s behavior is modeled by a parametric policy π θ : S Ñ P p
A q belonging to a parametric policy space Π Θ “t π θ : θ P Θ Ď R d u .
The interaction between an agent and the MDP generates a trajectory τ “p s 0 ,a 0 ,s 1 ,a 1 ,...,s H ´ 1 ,a H ´ 1 ,s H q where H P N is the trajectory length and s 0 „ D 0 , a t „ π θ p¨| s t q , s t ` 1 „ P p¨| s t ,a t q for all t Pt 0 ,...,H ´ 1 u .
Given a trajectory τ , the return is the discounted sum of the rewards R p τ q“ ř H ´ 1 t “ 0 γ t R p s t ,a t q .
For a policy π θ P Π Θ , we denote with p p¨| θ
q the induced trajectory distribution: p p τ | θ q“ D 0 p s 0 q ś H ´ 1 t “ 0 π θ p
a t | s t q P p s t ` 1 | s t ,a t q .
An agent aims at ﬁnding a parametrization maximizing the expected return J p θ q [7]:
[Equation]
where
[Equation]
In the remainder of the paper, we will keep the presentation as general as possible, introducing the results for arbitrary distributions.
Then, we will particularize for the parametric PO setting.
Minimum–Variance Behavioral Distribution
In this section, we revise Off-VM, i.e., the problem of ﬁnding a behavioral distribution Q P P p X q that induces an IS estimate p µ P { Q with minimum variance, knowing the (ﬁxed) target distribution
P P P p X q and function f P B p X , r 0 , 8qq .
Furthermore, we do not enforce any restrictions on the possible forms of the behavioral distribution Q P P p X q .
The problem and the corresponding well-known minimum-variance behavioral distribution Q ˚ are stated in the following [20, 19]:
[Equation]
We observe that the IS estimator p µ P { Q ˚ is non-stochastic, equal to the quantity we aim to estimate, i.e., p µ P { Q ˚ “ E x „ P r f p x qs .
This suggests that the construction of Q ˚ is infeasible as it requires knowledge of E x „
P r f p x qs .
Since Q ˚ generates a non-stochastic estimator, it not only leads to zero-variance but, clearly, simultaneously minimizes the absolute central moments of any order.
A second, and most remarkable property, is that Q ˚ is a performance improvement w.r.t.
P , i.e., the expectation of f under Q ˚ is larger than the expectation of f under the target distribution P [30]:
[Equation]
It is worth noting that the magnitude of the improvement is directly related to the reduction in variance
V ar x „ P r f p x qs .
Equation (2) suggests an appealing connection between the problem of ﬁnding the minimum-variance behavioral distribution (Off-VM) and the problem of ﬁnding a target distribution that maximizes the expectation E x „
P r f p x qs (Off-PL).
In other words, we could employ Off-VM as a performance improvement tool, by repeatedly solving the problem in Equation (1).
In the following two sections, we will delve into the properties of the repeated construction of the minimum-variance distribution as a performance improvement tool under two assumptions: (i) there are no restrictions in the choice of the behavioral distribution Q P P p X q (Section 4); (ii) the behavioral distribution must be chosen within a subset Q P Q Ď P p X q (Section 5).
In both cases, we will address the following three questions:
(Q1)
Does this procedure always generate a distribution that is a performance improvement ?
(Q2)
Does this procedure converge to a (global or local) maximum of f ?
(Q3)
Can we quantify the divergence between two consecutive distributions,
i
e., does this procedure enforce a trust region ?
Unconstrained Probability Distribution Space
In Section 3, we have seen that Q ˚ is a performance improvement w.r.t.
P .
We now generalize of this construction, by composing function f with a non-negative monotonic strictly-increasing function h : r 0 , 8qÑr 0 , 8q .
The rationale behind this choice is that if h is strictly-increasing, then h
˝ f has the same maxima as f .
We start deﬁning the operator
I h
˝ f : P p X qÑ P p
X q :
[Equation]
Thus, I h
˝ f takes as input a target distribution P P P p X q , a function h
˝ f P B p X , r 0 , 8qq , and outputs the minimum-variance behavioral distribution for the IS estimation of E x „ P r h p
f p x qqs , i.e.,
Q ˚ “ I h
˝ f r P s .
Intuitively, looking at Equation (3), by iterating the application of I h
˝
f , we will obtain distributions tending to assign larger probability mass to points x P X with high values of f p x q .
Concerning (Q1) , the following result, due to [11], generalizes Equation (2) showing that whenever h is increasing, we can prove that I h
˝ f r P s is a performance improvement w.r.t.
P .
Proposition 4.1 (Proposition 9 of [11]) .
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic increasing.
Then, I h
˝ f r P s is a performance improvement w.r.t.
P :
[Equation]
It is worth noting that, since h is a monotonic increasing function, we have that
C ov x „ P r h p
f
p x qq ,f p x qsě 0 [6].
The following sections tackle questions (Q2) and (Q3) .
4.1 Convergence Properties
We now address question (Q2) , analyzing the effect of repeatedly applying operator
I h
˝ f .
More formally, let us consider an initial distribution P P P p X q , and suppose to iterate the application of the operator I h ˝ f , generating the sequence of distributions p Q k q k P N , where Q 0 “ P and for every k P N ěwe have Q k
“ I h
˝ f r Q k ´ 1 s“p
I h
˝ f q k r P s .
The following result shows that, under certain conditions, the operator I h
˝ f admits ﬁxed points and the sequence p Q k q k P N converges to a distribution Q 8 that assigns probability to the global maxima of f , restricted to the support of P .
Theorem 4.2.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, the following statements hold:
(i) P is a ﬁxed point of I h
˝ f , i.e., I h
˝ f r P s“ P a.s., if and only if V ar x „ P r f p x qs“ 0 ; (ii) let X ˚ “ argmax x P supp p P q t f p x qu be the set of maxima of f restricted to the support of P .
If X ˚ is non-empty and measurable then, the repeated application of I h
˝
f converges to a distribution Q 8 “ lim k Ñ8 p
I h
˝ f q k r P s with support X ˚ .
In particular:
[Equation]
Some remarks are in order.
First, all three properties are independent of the function h as long as it is non-negative and monotonically increasing.
This is expected since, under this condition, h ˝ f admits the same set of global optima of f .
Second, as a corollary to point (i), any deterministic P is a ﬁxed point of I h
˝ f .
Finally, from point (ii), we deduce that if we select P that assigns non-zero probability to all points in X , i.e., supp p P q“ X , the iterated application of I h ˝
f converges to the distribution
Q 8 such that E x „ Q 8 r f p x qs“ max x P X t f p x qu , i.e., we are performing a global optimization of f .
4.2 Implicit Trust Region
The reader might wonder what are the advantages of casting the optimization of function f as such an iterative procedure.
The reason lies in question (Q3) .
We now prove that we are able to naturally control the divergence between two consecutive distributions Q k and Q k ` 1 “ I h
˝ f r Q k s , with the effect of enforcing an implicit trust region.
The following result shows how it is possible to obtain a bound on the α -Rényi divergence between two consecutive distributions.
Theorem 4.3.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, for every α Pr 0 , 8s , it holds that:
[Equation]
−     x f ( x )  40   k
E x ∼
Q k [ f ( x ) ]   10 −10 −10k
D K L (
Q k k
Q k −) β
= 0 .β = 0 .β
=
.β
=
β =
β =β =
In particular, for α “ 1 it holds that:
[Equation]
For α “ 2 , we obtain D 2 p
I h
˝ f r P s} P q“ log
[Figure]
[Equation]
Thus, the divergence is large when the variance of h p f p x qq is.
The result is particularly remarkable as we are able to control the Rényi divergences of any order α Pr 0 , 8s .
This is a relevant achievement since the trust regions commonly used, like KL-divergence [38], are unable to control higher-order divergences that can still be inﬁnite.
We can also appreciate the role of the increasing function h that works as a regularizer with the effect of controlling the width of the trust region.
The following example shows that the faster h increases, the larger the induced trust region becomes.
Example 4.1.
We consider (a slight variation of) the one-dimensional Ackley function [1]: f p x q“ ´ 5 ` 20exp p´ 0 .
1414 | x |q` exp p 0 .
5 p cos p 2 πx q` 1 qq` e , shown in Figure 1 (left) and the class of increasing functions p h
˝ f
qp x q“ f p x q β where β ě 0 .
We consider an initial uniform distribution
P “ Uni pr´ 5 , 5 sq .
In Figure 1, we plot the expectation of distribution Q k “p I h
˝
f q k r P s (center) and the KL-divergence between two consecutive distributions (right), as a function of the number of applications k , for the different β values.
We observe that convergence to the global optimum ( x ˚ “and f p x ˚ q“ 15 ) is faster for higher powers which, at the same time, lead to larger trust regions.
Constrained Probability Distribution Space
The approach we have presented in Section 4 can be effectively applied when there are no restrictions on the class of distributions that can be played, i.e., we can select Q in the whole space
P p
X q .
This is for instance the case of multi-armed bandit problems where any distribution over the arms can be played, but not the case of MDPs in which trajectory distributions are governed by the transition model and are, naturally, constrained.
More formally, when consider a class of distributions
Q Ď P p X q , even if P P Q , the distribution
I h
˝ f r P s might not belong to Q .
Furthermore, while
I h
˝ f r P s minimizes all absolute central α -moments of the IS estimator, as it leads to a non-stochastic estimator (Section 3), there may exist different distributions in Q minimizing the different absolute central α -moments:
[Equation]
Apart from α “ 2 , where the problem in Equation (4) reduces to Equation (1), for general value of α Pr 0 , 8s , the optimization is not straightforward (e.g., Equation (4) is not differentiable for α Pp 0 , 2 q ).
The following result shows that performing a moment projection through the α -Rényi divergence is a reasonable surrogate for minimizing the absolute central α -moments of Equation (4).
Proposition 5.1.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Then, for any α Pp 1 , 8q , it holds that:
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
Thus, having considered the subset of distributions Q Ď P p X q , whenever I h ˝ f r P sR Q , we replace it with the corresponding moment projection performed through the α -Rényi divergence:
[Equation]
In the following sections, we shall address the questions (Q1) , (Q2) , and (Q3) .
5.1 Performance Improvement
In Proposition 4.1, we have seen that, whenever h is strictly-increasing, I h
˝ f r P s is a performance improvement w.r.t.
P , evaluated under function f (and also under the composition between f and any strictly-increasing function).
In this section, we address question (Q1) , showing that, when considering a subset of distributions Q Ď P p X q , the performance improvement cannot be in general guaranteed for f , but just for a speciﬁc monotonic transformation of f , depending on h and α .
Theorem 5.2.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Let Q Ď P p X q , Q P Q , and α Pr 0 , 8s , then, it holds that:
[Equation]
¯ .
In particular, for α “ 1 , it holds that [11, Proposition 6]:
[Equation]
The result shows that by minimizing the α -moment of the transformed function h
˝ f , we are able to guarantee a performance improvement on the function p¨q α ˝ h ˝ f .
The result holds provided that
D α p
I h
˝ f r P s} Q qď D
α p
I h
˝ f r P s} P q , which is always guaranteed when P P Q and Q “ Q : , being
Q : deﬁned in Equation (5) as the minimizer of the second divergence term.
In particular, if we select h “p¨q 1 { α , the guarantee holds for the function f directly.
For all other choices, the performance improvement can be guaranteed for a monotonic transformation of f only.
5.2 Convergence Properties
We now turn to (Q2) .
By using Equation (5) as an iterate Q k ` 1 P argmin Q P Q t D α p
I h
˝ f r Q k s} Q qu to generate a sequence of distributions p Q k q k P N , we are not guaranteed to converge to any ﬁxed-point distribution Q 8 , differently form the unconstrained setting (Theorem 4.2).
This is because the minimization might yield multiple solutions.
Nevertheless, we are able to provide guarantees on the ﬁnal divergence value and on the performance of the distributions Q k .
Theorem 5.3.
Let P P P p X q , f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictlyincreasing.
Let Q Ď P p X q
and suppose that h ˝ f is bounded from above, then, the iterate
Q k ` 1 P argmin Q P Q t D α p
I h
˝ f r Q k s} Q qu (where possible ties are broken arbitrarily) satisﬁes:
(i) the sequence of divergences D α p
I h
˝ f r Q k s} Q k q is convergent; (ii) the sequence of expectations
E x „ Q k r h p f p x qq α s is non-decreasing in k P N and converges to a stationary point of E x „ Q r h p
f p x qq α s w.r.t.
Q P Q .
The convergence of the sequences D α p
I h
˝ f r Q k s} Q k q and E x „ Q k r h p f p x qq α s is derived by the performance improvement result of Theorem 5.2.
The important point of Theorem 4.2 is that we achieve convergence to a stationary point of E x „ Q r h p
f
p x qq α s .
If Q is a parametric space Q Θ “t Q θ P P p X q : θ P Θ Ď R d u , then we are guaranteed to stop when E x „ Q θ r ∇ θ log q θ p x q h p
f p x qq α s“ 0 , like for a general policy gradient [31] method maximizing h p f p x qq α .
Compared to the result for the unconstrained distribution space (Theorem 4.2), we loose the convergence to a ﬁxed point.
This property can be recovered under the assumption that the iterate in Equation (5) admits a unique solution for every P .
In such a case, we will converge to a distribution Q 8 “ argmin Q P Q t D α p
I h
˝ f r Q s} Q qu .
5.3 Implicit Trust Region
In Theorem 4.3, we have proved that the α -Rényi divergence between I h
˝ f r P s and P is bounded.
In this section, we answer (Q3) , wondering whether similar properties hold when we consider a limited set of distributions Q Ď P p X q .
The following result shows that, under a particular form of convexity [42] of Q , we are able to control the trust region as well.
Theorem 5.4.
Let f P B p X , r 0 , 8qq , and h : r 0 , 8qÑr 0 , 8q monotonic strictly-increasing.
Let Q Ď P p X q be a p 1 ´ α q -convex set [42, Deﬁnition 4], P P Q , Q : P argmin Q P Q t D α p
I h
˝ f r P s} Q qu , and α Pr 0 , 8s , then it holds that:
[Equation]
Therefore, we are always guaranteed that the trust region induced by Q : is tighter compared to the one induced by Q ˚ “ I h
˝ f r P s computed in Theorem 4.3, i.e., D α
[Equation]
Policy Optimization via Optimal Policy Evaluation
In the previous sections, we have discussed the properties of the distributions that minimize the absolute central α -moments of the IS estimator, when the sampling distributions is chosen without restrictions (Section 4) or within a set of distributions (Section 5).
In this section, we employ these results to build a sample-based Off-PL algorithm, which uses Off-VM as an inner loop.
The pseudocode of the algorithm, named Policy Optimization via Optimal Policy Evaluation (PO 2 PE), is reported in Algorithm 1.
For generality of presentation, we consider a parametric distribution space
Q Θ “t Q θ P P p X q : θ P Θ Ď R d u , that is a common setting encountered in PO.
[Table]
Optimization
[Table]
,J do
Evaluation
Collect n samples θ i,j minimizing f
D j send
8 end
The basic structure of PO 2 PE consists of two nested loops.
Given a target distribution q θ i , the inner loop aims at performing the
Evaluation of the performance of q θ i .
At each inner iteration j Pr J s , it collects samples D i,j with the current behavioral distribution q θ i,j and employs them, together with all the samples collected so far p D i,k q k Pr j s , to com- pute the next behavioral distribution q θ i,j ` 1 , with the goal of minimizing the absolute central αmoment.
This process is governed by two hyperparameters: h the transformation function and α the moment order.
The outer loop, instead, aims to perform the
Optimization of the target distribution q θ i .
At the end of each outer iteration i Pr I s , the target distribution q θ i ` 1 is updated with the last behavioral distribution produced by the inner loop q θ i,J ` 1 .
To get a usable algorithm, we need to further characterize how the samples are collected (Line 4), particularizing for the PO setting, and how to perform the optimization from samples (Line 5).
Sample-based Optimization
The problem of ﬁnding the next behavioral distribution parameter θ i,j ` 1 using the samples collected so far p D i,k q k Pr j s is in all regards an off-policy learning problem.
Let us deﬁne Φ i,j “
1 j ř k
Pr j s q θ i,k as the mixture of the j behavioral distributions experienced so far in the inner loop.
Instead of directly estimating D α p
I h
˝ f r Q
θ i s} Q
θ qq , we refer to the (non-central) α -moment, which is connected to the original objective through Proposition 5.1.
Since we have samples coming from different behavioral distributions, we can use a multiple IS estimator [43]:
[Equation]
The (a) factor takes into account that we are using samples collected with the mixture Φ i,j to estimate an expectation under q θ , whereas the factor (b) is the actual variable we want to compute the expectation of, i.e., the α -moment.
It is simple to prove that the expectation of p d α is indeed the α -moment [32].
To perform the minimization of Equation (6), we employ a variance correction to mitigate the effect of ﬁnite samples [27], theoretically grounded in the following result.
[Figure]
ag
[Figure]
r
[Figure]
t u r n
Cartpole
[Figure]
Inverted Double Pendulum
[Figure]
Swimmer
[Figure]
Theorem 6.1.
Let Q Θ Ď P p X q be a set of parametric distributions and let θ , θ i P Θ .
If } h ˝ f } 8 ď m , then, if all samples are independent, for every δ Pr 0 , 1 s , with probability at least 1 ´ δ it holds that:
E
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
Some remarks are in order.
First, the integral within the square root is an upper bound to the variance of the α -moment estimator p
d α p
I h
˝ f r Q
θ i s} Q θ ;Φ i,j q .
In particular, when θ “ θ i , we obtain the exponentiated Rényi divergence, as illustrated in [28].
When all involved distributions are Guassians, it is possible to provide a closed-form tight bound on this quantity (Appendix C).
Second, unlike the results available in the literature about concentration of IS estimator, without correction or transformation, we are able to provide an exponential concentration inequality (dependence on delta of the form log p 1 { δ q ), instead of a polynomial concentration (dependence of the form 1 { δ ).
This is due to the fact that we are dealing with random variables that are bounded to zero from below and they allow applying stronger unilateral Bernstein’s concentration inequalities [2].
The reader might object that to optimize the proposed objective function, designed to enforce an implicit trust region, we are actually introducing an additional correction term.
This is necessary for theoretical purposes, but, as we shall see in the Section 7, the need for a penalization or constraint is signiﬁcantly less relevant than in existing approaches, like TRPO [38], or POIS [27].
Sample Collection
The sample collection (Line 4) depend on the kind of problem we are dealing with.
Speciﬁcally, for the PO setting, q θ “ p p¨| θ q is the trajectory distribution induced by policy π θ , and function f corresponds to the trajectory return R p τ q .
At each inner iteration j Pr J s , we sample n trajectories t τ l u l Pr n s independently with the policy π θ i,j and we build the dataset D i,j “ tp τ l , R p τ l qqu l Pr n s .
The correction term in Theorem 6.1 has to be estimated from samples as well, as done for the Rényi divergence in [27], since it involves integrals between trajectory distributions.
Experimental Evaluation
In this section, we provide the experimental evaluation of PO 2 PE on continuous control tasks.
We ﬁrst compare the learning performance of PO 2 PE with POIS [27] and TRPO [38] on four benchmarks.
Then, we dive into two relevant aspects of PO 2 PE: its robustness to small batch sizes and the effect of the transformation function h .
All experiments are conducted with Gaussian policies, linear in the state variables, with ﬁxed variance.
The experimental details are reported in Appendix D.
Comparison with POIS and TRPO In Figure 2, we show the average return as a function of the number of collected episodes, with a batch size n “ 100 , using α “ 2 , h “ Id , and one inner iteration ( J “ 1 ).
In the Cartpole environment, we observe that the performance of PO 2 PE is slightly above that of POIS.
Instead, TRPO converges to a suboptimal policy that fails keeping the pole in the vertical position.
In the Inverted Double Pendulum experiment, the gap between PO 2 PE and the baselines is more evident, whereas in the Mountain Car domain, while POIS and TRPO display a similar convergence speed, PO 2 PE reaches the optimal performance faster.
Finally, in the Mujoco
[Figure]
t u r n n =
[Figure] [Figure]
n =
[Figure] [Figure]
Swimmer domain [41], PO 2 PE and TRPO clearly outperform POIS.
In all three experiments, we appreciate the small variance of PO 2 PE across the different runs.
Robustness to Small Batch Sizes Based on the previous results, we further investigate the properties of PO 2 PE in terms of variance control.
In the Cartpole domain, we test the robustness to the reduction of the batch size.
In Figure 3, we show the average return as a function of the number of collected episodes for batch sizes n Pt 11 , 50 u and different number of inner iterations J .
Also considering the n “ 100 case (Figure 2), we notice, as expected, that the variance of each setting increases overall as n decreases.
Nevertheless, PO 2 PE proves to be robust, always succeeding in reaching the optimal performance.
Differently, POIS suffers the reduced batch size, while TRPO always converging to the same suboptimal policy.
The desirable behavior of PO 2 PE is indeed an effect of the kind of objective function we employ that explicitly accounts for the variance of the estimator, trying to minimize it, and, as we have shown in the previous sections, it allows enforcing an implicit trust region.
Concerning the number of inner iterations J , although all considered cases approach the optimal performance, a small number of inner iterations seem to be beneﬁcial for the stability.
Effect of the Function h
While previous experiments we consider h to be the identity function, we now investigate the effects of using h “p¨q β , i.e., a power function.
In Figure 4, we show the learning curves of the Inverted Double Pendulum for different values of β .
We notice that for β close to(0.5, 1, 2) the curves are not very dissimilar, while for too extreme powers (0.1 and 4) the learning performance degrades.
This example shows an interesting phenomenon, i.e., even if we optimize a power of return, within certain limits, we are still able to converge to a (near-)optimal policy.
Discussion and Conclusions
In this paper, we have deepened the study of importance sampling beyond its usage as a passive tool for off-policy evaluation and learning.
We imported the role of IS as a variance reduction active tool, typical of the Monte Carlo simulation ﬁeld, to the off-policy learning setting.
We have illustrated that by minimizing the absolute central α -moment of the IS estimator we are able to guarantee the performance improvement for a monotonic transformation of the original objective function and eventually converge, at least, to a stationary point.
Interestingly, this approach is able to naturally induce a trust region, mitigating the need for an explicit penalization or constraint.
The experimental evaluation conﬁrmed our theoretical ﬁndings.
PO 2 PE is able to outperform POIS and TRPO on several continuous control tasks.
Remarkably, our algorithm has proved to be robust to the reduction of the batch size and this represents a beneﬁcial effect of the implicit trust region enforcement.
We believe that this work contributes to shed light on an appealing facet of off-policy learning with possible new research opportunities.
Future works include an extension of the convergence analysis to the case in which samples are involved and an experimentation of PO 2 PE coupled with more complex policy architectures.
[-First of all, we observe that since h is monotonically strictly-increasing it holds that V ar x „ P r f p x qs“ 
-]
[-Let us ﬁrst observe that for λ “ 0 , we have Q 
-]