A Boosting Approach to Reinforcement Learning
Abstract
Reducing reinforcement learning to supervised learning is a well-studied and effective approach that leverages the beneﬁts of compact function approximation to deal with large-scale Markov decision processes.
Independently, the boosting methodology (e.g. AdaBoost) has proven to be indispensable in designing efﬁcient and accurate classiﬁcation algorithms by combining inaccurate rules-of-thumb .
In this paper, we take a further step: we reduce reinforcement learning to a sequence of weak learning problems.
Since weak learners perform only marginally better than random guesses, such subroutines constitute a weaker assumption than the availability of an accurate supervised learning oracle.
We prove that the sample complexity and running time bounds of the proposed method do not explicitly depend on the number of states.
While existing results on boosting operate on convex losses, the value function over policies is non-convex.
We show how to use a non-convex variant of the
Frank-Wolfe method for boosting, that additionally improves upon the known sample complexity and running time even for reductions to supervised learning.
Introduction
In reinforcement learning, Markov decision processes (MDP) model the mechanism of learning from rewards, as opposed to examples.
Although the case of tabular MDPs is well understood, the main challenge in applying RL in the real-world is the size of the state space in practical domains.
This challenge of ﬁnding efﬁcient and provable algorithms for MDPs with large state space is the focus of our study.
Various techniques have been suggested and applied to cope with very large
MDPs.
One class of approaches attempts to approximate either the value or the transition function of the underlying MDP by using a parametric function class.
Such approaches invariably make strong realizability assumptions to produce global optimality guarantees.
Another class of approaches, so-called direct methods, produces a near-optimal policy that maximizes the expected return from a given policy class.
To deal with the challenge of large (possibly innumerable) policy classes, a popular strategy [23] is to the frame policy search as a sequence of supervised learning problems.
Such approaches yield global optimality guarantees under state coverage assumptions without reliance on realizability, and have inspired practical adaptations for sampling-based policy search.
In this paper, we study another methodology to derive provable algorithms for reinforcement learning: ensemble methods for aggregating weak or approximate algorithms into substantially more accurate solutions.
Our proposal extends the methodology of boosting, typically used to solve supervised learning instances [31], to reinforcement learning.
A typical boosting algorithm (e.g. AdaBoost)
[Table] [Table]
iteratively constructs a near-optimal classiﬁer by combining computationally cheap, yet inaccurate rules-of-thumb . Unlike RL reductions to supervised learning which assume the existence of an efﬁcient and accurate classiﬁcation or regression procedure, the proposed algorithms builds on learning algorithms that perform only ever-so-slightly better than a random guess, and which thus may be produced cheaply both in computational and statistical terms.
Concretely, we assume access to a weak learner: an efﬁcient sample-based procedure that is capable of generating an approximate solution to any weighted multi-class objective over a ﬁxed policy class.
We describe an algorithm that iteratively calls this procedure on carefully constructed new objectives, and aggregates the solution into a single policy.
We prove that after sufﬁciently many iterations, our resulting policy has competitive global gurantees on performacnce.
Interestingly, unlike boosting algorithms for regression and classiﬁcation, our resulting aggregation of weak learners is non-linear.
1.1 Challenges and techniques
Reinforcement learning is quite different from supervised learning and several difﬁculties have to be circumvented for boosting to work.
Among the challenges that the reinforcement learning setting presents, consider the following,
(a) The value function is not a convex or concave function of the policy.
This is true even in the tabular case, and even more so if we use a parameterized policy class.
(b) The transition matrix is unknown, or prohibitively large to manipulate for large state spaces.
This means that even evaluation of a policy cannot be exact, and can only be computed approximately.
(c) It is unrealistic to expect a weak learner that attains near-optimal value for a given linear objective over the policy class.
At most one can hope for a multiplicative and/or additive approximation of the overall value.
Our approach overcomes these challenges by applied several new as well as recently developed techniques.
To overcome the nonconvexity of the value function, we use a novel variant of the
Frank-Wolfe optimization algorithm that simultaneously delivers on two guarantees.
First, it ﬁnds a ﬁrst order stationary point with near-optimal rate.
Secondly, if the objective happens to admit a certain gradient domination property, an important generalization of convexity, it also guarantees near optimal value.
The application of the nonconvex Frank-Wolfe method is justiﬁed due to previous recent investigation of the policy gradient algorithm [2, 1], which identiﬁed conditions under which the value function is gradient dominated.
The second information-theoretic challenge of the unknown transition function is overcome by careful algorithmic design: our boosting algorithm requires only samples of the transitions and rewards, obtained by rollouts on the MDP.
The third challenge is perhaps the most difﬁcult to overcome.
Thus far, the use of the Frank-Wolfe method in reinforcement learning did not include a multiplicative approximation, which is critical for our application.
We adapt the techniques used for boosting in online convex optimization [18] with a multiplicative weak learner to our setting, by non-linearly aggregating (using a 2-layer network) the weak learners.
This aspect is perhaps of general interest to boosting algorithm design, which is mostly based on linear aggregation.
1.2 Our contributions
Our main contribution is a novel efﬁcient boosting algorithm for reinforcement learning.
Our techniques apply in various settings and the sample complexity bounds of all of our results are summarized in Tables 1 and 2.
The input to this algorithm is a weak learning method capable of approximately solving a weighted multi-class problem instance over a certain policy class.
The output of the algorithm is a policy which does not belong to the original class considered, hence being an instance of improper learning.
It is rather a non-linear aggregation of policies from the original class, according to a two-layer neural network.
This is a result of the two-tier structure of our algorithm: an outer loop of non-convex
Frank-Wolfe method, and an inner loop of online convex optimization based boosting.
The ﬁnal policy comes with provable global optimality guarantees.
Beyond novelty of techniques, an important contribution (Table 1) of our work is to highlight the quantitative difference in guarantees that depend on the mode of accessing the MDP (episodic rollouts vs. access to an exploratory reset distrbution) and the nature of the weak learners (online vs statistical), thus indicating that some algorithmic choices may be preferable compared to others in terms of speed of convergence and sample complexity.
As with existing reductions to supervised learning [23], these global convergence guarantees happen under appropriate state coverage assumptions either via access to a reset distribution that has some overlap with the state distribution of the optimal policy, or by constraining the policy class to policies that explore sufﬁciently.
Yet another contribution of our work is to show an improved sample complexity result in the latter setting, even when considering reductions to supervised learning instances .
This improvement in convergence in well-studied settings is documented in Table 2.
1.3 Related work
Reinforcement learning approaches for dealing with large-scale MDPs rely on function approximation [34].
Such function approximation may be performed on the underlying conditional probability of transition (e.g. [33, 20]) or the value function (e.g. [36, 35]).
The provable guarantees in such methods come at the cost of strong realizability assumptions.
In contrast, the so-called direct approaches attempt policy search over an appropriate policy class [2, 1], and rely on making making incremental updates, such as Conservative Policy Iteration (CPI) [23, 32], and Policy Search by
Dynamic Programming (PSDP)[5].
These provide convergence guarantees under appropriate state coverage assumptions comparable to ones made in this work.
Our boosting approach for provable RL builds on the vast literature of boosting for supervised learning [31], and recently online learning [26, 11, 12, 6, 21, 22].
One of the crucial techniques important for our application is the extension of boosting to the online convex optimization setting, with bandit information [9], and critically with a multiplicative weak learner [18].
This latter technique implies a non-linear aggregation of the weak learners.
Non-linear boosting was only recently investigated in the context of classiﬁcation [4], where it was shown to potentially enable signiﬁcantly more efﬁcient boosting.
Another work on boosting in the context of control of dynamical systems [3].
However, this work critically requires knowledge of the underlying dynamics (transitions) and makes convexity assumptions, which we do not, and cannot cope with a multiplicative approximate weak learner.
The Frank-Wolfe algorithm is extensively used in machine learning, see e.g. [19], references therein, and recent progress in stochastic Frank-Wolfe methods [15, 27, 10, 38].
Recent literature has applied a variant of this algorithm to reinforcement learning in the context of state space exploration [17].
Preliminaries
Optimization.
We say that a differentiable function f : K (cid:55)→ R over some domain
K ⊂ R d is L -smooth with respect to some norm (cid:107) · (cid:107) ∗ if for every x, y ∈ K we have (cid:12)(cid:12)
f
( y )
− f ( x ) − ∇ f ( x ) (cid:62) ( y − x ) (cid:12)(cid:12) ≤
L 2 (cid:107) x − y (cid:107) 2 ∗ .
We deﬁne the projection Γ : R | A | → ∆ A , with respect to a set A , where ∆ A denotes the probability simplex over A .
For any x ∈ R | A | , Γ[ x ] = arg min y ∈
∆
A (cid:107) x − y (cid:107) .
An important generalization of the property of convexity we use henceforth is that of gradient domination.
Deﬁnition 1 (Gradient Domination) .
A function f : K → R is said to be ( κ, τ, K 1 , K 2 ) -locally gradient dominated (around K 1 by K 2 ) if for all x ∈ K 1 , it holds that
[Equation]
Markov decision process.
An inﬁnite-horizon discounted Markov Decision Process (MDP)
M = ( S, A, P, r, γ, d 0 ) is speciﬁed by: a state space S , an action space A , a transition model P where
P ( s (cid:48) | s, a ) denotes the probability of immediately transitioning to state s (cid:48) upon taking action a at state s , a reward function r : S
×
A → [0 , 1] where r ( s, a ) is the immediate reward associated with taking action a at state s , a discount factor γ ∈ [0 , 1) ; a starting state distribution d 0 over S .
For any inﬁnite-length state-action sequence (hereafter, called a trajectory), we assign the following value V ( ς = ( s 0 , a 0 , s 1 , a 1 , . . . ))
= (cid:80) ∞ t =
γ t r ( s t , a t ) .
The agent interacts with the MDP through the choice of stochastic policy π : S → ∆ A it executes.
The execution of such a policy induces a distribution over trajectories ς
= ( s 0 , a 0 , . . . )
as P ( ς | π )
=
d 0 ( s 0 ) (cid:81) ∞ t =
( P ( s t +1 | s t , a t ) π ( a t | s t )) .
Using this description we can associate a state V π ( s ) and state-action Q π ( s, a ) value function with any policy π .
For an arbitrary distribution d over S , deﬁne:
[Equation]
Here the expectation is with respect to the randomness of the trajectory induced by π in M .
When convenient, we shall use V π to denote V πd 0 , and V ∗ to denote max π V π .
Similarly, to any policy π , one may ascribe a (discounted) state-visitation distribution d
π = d πd 0 .
[Equation]
Modes of Accessing the MDP.
We henceforth consider two modes of accessing the MDP, that are standard in the reinforcement learning literature, and provide different results for each.
The ﬁrst natural access model is called the episodic rollout setting.
This mode of interaction allows us to execute a policy, stop and restart at any point, and do this multiple times.
Another interaction model we consider is called rollout with ν -restarts.
This is similar to the episodic setting, but here the agent may draw from the MDP a trajectory seeded with an initial state distribution ν (cid:54) =
d 0 .
This interaction model was considered in prior work on policy optimization [23, 2].
The motivation for this model is two-fold: ﬁrst, ν can be used to incorporate priors (or domain knowledge) about the state coverage of the optimal policy; second, ν provides a mechanism to incorporate exploration into policy optimization procedures.
2.1 Weak learning
Our boosting algorithms henceforth call upon weak learners to generate weak policies.
We formalize the notion of a weak learner next.
We consider two types of weak learners, and give different end results based on the different assumptions: weak supervised and weak online learners.
In the discussion below, let π Rand be a uniformly random policy, i.e. ∀ ( s, a ) ∈ S
× A, π Rand ( a | s ) = 1 / | A | .
The formal deﬁnition and results for the online setting are deferred to the appendix.
In what follows we deﬁne the supervised weak learning model.
The natural way to deﬁne weak learning is an algorithm whose performance is always slight better than that of random policy, one that chooses an action uniformly at random at any given state.
However, in general no learner can outperform a random learner over all label distributions.
This motivates the literature on agnostic boosting [24, 8, 18] that deﬁnes a weak learner as one that can approximate the best policy in a given policy class.
Deﬁnition 2 (Weak Supervised Learner) .
Let α ∈ (0 , 1] .
Consider a class L of linear loss functions (cid:96) :
R A
→ R , a family D of distributions that are supported over S × L , and policy class Π .
A weak supervised learning algorithm, for every ε, δ > 0 , given m ( ε, δ ) = log |W| εlog 1 δ samples D m from any distribution D ∈ D outputs a policy W ( D
m ) ∈
Π such that with probability 1 − δ ,
[Equation]
Note that the weak learner outputs a policy in Π which is approximately competitive against the class Π .
As an additional relaxation, instead of requiring that the weak learning guarantee holds for all distributions, in our setup, it will be sufﬁcient that the weak learning assumption holds over natural distributions.
Speciﬁcally, we deﬁne a class of natural distributions D , such that D ∈ D if and only if there exists some π ∈ Π such that, D ( s ) =
(cid:82) (cid:96) D ( s, (cid:96) )
dµ ( (cid:96) )
=
d π ( s ) .
In particular, while a natural distribution may have arbitrary distribution over labels, its marginal distribution over states must be realizable as the state distribution of some policy in Π over the MDP M .
Therefore, the complexity of weak learning adapts to the complexity of the MDP itself.
As an extreme example, in stochastic contextual bandits where policies do not affect the distribution of states (say d 0 ), it is sufﬁcient that the weak learning condition holds with respect to all couplings of a single distribution d 0 .
Algorithm & Main Results
In this section we describe our RL boosting algorithm.
Here we focus on the case where a supervised weak learning is provided.
The online weak learners variant of our result is detailed in the appendix.
We next deﬁne several deﬁnitions and algorithmic subroutines required for our method.
3.1 Policy aggregation
For a base class of policies Π , our algorithm incrementally builds a more expressive policy class by aggregating base policies via both linear combinations and non-linear transformations.
In effect, the algorithm produces a ﬁnite-width depth-2 circuit over some subset of the base policy class.
That is, our approach can be thought of as an aggregation of base policies, which forms a 2-layer neural network, as depicted in Figure 1.
The leaves of the tree are the policies π ∈ Π the base policy class.
These are then linearly aggregated to form the ﬁrst layer of the tree, denoted ˜ π 1 , ˜ π 2 in Figure 1.
Next, each linear combination of policies in the overall aggregation undergoes a projection operation.
The projection may be viewed as a non-linear activation function, such as ReLU, in deep learning terms.
Note that the projection of any function from S to R | A | produces a policy, i.e. a mapping from states to distributions over actions.
In the analysis of our algorithm we give a particular projection operation Γ[ · ] which allows us to yield the desired guarantees.
Deﬁnition 3 (Policy Projection) .
Given ˜ π : S → R | A | , deﬁne a projected policy π = Γ[˜ π ] to be a policy such that simultaneously for all s ∈ S , it holds that π ( ·| s ) =
Γ [˜ π ( s )] .
Deﬁnition 4 (Policy Tree) .
A Policy Tree Π ⊆ S →
∆ A with respect to Π ⊆ S → ∆ A some base policy class, and N, T ∈ N , is a linear combination of T projected policies Γ[˜ π ] , where each ˜ π is a linear combination of N base policies π ∈ Π .
This ﬁnal deﬁnition describes the set of possible outputs of the boosting procedure.
It is important that the policy that the boosting algorithm outputs can be evaluated efﬁciently.
In the appendix we show it is indeed the case (see Lemma 12).
Hereafter, we refer to a Policy Tree with respect to Π , N and T , as Π for N, T =
O (poly( | A | , (1 − γ ) − 1 , ε − 1 , α − 1 , log δ − 1 ))
speciﬁed later.
3.2 Main results
Next, we give the main results of our RL boosting algorithm via weak supervised learning, speciﬁed in Algorithm 1.
¯ π Γ[˜ π 1 ] ˜ ππππΓ[˜ π 2 ] ˜ ππππ
A Policy Tree ¯ π
To state the results, we need the following deﬁnitions.
The ﬁrst generalizes the policy completeness notion from [32].
It may be seen as the policy-equivalent analogue of inherent bellman error [28].
Intuitively, it measures the degree to which a policy in Π can best approximate the bellman operator in an average sense with respect to the state distribution induced by a policy from Π .
Deﬁnition 5 (Policy Completeness) .
For any initial state distribution µ , and policy classes Π , Π , deﬁne [Equation] [Equation] π ∗
[Equation]
Deﬁnition 6 (Distribution Mismatch) .
Let π ∗ =
arg max π V π , and ν a ﬁxed initial state distribution (see section 2).
Deﬁne the following distribution mismatch coefﬁcients: C ∞
[Equation]
The above notion of the distribution mismatch coefﬁcient is often useful to characterize the exploration problem faced by policy optimization algorithms.
We now give the main result for the output of our
RL boosting algorithm, assuming supervised weak learners.
Theorem 7.
Algorithm 1 samples T ( MN + P ) episodes of length ˜ O
( 11 − γ ) with probability 1 − δ .
In the episodic model, with µ = d 0 , for η 1 ,t = min { 1 , 2 C ∞ t } , T = O (cid:16) C 2 ∞ (1 − γ ) 3 ε (cid:17) , N = (cid:16) 16 | A | C ∞ (1 − γ ) 2 α(cid:15) (cid:17) 2 ,
M = m (cid:16) (1 − γ ) 2 αε C ∞ | A | , δ NT (cid:17) , with probability 1 − δ , V ∗ − V π ≤ C ∞ E 1 − γ + ε.
In the ν -reset model, with µ = ν , for η 1 ,t = (cid:113) 8 γ (1 − γ ) 2 | A | 2 T , T = 8 D 2 ∞ (1 − γ ) 6 ε 2 , N = (cid:16)
| A | D ∞ (1 − γ ) 3 α(cid:15) (cid:17) 2 ,
M = m (cid:16) (1 − γ ) 3 αε 8 | A | D ∞ , δ 2 NT (cid:17) , with probability 1 − δ , V ∗ − V π ≤ D ∞ E ν (1 − γ ) 2 + ε.
Sample complexities:
If m ( ε, δ ) = log |W| εlog 1 δ for some measure of weak learning complexity |W| ,
[Equation]
[Equation] [Equation] [Equation]
where
Q i A
[Equation]
vector (cid:100) Q πs h , such that for all a ∈ A , (cid:100) Q πs h ( a ) =
| A | · R ( s h , a (cid:48) )
· I a = a (cid:48) .
5: return ( s h , (cid:100) Q πs h ) .
the algorithm samples ˜ O (cid:16) C 6 ∞ | A | 4 log |W| (1 − γ )
α 4 ε 5 (cid:17) episodes in the episodic model, and ˜ O (cid:16) D 6 ∞ | A | 4 log |W| (1 − γ )
α 4 ε 6 (cid:17) in the ν -reset model.
Theorem 7 above pertains to the case where a weak learning algorithm is available.
However, another main result is given by considering the simpler approach of reduction of RL to a strong supervised learning algorithm.
In particular, when running our main boosting algorithm, we can replace the call to Internal Boost (in Line 4 of Algorithm 1) with a call to a strong supervised learning algorithm.
By a similar analysis to that of Theorem 7 we obtain the following corollary.
Corollary 8.
Let m ( ε, δ ) = log |W| εlog 1 δ for some measure of weak learning complexity |W| .
When run with a supervised learning oracle (Deﬁnition 2 with α = 1 , i.e. N = 1 ) as the Internal boosting,
Algorithm 1 samples ˜ O (cid:16) C 3 ∞ log |W| ε(cid:17) episodes in the episodic model, and ˜ O (cid:16) D 4 ∞ log |W| ε(cid:17) in the ν -reset model, to guarantee V ∗ − V π ≤ C ∞ E 1 − γ + ε with probability 1 − δ in the episodic model and
V ∗ − V π ≤ D ∞ E ν (1 − γ ) 2 + ε in the ν -reset model.
We note that this result is an improvement over previous results in terms of sample complexity requirement of the algorithm.
In particular, in [23], Theorem 4.4 and Corollary 4.5 achieve the same guarantee using O (1 /ε 4 ) samples regardless of the MDP access model.
Brieﬂy, CPI utilizes 1 /ε 2 calls to an ε -optimal supervised learning oracle (each call needing 1 /ε 2 samples) to reach a ε -local optima of the value function.
Under requisite state coverage assumptions, this translates to ε -function value suboptimality.
Indeed, such mode of analysis via ﬁrst arguing for convergence to a local optima for the CPI algorithm can be shown to be tight.
The improvement in our case for the episodic access model comes from the insight that it is possible to make direct claims on the function value sub-optimality (second part of Theorem 9), bypassing the need for making a claim on the local optimality, in the gradient-dominated case.
3.3 Trajectory sampler
In Algorithm 3 we describe an episodic sampling procedure, that is used in our sample-based RL boosting algorithms described above.
For a ﬁxed initial state distribution µ , and any given policy π , we apply the following sampling procedure: start at an initial state s 0 ∼ µ , and continue to act thereafter in the MDP according to any policy π , until termination.
With this process, it is straightforward to both sample from the state visitation distribution s ∼ d π , and to obtain unbiased samples of Q π ( s, · ) ; see Algorithm 3 for the detailed process.
Sketch of the analysis
Non-convex Frank-Wolfe.
We give an abstract high-level procedural template that the previously introduced RL boosters operate in.
This is based on a variant of the Frank-Wolfe optimization technique [14], adapted to non-convex and gradient dominated function classes (see Deﬁnition 1).
The Frank-Wolfe (FW) method assumes oracle access to a black-box linear optimizer, denoted O , and utilizes it by iteratively making oracle calls with modiﬁed objectives, in order to solve the harder task of convex optimization.
Analogously, boosting algorithms often assume oracle access to a ”weak” learner, which are utilized by iteratively making oracle calls with modiﬁed objective, in order to obtain a ”strong” learner, with boosted performance.
In the RL setting, the objective is in fact non-convex, but exhibits gradient domination.
By adapting Frank-Wolfe technique to this setting, we will in subsequent section obtain guarantees for the algorithms given in Section 3.
Oracle:
Denote by
O a black-box oracle to an ( (cid:15) 0 , K 2 ) -approximate linear optimizer over a convex set K ⊆ R d such that for any given v ∈ R d , we have v (cid:62) O ( v ) ≥ max u ∈K 2 v (cid:62) u − (cid:15) 0 . [Table]
[Table]
O ( ∇ t − 1 ) , where ∇ t − 1 = ∇ f ( x t − 1 ) . Set x t = (1 − η t ) x t − 1 + η t z t .
5: end for 6: return x := x t (cid:48) where t (cid:48) = arg min t ∇ (cid:62) t ( z t − x t ) .
Theorem 9.
Let f : K
→ R be L -smooth in some norm (cid:107) · (cid:107) ∗ , bounded for all x ∈ K , | f ( x ) | ≤ H for some H > 0 , and let the diameter of K in (cid:107) · (cid:107) ∗ be D .
Then, for a ( (cid:15) 0 , K 2 ) -linear optimization oracle O , and η t = η = (cid:113) 4 H LD 2 T , the output ¯ x of Algorithm 4 satisﬁes
[Equation]
Furthermore, if f is ( κ, τ, K 1 , K 2 ) -locally gradient-dominated and x 0 , . . .
x T ∈ K 1 , then the output ¯ x of Algorithm 4 where η t =
min { 1 , 2 κt } satisﬁes the bound on the right.
We sketch the high-level ideas of the proof of our main result, stated in Theorem 7, and refer the reader to the appendix for the formal proof.
We will establish an equivalence between RL Boosting (Algorithm 1) and the variant of the Frank-Wolfe algorithm (Algorithm 4).
This abstraction allows us to obtain the novel convergence guarantees given in Theorem 7.
Throughout the analysis, we use the notation ∇ π V π to denote the gradient of the value function with respect to the | S | × | A | -sized representation of the policy π , namely the functional gradient of V π .
Internal-boosting weak learners.
The Frank-Wolfe algorithm utilizes an inner gradient optimization oracle as a subroutine.
To implement this oracle using approximate optimizers, we utilize yet another variant of the FW method as “internal-boosting” for the weak learners, by employing an adapted analysis of [18] that is stated in Claim 10 below.
Let D t be the distribution induced by the trajectory sampler in round t .
Claim 10.
Let β = (cid:112) 1 /αN , η 2 ,n = min { 2 /n, 1 } .
π (cid:48) t produced by Algorithm 1 satisﬁes
[Equation]
From weak learning to linear optimization, Next, we give an important observation which allows us to re-state the guarantee in the previous subsection in terms of linear optimization over functional gradients.
The key observation here is that the expensive optimizing procedure for ( ∇ π V π ) (cid:62) π (cid:48) , which in particular requires iterating over all states in S , can be instead replaced with sampling from an appropriate distribution D (via Algorithm 3).
These sample pairs ( s, (cid:99) Q π ( s, · )) could then be fed to our weak learning algorithm, which guarantees generalization.
Lemma 11.
Applying Algorithm 3 for any given policy π yields an unbiased estimate of the gradient, such that for any π (cid:48) , ( ∇ π V πµ ) (cid:62) π (cid:48) =
E ( s, (cid:99) Q π ( s, · ))
∼D (cid:104) (cid:99)
Q π ( s, · ) (cid:62) π (cid:48) ( ·| s ) (cid:105) / (1 − γ ) , where π (cid:48) ( ·| s ) ∈
∆ A ,
D is the distribution induced on the outputs of Algorithm 3, for the policy π and initial distribution µ .
Experiments
The primary contribution of the present work is theoretical.
Nevertheless, we empirically test our proposal with the experiment designed to elicit qualitative properties of the proposed algorithm, instead of aiming to achieve the state-of-the-art.
To validate our results, we check if the proposed algorithm is indeed capable of boosting the accuracy of concrete instantiations of weak learners.
We use depth-3 decision trees, with the implementation adapted from Scikit-Learn [29], as our base weak learner.
This choice of weak learner is particularly suitable for boosting, because it is an impoverished policy class in a representational sense and hence it is reasonable to expect that it may do only slightly better than random guessing with respect to the classiﬁcation loss.
We consider the performance of the boosting algorithm (Algorithm 1) across multiple rounds of boosting or number of weak learners to that of supervised-learning-based policy iteration; the computational burden of the algorithm scales linearly with the latter.
Throughout all the experiments, we used η = 0 .
To speed up computation, the plots below were generated by retaining the 3 most recent policies of every iteration in the policy mixture.
We evaluated these on the CartPole and the LunarLander environments.
The results demonstrate the proposed RL boosting algorithm succeeds in maximizing rewards while using few weak learners (equivalently, within a few rounds of boosting).
Conclusions
Building on recent advances in boosting for online convex optimization and bandits, we have described a boosting algorithm for reinforcement learning over large state spaces with provable guarantees.
We see this as a ﬁrst attempt at using a tried-and-tested methodology from supervised learning to RL.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] in the supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] A Notation: List of Symbols
Weak Learning and Boosting
α
Weak learning parameter
T
Number of boosting iterations
N
Number of internal-boosting iterations
M
Number of internal-boosting episodes Γ[ · ]
Policy projection Π
Policy class Π
Policy-Tree class (w.r.t Π , Γ , N and T )
Markov Decision Process
S
State space
A
Action space ∆
A
Probability simplex over actions
Q π ( s, a )
Q function
V π ( s )
Value function d ( s 0 )
Initial state distribution d πd ( s )
State-visitation distribution w.r.t π, d γ
Discount factor
E µ ( Π , Π)
Policy completeness µ, ν
Used for different initial state distributions
C ∞
Distribution mismatch if µ = d
D ∞
Distribution mismatch if µ = ν (cid:54) =
d
Optimization
K
Decision set
L
Smoothness of the objective
H
Upper bound on the range of function value
D
Upper bound on Euclidean diameter
B Appendix
It is important that the policy that the boosting algorithm outputs can be evaluated efﬁciently.
Towards that end, we give the following claim.
Claim 12.
For any π ∈ Π (Π , N, T ) , π ( ·| s ) for any s ∈ S can be evaluated using TN base policy evaluations and O ( T × ( NA + A log A )) arithmetic and logical operations.
Proof.
Since π ∈ Π (Π , N, T ) , it is composed of TN base policies.
Producing each aggregated function takes NA additions and multiplications; there are T of these.
Each projection takes time equivalent to sorting | A | numbers, due to a water-ﬁlling algorithm [13]; these are also T in number.
The ﬁnal linear transformation takes an additional TA operations. C RL Boosting via Weak Online Learning The second model of weak learning we consider requires a stronger assumption, but will give us better sample and oracle complexity bounds henceforth. Deﬁnition 13 (Weak Online Learner) .
Let α ∈ (0 , 1) .
Consider a class L of linear loss functions (cid:96) :
R A
→ R .
A weak online learning algorithm, for every M > 0 , incrementally for each timestep computes a policy W m ∈ Π and then observes the state-loss pair ( s, (cid:96) t )
∈ S ×
L such that
M
[Equation]
=
[Equation]
Assumption 1 (Weak Online Learning) .
The booster has access to a weak online learning oracle (Deﬁnition 13) over the policy class Π , for some α ∈ (0 , 1) .
Remark 14.
A similar remark about natural distributions applies to the online weak learner.
In particular, it is sufﬁcient the guarantee in 13 holds for arbitrary sequence of loss functions with high probability over the sampling of the state from d π for some π ∈ Π .
Although stronger than supervised weak learning, this oracle can be interpreted as a relaxation of the online weak learning oracle considered in [8, 9, 18].
A similar model of hybrid adversarial-stochastic online learning was considered in [30, 25, 7].
In particular, it is known [25] that unlike online learning, the capacity of a hypothesis class for this model is governed by its VC dimension (vs. Littlestone dimension).
1: Initialize a policy π 0 ∈ Π arbitrarily.
2: for t = 1 to T do 3: Initialize online weak learners W 1 , . . .
W N . 4: for m = 1 to M do 5: Execute π t − 1 once with initial
state distribution µ via Algorithm 3, to get ( s t,m , (cid:98) Q t,m ) .
Choose ˜ π t,m, 0 ∈ Π arbitrarily.
7: for n = 1 to N do 8:
Set ˜ π t,m,n = (1 − η 2 ,n )˜
π t,m,n − 1 + η 2 ,n α W n .
9: end for 10:
Pass to each W n the following loss linear f t,m,n : [Equation] [Equation] [Equation]
s i )) .
where G =
A 1 − γ , β = 2 γ (1 − γ ) 3 and f i , (cid:98) Q i ∈ R | A | y i =
arg min
y ∈
∆ A {− (cid:98) Q (cid:62)
[Equation]
,m y + G min z
[Equation]
∆ A (cid:107)
[Equation]
y (cid:107) + (cid:107)
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
β } 11: end for 12:
Declare π (cid:48) t = 1 M (cid:80)
Mm =1 Γ [˜ π t,m,N ] .
Choose η 1 ,t = min { 1 , 2 C ∞ t } if µ = d 0 else set η 1 ,t = (cid:113) 8 γ (1 − γ ) 2 | A | 2 T .
Update π t
=
(1 − η 1 ,t ) π t − 1 + η 1 ,t π (cid:48) t .
15: end for 16: Run each policy π t for P rollouts to compute an empirical estimate (cid:100)
V π t of the expected return.
17: return ¯ π := π t (cid:48) where t (cid:48) =
arg max t (cid:100) V π t .
Theorem 15.
Algorithm 5 samples TM episodes of length 11 − γ log TMδ with probability 1 − δ .
In the episodic model, Algorithm 5 guarantees as long as T = 16 C 2 ∞
[Table]
[Figure]
[Equation]
[Table]
[Figure]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Table]
[Figure]
[Equation]
[Table]
[Figure]
[Table]
[Table]
, we have with probability
[Table]
δ
[Equation]
{+In the ν -reset model, Algorithm 1 guarantees as long as T = 100 D 2 ∞ (1 − γ ) 6 ε 2 , N = (cid:16) 20 | A | D ∞ (1 − γ ) 3 α(cid:15) (cid:17) 2 , M = max (cid:26)(cid:16) 
+}
| A | D ∞
[Equation]
[Equation]
[Equation]
log Tδ (cid:17) 2 , 10 |
[Equation]
D
[Equation]
M )
[Equation]
[Equation]
[Equation]
(cid:27) , µ = ν , we have with probability 1 − δ
V
[Equation]
If R W ( M ) = (cid:112) M log |W| for some measure of weak learning complexity |W| , the algorithm samples
˜ O (cid:16) C 4 ∞
| A | 2 log |W| (1 − γ ) 7 α 2 ε 3 (cid:17) episodes in the episodic model, and ˜ O (cid:16) D 4 ∞ | A | 2 log |W| (1 − γ )
α 2 ε 4 (cid:17) in the ν -reset model.
D Analysis for Boosting with Weak Supervised Learning (Proof of Theorem 7)
Theorem (Formal version of Theorem 7) .
Algorithm 1 samples TMN episodes of length 11 − γ log TMNδ with probability 1 − δ .
{+In the episodic model, Algorithm 1 guarantees as long as T = 16 C 2 ∞ ) (1 − γ ) 3 ε , N = (cid:16) 
+}
| A | C ∞
[Equation]
1 − γ ) 2 α(cid:15) (cid:17) 2 , M = m (cid:16) (1 − γ ) 2 αε 8 C ∞
| A | , δNT (cid:17) , µ = d 0 , we have with probability 1 − δ
[Equation]
In the ν -reset model, Algorithm 1 guarantees as long as T =
[Equation]
2 , M = m (cid:16) (1 − γ ) 3 αε 8 | A | D ∞ , δ 2 NT (cid:17) , µ = ν , we have with probability 1 − δ
[Equation]
If m ( ε, δ ) = log |W| εlog 1 δ for some measure of weak learning complexity |W| , the algorithm samples ˜ O (cid:16) C 6 ∞
{+| A | 4 log |W| (1 − γ ) 
+}
{+α 4 ε 5 (cid:17) episodes in the episodic model, and ˜ O (cid:16) D 6 ∞ | A | 4 log |W| (1 − γ ) 
+}
α 4 ε 6 (cid:17) in the ν -reset model.
Proof of Theorem 7.
The broad scheme here is to utilize an equivalence between Algorithm 1 and
Algorithm 4 on the function V π (or V πν in the ν -reset model), to which Theorem 9 applies.
To this end, ﬁrstly, note V π is 11 − γ -bounded.
Deﬁne a norm (cid:107) · (cid:107) ∞ , 1 : R | S |×| A | → R as (cid:107) x (cid:107) 1 , ∞ = max s ∈ S (cid:80) a ∈ A | x s,a | .
Further, observe that for any policy π : S → ∆ A , (cid:107) π (cid:107) ∞ , 1 = 1 .
The following lemma speciﬁes the smoothness of V π in this norm.
Lemma 16.
V π is
[Equation]
To be able to interpret Algorithm 1 as an instantiation of the algorithmic template Algorithm 4 presents, we need to show that π (cid:48) t (Line 3-10) serves as an approximate linear optimizer for ∇ V π t − 1 .
This will imply that the iterates produced by the two algorithms coincide.
Indeed, Claim 17 demonstrates that π (cid:48) t serves a linear optimizer over gradients of the function V π ; the suboptimality speciﬁes (cid:15) 0 .
Claim 17.
Let β = (cid:113) 1 αN , and η 2 ,n = min { 2 n , 1 } .
Then, for any t , π (cid:48) t produced by Algorithmsatisﬁes with probability 1 − δ
[Equation]
Finally, observe that it is by construction that π t ∈ Π .
{+Therefore, in terms of the previous section, K is the class of all policies, K 
+}
= Π , K 2 = Π .
In the episodic model, we wish to invoke the second part of Theorem 9.
The next lemma establishes gradient-domination properties of V π to support this.
Lemma 18.
V π is (cid:16) C ∞ , 11 − γ C ∞
E (Π , Π ) , Π , Π (cid:17) -gradient dominated, i.e. for any π ∈ Π :
[Equation]
Deriving κ, τ from the above lemma along with (cid:15) 0 from Claim 17, as a consequence of the second part of Theorem 9, we have with probability 1 − NTδ
[Equation]
Similarly, in the ν -reset model, the ﬁrst part of Theorem 9 provides a local-optimality guarantee for
V πν .
Lemma 19 provides a bound on the function-value gap (on V π ) provided such local-optimality conditions.
Lemma 19.
For any π ∈ Π , we have
[Equation]
Again, using the bound on max π (cid:48) ∈ Π ( ∇ V ¯ πν ) (cid:62) ( π (cid:48) − ¯ π ) Theorem 9 provides, we have that with probability 1 − 2 NTδ
[Equation]
δ
E Analysis for Boosting with Weak Online Learning (Proof of Theorem 15)
Proof of Theorem 15.
Similar to the proof of Theorem 7, we establish an equivalence between
Algorithm 1 and Algorithm 4 on the function V π (or V πν in the ν -reset model), to which Theoremapplies provided smoothness (see Lemma 16).
Indeed, Claim 20 demonstrates π (cid:48)
t serves a linear optimizer over gradients of the function V π , and provides a bound on (cid:15) 0 .
As before, observe that it is by construction that π t ∈ Π .
Claim 20.
Let β = (cid:113) 1 αN , and η 2 ,n = min { 2 n , 1 } .
Then, for any t , π (cid:48) t produced by Algorithmsatisﬁes with probability 1 − δ
[Equation]
In the episodic model, one may combine the second part of Theorem 9, which provides a bound on function-value gap for gradient dominated functions, which Lemma 18 guarantees, to conclude with probability 1 − Tδ [Equation] Similarly, in the ν -reset model, Lemma 19 provides a bound on the function-value gap provided local-optimality conditions, which the ﬁrst part of Theorem 9 provides for.
Again, with probability 1 − Tδ
[Equation] F Proofs of Supporting Claims F.1 Guarantees on the sampling algorithm
Proof of Lemma 11.
Recall ∇ π V π denotes the gradient with respect to the | S | × | A | -sized representation of the policy π – the functional gradient. Then, using the policy gradient theorem [37, 34], it is given by,
[Equation]
The following sources of randomness are at play in the sampling algorithm (Algorithm 3): the distribution d π (which encompasses the discount-factor-based random termination, the transition probability, and the stochasticity of π ), and the uniform sampling over A .
For a ﬁxed s, π , denote by
Q πs as the distribution over (cid:99) Q π ( s, · ) ∈ R A , induced by all the aforementioned randomness sources.
To conclude the claim, observe that by construction
[Equation]
F.2 Non-convex Frank-
Wolfe method (Theorem 9)
Proof of Theorem 9.
Non-convex general case.
Note that for any timestep t , it holds due to smoothness that
[Equation]
[Equation]
Let t (cid:48) =
arg min t f ( x t ) − f ( x t − 1 ) .
Note that by telescoping over function-value differences across successive iterates, we get
[Equation]
Combining with (4), and plugging in η , we get
[Equation]
To conclude the claim for the non-convex general case, observe that since z t (cid:48)
=
O ( ∇ t
[Equation]
− 1 ) , it follows by the oracle deﬁnition that
[Equation]
Gradient-dominated case.
Let x ∗ =
arg max x ∈K f ( x ) and let h t
= f ( x ∗ ) − f ( x t ) .
[Equation]
The theorem then follows from the following claim.
Claim 21.
Let C ≥ 1 .
Let g t be a H -bounded positive sequence such that
[Equation]
Then choosing σ t = min { 1 , 2 Ct } implies g t ≤ 2 C 2 max { 2 D,H } t +
[Equation]
F.3 Smoothness of value function (Lemma 16)
Proof of Lemma 16.
Consider any two policies π, π (cid:48) .
Using the Performance Difference Lemma (Lemma 3.2 in [2], e.g.) and Equation ??
, we have
[Equation]
The last inequality uses the fact that max s,a Q π ( s, a ) ≤ 1 1 − γ .
It sufﬁces to show (cid:107) d π (cid:48) − d π (cid:107)≤ γ 1 − γ (cid:107) π (cid:48)
− π (cid:107) ∞ , 1 .
To establish this, consider the Markov operator P π ( s (cid:48) | s )
= (cid:80) a ∈ A P ( s (cid:48) | s, a ) π ( a | s ) induced by a policy π on MDP M .
For any distribution d supported on S , we have
[Equation]
Using sub-additivity of the l 1 norm and applying the above observation t times, we have for any t (cid:107) (
P π (cid:48)
−
P π d
≤ t
−
(cid:107) ∞ , 1 .
Finally, observe that
[Equation]
F.4 Gradient domination (Lemma 18 and Lemma 19)
Proof of Lemma 18.
Invoking Lemma 4.1 from [2] with µ = d 0 , we have
[Equation]
Finally [Equation] with the aid of Equation ?? [Equation]
observe that
[Equation]
Proof of Lemma 19.
Invoking Lemma 4.1 from [2] with µ = ν , we have [Equation] Again [Equation]
with the aid of Equation ??
[Equation]
observe that
[Equation]
F.5 Supervised linear optimization guarantees
Proof of Claim 10.
The internal boosting subroutine of Algorithm 1, that is presented in Algorithm 5, is an instantiation of Algorithm 3 from [18], specializing the decision set to be ∆ A .
To note the equivalence, note that in [18] the algorithm is stated assuming that the center-of-mass of the decision set is at the origin (after a coordinate transform); correspondingly, the update rule in Algorithm 1 can be written as [Equation] For any state s , π ( ·| s ) =
1 A 1 | A | corresponds to the center-of-mass of ∆ A .
Finally, note that maximizing f (cid:62) x over x ∈ K is equivalent to minimizing ( − f ) (cid:62) x over the same domain.
Therefore, we can apply previous result on boosting for statistical learning from [18] (Theorem 13).
Note that (cid:99) Q π ( s, · ) produced by Algorithm 3 satisﬁes (cid:107) (cid:99) Q π ( s, · ) (cid:107) = | A | 1 − γ . Let D t be the distribution induced by the trajectory sampler in round t .
This yields the bound in the claim.
Proof of Claim 17.
Lemma 11 allows us to restate the guarantees from Claim 10 in terms of linear optimization over functional gradients.
The conclusion thus follows immediately by combining
Lemma 11 and Theorem 10.
F.6 Online linear optimization guarantees (Claim 20)
Proof of Claim 20.
In a similar vein to the proof of Claim 17, here we state the a result on boosting for online convex optimization (OCO) from [18] (Theorem 6), the counterpart of Theorem ??
for the online weak learning case.
Theorem 22.
Let β = (cid:113) 1 αN , and η 2 ,n = min { 2 n , 1 } .
Then, for any t , Γ[˜ π t,m,N ] produced by
Algorithm 5 satisﬁes
[Equation]
Next we invoke online-to-batch conversions.
Note that in Algorithm 5, ( s t,m , ˆ Q t,m ) for any ﬁxed t is sampled i
[Equation]
i.d. from the same distribution.
Therefore, we can apply online-to-batch results,
i
[Equation]
e.
Theorem 9.5 in [16], on Theorem 22 to get
[Equation]
We ﬁnally invoke Lemma 11.
F.7 Remaining proofs (Claim 21)
Proof of Claim 21.
Let T ∗ = arg max t { t : t ≤ 2 C } .
For any t ≤ T ∗ , we have σ t = 1 and g t ≤ H ≤ 2 C 2 H t .
For t ≥ T ∗ , we proceed by induction.
The base case ( t = T ∗ ) is true by the previous display.
Now, assume g t − 1 ≤ 2 C 2 max { 2 D,H } t −+ CE for some t > T ∗ .
[Equation]