Occupancy Information Ratio: Infinite-Horizon, Information-Directed, Parameterized Policy Search *
Abstract
We develop a new measure of the exploration/exploita- tion trade-off in infinite-horizon reinforcement learning (RL) problems called the occupancy information ratio (OIR), which is comprised of a ratio between the infinite-horizon average cost of a policy and the entropy of its induced long-term state occupancy measure.
Modifying the classic RL objective in this way yields policies that strike an optimal balance be- tween exploitation and exploration, providing a new tool for addressing the exploration/exploitation trade-off in RL.
The paper develops for the first time policy gradient and actorcritic algorithms for OIR optimization based upon a new en- tropy gradient theorem, and establishes both asymptotic and non-asymptotic convergence results with global optimality guarantees.
In experiments, these methodologies outperform several deep RL baselines in problems with sparse rewards.
Introduction
Stochastic optimization problems, where the goal is to optimize an objective that takes the form of an expectation, are ubiquitous in machine learning.
Classic methods for analyzing stochastic optimization techniques like stochastic gradient descent frequently rely on the well-known connections between stochastic approximation (Robbins and
Monro 1951) and dynamical systems (Borkar 2008).
Policy optimization in reinforcement learning (RL) (Sutton and
Barto 2018) can be viewed as a special form of stochastic optimization problem, where the expectation in the objective is taken over the trajectories induced by an agent’s policy when interacting with its environment.
In the model-free
RL setting, where the agent’s environment is unknown, however, the policy optimization problem features an additional, external imperative: the environment must be explored in order to learn an effective policy.
This so-called exploration/- exploitation trade-off justifies attempts to alter the landscape of the underlying dynamical system by augmenting the corresponding optimization objective to appropriately balance exploration with exploitation.
In control of dynamical systems, this exploration/exploitation trade-off is often addressed by separating the steps of planning and control: first one conducts state estimation (system identification ( ˚Astr¨om and Eykhoff 1971)) over a stochastic system, and then solves for actuation parameters in terms of it (Bertsekas and Shreve 1996).
The utility of RL-based methods for control of dynamical systems is wellestablished (Recht 2019; Meyn 2022). Indeed, RL methods for control have been placed on solid theoretical footing for a variety of classic control problems, including for linear quadratic regulators (LQR) (Fazel et al. 2018;
Bhandari and Russo 2019; Lale et al. 2022), the linear quadratic Gaussian (LQG) problem (Tang, Zheng, and Li 2021; Uehara et al. and ensuring safety (Berkenkamp et al. {+
 ; Cheng et al. 2019).+}
Importantly, many of these works have recognized the importance of balancing exploration of the system with exploitation of current knowledge to achieve critical goals like safe state identification (Berkenkamp et al. 2017), improved sample efficiency (Bhandari and Russo 2019), and improved stability (Lale et al. 2022). Recently, new approaches to addressing the exploration/- exploitation trade-off in the RL context have led to key theoretical advances.
In the context of policy gradient methods, the recent works (Bhandari and Russo 2019; Agarwal et al. 2020a; Mei et al. 2020) have provided new insights into the relationships between system exploration, convergence rates, and global optimality.
A key insight from these works is that the right amount of exploration accelerates convergence without sacrificing global optimality.
The prior works (Russo and Van Roy 2014; Lu et al. 2021) in multi-armed bandits (MABs) and RL, on the other hand, seek to balance the goals of exploration and exploitation explicitly: by minimizing an information ratio , defined as the ratio of cost incurred – formulated as regret – to information acquired.
A key insight of these works is that explicitly optimizing the rate of reward accrued per quantity information acquired about the system leads to more intelligent exploration behaviors and improved regret.
Despite the advantages of information ratio-based techniques and the recent advances in the theory of policy gradient methods, however, the development of policy gradient methods for information ratio objectives remains unexplored.
Our goal is to develop information ratio optimization approaches for infinite-horizon RL problems that can operate in high-dimensional, possibly continuous spaces.
The information ratio of (Lu et al. 2021) considers information gain of a policy over a fixed time-horizon, which in the infinite-horizon setting requires conditioning over an infinite trajectory.
Moreover, the deep Q-learning-based methods proposed in (Lu et al. 2021) are inherently restricted to the finite action space case, limiting their scalability.
To improve this scaling, operating in parameter space instead is required, for which policy gradient methods are most natural (Lillicrap et al. 2015; Schulman et al. 2017;
Haarnoja et al. 2018).
Our goal thus requires a definition of informativeness that is amenable to policy search in parameter space.
Occupancy measure entropy has recently been used as an optimization objective (Hazan et al. 2019; Lee et al.
; Zhang et al. 2020a) that captures the amount of information about the environment that a policy provides through the Kullback–Leibler divergence of its state occupancy measure (also called state marginal distribution) from a uniform distribution.
This motivates us to take occupancy measure entropy, or occupancy information , of a policy as the fundamental quantity defining its informativeness.
Based on this definition, we develop a new objective called the occupancy information ratio , or OIR, which captures the exploration/exploitation trade-off as defined by the ratio of long-term average cost to occupancy information of a policy.
We underscore that OIR is the first RL objective to concisely trade off exploration and exploitation and to which parameterized policies also naturally apply , making it the first to scale well to large spaces.
Consequently, we are able to derive policy gradient (PG) and actor-critic (AC) algorithms to optimize the OIR in parameter space.
Moreover, through connections to quasiconcave programming and a novel application of the perspective transform that arises in fractional programming, we establish that this objective has no spurious extrema (Theorem 6), extending the hidden concavity arguments of (Zhang et al.
2020a)
to hidden quasiconcavity .
Hence our newly derived PG and
AC algorithms exhibit convergence to global optimality as established in Theorems 8 and 11.
In experiments, we illustrate that the OIR yields policies that avoid spurious, suboptimal behavior in practice, whereas benchmarks exhibit a tendency to become stuck.
Due to the importance of the exploration/exploitation trade-off in achieving critical goals like safety, improved stability, and sample efficiency, we believe OIR-based methods are of interest to those working at the intersection of learning and dynamical systems.
Problem Formulation
In this section we describe our problem setting and formulate the occupancy information ratio objective.
We first define an underlying Markov decision process, then formulate the OIR as an objective to be optimized over it.
Markov Decision Processes.
Consider an average-cost
MDP described by the tuple ( S , A , p, c ) , where S is the finite state space, A is the finite action space, p : S ×
A → D ( S ) is the transition probability kernel mapping state-action pairs to distributions over the state space, and c : S ×
A → R + is the cost function mapping state-action pairs to positive scalars.
In this setting, at time-step t , the agent is in state s t , chooses an action a t according to a policy π : S → D ( A ) mapping states to distributions over A , incurs cost c ( s t , a t ) , and then the system transitions into a new state s t +1 ∼ p ( ·| s t , a t ) .
Since we are interested in policy gradient methods, we give the following definitions with respect to a parameterized family { π θ : S → D ( A ) } θ ∈ Θ of policies, where Θ ⊂ R d is some set of permissible policy parameters.
Note that analogous definitions apply to any policy π .
For any θ ∈ Θ , let d θ ( s ) =
lim t →∞ P ( s t = s |
π θ ) denote the steady-state occupancy measure over S induced by π θ , which we assume to be independent of the initial startstate.
In addition, let λ θ ( s, a ) =
lim t →∞ P ( s t = s, a t = a | π θ ) denote the state-action occupancy measure induced by π θ over S × A .
Notice that λ θ ( s, a )
=
d θ ( s ) π θ ( a | s ) .
Furthermore, let J ( θ )
=
(cid:80) s
d θ ( s ) (cid:80) a π θ ( a | s ) c
( s, a ) denote the long-run average cost of using policy π θ .
Finally, given θ , define the entropy of the state occupancy measure induced by π θ to be H ( d θ ) = − (cid:80) s d θ ( s ) log d θ ( s ) .
This quantity measures how well π θ covers the state space S in the long run.
Occupancy Information Ratio.
In this paper we consider the OIR objective
[Equation]
where κ > − min θ H ( d θ ) is a user-specific constant, discussed in Remark 1.
Given an MDP ( S , A , p, c ) , our goal is to find a policy parameter θ ∗ such that π θ ∗ minimizes (1) over the MDP , i.e., subject to its costs and dynamics.
As
J ( θ ) and H ( d θ ) are both long-run, infinite-horizon quantities, we regard (1) as an infinite-horizon objective .
Remark 1.
Since κ scales the relative importance of H ( d θ ) in (1) , it can be viewed (and used) as a regularizer.
When minimizing a function f ( x ) , one frequently considers a regularized objective function f ( x ) + κ ∥ x ∥ , where κ ≥ 0 .
Here, the larger κ becomes, the more important the regularization term becomes with respect to the objective function.
In contrast, for (1) , the relative importance of the entropy term actually diminishes as κ becomes larger: when κ is small, even minor changes in the value of H ( d θ ) can have a large effect on the value of ρ ( θ ) ; when κ is large, on the other hand, even significant perturbations of the value of H ( d θ ) have little effect on the value of ρ ( θ ) .
Remark 2.
Though we stipulated that κ > − min θ H ( d θ ) in the definition of the OIR above, letting κ < − max θ H ( d θ ) has an important interpretation as well.
When κ < − max θ H ( d θ ) and J ( θ ) ≥ 0 , for all θ ∈ Θ , clearly the OIR ρ ( θ ) will always be non-positive.
Because of this, minimizing the OIR will in fact minimize the ratio of − J ( θ ) to the absolute value | κ + H ( d θ ) | .
This means that the expected cost J ( θ ) of the underlying MDP is instead treated as an expected reward to be maximized, and any algorithm for minimizing the OIR will therefore balance maximizing the reward J ( θ ) with maximizing the shifted entropy | κ + H ( d θ ) | .
This allows the OIR framework to accommodate rewards by simply replacing the cost function c in the MDP with a reward function r , and choosing κ < − max θ H ( d θ ) .
Policy Gradients
Sampling the gradient of (1) is not straightforward using ex- isting tools, as obtaining stochastic estimates of ∇ ρ ( θ ) involves estimating
[Equation]
Though we can use the classical policy gradient theorem [cf.
Eq.
(3)] to estimate ∇ J ( θ )
and we can empirically estimate
J ( θ ) and H ( d θ ) , it is not obvious how to estimate ∇ H ( d θ ) .
In what follows we prove an entropy gradient theorem that allows us to estimate ∇ H ( d θ ) and consequently ∇ ρ ( θ ) .
Policy Gradient Preliminaries.
Given an MDP ( S , A , p, c ) and policy π θ , two important objects from the RL literature are the relative state value func- tion
[Equation]
and the relative action value function Q
[Equation]
s
[Equation]
a )
[Equation]
Under the assumption that π θ ( a | s ) is differentiable in θ , for all s ∈ S , a ∈ A , classic policy gradient methods minimize
J ( θ ) by taking stochastic gradient descent steps in the direction −∇ J ( θ ) .
We are guaranteed by the policy gradient theorem (Sutton et al. 1999)
that, under certain conditions,
[Equation]
By following policy π θ , we can sample from the right-hand side of (3) to estimate ∇ J ( θ ) , then use this to perform stochastic gradient descent.
Cross-Entropy Gradient.
To estimate ∇ ρ ( θ ) we must know how to estimate ∇ H ( d θ ) .
Fortunately, by using the relationship between entropy and cross-entropy, ∇ H ( d θ ) can be estimated in a straightforward manner.
Given two policy parameters θ and θ ′ , define the cross-entropy between d θ and d θ ′ to be CE ( d θ , d θ ′ )
=
− (cid:80) s d θ ( s ) log d
θ ′ ( s ) and their
Kullback-Leibler (KL) divergence to be D KL ( d θ || d θ ′ )
= (cid:80) s log (cid:16)
d θ ( s ) d θ ′ ( s ) (cid:17) d θ ( s ) .
Also recall the useful fact that
CE [Equation] d
[Equation]
|| d θ
[Equation]
We now have:
Lemma 1.
For any θ ′ ∈ Θ ,
[Equation]
This establishes an important fact: we can estimate the entropy gradient ∇ H ( d θ ) | θ = θ t by instead estimating the cross-entropy gradient ∇ CE ( d θ , d θ t ) | θ = θ t .
At first glance, this simply substitutes one problem for another.
However, given a fixed θ t , for any θ , we can use the classic policy gradient theorem (3) to obtain a tractable expression for ∇ CE ( d θ , d θ t )
| θ = θ t , as described next.
Entropy and OIR Policy Gradients.
Our next results provide tractable gradient expressions enabling policy gradient algorithms for maximizing H ( d θ ) and minimizing (1). = θ t .
Theorem 2.
Let an MDP ( S , A , p, c ) and a differentiable parametrized policy class { π θ } θ ∈ Θ be given, and recall the definition above of the state occupancy measure
d θ induced by π θ on S .
Fix a policy parameter iterate θ t at time-step t .
The gradient ∇ H ( d θ ) | θ = θ t [cf.
(4) ] with respect to the policy parameters θ of the state occupancy measure entropy
H ( d θ ) , evaluated at θ = θ t , satisfies ∇ H ( d θ ) (cid:12)(cid:12)
[Equation]
With Theorem 2 in hand, we have the following OIR policy gradient theorem:
Theorem 3.
Let an MDP ( S , A , p, c ) , a differentiable parametrized policy class { π θ } θ ∈ Θ , and a constant κ ≥ 0 be given, and recall the definitions of the average cost J ( θ ) , state occupancy measure d θ , and entropy H ( d θ ) .
Fix a pol- icy parameter iterate θ t at time-step t .
The gradient ∇ ρ ( θ t )
[cf.
(2) ] with respect to the policy parameters θ of the OIR ρ ( θ )
[cf.
(1) ], evaluated at θ = θ t , satisfies ∇ ρ ( θ t ) =
[Equation]
where δ
[Equation]
The claim follows by combining equations (2) and (3) with Theorem 2.
Armed with Theorem 3, we next develop policy gradient algorithms for minimizing the OIR
. Algorithms
In this section we derive two policy search schemes for minimizing (1).
Throughout this section, we will assume that an average-cost MDP ( S , A , p, c ) is fixed.
The reward setting can be accommodated with minor changes by Remark 2.
Information-Directed REINFORCE.
We present
Information-Directed REINFORCE (ID-REINFORCE), which builds on the class REINFORCE algorithm
(Williams 1992) to minimize the more complicated objective (1).
At each time-step t , the algorithm generates a trajectory using the current policy π θ t .
It then forms estimates of J ( θ t ) and H ( d θ t ) and in turn uses these to estimate ∇ ρ ( θ t ) by leveraging (6).
This gradient estimate is then used to update the policy parameters.
Note that, in order to estimate
H ( d θ t ) , it is necessary to first estimate
d θ t .
This task is addressed both implicitly and explicitly in previous works (Hazan et al. 2019; Lee et al. 2019;
Zhang et al. 2020a)
As in (Hazan et al. 2019), for ease of exposition we assume access to an oracle D ENSITY E STIMATOR that returns the occupancy measure d θ = D
ENSITY E STIMATOR
( θ ) when provided with input policy parameter θ ∈ Θ .
When S is finite and not too large, D ENSITY E STIMATOR can be implemented with a count-based estimator.
We focus on this setting in this paper.
Pseudocode for ID-REINFORCE can be found in Algorithm 1.
Information-Directed Actor-Critic.
We next present the
Information-Directed Actor-Critic (IDAC) algorithm, a variant of the classic actor-critic algorithm (Konda 2002; Bhatnagar et al.
with two critics: the standard critic corresponding to average cost J ( θ ) , and an entropy critic corresponding to the shadow MDPs ( S , A , p, r t ) , t ≥ 0 , where
1: Initialization: Select rollout length K , step-sizes η >and τ ∈ (0 , 1] , parametrized policy class { π θ } θ ∈ Θ , and entropy additive constant κ ≥ 0 .
Randomly sample sand θ 0 , select µ H − 1 , µ J − 1 > 0 , and set t ← 0 .
2: repeat 3:
Generate trajectory { ( s i , a i ) } i =1 ,...,K using π θ t 4: (cid:91) J ( θ t ) = 1 K (cid:80)
Ki =1 c ( s i , a i ) 5: µ Jt = (1 − τ ) µ Jt − 1 + τ (cid:91) J ( θ t ) 6:
d θ t = D ENSITY E STIMATOR
( θ t ) 7: (cid:92) H ( d θ t )
= 1 K (cid:80) Ki
=1 ( − log d θ t ( s i )) 8: µ Ht
=
(1 − τ ) µ Ht − 1 + τ (cid:92) H ( d θ t ) 9: for i = 1 , . . .
, k do 10: δ Ji
= c ( s i , a i ) − µ Jt 11: δ Hi = − log
d θ t ( s i ) − µ Ht 12: ψ i = ∇ log π θ t ( a i | s i ) 13: end for 14: (cid:92) ∇ ρ ( θ t )
=
[ κ + µ Ht ] 2 1 K (cid:80) Ki =1 (cid:2)
δ Ji (cid:0) κ + µ Ht (cid:1) − µ
Jt δ Hi (cid:3) ψ i 15: θ t +1 =
θ t − η (cid:92) ∇ ρ ( θ t ) 16:
t ← t +17: until convergence r t ( s, a ) = − log d θ t ( s ) is the shadow reward discussed in the proof of Theorem 2. We assume access to the D ENSITY E STIMATOR oracle throughout. For IDAC, we modify the classic actor-critic scheme by: (i) introducing an entropy critic to estimate the entropy gradient, and (ii) altering the policy update to take a gradient descent step in the direction −∇ ρ ( θ t ) instead of −∇ J ( θ t ) .
At time-step t , the algorithm computes two different TD errors: one corresponds to the critic for the MDP ( S , A , p, c ) , while the other corresponds to the critic for the shadow MDP ( S , A , p, r t ) .
Next, the cost and entropy critic TD errors are used to update their respective critics, which are in turn combined to perform the actor update.
Pseudocode for IDAC can be found in Algorithm 2.
Density Estimation Issue.
As discussed above, these algorithms estimate the state density with a count-based estimator, which can be inefficient in continuous, highdimensional spaces.
There are two main options for overcoming this issue.
First, more sophisticated density estimation procedures may be used to directly estimate the state occupancy measure in continuous, higher-dimensional spaces.
Recent works including (Hazan et al.
2019; Lee et al.
2019) have effectively leveraged kernel density estimation and variational autoencoders to maximize state entropy in the RL setting.
These techniques can be extended to the OIR setting to allow our methods to handle continuous, higherdimensional state spaces.
Second, eliminating the need to perform density estimation altogether by directly estimating state entropy is another viable option.
Indeed, recent works in unsupervised RL have leveraged particle-based entropy estimation techniques from statistics (Singh et al. 2003) to efficiently perform maximum state entropy exploration in continuous, high-dimensional domains (Yarats et al. 2021;
Liu and Abbeel 2021; Mutti, Pratissoli, and Restelli 2021).
s 0 , θ 0 , ω J 0 , ω H 0 , select µ H − 1 , µ J − 1 > 0 , and set t ← 0 .
2: repeat 3:
Generate trajectory { ( s i , a i ) } i =1 ,.
[Equation]
.,K using π θ t 4:
[Equation]
[Equation]
[Equation]
7: for
[Equation]
= 1 ,
[Equation]
, K do 8: Set
[Equation]
[Equation]
[Equation]
[Equation]
12: end for 13:
[Equation] [Equation]
[Equation]
[Equation]
[Equation]
←
[Equation]
18: until convergence These methods can be extended to obtain practical OIR methods that avoid density estimation altogether.
Building on these techniques, the density estimation issue can be mitigated or eliminated, providing an important future direction.
Theoretical Results
In this section we provide the following key results underpinning policy search for the OIR problem: all stationary points of ρ ( θ ) are in fact global minimizers; the stochastic gradient descent scheme underlying ID-REINFORCE enjoys a non-asymptotic convergence rate depending on κ , the policy class, and ergodicity properties of the underyling
MDP; IDAC enjoys asymptotic, almost sure (a.s.) convergence to a neighborhood of a stationary point.
Taken together, these results prove that both algorithms converge to globally optimal solutions under suitable conditions.
Stationarity Implies Global Optimality
As we will see, the OIR optimization problem enjoys a pow- erful hidden quasiconcavity property: under certain conditions on the set Θ and the policy class { π θ } θ ∈ Θ , stationary points of ρ ( θ ) correspond to global optima of the OIR minimization problem
[Equation]
This result is surprising, as the objective function ρ ( θ ) is typically highly non-convex.
Let Θ ⊂ R k be convex and let a parametrized policy class { π θ } θ ∈ Θ be given.
Let λ : Θ →
D ( S ×A ) be a function mapping each parameter vector θ ∈ Θ to the state-action occupancy measure λ ( θ ) :
=
λ θ :=
λ
π θ induced by the policy π θ over S×A .
We make the following assumptions.
Assumption 4.
The set Θ is compact.
For any s ∈ S , a ∈
A , the function π θ ( a | s ) is continuously differentiable with respect to θ on Θ , and the Markov chain induced by π θ on
S is ergodic.
Assumption 5.
The following statements hold:
λ ( · ) gives a bijection between Θ and its image λ (Θ) , and λ (Θ) is compact and convex.
Let h ( · ) :
= λ − 1 ( · ) denote the inverse mapping of λ ( · ) .
h ( · ) is Lipschitz continuous.
The Jacobian matrix ∇ λ ( θ ) is Lipschitz on Θ .
We have the following theorem.
Theorem 6.
Let Assumptions 4 and 5 hold.
Let θ ∗ be a stationary point of (7) , i.e., ∇ ρ ( θ ∗ ) = 0 .
Then θ ∗ is globally optimal for (7) .
This powerful hidden quasiconcavity property implies that any policy gradient algorithm that can be shown to converge to a stationary point of the OIR optimization problem min θ ∈ Θ ρ ( θ ) in fact converges to a global optimum.
This greatly strengthens the convergence results provided next by guaranteeing that they apply to global optima.
In contrast to the global optimality guarantees for tabular, softmax policy search established in (Bhandari and Russo 2019; Agarwal et al. 2020b; Mei et al. 2020;
Zhang et al.
2020a;
Bedi et al.
using persistent exploration conditions, our result instead builds on hidden concavity arguments from (Zhang et al.
2020a), which apply to parameterized policies.
However, Theorem 6 generalizes these results in important ways.
First, it applies to ratio objectives, which have not been addressed in prior work.
In addition, we establish hidden quasiconcavity for ratio objectives, not hidden concavity , which requires reformulation via a novel application of the perspective transform.
Theorem 6 is thus a strict generalization of existing results for the landscape of RL objectives.
Non-Asymptotic Convergence Rate
Next, we establish a non-asymptotic convergence rate for the following projected gradient descent scheme for solving the
OIR minimization problem (7):
[Equation]
for a fixed stepsize η > 0 , where Proj Θ denotes euclidean projection onto Θ and the second equality holds by the convexity of Θ .
Note that (8) is a reformulation of IDREINFORCE with null gradient estimation error and projection onto the set Θ ; we assume the projection operation for the purposes of analysis, and we discuss the gradient estimation issue at the end of this subsection.
Let Θ ⊂ R k , { π θ } θ ∈ Θ , and λ : Θ → D ( S × A ) be as in the previous section.
Consider the mapping ζ : D ( S ×A ) →
R |S||A| +1 , defined to be ζ ( λ ) =
( λ/c ⊤ λ, 1 /c ⊤ λ ) , where c ∈
R |S||A| , c > 0 is a vector of positive costs.
Notice that, under the ergodicity conditions in Assumption 4 and properties of entropy, min θ ρ ( θ ) > 0 and max θ ρ ( θ ) < ∞ .
In addition to
Assumptions 4 and 5, we will need the following.
Assumption 7.
∇ ρ ( θ ) is Lipschitz and L > 0 is the smallest number such that ∥∇ ρ ( θ ) − ∇ ρ ( θ ′ )
∥ ≤ L ∥
θ − θ ′ ∥ , for all θ, θ ′ ∈ Θ .
We have the following convergence rate result for the projected gradient descent scheme (8).
Theorem 8.
Let Assumptions 4, 5, and 7 hold.
Let
D ζ = max z,z ′ ∈ ( ζ ◦ λ )(Θ) ∥
z − z ′
∥ denote the diameter of the convex, compact set ( ζ ◦ λ )(Θ) .
Define M = max θ ∈ Θ ρ ( θ ) , m = min θ ∈ Θ ρ ( θ ) , K = max { m 2 L, M 2 m 2 L } , and L 1 = max { L, M 2 L } .
Then, with η = 1 /K , for all t ≥ 0 ,
[Equation]
Coupled with Theorem 6, this result provides a nonasymptotic convergence rate to global optimality for algorithms solving the OIR minimization problem (7).
Remark 3.
When compared with the corresponding result in (Zhang et al.
2020a), to which it is closely related, the bound (9) of Theorem 8 contains an interesting dependence on the user-specified κ , the policy class { π θ } θ ∈ Θ , and the underlying MDP.
The presence of M = max θ ∈ Θ ρ ( θ ) =
max θ
[ J ( θ ) / ( κ + H
( d θ ))] in the bound (9) suggests that the convergence rate depends on the value of κ as well as the minimal possible value of H ( d θ ) over θ ∈ Θ .
To see why, let
C = max θ ∈ Θ J ( θ ) and notice that
[Equation]
On the one hand, when the MDP dynamics and policy class are such that min θ ∈ Θ H ( d θ ) is large, then M will be closer to 0, yielding a tighter bound in (9) .
This suggests that it may be easier to optimize the OIR over MDPs and/or policy classes that tend to be “more ergodic”.
When both κ and min θ ∈ Θ H ( d θ ) are close to 0, on the other hand, M may be very large, resulting in a looser bound in (9) .
This highlights the usefulness of κ , as choosing larger κ values can be used to smooth the objective function ρ ( θ ) and thereby lead to stabler convergence when optimizing the OIR over MDPs and policy classes that tend to be “less ergodic”.
In the preceding theorem, we assume “exact policy gradient,” or zero stochastic approximation error.
This is limited to Theorem 8, whereas Theorem 11 below allows stochastic approximation error and Theorem 6 above is independent of estimation issues.
Though this assumption is a drawback for Theorem 8, it allows us to succinctly focus on a core insight of this work: hidden quasiconcavity unlocks an information-dependent convergence rate to global optimality.
We also note that, for REINFORCE-like algorithms like those considered in Theorem 8, long rollouts enable more accurate gradient estimates, for which the existing assumptions approximately apply.
A precise treatment of gradient estimation error versus rollout length is an important direction future work, and can be achieved by extending the analysis in (Zhang et al. 2021) to the OIR problem.
We conclude this section by proving almost sure (a.s.) convergence of IDAC to a neighborhood of a stationary point of (7).
By Theorem 6, this implies IDAC converges a.s. to a neighborhood of a global optimum.
This is much stronger than existing asymptotic results for actor-critic schemes, which typically guarantee convergence to a neighborhood of a local optimum or saddle point (Bhatnagar et al. 2009;
Zhang et al.
2020b; Agarwal et al. 2020b).
We analyze the algorithm as given in Algorithm 2 under the assumption that τ t = α t , for all t ≥ 0 , that K = 1 , and with the addition of a projection operation to the policy update:
[Equation]
where Γ : R d → Θ maps any parameter θ ∈ R d back onto the compact set Θ ⊂ R d of permissible policy parameters.
This projection, which is common in the actor-critic and broader two-timescale stochastic approximation literatures (see, e.g., (Kushner and Yin 2003; Borkar 2008; Bhatnagar et al. 2009)) is for purposes of theoretical analysis, and is typically not needed in practice.
In addition to Assumption 4, we impose the following:
Assumption 9.
Stepsizes { α t } , { β t } satisfy (cid:80) t α t
[Equation]
(cid:80) t β t = ∞ , (cid:80) t
α 2 t + β 2 t < ∞ , lim t β t α t = 0 .
Assumption 10.
The value function approximators v ω are linear, i.e., v ω ( s )
= ω ⊤ ϕ ( s ) , where ϕ ( s ) =
[ ϕ 1 ( s ) · · · ϕ K ( s )]
⊤ ∈ R K is the feature vector associated with s ∈ S .
The feature vectors ϕ ( s ) are uniformly bounded for any s ∈ S , and the feature matrix Φ =
[ ϕ ( s )]
⊤ s ∈S ∈
R |S|× K has full column rank.
For any u ∈ R K , Φ u ̸ = 1 , where 1 is the vector of all ones.
Assumptions 4, 9, and 10 are standard in two-timescale convergence analyses for actor-critic algorithms (Bhatnagar et al. 2009).
Moreover, we consider neural network parameterizations in our experiments and observe favorable convergence behavior, so we believe Assumption 10 can be relaxed.
Consider the ordinary differential equation (ODE) [Equation] where ˆΓ( ∇ ρ ( θ )) :
=
lim η → 0 + [ γ ( θ + η ∇ ρ ( θ )) − θ ]
/η .
We note here that (12) can be interpreted as the projected ODE ˙ θ = ∇ ρ ( θ ) + z ( θ ) , where z ( θ ) is the minimal force necessary to project θ back onto Θ .
We now present the main result of this subsection, which establishes convergence of the actor-critic algorithm.
Theorem 11.
Let Z denote the set of asymptotically stable equilibria of the ODE (12) .
Given any ε > 0 , define
Z ε = { z | inf z ′ ∈Z ∥
z − z ′
∥ ≤ ε } .
For any θ ∈ Θ , let ε θ =
( ϵ Jθ [ κ + H ( d θ )] − J ( θ )
ϵ Hθ ) /
([ κ + H ( d θ )] 2 ) .
Under
Assumptions 4, 9, and 10, given any ε > 0 , there exists δ > 0 such that, for { θ t } obtained from Algorithm 2 with projection (11) ,
if sup t ∥ ϵ θ t ∥ <
δ , then θ t → Z ε a.s. as t → ∞ .
Combined with Theorem 6, Theorem 11 establishes almost sure convergence of IDAC to a neighborhood of a global optimum of the OIR minimization problem (7).
Note that if the linear approximation and features are expressive enough, then ε will be small or even zero.
Experiments
The experiments presented in this section demonstrate that
OIR policy gradient methods avoid spurious behavior, while state-of-the-art methods can become overconfident and settle into suboptimality.
In particular, when the reward signal is sparse, OIR methods can lead to improved performance when compared with vanilla RL methods.
For Fig.
1, we first compared a neural network version of IDAC with the
Stable Baselines 3 implementations (Raffin et al.
2019) environment depicted in Fig.
2 in the appendix; for Fig.
3 presented in the appendix, we also compared tabular versions of IDAC and vanilla actor-critic with softmax policies on the
GridWorld2 depicted in Fig.
Details on the gridworld implementations are provided in the appendix.
In all cases, the vanilla methods prematurely converge to suboptimal policies, whereas the OIR-based methods solve the problem.
This illustrates that, in sparse-reward environments, the inherent skepticism of OIR-based policy gradient methods can lead to improved performance.
Conclusion
In this paper we have addressed the exploration/exploitation trade-off in reinforcement learning via a new RL objective: the OIR. Interesting future directions include clarifying the relationship between optimal solutions to the OIR and vanilla problems, development of continuous-spaces version of IDAC, and thorough empirical evaluation of deep RL variants of IDAC on a range of benchmark problems.
Environments
Each gridworld is composed of an n × m grid of states, S = { 0 , . .
n − 1 } × { 0 , . .
, m − 1 } , along with a designated start state s start , designated goal state s goal , and a set B ⊂ S of blocked states which the agent is not permitted to enter.
Episodes are of fixed length K , and the agent begins each episode in state s start .
In a given state s = ( i, j ) , the agent chooses an action a ∈ { stay , up , down , left , right } .
The agent then attempts to move in the direction corresponding to the action selected: if the selected action would move the agent off the grid or into a blocked state, the agent remains in s ; otherwise, the agent moves into (or remains in) the state corresponding to the action selected.
For example, if a = up is chosen, the agent attempts to move to state s ′ =
( i, j − 1) .
If s ′ is off the grid (i.e. j − 1 < 0 ) or s ′ ∈ B , the agent remains in s .
Otherwise, the agent transitions to s ′ .
Finally, let A ( s ) denote the set of all actions at s that do not lead off the grid or into a blocked state; the cost function is then given by:
[Equation]
and a ∈ A ( s ) ,
[Equation]
blocked
[Equation]
where 0 < c goal < c allowed < c blocked .
A policy minimizing J ( θ ) will move as quickly as possible to s goal while always choosing actions within A ( s ) .
Because of this, when a problem is small enough that the agent can reach the goal state quickly and remain in it for most of the episode, the optimal average cost should be close to 1.
A policy minimizing ρ ( θ ) , on the other hand, will seek to balance minimizing J ( θ ) with maximizing H ( d θ ) , while avoiding actions a / ∈ A ( s ) .
For the first set of experiments, we considered the Gridworld1 , 2 , and 3 environments shown in Figure
[Figure]
GridWorld
S
G
GridWorld
[Figure]