Reward-Free Policy Space Compression for Reinforcement Learning
Abstract
In reinforcement learning, we encode the potential behaviors of an agent interacting with an environment into an inﬁnite set of policies, called policy space , typically represented by a family of parametric functions.
Dealing with such a policy space is a hefty challenge, which often causes sample and computational inefﬁciencies.
However, we argue that a limited number of policies is actually relevant when we also account for the structure of the environment and of the policy parameterization, as many of them would induce very similar interactions, i.e., state-action distributions.
In this paper, we seek for a reward-free compression of the policy space into a ﬁnite set of representative policies, such that, given any policy π , the minimum Rényi divergence between the stateaction distributions of the representative policies and the state-action distribution of π is bounded.
We show that this compression of the policy space can be formulated as a set cover problem, and it is inherently NP-hard.
Nonetheless, we propose a game-theoretic reformulation for which a locally optimal solution can be efﬁciently found by iteratively stretching the compressed space to cover the most challenging policy.
Finally, we provide an empirical evaluation to illustrate the compression procedure in simple domains, and its ripple effects in reinforcement learning.
1. Introduction
In the Reinforcement Learning (RL) (Sutton & Barto, 2018) framework, an artiﬁcial agent interacts with an environment, typically modeled through a Markov Decision Process (MDP) (Puterman, 2014), to maximize some form of long-term performance, which is usually the sum of the discounted rewards collected in the process.
The agent’s behavior is encoded in a Markovian policy , i.e., a function that maps the current state of the environment with a probability distribution over the next action to be taken.
In principle, if the underlying MDP is small enough, we can represent a
Markovian policy with a table that includes an entry for each state-action pair, and we call it a tabular policy.
However, most relevant scenarios have too many (possibly inﬁnite) states and actions to allow for a tabular representation.
In this case, we can turn to function approximation (Sutton &
Barto, 2018) to encode the policy within a family of parametric functions, e.g., a linear basis combination or a deep neural network, and we call it a parametric policy.
This set of parametric policies, which we call the policy space , is typically inﬁnite.
Therefore, learning a policy that maximizes the performance can be a hefty challenge, and the sheer size of the policy space often causes sample and computation inefﬁciencies.
A setting where these inefﬁciencies arise clearly and naturally is Policy Optimization (PO) (Deisenroth et al., 2013).
In PO, we aim to ﬁnd a policy that maximizes the performance within the policy space, i.e., an optimal policy, with the least amount of interactions (Sutton et al., 1999; Silver et al., 2014; Schulman et al., 2015; Metelli et al., 2018).
If we also account for the performance of the policies that are actually deployed to collect these interactions, we come up with an online PO (Papini et al., 2019).
In this setting, we try to minimize the regret that the agent suffers by taking interactions with a sub-optimal behavior before converging to an optimal policy.
Recent results showed that the regret of online PO is directly related to the size of the policy space (Papini et al., 2019; Metelli et al., 2020a).
In particular, online PO with a ﬁnite policy space can enjoy a constant regret, i.e., it does not scale with the number of interactions, under certain conditions (Metelli et al., 2020a).
Instead, the regret of online PO with an inﬁnite policy space does scale with the square root of the number of interactions in general (Papini et al., 2019), which means that we only have asymptotic guarantees of reaching an optimal policy.
In view of these results, one could wonder whether the expressive power of an inﬁnite policy space is worth the additional regret it causes: Are all of these inﬁnitely many policies really necessary for PO?
The expressive power of a policy space is related to the different distributions that its policies can induce over the states and actions of the environment, as the whole point of PO is to ﬁnd a policy that maximizes the probability of reaching state-action pairs associated with high rewards.
However, different parameterizations might actually induce equivalent policies due to the speciﬁc structure of the policy space.
Similarly, even different policies can induce the same state-action distribution in a given environment.
These two types of policies are arguably redundant for PO and we would like to ﬁnd a policy space that does not include either.
Especially, we aim to answer the following question:
Having an inﬁnite parametric policy space Θ in a given environment M , can we compress Θ into a ﬁnite subset that retains most of its expressive power?
In this paper, we formulate this question into the Policy
Space Compression problem, where we exploit the inherent structure of M and Θ to compute the compressed policy space.
The general idea is to identify a ﬁnite set of representative policies, such that for any policy π of the original space, the minimum Rényi divergence between the stateaction distributions of the representative policies and the state-action distribution of π is bounded by a given constant.
This compression is agnostic to the reward function, and thus the resulting policy space can beneﬁt the computational and sample complexity of any RL task one can later specify over M , as it is typical in reward-free RL (Hazan et al., 2019; Jin et al., 2020a).
Especially, the paper includes the following contributions.
First, we provide a formal deﬁnition of the policy space compression problem (Section 3).
We note that the problem can be formulated equivalently as a set cover, and that ﬁnding an optimal compression of the policy space is NP-hard in general (Feige, 1998).
Despite this negative result, we propose a game-theoretic reformulation (Section 4) that casts the problem to the one of reaching a differential Stackelberg equilibrium (Fiez et al., 2020) of a two-player sequential game, in which the ﬁrst player tries to cover the policy space with a ﬁnite set of policies and the second player tries to ﬁnd a policy that falls outside this coverage.
Then, we present an algorithm (Section 5) to efﬁciently compute a compression of the policy space by repeatedly solving, with a ﬁrst-order method, the two-player game for an increasing number of covering policies, until the compression requirement is met globally.
In Section 6, we provide a theoretical analysis of the performance guarantees attained by the compressed policy space in relevant RL tasks.
Finally, in Section 7 we provide a brief numerical validation of both the compression algorithm and RL with the compressed policy space.
The proofs of the theorems can be found in Appendix A.
2. Preliminaries
In this section, we introduce the essential background on controlled Markov processes, policy optimization, importance sampling estimation and its relationship with the
Rényi divergence.
2.1. Controlled Markov Processes
A discrete-time Controlled Markov Process (CMP) is deﬁned as a tuple M :=
( S , A , P, µ, γ ) , in which S is the state space,
A is the action space, P : S ×
A → ∆( S ) is a transition model such that the next state is drawn as s (cid:48) ∼ P ( ·| s, a ) given the current state s ∈ S and action a ∈ A , µ : ∆( S ) is an initial state distribution such that the initial state is drawn as s ∼ µ ( · ) , and γ ∈ (0 , 1] is the discount factor.
The behavior of an agent interacting with a CMP can be modeled through a Markovian parametric policy π θ : S → ∆( A ) such that an action is drawn as a ∼ π θ ( ·| s ) given the current state s ∈ S , where
θ ∈ Θ ⊆
R m are the policy parameters, and the set Π Θ is called the policy space .
A policy π θ induces a γ -discounted state distribution d sπ θ : ∆( S ) over the state space of the CMP
M , which is given by d sπ θ ( s ) =
(1 − γ ) (cid:80) ∞
t =1 γ t Pr ( s t = s ) or the equivalent recursive relation d sπ θ ( s ) =
(1 − γ ) µ ( s ) − γ (cid:82)
[Equation]
ilarly, we deﬁne the γ -discounted state-action distribution d saπ θ : ∆( S × A ) given by d saπ θ ( s, a ) =
π θ ( a | s ) d sπ θ ( s ) .
With a slight overloading of notation, we will indifferently denote the parametric policy space Π Θ by Θ , a parametric policy π θ ∈ Π Θ by θ , and its induced distributions d sπ θ ( s ) , d saπ θ ( s, a ) by d s θ ( s ) , d sa θ ( s, a ) .
2.2. Policy Optimization
The process of looking for the policy that maximizes the agent’s performance on a given RL task with a direct search in the policy space is called Policy Optimization (PO) (Deisenroth et al., 2013).
The task is generally modeled through a Markov Decision Process (MDP) (Puterman, 2014)
M R :=
M ∪ R , i.e., the combination of a CMP M and a reward function R : S ×
A → [ − R max , R max ] such that R ( s, a ) is the bounded reward that the agent collects by selecting action a ∈ A in state s ∈ S , and R max < ∞ .
The agent’s performance is deﬁned by the expected sum of discounted rewards collected by its policy, i.e.,
[Equation]
A Monte-Carlo estimate of the performance can be computed from a batch of N samples { s n , a n } Nn =1 taken with the policy π θ in the γ -discounted MDP M R as (cid:98) J ( θ ) =(1 − γ ) N (cid:80)
Nn =1 R ( s n , a n ) .
2.3. Importance Sampling and Rényi Divergence
Importance Sampling (IS) (Cochran, 2007; Owen, 2013) is a common technique to estimate the expectation of a function under a target distribution by taking samples from a different distribution.
In PO, importance sampling allows for estimating the performance of a target policy π θ (cid:48) through a batch of samples { s n , a n } Nn =1 taken with a policy π θ .
Especially, we deﬁne the importance weight w θ (cid:48) /
θ ( s, a ) :
=
d sa θ (cid:48) ( s, a ) /d
sa θ ( s, a ) .
A Monte-Carlo estimate of J ( θ (cid:48) ) via importance sampling is given by (cid:98)
[Equation] IS
[Equation]
The latter estimator (cid:98) J IS ( θ (cid:48) / θ ) is known to be unbiased, i.e., E θ [ (cid:98)
J IS ( θ (cid:48) / θ )]
=
J ( θ (cid:48) )
(Owen, 2013).
However, (cid:98) J IS ( θ (cid:48) / θ ) might suffer from a large variance whenever the importance weights w θ (cid:48) /
θ ( s, a ) have a large variance.
The variance of the importance weights is related to the exponentiated 2-Rényi divergence D 2 ( d sa θ (cid:48)
||
d sa θ ) (Rényi et al., 1961) through V ar ( s,a ) ∼ d sa θ [ w θ (cid:48) /
θ ( s, a )]
= D 2 ( d sa θ (cid:48)
||
d sa θ ) − 1 (Cortes et al., 2010), where
[Equation]
d
[Equation] θ ( s, a ) (cid:18) d sa θ (cid:48) ( s, a ) d sa θ ( s, a ) (cid:19)d s d a. The last result has been employed in (Metelli et al., 2018) to upper bound the variance of the importance sampling estimator (cid:98) J IS ( θ (cid:48) / θ ) as V ar ( s,a ) ∼ d sa θ [ (cid:98) J IS ( θ (cid:48) / θ )] ≤ (cid:0) R max− γ (cid:1) 2 D 2 ( d sa θ (cid:48)
||
d sa θ ) /N .
In the following, we will refer to the exponentiated 2-Rényi divergence as the Rény divergence.
3. The Policy Space Compression Problem
Let us suppose to have a CMP M the agent can interact with, and a parametric policy space Θ from which the agent can select its strategy of interaction.
For the common parameterization choices, ranging from linear policies to deep neural networks, the policy space Θ is typically inﬁnite.
Dealing with such a large policy space to address the usual
RL tasks, e.g., ﬁnding a convenient task-agnostic sampling strategy (Hazan et al., 2019) or seeking for an optimal policy within the set (Deisenroth et al., 2013), it is often a huge challenge.
Furthermore, many policies in Θ are unnecessary for these purposes, as they induce very similar interactions, and thus they have very similar performance.
On the one hand, different policy parameters θ ∈ Θ might induce nearly identical distributions over actions.
On the other hand, even different distributions over actions can lead to comparable state-action distributions due to the structure of the environment.
Since we do not have any reward encoded in M , it would be unwise to deem any state-action distribution irrelevant without additional information on the task structure.
In this work, we aim to identify a subset of the policy space Θ (cid:48) ⊆
Θ that retains most of the expressive power of Θ , i.e., the set of the state-action distributions it can induce, while dramatically reducing its size, to the advantage of the computational and sample efﬁciency of future
RL tasks.
Especially, we consider a σ -soft compression of Θ , where for any policy θ ∈ Θ we would like to have a policy θ (cid:48)
∈ Θ (cid:48) such that the Rényi divergence between their respective state-action distributions d sa θ , d sa θ (cid:48) is bounded by a positive constant σ .
The Rényi divergence is particularly convenient in this setting due to its relationship with the variance of the importance sampling in the off-policy estimation (Cortes et al., 2010; Metelli et al., 2018).
The following statement provides a more formal deﬁnition of this σ -soft compression.
Deﬁnition 3.1 ( σ -compression) .
Let M be a CMP, let Θ be a parametric policy space for M , and let σ > 0 be a constant.
We call Θ σ a σ -compression of Θ in M if it holds that | Θ σ | < ∞ and
[Equation]
We call the task of ﬁnding a σ -compression of Θ in M the policy space compression problem.
Notably, for some
M , Θ , σ , a σ -compression of Θ in M might not exist, as all the policies θ ∈ Θ might induce relevant state-action distributions.
Otherwise, we say that the compression is feasible .
In this case, given M and Θ , we would like to extract the smallest set of policies Θ (cid:48) that is a σ -compression of Θ in M , and then keep this reduced policy space to address any RL task one can deﬁne over M .
Let Ω Θ := { d sa θ | ∀ θ ∈ Θ } be the set of state-action distributions induced by the policy space Θ , the compression problem can be formulated as a typical set cover problem , i.e., minimize
[Equation]
subject to (cid:88)
[Equation]
where the positive integers x ω denote the state-action distributions that are active in the covering, and the corresponding σ -compression of Θ in M can be retrieved as Θ σ = { θ ∈ Θ | d sa θ = ω
∧ x ω = 1 } .
Unfortunately, the problem (1) is known to be NP-hard (Feige, 1998), and even building an instance is far-fetched when Ω Θ is an inﬁnite set.
Two aspects arguably make this problem extremely hard:
On the one hand, we are looking for an efﬁcient solution in the number of active state-action distributions, secondly, we are covering the set Ω Θ all at once rather than incrementally.
Instead of considering common relaxations of (1) (Johnson, 1974; Lovász, 1975), which would not strictly meet the requirements of Deﬁnition 3.1 (Feige, 1998), in the next section we build on these insights to reformulate the policy space compression problem in a tractable way.
4. A Game-Theoretic Reformulation
Due to its inherent hardness, we aim to ﬁnd a tractable reformulation of the policy space compression problem (1) whose solution is a valid σ -compression of Θ in M .
To this end, let us consider a game-theoretic perspective to the set cover problem.
In this perspective, a player distributes a set of K policies ( θ 1 , . .
, θ K ) ∈
Θ K with the intention of covering the set of state-action distributions Ω Θ .
A second player tries to ﬁnd a policy µ ∈ Θ that is not well covered by ( θ 1 , . .
, θ K ) , i.e., a policy that maximizes the Rényi divergence between its state-action distribution and the one of the closest θ k ∈ ( θ 1 , . .
, θ K ) .
The former player moves ﬁrst, and we call it a leader .
The latter player makes his move in response to the other player, and it is then called a follower .
The two-player, zero-sum, sequential game that we have informally described can be represented as the optimization problem
[Equation]
where θ =
( θ 1 , . .
, θ K ) and [ K ] = { 0 , . . .
, K } .
It is straightforward to see that if the σ -compression is feasible for Θ in M and K is large enough, then any optimal leader’s strategy for the game (2), i.e., θ ∗ ∈ arg max θ ∈ Θ K min µ ∈ Θ
f
( θ , µ ) , is a σ -compression of Θ in M .
Unfortunately, f ( θ , µ ) is a non-convex non-concave function, and ﬁnding a globally optimal strategy for the game (2) is still a NP-hard problem.
However, we do not actually need to ﬁnd a globally optimal strategy for the leader, as any θ ∈ Θ K such that min µ ∈ Θ
f
( θ , µ ) ≤ σ would be a valid σ -compression of Θ .
Thus, we might instead target a locally optimal strategy for (2), which is a stationary point of f and it is both a local maximum w.r.t.
θ and a local minimum w.r.t.
µ .
In the next statement, we formalize this solution concept as a Differential Stackelberg Equilibrium (DSE) (Fiez et al., 2020).
Deﬁnition 4.1 (Differential Stackelberg (Fiez et al., 2020)) .
The joint strategy ( θ ∗ , µ ∗ ) ∈
Θ K +1 in which θ ∗ k ∈ arg min k ∈
[ K ] ( d sa µ ∗ ||
d sa θ ∗ k ) is a differential Stackelberg equilibrium of the game (2) if it holds ∇ θ ∗
k f ( θ ∗ , µ ∗ )
= 0 , ∇ µ ∗ f
( θ ∗ , µ ∗ )
= 0 , |∇
θ ∗ k ∇ (cid:62) θ ∗ k f ( θ ∗ , µ ∗ ) | > 0 , and |∇ µ ∗ ∇ (cid:62) µ ∗ f
( θ ∗ , µ ∗ ) | < 0 .
Luckily, several recent works have established a favorable complexity for the problem of ﬁnding a DSE (Jin et al., 2020b; Fiez et al., 2020; Fiez & Ratliff, 2020) in a sequential game.
Especially, Jin et al.
(Jin et al., 2020b) showed that a basic ﬁrst-order method, i.e., Gradient Descent Ascent (GDA), with an inﬁnite time-scale separation between the leader’s and follower’s updates is guaranteed to converge to a DSE under mild conditions.
This result might be surprising, as we started with a fundamentally hard problem (1) and ended up with a way easier formulation (2) that we can address with a common methodology, without making any strong assumption on the structure of the problem.
However, we still have to deal with two crucial issues to solve the policy space compression problem through the game-theoretic formulation.
On the one hand, it is not enough to look at the value f
( θ ∗ , µ ∗ ) attained by a DSE ( θ ∗ , µ ∗ ) to guarantee that θ is a σ -compression of Θ , as we should check that max µ ∈ Θ
f ( θ ∗ , µ ) ≤ σ , where µ is a global maximizer.
On the other hand, it is not clear how to set a convenient value of
K beforehand.
In the next section, we present a ﬁrst-order method that addresses these two issues by ﬁnding a DSE of iteratively larger instances of the game (2) (which we will henceforth call the cover game ) until a conservative approximation of the global condition max µ ∈ Θ f ( θ ∗ , µ ) ≤ σ is ﬁnally met.
5. An Algorithm to Solve the Policy Space Compression
Optimization problems of the kind of (2) are typically addressed with a GDA procedure, in which the leader’s parameters ( θ ) and the follower’s parameters ( µ ) are updated iteratively according to
[Equation]
where ∇ θ f ( θ , µ ) and ∇ µ f ( θ , µ ) are the respective gradients of the joint objective function.
Especially, if we consider a sufﬁciently large time-scale separation τ := β/α , we are guaranteed to converge to a DSE of the game (2) (Jin et al., 2020b; Fiez & Ratliff, 2020).
In this case, we can consider τ = ∞ , which means we update the follower’s parameters until a stationary point is reached, i.e., ∇ µ f ( θ , µ ) = 0 , before updating the leader’s parameters.
However, to instantiate the cover game, we still need to specify the number K of leader-controlled policies θ =
( θ 1 , . .
, θ K ) .
A straightforward solution is to start with a small number of policies ﬁrst, say K
= 1 , then retrieve a DSE ( θ ∗ , µ ∗ ) via GDA for a cover-game instance with K policies, and ﬁnally check if the resulting leader’s strategy θ ∗ meets the global requirement max µ ∈ Θ f ( θ ∗ , µ ) ≤ σ . If the answer is positive, the policy space compression problem is solved, and θ ∗ is a σ -compression of Θ in M . Otherwise, we increment K and we repeat the process to see if we can solve the problem with more policies in θ .
If the policy space compression problem is feasible, with this simple procedure we are guaranteed to
Algorithm
Input : CMP M , policy space Θ , constant σ initialize K = 0 and the cover guarantee Z θ = ∞ while Z θ > σ do
K ← K +initialize the leader θ =
( θ 1 , . .
, θ K ) ∈
Θ K for epoch
= 1 , 2 , . .
, until convergence do compute the best response µ br to θ identify the active leader’s component w.r.t.
µ br update the leader θ k ← θ k − α ∇ θ k f ( θ , µ br ) end for compute the cover guarantee Z θ with (6) end while
Output : return θ , i.e., a σ -compression of Θ in M get a valid σ -compression eventually.
We call this method the Policy Space Compression Algorithm (PSCA) and we report the pseudocode in Algorithm 1.
In the following sections, we describe in details how the optimization of the follower’s parameters (Section 5.1) and the leader’s parameters (Section 5.2) are carried out in an adaptation of the GDA method to the speciﬁc setting of the cover game.
In Section 5.3, we discuss how to verify the global requirement max µ ∈ Θ f ( θ ∗ , µ ) ≤ σ without actually having to ﬁnd a globally optimal follower’s strategy, but instead optimizing a surrogate objective through a tractable linear program.
5.1. Optimizing the Follower’s Parameters
In principle, we would like to compute the gradient ∇ µ f ( θ , µ ) to perform the update µ ← µ + β ∇ µ f ( θ , µ ) as in a common GDA procedure.
Unfortunately, the objective function f ( θ , µ ) =
min k ∈ [ K ] D 2 ( d sa µ || d sa θ k ) is not differentiable due to the minimum over the K components of θ .
However, only the leader’s component θ k that attains the minimum of f is actually relevant for the follower’s update, as the other K − 1 components do not affect the value of the objective.
Thus, we call θ k ∈ arg min θ i ∈
θ D [Equation] d sa µ || d sa θ k ) the active leader’s component .
Conveniently, we can update the follower’s parameters w.r.t.
the gradient ∇ µ f ( θ k , µ ) , which is differentiable w.r.t.
µ .
The following proposition provides the formula for this gradient.
Proposition 5.1 (Follower’s Gradient) .
Let ( θ , µ ) ∈
Θ K , the gradient of f ( θ , µ ) w.r.t.
µ is given by
[Equation]
where θ k is the leader’s component such that θ k ∈
[Equation]
To perform a full optimization of the follower’s parameters, we just need to repeatedly apply the gradient ascent update with the gradient ∇ µ f ( θ , µ ) computed as in (3).
Under mild conditions on the learning rate (Robbins & Monro, 1951), this process is guaranteed to converge to a stationary point such that ∇ µ f ( θ , µ ) = 0 .
We call the follower’s parameters µ at this stationary point the best response to the leader’s parameter θ , and we denote it as µ br .
5.2. Optimizing the Leader’s Parameters
Whenever the follower converges at the best response µ br to the current leader’s parameters, we would like to make an update to θ in the direction of the gradient ∇ θ f
( θ , µ ) , i.e., θ ← θ − α ∇ θ f
( θ , µ ) .
Just as before, we can pre-compute the active leader’s component θ k ∈ arg min θ i ∈
θ D 2 ( d sa µ || d sa θ i ) to make an update to θ k in the direction of the gradient ∇ θ k f ( θ k , µ ) , which is differentiable in θ k .
Indeed, an update to any other leader’s component would not have a meaningful impact on the value of the objective, whereas updating θ k with a sufﬁciently small learning rate α is guaranteed to decrease f ( θ , µ ) , possibly forcing the follower to change its best response in the next epoch.
The following proposition provides the formula for the gradient.
Proposition 5.2 (Leader’s Gradient) .
Let ( θ , µ ) ∈
Θ K , the gradient of f ( θ , µ ) w.r.t.
θ k is given by
[Equation]
5.3. Assessing the Global Value of the Leader’s Parameters
The last missing piece of the PSCA algorithm requires verifying that the leader’s strategy in the DSE ( θ ∗ , µ ∗ ) obtained from the GDA procedure is actually a σ -compression of Θ in M .
In principle, we should verify that min k ∈ [ K ] D 2 ( θ ∗ , µ ) ≤ σ for any µ ∈ Θ , which is equivalent to controlling if max µ ∈ Θ
f ( θ ∗ , µ ) ≤ σ .
Unfortunately, the follower’s strategy µ ∗ is only locally optimal.
Thus, checking f ( θ ∗ , µ ∗ ) ≤ σ is not sufﬁcient, as the globally optimal follower’s strategy might attain a greater value of f than µ ∗ .
Instead, we should check Z θ ∗ ≤ σ , where Z θ ∗ is given by
[Equation]
[Equation]
which can be written as a quadratically constrained quadratic program (see Appendix B.1).
It might come as no surprise that solving this problem is NP-hard.
Indeed, this is equivalent to the problem (2) with a ﬁxed leader’s strategy θ ∗ , but the objective f ( θ ∗ , µ ) is still non-concave w.r.t.
µ .
Luckily, we can reformulate this NP-hard problem in the surrogate linear program (see Appendix B.2):
[Equation]
where the value ( Z θ ∗ ) 2 is a conservative approximation of
Z θ ∗ , as stated in the following theorem:
Theorem 5.3.
The value ( Z θ ∗ ) 2 is an upper bound to the value Z θ ∗ , i.e., ( Z θ ∗ ) 2 ≥ Z θ ∗ , ∀ θ ∗ ∈
Θ K .
6. Theoretical Guarantees of RL with a Compressed Policy Space
In the previous sections, we have motivated the pursuit of a compression of the original policy space Θ in the CMP
M as a way to improve the computational and sample efﬁciency of solving RL tasks deﬁned upon M , and we have presented a viable methodology for extracting such a compression Θ σ .
Since this compression procedure induces a loss, albeit bounded, in the expressive power of the policy space, it is worth investigating the performance guarantees that we have when addressing RL tasks with Θ σ .
We ﬁrst analyze policy evaluation (Section 6.1) and then policy optimization (Section 6.2).
The reported theoretical results mostly combine techniques from (Metelli et al., 2018; Papini et al., 2019).
6.1. Policy Evaluation
In policy evaluation (Sutton & Barto, 2018), we aim to estimate the performance J ( θ ) of a target policy θ ∈ Θ through sampled interactions with an MDP M R .
In our case, we can only draw samples with the policies in Θ σ , and we have to provide an off-policy estimate of J ( θ ) via importance sampling.
Since for any target policy θ we are guaranteed to have a sampling policy θ (cid:
) ∈ Θ σ such that D 2 ( d sa θ ||
d sa θ (cid:48) ) ≤ σ , by choosing a convenient sampling policy in Θ σ , we can enjoy the following guarantee on the error we would make when evaluating any target policy θ ∈ Θ in any MDP M R one can build upon M .
Theorem 6.1 (Evaluation Error) .
Let Θ σ be a σcompression of Θ in M , let R be a reward function for
M uniformly bounded by R max , let θ ∈ Θ be a target policy, and let δ ∈ (0 , 1) be a conﬁdence.
There exists θ (cid:48)
∈
Θ σ such that, given N samples from θ (cid:48) , the error of the importance sampling evaluation of J ( θ ) in M R , i.e., (cid:98) J IS [Equation]
[Equation]
=
[Equation]
is upper bounded as |
[Equation]
with probability at least 1 − δ .
Notably, given a budget of samples N , a conﬁdence δ , and a requirement on the evaluation error beforehand, we could select a proper σ to build a σ -compression that meets the requirement in any policy evaluation task.
However, choosing a sampling policy θ (cid:48) ∈
Θ σ that is best suited for a given task might be non-trivial.
Thus, one can instead take a batch of N k samples with each policy in Θ σ , and then perform the policy evaluation via Multiple Importance Sampling (MIS) (Owen, 2013; Papini et al., 2019).
Corollary 6.2.
Let Θ σ be a σ -compression of Θ in M such that | Θ σ | = K , let R be a reward function for M uniformly bounded by R max , let θ ∈ Θ be a target policy, and let δ ∈ (0 , 1) be a conﬁdence.
Given N k samples from each θ k ∈ Θ σ , the error of the multiple importance sampling evaluation of J ( θ ) in M R , i.e.,
[Equation]
/ θ 1 , . . , θ K ) = (1 − γ ) K (cid:88) k
=
N k (cid:88) n = d sa
[Equation]
[Equation]
a
[Equation]
k ) (cid:80)
[Equation]
=
[Equation]
N
[Equation] d sa
[Equation]
j ( s n
[Equation]
, a n,k )
R ( s n,k , a n,k ) , is upper bounded as | J ( θ ) − (cid:98) J MIS ( θ / θ 1 , . .
, θ K ) | ≤ R max 1 − γ (cid:112) D 2 ( d sa θ || Φ) /δN
with probability at least 1 − δ , where N = (cid:80)
Kk =
1 N k is a number of samples and Φ = (cid:80)
Kk =1 N k N d sa θ k is a ﬁnite mixture.
Thanks to the result in (Metelli et al., 2020b, Theorem 1), in tabular MDPs the evaluation error of the MIS estimator is guaranteed to be lower than the one of the IS estimator of
Theorem 6.1 (as long as N k ≥ N , where N is the number of samples considered by the IS estimator).
6.2. Policy Optimization
In policy optimization (see Section 2.2), we seek for the policy θ that maximizes J ( θ ) within a parametric policy space.
In principle, we could look for the policy that maximizes the performance within the σ -compression Θ σ , which can be found efﬁciently with the OPTIMIST algorithm (Papini et al., 2019).
Especially, in this setting OPTIMIST yields constant regret for tabular MDPs (Metelli et al., 2020a), as the set Θ σ is ﬁnite and it is composed of stochastic policies such that ∀ θ , θ
(cid:48) ∈
Θ σ , D 2 ( d sa θ ||
d sa θ (cid:48) ) < ∞ .
However, this optimal policy within Θ σ might be sub-optimal w.r.t.
the optimal policy within the original policy space Θ .
We can still upper bound this sub-optimality, as reported in the following theorem.
Theorem 6.3 (Policy Optimization in Θ σ ) .
Let Θ σ be a σ -compression of Θ in M , and let R be a reward function for M uniformly bounded by R max .
The policy θ ∗ σ ∈ arg max θ ∈ Θ σ J ( θ ) is (cid:15) -optimal for the MDP M R , where (cid:15) :
[Equation]
Notably, the latter guarantee does not involve any estimation, and the policy θ ∗ can be obtained in a ﬁnite number of interactions.
Nonetheless, one can shrink even more the sub-optimality (cid:15) , and without deteriorating the sample complexity, by coupling the OPTIMIST algorithm with an additional ofﬂine optimization procedure.
The idea is to return the policy θ ∈ Θ that maximizes the importance sampling evaluation obtained with the samples from the policies in Θ σ .
Theorem 6.4 (Off-Policy Optimization in Θ ) .
Let Θ σ be a σ -compression of Θ in M such that | Θ σ | = K , let R be a reward function for M uniformly bounded by R max , and let δ ∈ (0 , 1) be a conﬁdence.
Given N k samples from each θ k ∈ Θ σ , we can recover an (cid:15) -optimal policy for M R as (cid:0) _ , θ ∗ IS (cid:1) ∈
[Equation]
such that (cid:15) := [Equation] ≤ R
[Equation]
σ
[Equation]
N
[Equation]
δ with probability at least 1 − δ .
Although, contrary to the guarantee in Theorem 6.3, (cid:15) vanishes with the number of samples in the latter result, solving problem (7) is non-trivial in general, as the policy space Θ is often inﬁnite.
7. Numerical Validation
In this section, we provide a brief numerical validation of the policy space compression problem (Section 7.1) and how it beneﬁts RL (Section 7.2, 7.3).
To the purpose of the analysis, we consider the River Swim domain (Strehl & Littman, 2008), in which an agent navigates a chain of six states by taking one of two actions: either swim up , to move upstream towards the upper states, or swim down , to go downstream back to the lower states.
In Appendix C, we report further details on the experimental settings, along with some additional results in a Grid World environment.
We leave as future work a more extensive experimental evaluation of the policy space compression problem beyond toy domains.
7.1. Policy Space Compression
In the River Swim, we consider the policy space Θ ⊆ R
|S|×
( |A|− 1) of the softmax policies π θ ( a | s ) =
exp( θ sa ) / (cid:80) j ∈A exp( θ sj ) , and we seek for a compression Θ σ with the requirement σ = 10 , such that Θ σ is a valid σ -compression if min θ ∈ Θ σ max µ ∈ Θ
f ( θ , µ ) ≤ 10 .
In Figure 1a, we report the values of Z = max µ ∈ Θ f
( θ , µ ) (5) and its upper bound Z ≥ Z (6).
Especially, we can see that
PSCA effectively founds a valid σ -compression Θ σ of just
K = 3 policies (Figure 1a, left), and that the values of Z and Z smoothly decreases during the GDA procedure for a ﬁxed number of policies (Figure 1a, right).
Notably, K =policies are actually sufﬁcient to meet the σ requirement in this setting.
However, PSCA cannot access Z but its conservative approximation Z , and thus stops whenever Z ≤ σ .
In Appendix C, we report an illustration of the obtained policies θ k ∈ Θ σ .
7.2. Policy Evaluation with a Compressed Policy Space
We now aim to show that the obtained σ -compression Θ σ can be employed with beneﬁt in the most challenging policy evaluation task one can deﬁne in the River Swim, which is the off-policy evaluation of an (cid:15) -greedy policy θ for the reward function R that assigns R max = 100 for taking the action swim up in the ﬁnal state.
In Figure 1c, we show that sampling with the policies θ 1 , θ 2 ∈ Θ σ lead to an IS off-policy evaluation that is comparable to the exact
J ( θ ) (dashed line) and its on-policy estimate ( θ ).
Even by sampling from a uniform mixture of the policies in Θ σ , the performance of the MIS evaluation is signiﬁcantly better than the one obtained by a uniform mixture of three random policies ( Θ 3 ), as reported in Figure 1d.
For both the IS and the MIS regime, we provide the empirical evaluations (on the left) and the hindsight evaluations (right) obtained with the exact values of the importance weights w θ / θ (cid:48) and the conﬁdence bounds of the Theorem 6.1, 6.2 respectively.
7.3. Compression for Policy Optimization
Finally, we show that the compression Θ σ allows for efﬁcient policy optimization.
We consider the same reward function R of the previous section, and the OPTIMIST (Papini et al., 2019) algorithm equipped with Θ σ , or a uniform discretization of the original policy space Θ with either three policies ( Θ 3 ) or twenty policies ( Θ 20 ).
In Figure 1b, we show that OPTIMIST with Θ σ swiftly converges (less than ﬁve iterations) to the optimal policy within the space.
Instead, the policy space Θ 3 leads to a huge sub-optimality in the ﬁnal performance, and OPTIMIST with Θ 20 is way slower to converge to the optimal policy within the space.
These results are a testament of the ability of PSCA to incorporate the peculiar structure of the domain in a small set of representative policies Θ σ , and to allows for a remarkable balance between sample efﬁciency and sub-optimality in subsequent policy optimization.
8. Conclusions
In this paper, we considered the problem of compressing an inﬁnite parametric policy space into a ﬁnite set of representative policies for a given environment.
First, we provided a formal deﬁnition of the problem, and we highlighted its inherent hardness.
Then, we proposed a tractable gametheoretic reformulation, for which a locally optimal solution can be efﬁciently found through an iterative GDA procedure.
Finally, we provided a theoretical characterization of the guarantees that the compression brings to subsequent RL [Figure] [Figure] number of policies
Z Z
[Figure] 200   iteration Z Z σ (a) Policy Space Compression       iteration J ( θ )
OPTIM.
( Θ σ ) OPTIM.
( Θ 3 ) OPTIM( Θ 20 ) (b) Policy Optimization θ
[Figure]
θ
[Figure]
θ
[Figure]
[Figure]
J
[Figure]
( θ ) θ 1 θ 2 θ
[Figure]
U  b
J IS ( θ )
[Figure]
IS
[Figure]
Evaluation Θ σ Θ 3  b
J MIS ( θ ) Θ σ Θ 3  b
J MIS ( θ )
[Figure]
tasks, and a numerical validation of the proposed approach.
Future works might target the compression problem from interactions with an unknown environment, to pave the way for scalable policy space compression.
Especially, such an extension would require sample-based estimates of the gradients (3), (4), and the global guarantee (6).
Whereas estimating the gradients of state-action distributions is not an easy feat, previous works provide useful inspiration (Morimura et al., 2010; Schroecker & Isbell, 2017; Schroecker et al., 2018).
Other interesting future directions include an extension of the policy space compression problem to the parameter-based perspective (Sehnke et al., 2008; Metelli et al., 2018;
Papini et al., 2019), and the development of policy optimization algorithms that are tailored to exploit a compression of the policy space.
A. Proofs A.1. Proofs of Section
Proposition 5.1 (Follower’s Gradient) .
Let ( θ , µ ) ∈
Θ K , the gradient of f ( θ , µ ) w.r.t.
µ is given by
[Equation]
where θ k is the leader’s component such that θ k ∈ arg [Equation]
Proof.
Let θ k be the active leader’s component, i.e., θ k ∈ arg min θ i ∈ θ D 2 ( d sa µ || d sa θ i ) .
We can compute the gradient of the objective f ( θ , µ ) w.r.t.
µ as
[Equation]
Proposition 5.2 (Leader’s Gradient) .
Let ( θ , µ ) ∈
Θ K , the gradient of f ( θ , µ ) w.r.t.
θ k is given by
[Equation]
Proof.
We can compute the gradient of the objective f ( θ , µ ) w.r.t.
θ k ∈ θ as
[Equation]
Theorem 5.3.
The value ( Z θ ∗ ) 2 is an upper bound to the value Z θ ∗ , i.e., ( Z θ ∗ ) 2 ≥ Z θ ∗ , ∀ θ ∗ ∈
Θ K .
Proof.
The result is straightforward from
[Equation]
A.2. Proofs of Section
Lemma A.1 (Variance of the IS Estimator) .
Let M be a CMP, and let θ ∈ Θ be a target policy.
Let { s n , a n } Nn =1 be a sample of state-action pairs taken with the policy θ (cid:48) in M .
Then, the variance of the importance sampling evaluation of
J ( θ ) in M , i.e., (cid:98) J IS ( θ / θ (cid:48) )
=
[Equation]
N [Equation] Nn =
[Equation]
w [Equation] R
[Equation]
s n , a n ) , can be upper bounded as
[Equation]
Proof.
The proof follows the derivation in (Metelli et al., 2018, Lemma 4.1).
When considering state-action pairs (as opposed to trajectories in (Metelli et al., 2018)) one should account for the dependency between state-actions in the same trajectory.
Here we consider a batch of N samples taken within a single trajectory, in which the dependency vanishes as the
CMP mixes to the steady-state, and thus it can be neglected.
Especially, we write
[Equation]
In episodic settings, one can reﬁne this result to account for dependent data by exploiting the Bellman equation of the variance (see Sobel, 1982; Xie et al., 2019).
Theorem 6.1 (Evaluation Error) .
Let Θ σ be a σ -compression of Θ in M , let R be a reward function for M uniformly bounded by R max , let θ ∈ Θ be a target policy, and let δ ∈ (0 , 1) be a conﬁdence.
There exists θ (cid:48)
∈
Θ σ such that, given N samples from θ (cid:48) , the error of the importance sampling evaluation of J ( θ ) in M R , i.e., (cid:98) J IS ( θ / θ (cid:48) )
= 1 (1 − γ ) N (cid:80)
Nn =1 w θ / θ (cid:48) ( s n , a n ) R ( s n , a n ) , is upper bounded as | J ( θ ) − (cid:98)
J IS ( θ / θ (cid:48) )
| ≤ R max 1 − γ (cid:112) σ/δN with probability at least 1 − δ .
Proof.
We would like to bound the difference | J ( θ ) − (cid:98)
J IS ( θ / θ (cid:48) )
| for a policy θ (cid:48) ∈
Θ σ .
By the deﬁnition of σ -compression, there exists at least a policy θ (cid:48)
∈ Θ σ such that D 2 ( d sa θ ||
d sa θ (cid:48) ) ≤ σ .
Since the IS estimator (cid:98) J IS ( θ / θ (cid:48) ) is unbiased, and
V ar ( s,a ) ∼ d sa θ (cid:48) (cid:2) (cid:98) J IS ( θ / θ (cid:48) ) (cid:3) < ∞ through Lemma A.1, we can use the Chebichev’s inequality to write, ∀ (cid:15) > 0 ,
[Equation]
Then, by calling δ = V ar ( s,a
[Equation]
and considering the complimentary event, we get
[Equation]
where we upper bounded the variance of (cid:98) J IS ( θ / θ (cid:48) )
as in Lemma A.1 and the Rényi D 2 ( d sa θ || d sa θ (cid:48) ) with σ .
Corollary 6.2.
Let Θ σ be a σ -compression of Θ in M such that | Θ σ | = K , let R be a reward function for M uniformly bounded by R max , let θ ∈ Θ be a target policy, and let δ ∈ (0 , 1) be a conﬁdence.
Given N k samples from each θ k ∈ Θ σ , the error of the multiple importance sampling evaluation of J ( θ ) in M R , i.e.,
[Equation]
(cid:88) [Equation]
= [Equation] R
[Equation]
s
[Equation]
, is upper bounded as | J ( θ ) − (cid:98) J MIS ( θ / θ 1 , . .
, θ K ) | ≤ R max 1 − γ (cid:112) D 2 ( d sa θ || Φ) /δN
with probability at least 1 − δ , where
N = (cid:80) Kk
=1 N k is a number of samples and Φ = (cid:80)
Kk =
N k N d sa θ k is a ﬁnite mixture.
Proof.
Through the combination of (Papini et al., 2019, Lemma 1) and Lemma A.1, it is straightforward to derive
[Equation]
Then, similarly as in Theorem 6.1, we can use the Chebichev’s inequality to write, ∀ (cid:15) > 0 ,
[Equation]
By calling δ =
[Equation]
and considering the complimentary event, we get
[Equation]
where we upper bounded the variance of (cid:98) J MIS ( θ / θ 1 , .
, θ K ) as in (8).
Theorem 6.3 (Policy Optimization in Θ σ ) .
Let Θ σ be a σ -compression of Θ in M , and let R be a reward function for M uniformly bounded by R max .
The policy θ ∗ σ ∈ arg max θ ∈ Θ σ J ( θ ) is (cid:15) -optimal
for the MDP M R , where (cid:15) :
= | max [Equation] ∈ [Equation]
θ
[Equation]
− J ( θ ∗
[Equation]
) |
[Equation]
Proof.
Let be θ ∗ ∈ arg max θ ∈ Θ J ( θ ) .
From the deﬁnition of σ -compression we have that there exists at least a policy θ (cid:48)
∈ Θ σ such that D 2 ( d sa θ ∗ ||
d sa θ (cid:48) ) ≤ σ .
Then, we can write
[Equation]
where (9) is from the deﬁnition of J given in Section 2.2, (11) is obtained from (10) through the Pinsker’s inequality, and (12) derives from d KL ( p || q ) =
d 1 ( p || q ) ≤ d 2 ( p || q ) =
D 2 ( p || q ) , which is straightforward from the deﬁnition of Rényi divergence.
Finally, it is trivial to see that J ( θ ∗ σ ) ≥ J ( θ (cid:48) ) for θ ∗ σ ∈ arg max θ ∈ Θ σ J ( θ ) .
Theorem 6.4 (Off-Policy Optimization in Θ ) .
Let Θ σ be a σ -compression of Θ in M such that | Θ σ | = K , let R be a reward function for M uniformly bounded by R max , and let δ ∈ (0 , 1) be a conﬁdence.
Given N k samples from each θ k ∈ Θ σ , we can recover an (cid:15) -optimal policy for M R as
[Equation]
such that
[Equation]
probability
[Equation]
Proof.
Thanks to the deﬁnition of σ -compression and the guarantee provided by Theorem 6.1, from the collected samples we have that there exists θ k ∈ Θ σ such that
[Equation]
holds ∀ θ ∈ Θ with probability at least 1 − δ/ 2 .
Then, let θ ∗ IS be a policy obtained as in (7), and let θ ∗ ∈ arg max θ ∈ Θ J ( θ ) .
We consider the event in which J ( θ ∗ IS ) falls below its lower conﬁdence bound and J ( θ ∗ ) exceeds its upper conﬁdence bound.
It is easy to see that this event happens with probability at most δ , whereas the complimentary event guarantees that
[Equation]
B. Optimization Problems B.1. Quadratic Program Formulation of (5)
The optimization problem in (5) can be formulated into a quadratically constrained quadratic program as maximize z ∈ R , ω ∈ R SA z subject to
[Equation]
B.2. Linear Program Formulation of (6)
The optimization problem in (6) can be formulated into a linear program as maximize z ∈ R , ω ∈ R SA z subject to
[Equation]
s | s (cid:48) , a (cid:48) )
d s (cid:48)
d a (cid:48) , ∀ s ∈ S ω ( s, a ) ≥ 0 , ∀ s ∈ S , ∀ a ∈ A .
C. Further Details on the Numerical Validation
In Section 7.1, we commented the results of PSCA in the River Swim domain.
For the sake of clarity, here we report an illustration of the River Swim CMP (Figure 2a), heatmap visualizations of the policies in the σ -compression obtained by PSCA (Figure 2b-2d), and the set of parameters we employed ( σ = 10 , α = 0 .
005 , β = 0 . 1 ).
We further report the results of an additional policy space compression experiment in a Gridworld domain ( |S| = 9 , |A| = 4 ).
In this setting, we considered σ = 40 , α = 0 .
005 , β = 0 .
1 , and the resulting σ -compression is composed of K = 4 policies (a visualization is provided in Figure 3a-3d).
In Section 7.2, we reported a set of policy evaluation experiments in the River Swim domain.
Especially, we considered an
IS off-policy evaluation setting, in which we take a batch of samples with each policy θ k ∈ Θ σ , or with a uniform policy θ U , or with the target policy itself θ .
For every policy, the batch is composed of N = 100000 samples, and it is obtained by drawing 5000 trajectories of 20 steps.
Similarly, we considered a MIS off-policy evaluation setting, in which we take a batch of samples with the σ -compression Θ σ , or a set of three random policies Θ 3 .
In both the cases, the batch is composed of N = 300000 samples ( N k = 100000 for each policy in the space), obtained by drawing 15000 trajectories of 20 steps.
In Section 7.3, we reported a policy optimization experiment in the River Swim domain.
To run this experiment, we implemented the action-based formulation of the OPTIMIST algorithm (Papini et al., 2019, Algorithm 1).
For each seed, we run the algorithm for 100 iterations, in each iteration we collect N = 1000 samples, which are obtained from 50 trajectories of 20 steps.
The value of the importance weights truncation M and the conﬁdence schedule δ t are taken from the theoretical analysis in (Papini et al., 2019).
[Figure] [Figure]