T OWARDS A U NIFIED P OLICY A BSTRACTION T HEORY AND R EPRESENTATION L EARNING A PPROACH IN M ARKOV D ECISION P ROCESSES
A BSTRACT
Lying on the heart of intelligent decision-making systems, how policy is represented and optimized is a fundamental problem.
The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios.
Towards a desirable surrogate policy space, recently policy representation in a low-dimensional latent space has shown its potential in improving both the evaluation and optimization of policy.
The key question involved in these studies is by what criterion we should abstract the policy space for desired compression and generalization.
However, both the theory on policy abstraction and the methodology on policy representation learning are less studied in the literature.
In this work, we make very first efforts to fill up the vacancy.
First, we propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels.
Then, we generalize them to three policy metrics that quantify the distance (i.e., similarity) of policies, for more convenient use in learning policy representation.
Further, we propose a policy representation learning approach based on deep metric learning.
For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively.
Our experiments are conducted in both policy optimization and evaluation problems, containing trust-region policy optimization (TRPO), diversity-guided evolution strategy (DGES) and off-policy evaluation (OPE).
Somewhat naturally, the experimental results indicate that there is no a universally optimal abstraction for all downstream learning problems; while the influence-irrelevance policy abstraction can be a generally preferred choice.
I NTRODUCTION
How to obtain the optimal policy is the ultimate problem in many decision-making systems, such as
Game Playing (Mnih et al., 2015), Robotics Manipulation (Smith et al., 2019), Medicine Discovery (Schreck et al., 2019).
Policy, the central notion in the aforementioned problem, defines the agent’s behavior under specific circumstances.
Towards solving the problem, a lot of works carry out studies on policy with different focal points, e.g., how policy can be well represented (Ma et al., 2020; Urain et al., 2020), how to optimize policy (Schulman et al., 2017a;
Ho & Ermon, 2016) and how to analyze and understand agents’ behaviors (Zheng et al., 2018; Hansen & Ostermeier, 2001).
The root challenge to the studies on policy is the large scale and the high complexity of policy space, especially in real-world scenarios.
As a consequence, the difficulty of policy learning is escalated severely.
Intuitively and naturally, such issues can be significantly alleviated if we have an ideal surrogate policy space, which are compact in scale while keep the key features of policy space.
Related to this direction, low-dimensional latent representation of policy plays an important role in Reinforcement Learning (RL) (Tang et al., 2020),
Opponent Modeling (Grover et al., 2018),
Fast Adaptation (Raileanu et al., 2020; Sang et al., 2022), Behavioral Characterization (Kanervisto et al., 2020) and etc.
In these domains, a few preliminary attempts have been made in devising different policy representations.
Most policy representations introduced in prior works resort to encapsulating the information of policy distribution under interest states (Harb et al., 2020; Pacchiano et al., 2020), e.g., learning policy embedding by encoding policy’s state-action pairs (or trajectories) and optimizing a policy recovery objective (Grover et al., 2018; Raileanu et al., 2020).
Rather than policy distribution, some other works resort to the information of policy’s influence on the environment, e.g., state(-action) visitation distribution induced by the policy (Kanervisto et al., 2020;
Mutti et al., 2021).
Recently, Tang et al.
(2020) offers several methods to learn policy representation through policy contrast or recovery from both policy network parameters and interaction experiences.
Put shortly, the key question of policy representation learning is by what criterion we should abstract the policy space for desired compression and generalization.
Unfortunately, both a unified theory on policy abstraction and a systematic methodology on policy representation are currently missing.
In this paper, we make first efforts to fill up the plank in both the theory and methodology.
Inspired by the state abstraction theory (Li et al., 2006), first we introduce a unified theory of policy abstraction.
We start from proposing three types of policy abstraction: distribution-irrelevance , influence-irrelevance , and value-irrelevance .
They follow different abstraction criteria, each of which concerns distinct features of policy.
Concretely, we utilize the exact equivalence relations between policies and derive the corresponding policy abstractions.
Further, we generalize the exact equivalence relations to policy metrics, allowing quantitatively measure the distance (i.e., similarity) between policies.
Such policy metrics are more informative than the binary outcomes of policy equivalence and thus provide more usefulness in policy representation learning.
Moreover, towards applying practical policy representation in downstream learning problems, we introduce a policy representation learning approach based on deep metric learning (Kaya & Bilge, 2019).
We propose an alignment loss for a unified objective function of learning with different policy metrics.
The policy representation is learned to render the abstraction criterion through minimizing the difference between the distance of policy embeddings and the quantity measure by the policy metrics.
In particular, we use Maximum
Mean Discrepancy (Gretton et al., 2012; Nguyen-Tang et al., 2021) for efficient empirical estimation of the policy metrics; and we adopt Layer-wise Permutation-invariant Encoder (Tang et al., 2020) for structure-aware encoding of the parameters of policy network.
In addition to the theoretical understanding of policy abstraction, we further investigate the empirical efficacy of different policy metrics and representations in characterizing policy difference and conveying policy generalization respectively.
We conduct experiments in both policy optimization and policy evaluation problems.
For policy optimization, we adopt Trust-Region Policy Optimization (TRPO) and Diversity-Guided Evolution Strategy (DGES) as the problem settings from (Kanervisto et al., 2020), covering both gradient-based and gradient-free policy optimization.
For policy evaluation, we consider Off-policy Evaluation (OPE).
In particular, we establish a series of OPE settings with different configurations of training data and generalization tasks.
These settings reflect the circumstances often encountered in RL.
Our experimental results indicate that, somewhat naturally, there is no a universally optimal abstraction for all downstream learning problems.
Additionally, it turns out that the influence-irrelevance abstraction can be a preferred choice in general cases.
Our main contributions are summarized as follows: 1) We focus on the general policy abstraction problem and to our knowledge, we propose a unified theory of policy abstraction along with several policy metrics for the first time.
We propose a unified policy representation learning approach based on deep metric learning.
We empirically evaluate the efficacy of our proposed policy representations in multiple fundamental problems (i.e., TRPO, DGES and OPE).
B ACKGROUND
Reinforcement Learning We consider a Markov Decision Process (MDP) (Puterman, 2014) typically defined by a five-tuple ⟨ S, A, P, R, γ ⟩ , with the state space S , the action space A , the transition probability P : S ×
A → ∆( S ) , the reward function R : S ×
A → R and the discount factor γ ∈ [0 , 1) .
∆( X ) denotes the probability distribution over X .
A stationary policy π : S → ∆( A ) is a mapping from states to action distributions, which defines how to behave under specific states.
An agent interacts with the MDP at discrete timesteps by its policy π , generating trajectories with s 0 ∼ ρ 0 ( · ) , a t ∼ π ( ·| s t ) , s t +1 ∼ P ( · | s t , a t ) and r t =
R ( s t , a t ) , where ρ 0 is the initial state distribution.
We use P π ( s ′ | s ) =
E a ∼ π ( ·| s ) P ( s ′ | s, a ) to denote the distribution of next state s ′ when performing policy π at state s .
For a policy π , the return G t = (cid:80) ∞
t =0 γ t r t is the random variable for the sum of discounted rewards while following π , whose distribution is denoted by Z π .
The value function of policy π defines the expected return for state s , i.e., V π ( s ) =
E π
[ G t | s 0 = s ] .
The goal of an RL agent is to learn an optimal policy π ∗ that maximizes J ( π ) =
E s 0 ∼ ρ 0 ( · ) [ V π ( s 0 )] .
Metric Learning
Here we recall the standard definition of metrics which is central to our work.
Definition 1 (Metrics (Royden, 1968)) .
Let X be a non-empty set of data elements and a metric is a real-valued function d : X × X → [0 , ∞ ) such that for all x, y, z ∈ X : 1) d ( x, y )
=
⇐⇒ x = y ; 2) d ( x, y ) =
d ( y, x ) ; 3) d ( x, y ) ≤ d ( x, z ) + d ( z, y ) .
A pseudo-metric d is a metric with the first condition replaced by x = y =
⇒ d ( x, y ) = 0 .
The combination ⟨ X, d ⟩ is called a metric space.
A metric d is often used to quantify the distance between two data elements in a general sense.
In this paper, we will also use metric to stand for pseudo-metric for brevity.
Typically metric learning aims to reduce the distance between similar data and increase the distance between dissimilar data.
With nonlinear transformation offered by deep neural networks, Deep Metric Learning allows us to find such optimal metrics by optimizing a latent representation space of raw data.
P OLICY A BSTRACTION T HEORY
Inspired by the state abstraction theory (Li et al., 2006), in this section, we make the first effort in proposing a unified policy abstraction theory.
First, we propose the formal definition of three types for policy abstraction; then, we generalize the abstractions to three types of policy metrics.
Finally, we analyze the properties of policy abstraction and compare them in several Gridworld MDPs.
3. P OLICY A BSTRACTION
First of all, following the classic definition of an abstraction (Giunchiglia & Walsh, 1992), we propose a general definition of policy abstraction as follows:
Definition 2 (Policy Abstraction) .
A policy abstraction f : Π → X , is a mapping from ground policy space Π to an abstract space X .
f ( π ) ∈ X is the abstract policy representation corresponding to ground policy π ∈ Π , and the inverse image f − 1 ( χ ) with χ ∈ X , is the set of ground policies that correspond to χ under abstraction function f .
It is apparent that there are many such abstractions since we may have many possible ways to partition the policy space.
However, we are only interested in some useful ones among them that follow specific abstraction criteria to preserve the important features related to decision making.
In this paper, we present three types of policy abstraction which are defined below:
Definition 3.
Given an MDP and a ground policy space Π , for any two policies π i , π j ∈ Π , we define three types of policy abstraction as follows:
A distribution-irrelevance abstraction ( f π ) is such that for all s ∈ S , a ∈ A , f π ( π i ) =
f π ( π j ) implies that π i ( a | s ) =
π j ( a | s ) .
An influence-irrelevance abstraction ( f P π ) is such that for all s, s ′ ∈ S , f P π ( π i )
=
f P π ( π j ) implies that P π i ( s ′ | s ) =
P π j ( s ′ | s ) .
A value-irrelevance abstraction ( f V π ) is such that for all s ∈ S , f V π ( π i ) =
f V π ( π j ) implies that V π i ( s ) =
V π j ( s ) .
These abstractions aggregate policies based on the corresponding equivalence relations with respective concerns on different features of policy.
Intuitively, the distribution-irrelevance abstraction ( f π ) preserves the action distribution of the policy; the influence-irrelevance abstraction ( f P π ) preserves the state transition distribution induced by the policy, i.e., the influence caused by the policy on the environment; and value-irrelevance abstraction ( f V π ) preserves the value function of the policy.
In addition to the policy abstractions introduced in Definition 3, we provide some other ones in Appendix
A.2.
Moreover, we revisit the policy abstractions adopted in prior related works and summarize them from the angle of our policy abstraction theory in Table 4 of Appendix B.
3. P OLICY M ETRICS
The policy abstractions allow us to aggregate policies according to equivalence relation.
However, exact equivalence is rarely encountered in continuous policy space (e.g., the usual case with neural policies), thus useful abstraction can be seldom obtained.
Moreover, the equivalence relation offers only qualitative (i.e., binary) outcomes and is incapable of measuring the similarity between policies, which is significant to policy representation learning.
To this end, we generalize the policy abstractions to policy metrics which quantitatively measures the distance between two policies.
Corresponding to the three types of policy abstraction, we define the following three policy metrics:
Definition 4.
Given an MDP, a ground policy space Π , a state distribution p ( s ) and a distribution (pseudo-)metric D ( · , · ) , for any two policies π i , π j ∈ Π , we define three policy metrics as follows:
[Table]
A distribution-irrelevance metric: d π ( π i , π j ) =
E s ∼ p
( s )
[ D ( π i ( a | s ) , π j ( a | s ))] .
An influence-irrelevance metric: d P π ( π i , π j ) =
E s ∼ p
( s )
[ D ( P π i ( s ′ | s ) , P π j ( s ′ | s ))] .
A value-irrelevance metric: d V π ( π i , π j ) =
E s ∼ p
( s )
[ D ( Z π i ( s ) , Z π j ( s ))] .
These metrics follow the same abstraction criteria as in Definition 3,
i
[Table]
e., the irrelevance regarding action distribution, influence and value, measuring the similarity of policies by the distance at respective levels.
Compared to the binary outcomes offered by the equivalence relations, the metrics defined here are continuous, thus are more informative in comparing and representing policies in finer views.
Specially, one may see that the equivalence relations used in Definition 3 induce corresponding discrete pseudo-metrics, e
[Table]
g., d Eq π ( π i , π j ) =
if f π ( π i ) =
f π ( π j ) [Table] and 1 otherwise.
Notice the metrics proposed above depends on the distribution metric D and state distribution p ( s ) .
For D , typical choices can be Jeffreys Divergence (Jeffreys, 1946) and Maximum Mean Discrepancy (MMD) (Nguyen-Tang et al., 2021).
For p ( s ) , intuitively, it should be the distribution of states we are interested in when comparing two policies.
We defer the concrete choices for practical implementation of these metrics in Section 4.
3. P ROPERTIES OF THE A BSTRACTIONS
Superficially, the three abstractions proposed preserve features that are progressively more relevant to decision making in the learning task, but essentially, what is the relationship between the three abstractions?
To investigate the problem, we define the fineness of policy abstractions similar to the one for state abstractions used in (Li et al., 2006), to prove how the three abstractions are related.
Definition 5 (Abstraction Fineness) .
Let F Π denotes the set of abstractions on ground policy space Π .
Suppose f 1 , f 2 ∈ F Π .
We say f 1 is finer than f 2 , denoted f 1 ⪰ f 2 , iff ∀ π 1 , π 2 ∈ Π , f 1 ( π 1 ) =
f 1 ( π 2 ) implies f 2 ( π 1 ) =
f 2 ( π 2 ) .
If , f 1 ̸ = f 2 , then f 1 is strictly finer than f 2 , denoted f 1 ≻ f 2 .
In contrast, we may also say f 2 is (strictly) coarser than f 1 , denoted f 2 ⪯ f 1 ( f 2 ≺ f 1 ).
It is easy to see the relation ⪰ satisfies self-reflexivity, antisymmetry and transitivity, thus it is a partial ordering.
Consider the set of possible policy abstractions, while the coarsest abstraction ( f 0 ) is the trivial representation where all policies are treated as the same; while the finest abstraction is the identity representation, e.g., f Θ ( π θ )
= θ for a policy neural network parameterized with θ ∈ Θ .
With the partial ordering ⪰ , we further derive the following theory.
Theorem 3.1.
Under the Definition 3 and 5, we have ( f Θ ⪰ )
f π ⪰ f P π ⪰ f V π ( ⪰ f 0 ) .
The proof is provided in Appendix A.1.
The theorem declares how the three policy abstractions are related to each other in the sense of abstraction fineness with the two extreme cases ( f Θ , f 0 ) for reference.
The coarser the abstraction is, the more the original policy space is abstracted.
In Table 1, we summarize the properties of different policy abstractions, regarding abstraction criteria, abstraction fineness and task relevance.
The major conclusion is that there is an inverse relation between abstraction fineness and task relevance.
Except for the two extreme cases ( f Θ , f 0 ) that are totally task-independent, the policy abstraction becomes more task-relevant as the abstraction criterion concerns more policy features related to the learning task.
The distribution-irrelevance abstraction f π concerns the policy behavior in the learning task, defined by action distribution at interested states; meanwhile f π is coarser than f Θ since the same policy behavior can be realized by non-unique policy parameter.
Taking one step closer to the task, the influence-irrelevance abstraction f P π cares about the state transition dynamics induced by policy behavior.
Obviously, f P π is coarser than f π as different behaviors may induce the same transition distribution.
The value-irrelevance abstraction f
V π further involves the rewards of long-term dynamics, thus is the most task-relevant and coarsest among the three types of policy abstraction.
Empirical Comparison of Policy Metrics in Gridworld MDPs To compare these policy abstractions in a quantitative view, we demonstrate how the distances of two policies measured by the [Figure] corresponding policy metrics differ in several Gridworld MDPs.
We use Distinct Policies , Doorway from (Kanervisto et al., 2020) and design a new task, Key Action for simple prototypes of tasks with different features; moreover, we increase the stochasticity of the environment for a better evaluation.
In particular, E s ∼ p ( s ) D ( · , · ) is calculated by average the absolute differences over all states.
The illustrations and results are shown in Fig.
1 while more results can be found in Appendix C.
We observe that the distribution-irrelevance metric d π may fail to show the difference in dynamics and outcome between the two policies, e.g., in Doorway.
This is because d π measures the difference in the action distribution itself, i.e., independent of the dynamics as well as the increasing stochasticity of the environment.
This issue may be resolved by using a designated distribution that concentrates over key states.
Conversely, the value-irrelevance d V π measures the difference in the outcomes of the two policies regardless their differences in action distribution and dynamics, e.g., in Distinct Policies.
Another discovery is that d V π quickly degenerates and turns to be not informative as the increase of stochasticity, showing its poor robustness.
By contrast, the influence-irrelevance metric d P π is a sweet intermediate point, consistently keeping the ability of distinguishing the two policies across all the environments and stochasticity configurations.
In a summary, different policy abstractions and metrics may yield different outcomes for the same two policies and the optimality depends on the specific downstream learning problem concerned.
Later, we evaluate these options in representative downstream RL problems in Section 5 and 6 for useful insights.
P OLICY R EPRESENTATION L EARNING A PPROACH
The next question concerned in practice is: how can we learn the representation of RL policies (usually modeled by NNs) in a general way?
Based on the policy metrics introduced above, we propose a policy representation learning approach by following the principle of Deep Metric Learning.
4. L EARNING P OLICY R EPRESENTATION BY E MBEDDING A LIGNMENT
The policy metrics proposed in previous section measure the quantitative relationship between policies from different perspectives of the policy abstraction criteria.
For a unified objective function of learning from different policy metrics, we use the alignment loss , with which the difference between the distances of two policies in the representation space and in the policy metric space is minimized.
Concretely, consider a policy representation function f ψ , and the alignment loss can be formalized as,
[Equation]
where we consider d ∗ ∈ { d π , d P π , d V π } and η is the weight for scaling.
As we can see, L AL ( ψ ) consists of two metrics, i.e., the L 2 distance function ( ∥·∥ 2 ) of two inputs and the policy metric ( d ∗ ( · , · ) ).
Intuitively, minimizing the alignment loss is to align the two metrics by optimizing the policy representation function f ψ .
By this means, we are able to learn different policy representation functions, which maps the ground policy π ∈ Π to the latent embedding χ π = f ψ ( π ) ∈ X .
The embedding preserves the policy features corresponding to the abstraction criteria reflected by the specific policy metric considered.
For a practical implementation, the following problems are the estimation of policy metrics d ∗ and the realization of the training for policy representation function f ψ , which are detailed in the next two subsections respectively.
In the literature of learning policy representation, representative methods follow the principle of behavior recovering (Grover et al., 2018) and policy contrast (Tang et al., 2020).
To our knowledge, none of prior works take a systematic view of policy abstraction.
In Table 4, we show that prior methods are specific instances of one of our proposed policy abstractions that differ in realization.
4. E STIMATING P OLICY M ETRICS VIA M AXIMUM M EAN D ISCREPANCY
Given a tractable metric D and a state distribution p , the policy metrics (i.e., d π , d P π , d V π ) can be calculated exactly if the probability distributions (i.e., π , P π , Z π ) are available.
However, this is usually infeasible in practice; instead, in more regular cases, only finite samples of policy interaction are available.
Although the empirical distributions can be estimated in simple MDPs where the state-action space is finite (as in Appendix D), unfortunately, approximating the distributions and computing the metrics are non-trivial, especially with high-dimensional continuous state-action space.
Therefore, we estimate the policy metrics directly from the samples, bypassing estimating the empirical distributions (i.e., (cid:101) π , (cid:101) P π , (cid:101) Z π ).
In particular, we adopt MMD (Nguyen-Tang et al., 2021; Sejdinovic et al., 2012) as the distribution metric, i.e., let D be D MMD .
MMD measures the maximum value of the mean discrepancy of two distributions regarding all possible functions in a predefined family.
Conventionally, let the class of functions h :
X → R be a unit ball in a Reproducing Kernel
Hilbert Space (RKHS) H associated with a continuous kernel k ( · , · ) on X , p, q be two distribution defined on X , x, x ′ and y,
y ′ be i.i.d.
samples from p and q respectively, the MMD is defined as:
[Equation]
where µ p = (cid:82) X k ( x, · ) p ( d x ) is the mean embedding of p into H (Smola et al., 2007).
Thus, MMD can be empirically estimated with samples { x i } Ni =1 ∼ p and { y i } Mi =1 ∼ q :
[Equation]
According to Eq.
3, we can estimate the policy metrics d π , d P π , d V π empirically from the samples { a i } , { s ′ i } , { G i } of different policies respectively, under the sampled states { s i } for the expectation E p ( s ) .
However, it is often impractical to obtain multiple samples under the same state.
Thus, we resort to estimating the surrogates, e.g., ˆ d P π ( π i , π j ) =
D ( P π i ( s, s ′ ) , P π j ( s, s ′ )) for d P π , where the joint distributions rather than the state-conditioned distributions are measured.
We use Gaussian kernel by default, i.e., k ( x, x ′ ) =
exp (cid:0) − || x − x ′ ||
22 2 σ(cid:1) .
Consequently, the empirical estimates of the policy metrics serve as the self-supervision in Eq. 1.
4. R EALIZING THE T RAINING OF P OLICY R EPRESENTATION F UNCTION
With the empirical policy metrics provided in previous section, the training of policy representation is straightforward with a differentiable function f ψ by optimizing the alignment loss (Eq. 1).
The realization of policy representation function concerns two aspects: 1) the choice of policy data (or original representation) and 2) the construction of f ψ (i.e., how policy data is encoded).
For the first aspect, we focus on parameterized policy π θ (typically by a neural network) and use policy parameter θ as the policy data.
One may recall that θ itself can be viewed as the finest representation obtained by policy abstraction f Θ in Table 1.
Such an original representation (i.e., θ ) is high-dimensional and highly nonlinear, offering no help in the compression and generalization of policy space.
In addition, we are aware that in some cases the policy parameters may be not available, and thus the interaction experiences generated by the policy can be alternative policy data, as used in (Grover et al., 2018; Tang et al., 2020).
Our policy representation learning approach is compatible with such alternatives with the need of possible slight modifications.
For the second aspect, we adopt
Layer-wise Permutation-invariant Encoder (LPE) (Tang et al., 2020) as the implementation choice of f ψ , which has demonstrated the effectiveness in encoding conventional policy networks.
To be specific, for the parameter θ = { W i , b i } ki =0 of policy π , i.e., the weights and biases of k -layer MLP, the weight W i ∈ R l i × l i +1 and bias b i ∈ R 1 × l i +
( l i is the unit number of the i -layer; l 0 and l k are for the input and output layers) are concatenated ( ⊕ ) and transposed, followed by a MLP ( f ψ,i ) and a mean-reduce operation ( MR ), resulting in a layer embedding z i ; Thereafter, the policy embedding is obtained by concatenating the embedding of each layer.
Formally,
[Equation]
Each row of [ W i ⊕ b i ] ⊤ , indexing by the subscript j, · , describes a transformation of the i -layer into the next layer.
All the rows are fed into f ψ,i separately and are then averaged into z i .
In a consequence, the policy embedding serves as the compact representation of the policy network by summarizing the transformations made by the each layer of it.
The significant difference between
LPE and a straightforward MLP encoder is that, LPE provides structure-aware representation, i.e., both the intra-layer and inter-layer structures are explicitly considered.
Intuitively, this alleviates the difficulty of learning representation from the policy network parameters.
Other advanced encoder structures are beyond the scope of this work and we leave them as future work.
Till now, we can update the parameters of LPE ψ = { ψ i } ki =0 by optimizing Eq.
1 with the policy samples from some given dataset of policies and the empirical policy metrics estimated accordingly.
Depending on the specific choice of policy metric, the policy representation is learned to render the policy abstraction in Table 1, starting from f Θ and going downwards to the corresponding level.
A PPLYING P OLICY A BSTRACTION TO P OLICY O PTIMIZATION
Despite the theoretical understanding of the policy abstraction, we have no idea about how the derived policy metrics behave in different downstream learning problems.
To shed some light on this, we evaluate the efficacy of the policy metrics proposed in Sec. 3.2 in policy optimization, including
Trust-Region Policy Optimization (TRPO) and Diversity-Guided Evolutionary Strategy (DGES).
Trust-Region Policy Optimization We adopt TRPO problem as the first test stone for our policy abstractions.
Specifically, the objective of TRPO problem is to maximize the policy return while constraining the difference between old and new policies: J TRPO ( θ ) =
E τ ∼ P
πθ [ R ( τ )] , s.t., d ∗ ( π θ , π θ old ) ≤ σ , where σ is a threshold.
For our experiments, we consider the policy metrics d ∗ ∈ { d π , d P π , d V π } .
In another word, the learning agent checks if the difference measured by the policy metrics are larger than σ for each policy update.
In this experiment, the original
TRPO (Schulman et al., 2015) is generalized to incorporate different alternative metrics for the trust-region constraint.
Thus, we can evaluate the efficacy of the different trust regions provided by our proposed policy metrics, shedding some light on what policy features we care the most in TRPO.
[Figure]
We adopt a Gridworld environment where the agent can move to one of N directions at each grid and only one direction yields high reward (Kanervisto et al., 2020).
The results are shown in Fig. 2(a).
We observe that all our TRPO variants (i.e., TRPO- f ∗ ) outperform Vanilla-PO (i.e., no trust-region constraint used), demonstrating the effectiveness of our policy abstractions.
Moreover, TRPO-
f π outperforms the others.
This is because f π follows the abstraction criterion regarding action distribution, thus pertains to the essence of TRPO.
By contrast, f P π and f V π utilize coarser abstraction which does not hold the features of action distribution.
In addition, we demonstrate the superiority of our policy abstractions when compared with existing related methods in Appendix E.1.
Diversity-Guided Evolution Strategy Next, we adopt DGES problem as the second test stone for our policy abstractions.
Formally, the objective of DGES problem is to maximize the policy return of the current policy π θ and maximize its policy difference to the ancestor policy ¯ π : J DGES ( θ ) =
E τ ∼ P πθ [ R ( τ )]
+ β (cid:80)
Np =1 d ∗ ( π θ , ¯ π ) , where β ≥ 0 is the weight.
Similarly, we consider the policy metrics d ∗ ∈ { d π , d P π , d V π } .
Here, the choices of policy metrics realize the population diversity in different ways.
We aim at exploring the diversity concerning which policy feature is the most effective in DGES.
To explore this, we leverage the Point environment with deceptive rewards (Pacchiano et al., 2020).
The results are reported in Fig. 2(b).
In comparison to Vanilla-ES (i.e., β = 0 ), optimizing policy diversity (i.e., β > 0 ) based on our policy metrics (i.e., DGES- f ∗ ) does help exploration and thus leads to better performance.
In particular, DGES- f P π performs the best.
Since ES optimizes policy in a gradient-free fashion, the evolution process concerns only policy return.
Therefore, the distribution-irrelevance abstraction f π (i.e., the winner in the TRPO experiment) can be redundant since multiple action distributions may have the same outcome (i.e., influence and value).
For the value-irrelevance abstraction f V π , it turns to be too fine to contain the features of policy behavior (i.e., action distribution and influence).
Therefore, the influence-irrelevance abstraction f P π serves as a sweet point.
Furthermore, we provide additional comparative evaluation in Appendix E.2.
A PPLYING P OLICY A BSTRACTION TO O FF - POLICY E VALUATION
After the investigation in policy optimization, now we move to policy evaluation.
Typical OPE (Fu et al., 2021; Harb et al., 2020) focuses on using offline data to evaluate unseen policies.
Likewise, we are interested in studying the value generalization performance on unseen policies of the representations learned regarding different policy abstractions.
The appealing characteristic of policy representation in value generalization has been studied in (Tang et al., 2020), where a Policy-extended
Value Function Approximator (PeVFA, V ( χ π ) ) takes as input the policy representation χ π approximates the values of multiple policies and offers implicit value generalization among the policy representation space.
For policy data collection, we run PPO (Schulman et al., 2017b) in OpenAI Gym continuous control tasks: InvertedDoublePendulum-v2 (IDP-v2) and LunarLanderContinuous-v2 (LLC-v2) (Brockman et al., 2016).
By collecting the policies at intervals during the learning process, we build an offline policy set, based on which we train our policy representations and a PeVFA V ( χ π ) .
For concrete problem settings, we establish both weak and strong generalization OPE scenarios which differs at the difficulty of evaluating the unseen policies.
For the weak generalization scenario ( easy ), we sample training data uniformly from the whole band of the policy set.
For the strong generalization scenario ( hard ), we separate the policy set and use the low-performance policies for the training data, with the rest taken as the unseen policies to evaluate.
For both the settings, the ratio of sampling and separation is set to be 20%, 40%, or 80%.
We report the results of the ratio 20% in Table 2 and leave the results of other ratios in Appendix G. For evaluation protocols, we report the evaluation (testing) error of unseen policies ( T-error ) and the generalization gap ( G-gap ), i.e., the difference between training and testing error.
We denote different policy representations by their underlying policy abstraction (e.g., f π ) correspondingly.
Complete details can be found in Appendix F.
Weak Generalization Scenario in OPE
First, we study the empirical comparison in the weak generalization scenario.
Table 2 reports the results of value generalization for the policy representations learned based on corresponding policy abstractions.
To be specific, the f Θ denotes directly using policy parameters θ as policy representations (i.e., no representation training).
For our proposed policy abstractions f π , f P π , f V π , we learn the representations for them according to Eq.
1 based on the
LPE and MMD estimation (Sec. 4.3).
To further complete the comparison, we include two additional representations f RE and f EL : f RE uses a randomly initialized LPE with no further training while f
EL uses the LPE trained by the end-to-end OPE loss (see Appendix F.2) respectively.
Note that f EL can be viewed as a variant of f V π since it also learns from values but does not optimize the alignment loss.
Besides, we also include the representation ( f CL ) learned by unsupervised contrastive learning based on InfoNCE loss (van den Oord et al., 2018), as proposed in (Tang et al., 2020).
From the Table 2 (Weak Generalization), we can observe that f π , f P π , f V π outperforms f Θ , f RE , and f EL in both IDP-v2 and LLC-v2.
This demonstrates the effectiveness and superiority of our proposed representations in value function approximation and generalization.
f EL is significantly better than the f Θ and f RE , indicating the advantages of LPE structure and training.
We can observe that contrastive policy representation f CL performs poorly.
We postulate that with less training data available at the 20% sampling ratio, the f CL with emphasis on policy instance-level comparison suffers from higher evaluation error and generalization gap.
The superiority of f V π compared to f EL from the Table 2 demonstrates the effectiveness of alignment loss.
This is because although both f V π , f EL learn policy representation from the information of policy value, naive end-to-end training is less effective than alignment optimization which establishes the representation space based on the [Table] policy metrics.
In general, the value generalization results among our abstractions f π , f P π , f V π do not differ much.
This is mainly because in the weak generalization setting, the unseen policies obey the same distribution as the training policies, thus posing less difficulty of value generalization.
Strong Generalization Scenario in OPE
Now we move to the study in the strong generalization scenario and similarly the results are reported in Table 2 (Strong Generalization).
Compared to the weak generalization scenario, the overall T-error and G-gap are significantly higher in the strong generalization scenario.
This is reasonable because there is a larger performance difference between the training and unseen policies.
In other words, the unseen policies belong to out-of-distribution data.
The f V π obtains the lowest evaluation error on the two environments, which indicates the valueirrelevance abstraction with higher task relevance may be best suited for the strong generalization setting.
For the explanation, since the objective of OPE lies at the value function approximation and generalization, we consider that the value-irrelevance principle of f V π is consistent to the objective and thus fits naturally.
With only low-performance policies for the training data at the 20% sampling ratio ( hardest ), the results of other abstractions including our proposed f π and f P π on the task are poor.
The main reason may be that under the strong generalization setting, there is a large data-shift between the training and unseen policies.
f π and f P π fails to learn a policy abstraction with generalization ability in the absence of diversity policies.
Nevertheless, from the Table9, 10, as the sampling ratio increase and training policies become more diverse, the advantage of f π and f P π over other policy abstractions gradually emerge.
Moreover, in the hardest case, f P π is better than f π , which shows that the influence-irrelevance policy abstraction may be a general policy abstraction option.
For the other baselines, f Θ still shows few competition.
For f RE , f EL , f CL , they falls behind f V π while slightly outperforms f P π and f π in Table 2.
Such slight advantages no long holds in the settings of higher sampling ratios (i.e., 40% in Table 9 and 80% in Table 10).
Unlike weak generalization scenario, f CL is not so bad in strong generalization scenario.
The main reason is that encountering hard policy evaluation tasks (Strong Generalization), other methods suffer from performance degradation and are no longer superior to contrastive learning.
In contrast, contrastive learning based on policy instance-level comparison maintains a relatively good result.
Other Experiments In addition to Table 2, we provide more results under different settings of data amount in Appendix G.1, for both the weak and strong generalization scenarios.
Other comprehensive studies (e.g., extrapolation behaviors, visualization) can be found in Appendix G.2,G.3.
C ONCLUSION & L IMITATIONS
In this work, we introduce a unified policy abstraction theory, including three major types of policy abstraction, and corresponding policy metrics derived from the abstraction, as well as the analysis of their properties.
We further propose a policy representation learning approach based on deep metric learning.
We empirically evaluate the efficacy of different policy abstraction in both policy optimization (i.e., TRPO, DGES) and off-policy evaluation (OPE).
For limitations and future work, we only provide the theory on the fineness of policy abstraction, while provide no theory on the optimality, although the optimality ought to depend on the downstream problem considered.
For policy representation learning, the alignment loss and MMD metric are not the only choices; besides, other representation learning principles (Bardes et al., 2021) are potential.