Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction
Abstract
We consider the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where the goal is to estimate the performance of an evaluation policy, π e , using a fixed dataset, D , collected by one or more policies that may be different from π e .
Current OPE algorithms may produce poor OPE estimates under policy distribution shift i.e., when the probability of a particular state-action pair occurring under π e is very different from the probability of that same pair occurring in D
Voloshin et al.
Fu et al.
In this work, we propose to improve the accuracy of OPE estimation by projecting the ground state-space into a lowerdimensional state-space using concepts from the state abstraction literature in RL.
Specifically, we consider marginalized importance sampling (MIS) OPE algorithms which compute distribution correction ratios to produce their OPE estimate.
In the original state-space, these ratios may have high variance which may lead to high variance OPE.
However, we prove that in the lower-dimensional abstract state-space the ratios can have lower variance resulting in lower variance OPE.
We then present a minimax optimization problem that incorporates the state abstraction.
Finally, our empirical evaluation on difficult, high-dimensional state-space OPE tasks shows that the abstract ratios can make MIS OPE estimators achieve lower mean-squared error and more robust to hyperparameter tuning than the ground ratios.
Introduction
This study focuses on the problem of off-policy evaluation (OPE)
Fu et al.
; Voloshin et al.
(2021), where the goal is to evaluate a policy of interest by leveraging offline data generated by possibly different policies.
Solving the OPE problem would enable us to estimate the performance of a potentially risky policy without having to actually deploy it.
The core OPE problem is to produce accurate policy value estimates under policy distribution shift.
This problem is particularly difficult on tasks with high-dimensional state-spaces Voloshin et al.
(2021); Fu et al.
For example, consider the AntUMaze problem illustrated on the left side of Figure 1.
In this task, an ant-like robot with a high-dimensional state representation moves in a U-shaped maze and receives a reward only for reaching a specific 2 D coordinate goal location.
The state-space of this task includes information such as 2 D location, ant limb angles, torso orientation etc., resulting in a 29 -dimensional state-space.
The OPE task is to evaluate the performance of a particular policy’s ability to take the ant to the 2 D goal location using data that may be collected by different policies.
Policy distribution shift is common in this type of high-dimensional task since the chances of different policies inducing similar limb angles, torso orientations, paths traversed etc. are incredibly slim.
Notice, however, that while different policies may induce different body configurations, they may traverse similar 2 D paths since all (successful) policies must move the ant through roughly the same path to reach the goal.
Moreover, the only critical information needed from the state-space to determine the ant’s per-step reward are its 2 D coordinates.
Motivated by this observation, we propose to improve the accuracy of OPE algorithms on high-dimensional state-space tasks by projecting the high-dimensional state-space into a lower-dimensional space.
This idea is illustrated on the right side of Figure 1 where the ant is reduced to a 2 D point-mass.
With this general motivation in mind, in this paper, we leverage concepts from the state abstraction literature Li et al.
(2006) to improve the accuracy of marginalized importance sampling (MIS) OPE algorithms which estimate state-action density correction ratios to compute a policy value estimate
Liu et al.
(2018a);
Xie et al.
Due to the low chances of similarity between states of policies in high-dimensional state-spaces, current MIS algorithms can produce high variance state-action density ratios, resulting in high variance OPE estimates.
However, if we are given a suitable state abstraction function, we can project the high-dimensional ground state-space into a lower-dimensional abstract state-space.
The projection step increases the chances of similarity between these lower-dimensional states, resulting in low variance density ratios and OPE estimates.
Preliminaries
In this section, we discuss background information.
2.1 Notation and Problem Setup
We consider an infinite-horizon Markov decision process (MDP), M = ⟨S , A , R , P, γ, d 0 ⟩ , where S is the state-space,
A is the action-space, R : S×A → ∆([0 , ∞ )) is the reward function, P : S×A → ∆(
S ) is the transition dynamics function, γ ∈ [0 , 1) is the discount factor, and d 0 ∈ ∆( S ) is the initial state distribution.
The agent acting, according to policy π , in the MDP generates a trajectory: s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ... , where s 0 ∼ d 0 , a t ∼ π ( ·| s t ) , r t ∼ R ( s t , a t ) , and s t
+1 ∼ P ( ·| s t , a t ) for t ≥ 0 .
We define r ( s, a ) :
=
E r ∼R ( s,a ) [ r ] and the agent’s discounted state-action occupancy measure under policy π as d π ( s, a ) :
=
lim T →∞ (cid:16)(cid:80) T − {+1 t =
+} γ t d π ( s t , a t ) (cid:17) /
{+(cid:16)(cid:80) T − 1 t =
+}
γ t (cid:17) , where d π ( s t , a t ) is the probability the agent will be in state s and take action a at time-step t under policy π .
Finally, we define the performance of policy π to be its average reward, ρ ( π ) :
=
E ( s,a ) ∼ d π ,r ∼R ( s,a ) [ r ] .
2.2 Off-Policy Evaluation (OPE)
In behavior-agnostic OPE, the goal is to estimate the performance of an evaluation policy π e given only a fixed offline data set of transition tuples, D :=
{ ( s i , a i , s ′ i , r i ) } mTi =1 , where ( s i , a i ) ∼ d D , m is the batch size (# of trajectories), and T is the fixed length of each trajectory, generated by unknown and possibly multiple behavior policies.
The difficulty in OPE is to estimate ρ ( π e ) under d π e given samples only from d D .
We define the average-reward in dataset D to be ¯ r D := E ( s,a ) ∼ d D ,r ∼R ( s,a ) [ r ] .
As in prior OPE work, we assume that if d π e ( s, a ) > 0 then d D ( s, a ) > 0 .
Empirically, we measure the accuracy of an estimate ˆ ρ ( π e ) by generating M datasets and then computing the relative mean-squared error (MSE):
MSE (ˆ ρ ( π e )) :
= 1 M (cid:80) Mi
=1 ( ρ ( π e ) −
ˆ ρ
i ( π e )) 2 ( ρ ( π e ) − ¯ r D i ) 2 ,
where ˆ ρ i ( π e ) is computed using dataset D i and
¯ r D
i is the average reward in D i .
2.2.1 Marginalized Importance Sampling (MIS)
In this work, we focus on MIS methods, which evaluate π e by using the ratio between d π e and d D .
That is, MIS methods evaluate π e by estimating ρ ( π e ) :
=
E ( s,a ) ∼ d D ,r ∼R ( s,a ) [ ζ ( s, a ) r ] , where ζ ( s, a ) :
= d π e ( s, a ) /d D ( s, a ) is the state-action density ratio for state-action pair ( s, a ) and d π ( s, a ) =
d π ( s ) π ( a | s ) .
When the true ζ is known, the empirical estimate of ρ ( π e ) is:
[Equation]
where N is the number of samples.
In practice, however, ζ is unknown and must be estimated.
One set of ζ -estimation algorithms is the DICE family Yang et al.
While there are many variations, the general DICE optimization problem is:
[Equation]
where the solution to the optimization, ζ ∗ ( s, a ) , are the true ratios.
The estimator we present in
Section 3 builds upon the DICE framework.
2.3 State Abstractions
We define a state abstraction function as a mapping ϕ : S → S ϕ , where S is called the ground state-space and
S ϕ is called the abstract state-space.
We consider state abstraction functions that partition the ground state-space into disjoint sets.
We can use ϕ to project the original MDP into a new abstract MDP with the same action-space A and reward and transition dynamics functions defined as: R ϕ ( s ϕ , a )
= (cid:80) s
∈ ϕ − 1 ( s ϕ ) w ( s ) R ( s, a ) and P ϕ ( s ′ ϕ | s ϕ , a ) = (cid:80) s
[Equation]
w
[Equation]
s ) P ( s
[Equation]
| s, a ) , where w : S → [0 , 1] is a ground state weighting function where for each s ϕ , (cid:80) s ∈ ϕ − 1 ( s ϕ ) w ( s )
= 1 Li et al.
Similarly a policy can be transformed into its abstract equivalent as: π ϕ ( a | s ϕ ) = (cid:80) s ∈ ϕ − 1 ( s ϕ ) w ( s ) π ( a | s ) . In this work, we use w π ( s ) =
d
[Equation]
( s ) (cid:80)
[Equation]
d π ( s ′ ) and only consider abstractions that satisfy:
Assumption 1 (Reward distribution equality) .
∀ s 1 , s 2 ∈ S such that s 1 , s 2 ∈ s ϕ , ∀ a, R ( s 1 , a ) =
R ( s 2 , a ) .
Abstract MIS
Marginalized IS methods may suffer from high variance in high-dimensional state-spaces.
To potentially reduce this high variance, we propose to first use ϕ to project D into the abstract statespace to obtain: D ϕ :
= { ( s ϕ , a, r ϕ , s ′ ϕ ) } where s ϕ =
ϕ ( s ) and r ϕ ( s, a ) = r ( s, a ) ∀ s ∈ s ϕ , and then use the following estimator on D ϕ to estimate π ϕe :
[Equation]
where N is the number of samples, d π ϕ ( s ϕ , a )
=
d π ϕ ( s ϕ )
π ϕ ( a | s ϕ ) with d π ϕ ( s ϕ ) =
(cid:80) s
∈ ϕ − 1 ( s ϕ )
d π ( s ) and π ϕ constructed using w π .
In the remainder of this section, we present theoretical results on the statistical properties of the abstract ratios and the OPE estimator given in Equation (3).
We then present a minimax optimization problem based on the DICE framework that incorporates state abstraction.
3.1 Theoretical Results
We now present the statistical properties of the abstract ratios and our estimator assuming it has access to the true abstract state-action ratios.
[-( s 1 ,a 0 ) ≈ 
 and d πe ( s 2 ,a 0 ) d π D ( s 2 ,a 0 ) ≈ 100 can lead-]Due to space constraints, we defer proofs to Appendix
A.4.
We have Theorem 1, in which we prove that projecting S → S ϕ can lower the variance of density ratios:
Theorem 1.
Var
[Equation]
where equality holds only if ϕ is the identity function i.e. ϕ ( s )
=
s, ∀ s ∈ S
and/or if ∀ s 1 , s 2 ∈ S such that ∀ s 1 , s 2 ∈ s ϕ and for a given action a , d
[Equation]
Furthermore, we prove our abstract estimator is unbiased (Theorem 3 in Appendix A.4) and strongly consistent (Theorem 2 and Corollary 1):
Theorem 2.
Our estimator, ˆ ρ ( π ϕe ) , given in Equation 3 is an asymptotically consistent estimator of ρ ( π e ) in terms of MSE: lim N →∞ E
[(ˆ ρ ( π ϕe ) − ρ ( π e )) 2 ] = 0 .
3.2 MIS OPE with Abstract DICE
To test the effectiveness of state abstraction for OPE, we evaluate the DICE framework on D ϕ .
We focus on BestDICE Yang et al.
(2020), and call our algorithm AbstractBestDICE, which solves the following optimization problem:
[Equation]
where ν : S ϕ ×A → R , λ ∈ R , and ζ : S ϕ ×A → R ≥ 0 .
The solution to this optimization, ζ ∗ ( s ϕ , a ) , is then plugged into the estimator given in Equation (3) to get an OPE estimate.
Note that we have not proven whether this optimization recovers the true abstract density ratios.
However, in Sectionwe show that AbstractBestDICE can still lead to accurate OPE in high-dimensional state-spaces.
Empirical Study
We will now show how projecting S → S ϕ can produce data-efficient and stable OPE estimates in practice.
4.1 Empirical Setup
In this section, we describe the algorithms and domains of our empirical study.
Due to space constraints, we defer supporting details to the appendix (A.5 and A.6).
4.1.1 Algorithms
We compare AbstractBestDICE to ground BestDICE.
As also reported by Yang et al.
Fu et al.
(2021), we found in preliminary experiments that BestDICE performed much better than the other DICE variants such as DualDICE Nachum et al.
(2019), GenDICE Zhang et al.
(2020a), etc.
4.1.2 Domains
We focus on high-dimensional state-space tasks, which have been known to be particularly challenging for DICE methods Fu et al.
We specify the fixed ϕ for each environment in Appendix A.6.
• Reacher .
A robotic arm tries to move to a goal location.
Here, s ∈ R 11 , a ∈ R 2 , and s ϕ ∈ R 4 .
• Walker2D .
A bi-pedal robot tries to move as fast as possible.
Here, s ∈ R 18 , a ∈ R 6 , and s ϕ ∈ R 3 .
• Pusher .
A robotic arm tries to push an object to a goal location.
Here, s ∈ R 23 , a ∈ R 7 , and s ϕ ∈ R 6 .
• AntUMaze .
This sparse-reward task requires an ant to move from one end of the U-shaped maze to the other end.
Here, s ∈ R 29 , a ∈ R 8 , and s ϕ ∈ R 2 .
4.2 Empirical Results
In this section
[Figure]
we describe our main empirical results; additional experiments can be found in appendix A.6.
4.2.1 Data-Efficiency
Figure 2 shows the results of our (relative) MSE vs. batch size experiment for the function approximation case. For a given batch size, we train each algorithm for 100 k epochs with different hyperparameters sets, record the (relative) MSE on the last epoch by each hyperparameter set
[Figure]
and plot the lowest MSE achieved by these hyperparameter sets.
We find that AbstractBestDICE is able to achieve lower MSE than BestDICE for a given batch size.
We note that while hyperparameter tuning is difficult in OPE, in this experiment, we aim to evaluate each algorithm assuming each had favorable hyperparameters. [Figure] l [Figure]
t i v e )
[Figure] ( ρ ( π e ) ) [Figure] 4.2.2 Hyperparameter Robustness
Finally [Figure] we study the robustness of these algorithms to hyperparameters tuning.
In practical OPE, hyperparameter tuning with respect to MSE is impractical since the true ρ ( π e ) is unknown Fu et al.
; Paine et al.
Thus, we want OPE algorithms to be as robust as possible to [Figure] hyperparameter tuning.
The main hyperparameters for DICE are the learning rates of ζ and ν , α ζ and α ν .
We focus on small batch sizes, where we would expect high sensitivity.
The results of this study are in Figure 3.
We find that AbstractBestDICE has a less volatile MSE than BestDICE (also see appendix A.6).
Related Work
MIS and Off-Policy Evaluation .
There have been broadly three families of MIS algorithms in the OPE literature to estimate state-action density ratios.
One is the DICE family, which includes: minimax-weight learning Uehara et al.
, DualDice Nachum et al.
(2019), GenDICE Zhang et al.
(2020a), GradientDICE Zhang et al.
(2020b), and BestDICE Yang et al.
The second family of MIS algorithms is the COP-TD algorithm Hallak and Mannor (2017);
Gelada and Bellemare (2019), which learns the state density ratios with an online TD-styled update.
The third family is the variational power method Wen et al.
(2020) algorithm which generalizes the power iteration method to estimate density ratios.
While our focus has been on MIS algorithms, there are many other OPE algorithms such as model-based methods Zhang et al.
(2021b)
; Hanna et al.
; Liu et al.
(2018b)
, fitted-Q evaluation Le et al.
and IS Precup et al.
; Thomas (2015); Hanna et al.
State Abstraction .
The literature on state abstraction is extensive Singh et al.
; Dietterich (1999); Ferns et al. (2011); Li et al. (2006); Abel (2020).
However, much of this work has been exclusively focused on building a theory of abstraction and learning optimal policies.
To the best of our knowledge, no work has leveraged state abstraction to improve the accuracy of OPE algorithms.
Summary and Future Work
In this work, we showed that we can improve the accuracy of OPE estimates by projecting the original ground state-space into a lower-dimensional abstract state-space using state abstraction and performing OPE in the resulting abstract Markov decision process.
We showed that AbstractBestDICE obtained more accurate estimates with added hyperparameter robustness on difficult, high-dimensional state-space tasks.
As for future work it would be interesting to leverage existing ideas Gelada et al.
; Zhang et al.
(2021a) to learn ϕ instead of using a fixed ϕ .
Another interesting direction would be to apply abstraction to other OPE algorithms.
While this work focused exclusively on MIS algorithms, a promising direction will be to apply abstraction techniques to model-based, trajectory IS, and value-function based OPE.
Acknowledgements
Support for this research was provided by the Office of the Vice Chancellor for Research and
Graduate Education at the University of Wisconsin — Madison with funding from the Wisconsin
Alumni Research Foundation.
The authors thank the anonymous reviewers, Nicholas Corrado, Ishan
Durugkar and Subhojyoti Mukherjee for their helpful comments in improving this work.
A.2 Preliminaries
This section provides the supporting lemmas and definitions that we leverage to prove our lemmas and theorems.
Definition 1 (Almost Sure Convergence) .
A sequence of random variables, ( X n ) ∞ n =1 , almost surely converges to the random variable, X if
[Equation]
We write X n a.s.
→ X to denote that the sequence ( X n ) ∞ n =1 converges almost surely to X .
Definition 2 ((Strongly) Consistent Estimator) .
Let θ be a real number and (ˆ θ n ) ∞ n =1 be an infinite sequence of random variables.
We call ˆ θ n a (strongly) consistent estimator of θ if and only if ˆ θ n a.s.
→ θ .
Lemma 1.
If ( X i ) ∞ i =1 is a sequence of uniformly bounded real-valued random variables, then
X n a.s.
→ X if and only if lim N →∞ E
[( X n − X ) 2 ] = 0 .
Proof
See Lemma 3 in Thomas and Brunskill (2016).
A.3 Assumptions and Definitions
In the main paper, we provided the major assumptions required for our theoretical and empirical work relevant to abstraction and OPE.
Here we provide supporting assumptions typically used in the OPE literature used for the theoretical analysis.
Assumption 2 (Coverage) .
For all ( s, a ) ∈ S × A , if π e ( a | s ) > 0 then π b ( a | s ) > 0 .
Assumption 3 (Non-negative reward) .
We assume that the reward function is bounded between [0 , ∞ ) .
Definition 3 (Ground state normalized weightings) .
For a given policy π , each ground state s ∈ s ϕ , has a state aggregation weight, w π ( s ) =
d π ( s ) (cid:80) s ′∈ ϕ − 1( sϕ ) d
π ( s ′ ) , where d π ( s ) is the discounted stateoccupancy measure of π .
A.4 Proofs
In the proofs below [Equation] we denote the collection of behavior policies that generated D with π D .
That is, π D is the conditional probability of an action occurring in a given state in the data.
Similarly, we also have π ϕ D .
These minor changes give us d D = d π D and d D ϕ =
d π ϕ D .
Lemma 2.
For an arbitrary function, f , E s
[Equation]
Proof
[Equation]
where (a) is due to Definition 3 and we can replace s ϕ with ϕ ( s ) when we know s ∈ s ϕ .
Theorem 1.
Var
[Equation]
Proof
Before comparing the variances
[Equation]
we note that due to Assumption 2 and Lemma 2:
[Equation]
Denote, V g := Var [Equation] d
[Equation]
π e
[Equation]
d
[Equation]
and V ϕ :=
Var
[Equation]
Now consider the difference between the two variances
[Equation]
We can analyze this difference by looking at one abstract state and one action and all the states that belong to it.
That is, for a fixed abstract state, s ϕ , and fixed action, a , we have:
[Equation]
where (a) is due to Definition 3.
If we can show that D ′ ≥ 0 for all possible sizes of | s ϕ | , we will the have the original difference, D , is a sum of only non-negative terms, thus proving Theorem 1.
We will prove D ′ ≥ 0 by inductive proof on the size of | s ϕ | from 1 to some n ≤ |S| .
Let our statement to prove, P ( n ) be that D ′ ≥ 0 where n = | s ϕ | .
This is trivially true for P (1) where the ground state equals the abstract state.
Now consider the inductive hypothesis, P ( n ) is true for n ≥ 1 .
Now with the inductive step, we must show that P ( n + 1) is true given P ( n ) is true.
Starting with the inductive hypothesis:
[Equation]
We define S :
= (cid:16)(cid:80) s ∈
[Equation]
(cid:17) , C := ( d π ϕ e ( s ϕ )
[Equation]
ϕe ( a | s ϕ ) , and C ′ : = d π ϕ D ( s ϕ ) π ϕ D ( a | s ϕ ) . After making the substitutions, we have:
[Equation]
We have the above result holding true for when the | s ϕ | = n .
Now consider the inductive step in relation to the inductive hypothesis where a new state, s n +1 is added to the abstract state.
We have the following difference:
[Equation]
a | s n +1 )
For ease in notation, let x = d π e ( s n +1 )
π e ( a | s n +1 ) and y =
d π D ( s n +1 )
π D ( a | s n +1 ) .
The above difference is then:
[Equation]
The above difference, D ′′ , is minimized most when C is as large as possible.
From the inductive hypothesis, we have C ≤ √ SC ′ .
The minimum difference can be written as:
[Equation]
So we have D ′′ ≥ 0 for | s ϕ | = n + 1 , which means D ′ ≥ 0 .
We have showed that P ( n ) is true for all n .
We now have the original difference, D , to be a sum of non-negative terms after performing this same grouping for all abstract states and actions, which results in: Var
[Equation]
Thus, we have:
[Equation]
Proposition 1.
If Assumption 1 holds, the average reward of ground policy π executed in ground
MDP M , ρ ( π ) , is equal to the average reward of abstract policy π ϕ executed in abstract MDP M ϕ constructed with w π , ρ ( π ϕ ) .
That is, ρ ( π ) =
ρ ( π ϕ ) .
Proof
Consider the definition of R ϕπ :
[Equation]
where (a) is due to Definition [Equation] (b) is due to Definition 3 and Assumption
Theorem 3.
If Assumption 1 holds, our estimator, ˆ ρ ( π ϕe ) as defined in Equation 3, is an unbiased estimator of ρ ( π e ) .
Proof
We first consider the expectation of a single sample, X = d πϕe ( s ϕ ) π ϕe ( a
[Equation]
where (c) is due to Definition 3 and Assumption 1, (d) is due to Assumption 2, and (e) is due to
Proposition 1.
We have the bias defined as:
[Equation]
Bias [ˆ
[Equation]
=where (a) is due to linearity of expectation and (b) is due to expectation of a single sample.
Theorem 2.
Our estimator, ˆ ρ ( π ϕe ) , given in Equation 3 is an asymptotically consistent estimator of ρ ( π e ) in terms of MSE: lim N →∞ E
[(ˆ ρ ( π ϕe ) − ρ ( π e )) 2 ] = 0 .
Proof
We have the MSE of ˆ ρ ( π ϕe ) w.r.t ρ ( π e ) defined in terms of the bias and variance as follows:
[Equation]
where (a) is because ˆ ρ is an unbiased estimator as shown in Theorem 3.
Due to Assumptions 2 and 3, (cid:32)(cid:80)
[Equation]
is a bounded value.
Thus, as mT → ∞ , Var [ˆ ρ ( π ϕe )] → 0 .
We then have lim mT →∞ E
[(ˆ ρ ( π ϕe ) − ρ ( π e )) 2 ] = 0 .
Thus, the estimator ˆ ρ ( π ϕe ) is consistent in MSE.
Corollary 1.
If Assumption 1 holds, then our estimator, ˆ ρ ( π ϕe ) as defined in Equation 3 is an asymptotically strongly consistent estimator of ρ ( π e ) .
Proof
Theorem 2 showed that ˆ ρ ( π ϕe ) is consistent in terms of MSE.
Then by applying Lemma 1, we have ˆ ρ ( π ϕe ) to be an asymptotically strongly consistent estimator of ρ ( π e ) .
That is,
[Equation]
A.5 Tabular Experiments and Details with True Ratios
We conduct the following tabular experiment on the MDP pictured in Figure 4.
All trajectories are 100 time-steps long.
[-where β ϕ 
-]
[Figure]
We conduct an experiment on the TwoPath MDP to estimate ρ ( π e ) where we apply the ground estimator given in Equation (1) and our abstract estimator given in Equation (3), assuming both have access to their respective true ratios .
The results of this experiment are illustrated in Figure 5(a).
We can observe that the abstract estimator with the true abstract ratios produces substantially more data-efficient and lower variance OPE estimates for different batch sizes compared to the ground equivalent.
• Reacher Brockman et al.
A robotic arm tries to move to a goal location.
Here, s ∈ R 11 and a ∈ R 2 .
Since the reward function is the Euclidean distance between the arm and goal, ϕ extracts only the arm-to-goal vector, and angular velocities from the ground state, resulting in s ϕ ∈ R 4 .
All trajectories are 200 time-steps long. • Walker2D Brockman et al. A bi-pedal robot tries to move as fast as possible. Here, s ∈ R 18 and a ∈ R 6 .
We use the Euclidean distance from the start location as the reward function and use a ϕ that extracts x and z coordinates and top angle of the walker’s body, resulting in s ϕ ∈ R 3 . All trajectories are 500 time-steps long. • Pusher Brockman et al. A robotic arm tries to push an object to a goal location.
Here, s ∈ R 23 and a ∈ R 7 . Since the reward function is the Euclidean distance between object and arm and object and goal, ϕ extracts only object-to-arm and object-to-goal vectors, resulting in s ϕ ∈ R 6 . All trajectories are 300 time-steps long. • AntUMaze Fu et al.
This sparse-reward task requires an ant to move from one end of the U-shaped maze to the other end.
Here, s ∈ R 29 and a ∈ R 8 .
We use the “play" version where the goal location is fixed. Since the reward function is +1 only if the 2 D location of the ant is at a certain Euclidean distance from the 2 D goal location, ϕ extracts only the 2 D coordinates of the ant, resulting in s ϕ ∈ R 2 . All trajectories are 500 time-steps long. A.6.2 Oracle ρ ( π e ) Values
On each domain, we executed π e for 200 episodes and averaged the results.
A.6.3 Policies
For each of the domains, we used the following policies:
• Reacher: We trained a policy using PPO Schulman et al.
π e was the trained policy after 100 k time-steps with a standard deviation of 0 .
1 on the action dimensions while π b used 0 .
5 as the standard deviation.
• Walker2D: We trained a policy using PPO Schulman et al.
π e was the trained policy after 100 k time-steps with a standard deviation of 0 .
1 on the action dimensions while π b used 0 .
5 as the standard deviation.
• Pusher: We trained a policy using PPO Schulman et al.
π e was the trained policy after 100 k time-steps with a standard deviation of 0 .
1 on the action dimensions while π b used 0 .
5 as the standard deviation.
• AntUMaze:
We used the policies made available Fu et al.
π e was the final 10th snapshot saved and π b was the 5th snapshot.
Each also had 0 .
1 standard deviation on the action dimensions.
A.6.4 Trajectory Length
For each of the domains, the trajectory length is: 200 for Reacher, 500 for Walker2D, 300 for Pusher, and 500 for AntUMaze.
A.6.5 Hyperparameters
For BestDICE and AbstractBestDICE, we fixed the following hyperparameters:
• γ = 0 .
995 in all experiments.
• Neural net architecture: All neural networks are 2 layers with 64 hidden units using tanh activation.
• Unit mean constraint learning rate Zhang et al.
(2020b)
; Yang et al.
(2020): λ = 1 e − 3 .
• Optimizer: Adam optimizer with default parameters in Pytorch.
• Positivity constraint: squaring function on the last layer of the neural network.
We conducted a search for the learning rate of ν ( α ν ) and learning rate of ζ ( α ζ ), .
The learning rate search for ( α ν , α ζ ) was over { (5 e − 5 , 5 e − 5 ) , (1 e − 4 , 1 e − 4 ) , (3 e − 4 , 3 e − 4 ) , (7 e − 4 , 7 e − 4 ) , (1 e − 3 , 1 e − 3 ) } .
The optimal hyperparameters ( α ν = α ζ ) for each environment and batch size were: {+Reacher 5 e −1 e −1 e −
 e −
+}
e −7 e −1 e −7 e − Walker2D 1 e −5 e −5 e −5 e −5 e −5 e −5 e −5 e − Pusher 5 e −5 e −5 e −5 e −1 e −5 e −5 e −5 e − AntUMaze 3 e −5 e −5 e −5 e −5 e −5 e −5 e −5 e −
Reacher 5 e −1 e −5 e −1 e −5 e −1 e −3 e −1 e − Walker2D 5 e −3 e −7 e −7 e −7 e −1 e −3 e −1 e − Pusher 5 e −1 e −1 e −5 e −1 e −5 e −1 e −5 e − AntUMaze 1 e −3 e −5 e −5 e −5 e −5 e −5 e −5 e −
A.6.6 Empirical Estimator
In practice we use a weighted importance sampling Yang et al.
(2020) approach for the function approximation cases to estimate ρ ( π e ) (same for BestDICE):
[Equation]
A.6.7 Misc Abstraction Details
• For Walker2D, we modified the default reward function from incremental distance covered at each time-step to distance from start location at each time-step to ensure Assumption 1 is satisfied.
• For AntUMaze, the reward function is originally r ( s ′ )
i.e. it is based on the next state that the ant moves to.
To ensure Assumption 1 is satisfied, we changed this reward function to be of the current state, r ( s ) .
A.6.8 Additional Results
Additional Hyperparameter Robustness Results In general, we can see AbstractBestDICE can be much more robust than BestDICE to hyperparameter tuning.
Training Stability In Figure 7 we show that ϕ can improve training stability.
Abstract Quality and Data-Efficiency .
We find that not all abstractions that satisfy Assumptionlead to better performance.
For example, the following are valid abstractions on the Reacher task: 1) the Euclidean distance between the arm and goal, s ϕ ∈ R and the 3D vector between the arm and goal, s ϕ ∈ R 3 (Figure 8).
However, in practice we found that these were unreliable.
One possible reason for this unreliability is that these abstractions are incredibly extreme and the algorithm may be unable to differentiate between abstract state, resulting in outputting similar ζ ϕ ( s ϕ , a ) ∀ s ϕ .
A.7 Hardware For Experiments
• Distributed cluster on HTCondor framework • Intel(R)
Xeon(R) CPU E5-2470 0 @ 2.30GHz • RAM: 5GB • Disk space: 4GB
[Figure]
[-Assumption 2 (transition dynamics) violated 0 .0 .0 .
-]
t i v
[Figure] [Figure]
( π
[Figure]
[Figure]
(a) Reacher (Batch size: 5)
[Figure]
[Figure] [Figure]
(Batch
[Figure]
(d) Walker2D (Batch size: 5)
[Figure]
(e)
[Figure] [Figure]
[Figure] [Figure]