B LACK - BOX O FF - POLICY E STIMATION FOR I NFINITE -H ORIZON R EINFORCEMENT L EARNING
A BSTRACT
Off-policy estimation for long-horizon problems is important in many real-life applications such as healthcare and robotics, where high-ﬁdelity simulators may not be available and on-policy evaluation is expensive or impossible.
Recently,
Liu et al.
(2018) proposed an approach that avoids the curse of horizon suffered by typical importance-sampling-based methods.
While showing promising results, this approach is limited in practice as it requires data be drawn from the stationary distribution of a known behavior policy.
In this work, we propose a novel approach that eliminates such limitations.
In particular, we formulate the problem as solving for the ﬁxed point of a certain operator. Using tools from Reproducing Kernel Hilbert Spaces (RKHSs), we develop a new estimator that computes importance ratios of stationary distributions, without knowledge of how the off-policy data are collected.
We analyze its asymptotic consistency and ﬁnite-sample generalization.
Experiments on benchmarks verify the effectiveness of our approach.
I NTRODUCTION
As reinforcement learning (RL) is increasingly applied to crucial real-life problems like robotics, recommendation and conversation systems, off-policy estimation becomes even more critical.
The task here is to estimate the average long-term reward of a target policy, given historical data collected by (possibly unknown) behavior policies.
Since the reward and next state depend on what action the policy chooses, simply averaging rewards in off-policy data does not estimate the target policy’s long-term reward.
Instead, proper correction must be made to remove the bias in data distribution.
One approach is to build a simulator that mimics the reward and next-state transitions in the real world, and then evaluate the target policy against the simulator (e.g., Fonteneau et al., 2013; Ie et al., 2019).
While the idea is natural, building a high-ﬁdelity simulator could be extensively challenging in numerous domains, such as those that involve human interactions.
Another approach is to use propensity scores as importance weights, so that we could use the weighted average of rewards in off-policy data as a suitable estimate of the average reward of the target policy.
The latter approach is more robust, as it does not require modeling assumptions about the real world’s dynamics.
It often ﬁnds more success in short-horizon problems like contextual bandits, but its variance often grows exponentially in the horizon, a phenomenon known as “the curse of horizon” (Liu et al., 2018).
To address this challenge, Liu et al.
(2018) proposed to solve an optimization problem of a minimax nature, whose solution directly estimates the desired propensity score of states under the stationary distribution , avoiding an explicit dependence on horizon.
While their method is shown to give more accurate predictions than previous algorithms, it is limited in several important ways:
• The method requires that data be collected by a known behavior policy.
In practice, however, such data are often collected over an extended period of time by multiple, unknown behavior policies.
For example, observational healthcare data typically contain patient records, whose treatments were provided by different doctors in multiple hospitals, each following potentially different procedures that are not always possible to specify explicitly.
• The method requires that the off-policy data reach the stationary distribution of the behavior policy.
In reality, it may take a very long time for a trajectory to reach the stationary distribution, which may be impractical due to various reasons like costs and missing data.
In this paper, we introduce a novel approach for the off-policy estimation problem that overcome these drawbacks.
The main contributions of our work are three-fold:
• We formulate the off-policy estimation problem into one of solving for the ﬁxed point of an operator.
Different from the related, and similar, Bellman operator that goes forward in time, this operator is backward in time.
• We develop a new algorithm, which does not have the aforementioned limitations of Liu et al.
(2018), and analyze its generalization bounds.
Speciﬁcally, the algorithm does not require that the off-policy data come from the stationary distribution, or that the behavior policy be known.
• We empirically demonstrate the effectiveness of our method on several classic control benchmarks.
In particular, we show that, unlike Liu et al.
(2018), our method is effective even if the off-policy data has not reached the stationary distribution.
In the next section, we give a brief overview of recent and related works.
We then move to describing the problem setting that we have used in the course of the paper and our off-policy estimation approach.
Finally, we present several experimental results to show the effectiveness of our method.
Notation.
In the following, we use ∆( X ) to denote the set of distributions over a set X .
The (cid:96) 2 norm of vector x is (cid:107) x (cid:107) .
Given a real-valued function f deﬁned on some set X , let (cid:107) f (cid:107) 2 :=
(cid:113)(cid:82) X f ( x ) 2 dx .
Finally, we denote by [ n ] the set { 1 , 2 , . . . , n } , and 1 { A } the indicator function.
R ELATED WORKS
Our work focuses on estimating a scalar (average long-term reward) that summarizes the quality of a policy and has extensive applications in practice.
This is different from value function or policy learning from off-policy data (e.g., Precup et al., 2001; Maei et al., 2010; Sutton et al., 2016;
Munos et al., 2016; Metelli et al., 2018), where the major goal is to ensure stability and convergence.
Yet, these two problems share numerous core techniques, such as importance reweighting and doubly robustness.
Off-policy estimation and evaluation can also be used as a component for policy optimization (e.g., Jiang & Li, 2016; Gelada & Bellemare, 2019; Liu et al., 2019; Zhang et al., 2019).
Importance reweighting, or inverse propensity scoring, has been used for off-policy RL (e.g., Precup et al., 2001;
Murphy et al., 2001; Li et al., 2015; Munos et al., 2016; Hanna et al., 2017; Xie et al., 2019).
Its accuracy can be improved by various techniques (Jiang & Li, 2016; Thomas & Brunskill, 2016;
Guo et al., 2017; Farajtabar et al., 2018).
However, these methods typically have a variance that grows exponentially with the horizon, limiting their application to mostly short-horizon problems like contextual bandits (Dudík et al., 2011; Bottou et al., 2013).
There have been recent efforts to avoid the exponential blow-up of variance in basic inverse propensity scoring.
A few authors explored the alternative to estimate the propensity score of a state’s stationary distribution (Liu et al., 2018; Gelada & Bellemare, 2019), when behavior policies are known.
Later,
Nachum et al.
(2019) extended this idea to situations with unknown behavior policies.
However, their approach only works for the discounted reward criterion.
In contrast, our work considers the more general and challenging undiscounted criterion.
In the next section, we brieﬂy mention the setting under which we study this problem and then present our black-box off-policy estimator.
Our black-box estimator is inspired by previous work for black-box importance sampling (Liu & Lee, 2017).
Interestingly, the authors show that it is beneﬁcial to estimate propensity scores from data without using knowledge of the behavior distribution (called proposal distribution in that paper), even if it is available; see also Henmi et al.
(2007) for related arguments.
Similar beneﬁts may exist for our black-box off-policy estimator developed here, although a systematic study is outside the scope of this paper.
P ROBLEM S ETTING
Consider a Markov decision process (MDP) (Puterman, 1994)
M = (cid:104)S , A , P, R, p 0 , γ (cid:105) , where S and
A are the state and action spaces, P is the transition probability function, R is the reward function, p 0 ∈ ∆( S ) is the initial state distribution, and γ ∈ [0 , 1] is the discount factor.
A policy π maps states to a distribution over actions: π : S (cid:55)→ ∆( A ) , and π ( a | s ) is the probability of choosing action a in state s by policy π .
With a ﬁxed π , a trajectory τ = ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . . )
is generated as follows:
[Equation]
Given a target policy π , we consider two reward criteria, undiscounted ( γ = 1) and discounted ( γ < 1 ), where E π [ · ] indicates the trajectory τ is controlled by policy π :
[Equation]
In the above, d π is the stationary distribution over S × A , which exists and is unique under certain assumptions (Levin & Peres, 2017).
The γ < 1 case can be reduced to the undiscounted case of γ = 1 , but not vice versa.
Indeed, one can show that the discounted reward in equation 2 can be interpreted as the stationary distribution of an induced Markov process, whose transition function is a mixture of P and the initial-state distribution p 0 .
We refer interested readers to Appendix A for more details.
Accordingly, in the following and without the loss of generality, we will merely focus on the more general undiscounted criterion in equation 1, and suppress the unnecessary dependency on p 0 and γ .
In the off-policy estimation problem, we are interested in estimating ρ π for a given target policy π .
However, instead of having access to on-policy trajectories generated by π , we have a set of n transitions collected by some unknown (i.e., “ black-box ” or behavior-agnostic (Nachum et al., 2019)) behavior mechanism π BEH :
[Equation]
Therefore, the goal of off-policy estimation is to estimate ρ π based on D , for a given target policy π .
The setting we described above is quite general, covering a number of situations.
For example, π BEH might be a single policy and D might consist of one or multiple trajectories collected by π BEH .
In this special case, s (cid:48)
i =
s i +1 for 1 <
i < n ; this is the off-policy RL scenario widely studied (e.g.,
Precup et al., 2001; Sutton et al., 2016; Munos et al., 2016; Liu et al., 2018; Gelada & Bellemare, 2019).
Furthermore, if π BEH = π , we recover the on-policy setting.
On the other hand, π BEH and
D can consist of multiple policies and their corresponding trajectories.
In this situation, unlike the single policy case s (cid:48) i and s i +1 might originate from two distinct policies.
In general, one can consider π BEH as a distribution over S × A where ( s i , a i ) in D are sampled from.
Having introduced the general setting of the problem, we will describe our estimation approach in the next section.
B LACK - BOX ESTIMATION
Our estimator is based on the following operator deﬁned on functions over S × A .
For discrete state-action spaces, given any d ∈ R S×A ,
[Equation]
While we will develop the rest of the paper using the discrete version above for simplicity, the continuous version can be similarly obtained without affecting the estimator and results:
[Equation]
where P is now interpreted as the transition kernel.
We should note that B π is indeed different from the Bellman operator (Puterman, 1994); although they have some similarities.
In particular, given some state-action pair ( s, a ) , the Bellman operator is deﬁned using next state s (cid:48) of ( s, a ) , while B π is deﬁned using previous state-actions ( ξ, α ) that transition to s .
It is in this sense that B π is backward (in time).
Furthermore, as we will show later, d has the interpretation of a distribution over S × A .
Therefore, B π describes how visitation ﬂows from ( ξ, α ) to ( s, a ) and hence, we call it the backward ﬂow operator.
Note that similar forms of B π have appeared in the literature, usually used to encode constraints in a dual linear program for an
MDP (e.g., Wang et al., 2007; Wang, 2017; Dai et al., 2018).
However, the application of B π for the off-policy estimation problem as considered here appears new to the best of our knowledge.
An important property of B π is that, under certain assumptions, the stationary distribution d π is the unique ﬁxed point that lies in ∆( S × A ) (Levin & Peres, 2017):
[Equation]
This property is the key element we use to derive our estimator as we describe in the following.
4. B LACK - BOX ESTIMATOR
In most cases, off-policy estimation involves a weighted average of observed rewards r i in D .
We therefore aim to directly estimate these (non-negative) weights which we denote by w = { w i } ∈ ∆([ n ]) ; that is, w i ≥ 0 for i ∈ [ n ] and (cid:80) ni
=
w i = 1 .
Note that the normalization of w may be ensured by techniques such as self-normalized importance sampling (Liu, 2001).
Once such a w is obtained, the estimated reward is given by:
[Equation]
Effectively, any w ∈ ∆([ n ]) deﬁnes an empirical distribution which we denote by d w over S × A :
[Equation]
Equation 6 is equivalent to ˆ ρ π = E ( s,a ) ∼ d w [ r ] .
Comparing it to equation 1, we naturally want to optimize w so that d w is close to d π .
Therefore, inspired by the ﬁxed-point property of d π in equation 5, the problem naturally becomes one of minimizing the discrepancy between d w and B π d w .
In practice, w is often represented in a parametric way:
[Equation]
where W ( . ) is a parametric model, such as neural networks, with parameters ω ∈ Ω .
We have now reached the following optimization problem:
[Equation]
where D ( · (cid:107) · ) is some discrepancy function between distributions.
In practice, B is unknown, and must be approximated by samples in the dataset D :
[Equation]
Clearly, ˆ B π d w is a valid distribution over S ×
A induced by w and D , and the black-box estimator solves for w by minimizing D ( d w (cid:107) ˆ B π d w ) .
4. B LACK - BOX ESTIMATOR WITH MMD
There are different choices for D ( · (cid:107) · ) in equation 9, and multiple approaches to solve it (e.g.,
Nguyen et al., 2010; Dai et al., 2017).
Here, we describe one such algorithm based on Maximum
Mean Discrepancy (MMD) (Muandet et al., 2017a).
For simplicity, the discussion in this subsection assumes S × A is ﬁnite, but the extension to continuous S
×
A is immediate.
Let k ( · , · ) be a positive deﬁnite kernel function deﬁned on ( S ×A ) 2 .
Given two real-valued functions, f and g , deﬁned on S ×
A we deﬁne the following bilinear functional
[Equation]
Clearly, we have k [ f ; f ] ≥ 0 for any f due to the positive deﬁniteness of k .
In addition, k is called strictly integrally positive deﬁnite if k [ f ; f ] = 0 implies (cid:107) f (cid:107) 2 = 0 .
Let H be the reproducing kernel Hilbert space (RKHS) associated with the kernel function k .
This is the unique Hilbert space that includes functions that can be expressed as a sum of countably many terms: f ( · ) =
(cid:80) i u i k
(( s i , a i ) , · ) , where { u i } ⊂ R , and { ( s i , a i ) } ⊆
S × A .
The space H is equipped with an inner product deﬁned as follow: given f, g ∈ H such that f = (cid:80) i u i k
(( s i , a i ) , · ) and g = (cid:80) i v i k
(( s i , a i ) , · ) , the inner product is (cid:104) f, g (cid:105) H :
= (cid:80) i,j u i v j k
( s i , a i ) , ( s j , a j )) , which induces the norm deﬁned by (cid:107) f (cid:107) H :=
(cid:113)(cid:80) i,j u i u j k
(( s i , a i ) , ( s j , a j )) .
Given H , the maximum mean discrepancy between two distributions, µ 1 and µ 2 , is deﬁned by
[Equation]
Here, f may be considered as a discriminator, playing a similar role as the discriminator network in generative adversarial networks (Goodfellow et al., 2014), to measure the difference between µ 1 and µ 2 .
A useful property of MMD is that it admits a closed-form expression (Gretton et al., 2012):
[Equation]
where k [ · ; · ] is deﬁned in equation 10, and we used the bilinear property k [ · ; · ] .
Interested readers are referred to surveys (e.g. Berlinet & Thomas-Agnan, 2011; Muandet et al., 2017b) for more background on RKHS and MMD.
Applying MMD to our objective, we have
[Equation]
In the above, both d w and ˆ B π d w are simply probability mass functions on a ﬁnite subset of S × A , consisting of state-actions encountered in D .
It follows immediately from equation 10 that
[Equation]
Deﬁning K i,j := K (0) i,j − 2 K (1) i,j + K (2) i,j , we can express the objective as a function of ω (since { w i } depends on ω ; see equation 8): (cid:96)
[Equation]
[Equation]
Remark 4.1.
Mini-batch training is an effective approach to solve large-scale problems.
However, the objective (cid:96) ( ω ) is not in a form that is ready for mini-batch training, as w i requires normalization (equation 8) that involves all data in D .
Instead, we may equivalently minimize L ( ω ) :
= log (cid:96) ( ω ) , which can be turned into a form that allow mini-batch training, using a trick that is also useful in other machine learning contexts (e.g., Jean et al., 2015).
See Appendix D for more details.
Algorithm 1 in Appendix E summarizes our estimator.
We next present theoretical analysis of our approach
4. T HEORETICAL A NALYSIS
Consistency.
The following theorem shows that the exact minimizer of equation 9 coincides with the ﬁxed point of B π , and the objective function measures the norm of the estimation error in an induced RKHS.
To simplify exposition, we assume x =
( s, a ) and x (cid:48)
=
( s (cid:48) , a (cid:48) )
a successive actionstate pair following x : x (cid:48) ∼ d π ( · | x ) , where d π ( x (cid:48) | x ) is the transition probability from x to x (cid:48) , that is, d π ( x (cid:48)
| x ) = P ( s (cid:48) | s, a ) π ( a (cid:48) | s (cid:48) ) .
Similarly, we denote by (¯ x, ¯ x (cid:48) )
=
((¯ s, ¯ a ) , (¯ s (cid:48) , ¯ a (cid:48) ))
an independent copy of ( x, x (cid:48) ) .
Theorem 4.1.
Suppose k is strictly integrally positive deﬁnite, and d π is the unique ﬁxed point of B π in equation 5.
Then, for any d ∈ ∆( S × A ) ,
[Equation]
Furthermore, D k ( d || B π d ) equals an MMD between d and d π , with a transformed kernel:
[Equation]
where ˜ k ( x, ¯ x ) is a positive deﬁnite kernel, deﬁned by
[Equation]
where the expectation is with respect to x (cid:48) ∼ d π ( · | x ) and ¯ x (cid:48) ∼ d π ( · | ¯ x ) , with x (cid:48) and ¯ x (cid:48) drawn independently.
Generalization.
We next give a sample complexity analysis.
In practice, the estimated weight ˆ w is based on minimizing the empirical loss D k ( d w || ˆ B π d w ) , where B π is replaced by the empirical approximation ˆ B π .
The following theorem compares the empirical weights ˆ w with the oracle weight w ∗ obtained by minimizing the expected loss D k ( d w || B π d w ) , with the exact transition operator B π .
Theorem 4.2.
Assume the weight function is decided by w i
=
W ( s i , a i ; ω ) /n .
Denote by W = { ˜ W ( · ; ω ): ω ∈ Ω } the model class of W ( · ; ω ) .
Assume ˆ w is the minimizer of the empirical loss
D k ( d w || ˆ B π d w ) and w ∗ the minimizer of expected loss D k ( d w || B π d w ) .
Assume { x i } ni
=1 are i.i.d.
samples.
Then, with probability 1 − δ we have
[Equation]
where R n ( W ) denotes the expected Rademacher complexity of W with data size n , and r max := max( (cid:107)W(cid:107) ∞ , sup x (cid:112) k
( x, x )) , with (cid:107)W(cid:107) ∞ :=
sup {(cid:107) W (cid:107) ∞ : W ∈ W} .
This suggests a generalization error of O (1 / √ n ) if R n ( W )
=
O (1 / √ n ) , which is typical for parametric families of functions.
E XPERIMENTS
In this section, we present experiments to compare the performance of our proposed method with other baselines on the off-policy evaluation problem.
In general and for each experiment, we use a behavior policy π BEH to generate trajectories of length T BEH .
We then use these generated samples from a behavior policy to estimate the expected reward of a given target policy π .
To compare our approach with other baselines, we use the root mean squared error (RMSE) with respect to the average long-term reward of the target policy π .
The latter is estimated using a trajectory of length
T TAR (cid:29) 1 .
In particular, we compare our proposed black-box approach with the following baselines:
• naive averaging baseline in which we simply estimate the expected reward of a target policy by averaging the rewards over the trajectories generated by
the
behavior policy.
• model-based baseline where we use the kernel regression technique with a standard Gaussian RBF kernel.
We set the bandwidth of the kernel to the median (or 25 th or 75 th percentiles) of the pairwise euclidean distances between the observed data points.
• inverse propensity score (IPS) baseline introduced by Liu et al.
We will ﬁrst use a simple MDP from Thomas & Brunskill (2016) to highlight the IPS drawback we previously mentioned in Section 1.
We then move to classical control benchmarks.
[Figure]
5. T OY E XAMPLE
The ModelWin domain ﬁrst introduced in Thomas & Brunskill (2016) is a fully observable MDP with three states and two actions as denoted in Figure 1(a).
The agent always begins in s 1 and should choose between two actions a 1 and a 2 .
If the agent chooses a 1 , then with probability of p and 1 − p it makes a transition to s 2 and s 3 and receives a reward of r = 1 and r = − 1 , respectively.
On the other hand, if the agent chooses a 2 , then with probability of p and 1 − p it makes a transition to s 3 with the reward of r = − 1 and s 2 with the reward of r = 1 , respectively.
Once the agent is in either s 2 or s 3 , it goes back to the s 1 in the next step without any reward.
In our experiments, we set p = 0 .
We deﬁne the behavior and target policies as the following.
In the target policy, once the agent is in s 1 , it chooses a 1 and a 2 with the probability of 0.9 and 0.1, respectively.
On the other hand and for the behavior policy, once the agent is in s 1 , it chooses a 1 and a 2 with the probability of 0.
and 0.3, respectively.
We calculate the average on-policy reward from samples based on running a trajectory of length T TAR = 50 , 000 collected by the target policy.
We estimate this on-policy reward using trajectories of length T BEH ∈ { 4 , 8 , 16 , 32 , 64 , 128 } collected by the behavior policy.
In each case, we set the number of trajectories such that the total number of transitions (i.e., T BEH times the number of trajectories) is kept constant.
For example, for T BEH = 4 and T BEH =
we use 50, and 25,000 trajectories, respectively.
Since the problem has ﬁnitely many state-actions, we use the tabular method and hence, equation 11 turns into a quadratic programming.
We then report the result of each setting based on 10 Monte-Carlo samples.
As we can see in Figure 1(b), the naive averaging method performs poorly consistently and independent of the length of trajectories collected by the behavior policies.
On the other hand, IPS performs poorly when the collected trajectories have short-horizon and gets better as the horizon length of trajectories get larger.
This is expected for IPS — as mentioned in Section 1, it requires data be drawn from the stationary distribution.
In contrast, as shown in Figure 1(b), our black-box approach performance is independent of the horizon length, and substantially better.
5. C LASSIC C ONTROL
We now focus on four classic control problems.
We begin by brieﬂy describing each problem and then compare the performance of our method with other approaches on these problems.
Note that for these problems are episodic, we convert them into inﬁnite-horizon problems by resetting the state to a random start state once the episode terminates.
Pendulum.
In this environment, the goal is to control a pendulum in a vertical position.
State variables are the pole angle θ and velocity ˙ θ .
The action a is the torque in the set {− 2 , − 1 , 0 , 1 , 2 } applied to the base.
We set the reward function to − ( θ 2 + 0 .
1 ˙ θ 2 + 0 .
001 a 2 ) .
Mountain Car.
For this problem, the goal is to drive up the car to top of a hill.
Mountain Car has a state space of R 2 (the position and speed of the car) and three possible actions (negative, positive, or zero acceleration).
We set the reward to +100 when the car reaches the goal and -1 otherwise.
Cartpole.
The goal here is to prevent an attached pole to a cart from falling by changing the cart’s velocity.
Cartpole has a state space of R 4 (cart position, velocity, pole angle and velocity) and two possible actions (moving left or right).
Reward is -100 when the pole falls and +1 otherwise.
Acrobot.
In this problem, our goal is to swing a 2-link pendulum above the base.
Acrobot has a state space of R 6 ( sin( . ) and cos( . ) of both angles and angular velocities) and three possible actions (applying +1, 0 or -1 torque on the joint).
Reward is +100 for reaching the goal and -1 otherwise.
For each environment, we train a near-optimal policy π + using the Neural Fitted Q Iteration algorithm (Riedmiller, 2005).
We then set the behavior and target policies as π BEH =
α 1 π
+ + (1 − α 1 ) π − and π = α 2 π + + (1 − α 2 ) π − , where π − denotes a random policy, and 0 ≤ α 1 , α 2 ≤ 1 are two constant values making the behavior policy distinct from the target policy.
In our experiments, we set α 1 = 0 .and
α 2 = 0 .
In order to calculate the on-policy reward, we use a single trajectory collected by π with T TAR = 50 , 000 .
For off-policy data, we use multiple trajectories collected by π BEH with
T BEH = 200 .
In all the cases, we use a 3-layer (having 30, 20, and 10 hidden neurons) feed-forward neural network with the sigmoid activation function as our parametric model in equation 8.
For each setting, we report results based on 20 Monte-Carlo samples.
Figure 2 shows the log of RMSE w.r.t.
the target policy reward as we change the number of trajectories collected by the behavior policy.
We should note that all methods except the naive averaging method have hyperparameters to be tuned.
For each method, the optimal set of parameters might depend on the number of trajectories (i.e., size of the training data).
However, in order to avoid excessive tuning and to show how much each method is robust to a change in the number of trajectories, we only tune different methods based on 50 trajectories and use the same set of parameters for other settings.
As we can see, the naive averaging performance is almost independent of the number of trajectories.
Our method outperforms other approaches on three environments and it is only the Acrobot in which IPS performs comparably to our black-box approach.
In order to have a robust evaluation against outliers, we have plotted the median and error bars at 25 th and 75 th percentiles in Figure 3.
If we compare the
Figures 2 and 3, we notice that the trend of results is almost the same in both.
In Appendix G, we have studied how changing α 1 of the behavior policy affects the ﬁnal RMSE.
C ONCLUSIONS
In this paper, we presented a novel approach for solving the off-policy estimation problem in the long-horizon setting.
Our method formulates the problem as solving for the ﬁxed point of a “backward ﬂow” operator.
We showed that unlike previous works, our approach does not require the knowledge of the behavior policy or stationary off-policy data.
We presented experimental results to show the effectiveness of our approach compared to previous baselines.
In the future, we plan to use structural domain knowledge to improve the estimator and consider a random time horizon in episodic RL.
A R EDUCTION FROM D ISCOUNTED TO U NDISCOUNTED R EWARD
The same reduction is used in Liu et al.
For completeness, we give the derivation details here, for the case of ﬁnite state/actions.
The derivation can be extended to general state-action spaces, with proper adjustments in notation.
Let τ =
( s 0 , a 0 , r 0 , s 1 , a 1 , . . .
) be a trajectory generated by π , and d t ∈ ∆( S ×A ) be the distribution of ( s t , a t ) .
Clearly,
[Equation]
Using a matrix form
[Equation]
the recursion above can be written equivalently as d t +1 =
P T π d t , where P π is given by
P π ( s, a | ξ, α ) =
P ( s | ξ, α ) π ( a | s ) .
The discounted reward of policy π is
[Equation]
with
[Equation]
Multiplying both sides of above by γP T π , we have
[Equation]
Therefore, d
[Equation]
Accordingly, d π,γ is the ﬁxed point of an induced transition matrix given by P π,λ :=
γP π + (1 − γ ) d 0 1 T .
This completes the reduction from the discounted to the undiscounted case.
B P ROOF OF T HEOREM 4.
Note that
[Equation]
Following the deﬁnition of the strictly integrally positive deﬁnite kernels, we have that D k ( d (cid:107) B π d ) =
implies d − B π d = 0 , which in turn implies d = d π by the uniqueness assumption of d π .
We have thus proved the ﬁrst claim.
For the second claim, deﬁne δ w = d − d π .
Since d π − B π d π = 0 , we have
[Equation]
=
[Equation]
δ
[Equation]
π
[Equation]
w )] .
Recalling that B π d ( x ) = (cid:80) x 0 P π ( x | x 0 ) d ( x 0 ) , we have
D k ( d (cid:107) B π d ) =
k [( δ w − B π δ w ); ( δ w − B π δ w )]
= (cid:88) x, ¯ x k ( x, ¯ x )( δ w ( x ) − B π δ w ( x ))
( δ w (¯ x ) − B π δ w (¯ x ))
= (cid:88)
x, ¯ x k ( x, ¯ x ) (cid:32) δ w ( x ) − (cid:88)
[Equation]
Deﬁne the adjoint operator of B π ,
[Equation]
Denote by P xπ the operator applied on k ( x, ¯ x ) in terms of variable x , that is, P xπ k ( x, ¯ x ) := (cid:80) x (cid:48)
P π ( x (cid:48) | x ) k
( x (cid:48) , ¯ x ) .
This gives
[Equation]
completing the proof.
C P ROOF OF T HEOREM 4.
First, note that the error can be decomposed as follows:
[Equation]
where we deﬁne
[Equation]
Therefore, we just need to bound Z .
Denote by B k :=
{ f : f ∈ H , (cid:107) f (cid:107) H ≤ 1 } the unit ball of RKHS.
Deﬁne (cid:107)B k (cid:107) := sup f ∈B k and R n ( B k ) the expected Rademacher complexity of B k of data size n .
From classical RKHS theory (see Lemma C.2 below), we know that (cid:107)B k (cid:107) ∞ ≤ r k and R n ( B k ) ≤ r k / √ n , where
r k := (cid:112) sup x ∈X k
( x, x (cid:48) ) .
It remains to calculate Z := sup w ∈W D k ( ˆ B π d w (cid:107) B π d w )) .
Recall from the deﬁnition of D k that
[Equation]
Recall that { ( x i , x (cid:48) i ) } ni
=1 is a set of transitions with x (cid:48)
i ∼ d π ( · | x i ) , where d π denotes the transition probability under policy π .
Following the deﬁnitions of ˆ B π d w and B π d w , we have
[Equation]
[Equation]
[Equation]
where ¯ x (cid:48)
i ∼ d π ( · | x i ) is an independent copy of x (cid:48)
i that follows the same distribution.
Note that ¯ x (cid:48)
i is introduced only for the sample complexity analysis.
Note that Z is a random variable, and E [ Z ] denotes its expectation w.r.t.
random data { x i , x (cid:48) i } ni =1 .
We assume different ( x i , x (cid:48) i ) are independent with each other.
First, by McDiarmid inequality, we have
[Equation]
This is because when changing each data point ( x i , x (cid:48) i ) , the maximum change on Z is at most 2 (cid:107)W(cid:107) ∞ (cid:107)B k (cid:107) ∞ /n .
Therefore, we have Z ≤ [Equation] [ Z ] + [Equation]
2 log(
[Equation] δ [Equation] /n with probability at least 1 − δ .
Accordingly, we now just need to bound E
[ Z ] :
[Equation]
(because { σ i } are i.i
[Equation]
d. Rademacher random variables)
[Equation]
, where
[Equation]
By Lemma C.1 below
[Equation]
we have
[Equation]
Combining the bounds, we have with probability 1 − δ ,
[Equation]
n
[Equation]
Plugging Lemma C.2, we have
[Equation]
log(1 /δ ) / 8 (cid:17) √ n .
Assume r max = max( (cid:107)W(cid:107) ∞ , r k ) .
We have
[Equation]
Lemma C.1.
Denote by (cid:107)W(cid:107) ∞
=
sup {(cid:107) f (cid:107) ∞ : f ∈ W} the super norm of a function set W .
Then,
R n ( W ⊗ B k ) ≤ 2 ( (cid:107)W(cid:107) ∞ + (cid:107)B k (cid:107) ∞ ) ( R n ( W ) + R n ( B k )) .
Proof.
Note that
[Equation]
Note that x 2 is 2( (cid:107)W(cid:107) ∞ + (cid:107)B k
[Equation]
∞ )
-Lipschitz on interval [ − (cid:107)W(cid:107) ∞ − (cid:107)B k (cid:107) ∞ , (cid:107)W(cid:107) ∞
+ (cid:107)B k (cid:107) ∞ ] .
Applying Lemma C.1 of Liu & Wang (2018), we have
[Equation]
where W ⊕ B k = { f ( x ) + g ( x (cid:48) ): f ∈ W , g ∈ B k } , and
[Equation]
[Equation]
Remark The same result holds when w is deﬁned as a function of the whole transition pair ( x, x (cid:48) ) , that is, w i =
w ( x i , x (cid:48) i ) .
Lemma C.2.
Let H be the RKHS with a positive deﬁnite kernel k ( x, x (cid:48) ) on domain X .
Let B k
= { f ∈ H k : (cid:107) f (cid:107) H k ≤ 1 } be the unit ball of H k , and r k = (cid:112) sup x ∈X k ( x, x (cid:48) ) .
Then,
[Equation]
Proof.
These are standard results in RKHS theory, and we give a proof for completeness.
For (cid:107)B k (cid:107) ∞ , we just note that for any f ∈ B k and x ∈ X ,
[Equation]
The Rademacher complexity can be bounded as follows:
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
n (cid:88) i =
[Equation]
x
[Equation]
· ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
D M INI - BATCH TRAINING
The objective (cid:96) ( ω ) is not in a form that is ready for mini-batch training.
It is possible to yield better scalability with a trick that has been found useful in other machine learning contexts (e.g., Jean et al., 2015).
We start with a transformed objective:
[Equation]
Then, ∇
[Equation]
where ˆ E ij [ · ] and ˆ E i [ · ] correspond to two properly deﬁned discrete distributions deﬁned on D 2 and D , respectively.
Clearly, ∇ L can now be approximated by mini-batches by drawing random samples from D 2 or D to approximate ˆ E ij and ˆ E i .
E P SEUDO -C ODE OF A LGORITHM
This section includes the pseudo-code of our algorithm that we described in Section 4.
Inputs: Dataset D = { ( s i , a i , r i , s (cid:48) i ) } 1 ≤ i ≤ n of behavior policy π BEH , kernel function k , target policy π , parametric model W ( . ) .
Output: Target policy value estimate ˆ ρ π .
1: Estimate important weights { w i } 1 ≤ i ≤ n by solving the optimization problem in equation 11. return ˆ ρ
π = (cid:80) ni
=1 w i r i (see equation 6).
F B IAS -V ARIANCE C OMPARISON
In this section, we compare the performance of our approach from the bias and variance perspective.
In particular, we focus on the ModelWin MDP (Figure 1).
In order to see the impact of training data size on the performance of different estimators, we consider different number of trajectories of length 4 .
For each setting, we have 200 independent runs and calculate the bias and variance of each method based on them.
We compare our approach with Liu et al.
(2018) and Nachum et al.
(2019) that are two state-of-the-art methods in the off-policy estimation problem.
Figure 4 shows the results of this comparison.
As we can see, our method has a smaller bias but larger variance compared to other approaches on this problem.
As mentioned before, DualDICE (Nachum et al., 2019) does not cover the undiscounted reward criterion.
Therefore, instead of γ = 1 , we have used γ = 0 .
9999 in the code shared by the authors.
This induces some bias but reduces the variance of this method.
To conﬁrm this observation, we further decreased the γ to 0.9 in DualDICE and observed that reducing γ in this algorithm increases the bias but reduces the variance at the same time.
The comparison to IPS of Liu et al.
(2018) highlights the signiﬁcance of an assumption needed by
IPS: the data must be drawn (approximately) from the stationary distribution of the behavior policy.
As the trajectory length is 4 in the experiment, this assumption is apparently violated, thus the high bias of IPS.
In contrast, our method does not rely on such an assumption, so is more robust.
G S ENSITIVITY
Finally, in Figure 5 we measure how robust our approach is to changing the behavior policy compared to other methods.
In particular, we vary α 1 that corresponds to the behavior policy to measure how the
RMSE is affected.
While α 2 is ﬁxed to 0.9, in each experiment we choose α 1 from { 0 . 7 , 0 . 5 , 0 . 3 , 0 . 1 } .
For each experiment, we use data from 50 trajectories (with T BEH = 200 ) collected by the behavior policy and report results based on 20 Monte-Carlo samples.
According to Figure 5, as α 1 diverges more from α 2 , the performance of all the methods degrade while our method is the least affected.
It is worth mentioning that for the Mountain Car problem and α 1 = 0 .
1 , the behavior policy is close to a random policy and hence the car has not been able to drive up to top of the hill.
This means that all the methods have constantly received a reward of − 1 during all the steps and hence the estimated on-policy reward has been -1 for all the methods as well.
Therefore, the RMSE of all four methods are equal in this case.