Original: I NFINITE - HORIZON O FF -P OLICY P OLICY E VALUATION WITH M ULTIPLE B EHAVIOR P OLICIES
Revised: I NFINITE - HORIZON O FF -P OLICY P OLICY E VALUATION WITH M ULTIPLE B EHAVIOR P OLICIES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.
Revised: We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.

Original: Recent work has shown the key role played by the state or state-action stationary distribution corrections in the inﬁnite horizon context for off-policy policy evaluation.
Revised: Recent work has shown the key role played by the state or state-action stationary distribution corrections in the inﬁnite horizon context for off-policy policy evaluation.

Original: We propose estimated mixture policy (EMP), anovel class of partially policy-agnostic methods to accurately estimate those quantities.
Revised: We propose estimated mixture policy (EMP), a partially policy-agnostic methods to accurately estimate those quantities.

Original: With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction.
Revised: With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction.

Original: In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers signiﬁcantly improved accuracy compared to the state-of-the-art methods.
Revised: In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers signiﬁcantly improved accuracy compared to the state-of-the-art methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In many real-world decision-making scenarios, evaluating a novel policy by directly executing it in the environment is generally costly and can even be downright risky.
Revised: In many real-world decision-making scenarios, evaluating a novel policy by directly executing it in the environment is generally costly and can even be downright risky.

Original: Examples include evaluating a recommendation policy (Swaminathan et al., 2017; Zheng et al., 2018), a treatment policy (Hirano et al., 2003; Murphy et al., 2001), and a trafﬁc light control policy (Van der Pol & Oliehoek, 2016).
Revised: Examples include evaluating a recommendation policy (Swaminathan et al., 2017; Zheng et al., 2018), a treatment policy (Hirano et al., 2003; Murphy et al., 2001), and a trafﬁc light control policy (Van der Pol & Oliehoek, 2016).

Original: Off-policy policy evaluation methods (OPPE) utilize a set of previously-collected trajectories (for example, website interaction logs, patient trajectories, or robot trajectories) to estimate the value of a novel decision-making policy without interacting with the environment (Precup et al., 2001; Dud´ık et al., 2011).
Revised: Off-policy policy evaluation methods (OPPE) utilize a set of previously-collected trajectories (for example, website interaction logs, patient trajectories, or robot trajectories) to estimate the value of a novel decision-making policy without interacting with the environment (Precup et al., 2001; Dud ´ ık et al., 2011).

Original: For many reinforcement learning applications, the value of the decision is deﬁned in a long- or inﬁnite-horizon, which makes OPPE more challenging.
Revised: For many reinforcement learning applications, the value of the decision is deﬁned in a long- or inﬁnite-horizon, which makes OPPE more challenging.

Original: The state-of-the-art methods for inﬁnite-horizon off-policy policy evaluation rely on learning (discounted) state stationary distribution corrections or ratios .
Revised: The state-of-the-art methods for inﬁnite-horizon off-policy policy evaluation rely on learning (discounted) state stationary distribution corrections or ratios .

Original: In particular, for each state in the environments, these methods estimate the likelihood ratio of the long-term probability measure for the state to be visited in a trajectory generated by the target policy , normalized by the probability measure generated by the behavior policy .
Revised: In particular, for each state in the environments, these methods estimate the likelihood ratio of the long-term probability measure for the state to be visited in a trajectory generated by the target policy , normalized by the probability measure generated by the behavior policy .

Original: This approach can effectively avoid the exponentially high variance compared to the more classic importance sampling (IS) estimation methods (Precup, 2000;
Revised: This approach can effectively avoid the exponentially high variance compared to the more classic importance sampling (IS) estimation methods (pre; Dud´ık et al., 2011;

Original: Dud´ık et al., 2011; Hirano et al., 2003; Wang et al., 2017; Murphy et al., 2001), especially for inﬁnitehorizon policy evaluation (Liu et al., 2018; Nachum et al., 2019; Hallak & Mannor, 2017).
Revised: Hirano et al., 2003; Wang et al., 2017; Murphy et al., 2001), especially for inﬁnite-horizon policy evaluation (Liu et al., 2018; Nachum et al., 2019; Hallak & Mannor, 2017).

Original: However, learning state stationary distribution requires detailed information on distributions of the behavior policy, and we call them policy-aware methods.
Revised: However, learning state stationary distribution requires detailed information on distributions of the behavior policy, and we call them policy-aware methods.

Original: As a consequence, policy-aware methods are difﬁcult to apply when off-policy data are pre-generated by multiple behavior policies or when the behavior policy’s form is unknown.
Revised: As a consequence, policy-aware methods are difﬁcult to apply when off-policy data are pre-generated by multiple behavior policies or when the behavior policy’s form is unknown.

Original: To address this issue, Nachum et al.
Revised: To address this issue, Nachum et al.

Original: (2019) proposes a policy-agnostic method, DualDice, which learns the joint state-action stationary distribution correction that is much higher dimension, and therefore needs more model parameters than the state stationary distribution.
Revised: (2019) proposes a policy-agnostic method, DualDice, which learns the joint state-action stationary distribution correction that is much higher dimension, and therefore needs more model parameters than the state stationary distribution.

Original: Besides, there is no theoretic comparison between policy-aware and policy-agnostic methods.
Revised: Besides, there is no theoretic comparison between policy-aware and policy-agnostic methods.

Original: In this paper, we propose a partially policy-agnostic method, EMP (estimated mixture policy) for inﬁnite-horizon off-policy policy evaluation with multiple known or unknown behavior policies.
Revised: In this paper, we propose a OPPE method with behavior policy learning, EMP (estimated mixture policy) for inﬁnite-horizon off-policy policy evaluation with multiple known or unknown behavior policies.

Original:  EMP is partially policy-agnostic  in the since that it does notnecessarily require knowledge of theindividual behavior policies.
Revised: We call EMP a partially policy-agnostic method in the sense that, EMP does not require any information on each“physical” behavior policy, instead, it utilizes some aggregated information of the behavior policies learned from data. In detail, EMP includes a pre-estimation step using certain parametric model to learn a “virtual” policy (we call it the mixture policy and formally deﬁne it in Section 4).

Original: Instead, it involves a pre-estimation step to estimate a single mixed policy that will be deﬁned formally later.
Revised: Hence, its performance depends on the accuracy of mixture policy estimation.

Original: Like the method in Liu et al.
Revised: Like the method in Liu et al.

Original: , EMP  also learns the state stationary distribution correction, so it remains computationally cheap and is scalable in terms of the number of behavior policies.
Revised: (2018), EMP obtain OPPE also via learning the state stationary distribution correction, so it remains computationally cheap and is scalable in terms of the number of behavior policies.

Original: Inspired by Hanna et al.
Revised: Besides, inspired by Hanna et al.

Original: (2019), we construct a theoretical bound for the mean square error (MSE) of the  stationary distribution corrections learned by EMP. In particular, we show that in the single-behavior policy setting, EMP yields smaller MSE than the policy-aware method.
Revised: (2019), we provide a theoretic guarantee that EMP yields smaller mean square error (MSE) than the policy-aware methods in stationary distribution corrections learning, even in the single-behavior policy setting.

Original: On the other hand, compared to DualDice, EMP learns the state stationary distribution correction of smaller dimension, more importantly the estimation of the mixture policy can be considered as an inductive bias as far as the stationary distribution correction is concerned, and hence could achieve better performance when the pre-estimation is not expensive.
Revised: On the other hand, compared to DualDice, EMP learns the state stationary distribution correction of smaller dimension, more importantly the estimation of the mixture policy can be considered as an inductive bias as far as the stationary distribution correction is concerned, and hence could achieve better performance when the pre-estimation is not expensive.

Original: In addition, we propose an ad-hoc improvement of EMP, whose theoretical analysis is left for future studies.
Revised: In addition, we propose an ad-hoc improvement of EMP, whose theoretical analysis is left for future studies.

Original: EMP is compared with both policy-aware and policy-agnostic methods in a set of continuous and discrete control tasks and shows signiﬁcant improvement.
Revised: EMP is compared with both policy-aware and policy-agnostic methods in a set of continuous and discrete control tasks and shows signiﬁcant improvement.

Revised:  B ACKGROUND AND R ELATED W ORK

Revised: We ﬁrst introduce the general setting of OPPE in inﬁnite horizon.

Revised: Then we review two families of

Revised: OPPE methods, based on importance sampling (IS) and stationary distribution correction learning, respectively.

Original: B ACKGROUND AND R ELATED W ORK 2. I NFINITE - HORIZON O FF - POLICY P OLICY E VALUATION
Revised: 2. I NFINITE - HORIZON O FF - POLICY P OLICY E VALUATION

Original: We consider a Markov Decision Process (MDP) and our goal is to estimate the inﬁnite-horizon average reward .
Revised: We consider a Markov Decision Process (MDP) and our goal is to estimate the inﬁnite-horizon average reward .

Original: The environment is speciﬁed by a tuple M = (cid:104) S, A, R, T (cid:105) , consisting of a state space, an action space, a reward function, and a transition probability function.
Revised: The environment is speciﬁed by a tuple M = (cid:104) S, A, R, T (cid:105) , consisting of a state space, an action space, a reward function, and a transition probability function.

Original: A policy π interacts with the environment iteratively, starting with an initial state s 0 .
Revised: A policy π interacts with the environment iteratively, starting with an initial state s 0 .

Original: At step n = 0 , 1 , ... , the policy produces a distribution π ( ·| s n ) over the actions A , from which an action a n is sampled and applied to the environment.
Revised: At step n = 0 , 1 , ... , the policy produces a distribution π ( ·| s n ) over the actions A , from which an action a n is sampled and applied to the environment.

Original: The environment stochastically produces a scalar reward r ( s n , a n ) and a next state s n +1 ∼ T ( ·| s n , a n ) .
Revised: The environment stochastically produces a scalar reward r ( s n , a n ) and a next state s n +1 ∼ T ( ·| s n , a n ) .

Original: The inﬁnite-horizon average reward under policy π is
Revised: The inﬁnite-horizon average reward under policy π is

Original: [Equation]
Revised: [Equation]

Original:  Without gathering new data, off-policy policy evaluation (OPPE) considers the problem of estimating the expected reward of a target policy π viaa pre-collected state-action-reward tuples from policies that are different from π , which are called behavior policies.
Revised: π [ r ( s n , a n )] . Without gathering new data, off-policy policy evaluation (OPPE) considers the problem of estimating the expected reward of a target policy π via pre-collected state-action-reward tuples from policies that are different from π , which are called behavior policies.

Original: In our paper, we consider the general setting that the data are generated by multiple behavior policies
Revised: In our paper, we consider the general setting that the data are generated by multiple behavior policies

Original: π j ( j =0 , 1 , .., m ) .
Revised: π j ( j = 1 , .., m ) .

Original: Most OPPE literature has focused on the single-behavior-policy case where m = 1 .
Revised: Most OPPE literature has focused on the single-behavior-policy case where m = 1 .

Original: In this case, we denote the behavior policy by π 0 .
Revised: In this case, we denote the behavior policy by π 0 to distinguish from the multiple-behavior-policy case.

Original: Roughly speaking, most OPPE methods can be grouped into two categories: importance-sampling(IS) based OPPE and stationary-distribution-correction based OPPE.
Revised: Roughly speaking, most OPPE methods can be grouped into two categories: importance-sampling(IS) based OPPE and stationary-distribution-correction based OPPE.

Original: 2. I MPORTANCE S AMPLING P OLICY E VALUATIONU SING E XACT AND E STIMATED B EHAVIOR P OLICY
Revised: 2. I MPORTANCE S AMPLING P OLICY E VALUATION

Original: As for short-horizon off-policy policy evaluation, importance sampling policy evaluation (IS) methods (Precup et al., 2001; Dud ´ ık et al., 2011; Swaminathan et al., 2017; Precup et al., 2000; Horvitz &
Revised: As for short-horizon off-policy policy evaluation, importance sampling policy evaluation (IS) methods (Precup et al., 2001; Dud´ık et al., 2011; Swaminathan et al., 2017; Precup et al., 2000; Horvitz &

Original: Thompson, 1952) have shown promising empirical results.
Revised: Thompson, 1952) have shown promising empirical results.

Original: The main idea of importance sampling based OPPE is using importance weighting π/π j to correct the mismatch between the target policy π and the behavior policy π j that generates the trajectory.
Revised: The main idea of importance sampling based OPPE is using importance weighting π/π j to correct the mismatch between the target policy π and the behavior policy π j that generates the trajectory.

Revised: One key element in our EMP method are inspired by importance sampling literature.

Original: Li et al.
Revised: Li et al.

Original: (2015) and Hanna et al.
Revised: (2015) and Hanna et al.

Original: (2019) show that using estimated behavior policy in the importance weighting can obtain importance sampling estimation with smaller mean square error (MSE).
Revised: (2019) show that using estimated behavior policy in the importance weighting can reduce the mean square error (MSE).

Original: EMP also uses estimated policy, but there are two key difference between EMP and the previous works: (1) EMP is not an IS-based method, it involves a min-max problem; (2) EMP focuses on multiplebehavior-policy setting while previous works have focused on single-behavior setting.
Revised: EMP also uses estimated policy, but there are two key difference between EMP and the previous works: (1) EMP is not an IS-based method, it involves a min-max problem; (2) EMP focuses on multiple-behavior-policy setting while these papers have focused on single-behavior setting.

Original: 2. P OLICY E VALUATION VIA L EARNING S TATIONARY D ISTRIBUTION C ORRECTION
Revised: 2. P OLICY E VALUATION VIA L EARNING S TATIONARY D ISTRIBUTION C ORRECTION

Original: The state-of-the-art methods for long-horizon off-policy policy evaluation are stationary-distributioncorrection based (Liu et al., 2018; Nachum et al., 2019; Hallak & Mannor, 2017).
Revised: The state-of-the-art methods for long-horizon off-policy policy evaluation are stationary-distributioncorrection based (Liu et al., 2018; Nachum et al., 2019; Hallak & Mannor, 2017).

Original: Let d π 0 ( s ) and d π ( s ) be the stationary distribution of state s under the behavior policy π 0 and target policy π respectively.
Revised: Let d π 0 ( s ) and d π ( s ) be the stationary distribution of state s under the behavior policy π 0 and target policy π respectively.

Original: The main idea  is directly applying importance weighting by ω =
Revised: The main idea of such methods is directly applying importance weighting by ω =

Original: d π /d π 0 on the stationary state-visitation distributions to avoid the exploding variance suffered from IS, and estimate the average reward as
Revised: d π /d π 0 on the stationary state-visitation distributions to avoid the exploding variance suffered from IS, and estimate the average reward as

Original: [Equation]
Revised: [Equation]

Original: For example, Liu et al.
Revised: For example, Liu et al.

Original: (2018) uses min-max approach to estimate ω directly from the data.
Revised: (2018) uses min-max approach to estimate ω directly from the data.

Original: This class of methods require exact knowledge of behavior policy π 0 and are not straightforward to apply in multiple-behavior-policy setting.
Revised: This class of methods require exact knowledge of behavior policy π 0 and are not straightforward to apply in multiple-behavior-policy setting.

Original: Recently, Nachum et al.
Revised: Recently, Nachum et al.

Original: (2019) proposes DualDice to overcome such limitation by learning the state-action stationary distribution correction ω ( s, a ) =
Revised: (2019) proposed DualDice to overcome such limitation by learning the state-action stationary distribution correction ω ( s, a ) =

Original: d π ( s ) π ( a | s ) /d
Revised: d π ( s ) π ( a | s ) /d

Original: π 0 ( s ) π 0 ( a | s ) .
Revised: π 0 ( s ) π 0 ( a | s ) .

Original:  S INGLE B EHAVIOR P OLICY 
Revised:  S INGLE B EHAVIOR P OLICY E STIMATION

Original: We ﬁrst consider the task of stationary distribution correction learning in  the simple case where the data are generated by a single behavior policy as previous state stationary distribution correction methods.
Revised: When the behavior policy π 0 in (1) is unknown, a natural idea is to estimate it from data. In this section, we focus on the standard case where the data are generated by a single behavior policy so that estimation of behavior policy is more straightforward.

Original: To explain the min-max problem formulation of the learning task, we ﬁrst breiﬂy review the method introduced by Liu et al.
Revised: We ﬁrst breiﬂy review the method introduced by Liu et al.

Original: (2018) in Section 3.1, which we shall refer as the BCH method in the rest of the paper. In Section 3.2, we show thatit is beneﬁcial to replace the exact values of the behavior policy in the min-max problem by their estimated values in two folds.
Revised: (2018) in Section 3.1, which we shall refer as the BCH method in the rest of the paper, to explain the min-max problem formulation of the stationary distribution correction learning task. In Section 3.2, we show that behavior policy estimation is beneﬁcial in two aspects.

Original: First, this extends the  method to application setting wherethe behavior policy is unknown.
Revised: First, it extends the stationary distribution correction method to settings where behavior policy is unknown.

Original: Second, even when the behavior policy is known with exact values, we prove that the stationary distribution correction learned by the min-max problem with estimated behavior policy  has smaller MSE.
Revised: Second, even when the behavior policy is known with exact values, we prove that the stationary distribution correction learned using behavior policy estimation has smaller MSE than that using exact values.

Original: We will deal with multiple-behavior-policy cases in Section 4.
Revised: Later, we will extend this behavior policy estimation idea to more general multiple-behavior-policy cases in Section 4.

Original: 3. L EARNING S TATIONARY D ISTRIBUTION C ORRECTION WITH EXACT B EHAVIOR P OLICY
Revised: 3. L EARNING S TATIONARY D ISTRIBUTION C ORRECTION WITH E

Original: Assume the data, consisting of state-action-next-state tuples, are generated by a single behavior policy π 0 , i.e. D = { ( s n , a n , s (cid:48) n ) : n = 1 , 2 , ..., N } .
Revised: Assume the data, consisting of state-action-next-state tuples, are generated by a single behavior policy π 0 , i.e. D = { ( s n , a n , s (cid:48) n ) : n = 1 , 2 , ..., N } .

Original: Recall that d π 0 and d π are the stationary state distribution under the behavior and target policy respectively, and ω =
Revised: Recall that d π 0 and d π are the stationary state distribution under the behavior and target policy respectively, and ω =

Original: d π /d π 0 is the stationary distribution correction .
Revised: d π /d π 0 is the stationary distribution correction .

Original: In the rest of Section 3, by slight notation abusion, we also denote d π ( s, a ) =
Revised: In the rest of Section 3, by slight notation abuse, we also denote d π ( s, a ) =

Original: d π ( s ) π ( a | s ) ,
Revised: d π ( s ) π ( a | s ) ,

Original: d π 0 ( s, a ) =
Revised: d π 0 ( s, a ) =

Original: d π 0 ( s )
Revised: d π 0 ( s )

Original: π 0 ( a | s ) and d π 0 ( s, a, s (cid:48) )
Revised: π 0 ( a | s ) and d π 0 ( s, a, s (cid:48) )

Original: =
Revised: =

Original: d π 0 ( s ) π 0 ( a | s ) T ( s (cid:48) | a, s ) .
Revised: d π 0 ( s ) π 0 ( a | s ) T ( s (cid:48) | a, s ) .

Original: We brieﬂy review the BCH method proposed by Liu et al.
Revised: We brieﬂy review the BCH method proposed by Liu et al.

Original: As d π ( s ) is the stationary distribution of s n as n → ∞ under policy π , it follows that:
Revised: As d π ( s ) is the stationary distribution of s n as n → ∞ under policy π , it follows that:

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any function f : S → R , 
Revised: Therefore, for any function f : S → R , (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Recall that d π 0 ( s, a, s
Revised: Recall that d π 0 ( s, a, s

Original: [Equation]
Revised: [Equation]

Original:  =d π [Equation]
Revised: ) =

Original: s
Revised: d

Original: [Equation]
Revised: [Equation]

Original: π

Original: [Equation]
Revised: [Equation]

Original: a | s 
Revised: s )

Original: [Equation]
Revised: [Equation]

Original: T

Original: [Equation] s [Equation] | a, s ) , so ω and the data sample satisfy the following equation
Revised: 0 ( a | s ) T ( s (cid:48) | a, s ) , so ω and the data sample satisfy the following equation

Original: [Equation]
Revised: [Equation]

Original: BCH solves the above equation via the following min-max problem:
Revised: BCH solves the above equation via the following min-max problem:

Original: [Equation]
Revised: [Equation]

Original: and use kernel method to solve ω .
Revised: and use kernel method to solve ω .

Original: The derivation of kernel method are put in Appendix A.
Revised: The derivation of kernel method is put in Appendix A.

Original: 3. L EARNING S TATIONARY D ISTRIBUTION C ORRECTION WITH E STIMATED B EHAVIOR P OLICY
Revised: 3. L EARNING S TATIONARY D ISTRIBUTION C ORRECTION WITH E STIMATED B EHAVIOR P OLICY

Original: The objective function in the min-max problem (2), evaluated by data sample, can be viewed as a one-step importance sampling estimation.
Revised: The objective function in the min-max problem (2), evaluated by data sample, can be viewed as a one-step importance sampling estimation.

Original: As shown in Hanna et al.
Revised: As shown in Hanna et al.

Original: (2019), importance sampling with estimated behavior policy has smaller MSE.
Revised: (2019), importance sampling with estimated behavior policy has smaller MSE.

Original: Motivated by this fact and the heuristic that better objective function evaluation will lead to more accurate solution, we show that the BCH method can also be improved by using estimated behavior policy and obtain smaller asymptotic MSE.
Revised: Motivated by this fact and the heuristic that better objective function evaluation will lead to more accurate solution, we show that the BCH method can also be improved by using estimated behavior policy to obtain smaller asymptotic MSE.

Original: We will use this result to build theoretic guarantee for the performance of EMP method in Section 4.
Revised: We will use this result to build theoretic guarantee for the performance of EMP method in Section 4.

Original: To formally state the theoretic result, we need  introduce more notation.
Revised: To formally state the theoretic result, we need to introduce more notation.

Original: Assume that we are given a class of stationary distribution correction Ω
Revised: Assume that we are given a class of stationary distribution correction Ω

Original: = { ω ( η ; s ) : η ∈ E η } , and there exists η 0 ∈ E
Revised: = { ω ( η ; s ) : η ∈ E η } , and there exists η 0 ∈ E

Original: η such that the true distribution correction ω
Revised: η such that the true distribution correction ω

Original: ( s )
Revised: ( s )

Original: = ω
Revised: = ω

Original: ( η 0 ; s ) .
Revised: ( η 0 ; s ) .

Original: Let ω (˜ η ; s ) be the stationary distribution correction learned by the min-max problem (2) and
Revised: Let ω (˜ η ; s ) be the stationary distribution correction learned by the min-max problem (2) and

Original: ω (ˆ η ; s ) be that learned by a min-max problem using estimated policy: [Equation]
Revised: ω (ˆ η ; s ) be that learned by a min-max problem similar to (2) with π 0 replaced with its estimation ˆ π 0 .

Original: The intuition is that the value of ˆ π 0 is estimated from the data sample and appears in the denominator, as a result, it could cancel out a certain amount of random error in data sample.
Revised: Intuitively, ˆ π 0 is estimated from the data sample and appears in the denominator, as a result, it could cancel out a certain amount of random error in data sample.

Original: We use a maximum likelihood method to estimate the behavior policies for discrete and continuous control tasks.

Original: The details are in Appendix E.2. Based on the proof techniques in Henmi et al.
Revised: Following this intuition and applying the proof techniques in Henmi et al.

Original: (2007), we establish the following theoretic guarantee that using estimated behavior policy  yields better estimates of the stationary distribution correction.
Revised: (2007), we establish the following theoretic guarantee that using estimated behavior policy 1 yields better estimates of the stationary distribution correction.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under some mild conditions, we have, asymptotically
Revised: Under Assumptions 1 and 2, we have, asymptotically

Original: [Equation]
Revised: [Equation]

Original: As a direct consequence, we derive the ﬁnite-sample error bound for ˆ η .
Revised: As a direct consequence, we derive the ﬁnite-sample error bound for ˆ η .

Original: Corollary 1.
Revised: Corollary 1.

Original: (informal) Let N be the number of ( s, a, s (cid:48) ) tuples in the data,
Revised: Let N be the number of ( s, a, s (cid:48) ) tuples in the data. Under Assumptions 1 and 2,

Original: [Equation]
Revised: [Equation]

Original: The precise conditions for Theorem 1 and Corollary 1 to hold and their proofs are in Appendix B.
Revised: Due to the space limit, we put the precise descriptions of Assumptions 1 and 2 (which involves details of the kernel method that solves the min-max problem (2)) for Theorem 1 and Corollary 1 to hold and their proofs are in Appendix B.

Original:  EMP FOR M ULTIPLE B EHAVIOR P OLICIES
Revised:  EMP FOR M ULTIPLE B EHAVIOR P OLICIES

Original: In this section, we shall propose our EMP method for off-policy policy evaluation with multipleknown or unknown behavior policies and establish theoreticresults on variance reduction of EMP.
Revised: In this section, we apply the behavior policy estimation idea and develop our EMP method for OPPE in settings of multiple behavior policies and establish theoretic variance reduction results for EMP.

Revised: In

Original: Before that, we ﬁrst give a detailed description on the  data sample and its distribution.
Revised: Section 4.1, we ﬁrst clarify what is the policy to estimate from data when there are multiple behavior policies. Then, we introduce EMP in Section 4.2 and establish variance reduction result in 4.3. 4. M IXTURE P OLICY AND M IXTURE S TATE D ISTRIBUTION Let’s ﬁrst take a closer look at the distribution of data generated by multiple behavior policies.

Revised: In particular, we show that the data from different behavior policies can be pooled together as if they are generated by a virtual“mixture policy” π M , which plays a key role in derivation of EMP method.

Original: Assume the state-action-next-state tuples are generated by m different unknown behavior policies π j , j = 1 , 2 , ..., m . Let d π j ( s ) be the stationary state distribution and N j be the number of state-action-next-state tuples  bypolicy π j , for j= 1 , 2 , ..., m .
Revised: In the multiple-behavior setting, we assume the state-action-next-state tuples are generated by m different unknown behavior policies π j , j = 1 , 2 , ..., m . For each j , there are N j state-action-nextstate tuples generated by π j and follows the corresponding stationary state distribution d π j .

Original: Let N = (cid:80) j N j and denote by w j = N j /N the proportion of data generated by policy π j .
Revised: Let N = (cid:80) j N j and denote by w j = N j /N the proportion of data generated by policy π j .

Original: We use D  to denote the data set and D  = { ( s j,n j , a j,n j , s (cid:48) j,n j ) : j = 1 , 2 , .., m, n j = 1 , 2 , ..., N j } .
Revised: We use D M to denote the data set, then D M = { ( s j,n j , a j,n j , s (cid:48) j,n j ) : j = 1 , 2 , .., m, n j = 1 , 2 , ..., N j } .

Original: Note that the policy label j in the subscript is only for notation clarity
Revised: Note that the policy label j in the subscript is only for notation clarity

Original: and it is not revealed in the data.
Revised: and it is not revealed in the data.

Original: Then,  a single ( s, a, s (cid:48) ) tuple simply follows the marginal distribution d 0 ( s, a, s (cid:48) ) :
Revised: Then, if we randomly draw a single ( s, a, s (cid:48) ) tuple from D M , its distribution function is the mixture of state-action-next-state tuple distributions generated by each behavior policy:

Original: = (cid:80)
Revised: [Equation]

Original: j w j d π j ( s ) π j ( a | s ) T ( s (cid:48) | a, s ) .

Original: With slight notation abusion, we write d 0 ( s, a ) =
Revised: With slight notation abuse, we write d M ( s ) =

Original: (cid:80) j w j d π j ( s ) π ( a | s ) .4. EMP M ETHOD
Revised: (cid:80) j w j d π j as the mixture state distribution .

Original: Now we derive the EMP method in the multiple-behavior-policy setting and explicitly explain what is the mixed policy to be estimated in EMP.

Original: Let d 0 := (cid:80)

Original: j w j d π j be the mixture of stationary distributions of the behavior policies.

Original: For each state-action pair ( a, s ) , deﬁne  π 0 ( a | s ) as the weighted average of the behavior policies:
Revised: For each state-action pair ( a, s ) , deﬁne the mixture policy π M ( a | s ) as the weighted average of the behavior policies:

Original: [Equation]
Revised: [Equation]

Original: It is easy to check that for each s , π 0 ( ·| s ) is a distribution on the action space and hence deﬁnes a policy by itself.

Original: We call π 0 the mixed policy .

Original: Let ω = d π /d 0 , which is a state distributionratio. Then, d 0 , π 0 and ω satisfy the following relation with the average reward R π .
Revised: The following result shows that, the multiple-behavior-policy data D M and the corresponding state distribution d M ( s ) can be viewed as if they were generated by the mixture policy π M

Original: Proposition 1.
Revised: Proposition 1.

Original: [Equation]
Revised: The state-action-next-station tuples generated by π M follows the distribution d M ( s, a, s (cid:48) ) . As a consequence, d M ( s ) is the stationary state distribution generated by π M . Let ω M ( s ) = d π ( s ) /d

Original: Besides, the state distribution ratio ω can be characterized by the stationary equation.
Revised: M ( s ) be the likelihood ratio of the state distribution generated by the target policy over the mixture state distribution. We can estimate the average reward by ω M :

Original: Proposition 2.
Revised: Proposition 2.

Original: The function
Revised: The average award satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  to 
Revised: Proposition 1 shows that d M ( s ) , the state distribution generated by multiple behavior policies, is equal to the state stationary distribution generated by π M .

Original: [Equation] constant [Equation] if and only
Revised: If π M is known, then, following the same argument of BCH, we can learn ω M ( s )

Original: [Equation]
Revised: = ω

Original: In the special case when m = 1 , i.e. the data are generated by a single behavior policy, Propositionreduces to Theorem 1 of Liu et al. The above two Propositions indicate that, to certain extend, the ( s, a, s(cid:48) ) tuples generated by multiple behavior policies can be pooled together and treated as if they are generated by a single behavior policy π 0  . 
Revised: ( η 0 ; s ) via a similar min-max problem as (2), by replacing d π 0 with d M and π 0 with π M , and then, estimate R π using (4), by replacing d π 0 with d M , π 0 with π M and ω with ω M . But π M is usually unknown.

Original: Note that expression of π 0 (4) involves not only the behavior policies but also the state stationary distributions.
Revised: Indeed, it involves not only the behavior polices π j , but also their stationary distributions d π j , which are unknown and hard to compute.

Original: In  EMP method, we shall use a pre-estimation step to generate an estimateˆ π 0 from the data.
Revised: In our EMP method, we will estimate π M directly from data, without learning π j and d π j .

Revised: In particular, we assume the mixture policy π M belongs to some parametric family, i.e. π M ( a | s ) =

Original: Based on Proposition 2, the state distribution ratio ω can be estimated by the following min-max problem
Revised: π ( θ M ; a, s ) . For instance, π ( θ ; a, s ) could be a regression model or a neural network. We then estimate θ M by MLE, i.e.

Original: [Equation]
Revised: [Equation]

Original: Finally, EMP estimates the average reward according to (5) where d 0 is approximate by data D . Applying Theorem 1, we show that using the estimated ˆ π 0 in EMP can actually reduce the  MSE of the learned stationary distribution ratio ω .
Revised: After this pre-estimation step, we replace the exact mixture policy π M with the estimated mixture policy π (ˆ θ M ; · ) and ﬁnally formulate the following min-max problem for EMP to learn ω M : [Equation] Applying Theorem 1, we build the following MSE bound for EMP method.

Original: Proposition 3.
Revised: Theorem 2.

Original: Under the same conditions of Theorem 1, if ω (˜ η ; s ) and ω (ˆ η ; s ) are the stationary distribution correction learned from (7) and from the same min-max problem but with exact value of π 0 , then, asymptotically
Revised: Under the same conditions of Theorem 1, if ω (˜ η ; s ) and ω (ˆ η ; s ) are the stationary distribution correction learned from (6) and from the same min-max problem but with exact value of π M , then, asymptotically

Original: [Equation]
Revised: [Equation]

Original: As a result, E [(ˆ η − η 0 ) 2 ] = O (cid:0) 1 N (cid:1) .
Revised: As a result, E [(ˆ η − η 0 ) 2 ] = O (cid:0) 1 N (cid:1) .

Original: 4. W HY P OOLING IS B ENEFICIAL FOR EMP
Revised: 4. W HY P OOLING IS B ENEFICIAL FOR EMP

Original: One important feature of EMP is that it pools the data from different policy behaviors together and treat them as if they are from a single mixed policy.
Revised: One important feature of EMP is that it pools the data from different policy behaviors together and treat them as if they are from a single mixture policy.

Original: Of course, pooling makes EMP applicable to settings with minimal information on the behavior policies, for instance, EMP does not even require the knowledge on the number of behavior policies.
Revised: Of course, pooling makes EMP applicable to settings with minimal information on the behavior policies, for instance, EMP does not even require the knowledge on the number of behavior policies.

Original: In this part, we show that, the pooling feature of
Revised: In this part, we show that, the pooling feature of

Original: EMP is not just a compromise to the lack of behavior policy information, it also leads to variance reduction in an intrinsic manner.
Revised: EMP is not just a compromise to the lack of behavior policy information, it also leads to variance reduction in an intrinsic manner.

Original: If instead, the data can be classiﬁed according to the behavior policies and treated separately, we can still use EMP,which reduces to (3), or any other single-behavior-policy method, to obtain the stationary distribution correction ω
Revised: If instead, the data are treated separately according to the behavior policies, we can still use EMP, or any OPPE method for single behavior policy, to obtain the stationary distribution correction ω

Original: j = d π /d π j for each behavior policy.
Revised: j = d π /d π j for each behavior policy.

Original: Given ω j , a common approach for variance reduction is to apply multiple importance sampling (MIS) (Tirinzoni et al., 2019; Veach & Guibas, 1995) technique and the average reward estimator is of the form
Revised: Given ω j , a common approach for variance reduction is to apply multiple importance sampling (MIS) (Tirinzoni et al., 2019; Veach & Guibas, 1995) technique and the average reward estimator is of the form

Original: [Equation]
Revised: [Equation]

Original: where the function h is often referred to as heuristics and must be a partition of unity, i.e., (cid:80) j h j ( s )
Revised: where the function h is often referred to as heuristics and must be a partition of unity, i.e., (cid:80) j h j ( s )

Original: = 1 for all s ∈ S .
Revised: = 1 for all s ∈ S .

Original: It has been proved by (Veach & Guibas, 1995) that MIS is unbiased, and, for given w j = N j /N , there is an optimal heuristic function to minimize the variance of ˆ R MIS .
Revised: It has been proved by Veach & Guibas (1995) that MIS is unbiased, and, for given w j = N j /N , there is an optimal heuristic function to minimize the variance of ˆ R MIS .

Original: Proposition 4.
Revised: Proposition 3.

Revised: (Veach & Guibas, 1995)

Original: For MIS with ﬁxed values of w j , j = 1 , 2 , ..., m , among all possible values of heuristics h , the balanced heuristic
Revised: For MIS with ﬁxed values of w j , j = 1 , 2 , ..., m , among all possible values of heuristics h , the balanced heuristic

Original: [Equation]
Revised: [Equation]

Original:  reaches the minimal variance.
Revised: and s ∈ S, reaches the minimal variance.

Original: Plug the optimal heuristic h j ( s ) into MIS estimator (8), and we will obtain that the optimal MIS estimator coincides with the EMP estimator (5), i.e. [Equation]
Revised: Plug the optimal heuristic h j ( s ) into MIS estimator (7), and we will obtain that the optimal MIS estimator coincides with the EMP estimator (4).

Original: In this light, by pooling the data together and directly learning ω . EMP also learns the optimal MIS weight inexplicitly.
Revised: In this light, by pooling the data together and directly learning ω M , EMP also learns the optimal MIS weight inexplicitly.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we conduct experiments in three discrete-control tasks Taxi, Singlepath,Gridworld and one continuous-control task Pendulum (see Appendix E.1 for the details), with following purposes:
Revised: In this section, we evaluate EMP on OPPE problems in three discrete-control tasks Taxi, Singlepath,

Original: (i) to compare the performance of distribution correction learning using policy-aware, policy- agnostic and partially agnostic-policy methods (in Sec 5.1); (ii) to compare the performance ofthe proposed EMP with existing OPPE methods (in Sec 5.1 and 5.2); (iii) to explore potential improvement of EMP methods (in Sec 5.2).
Revised: Gridworld and one continuous-control task Pendulum (see Appendix D.1 for the details), in both single-behavior-policy (Section 5.1)and multiple-behavior-policy settings (Section 5.2), with following purposes: (i) to compare the performance of EMP with existing OPPE methods; (ii) to validate the theoretical properties for EMP; (iii) to explore potential improvement of EMP methods for future study.

Original: We will release the codes with the publication of this paper for relevant study.
Revised: We will release the codes with the publication of this paper for relevant study.

Original: 5. R ESULTS FOR S INGLE B EHAVIOR P OLICY
Revised: 5. R ESULTS FOR S INGLE B EHAVIOR P OLICY

Original: In this section, we compare the EMP method with the BCH method and step-wise importance sampling (IS) in the setting of single-behavior policy, i.e. the data is generated from a single behavior policy.
Revised: In this section, we compare the EMP method with BCH, DualDice and step-wise importance sampling (IS) in the setting of single-behavior policy, i.e. the data is generated from a single behavior policy.

Original: Experiment Set-up.
Revised: Experiment Set-up.

Original: A single behavior policywhich is learned by a certain reinforcement learning algorithm 1 for evaluating BCH and IS.
Revised: A single behavior policy is learned by a certain reinforcement learning algorithm 2 for evaluating BCH and IS.

Original: This single behavior policy then generates a set of trajectories consisting of s-a-s-r tuples.
Revised: This single behavior policy then generates a set of trajectories consisting of s-a-s-r tuples.

Original: These tuples are used to estimate the behaviour policy for EMP methods as well as estimating the stationary distribution corrections for estimating the average step reward of the target policy.
Revised: These tuples are used to estimate the behaviour policy in EMP method [Figure] [Figure] as well as estimating the stationary distribution corrections for estimating the average step reward of the target policy.

Original: Stationary Distribution Learning Performance.
Revised: Stationary Distribution Learning Performance.

Original: We choose the Taxi domain as an example to compare the stationary distribution ˆ d π true and ˆ d π esti learned by BCH  and EMP.
Revised: To validate Theorem 1, we use the Taxi domain as an example to compare the stationary distribution ˆ d π true and ˆ d π esti learned by BCH (using exact behavior policy) and EMP (using estimated behavior policy).

Original: Figure 1(a) shows the scatter pairs ( ˆ d π true , d π ) and ( ˆ d π esti , d π ) estimated by 200 trajectories of 200 steps.
Revised: Figure 1(a) plots the scatter pairs ( ˆ d π true , d π ) and ( ˆ d π esti , d π ) estimated by 200 trajectories of 200 steps.

Original: It shows that ˆ d π esti approximate d π better than ˆ d π true .
Revised: It shows that ˆ d π esti approximate d π better than ˆ d π true .

Original: Figure 1(b) and Figure 1(b) compare the TV distance from ˆ d π true and ˆ d π esti to d π under different data sample sizes.
Revised: Figure 1(b) and Figure 1(b) compare the TV distance from ˆ d π true and ˆ d π esti to d π .

Original: The results indicate that both ˆ d π true and ˆ d π esti converge, while ˆ d π esti converges faster and is signiﬁcantly closer to d π when the data size is small.
Revised: The results indicate that both ˆ d π true and ˆ d π esti converge, while ˆ d π esti converges faster and is signiﬁcantly closer to d π when the data size is small.

Original: These observations are well consistent with Theorem 1.
Revised: These observations are well consistent with Theorem 1.

Original: Policy Evaluation Performance.
Revised: Policy Evaluation Performance.

Original: Figure 2 reports the MSE of policy evaluation by EMP, BCH and
Revised: Figure 2 reports the MSE of policy evaluation by EMP, BCH,

Original:  IS methods for the 4 different environments.
Revised: DualDice and IS methods for the 4 different environments.

Original: We observe that, (i) EMP consistently obtains smaller[Figure] [Figure] MSE than the other two methods for different sample scales and different environments.
Revised: We observe that, (i) EMP consistently obtains smaller MSE than the other three methods for different sample scales and different environments.

Original: (ii)
Revised: (ii)

Original: The performance of EMP and BCH  improves as the number of trajectories and length of horizons increase, while the IS method suffers from growing variance.
Revised: The performance of EMP, BCH and DualDice improves as the number of trajectories and length of horizons increase, while the IS method suffers from growing variance.

Original: our method correctly estimates the true density ratio over the state space. Partially Policy-agnostic versus Policy-agnostic OPPE. Figure 3 reports the comparison results for the policy-aware BCH, partially policy-agnostic EMP and a policy-agnostic method, which we call it state-action distribution learning (SADL) and whose formal formulation is given in Appendix D. The results show that all three methods obtain improvement as the number of length of trajectories increase.
Revised: [Figure]

Original: Roughly speaking, both EMP and SADL outperform BCH. The policy-agnostic SADL is better than EMP in the cases of small sample size.
Revised: Pendulum

Original: But when the sample size increases so that the estimated behavior policy is more accurate, EMP gradually exceeds SADL. Remark: In our implementation of SADL, we use the same min-max formulation and optimization solver as EMP so that the comparison could shed more lights on the impact of behavior policy information on the performance of off-policy policy evaluation. We will report the comparison result between EMP and DualDice once the code is released.
Revised: [Figure]

Original: 5. R ESULTS FOR M ULTIPLE U NKNOWN B EHAVIOR P OLICIES
Revised: 5. R ESULTS FOR M ULTIPLE U NKNOWN B EHAVIOR P OLICIES

Original: As for multiple behavior policies, we conduct experiments in policy-aware and partially policyagnostic settings.
Revised: In this section [Figure] we compare the performance of EMP with a multiple-behavior version of BCH method,

Original: We report the results of partially policy-agnostic setting in this section and the policy-aware setting is described in Appendix E.4. Because partially policy-agnostic version consistently achieves better performance.
Revised: DualDice and MIS (Precup et al., 2000)

Original: [Figure]
Revised: [Figure]

Revised: and explore potential improvement of EMP methods.

Original: Experiment Set-up.
Revised: Experiment Setup.

Original: We implement the following 4 methods: (1) the proposed EMP; (2) the multiple importance sampling (MIS) method as in Tirinzoni et al.
Revised: For each environment, we use 5 behavior policies to generate the data. They are obtained by reinforcement learning after different number of training steps, i

Original: (2019) using balanced heuristics; (3) EMP (single), in which we apply EMP for each subgroup of samples generated bya single behavior policy to obtain one OPPE value and ﬁnally output their average; (4) KL-EMP, which is an ad-hoc improvement of EMP using more information on the behavior policies and whose implementation details are given in Appendix E.3.
Revised: [Figure] e. the ﬁrst 20%, 40%, 50%, 70%, 90% of training steps by which we learn the target policy. To form the multiple-behaviorpolicy samples, we generate same number of trajectories of equal length from each policy.

Original: Policy Evaluation Performance.
Revised: Policy Evaluation Performance.

Revised: We implement 4 methods: EMP, BCH, DualDice and WIS introduced in pre using balanced heuristics.

Revised: To compare with policy-aware BCH in the multiplebehavior-policy setting, we develop an extension BCH method called BCH (pooled), which utilize the exact values of all the behavior policies 4 .

Original: Figure  4 reports the log MSE of the 4 methods in different environments with different sample scales.
Revised: Figure 3 shows that, in all 4 environments, we found EMP consistently obtain smaller MSE than the other three methods.

Original: It shows that the proposed EMP outperforms both MIS and EMP (single).
Revised: Pooling is Beneﬁcial. To validate the variance reduction result Proposition 3, we compare EMP with its variation EMP (single) in which trajectories from different behaviors are not pooled. In detail,

Original: It is interesting to note that in EMP (single), actually more information on the behavior policies is learned than in EMP, but the learned stationary distribution corrections are mixed withnaively equal weights. So, the advantage of EMP over EMP (single) can be probably attributed to (1) the robustness due to less required information on behavior policies; (2) a near-optimal weighted average that is automatically learned by pooling together the  samples from different behavior policies.
Revised: EMP (single) applies EMP to each group of data generated by same behavior policy and return the mean average reward estimation. Figure 4 shows that EMP outperforms EMP (single), which is consistent with the theoretic variance reduction result. On the other hand, the optimality of EMP in Proposition 3 holds for ﬁxed w j , we now explore the possibility of further variance reduction via optimizing w j , i.e. , by re-weighting the proportion of samples generated by different behavior policies.

Original: On the other hand, we see that the performance of KL-EMP has greater improvement with the increase of sample size and eventually outperform EMP in cases of large sample size. This is because, KL-EMP replaces the ﬁxed sample proportion (i.e. w j as deﬁned in Section 4.2) with a KL divergence-based proportion, which is better estimated with more data sample.
Revised: In detail, we implement a variation of EMP in which the data samples are re-weighted according to the KL-divergence between its behavior policy and the target policy 5 . Figure 4 also shows that the performance of KL-EMP has greater improvement with the increase of sample size and could outperform EMP in cases of large sample size where the KL-divergence is better estimated.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we advocate the viewpoint of partial policy-awareness and the beneﬁts of estimating a  mixture policy for off-policy policy evaluation.
Revised: In this paper, we advocate the viewpoint of partial policy-awareness and the beneﬁts of estimating a “virtual” mixture policy for off-policy policy evaluation.

Original: The theoretical results of reduced variance coupled with experimental results illustrate the power of this class of methods.
Revised: The theoretical results of reduced variance coupled with experimental results illustrate the power of this class of methods.

Original: One key question that still remains is the following: if we are willing to estimate the individual behavior policies, can we further improve EMP by developing an efﬁcient algorithm to compute the optimal weights?
Revised: One key question that still remains is the following: if we are willing to estimate the individual behavior policies, can we further improve EMP by developing an efﬁcient algorithm to compute the optimal weights?

Original: One other question is a direct comparison of DualDice and EMP when the code of DualDice is released, this will allow us to see the props and cons of inductive bias offered by the Bellman equation used by DualDice and direct estimation of the mixture policy used by EMP.
Revised: The preliminary experiment results suggest that the answer would be yes, and we will leave this for future study.

Original: A K ERNEL M ETHOD
Revised: A K ERNEL M ETHOD

Original: We use the reproducing kernel Hilbert space to solve the mini-max problem of BCH (Liu et al.
Revised: We use the reproducing kernel Hilbert space to solve the mini-max problem of BCH (Liu et al.

Original: The key property of RKHS we leveraged is called reproducing property .
Revised: The key property of RKHS we leveraged is called reproducing property .

Original: The reproducing property claims, for any function f ∈ H ( H is a RKHS), the evaluation of f at point x equals its inner product with another function in RKHS:
Revised: The reproducing property claims, for any function f ∈ H ( H is a RKHS), the evaluation of f at point x equals its inner product with another function in RKHS:

Original: f ( s ) =
Revised: f ( s ) =

Original: (cid:104) f, k ( s, · ) (cid:105) H .
Revised: (cid:104) f, k ( s, · ) (cid:105) H .

Original: Given the objective function of BCH L ( w, f ) =
Revised: Given the objective function of BCH L ( w, f ) =

Original: E ( s,a,s [Equation] ) ∼ d π 0 [( ω ( s ) [Equation]
Revised: E ( s,a,s (cid:48) ) ∼ d π 0 [( ω ( s ) π ( a | s π 0 ( a | s ) − ω )

Original: ( a | s ) − ω ) f ( s (cid:48) )] .
Revised: f ( s (cid:48) )] .

Original: We use the reproducing property to obtain the closed form representation of max f ∈F L ( w, f ) 2 , which is shown as follows:
Revised: We use the reproducing property to obtain the closed form representation of max f ∈F L ( w, f ) 2 , which is shown as follows:

Original: [Equation]
Revised: [Equation]

Original: This equation has been proved in BCH Liu et al.
Revised: This equation has been proved in BCH Liu et al.

Original: B P ROOF OF T HEOREMB. A SSUMPTIONS
Revised: B P ROOF OF T HEOREMB. A SSUMPTIONS

Original: In this appendix, we provide the mathematical details and proof of Theorem 1.
Revised: In this appendix, we provide the mathematical details and proof of Theorem 1.

Original: We ﬁrst introduce some notations and assumptions.
Revised: We ﬁrst introduce some notations and assumptions.

Original: We assume the behavior policy π 0 ( a | s ) belongs to a class of policies
Revised: We assume the behavior policy π 0 ( a | s ) belongs to a class of policies

Original: Π = { π ( θ ; a, s ) : θ ∈ E θ } , where F θ is the parameter space, i.e. there exists θ 0 ∈ E 1 such thatπ 0 ( a | s )
Revised: Π = { π ( θ ; a, s ) : θ ∈ E θ } , where E θ is the parameter space, i.e. there exists θ 0 ∈ E θ such that

Original:  =
Revised: π 0 ( a | s ) =

Original: π ( θ 0 ; a, s ) .
Revised: π ( θ 0 ; a, s ) .

Original: The estimated behavior policy ˆ π 0  =
Revised: The estimated behavior policy ˆ π 0 ( a | s ) =

Original: π ˆ θ  is obtained via maximum likelihood method, i.e.
Revised: π (ˆ θ ; a, s ) is obtained via maximum likelihood method, i.e.

Revised: ˆ θ = arg

Original: [Equation]
Revised: [Equation]

Original: We assume central limit theorem holds for ˆ θ .
Revised: We assume central limit theorem holds for ˆ θ :

Revised: Assumption 1.

Revised: (CLT of MLE)

Original:  Recall that we have assumed in Section 3.2 that the true stationary distribution correction ω ( s ) =
Revised: [Equation] Recall that we have assumed in Section 3.2 that the true stationary distribution correction ω ( s ) =

Original: ω ( η 0 ; s ) .
Revised: ω ( η 0 ; s ) .

Original: Using the kernel method introduced in Appendix A, our estimation ˆ ω ( s )=
Revised: Using the kernel method introduced in Appendix A, we estimate ˆ ω ( s )

Original:  ω (ˆ η ; s ) is obtained via
Revised: = ω (ˆ η ; s ) by [Equation] i

Original: [Equation]
Revised: [Equation]

Original: , with x i =
Revised: with x i =

Original: ( s i , a i , s (cid:48) i ) and
Revised: ( s i , a i , s (cid:48) i ) and

Revised: G

Revised: [Equation]

Revised: η,

Original: [Equation]
Revised: [Equation]

Revised: θ

Revised: [Equation]

Revised: ( x i , x

Revised: [Equation]

Revised: The BCH method estimate ˜ θ by

Revised: [Equation]

Original: Assumption 1.
Revised: Assumption 2.

Original: We assume the following regularity conditions on G :
Revised: We assume the following regularity conditions on G :

Original: G is second order differentiable.
Revised: G is second order differentiable.

Original: E [ ∂ η ∂ θ G ( η 0 , θ 0 ; x i , x j )] is ﬁnite.
Revised: E [ ∂ η ∂ θ G ( η 0 , θ 0 ; x i , x j )] is ﬁnite.

Original: E [ ∂ 2 η G ( η 0 , θ 0 ; x i , x j )] is ﬁnite and non-zero.
Revised: E [ ∂ 2 η G ( η 0 , θ 0 ; x i , x j )] is ﬁnite and non-zero.

Original: E [ ∂ η G ( η 0 , θ 0 ; x i , x j ) 2 ] is ﬁnite.
Revised: E [ ∂ η G ( η 0 , θ 0 ; x i , x j ) 2 ] is ﬁnite.

Original: Here we simply write E x i ∼ d π 0 ,x j ∼ d π 0 as E for the simplicity of notation.
Revised: Here we simply write E x i ∼ d π 0 ,x j ∼ d π 0 as E for the simplicity of notation.

Original: B. P ROOF OF T HEOREM
Revised: Theorem 1. Under Assumptions 1 and 2, we have, asymptotically [Equation]

Original: Proof.
Revised: Proof.

Original: Following the kernel method,
Revised: Following the kernel method,

Original: [Equation]
Revised: [Equation]

Original: Then, (cid:80) 1 ≤ i,j ≤ N ∂ η G(ˆ η, ˆ θ ; ( x i , x j )) = 0 , we have
Revised: G

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: η, ˆ θ ; ( x i , x j )) with x i = ( s i , a i , s (cid:48) i ) and s (cid:48) i (cid:44) s i

Original: [Equation]
Revised: +1 .

Original:  we have
Revised: Then, (cid:80) 1 ≤ i,j ≤ N ∂ η G (ˆ η, ˆ θ ; ( x i , x j )) = 0 , we have

Original: [Equation]
Revised: [Equation]

Revised: By deﬁnition, ˜ η is the solution to the optimization problem max η (cid:80) 1 ≤ i,j ≤ N G ( η, θ 0 ; ( x i , x j )) .

Original: and 0 =
Revised: Therefore, 0 =

Original: [Equation]
Revised: [Equation]

Original: Deﬁne S( θ ; ( x i , x j ))
Revised: Deﬁne S ( θ ; ( x i , x j ))

Original: =
Revised: =

Original: log( π ( θ ; s i , a i ))
Revised: log( π ( θ ; s i , a i ))

Original: + log( π ( θ ; s j , a j )) .
Revised: + log( π ( θ ; s j , a j )) .

Original: According to our estimation method,
Revised: Similarly, ˆ θ is the optimal solution to arg max θ (cid:80) 1 ≤ i,j ≤ N S ( θ ; ( x i , x j )) . Therefore, 0 =

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: 0 =
Revised: ) E (cid:2) ∂

Original: [Equation]
Revised: [Equation]

Revised: S ( θ 0 ; ( x 1 , x 2 )) (cid:3) + o p (1) .

Original: Following the proof of Theorem 1 of (Henmin et al.2007)
Revised: Following the proof of Theorem 1 of (Henmin et al.

Original: [Equation]

Original:  it sufﬁces to prove that
Revised: 2007), it sufﬁces to prove that

Original: [Equation]
Revised: [Equation]

Original: One can check
Revised: One can check

Original: [Equation]
Revised: [Equation]

Original: The last equality holds because ( x 1 , x 2 ) ∼ d π 0 ( s 1 ) π ( x [Equation] η 0 ) ⊗d π [Equation]
Revised: The last equality holds because ( x 1 , x 2 ) ∼ d π 0 ( s 1 ) π ( x 1 ; η 0 ) ⊗

Original: s

Original: [Equation] π [Equation] x 2 ; η 0 ) .
Revised: d π 0 ( s 2 ) π ( x 2 ; η 0 ) .

Original: Besides, we have
Revised: Besides, we have

Original: [Equation]
Revised: [Equation]

Original: Then [Equation] we deriveE [
Revised: Then, we derive

Original: [Equation]

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: Here, we deﬁne
Revised: we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Therefore E [ H ( η 0 , θ 0 ; ( x 1 , x 2 ))] = 0 .
Revised: Therefore E [ H ( η [Equation] θ 0 ; ( x 1 , x 2 ))] = 0 .

Original: So we obtain (10).
Revised: So we obtain (8).

Original: B. P ROOF OF C OROLLARY
Revised: B. P ROOF OF C OROLLARY

Revised: Corollary 1.

Revised: Let N be the number of ( s, a, s (cid:48) ) tuples in the data.

Revised: Under Assumptions 1 and 2,

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the prove of Theorem
Revised: In the prove of Theorem

Original: [Equation]
Revised: [Equation]

Original: we see that
Revised: we see that

Original: [Equation]
Revised: [Equation]

Original: with K 1 = E (cid:2) ∂ 2 η G ( η 0 , θ 0 ; ( x 1 , x 2 )) (cid:3) and K 2 = E [ ∂ θ ∂ η G ( η 0 , θ 0 ; ( x 1 , x 2 ))] .
Revised: with K 1 = E (cid:2) ∂ 2 η G ( η 0 , θ 0 ; ( x 1 , x 2 )) (cid:3) and K 2 = E [ ∂ θ ∂ η G ( η 0 , θ 0 ; ( x 1 , x 2 ))] .

Original: Therefore,
Revised: Therefore,

Original: E [
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: ˆ

Original: [Equation] − η 0 ) 2 ] [Equation] We assume that CLT holds for the maximum likelihood estimator ˆ θ , i.e. E [(ˆ θ − θ 0 ) 2 ]
Revised: We assume that CLT holds for the maximum likelihood estimator ˆ θ , i.e. E [(ˆ θ − θ 0 ) 2 ]

Original: = O (1 /N ) .
Revised: = O (1 /N ) .

Original: Besides, as E [ ∂ η G ( η 0 , θ 0 ; ( x i , x j ))]
Revised: Besides, as E [ ∂ η G ( η 0 , θ 0 ; ( x i , x j ))]

Original: = 0 , under Condition 4 of Assumption 1, , we can apply the central limit theorem (for stationary Markov chain) and have
Revised: = 0 , under Condition 4 of Assumption 2, , we can apply the central limit theorem (for stationary Markov chain) and have

Original: E

Original: [Equation]
Revised: [Equation]

Original: Therefore,

Original: E [(ˆ η − η 0 ) 2 ] =

Original: O (1 /N ) .

Original: C P ROOFS OF P ROPOSITIONS FOR EMP
Revised: C P ROOFS OF T HEORETIC R ESULTS FOR EMP

Original: Proof of Proposition 1.
Revised: Propostion 1.

Original: [Equation]
Revised: The state-action-next-station tuples generated by π M follows the distribution d M ( s, a, s (cid:48) ) .

Original: Proof of Proposition 2. If ω = d π /d 0 , based on the stationary equation
Revised: As a consequence, d M ( s ) is the stationary state distribution generated by π M .

Original: [Equation]
Revised: Proof.

Original:  any 
Revised: It sufﬁces to check that for any s (cid:48) ,

Original: [Equation]
Revised: [Equation]

Original:  we have
Revised: For each behavior policy π j , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore, (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: Note that the left hand side is simple d M ( s (cid:48) ) .

Original: On the opposite way
Revised: In the right hand side, (cid:88)

Original: [Equation]
Revised: [Equation]

Original: if
Revised: and then we immediately obtain (9).

Original: [Equation]
Revised: Propostion 2.

Original: for all function f :
Revised: The average award satisﬁes

Original: [Equation]
Revised: [Equation]

Original: → R , we should have
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Therefore, d 0 ω satisfy the stationary equation and must equal to d π (up to a constant).

Original: Proof of Proposition 3.
Revised: Theorem 2.

Original: The proof follows immediately from that of Theorem 1.
Revised: Under the same conditions of Theorem 1, if ω (˜ η ; s ) and ω (ˆ η ; s ) are the stationary distribution correction learned from (6) and from the same min-max problem but with exact value of π M , then, asymptotically [Equation] As a result, E [(ˆ η − η 0 ) 2 ] = O (cid:0) 1 N (cid:1) . Proof.

Revised: The proof follows immediately from that of Theorem 1.

Original: In particular, assume π 0 ∈ { π ( θ ; a, s ) : θ ∈ E θ } and the estimated ˆ π 0 =
Revised: In particular, assume π M ∈ { π ( θ ; a, s ) : θ ∈ E θ } and the estimated ˆ π M =

Original: π
Revised: π

Original: (ˆ θ ; · ) is obtained via
Revised: ( ˆ θ ; · ) is obtained via

Original: [Equation]
Revised: [Equation]

Original: The rest part of the proof follows the same argument in the proof of Theorem 1.
Revised: The rest part of the proof follows the same argument in the proof of Theorem 1.

Original: D S TATE -A CTION D ISTRIBUTION L EARNING

Original: Here we propose a behavior-agnostic approach that evaluates the target policy through learning occupation distribution correction instead of stationary distribution correction.

Original: Recall that the occupation distribution the stationary distribution of the state-action pair d π ( s ) π ( a | s ) .

Original: We deﬁne occupation distribution correction as d π ( s ) π ( a | s ) /

Original: ( d π 0 ( s ) π 0 ( a | s )) .

Original: Since π ( a | s ) is known, we denote u ( a, s ) =

Original: d π ( s ) /

Original: ( d π 0 ( s ) π 0 ( a | s )) and formulate a min-max problem to learn u .

Original: Following this notation, Equation (1) can be written as:

Original: [Equation]

Original: For any test function f ( s (cid:48) , a (cid:48) ) :

Original: S × R → R and any probability density function ω ( a (cid:48) )

Original: with respect to a (cid:48) ∈ A , the equation above implies that:

Original: [Equation]

Original: which is equivalent to,

Original: [Equation]

Original: This suggest the following mini-max problem can be used to estimate u ( s, a )

Original: [Equation]

Original: where L ( u, f ) := E ( s,a,s (cid:48) ) ∼ d π 0 (cid:2) u ( s,

Original: [Equation]

Original: ) ω

Original: [Equation]

Original: a )

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: We simplify this mini-max problem into a minimization problem using the kernel method.

Original: Theorem 2 gives the closed form representation of D ( u ) when F is a unit ball in a RKHS with kernel k .

Original: Theorem 2.

Original: Assume H is a RKHS of functions

Original: f

Original: ( s, a ) with a positive deﬁnite kernel k (( s, a ) , ( s (cid:48) , a (cid:48) )) , and deﬁne F := { f ∈ H :

Original: || f || H ≤ 1 } to be the unit ball of H .

Original: We have max f ∈F L ( u, f ) 2 = =

Original: E ( s,a,s (cid:48) ) ∼ d π 0 ,a (cid:48) ∼ ω, (¯ s, ¯ a, ¯ s (cid:48) ) ∼ d π 0 , ¯ a (cid:48) ∼ ω

Original: [ u ( s, a ) u (¯ s,

Original: ¯ a ) ω

Original: ( a ) ω (¯ a ) k

Original: (( s, a ) , (¯ s, ¯ a ))

Original: + u ( s, a ) u (¯ s, ¯ a ) π ( a | s ) π

Original: (¯ a | ¯ s ) k

Original: (( s (cid:48) , a (cid:48) ) , (¯ s (cid:48) , ¯ a (cid:48) ))

Original: − u ( s, a ) u (¯ s, ¯ a ) ω ( a ) π (¯ a | ¯ s ) k

Original: (( s, a ) , (¯ s (cid:48) , ¯ a (cid:48) ))

Original: − u ( s, a ) u (¯ s, ¯ a ) ω (¯ a ) π ( a | s ) k

Original: ((¯ s, ¯ a ) , ( s (cid:48) , a (cid:48) ))] .

Original: E E XPERIMENTAL D ETAILS E. E NVIRONMENT D ESCRIPTION
Revised: D E XPERIMENTAL D ETAILS D. E NVIRONMENT D ESCRIPTION

Original: Taxi Taxi Dietterich & G (2000) is a 5 × 5 grid world simulating a taxi movement.
Revised: Taxi (Dietterich & G, 2000) is a 5 × 5 grid world simulating a taxi movement.

Original: Six actions are contained in Taxi: moves North, East, South, West, pick up and drop off a passenger.
Revised: Six actions are contained in Taxi: moves North, East, South, West, pick up and drop off a passenger.

Original: A reward ofis received when it picks up a passenger or drops her/he off at the right place, and a reward of -1 for each time step.
Revised: A reward ofis received when it picks up a passenger or drops her/he off at the right place, and a reward of -1 for each time step.

Original: The passengers are allow to randomly appear and disappear at every corner of the map at each time step.
Revised: The passengers are allow to randomly appear and disappear at every corner of the map at each time step.

Original: The 5 × 5 grid size yields 2000 states in total (25 × 24 × 5, corresponding totaxi locations, 24 passenger appearance status and 5 taxi status (empty or with one of 4 destinations)).
Revised: The 5 × 5 grid size yields 2000 states in total (25 × 24 × 5, corresponding totaxi locations, 24 passenger appearance status and 5 taxi status (empty or with one of 4 destinations)).

Original: Gridworld Gridworld Thomas & Brunskill (2016) is a 4 × 4 grid world which including one reward state, one terminate state and one ﬁre state and thirteen normal state.
Revised: Gridworld (Thomas & Brunskill, 2016) is a 4 × 4 grid world which including one reward state, one terminate state and one ﬁre state and thirteen normal state.

Original: Four action can be taken in this environment: up, down, left and right.
Revised: Four action can be taken in this environment: up, down, left and right.

Original: A reward of -1 will be received while the agent in normal states, 1 reward is obtained in reward state, 100 reward is got in terminate state and -11 reward will got in ﬁre state.
Revised: A reward of -1 will be received while the agent in normal states, 1 reward is obtained in reward state, 100 reward is got in terminate state and -11 reward will got in ﬁre state.

Original: SinglePathThis environment has 5 states, 2 actions.
Revised: SinglePath has 5 states, 2 actions.

Original: The agent begins in state 0 and both actions either take the agent from state n to state n + 1 or cause the agent to remain in state n. If the agent arrives at a new state, it will receive a +1 reward, otherwise it will get a 1 reward.
Revised: The agent begins in state 0 and both actions either take the agent from state n to state n + 1 or cause the agent to remain in state n. If the agent arrives at a new state, it will receive a +1 reward, otherwise it will get a 1 reward.

Original: PendulumPendulum has a continuous state space of R 3 which describes the triangle of and a action space of [ − 2 , 2] .
Revised: Pendulum has a continuous state space of R 3 which describes the triangle of and a action space of [ − 2 , 2] .

Original: E. B EHAVIOR P OLICY E STIMATION
Revised: D. M IXTURE P OLICY E STIMATION

Original: EMP employs the maximum likelihood method as in Tirinzoni et al. (2019) to estimate the mixed policy ˆ π 0 as
Revised: For the discrete environments Taxi, Gridworld and SinglePath, the MLE estimate (5) coincide with count-frequency, and therefore, we directly estimate

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: For the continuous environment Pendulum, we use a neural network to model the policy. In detail, we train a two-layer MLP neural network to estimate the policy.

Original: As for discrete control tasks, the optimal ˆ π 0 coincides with the count-frequency.
Revised: The size of the two hidden layers are both 32 with the learning rate 0.001 and tanh activation function.

Original: E. C OMPUTATION OF KL-W EIGHTS
Revised: We use MEL (5) and Adam optimizer to train the neural network with batch size 128. D. KL-EMP In EMP algorithm, the proportion of samples from policy π j in the data buffer, w j , is ﬁxed.

Original: In an ad-hoc way, we optimize the weights w j according to the KL-divergence between the behavior policies and the target policy, which can be estimated directly from the data.
Revised: In an ad-hoc way, we optimize the weights w KLj according to the KL-divergence between the behavior policy π j and the target policy. Then, we will generate a new data buffer as follows.

Original: For ﬁnite-state space, we propose to choose
Revised: First, we sample j ∈ { 1 , 2 , ..., m } with probability w KL j . Then, given j , we sample uniformly from the subgroup of data generated by j . Note that, to implement KL-EMP, one do not need to know the exact value of π j , but one need to know which data are generated from which policy. In the numerical experiment, we use the following formula to compute w KL for ﬁnite state space:

Original: [Equation]
Revised: [Equation]

Original: To implement this method for inﬁnite- or continuous-state space in the numerical experiments, we replace the set of all possible states S in (12) with the set of all states that has been visited in the data buffer. 
Revised: To implement this method for inﬁnite or continuous state space, we replace the set of all possible states S in (10) with the set of all states that has been visited in the data buffer. Besides, the behavior policy π k is unknown, to estimate the KL-divergence D KL ( π ( ·| s ) |

Revised: π k ( ·| s )) , we use a neural network to learn π k from the subgroup of data that are generated from policy π k .

Original: The numerical results show that using the KL weights { w KLj } could achieve smaller MSE compared to using { w j } as given by the data sample.
Revised: The numerical results show that using the KL weights { w KLj } could achieve smaller MSE compared to using { w j } as given by the data sample.

Original: We believe this approach deserves more careful analysis in future research studies.
Revised: We believe this approach deserves more careful analysis in future research studies.

Original: [Figure]
Revised: [Figure]

Original: E. A DDITIONAL E XPERIMENT R ESULTS
Revised: D. A DDITIONAL E XPERIMENT R ESULTS

Original: BCH, EMP  and KL-EMP  have both policy-aware and partially policy-agnostic versions in multiple behavior policies.
Revised: Note that EMP, EMP (single) and KL-EMP all have their policy-aware analogues.

Original: In  policy-aware BCH we ﬁrst apply BCH for each subgroup of samples generated by each behavior policy followed by output their average value. As for partially policy-agnostic BCH, it is equal to EMP (single).
Revised: In order to test the variance reduction effect of policy estimation, we implement the policy-aware version for each EMP-type algorithm and compare their performances. The policy-aware version of EMP (single) is naive BCH, in which we ﬁrst apply BCH to each behavior policy and then return the estimation average.

Original: The policy-aware version of EMP is named as BCH (pooled).
Revised: The policy-aware version of EMP is named as BCH (pooled).

Original: In BCH (pooled), the corresponding min-max problem formation is
Revised: In BCH (pooled), the corresponding min-max problem formation is

Original: [Equation]
Revised: [Equation]

Original: They both pool the data from different behavior policies together and the main difference is that BCH (pooled) uses the exact behavior policies.
Revised: They both pool the data from different behavior policies together and the main difference is that BCH (pooled) uses the exact behavior policies.

Original: The policy-aware version of KL-EMP is called BCH (KL-polled).
Revised: The policy-aware version of KL-EMP is called BCH (KL-polled).

Original: The main difference between
Revised: The main difference between

Original: BCH (KL-polled) and BCH (polled) is that BCH (KL-polled) utilizes KL-divergence to calculate the weights.
Revised: BCH (KL-polled) and BCH (polled) is that BCH (KL-polled) utilizes KL-divergence to optimize the weights.

Original: The resultsof the two versions are shown in Figure 5.
Revised: Comparison results are shown in Figure 5.

Original: We observe that the partially policy-agnostic version OPPE consistently outperform the policy-aware version OPPE.
Revised: We observe that the partially policy-agnostic methods consistently outperform their policy-aware analogues.