Original: G EN DICE: G ENERALIZED O FFLINE E STIMATION OF S TATIONARY V ALUES
Revised: G EN DICE: G ENERALIZED O FFLINE E STIMATION OF S TATIONARY V ALUES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities deﬁned by the stationary distribution of a Markov chain.
Revised: An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities deﬁned by the stationary distribution of a Markov chain.

Original: In many real-world applications, access to the underlying transition operator is limited to a ﬁxed set of data that has already been collected, without additional interaction with the environment being available.
Revised: In many real-world applications, access to the underlying transition operator is limited to a ﬁxed set of data that has already been collected, without additional interaction with the environment being available.

Original: We show that consistent estimation remains possible in this scenario, and that effective estimation can still be achieved in important applications.
Revised: We show that consistent estimation remains possible in this challenging scenario and that effective estimation can still be achieved in important applications.

Original: Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization.
Revised: Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution and exploiting constraint reformulations based on variational divergence minimization.

Original: The resulting algorithm, GenDICE, is straightforward and effective.
Revised: The resulting algorithm, GenDICE, is straightforward and effective.

Original: We prove the consistencyof the method under general conditions, provide a detailed error analysis, and demonstrate strong empirical performance on benchmark tasks, including off-line PageRank and off-policy policy evaluation.
Revised: We prove its consistency under general conditions, provide an error analysis, and demonstrate strong empirical performance on benchmark problems, including off-line PageRank and off-policy policy evaluation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Estimation of quantities deﬁned by the stationary distribution of a Markov chain lies at the heart of many scientiﬁc and engineering problems.
Revised: Estimation of quantities deﬁned by the stationary distribution of a Markov chain lies at the heart of many scientiﬁc and engineering problems.

Original: Famously, the steady-state distribution of a random walk on the World Wide Web provides the foundation of the PageRank algorithm (Langville & Meyer, 2004).
Revised: Famously, the steady-state distribution of a random walk on the World Wide Web provides the foundation of the PageRank algorithm (Langville & Meyer, 2004).

Original: In many areas of machine learning, Markov chain Monte Carlo (MCMC) methods are used to conduct approximate Bayesian inference by considering Markov chains whose equilibrium distribution is a desired posterior (Andrieu et al., 2002).
Revised: In many areas of machine learning, Markov chain Monte Carlo (MCMC) methods are used to conduct approximate Bayesian inference by considering Markov chains whose equilibrium distribution is a desired posterior (Andrieu et al., 2002).

Original: An example from engineering is queueing theory, where the queue lengths and waiting time under the limiting distribution have been extensively studied (Gross et al., 2018).
Revised: An example from engineering is queueing theory, where the queue lengths and waiting time under the limiting distribution have been extensively studied (Gross et al., 2018).

Original: As we will also see below, stationary distribution quantities are of fundamental importance in reinforcement learning (RL) (e.g., Tsitsiklis & Van Roy, 1997).
Revised: As we will also see below, stationary distribution quantities are of fundamental importance in reinforcement learning (RL) (e.g., Tsitsiklis & Van Roy, 1997).

Original: Classical algorithms for estimating stationary distribution quantities rely on the ability to sample next states from the current state by directly interacting with the environment (as in on-line RL or
Revised: Classical algorithms for estimating stationary distribution quantities rely on the ability to sample next states from the current state by directly interacting with the environment (as in on-line RL or

Original: MCMC), or even require the transition probability distribution to be given explicitly (as in PageRank).
Revised: MCMC), or even require the transition probability distribution to be given explicitly (as in PageRank).

Original: Unfortunately, these classical approaches are inapplicable when direct access to the environment is not available, which is often the case in practice.
Revised: Unfortunately, these classical approaches are inapplicable when direct access to the environment is not available, which is often the case in practice.

Original: There are many practical scenarios where a collection of sampled trajectories is available, having been collected off-line by an external mechanism that chose states and recorded the subsequent next states.
Revised: There are many practical scenarios where a collection of sampled trajectories is available, having been collected off-line by an external mechanism that chose states and recorded the subsequent next states.

Original: Given such data, we still wish to estimate a stationary quantity.
Revised: Given such data, we still wish to estimate a stationary quantity.

Original: One important example is off-policy policy evaluation in RL, where we wish to estimate the value of a policy different from that used to collect experience.
Revised: One important example is off-policy policy evaluation in RL, where we wish to estimate the value of a policy different from that used to collect experience.

Original: Another example is off-line PageRank (OPR), where we seek to estimate the relative importance of webpages given a sample of the web graph.
Revised: Another example is off-line PageRank (OPR), where we seek to estimate the relative importance of webpages given a sample of the web graph.

Original: Motivated by the importance of these off-line scenarios, and by the inapplicability of classical methods, we study the problem of off-line estimation of stationary values via a stationary distribution corrector .
Revised: Motivated by the importance of these off-line scenarios, and by the inapplicability of classical methods, we study the problem of off-line estimation of stationary values via a stationary distribution corrector .

Original: Instead of having access to the transition probabilities or a next-state sampler, we assume only access to a ﬁxed sample of state transitions, where states have been sampled from an unknown distribution and next-states are sampled according to the Markov chain’s transition operator.
Revised: Instead of having access to the transition probabilities or a next-state sampler, we assume only access to a ﬁxed sample of state transitions, where states have been sampled from an unknown distribution and next-states are sampled according to the Markov chain’s transition operator.

Original: This off-line setting is distinct from that considered by most MCMC or on-line RL methods, where it is assumed that new observations can be continually sampled by demand from the environment.

Original: The off-line setting is indeed more challenging than its more traditional on-line counterpart, given that one must infer an asymptotic quantity from ﬁnite data.
Revised: The off-line setting is indeed more challenging than its more traditional on-line counterpart, given that one must infer an asymptotic quantity from ﬁnite data.

Original: Nevertheless, we develop techniques that still allow consistent estimation under general conditions, and provide effective estimates in practice.
Revised: Nevertheless, we develop techniques that still allow consistent estimation under general conditions, and provide effective estimates in practice.

Original: The main contributions of this work are:
Revised: The main contributions of this work are:

Original: • We formalize the problem of off-line estimation of stationary quantities, which captures a wide range of practical applications.
Revised: • We formalize the problem of off-line estimation of stationary quantities, which captures a wide range of practical applications.

Original: • We propose a novel stationary distribution estimator, GenDICE, for this task.
Revised: • We propose a novel stationary distribution estimator, GenDICE, for this task.

Original: The resulting algorithm is based on a new dual embedding formulation for divergence minimization, with a carefully designed mechanism that explicitly eliminates degenerate solutions.
Revised: The resulting algorithm is based on a new dual embedding formulation for divergence minimization, with a carefully designed mechanism that explicitly eliminates degenerate solutions.

Original: • We theoretically establish consistency and other statistical properties of GenDICE, and empirically demonstrate that it achieves signiﬁcant improvements on several behavior-agnostic offpolicy evaluation benchmarks and an off-line version of PageRank.
Revised: • We theoretically establish consistency and other statistical properties of GenDICE, and empirically demonstrate that it achieves signiﬁcant improvements on several behavior-agnostic offpolicy evaluation benchmarks and an off-line version of PageRank.

Original: The methods we develop in this paper fundamentally extend recent work in off-policy policy evaluation (Liu et al., 2018; Nachum et al., 2019) by introducing a new formulation that leads to a more general, and as we will show, more effective estimation method.
Revised: The methods we develop in this paper fundamentally extend recent work in off-policy policy evaluation (Liu et al., 2018; Nachum et al., 2019) by introducing a new formulation that leads to a more general, and as we will show, more effective estimation method.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We ﬁrst introduce off-line PageRank (OPR) and off-policy policy evaluation (OPE) as two motivating domains, where the goal is to estimate stationary quantities given only off-line access to a set of sampled transitions from an environment.
Revised: We ﬁrst introduce off-line PageRank (OPR) and off-policy policy evaluation (OPE) as two motivating domains, where the goal is to estimate stationary quantities given only off-line access to a set of sampled transitions from an environment.

Original: Off-line PageRank (OPR)
Revised: Off-line PageRank (OPR)

Original: The celebrated PageRank algorithm (Page et al., 1999) deﬁnes the ranking of a web page in terms of its asymptotic visitation probability under a random walk on the (augmented) directed graph speciﬁed by the hyperlinks.
Revised: The celebrated PageRank algorithm (Page et al., 1999) deﬁnes the ranking of a web page in terms of its asymptotic visitation probability under a random walk on the (augmented) directed graph speciﬁed by the hyperlinks.

Original: If we denote the World Wide Web by a directed graph G = ( V, E ) with vertices (web pages) v ∈ V and edges (hyperlinks) ( v, u ) ∈ E ,
Revised: If we denote the World Wide Web by a directed graph G = ( V, E ) with vertices (web pages) v ∈ V and edges (hyperlinks) ( v, u ) ∈ E ,

Original: PageRank considers the random walk deﬁned by the Markov transition operator v → u :
Revised: PageRank considers the random walk deﬁned by the Markov transition operator v → u :

Original: [Equation]
Revised: [Equation]

Original: where | v | denotes the out-degree of vertex v and η ∈ [0 , 1) is a probability of “teleporting” to any page uniformly.
Revised: where | v | denotes the out-degree of vertex v and η ∈ [0 , 1) is a probability of “teleporting” to any page uniformly.

Original: Deﬁne d t ( v ) :
Revised: Deﬁne d t ( v ) :

Original: = P ( s t = v | s 0 ∼ µ 0 , ∀ i < t, s i +1 ∼ P ( ·| s i )) , where µ 0 is the initial distribution over vertices, then the original PageRank algorithm explicitly iterates for the limit
Revised: = P ( s t = v | s 0 ∼ µ 0 , ∀ i < t, s i +1 ∼ P ( ·| s i )) , where µ 0 is the initial distribution over vertices, then the original PageRank algorithm explicitly iterates for the limit

Original: [Equation]
Revised: [Equation]

Original: The classical version of this problem is solved by tabular methods that simulate Equation 1.
Revised: The classical version of this problem is solved by tabular methods that simulate Equation 1.

Original: However, we are interested in a more scalable off-line version of the problem where the transition model is not explicitly given.
Revised: However, we are interested in a more scalable off-line version of the problem where the transition model is not explicitly given.

Original: Instead, consider estimating the rank of a particular web page v (cid:48) from a large web graph, given only a sample D
Revised: Instead, consider estimating the rank of a particular web page v (cid:48) from a large web graph, given only a sample D

Original: = { ( v, u ) i } Ni =1 from a random walk on G as speciﬁed above.
Revised: = { ( v, u ) i } Ni =1 from a random walk on G as speciﬁed above.

Original: We would still like to estimate d ( v (cid:48) )
Revised: We would still like to estimate d ( v (cid:48) )

Original: based on this data.
Revised: based on this data.

Original: First, note that if one knew the distribution p by which any vertex v appeared in D , the target quantity could be re-expressed by a simple importance ratio d ( v (cid:48) )
Revised: First, note that if one knew the distribution p by which any vertex v appeared in D , the target quantity could be re-expressed by a simple importance ratio d ( v (cid:48) )

Original: =
Revised: =

Original: E v ∼ p (cid:104)
Revised: E v ∼ p (cid:104)

Original: d ( v ) p ( v ) 1 v = v (cid:48) (cid:105) .
Revised: d ( v ) p ( v ) 1 v = v (cid:48) (cid:105) .

Original: Therefore, if one had the correction ratio function τ ( v ) =
Revised: Therefore, if one had the correction ratio function τ ( v ) =

Original: d ( v ) p ( v ) , an estimate of d ( v (cid:48) ) can easily be recovered via d ( v (cid:48) )
Revised: d ( v ) p ( v ) , an estimate of d ( v (cid:48) ) can easily be recovered via d ( v (cid:48) )

Original: ≈ ˆ p ( v (cid:48) )
Revised: ≈ ˆ p ( v (cid:48) )

Original: τ ( v (cid:48) ) , where ˆ p ( v (cid:48) ) is the empirical probability of v (cid:48) estimated from D .
Revised: τ ( v (cid:48) ) , where ˆ p ( v (cid:48) ) is the empirical probability of v (cid:48) estimated from D .

Original: The main attack on the problem we investigate is to recover a good estimate of the ratio function τ .
Revised: The main attack on the problem we investigate is to recover a good estimate of the ratio function τ .

Original: Policy Evaluation(Preliminaries)
Revised: Policy Evaluation

Original: An important generalization of this stationary value estimation problem arises in RL in the form of policy evaluation.
Revised: An important generalization of this stationary value estimation problem arises in RL in the form of policy evaluation.

Original: Consider a Markov Decision Process (MDP)
Revised: Consider a Markov Decision Process (MDP)

Original: M = (cid:104) S, A, P , R, γ, µ 0 (cid:105) (Puterman, 2014), where S is a state space, A is an action space, P ( s (cid:48) | s, a ) denotes the transition dynamics, R is a reward function, γ ∈ (0 , 1] is a discounted factor, and µ 0 is the initial state distribution.
Revised: M = (cid:104) S, A, P , R, γ, µ 0 (cid:105) (Puterman, 2014), where S is a state space, A is an action space, P ( s (cid:48) | s, a ) denotes the transition dynamics, R is a reward function, γ ∈ (0 , 1] is a discounted factor, and µ 0 is the initial state distribution.

Original: Given a policy, which chooses actions in any state s according to the probability distribution π ( ·| s ) , a trajectory β =
Revised: Given a policy, which chooses actions in any state s according to the probability distribution π ( ·| s ) , a trajectory β =

Original: ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . . )
Revised: ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . . )

Original: is generated by ﬁrst sampling the initial state s 0 ∼ µ 0 , and then for t ≥ 0 , a t ∼ π ( ·| s t ) , r t ∼ R ( s t , a t ) , and s t +1 ∼ P ( ·| s t , a t ) .
Revised: is generated by ﬁrst sampling the initial state s 0 ∼ µ 0 , and then for t ≥ 0 , a t ∼ π ( ·| s t ) , r t ∼ R ( s t , a t ) , and s t +1 ∼ P ( ·| s t , a t ) .

Original: The value of a policy π is the expected per-step reward deﬁned as:
Revised: The value of a policy π is the expected per-step reward deﬁned as:

Original: Average: R ( π ) :=

Original: lim T →∞ 1 T +1 E (cid:104)(cid:80)

Original: [-Tt =
 r t (cid:105) , Discounted: R γ ( π ) := (1 − γ ) E [ (cid:80) ∞ t =0 γ t r t ] .-]
Revised: [-Tt =
 r t (cid:105) , Discounted: R γ ( π ) := (1 − γ ) E [ (cid:80) ∞ t =0 γ t r t ] .-][Equation]

Original: Here the expectation is taken with respect to the randomness in the state-action pair
Revised: In the above, the expectation is taken with respect to the randomness in the state-action pair

Original: P ( s (cid:48) | s, a ) π ( a (cid:48) | s (cid:48) ) and the reward R ( s t , a t ) .
Revised: P ( s (cid:48) | s, a ) π ( a (cid:48) | s (cid:48) ) and the reward R ( s t , a t ) .

Original: Without loss of generality, we assume the limit exists for the average case, and hence R ( π ) is ﬁnite.
Revised: Without loss of generality, we assume the limit exists for the average case, and hence R ( π ) is ﬁnite.

Original: Behavior-agnostic Off-Policy Evaluation (OPE)
Revised: Behavior-agnostic Off-Policy Evaluation (OPE)

Original: A natural version of policy evaluation that often arises in practice is to estimate R γ ( π ) or R ( π ) given a ﬁxed dataset D
Revised: An important setting of policy evaluation that often arises in practice is to estimate R γ ( π ) or R ( π ) given a ﬁxed dataset D

Original: = { ( s, a, r, s (cid:48) )
Revised: = { ( s, a, r, s (cid:48) )

Original: i } Ni =1 ∼
Revised: i } Ni =1 ∼

Original: P ( s (cid:48) | s, a ) p ( s, a ) , where p ( s, a ) is an unknown distribution induced by multiple unknown behavior policies.
Revised: P ( s (cid:48) | s, a ) p ( s, a ) , where p ( s, a ) is an unknown distribution induced by multiple unknown behavior policies.

Original: This problem is different from the classical form of OPE, where it is assumed that a known behavior policy π b is used to collect transitions; in the behavior-agnostic scenario we are considering here, standard importance sampling (IS) estimators (Precup et al., 2000) cannot be applied.
Revised: This problem is different from the classical form of OPE, where it is assumed that a known behavior policy π b is used to collect transitions. In the behavior-agnostic scenario, however, typical importance sampling (IS) estimators (e.g., Precup et al., 2000) do not apply. Even if one can assume

Revised: D consists of trajectories where the behavior policy can be estimated from data, it is known that that straightforward IS estimators suffer a variance exponential in the trajectory length, known as the “curse of horizon” (Jiang & Li, 2016; Liu et al., 2018).

Original: Let d πt ( s, a )
Revised: Let d πt ( s, a )

Original: =
Revised: =

Original: P ( s t = s, a t = a | s 0 ∼ µ 0 , ∀ i < t, a i ∼ π ( ·| s i ) ,
Revised: P ( s t = s, a t = a | s 0 ∼ µ 0 , ∀ i < t, a i ∼ π ( ·| s i ) ,

Original: s i +1 ∼ P ( ·| s i , a i )) .
Revised: s i +1 ∼ P ( ·| s i , a i )) .

Original: The stationary distribution can then be deﬁned as
Revised: The stationary distribution can then be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: From this deﬁnition note that R ( π ) and R γ ( π ) can be equivalently re-expressed as
Revised: With this deﬁnition, R ( π ) and R γ ( π ) can be equivalently re-expressed as

Original: [Equation]
Revised: [Equation]

Original: Here we see once again that if we had the correction ratio function τ ( s, a ) = µ πγ ( s,a ) p ( s,a ) a straightforward estimate of R γ ( π ) could be recovered via R γ ( π )
Revised: Here we see once again that if we had the correction ratio function τ ( s, a ) = µ πγ ( s,a ) p ( s,a ) a straightforward estimate of R γ ( π ) could be recovered via R γ ( π )

Original: ≈
Revised: ≈

Original: E ˆ p [ τ ( s, a ) R ( s, a )] , where ˆ p ( s, a ) is an empirical estimate of p ( s, a ) .
Revised: E ˆ p [ τ ( s, a ) R ( s, a )] , where ˆ p ( s, a ) is an empirical estimate of p ( s, a ) .

Original: In this way, the behavior-agnostic OPE problem can be reduced to estimating the correction ratio function τ , as above.
Revised: In this way, the behavior-agnostic OPE problem can be reduced to estimating the correction ratio function τ , as above.

Original: We note that Liu et al.
Revised: We note that Liu et al.

Original: (2018) and Nachum et al.
Revised: (2018) and Nachum et al.

Original: (2019) also exploit Equation 5 to reduce OPE to stationary distribution correction, but these prior works are distinct from the current proposal in different ways.
Revised: (2019) also exploit Equation 5 to reduce OPE to stationary distribution correction, but these prior works are distinct from the current proposal in different ways.

Original: First, the inverse propensity score (IPS) method of Liu et al.
Revised: First, the inverse propensity score (IPS) method of Liu et al.

Original: (2018) assumes the transitions are sampled from a single behavior policy, which must be known beforehand; hence that approach is not applicable in behavior-agnostic OPE setting.
Revised: (2018) assumes the transitions are sampled from a single behavior policy, which must be known beforehand; hence that approach is not applicable in behavior-agnostic OPE setting.

Original: Second, the recent DualDICE algorithm (Nachum et al., 2019) is also a behavior-agnostic OPE estimator, but its derivation relies on a change-of-variable trick that is only valid for γ < 1 .
Revised: Second, the recent DualDICE algorithm (Nachum et al., 2019) is also a behavior-agnostic OPE estimator, but its derivation relies on a change-of-variable trick that is only valid for γ < 1 .

Original: This previous formulation becomes unstable when γ → 1 , as shown in Section 6 and
Revised: This previous formulation becomes unstable when γ → 1 , as shown in Section 6 and

Original: Appendix E.
Revised: Appendix E.

Original: The behavior-agnostic OPE estimator we derive below in Section 3 is applicable both when γ = 1 and γ ∈ (0 , 1) .
Revised: The behavior-agnostic OPE estimator we derive below in Section 3 is applicable both when γ = 1 and γ ∈ (0 , 1) .

Original: This connection is why we name the new estimator GenDICE, for GENeralized stationary DIstribution Correction Estimation .
Revised: This connection is why we name the new estimator GenDICE, for GENeralized stationary DIstribution Correction Estimation .

Original:  G EN DICE
Revised:  G EN DICE

Original: As noted, there are important estimation problems in the Markov chain and MDP settings that can be recast as estimating a stationary distribution correction ratio.
Revised: As noted, there are important estimation problems in the Markov chain and MDP settings that can be recast as estimating a stationary distribution correction ratio.

Original: We ﬁrst outline the conditions that characterize the correction ratio function τ , upon which we construct the objective for the GenDICE estimator, and design efﬁcient algorithm for optimization.
Revised: We ﬁrst outline the conditions that characterize the correction ratio function τ , upon which we construct the objective for the GenDICE estimator, and design efﬁcient algorithm for optimization.

Original: We will develop our approach for the more general MDP setting, with the understanding that all the methods and results can be easily specialized to the Markov chain setting.
Revised: We will develop our approach for the more general MDP setting, with the understanding that all the methods and results can be easily specialized to the Markov chain setting.

Original: 3. E STIMATING S TATIONARY D ISTRIBUTION C ORRECTION
Revised: 3. E STIMATING S TATIONARY D ISTRIBUTION C ORRECTION

Original: The stationary distribution µ π γ deﬁned in Equation 4 can also be characterized via
Revised: The stationary distribution µ πγ deﬁned in Equation 4 can also be characterized via

Original: [Equation]
Revised: [Equation]

Original: At ﬁrst glance, this equation shares a superﬁcial similarity to the Bellman equation, but there is a fundamental difference.
Revised: At ﬁrst glance, this equation shares a superﬁcial similarity to the Bellman equation, but there is a fundamental difference.

Original: The Bellman operator recursively integrates out future ( s (cid:48) , a (cid:48) ) pairs to characterize a current pair ( s, a ) value, whereas the distribution operator T deﬁned in Equationoperates in the reverse temporal direction.
Revised: The Bellman operator recursively integrates out future ( s (cid:48) , a (cid:48) ) pairs to characterize a current pair ( s, a ) value, whereas the distribution operator T deﬁned in Equationoperates in the reverse temporal direction.

Original: When γ ≤ 1 , Equation 6 always has a ﬁxed-point solution.
Revised: When γ < 1 , Equation 6 always has a ﬁxed-point solution.

Original: For γ = 1 , in the discrete case, the ﬁxedpoint exists as long as T is ergodic; in the continuous case, the conditions for ﬁxed-point existence become more complicated (Meyn & Tweedie, 2012) and beyond the scope of this paper.
Revised: For γ = 1 , in the discrete case, the ﬁxedpoint exists as long as T is ergodic; in the continuous case, the conditions for ﬁxed-point existence become more complicated (Meyn & Tweedie, 2012) and beyond the scope of this paper.

Original: The development below is based on a divergence D and the following default assumption.
Revised: The development below is based on a divergence D and the following default assumption.

Original: Assumption 1 (Markov chain regularity)
Revised: Assumption 1 (Markov chain regularity)

Original: For the given target policy π , the resulting state-action transition operator T has a unique stationary distribution µ that satisﬁes D ( T ◦ µ (cid:107) µ ) = 0 .
Revised: For the given target policy π , the resulting state-action transition operator T has a unique stationary distribution µ that satisﬁes D ( T ◦ µ (cid:107) µ ) = 0 .

Original: In the behavior-agnostic setting we consider, one does not have direct access to P for element-wise evaluation or sampling, but instead is given a ﬁxed set of samples from P ( s (cid:48) | s, a ) p ( s, a ) with {+respect to some distribution p ( s, a ) over S × A . Deﬁne T pγ,µ 0 to be a mixture of µ 0 π and T p ; i.e. , let T pγ,µ 
 (( s (cid:48) , a (cid:48) ) , ( s, a )) :+}
Revised: In the behavior-agnostic setting we consider, one does not have direct access to P for element-wise evaluation or sampling, but instead is given a ﬁxed set of samples from P ( s (cid:48) | s, a ) p ( s, a ) with {+respect to some distribution p ( s, a ) over S × A . Deﬁne T pγ,µ 0 to be a mixture of µ 0 π and T p ; i.e. , let T pγ,µ 
 (( s (cid:48) , a (cid:48) ) , ( s, a )) :+}

Revised: =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Obviously, conditioning on ( s, a, s (cid:48) ) one could easily sample a (cid:48) ∼ π ( a (cid:48) | s (cid:48) ) to form ( s, a, s (cid:48) , a (cid:48) ) ∼
Revised: Obviously, conditioning on ( s, a, s (cid:48) ) one could easily sample a (cid:48) ∼ π ( a (cid:48) | s (cid:48) ) to form ( s, a, s (cid:48) , a (cid:48) ) ∼

Original: T p (( s (cid:48) , a (cid:48) ) , ( s, a )) ; similarly, a sample ( s (cid:48) , a (cid:48) ) ∼ µ 0 ( s (cid:48) )
Revised: T p (( s (cid:48) , a (cid:48) ) , ( s, a )) ; similarly, a sample ( s (cid:48) , a (cid:48) ) ∼ µ 0 ( s (cid:48) )

Original: π ( a (cid:48) | s (cid:48) ) could be formed from s (cid:48) .
Revised: π ( a (cid:48) | s (cid:48) ) could be formed from s (cid:48) .

Original: Mixing such samples with probability γ and 1 − γ respectively yields a sample ( s, a, s (cid:48) , a (cid:48) ) ∼
Revised: Mixing such samples with probability γ and 1 − γ respectively yields a sample ( s, a, s (cid:48) , a (cid:48) ) ∼

Original: T pγ,µ 

Revised: T pγ,µ

Original: (( s (cid:48) , a (cid:48) ) , ( s, a )) .
Revised: (( s (cid:48) , a (cid:48) ) , ( s, a )) .

Original: Based on these observations, the stationary condition for the ratio from Equation 6 can be re-expressed in terms of T pγ,µ 0 as
Revised: Based on these observations, the stationary condition for the ratio from Equation 6 can be re-expressed in terms of T pγ

Original: [Equation]
Revised: [Equation]

Original: where τ ∗ ( s, a ) :
Revised: where τ ∗ ( s, a ) :

Original: = µ ( s,a ) p ( s,a ) is the correction ratio function we seek to estimate.
Revised: = µ ( s,a ) p ( s,a ) is the correction ratio function we seek to estimate.

Original: One natural approach to estimating τ ∗ is to match the LHS and RHS of Equation 8 with respect to some divergence D ( ·(cid:107)· ) over the empirical samples.
Revised: One natural approach to estimating τ ∗ is to match the LHS and RHS of Equation 8 with respect to some divergence D ( ·(cid:107)· ) over the empirical samples.

Original: That is, we consider estimating τ ∗ by solving the optimization problem
Revised: That is, we consider estimating τ ∗ by solving the optimization problem

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Although this forms the basis of our approach, there are two severe issues with this naive formulation that ﬁrst need to be rectiﬁed: i) Degenerate solutions: When γ = 1 , the operator T pγ =1 ,µ 0 is invariant to constant rescaling: if τ ∗ =
Revised: Although this forms the basis of our approach, there are two severe issues with this naive formulation that ﬁrst need to be rectiﬁed: i) Degenerate solutions: When γ = 1 , the operator T pγ =1 ,µ 0 is invariant to constant rescaling: if τ ∗ =

Original: T pγ =1 ,µ 0 ◦ τ ∗ then cτ ∗ =
Revised: T pγ =1 ,µ 0 ◦ τ ∗ then cτ ∗ =

Original: T pγ =1 ,µ 0 ◦ ( cτ ∗ ) for any c ≥ 0 .
Revised: T pγ =1 ,µ 0 ◦ ( cτ ∗ ) for any c ≥ 0 .

Original: Therefore, simply minimizing the divergence D (cid:0) T pγ =1 ,µ 0 ◦ τ (cid:107) p · τ (cid:1) cannot provide a desirable estimate of τ ∗ .
Revised: Therefore, simply minimizing the divergence D (cid:0) T pγ =1 ,µ 0 ◦ τ (cid:107) p · τ (cid:1) cannot provide a desirable estimate of τ ∗ .

Original: In fact, in this case the trivial solution τ ∗ ( s, a ) = 0 cannot be eliminated.
Revised: In fact, in this case the trivial solution τ ∗ ( s, a ) = 0 cannot be eliminated.

Original: ii) Intractable objective: The divergence D (cid:0) T pγ,µ 0 ◦ τ (cid:107) p · τ (cid:1) involves the computation of T pγ,µ 0 ◦ τ , which in general involves an intractable integral.
Revised: ii) Intractable objective: The divergence D (cid:0) T pγ,µ 0 ◦ τ (cid:107) p · τ (cid:1) involves the computation of T pγ,µ 0 ◦ τ , which in general involves an intractable integral.

Original: Thus, evaluation of the exact objective is intractable, and neglects the assumption that we only have access to samples from T pγ,µ 0 and are not able to evaluate it at arbitrary points.
Revised: Thus, evaluation of the exact objective is intractable, and neglects the assumption that we only have access to samples from T pγ,µ 0 and are not able to evaluate it at arbitrary points.

Original: We address each of these two issues in a principled manner.
Revised: We address each of these two issues in a principled manner.

Original: 3. E LIMINATING DEGENERATE SOLUTIONS
Revised: 3. E LIMINATING DEGENERATE SOLUTIONS

Original: To avoid degenerate solutions when γ = 1 , we ensure that the solution is a proper density ratio; that is, the property τ ∈ Ξ := { τ ( · ) ≥ 0 , E p [ τ ] = 1 } must be true of any τ that is a ratio of some density to p .
Revised: To avoid degenerate solutions when γ = 1 , we ensure that the solution is a proper density ratio; that is, the property τ ∈ Ξ := { τ ( · ) ≥ 0 , E p [ τ ] = 1 } must be true of any τ that is a ratio of some density to p .

Original: This provides an additional constraint that we add to the optimization formulation
Revised: This provides an additional constraint that we add to the optimization formulation

Original: [Equation]
Revised: [Equation]

Original: With this additional constraint, it is obvious that the trivial solution τ ( s, a ) = 0 is eliminated as an infeasible point of Eqn (10), along with other degenerate solutions
Revised: With this additional constraint, it is obvious that the trivial solution τ ( s, a ) = 0 is eliminated as an infeasible point of Eqn (10), along with other degenerate solutions

Original: τ ( s, a ) =
Revised: τ ( s, a ) =

Original: cτ ∗ ( s, a ) with c (cid:54) = 1 .
Revised: cτ ∗ ( s, a ) with c (cid:54) = 1 .

Original: Unfortunately, exactly solving an optimization with expectation constraints is very complicated in general (Lan & Zhou, 2016), particularly given a nonlinear parameterization for τ .
Revised: Unfortunately, exactly solving an optimization with expectation constraints is very complicated in general (Lan & Zhou, 2016), particularly given a nonlinear parameterization for τ .

Original: The penalty method (Luenberger & Ye, 2015) provides a much simpler alternative, where a sequence of regularized problems are solved
Revised: The penalty method (Luenberger & Ye, 2015) provides a much simpler alternative, where a sequence of regularized problems are solved

Original: [Equation]
Revised: [Equation]

Original: 11) with λ increasing.
Revised: with λ increasing.

Original: The drawback of the penalty method is that it generally requires λ → ∞ to ensure the strict feasibility, which is still impractical, especially in stochastic gradient descent.
Revised: The drawback of the penalty method is that it generally requires λ → ∞ to ensure the strict feasibility, which is still impractical, especially in stochastic gradient descent.

Original: The inﬁnite λ may induce unbounded variance in the gradient estimator, and thus, divergence in optimization.
Revised: The inﬁnite λ may induce unbounded variance in the gradient estimator, and thus, divergence in optimization.

Original: However, by exploiting the special structure of the solution sets to Equation 11, we can show that, remarkably, it is unnecessary to increase λ .
Revised: However, by exploiting the special structure of the solution sets to Equation 11, we can show that, remarkably, it is unnecessary to increase λ .

Original: Theorem 1 For γ ∈ (0 , 1] and any λ > 0 , the solution to Equation 11 is given by τ ∗ ( s, a ) =
Revised: Theorem 1 For γ ∈ (0 , 1] and any λ > 0 , the solution to Equation 11 is given by τ ∗ ( s, a ) =

Original: u ( s,a ) p ( s,a ) .
Revised: u ( s,a ) p ( s,a ) .

Original: The detailed proof for Theorem 1 is given in Appendix A.1.
Revised: The detailed proof for Theorem 1 is given in Appendix A.1.

Original: By Theorem 1, we can estimate the desired correction ratio function τ ∗ by solving only one optimization with an arbitrary λ > 0 .
Revised: By Theorem 1, we can estimate the desired correction ratio function τ ∗ by solving only one optimization with an arbitrary λ > 0 .

Original: 3. E XPLOITING DUAL EMBEDDING
Revised: 3. E XPLOITING DUAL EMBEDDING

Original: The optimization in Equation 11 involves the integrals (cid:0) T pγ,µ 0 ◦ τ (cid:1) and E p [ τ ] inside nonlinear loss functions, hence appears difﬁcult to solve.
Revised: The optimization in Equation 11 involves the integrals (cid:0) T pγ,µ 0 ◦ τ (cid:1) and E p [ τ ] inside nonlinear loss functions, hence appears difﬁcult to solve.

Original: Moreover, obtaining unbiased gradients with a naive approach requires double sampling (Baird, 1995).
Revised: Moreover, obtaining unbiased gradients with a naive approach requires double sampling (Baird, 1995).

Original: Instead, we bypass both difﬁculties by applying a dual embedding technique (Dai et al., 2016; 2018).
Revised: Instead, we bypass both difﬁculties by applying a dual embedding technique (Dai et al., 2016; 2018).

Original: In particular, we assume the divergence D is in the form of an f -divergence (Nowozin et al., 2016)
Revised: In particular, we assume the divergence D is in the form of an f -divergence (Nowozin et al., 2016)

Original: [Equation]
Revised: [Equation]

Original: ds da where φ ( · ) : R + → R is a convex, lower-semicontinuous function with φ (1) = 0 .
Revised: ds da where φ ( · ) : R + → R is a convex, lower-semicontinuous function with φ (1) = 0 .

Original: Plugging this into J ( τ ) in Equation 11 we can easily check the convexity of the objective
Revised: Plugging this into J ( τ ) in Equation 11 we can easily check the convexity of the objective

Original: Theorem 2 For an f -divergence with valid φ deﬁning D φ , the objective J ( τ ) is convex w.r.t.
Revised: Theorem 2 For an f -divergence with valid φ deﬁning D φ , the objective J ( τ ) is convex w.r.t.

Original: τ .
Revised: τ .

Original: The detailed proof is provided in Appendix A.2.
Revised: The detailed proof is provided in Appendix A.2.

Original: Recall that a suitable convex function can be represented as φ ( x ) = max f x · f − φ ∗ ( f ) , where φ ∗ is the Fenchel conjugate of φ ( · ) .
Revised: Recall that a suitable convex function can be represented as φ ( x ) = max f x · f − φ ∗ ( f ) , where φ ∗ is the Fenchel conjugate of φ ( · ) .

Original: In particular, we have the representation 12 x 2 = max u ux − 12 u 2 , which allows us to re-express the objective as
Revised: In particular, we have the representation 12 x 2 = max u ux − 12 u 2 , which allows us to re-express the objective as

Original: [Equation]
Revised: [Equation]

Original: Applying the interchangeability principle (Shapiro et al., 2014; Dai et al., 2016), one can replace the inner max in the ﬁrst term over scalar f to maximize over a function f ( · , · ) : S ×
Revised: Applying the interchangeability principle (Shapiro et al., 2014; Dai et al., 2016), one can replace the inner max in the ﬁrst term over scalar f to maximize over a function f ( · , · ) : S ×

Original: A → R min τ
Revised: A → R [Equation]

Original: [Equation]
Revised: [Equation]

Original: This yields the main optimization formulation, which avoids the aforementioned difﬁculties and is well-suited for practical optimization as discussed in Section 3.4.
Revised: This yields the main optimization formulation, which avoids the aforementioned difﬁculties and is well-suited for practical optimization as discussed in Section 3.4.

Original: Remark (Other divergences):
Revised: Remark (Other divergences):

Original: In addition to f -divergence, the proposed estimator Equationis compatible with other divergences, such as the integral probability metrics (IPM) (M ¨ uller, 1997;
Revised: In addition to f -divergence, the proposed estimator Equationis compatible with other divergences, such as the integral probability metrics (IPM) (M¨uller, 1997;

Original: Sriperumbudur et al., 2009), while retaining consistency.
Revised: Sriperumbudur et al., 2009), while retaining consistency.

Original: Based on the deﬁnition of the IPM, these divergences directly lead to min - max optimizations similar to Equation 13 with the identity function as φ ∗ ( · ) and different feasible sets for the dual functions.
Revised: Based on the deﬁnition of the IPM, these divergences directly lead to min - max optimizations similar to Equation 13 with the identity function as φ ∗ ( · ) and different feasible sets for the dual functions.

Original: Speciﬁcally, maximum mean discrepancy (MMD) (Smola et al., 2006) requires (cid:107) f (cid:107) H k ≤ 1 where H k denotes the RKHS with kernel k ; the Dudley metric (Dudley, 2002) requires (cid:107) f (cid:107) BL ≤ 1 where (cid:107) f (cid:107)
Revised: Speciﬁcally, maximum mean discrepancy (MMD) (Smola et al., 2006) requires (cid:107) f (cid:107) H k ≤ 1 where H k denotes the RKHS with kernel k ; the Dudley metric (Dudley, 2002) requires (cid:107) f (cid:107) BL ≤ 1 where (cid:107) f (cid:107)

Original: BL := (cid:107) f (cid:107) ∞
Revised: BL := (cid:107) f (cid:107) ∞

Original: + (cid:107)∇ f (cid:107) 2 ; and
Revised: + (cid:107)∇ f (cid:107) 2 ; and

Original: Wasserstein distance (Arjovsky et al., 2017) requires (cid:107)∇ f (cid:107) 2 ≤ 1 .
Revised: Wasserstein distance (Arjovsky et al., 2017) requires (cid:107)∇ f (cid:107) 2 ≤ 1 .

Original: These additional requirements on the dual function might incur some extra difﬁculty in practice.
Revised: These additional requirements on the dual function might incur some extra difﬁculty in practice.

Original: For example, with Wasserstein distance and the Dudley metric, we might need to include an extra gradient penalty (Gulrajani et al., 2017), which requires additional computation to take the gradient through a gradient.
Revised: For example, with Wasserstein distance and the Dudley metric, we might need to include an extra gradient penalty (Gulrajani et al., 2017), which requires additional computation to take the gradient through a gradient.

Original: Meanwhile, the consistency of the surrogate loss under regularization is not clear.
Revised: Meanwhile, the consistency of the surrogate loss under regularization is not clear.

Original: For MMD, we can obtain a closed-form solution for the dual function, which saves the cost of the inner optimization (Gretton et al., 2012), but with the tradeoff of requiring two independent samples in each outer optimization update.
Revised: For MMD, we can obtain a closed-form solution for the dual function, which saves the cost of the inner optimization (Gretton et al., 2012), but with the tradeoff of requiring two independent samples in each outer optimization update.

Original: Moreover, MMD relies on the condition that the dual function lies in some RKHS, which introduces additional kernel parameters to be tuned and in practice may not be sufﬁciently ﬂexible compared to neural networks.
Revised: Moreover, MMD relies on the condition that the dual function lies in some RKHS, which introduces additional kernel parameters to be tuned and in practice may not be sufﬁciently ﬂexible compared to neural networks.

Original: 3. A P RACTICAL A LGORITHM
Revised: 3. A P RACTICAL A LGORITHM

Original: We have derived a consistent stationary distribution correction estimator in the form of a min - max saddle point optimization Equation 13.
Revised: We have derived a consistent stationary distribution correction estimator in the form of a min - max saddle point optimization Equation 13.

Original: Here, we present a practical instantiation of GenDICE with a concrete objective and parametrization.
Revised: Here, we present a practical instantiation of GenDICE with a concrete objective and parametrization.

Original: We choose the χ 2 -divergence, which is an f -divergence with φ ( x ) =
Revised: We choose the χ 2 -divergence, which is an f -divergence with φ ( x ) =

Original: ( x − 1) 2 and φ ∗ ( y ) =
Revised: ( x − 1) 2 and φ ∗ ( y ) =

Original: y + y 2 4 .
Revised: y + y 2 4 .

Original: The objective becomes
Revised: The objective becomes

Original: [Equation]
Revised: [Equation]

Original: There two major reasons for adopting χ 2 -divergence: i)
Revised: There two major reasons for adopting χ 2 -divergence: i)

Original: In the behavior-agnostic OPE problem, we mainly use the ratio correction function for estimating (cid:98) E p [ˆ τ ( s, a ) R ( s, a )] , which is an expectation.
Revised: In the behavior-agnostic OPE problem, we mainly use the ratio correction function for estimating (cid:98) E p [ˆ τ ( s, a ) R ( s, a )] , which is an expectation.

Original: Recall that the error between the estimate and ground-truth can then be bounded by total variation, which is a lower bound of χ 2 -divergence
Revised: Recall that the error between the estimate and ground-truth can then be bounded by total variation, which is a lower bound of χ 2 -divergence.

Original: . ii)
Revised: ii)

Original: For the alternative divergences, the conjugate of the KL -divergence involves exp ( · ) , which may lead to instability in optimization; while the IPM variants introduce extra constraints on dual function, which may be difﬁcult to be optimized.
Revised: For the alternative divergences, the conjugate of the KL -divergence involves exp ( · ) , which may lead to instability in optimization; while the IPM variants introduce extra constraints on dual function, which may be difﬁcult to be optimized.

Original: The conjugate function of χ 2 -divergence en- joys suitable numerical properties and provides squared regularization.
Revised: The conjugate function of χ 2 -divergence enjoys suitable numerical properties and provides squared regularization.

Original: We have provided anadditional empirical ablation study that investigates the alternative divergences in Appendix E.3.
Revised: We have provided an empirical ablation study that investigates the alternative divergences in Section 6.3.

Original: To parameterize the correction ratio τ and dual function f we use neural networks, τ ( s, a )
Revised: To parameterize the correction ratio τ and dual function f we use neural networks, τ ( s, a )

Original: =
Revised: =

Original: nn w τ ( s, a ) and f ( s, a ) =
Revised: nn w τ ( s, a ) and f ( s, a ) =

Original: nn w f ( s, a ) , where w τ and w f denotes the parameters of τ and f respectively.
Revised: nn w f ( s, a ) , where w τ and w f denotes the parameters of τ and f respectively.

Original: Since the optimization requires τ to be non-negative, we add an extra positive neuron, such as exp ( · ) , log (1 + exp ( · )) or ( · ) 2 at the ﬁnal layer of nn w τ ( s, a ) .
Revised: Since the optimization requires τ to be non-negative, we add an extra positive neuron, such as exp ( · ) , log (1 + exp ( · )) or ( · ) 2 at the ﬁnal layer of nn w τ ( s, a ) .

Original: We empirically compare the different positive neurons in Section 6.3.
Revised: We empirically compare the different positive neurons in Section 6.3.

Original: For these representations, and unbiased gradient estimator ∇ ( w τ ,u,w f ) J ( τ, u, f ) can be obtained straightforwardly, as shown in Appendix B.
Revised: For these representations, and unbiased gradient estimator ∇ ( w τ ,u,w f ) J ( τ, u, f ) can be obtained straightforwardly, as shown in Appendix B.

Original: This allows us to apply stochastic gradient descent to solve the saddle-point problem Equation 14 in a scalable manner, as illustrated in Algorithm 1.
Revised: This allows us to apply stochastic gradient descent to solve the saddle-point problem Equation 14 in a scalable manner, as illustrated in Algorithm 1.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: We provide a theoretical analysis for the proposed GenDICE algorithm, following a similar learning setting and assumptions to (Nachum et al., 2019).
Revised: We provide a theoretical analysis for the proposed GenDICE algorithm, following a similar learning setting and assumptions to (Nachum et al., 2019).

Original: Assumption 2 The target stationary correction are bounded, (cid:107) τ ∗ (cid:107) ∞ ≤ C < ∞ .
Revised: Assumption 2 The target stationary correction are bounded, (cid:107) τ ∗ (cid:107) ∞ ≤ C < ∞ .

Original: The main result is summarized in the following theorem.
Revised: The main result is summarized in the following theorem.

Original: A formal statement, together with the proof, is given in Appendix C.
Revised: A formal statement, together with the proof, is given in Appendix C.

Original: Theorem 3 (Informal) Under mild conditions, with learnable F and H , the error in the objective between the GenDICE estimate, ˆ τ , to the solution τ ∗ ( s, a ) =
Revised: Theorem 3 (Informal) Under mild conditions, with learnable F and H , the error in the objective between the GenDICE estimate, ˆ τ , to the solution τ ∗ ( s, a ) =

Original: u ( s,a ) p ( s,a ) is bounded by 
Revised: u ( s,a ) p ( s,a ) is bounded by E

Original: [Equation]
Revised: [Equation]

Original: where E [ · ] is w.r.t.
Revised: where E [ · ] is w.r.t.

Original: the randomness in D and in the optimization algorithms, (cid:15) opt is the optimization error, and (cid:15) approx ( F , H ) is the approximation induced by ( F , H ) for parametrization of ( τ, f ) .
Revised: the randomness in D and in the optimization algorithms, (cid:15) opt is the optimization error, and (cid:15) approx ( F , H ) is the approximation induced by ( F , H ) for parametrization of ( τ, f ) .

Original: The theorem shows that the suboptimality of GenDICE’s solution, measured in terms of the objective function value, can be decomposed into three terms: (1) the approximation error (cid:15) approx , which is controlled by the representation ﬂexibility of function classes; (2) the estimation error due to sample randomness, which decays at the order of 1 / √ N ; and (3) the optimization error, which arises from the suboptimality of the solution found by the optimization algorithm.
Revised: The theorem shows that the suboptimality of GenDICE’s solution, measured in terms of the objective function value, can be decomposed into three terms: (1) the approximation error (cid:15) approx , which is controlled by the representation ﬂexibility of function classes; (2) the estimation error due to sample randomness, which decays at the order of 1 / √ N ; and (3) the optimization error, which arises from the suboptimality of the solution found by the optimization algorithm.

Original: As discussed in Appendix C, in special cases, this suboptimality can be bounded below by a divergence between ˆ τ and τ ∗ , and therefore directly bounds the error in the estimated policy value.
Revised: As discussed in Appendix C, in special cases, this suboptimality can be bounded below by a divergence between ˆ τ and τ ∗ , and therefore directly bounds the error in the estimated policy value.

Original: There is also a tradeoff between these three error terms.
Revised: There is also a tradeoff between these three error terms.

Original: With more ﬂexible function classes ( e.g. , neural networks) for F and H , the approximation error (cid:15) approx becomes smaller.
Revised: With more ﬂexible function classes ( e.g. , neural networks) for F and H , the approximation error (cid:15) approx becomes smaller.

Original: However, it may increase the estimation error (through the constant in front of 1 / √ N ) and the optimization error (by solving a harder optimization problem).
Revised: However, it may increase the estimation error (through the constant in front of 1 / √ N ) and the optimization error (by solving a harder optimization problem).

Original: On the other hand, if F and H are linearly parameterized, estimation and optimization errors tend to be smaller and can often be upper-bounded explicitly in Appendix C.3.
Revised: On the other hand, if F and H are linearly parameterized, estimation and optimization errors tend to be smaller and can often be upper-bounded explicitly in Appendix C.3.

Original: However, the corresponding approximation error will be larger.
Revised: However, the corresponding approximation error will be larger.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Off-policy  evaluation with importance sampling (IS) has has been explored in the contextual bandits (Strehl et al., 2010; Dud´ık et al., 2011;
Revised: Off-policy Policy Evaluation Off-policy policy evaluation with importance sampling (IS) has has been explored in the contextual bandits (Strehl et al., 2010; Dud´ık et al., 2011;

Original: Wang et al., 2017), and episodic RL settings (Murphy et al., 2001; Precup et al., 2001), achieving many empirical successes (e.g., Strehl et al., 2010; Dud´ık et al., 2011; Bottou et al., 2013).
Revised: Wang et al., 2017), and episodic RL settings (Murphy et al., 2001; Precup et al., 2001), achieving many empirical successes (e.g., Strehl et al., 2010; Dud´ık et al., 2011; Bottou et al., 2013).

Original: Unfortunately, IS-based methods suffer from exponential variance in long-horizon problems, known as the “curse of horizon” (Liu et al., 2018).
Revised: Unfortunately, ISbased methods suffer from exponential variance in long-horizon problems, known as the “curse of horizon” (Liu et al., 2018).

Original: A few variancereduction techniques have been introduced, but still cannot eliminate this fundamental issue (Jiang & Li, 2015; Thomas & Brunskill, 2016;
Revised: A few variance-reduction techniques have been introduced, but still cannot eliminate this fundamental issue (Jiang & Li, 2016; Thomas & Brunskill, 2016;

Original: Guo et al., 2017).
Revised: Guo et al., 2017).

Original: By rewriting the accumulated reward as an expectation w.r.t.
Revised: By rewriting the accumulated reward as an expectation w.r.t.

Original: a stationary distribution, Liu et al.
Revised: a stationary distribution, Liu et al.

Original: ; Gelada & Bellemare (2019) recast OPE as estimating a correction ratio function, which signiﬁcantly alleviates variance.
Revised: ; Gelada & Bellemare (2019) recast OPE as estimating a correction ratio function, which signiﬁcantly alleviates variance.

Original: However, these methods still require the off-policy data to be collected by a single and known behavior policy, which restricts their practical applicability.
Revised: However, these methods still require the off-policy data to be collected by a single and known behavior policy, which restricts their practical applicability.

Original: The only published algorithm in the literature, to the best of our knowledge, that solves agnostic-behavior off-policy evaluation is DualDICE (Nachum et al., 2019).
Revised: The only published algorithm in the literature, to the best of our knowledge, that solves agnostic-behavior off-policy evaluation is DualDICE (Nachum et al., 2019).

Original: However, DualDICE was developed for discounted problems and its results become unstable when the discount factor approaches 1 (see below).
Revised: However, DualDICE was developed for discounted problems and its results become unstable when the discount factor approaches 1 (see below).

Original: By contrast, GenDICE can cope with the more challenging problem of undiscounted reward estimation in the general behavior-agnostic setting.
Revised: By contrast, GenDICE can cope with the more challenging problem of undiscounted reward estimation in the general behavior-agnostic setting.

Original: Note that standard model-based methods (Sutton & Barto, 1998), which estimate the transition and reward models directly then calculate the expected reward based on the learned model, are also applicable to the behavior-agnostic setting considered here.
Revised: Note that standard model-based methods (Sutton & Barto, 1998), which estimate the transition and reward models directly then calculate the expected reward based on the learned model, are also applicable to the behavior-agnostic setting considered here.

Original: Unfortunately, model-based methods typically rely heavily on modeling assumptions about rewards and transition dynamics.
Revised: Unfortunately, model-based methods typically rely heavily on modeling assumptions about rewards and transition dynamics.

Original: In practice, these assumptions do not always hold, and the evaluation results can become unreliable.
Revised: In practice, these assumptions do not always hold, and the evaluation results can become unreliable.

Revised: Markov Chain Monte Carlo Classical MCMC (Brooks et al., 2011; Gelman et al., 2013) aims at sampling from µ π by iteratively simulting from the transition operator.

Revised: It requires continuous interaction with the transition operator and heavy computational cost to update many particles.

Revised: Amortized SVGD (Wang & Liu, 2016) and Adversarial MCMC (Song et al., 2017; Li et al., 2019) alleviate this issue via combining with neural network, but they still interact with the transition operator directly, i.e. , in an on-policy setting.

Revised: The major difference of our GenDICE is the learning setting: we only access the off-policy dataset, and cannot sample from the transition operator.

Original: For more related work on MCMC, density ratio estimation  and PageRank, please refer to Appendix F.
Revised: The proposed GenDICE leverages stationary density ratio estimation for approximating the stationary quantities, which distinct it from classical methods. Density Ratio Estimation Density ratio estimation is a fundamental tool in machine learning and much related work exists. Classical density ratio estimation includes moment matching (Gretton et al., 2008), probabilistic classiﬁcation (Bickel et al., 2007), and ratio matching (Nguyen et al., 2008; Sugiyama et al., 2008; Kanamori et al., 2009).

Revised: These classical methods focus on estimating the ratio between two distributions with samples from both of them, while GenDICE estimates the density ratio to a stationary distribution of a transition operator, from which even one sample is difﬁcult to obtain.

Revised: PageRank Yao & Schuurmans (2013) developed a reverse-time RL framework for PageRank via solving a reverse Bellman equation, which is less sensitive to graph topology and shows faster adaptation with graph change.

Revised: However, Yao & Schuurmans (2013) still considers the online manner, which is different with our OPR setting.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate GenDICE on OPE and OPR problems.
Revised: In this section, we evaluate GenDICE on OPE and OPR problems.

Original: For OPE, we use one or multiple behavior policies to collect a ﬁxed number of trajectories at some ﬁxed trajectory length.
Revised: For OPE, we use one or multiple behavior policies to collect a ﬁxed number of trajectories at some ﬁxed trajectory length.

Original: This data is used to recover a correction ratio function for a target policy π that is then used to estimate the average reward in two different settings: i ) average reward; and ii ) discounted reward.
Revised: This data is used to recover a correction ratio function for a target policy π that is then used to estimate the average reward in two different settings: i ) average reward; and ii ) discounted reward.

Original: In both settings, we compare with a model-based approach and step-wise weighted IS (Precup et al., 2000).
Revised: In both settings, we compare with a model-based approach and step-wise weighted IS (Precup et al., 2000).

Original: We also compare to Liu et al.
Revised: We also compare to Liu et al.

Original: (2018) (referred to as “IPS” here) in the Taxi domain with a learned behavior policy 1 .
Revised: (2018) (referred to as “IPS” here) in the Taxi domain with a learned behavior policy 1 .

Original: We speciﬁcally compare to DualDICE (Nachum et al., 2019) in the discounted reward setting, which is a direct and current state-of-the-art baseline.
Revised: We speciﬁcally compare to DualDICE (Nachum et al., 2019) in the discounted reward setting, which is a direct and current state-of-the-art baseline.

Original: For OPR, the main comparison is with the model-based method, where the transition operator is empirically estimated and stationary distribution recovered via an exact solver.
Revised: For OPR, the main comparison is with the model-based method, where the transition operator is empirically estimated and stationary distribution recovered via an exact solver.

Original: We validate GenDICE in both tabular and continuous cases, and perform an ablation study to further demonstrate its effectiveness.
Revised: We validate GenDICE in both tabular and continuous cases, and perform an ablation study to further demonstrate its effectiveness.

Original: All results are based on 20 random seeds, with mean and standard deviation plotted.
Revised: All results are based on 20 random seeds, with mean and standard deviation plotted.

Original: Our code will be publicly available for reproduction.
Revised: Our code is publicly available at https://github.com/zhangry868/GenDICE .

Original: 6. T ABULAR C ASE
Revised: 6. T ABULAR C ASE

Original: [Equation]

Original: -BasedGenDICE (ours) 1k 2k 5k 10k 20k 50k # of Samples -3.

Original: BA Graph (500 Nodes) 2k 5k 10k 20k 50k 100k # of Samples -2.

Original: l o g K L - D i v e r gen c e

Original: Citeseer 2k 5k 10k 20k 50k 100k # of Samples -2.8 -2.

Original: Cora Ofﬂine PageRank on Graphs One direct application of GenDICE is off-line PageRank (OPR).
Revised: Ofﬂine PageRank on Graphs One direct application of GenDICE is off-line PageRank (OPR).

Original: We test GenDICE on a Barabasi-Albert (BA) graph (synthetic), and two realworld graphs, Cora and Citeseer.
Revised: We test GenDICE on a Barabasi-Albert (BA) graph (synthetic), and two real-world graphs, Cora and Citeseer.

Original: Details of the graphs are given in Appendix D.
Revised: Details of the graphs are given in Appendix D.

Original: We use the log KL -divergence between estimated stationary distribution and the ground truth as the evaluation metric, with the ground truth computed by an exact solver based on the exact transition operator of the graphs.
Revised: We use the log KL -divergence between estimated stationary distribution and the ground truth as the evaluation metric, with the ground truth computed by an exact solver based on the exact transition operator of the graphs.

Original: We compared GenDICE with modelbased methods in terms of the sample efﬁciency.
Revised: We compared GenDICE with model-based methods in terms of the sample efﬁciency.

Original: From the results in Figure 1, GenDICE outperforms the modelbased method when limited data is given.
Revised: From the results in Figure 1, GenDICE outperforms the model-based method when limited data is given.

Original: Even with 20 k samples for a BA graph with 100 nodes, where a transition matrix has 10 k entries, GenDICE still shows better performance in the ofﬂine setting.
Revised: Even with 20 k samples for a BA graph with 100 nodes, where a transition matrix has 10 k entries, GenDICE still shows better performance in the ofﬂine setting.

Original: This is reasonable since GenDICE directly estimates the stationary distribution vector or ratio, while the model-based method needs to learn an entire transition matrix that has many more parameters. 
Revised: This is reasonable since GenDICE directly estimates the stationary distribution vector or ratio, while the model-based method needs to learn an entire transition matrix that has many more parameters. 0.5k 1k 2k 5k 10k 20k # of Samples -5. l o g K L - D i v e r g e n c e BA

Revised: [Figure]

Original: Off-Policy Evaluation with Taxi We use a similar taxi domain as in Liu et al.
Revised: [Figure] [Figure] [Figure] (ours) 1k 2k 5k

Revised: [Figure]

Revised: 20k 50k

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: l o g K L - D i v e r g e n

Revised: [Figure]

Revised: 2k 5k

Revised: [Figure]

Revised: [Figure]

Revised: Off-Policy Evaluation with Taxi We use a similar taxi domain as in Liu et al.

Original: (2018), where a grid size of 5 × 5 yields 2000 states in total ( 25 × 16 × 5 , corresponding to 25 taxi locations, 16 passenger appearance status and 5 taxi status).
Revised: (2018), where a grid size of 5 ×yields 2000 states in total ( 25 × 16 × 5 , corresponding to 25 taxi locations, 16 passenger appearance status and 5 taxi status).

Original: We set the target policy to a ﬁnal policy π after running tabular Q-learning for 1000 iterations, and set another policy π + after 950 iterations as the base policy.
Revised: We set the target policy to a ﬁnal policy π after running tabular Q-learning for 1000 iterations, and set another policy π + after 950 iterations as the base policy.

Original: The behavior policy is a mixture controlled by α as π b =
Revised: The behavior policy is a mixture controlled by α as π b =

Original: (1 − α )
Revised: (1 − α )

Original: π
Revised: π

Original: + απ + .
Revised: + απ + .

Original: For the model-based method, we use a tabular representation for the reward and transition functions, whose entries are estimated from behavior data.
Revised: For the model-based method, we use a tabular representation for the reward and transition functions, whose entries are estimated from behavior data.

Original: For IS and IPS, we ﬁt a policy via behavior cloning to estimate the policy ratio.
Revised: For IS and IPS, we ﬁt a policy via behavior cloning to estimate the policy ratio.

Original: In this speciﬁc setting, our methods achieve better results compared to IS, IPS and the model-based method.
Revised: In this speciﬁc setting, our methods achieve better results compared to IS, IPS and the model-based method.

Original: Interestingly, with longer horizons, IS cannot improve as much as other methods even with more data, while GenDICE consistently improve and achieves much better results than the baselines.
Revised: Interestingly, with longer horizons, IS cannot improve as much as other methods even with more data, while GenDICE consistently improve and achieves much better results than the baselines.

Original: DualDICE only works with γ < 1 .
Revised: DualDICE only works with γ < 1 .

Original: GenDICE is more stable than DualDICE when γ becomes larger (close to 1 ), while still showing competitive performance for smaller discount factors γ .
Revised: GenDICE is more stable than DualDICE when γ becomes larger (close to 1 ), while still showing competitive performance for smaller discount factors γ .

Original: [Figure]
Revised: [Figure]

Original: 6. C ONTINUOUS C ASE
Revised: 6. C ONTINUOUS C ASE

Original: We further test our method for OPE on three control tasks: a discrete-control task Cartpole and two continuous-control tasks Reacher and HalfCheetah.
Revised: We further test our method for OPE on three control tasks: a discrete-control task Cartpole and two continuous-control tasks Reacher and HalfCheetah.

Original: In these tasks, observations (or states) are continuous, thus we use neural network function approximators and stochastic optimization.
Revised: In these tasks, observations (or states) are continuous, thus we use neural network function approximators and stochastic optimization.

Original: Since
Revised: Since

Original: DualDICE (Nachum et al., 2019) has shown the state-of-the-art performance on discounted OPE, we mainly compare with it in the discounted reward case.
Revised: DualDICE (Nachum et al., 2019) has shown the state-of-the-art performance on discounted OPE, we mainly compare with it in the discounted reward case.

Original: We also compare to IS with a learned policy via behavior cloning and a neural model-based method, similar to the tabular case, but with neural network as the function approximator.
Revised: We also compare to IS with a learned policy via behavior cloning and a neural model-based method, similar to the tabular case, but with neural network as the function approximator.

Original: All neural networks are feed-forward with two hidden layers of dimension 64 and tanh activations.
Revised: All neural networks are feed-forward with two hidden layers of dimension 64 and tanh activations.

Original: More details can be found in Appendix D.
Revised: More details can be found in Appendix D.

Original: Due to limited space, we put the discrete control results in Appendix E and focus on the more challenging continuous control tasks.
Revised: Due to limited space, we put the discrete control results in Appendix E and focus on the more challenging continuous control tasks.

Original: Here, the good performance of IS and model-based methods in
Revised: Here, the good performance of IS and model-based methods in

Original: Section 6.1 quickly deteriorates as the environment becomes complex, i.e. , with a continuous action space.
Revised: Section 6.1 quickly deteriorates as the environment becomes complex, i.e. , with a continuous action space.

Original: Note that GenDICE is able to maintain good performance in this scenario, even when using function approximation and stochastic optimization.
Revised: Note that GenDICE is able to maintain good performance in this scenario, even when using function approximation and stochastic optimization.

Original: This is reasonable because of the difﬁculty of ﬁtting to the coupled policy-environment dynamics with a continuous action space.
Revised: This is reasonable because of the difﬁculty of ﬁtting to the coupled policy-environment dynamics with a continuous action space.

Original: Here we also empirically validate GenDICE with off-policy data collected by multiple policies.
Revised: Here we also empirically validate GenDICE with off-policy data collected by multiple policies.

Original: As illustrated in Figure 3, all methods perform better with longer trajectory length or more trajectories.
Revised: As illustrated in Figure 3, all methods perform better with longer trajectory length or more trajectories.

Original: When α becomes larger, i.e. , the behavior policies are closer to the target policy, all methods performs better, as expected.
Revised: When α becomes larger, i.e. , the behavior policies are closer to the target policy, all methods performs better, as expected.

Original: Here, GenDICE demonstrates good performance both on averagereward and discounted reward cases in different settings.
Revised: Here, GenDICE demonstrates good performance both on averagereward and discounted reward cases in different settings.

Original: The right two ﬁgures in each row show the log MSE curve versus optimization steps, where GenDICE achieves the smallest loss.
Revised: The right two ﬁgures in each row show the log MSE curve versus optimization steps, where GenDICE achieves the smallest loss.

Original: In the discounted reward case, GenDICE shows signiﬁcantly better and more stable performance than the strong baseline, DualDICE.
Revised: In the discounted reward case, GenDICE shows signiﬁcantly better and more stable performance than the strong baseline, DualDICE.

Original: Figure 4 also shows better performance of GenDICE than all baselines in the more challenging HalfCheetah domain.
Revised: Figure 4 also shows better performance of GenDICE than all baselines in the more challenging HalfCheetah domain [Figure] [Figure]

Original: 6. A BLATION S TUDY
Revised: 6. A BLATION S TUDY

Original: Finally, we conduct an ablation study on GenDICE to study its robustness and implementation sensitivities.
Revised: Finally, we conduct an ablation study on GenDICE to study its robustness and implementation sensitivities.

Original: We investigate the effects of learning rate, activation function, discount factor, and the speciﬁcally designed ratio constraint.
Revised: We investigate the effects of learning rate, activation function, discount factor

Original: We further demonstrate the effect of the choice of divergences and the penalty weight in Appendix E.3.

Original: [Figure]
Revised: [Figure]

Revised: and the speciﬁcally designed ratio constraint.

Revised: We further demonstrate the effect of the choice of divergences and the penalty weight.

Original: Effects of the Learning Rate Since we are using neural network as the function approximator, and stochastic optimization, it is necessary to show sensitivity to the learning rate with { 0 . 0001 , 0 . 0003 , 0 .
Revised: Effects of the Learning Rate Since we are using neural network as the function approximator, and stochastic optimization, it is necessary to show sensitivity to the learning rate with { 0 . 0001 , 0 . 0003 , 0 .

Original: 001 , 0 . 003 } , with results in Figure 5.
Revised: 001 , 0 . 003 } , with results in Figure 5.

Original: When α = 0 .
Revised: When α = 0 .

Original: 33 , i.e. , the OPE tasks are relatively easier and GenDICE obtains better results at all learning rate settings.
Revised: 33 , i.e. , the OPE tasks are relatively easier and GenDICE obtains better results at all learning rate settings.

Original: However, when α = 0 . 0 , i.e. , the estimation becomes more difﬁcult and only GenDICE only obtains reasonable results with the larger learning rate.
Revised: However, when α = 0 . 0 , i.e. , the estimation becomes more difﬁcult and only GenDICE only obtains reasonable results with the larger learning rate.

Original: Generally, this ablation study shows that the proposed method is not sensitive to the learning rate, and is easy to train.
Revised: Generally, this ablation study shows that the proposed method is not sensitive to the learning rate, and is easy to train.

Original: Activation Function of Ratio Estimator We further investigate the effects of the activation function on the last layer, which ensure the non-negative outputs required for the ratio.
Revised: Activation Function of Ratio Estimator We further investigate the effects of the activation function on the last layer, which ensure the non-negative outputs required for the ratio.

Original: To better understand which activation function will lead to stable trainig for the neural correction estimator, we empirically compare using i ) ( · ) 2 ; ii ) log(1 + exp( · )) ; and iii ) exp( · ) .
Revised: To better understand which activation function will lead to stable trainig for the neural correction estimator, we empirically compare using i ) ( · ) 2 ; ii ) log(1 + exp( · )) ; and iii ) exp( · ) .

Original: In practice, we use the ( · )since it achieves low variance and better performance in most cases, as shown in Figure 5.
Revised: In practice, we use the ( · )since it achieves low variance and better performance in most cases, as shown in Figure 5.

Original: Effects of Discount Factors We vary γ ∈ { 0 . 95 , 0 . 99 , 0 .
Revised: Effects of Discount Factors We vary γ ∈ { 0 . 95 , 0 . 99 , 0 .

Original: 999 , 1 . 0 } to probe the sensitivity of GenDICE.
Revised: 999 , 1 . 0 } to probe the sensitivity of GenDICE.

Original: Speciﬁcally, we compare to DualDICE, and ﬁnd that GenDICE is stable, while
Revised: Speciﬁcally, we compare to DualDICE, and ﬁnd that GenDICE is stable, while

Original: DualDICE becomes unstable when the γ becomes large, as shown in Figure 6.
Revised: DualDICE becomes unstable when the γ becomes large, as shown in Figure 6.

Original: GenDICE is also more general than DualDICE, as it can be applied to both the average and discounted reward cases.
Revised: GenDICE is also more general than DualDICE, as it can be applied to both the average and discounted reward cases.

Original: Effects of Ratio Constraint
Revised: Effects of Ratio Constraint

Original: In Section 3, we highlighted the importance of the ratio constraint.
Revised: In Section 3, we highlighted the importance of the ratio constraint.

Original: Here we investigate the trivial solution issue without the constraint.
Revised: Here we investigate the trivial solution issue without the constraint.

Original: The results in Figure 6 demonstrate the necessity of adding the constraint penalty, since a trivial solution prevents an accurate corrector from being recovered (green line in left two ﬁgures). 
Revised: The results in Figure 6 demonstrate the necessity of adding the constraint penalty, since a trivial solution prevents an accurate corrector from being recovered (green line in left two ﬁgures). [Figure]

Original: C ONCLUSION
Revised: Effects of the Choice of Divergences We empirically test the GenDICE with several other alternative divergences, e g. , Wasserstein- 1 distance, Jensen-Shannon divergence, KL -divergence, Hellinger divergence, and MMD. To avoid the effects of other factors in the estimator, e.g. , function parametrization, we focus on the ofﬂine PageRank task on BA graph with 100 nodes and 10 k ofﬂine samples.

Revised: All the experiments are evaluated withrandom trials.

Revised: To ensure the dual function to be 1 -Lipchitz, we add the gradient penalty.

Revised: Besides, we use a learned

Revised: Gaussian kernel in MMD, similar to Li et al.

Revised: As we can see in Figure 7(a), the GenDICE estimator is compatible with many different divergences.

Revised: Most of the divergences, with appropriate extra techniques to handle the difﬁculties in optimization and carefully tuning for extra parameters, can achieve similar performances, consistent with phenomena in the variants of GANs (Lucic et al., 2018).

Revised: However, KL -divergence is an outlier, performing noticeably worse, which might be caused by the ill-behaved exp ( · ) in its conjugate function.

Revised: The χ 2divergence and JS-divergence are better, which achieve good performances with fewer parameters to be tuned.

Revised: Effects of the Penalty Weight

Revised: The results of different penalty weights λ are illustrated in Figure 7(b).

Revised: We vary the λ ∈ [0 . 1 , 5] with χ 2 -divergence.

Revised: Within a large range of λ , the performances of the proposed GenDICE are quite consistent, which justiﬁes Theorem 1.

Revised: The penalty multiplies with λ .

Revised: Therefore, with λ increases, the variance of the stochastic gradient estimator also increases, which explains the variance increasing in large λ in Figure 7(b).

Revised: In practice, λ = 1 is a reasonable choice for general cases.

Revised:  C ONCLUSION

Original: In this paper, we proposed a novel algorithm GenDICE for general stationary distribution correction estimation, which can handle both the discounted and average stationary distribution given multiple behavior-agnostic samples.
Revised: In this paper, we proposed a novel algorithm GenDICE for general stationary distribution correction estimation, which can handle both the discounted and average stationary distribution given multiple behavior-agnostic samples.

Original: Empirical results on off-policy evaluation and ofﬂine PageRank show the superiority of proposed method over the existing state-of-the-art methods.
Revised: Empirical results on off-policy evaluation and ofﬂine PageRank show the superiority of proposed method over the existing state-of-the-art methods.

Revised: A CKNOWLEDGMENTS

Revised: The authors would like to thank Oﬁr Nachum, the rest of the Google Brain team and the anonymous reviewers for helpful discussions and feedback.

Original: A P ROPERTIES OF G EN DICE
Revised: A P ROPERTIES OF G EN DICE

Original: For notation simplicity, we denote x =
Revised: For notation simplicity, we denote x =

Original: ( s, a ) ∈ Ω :=
Revised: ( s, a ) ∈ Ω :=

Original: S × A and P π ( x (cid:48)
Revised: S × A and P π ( x (cid:48)

Original: | x ) :
Revised: | x ) :

Original: = π ( a (cid:48) | s (cid:48) )
Revised: = π ( a (cid:48) | s (cid:48) )

Original: P ( s (cid:48) | s, a ) .
Revised: P ( s (cid:48) | s, a ) .

Original: Also deﬁne (cid:107) f (cid:107) p, 2 :=
Revised: Also deﬁne (cid:107) f (cid:107) p, 2 :=

Original: (cid:104) f, f (cid:105) p = (cid:82) f
Revised: (cid:104) f, f (cid:105) p = (cid:82) f

Original: ( x ) 2 p ( x ) dx .
Revised: ( x ) 2 p ( x ) dx .

Original: We make the following assumption to ensure the existence of the stationary distribution.
Revised: We make the following assumption to ensure the existence of the stationary distribution.

Original: Our discussion is all based on this assumption.
Revised: Our discussion is all based on this assumption.

Original: Assumption 1 Under the target policy, the resulted state-action transition operator T has a unique stationary distribution in terms of the divergence D ( ·||· ) .
Revised: Assumption 1 Under the target policy, the resulted state-action transition operator T has a unique stationary distribution in terms of the divergence D ( ·||· ) .

Original: If the total variation divergence is selected, the Assumption 1 requires the transition operator should be ergodic, as discussed in Meyn & Tweedie (2012).
Revised: If the total variation divergence is selected, the Assumption 1 requires the transition operator should be ergodic, as discussed in Meyn & Tweedie (2012).

Original: A. C ONSISTENCY OF THE E STIMATOR
Revised: A. C ONSISTENCY OF THE E STIMATOR

Original: Theorem 1 For arbitrary λ > 0 , the solution to the optimization Eqn (11) is u ( s,a ) p ( s,a ) for γ ∈ (0 , 1] .
Revised: Theorem 1 For arbitrary λ > 0 , the solution to the optimization Eqn (11) is u ( s,a ) p ( s,a ) for γ ∈ (0 , 1] .

Original: Proof For γ ∈ (0 , 1) , there is not degenerate solutions to D (cid:0)(cid:0) T pγ,µ 0 ◦ τ (cid:1) || p · τ (cid:1) .
Revised: Proof For γ ∈ (0 , 1) , there is not degenerate solutions to D (cid:0)(cid:0) T pγ,µ 0 ◦ τ (cid:1) || p · τ (cid:1) .

Original: The optimal solution is a density ratio.
Revised: The optimal solution is a density ratio.

Original: Therefore, the extra penalty (cid:0) E p ( x ) [ τ ( x )] − 1 (cid:1) 2 does not affect the optimality for ∀ λ > 0 .
Revised: Therefore, the extra penalty (cid:0) E p ( x ) [ τ ( x )] − 1 (cid:1) 2 does not affect the optimality for ∀ λ > 0 .

Original: When γ = 1 , for ∀ λ > 0 , recall both D (cid:0)(cid:0) T pγ,µ 0 ◦ τ (cid:1) || p · τ (cid:1) and (cid:0) E p ( x ) [ τ ( x )] − 1 (cid:1) 2 are nonnegative, and the the density ratio µ ( x ) p ( x ) leads to zero for both terms.
Revised: When γ = 1 , for ∀ λ > 0 , recall both D (cid:0)(cid:0) T pγ,µ 0 ◦ τ (cid:1) || p · τ (cid:1) and (cid:0) E p ( x ) [ τ ( x )] − 1 (cid:1) 2 are nonnegative, and the the density ratio µ ( x ) p ( x ) leads to zero for both terms.

Original: Then, the density ratio is a solution to J ( τ ) .
Revised: Then, the density ratio is a solution to J ( τ ) .

Original: For any other non-negative function τ ( x ) ≥ 0 , if it is the optimal solution to
Revised: For any other non-negative function τ ( x ) ≥ 0 , if it is the optimal solution to

Original: J ( τ ) , then, we have
Revised: J ( τ ) , then, we have

Original: [Equation]
Revised: [Equation]

Original: We denote µ ( x ) =
Revised: We denote µ ( x ) =

Original: p ( x ) τ ( x ) , which is clearly a density function.
Revised: p ( x ) τ ( x ) , which is clearly a density function.

Original: Then, the optimal conditions in
Revised: Then, the optimal conditions in

Original: Equation 15 imply
Revised: Equation 15 imply

Original: [Equation]
Revised: [Equation]

Original: or equivalently, µ is the stationary distribution of T .
Revised: or equivalently, µ is the stationary distribution of T .

Original: We have thus shown the optimal τ ( x ) = µ ( x ) p ( x ) is the target density ratio.
Revised: We have thus shown the optimal τ ( x ) = µ ( x ) p ( x ) is the target density ratio.

Original: A. C ONVEXITY OF THE O BJECTIVE
Revised: A. C ONVEXITY OF THE O BJECTIVE

Original: Proof Since the φ is convex
Revised: Proof Since the φ is convex

Original: [Equation]
Revised: [Equation]

Original: we consider the Fenchel dual representation of the f -divergence
Revised: we consider the Fenchel dual representation of the f -divergence

Original: D φ
Revised: D φ

Original: [Equation]
Revised: [Equation]

Original: It is obviously (cid:96) ( τ, f ) is convex 
Revised: It is obviously (cid:96) ( τ, f ) is convex τ τ τ

Original: Algorithm
Revised: Algorithm

Original: Inputs :
Revised: Inputs :

Original: Convex function φ and its Fenchel conjugate φ ∗ , off-policy data D
Revised: Convex function φ and its Fenchel conjugate φ ∗ , off-policy data D

Original: = { ( s ( i ) , a ( i ) , r ( i ) , s (cid:48)
Revised: = { ( s ( i ) , a ( i ) , r ( i ) , s (cid:48)

Original: ( i ) )
Revised: ( i ) )

Original: } Ni =1 , initial state s∼ µ 0 , target policy π , distribution corrector nn w τ ( · , · ) , nn w f ( · , · ) , constraint scalar u , learning rates η τ , η f , η u , number of iterations K , batch size B .
Revised: } Ni =1 , initial state s∼ µ 0 , target policy π , distribution corrector nn w τ ( · , · ) , nn w f ( · , · ) , constraint scalar u , learning rates η τ , η f , η u , number of iterations K , batch size B .

Original: for t = 1 , . . .
Revised: for t = 1 , . . .

Original: K do
Revised: K do

Original: Sample batch { ( s ( i ) , a ( i ) , r ( i ) , s (cid:48)
Revised: Sample batch { ( s ( i ) , a ( i ) , r ( i ) , s (cid:48)

Original: ( i ) )
Revised: ( i ) )

Original: } Bi =1 from D .
Revised: } Bi =1 from D .

Original: Sample batch { s ( i ) 0 } Bi =1 from µ 0 .
Revised: Sample batch { s ( i ) 0 } Bi =1 from µ 0 .

Original: Sample actions a (cid:48)
Revised: Sample actions a (cid:48)

Original: ( i ) ∼ π ( s (cid:48) ( i ) ) , for i
Revised: ( i ) ∼ π ( s (cid:48) ( i ) ) , for i

Original: = 1 , . .
Revised: = 1 , . .

Original: , B .
Revised: , B .

Original: Sample actions a ( i ) 0 ∼ π ( s ( i ) 0 ) , for i = 1 , . . .
Revised: Sample actions a ( i ) 0 ∼ π ( s ( i ) 0 ) , for i = 1 , . . .

Original: , B .
Revised: , B .

Original: Compute empirical loss ˆ J χ 2 ( τ, u, f )
Revised: Compute empirical loss ˆ J χ 2 ( τ, u, f )

Original: = (1 − γ ) E µ 0 π
Revised: = (1 − γ ) E µ 0 π

Original: [ f ( s, a )] + γ E T p
Revised: [ f ( s, a )] + γ E T p

Original: [ τ ( s, a ) f ( s (cid:48) , a (cid:48) )]
Revised: [ τ ( s, a ) f ( s (cid:48) , a (cid:48) )]

Original: − E p (cid:2) τ ( s, a ) (cid:0) f ( s, a ) + 14 f 2 ( s, a ) (cid:1)(cid:3) + λ (cid:16) E p [ uτ ( s, a ) − u ] − u 2 2 (cid:17) .
Revised: − E p (cid:2) τ ( s, a ) (cid:0) f ( s, a ) + 14 f 2 ( s, a ) (cid:1)(cid:3) + λ (cid:16) E p [ uτ ( s, a ) − u ] − u 2 2 (cid:17) .

Original: Update w τ ← w τ − η τ ∇ θ τ ˆ J χ 2 .
Revised: Update w τ ← w τ − η τ ∇ θ τ ˆ J χ 2 .

Original: Update w f ← w f + η f ∇ θ f ˆ J χ 2 .
Revised: Update w f ← w f + η f ∇ θ f ˆ J χ 2 .

Original: Update u ← u + η u ∇ u ˆ J χ 2 .
Revised: Update u ← u + η u ∇ u ˆ J χ 2 .

Original: end for
Revised: end for

Original: Return nn w τ .
Revised: Return nn w τ .

Original: B A LGORITHM D ETAILS
Revised: B A LGORITHM D ETAILS

Original: We provide the unbiased gradient estimator for ∇ w τ ,u,w f J ( τ, u, f ) in Eqn (14) below:
Revised: We provide the unbiased gradient estimator for ∇ w τ ,u,w f J ( τ, u, f ) in Eqn (14) below:

Original: [Equation]
Revised: [Equation]

Original: Then, we have the psuedo code which applies SGD for solving Eqn (14)
Revised: Then, we have the psuedo code which applies SGD for solving Eqn (14)

Original: [Equation]
Revised: [Equation]

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: For convenience
Revised: For convenience

Original: [Equation]
Revised: [Equation]

Original: we repeat here the notation deﬁned in the main text.
Revised: we repeat here the notation deﬁned in the main text.

Original: The saddle-point reformulation of the objective function of GenDICE is:
Revised: The saddle-point reformulation of the objective function of GenDICE is:

Original: [Equation]
Revised: [Equation]

Original: To avoid the numerical inﬁnity in D φ ( ·||· ) , we induced the bounded version as
Revised: To avoid the numerical inﬁnity in D φ ( ·||· ) , we induced the bounded version as

Original: [Equation]
Revised: [Equation]

Original: in which D Cφ ( ·||· ) is still a valid divergence, and therefore the optimal solution τ ∗ is still the stationary density ratio µ ( x ) p ( x ) .
Revised: in which D Cφ ( ·||· ) is still a valid divergence, and therefore the optimal solution τ ∗ is still the stationary density ratio µ ( x ) p ( x ) .

Original: We denote the (cid:98) J ( τ, µ, f ) as the empirical surrogate of J ( τ, µ, f ) on samples D = (cid:16) ( x, x (cid:48) )
Revised: We denote the (cid:98) J ( τ, µ, f ) as the empirical surrogate of J ( τ, µ, f ) on samples D = (cid:16) ( x, x (cid:48) )

Original: N i =1 (cid:17) ∼ T pγ,µ 

Revised: N i =1 (cid:17) ∼ T pγ,µ

Original: ( x, x (cid:48) ) with the optimal solution in ( H , F , R ) as (cid:16) ˆ τ ∗H , (cid:98) u ∗ , (cid:98)
Revised: ( x, x (cid:48) ) with the optimal solution in ( H , F , R ) as (cid:16) ˆ τ ∗H , (cid:98) u ∗ , (cid:98)

Original: f ∗F (cid:17) .
Revised: f ∗F (cid:17) .

Original: Furthermore, denote
Revised: Furthermore, denote

Original: [Equation]
Revised: [Equation]

Original: with optimal ( f ∗ , u ∗ ) , and
Revised: with optimal ( f ∗ , u ∗ ) , and

Original: [Equation]
Revised: [Equation]

Original: We apply some optimization algorithm for (cid:98) J ( τ, u, f ) over space ( H , F , R ) , leading to the output (cid:16) ˆ τ, (cid:98) u, (cid:98) f (cid:17) .
Revised: We apply some optimization algorithm for (cid:98) J ( τ, u, f ) over space ( H , F , R ) , leading to the output (cid:16) ˆ τ, (cid:98) u, (cid:98) f (cid:17) .

Original: Under Assumption 2, we need only consider (cid:107) τ (cid:107) ∞ ≤ C , then, the corresponding dual u = E p ( τ ) − 1 ⇒ u ∈ U := {| u | ≤ ( C + 1) } .
Revised: Under Assumption 2, we need only consider (cid:107) τ (cid:107) ∞ ≤ C , then, the corresponding dual u = E p ( τ ) − 1 ⇒ u ∈ U := {| u | ≤ ( C + 1) } .

Original: We choose the φ ∗ ( · ) is a κ -Lipschitz continuous, then, the
Revised: We choose the φ ∗ ( · ) is a κ -Lipschitz continuous, then, the

Original: J ( τ, u, f ) isa C P
Revised: J ( τ, u, f ) is

Original: [Equation]
Revised: [Equation]

Original: ,κ,
Revised: P

Original: [Equation]
Revised: [Equation]

Original: = max
Revised: = max

Original: [Equation]
Revised: [Equation]

Original: + 
Revised: + 1)

Original: [Equation]
Revised: [Equation]

Original: (cid:0) λ +
Revised: λ +

Original: [Equation]
Revised: [Equation]

Original: Lipschitz continuous function w.r.t.
Revised: Lipschitz continuous function w.r.t.

Original: ( f, u ) with the norm (cid:107) ( f, u ) (cid:107) p, 1 := (cid:82) | f
Revised: ( f, u ) with the norm (cid:107) ( f, u ) (cid:107) p, 1 := (cid:82) | f

Original: ( x ) | p ( x ) dx + | u | , and
Revised: ( x ) | p ( x ) dx + | u | , and

Original: C φ,C,λ := (cid:0) C + λ ( C + 1) + max t ∈{− C,C } ( − φ ( t )) (cid:1) -Lipschitz continuous function w.r.t.
Revised: C φ,C,λ := (cid:0) C + λ ( C + 1) + max t ∈{− C,C } ( − φ ( t )) (cid:1) -Lipschitz continuous function w.r.t.

Original: τ with the norm (cid:107) τ (cid:107) p, 1 := (cid:82) | τ ( x ) | p ( x ) dx .
Revised: τ with the norm (cid:107) τ (cid:107) p, 1 := (cid:82) | τ ( x ) | p ( x ) dx .

Original: We consider the error between ˆ τ and τ ∗ using standard arguments (Shalev-Shwartz & Ben-David, 2014; Bach, 2014), i.e. ,
Revised: We consider the error between ˆ τ and τ ∗ using standard arguments (Shalev-Shwartz & Ben-David, 2014; Bach, 2014), i.e. ,

Original: [Equation]
Revised: [Equation]

Original: The discrepancy d ( τ, τ ∗ ) ≥ 0 and d ( τ, τ ∗ ) = 

Revised: The discrepancy d ( τ, τ ∗ ) ≥ 0 and d ( τ, τ ∗ ) =

Original: if and only if p · τ is stationary distribution of T in the weak sense of D φ ( ·||· ) .
Revised: if and only if p · τ is stationary distribution of T in the weak sense of D φ ( ·||· ) .

Original: Remark:
Revised: Remark:

Original: In some special cases, the suboptimality also implies the distance between ˆ τ and τ ∗ .
Revised: In some special cases, the suboptimality also implies the distance between ˆ τ and τ ∗ .

Original: Speciﬁcally,for γ = 1 , if the transition operator P π can be represented as P π = Q Λ Q − 1 where
Revised: Speciﬁcally,for γ = 1 , if the transition operator P π can be represented as P π = Q Λ Q − 1 where

Original: Q denotes the (countable) eigenfunctions and Λ denotes the diagonal matrix with eigenvalues, the largest of which is 1 .
Revised: Q denotes the (countable) eigenfunctions and Λ denotes the diagonal matrix with eigenvalues, the largest of which is 1 .

Original: We consider φ ( · ) as identity and f ∈ F := (cid:110) span ( Q ) , (cid:107) f (cid:107) p, 2 ≤ 1 (cid:111) , then the d ( τ, τ ∗ ) will bounded from below by a metric between τ and τ ∗ .
Revised: We consider φ ( · ) as identity and f ∈ F := (cid:110) span ( Q ) , (cid:107) f (cid:107) p, 2 ≤ 1 (cid:111) , then the d ( τ, τ ∗ ) will bounded from below by a metric between τ and τ ∗ .

Original: Particularly, we have
Revised: Particularly, we have

Original: [Equation]
Revised: [Equation]

Original: Rewrite τ = ατ ∗ + ζ , where ζ ∈ span (cid:0) Q \ τ ∗ (cid:1) , then
Revised: Rewrite τ = ατ ∗ + ζ , where ζ ∈ span (cid:0) Q \ τ ∗ (cid:1) , then

Original: [Equation]
Revised: [Equation]

Original: Recall the optimality of τ ∗ , i.e. , D φ (cid:16)(cid:0) T pγ,µ 0 ◦ τ (cid:1) ∗ || p · τ ∗ (cid:17) = 0 , we have
Revised: Recall the optimality of τ ∗ , i.e. , D φ (cid:16)(cid:0) T pγ,µ 0 ◦ τ (cid:1) [Equation] || p · τ ∗ (cid:17) = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: C. E RROR D ECOMPOSITION
Revised: C. E RROR D ECOMPOSITION

Original: We start with the following error decomposition:
Revised: We start with the following error decomposition:

Original: [Equation]
Revised: [Equation]

Original: • For (cid:15) 1 , we have
Revised: • For (cid:15) 1 , we have

Original: [Equation]
Revised: [Equation]

Original: We consider the terms one-by-one.
Revised: We consider the terms one-by-one.

Original: By deﬁnition, we have
Revised: By deﬁnition, we have

Original: [Equation]
Revised: [Equation]

Original: which is induced by introducing F for dual approximation.
Revised: which is induced by introducing F for dual approximation.

Original: For the third term L (ˆ τ ∗H ) − J (ˆ τ ∗H ) , we have
Revised: For the third term L (ˆ τ ∗H ) − J (ˆ τ ∗H ) , we have

Original: [Equation]
Revised: [Equation]

Original: ,u ∈ U J (ˆ τ ∗H , u, f )
Revised: u ∈ U J (ˆ τ ∗H , u, f )

Original: − max f,u ∈ U J (ˆ τ ∗H , u, f ) ≤ 0 .
Revised: − max f,u ∈ U J (ˆ τ ∗H , u, f ) ≤ 0 .

Original: For the termL
Revised: For the term

Original: [Equation]
Revised: [Equation]

Original: (22) where we deﬁne (cid:15) est := sup τ ∈H ,f ∈F,u ∈ U (cid:12)(cid:12)(cid:12) J ( τ, u, f ) − (cid:98) J ( τ, u, f ) (cid:12)(cid:12)(cid:12) .
Revised: 22) where we deﬁne (cid:15) est := sup τ ∈H [Equation] ∈F

Revised: [Equation]

Revised: u ∈ U

Revised: [Equation]

Revised: , u

Revised: [Equation]

Revised: f ) −

Revised: [Equation]

Revised: u

Revised: [Equation]

Revised: f

Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we can now bound (cid:15) 1 as
Revised: we can now bound (cid:15) 1 as

Original: [Equation]
Revised: [Equation]

Original: • For (cid:15) 2 , we have
Revised: • For (cid:15) 2 , we have

Original: [Equation]
Revised: [Equation]

Original: We consider the terms from right to left.
Revised: We consider the terms from right to left.

Original: For the term J ( τ ∗H ) − J ( τ ∗ ) , we have
Revised: For the term J ( τ ∗H ) − J ( τ ∗ ) , we have

Original: [Equation]
Revised: [Equation]

Original: which is induced by restricting the function space to H .
Revised: which is induced by restricting the function space to H .

Original: The second term is nonpositive, due to the optimality of ( u ∗ , f ∗ ) .
Revised: The second term is nonpositive, due to the optimality of ( u ∗ , f ∗ ) .

Original: The ﬁnal inequality comes from the fact that J ( τ, u, f ) is C φ,C,λLipschitz w.r.t.
Revised: The ﬁnal inequality comes from the fact that J ( τ, u, f ) is C φ,C,λLipschitz w.r.t.

Original: τ .
Revised: τ .

Original: For the term L (ˆ τ ∗H ) − J ( τ ∗H ) , by deﬁnition
Revised: For the term L (ˆ τ ∗H ) − J ( τ ∗H ) , by deﬁnition

Original: [Equation]
Revised: [Equation]

Original: For the term L (ˆ τ ∗H ) − L ( τ ∗H ) , we have
Revised: For the term L (ˆ τ ∗H ) − L ( τ ∗H ) , we have

Original: [Equation]
Revised: [Equation]

Original: where the second term is nonpositive, thanks to the optimality of ˆ τ ∗H .
Revised: where the second term is nonpositive, thanks to the optimality of ˆ τ ∗H .

Original: Finally, for the term J (ˆ τ ∗H ) − J ( τ ∗H ) , using the same argument in Equation 21, we have
Revised: Finally, for the term J (ˆ τ ∗H ) − J ( τ ∗H ) , using the same argument in Equation 21, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can bound (cid:15) 2 by
Revised: Therefore, we can bound (cid:15) 2 by

Original: [Equation]
Revised: [Equation]

Original: In sum, we have
Revised: In sum, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: In the following sections, we will bound the (cid:15) est and ˆ (cid:15) opt .
Revised: In the following sections, we will bound the (cid:15) est and ˆ (cid:15) opt .

Original: C. S TATISTICAL E RROR
Revised: C. S TATISTICAL E RROR

Original: In this section, we analyze the statistical error
Revised: In this section, we analyze the statistical error

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: We mainly focus on the batch RL setting with i.i.d.
Revised: We mainly focus on the batch RL setting with i.i.d.

Original: samples D
Revised: samples D

Original: =
Revised: =

Original: [( x i , x (cid:48) i )]
Revised: [( x i , x (cid:48) i )]

Original: Ni =1 ∼ T p ( x, x (cid:48) ) , which has been studied by previous authors (e.g., Sutton et al., 2012; Nachum et al., 2019).
Revised: Ni =1 ∼ T p ( x, x (cid:48) ) , which has been studied by previous authors (e.g., Sutton et al., 2012; Nachum et al., 2019).

Original: However, as discussed in the literature (Antos et al., 2008; Lazaric et al., 2012; Dai et al., 2018; Nachum et al., 2019), using the blocking technique of Yu (1994), the statistical error provided here can be generalized to β -mixing samples in a single sample path.
Revised: However, as discussed in the literature (Antos et al., 2008; Lazaric et al., 2012; Dai et al., 2018; Nachum et al., 2019), using the blocking technique of Yu (1994), the statistical error provided here can be generalized to β -mixing samples in a single sample path.

Original: We omit this generalization for the sake of expositional simplicity.
Revised: We omit this generalization for the sake of expositional simplicity.

Original: To bound the (cid:15) est , we follow similar arguments by Dai et al.
Revised: To bound the (cid:15) est , we follow similar arguments by Dai et al.

Original: ; Nachum et al.
Revised: ; Nachum et al.

Original: (2019) via the covering number.
Revised: (2019) via the covering number.

Original: For completeness, the deﬁnition is given below.
Revised: For completeness, the deﬁnition is given below.

Original: The Pollard’s tail inequality bounds the maximum deviation via the covering number of a function class:
Revised: The Pollard’s tail inequality bounds the maximum deviation via the covering number of a function class:

Original: Lemma 4 (Pollard (2012)) Let G be a permissible class of Z → [ − M, M ] functions and { Z i } Ni = are i.i.d.
Revised: Lemma 4 (Pollard (2012)) Let G be a permissible class of Z → [ − M, M ] functions and { Z i } Ni = are i.i.d.

Original: samples from some distribution.
Revised: samples from some distribution.

Original: Then, for any given (cid:15) > 0 ,
Revised: Then, for any given (cid:15) > 0 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The covering number can then be bounded in terms of the function class’s pseudo-dimension:
Revised: The covering number can then be bounded in terms of the function class’s pseudo-dimension:

Original: Lemma 5 (Haussler (1995), Corollary 3)
Revised: Lemma 5 (Haussler (1995), Corollary 3)

Original: For any set X , any points x 1: N ∈ X N , any class F of functions on X taking values in [0 , M ] with pseudo-dimension D F < ∞ , and any (cid:15) > 0 ,
Revised: For any set X , any points x 1: N ∈ X N , any class F of functions on X taking values in [0 , M ] with pseudo-dimension D F < ∞ , and any (cid:15) > 0 ,

Original: [Equation]
Revised: [Equation]

Original: The statistical error (cid:15) est can be bounded using these lemmas.
Revised: The statistical error (cid:15) est can be bounded using these lemmas.

Original: Lemma 6 (Stochastic error)
Revised: Lemma 6 (Stochastic error)

Original: Under the Assumption 2, if φ ∗ is κ -Lipschitz continuous and the psuedo-dimension of H and F are ﬁnite, with probability at least 1 − δ , we have
Revised: Under the Assumption 2, if φ ∗ is κ -Lipschitz continuous and the psuedo-dimension of H and F are ﬁnite, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof The proof works by verifying the conditions in Lemma 4 and computing the covering number.
Revised: Proof The proof works by verifying the conditions in Lemma 4 and computing the covering number.

Original: Denote the h τ,u,f ( x, x (cid:48) )
Revised: Denote the h τ,u,f ( x, x (cid:48) )

Original: = (1 − γ ) f
Revised: = (1 − γ ) f

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: + γτ ( x ) f ( x (cid:48) )
Revised: + γτ ( x ) f ( x (cid:48) )

Original: − τ ( x ) φ ∗ ( f ( x ))
Revised: − τ ( x ) φ ∗ ( f ( x ))

Original: + λuτ ( x ) − λu − λ 1 2 u 2 , we will apply Lemma 4 with Z = Ω × Ω , Z i =
Revised: + λuτ ( x ) − λu − λ 1 2 u 2 , we will apply Lemma 4 with Z = Ω × Ω , Z i =

Original: ( x i , x (cid:48) i ) , and G = h H×F× U .
Revised: ( x i , x (cid:48) i ) , and G = h H×F× U .

Original: We check the boundedness of h ζ,u,f ( x, x (cid:48) ) .
Revised: We check the boundedness of h ζ,u,f ( x, x (cid:48) ) .

Original: Based on Assumption 2, we only consider the τ ∈ H and u ∈ U bounded by C and C + 1 .
Revised: Based on Assumption 2, we only consider the τ ∈ H and u ∈ U bounded by C and C + 1 .

Original: We also rectify the (cid:107) f (cid:107) ∞ ≤ C .
Revised: We also rectify the (cid:107) f (cid:107) ∞ ≤ C .

Original: Then, we can bound the (cid:107) h (cid:107) ∞ : (cid:107) h τ,u,f (cid:107) ∞ ≤ (1 + (cid:107) τ (cid:107) ∞ ) (cid:107) f (cid:107) ∞
Revised: Then, we can bound the (cid:107) h (cid:107) ∞ : (cid:107) h τ,u,f (cid:107) ∞ ≤ (1 + (cid:107) τ (cid:107) ∞ ) (cid:107) f (cid:107) ∞

Original: + (cid:107) τ (cid:107) ∞ (cid:18)
Revised: + (cid:107) τ (cid:107) ∞ (cid:18)

Original: [Equation]
Revised: [Equation]

Original: where C φ = max t ∈ [ − C,C ] − φ ∗ ( t ) .
Revised: where C φ = max t ∈ [ − C,C ] − φ ∗ ( t ) .

Original: Thus, by Lemma
Revised: Thus, by Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Next, we check the covering number of G .
Revised: Next, we check the covering number of G .

Original: Firstly, we bound the distance in G ,
Revised: Firstly, we bound the distance in G ,

Original: [Equation]
Revised: [Equation]

Original: which leads to
Revised: which leads to

Original: [Equation]
Revised: [Equation]

Original: For the set U
Revised: For the set U

Original: =
Revised: =

Original: [ − C −
Revised: [ − C −

Original: [Equation]
Revised: [Equation]

Original: C + 1]
Revised: C + 1]

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Denote the pseudo-dimension of H and F as D H and D F , respectively, we have
Revised: Denote the pseudo-dimension of H and F as D H and D F , respectively, we have

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: , where D 1 = D H + D F + 1 and
Revised: , where D 1 = D H + D F + 1 and

Original: [Equation]
Revised: [Equation]

Original: Combine this result with Equation 23, we obtain the bound for the statistical error:
Revised: Combine this result with Equation 23, we obtain the bound for the statistical error:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Setting (cid:15)
Revised: Setting (cid:15)

Original: [Equation]
Revised: [Equation]

Original: with C 2 = max (cid:16) (8 C 1 ) 2 D 1 , 512 MD 1 , 512 M, 1 (cid:17) , we have
Revised: with C 2 = max (cid:16) (8 C 1 ) 2 D 1 , 512 MD 1 , 512 M, 1 (cid:17) , we have

Original: [Equation]
Revised: [Equation]

Original: C. O PTIMIZATION E RROR
Revised: C. O PTIMIZATION E RROR

Original: In this section, we investigate the optimization error
Revised: In this section, we investigate the optimization error

Original: [Equation]
Revised: [Equation]

Original: Notice our estimator min τ ∈H max f ∈F ,u ∈ U (cid:98) J ( τ, u, f ) is compatible with different parametrizations for ( H , F ) and different optimization algorithms, the optimization error will be different.
Revised: Notice our estimator min τ ∈H max f ∈F ,u ∈ U (cid:98) J ( τ, u, f ) is compatible with different parametrizations for ( H , F ) and different optimization algorithms, the optimization error will be different.

Original: For the general neural network for ( τ, f ) , although there are several progress recently (Lin et al., 2018;
Revised: For the general neural network for ( τ, f ) , although there are several progress recently (Lin et al., 2018;

Original: Jin et al., 2019; Lin et al., 2019) about the convergence to a stationary point or local minimum, it remains a largely open problem to quantify the optimization error, which is out of the scope of this paper.
Revised: Jin et al., 2019; Lin et al., 2019) about the convergence to a stationary point or local minimum, it remains a largely open problem to quantify the optimization error, which is out of the scope of this paper.

Original: Here, we mainly discuss the convergence rate with tabular, linear and kernel parametrization for ( τ, f ) .
Revised: Here, we mainly discuss the convergence rate with tabular, linear and kernel parametrization for ( τ, f ) .

Original: Particularly, we consider the linear parametrization particularly, i.e. , τ ( x ) = 
Revised: Particularly, we consider the linear parametrization particularly, i.e. , τ ( x ) = w (cid:62)

Original: σ (cid:0) w (cid:62) τ ψ  ( x ) (cid:1) , f ( x ) = w (cid:62) f ψ  ( x ) , and σ ( · ) :
Revised: τ ψ τ ( x ) with { w τ , ψ τ ( x ) ≥ 0 } and f ( x ) = w (cid:62)

Original: R → R + is convex. There are many choices of the σ ( · ) , e.g. , exp ( · ) , log (1 + exp ( · )) and ( · )2 .
Revised: f ψ f ( x ) .

Original: Obviously, even with such nonlinear mapping, the (cid:98) J ( τ, u, f ) is still convex-concave w.r.t ( w τ , w f , u ) by the convex composition rule.
Revised: With such parametrization, the (cid:98) J ( τ, u, f ) is still convexconcave w.r.t ( w τ , w f , u ) .

Original: We can bound the ˆ (cid:15) opt by the primal-dual gap (cid:15) gap : ˆ (cid:15) opt
Revised: We can bound the ˆ (cid:15) opt by the primal-dual gap (cid:15) gap : ˆ (cid:15) opt

Original: [Equation]
Revised: [Equation]

Original: With vanilla SGD, we have (cid:15) gap
Revised: With vanilla SGD, we have (cid:15) gap

Original: = O (cid:16) 1 √ T (cid:17) , where T is the optimization steps (Nemirovski et al., 2009).
Revised: = O (cid:16) 1 √ T (cid:17) , where T is the optimization steps (Nemirovski et al., 2009).

Original: Therefore, (cid:15) opt =
Revised: Therefore, (cid:15) opt =

Original: E [ˆ (cid:15) opt ] =
Revised: E [ˆ (cid:15) opt ] =

Original: O (cid:16) 1 √ T (cid:17) , where the E [ · ] is taken w.r.t.
Revised: O (cid:16) 1 √ T (cid:17) , where the E [ · ] is taken w.r.t.

Original: randomness in SGD.
Revised: randomness in SGD.

Original: C. C OMPLETE E RROR A NALYSIS
Revised: C. C OMPLETE E RROR A NALYSIS

Original: We are now ready to state the main theorm in a precise way:
Revised: We are now ready to state the main theorm in a precise way:

Original: Theorem
Revised: Theorem

Original: Under Assumptions 2 and 1 , the stationary distribution µ exists, i.e. , max f ∈F ∗ E T ◦ µ [ f ] − E µ [ φ ∗ ( f )]
Revised: Under Assumptions 2 and 1 , the stationary distribution µ exists, i.e. , max f ∈F ∗ E T ◦ µ [ f ] − E µ [ φ ∗ ( f )]

Original: = 0 .
Revised: = 0 .

Original: If the φ ∗ ( · ) is κ -Lipschitz continuous, (cid:107) f (cid:107) ∞ ≤ C < ∞ , ∀ f ∈ F ∗ , and the psuedo-dimension of H and F are ﬁnite, the error between the GenDICE estimate to τ ∗ ( x ) =
Revised: If the φ ∗ ( · ) is κ -Lipschitz continuous, (cid:107) f (cid:107) ∞ ≤ C < ∞ , ∀ f ∈ F ∗ , and the psuedo-dimension of H and F are ﬁnite, the error between the GenDICE estimate to τ ∗ ( x ) =

Original: u ( x ) p ( x ) is bounded by
Revised: u ( x ) p ( x ) is bounded by

Original: [Equation]
Revised: [Equation]

Original: where E [ · ] is w.r.t.
Revised: where E [ · ] is w.r.t.

Original: the randomness in sample D and in the optimization algorithms.
Revised: the randomness in sample D and in the optimization algorithms.

Original: (cid:15) opt is the optimization error, and (cid:15) approx ( F , H ) is the approximation induced by ( F , H ) for parametrization of ( τ, f ) .
Revised: (cid:15) opt is the optimization error, and (cid:15) approx ( F , H ) is the approximation induced by ( F , H ) for parametrization of ( τ, f ) .

Original: Proof We have the total error as E
Revised: Proof We have the total error as E

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) approx :
Revised: where (cid:15) approx :

Original: = 2 C T ,κ,λ (cid:15) approx
Revised: = 2 C T ,κ,λ (cid:15) approx

Original: ( F ) + C φ, [Equation] ,λ [Equation]
Revised: ( F ) + C φ,C,λ (cid:15) approx

Original: approx ( H ) .
Revised: ( H ) .

Original: For (cid:15) opt , we can apply the results for
Revised: For (cid:15) opt , we can apply the results for

Original: SGD in Appendix C.3.
Revised: SGD in Appendix C.3.

Original: We can bound the E [ (cid:15) est ] by Lemma 6.
Revised: We can bound the E [ (cid:15) est ] by Lemma 6.

Original: Speciﬁcally, we have
Revised: Speciﬁcally, we have

Original: [Equation]
Revised: [Equation]

Original:  by setting δ = 1 √ N .
Revised: N (cid:33) , by setting δ = 1 √ N .

Original: Plug all these bounds into Equation 25, we achieve the conclusion.
Revised: Plug all these bounds into Equation 25, we achieve the conclusion.

Original: D E XPERIMENTAL S ETTINGS D. T ABULAR C ASE
Revised: D E XPERIMENTAL S ETTINGS D. T ABULAR C ASE

Original: For the Taxi domain, we follow the same protocol as used in Liu et al.
Revised: For the Taxi domain, we follow the same protocol as used in Liu et al.

Original: The behavior and target policies are also taken from Liu et al.
Revised: The behavior and target policies are also taken from Liu et al.

Original: (2018) (referred in their work as the behavior policy for α = 0 ).
Revised: (2018) (referred in their work as the behavior policy for α = 0 ).

Original: We use a similar taxi domain, where a grid size of 5 × 5 yields 2000 states in total ( 25 × 16 × 5 , cor- responding to 25 taxi locations, 16 passenger appearance status and 5 taxi status).
Revised: We use a similar taxi domain, where a grid size of 5 × 5 yields 2000 states in total ( 25 × 16 × 5 , corresponding to 25 taxi locations, 16 passenger appearance status and 5 taxi status).

Original: We set our target policy as the ﬁnal policy π ∗ after running Q-learning (Sutton & Barto, 1998) for 1000 iterations, and set another policy π + after 950 iterations as our base policy.
Revised: We set our target policy as the ﬁnal policy π ∗ after running Q-learning (Sutton & Barto, 1998) for 1000 iterations, and set another policy π + after 950 iterations as our base policy.

Original: The behavior policy is a mixture policy controlled by α as π = (1 − α )
Revised: The behavior policy is a mixture policy controlled by α as π = (1 − α )

Original: π ∗ + απ + , i.e. , the larger α is, the behavior policy is more close to the target policy.
Revised: π ∗ + απ + , i.e. , the larger α is, the behavior policy is more close to the target policy.

Original: In this setting, we solve for the optimal stationary ratio τ exactly using matrix operations.
Revised: In this setting, we solve for the optimal stationary ratio τ exactly using matrix operations.

Original: Since Liu et al.
Revised: Since Liu et al.

Original: (2018) perform a similar exact solve for | S | variables µ ( s ) , for better comparison we also perform our exact solve with respect to | S | variables τ ( s ) .
Revised: (2018) perform a similar exact solve for | S | variables µ ( s ) , for better comparison we also perform our exact solve with respect to | S | variables τ ( s ) .

Original: Speciﬁcally, the ﬁnal objective of importance sampling will require knowledge of the importance weights µ ( a | s ) /p
Revised: Speciﬁcally, the ﬁnal objective of importance sampling will require knowledge of the importance weights µ ( a | s ) /p

Original: ( a | s ) .
Revised: ( a | s ) .

Original: [Table]
Revised: [Table]

Original: For ofﬂine PageRank, the graph statistics are illustrated in Table 1, and the degree statistics and graph visualization are shown in Figure 7.
Revised: For ofﬂine PageRank, the graph statistics are illustrated in Table 1, and the degree statistics and graph visualization are shown in Figure 8.

Original: For the BarabasiAlbert (BA) Graph, it begins with an initial connected network of m 0 nodes in the network.
Revised: For the BarabasiAlbert (BA) Graph, it begins with an initial connected network of m 0 nodes in the network.

Original: Each new node is connected to m ≤ m 0 existing nodes with a probability that is proportional to the number of links that the existing nodes already have.
Revised: Each new node is connected to m ≤ m 0 existing nodes with a probability that is proportional to the number of links that the existing nodes already have.

Original: Intuitively, heavily linked nodes (‘hubs’) tend to quickly accumulate even more links, while nodes with only a few links are unlikely to be chosen as the destination for a new link.
Revised: Intuitively, heavily linked nodes (‘hubs’) tend to quickly accumulate even more links, while nodes with only a few links are unlikely to be chosen as the destination for a new link.

Original: The new nodes have a ‘preference’ to attach themselves to the already heavily linked nodes.
Revised: The new nodes have a ‘preference’ to attach themselves to the already heavily linked nodes.

Original: For two real-world graphs, it is built upon the real-world citation networks.
Revised: For two real-world graphs, it is built upon the real-world citation networks.

Original: In our experiments, the weights of the BA graph is randomly drawn from a standard Gaussian distribution with normalization to ensure the property of the transition matrix.
Revised: In our experiments, the weights of the BA graph is randomly drawn from a standard Gaussian distribution with normalization to ensure the property of the transition matrix.

Original: The ofﬂine data is collected by a random walker on the graph, which consists the initial state and next state in a single trajectory.
Revised: The ofﬂine data is collected by a random walker on the graph, which consists the initial state and next state in a single trajectory.

Original: In experiments, we vary the number of off-policy samples to validate the effectiveness of GenDICE with limited ofﬂine samples provided.
Revised: In experiments, we vary the number of off-policy samples to validate the effectiveness of GenDICE with limited ofﬂine samples provided.

Original: [Figure]
Revised: [Figure]

Original: D. C ONTINUOUS C ASE
Revised: D. C ONTINUOUS C ASE

Original: We use the Cartpole, Reacher and HalfCheetah tasks as given by OpenAI Gym.
Revised: We use the Cartpole, Reacher and HalfCheetah tasks as given by OpenAI Gym.

Original: In importance sampling, we learn a neural network policy via behavior cloning, and use its probabilities for computing importance weights π ∗ ( a | s ) /π
Revised: In importance sampling, we learn a neural network policy via behavior cloning, and use its probabilities for computing importance weights π ∗ ( a | s ) /π

Original: ( a | s ) .
Revised: ( a | s ) .

Original: All neural networks are feed-forward with two hidden layers of dimension 64 and tanh activations.
Revised: All neural networks are feed-forward with two hidden layers of dimension 64 and tanh activations.

Original: Discrete Control Tasks We modify the Cartpole task to be inﬁnite horizon: We use the same dynamics as in the original task but change the reward to be − 1 if the original task returns a termination (when the pole falls below some threshold) and 1 otherwise.
Revised: Discrete Control Tasks We modify the Cartpole task to be inﬁnite horizon: We use the same dynamics as in the original task but change the reward to be − 1 if the original task returns a termination (when the pole falls below some threshold) and 1 otherwise.

Original: We train a policy on this task with standard Deep Q-Learning (Mnih et al., 2013) until convergence.
Revised: We train a policy on this task with standard Deep Q-Learning (Mnih et al., 2013) until convergence.

Original: We then deﬁne the target policy π ∗ as a weighted combination of this pre-trained policy (weight 0 . 7 ) and a uniformly random policy (weight 0 . 3 ).
Revised: We then deﬁne the target policy π ∗ as a weighted combination of this pre-trained policy (weight 0 . 7 ) and a uniformly random policy (weight 0 . 3 ).

Original: The behavior policy π for a speciﬁc 0 ≤ α ≤ 1 is taken to be a weighted combination of the pre-trained policy (weight 0 .
Revised: The behavior policy π for a speciﬁc 0 ≤ α ≤ 1 is taken to be a weighted combination of the pre-trained policy (weight 0 .

Original: 55 + 0 .
Revised: 55 + 0 .

Original: 15 α ) and a uniformly random policy (weight 0 .
Revised: 15 α ) and a uniformly random policy (weight 0 .

Original: 45 − 0 .
Revised: 45 − 0 .

Original: 15 α ).
Revised: 15 α ).

Original: We train each stationary distribution correction estimation method using the Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from { 0 . 0001 , 0 . 0003 , 0 . 001 , 0 . 003 } and choose the best one as 0 . 0003 .
Revised: We train each stationary distribution correction estimation method using the Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from { 0 . 0001 , 0 . 0003 , 0 . 001 , 0 . 003 } and choose the best one as 0 . 0003 .

Original: Continuous Control Tasks For the Reacher task, we train a deterministic policy until convergence via DDPG (Lillicrap et al., 2015).
Revised: Continuous Control Tasks For the Reacher task, we train a deterministic policy until convergence via DDPG (Lillicrap et al., 2015).

Original: We deﬁne the target policy π as a Gaussian with mean given by the pre-trained policy and standard deviation given by 0 . 1 .
Revised: We deﬁne the target policy π as a Gaussian with mean given by the pre-trained policy and standard deviation given by 0 . 1 .

Original: The behavior policy π b for a speciﬁc 0 ≤ α ≤ 1 is taken to be a Gaussian with mean given by the pre-trained policy and standard deviation given by 0 .
Revised: The behavior policy π b for a speciﬁc 0 ≤ α ≤ 1 is taken to be a Gaussian with mean given by the pre-trained policy and standard deviation given by 0 .

Original: 4 − 0 .
Revised: 4 − 0 .

Original: 3 α .
Revised: 3 α .

Original: We train each stationary distribution correction estimation method using the
Revised: We train each stationary distribution correction estimation method using the

Original: Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from { 0 . 0001 , 0 . 0003 , 0 . 001 , 0 . 003 } and the optimal learning rate found was 0 . 003 ).
Revised: Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from { 0 . 0001 , 0 . 0003 , 0 . 001 , 0 . 003 } and the optimal learning rate found was 0 . 003 ).

Original: For the HalfCheetah task, we also train a deterministic policy until convergence via DDPG (Lillicrap et al., 2015).
Revised: For the HalfCheetah task, we also train a deterministic policy until convergence via DDPG (Lillicrap et al., 2015).

Original: We deﬁne the target policy π as a Gaussian with mean given by the pre-trained policy and standard deviation given by 0 . 1 .
Revised: We deﬁne the target policy π as a Gaussian with mean given by the pre-trained policy and standard deviation given by 0 . 1 .

Original: The behavior policy π b for a speciﬁc 0 ≤ α ≤ 1 is taken to be a Gaussian with mean given by the pre-trained policy and standard deviation given by 0 .
Revised: The behavior policy π b for a speciﬁc 0 ≤ α ≤ 1 is taken to be a Gaussian with mean given by the pre-trained policy and standard deviation given by 0 .

Original: 2 − 0 .
Revised: 2 − 0 .

Original: 1 α .
Revised: 1 α .

Original: We train each stationary distribution correction estimation method using the Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from { 0 . 0001 , 0 . 0003 , 0 . 001 , 0 . 003 } and the optimal learning rate found was 0 . 003 .
Revised: We train each stationary distribution correction estimation method using the Adam optimizer with batches of size 2048 and learning rates chosen using a hyperparameter search from { 0 . 0001 , 0 . 0003 , 0 . 001 , 0 . 003 } and the optimal learning rate found was 0 . 003 .

Original: E A DDITIONAL E XPERIMENTS E. OPE FOR D ISCRETE C ONTROL
Revised: E A DDITIONAL E XPERIMENTS E. OPE FOR D ISCRETE C ONTROL

Original: On the discrete control task, we modify the Cartpole task to be inﬁnite horizon: the original dynamics is used but with a modiﬁed reward function: the agent will receive − 1 if the environment returns a termination ( i.e. , the pole falls below some threshold) and 1 otherwise.
Revised: On the discrete control task, we modify the Cartpole task to be inﬁnite horizon: the original dynamics is used but with a modiﬁed reward function: the agent will receive − 1 if the environment returns a termination ( i.e. , the pole falls below some threshold) and 1 otherwise.

Original: As shown in Figure 3, our method shows competitive results with IS and Model-Based in average reward case, but our proposed method ﬁnally outperforms these two methods in terms of log MSE loss.
Revised: As shown in Figure 3, our method shows competitive results with IS and Model-Based in average reward case, but our proposed method ﬁnally outperforms these two methods in terms of log MSE loss.

Original: Speciﬁcally, it is relatively difﬁcult to ﬁt a policy with data collected by multiple policies, which renders the poor performance of IS.
Revised: Speciﬁcally, it is relatively difﬁcult to ﬁt a policy with data collected by multiple policies, which renders the poor performance of IS.

Original: policy closer to the target policy; the same in other ﬁgures); M1:

Original: α = [0 . 0 , 0 .

Original: 33] ; M2: α = [0 . 0 , 0 .

Original: E. A DDITIONAL R ESULTS ON C ONTINUOUS C ONTROL
Revised: E. A DDITIONAL R ESULTS ON C ONTINUOUS C ONTROL

Original: In this section, we show more results on the continuous control tasks, i.e. , HalfCheetah and Reacher.
Revised: In this section, we show more results on the continuous control tasks, i.e. , HalfCheetah and Reacher.

Original: Figure 9 shows the log MSE towards training steps, and GenDICE outperforms other baselines with different behavior policies.
Revised: Figure 10 shows the log MSE towards training steps, and GenDICE outperforms other baselines with different behavior policies.

Original: Figure 10 better illustrates how our method beat other baselines, and can accurately estimate the reward of the target policy.
Revised: Figure 11 better illustrates how our method beat other baselines, and can accurately estimate the reward of the target policy.

Original: Besides, Figure 11 shows GenDICE gives better reward estimation of the target policy.
Revised: Besides, Figure 12 shows GenDICE gives better reward estimation of the target policy.

Original: In these ﬁgures, the left three ﬁgures show the performance with off-policy dataset collected by single behavior policy from more difﬁcult to easier tasks.
Revised: In these ﬁgures, the left three ﬁgures show the performance with off-policy dataset collected by single behavior policy from more difﬁcult to easier tasks.

Original: The right two ﬁgures show the results, where off-policy dataset collected by multiple behavior policies.
Revised: The right two ﬁgures show the results, where off-policy dataset collected by multiple behavior policies.

Original: Figure 12 shows the ablation study results in terms of estimated rewards.
Revised: Figure 13 shows the ablation study results in terms of estimated rewards.

Original: The left two ﬁgures shows the effects of different learning rate.
Revised: The left two ﬁgures shows the effects of different learning rate.

Original: When α = 0 .
Revised: When α = 0 .

Original: 33 , i.e. , the OPE tasks are relatively easier,
Revised: 33 , i.e. , the OPE tasks are relatively easier,

Original: GenDICE gets relatively good results in all learning rate settings.
Revised: GenDICE gets relatively good results in all learning rate settings.

Original: However, when α = 0 . 0 , i.e. , the estimation becomes more difﬁcult, only GenDICE in larger learning rate gets reasonable estimation.
Revised: However, when α = 0 . 0 , i.e. , the estimation becomes more difﬁcult, only GenDICE in larger learning rate gets reasonable estimation.

Original: Interestingly, we can see with larger learning rates, the performance becomes better, and when learning rate is 0 .
Revised: Interestingly, we can see with larger learning rates, the performance becomes better, and when learning rate is 0 .

Original: 001 with α = 0 . 

Revised: 001 with α = 0 .

Original: , the variance is very high, showing some cases the estimation becomes more accurate.
Revised: , the variance is very high, showing some cases the estimation becomes more accurate.

Original: The right three ﬁgures show different activation functions with different behavior policy.
Revised: The right three ﬁgures show different activation functions with different behavior policy.

Original: The square and softplus function works well; while the exponential function shows poor performance under some settings.
Revised: The square and softplus function works well; while the exponential function shows poor performance under some settings.

Original: In practice, we use the square function since its low variance and better performance in most cases.
Revised: In practice, we use the square function since its low variance and better performance in most cases.

Original: [Figure]
Revised: [Figure]

Original: row shows the estimated average step reward over training and different behavior policies (higher α corresponds to a behavior policy closer to the target policy.
Revised: to a behavior policy closer to the target policy.

Original: E. M ORE A BLATION S TUDY
Revised: E. C OMPARISON WITH SELF - NORMALIZATION TRICK

Original: Although any valid divergence between p · τ and (cid:0) T pγ,µ 0 ◦ τ (cid:1) in our estimator is consistent, which will always lead to the stationary distribution correction ratio asymptotically, and any λ > 0 will guarantee the normalization constraint, i.e. , E p [ τ] = 1 , as we discussed in main text, different We test the GenDICE with several other alternative divergences, e.g. , Wasserstein- 1 distance,
Revised: The self-normalization trick used in (Liu et al., 2018) encodes the normalization constraint in τ , while the principled optimization technique is considered in GenDICE. Further, the selfnormalization trick will lead to several disadvantages theoretically, in both statistical and computational aspects :

Original: Jensen-Shannon divergence, KL -divergence, Hellinger divergence, and MMD. To ensure the dual function to be 1 -Lipchitz, we add the gradient penalty. We use a learned Gaussian kernel in MMD, similar to Li et al. As we can see in Figure 13(a), with these different divergences, the proposed GenDICE estimator can always achieve signiﬁcantly better performance compared with the model-based estimator, showing that the GenDICE estimator is compatible with many different divergences.
Revised: i)

Original: Most of the divergences, with appropriate extra techniques to handle the difﬁculties in optimization and carefully tuning for extra parameters, can achieve similar performances, consistent with phenomena in the variants of GANs (Lucic et al., 2018). However, KL -divergence is an outlier, performing noticeably worse, which might be caused by the ill-behaved exp ( · ) in its conjugate function. The χ 2 -divergence and JS-divergence are better, which achieve good performances with fewer parameters to be tuned. The effect of the penalty weight λ is illustrated the in Figure 13(b).
Revised: It will generally not produce an unbiased solution.

Original: We vary the λ ∈ [0 . 1 , 5] with χ 2divergence in the GenDICE estimator. Within a large range of λ , the performances of the proposed GenDICE are quite consistent, which justiﬁes Theorem 1.
Revised: Although 1 |D| (cid:80) ( s,a ) ∈D τ ( s, a ) is an unbiased estimator for E [ τ ] , the plugin estimator

Original: The penalty multiplies with λ . Therefore, with λ increases, the variance of the stochastic gradient estimator also increases, which explains the variance increasing in large λ in Figure 13(b). In practice, λ = 1 is a reasonable choice for general cases. L o g K LD i v e
Revised: [Figure]

Original: r g e n c e
Revised: will be biased

Original: JS Wasserstein
Revised: [Figure]

Original: Hellinger MMD
Revised: ii) It will induce more computational cost.

Original: KL
Revised: Speciﬁcally, the self-normalized ratio will be in the form of τ

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: s,

Original: [Figure]
Revised: [Figure]

Revised: s,

Original: [Figure]
Revised: [Figure]

Original: Markov Chain Monte Carlo Classical MCMC (Brooks et al., 2011; Gelman et al., 2013) aims at sampling from µ π by iteratively simulting from the transition operator. It requires continuous interaction with the transition operator and heavy computational cost to update many particles.
Revised: ) , which requires to go through all the samples in training set D even for just estimating one stochastic gradient, and thus, is prohibitive for large dataset.

Original: Amor- tized SVGD (Wang & Liu, 2016) andAdversarial MCMC (Song et al., 2017; Li et al., 2019) alleviate this issue via combining with neural network, but they still interact with the transition operator directly, i.e. , in an on-policy setting.
Revised: Empirically, self-normalization is the most natural and the ﬁrst idea we tried during this project.

Original: The major difference of our GenDICE is the learning setting: we only access the off-policy dataset, and cannot sample from the transition operator.
Revised: We have some empirical results about this method in the OPR setting.

Original: The proposed
Revised: [Table]

Original: GenDICE leverages stationary density ratio estimation for approximating the stationary quantities, which distinct it from classical methods.
Revised: Despite the additional computational cost, it performs worse than the proposed regularization technique used in the current version of GenDICE.

Original: Density Ratio Estimation Density ratio estimation is a fundamental tool in machine learning and much related work exists.

Original: Classical density ratio estimation includes moment matching (Gretton et al.), probabilistic classiﬁcation (Bickel et al., 2007), and ratio matching (Nguyen et al., 2008;
Revised: Table 2 shows a comparison between selfnormalization and regularization on OPR with χ 2divergence for BA graph with 100 nodes, 10,000 ofﬂine samples, and 20 trials.

Original: Sugiyama et al., 2008; Kanamori et al., 2009). These classical methods focus on estimating the ratio between two distributions with samples from both of them, while GenDICE estimates the density ratio to a stationary distribution of a transition operator, from which even one sample is difﬁcult to obtain. PageRank Yao & Schuurmans (2013) developed a reverse-time RL framework for PageRank via solving a reverse Bellman equation, which is less sensitive to graph topology and shows faster adaptation with graph change. However, Yao & Schuurmans (2013) still considers the online manner, which is different with our OPR setting.
Revised: We stop the algorithms in the same running-time budget.