Original: Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Models
Revised: Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Models

Original: Abstract
Revised: Abstract

Original: We study the problem of off-policy evaluation (OPE) for episodic Partially Observable Markov Decision Processes (POMDPs) with continuous states.
Revised: We study the problem of off-policy evaluation (OPE) for episodic Partially Observable Markov Decision Processes (POMDPs) with continuous states.

Original: Motivated by the recently proposed proximal causal inference framework, we develop a non-parametric identiﬁcation result for estimating the policy value via a sequence of so-called V-bridge functions with the help of time-dependent proxy variables.
Revised: Motivated by the recently proposed proximal causal inference framework, we develop a non-parametric identiﬁcation result for estimating the policy value via a sequence of so-called V-bridge functions with the help of time-dependent proxy variables.

Original: We then develop a ﬁtted-Q-evaluation-type algorithm to estimate V-bridge functions recursively, where a non-parametric instrumental variable (NPIV) problem is solved at each step.
Revised: We then develop a ﬁtted-Q-evaluation-type algorithm to estimate V-bridge functions recursively, where a non-parametric instrumental variable (NPIV) problem is solved at each step.

Original: By analyzing this challenging sequential NPIV problem, we establish the ﬁnite-sample error bounds for estimating the V-bridge functions and accordingly that for evaluating the policy value, in terms of the sample size, length of horizon and so-called (local) measure of ill-posedness at each step.
Revised: By analyzing this challenging sequential NPIV problem, we establish the ﬁnite-sample error bounds for estimating the V-bridge functions and accordingly that for evaluating the policy value, in terms of the sample size, length of horizon and so-called (local) measure of ill-posedness at each step.

Original: To the best of our knowledge, this is the ﬁrst ﬁnite-sample error bound for OPE in POMDPs under non-parametric models.
Revised: To the best of our knowledge, this is the ﬁrst ﬁnite-sample error bound for OPE in POMDPs under non-parametric models.

Original:  Introduction
Revised:  Introduction

Original: In practical reinforcement learning (RL) a representation of the full state which makes the system
Revised: In practical reinforcement learning (RL) a representation of the full state which makes the system

Original: Markovian and therefore amenable to most existing RL algorithms is not known a priori.
Revised: Markovian and therefore amenable to most existing RL algorithms is not known a priori.

Original: Decision makers are often facing so-called partial observability of the state information, which signiﬁcantly hinders the task of RL.
Revised: Decision makers are often facing so-called partial observability of the state information, which signiﬁcantly hinders the task of RL.

Original: In general, agents have to maintain all historical information and establish a belief system on the hidden state for optimal decision making.
Revised: In general, agents have to maintain all historical information and establish a belief system on the hidden state for optimal decision making.

Original: A partially observable Markov decision process (POMDP) is often used to model the data generating process.
Revised: A partially observable Markov decision process (POMDP) is often used to model the data generating process.

Original: See examples in robotics [Rafferty et al., 2011], precision medicine [Tsoukalas et al., 2015], stochastic game
Revised: See examples in robotics [Rafferty et al., 2011], precision medicine [Tsoukalas et al., 2015], stochastic game

Original: [Hansen et al., 2004] and many others.
Revised: [Hansen et al., 2004] and many others.

Original: However, it is well known that learning optimal policies in POMDP is computationally intractable [Papadimitriou and Tsitsiklis, 1987].
Revised: However, it is well known that learning optimal policies in POMDP is computationally intractable [Papadimitriou and Tsitsiklis, 1987].

Original: The issue of partial observability becomes more serious in the batch setting, where agents are not able to actively collect additional data and further explore the environment.
Revised: The issue of partial observability becomes more serious in the batch setting, where agents are not able to actively collect additional data and further explore the environment.

Original: For example, standard off-policy evaluation (OPE) methods, which aim to learn a policy value from the batch data generated from some behavior policy, would fail to give a consistent estimate because of unobserved state variables.
Revised: For example, standard off-policy evaluation (OPE) methods, which aim to learn a policy value from the batch data generated from some behavior policy, would fail to give a consistent estimate because of unobserved state variables.

Original: Due to this practical concern, there is a recent line of research studying the OPE under the framework of a confounded POMDP, where the behavior policy to generate the batch data is allowed to depend on some unobserved state variables [e.g., Tennenholtz et al., 2020, Nair and Jiang, 2021, Bennett and Kallus, 2021, Shi et al., 2021].
Revised: Due to this practical concern, there is a recent line of research studying the OPE under the framework of a confounded POMDP, where the behavior policy to generate the batch data is allowed to depend on some unobserved state variables [e.g., Tennenholtz et al., 2020, Nair and Jiang, 2021, Bennett and Kallus, 2021, Shi et al., 2021].

Original: Their identiﬁcation results on the policy value are inspired by the negative controls or so-called proxy variables in the literature of causal inference [e.g., Miao et al., 2018a, Tchetgen Tchetgen et al., 2020].
Revised: Their identiﬁcation results on the policy value are inspired by the negative controls or so-called proxy variables in the literature of causal inference [e.g., Miao et al., 2018a, Tchetgen Tchetgen et al., 2020].

Original: A building block of these results is the existence of some bridge functions, namely Q / V -bridge or weight-bridge functions, which are projections of the Q / V -functions or importance weights deﬁned over the original state space onto the observation space.
Revised: A building block of these results is the existence of some bridge functions, namely Q / V -bridge or weight-bridge functions, which are projections of the Q / V -functions or importance weights deﬁned over the original state space onto the observation space.

Original: The corresponding statistical estimation of these bridge functions mainly relies on solving linear integral equations [e.g., Kress, 1989].
Revised: The corresponding statistical estimation of these bridge functions mainly relies on solving linear integral equations [e.g., Kress, 1989].

Original: Different from the tabular case studied by Tennenholtz et al.
Revised: Different from the tabular case studied by Tennenholtz et al.

Original: [2020] and Nair and Jiang [2021] and linear models studied by Shi et al.
Revised: [2020] and Nair and Jiang [2021] and linear models studied by Shi et al.

Original: [2021] theoretically, solving linear integral equations with non-parametric models in the continuous state/observation space are known to be challenging due to the potential ill-posedness [Chen and Reiss, 2011], leading to slow statistical convergence rates.
Revised: [2021] theoretically, solving linear integral equations with non-parametric models in the continuous state/observation space are known to be challenging due to the potential ill-posedness [Chen and Reiss, 2011], leading to slow statistical convergence rates.

Original: However, existing theoretical results developed by Bennett and
Revised: However, existing theoretical results developed by Bennett and

Original: Kallus [2021] and Shi et al.
Revised: Kallus [2021] and Shi et al.

Original: [2021] require fast enough convergence rates for these bridge function estimators in order to establish the asymptotic normality of their estimators for OPE, which could be illusive when the problem is seriously ill-posed under non-parametric models.
Revised: [2021] require fast enough convergence rates for these bridge function estimators in order to establish the asymptotic normality of their estimators for OPE, which could be illusive when the problem is seriously ill-posed under non-parametric models.

Original: This is different from the supervised learning where a fast enough convergence rate can be easily achieved under non-parametric models.
Revised: This is different from the supervised learning where a fast enough convergence rate can be easily achieved under non-parametric models.

Original: Therefore, to ﬁll this important theoretical gap, it is necessary to study the ﬁnite-sample performance of OPE of which bridge functions are estimated non-parametrically.
Revised: Therefore, to ﬁll this important theoretical gap, it is necessary to study the ﬁnite-sample performance of OPE of which bridge functions are estimated non-parametrically.

Original: Motivated by these, in this paper, we study the OPE for confounded and episodic POMDPs with continuous states, where we non-parametrically estimate V -bridge functions.
Revised: Motivated by these, in this paper, we study the OPE for confounded and episodic POMDPs with continuous states, where we non-parametrically estimate V -bridge functions.

Original: Our main contribution to the literature is three-fold.
Revised: Our main contribution to the literature is three-fold.

Original: First, relying on some time-dependent proxy variables, we establish a non-parametric identiﬁcation result for OPE using V -bridge functions for time-inhomogeneous confounded POMDPs.
Revised: First, relying on some time-dependent proxy variables, we establish a non-parametric identiﬁcation result for OPE using V -bridge functions for time-inhomogeneous confounded POMDPs.

Original: Based on the identiﬁcation result, we develop a new ﬁtted-Q-evaluation(FQE)- type approach to estimating V -bridge functions recursively and obtain an estimator for OPE based on the bridge function estimators.
Revised: Based on the identiﬁcation result, we develop a new ﬁtted-Q-evaluation(FQE)- type approach to estimating V -bridge functions recursively and obtain an estimator for OPE based on the bridge function estimators.

Original: At each step of our algorithm, we propose to ﬁt a non-parametric instrumental variable (NPIV) regression using a min-max estimation method, i.e., solving a linear integral equation with a non-parametric model.
Revised: At each step of our algorithm, we propose to ﬁt a non-parametric instrumental variable (NPIV) regression using a min-max estimation method, i.e., solving a linear integral equation with a non-parametric model.

Original: Our algorithm can be viewed as a sequential NPIV estimation, which is not well studied in the literature.
Revised: Our algorithm can be viewed as a sequential NPIV estimation, which is not well studied in the literature.

Original: Second and most importantly, we establish the ﬁnite-sample error bound for estimating V -bridge functions and accordingly that for evaluating the policy value, in terms of the sample size , length of horizon and (local) measure of ill-posedness at each step.
Revised: Second and most importantly, we establish the ﬁnite-sample error bound for estimating V -bridge functions and accordingly that for evaluating the policy value, in terms of the sample size , length of horizon and (local) measure of ill-posedness at each step.

Original: Unlike the well studied standard NPIV model in the econometrics literature [e.g., Ai and
Revised: Unlike the well studied standard NPIV model in the econometrics literature [e.g., Ai and

Original: Chen, 2003, Newey and Powell, 2003] where the response variable is directly observed, the response variable in our NPIV model at each step of the algorithm relies on the model estimate at its previous step.
Revised: Chen, 2003, Newey and Powell, 2003] where the response variable is directly observed, the response variable in our NPIV model at each step of the algorithm relies on the model estimate at its previous step.

Original: This difference makes our theoretical analysis substantially difﬁcult.
Revised: This difference makes our theoretical analysis substantially difﬁcult.

Original: By carefully characterizing the statistical error due to the NPIV estimation at each step and more importantly, its propagation effect on future estimates, we are able to establish the ﬁrst ﬁnite-sample result of OPE for confounded
Revised: By carefully characterizing the statistical error due to the NPIV estimation at each step and more importantly, its propagation effect on future estimates, we are able to establish the ﬁrst ﬁnite-sample result of OPE for confounded

Original: POMDPs under non-parametric models, which achieves a polynomial order over the length of horizon and sample size.
Revised: POMDPs under non-parametric models, which achieves a polynomial order over the length of horizon and sample size.

Original: Finally, our theoretical results on the sequential NPIV estimation are generally applicable to other sequential-type conditional moment restriction problems.
Revised: Finally, our theoretical results on the sequential NPIV estimation are generally applicable to other sequential-type conditional moment restriction problems.

Original: The development of the uniform ﬁnite-sample error bounds of the NPIV estimation, extending the pointwise result in the previous literature such as Dikkala et al.
Revised: The development of the uniform ﬁnite-sample error bounds of the NPIV estimation, extending the pointwise result in the previous literature such as Dikkala et al.

Original: [2020], may be of independent interest.
Revised: [2020], may be of independent interest.

Original:  Related Work
Revised:  Related Work

Original: Recently there is a surge of interest in studying OPE with unobserved variables in the sequential decision making problem.
Revised: Recently there is a surge of interest in studying OPE with unobserved variables in the sequential decision making problem.

Original: Speciﬁcally, Zhang and Bareinboim [2016] are among the ﬁrst who proposed the framework of confounded MDPs, which essentially considers i.i.d.
Revised: Speciﬁcally, Zhang and Bareinboim [2016] are among the ﬁrst who proposed the framework of confounded MDPs, which essentially considers i.i.d.

Original: confounders in the dynamic system and therefore preserves the Markovian property.
Revised: confounders in the dynamic system and therefore preserves the Markovian property.

Original: Along this direction, OPE methods are developed under various identiﬁcation conditions such as partial identiﬁcation using sensitivity analysis [Namkoong et al., 2020, Kallus and Zhou, 2020, Bruns-Smith, 2021], instrumental variable or mediator assisted OPE
Revised: Along this direction, OPE methods are developed under various identiﬁcation conditions such as partial identiﬁcation using sensitivity analysis [Namkoong et al., 2020, Kallus and Zhou, 2020, Bruns-Smith, 2021], instrumental variable or mediator assisted OPE

Original: [Liao et al., 2021, Li et al., 2021, Shi et al., 2022] and many others.
Revised: [Liao et al., 2021, Li et al., 2021, Shi et al., 2022] and many others.

Original: Another line of research focuses on more general
Revised: Another line of research focuses on more general

Original: confounded POMDP models , where the
Revised: confounded POMDP models , where the

Original: Markovian assumption is violated, under which several point estimation results were developed such as the aforementioned proxy variables related methods [Tennenholtz et al., 2020, Deaner, 2018,
Revised: Markovian assumption is violated, under which several point estimation results were developed such as the aforementioned proxy variables related methods [Tennenholtz et al., 2020, Deaner, 2018,

Original: Ying et al., 2021, Bennett and Kallus, 2021, Nair and Jiang, 2021, Shi et al., 2021], spectral methods in undercomplete POMDPs [Hsu et al., 2012, Anandkumar et al., 2014, Jin et al., 2020] and predictive state representation related methods [Littman and Sutton, 2001, Singh et al., 2012, Cai et al., 2022].
Revised: Ying et al., 2021, Bennett and Kallus, 2021, Nair and Jiang, 2021, Shi et al., 2021], spectral methods in undercomplete POMDPs [Hsu et al., 2012, Anandkumar et al., 2014, Jin et al., 2020] and predictive state representation related methods [Littman and Sutton, 2001, Singh et al., 2012, Cai et al., 2022].

Original: Our proposed method, which uses proxy variables for OPE, is closely related to those recently developed by Bennett and Kallus [2021], Shi et al.
Revised: Our proposed method, which uses proxy variables for OPE, is closely related to those recently developed by Bennett and Kallus [2021], Shi et al.

Original: [2021], and Ying et al.
Revised: [2021], and Ying et al.

Original: Bennett and
Revised: Bennett and

Original: Kallus [2021] and Ying et al.
Revised: Kallus [2021] and Ying et al.

Original: [2021] studied episodic POMDPs (or complex longitudinal studies) and mainly focused on developing asymptotic normality results of their policy value estimators.
Revised: [2021] studied episodic POMDPs (or complex longitudinal studies) and mainly focused on developing asymptotic normality results of their policy value estimators.

Original: Their results rely on some high level rate conditions on the bridge function estimation, which are unknown if they would be satisﬁed when using non-parametric models due to the aforementioned measure of ill-posedness.
Revised: Their results rely on some high level rate conditions on the bridge function estimation, which are unknown if they would be satisﬁed when using non-parametric models due to the aforementioned measure of ill-posedness.

Original: Shi et al.
Revised: Shi et al.

Original: [2021] mainly focused on time-homogeneous inﬁnite-horizon
Revised: [2021] mainly focused on time-homogeneous inﬁnite-horizon

Original: POMDPs and developed asymptotic normality for their estimators under similar high-level conditions, which therefore has the same issue.
Revised: POMDPs and developed asymptotic normality for their estimators under similar high-level conditions, which therefore has the same issue.

Original: Besides, while Shi et al.
Revised: Besides, while Shi et al.

Original: [2021] also established ﬁnite-sample bounds for their bridge function estimation and corresponding OPE, they only study the tabular case or linear/parametric models, where the issue of ill-posedness does not exist .
Revised: [2021] also established ﬁnite-sample bounds for their bridge function estimation and corresponding OPE, they only study the tabular case or linear/parametric models, where the issue of ill-posedness does not exist .

Original: In this paper, we provide a systematic investigation on the estimation of V -bridge functions and establish ﬁnite-sample guarantees for them and the corresponding OPE under non-parametric models.
Revised: In this paper, we provide a systematic investigation on the estimation of V -bridge functions and establish ﬁnite-sample guarantees for them and the corresponding OPE under non-parametric models.

Original: Speciﬁcally, we tackle the challenging episodic setting, where V -bridge functions are estimated sequentially.
Revised: Speciﬁcally, we tackle the challenging episodic setting, where V -bridge functions are estimated sequentially.

Original: Without carefully controlling the effect of ill-posedness at each step and its propagation effect on future steps, the estimation error for these V -bridge functions and also that for OPE could be exponentially large in terms of the length of horizon.
Revised: Without carefully controlling the effect of ill-posedness at each step and its propagation effect on future steps, the estimation error for these V -bridge functions and also that for OPE could be exponentially large in terms of the length of horizon.

Original: Motivated by the chaining argument in the empirical process theory, we successfully disentangle the effects of ill-posedness on the current step and future steps separately and thus establish ﬁnite-sample bounds for V -bridge functions and OPE both with a polynomial dependence on the length of horizon, which are new theoretical results we contribute to the literature.
Revised: Motivated by the chaining argument in the empirical process theory, we successfully disentangle the effects of ill-posedness on the current step and future steps separately and thus establish ﬁnite-sample bounds for V -bridge functions and OPE both with a polynomial dependence on the length of horizon, which are new theoretical results we contribute to the literature.

Original: Since our V -bridge function estimation can be formulated as a sequential NPIV problem, it is natually related to classical NPIV estimations, which have been extensively studied in the econometrics literature [see, e.g., Newey and Powell, 2003, Ai and Chen, 2003, 2012, Hall and Horowitz, 2005,
Revised: Since our V -bridge function estimation can be formulated as a sequential NPIV problem, it is natually related to classical NPIV estimations, which have been extensively studied in the econometrics literature [see, e.g., Newey and Powell, 2003, Ai and Chen, 2003, 2012, Hall and Horowitz, 2005,

Original: Chen and Reiss, 2011, Chen and Christensen, 2018, Darolles et al., 2011, Blundell et al., 2007, for earlier reference].
Revised: Chen and Reiss, 2011, Chen and Christensen, 2018, Darolles et al., 2011, Blundell et al., 2007, for earlier reference].

Original: Recently there is also a growing interest in the min-max estimation for NPIV models [see, e.g., Muandet et al., 2020, Dikkala et al., 2020, Hartford et al., 2017, for some recent developments].
Revised: Recently there is also a growing interest in the min-max estimation for NPIV models [see, e.g., Muandet et al., 2020, Dikkala et al., 2020, Hartford et al., 2017, for some recent developments].

Original: As commented before, existing theoretical results for standard NPIV models cannot be directly applied to our setting due to the sequential structure of our FQE algorithm, so we need to develop new theory to address our setting.
Revised: As commented before, existing theoretical results for standard NPIV models cannot be directly applied to our setting due to the sequential structure of our FQE algorithm, so we need to develop new theory to address our setting.

Original: Technically, in order to establish a polynomial-order ﬁnite-sample error bound over the length of horizon for OPE, which is particularly important in RL, we decompose the measure of ill-posedness at each step of our sequential NPIV estimation into two components: the so-called (local) measure of one-step transition ill-posedness and the standard (local) measure of ill-posedness [e.g., Chen and Pouzo, 2012].
Revised: Technically, in order to establish a polynomial-order ﬁnite-sample error bound over the length of horizon for OPE, which is particularly important in RL, we decompose the measure of ill-posedness at each step of our sequential NPIV estimation into two components: the so-called (local) measure of one-step transition ill-posedness and the standard (local) measure of ill-posedness [e.g., Chen and Pouzo, 2012].

Original: Thanks to this novel decomposition, the effect of the ﬁrst component on the estimation error of V -bridge functions and OPE is multiplicative but can be properly controlled while that of the second component could be large but is only cumulative.
Revised: Thanks to this novel decomposition, the effect of the ﬁrst component on the estimation error of V -bridge functions and OPE is multiplicative but can be properly controlled while that of the second component could be large but is only cumulative.

Original: See
Revised: See

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: Finally, we remark that Ai and Chen [2012] also studied the sequential NPIV estimation problem, where the non-parametric components are estimated jointly.
Revised: Finally, we remark that Ai and Chen [2012] also studied the sequential NPIV estimation problem, where the non-parametric components are estimated jointly.

Original: However, this method could be computationally inefﬁcient in RL with a long horizon.
Revised: However, this method could be computationally inefﬁcient in RL with a long horizon.

Original: More importantly, their results are built on the nested structure among conditional moment restriction models, which are not satisﬁed in our setting.
Revised: More importantly, their results are built on the nested structure among conditional moment restriction models, which are not satisﬁed in our setting.

Original:  Preliminaries and Notations
Revised:  Preliminaries and Notations

Original: In this section, we introduce the framework of discrete-time confounded POMDPs and its related OPE problem.
Revised: In this section, we introduce the framework of discrete-time confounded POMDPs and its related OPE problem.

Original: Consider an episodic and confounded POMDP denoted by M =
Revised: Consider an episodic and confounded POMDP denoted by M =

Original: ( S , U , A , T, P , r ) , with S and U as the observed and unobserved continuous state spaces respectively,
Revised: ( S , U , A , T, P , r ) , with S and U as the observed and unobserved continuous state spaces respectively,

Original: A as the discrete action space, T as the length of horizon, P = { P t } Tt =1 as the transition kernel over S × U × A to S × U , and r = { r t } Tt =1 as the reward function over S × U × A .
Revised: A as the discrete action space, T as the length of horizon, P = { P t } Tt =1 as the transition kernel over S × U × A to S × U , and r = { r t } Tt =1 as the reward function over S × U × A .

Original: S can also be treated as the observation space in the classical POMDP.
Revised: S can also be treated as the observation space in the classical POMDP.

Original: Then the process of M can be summarized as { S t , U t , A t , R t } Tt = with S t and U t as observed and unobserved state variables, A t as the action, and R t as the reward, where r t ( s, u, a ) =
Revised: Then the process of M can be summarized as { S t , U t , A t , R t } Tt = with S t and U t as observed and unobserved state variables, A t as the action, and R t as the reward, where r t ( s, u, a ) =

Original: E [ R t | S t = s, U t
Revised: E [ R t | S t = s, U t

Original: = u, A t = a ] for any ( s, u, a ) ∈ S
Revised: = u, A t = a ] for any ( s, u, a ) ∈ S

Original: × U × A .
Revised: × U × A .

Original: For simplicity, we assume that | R t | ≤ 1 uniformly in 1 ≤ t ≤ T .
Revised: For simplicity, we assume that | R t | ≤ 1 uniformly in 1 ≤ t ≤ T .

Original: The goal of OPE in a confounded POMDP is to evaluate the performance of a target policy using the batch data collected by some behavior policy.
Revised: The goal of OPE in a confounded POMDP is to evaluate the performance of a target policy using the batch data collected by some behavior policy.

Original: In this paper, the target policy we focus on is a sequence of functions mapping from the state space S to a probability mass function over the action space A , denoted by π = { π t } Tt =1 , where π t ( a | s ) is the probability of choosing an action A t = a given the state value S t = s .
Revised: In this paper, the target policy we focus on is a sequence of functions mapping from the state space S to a probability mass function over the action space A , denoted by π = { π t } Tt =1 , where π t ( a | s ) is the probability of choosing an action A t = a given the state value S t = s .

Original: We remark that our proposed identiﬁcation results stated in Sectioncan be generalized to other policies such as history-dependent ones.
Revised: We remark that our proposed identiﬁcation results stated in Sectioncan be generalized to other policies such as history-dependent ones.

Original: Given a target policy π , deﬁne its state value function as
Revised: Given a target policy π , deﬁne its state value function as

Original: [Equation]
Revised: [Equation]

Original: where E π denotes the expectation with respect to the distribution whose action at decision time t follows π t for any t ≥ 1 .
Revised: where E π denotes the expectation with respect to the distribution whose action at decision time t follows π t for any t ≥ 1 .

Original: We consider the batch setting, where the observed action A t is generated by some behavior policy ˜ π bt depending on both S t and U t for 1 ≤ t ≤ T .
Revised: We consider the batch setting, where the observed action A t is generated by some behavior policy ˜ π bt depending on both S t and U t for 1 ≤ t ≤ T .

Original: We aim to use the batch data to estimate the policy value of a target policy π , which is deﬁned as
Revised: We aim to use the batch data to estimate the policy value of a target policy π , which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where E denotes the expectation with respect to the behavior policy.
Revised: where E denotes the expectation with respect to the behavior policy.

Original: Due to the unobserved U t , standard OPE methods that rely on the Markovianity will give bias estimations.
Revised: Due to the unobserved U t , standard OPE methods that rely on the Markovianity will give bias estimations.

Original: In the following, we introduce an identiﬁcation result for estimating the policy value using some proxy variables.
Revised: In the following, we introduce an identiﬁcation result for estimating the policy value using some proxy variables.

Original: Notations : For two sequences { (cid:36) ( n ) } n ≥ 1 and { θ ( n ) } n ≥ 1 , the notation (cid:36) ( n ) (cid:38) θ ( n ) (resp.
Revised: Notations : For two sequences { (cid:36) ( n ) } n ≥ 1 and { θ ( n ) } n ≥ 1 , the notation (cid:36) ( n ) (cid:38) θ ( n ) (resp.

Original: (cid:36) ( n ) (cid:46) θ ( n ) ) means that there exists a sufﬁciently large constant (resp. small) constant c 1 > 0 (resp.
Revised: (cid:36) ( n ) (cid:46) θ ( n ) ) means that there exists a sufﬁciently large constant (resp. small) constant c 1 > 0 (resp.

Original: c 2 > 0 ) such that (cid:36) ( N ) ≥ c 1 θ ( N ) (resp.
Revised: c 2 > 0 ) such that (cid:36) ( N ) ≥ c 1 θ ( N ) (resp.

Original: (cid:36) ( n ) ≤ c 2 θ ( n ) ).
Revised: (cid:36) ( n ) ≤ c 2 θ ( n ) ).

Original: We use (cid:36) ( n ) (cid:16) θ ( n ) when (cid:36) ( n ) (cid:38) θ ( n ) and (cid:36) ( n ) (cid:46) θ ( n ) .
Revised: We use (cid:36) ( n ) (cid:16) θ ( n ) when (cid:36) ( n ) (cid:38) θ ( n ) and (cid:36) ( n ) (cid:46) θ ( n ) .

Original: For any random variable X , we use L q { X } to denote the class of all measurable functions with ﬁnite q -th moments for 1 ≤ q ≤ ∞ .
Revised: For any random variable X , we use L q { X } to denote the class of all measurable functions with ﬁnite q -th moments for 1 ≤ q ≤ ∞ .

Original: Then the L q -norm is denoted by (cid:107) • (cid:107)
Revised: Then the L q -norm is denoted by (cid:107) • (cid:107)

Original: L q { X } .
Revised: L q { X } .

Original: When there is no confusion in the underlying distribution, we also write it as (cid:107) • (cid:107) L q or (cid:107) • (cid:107) q .
Revised: When there is no confusion in the underlying distribution, we also write it as (cid:107) • (cid:107) L q or (cid:107) • (cid:107) q .

Original: In particular, (cid:107) • (cid:107) ∞ denotes the sup-norm.
Revised: In particular, (cid:107) • (cid:107) ∞ denotes the sup-norm.

Original: In addition, we use Big O and small o as the convention.
Revised: In addition, we use Big O and small o as the convention.

Original:  Identiﬁcation Results
Revised:  Identiﬁcation Results

Original: Inspired by the proximal causal inference recently proposed by Tchetgen Tchetgen et al.
Revised: Inspired by the proximal causal inference recently proposed by Tchetgen Tchetgen et al.

Original: [2020], we develop a non-parametric identiﬁcation result for estimating V ( π ) , which is similar to those by Bennett and Kallus [2021] and Shi et al.
Revised: [2020], we develop a non-parametric identiﬁcation result for estimating V ( π ) , which is similar to those by Bennett and Kallus [2021] and Shi et al.

Original: Assume that we can additionally observe the so-called reward-inducing proxy variables W t that are only related to the action A t through ( S t , U t ) and action-inducing proxy variables Z t that are only related to the reward R t through ( S t , U t ) at each decision time t .
Revised: Assume that we can additionally observe the so-called reward-inducing proxy variables W t that are only related to the action A t through ( S t , U t ) and action-inducing proxy variables Z t that are only related to the reward R t through ( S t , U t ) at each decision time t .

Original: See Figure 1 for a directed acyclic graph (DAG) to illustrate their relationships and a time series data example in Miao et al.
Revised: See Figure 1 for a directed acyclic graph (DAG) to illustrate their relationships and a time series data example in Miao et al.

Original: [2018b].
Revised: [2018b].

Original: For another example, the action-inducing proxy variables Z t can be deﬁned as the observed history before time t , then Z t and related arrows in Figure 1 can be removed.
Revised: For another example, the action-inducing proxy variables Z t can be deﬁned as the observed history before time t , then Z t and related arrows in Figure 1 can be removed.

Original: Detailed assumptions and discussion are given in Appendix A. Denote the spaces of { Z t } Tt =1 and { W t } Tt =1 by W and Z respectively.
Revised: Detailed assumptions and discussion are given in Appendix A. Denote the spaces of { Z t } Tt =1 and { W t } Tt =1 by W and Z respectively.

Original: U t − 1 S t −
Revised: U t − 1 S t −

Original: Z t −
Revised: Z t −

Original: W t −
Revised: W t −

Original: A t −
Revised: A t −

Original: R t −
Revised: R t −

Original: U t S t
Revised: U t S t

Original: Z t
Revised: Z t

Original: W t
Revised: W t

Original: A t
Revised: A t

Original: R t
Revised: R t

Original: U t +1 S t +
Revised: U t +1 S t +

Original: Z t +
Revised: Z t +

Original: W t +
Revised: W t +

Original: A t +
Revised: A t +

Original: R t +
Revised: R t +

Original: Since the states { U t } Tt =1 are unmeasured, we cannot estimate the value function by the celebrated
Revised: Since the states { U t } Tt =1 are unmeasured, we cannot estimate the value function by the celebrated

Original: Bellman equation.
Revised: Bellman equation.

Original: However, with the help of confounding proxies { W t , Z t } Tt =1 , the value of a target policy π can be non-parametrically identiﬁed using observed variables under proper assumptions.
Revised: However, with the help of confounding proxies { W t , Z t } Tt =1 , the value of a target policy π can be non-parametrically identiﬁed using observed variables under proper assumptions.

Original: To proceed, we deﬁne a class of V -bridge functions (or V -bridges for short) { v πt } Tt =1 deﬁned over
Revised: To proceed, we deﬁne a class of V -bridge functions (or V -bridges for short) { v πt } Tt =1 deﬁned over

Original: W × S such that for every ( s, u ) ∈ S
Revised: W × S such that for every ( s, u ) ∈ S

Original: × U and t ≥ 1 ,
Revised: × U and t ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: If such V -bridges exist, then we obtain the following identiﬁcation result for the policy value in (2).
Revised: If such V -bridges exist, then we obtain the following identiﬁcation result for the policy value in (2).

Original: Proposition 4.1 (Identiﬁcation) .
Revised: Proposition 4.1 (Identiﬁcation) .

Original: If there exist { v πt } T t =1 that satisfy (3), then the value of target policy π can be identiﬁed by V ( π ) =
Revised: If there exist { v πt } T t =1 that satisfy (3), then the value of target policy π can be identiﬁed by V ( π ) =

Original: E [ v π 1 ( W 1 , S 1 )] .
Revised: E [ v π 1 ( W 1 , S 1 )] .

Original: Note that V -bridges { v πt } Tt =1 that satisfy (3) are not necessarily unique, but we can uniquely identify
Revised: Note that V -bridges { v πt } Tt =1 that satisfy (3) are not necessarily unique, but we can uniquely identify

Original: V ( π ) based on any of them.
Revised: V ( π ) based on any of them.

Original: Next, we provide a theoretical guarantee for the existence of V -bridges { v πt } Tt =1 in terms of a sequence of linear integral equations.
Revised: Next, we provide a theoretical guarantee for the existence of V -bridges { v πt } Tt =1 in terms of a sequence of linear integral equations.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For a POMDP model of which variables satisfy the relationships illustrated in Figureand some regularity conditions given in Appendix A, there always exist V -bridges { v πt } Tt =1 satisfying (3).
Revised: For a POMDP model of which variables satisfy the relationships illustrated in Figureand some regularity conditions given in Appendix A, there always exist V -bridges { v πt } Tt =1 satisfying (3).

Original: With v πT +

Revised: With v πT +

Original: =
Revised: =

Original: 0 , a particular sequence of V -bridges { v πt } Tt =1 can be obtained by solving the following linear integral equations:
Revised: 0 , a particular sequence of V -bridges { v πt } Tt =1 can be obtained by solving the following linear integral equations:

Original: [Equation]
Revised: [Equation]

Original: where { q πt } Tt =1 are Q -bridges deﬁned over W × S
Revised: where { q πt } Tt =1 are Q -bridges deﬁned over W × S

Original: ×
Revised: ×

Original: A such that
Revised: A such that

Original: [Equation]
Revised: [Equation]

Original: for every ( s, u, a ) ∈ S × U × A and t ≥ 1 , and v πt ( w, s ) = (cid:80) a ∈A π t ( a | s ) q πt ( w, s, a ) .
Revised: for every ( s, u, a ) ∈ S × U × A and t ≥ 1 , and v πt ( w, s ) = (cid:80) a ∈A π t ( a | s ) q πt ( w, s, a ) .

Original: Clearly
Revised: Clearly

Original: Q -bridges { q πt } Tt =1 also exist.
Revised: Q -bridges { q πt } Tt =1 also exist.

Original: Theorem 4.1 guarantees the existence of both V -bridges and Q -bridges, and also provides a natural procedure (4) to ﬁnd { v πt } Tt =1 and eventually estimate the policy value V ( π ) .
Revised: Theorem 4.1 guarantees the existence of both V -bridges and Q -bridges, and also provides a natural procedure (4) to ﬁnd { v πt } Tt =1 and eventually estimate the policy value V ( π ) .

Original: Then based on
Revised: Then based on

Original: Proposition 4.1 and Theorem 4.1, we can perform OPE via Algorithm 1 in the population level.
Revised: Proposition 4.1 and Theorem 4.1, we can perform OPE via Algorithm 1 in the population level.

Original: Speciﬁcally at each step we will solve (4) via a non-parametric model, which is a NPIV problem.
Revised: Speciﬁcally at each step we will solve (4) via a non-parametric model, which is a NPIV problem.

Original: 1 Input: { ( S t , W t , Z t , A t , R t ) } Tt =1 , a target policy π = { π t } Tt =1 .
Revised: 1 Input: { ( S t , W t , Z t , A t , R t ) } Tt =1 , a target policy π = { π t } Tt =1 .

Original: 2 Let v πT +1 = 0 .
Revised: 2 Let v πT +1 = 0 .

Original: 3 Repeat for t = T, . . .
Revised: 3 Repeat for t = T, . . .

Original: Solve v πt and q πt by E (cid:8) q πt ( W t , S t , A t ) − R t − v πt
Revised: Solve v πt and q πt by E (cid:8) q πt ( W t , S t , A t ) − R t − v πt

Original: +

Revised: +

Original: ( W t +1 , S t +1 )
Revised: ( W t +1 , S t +1 )

Original: | Z t , S t , A t (cid:9) =
Revised: | Z t , S t , A t (cid:9) =

Original: with v πt ( W t , S t ) (cid:44) (cid:80) a ∈A π t ( a | S t ) q πt ( W t , S t , a ) .
Revised: with v πt ( W t , S t ) (cid:44) (cid:80) a ∈A π t ( a | S t ) q πt ( W t , S t , a ) .

Original: 5 Output: V ( π ) =
Revised: 5 Output: V ( π ) =

Original: E [ v π 1 ( W 1 , S 1 )] .
Revised: E [ v π 1 ( W 1 , S 1 )] .

Original:  Estimation
Revised:  Estimation

Original: In this section, we discuss how to estimate V ( π ) using batch data based on results given in Theorem 4.1 and Algorithm 1.
Revised: In this section, we discuss how to estimate V ( π ) using batch data based on results given in Theorem 4.1 and Algorithm 1.

Original: Let a pre-collected training dataset be D n = { ( S t,i , W t,i , Z t,i , A t,i , R t,i )
Revised: Let a pre-collected training dataset be D n = { ( S t,i , W t,i , Z t,i , A t,i , R t,i )

Original: Tt =1 : i = 1 , . . . , n } , which consists of n i.i.d.
Revised: Tt =1 : i = 1 , . . . , n } , which consists of n i.i.d.

Original: copies of the observable trajectory ( S t , W t , Z t , A t , R t )
Revised: copies of the observable trajectory ( S t , W t , Z t , A t , R t )

Original: Tt
Revised: Tt

Original: = of a confounded POMDP.
Revised: = of a confounded POMDP.

Original: Following Algorithm 1, we develop a FQE-type approach where we propose to solve a min-max problem for estimating v πt at the t -th step using the idea of Dikkala et al.
Revised: Following Algorithm 1, we develop a FQE-type approach where we propose to solve a min-max problem for estimating v πt at the t -th step using the idea of Dikkala et al.

Original: [2020], and then apply Proposition 4.1 for OPE.
Revised: [2020], and then apply Proposition 4.1 for OPE.

Original: For convenience, we ﬁrst rewrite the linear integral equations (4) for solving V -bridges in terms of operators.
Revised: For convenience, we ﬁrst rewrite the linear integral equations (4) for solving V -bridges in terms of operators.

Original: Deﬁne an operator (cid:101) P t : L 2 {R×W ×S} →
Revised: Deﬁne an operator (cid:101) P t : L 2 {R×W ×S} →

Original: L 2 {Z ×S×A} such that [ (cid:101) P t g ]
Revised: L 2 {Z ×S×A} such that [ (cid:101) P t g ]

Original: ( Z t , S t , A t ) =
Revised: ( Z t , S t , A t ) =

Original: E [ g ( R t , W t
Revised: E [ g ( R t , W t

Original: +1 , S t +1 )
Revised: +1 , S t +1 )

Original: | Z t , S t , A t ] for any g ∈ L 2 {R × W × S} .
Revised: | Z t , S t , A t ] for any g ∈ L 2 {R × W × S} .

Original: Deﬁne another operator P t :
Revised: Deﬁne another operator P t :

Original: L 2 {W × S × A} → L 2 {Z × S × A} such that for any h ∈ L 2 {W × S × A} , [ P t h ]( Z t , S t , A t ) =
Revised: L 2 {W × S × A} → L 2 {Z × S × A} such that for any h ∈ L 2 {W × S × A} , [ P t h ]( Z t , S t , A t ) =

Original: E [ h ( W t , S t , A t ) | Z t , S t , A t ] .
Revised: E [ h ( W t , S t , A t ) | Z t , S t , A t ] .

Original: Motivated by (4), we deﬁne the V-bridge transition operator P πt :
Revised: Motivated by (4), we deﬁne the V-bridge transition operator P πt :

Original: L 2 {R × W × S} → L 2 {W × S} such that
Revised: L 2 {R × W × S} → L 2 {W × S} such that

Original: [Equation]
Revised: [Equation]

Original: , where P t g = P t
Revised: , where P t g = P t

Original: [Equation]
Revised: [Equation]

Original: all g
Revised: all g

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: R × W
Revised: R × W

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: In particular, (cid:104) π t ( · | S t ) , [ P t g ]( W t , S t , · ) (cid:105) (cid:44) (cid:80) a ∈A π t ( a | S t )
Revised: In particular, (cid:104) π t ( · | S t ) , [ P t g ]( W t , S t , · ) (cid:105) (cid:44) (cid:80) a ∈A π t ( a | S t )

Original: [ P t g ]( W t , S t , a ) , and (cid:101)
Revised: [ P t g ]( W t , S t , a ) , and (cid:101)

Original: P t g is invertible by P t .
Revised: P t g is invertible by P t .

Original: The invertibility is ensured by Assumption 8 in Appendix A.
Revised: The invertibility is ensured by Assumption 8 in Appendix A.

Original: Then by the deﬁnition of V -bridges and (4), we can identify { v πt } Tt =1 via solving
Revised: Then by the deﬁnition of V -bridges and (4), we can identify { v πt } Tt =1 via solving

Original: [Equation]
Revised: [Equation]

Original: To ﬁnd the estimated V-bridges { ˆ v πt } Tt =1 , it sufﬁces to estimate P πt .
Revised: To ﬁnd the estimated V-bridges { ˆ v πt } Tt =1 , it sufﬁces to estimate P πt .

Original: Note that one can regard (6) as a series of conditional moment model restrictions and we propose to solve them via a sequential
Revised: Note that one can regard (6) as a series of conditional moment model restrictions and we propose to solve them via a sequential

Original: NPIV estimation.
Revised: NPIV estimation.

Original: In particular, at the t -th step, we adopt the min-max estimation method proposed by Dikkala et al.
Revised: In particular, at the t -th step, we adopt the min-max estimation method proposed by Dikkala et al.

Original: [2020] to estimate P πt non-parametrically as follows: (cid:98) P πt g
Revised: [2020] to estimate P πt non-parametrically as follows: (cid:98) P πt g

Original: =
Revised: =

Original: (cid:68) π t , (cid:98) P t g (cid:69) , where
Revised: (cid:68) π t , (cid:98) P t g (cid:69) , where

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) f
Revised: where (cid:107) f

Original: (cid:107) 2 n = n − 1 (cid:80) ni
Revised: (cid:107) 2 n = n − 1 (cid:80) ni

Original: =1 f 2 ( Z t,i , S t,i , A t,i ) for f ∈ F ( t ) .
Revised: =1 f 2 ( Z t,i , S t,i , A t,i ) for f ∈ F ( t ) .

Original: H ( t ) on W × S × A and F ( t ) on
Revised: H ( t ) on W × S × A and F ( t ) on

Original: Z × S × A are two user-deﬁned function spaces endowed with norms (cid:107) • (cid:107) H ( t ) and (cid:107) • (cid:107) F ( t ) respectively, λ, µ, M, δ > 0 are tuning parameters, and Ψ t,n ( h, f, g )
Revised: Z × S × A are two user-deﬁned function spaces endowed with norms (cid:107) • (cid:107) H ( t ) and (cid:107) • (cid:107) F ( t ) respectively, λ, µ, M, δ > 0 are tuning parameters, and Ψ t,n ( h, f, g )

Original: =
Revised: =

Original: n − 1 (cid:80) ni
Revised: n − 1 (cid:80) ni

Original: =

Revised: =

Original: [ h ( W t,i , S t,i , A t,i ) − ( T − t +1) − 1 g
Revised: [ h ( W t,i , S t,i , A t,i ) − ( T − t +1) − 1 g

Original: ( R t,i , W t +1 ,i , S t +1 ,i )]
Revised: ( R t,i , W t +1 ,i , S t +1 ,i )]

Original: f ( Z t,i , S t,i , A t,i ) , where g ( R t , W t +1 , S t +1 ) =
Revised: f ( Z t,i , S t,i , A t,i ) , where g ( R t , W t +1 , S t +1 ) =

Original: R t + ¯ g ( W t +1 , S t +1 ) for some ¯ g ∈ G ( t +1) on W × S , endowed with norm (cid:107) • (cid:107) G ( t +1) .
Revised: R t + ¯ g ( W t +1 , S t +1 ) for some ¯ g ∈ G ( t +1) on W × S , endowed with norm (cid:107) • (cid:107) G ( t +1) .

Original: The rational behind (7) is that when λ, λµ → 0 and λM/δ 2 (cid:16) 1 , the following two population-version min-max optimization problems
Revised: The rational behind (7) is that when λ, λµ → 0 and λM/δ 2 (cid:16) 1 , the following two population-version min-max optimization problems

Original: [Equation]
Revised: [Equation]

Original: have the same solution h when the space F ( t ) of testing functions is rich enough.
Revised: have the same solution h when the space F ( t ) of testing functions is rich enough.

Original: Note that ( T − t + 1) − 1 used above and in (7) are for scaling purpose.
Revised: Note that ( T − t + 1) − 1 used above and in (7) are for scaling purpose.

Original: After T steps, we output our estimator for the policy value based on the empirical counterpart of
Revised: After T steps, we output our estimator for the policy value based on the empirical counterpart of

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Our FQE-type algorithm is summarized in Algorithms 2 and 3 in Appendix E.
Revised: Our FQE-type algorithm is summarized in Algorithms 2 and 3 in Appendix E.

Original:  Theoretical Results
Revised:  Theoretical Results

Original: In this section, we establish the ﬁnite-sample bounds for the L 2 error of estimating V -bridge v πand the error of OPE, in terms of the sample size, length of horizon and two (local) measures of ill-posedness.
Revised: In this section, we establish the ﬁnite-sample bounds for the L 2 error of estimating V -bridge v πand the error of OPE, in terms of the sample size, length of horizon and two (local) measures of ill-posedness.

Original: Our bounds also rely on the critical radii of certain spaces related to the user-deﬁned function spaces H ( t ) and F ( t ) in (7), and also G ( t ) of V -bridge functions.
Revised: Our bounds also rely on the critical radii of certain spaces related to the user-deﬁned function spaces H ( t ) and F ( t ) in (7), and also G ( t ) of V -bridge functions.

Original: Technical preliminaries.
Revised: Technical preliminaries.

Original: Before presenting our main results, we ﬁrst introduce some concepts from the empirical process theory [Wainwright, 2019].
Revised: Before presenting our main results, we ﬁrst introduce some concepts from the empirical process theory [Wainwright, 2019].

Original: Deﬁnition 6.1 (Local Rademacher Complexity) .
Revised: Deﬁnition 6.1 (Local Rademacher Complexity) .

Original: Given any real-valued function class F deﬁned over a random vector X and any radius δ > 0 , the local Rademacher complexity is given by
Revised: Given any real-valued function class F deﬁned over a random vector X and any radius δ > 0 , the local Rademacher complexity is given by

Original: [Equation]
Revised: [Equation]

Original: where { X i } ni =1 are i.i.d.
Revised: where { X i } ni =1 are i.i.d.

Original: copies of X and { (cid:15) i } ni =1 are i.i.d.
Revised: copies of X and { (cid:15) i } ni =1 are i.i.d.

Original: Rademacher random variables.
Revised: Rademacher random variables.

Original: By bounding the local Rademacher complexity, which measures the complexity of the functional class F locally in a neighborhood of the ground truth, we can control the error rate of the proposed
Revised: By bounding the local Rademacher complexity, which measures the complexity of the functional class F locally in a neighborhood of the ground truth, we can control the error rate of the proposed

Original: V -bridge estimator in each step.
Revised: V -bridge estimator in each step.

Original: A crucial parameter for local Rademacher complexity of a function class F is called critical radius .
Revised: A crucial parameter for local Rademacher complexity of a function class F is called critical radius .

Original: Deﬁnition 6.2 (Critical Radius) .
Revised: Deﬁnition 6.2 (Critical Radius) .

Original: Assume that F is a star-shaped function class, i.e. αf ∈ F for any f ∈ F and scalar α ∈ [0 , 1] , and also that F is b -uniformly bounded, i.e., (cid:107) f (cid:107) ∞ ≤ b < ∞ , ∀ f ∈ F .
Revised: Assume that F is a star-shaped function class, i.e. αf ∈ F for any f ∈ F and scalar α ∈ [0 , 1] , and also that F is b -uniformly bounded, i.e., (cid:107) f (cid:107) ∞ ≤ b < ∞ , ∀ f ∈ F .

Original: The critical radius of F , denoted by δ n , is the solution to the inequality R n ( F , δ ) ≤ δ 2 /b.
Revised: The critical radius of F , denoted by δ n , is the solution to the inequality R n ( F , δ ) ≤ δ 2 /b.

Original: Additional Notations : We assume that the test functions f belong to a star shaped, symmetric space
Revised: Additional Notations : We assume that the test functions f belong to a star shaped, symmetric space

Original: F ( t ) ⊆
Revised: F ( t ) ⊆

Original: L 2 ( Z × S × A ) endowed with norm (cid:107) · (cid:107) F ( t ) .
Revised: L 2 ( Z × S × A ) endowed with norm (cid:107) · (cid:107) F ( t ) .

Original: For brevity of notation, hereafter we suppress the time-step indicator ( t ) in the context unless necessary.
Revised: For brevity of notation, hereafter we suppress the time-step indicator ( t ) in the context unless necessary.

Original: For a function space F , we deﬁne α F = { αf : f ∈ F} , for some α ∈ R .
Revised: For a function space F , we deﬁne α F = { αf : f ∈ F} , for some α ∈ R .

Original: Deﬁne F B = { f ∈ F : (cid:107) f (cid:107) 2 F ≤ B } , for any B > 0 .
Revised: Deﬁne F B = { f ∈ F : (cid:107) f (cid:107) 2 F ≤ B } , for any B > 0 .

Original: Deﬁne the projected root mean squared error (cid:107) proj t f (cid:107) 2 = (cid:112) E
Revised: Deﬁne the projected root mean squared error (cid:107) proj t f (cid:107) 2 = (cid:112) E

Original: { E [ f ( X ) | Z t , S t , A t ] } 2 , for any squared integrable f with respect to the conditional distribution of X given ( Z t , S t , A t ) .
Revised: { E [ f ( X ) | Z t , S t , A t ] } 2 , for any squared integrable f with respect to the conditional distribution of X given ( Z t , S t , A t ) .

Original: Standard (Local) Measures of ill-posedness : Let ¯ τ 1 = sup g ∈G (1) (cid:107) g
Revised: Standard (Local) Measures of ill-posedness : Let ¯ τ 1 = sup g ∈G (1) (cid:107) g

Original: ( W 1 , S 1 ) (cid:107) 2 / (cid:107) E
Revised: ( W 1 , S 1 ) (cid:107) 2 / (cid:107) E

Original: [ g ( W 1 , S 1 ) |
Revised: [ g ( W 1 , S 1 ) |

Original: Z 1 , S 1 ] (cid:107) 2 be the measure of ill-posedness for G (1) ( W 1 × S 1 ) projected on Z 1 × S 1 .
Revised: Z 1 , S 1 ] (cid:107) 2 be the measure of ill-posedness for G (1) ( W 1 × S 1 ) projected on Z 1 × S 1 .

Original: Let τ t =
Revised: Let τ t =

Original: sup h ∈H ( t ) (cid:107) h
Revised: sup h ∈H ( t ) (cid:107) h

Original: ( W t , S t , A t ) (cid:107) 2 / (cid:107) proj t h ( W t , S t , A t ) (cid:107) 2 be the standard measure of ill-posedness for
Revised: ( W t , S t , A t ) (cid:107) 2 / (cid:107) proj t h ( W t , S t , A t ) (cid:107) 2 be the standard measure of ill-posedness for

Original: H ( t ) ( W × S × A ) projected on Z × S × A .
Revised: H ( t ) ( W × S × A ) projected on Z × S × A .

Original: It can be seen that ¯ τ 1 , τ t ≥ 1 for t ≥ 1 .
Revised: It can be seen that ¯ τ 1 , τ t ≥ 1 for t ≥ 1 .

Original: Indeed we only require measuring ¯ τ 1 and τ t locally .
Revised: Indeed we only require measuring ¯ τ 1 and τ t locally .

Original: See more details in Appendix C.
Revised: See more details in Appendix C.

Original: 2. Results
Revised: 2. Results

Original: We ﬁrst give Assumption 1 used to develop our theoretical results below.
Revised: We ﬁrst give Assumption 1 used to develop our theoretical results below.

Original: Assumption 1.
Revised: Assumption 1.

Original: For each t = 1 , . . .
Revised: For each t = 1 , . . .

Original: , T ,
Revised: , T ,

Original: (1) Closeness.
Revised: (1) Closeness.

Original: For any g ∈ G ( t +1) , P t ( g + R t ) ∈ H ( t ) ; For any h ∈ H ( t ) , (cid:104) π t , h
Revised: For any g ∈ G ( t +1) , P t ( g + R t ) ∈ H ( t ) ; For any h ∈ H ( t ) , (cid:104) π t , h

Original: (cid:105) ∈ G ( t ) .
Revised: (cid:105) ∈ G ( t ) .

Original: (2) For any h ∈ ( T − t ) H ( t +1) , we have (cid:107)P t (cid:16) R t + (cid:104) π
Revised: (2) For any h ∈ ( T − t ) H ( t +1) , we have (cid:107)P t (cid:16) R t + (cid:104) π

Original: [Table]
Revised: [Table]

Original: +1 ,h (cid:105) T − t +
Revised: +1 ,h (cid:105) T − t +

Original: [Table]
Revised: [Table]

Original: (cid:107) 2 H
Revised: (cid:107) 2 H

Original: [Table]
Revised: [Table]

Original: t ) ≤
Revised: t ) ≤

Original: [Table]
Revised: [Table]

Original: H
Revised: H

Original: [Table]
Revised: [Table]

Original: (3) There exists a constant C G > 0 such that (cid:107) (cid:104)
Revised: (3) There exists a constant C G > 0 such that (cid:107) (cid:104)

Original: π t , h (cid:105) (cid:107) 2 G ( t ) ≤ C G (cid:107) h (cid:107) 2 H ( t ) , for h ∈ H ( t ) .
Revised: π t , h (cid:105) (cid:107) 2 G ( t ) ≤ C G (cid:107) h (cid:107) 2 H ( t ) , for h ∈ H ( t ) .

Original: (4) q πt ∈ ( T − t + 1) H ( t ) ( W , S , A ) and (cid:107) q πT (cid:107) 2 H ( T ) ≤ M H , where M H > 0 is a constant.
Revised: (4) q πt ∈ ( T − t + 1) H ( t ) ( W , S , A ) and (cid:107) q πT (cid:107) 2 H ( T ) ≤ M H , where M H > 0 is a constant.

Original: (5) Testing function class F ( t ) is sufﬁciently rich such that there exists L > 0 , (cid:107) f ∗ − proj t
Revised: (5) Testing function class F ( t ) is sufﬁciently rich such that there exists L > 0 , (cid:107) f ∗ − proj t

Original: h t (cid:107) 2 ≤ η ( t ) n , where f ∗ ∈ arg min f ∈F ( t ) L 2 (cid:107) ht (cid:107) 2 H (cid:107) f − proj t h t (cid:107) 2 , for all h t ∈ H ( t ) .
Revised: h t (cid:107) 2 ≤ η ( t ) n , where f ∗ ∈ arg min f ∈F ( t ) L 2 (cid:107) ht (cid:107) 2 H (cid:107) f − proj t h t (cid:107) 2 , for all h t ∈ H ( t ) .

Original: (6) Behavior policies: there exists a constant b π such that π bt ( a | s )
Revised: (6) Behavior policies: there exists a constant b π such that π bt ( a | s )

Original: (cid:44) E
Revised: (cid:44) E

Original: [˜ π bt ( a | U t , S t ) | S t = s ] ≥ b π > 0 for all ( s, a ) ∈ S × A .
Revised: [˜ π bt ( a | U t , S t ) | S t = s ] ≥ b π > 0 for all ( s, a ) ∈ S × A .

Original: Assumption 1 (1) is similar to Bellman completeness, which has been widely used in RL without unobserved states [e.g., Antos et al., 2008].
Revised: Assumption 1 (1) is similar to Bellman completeness, which has been widely used in RL without unobserved states [e.g., Antos et al., 2008].

Original: Note that both G ( t ) and H ( t ) can be chosen as inﬁnitedimensional spaces, e.g., RKHSs.
Revised: Note that both G ( t ) and H ( t ) can be chosen as inﬁnitedimensional spaces, e.g., RKHSs.

Original: Hence this assumption is relatively mild.
Revised: Hence this assumption is relatively mild.

Original: Assumption 1 (2) requires the operator P t to be bounded, which can be ensured under some continuity conditions on transition kernels
Revised: Assumption 1 (2) requires the operator P t to be bounded, which can be ensured under some continuity conditions on transition kernels

Original: [Kress, 1989].
Revised: [Kress, 1989].

Original: Assumption 1 (3) is a technical condition for controlling the complexity of
Revised: Assumption 1 (3) is a technical condition for controlling the complexity of

Original: G by H .
Revised: G by H .

Original: Assumption 1 (4) essentially assumes that we can model q π (and v π ) correctly at each t -step, which is again mild as H ( t ) for t ≥ 1 can all be chosen as inﬁnite-dimensional spaces.
Revised: Assumption 1 (4) essentially assumes that we can model q π (and v π ) correctly at each t -step, which is again mild as H ( t ) for t ≥ 1 can all be chosen as inﬁnite-dimensional spaces.

Original: This assumption is also called realizability of value functions, which is commonly seen in the literature of RL [e.g., Antos et al., 2008].
Revised: This assumption is also called realizability of value functions, which is commonly seen in the literature of RL [e.g., Antos et al., 2008].

Original: Assumption 1 (5) is imposed to ensure that the space of testing functions F is large enough so that we are able to capture the conditional expectation operator in each min-max estimation (7).
Revised: Assumption 1 (5) is imposed to ensure that the space of testing functions F is large enough so that we are able to capture the conditional expectation operator in each min-max estimation (7).

Original: Assumption 1 (6) basically requires a full coverage of our batch data generating process induced by the behavior policy, which is widely used in OPE [Precup, 2000, Antos et al., 2008].
Revised: Assumption 1 (6) basically requires a full coverage of our batch data generating process induced by the behavior policy, which is widely used in OPE [Precup, 2000, Antos et al., 2008].

Original: Next, we provide a key decomposition of the L 2 error for V -bridge estimation.
Revised: Next, we provide a key decomposition of the L 2 error for V -bridge estimation.

Original: Theorem 6.1 (Error decomposition) .
Revised: Theorem 6.1 (Error decomposition) .

Original: Under Assumption 1 (1) and (6), we can decompose the Lerror of the estimated V -bridge by
Revised: Under Assumption 1 (1) and (6), we can decompose the Lerror of the estimated V -bridge by

Original: [Equation]
Revised: [Equation]

Original: where the measures of one-step transition ill-posedness C ( t ) 1 , 0 (cid:44) 1 and C ( t )
Revised: where the measures of one-step transition ill-posedness C ( t ) 1 , 0 (cid:44) 1 and C ( t )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: , 2 ≤ t (cid:48) ≤ t ≤ T are deﬁned after Corollary 6.2.
Revised: , 2 ≤ t (cid:48) ≤ t ≤ T are deﬁned after Corollary 6.2.

Original: Theorem 6.1 shows that there are four key components for upper bounding the L 2 error of ˆ v π 1 .
Revised: Theorem 6.1 shows that there are four key components for upper bounding the L 2 error of ˆ v π 1 .

Original: The ﬁrst component is the probability ratio (cid:107) π t /π bt (cid:107) ∞ , which is used to measure the distributional mismatch between the target and behavior policies.
Revised: The ﬁrst component is the probability ratio (cid:107) π t /π bt (cid:107) ∞ , which is used to measure the distributional mismatch between the target and behavior policies.

Original: The second component is (cid:107) proj t
Revised: The second component is (cid:107) proj t

Original: ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 , the one-step projected error of (cid:98) P t to P t , where ˆ v πt +1 is the estimate for v πt +1 depending on the observed data after t -step.
Revised: ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 , the one-step projected error of (cid:98) P t to P t , where ˆ v πt +1 is the estimate for v πt +1 depending on the observed data after t -step.

Original: We remark that this is different from the analysis in the standard NPIV estimation with a directly measured outcome.
Revised: We remark that this is different from the analysis in the standard NPIV estimation with a directly measured outcome.

Original: Hence the results, e.g., from Dikkala et al.
Revised: Hence the results, e.g., from Dikkala et al.

Original: [2020], cannot be directly applied to bound this component.
Revised: [2020], cannot be directly applied to bound this component.

Original: The last two components are related to the (local) measure of ill-posedness.
Revised: The last two components are related to the (local) measure of ill-posedness.

Original: The third component τ t is the measure of ill-posedness for characterizing the difﬁculty of estimating q πt by (4) using H ( t ) at the t -th step. {
Revised: The third component τ t is the measure of ill-posedness for characterizing the difﬁculty of estimating q πt by (4) using H ( t ) at the t -th step. {

Original: τ t } Tt =1 are similar to those used in the standard NPIV estimation such as Chen and Reiss [2011], and the effect of each τ t on the upper bound is cumulative.
Revised: τ t } Tt =1 are similar to those used in the standard NPIV estimation such as Chen and Reiss [2011], and the effect of each τ t on the upper bound is cumulative.

Original: The last component { Π tt (cid:48) =
Revised: The last component { Π tt (cid:48) =

Original: 1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 quantify the propagation effect of estimation errors in previous steps on the last step of estimating v π 1 , which is multiplicative in terms of C ( t ) t (cid:48) ,t (cid:48) − 1 .
Revised: 1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 quantify the propagation effect of estimation errors in previous steps on the last step of estimating v π 1 , which is multiplicative in terms of C ( t ) t (cid:48) ,t (cid:48) − 1 .

Original: We call C ( t ) t (cid:48) ,t (cid:48) − 1 the measure of one-step transition ill-posedness from t (cid:48) to t (cid:48) − 1 related to t -step NPIV estimations.
Revised: We call C ( t ) t (cid:48) ,t (cid:48) − 1 the measure of one-step transition ill-posedness from t (cid:48) to t (cid:48) − 1 related to t -step NPIV estimations.

Original: Next we provide detailed bounds for the second and last components.
Revised: Next we provide detailed bounds for the second and last components.

Original: The discussion of the third component can be found in Appendix C.4.
Revised: The discussion of the third component can be found in Appendix C.4.

Original: Component 2: one-step projected error .
Revised: Component 2: one-step projected error .

Original: In the following, we show that (cid:107) proj t
Revised: In the following, we show that (cid:107) proj t

Original: ( (cid:98) P t −P t )(ˆ v πt +1 + R t ) (cid:107)is bounded by the critical radii of some spaces deﬁned as balls H ( t ) B , G ( t +1)
Revised: ( (cid:98) P t −P t )(ˆ v πt +1 + R t ) (cid:107)is bounded by the critical radii of some spaces deﬁned as balls H ( t ) B , G ( t +1)

Original: C G ( T − t +1)
Revised: C G ( T − t +1)

Original: M H in hypothesis spaces H ( t ) , G ( t +1) respectively and a ball F ( t ) 3 M in testing space F ( t ) , for some ﬁxed constants
Revised: M H in hypothesis spaces H ( t ) , G ( t +1) respectively and a ball F ( t ) 3 M in testing space F ( t ) , for some ﬁxed constants

Original: M, B > 0 such that functions in H ( t ) B and F ( t ) 3 M have uniformly bounded ranges in [ − 1 , 1] for all 1 ≤ t ≤ T .
Revised: M, B > 0 such that functions in H ( t ) B and F ( t ) 3 M have uniformly bounded ranges in [ − 1 , 1] for all 1 ≤ t ≤ T .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Ξ ( t ) =
Revised: Ξ ( t ) =

Original: { ( s t , w t , z t , a t ) (cid:55)→ r
Revised: { ( s t , w t , z t , a t ) (cid:55)→ r

Original: [ h − h
Revised: [ h − h

Original: [Equation]
Revised: [Equation]

Original: g ]( w t , s t , a t ) f
Revised: g ]( w t , s t , a t ) f

Original: L 2 B ( z t , s t , a t ) :
Revised: L 2 B ( z t , s t , a t ) :

Original: h
Revised: h

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: G
Revised: G

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , r ∈ [0 , 1] } , where h ∗ g ∈ H ( t ) is the solution to E [ h ( W t , S t , A t ) − g ( W t +1 , S t +1 ) | Z t , S t , A t ] = 0 , and f L 2 B = arg min f ∈F ( t )
Revised: , r ∈ [0 , 1] } , where h ∗ g ∈ H ( t ) is the solution to E [ h ( W t , S t , A t ) − g ( W t +1 , S t +1 ) | Z t , S t , A t ] = 0 , and f L 2 B = arg min f ∈F ( t )

Original: L 2 B (cid:107) f − proj t
Revised: L 2 B (cid:107) f − proj t

Original: ( h − h ∗ g ) (cid:107) 2 for a given L > 0 .
Revised: ( h − h ∗ g ) (cid:107) 2 for a given L > 0 .

Original: An upper bound for (cid:107) proj t ( (cid:98) P t −P t )(ˆ v πt +

Revised: An upper bound for (cid:107) proj t ( (cid:98) P t −P t )(ˆ v πt +

Original: +
Revised: +

Original: R t ) (cid:107) 2 is given in Theorem 6.2.
Revised: R t ) (cid:107) 2 is given in Theorem 6.2.

Original: Theorem 6.2.
Revised: Theorem 6.2.

Original: Suppose that Assumption 1 holds.
Revised: Suppose that Assumption 1 holds.

Original: Let δ ( t ) n = ¯ δ ( t ) n + c 0 (cid:113) log( c 1 T/ζ ) n for some universal constants c 0 , c 1 > 0 where ¯ δ ( t ) n is the upper bound of the critical radii of F ( t ) 3 M , Ω ( t ) and Ξ ( t ) .
Revised: Let δ ( t ) n = ¯ δ ( t ) n + c 0 (cid:113) log( c 1 T/ζ ) n for some universal constants c 0 , c 1 > 0 where ¯ δ ( t ) n is the upper bound of the critical radii of F ( t ) 3 M , Ω ( t ) and Ξ ( t ) .

Original: Assume that the approximation error in Assumption 1 (5) can be bounded by η ( t ) n ≤ δ ( t ) n .
Revised: Assume that the approximation error in Assumption 1 (5) can be bounded by η ( t ) n ≤ δ ( t ) n .

Original: Furthermore, letting tuning parameters satisfy Mλ (cid:16) ( δ ( t ) n ) 2 and µ ≥
Revised: Furthermore, letting tuning parameters satisfy Mλ (cid:16) ( δ ( t ) n ) 2 and µ ≥

Original: O
Revised: O

Original: ( L 2 + M/B ) , with probability at least 1 − ζ , we have
Revised: ( L 2 + M/B ) , with probability at least 1 − ζ , we have

Original: [Equation]
Revised: [Equation]

Original: for all 1 ≤ t ≤ T
Revised: for all 1 ≤ t ≤ T

Original: [Equation]
Revised: [Equation]

Original: Depending on the choices of H ( t ) , G ( t +1) , and F ( t ) , we can obtain different ﬁnite-sample error bounds of the one-step projected error for each t .
Revised: Depending on the choices of H ( t ) , G ( t +1) , and F ( t ) , we can obtain different ﬁnite-sample error bounds of the one-step projected error for each t .

Original: Below we provide two examples.
Revised: Below we provide two examples.

Original: Corollary 6.1.
Revised: Corollary 6.1.

Original: Let F ( t ) , H ( t ) and G ( t +1) be VC-subgraph classes with VC dimensions V ( F ( t ) ) ,
Revised: Let F ( t ) , H ( t ) and G ( t +1) be VC-subgraph classes with VC dimensions V ( F ( t ) ) ,

Original: V ( H ( t ) ) and V ( G ( t +1) ) respectively.
Revised: V ( H ( t ) ) and V ( G ( t +1) ) respectively.

Original: Then with probability at least 1 − ζ , for all 1 ≤ t ≤ T , (cid:107) proj
Revised: Then with probability at least 1 − ζ , for all 1 ≤ t ≤ T , (cid:107) proj

Original: [Equation]
Revised: [Equation]

Original: The deﬁnition of the VC-subgraph class can be found in, e.g., Wainwright [2019].
Revised: The deﬁnition of the VC-subgraph class can be found in, e.g., Wainwright [2019].

Original: This is a broad class.
Revised: This is a broad class.

Original: For example, if one lets each of F ( t ) , H ( t ) and G ( t +1) be a linear space F
Revised: For example, if one lets each of F ( t ) , H ( t ) and G ( t +1) be a linear space F

Original: = { θ (cid:62) φ ( · ) : θ ∈ R d } with basis functions φ ( · ) , then V ( F ) =
Revised: = { θ (cid:62) φ ( · ) : θ ∈ R d } with basis functions φ ( · ) , then V ( F ) =

Original: d + 1 .
Revised: d + 1 .

Original: Then the upper bound for the one-step projected error becomes O (( T − t + 1) 2 . 5 d/ √ n ) .
Revised: Then the upper bound for the one-step projected error becomes O (( T − t + 1) 2 . 5 d/ √ n ) .

Original: Corollary 6.2.
Revised: Corollary 6.2.

Original: Let H ( t ) , G ( t +1) and F ( t ) be reproducing kernel Hilbert spaces (RKHSs) equipped with kernels K H ( t ) , K G ( t +1) and K F ( t ) respectively.
Revised: Let H ( t ) , G ( t +1) and F ( t ) be reproducing kernel Hilbert spaces (RKHSs) equipped with kernels K H ( t ) , K G ( t +1) and K F ( t ) respectively.

Original: For a given positive deﬁnite kernel K , we denote its nonincreasing eigenvalue sequence by { λ ↓ j ( K ) } ∞ j =1 .
Revised: For a given positive deﬁnite kernel K , we denote its nonincreasing eigenvalue sequence by { λ ↓ j ( K ) } ∞ j =1 .

Original: We consider two scenarios for { λ ↓ j ( K ) } ∞ j =1 .
Revised: We consider two scenarios for { λ ↓ j ( K ) } ∞ j =1 .

Original: (1) Polynomial eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ aj − 2 α H , λ ↓ j ( K G ( t +1) )
Revised: (1) Polynomial eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ aj − 2 α H , λ ↓ j ( K G ( t +1) )

Original: ≤ aj − 2 α G and
Revised: ≤ aj − 2 α G and

Original: λ ↓ j ( K F ( t ) ) ≤ aj − 2 α F for constants α H , α G , α F > 1 / 2 and a > 0 , then under the assumptions in Theorem 6.2, with probability at least 1 − ζ , for all 1 ≤ t ≤ T , we have
Revised: λ ↓ j ( K F ( t ) ) ≤ aj − 2 α F for constants α H , α G , α F > 1 / 2 and a > 0 , then under the assumptions in Theorem 6.2, with probability at least 1 − ζ , for all 1 ≤ t ≤ T , we have

Original: [Equation]
Revised: [Equation]

Original: (2) Exponential eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ a 1 e − a 2 j β H , λ ↓ j ( K G ( t +1) ) ≤ a 1 e − a 2 j β G and λ ↓ j ( K F
Revised: (2) Exponential eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ a 1 e − a 2 j β H , λ ↓ j ( K G ( t +1) ) ≤ a 1 e − a 2 j β G and λ ↓ j ( K F

Original: t
Revised: t

Original: ) ≤
Revised: ) ≤

Original: a e − a 2 j β F ,
Revised: a e − a 2 j β F ,

Original: for
Revised: for

Original: constants a 1 , a 2 , β H ,
Revised: constants a 1 , a 2 , β H ,

Original: β
Revised: β

Original: G , β F > 0 , then under the assumptions in
Revised: G , β F > 0 , then under the assumptions in

Original: Theorem 6.2, with probability at least 1 − ζ , for all 1 ≤ t ≤ T , we have
Revised: Theorem 6.2, with probability at least 1 − ζ , for all 1 ≤ t ≤ T , we have

Original: [Equation]
Revised: [Equation]

Original: Kernels of the two types of eigen-decay considered above are very common.
Revised: Kernels of the two types of eigen-decay considered above are very common.

Original: For example, the kernel of the α -order Soblev space with α > 1 / 2 , has a polynomial eigen-decay while the Gaussian kernel has an exponential eigen-decay, with β = 2 for Lebesgue measure on real line and β = 1 on a compact domain [Wei et al., 2017].
Revised: For example, the kernel of the α -order Soblev space with α > 1 / 2 , has a polynomial eigen-decay while the Gaussian kernel has an exponential eigen-decay, with β = 2 for Lebesgue measure on real line and β = 1 on a compact domain [Wei et al., 2017].

Original: Components 4: measure of one-step transition ill-posedness .
Revised: Components 4: measure of one-step transition ill-posedness .

Original: We ﬁrst provide more insights on { Π tt (cid:48)
Revised: We ﬁrst provide more insights on { Π tt (cid:48)

Original: =1 C ( t ) t (cid:48)
Revised: =1 C ( t ) t (cid:48)

Original: [Equation]
Revised: [Equation]

Original: t (cid:48) − 1 } Tt =1 before providing an upper bound.
Revised: t (cid:48) − 1 } Tt =1 before providing an upper bound.

Original: We formally deﬁne the local measure of one-step transition ill-posedness C ( t ) t (cid:48)
Revised: We formally deﬁne the local measure of one-step transition ill-posedness C ( t ) t (cid:48)

Original: +1 ,t (cid:48) recursively based on C ( t ) t,t − 1 to C ( t ) 2 , 1 as
Revised: +1 ,t (cid:48) recursively based on C ( t ) t,t − 1 to C ( t ) 2 , 1 as

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: t ) t
Revised: t ) t

Original: [Equation]
Revised: [Equation]

Original: ,t
Revised: ,t

Original: [Equation]
Revised: [Equation]

Original: (cid:44)
Revised: (cid:44)

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: with C ( t ) t +1 ,t (cid:44) 1 for each t = 1 , . . .
Revised: with C ( t ) t +1 ,t (cid:44) 1 for each t = 1 , . . .

Original: , T .
Revised: , T .

Original: For C ( t ) t,t − 1 , we can upper bound (cid:107) E [ˆ v π ( W t , S t ) | Z t , S t ] (cid:107) 2 by the projected error (cid:107) proj t ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 multiplied by the ill-posedness τ t .
Revised: For C ( t ) t,t − 1 , we can upper bound (cid:107) E [ˆ v π ( W t , S t ) | Z t , S t ] (cid:107) 2 by the projected error (cid:107) proj t ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 multiplied by the ill-posedness τ t .

Original: By Theorem 6.2, the projected error can be well controlled by δ ( t ) n with high probability, so C ( t ) t,t − 1 can be deﬁned locally.
Revised: By Theorem 6.2, the projected error can be well controlled by δ ( t ) n with high probability, so C ( t ) t,t − 1 can be deﬁned locally.

Original: Therefore, we can provide an upper bound for the denominator sequentially and deﬁne all
Revised: Therefore, we can provide an upper bound for the denominator sequentially and deﬁne all

Original: C ( t ) t (cid:48)
Revised: C ( t ) t (cid:48)

Original: +1 ,t (cid:48) locally, which indicates that all C ( t ) t (cid:48) +1 ,t (cid:48) could be small.
Revised: +1 ,t (cid:48) locally, which indicates that all C ( t ) t (cid:48) +1 ,t (cid:48) could be small.

Original: For example, if we use the observed history as the action-inducing proxy, then σ ( Z 1 × S 1 ) ⊂ σ ( Z 2 × S 2 ) ⊂ · · · ⊂ σ ( Z T × S T ) is a ﬁltration.
Revised: For example, if we use the observed history as the action-inducing proxy, then σ ( Z 1 × S 1 ) ⊂ σ ( Z 2 × S 2 ) ⊂ · · · ⊂ σ ( Z T × S T ) is a ﬁltration.

Original: In this case, (cid:81) tt (cid:48)
Revised: In this case, (cid:81) tt (cid:48)

Original: =2 C ( t ) t (cid:48) ,t (cid:48) − 1 are expected to be small for t ≥ 2 if the target policy is stationary.
Revised: =2 C ( t ) t (cid:48) ,t (cid:48) − 1 are expected to be small for t ≥ 2 if the target policy is stationary.

Original: While this can enlarge the critical radii δ ( t ) n due to the dimension of the action-inducing proxy, this only affects one-step errors.
Revised: While this can enlarge the critical radii δ ( t ) n due to the dimension of the action-inducing proxy, this only affects one-step errors.

Original: See detailed discussion in
Revised: See detailed discussion in

Original: Appendix B. Motivated by this, it is reasonable to impose Assumption 2 below on C ( t ) t (cid:48)
Revised: Appendix B. Motivated by this, it is reasonable to impose Assumption 2 below on C ( t ) t (cid:48)

Original: +1 ,t (cid:48) .
Revised: +1 ,t (cid:48) .

Original: Assumption 2.
Revised: Assumption 2.

Original: For every t ≥ 2 and 2 ≤ m ≤ t , C ( t ) m,m − 1 ≤ 1 + a t m αt with time-dependent constants a t > 0 , α t ≥ α > 1 .
Revised: For every t ≥ 2 and 2 ≤ m ≤ t , C ( t ) m,m − 1 ≤ 1 + a t m αt with time-dependent constants a t > 0 , α t ≥ α > 1 .

Original: Corollary 6.3.
Revised: Corollary 6.3.

Original: If Assumption 2 holds, then (cid:81) tt (cid:48)
Revised: If Assumption 2 holds, then (cid:81) tt (cid:48)

Original: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 ≤ exp { a t ζ ( α t ) } , where ζ ( α t ) =
Revised: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 ≤ exp { a t ζ ( α t ) } , where ζ ( α t ) =

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: n =1 (1 /n )
Revised: n =1 (1 /n )

Original: α t is uniformly bounded for t ≥ 1 .
Revised: α t is uniformly bounded for t ≥ 1 .

Original: Main result: error bounds for V -bridge estimation and OPE .
Revised: Main result: error bounds for V -bridge estimation and OPE .

Original: Deﬁne trans-ill = max 1 ≤ t ≤ T exp { a t ζ ( α t ) } and let ill max = ¯ τ 1 max 1 ≤ t ≤ T τ t (cid:107) π t /π bt (cid:107) ∞ .
Revised: Deﬁne trans-ill = max 1 ≤ t ≤ T exp { a t ζ ( α t ) } and let ill max = ¯ τ 1 max 1 ≤ t ≤ T τ t (cid:107) π t /π b t (cid:107) ∞ .

Original: Summarizing all aforementioned results, we have the following main theorem based on the polynomial eigen-decay case in
Revised: Summarizing all aforementioned results, we have the following main theorem based on the polynomial eigen-decay case in

Original: Corollary 6.2.
Revised: Corollary 6.2.

Original: Other cases can be found in Appendix B.
Revised: Other cases can be found in Appendix B.

Original: Theorem 6.3 (Finite-sample error bounds for V -bridges and policy value) .
Revised: Theorem 6.3 (Finite-sample error bounds for V -bridges and policy value) .

Original: Under Assumptionsand 2, and assumptions in Theorem 6.2 and Corollary 6.2 (1), with probability at least 1 − ζ , we have
Revised: Under Assumptionsand 2, and assumptions in Theorem 6.2 and Corollary 6.2 (1), with probability at least 1 − ζ , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 6.3 provides the ﬁrst ﬁnite-sample error bound for OPE under confounded and episodic
Revised: Theorem 6.3 provides the ﬁrst ﬁnite-sample error bound for OPE under confounded and episodic

Original: POMDPs in terms of the sample size, length of horizon and two ( local ) measures of ill-posedness.
Revised: POMDPs in terms of the sample size, length of horizon and two ( local ) measures of ill-posedness.

Original: Without considering the measures of ill-posedness, the derived error bound for V -bridge function nearly achieves the optimal L 2 -convergence rate in the classical non-parametric regression [Stone, 1982].
Revised: Without considering the measures of ill-posedness, the derived error bound for V -bridge function nearly achieves the optimal L 2 -convergence rate in the classical non-parametric regression [Stone, 1982].

Original: Moreover, our OPE error bound depends on a polynomial order of T , i.e., T 7 / 2 , which is larger than the standard O ( T 3 ) in the OPE without unobserved variables.
Revised: Moreover, our OPE error bound depends on a polynomial order of T , i.e., T 7 / 2 , which is larger than the standard O ( T 3 ) in the OPE without unobserved variables.

Original: However, when the function class consider in (7) grows with the sample size n , ill max will also increase and therefore the convergence rates in Theorem 6.3 could be much slower.
Revised: However, when the function class consider in (7) grows with the sample size n , ill max will also increase and therefore the convergence rates in Theorem 6.3 could be much slower.

Original: We further study a case when we can control the local measures of ill-posedness  τ t  , by assuming that λ min (Γ ( t ) m ) ≥ ν mand other regularity conditions in Lemma C.1 uniformly for all 1 ≤ t ≤ T , and ( S t , A t ) ∈
Revised: Next we study a case when we can control the local measures of ill-posedness { τ t } Tt =1 , by assuming that λ min (Γ ( t ) m ) ≥ ν m for all 1 ≤ t ≤ T almost surely and other regularity conditions in

Original: S × A , where Γ ( t ) m
Revised: Lemma C.1, where Γ ( t ) m

Original: (cid:44) E (cid:110) E [ e ( t )
Revised: (cid:44) E (cid:110) E [ e ( t )

Original: I ( W t , S t , A t ) | Z t , S t , A t ] E [ e ( t ) I ( W t , S t , A t ) | Z t , S t , A t ] (cid:62) (cid:111) with e ( t ) I =
Revised: I ( W t , S t , A t ) | Z t , S t , A t ] E [ e ( t ) I ( W t , S t , A t ) | Z t , S t , A t ] (cid:62) (cid:111) with e ( t ) I =

Original: ( e ( t ) 1 , . . .
Revised: ( e ( t ) 1 , . . .

Original: , e ( t ) m ) are the ﬁrst m eigenfunctions of kernel K H ( t ) .
Revised: , e ( t ) m ) as the ﬁrst m eigenfunctions of kernel K H ( t ) .

Original: Similar argument can be applied to  ¯ τ 1 , which is omitted here for simplicity.
Revised: Similar conditions can be imposed to control ¯ τ 1 , which is omitted here for simplicity.

Original: Let η ( n, T, ζ, α H , α F , α G , b ) (cid:44) 
Revised: Let η ( n, T, ζ, α H , α F , α G , b ) (cid:44) T [Equation] log

Original: [Equation]
Revised: [Equation]

Revised: , with b deﬁned below.

Original: Corollary 6.4.
Revised: Corollary 6.4.

Original: If assumptions in Theorem 6.3 holds and ν m ≥ m − 2 b  , then
Revised: If assumptions in Theorem 6.3 holds and ν m ≥ m − 2 b for some b ≥ 0 , then

Original: [Equation]
Revised: [Equation]

Original:  × trans-ill × η ( n, T, ζ, α H , α F , α G , b ) .
Revised: ¯ τ 1 max 1 ≤ t ≤ T (cid:107) π t /π bt (cid:107) ∞ × trans-ill × η ( n, T, ζ, α H , α F , α G , b ) .

Original: Corollary C.5.2 considers the mildly ill-posed case, i.e., ν m ≥ m − 2 b , and shows that the local measure of ill-posedness can deteriorate the sample complexity by factor α H / ( α H + b ) .
Revised: Corollary C.5.2 considers the mildly ill-posed case, i.e., ν m ≥ m − 2 b , and shows that the local measure of ill-posedness can deteriorate convergence rate of ˆ v π 1 signiﬁcantly.

Original: If b is large  or  severely ill-posed case is considered (i.e., ν m decays exponentially fast, see Appendix C.5.2), then the convergence rate of V -bridge estimation could be much slower and the typical requirement on the nuisance parameter for achieving asymptotic normality for the policy value will fail.
Revised: If b is large relative to α H or further severely ill-posed case is considered (i.e., ν m decays exponentially fast, see Appendix C.5.2), then the convergence rate of V -bridge estimation could be much slower and the typical requirement on the nuisance parameter for achieving asymptotic normality for the policy value will fail.

Original: On the other hand, it can be seen that when b = 0 , the ﬁnite sample error bounds match the results in Theorem 6.3.
Revised: On the other hand, it can be seen that when b = 0 , the ﬁnite sample error bounds match the results in Theorem 6.3.

Original:  Simulation
Revised:  Simulation

Original: In this section, we perform a simulation study to evaluate the performance of our proposed OPE estimation and to verify the ﬁnite-sample error bound of our OPE estimator in Theorem 6.3.
Revised: In this section, we perform a simulation study to evaluate the performance of our proposed OPE estimation and to verify the ﬁnite-sample error bound of our OPE estimator in Theorem 6.3.

Original: Let S
Revised: Let S

Original: = R 2 , U = R , W = R , Z = R , and A = { 1 , − 1 } .
Revised: = R 2 , U = R , W = R , Z = R , and A = { 1 , − 1 } .

Original: At time t , the hidden state U t , two proximal variables Z t , W t satisfy the following multivariate normal distribution given ( S t , A t ) :
Revised: At time t , the hidden state U t , two proximal variables Z t , W t satisfy the following multivariate normal distribution given ( S t , A t ) :

Original: [Equation]
Revised: [Equation]

Original: 9) where parameters are given in the Appendix.
Revised: 9) where parameters are given in the Appendix.

Original: The behavior policy is given by ˜ π bt ( A t | U t , S t )
Revised: The behavior policy is given by ˜ π bt ( A t | U t , S t )

Original: =
Revised: =

Original: expit (cid:8) − A t (cid:0) t 0 + t u U t + t (cid:62) s S t (cid:1)(cid:9) , where t 0 = 0 , t u = 1 , and t (cid:62)
Revised: expit (cid:8) − A t (cid:0) t 0 + t u U t + t (cid:62) s S t (cid:1)(cid:9) , where t 0 = 0 , t u = 1 , and t (cid:62)

Original: s =
Revised: s =

Original: [ − 0 .
Revised: [ − 0 .

Original: 5 , − 0 .
Revised: 5 , − 0 .

Original: Then by Assumption 1 (6), π bt ( A t | S t ) =
Revised: Then by Assumption 1 (6), π bt ( A t | S t ) =

Original: expit {− A t (cid:0) t 0 + t u κ 0 + ( t s + t
Revised: expit {− A t (cid:0) t 0 + t u κ 0 + ( t s + t

Original: u κ s ) (cid:62) S t (cid:1) } .
Revised: u κ s ) (cid:62) S t (cid:1) } .

Original: The initial S 1 is uniformly sampled from R 2 .
Revised: The initial S 1 is uniformly sampled from R 2 .

Original: At time t , given ( S t , U t , A t ) , we generate S t +

Revised: At time t , given ( S t , U t , A t ) , we generate S t +

Original: =
Revised: =

Original: S t + A t U t 1 2 + e S t
Revised: S t + A t U t 1 2 + e S t

Original: +1 , where 1 2 =
Revised: +1 , where 1 2 =

Original: [1 , 1] (cid:62) and the random error e S t +1 ∼ N ([0 , 0] (cid:62) , I 2 ) with I 2 denoting the 2 -by- 2 identity matrix.
Revised: [1 , 1] (cid:62) and the random error e S t +1 ∼ N ([0 , 0] (cid:62) , I 2 ) with I 2 denoting the 2 -by- 2 identity matrix.

Original: The reward is given by R t
Revised: The reward is given by R t

Original: = expit (cid:8) 

Revised: = expit (cid:8)

Original: A t ( U t + [1 , − 2] S t ) (cid:9) + e t , where e t ∼ Uniform [ − 0 .
Revised: A t ( U t + [1 , − 2] S t ) (cid:9) + e t , where e t ∼ Uniform [ − 0 .

Original: One can verify that our simulation setting satisﬁes the conditions in Section A.1 so that our method can be applied.
Revised: One can verify that our simulation setting satisﬁes the conditions in Section A.1 so that our method can be applied.

Original: 2222222 6 Horizon T 

Revised: 2222222 6 Horizon T

Original: M e a n a b s o l u t e e rr o r
Revised: M e a n a b s o l u t e e rr o r

Original: Training set size n= 22222 12 Training set size n 

Revised: Training set size n= 22222 12 Training set size n

Original: M e a n a b s o l u t e e rr o r
Revised: M e a n a b s o l u t e e rr o r

Original: [Figure]
Revised: [Figure]

Original: T
Revised: T

Original: [Figure]
Revised: [Figure]

Original: We choose F ( t ) and H ( t ) as RKHSs endowed with Gaussian kernels, with bandwidths selected according to the median heuristic trick by Fukumizu et al.
Revised: We choose F ( t ) and H ( t ) as RKHSs endowed with Gaussian kernels, with bandwidths selected according to the median heuristic trick by Fukumizu et al.

Original: [2009] for each 1 ≤ t ≤ T .
Revised: [2009] for each 1 ≤ t ≤ T .

Original: The pool of scaling factors SCALE contains 30 positive numbers spaced evenly on a log scale between 0.001 to 0.05.
Revised: The pool of scaling factors SCALE contains 30 positive numbers spaced evenly on a log scale between 0.001 to 0.05.

Original: The number of cross-validation partition K = 5 .
Revised: The number of cross-validation partition K = 5 .

Original: The true target policy value of π is estimated by the mean cumulative rewards of 50 , 000 Monte Carlo trajectories with policy π .
Revised: The true target policy value of π is estimated by the mean cumulative rewards of 50 , 000 Monte Carlo trajectories with policy π .

Original: We compare our
Revised: We compare our

Original: OPE estimator (cid:98) V ( π ) with the target policy value by computing mean absolute error (MAE) for each setting of ( n, T ) , as reported in Figure 2.
Revised: OPE estimator (cid:98) V ( π ) with the target policy value by computing mean absolute error (MAE) for each setting of ( n, T ) , as reported in Figure 2.

Original: Figure 2 validate the derived ﬁnite-sample error bound of our OPE estimator in Theorem 6.3.
Revised: Figure 2 validate the derived ﬁnite-sample error bound of our OPE estimator in Theorem 6.3.

Original: Speciﬁcally, Figure 2 (a) shows that the OPE estimation error is polynomial in T , but with an order slightly smaller than O ( T 7 / 2 ) as stated in Theorem 6.3.
Revised: Speciﬁcally, Figure 2 (a) shows that the OPE estimation error is polynomial in T , but with an order slightly smaller than O ( T 7 / 2 ) as stated in Theorem 6.3.

Original: Figure(b) shows that the convergence rate in terms of the sample size n for our OPE estimator is slower than O ( n − 1 / 2 ) , which also justiﬁes our theoretical results.
Revised: Figure(b) shows that the convergence rate in terms of the sample size n for our OPE estimator is slower than O ( n − 1 / 2 ) , which also justiﬁes our theoretical results.

Original:  Discussion
Revised:  Discussion

Original: In this paper, we propose a non-parametric identiﬁcation and estimation method for OPE in episodic confounded POMDPs with continuous states, relying on time-dependent proxy variables.
Revised: In this paper, we propose a non-parametric identiﬁcation and estimation method for OPE in episodic confounded POMDPs with continuous states, relying on time-dependent proxy variables.

Original: We develop a ﬁtted- Q -evaluation-type algorithm for estimating the V -bridge functions sequentially and for OPE based on the estimated V -bridges.
Revised: We develop a ﬁtted- Q -evaluation-type algorithm for estimating the V -bridge functions sequentially and for OPE based on the estimated V -bridges.

Original: The ﬁrst ﬁnite-sample error bound for estimating the policy value under confounded POMDPs is established, which achieves a polynomial order with respect to the sample size and the length of horizon.
Revised: The ﬁrst ﬁnite-sample error bound for estimating the policy value under confounded POMDPs is established, which achieves a polynomial order with respect to the sample size and the length of horizon.

Original: Our OPE results can serve as a foundation for developing new policy optimization algorithms in the confounded POMDP, which We will leave for future work.
Revised: Our OPE results can serve as a foundation for developing new policy optimization algorithms in the confounded POMDP, which We will leave for future work.

Original: Acknowledgement
Revised: Acknowledgement

Original: Zhang’s research is partially supported by GW University Facilitating Fund.
Revised: Zhang’s research is partially supported by GW University Facilitating Fund.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] The paper’s contributions and scope are summarized in the abstract and then emphasized in Sections 1 and 2.
Revised: [Yes] The paper’s contributions and scope are summarized in the abstract and then emphasized in Sections 1 and 2.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We did not provide results of minimax lower bound for our estimation problem.
Revised: [Yes] We did not provide results of minimax lower bound for our estimation problem.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Main assumptions are stated in Sections 4 and 6.
Revised: [Yes] Main assumptions are stated in Sections 4 and 6.

Original: Some regularity conditions are provided in Appendix A. (b) Did you include complete proofs of all theoretical results?
Revised: Some regularity conditions are provided in Appendix A. (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs for theoretical results are provided in Appendices C and D. 3.
Revised: [Yes] All proofs for theoretical results are provided in Appendices C and D. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We attached the code and instructions in the supplemental materials.
Revised: [Yes] We attached the code and instructions in the supplemental materials.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Provided in Appendix F (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] Provided in Appendix F (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (c) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (d) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (d) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]