Original: Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric ModelsAnonymous Author(s) AfﬁliationAddress
Revised: Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Models

Original: Abstract
Revised: Abstract

Original: We study the problem of off-policy evaluation (OPE) for episodic Partially Observable Markov Decision Processes (POMDPs) with continuous states.
Revised: We study the problem of off-policy evaluation (OPE) for episodic Partially Observable Markov Decision Processes (POMDPs) with continuous states.

Original: Motivatedby the recently proposed proximal causal inference framework, we develop anon-parametric identiﬁcation result for estimating the policy value via a sequenceof so-called V-bridge functions with the help of time-dependent proxy variables.
Revised: Motivated by the recently proposed proximal causal inference framework, we develop a non-parametric identiﬁcation result for estimating the policy value via a sequence of so-called V-bridge functions with the help of time-dependent proxy variables.

Original: We then develop a ﬁtted-Q-evaluation-type algorithm to estimate V-bridge functions recursively, where a non-parametric instrumental variable (NPIV) problem issolved at each step.
Revised: We then develop a ﬁtted-Q-evaluation-type algorithm to estimate V-bridge functions recursively, where a non-parametric instrumental variable (NPIV) problem is solved at each step.

Original: By analyzing this challenging sequential NPIV estimation, weestablish the ﬁnite-sample error bounds for estimating the V-bridge functions andaccordingly that for evaluating the policy value, in terms of the sample size, lengthof horizon and so-called (local) measure of ill-posedness at each step.
Revised: By analyzing this challenging sequential NPIV problem, we establish the ﬁnite-sample error bounds for estimating the V-bridge functions and accordingly that for evaluating the policy value, in terms of the sample size, length of horizon and so-called (local) measure of ill-posedness at each step.

Original: To the bestof our knowledge, this is the ﬁrst ﬁnite-sample error bound for OPE in POMDPsunder non-parametric models.
Revised: To the best of our knowledge, this is the ﬁrst ﬁnite-sample error bound for OPE in POMDPs under non-parametric models.

Original:  Introduction
Revised:  Introduction

Original: In practical reinforcement learning (RL), a representation of the full state which makes the system
Revised: In practical reinforcement learning (RL) a representation of the full state which makes the system

Original: Markovian and therefore amenable to most existing RL algorithms is not known a priori.
Revised: Markovian and therefore amenable to most existing RL algorithms is not known a priori.

Original: Decisionmakers are often facing so-called partial observability of the state information, which signiﬁcantlyhinders the task of RL.
Revised: Decision makers are often facing so-called partial observability of the state information, which signiﬁcantly hinders the task of RL.

Original: In general, agents have to maintain all historical information and establisha belief system on the hidden state for optimal decision making.
Revised: In general, agents have to maintain all historical information and establish a belief system on the hidden state for optimal decision making.

Original: A partially observable Markovdecision process (POMDP) is often used to model the data.
Revised: A partially observable Markov decision process (POMDP) is often used to model the data generating process.

Original: See examples in robotics [Raffertyet al., 2011], precision medicine [Tsoukalas et al., 2015], stochastic game [Hansen et al., 2004] andmany others.
Revised: See examples in robotics [Rafferty et al., 2011], precision medicine [Tsoukalas et al., 2015], stochastic game [Hansen et al., 2004] and many others.

Original: However, it is well known that learning optimal policies in POMDP is computationallyintractable [Papadimitriou and Tsitsiklis, 1987].
Revised: However, it is well known that learning optimal policies in POMDP is computationally intractable [Papadimitriou and Tsitsiklis, 1987].

Original: The issue of partial observability becomes moreserious in the batch setting, where agents are not able to actively collect additional data and furtherexplore the dynamic system.
Revised: The issue of partial observability becomes more serious in the batch setting, where agents are not able to actively collect additional data and further explore the environment.

Original: For example, standard off-policy evaluation (OPE) methods, which aimto learn a policy value from the batch data generated from some behavior policy, would fail to give aconsistent estimate because of unobserved state variables.
Revised: For example, standard off-policy evaluation (OPE) methods, which aim to learn a policy value from the batch data generated from some behavior policy, would fail to give a consistent estimate because of unobserved state variables.

Original: Due to this practical concern, there is a recent line of research studying the OPE under the frameworkof a confounded POMDP, where the behavior policy to generate the batch data is allowed to dependon some unobserved state variables [e.g., Tennenholtz et al., 2020, Nair and Jiang, 2021, Bennettand Kallus, 2021, Shi et al., 2021].
Revised: Due to this practical concern, there is a recent line of research studying the OPE under the framework of a confounded POMDP, where the behavior policy to generate the batch data is allowed to depend on some unobserved state variables [e.g., Tennenholtz et al., 2020, Nair and Jiang, 2021, Bennett and Kallus, 2021, Shi et al., 2021].

Original: Their identiﬁcation results on the policy value are inspired bythe negative controls or so-called proxy variables in the literature of causal inference [e.g., Miaoet al., 2018a, Tchetgen  et al., 2020].
Revised: Their identiﬁcation results on the policy value are inspired by the negative controls or so-called proxy variables in the literature of causal inference [e.g., Miao et al., 2018a, Tchetgen Tchetgen et al., 2020].

Original: A building block of these results is the existence of somebridge functions, namely Q / V -bridge or weight-bridge functions, which are projections of the Q / Vfunctions or importance weights deﬁned over the original state space onto the observation space.
Revised: A building block of these results is the existence of some bridge functions, namely Q / V -bridge or weight-bridge functions, which are projections of the Q / V -functions or importance weights deﬁned over the original state space onto the observation space.

Original: The corresponding statistical estimation of these bridge functions mainly relies on solving some linearintegral equations [e.g., Kress, 1989].
Revised: The corresponding statistical estimation of these bridge functions mainly relies on solving linear integral equations [e.g., Kress, 1989].

Original: Different from the tabular case studied by Tennenholtz et al.[2020] and Nair and Jiang [2021], solving linear integral equations with non-parametric models inthe continuous state/observation space are known to be challenging due to the potential ill-posedness[Chen and Reiss, 2011], leading to slow statistical convergence rates.
Revised: Different from the tabular case studied by Tennenholtz et al. [2020] and Nair and Jiang [2021] and linear models studied by Shi et al. [2021] theoretically, solving linear integral equations with non-parametric models in the continuous state/observation space are known to be challenging due to the potential ill-posedness [Chen and Reiss, 2011], leading to slow statistical convergence rates.

Original: However, existing theoreticalresults developed by Bennett and Kallus [2021] and Shi et al.
Revised: However, existing theoretical results developed by Bennett and Kallus [2021] and Shi et al.

Original: [2021] require fast enough convergencerates for these bridge function estimators in order to establish the asymptotic normality results oftheir estimators for OPE, which could be illusive when using non-parametric models.
Revised: [2021] require fast enough convergence rates for these bridge function estimators in order to establish the asymptotic normality of their estimators for OPE, which could be illusive when the problem is seriously ill-posed under non-parametric models.

Revised: This is different from the supervised learning where a fast enough convergence rate can be easily achieved under non-parametric models.

Original: Therefore, toﬁll this important theoretical gap, it is important to study the ﬁnite-sample performance of OPE ofwhich bridge functions are estimated non-parametrically.
Revised: Therefore, to ﬁll this important theoretical gap, it is necessary to study the ﬁnite-sample performance of OPE of which bridge functions are estimated non-parametrically.

Original: Motivated by these, in this paper, we study the OPEproblem for confounded and episodic POMDPswith continuous states where we non-parametrically estimate V -bridge functions.
Revised: Motivated by these, in this paper, we study the OPE for confounded and episodic POMDPs with continuous states, where we non-parametrically estimate V -bridge functions.

Original: Our main contribution to the literature is three-fold.
Revised: Our main contribution to the literature is three-fold.

Original: First, relying on some time-dependent proxy variables, we establisha non-parametric identiﬁcation result for OPE using V -bridge functions for time-inhomogeneousconfounded POMDPs.
Revised: First, relying on some time-dependent proxy variables, we establish a non-parametric identiﬁcation result for OPE using V -bridge functions for time-inhomogeneous confounded POMDPs.

Original: Based on the identiﬁcation result, we develop a new ﬁtted-Q-evaluation(FQE)-type approach to estimating V -bridge functions recursively and obtain an estimator for OPE basedon the bridge function estimators.
Revised: Based on the identiﬁcation result, we develop a new ﬁtted-Q-evaluation(FQE)- type approach to estimating V -bridge functions recursively and obtain an estimator for OPE based on the bridge function estimators.

Original: At each step of our algorithm, we propose to ﬁt a non-parametricinstrumental variable (NPIV) regression using a min-max estimation method, i.e., solving a linearintegral equation with a non-parametric model.
Revised: At each step of our algorithm, we propose to ﬁt a non-parametric instrumental variable (NPIV) regression using a min-max estimation method, i.e., solving a linear integral equation with a non-parametric model.

Original: Our algorithm can be viewed as a sequential NPIVestimation, which is not well studied in the literature.
Revised: Our algorithm can be viewed as a sequential NPIV estimation, which is not well studied in the literature.

Original: Second and most importantly, we establishthe ﬁnite-sample error bound for estimating V -bridge functions and accordingly that for evaluatingthe policy value, in terms of the sample size , length of horizon and (local) measure of ill-posednessat each step.
Revised: Second and most importantly, we establish the ﬁnite-sample error bound for estimating V -bridge functions and accordingly that for evaluating the policy value, in terms of the sample size , length of horizon and (local) measure of ill-posedness at each step.

Original: Unlike the well studied standard NPIV model in the econometrics literature [e.g., Aiand Chen, 2003, Newey and Powell, 2003] where the response variable is directly observed, theresponse variable in our NPIV model at each step of the algorithm relies on the model estimate atits previous step.
Revised: Unlike the well studied standard NPIV model in the econometrics literature [e.g., Ai and Chen, 2003, Newey and Powell, 2003] where the response variable is directly observed, the response variable in our NPIV model at each step of the algorithm relies on the model estimate at its previous step.

Original: This difference makes our theoretical analysis substantially difﬁcult.
Revised: This difference makes our theoretical analysis substantially difﬁcult.

Original: By carefullycharacterizing the statistical error due to the NPIV estimation at each step and more importantly,its propagation effect on future estimates, we are able to establish the ﬁrst ﬁnite-sample result of 
Revised: By carefully characterizing the statistical error due to the NPIV estimation at each step and more importantly, its propagation effect on future estimates, we are able to establish the ﬁrst ﬁnite-sample result of OPE for confounded

Original: OPE for confounded POMDPs under non-parametric models, which achieves a polynomial orderover the length of horizon and sample size.
Revised: POMDPs under non-parametric models, which achieves a polynomial order over the length of horizon and sample size.

Original: Finally, our theoretical results on the sequential NPIVestimation are generally applicable to other sequential-type conditional moment estimation problems.
Revised: Finally, our theoretical results on the sequential NPIV estimation are generally applicable to other sequential-type conditional moment restriction problems.

Original: The development of the uniform ﬁnite-sample error  of the NPIV estimation, extending the pointwiseresult in the previous literature such as Dikkala et al.
Revised: The development of the uniform ﬁnite-sample error bounds of the NPIV estimation, extending the pointwise result in the previous literature such as Dikkala et al.

Original: [2020], may be of independent interest.
Revised: [2020], may be of independent interest.

Original:  Related Work
Revised:  Related Work

Original: Recently there is a surge of interest in studying OPE with unobserved variables in the sequentialdecision making problem.
Revised: Recently there is a surge of interest in studying OPE with unobserved variables in the sequential decision making problem.

Original: Speciﬁcally, Zhang and Bareinboim [2016] are among the ﬁrst whoproposed the framework of confounded MDPs, which essentially considers i.i.d.
Revised: Speciﬁcally, Zhang and Bareinboim [2016] are among the ﬁrst who proposed the framework of confounded MDPs, which essentially considers i.i.d.

Original: confounders in thedynamic system and therefore preserves the Markovian property.
Revised: confounders in the dynamic system and therefore preserves the Markovian property.

Original: Along this direction, OPE methodsare developed under various identiﬁcation conditions such as partial identiﬁcation using sensitivityanalysis [Namkoong et al., 2020, Kallus and Zhou, 2020, Bruns-Smith, 2021], instrumental variableor mediator assisted OPE
Revised: Along this direction, OPE methods are developed under various identiﬁcation conditions such as partial identiﬁcation using sensitivity analysis [Namkoong et al., 2020, Kallus and Zhou, 2020, Bruns-Smith, 2021], instrumental variable or mediator assisted OPE

Original: [Liao et al., 2021, Li et al., 2021, Shi et al., 2022] and many others.
Revised: [Liao et al., 2021, Li et al., 2021, Shi et al., 2022] and many others.

Original: Anotherline of research focuses on more general  POMDP models (e.g., confounded POMDPs) where the
Revised: Another line of research focuses on more general confounded POMDP models , where the

Original: Markovian assumption is violated, under which several point estimation results were developed suchas the aforementioned negative controls/proxy variables related methods [Tennenholtz et al., 2020, 
Revised: Markovian assumption is violated, under which several point estimation results were developed such as the aforementioned proxy variables related methods [Tennenholtz et al., 2020, Deaner, 2018,

Original: Deaner, 2018, Ying et al., 2021, Bennett and Kallus, 2021, Nair and Jiang, 2021, Shi et al. , 2021],spectral methods in undercomplete POMDPs [Hsu et al., 2012, Anandkumar et al., 2014, Jin et al. ,2020] and predictive state representation related methods [Littman and Sutton, 2001, Singh et al. ,2012, Cai et al., 2022].
Revised: Ying et al., 2021, Bennett and Kallus, 2021, Nair and Jiang, 2021, Shi et al., 2021], spectral methods in undercomplete POMDPs [Hsu et al., 2012, Anandkumar et al., 2014, Jin et al., 2020] and predictive state representation related methods [Littman and Sutton, 2001, Singh et al., 2012, Cai et al., 2022].

Original: Our proposed method, which uses proxy variables for OPE, is closely related to those recentlydeveloped by Bennett and Kallus [2021], Shi et al.
Revised: Our proposed method, which uses proxy variables for OPE, is closely related to those recently developed by Bennett and Kallus [2021], Shi et al.

Original: [2021], and Ying et al.
Revised: [2021], and Ying et al.

Original: Bennett and
Revised: Bennett and

Original: Kallus [2021] and Ying et al.
Revised: Kallus [2021] and Ying et al.

Original: [2021] studied episodic POMDPs (or complex longitudinal studies)and mainly focused on developing asymptotic normality results of their policy value estimators.
Revised: [2021] studied episodic POMDPs (or complex longitudinal studies) and mainly focused on developing asymptotic normality results of their policy value estimators.

Original: Their results rely on some high level rate conditions on the bridge function estimation, which areunknown if they would be satisﬁed when using non-parametric models due to the aforementionedmeasure of ill-posedness.
Revised: Their results rely on some high level rate conditions on the bridge function estimation, which are unknown if they would be satisﬁed when using non-parametric models due to the aforementioned measure of ill-posedness.

Original: Shi et al.
Revised: Shi et al.

Original: [2021] mainly focused on time-homogeneous inﬁnite-horizon
Revised: [2021] mainly focused on time-homogeneous inﬁnite-horizon

Original: POMDPs and developed asymptotic normality for their estimators under similar high-level conditions,which therefore has the same issue.
Revised: POMDPs and developed asymptotic normality for their estimators under similar high-level conditions, which therefore has the same issue.

Original: Besides, while Shi et al.
Revised: Besides, while Shi et al.

Original: [2021] also established ﬁnite-samplebounds for their bridge function estimation and corresponding OPE, they only study the tabularcase or linear/parametric models, where the issue of ill-posedness does not exist .
Revised: [2021] also established ﬁnite-sample bounds for their bridge function estimation and corresponding OPE, they only study the tabular case or linear/parametric models, where the issue of ill-posedness does not exist .

Original: In this paper, weprovide a systematic investigation on the estimation of V -bridge functions and establish ﬁnite-sampleguarantees for them and the corresponding OPE under non-parametric models.
Revised: In this paper, we provide a systematic investigation on the estimation of V -bridge functions and establish ﬁnite-sample guarantees for them and the corresponding OPE under non-parametric models.

Original: Speciﬁcally, wetackle the challenging episodic setting, where V -bridge functions are estimated sequentially.
Revised: Speciﬁcally, we tackle the challenging episodic setting, where V -bridge functions are estimated sequentially.

Original: Withoutcarefully controlling the effect of ill-posedness at each step and its propagation effect on future steps,the estimation error for these V -bridge functions and also that for OPE could be exponentially largein terms of the length of horizon.
Revised: Without carefully controlling the effect of ill-posedness at each step and its propagation effect on future steps, the estimation error for these V -bridge functions and also that for OPE could be exponentially large in terms of the length of horizon.

Original: Motivated by the chaining argument in the empirical process theory,we successfully disentangle the effects of ill-posedness on the current step and future steps separatelyand thus establish ﬁnite-sample bounds for V -bridge functions and OPE both with a polynomialdependence on the length of horizon, which are new theoretical results we contribute to the literature.
Revised: Motivated by the chaining argument in the empirical process theory, we successfully disentangle the effects of ill-posedness on the current step and future steps separately and thus establish ﬁnite-sample bounds for V -bridge functions and OPE both with a polynomial dependence on the length of horizon, which are new theoretical results we contribute to the literature.

Original: Since our V -bridge function estimation can be formulated as a sequential NPIV problem, it is natuallyrelated to classical NPIV estimations, which have been extensively studied in the econometricsliterature [see, e.g., Newey and Powell, 2003, Ai and Chen, 2003, 2012, Hall and Horowitz, 2005,
Revised: Since our V -bridge function estimation can be formulated as a sequential NPIV problem, it is natually related to classical NPIV estimations, which have been extensively studied in the econometrics literature [see, e.g., Newey and Powell, 2003, Ai and Chen, 2003, 2012, Hall and Horowitz, 2005,

Original: Chen and Reiss, 2011, Chen and Christensen, 2018, Darolles et al., 2011, Blundell et al., 2007, forearlier reference].
Revised: Chen and Reiss, 2011, Chen and Christensen, 2018, Darolles et al., 2011, Blundell et al., 2007, for earlier reference].

Original: Recently there is also a growing interest in the min-max estimation in NPIVmodels [see, e.g., Muandet et al., 2020, Dikkala et al., 2020, Hartford et al., 2017, for some recentdevelopments].
Revised: Recently there is also a growing interest in the min-max estimation for NPIV models [see, e.g., Muandet et al., 2020, Dikkala et al., 2020, Hartford et al., 2017, for some recent developments].

Original: As commented before, existing theoretical results for standard NPIV models cannotbe directly applied to our setting due to the sequential structure of our FQE algorithm, so we needto develop new theory to address our setting.
Revised: As commented before, existing theoretical results for standard NPIV models cannot be directly applied to our setting due to the sequential structure of our FQE algorithm, so we need to develop new theory to address our setting.

Original: Technically, in order to establish a polynomial-orderﬁnite-sample error bound over the length of horizon for OPE, which is particularly important in RL,we decompose the measure of ill-posedness at each step of our sequential NPIV estimation into twocomponents: the so-called (local) measure of one-step transition ill-posedness and the standard (local)measure of ill-posedness [e.g., Chen and Pouzo, 2012].
Revised: Technically, in order to establish a polynomial-order ﬁnite-sample error bound over the length of horizon for OPE, which is particularly important in RL, we decompose the measure of ill-posedness at each step of our sequential NPIV estimation into two components: the so-called (local) measure of one-step transition ill-posedness and the standard (local) measure of ill-posedness [e.g., Chen and Pouzo, 2012].

Original: Thanks to this novel decomposition, the effectof the ﬁrst component on the estimation error of V -bridge functions and OPE is multiplicative but canbe properly controlled while that of the second component could be large but is only cumulative.
Revised: Thanks to this novel decomposition, the effect of the ﬁrst component on the estimation error of V -bridge functions and OPE is multiplicative but can be properly controlled while that of the second component could be large but is only cumulative.

Original: See
Revised: See

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: Finally, we remark that Ai and Chen [2012] also studied the sequential NPIV estimationproblem, where the non-parametric components are estimated jointly.
Revised: Finally, we remark that Ai and Chen [2012] also studied the sequential NPIV estimation problem, where the non-parametric components are estimated jointly.

Original: However, this method could becomputationally inefﬁcient in RL with a long horizon.
Revised: However, this method could be computationally inefﬁcient in RL with a long horizon.

Original: More importantly, their results are built on thenested structure among conditional moment restriction models, which are not satisﬁed in our setting.
Revised: More importantly, their results are built on the nested structure among conditional moment restriction models, which are not satisﬁed in our setting.

Original:  Preliminaries and Notations
Revised:  Preliminaries and Notations

Original: In this section, we introduce the framework of discrete-time confounded POMDPs and its related OPE problem.
Revised: In this section, we introduce the framework of discrete-time confounded POMDPs and its related OPE problem.

Original: Consider an episodic and confounded POMDP denoted by M =
Revised: Consider an episodic and confounded POMDP denoted by M =

Original: ( S , U , A , T, P , r ) ,with S and U as the observed and unobserved continuous state spaces respectively,
Revised: ( S , U , A , T, P , r ) , with S and U as the observed and unobserved continuous state spaces respectively,

Original: A as the discreteaction space, T as the length of horizon, P = { P t } Tt =1 as the transition kernel over S × U × A to S × U , and r = { r t } Tt =1 as the reward function over S ×  A .
Revised: A as the discrete action space, T as the length of horizon, P = { P t } Tt =1 as the transition kernel over S × U × A to S × U , and r = { r t } Tt =1 as the reward function over S × U × A .

Original: S can also be treated as the observationspace in the classical POMDP.
Revised: S can also be treated as the observation space in the classical POMDP.

Original: Then the process of M can be summarized as { S t , U t , A t , R t } Tt =1with S t and U t as observed and unobserved state variables, A t as the action, and R t as the reward,where r t ( S t , A t ) =
Revised: Then the process of M can be summarized as { S t , U t , A t , R t } Tt = with S t and U t as observed and unobserved state variables, A t as the action, and R t as the reward, where r t ( s, u, a ) =

Original: E [ R t | S t , A t  ]  .
Revised: E [ R t | S t = s, U t = u, A t = a ] for any ( s, u, a ) ∈ S × U × A .

Original: For simplicity, we assume that | R t | ≤ 1 uniformly in 1 ≤ t ≤ T .
Revised: For simplicity, we assume that | R t | ≤ 1 uniformly in 1 ≤ t ≤ T .

Original: The goal of OPE in a confounded POMDP is to evaluate the performance of a target policy usingthe batch data collected by some behavior policy.
Revised: The goal of OPE in a confounded POMDP is to evaluate the performance of a target policy using the batch data collected by some behavior policy.

Original: In this paper, the target policy we focus on is asequence of functions mapping from the state space S to a probability mass function over the actionspace A , denoted by π = { π t } Tt =1 , where π t ( a | s ) is the probability of choosing an action A t = agiven the state value S t = s .
Revised: In this paper, the target policy we focus on is a sequence of functions mapping from the state space S to a probability mass function over the action space A , denoted by π = { π t } Tt =1 , where π t ( a | s ) is the probability of choosing an action A t = a given the state value S t = s .

Original: We remark that our proposed identiﬁcation results stated in Section 4can be generalized to other policies such as history-dependent ones.
Revised: We remark that our proposed identiﬁcation results stated in Sectioncan be generalized to other policies such as history-dependent ones.

Original: Given a target policy π , deﬁneits state value function as
Revised: Given a target policy π , deﬁne its state value function as

Original: [Equation]
Revised: [Equation]

Original: where E π denotes the expectation with respect to the distribution whose action at decision time tfollows π t for any t ≥ 1 .
Revised: where E π denotes the expectation with respect to the distribution whose action at decision time t follows π t for any t ≥ 1 .

Original: We consider the batch setting, where the observed action A t is generatedby some behavior policy ˜ π bt depending on both S t and U t for 1 ≤ t ≤ T .
Revised: We consider the batch setting, where the observed action A t is generated by some behavior policy ˜ π bt depending on both S t and U t for 1 ≤ t ≤ T .

Original: We aim to use this batchdata to estimate the policy value of a target policy π , which is deﬁned as
Revised: We aim to use the batch data to estimate the policy value of a target policy π , which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: wherewe use E to denote the expectation with respect to the behavior policy.
Revised: where E denotes the expectation with respect to the behavior policy.

Original: Since U t is unobserved,standard OPE methods that rely on the Markov assumption willin general fail and give bias estimations.
Revised: Due to the unobserved U t , standard OPE methods that rely on the Markovianity will give bias estimations.

Original: In the following, we introduce an identiﬁcation result for estimating the policy value with thehelp of some proxy variables.
Revised: In the following, we introduce an identiﬁcation result for estimating the policy value using some proxy variables.

Original: Notations :  { (cid:36) ( n ) } n ≥ 1 and { θ ( n ) } n ≥ 1 , the notation (cid:36) ( n ) (cid:38) θ ( n ) (resp.
Revised: Notations : For two sequences { (cid:36) ( n ) } n ≥ 1 and { θ ( n ) } n ≥ 1 , the notation (cid:36) ( n ) (cid:38) θ ( n ) (resp.

Original: (cid:36) ( n ) (cid:46) θ ( n ) ) meansthat there exists a sufﬁciently large constant (resp. small) constant c 1 > 0 (resp.
Revised: (cid:36) ( n ) (cid:46) θ ( n ) ) means that there exists a sufﬁciently large constant (resp. small) constant c 1 > 0 (resp.

Original: c 2 > 0 ) suchthat (cid:36) ( N ) ≥ c 1 θ ( N ) (resp.
Revised: c 2 > 0 ) such that (cid:36) ( N ) ≥ c 1 θ ( N ) (resp.

Original: (cid:36) ( n ) ≤ c 2 θ ( n ) ).
Revised: (cid:36) ( n ) ≤ c 2 θ ( n ) ).

Original: We use (cid:36) ( n ) (cid:16) θ ( n ) when (cid:36) ( n ) (cid:38) θ ( n ) and(cid:36) ( n ) (cid:46) θ ( n ) .
Revised: We use (cid:36) ( n ) (cid:16) θ ( n ) when (cid:36) ( n ) (cid:38) θ ( n ) and (cid:36) ( n ) (cid:46) θ ( n ) .

Original: For any random variable X , we use L q ( X ) to denote the class of all measurablefunctions with ﬁnite q -th moments for 1 ≤ q ≤ ∞ .
Revised: For any random variable X , we use L q { X } to denote the class of all measurable functions with ﬁnite q -th moments for 1 ≤ q ≤ ∞ .

Original: Then the L q -norm is denoted by (cid:107) • (cid:107)
Revised: Then the L q -norm is denoted by (cid:107) • (cid:107)

Original: L q ( X ) .
Revised: L q { X } .

Original: When there is no confusion in the underlying distribution, we also write it as (cid:107) • (cid:107) L q or (cid:107) • (cid:107) q .
Revised: When there is no confusion in the underlying distribution, we also write it as (cid:107) • (cid:107) L q or (cid:107) • (cid:107) q .

Original: Inparticular, (cid:107) • (cid:107) ∞ denotes the sup-norm.
Revised: In particular, (cid:107) • (cid:107) ∞ denotes the sup-norm.

Original: In addition, we use Big O and small o as the convention.
Revised: In addition, we use Big O and small o as the convention.

Original:  Identiﬁcation Results
Revised:  Identiﬁcation Results

Original: Inspired by the proximal causal inference recently proposed by Tchetgen  et al.
Revised: Inspired by the proximal causal inference recently proposed by Tchetgen Tchetgen et al.

Original: [2020], we developa non-parametric identiﬁcation result for estimating V ( π ) , which is similar to those by Bennettand Kallus [2021] and Shi et al.
Revised: [2020], we develop a non-parametric identiﬁcation result for estimating V ( π ) , which is similar to those by Bennett and Kallus [2021] and Shi et al.

Original: Assume that we can additionally observe the so-calledreward-inducing proxy variables W t that are only related to the action A t through ( S t , U t ) andaction-inducing proxy variables Z t that are only related to the reward R t through ( S t , U t ) at eachdecision time t .
Revised: Assume that we can additionally observe the so-called reward-inducing proxy variables W t that are only related to the action A t through ( S t , U t ) and action-inducing proxy variables Z t that are only related to the reward R t through ( S t , U t ) at each decision time t .

Original: See Figure 1 for a directed acyclic graph (DAG) to illustrate their relationships anda time series data example in Miao et al.
Revised: See Figure 1 for a directed acyclic graph (DAG) to illustrate their relationships and a time series data example in Miao et al.

Original: [2018b].
Revised: [2018b].

Original: For another example, the action-inducing proxyvariables Z t can beimplicitly deﬁned as the observed history before time t , then Z t and relatedarrows in Figure 1 can be removed.
Revised: For another example, the action-inducing proxy variables Z t can be deﬁned as the observed history before time t , then Z t and related arrows in Figure 1 can be removed.

Original: Detailed assumptions and discussion are given in Appendix A. Denote the spaces of { Z t } Tt =1 and { W t } Tt =1 by W and Z respectively. 
Revised: Detailed assumptions and discussion are given in Appendix A. Denote the spaces of { Z t } Tt =1 and { W t } Tt =1 by W and Z respectively. U t − 1 S t −

Original: [Figure]
Revised: Z t − W t − A t −

Original: [Equation]
Revised: R t − U t S t Z t W t

Original: [Figure]
Revised: A t R t U t +1 S t + Z t +

Original:  Since the states { U t } Tt =1 are unmeasured, we cannot estimate the value function by the celebrated
Revised: W t + A t + R t + Since the states { U t } Tt =1 are unmeasured, we cannot estimate the value function by the celebrated

Original: Bellman equation.
Revised: Bellman equation.

Original: However, with the help of confounding proxies { W t , Z t } Tt =1 , the value of a targetpolicy π can be non-parametrically identiﬁed using observed variables under proper assumptions.
Revised: However, with the help of confounding proxies { W t , Z t } Tt =1 , the value of a target policy π can be non-parametrically identiﬁed using observed variables under proper assumptions.

Original: To proceed, we deﬁne a class of V -bridge functions (or V -bridges for short) { v πt } Tt =1 deﬁned over
Revised: To proceed, we deﬁne a class of V -bridge functions (or V -bridges for short) { v πt } Tt =1 deﬁned over

Original: W × S such that for every ( s, u ) ∈ S
Revised: W × S such that for every ( s, u ) ∈ S

Original: × U and t ≥ 1 ,
Revised: × U and t ≥ 1 ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If such V -bridges exist, then we obtain the following identiﬁcation result for the policy value in (2).
Revised: If such V -bridges exist, then we obtain the following identiﬁcation result for the policy value in (2).

Original: Proposition 4.1 (Identiﬁcation) .
Revised: Proposition 4.1 (Identiﬁcation) .

Original: If there exist { v πt } Tt =1 that satisfy (3), then the value of  policy πcan be identiﬁed by V ( π ) =
Revised: If there exist { v πt } T t =1 that satisfy (3), then the value of target policy π can be identiﬁed by V ( π ) =

Original: E [ v π 1 ( W 1 , S 1 )] .
Revised: E [ v π 1 ( W 1 , S 1 )] .

Original: Note that V -bridges { v πt } Tt =1 that satisfy (3) are not necessarily unique, but we can  identify V ( π )based on any of them.
Revised: Note that V -bridges { v πt } Tt =1 that satisfy (3) are not necessarily unique, but we can uniquely identify V ( π ) based on any of them.

Original: Next, we provide a theoretical guarantee for the existence of V -bridges{ v πt } Tt =1 in terms of a sequence of linear integral equations.
Revised: Next, we provide a theoretical guarantee for the existence of V -bridges { v πt } Tt =1 in terms of a sequence of linear integral equations.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For a POMDP model of which variables satisfy the relationships illustrated in Figure 1and some regularity conditions given in Appendix A, there always exist V -bridges { v πt } Tt =1 satisfying(3).
Revised: For a POMDP model of which variables satisfy the relationships illustrated in Figureand some regularity conditions given in Appendix A, there always exist V -bridges { v πt } Tt =1 satisfying (3).

Original: With v πT +

Revised: With v πT +

Original: =
Revised: =

Original: 0 , a particular sequence of V -bridges { v πt } Tt =1 can be obtained by solving thefollowing linear integral equations:
Revised: 0 , a particular sequence of V -bridges { v πt } Tt =1 can be obtained by solving the following linear integral equations:

Original: [Equation]
Revised: [Equation]

Original: where { q πt } Tt =1 are Q -bridges deﬁned over W × S
Revised: where { q πt } Tt =1 are Q -bridges deﬁned over W × S

Original: ×
Revised: ×

Original: A such that
Revised: A such that

Original: [Equation]
Revised: [Equation]

Original: for every ( s, u, a ) ∈ S × U × A and t ≥ 1 , and v πt ( w, s ) = (cid:80) a ∈A π t ( a | s ) q πt ( w, s, a ) .
Revised: for every ( s, u, a ) ∈ S × U × A and t ≥ 1 , and v πt ( w, s ) = (cid:80) a ∈A π t ( a | s ) q πt ( w, s, a ) .

Original: Clearly(4)also guarantees the existence of Q -bridges { q πt } Tt =1 .
Revised: Clearly Q -bridges { q πt } Tt =1 also exist.

Original: Theorem 4.1 guarantees the existence of both V -bridges{ v πt } Tt =1 and Q -bridges, and also provides anatural procedure (4) to ﬁnd { v πt } Tt =1 . in order to eventually estimate the policy value V ( π ) .
Revised: Theorem 4.1 guarantees the existence of both V -bridges and Q -bridges, and also provides a natural procedure (4) to ﬁnd { v πt } Tt =1 and eventually estimate the policy value V ( π ) .

Original: Thendue to Proposition 4.1 and Theorem 4.1, we can perform OPE via Algorithm 1 in the population sense.
Revised: Then based on Proposition 4.1 and Theorem 4.1, we can perform OPE via Algorithm 1 in the population level.

Original: At the t -th step, we propose to solve (4) with a non-parametric model, which is a NPIV problem.
Revised: Speciﬁcally at each step we will solve (4) via a non-parametric model, which is a NPIV problem.

Original: 1 Input: { ( S t , Z t , W t , A t , R t ) T t =1 , a target policy π = { π t 
Revised: 1 Input: { ( S t , W t , Z t , A t , R t ) } Tt =1 , a target policy π = { π t } Tt =1 .

Original: T t =1 . 2 Let v πT +1 = 
Revised: 2 Let v πT +1 = 0 .

Original: 3 Repeat for t = T, . . .
Revised: 3 Repeat for t = T, . . .

Original: , 1 : Solve v πt and q πt by E (cid:8) q πt ( W t , S t , A t ) − R t − v πt
Revised: Solve v πt and q πt by E (cid:8) q πt ( W t , S t , A t ) − R t − v πt

Original: +

Revised: +

Original: ( W t +1 , S t 
Revised: ( W t +1 , S t +1 )

Original: +

Original: 1 ) | Z t  S t  A t (cid:9) 
Revised: | Z t , S t , A t (cid:9) =

Original: = 0 with v πt ( W t , S t ) (cid:44) (cid:80) a ∈A π t ( a | S t ) q πt 
Revised: with v πt ( W t , S t ) (cid:44) (cid:80) a ∈A π t ( a | S t ) q πt ( W t , S t , a ) .

Original: ( W t , S t ,

Original: a Output: V  π ) =
Revised: 5 Output: V ( π ) =

Original: E [ v π 1 ( S 1 , W 1 )] .
Revised: E [ v π 1 ( W 1 , S 1 )] .

Original:  Estimation
Revised:  Estimation

Original: In this section, we discuss how to estimate V ( π ) using batch data based on results given in Theorem4.1 and Algorithm 1.
Revised: In this section, we discuss how to estimate V ( π ) using batch data based on results given in Theorem 4.1 and Algorithm 1.

Original: Let a pre-collected training dataset be D n = { ( S i,t , W i,t , Z i,t , A 
Revised: Let a pre-collected training dataset be D n = { ( S t,i , W t,i , Z t,i , A t,i , R t,i )

Original: i,t , R i,t ) Tt =1 :i = 1 , . . . , n } , which consists of n i.i.d.
Revised: Tt =1 : i = 1 , . . . , n } , which consists of n i.i.d.

Original: copies of the observable variables ( S t , W t , Z t , A t , R t ) Tt =1 ofa confounded POMDP.
Revised: copies of the observable trajectory ( S t , W t , Z t , A t , R t ) Tt = of a confounded POMDP.

Original: Following Algorithm 1, we develop a FQE-type approach where we proposeto solve a min-max problem for estimating v πt at the t -th step using the idea of Dikkala et al.
Revised: Following Algorithm 1, we develop a FQE-type approach where we propose to solve a min-max problem for estimating v πt at the t -th step using the idea of Dikkala et al.

Original: [2020],1 ≤ t ≤ T , and then apply Proposition 4.1 for OPE.
Revised: [2020], and then apply Proposition 4.1 for OPE.

Original: For convenience, we ﬁrst rewrite the linear integral equations (4) to solve V -bridges in terms of operators.
Revised: For convenience, we ﬁrst rewrite the linear integral equations (4) for solving V -bridges in terms of operators.

Original: Deﬁne an operator (cid:101) P t : L 2 {R × W × S} →
Revised: Deﬁne an operator (cid:101) P t : L 2 {R×W ×S} →

Original: L 2 {Z × A × S} such that [ (cid:101) P πt g ]( Z t , A t , S t ) =
Revised: L 2 {Z ×S×A} such that [ (cid:101) P t g ] ( Z t , S t , A t ) =

Original: 196 E [(cid:104) g ( R t , W t+1 , · , S t +1 ) , π t +1 ( · | S t +1 ) (cid:105) | Z t , A t , S t ] for any g ∈ L 2 {R × W × A × S} , where(cid:104)
Revised: E [ g ( R t , W t

Original: g ( R t , W t +1 , · , S t +1) ,π t +1 ( · | S t +1 )(cid:105) (cid:44) (cid:80) a ∈A g
Revised: +1 , S t +1 )

Original: ( R t , W t+1 ,a, S t+1 ) , π t +1 ( a | S t +1 ) .
Revised: | Z t , S t , A t ] for any g ∈ L 2 {R × W × S} .

Original: Deﬁne another operator P t :
Revised: Deﬁne another operator P t :

Original: L 2 {W × A × S} → L 2 {Z × A × S} such that for any h ∈ L 2 {W × A × S} ,[ P t h ]( Z t , A t , S t ) =
Revised: L 2 {W × S × A} → L 2 {Z × S × A} such that for any h ∈ L 2 {W × S × A} , [ P t h ]( Z t , S t , A t ) =

Original: E [ h ( W t , A t , S t ) | Z t , A t , S t ] .
Revised: E [ h ( W t , S t , A t ) | Z t , S t , A t ] .

Original: Motivated by (4), we deﬁne the V-bridge transition operator P πt :L 2 {R × W × S} →
Revised: Motivated by (4), we deﬁne the V-bridge transition operator P πt :

Original: L 2  {W × S} such that 
Revised: L 2 {R × W × S} → L 2 {W × S} such that [Equation]

Original: P πt g = (cid:104) π t ,  P t g (cid:105) , where P t  g =
Revised: , where P t g = P t [Equation] all g [Equation]

Revised: L

Revised: [Equation]

Revised: R × W

Original: P t − 1 (cid:101) P t g for all g ∈ L 2 {R × W × S} , (cid:104) π t ( · | S t ) , [ P t g ]( W t ,· , S t  ) (cid:105) (cid:44) (cid:80) a ∈A π t ( a | S t )
Revised: [Equation] S [Equation] In particular, (cid:104) π t ( · | S t ) , [ P t g ]( W t , S t , · ) (cid:105) (cid:44) (cid:80) a ∈A π t ( a | S t )

Original: [ P t g ]( W t ,· , S t  ) , and (cid:101) P t g  invertible by P t .
Revised: [ P t g ]( W t , S t , a ) , and (cid:101) P t g is invertible by P t .

Original: Theinvertibility is ensured by Assumption 8 in Appendix A.
Revised: The invertibility is ensured by Assumption 8 in Appendix A.

Original: Then by the deﬁnition of V -bridges and (4), we can identify { v πt } Tt =1equivalently via solvingv πt =
Revised: Then by the deﬁnition of V -bridges and (4), we can identify { v πt } Tt =1 via solving [Equation]

Original: P πt ( v πt +1 + R t ) , for t ≥ 1 ,(6) so it sufﬁces to estimate P πt .
Revised: To ﬁnd the estimated V-bridges { ˆ v πt } Tt =1 , it sufﬁces to estimate P πt .

Original: We regard (6) as a series of conditional moment model restrictions andpropose to solve them via a sequential NPIV estimation.
Revised: Note that one can regard (6) as a series of conditional moment model restrictions and we propose to solve them via a sequential NPIV estimation.

Original: In particular, at the t -th step, we adopt themin-max estimation method proposed by Dikkala et al.
Revised: In particular, at the t -th step, we adopt the min-max estimation method proposed by Dikkala et al.

Original: [2020] to estimate P πt non-parametrically asfollows: (cid:98) P πt g
Revised: [2020] to estimate P πt non-parametrically as follows: (cid:98) P πt g

Original: =
Revised: =

Original: (cid:68) π t , (cid:98) P t g (cid:69) , where
Revised: (cid:68) π t , (cid:98) P t g (cid:69) , where

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) f
Revised: where (cid:107) f

Original: (cid:107) 2 n = n − 1 (cid:80) nk
Revised: (cid:107) 2 n = n − 1 (cid:80) ni

Original: =1 f 2 ( Z t,k , A t,k , S t,k ) , the user-deﬁned function spaces H ( t ) on W × A × Sand F ( t ) on Z ×A × S  are  endowed with norms (cid:107) • (cid:107) H ( t ) and (cid:107) • (cid:107) F ( t ) respectively,and tuningparameters λ, µ, M, δ > 0 , and [Equation]
Revised: =1 f 2 ( Z t,i , S t,i , A t,i ) for f ∈ F ( t ) . H ( t ) on W × S × A and F ( t ) on Z × S × A are two user-deﬁned function spaces endowed with norms (cid:107) • (cid:107) H ( t ) and (cid:107) • (cid:107) F ( t ) respectively, λ, µ, M, δ > 0 are tuning parameters, and Ψ t,n ( h, f, g )

Original: n [Equation]
Revised: =

Original: where g ( R t , W t +1 , S t +1 )
Revised: n − 1 (cid:80) ni

Original: ={+=
+}
Revised: {+=
+}

Revised: [ h ( W t,i , S t,i , A t,i ) − ( T − t +1) − 1 g

Revised: ( R t,i , W t +1 ,i , S t +1 ,i )]

Revised: f ( Z t,i , S t,i , A t,i ) , where g ( R t , W t +1 , S t +1 ) =

Original: R t + ¯ g ( W t +1 , S t +1 ) for some ¯ g ∈ G ( t +1) on W × S , endowed withnorm (cid:107) • (cid:107) G ( t +1) .
Revised: R t + ¯ g ( W t +1 , S t +1 ) for some ¯ g ∈ G ( t +1) on W × S , endowed with norm (cid:107) • (cid:107) G ( t +1) .

Original: The rational behind (7) is that when λ, λµ → 0 and λM/δ 2 (cid:16) 1 , the following two population-versionmin-max optimization problems[Equation] and
Revised: The rational behind (7) is that when λ, λµ → 0 and λM/δ 2 (cid:16) 1 , the following two population-version min-max optimization problems

Original: [Equation]
Revised: [Equation]

Original: have the same solution h when the space F ( t ) of testing functions is rich enough.
Revised: have the same solution h when the space F ( t ) of testing functions is rich enough.

Original: Note that [Equation] +1 216 used above and in (7) are for scaling purpose.
Revised: Note that ( T − t + 1) − 1 used above and in (7) are for scaling purpose.

Original: After T steps, we output our estimator for the policy value based on the empiricalcounterpart ofProposition 4.1.
Revised: After T steps, we output our estimator for the policy value based on the empirical counterpart of

Original: [-Our FQE-type algorithm is summarized in Algorithm 2. Algorithm 2: A FQE-type algorithm with a sequential NPIV estimation ˆ v πT +
 = 0 .-]
Revised: [-Our FQE-type algorithm is summarized in Algorithm 2. Algorithm 2: A FQE-type algorithm with a sequential NPIV estimation ˆ v πT +
 = 0 .-]Proposition 4.1.

Original: 3 Repeat for t = T, . . . Solve (cid:98) P t by (7) and compute ˆ v πt = (cid:98) P πt (ˆ v πt +1 + R t ) where (cid:98) P πt g = (cid:68) π t , (cid:98) P t g (cid:69) .
Revised: Our FQE-type algorithm is summarized in Algorithms 2 and 3 in Appendix E.

Original: 5 Output: (cid:98) V ( π ) = n − 1 (cid:80)

Original: nk

Original: =1 ˆ v π 1 ( W 1 ,k , S 1 ,k ) .

Original:  Theoretical Results
Revised:  Theoretical Results

Original: In this section, we establish the ﬁnite-sample bounds for the L 2 error of estimating V -bridge v π 1and the error of OPE, in terms of the sample size, length of horizon and two (local) measures ofill-posedness.
Revised: In this section, we establish the ﬁnite-sample bounds for the L 2 error of estimating V -bridge v πand the error of OPE, in terms of the sample size, length of horizon and two (local) measures of ill-posedness.

Original: Our bounds also reply on the critical radii of certain spaces related to user-deﬁnedfunction spaces H ( t ) and F ( t ) in (7), and also G ( t ) of V -bridge functions.1.
Revised: Our bounds also rely on the critical radii of certain spaces related to the user-deﬁned function spaces H ( t ) and F ( t ) in (7), and also G ( t ) of V -bridge functions.

Original: Technical preliminaries.
Revised: Technical preliminaries.

Original: To present our main results, we ﬁrst introduce some concepts from theempirical process theory [Wainwright, 2019].
Revised: Before presenting our main results, we ﬁrst introduce some concepts from the empirical process theory [Wainwright, 2019].

Original: Deﬁnition 6.1 (Local Rademacher Complexity) .
Revised: Deﬁnition 6.1 (Local Rademacher Complexity) .

Original: Given any real-valued function class F deﬁnedover a random vector X and any radius δ > 0 , the local Rademacher complexity is given by[Equation] n ( F , δ ) = E (cid:15),X (cid:34) sup
Revised: Given any real-valued function class F deﬁned over a random vector X and any radius δ > 0 , the local Rademacher complexity is given by

Original: [Equation]
Revised: [Equation]

Original: : (cid:107) f (cid:107) n ≤ δ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)n n (cid:88) i = (cid:15) i f ( X i ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35) , (8) where { X i } ni =1 are i.i.d.
Revised: where { X i } ni =1 are i.i.d.

Original: copies of X and { (cid:15) i } ni =1 are i.i.d.
Revised: copies of X and { (cid:15) i } ni =1 are i.i.d.

Original: Rademacher random variables.
Revised: Rademacher random variables.

Original: Let

Original: R n ( F ) (cid:44) R n ( F , ∞ ) denote the non-localized Rademacher complexity.

Original: By bounding the local Rademacher complexity, which measures the complexity of the functionalclass F locally in a neighborhood of the ground truth, we can control the error rate of the proposed
Revised: By bounding the local Rademacher complexity, which measures the complexity of the functional class F locally in a neighborhood of the ground truth, we can control the error rate of the proposed

Original: V -bridge estimator in each step.
Revised: V -bridge estimator in each step.

Original: A crucial parameter for local Rademacher complexity of a functionclass F is the critical radius .
Revised: A crucial parameter for local Rademacher complexity of a function class F is called critical radius .

Original: Deﬁnition 6.2 (Critical Radius) .
Revised: Deﬁnition 6.2 (Critical Radius) .

Original: Assume that F is a star-shaped function class, i.e. αf ∈ F for anyf ∈ F and scalar α ∈ [0 , 1] , and also that F is b -uniformly bounded, i.e., (cid:107) f (cid:107) ∞ ≤ b < ∞ , ∀ f ∈ F .
Revised: Assume that F is a star-shaped function class, i.e. αf ∈ F for any f ∈ F and scalar α ∈ [0 , 1] , and also that F is b -uniformly bounded, i.e., (cid:107) f (cid:107) ∞ ≤ b < ∞ , ∀ f ∈ F .

Original: The critical radius of F , denoted by δ n , is the solution to the inequality R n ( F , δ ) ≤ δ 2 /b.
Revised: The critical radius of F , denoted by δ n , is the solution to the inequality R n ( F , δ ) ≤ δ 2 /b.

Original: Additional Notations : We assume that the test functions f belong to a star shaped, symmetric space
Revised: Additional Notations : We assume that the test functions f belong to a star shaped, symmetric space

Original: F ( t ) ⊆
Revised: F ( t ) ⊆

Original: L 2 ( Z ×A × S  ) endowed with norm (cid:107) · (cid:107) F ( t ) .
Revised: L 2 ( Z × S × A ) endowed with norm (cid:107) · (cid:107) F ( t ) .

Original: For brevity of notation, hereafter we suppressstep indicator ( t ) in the context unless needed to specify it.
Revised: For brevity of notation, hereafter we suppress the time-step indicator ( t ) in the context unless necessary.

Original: For a function space F , we deﬁneα F = { αf : f ∈ F} ,  α ∈ R .
Revised: For a function space F , we deﬁne α F = { αf : f ∈ F} , for some α ∈ R .

Original: Deﬁne F B = { f ∈ F : (cid:107) f (cid:107) 2 F ≤ B } , for any B > 0 .
Revised: Deﬁne F B = { f ∈ F : (cid:107) f (cid:107) 2 F ≤ B } , for any B > 0 .

Original: Deﬁnethe projected root mean squared error (cid:107) proj t f (cid:107) [-
-]
Revised: Deﬁne the projected root mean squared error (cid:107) proj t f (cid:107) [-
-]2 = (cid:112) E

Original: = (cid:112) E { E [ f ( X ) | Z t , A t , S t ] } 2 , for any squaredintegrable f with respect to the conditional distribution of X given ( Z t , A t , S t ) .
Revised: { E [ f ( X ) | Z t , S t , A t ] } 2 , for any squared integrable f with respect to the conditional distribution of X given ( Z t , S t , A t ) .

Original: Standard (Local) Measures of ill-posedness : Let  τπ 1 = sup g ∈G (1) (cid:107) g
Revised: Standard (Local) Measures of ill-posedness : Let ¯ τ 1 = sup g ∈G (1) (cid:107) g

Original: ( W 1 , S 1 ) (cid:107) 2 / (cid:107) Eπ 1 [ g ( W 1 , S 1 ) |
Revised: ( W 1 , S 1 ) (cid:107) 2 / (cid:107) E [ g ( W 1 , S 1 ) |

Original: Z 1 , S 1 ] (cid:107) 2 be the measure of ill-posedness for G (1) ( W 1 × S 1 ) projected on Z 1 × S 1 .
Revised: Z 1 , S 1 ] (cid:107) 2 be the measure of ill-posedness for G (1) ( W 1 × S 1 ) projected on Z 1 × S 1 .

Original: Let τ t =
Revised: Let τ t =

Original: sup h ∈H ( t ) (cid:107) h
Revised: sup h ∈H ( t ) (cid:107) h

Original: ( W t , S t , A t ) (cid:107) 2 / (cid:107) proj t h ( W t , S t , A t ) (cid:107) 2 be the standard measure of ill-posedness for
Revised: ( W t , S t , A t ) (cid:107) 2 / (cid:107) proj t h ( W t , S t , A t ) (cid:107) 2 be the standard measure of ill-posedness for

Original: H ( t ) ( Wt × A t × S t ) projected on Zt × A t × S t .
Revised: H ( t ) ( W × S × A ) projected on Z × S × A .

Original: It can be seen that  τπ 1 , τ t ≥ 1 for t ≥ 1 .
Revised: It can be seen that ¯ τ 1 , τ t ≥ 1 for t ≥ 1 .

Original: Indeedwe only require measuring  τπ 1 and τ t locally .
Revised: Indeed we only require measuring ¯ τ 1 and τ t locally .

Original: See more details in Appendix C.2.
Revised: See more details in Appendix C.

Original: Results.
Revised: 2. Results

Original: We ﬁrst give Assumption 1 used to develop our theoretical results below.
Revised: We ﬁrst give Assumption 1 used to develop our theoretical results below.

Original: Assumption 1.
Revised: Assumption 1.

Original: For each t = 1 , . . .
Revised: For each t = 1 , . . .

Original: , T ,
Revised: , T ,

Original: (1) Closeness.
Revised: (1) Closeness.

Original: For any g ∈ G ( t +1) , P t ( g + R t ) ∈ H ( t ) ; for any h ∈ H ( t ) , (cid:104) π t , h
Revised: For any g ∈ G ( t +1) , P t ( g + R t ) ∈ H ( t ) ; For any h ∈ H ( t ) , (cid:104) π t , h

Original: (cid:105) ∈ G ( t ) .(2)
Revised: (cid:105) ∈ G ( t ) .

Original:  For any h ∈ ( T − t + 2) H ( t +1)( W , A , S ) , we have (cid:107)P t (cid:16) R t + (cid:104) 
Revised: (2) For any h ∈ ( T − t ) H ( t +1) , we have (cid:107)P t (cid:16) R t + (cid:104) π

Original: π t +1 ,h (cid:105) T − t +(cid:17) (cid:107) 2 H ≤ (cid:107) h T − t +2 (cid:107) 2 H .(3)
Revised: [Table] +1 ,h (cid:105) T − t +

Original: There exists a constant C G > 0 such that (cid:107) (cid:104)
Revised: [Table]

Original: π t , h (cid:105) (cid:107) 2 G ≤ C G (cid:107)
Revised: (cid:107) 2 H [Table] t ) ≤ [Table]

Original:  h (cid:107) 2 H  , for h ∈ H ( t ) ( W × A × S ) .(4)
Revised: H [Table] (3) There exists a constant C G > 0 such that (cid:107) (cid:104) π t , h (cid:105) (cid:107) 2 G ( t ) ≤ C G (cid:107) h (cid:107) 2 H ( t ) , for h ∈ H ( t ) .

Original:  q πt ∈ ( T − t + 1) H ( t ) ( W ,A , S  ) and (cid:107) q πT (cid:107) 2 H  ≤ M H , where M H > 0 is a constant.(5) Testing function class F ( t ) is sufﬁciently rich such that there exists L > 0 , (cid:107) f ∗ − proj t
Revised: (4) q πt ∈ ( T − t + 1) H ( t ) ( W , S , A ) and (cid:107) q πT (cid:107) 2 H ( T ) ≤ M H , where M H > 0 is a constant. (5) Testing function class F ( t ) is sufﬁciently rich such that there exists L > 0 , (cid:107) f ∗ − proj t

Original: h t (cid:107) 2 ≤ η ( t ) n ,where f ∗ ∈ arg min f ∈F ( t ) L 2 (cid:107) ht (cid:107) 2 H (cid:107) f − proj t h t (cid:107) 2 , for all h t ∈ H (W t , A t , S t ) .(6)
Revised: h t (cid:107) 2 ≤ η ( t ) n , where f ∗ ∈ arg min f ∈F ( t ) L 2 (cid:107) ht (cid:107) 2 H (cid:107) f − proj t h t (cid:107) 2 , for all h t ∈ H ( t ) .

Original:  Behavior policies. There exists a constant b π such that π bt ( a | s ) (cid:44) E
Revised: (6) Behavior policies: there exists a constant b π such that π bt ( a | s ) (cid:44) E

Original: [˜ π bt ( a | U t , S t ) | S t = s ] ≥b π > 0 for all ( s, a ) ∈ S × A .
Revised: [˜ π bt ( a | U t , S t ) | S t = s ] ≥ b π > 0 for all ( s, a ) ∈ S × A .

Original: Assumption 1 (1) and (5) are similar to Bellman completeness widely used in RL without unobservedstates [e.g., Antos et al., 2008].
Revised: Assumption 1 (1) is similar to Bellman completeness, which has been widely used in RL without unobserved states [e.g., Antos et al., 2008].

Revised: Note that both G ( t ) and H ( t ) can be chosen as inﬁnitedimensional spaces, e.g., RKHSs.

Revised: Hence this assumption is relatively mild.

Original: Assumption 1 (2) requires the operator P t to be bounded, which canbe ensured under some continuity conditions on transition kernels
Revised: Assumption 1 (2) requires the operator P t to be bounded, which can be ensured under some continuity conditions on transition kernels

Original: [Kress, 1989].
Revised: [Kress, 1989].

Original: Assumption 1 (3) isa technical condition for controlling the complexity of G  .
Revised: Assumption 1 (3) is a technical condition for controlling the complexity of G by H .

Original: Assumption 1 (4) essentially assumes thatwe can model q π (and v π ) correctly at each t -step, which is  mild as H ( t )( W , A , S ) for t ≥ 1 are allinﬁnite-dimensional spaces.
Revised: Assumption 1 (4) essentially assumes that we can model q π (and v π ) correctly at each t -step, which is again mild as H ( t ) for t ≥ 1 can all be chosen as inﬁnite-dimensional spaces.

Revised: This assumption is also called realizability of value functions, which is commonly seen in the literature of RL [e.g., Antos et al., 2008].

Revised: Assumption 1 (5) is imposed to ensure that the space of testing functions F is large enough so that we are able to capture the conditional expectation operator in each min-max estimation (7).

Original: Assumption 1 (6) basically requires a full coverage in our batch data,which is widely used in OPE [Precup, 2000, Antos et al., 2008].
Revised: Assumption 1 (6) basically requires a full coverage of our batch data generating process induced by the behavior policy, which is widely used in OPE [Precup, 2000, Antos et al., 2008].

Original: We ﬁrst provide a key decompositionof the L 2 error for V -bridge estimation.
Revised: Next, we provide a key decomposition of the L 2 error for V -bridge estimation.

Original: Theorem 6.1 (Error decomposition) .
Revised: Theorem 6.1 (Error decomposition) .

Original: Under Assumption 1 (1) and (6), we can decompose the L 2 ( W × S ) error of the estimated V -bridge by
Revised: Under Assumption 1 (1) and (6), we can decompose the Lerror of the estimated V -bridge by

Original: [Equation]
Revised: [Equation]

Original: where the measures of one-step transition ill-posedness C ( t ) 1 , 0 (cid:44) 1 and C ( t )t
Revised: where the measures of one-step transition ill-posedness C ( t ) 1 , 0 (cid:44) 1 and C ( t )

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]

Original: − 1 , 2 ≤ t

Original: [Equation]
Revised: [Equation]

Original: ≤ t , 2 ≤ t  ≤ Tare deﬁned after Corollary 6.2.
Revised: , 2 ≤ t (cid:48) ≤ t ≤ T are deﬁned after Corollary 6.2.

Original: Theorem 6.1 shows that there are four key components forthe upper bound of the L 2 error of ˆ v π 1 .
Revised: Theorem 6.1 shows that there are four key components for upper bounding the L 2 error of ˆ v π 1 .

Original: The ﬁrst component is the probability ratio (cid:107) π t /π bt (cid:107) ∞ , which is used to measure the distributionalmismatch between the target and behavior policies, which is the fundamental challenge and irreduciblein ofﬂine RL [Levine et al., 2020].
Revised: The ﬁrst component is the probability ratio (cid:107) π t /π bt (cid:107) ∞ , which is used to measure the distributional mismatch between the target and behavior policies.

Original: The second component is (cid:107) proj t
Revised: The second component is (cid:107) proj t

Original: ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 , theone-step projected error of (cid:98) P t to P t  where ˆ v πt +1 is the estimate for v πt +1 depending on the observeddata.
Revised: ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 , the one-step projected error of (cid:98) P t to P t , where ˆ v πt +1 is the estimate for v πt +1 depending on the observed data after t -step.

Original: This is different from the  standard NPIV estimation with a directly measured outcome so theresult, e.g., from Dikkala et al.
Revised: We remark that this is different from the analysis in the standard NPIV estimation with a directly measured outcome. Hence the results, e.g., from Dikkala et al.

Original: [2020], cannot be directly applied to bound this component.
Revised: [2020], cannot be directly applied to bound this component.

Original: The lasttwo components are related to the (local) measure of ill-posedness.
Revised: The last two components are related to the (local) measure of ill-posedness.

Original: The third component τ t is themeasure of ill-posedness for characterizing the difﬁculty of estimating q πt by (4) using H ( t ) at thet -th step. 
Revised: The third component τ t is the measure of ill-posedness for characterizing the difﬁculty of estimating q πt by (4) using H ( t ) at the t -th step. {

Original: Here { τ t } Tt =1 are similar to those used in the standard NPIV estimation such as Chen and Reiss [2011]. Obviously the effect of { τ t} Tt =1 on the upper bound is cumulative.
Revised: τ t } Tt =1 are similar to those used in the standard NPIV estimation such as Chen and Reiss [2011], and the effect of each τ t on the upper bound is cumulative.

Original: The last component{ Π tt (cid:48) =
Revised: The last component { Π tt (cid:48) =

Original: 1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 quantiﬁes the propagation effect of estimation errors in previous steps on thelast step of estimating v π 1 , which is multiplicative in terms of C ( t ) t (cid:48) ,t (cid:48) − 1 .
Revised: 1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 quantify the propagation effect of estimation errors in previous steps on the last step of estimating v π 1 , which is multiplicative in terms of C ( t ) t (cid:48) ,t (cid:48) − 1 .

Original: We call C ( t ) t (cid:48) ,t (cid:48) − 1 the measure of one-step transition ill-posedness from t (cid:48) to t (cid:48) − 1 related to t -step NPIV estimations.
Revised: We call C ( t ) t (cid:48) ,t (cid:48) − 1 the measure of one-step transition ill-posedness from t (cid:48) to t (cid:48) − 1 related to t -step NPIV estimations.

Original: Next weprovide detailed bounds for the second and last components.
Revised: Next we provide detailed bounds for the second and last components.

Original: The discussion of the third componentcan be found in Appendix.
Revised: The discussion of the third component can be found in Appendix C.4.

Original: Component 2: one-step projected error .
Revised: Component 2: one-step projected error .

Original: In the following, we show that (cid:107) proj t
Revised: In the following, we show that (cid:107) proj t

Original: ( (cid:98) P t −P t )(ˆ v πt +1 + R t ) (cid:107) 2is bounded by the critical radii of some spaces deﬁned as balls H ( t ) 3 M , G ( t +1) 
Revised: ( (cid:98) P t −P t )(ˆ v πt +1 + R t ) (cid:107)is bounded by the critical radii of some spaces deﬁned as balls H ( t ) B , G ( t +1) C G ( T − t +1)

Original: D in hypothesis spaces H ( t ) , G ( t +1)  and a ball F ( t ) B in testing space F ( t ) , for some ﬁxed constants M, B > 0 such thatfunctions in H ( t ) 3 M and F ( t ) B have uniformly bounded ranges in [ − 1 , 1] for all 1 ≤ t ≤ T .
Revised: M H in hypothesis spaces H ( t ) , G ( t +1) respectively and a ball F ( t ) 3 M in testing space F ( t ) , for some ﬁxed constants M, B > 0 such that functions in H ( t ) B and F ( t ) 3 M have uniformly bounded ranges in [ − 1 , 1] for all 1 ≤ t ≤ T .

Original: LetΩ
Revised: Let

Original: [Equation]
Revised: [Equation]

Original:  (s t ,
Revised: Ξ ( t ) =

Original: [Equation]

Original:  t , z t , [Equation] t , s [Equation]
Revised: { ( s t , w t , z t , a t ) (cid:55)→ r

Original: +1 , w [Equation] +1 ) (cid:55)→ r ( h
Revised: [ h − h

Original: [Equation]
Revised: [Equation]

Original: g ( w t ,a t , s t) − g ( w t +1 , s t +1 ))
Revised: g ]( w t , s t , a t ) f

Original: f ( z t ,a t , s t) : g ∈ G ( t +1) D , f ∈ F ( t ) 3 M , r ∈ [0 , 1] } , and Ξ ( t ) = { ( s t , w t , z t , a [Equation]
Revised: L 2 B ( z t , s t , a t ) :

Original: (cid:55)→ [Equation] [Equation] ]( w
Revised: h

Original: [Equation]
Revised: [Equation]

Original: , a

Original: [Equation]
Revised: [Equation]

Original: , s t
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ( z t , a t , s t ) : [Equation]
Revised: G [Equation] [Equation]

Original: , r ∈ [0 , 1] } , where h ∗ g ∈ H ( t ) is the solution to E [ h ( W t , A t , S t ) − g ( W t +1 , S t +1 ) | Z t , A t , S t ] = 0 , and f L 2 B =arg min f ∈F ( t )
Revised: , r ∈ [0 , 1] } , where h ∗ g ∈ H ( t ) is the solution to E [ h ( W t , S t , A t ) − g ( W t +1 , S t +1 ) | Z t , S t , A t ] = 0 , and f L 2 B = arg min f ∈F ( t )

Original: L 2 B (cid:107) f − proj t
Revised: L 2 B (cid:107) f − proj t

Original: ( h − h ∗ g ) (cid:107) 2 for a given L > 0 .
Revised: ( h − h ∗ g ) (cid:107) 2 for a given L > 0 .

Original: An upper bound for (cid:107) proj t ( (cid:98) P t −P t )(ˆ v πt +

Revised: An upper bound for (cid:107) proj t ( (cid:98) P t −P t )(ˆ v πt +

Original: +
Revised: +

Original: R t ) (cid:107) 2 is given in Theorem 6.2.
Revised: R t ) (cid:107) 2 is given in Theorem 6.2.

Original: Theorem 6.2.
Revised: Theorem 6.2.

Original: Suppose that Assumption 1 holds.
Revised: Suppose that Assumption 1 holds.

Original: Let δ ( t ) n = ¯ δ ( t ) n + c 0 (cid:113) log( c 1 T/ζ ) n for universalconstants c 0 , c 1 > 0 where ¯ δ ( t ) n is the upper bound of the critical radii of F ( t ) 3 M , Ω ( t ) and Ξ ( t ) .
Revised: Let δ ( t ) n = ¯ δ ( t ) n + c 0 (cid:113) log( c 1 T/ζ ) n for some universal constants c 0 , c 1 > 0 where ¯ δ ( t ) n is the upper bound of the critical radii of F ( t ) 3 M , Ω ( t ) and Ξ ( t ) .

Original: Assume that the approximation error in Assumption 1 (5) can be bounded by η ( t ) n ≤ δ ( t ) n .Then [Equation] where D = G ( t +1) C G ( T − t +1)
Revised: Assume that the approximation error in Assumption 1 (5) can be bounded by η ( t ) n ≤ δ ( t ) n .

Original: M H .

Original: Furthermore, letting tuning parameters satisfy Mλ (cid:16) ( δ ( t ) n ) 2 and µ ≥
Revised: Furthermore, letting tuning parameters satisfy Mλ (cid:16) ( δ ( t ) n ) 2 and µ ≥

Original: O
Revised: O

Original: ( L 2 + M/B ) , with probability at least 1 − ζ , we have
Revised: ( L 2 + M/B ) , with probability at least 1 − ζ , we have

Original: [Equation]
Revised: [Equation]

Original: for all 1 ≤ t ≤ T uniformly .
Revised: for all 1 ≤ t ≤ T [Equation]

Original: With common choices of H ( t ) , G ( t +1) , and F ( t ) , we can obtain the ﬁnite-sample error bounds of theone-step projected error for each t .
Revised: Depending on the choices of H ( t ) , G ( t +1) , and F ( t ) , we can obtain different ﬁnite-sample error bounds of the one-step projected error for each t .

Original: Below we provide two examples.
Revised: Below we provide two examples.

Original: Corollary 6.1.
Revised: Corollary 6.1.

Original: Let F ( t ) , H ( t ) and G ( t +1) be VC-subgraph classes with VC dimensions V ( F ( t ) ) ,
Revised: Let F ( t ) , H ( t ) and G ( t +1) be VC-subgraph classes with VC dimensions V ( F ( t ) ) ,

Original: V ( H ( t ) ) and V ( G ( t +1) ) respectively.
Revised: V ( H ( t ) ) and V ( G ( t +1) ) respectively.

Original: Then with probability at least 1 − ζ , for all 1 ≤ t ≤ T uniformly,(cid:107) projt ( [Equation] P [Equation]
Revised: Then with probability at least 1 − ζ , for all 1 ≤ t ≤ T , (cid:107) proj

Original: [Equation]

Original: πt

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 2 (cid:46) ( T − t + 1) 2 .(cid:114)

Original: log( c 1 T/ζ ) max { V ( F ( t ) ) , V ( H ( t ) ) , V ( G ( t +1) ) } n .

Original: The deﬁnition of the VC-subgraph class can be found in, e.g., Wainwright [2019].
Revised: The deﬁnition of the VC-subgraph class can be found in, e.g., Wainwright [2019].

Original: This is a broad class.
Revised: This is a broad class.

Original: For example, if one lets each of F ( t ) , H ( t ) and G ( t +1) be a linear space F
Revised: For example, if one lets each of F ( t ) , H ( t ) and G ( t +1) be a linear space F

Original: = { θ (cid:62) φ ( · ) : θ ∈ R d }with basis functions φ ( · ) , then V ( F ) =
Revised: = { θ (cid:62) φ ( · ) : θ ∈ R d } with basis functions φ ( · ) , then V ( F ) =

Original: d + [-
-]
Revised: d + [-
-]1 .

Original: so the upper bound for the one-step projected errorbecomes O (( T − t + 1) 2 . 5 d/ √ n ) .
Revised: Then the upper bound for the one-step projected error becomes O (( T − t + 1) 2 . 5 d/ √ n ) .

Original: Corollary 6.2.
Revised: Corollary 6.2.

Original: Let H ( t ) , G ( t +1) and F ( t ) be reproducing kernel Hilbert spaces (RKHSs) equippedwith kernels K H ( t ) , K G ( t +1) and K F ( t ) respectively.
Revised: Let H ( t ) , G ( t +1) and F ( t ) be reproducing kernel Hilbert spaces (RKHSs) equipped with kernels K H ( t ) , K G ( t +1) and K F ( t ) respectively.

Original: For a given positive deﬁnite kernel K , we denoteits decreasing eigenvalue sequence by { λ ↓ j ( K ) } ∞ j =1 .
Revised: For a given positive deﬁnite kernel K , we denote its nonincreasing eigenvalue sequence by { λ ↓ j ( K ) } ∞ j =1 .

Original: We consider two scenarios for { λ ↓ j ( K ) } ∞ j =1 .(1)
Revised: We consider two scenarios for { λ ↓ j ( K ) } ∞ j =1 .

Original:  Polynomial eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ aj − 2 α H , λ ↓ j ( K G ( t +1) )
Revised: (1) Polynomial eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ aj − 2 α H , λ ↓ j ( K G ( t +1) )

Original: ≤ aj − 2 α G and λ ↓ j ( K F ( t ) ) ≤aj − 2 α F for constants α H , α G , α F > 1 / 2 and a > 0 , then under the assumptions in Theorem 6.2,with probability at least 1 − ζ , for all 1 ≤ t ≤ T uniformly, we have
Revised: ≤ aj − 2 α G and λ ↓ j ( K F ( t ) ) ≤ aj − 2 α F for constants α H , α G , α F > 1 / 2 and a > 0 , then under the assumptions in Theorem 6.2, with probability at least 1 − ζ , for all 1 ≤ t ≤ T , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] ) Exponential eigen -decay : 
Revised: (2) Exponential eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ a 1 e − a 2 j β H , λ ↓ j ( K G ( t +1) ) ≤ a 1 e − a 2 j β G and λ ↓ j ( K F

Original: If λ ↓ j ( K H ( t)
Revised: t

Original:  ≤ a1 e − a 2 j β H
Revised: ) ≤ a e − a 2 j β F ,

Original: , λ
Revised: for

Original: ↓ j ( K G ( t +1) ) ≤ a 1 e
Revised: constants a 1 , a 2 , β H ,

Original: − a
Revised: β

Original: 2 j β G andλ ↓ j ( K F ( t ) ) ≤ a 1 e − a 2 j β F , for constants a 1 , a 2 , β H , β G , β F > 0 , then under the assumptions in
Revised: G , β F > 0 , then under the assumptions in

Original: Theorem 6.2,then with probability at least 1 − ζ , for all 1 ≤ t ≤ T uniformly, we have[Equation] n [Equation]
Revised: Theorem 6.2, with probability at least 1 − ζ , for all 1 ≤ t ≤ T , we have

Original: n

Original: [Equation]
Revised: [Equation]

Original: The kernels of the two types of eigen-decay  above are very common.
Revised: Kernels of the two types of eigen-decay considered above are very common.

Original: For example, the kernel of theα -order Soblev space with α > 1 / 2 , has a polynomial eigen-decay while the Gaussian kernel hasan exponential eigen-decay, with β = 2 for Lebesgue measure on real line and β = 1 on a compactdomain [Wei et al., 2017].
Revised: For example, the kernel of the α -order Soblev space with α > 1 / 2 , has a polynomial eigen-decay while the Gaussian kernel has an exponential eigen-decay, with β = 2 for Lebesgue measure on real line and β = 1 on a compact domain [Wei et al., 2017].

Original: Components 4: measure of one-step transition ill-posedness .
Revised: Components 4: measure of one-step transition ill-posedness .

Original: We ﬁrst provide more insights on{ Π tt (cid:48)
Revised: We ﬁrst provide more insights on { Π tt (cid:48)

Original: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 before providing an upper bound.
Revised: =1 C ( t ) t (cid:48) [Equation] t (cid:48) − 1 } Tt =1 before providing an upper bound.

Original: We formally deﬁne the local measure ofone-step transition ill-posedness C ( t ) t (cid:48)
Revised: We formally deﬁne the local measure of one-step transition ill-posedness C ( t ) t (cid:48)

Original: +1 ,t (cid:48) recursively based on C ( t ) t,t − 1 to C ( t ) 2 , 1 as[Equation] [Table]
Revised: +1 ,t (cid:48) recursively based on C ( t ) t,t − 1 to C ( t ) 2 , 1 as

Original: [Equation] [Table] [Equation] [Table]
Revised: C

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: t ) t

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: ,t

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: (cid:44)

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: subject to

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: 2 ∞ t − 1 (cid:89) s

Original: = t (cid:48) + C ( t ) s +1 ,s , with C ( t ) t +1 ,t (cid:44) 1 for each t = 1 , . . 
Revised: with C ( t ) t +1 ,t (cid:44) 1 for each t = 1 , . . .

Original: , T .
Revised: , T .

Original: For C ( t ) t,t − 1 , we can upper bound (cid:107) Eπ t [ˆ v π ( W t , S t ) | Z t , S t ] (cid:107) 2by the projected error (cid:107) proj t ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 multiplied by the ill-posedness τ t .
Revised: For C ( t ) t,t − 1 , we can upper bound (cid:107) E [ˆ v π ( W t , S t ) | Z t , S t ] (cid:107) 2 by the projected error (cid:107) proj t ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 multiplied by the ill-posedness τ t .

Original: By Theorem6.2, the projected error can be well controlled by δ ( t ) n with high probability, so C ( t ) t,t − 1 can be deﬁnedlocally.
Revised: By Theorem 6.2, the projected error can be well controlled by δ ( t ) n with high probability, so C ( t ) t,t − 1 can be deﬁned locally.

Original: Therefore, we can provide an upper bound for the denominator sequentially and deﬁne all
Revised: Therefore, we can provide an upper bound for the denominator sequentially and deﬁne all

Original: C ( t ) t (cid:48)
Revised: C ( t ) t (cid:48)

Original: +1 ,t (cid:48) locally, which indicates that all C ( t ) t (cid:48) +1 ,t (cid:48) could be small.
Revised: +1 ,t (cid:48) locally, which indicates that all C ( t ) t (cid:48) +1 ,t (cid:48) could be small.

Original: For example, if we use the observed history as the action-inducing proxy,e.g., Z t =( Z t − 1 ; S t − 1 , W t − 1 , A t − 1 , R t − 1 ) with some pre-observed Z 1 , then σ ( Z 1 × S 1 ) ⊂ σ ( Z 2 × S 2 ) ⊂· · · ⊂ σ ( Z T × S T ) is a ﬁltration.
Revised: For example, if we use the observed history as the action-inducing proxy, then σ ( Z 1 × S 1 ) ⊂ σ ( Z 2 × S 2 ) ⊂ · · · ⊂ σ ( Z T × S T ) is a ﬁltration.

Original: Therefore, (cid:81) tt (cid:48)
Revised: In this case, (cid:81) tt (cid:48)

Original: =2 C ( t ) t (cid:48) ,t (cid:48) − 1 = 1 for t ≥ 2 if the target policy isstationary.
Revised: =2 C ( t ) t (cid:48) ,t (cid:48) − 1 are expected to be small for t ≥ 2 if the target policy is stationary.

Original: While this can enlarge the critical radii δ ( t ) n since thetesting space F ( t ) ( Z × A × S )will have dimension of O ( t ) , this only affects one-step errors.Practitioners can also learn a lowdimensional representation of Z t to reduce the one-step statistical errors.
Revised: While this can enlarge the critical radii δ ( t ) n due to the dimension of the action-inducing proxy, this only affects one-step errors.

Original: See detailed discussion in
Revised: See detailed discussion in

Original: Appendix B. Motivated by this, it is reasonable to impose Assumption 2  on C ( t ) t (cid:48)
Revised: Appendix B. Motivated by this, it is reasonable to impose Assumption 2 below on C ( t ) t (cid:48)

Original: +1 ,t (cid:48) .
Revised: +1 ,t (cid:48) .

Original: Assumption 2.
Revised: Assumption 2.

Original: For every t ≥ 2 and 2 ≤ m ≤ t , C ( t ) m,m − 1 ≤ 1 + a t m αt with  constants a t  , α t  > 0 .
Revised: For every t ≥ 2 and 2 ≤ m ≤ t , C ( t ) m,m − 1 ≤ 1 + a t m αt with time-dependent constants a t > 0 , α t ≥ α > 1 .

Original: Then we can provide an upper bound for the propagation effect factor (cid:81) tt (cid:48)

Original: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 .

Original: Corollary 6.3.
Revised: Corollary 6.3.

Original: If Assumption 2 holds, then (cid:81) tt (cid:48)
Revised: If Assumption 2 holds, then (cid:81) tt (cid:48)

Original: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 ≤ exp { a t ζ ( α t ) } , 
Revised: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 ≤ exp { a t ζ ( α t ) } , where ζ ( α t ) =

Original: where ζ ( α t ) = 338 (cid:80) ∞ n =1 (1 /n )
Revised: (cid:80) ∞ n =1 (1 /n )

Original: α  is uniformly bounded for t ≥ 1 .
Revised: α t is uniformly bounded for t ≥ 1 .

Original: Main result: error bounds for V -bridge estimation and OPE .
Revised: Main result: error bounds for V -bridge estimation and OPE .

Original: Deﬁne trans-ill =max 1 ≤ t ≤ T exp { a t ζ ( α t ) } and let ill max =  τπ 1 max 1 ≤ t ≤ T τ t (cid:107) π t /π bt (cid:107)2 ∞ .
Revised: Deﬁne trans-ill = max 1 ≤ t ≤ T exp { a t ζ ( α t ) } and let ill max = ¯ τ 1 max 1 ≤ t ≤ T τ t (cid:107) π t /π bt (cid:107) ∞ .

Original: Summarizing all aforementioned results, we have the following main theorem based on the polynomial eigen-decay case in
Revised: Summarizing all aforementioned results, we have the following main theorem based on the polynomial eigen-decay case in

Original: Corollary 6.2.
Revised: Corollary 6.2.

Original: Other cases can be found in Appendix B.
Revised: Other cases can be found in Appendix B.

Original: Theorem 6.3 (Finite-sample error bounds for V -bridges and policy value) .
Revised: Theorem 6.3 (Finite-sample error bounds for V -bridges and policy value) .

Original: Under Assumptions 1and 2, and assumptions in Theorem 6.2 and Corollary 6.2 (1), with probability at least 1 − ζ , we have(cid:107) v π 1 − ˆ v π 1 (cid:107) 2 (cid:46) ill max × trans-ill × T 7 / 2 (cid:112) log( c 1 T/ζ ) n − 1 2+max { 1 /α H , 1 /α G , 1 /α F} , and |V ( π ) − ˆ V ( π ) | (cid:46) ill max × trans-ill × T 7 / 2 (cid:112) log( c
Revised: Under Assumptionsand 2, and assumptions in Theorem 6.2 and Corollary 6.2 (1), with probability at least 1 − ζ , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 6.3 provides the ﬁrst ﬁnite-sample error bound for OPE under confounded and episodic
Revised: Theorem 6.3 provides the ﬁrst ﬁnite-sample error bound for OPE under confounded and episodic

Original: POMDPs in terms of the sample size, length of horizon and two ( local ) measures of ill-posedness.
Revised: POMDPs in terms of the sample size, length of horizon and two ( local ) measures of ill-posedness.

Original: Aswe can see, the derived error bound for V -bridges function  achieves the optimal L 2 -convergence ratein the classical non-parametric regression [Stone, 1982].
Revised: Without considering the measures of ill-posedness, the derived error bound for V -bridge function nearly achieves the optimal L 2 -convergence rate in the classical non-parametric regression [Stone, 1982].

Original: Moreover, our OPE error bound dependson a polynomial order of T , i.e., T 7 / 2 , which is larger than the standard O ( T 3 ) in the OPE withoutunobserved variables.
Revised: Moreover, our OPE error bound depends on a polynomial order of T , i.e., T 7 / 2 , which is larger than the standard O ( T 3 ) in the OPE without unobserved variables.

Original: We also remark that when the function class consider in (7) grows with thesample size n , ill max will also increase and therefore the convergence rates in Theorem 6.3 couldbe much slower. 
Revised: However, when the function class consider in (7) grows with the sample size n , ill max will also increase and therefore the convergence rates in Theorem 6.3 could be much slower. We further study a case when we can control the local measures of ill-posedness τ t , by assuming that λ min (Γ ( t ) m ) ≥ ν m and other regularity conditions in Lemma C.1 uniformly for all 1 ≤ t ≤ T , and ( S t , A t ) ∈ S × A , where Γ ( t ) m (cid:44) E (cid:110) E [ e ( t )

Revised: I ( W t , S t , A t ) | Z t , S t , A t ] E [ e ( t ) I ( W t , S t , A t ) | Z t , S t , A t ] (cid:62) (cid:111) with e ( t ) I =

Revised: ( e ( t ) 1 , . . .

Original: Therefore it will be interesting to study the minimax lower bounds for our problem.
Revised: , e ( t ) m ) are the ﬁrst m eigenfunctions of kernel K H ( t ) . Similar argument can be applied to ¯ τ 1 , which is omitted here for simplicity. Let η ( n, T, ζ, α H , α F , α G , b ) (cid:44) [Equation]

Revised: Corollary 6.4.

Revised: If assumptions in Theorem 6.3 holds and ν m ≥ m − 2 b , then

Revised: [Equation]

Revised: × trans-ill × η ( n, T, ζ, α H , α F , α G , b ) .

Revised: Corollary C.5.2 considers the mildly ill-posed case, i.e., ν m ≥ m − 2 b , and shows that the local measure of ill-posedness can deteriorate the sample complexity by factor α H / ( α H + b ) .

Revised: If b is large or severely ill-posed case is considered (i.e., ν m decays exponentially fast, see Appendix C.5.2), then the convergence rate of V -bridge estimation could be much slower and the typical requirement on the nuisance parameter for achieving asymptotic normality for the policy value will fail.

Revised: On the other hand, it can be seen that when b = 0 , the ﬁnite sample error bounds match the results in Theorem 6.3.

Revised:  Simulation

Revised: In this section, we perform a simulation study to evaluate the performance of our proposed OPE estimation and to verify the ﬁnite-sample error bound of our OPE estimator in Theorem 6.3.

Revised: Let S

Revised: = R 2 , U = R , W = R , Z = R , and A = { 1 , − 1 } .

Revised: At time t , the hidden state U t , two proximal variables Z t , W t satisfy the following multivariate normal distribution given ( S t , A t ) :

Revised: [Equation]

Revised: 9) where parameters are given in the Appendix.

Revised: The behavior policy is given by ˜ π bt ( A t | U t , S t )

Revised: =

Revised: expit (cid:8) − A t (cid:0) t 0 + t u U t + t (cid:62) s S t (cid:1)(cid:9) , where t 0 = 0 , t u = 1 , and t (cid:62)

Revised: s =

Revised: [ − 0 .

Revised: 5 , − 0 .

Revised: Then by Assumption 1 (6), π bt ( A t | S t ) =

Revised: expit {− A t (cid:0) t 0 + t u κ 0 + ( t s + t

Revised: u κ s ) (cid:62) S t (cid:1) } .

Revised: The initial S 1 is uniformly sampled from R 2 .

Original: {+At time t , given ( S t , U t , A t ) , we generate S t +
+}
Revised: {+At time t , given ( S t , U t , A t ) , we generate S t +
+}

Revised: =

Revised: S t + A t U t 1 2 + e S t

Revised: +1 , where 1 2 =

Revised: [1 , 1] (cid:62) and the random error e S t +1 ∼ N ([0 , 0] (cid:62) , I 2 ) with I 2 denoting the 2 -by- 2 identity matrix.

Revised: The reward is given by R t

Original: {+= expit (cid:8) 
+}
Revised: {+= expit (cid:8) 
+}

Revised: A t ( U t + [1 , − 2] S t ) (cid:9) + e t , where e t ∼ Uniform [ − 0 .

Revised: One can verify that our simulation setting satisﬁes the conditions in Section A.1 so that our method can be applied.

Original: {+2222222 6 Horizon T 
+}
Revised: {+2222222 6 Horizon T 
+}

Revised: M e a n a b s o l u t e e rr o r

Original: {+Training set size n= 22222 12 Training set size n 
+}
Revised: {+Training set size n= 22222 12 Training set size n 
+}

Revised: M e a n a b s o l u t e e rr o r

Revised: [Figure]

Revised: T

Revised: [Figure]

Revised: We choose F ( t ) and H ( t ) as RKHSs endowed with Gaussian kernels, with bandwidths selected according to the median heuristic trick by Fukumizu et al.

Revised: [2009] for each 1 ≤ t ≤ T .

Revised: The pool of scaling factors SCALE contains 30 positive numbers spaced evenly on a log scale between 0.001 to 0.05.

Revised: The number of cross-validation partition K = 5 .

Revised: The true target policy value of π is estimated by the mean cumulative rewards of 50 , 000 Monte Carlo trajectories with policy π .

Revised: We compare our

Revised: OPE estimator (cid:98) V ( π ) with the target policy value by computing mean absolute error (MAE) for each setting of ( n, T ) , as reported in Figure 2.

Revised: Figure 2 validate the derived ﬁnite-sample error bound of our OPE estimator in Theorem 6.3.

Revised: Speciﬁcally, Figure 2 (a) shows that the OPE estimation error is polynomial in T , but with an order slightly smaller than O ( T 7 / 2 ) as stated in Theorem 6.3.

Revised: Figure(b) shows that the convergence rate in terms of the sample size n for our OPE estimator is slower than O ( n − 1 / 2 ) , which also justiﬁes our theoretical results.

Revised:  Discussion

Revised: In this paper, we propose a non-parametric identiﬁcation and estimation method for OPE in episodic confounded POMDPs with continuous states, relying on time-dependent proxy variables.

Revised: We develop a ﬁtted- Q -evaluation-type algorithm for estimating the V -bridge functions sequentially and for OPE based on the estimated V -bridges.

Revised: The ﬁrst ﬁnite-sample error bound for estimating the policy value under confounded POMDPs is established, which achieves a polynomial order with respect to the sample size and the length of horizon.

Revised: Our OPE results can serve as a foundation for developing new policy optimization algorithms in the confounded POMDP, which We will leave for future work.

Revised: Acknowledgement

Revised: Zhang’s research is partially supported by GW University Facilitating Fund.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] The paper’s contributions and scope are summarized inthe abstract and then emphasized in Sections 1 and 2.(b)
Revised: [Yes] The paper’s contributions and scope are summarized in the abstract and then emphasized in Sections 1 and 2.

Original:  Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We did not provide results ofminimax lower bound for our estimation problem.(c)
Revised: [Yes] We did not provide results of minimax lower bound for our estimation problem.

Original:  Did you discuss any potential negative societal impacts of your work?[N/A] The
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: proposed work is to characterize the ﬁnite-sample error bound for OPE in confounded

Original: and episodic POMDP under non-parametric models.

Original: The emphasis is on theories,

Original: without limitations on speciﬁc applications.

Original: As a result, it’s not applicable to talk about

Original: negative societal impacts.

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Main assumptions are stated in Sections 4 and 6.
Revised: [Yes] Main assumptions are stated in Sections 4 and 6.

Original: Some regularity conditions are provided in Appendix A. (b) Did you include complete proofs of all theoretical results?
Revised: Some regularity conditions are provided in Appendix A. (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs for theoret- ical results are provided in Appendices C and D. 
Revised: [Yes] All proofs for theoretical results are provided in Appendices C and D. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [Yes] We attached the code and instructions in the supplemental materials.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?[N/A](d)
Revised: [Yes] Provided in Appendix F (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original:  Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]4.
Revised: [Yes] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] ) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] [N/A] ) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (c) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] ) Did you discuss whether the data you are using/curating contains personally identiﬁable[Table] information or offensive content?
Revised: [N/A] (d) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... [Table] Did you include the full text of instructions given to participants and screenshots, if [Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: applicable N participant risks,  to Institutional Review 
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: applicable [Table] you include the estimated hourly wage paid to participants and the total amount spent on participant compensation
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]