Original: Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Models
Revised: Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Models

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddressemail
Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: We study the problem of off-policy evaluation (OPE) for episodic Partially Observable Markov Decision Processes (POMDPs) with continuous states.
Revised: We study the problem of off-policy evaluation (OPE) for episodic Partially Observable Markov Decision Processes (POMDPs) with continuous states.

Original: Motivatedby the recently proposed proximal causal inference framework, we develop anon-parametric identiﬁcation result for estimating the policy value via a sequenceof so-called V-bridge functions with the help of time-dependent proxy variables.
Revised: Motivatedby the recently proposed proximal causal inference framework, we develop anon-parametric identiﬁcation result for estimating the policy value via a sequenceof so-called V-bridge functions with the help of time-dependent proxy variables.

Original: We then develop a ﬁtted-Q-evaluation-type algorithm to estimate V-bridge functions recursively, where a non-parametric instrumental variable (NPIV) problem issolved at each step.
Revised: We then develop a ﬁtted-Q-evaluation-type algorithm to estimate V-bridge functions recursively, where a non-parametric instrumental variable (NPIV) problem issolved at each step.

Original: By analyzing this challenging sequential NPIV problem, weestablish the ﬁnite-sample error bounds for estimating the V-bridge functions andaccordingly that for evaluating the policy value, in terms of the sample size, lengthof horizon and so-called (local) measure of ill-posedness at each step.
Revised: By analyzing this challenging sequential NPIV estimation, weestablish the ﬁnite-sample error bounds for estimating the V-bridge functions andaccordingly that for evaluating the policy value, in terms of the sample size, lengthof horizon and so-called (local) measure of ill-posedness at each step.

Original: To the bestof our knowledge, this is the ﬁrst ﬁnite-sample error bound for OPE in POMDPsunder non-parametric models.
Revised: To the bestof our knowledge, this is the ﬁrst ﬁnite-sample error bound for OPE in POMDPsunder non-parametric models.

Original:  Introduction
Revised:  Introduction

Original: In practical reinforcement learning (RL), a representation of the full state which makes the system
Revised: In practical reinforcement learning (RL), a representation of the full state which makes the system

Original: Markovian and therefore amenable to most existing RL algorithms is not known a priori.
Revised: Markovian and therefore amenable to most existing RL algorithms is not known a priori.

Original: Decisionmakers are often facing so-called partial observability of the state information, which signiﬁcantlyhinders the task of RL.
Revised: Decisionmakers are often facing so-called partial observability of the state information, which signiﬁcantlyhinders the task of RL.

Original: In general, agents have to maintain all historical information and establisha belief system on the hidden state for optimal decision making.
Revised: In general, agents have to maintain all historical information and establisha belief system on the hidden state for optimal decision making.

Original: A partially observable Markovdecision process (POMDP) is often used to model the data generating process.
Revised: A partially observable Markovdecision process (POMDP) is often used to model the data.

Original: See examples inrobotics [Rafferty et al., 2011], precision medicine [Tsoukalas et al., 2015], stochastic game [Hansenet al., 2004] and many others.
Revised: See examples in robotics [Raffertyet al., 2011], precision medicine [Tsoukalas et al., 2015], stochastic game [Hansen et al., 2004] andmany others.

Original: However, it is well known that learning optimal policies in POMDP iscomputationally intractable [Papadimitriou and Tsitsiklis, 1987].
Revised: However, it is well known that learning optimal policies in POMDP is computationallyintractable [Papadimitriou and Tsitsiklis, 1987].

Original: The issue of partial observabilitybecomes more serious in the batch setting, where agents are not able to actively collect additional dataand further explore the environment.
Revised: The issue of partial observability becomes moreserious in the batch setting, where agents are not able to actively collect additional data and furtherexplore the dynamic system.

Original: For example, standard off-policy evaluation (OPE) methods,which aim to learn a policy value from the batch data generated from some behavior policy, wouldfail to give a consistent estimate because of unobserved state variables.
Revised: For example, standard off-policy evaluation (OPE) methods, which aimto learn a policy value from the batch data generated from some behavior policy, would fail to give aconsistent estimate because of unobserved state variables.

Original: Due to this practical concern, there is a recent line of research studying the OPE under the frameworkof a confounded POMDP, where the behavior policy to generate the batch data is allowed to dependon some unobserved state variables [e.g., Tennenholtz et al., 2020, Nair and Jiang, 2021, Bennettand Kallus, 2021, Shi et al., 2021].
Revised: Due to this practical concern, there is a recent line of research studying the OPE under the frameworkof a confounded POMDP, where the behavior policy to generate the batch data is allowed to dependon some unobserved state variables [e.g., Tennenholtz et al., 2020, Nair and Jiang, 2021, Bennettand Kallus, 2021, Shi et al., 2021].

Original: Their identiﬁcation results on the policy value are inspired bythe negative controls or so-called proxy variables in the literature of causal inference [e.g., Miaoet al., 2018a, TchetgenTchetgen et al., 2020].
Revised: Their identiﬁcation results on the policy value are inspired bythe negative controls or so-called proxy variables in the literature of causal inference [e.g., Miaoet al., 2018a, Tchetgen et al., 2020].

Original: A building block of these results is the existence ofsome bridge functions, namely Q / V -bridge or weight-bridge functions, which are projections of the Q / V -functions or importance weights deﬁned over the original state space onto the observation space.
Revised: A building block of these results is the existence of somebridge functions, namely Q / V -bridge or weight-bridge functions, which are projections of the Q / Vfunctions or importance weights deﬁned over the original state space onto the observation space.

Original: The corresponding statistical estimation of these bridge functions mainly relies on solvinglinear integral equations [e.g., Kress, 1989].
Revised: The corresponding statistical estimation of these bridge functions mainly relies on solving some linearintegral equations [e.g., Kress, 1989].

Original: Different from the tabular case studied by Tennenholtzet al. [2020] and Nair and Jiang[2021] and linear models studied by Shi et al. [2021], solving linearintegral equations with non-parametric models in the continuous state/observation space are knownto be challenging due to the potential ill-posedness [Chen and Reiss, 2011], leading to slow statisticalconvergence rates.
Revised: Different from the tabular case studied by Tennenholtz et al.[2020] and Nair and Jiang [2021], solving linear integral equations with non-parametric models inthe continuous state/observation space are known to be challenging due to the potential ill-posedness[Chen and Reiss, 2011], leading to slow statistical convergence rates.

Original: However, existing theoretical results developed by Bennett and Kallus [2021] and Shi et al.
Revised: However, existing theoreticalresults developed by Bennett and Kallus [2021] and Shi et al.

Original: [2021] require fast enough convergence rates for these bridge function estimators in orderto establish the asymptotic normality of their estimators for OPE, which could be illusive when usingnon-parametric models.
Revised: [2021] require fast enough convergencerates for these bridge function estimators in order to establish the asymptotic normality results oftheir estimators for OPE, which could be illusive when using non-parametric models.

Original: Therefore, to ﬁll this important theoretical gap, it is important to study theﬁnite-sample performance of OPE of which bridge functions are estimated non-parametrically.
Revised: Therefore, toﬁll this important theoretical gap, it is important to study the ﬁnite-sample performance of OPE ofwhich bridge functions are estimated non-parametrically.

Original: Motivated by these, in this paper, we study the OPE  for confounded and episodic POMDPs withcontinuous states, where we non-parametrically estimate V -bridge functions.
Revised: Motivated by these, in this paper, we study the OPE problem for confounded and episodic POMDPswith continuous states where we non-parametrically estimate V -bridge functions.

Original: Our main contributionto the literature is three-fold.
Revised: Our main contribution to the literature is three-fold.

Original: First, relying on some time-dependent proxy variables, we establisha non-parametric identiﬁcation result for OPE using V -bridge functions for time-inhomogeneousconfounded POMDPs.
Revised: First, relying on some time-dependent proxy variables, we establisha non-parametric identiﬁcation result for OPE using V -bridge functions for time-inhomogeneousconfounded POMDPs.

Original: Based on the identiﬁcation result, we develop a new ﬁtted-Q-evaluation(FQE)-type approach to estimating V -bridge functions recursively and obtain an estimator for OPE basedon the bridge function estimators.
Revised: Based on the identiﬁcation result, we develop a new ﬁtted-Q-evaluation(FQE)-type approach to estimating V -bridge functions recursively and obtain an estimator for OPE basedon the bridge function estimators.

Original: At each step of our algorithm, we propose to ﬁt a non-parametricinstrumental variable (NPIV) regression using a min-max estimation method, i.e., solving a linearintegral equation with a non-parametric model.
Revised: At each step of our algorithm, we propose to ﬁt a non-parametricinstrumental variable (NPIV) regression using a min-max estimation method, i.e., solving a linearintegral equation with a non-parametric model.

Original: Our algorithm can be viewed as a sequential NPIVestimation, which is not well studied in the literature.
Revised: Our algorithm can be viewed as a sequential NPIVestimation, which is not well studied in the literature.

Original: Second and most importantly, we establish theﬁnite-sample error bound for estimating V -bridge functions and accordingly that for evaluating thepolicy value, in terms of the sample size , length of horizon and (local) measure of ill-posedness ateach step.
Revised: Second and most importantly, we establishthe ﬁnite-sample error bound for estimating V -bridge functions and accordingly that for evaluatingthe policy value, in terms of the sample size , length of horizon and (local) measure of ill-posednessat each step.

Original: Unlike the well studied standard NPIV model in the econometrics literature [e.g., Ai and Chen, 2003, Newey and Powell, 2003] where the response variable is directly observed, the responsevariable in our NPIV model at each step of the algorithm relies on the model estimate at its previousstep.
Revised: Unlike the well studied standard NPIV model in the econometrics literature [e.g., Aiand Chen, 2003, Newey and Powell, 2003] where the response variable is directly observed, theresponse variable in our NPIV model at each step of the algorithm relies on the model estimate atits previous step.

Original: This difference makes our theoretical analysis substantially difﬁcult.
Revised: This difference makes our theoretical analysis substantially difﬁcult.

Original: By carefully characterizingthe statistical error due to the NPIV estimation at each step and more importantly, its propagationeffect on future estimates, we are able to establish the ﬁrst ﬁnite-sample result ofOPE for confounded
Revised: By carefullycharacterizing the statistical error due to the NPIV estimation at each step and more importantly,its propagation effect on future estimates, we are able to establish the ﬁrst ﬁnite-sample result of

Original:  POMDPs under non-parametric models, which achieves a polynomial order over the length of horizonand sample size.
Revised: OPE for confounded POMDPs under non-parametric models, which achieves a polynomial orderover the length of horizon and sample size.

Original: Finally, our theoretical results on the sequential NPIV estimation are generallyapplicable to other sequential-type conditional moment restriction problems.
Revised: Finally, our theoretical results on the sequential NPIVestimation are generally applicable to other sequential-type conditional moment estimation problems.

Original: The development ofthe uniform ﬁnite-sample errorbounds of the NPIV estimation, extending the pointwise result in theprevious literature such as Dikkala et al.
Revised: The development of the uniform ﬁnite-sample error of the NPIV estimation, extending the pointwiseresult in the previous literature such as Dikkala et al.

Original: [2020], may be of independent interest.
Revised: [2020], may be of independent interest.

Original:  Related Work
Revised:  Related Work

Original: Recently there is a surge of interest in studying OPE with unobserved variables in the sequentialdecision making problem.
Revised: Recently there is a surge of interest in studying OPE with unobserved variables in the sequentialdecision making problem.

Original: Speciﬁcally, Zhang and Bareinboim [2016] are among the ﬁrst whoproposed the framework of confounded MDPs, which essentially considers i.i.d.
Revised: Speciﬁcally, Zhang and Bareinboim [2016] are among the ﬁrst whoproposed the framework of confounded MDPs, which essentially considers i.i.d.

Original: confounders in thedynamic system and therefore preserves the Markovian property.
Revised: confounders in thedynamic system and therefore preserves the Markovian property.

Original: Along this direction, OPE methodsare developed under various identiﬁcation conditions such as partial identiﬁcation using sensitivityanalysis [Namkoong et al., 2020, Kallus and Zhou, 2020, Bruns-Smith, 2021], instrumental variableor mediator assisted OPE
Revised: Along this direction, OPE methodsare developed under various identiﬁcation conditions such as partial identiﬁcation using sensitivityanalysis [Namkoong et al., 2020, Kallus and Zhou, 2020, Bruns-Smith, 2021], instrumental variableor mediator assisted OPE

Original: [Liao et al., 2021, Li et al., 2021, Shi et al., 2022] and many others.
Revised: [Liao et al., 2021, Li et al., 2021, Shi et al., 2022] and many others.

Original: Anotherline of research focuses on more general POMDP models (e.g., confounded POMDPs), where the
Revised: Anotherline of research focuses on more general POMDP models (e.g., confounded POMDPs) where the

Original: Markovian assumption is violated, under which several point estimation results were developed suchas the aforementioned negative controls/proxy variables related methods [Tennenholtz et al., 2020,
Revised: Markovian assumption is violated, under which several point estimation results were developed suchas the aforementioned negative controls/proxy variables related methods [Tennenholtz et al., 2020,

Original: Deaner, 2018, Ying et al., 2021, Bennett and Kallus, 2021, Nair and Jiang, 2021, Shi et al.
Revised: Deaner, 2018, Ying et al., 2021, Bennett and Kallus, 2021, Nair and Jiang, 2021, Shi et al.

Original: , 2021],spectral methods in undercomplete POMDPs [Hsu et al., 2012, Anandkumar et al., 2014, Jin et al.
Revised: , 2021],spectral methods in undercomplete POMDPs [Hsu et al., 2012, Anandkumar et al., 2014, Jin et al.

Original: ,2020] and predictive state representation related methods [Littman and Sutton, 2001, Singh et al.
Revised: ,2020] and predictive state representation related methods [Littman and Sutton, 2001, Singh et al.

Original: ,2012, Cai et al., 2022].
Revised: ,2012, Cai et al., 2022].

Original: Our proposed method, which uses proxy variables for OPE, is closely related to those recentlydeveloped by Bennett and Kallus [2021], Shi et al.
Revised: Our proposed method, which uses proxy variables for OPE, is closely related to those recentlydeveloped by Bennett and Kallus [2021], Shi et al.

Original: [2021], and Ying et al.
Revised: [2021], and Ying et al.

Original: Bennett and
Revised: Bennett and

Original: Kallus [2021] and Ying et al.
Revised: Kallus [2021] and Ying et al.

Original: [2021] studied episodic POMDPs (or complex longitudinal studies)and mainly focused on developing asymptotic normality results of their policy value estimators.
Revised: [2021] studied episodic POMDPs (or complex longitudinal studies)and mainly focused on developing asymptotic normality results of their policy value estimators.

Original: Their results rely on some high level rate conditions on the bridge function estimation, which areunknown if they would be satisﬁed when using non-parametric models due to the aforementionedmeasure of ill-posedness.
Revised: Their results rely on some high level rate conditions on the bridge function estimation, which areunknown if they would be satisﬁed when using non-parametric models due to the aforementionedmeasure of ill-posedness.

Original: Shi et al.
Revised: Shi et al.

Original: [2021] mainly focused on time-homogeneous inﬁnite-horizon
Revised: [2021] mainly focused on time-homogeneous inﬁnite-horizon

Original: POMDPs and developed asymptotic normality for their estimators under similar high-level conditions,which therefore has the same issue.
Revised: POMDPs and developed asymptotic normality for their estimators under similar high-level conditions,which therefore has the same issue.

Original: Besides, while Shi et al.
Revised: Besides, while Shi et al.

Original: [2021] also established ﬁnite-samplebounds for their bridge function estimation and corresponding OPE, they only study the tabularcase or linear/parametric models, where the issue of ill-posedness does not exist .
Revised: [2021] also established ﬁnite-samplebounds for their bridge function estimation and corresponding OPE, they only study the tabularcase or linear/parametric models, where the issue of ill-posedness does not exist .

Original: In this paper, weprovide a systematic investigation on the estimation of V -bridge functions and establish ﬁnite-sampleguarantees for them and the corresponding OPE under non-parametric models.
Revised: In this paper, weprovide a systematic investigation on the estimation of V -bridge functions and establish ﬁnite-sampleguarantees for them and the corresponding OPE under non-parametric models.

Original: Speciﬁcally, wetackle the challenging episodic setting, where V -bridge functions are estimated sequentially.
Revised: Speciﬁcally, wetackle the challenging episodic setting, where V -bridge functions are estimated sequentially.

Original: Withoutcarefully controlling the effect of ill-posedness at each step and its propagation effect on future steps,the estimation error for these V -bridge functions and also that for OPE could be exponentially largein terms of the length of horizon.
Revised: Withoutcarefully controlling the effect of ill-posedness at each step and its propagation effect on future steps,the estimation error for these V -bridge functions and also that for OPE could be exponentially largein terms of the length of horizon.

Original: Motivated by the chaining argument in the empirical process theory,we successfully disentangle the effects of ill-posedness on the current step and future steps separatelyand thus establish ﬁnite-sample bounds for V -bridge functions and OPE both with a polynomialdependence on the length of horizon, which are new theoretical results we contribute to the literature.
Revised: Motivated by the chaining argument in the empirical process theory,we successfully disentangle the effects of ill-posedness on the current step and future steps separatelyand thus establish ﬁnite-sample bounds for V -bridge functions and OPE both with a polynomialdependence on the length of horizon, which are new theoretical results we contribute to the literature.

Original: Since our V -bridge function estimation can be formulated as a sequential NPIV problem, it is natuallyrelated to classical NPIV estimations, which have been extensively studied in the econometricsliterature [see, e.g., Newey and Powell, 2003, Ai and Chen, 2003, 2012, Hall and Horowitz, 2005,
Revised: Since our V -bridge function estimation can be formulated as a sequential NPIV problem, it is natuallyrelated to classical NPIV estimations, which have been extensively studied in the econometricsliterature [see, e.g., Newey and Powell, 2003, Ai and Chen, 2003, 2012, Hall and Horowitz, 2005,

Original: Chen and Reiss, 2011, Chen and Christensen, 2018, Darolles et al., 2011, Blundell et al., 2007, forearlier reference].
Revised: Chen and Reiss, 2011, Chen and Christensen, 2018, Darolles et al., 2011, Blundell et al., 2007, forearlier reference].

Original: Recently there is also a growing interest in the min-max estimation for NPIVmodels [see, e.g., Muandet et al., 2020, Dikkala et al., 2020, Hartford et al., 2017, for some recentdevelopments].
Revised: Recently there is also a growing interest in the min-max estimation in NPIVmodels [see, e.g., Muandet et al., 2020, Dikkala et al., 2020, Hartford et al., 2017, for some recentdevelopments].

Original: As commented before, existing theoretical results for standard NPIV models cannotbe directly applied to our setting due to the sequential structure of our FQE algorithm, so we needto develop new theory to address our setting.
Revised: As commented before, existing theoretical results for standard NPIV models cannotbe directly applied to our setting due to the sequential structure of our FQE algorithm, so we needto develop new theory to address our setting.

Original: Technically, in order to establish a polynomial-orderﬁnite-sample error bound over the length of horizon for OPE, which is particularly important in RL,we decompose the measure of ill-posedness at each step of our sequential NPIV estimation into twocomponents: the so-called (local) measure of one-step transition ill-posedness and the standard (local)measure of ill-posedness [e.g., Chen and Pouzo, 2012].
Revised: Technically, in order to establish a polynomial-orderﬁnite-sample error bound over the length of horizon for OPE, which is particularly important in RL,we decompose the measure of ill-posedness at each step of our sequential NPIV estimation into twocomponents: the so-called (local) measure of one-step transition ill-posedness and the standard (local)measure of ill-posedness [e.g., Chen and Pouzo, 2012].

Original: Thanks to this novel decomposition, the effectof the ﬁrst component on the estimation error of V -bridge functions and OPE is multiplicative but canbe properly controlled while that of the second component could be large but is only cumulative.
Revised: Thanks to this novel decomposition, the effectof the ﬁrst component on the estimation error of V -bridge functions and OPE is multiplicative but canbe properly controlled while that of the second component could be large but is only cumulative.

Original: See
Revised: See

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: Finally, we remark that Ai and Chen [2012] also studied the sequential NPIV estimationproblem, where the non-parametric components are estimated jointly.
Revised: Finally, we remark that Ai and Chen [2012] also studied the sequential NPIV estimationproblem, where the non-parametric components are estimated jointly.

Original: However, this method could becomputationally inefﬁcient in RL with a long horizon.
Revised: However, this method could becomputationally inefﬁcient in RL with a long horizon.

Original: More importantly, their results are built on thenested structure among conditional moment restriction models, which are not satisﬁed in our setting.
Revised: More importantly, their results are built on thenested structure among conditional moment restriction models, which are not satisﬁed in our setting.

Original:  Preliminaries and Notations
Revised:  Preliminaries and Notations

Original: In this section, we introduce the framework of discrete-time confounded POMDPs and its related OPEproblem.
Revised: In this section, we introduce the framework of discrete-time confounded POMDPs and its related OPE problem.

Original: Consider an episodic and confounded POMDP denoted by M =
Revised: Consider an episodic and confounded POMDP denoted by M =

Original: ( S , U , A , T, P , r ) , with Sand U as the observed and unobserved continuous state spaces respectively,
Revised: ( S , U , A , T, P , r ) ,with S and U as the observed and unobserved continuous state spaces respectively,

Original: A as the discrete actionspace, T as the length of horizon, P = { P t } Tt =1 as the transition kernel over S × U × A to S × U ,and r = { r t } Tt =1 as the reward function over S ×U × A .
Revised: A as the discreteaction space, T as the length of horizon, P = { P t } Tt =1 as the transition kernel over S × U × A to S × U , and r = { r t } Tt =1 as the reward function over S × A .

Original: S can also be treated as the observationspace in the classical POMDP.
Revised: S can also be treated as the observationspace in the classical POMDP.

Original: Then the process of M can be summarized as { S t , U t , A t , R t } T t =1with
Revised: Then the process of M can be summarized as { S t , U t , A t , R t } Tt =1with

Original: S t and U t as observed and unobserved state variables, A t as the action, and R t as the reward,where r t ( s, u, a ) =
Revised: S t and U t as observed and unobserved state variables, A t as the action, and R t as the reward,where r t ( S t , A t ) =

Original: E [ R t | S t = s, U t = u, A t= a ]for any ( s, u, a ) ∈ S × U × A .
Revised: E [ R t | S t , A t ] .

Original: For simplicity,we assume that | R t | ≤ 1 uniformly in 1 ≤ t ≤ T .
Revised: For simplicity, we assume that | R t | ≤ 1 uniformly in 1 ≤ t ≤ T .

Original: The goal of OPE in a confounded POMDP is to evaluate the performance of a target policy usingthe batch data collected by some behavior policy.
Revised: The goal of OPE in a confounded POMDP is to evaluate the performance of a target policy usingthe batch data collected by some behavior policy.

Original: In this paper, the target policy we focus on is asequence of functions mapping from the state space S to a probability mass function over the actionspace A , denoted by π = { π t } Tt =1 , where π t ( a | s ) is the probability of choosing an action A t = agiven
Revised: In this paper, the target policy we focus on is asequence of functions mapping from the state space S to a probability mass function over the actionspace A , denoted by π = { π t } Tt =1 , where π t ( a | s ) is the probability of choosing an action A t = agiven

Original: the state value S t =
Revised: the state value S t =

Original: s .
Revised: s .

Original: We remark that our proposed identiﬁcation results stated in Section 4can be generalized to other policies such as history-dependent ones.
Revised: We remark that our proposed identiﬁcation results stated in Section 4can be generalized to other policies such as history-dependent ones.

Original: Given a target policy π , deﬁneits state value function as
Revised: Given a target policy π , deﬁneits state value function as

Original: [Equation]
Revised: [Equation]

Original: U [Equation] where E π denotes the expectation with respect to the distribution whose action at decision time tfollows π t for any t ≥ 1 .
Revised: where E π denotes the expectation with respect to the distribution whose action at decision time tfollows π t for any t ≥ 1 .

Original: We consider the batch setting, where the observed action A t is generatedby some behavior policy ˜ π bt depending on both S t and U t for 1 ≤ t ≤ T .
Revised: We consider the batch setting, where the observed action A t is generatedby some behavior policy ˜ π bt depending on both S t and U t for 1 ≤ t ≤ T .

Original: We aim to use the batch data to estimate the policy value of a target policy π , which is deﬁned as
Revised: We aim to use this batchdata to estimate the policy value of a target policy π , which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where we use E to denote the expectation with respect to the behavior policy.
Revised: where we use E to denote the expectation with respect to the behavior policy.

Original: Since U t is unobserved,standard OPE methods that rely on the Markov assumption will in general fail and give bias estimations.
Revised: Since U t is unobserved,standard OPE methods that rely on the Markov assumption will in general fail and give bias estimations.

Original: In the following, we introduce an identiﬁcation result for estimating the policy value with thehelp of some proxy variables.
Revised: In the following, we introduce an identiﬁcation result for estimating the policy value with thehelp of some proxy variables.

Original: Notations :For two sequences { (cid:36) ( n ) } n ≥ 1 and { θ ( n ) } n ≥ 1 , the notation (cid:36) ( n ) (cid:38) θ ( n ) (resp.
Revised: Notations : { (cid:36) ( n ) } n ≥ 1 and { θ ( n ) } n ≥ 1 , the notation (cid:36) ( n ) (cid:38) θ ( n ) (resp.

Original: (cid:36) ( n ) (cid:46)θ ( n ) ) means that there exists a sufﬁciently large constant (resp. small) constant c 1 > 0 (resp.
Revised: (cid:36) ( n ) (cid:46) θ ( n ) ) meansthat there exists a sufﬁciently large constant (resp. small) constant c 1 > 0 (resp.

Original: c 2 > 0 )such that (cid:36) ( N ) ≥ c 1 θ ( N ) (resp.
Revised: c 2 > 0 ) suchthat (cid:36) ( N ) ≥ c 1 θ ( N ) (resp.

Original: (cid:36) ( n ) ≤ c 2 θ ( n ) ).
Revised: (cid:36) ( n ) ≤ c 2 θ ( n ) ).

Original: We use (cid:36) ( n ) (cid:16) θ ( n ) when (cid:36) ( n ) (cid:38) θ ( n ) and(cid:36) ( n ) (cid:46) θ ( n ) .
Revised: We use (cid:36) ( n ) (cid:16) θ ( n ) when (cid:36) ( n ) (cid:38) θ ( n ) and(cid:36) ( n ) (cid:46) θ ( n ) .

Original: For any random variable X , we use L q { X } to denote the class of all measurablefunctions with ﬁnite q -th moments for 1 ≤ q ≤ ∞ .
Revised: For any random variable X , we use L q ( X ) to denote the class of all measurablefunctions with ﬁnite q -th moments for 1 ≤ q ≤ ∞ .

Original: Then the L q -norm is denoted by (cid:107) • (cid:107)
Revised: Then the L q -norm is denoted by (cid:107) • (cid:107)

Original: L q { X } .
Revised: L q ( X ) .

Original: When there is no confusion in the underlying distribution, we also write it as (cid:107) • (cid:107) L q or (cid:107) • (cid:107) q .
Revised: When there is no confusion in the underlying distribution, we also write it as (cid:107) • (cid:107) L q or (cid:107) • (cid:107) q .

Original: Inparticular, (cid:107) • (cid:107) ∞ denotes the sup-norm.
Revised: Inparticular, (cid:107) • (cid:107) ∞ denotes the sup-norm.

Original: In addition, we use Big O and small o as the convention.
Revised: In addition, we use Big O and small o as the convention.

Original:  Identiﬁcation Results
Revised:  Identiﬁcation Results

Original: Inspired by the proximal causal inference recently proposed by TchetgenTchetgen et al.
Revised: Inspired by the proximal causal inference recently proposed by Tchetgen et al.

Original: [2020],we develop a non-parametric identiﬁcation result for estimating V ( π ) , which is similar to thoseby Bennett and Kallus [2021] and Shi et al.
Revised: [2020], we developa non-parametric identiﬁcation result for estimating V ( π ) , which is similar to those by Bennettand Kallus [2021] and Shi et al.

Original: Assume that we can additionally observe theso-called reward-inducing proxy variables W t that are only related to the action A t through ( S t , U t )and action-inducing proxy variables Z t that are only related to the reward R t through ( S t , U t ) at eachdecision time t .
Revised: Assume that we can additionally observe the so-calledreward-inducing proxy variables W t that are only related to the action A t through ( S t , U t ) andaction-inducing proxy variables Z t that are only related to the reward R t through ( S t , U t ) at eachdecision time t .

Original: See Figure 1 for a directed acyclic graph (DAG) to illustrate their relationships anda time series data example in Miao et al.
Revised: See Figure 1 for a directed acyclic graph (DAG) to illustrate their relationships anda time series data example in Miao et al.

Original: [2018b].
Revised: [2018b].

Original: For another example, the action-inducing proxyvariables Z t can be  deﬁned as the observed history before time t , then Z t and related arrows in Figure1 can be removed.
Revised: For another example, the action-inducing proxyvariables Z t can be implicitly deﬁned as the observed history before time t , then Z t and relatedarrows in Figure 1 can be removed.

Original: Detailed assumptions and discussion are given in Appendix A. Denote the spacesof { Z t } Tt =1 and { W t } Tt =1 by W and Z respectively. 
Revised: Detailed assumptions and discussion are given in Appendix A. Denote the spaces of { Z t } Tt =1 and { W t } Tt =1 by W and Z respectively. [Figure] [Equation]

Original: [Figure]
Revised: [Figure]

Original: Since the states { U t } T t =1 are unmeasured, we cannot estimate the value function by the celebrated
Revised: Since the states { U t } Tt =1 are unmeasured, we cannot estimate the value function by the celebrated

Original: Bellman equation.
Revised: Bellman equation.

Original: However, with the help of confounding proxies { W t , Z t } Tt =1 , the value of a targetpolicy π can be non-parametrically identiﬁed using observed variables under proper assumptions.
Revised: However, with the help of confounding proxies { W t , Z t } Tt =1 , the value of a targetpolicy π can be non-parametrically identiﬁed using observed variables under proper assumptions.

Original: To proceed, we deﬁne a class of V -bridge functions (or V -bridges for short) { v πt } Tt =1 deﬁned over
Revised: To proceed, we deﬁne a class of V -bridge functions (or V -bridges for short) { v πt } Tt =1 deﬁned over

Original: W × S such that for every ( s, u ) ∈ S
Revised: W × S such that for every ( s, u ) ∈ S

Original: × U and t ≥ 1 ,
Revised: × U and t ≥ 1 ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If such V -bridges exist, then we obtain the following identiﬁcation result for the policy value in (2).
Revised: If such V -bridges exist, then we obtain the following identiﬁcation result for the policy value in (2).

Original: Proposition 4.1 (Identiﬁcation) .
Revised: Proposition 4.1 (Identiﬁcation) .

Original: If there exist { v πt } Tt =1 that satisfy (3), then the value of policy πcan be identiﬁed by V ( π ) =
Revised: If there exist { v πt } Tt =1 that satisfy (3), then the value of policy πcan be identiﬁed by V ( π ) =

Original: E [ v π 1 ( W 1 , S 1 )] .
Revised: E [ v π 1 ( W 1 , S 1 )] .

Original: Note that V -bridges { v π t } T t =1 that satisfy (3) are not necessarily unique, but we canuniquely identify V ( π ) based on any of them.
Revised: Note that V -bridges { v πt } Tt =1 that satisfy (3) are not necessarily unique, but we can identify V ( π )based on any of them.

Original: Next, we provide a theoretical guarantee for the existence of V -bridges{ v πt } Tt =1 in terms of a sequence of linear integral equations.
Revised: Next, we provide a theoretical guarantee for the existence of V -bridges{ v πt } Tt =1 in terms of a sequence of linear integral equations.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For a POMDP model of which variables satisfy the relationships illustrated in Figure 1and some regularity conditions given in Appendix A, there always exist V -bridges { v πt } Tt =1 satisfying
Revised: For a POMDP model of which variables satisfy the relationships illustrated in Figure 1and some regularity conditions given in Appendix A, there always exist V -bridges { v πt } Tt =1 satisfying(3).

Original: With v πT +

Revised: With v πT +

Original: =
Revised: =

Original: 0 , a particular sequence of V -bridges { v πt } Tt =1 can be obtained by solving thefollowing linear integral equations:
Revised: 0 , a particular sequence of V -bridges { v πt } Tt =1 can be obtained by solving thefollowing linear integral equations:

Original: [Equation]
Revised: [Equation]

Original: where { q πt } Tt =1 are Q -bridges deﬁned over W × S
Revised: where { q πt } Tt =1 are Q -bridges deﬁned over W × S

Original: ×
Revised: ×

Original: A such that
Revised: A such that

Original: [Equation]
Revised: [Equation]

Original: for every ( s, u, a ) ∈ S × U × A and t ≥ 1 , and v πt ( w, s ) = (cid:80) a ∈A π t ( a | s ) q πt ( w, s, a ) .
Revised: for every ( s, u, a ) ∈ S × U × A and t ≥ 1 , and v πt ( w, s ) = (cid:80) a ∈A π t ( a | s ) q πt ( w, s, a ) .

Original: Clearly  Q -bridges { q πt } Tt =1 also exist.
Revised: Clearly (4)also guarantees the existence of Q -bridges { q πt } Tt =1 .

Original: Theorem 4.1 guarantees the existence of both V -bridges  and Q -bridges, and also provides a naturalprocedure (4) to ﬁnd { v πt } Tt =1 and eventually estimate the policy value V ( π ) .
Revised: Theorem 4.1 guarantees the existence of both V -bridges { v πt } Tt =1 and Q -bridges, and also provides anatural procedure (4) to ﬁnd { v πt } Tt =1 . in order to eventually estimate the policy value V ( π ) .

Original: Then based on Proposition 4.1 and Theorem 4.1, we can perform OPE via Algorithm 1 in the population level.
Revised: Thendue to Proposition 4.1 and Theorem 4.1, we can perform OPE via Algorithm 1 in the population sense.

Original: Speciﬁcally at each step we will solve (4) via a non-parametric model, which is a NPIV problem.
Revised: At the t -th step, we propose to solve (4) with a non-parametric model, which is a NPIV problem.

Original: 1 Input: { ( S t , W t , Z t , A t , R t ) } Tt =1 , a target policy π = { π t} Tt =1 .
Revised: 1 Input: { ( S t , Z t , W t , A t , R t ) T t =1 , a target policy π = { π t

Original:  2 Let v πT +1 =0 .
Revised: T t =1 . 2 Let v πT +1 =

Original: 3 Repeat for t = T, . . .
Revised: 3 Repeat for t = T, . . .

Original:  Solve v πt and q πt by E (cid:8) q πt ( W t , S t , A t ) − R t − v πt
Revised: , 1 : Solve v πt and q πt by E (cid:8) q πt ( W t , S t , A t ) − R t − v πt

Original: +

Revised: +

Original: ( W t +1 , S t+1 )
Revised: ( W t +1 , S t

Revised: +

Original:  | Z t, S t, A t (cid:9)=
Revised: 1 ) | Z t S t A t (cid:9)

Original:  with v πt ( W t , S t ) (cid:44) (cid:80) a ∈A π t ( a | S t ) q πt( W t , S t , a ) .
Revised: = 0 with v πt ( W t , S t ) (cid:44) (cid:80) a ∈A π t ( a | S t ) q πt

Revised: ( W t , S t ,

Original: 5 Output: V( π ) =
Revised: a Output: V π ) =

Original: E [ v π 1 ( W 1 , S 1 )] .
Revised: E [ v π 1 ( S 1 , W 1 )] .

Original:  Estimation
Revised:  Estimation

Original: In this section, we discuss how to estimate V ( π ) using batch data based on results given in Theorem4.1 and Algorithm 1.
Revised: In this section, we discuss how to estimate V ( π ) using batch data based on results given in Theorem4.1 and Algorithm 1.

Original: Let a pre-collected training dataset be D n = { ( S t,i , W t,i , Z t,i , At,i , R t,i )
Revised: Let a pre-collected training dataset be D n = { ( S i,t , W i,t , Z i,t , A

Original:  Tt =1 :i = 1 , . . . , n } , which consists of n i.i.d.
Revised: i,t , R i,t ) Tt =1 :i = 1 , . . . , n } , which consists of n i.i.d.

Original: copies of the observable trajectory ( S t , W t , Z t , A t , R t ) Tt =1of a confounded POMDP.
Revised: copies of the observable variables ( S t , W t , Z t , A t , R t ) Tt =1 ofa confounded POMDP.

Original: Following Algorithm 1, we develop a FQE-type approach where wepropose to solve a min-max problem for estimating v πt at the t -th step using the idea of Dikkala et al.[2020], 1 ≤ t ≤ T , and then apply Proposition 4.1 for OPE.
Revised: Following Algorithm 1, we develop a FQE-type approach where we proposeto solve a min-max problem for estimating v πt at the t -th step using the idea of Dikkala et al. [2020],1 ≤ t ≤ T , and then apply Proposition 4.1 for OPE.

Original: For convenience, we ﬁrst rewrite the linear integral equations (4) for solving V -bridges in terms ofoperators.
Revised: For convenience, we ﬁrst rewrite the linear integral equations (4) to solve V -bridges in terms of operators.

Original: Deﬁne an operator (cid:101) P t : L 2 {R×W×S} → L 2 {Z×S×A} such that [ (cid:101) P πt g ]( Z t , S t , A t ) =
Revised: Deﬁne an operator (cid:101) P t : L 2 {R × W × S} → L 2 {Z × A × S} such that [ (cid:101) P πt g ]( Z t , A t , S t ) =

Original:  E [  g ( R t , W t 
Revised: 196 E [ (cid:104) g ( R t , W t +1 , · , S t +1 ) , π t +1 ( · | S t +1 ) (cid:105) | Z t , A t , S t ] for any g ∈ L 2 {R × W × A × S} , where(cid:104)

Original:  +1 ,  S t +1 ) 
Revised: g ( R t , W t +1 , · , S t +1 ) , π t +1 ( · | S t +1 ) (cid:105) (cid:44) (cid:80) a ∈A g

Original: | Z t  ,  S t  , A t ] for any g ∈ L 2 {R × W × S} .
Revised: ( R t , W t +1 , a, S t +1 ) , π t +1 ( a | S t +1 ) .

Original: Deﬁne another operator P t :
Revised: Deﬁne another operator P t :

Original: L 2 {W × S × A} → L 2 {Z × S × A} such that for any h ∈ L 2 {W × S × A} , [ P t h ]( Z t , S t , A t ) =199 E [ h ( W t , S t , A t ) | Z t , S t , A t ] .
Revised: L 2 {W × A × S} → L 2 {Z × A × S} such that for any h ∈ L 2 {W × A × S} ,[ P t h ]( Z t , A t , S t ) = E [ h ( W t , A t , S t ) | Z t , A t , S t ] .

Original: Motivated by (4), we deﬁne the V-bridge transition operator P πt : 
Revised: Motivated by (4), we deﬁne the V-bridge transition operator P πt : L 2 {R × W × S} →

Original: L 2{R × W × S} → L 2 {W × S} such that
Revised: L 2 {W × S} such that

Original: P πt g = (cid:104) π t , P t g (cid:105) , where P t g =
Revised: P πt g = (cid:104) π t , P t g (cid:105) , where P t g =

Original: P t − 1 (cid:101) P t g for all g ∈ L 2 {R × W × S} , (cid:104) π t ( · | S t ) , [ P t g ]( W t ,  S t, · ) (cid:105) (cid:44) (cid:80) a ∈A π t ( a | S t )
Revised: P t − 1 (cid:101) P t g for all g ∈ L 2 {R × W × S} , (cid:104) π t ( · | S t ) , [ P t g ]( W t , · , S t ) (cid:105) (cid:44) (cid:80) a ∈A π t ( a | S t )

Original: [ P t g ]( W t ,  S t, a ) , and (cid:101) P t g invertible by P t .
Revised: [ P t g ]( W t , · , S t ) , and (cid:101) P t g invertible by P t .

Original: Theinvertibility is ensured by Assumption 8 in Appendix A.
Revised: Theinvertibility is ensured by Assumption 8 in Appendix A.

Original: Then by the deﬁnition of V -bridges and (4), we can identify { v πt } Tt =1 equivalently via solving [Equation]
Revised: Then by the deﬁnition of V -bridges and (4), we can identify { v πt } Tt =1 equivalently via solvingv πt =

Original: Hence it sufﬁces to estimate P πt .
Revised: P πt ( v πt +1 + R t ) , for t ≥ 1 , (6) so it sufﬁces to estimate P πt .

Original: Note that one can regard (6) as a series of conditional momentmodel restrictions and we propose to solve them via a sequential NPIV estimation.
Revised: We regard (6) as a series of conditional moment model restrictions andpropose to solve them via a sequential NPIV estimation.

Original: In particular, atthe t -th step, we adopt the min-max estimation method proposed by Dikkala et al.
Revised: In particular, at the t -th step, we adopt themin-max estimation method proposed by Dikkala et al.

Original: [2020] to estimate P πt non-parametrically as follows: (cid:98) P π t g
Revised: [2020] to estimate P πt non-parametrically asfollows: (cid:98) P πt g

Original: =
Revised: =

Original: (cid:68) π t , (cid:98) P t g (cid:69) , where
Revised: (cid:68) π t , (cid:98) P t g (cid:69) , where

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) f
Revised: where (cid:107) f

Original: (cid:107) 2 n = n − 1 (cid:80) ni
Revised: (cid:107) 2 n = n − 1 (cid:80) nk

Original: =1 f 2 ( Z t,i , S t,i , A t,i ) , the user-deﬁned function spaces H ( t ) on W ×S × A and F ( t ) on Z × 
Revised: =1 f 2 ( Z t,k , A t,k , S t,k ) , the user-deﬁned function spaces H ( t ) on W × A × Sand F ( t ) on Z × A × S are endowed with norms (cid:107) • (cid:107) H ( t ) and (cid:107) • (cid:107) F ( t ) respectively, and tuningparameters λ, µ, M, δ > 0 , and [Equation]

Original: S
Revised: n

Original: ×
Revised: [Equation]

Original: A are endowed with norms (cid:107) • (cid:107) H (  t ) and (cid:107) • (cid:107) F ( t ) respectively, and tuningparameters λ, µ, M, δ > 0 , andΨ t,n ( h, f, g )= n − 1 (cid:80) ni
Revised: where g ( R t , W t +1 , S t +1 )

Original: [-=
-]
Revised: [-=
-]=

Original: [ h ( W t,i , S t,i , A t,i ) − ( T − t +1) − 1 g ( R t,i , W t +1 ,i , S t +1 ,i )] f ( Z t,i , S t,i , A t,i ) , where g ( R t , W t +1 , S t +1 ) = R t + ¯ g ( W t +1 , S t +1 ) for some ¯ g ∈ G ( t +1) on W × S , endowed withnorm (cid:107) • (cid:107) G
Revised: R t + ¯ g ( W t +1 , S t +1 ) for some ¯ g ∈ G ( t +1) on W × S , endowed withnorm (cid:107) • (cid:107) G

Original: ( t +1) .
Revised: ( t +1) .

Original: The rational behind (7) is that when λ, λµ → 0 and λM/δ 2 (cid:16) 1 , the following two population-versionmin-max optimization problems 
Revised: The rational behind (7) is that when λ, λµ → 0 and λM/δ 2 (cid:16) 1 , the following two population-versionmin-max optimization problems [Equation] and

Original: [Equation]
Revised: [Equation]

Original: have the same solution h when the space F ( t ) of testing functions is rich enough.
Revised: have the same solution h when the space F ( t ) of testing functions is rich enough.

Original: Note that( T − t + 1) − 1 used above and in (7) are for scaling purpose.
Revised: Note that [Equation] +1 216 used above and in (7) are for scaling purpose.

Original: After T steps, we output our estimator for the policy value based on the empirical counterpart of Proposition 4.1.
Revised: After T steps, we output our estimator for the policy value based on the empiricalcounterpart of Proposition 4.1.

Original: Our FQE-type algorithm is summarized in Algorithms 2 and 3 in Appendix E.{+Algorithm 2. Algorithm 2: A FQE-type algorithm with a sequential NPIV estimation ˆ v πT +
 = 0 .+}
Revised: Our FQE-type algorithm is summarized in {+Algorithm 2. Algorithm 2: A FQE-type algorithm with a sequential NPIV estimation ˆ v πT +
 = 0 .+}

Revised: 3 Repeat for t = T, . . .

Original: Theoretical Results
Revised: Solve (cid:98) P t by (7) and compute ˆ v πt = (cid:98) P πt (ˆ v πt +1 + R t ) where (cid:98) P πt g = (cid:68) π t , (cid:98) P t g (cid:69) . 5 Output: (cid:98) V ( π ) = n − 1 (cid:80)

Original:  In this section, we establish the ﬁnite-sample bounds for the L 2 error of estimating V -bridge v π 1and the error of OPE, in terms of the sample size, length of horizon and two (local) measures ofill-posedness.
Revised: nk =1 ˆ v π 1 ( W 1 ,k , S 1 ,k ) .  Theoretical Results In this section, we establish the ﬁnite-sample bounds for the L 2 error of estimating V -bridge v π 1and the error of OPE, in terms of the sample size, length of horizon and two (local) measures ofill-posedness.

Original: Our bounds also reply on the critical radii of certain spaces related tothe user-deﬁnedfunction spaces H ( t ) and F ( t ) in (7), and also G ( t ) of V -bridge functions.1.
Revised: Our bounds also reply on the critical radii of certain spaces related to user-deﬁnedfunction spaces H ( t ) and F ( t ) in (7), and also G ( t ) of V -bridge functions.1.

Original: Technical preliminaries.
Revised: Technical preliminaries.

Original: To present our main results, we ﬁrst introduce some concepts from theempirical process theory [Wainwright, 2019].
Revised: To present our main results, we ﬁrst introduce some concepts from theempirical process theory [Wainwright, 2019].

Original: Deﬁnition 6.1 (Local Rademacher Complexity) .
Revised: Deﬁnition 6.1 (Local Rademacher Complexity) .

Original: Given any real-valued function class F deﬁnedover a random vector X and any radius δ > 0 , the local Rademacher complexity is given by 
Revised: Given any real-valued function class F deﬁnedover a random vector X and any radius δ > 0 , the local Rademacher complexity is given by [Equation] n ( F , δ ) = E (cid:15),X (cid:34) sup

Original: [Equation]
Revised: [Equation]

Original:  where { X i } ni =1 are i.i.d.
Revised: : (cid:107) f (cid:107) n ≤ δ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)n n (cid:88) i = (cid:15) i f ( X i ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35) , (8) where { X i } ni =1 are i.i.d.

Original: copies of X and { (cid:15) i } ni =1 are i.i.d.
Revised: copies of X and { (cid:15) i } ni =1 are i.i.d.

Original: Rademacher random variables.
Revised: Rademacher random variables.

Revised: Let

Revised: R n ( F ) (cid:44) R n ( F , ∞ ) denote the non-localized Rademacher complexity.

Original: By bounding the local Rademacher complexity, which measures the complexity of the functionalclass F locally in a neighborhood of the ground truth, we can control the error rate of the proposed
Revised: By bounding the local Rademacher complexity, which measures the complexity of the functionalclass F locally in a neighborhood of the ground truth, we can control the error rate of the proposed

Original: V -bridge estimator in each step.
Revised: V -bridge estimator in each step.

Original: A crucial parameter for local Rademacher complexity of a functionclass F is called critical radius .
Revised: A crucial parameter for local Rademacher complexity of a functionclass F is the critical radius .

Original: Deﬁnition 6.2 (Critical Radius) .
Revised: Deﬁnition 6.2 (Critical Radius) .

Original: Assume that F is a star-shaped function class, i.e. αf ∈ F for anyf ∈ F and scalar α ∈ [0 , 1] , and also that F is b -uniformly bounded, i.e., (cid:107) f (cid:107) ∞ ≤ b < ∞ , ∀ f ∈ F .
Revised: Assume that F is a star-shaped function class, i.e. αf ∈ F for anyf ∈ F and scalar α ∈ [0 , 1] , and also that F is b -uniformly bounded, i.e., (cid:107) f (cid:107) ∞ ≤ b < ∞ , ∀ f ∈ F .

Original: The critical radius of F , denoted by δ n , is the solution to the inequality R n ( F , δ ) ≤ δ 2 /b.
Revised: The critical radius of F , denoted by δ n , is the solution to the inequality R n ( F , δ ) ≤ δ 2 /b.

Original: Additional Notations : We assume that the test functions f belong to a star shaped, symmetric space
Revised: Additional Notations : We assume that the test functions f belong to a star shaped, symmetric space

Original: F ( t ) ⊆
Revised: F ( t ) ⊆

Original: L 2 ( Z ×S × A  ) endowed with norm (cid:107) · (cid:107) F ( t ) .
Revised: L 2 ( Z × A × S ) endowed with norm (cid:107) · (cid:107) F ( t ) .

Original: For brevity of notation, hereafter we suppressthe time-step indicator ( t ) in the context unless necessary.
Revised: For brevity of notation, hereafter we suppressstep indicator ( t ) in the context unless needed to specify it.

Original: For a function space F , we deﬁneα F = { αf : f ∈ F} ,for some α ∈ R .
Revised: For a function space F , we deﬁneα F = { αf : f ∈ F} , α ∈ R .

Original: Deﬁne F B = { f ∈ F : (cid:107) f (cid:107) 2 F ≤ B } , for any B > 0 .
Revised: Deﬁne F B = { f ∈ F : (cid:107) f (cid:107) 2 F ≤ B } , for any B > 0 .

Original: Deﬁnethe projected root mean squared error (cid:107) proj t f (cid:107) 

Revised: Deﬁnethe projected root mean squared error (cid:107) proj t f (cid:107)

Original: = (cid:112) E { E [ f ( X ) | Z t , S t , A t ] } 2 , for any squaredintegrable f with respect to the conditional distribution of X given ( Z t , S t , A t ) .
Revised: = (cid:112) E { E [ f ( X ) | Z t , A t , S t ] } 2 , for any squaredintegrable f with respect to the conditional distribution of X given ( Z t , A t , S t ) .

Original: Standard (Local) Measures of ill-posedness : Let¯ τ  1 = sup g ∈G (1) (cid:107) g
Revised: Standard (Local) Measures of ill-posedness : Let τ π 1 = sup g ∈G (1) (cid:107) g

Original: ( W 1 , S 1 ) (cid:107) 2 / (cid:107) E  [ g ( W 1 , S 1 ) |
Revised: ( W 1 , S 1 ) (cid:107) 2 / (cid:107) E π 1 [ g ( W 1 , S 1 ) |

Original: Z 1 , S 1 ] (cid:107) 2 be the measure of ill-posedness for G (1) ( W 1 × S 1 ) projected on Z 1 × S 1 .
Revised: Z 1 , S 1 ] (cid:107) 2 be the measure of ill-posedness for G (1) ( W 1 × S 1 ) projected on Z 1 × S 1 .

Original: Let τ t =
Revised: Let τ t =

Original: sup h ∈H ( t ) (cid:107) h
Revised: sup h ∈H ( t ) (cid:107) h

Original: ( W t , S t , A t ) (cid:107) 2 / (cid:107) proj t h ( W t , S t , A t ) (cid:107) 2 be the standard measure of ill-posedness for
Revised: ( W t , S t , A t ) (cid:107) 2 / (cid:107) proj t h ( W t , S t , A t ) (cid:107) 2 be the standard measure of ill-posedness for

Original: H ( t ) ( W × S × A  ) projected on Z × S × A  .
Revised: H ( t ) ( W t × A t × S t ) projected on Z t × A t × S t .

Original: It can be seen that¯ τ  1 , τ t ≥ 1 for t ≥ 1 .
Revised: It can be seen that τ π 1 , τ t ≥ 1 for t ≥ 1 .

Original: Indeed we onlyrequire measuring¯ τ  1 and τ t locally .
Revised: Indeedwe only require measuring τ π 1 and τ t locally .

Original: See more details in Appendix C.2.
Revised: See more details in Appendix C.2.

Original: Results.
Revised: Results.

Original: We ﬁrst give Assumption 1 used to develop our theoretical results below.
Revised: We ﬁrst give Assumption 1 used to develop our theoretical results below.

Original: Assumption 1.
Revised: Assumption 1.

Original: For each t = 1 , . . .
Revised: For each t = 1 , . . .

Original: , T ,(1) Closeness.
Revised: , T , (1) Closeness.

Original: For any g ∈ G ( t +1) , P t ( g + R t ) ∈ H ( t ) ; For any h ∈ H ( t ) , (cid:104) π t , h
Revised: For any g ∈ G ( t +1) , P t ( g + R t ) ∈ H ( t ) ; for any h ∈ H ( t ) , (cid:104) π t , h

Original: (cid:105) ∈ G ( t ) .
Revised: (cid:105) ∈ G ( t ) .(2)

Original: (2) For any h ∈ ( T − t + 2) H ( t +1), we have (cid:107)P t
Revised: For any h ∈ ( T − t + 2) H ( t +1)

Original: [Table]
Revised: ( W , A , S ) , we have (cid:107)P t (cid:16) R t + (cid:104) π t +1 ,h (cid:105) T − t + (cid:17) (cid:107) 2 H ≤ (cid:107) h T − t +2 (cid:107) 2 H .(3)

Original: There exists a constant C G > 0 such that (cid:107) (cid:104) π t , h (cid:105) (cid:107) 2 G( t ) ≤ C G (cid:107) h (cid:107) 2 H( t ) , for h ∈ H ( t )  .(4)
Revised: There exists a constant C G > 0 such that (cid:107) (cid:104) π t , h (cid:105) (cid:107) 2 G ≤ C G (cid:107) h (cid:107) 2 H , for h ∈ H ( t ) ( W × A × S ) .(4)

Original: q πt ∈ ( T − t + 1) H ( t ) ( W ,S , A  ) and (cid:107) q πT (cid:107) 2 H( T ) ≤ M H , where M H > 0 is a constant.(5) Testing function class F ( t ) is sufﬁciently rich such that there exists L > 0 , (cid:107) f ∗ − proj t
Revised: q πt ∈ ( T − t + 1) H ( t ) ( W , A , S ) and (cid:107) q πT (cid:107) 2 H ≤ M H , where M H > 0 is a constant.(5) Testing function class F ( t ) is sufﬁciently rich such that there exists L > 0 , (cid:107) f ∗ − proj t

Original: h t (cid:107) 2 ≤ η ( t ) n ,where f ∗ ∈ arg min f ∈F ( t ) L 2 (cid:107) ht (cid:107) 2 H (cid:107) f − proj t h t (cid:107) 2 , for all h t ∈ H (  t ) .(6)
Revised: h t (cid:107) 2 ≤ η ( t ) n ,where f ∗ ∈ arg min f ∈F ( t ) L 2 (cid:107) ht (cid:107) 2 H (cid:107) f − proj t h t (cid:107) 2 , for all h t ∈ H ( W t , A t , S t ) .(6)

Original: Behavior policies: there exists a constant b π such that π bt ( a | s ) (cid:44) E
Revised: Behavior policies. There exists a constant b π such that π bt ( a | s ) (cid:44) E

Original: [˜ π bt ( a | U t , S t ) | S t =
Revised: [˜ π bt ( a | U t , S t ) | S t =

Original: s ] ≥b π > 0 for all ( s, a ) ∈ S × A .
Revised: s ] ≥b π > 0 for all ( s, a ) ∈ S × A .

Original: Assumption 1 (1) is similar to Bellman completeness, which has been widely used in RL withoutunobserved states [e.g., Antos et al., 2008].
Revised: Assumption 1 (1) and (5) are similar to Bellman completeness widely used in RL without unobservedstates [e.g., Antos et al., 2008].

Original: Note that both G ( t ) and H ( t ) can be chosen as inﬁnitedimensional spaces, e.g., RKHSs.

Original: Hence this assumption is relatively mild.

Original: Assumption 1 (2) requiresthe operator P t to be bounded, which can be ensured under some continuity conditions on transitionkernels [Kress, 1989].
Revised: Assumption 1 (2) requires the operator P t to be bounded, which canbe ensured under some continuity conditions on transition kernels [Kress, 1989].

Original: Assumption 1 (3) is a technical condition for controlling the complexity of Gby H .
Revised: Assumption 1 (3) isa technical condition for controlling the complexity of G .

Original: Assumption 1 (4) essentially assumes that we can model q π (and v π ) correctly at eacht -step, which isagain mild as H ( t )  for t ≥ 1 can all be chosen as inﬁnite-dimensional spaces, e.g., RKHSs.
Revised: Assumption 1 (4) essentially assumes thatwe can model q π (and v π ) correctly at each t -step, which is mild as H ( t ) ( W , A , S ) for t ≥ 1 are allinﬁnite-dimensional spaces.

Original: This assumption is also called realizability of value functions, which is commonly seen inthe literature of RL [e.g., Antos et al., 2008].

Original: Assumption 1 (5) is imposed to ensure that the space oftesting functions F is large enough so that we are able to capture the conditional expectation operatorin each min-max estimation (7).

Original: Assumption 1 (6) basically requires a full coverage of our batch datagenerating process induced by the behavior policy, which is widely used in OPE [Precup, 2000, Antoset al., 2008].
Revised: Assumption 1 (6) basically requires a full coverage in our batch data,which is widely used in OPE [Precup, 2000, Antos et al., 2008].

Original: Next, we ﬁrst provide a key decomposition of the L 2 error for V -bridge estimation.
Revised: We ﬁrst provide a key decompositionof the L 2 error for V -bridge estimation.

Original: Theorem 6.1 (Error decomposition) .
Revised: Theorem 6.1 (Error decomposition) .

Original: Under Assumption 1 (1) and (6), we can decompose the L 2error of the estimated V -bridge by
Revised: Under Assumption 1 (1) and (6), we can decompose the L 2 ( W × S ) error of the estimated V -bridge by

Original: [Equation]
Revised: [Equation]

Original: where the measures of one-step transition ill-posedness C ( t ) 1 , 0 (cid:44) 1 and C ( t ) t(cid:48) ,
Revised: where the measures of one-step transition ill-posedness C ( t ) 1 , 0 (cid:44) 1 and C ( t ) t

Original: [Equation]
Revised: [Equation]

Revised: t

Original: (cid:48) − 1 , 2 ≤ t (cid:48) ≤ t , 2 ≤ t ≤ Tare deﬁned after Corollary 6.2.
Revised: [Equation] − 1 , 2 ≤ t [Equation] ≤ t , 2 ≤ t ≤ Tare deﬁned after Corollary 6.2.

Original: Theorem 6.1 shows that there are four key components for  upper bounding the L 2 error of ˆ v π 1 .
Revised: Theorem 6.1 shows that there are four key components for the upper bound of the L 2 error of ˆ v π 1 .

Original: The ﬁrstcomponent is the probability ratio (cid:107) π t /π bt (cid:107) ∞ , which is used to measure the distributional mismatchbetween the target and behavior policies, which is the fundamental challenge and irreducible in ofﬂine RL [Levine et al., 2020].
Revised: The ﬁrst component is the probability ratio (cid:107) π t /π bt (cid:107) ∞ , which is used to measure the distributionalmismatch between the target and behavior policies, which is the fundamental challenge and irreduciblein ofﬂine RL [Levine et al., 2020].

Original: The second component is (cid:107) proj t
Revised: The second component is (cid:107) proj t

Original: ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 , the one-stepprojected error of (cid:98) P t to P t where ˆ v πt +1 is the estimate for v πt +1 depending on the observed data aftert -step.
Revised: ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 , theone-step projected error of (cid:98) P t to P t where ˆ v πt +1 is the estimate for v πt +1 depending on the observeddata.

Original: This issubstantially different from the standard NPIV estimation with a directly measuredoutcome. Hence the results, e.g., from Dikkala et al.
Revised: This is different from the standard NPIV estimation with a directly measured outcome so theresult, e.g., from Dikkala et al.

Original: [2020], cannot be directly applied to control thiscomponent.
Revised: [2020], cannot be directly applied to bound this component.

Original: The last two components are related to the (local) measure of ill-posedness.
Revised: The lasttwo components are related to the (local) measure of ill-posedness.

Original: The thirdcomponent τ t is the measure of ill-posedness for characterizing the difﬁculty of estimating q πt by (4)using H ( t ) at the t -th step.
Revised: The third component τ t is themeasure of ill-posedness for characterizing the difﬁculty of estimating q πt by (4) using H ( t ) at thet -th step.

Original: Here { τ t } Tt =1 are similar to those used in the standard NPIV estimationsuch as Chen and Reiss [2011].
Revised: Here { τ t } Tt =1 are similar to those used in the standard NPIV estimation such as Chen and Reiss [2011].

Original: Obviously the effect of { τ t } T t =1 on the upper bound is cumulative.
Revised: Obviously the effect of { τ t } Tt =1 on the upper bound is cumulative.

Original: The last component { Π tt (cid:48) =
Revised: The last component{ Π tt (cid:48) =

Original: 1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 quantiﬁes the propagation effect of estimation errors inprevious steps on the last step of estimating v π 1 , which is multiplicative in terms of C ( t ) t (cid:48) ,t (cid:48) − 1 .
Revised: 1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 quantiﬁes the propagation effect of estimation errors in previous steps on thelast step of estimating v π 1 , which is multiplicative in terms of C ( t ) t (cid:48) ,t (cid:48) − 1 .

Original: Wecall C ( t ) t (cid:48) ,t (cid:48) − 1 the measure of one-step transition ill-posedness from t (cid:48) to t (cid:48) − 1 related to t -step NPIVestimations.
Revised: We call C ( t ) t (cid:48) ,t (cid:48) − 1 the measure of one-step transition ill-posedness from t (cid:48) to t (cid:48) − 1 related to t -step NPIV estimations.

Original: Next we provide detailed bounds for the second and last components.
Revised: Next weprovide detailed bounds for the second and last components.

Original: The discussion ofthe third component can be found in Appendix.
Revised: The discussion of the third componentcan be found in Appendix.

Original: Component 2: one-step projected error .
Revised: Component 2: one-step projected error .

Original: In the following, we show that (cid:107) proj t
Revised: In the following, we show that (cid:107) proj t

Original: ( (cid:98) P t −P t )(ˆ v πt +1 + R t ) (cid:107) 2is bounded by the critical radii of some spaces deﬁned as balls H ( t ) 3 M , G ( t +1)
Revised: ( (cid:98) P t −P t )(ˆ v πt +1 + R t ) (cid:107) 2is bounded by the critical radii of some spaces deﬁned as balls H ( t ) 3 M , G ( t +1)

Original: D in hypothesis spaces
Revised: D in hypothesis spaces

Original: H ( t ) , G ( t +1)respectively and a ball F ( t ) B in testing space F ( t ) , for some ﬁxed constants M,D, B > 0such that functions in H ( t ) 3 M and F ( t ) B have uniformly bounded ranges in [ − 1 , 1] for all 1 ≤ t ≤ T . 
Revised: H ( t ) , G ( t +1) and a ball F ( t ) B in testing space F ( t ) , for some ﬁxed constants M, B > 0 such thatfunctions in H ( t ) 3 M and F ( t ) B have uniformly bounded ranges in [ − 1 , 1] for all 1 ≤ t ≤ T . LetΩ [Equation] ( s t ,

Revised: [Equation]

Original: LetΩ ( t ) = { ( s t , w t , z t , a t , s t +1 , w t +1 )
Revised: t , z t , [Equation] t , s [Equation]

Original:  (cid:55)→ r ( h∗ g ( w t , s t , a t ) − g ( w t +1 , s t +1 ))
Revised: +1 , w [Equation] +1 ) (cid:55)→ r ( h

Original: f ( z t  , s t  , a t  ) : g ∈ G ( t +1) D , f ∈ F ( t ) 3 M , r ∈ [0 , 1] } , and 
Revised: [Equation] g ( w t , a t , s t ) − g ( w t +1 , s t +1 )) f ( z t , a t , s t ) : g ∈ G ( t +1) D , f ∈ F ( t ) 3 M , r ∈ [0 , 1] } , and Ξ ( t ) = { ( s t , w t , z t , a

Original: Ξ ( t ) =
Revised: [Equation]

Original: { ( s t , w t , z t , a t ) (cid:55)→ r
Revised: (cid:55)→ [Equation] [Equation] ]( w

Original: [ h − h ∗ g ]( w t , s t , a t )
Revised: [Equation] , a [Equation]

Original: f L 2 B ( z t , s t , a t ) :h ∈ H ( t ) ,
Revised: , s t [Equation] ( z t , a t , s t ) :

Original: h − h ∗ g ∈ H ( t ) B , g ∈ G ( t +1) D , r ∈ [0 , 1] } , where h ∗ g ∈ H ( t ) is the solution to E [ h ( W t , S t , A t ) − g ( W t +1 , S t +1 ) | Z t , S t , A t ] = 0 , and f L 2 B =arg min f ∈F ( t )
Revised: [Equation] , r ∈ [0 , 1] } , where h ∗ g ∈ H ( t ) is the solution to E [ h ( W t , A t , S t ) − g ( W t +1 , S t +1 ) | Z t , A t , S t ] = 0 , and f L 2 B =arg min f ∈F ( t )

Original: L 2 B (cid:107) f − proj t
Revised: L 2 B (cid:107) f − proj t

Original: ( h − h ∗ g ) (cid:107) 2 for a given L > 0 .
Revised: ( h − h ∗ g ) (cid:107) 2 for a given L > 0 .

Original: An upper bound for (cid:107) proj t ( (cid:98) P t −P t )(ˆ v πt +

Revised: An upper bound for (cid:107) proj t ( (cid:98) P t −P t )(ˆ v πt +

Original: +
Revised: +

Original: R t ) (cid:107) 2 is given in Theorem 6.2.
Revised: R t ) (cid:107) 2 is given in Theorem 6.2.

Original: Theorem 6.2.
Revised: Theorem 6.2.

Original: Suppose that Assumption 1 holds.
Revised: Suppose that Assumption 1 holds.

Original: Let δ ( t ) n = ¯ δ ( t ) n + c 0 (cid:113) log( c 1 T/ζ ) n forsome universalconstants c 0 , c 1 > 0 where ¯ δ ( t ) n is the upper bound of the critical radii of F ( t ) 3 M , Ω ( t ) and Ξ ( t ) .
Revised: Let δ ( t ) n = ¯ δ ( t ) n + c 0 (cid:113) log( c 1 T/ζ ) n for universalconstants c 0 , c 1 > 0 where ¯ δ ( t ) n is the upper bound of the critical radii of F ( t ) 3 M , Ω ( t ) and Ξ ( t ) .

Original: Assume that the approximation error in Assumption 1 (5) can be bounded by η ( t ) n ≤ δ ( t ) n .
Revised: Assume that the approximation error in Assumption 1 (5) can be bounded by η ( t ) n ≤ δ ( t ) n .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: ) D where D = G ( t +1) C G ( T − t +1)
Revised: where D = G ( t +1) C G ( T − t +1)

Original: M H .
Revised: M H .

Original: Furthermore, letting tuning parameters satisfy
Revised: Furthermore, letting tuning parameters satisfy

Original: Mλ (cid:16) ( δ ( t ) n ) 2 and µ ≥
Revised: Mλ (cid:16) ( δ ( t ) n ) 2 and µ ≥

Original: O
Revised: O

Original: ( L 2 + M/B ) , with probability at least 1 − ζ , we have(cid:107) [Equation] t ( (cid:98) [Equation]
Revised: ( L 2 + M/B ) , with probability at least 1 − ζ , we have

Original: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: ) (cid:107)

Original: [Equation]

Original: (cid:46)

Original: [Equation] for all 1 ≤ t [Equation]
Revised: for all 1 ≤ t ≤ T uniformly .

Original: T. Depending on the choices of H ( t ) , G ( t +1) , and F ( t ) , we can obtain different ﬁnite-sample errorbounds of the one-step projected error for each t .
Revised: With common choices of H ( t ) , G ( t +1) , and F ( t ) , we can obtain the ﬁnite-sample error bounds of theone-step projected error for each t .

Original: Below we provide two examples.
Revised: Below we provide two examples.

Original: Corollary 6.1.
Revised: Corollary 6.1.

Original: Let F ( t ) , H ( t ) and G ( t +1) be VC-subgraph classes with VC dimensions V ( F ( t ) ) ,
Revised: Let F ( t ) , H ( t ) and G ( t +1) be VC-subgraph classes with VC dimensions V ( F ( t ) ) ,

Original: V ( H ( t ) ) and V ( G ( t +1) ) respectively.
Revised: V ( H ( t ) ) and V ( G ( t +1) ) respectively.

Original: Then with probability at least 1 − ζ , for all 1 ≤ t ≤ T ,
Revised: Then with probability at least 1 − ζ , for all 1 ≤ t ≤ T uniformly,(cid:107) proj t (

Original: [Equation]
Revised: [Equation]

Original: proj t
Revised: P [Equation] [Equation] πt

Original: [Equation]
Revised: [Equation]

Revised: (cid:107) 2 (cid:46) ( T − t + 1) 2 .(cid:114)

Revised: log( c 1 T/ζ ) max { V ( F ( t ) ) , V ( H ( t ) ) , V ( G ( t +1) ) } n .

Original: The deﬁnition of the VC-subgraph class can be found in, e.g., Wainwright [2019].
Revised: The deﬁnition of the VC-subgraph class can be found in, e.g., Wainwright [2019].

Original: This is a broad class.
Revised: This is a broad class.

Original: For example, if one lets each of F ( t ) , H ( t ) and G ( t +1) be a linear space F
Revised: For example, if one lets each of F ( t ) , H ( t ) and G ( t +1) be a linear space F

Original: = { θ (cid:62) φ ( · ) : θ ∈ R d }with basis functions φ ( · ) , then V ( F ) =
Revised: = { θ (cid:62) φ ( · ) : θ ∈ R d }with basis functions φ ( · ) , then V ( F ) =

Original: d + 1 .{+
+}
Revised: d + {+
+}

Original: Then the upper bound for the one-step projected errorbecomes O (( T − t + 1) 2 .
Revised: so the upper bound for the one-step projected errorbecomes O (( T − t + 1) 2 .

Original: 5 d/ √ n ) .
Revised: 5 d/ √ n ) .

Original: Corollary 6.2.
Revised: Corollary 6.2.

Original: Let H ( t ) , G ( t +1) and F ( t ) be reproducing kernel Hilbert spaces (RKHSs) equippedwith kernels K H ( t ) , K G ( t +1) and K F ( t ) respectively.
Revised: Let H ( t ) , G ( t +1) and F ( t ) be reproducing kernel Hilbert spaces (RKHSs) equippedwith kernels K H ( t ) , K G ( t +1) and K F ( t ) respectively.

Original: For a given positive deﬁnite kernel K , we denoteits decreasing eigenvalue sequence by { λ ↓ j ( K ) } ∞ j =1 .
Revised: For a given positive deﬁnite kernel K , we denoteits decreasing eigenvalue sequence by { λ ↓ j ( K ) } ∞ j =1 .

Original: We consider two scenarios for { λ ↓ j ( K ) } ∞ j =1 .
Revised: We consider two scenarios for { λ ↓ j ( K ) } ∞ j =1 .(1)

Original: (1) Polynomial eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ aj − 2 α H , λ ↓ j ( K G ( t +1) )
Revised: Polynomial eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ aj − 2 α H , λ ↓ j ( K G ( t +1) )

Original: ≤ aj − 2 α G and λ ↓ j ( K F ( t ) ) ≤ aj − 2 α F for constants α H , α G , α F > 1 / 2 and a > 0 , then under the assumptions in Theorem 6.2, with probability at least 1 − ζ , for all 1 ≤ t ≤ T , we 
Revised: ≤ aj − 2 α G and λ ↓ j ( K F ( t ) ) ≤aj − 2 α F for constants α H , α G , α F > 1 / 2 and a > 0 , then under the assumptions in Theorem 6.2,with probability at least 1 − ζ , for all 1 ≤ t ≤ T uniformly, we have [Equation]

Revised: [Equation]

Revised: ) Exponential

Original: have
Revised: eigen

Original: [Equation]
Revised: -decay : If λ ↓ j ( K H ( t ) ≤ a 1 e − a 2 j β H

Revised: , λ

Original: (2) Exponential eigen-decay : If λ ↓ j ( K H ( t ) ) ≤ a 1 e − a 2 j β H , λ ↓ j ( K G ( t +1) ) ≤ a 1 e− a 2 j β G and
Revised: ↓ j ( K G ( t +1) ) ≤ a 1 e

Revised: −

Original: λ ↓ j ( K F ( t ) ) ≤ a 1 e − a 2 j β F , for constants a 1 , a 2 , β H , β G , β F > 0 , then under the assumptions in
Revised: a 2 j β G andλ ↓ j ( K F ( t ) ) ≤ a 1 e − a 2 j β F , for constants a 1 , a 2 , β H , β G , β F > 0 , then under the assumptions in

Original: Theorem 6.2,  with probability at least 1 − ζ , for all 1 ≤ t ≤ T , we 
Revised: Theorem 6.2, then with probability at least 1 − ζ , for all 1 ≤ t ≤ T uniformly, we have

Original: have
Revised: [Equation] n [Equation] n

Original: [Equation]
Revised: [Equation]

Original: The kernels of the two types of eigen-decay above are very common.
Revised: The kernels of the two types of eigen-decay above are very common.

Original: For example, the kernel of theα -order Soblev space with α > 1 / 2 , has a polynomial eigen-decay while the Gaussian kernel hasan exponential eigen-decay, with β = 2 for Lebesgue measure on real line and β = 1 on a compactdomain [Wei et al., 2017].
Revised: For example, the kernel of theα -order Soblev space with α > 1 / 2 , has a polynomial eigen-decay while the Gaussian kernel hasan exponential eigen-decay, with β = 2 for Lebesgue measure on real line and β = 1 on a compactdomain [Wei et al., 2017].

Original: Components 4: measure of one-step transition ill-posedness .
Revised: Components 4: measure of one-step transition ill-posedness .

Original: We ﬁrst provide more insights on{ Π tt (cid:48)
Revised: We ﬁrst provide more insights on{ Π tt (cid:48)

Original: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 before providing an upper bound.
Revised: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 } Tt =1 before providing an upper bound.

Original: We formally deﬁne the local measure ofone-step transition ill-posedness C ( t ) t (cid:48)
Revised: We formally deﬁne the local measure ofone-step transition ill-posedness C ( t ) t (cid:48)

Original: +1 ,t (cid:48) recursively based on C ( t ) t,t − 1 to C ( t ) 2 , 1 asC ( t
Revised: +1 ,t (cid:48) recursively based on C ( t ) t,t − 1 to C ( t ) 2 , 1 as

Original: [Equation]
Revised: [Equation]

Original: t (cid:48)
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: t (cid:48)
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: 2 ∞ t − 1 (cid:89) s

Original:  with C ( t ) t +1 ,t (cid:44) 1 for each t = 1 , . ..
Revised: = t (cid:48) + C ( t ) s +1 ,s , with C ( t ) t +1 ,t (cid:44) 1 for each t = 1 , . .

Original: , T .
Revised: , T .

Original: For C ( t ) t,t − 1 , we can upper bound (cid:107) E  [ˆ v π ( W t , S t ) | Z t , S t ] (cid:107) 2by the projected error (cid:107) proj t ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 multiplied by the ill-posedness τ t .
Revised: For C ( t ) t,t − 1 , we can upper bound (cid:107) E π t [ˆ v π ( W t , S t ) | Z t , S t ] (cid:107) 2by the projected error (cid:107) proj t ( (cid:98) P t − P t )(ˆ v πt +1 + R t ) (cid:107) 2 multiplied by the ill-posedness τ t .

Original: By Theorem6.2, the projected error can be well controlled by δ ( t ) n with high probability, so C ( t ) t,t − 1 can be deﬁnedlocally.
Revised: By Theorem6.2, the projected error can be well controlled by δ ( t ) n with high probability, so C ( t ) t,t − 1 can be deﬁnedlocally.

Original: Therefore, we can provide an upper bound for the denominator sequentially and deﬁne all
Revised: Therefore, we can provide an upper bound for the denominator sequentially and deﬁne all

Original: C ( t ) t (cid:48)
Revised: C ( t ) t (cid:48)

Original: +1 ,t (cid:48) locally, which indicates that all C ( t ) t (cid:48) +1 ,t (cid:48) could be small.
Revised: +1 ,t (cid:48) locally, which indicates that all C ( t ) t (cid:48) +1 ,t (cid:48) could be small.

Original: For example, if we use the observed history as the action-inducing proxy, e.g., Z t =( Z t − 1 ; S t − 1 , W t − 1 , A t − 1 , R t − 1 ) with some pre-observed Z 1 , then σ ( Z 1 × S 1 ) ⊂
Revised: For example, if we use the observed history as the action-inducing proxy, e.g., Z t =( Z t − 1 ; S t − 1 , W t − 1 , A t − 1 , R t − 1 ) with some pre-observed Z 1 , then σ ( Z 1 × S 1 ) ⊂

Original: σ ( Z 2 × S 2 ) ⊂· · · ⊂ σ ( Z T × S T ) is a ﬁltration.
Revised: σ ( Z 2 × S 2 ) ⊂· · · ⊂ σ ( Z T × S T ) is a ﬁltration.

Original: Then (cid:81) tt (cid:48)
Revised: Therefore, (cid:81) tt (cid:48)

Original: =2 C ( t ) t (cid:48) ,t (cid:48) − 1 are expected to be small for t ≥ 2 if thetarget policy is stationary.
Revised: =2 C ( t ) t (cid:48) ,t (cid:48) − 1 = 1 for t ≥ 2 if the target policy isstationary.

Original: While this can enlarge the critical radii δ ( t ) n since the testing space F ( t ) ( Z × A × S ) will have dimension of O ( t ) , this only affects one-step errors.
Revised: While this can enlarge the critical radii δ ( t ) n since the testing space F ( t ) ( Z × A × S )will have dimension of O ( t ) , this only affects one-step errors.

Original: Practitioners canalso learn a low dimensional representation of Z t to reduce the one-step statistical errors.
Revised: Practitioners can also learn a lowdimensional representation of Z t to reduce the one-step statistical errors.

Original: See detaileddiscussion in Appendix B. Motivated by this, it is reasonable to impose Assumption 2 on C ( t ) t (cid:48)
Revised: See detailed discussion in Appendix B. Motivated by this, it is reasonable to impose Assumption 2 on C ( t ) t (cid:48)

Original: +1 ,t (cid:48) .
Revised: +1 ,t (cid:48) .

Original: Assumption 2.
Revised: Assumption 2.

Original: For every t ≥ 2 and 2 ≤ m ≤ t , C ( t ) m,m − 1 ≤ 1 + a t m αt with time-dependent constantsa t> 0 , α t≥ α > 1 .
Revised: For every t ≥ 2 and 2 ≤ m ≤ t , C ( t ) m,m − 1 ≤ 1 + a t m αt with constants a t , α t > 0 .

Original: Then we can provide an upper bound for the propagation effect factor (cid:81) tt (cid:48)
Revised: Then we can provide an upper bound for the propagation effect factor (cid:81) tt (cid:48)

Original: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 .
Revised: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 .

Original: Corollary 6.3.
Revised: Corollary 6.3.

Original: If Assumption 2 holds, then (cid:81) tt (cid:48)
Revised: If Assumption 2 holds, then (cid:81) tt (cid:48)

Original: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 ≤ exp { a t ζ ( α t ) } ,where ζ ( α t ) =
Revised: =1 C ( t ) t (cid:48) ,t (cid:48) − 1 ≤ exp { a t ζ ( α t ) } ,

Original: 343 (cid:80) ∞ n =1 (1 /n )
Revised: where ζ ( α t ) = 338 (cid:80) ∞ n =1 (1 /n )

Original: αt is uniformly bounded for t ≥ 1 .
Revised: α is uniformly bounded for t ≥ 1 .

Original: Main result: error bounds for V -bridge estimation and OPE .
Revised: Main result: error bounds for V -bridge estimation and OPE .

Original: Deﬁne trans-ill =max 1 ≤ t ≤ T exp { a t ζ ( α t ) } and let ill max =¯ τ  1 max 1 ≤ t ≤ T τ t (cid:107) π t /π bt (cid:107)  ∞ .
Revised: Deﬁne trans-ill =max 1 ≤ t ≤ T exp { a t ζ ( α t ) } and let ill max = τ π 1 max 1 ≤ t ≤ T τ t (cid:107) π t /π bt (cid:107) 2 ∞ .

Original: Summarizing all aforementioned results, we have the following main theorem based on the polynomial eigen-decay case in
Revised: Summarizing all aforementioned results, we have the following main theorem based on the polynomial eigen-decay case in

Original: Corollary 6.2.
Revised: Corollary 6.2.

Original: Other cases can be found in Appendix B.
Revised: Other cases can be found in Appendix B.

Original: Theorem 6.3 (Finite-sample error bounds for V -bridges and policy value) .
Revised: Theorem 6.3 (Finite-sample error bounds for V -bridges and policy value) .

Original: Under Assumptions 1and 2, and assumptions in Theorem 6.2 and Corollary 6.2 (1), with probability at least 1 − ζ , we have [Equation] ill max × trans-ill × [Equation]
Revised: Under Assumptions 1and 2, and assumptions in Theorem 6.2 and Corollary 6.2 (1), with probability at least 1 − ζ , we have(cid:107) v π 1 − ˆ v π 1 (cid:107) 2 (cid:46) ill max × trans-ill × T 7 / 2 (cid:112) log( c 1 T/ζ ) n − 1 2+max { 1 /α H , 1 /α G , 1 /α F} , and |V ( π ) − ˆ V ( π ) | (cid:46) ill max × trans-ill × T 7 / 2 (cid:112) log( c

Original: [Equation]

Original: (cid:112) log( c 1 T/ζ ) n − 1 2+max { 1 /α H , 1 /α G , 1 /α F} , and

Original: [Equation]

Original: (cid:46) ill

Original: [Equation]

Original: × trans-ill × T

Original: [Equation]
Revised: [Equation]

Original: / 2 (cid:112) log( c 1 T/ζ ) n −

Original: [Equation]

Original: /α F} . Theorem 6.3 provides the ﬁrst ﬁnite-sample error bound for OPE under confounded and episodic
Revised: Theorem 6.3 provides the ﬁrst ﬁnite-sample error bound for OPE under confounded and episodic

Original: POMDPs in terms of the sample size, length of horizon and two ( local ) measures of ill-posedness.
Revised: POMDPs in terms of the sample size, length of horizon and two ( local ) measures of ill-posedness.

Original: Aswe can see, the derived error bound for V -bridge function achieves the optimal L 2 -convergence ratein the classical non-parametric regression [Stone, 1982].
Revised: Aswe can see, the derived error bound for V -bridges function achieves the optimal L 2 -convergence ratein the classical non-parametric regression [Stone, 1982].

Original: Moreover, our OPE error bound dependson a polynomial order of T , i.e., T 7 / 2 , which is larger than the standard O ( T 3 ) in the OPE withoutunobserved variables.
Revised: Moreover, our OPE error bound dependson a polynomial order of T , i.e., T 7 / 2 , which is larger than the standard O ( T 3 ) in the OPE withoutunobserved variables.

Original: We also remark that when the function class consider in (7) grows with thesample size n , ill max will also increase and therefore the convergence rates in Theorem 6.3 couldbe much slower.
Revised: We also remark that when the function class consider in (7) grows with thesample size n , ill max will also increase and therefore the convergence rates in Theorem 6.3 couldbe much slower.

Original: Therefore it will be interesting to study the minimax lower bounds for our problem.Discussion
Revised: Therefore it will be interesting to study the minimax lower bounds for our problem.

Original: In this paper, we propose a non-parametric identiﬁcation and estimation method for the off-policyevaluation in episodic

Original: confounded POMDPs with continuous states, relying on time-dependent proxyvariables.

Original: We develop a ﬁtted- Q -evaluation-type algorithm for estimating the V -bridge functionssequentially and for OPE based on the estimated V -bridges.

Original: More importantly, we establish a ﬁrstﬁnite-sample error bound for the evaluation of policy value and the estimation of V -bridges, whichachieves a polynomial order in terms of the sample size and the length of horizon.

Original: Our work can serve as a foundation for developing new policy optimization algorithms in theconfounded POMDP.

Original: One intuitive approach is to mimic the ﬁtted- Q -iteration algorithm.

Original: Explicitly,at each step t , after obtaining the ﬁtted Q -bridges ˆ q πt by (7), we may ﬁt a non-parametric regressionto obtain ˜ q πt ( s, a ) =

Original: E [ˆ q πt ( W t , S t , A t ) | S t = s, A t = a ] with batch data { W t,i , S t,i , A t,i } ni =1 andlearn a greedy policy deﬁned by π t ( s ) = argmax a ∈A ˜ q πt ( s, a ) .

Original: We can then estimate the next Qbridge by computing ˆ q πt − 1 = (cid:98) P t − 1 (ˆ q πt ( W t , S t , π t ( S t ))

Original: + R t − 1 ) and repeat this process from t

Original: =

Original: Tto 1 .

Original: We will leave it for future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes] The paper’s contributions and scope are summarized in the abstract and then emphasized in Sections 1 and 2.
Revised: [Yes] The paper’s contributions and scope are summarized inthe abstract and then emphasized in Sections 1 and 2.(b)

Original: (b) Did you describe the limitations of your work?
Revised: Did you describe the limitations of your work?

Original: [Yes] We did not provide results of minimax lower bound for our estimation problem.
Revised: [Yes] We did not provide results ofminimax lower bound for our estimation problem.(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: The
Revised: The

Original: proposed work is to characterize the ﬁnite-sample error bound for OPE in confounded
Revised: proposed work is to characterize the ﬁnite-sample error bound for OPE in confounded

Original: and episodic POMDP under non-parametric models.
Revised: and episodic POMDP under non-parametric models.

Original: The emphasis is on theories,
Revised: The emphasis is on theories,

Original: without limitations on speciﬁc applications.
Revised: without limitations on speciﬁc applications.

Original: As a result, it’s not applicable to talk about
Revised: As a result, it’s not applicable to talk about

Original: negative societal impacts.
Revised: negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: Main
Revised: Main

Original: assumptions are stated in Sections 4 and 6.
Revised: assumptions are stated in Sections 4 and 6.

Original: Some regularity conditions are provided in
Revised: Some regularity conditions are provided in

Original: Appendix A.
Revised: Appendix A.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs for theoret-
Revised: [Yes] All proofs for theoret-

Original: ical results are provided in Appendices C and D.
Revised: ical results are provided in Appendices C and D.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We attached

Original: the code and instructions in the supplemental materials.

Original: [Table]
Revised: [N/A]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] Provided in Appendix E
Revised: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] [Table]
Revised: [N/A](d)

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?[Yes]
Revised: Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Table]
Revised: [N/A]4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: [Table]
Revised: (a)

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [N/A]

Original: [Table]

Original: (b) Did you mention the license of the assets?
Revised: ) Did you mention the license of the assets?

Original: [Yes] [Table]
Revised: [N/A] [N/A]

Original: (c) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: ) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (d) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: ) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A]
Revised: Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Revised: applicable

Original: (b) Did you describe any potential participant risks,with links to Institutional Review 
Revised: N participant risks, to Institutional Review applicable

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table]
Revised: you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation

Original: [N/A]
Revised: [N/A]

Original: [Table]