Original: W HEN
Revised: W HEN

Original: IS
Revised: IS

Original: O FFLINE
Revised: O FFLINE

Original: H
Revised: H

Original: YPERPARAMETER S ELECTION F EASIBLE FOR R EINFORCEMENT L EARNING ?
Revised: YPERPARAMETER S ELECTION F EASIBLE FOR R EINFORCEMENT L EARNING ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Hyperparameter selection is a critical procedure before deploying reinforcement learning algorithms in real-world applications.
Revised: Hyperparameter selection is a critical procedure before deploying reinforcement learning algorithms in real-world applications.

Original: However, hyperparameter selection prior to deployment requires selecting policies offline without online execution, which is a significant challenge known as offline policy selection.
Revised: However, hyperparameter selection prior to deployment requires selecting policies offline without online execution, which is a significant challenge known as offline policy selection.

Original: As yet, there is little understanding about the fundamental limitations of the offline policy selection problem.
Revised: As yet, there is little understanding about the fundamental limitations of the offline policy selection problem.

Original: To contribute to our understanding of this problem, in this paper, we investigate when sample efficient offline policy selection is possible
Revised: To contribute to our understanding of this problem, in this paper, we investigate when sample efficient offline policy selection is possible

Original: As off-policy policy evaluation (OPE) is a natural approach for policy selection, the sample complexity of offline policy selection is therefore upper-bounded by the number of samples needed to perform OPE.
Revised: As off-policy policy evaluation (OPE) is a natural approach for policy selection, the sample complexity of offline policy selection is therefore upper-bounded by the number of samples needed to perform OPE.

Original: In addition, we prove that the sample complexity of offline policy selection is also lower-bounded by the sample complexity of OPE.
Revised: In addition, we prove that the sample complexity of offline policy selection is also lower-bounded by the sample complexity of OPE.

Original: These results imply not only that offline policy selection is effective when OPE is effective, but also that sample efficient policy selection is not possible without additional assumptions that make OPE effective.
Revised: These results imply not only that offline policy selection is effective when OPE is effective, but also that sample efficient policy selection is not possible without additional assumptions that make OPE effective.

Original: Moreover, we theoretically study the conditions under which offline policy selection using Fitted Q evaluation (FQE) and the Bellman error is sample efficient.
Revised: Moreover, we theoretically study the conditions under which offline policy selection using Fitted Q evaluation (FQE) and the Bellman error is sample efficient.

Original: We conclude with an empirical study comparing FQE and Bellman errors for offline policy selection.
Revised: We conclude with an empirical study comparing FQE and Bellman errors for offline policy selection.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning a policy from offline datasets, known as offline reinforcement learning (RL) , has gained in popularity due to its practicality.
Revised: Learning a policy from offline datasets, known as offline reinforcement learning (RL) , has gained in popularity due to its practicality.

Original: Offline RL is useful for many real-world applications, as learning from online interaction may be expensive or dangerous (Levine et al., 2020).
Revised: Offline RL is useful for many real-world applications, as learning from online interaction may be expensive or dangerous (Levine et al., 2020).

Original: For example, training a stock-trading RL agent online may incur large losses before learning to perform well, and training self-driving cars in the real world may be too dangerous.
Revised: For example, training a stock-trading RL agent online may incur large losses before learning to perform well, and training self-driving cars in the real world may be too dangerous.

Original: Ideally we want to learn a good policy offline, before deployment.
Revised: Ideally we want to learn a good policy offline, before deployment.

Original: In practice, offline RL algorithms often have hyperparameters which require careful tuning (Wu et al., 2019) and whether or not we can select an effective hyperparameter is perhaps the most important component when comparing algorithms (Kumar et al., 2022).
Revised: In practice, offline RL algorithms often have hyperparameters which require careful tuning (Wu et al., 2019) and whether or not we can select an effective hyperparameter is perhaps the most important component when comparing algorithms (Kumar et al., 2022).

Original: A hyperparameter is a parameter which is specified by the algorithm designer, rather than being learned from data.
Revised: A hyperparameter is a parameter which is specified by the algorithm designer, rather than being learned from data.

Original: Some examples of hyperparameters for RL are the learning rate, function approximation class or regularization coefficients.
Revised: Some examples of hyperparameters for RL are the learning rate, function approximation class or regularization coefficients.

Original: When an RL algorithm is run with a specific hyperparameter configuration, it outputs a learned policy and/or a value estimate.
Revised: When an RL algorithm is run with a specific hyperparameter configuration, it outputs a learned policy and/or a value estimate.

Original: Each hyperparameter configuration produces a candidate policy and our goal is to select the best-performing policy.
Revised: Each hyperparameter configuration produces a candidate policy and our goal is to select the best-performing policy.

Original: This problem of finding the best-performing policy from a set of candidate policies is called policy selection .
Revised: This problem of finding the best-performing policy from a set of candidate policies is called policy selection .

Original: The typical way of finding the best-performing policy in RL is to perform several rollouts for each candidate policy in the environment, compute the average return for each policy, and then select the policy that produced the highest average return.
Revised: The typical way of finding the best-performing policy in RL is to perform several rollouts for each candidate policy in the environment, compute the average return for each policy, and then select the policy that produced the highest average return.

Original: This approach is often used in many industrial problems where A/B testing is available.
Revised: This approach is often used in many industrial problems where A/B testing is available.

Original: However, this approach assumes that these rollouts can be performed, which necessarily requires access to the environment or simulator, a luxury not afforded to offline RL algorithms.
Revised: However, this approach assumes that these rollouts can be performed, which necessarily requires access to the environment or simulator, a luxury not afforded to offline RL algorithms.

Original: As such, in offline RL, Monte Carlo rollouts cannot be performed to select candidate policies, and mechanisms to select policies with offline data are needed.
Revised: As such, in offline RL, Monte Carlo rollouts cannot be performed to select candidate policies, and mechanisms to select policies with offline data are needed.

Original: In the offline setting, a common approach to policy selection is to perform off-policy policy evaluation (OPE) to estimate the value of candidate policies from a fixed dataset and then select the policy with the highest estimated value.
Revised: In the offline setting, a common approach to policy selection is to perform off-policy policy evaluation (OPE) to estimate the value of candidate policies from a fixed dataset and then select the policy with the highest estimated value.

Original: Typical OPE algorithms include direct methods such as fitted Q evaluation (FQE) estimator (Le et al., 2019), importance sampling (IS) estimator (Sutton & Barto, 2018), doubly robust estimator (Jiang & Li, 2016), model-based estimator (Mannor et al., 2007), and marginalized importance sampling estimator (Xie et al., 2019).
Revised: Typical OPE algorithms include direct methods such as fitted Q evaluation (FQE) estimator (Le et al., 2019), importance sampling (IS) estimator (Sutton & Barto, 2018), doubly robust estimator (Jiang & Li, 2016), model-based estimator (Mannor et al., 2007), and marginalized importance sampling estimator (Xie et al., 2019).

Original: Empirically, Tang & Wiens (2021);
Revised: Empirically, Tang & Wiens (2021);

Original: Paine et al.
Revised: Paine et al.

Original: (2020) provide experimental results on offline policy selection using OPE.
Revised: (2020) provide experimental results on offline policy selection using OPE.

Original: Similarly,
Revised: Similarly,

Original: Doroudi et al.
Revised: Doroudi et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2022) use OPE estimators for policy selection.
Revised: (2022) use OPE estimators for policy selection.

Original: Offline policy selection has been mainly associated with OPE, since these two problem are closely related.
Revised: Offline policy selection has been mainly associated with OPE, since these two problem are closely related.

Original: It is known that OPE is a hard problem that requires an exponential number of samples to evaluate any given policy in the worst case (Wang et al., 2020), so OPE can be unreliable for policy selection.
Revised: It is known that OPE is a hard problem that requires an exponential number of samples to evaluate any given policy in the worst case (Wang et al., 2020), so OPE can be unreliable for policy selection.

Original: As a result, a follow-up question is: do the hardness results from OPE also hold for offline policy selection?
Revised: As a result, a follow-up question is: do the hardness results from OPE also hold for offline policy selection?

Original: If the answer is yes, then we would need to consider additional assumptions to enable sample efficient offline policy selection.
Revised: If the answer is yes, then we would need to consider additional assumptions to enable sample efficient offline policy selection.

Original: Moreover, offline policy selection should be easier than OPE intuitively, since estimating each policy accurately might not be necessary for policy selection.
Revised: Moreover, offline policy selection should be easier than OPE intuitively, since estimating each policy accurately might not be necessary for policy selection.

Original: There is mixed evidence that alternatives to
Revised: There is mixed evidence that alternatives to

Original: OPE might be effective.
Revised: OPE might be effective.

Original: Tang & Wiens (2021) show empirically that using TD errors perform poorly because they provide overestimates; they conclude OPE is necessary.
Revised: Tang & Wiens (2021) show empirically that using TD errors perform poorly because they provide overestimates; they conclude OPE is necessary.

Original: On the other hand, Zhang &
Revised: On the other hand, Zhang &

Original: Jiang (2021) perform policy selection without OPE, by selecting the value function that is closest to the optimal value function.
Revised: Jiang (2021) perform policy selection without OPE, by selecting the value function that is closest to the optimal value function.

Original: However, their method relies on having the optimal value function in the set of candidate value functions.
Revised: However, their method relies on having the optimal value function in the set of candidate value functions.

Original: It remains an open question about when, or even if, alternative approaches can outperform OPE for offline policy selection.
Revised: It remains an open question about when, or even if, alternative approaches can outperform OPE for offline policy selection.

Original: Unfortunately, there is little understanding about the offline policy selection problem to answer the aforementioned questions.
Revised: Unfortunately, there is little understanding about the offline policy selection problem to answer the aforementioned questions.

Original: Therefore, to provide a better understanding, we aim to investigate the question: When can we perform offline policy selection efficiently (with a polynomial sample complexity) for RL?
Revised: Therefore, to provide a better understanding, we aim to investigate the question: When can we perform offline policy selection efficiently (with a polynomial sample complexity) for RL?

Original: To this end, our contributions are as follows:
Revised: To this end, our contributions are as follows:

Original: • We show that the sample complexity of the offline policy selection problem is lower-bounded by the samples need to perform OPE.
Revised: • We show that the sample complexity of the offline policy selection problem is lower-bounded by the samples need to perform OPE.

Original: This implies no policy selection approach can be more sample efficient than OPE in the worst case.
Revised: This implies no policy selection approach can be more sample efficient than OPE in the worst case.

Original: On the other hand, OPE can be used for offline policy selection, so the sample complexity of policy selection is upper-bounded by the samples required for OPE.
Revised: On the other hand, OPE can be used for offline policy selection, so the sample complexity of policy selection is upper-bounded by the samples required for OPE.

Original: In particular, we show that a selection algorithm that simply chooses the policy with the highest IS estimate achieves a nearly minimax optimal sample complexity, which is exponential in the horizon.
Revised: In particular, we show that a selection algorithm that simply chooses the policy with the highest IS estimate achieves a nearly minimax optimal sample complexity, which is exponential in the horizon.

Original: • To circumvent exponential sample complexity, we need to make additional assumptions.
Revised: • To circumvent exponential sample complexity, we need to make additional assumptions.

Original: We identify when FQE, a commonly used OPE method, is efficient for offline policy selection.
Revised: We identify when FQE, a commonly used OPE method, is efficient for offline policy selection.

Original: Specifically we show that FQE is efficient for policy selection when the candidate policies are well covered by the offline dataset.
Revised: Specifically we discuss that FQE is efficient for policy selection when the candidate policies are well covered by the offline dataset.

Original: This theoretical result supports several empirical findings.
Revised: This theoretical result supports several empirical findings.

Original: • We explore the use of Bellman errors for policy selection and provide a theoretical argument and experimental evidence for the improved sample efficiency of using Bellman errors compared to FQE, under stronger assumptions such as deterministic dynamics and good data coverage.
Revised: • We explore the use of Bellman errors for policy selection and provide a theoretical argument and experimental evidence for the improved sample efficiency of using Bellman errors compared to FQE, under stronger assumptions such as deterministic dynamics and good data coverage.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In reinforcement learning (RL), the agent-environment interaction can be formalized as a finite horizon finite Markov decision process (MDP) M
Revised: In reinforcement learning (RL), the agent-environment interaction can be formalized as a finite horizon finite Markov decision process (MDP) M

Original: =
Revised: =

Original: ( S , A , H, ν, Q ) .
Revised: ( S , A , H, ν, Q ) .

Original: S is a set of states with size
Revised: S is a set of states with size

Original: S = |S| , and A is a set of actions with size A = |A| ,
Revised: S = |S| , and A is a set of actions with size A = |A| ,

Original: H ∈ Z + is the horizon, and ν ∈ ∆( S ) is the initial state distribution where ∆( S ) is the set of probability distributions over S .
Revised: H ∈ Z + is the horizon, and ν ∈ ∆( S ) is the initial state distribution where ∆( S ) is the set of probability distributions over S .

Original: Without loss of generality, we assume that there is only one initial state s 0 .
Revised: Without loss of generality, we assume that there is only one initial state s 0 .

Original: The reward R and next state S ′ are sampled from Q , that is, ( R, S ′ ) ∼ Q ( ·| s, a ) .
Revised: The reward R and next state S ′ are sampled from Q , that is, ( R, S ′ ) ∼ Q ( ·| s, a ) .

Revised: We assume the reward is bounded in [0 , r max ] almost surely so the total return of each episode is bounded in [0 , V max ] almost surely.

Original: The stochastic kernel Q induces a transition probability P : S ×
Revised: The stochastic kernel Q induces a transition probability P : S ×

Original: A → ∆( S ) , and a mean reward function r ( s, a ) which gives the mean reward when taking action a in state s .
Revised: A → ∆( S ) , and a mean reward function r ( s, a ) which gives the mean reward when taking action a in state s .

Original: A non-stationary policy is a sequence of memoryless policies ( π 0 , . . .
Revised: A non-stationary policy is a sequence of memoryless policies ( π 0 , . . .

Original: , π H − 1 ) where π h : S → ∆( A ) .
Revised: , π H − 1 ) where π h : S → ∆( A ) .

Original: We assume that the set of states reachable at time step h , S h ⊂ S , are disjoint, without loss of generality, because we can always define a new state space S ′ =
Revised: We assume that the set of states reachable at time step h , S h ⊂ S , are disjoint, without loss of generality, because we can always define a new state space S ′ =

Original: S × { 0 , 1 , 2 , . .
Revised: S × { 0 , 1 , 2 , . .

Original: , H − 1 } .
Revised: , H − 1 } .

Original: Then, it is sufficient to consider stationary policies π : S → ∆( A ) .
Revised: Then, it is sufficient to consider stationary policies π : S → ∆( A ) .

Original: Given a policy π , for any h ∈ [ H ] , and ( s, a ) ∈ S × A , we define the value function and the actionvalue function as v πh ( s ) :
Revised: Given a policy π , for any h ∈ [ H ] , and ( s, a ) ∈ S × A , we define the value function and the actionvalue function as v πh ( s ) :

Original: =
Revised: =

Original: E π [ (cid:80) H − 1 t = h r ( S t , A t ) | S h
Revised: E π [ (cid:80) H − 1 t = h r ( S t , A t ) | S h

Original: = s ] and q πh ( s, a ) :
Revised: = s ] and q πh ( s, a ) :

Original: =
Revised: =

Original: E π [ (cid:80) H − 1 t = h r ( S t , A t ) | S h
Revised: E π [ (cid:80) H − 1 t = h r ( S t , A t ) | S h

Original: = s,
Revised: = s,

Original: A h
Revised: A h

Original: = a ] , respectively.
Revised: = a ] , respectively.

Original: The expectation is with respect to P π , which is the probability measure on the random element ( S 0 , A 0 , R 0 , . . .
Revised: The expectation is with respect to P π , which is the probability measure on the random element ( S 0 , A 0 , R 0 , . . .

Original: , R H − 1 ) induced by the policy π .
Revised: , R H − 1 ) induced by the policy π .

Original: The optimal value function is defined by v ∗ h ( s ) :
Revised: The optimal value function is defined by v ∗ h ( s ) :

Original: = sup π v πh ( s ) , and the Bellman operator is defined by
Revised: = sup π v πh ( s ) , and the Bellman operator is defined by

Original: [Equation]
Revised: [Equation]

Original: The Bellman evaluation operator is defined by
Revised: The Bellman evaluation operator is defined by

Original: [Equation]
Revised: [Equation]

Original: We use J ( π ) to denote the value of the policy π , that is,  J ( π ) = v π ( s 0 ) .
Revised: We use J ( π ) to denote the value of the policy π , that is, the expected return from the initial state J ( π ) = v π ( s 0 ) .

Original: A policy π is optimal if J ( π ) = v ∗ ( s 0 ) .
Revised: A policy π is optimal if J ( π ) = v ∗ ( s 0 ) .

Original: In the offline RL setting, we are given a fixed set of transitions D with samples drawn from a data distribution µ .
Revised: In the offline RL setting, we are given a fixed set of transitions D with samples drawn from a data distribution µ .

Original: In this paper, we consider the setting where the data is collected by a behavior policy π b since the data collection scheme is more practical (Xiao et al., 2022) and is used to collect data for benchmark datasets (Fu et al., 2020).
Revised: In this paper, we consider the setting where the data is collected by a behavior policy π b since the data collection scheme is more practical (Xiao et al., 2022) and is used to collect data for benchmark datasets (Fu et al., 2020).

Original: We use d πh to denote the data distribution at the horizon h by following the policy π , that is, d πh ( s, a ) :
Revised: We use d πh to denote the data distribution at the horizon h by following the policy π , that is, d πh ( s, a ) :

Original: =
Revised: =

Original: P π ( S h
Revised: P π ( S h

Original: = s, A h
Revised: = s, A h

Original: = a ) , and µ h ( s, a ) :=
Revised: = a ) , and µ h ( s, a ) :=

Original: P π b ( S h
Revised: P π b ( S h

Original: =
Revised: =

Original: s, A h = a ) .
Revised: s, A h = a ) .

Original: Notation.
Revised: Notation.

Original: We use [ H ] to denote the set [ H ] := { 0 , 1 , . . .
Revised: We use [ H ] to denote the set [ H ] := { 0 , 1 , . . .

Original: , H − 1 } .
Revised: , H − 1 } .

Original: Given a value function q and a state-action distribution µ , the norm is defined as ∥ q ∥ p,µ :=
Revised: Given a value function q and a state-action distribution µ , the norm is defined as ∥ q ∥ p,µ :=

Original: ( (cid:80) s,a µ ( s, a )
Revised: ( (cid:80) s,a µ ( s, a )

Original: |
Revised: |

Original: q
Revised: q

Original: ( s, a ) | p ) 1 /p and the max norm is defined as ∥ q ∥ ∞ :
Revised: ( s, a ) | p ) 1 /p and the max norm is defined as ∥ q ∥ ∞ :

Original: =
Revised: =

Original: max s,a | q ( s, a ) | .
Revised: max s,a | q ( s, a ) | .

Original:  S AMPLE C OMPLEXITY OF O FFLINE P OLICY S ELECTION
Revised:  S AMPLE C OMPLEXITY OF O FFLINE P OLICY S ELECTION

Original: We consider the offline policy selection (OPS) problem and offline policy evaluation (OPE) problem.
Revised: We consider the offline policy selection (OPS) problem and offline policy evaluation (OPE) problem.

Original: We follow a similar notation and formulation used in Xiao et al.
Revised: We follow a similar notation and formulation used in Xiao et al.

Original: (2022) to formally describe these problem settings.
Revised: (2022) to formally describe these problem settings.

Original: The OPS problem for a fixed number of episodes n is given by the tuple ( S , A , H, ν, n, I ) .
Revised: The OPS problem for a fixed number of episodes n is given by the tuple ( S , A , H, ν, n, I ) .

Original: I is a set of instances of the form ( M, d b , Π) where M ∈ M ( S , A , H, ν ) specifies an MDP, d b is a distribution over a trajectory ( S 0 , A 0 , R 0 , . . .
Revised: I is a set of instances of the form ( M, d b , Π) where M ∈ M ( S , A , H, ν ) specifies an MDP, d b is a distribution over a trajectory ( S 0 , A 0 , R 0 , . . .

Original: , R H − 1 ) by running the behavior policy π b on M , and Π is a finite set of candidate policies.
Revised: , R H − 1 ) by running the behavior policy π b on M , and Π is a finite set of candidate policies.

Original: We consider the setting where Π has a small size and does not depend on S , A or H .
Revised: We consider the setting where Π has a small size and does not depend on S , A or H .

Original: An OPS algorithm takes as input a batch of data D , which contains n trajectories, and a set of candidate policies Π , and outputs a policy π ∈ Π .
Revised: An OPS algorithm takes as input a batch of data D , which contains n trajectories, and a set of candidate policies Π , and outputs a policy π ∈ Π .

Original: We say an OPS algorithm L is ( ε, δ ) -sound on instance ( M, d b , Π) if
Revised: We say an OPS algorithm L is ( ε, δ ) -sound on instance ( M, d b , Π) if

Original: [Equation]
Revised: [Equation]

Original: where π † is the best policy in Π .
Revised: where π † is the best policy in Π .

Original: We say an OPS algorithm L is ( ε, δ ) -sound on the problem ( S , A , H, ν, n, I ) if it is sound on any instance ( M, d b , Π) ∈ I .
Revised: We say an OPS algorithm L is ( ε, δ ) -sound on the problem ( S , A , H, ν, n, I ) if it is sound on any instance ( M, d b , Π) ∈ I .

Original: Given a pair ( ε, δ ) , the sample complexity of OPS is the smallest integer n such that there exists a behavior policy π b and an OPS algorithm L such that L is ( ε, δ ) -sound on the OPS problem ( S , A , H, ν, n, I ( π b )) where I ( π b ) denotes the set of instances with data distribution d b .
Revised: Given a pair ( ε, δ ) , the sample complexity of OPS is the smallest integer n such that there exists a behavior policy π b and an OPS algorithm L such that L is ( ε, δ ) -sound on the OPS problem ( S , A , H, ν, n, I ( π b )) where I ( π b ) denotes the set of instances with data distribution d b .

Original: That is, if the statistical complexity is lower-bounded by n , then, for any behavior policy π b , there exists an
Revised: That is, if the statistical complexity is lower-bounded by n , then, for any behavior policy π b , there exists an

Original: MDP M and a set of candidate policies Π such that any ( ε, δ ) -sound
Revised: MDP M and a set of candidate policies Π such that any ( ε, δ ) -sound

Original: OPS algorithm on ( M, d b , Π) requires at least n samples.
Revised: OPS algorithm on ( M, d b , Π) requires at least n samples.

Original: Similarly, the OPE problem for a fixed number of episodes n is given by ( S , A , H, ν, n, I ) .
Revised: Similarly, the OPE problem for a fixed number of episodes n is given by ( S , A , H, ν, n, I ) .

Original: I is a set of instances of the form ( M, d b , π ) where M and d b are defined as above, and π is a target policy.
Revised: I is a set of instances of the form ( M, d b , π ) where M and d b are defined as above, and π is a target policy.

Original: An OPE algorithm takes as input a batch of data D and a target policy π , and outputs an estimate of the policy value.
Revised: An OPE algorithm takes as input a batch of data D and a target policy π , and outputs an estimate of the policy value.

Original: We say an OPE algorithm L is ( ε, δ ) -sound on instance ( M, d b , π ) if
Revised: We say an OPE algorithm L is ( ε, δ ) -sound on instance ( M, d b , π ) if

Original: [Equation]
Revised: [Equation]

Original: We say an OPE algorithm L is ( ε, δ ) -sound on the problem ( S , A , H, ν, n, I ) if it is sound on any instance ( M, d b , π ) ∈ I .
Revised: We say an OPE algorithm L is ( ε, δ ) -sound on the problem ( S , A , H, ν, n, I ) if it is sound on any instance ( M, d b , π ) ∈ I .

Original: Note that ε should be less than V max / 2 otherwise the bound is trivial.
Revised: Note that ε should be less than V max / 2 otherwise the bound is trivial.

Original: 3. OPE ALGORITHM AS SUBROUTINE FOR OFFLINE POLICY SELECTION
Revised: 3. OPE ALGORITHM AS SUBROUTINE FOR OFFLINE POLICY SELECTION

Original: It is obvious that a sound OPE algorithm can be used for OPS, so the sample complexity of OPS is upper-bounded by the sample complexity of OPE up to a logarithmic factor.
Revised: It is obvious that a sound OPE algorithm can be used for OPS, so the sample complexity of OPS is upper-bounded by the sample complexity of OPE up to a logarithmic factor.

Original: We state this formally in the theorem below.
Revised: We state this formally in the theorem below.

Original: The proof can be found in Appendix A.1.
Revised: The proof can be found in Appendix A.1.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Upper bound on sample complexity of OPS)
Revised: (Upper bound on sample complexity of OPS)

Original: Given an MDP M , a data distribution d b , and a set of policies Π , suppose that, for any pair ( ε, δ ) , there exists an ( ε, δ ) -sound
Revised: Given an MDP M , a data distribution d b , and a set of policies Π , suppose that, for any pair ( ε, δ ) , there exists an ( ε, δ ) -sound

Original: OPE algorithm L on any OPE instance I ∈ { ( M, d b , π ) : π ∈ Π } with a sample size at most
Revised: OPE algorithm L on any OPE instance I ∈ { ( M, d b , π ) : π ∈ Π } with a sample size at most

Original: O ( N OPE ( S, A, H, ε, 1 /δ )) .
Revised: O ( N OPE ( S, A, H, ε, 1 /δ )) .

Original: Then there exists an ( ε, δ ) -sound
Revised: Then there exists an ( ε, δ ) -sound

Original: OPS algorithm for the OPS problem instance ( M, d b , Π) which requires at most O ( N OPE ( S, A, H, ε, | Π | /δ )) episodes.
Revised: OPS algorithm for the OPS problem instance ( M, d b , Π) which requires at most O ( N OPE ( S, A, H, ε, | Π | /δ )) episodes.

Original: In terms of the sample complexity, we have an extra (cid:112) log ( | Π | )
Revised: In terms of the sample complexity, we have an extra (cid:112) log ( | Π | )

Original: /n term for OPS due to the union bound.
Revised: /n term for OPS due to the union bound.

Original: For hyperparameter selection in practice, the size of the candidate set is often much smaller than n , so this extra term is negligible.
Revised: For hyperparameter selection in practice, the size of the candidate set is often much smaller than n , so this extra term is negligible.

Original: However, if the set is too large, complexity regularization (Bartlett et al., 2002) may need to be considered.
Revised: However, if the set is too large, complexity regularization (Bartlett et al., 2002) may need to be considered.

Original: 3. O FFLINE P OLICY SELECTION IS NOT EASIER THAN OPE
Revised: 3. O FFLINE P OLICY SELECTION IS NOT EASIER THAN OPE

Original: We have shown that OPS is sample efficient when OPE is sample efficient.
Revised: We have shown that OPS is sample efficient when OPE is sample efficient.

Original: However, it remains unclear whether OPS can be sample efficient when OPE is not.
Revised: However, it remains unclear whether OPS can be sample efficient when OPE is not.

Original: In the following theorem, we lower bound the sample complexity of OPS by the sample complexity of OPE.
Revised: In the following theorem, we lower bound the sample complexity of OPS by the sample complexity of OPE.

Original: As a result, both OPS and
Revised: As a result, both OPS and

Original: OPE suffer from the same hardness result, and we cannot expect OPS to be sample efficient under conditions where OPE is not sample efficient.
Revised: OPE suffer from the same hardness result, and we cannot expect OPS to be sample efficient under conditions where OPE is not sample efficient.

Original: Theorem 2 (Lower bound on sample complexity of OPS) .
Revised: Theorem 2 (Lower bound on sample complexity of OPS) .

Original: Suppose for any data distribution d b and any pair ( ε, δ ) with ε ∈ (0 , V max / 2) and δ ∈ (0 , 1) , there exists an MDP M and a policy π such that any ( ε, δ ) -sound
Revised: Suppose for any data distribution d b and any pair ( ε, δ ) with ε ∈ (0 , V max / 2) and δ ∈ (0 , 1) , there exists an MDP M and a policy π such that any ( ε, δ ) -sound

Original: OPE algorithm requires at least Ω( N OPE ( S, A, H, ε, 1 /δ )) episodes.
Revised: OPE algorithm requires at least Ω( N OPE ( S, A, H, ε, 1 /δ )) episodes.

Original: Then there exists an MDP M ′ with S ′ =
Revised: Then there exists an MDP M ′ with S ′ =

Original: S + 2 , H ′ =
Revised: S + 2 , H ′ =

Original: H + 1 , and a set of candidate policies such that for any pair ( ε, δ ) with ε ∈ (0 , V max / 3) and δ ∈ (0 , 1 /m ) where m :=
Revised: H + 1 , and a set of candidate policies such that for any pair ( ε, δ ) with ε ∈ (0 , V max / 3) and δ ∈ (0 , 1 /m ) where m :=

Original: ⌈ log( V max /ε )
Revised: ⌈ log( V max /ε )

Original: ⌉ ≥ 1 , any ( ε, δ ) -sound
Revised: ⌉ ≥ 1 , any ( ε, δ ) -sound

Original: OPS algorithm also requires at least Ω( N OPE ( S, A, H, ε, 1 / ( mδ ))))
Revised: OPS algorithm also requires at least Ω( N OPE ( S, A, H, ε, 1 / ( mδ ))))

Original: episodes.
Revised: episodes.

Original: The proof sketch is to construct an OPE algorithm that queries OPS as a subroutine.
Revised: The proof sketch is to construct an OPE algorithm that queries OPS as a subroutine.

Original: As a result, the sample complexity of OPS is lower bounded by the sample complexity of OPE.
Revised: As a result, the sample complexity of OPS is lower bounded by the sample complexity of OPE.

Original: We use the reduction first mentioned in Wang et al.
Revised: We use the reduction first mentioned in Wang et al.

Original: (2020), and present a proof in Appendix A.2.
Revised: (2020), and present a proof in Appendix A.2.

Original: There exist several hardness results for OPE in tabular settings and with linear function approximation (Yin & Wang, 2021; Wang et al., 2020).
Revised: There exist several hardness results for OPE in tabular settings and with linear function approximation (Yin & Wang, 2021; Wang et al., 2020).

Original: Theorem 2 implies that the same hardness results hold for OPS.
Revised: Theorem 2 implies that the same hardness results hold for OPS.

Original: We should not expect to have a sound OPS algorithm without additional assumptions.
Revised: We should not expect to have a sound OPS algorithm without additional assumptions.

Original: Theorem 2, however, does not imply that OPS and OPE are always equally hard.
Revised: Theorem 2, however, does not imply that OPS and OPE are always equally hard.

Original: There are instances where OPS is easy but OPE is not.
Revised: There are instances where OPS is easy but OPE is not.

Original: For example, when all policies in the candidate set all have the same value, any random policy selection is sound.
Revised: For example, when all policies in the candidate set all have the same value, any random policy selection is sound.

Original: However, OPE can still be difficult in such cases.
Revised: However, OPE can still be difficult in such cases.

Original: 3. I MPORTANCE S AMPLING ACHIEVES NEARLY MINIMAX OPTIMAL SAMPLE COMPLEXITY
Revised: 3. I MPORTANCE S AMPLING ACHIEVES NEARLY MINIMAX OPTIMAL SAMPLE COMPLEXITY

Original: We present an exponential lower bound on the sample complexity of OPE in Theorem 5, which uses the same construction from Xiao et al.
Revised: We present an exponential lower bound on the sample complexity of OPE in Theorem 5 in the appendix, which uses the same construction from Xiao et al.

Original: In short, we construct an MDP with A actions and 2 H states.

Original: Given any behavior policy π b , for h ∈ [ H ] , define a h = arg min a π b ( a | s h ) be the action that leads to the next state s h +1 from state s h , and all other actions lead to an absorbing state

Original: s

Original: oh .

Original: The only nonzero reward is in the last state s H − 1 , and the reward is sampled from a Bernoulli distribution.

Original: Consider a target policy that chooses a h for state s h for all h ∈

Original: [ H ] .

Original: In order to estimate the target policy up to error ε with probability 1 − δ , we must visit s H − 1 at least Ω(1 /ε 2 ) times, and the probability of visiting s H − 1 for each episode is at most 1 /A H .

Original: As a result, we need at least Ω( A H ln (1 /δ ) /ε 2 ) episodes for any sound OPE algorithm.

Original: By the lower bound on the sample complexity of OPE and Theorem 2, we now have a lower bound for OPS.
Revised: By the lower bound on the sample complexity of OPE and Theorem 2, we now have a lower bound for OPS.

Original: Corollary 1 (Exponential lower bound on the sample complexity of OPS) .
Revised: Corollary 1 (Exponential lower bound on the sample complexity of OPS) .

Original: For any positive integers
Revised: For any positive integers

Original: S, A, H with S > 2 H and a pair ( ε, δ ) with 0 < ε ≤ (cid:112) 1 / 8 , δ ∈ (0 , 1) , any ( ε, δ ) -sound
Revised: S, A, H with S > 2 H and a pair ( ε, δ ) with 0 < ε ≤ (cid:112) 1 / 8 , δ ∈ (0 , 1) , any ( ε, δ ) -sound

Original: OPS algorithm needs at least ˜ Ω ( A H − 1 /ε 2 ) episodes.
Revised: OPS algorithm needs at least ˜Ω( A H − 1 /ε 2 ) episodes.

Original: We can use a common OPE method, importance sampling (IS), with a random behavior policy for policy selection.
Revised: We can use a common OPE method, importance sampling (IS), with a random behavior policy for policy selection.

Original: Recall the IS estimator (Rubinstein, 1981) is given by
Revised: Recall the IS estimator (Rubinstein, 1981) is given by

Original: [Equation]
Revised: [Equation]

Original: where n is the number of episodes in the dataset D .
Revised: where n is the number of episodes in the dataset D .

Original: We now provide an upper bound on the sample complexity of OPS using IS.
Revised: We now provide an upper bound on the sample complexity of OPS using IS.

Original: The proof can be found in Appendix A.3.
Revised: The proof can be found in Appendix A.3.

Original: Theorem 3 (OPS using importance sampling) .
Revised: Theorem 3 (OPS using importance sampling) .

Original: Suppose the data collection policy is uniformly random, that is, π b ( a | s ) = 1 /A for all ( s, a ) ∈ S × A , and | G i | ≤ V max almost surely.
Revised: Suppose the data collection policy is uniformly random, that is, π b ( a | s ) = 1 /A for all ( s, a ) ∈ S × A , and | G i | ≤ V max almost surely.

Original: Then the selection algorithm L that selects the policy with the highest IS estimate is ( ε, δ ) -sound with
Revised: Then the selection algorithm L that selects the policy with the highest IS estimate is ( ε, δ ) -sound with

Original: O ( A H V max ln ( | Π | /δ ) /ε 2 ) episodes.
Revised: O ( A H V max ln ( | Π | /δ ) /ε 2 ) episodes.

Original: The theorem suggests that IS achieves a nearly minimax optimal sample complexity for OPS up to a factor A and logarithmic factors.
Revised: The theorem suggests that IS achieves a nearly minimax optimal sample complexity for OPS up to a factor A and logarithmic factors.

Original: There are other improved variants of IS, including per-decision
Revised: There are other improved variants of IS, including per-decision

Original: IS and weighted IS (Precup et al., 2000; Sutton & Barto, 2018).
Revised: IS and weighted IS (Precup et al., 2000; Sutton & Barto, 2018).

Original: However, none of these variants can help reduce sample complexity in the worst case because the lower bound in Corollary 1 holds for any OPS algorithm.
Revised: However, none of these variants can help reduce sample complexity in the worst case because the lower bound in Corollary 1 holds for any OPS algorithm.

Original: The result suggests that we need to consider additional assumptions on the environment, the data distribution, or the candidate set to obtain guarantees for OPS.
Revised: The result suggests that we need to consider additional assumptions on the environment, the data distribution, or the candidate set to obtain guarantees for OPS.

Revised: Note that Wang et al.

Revised: (2017) have shown that IS estimator achievew the minimax mean squared error for the OPE problem.

Revised: Our result shows that IS also achieves a (nearly) minimax sample complexity for the OPS problem.

Original: There are other examples where OPS is efficient by using particular OPE methods.
Revised: There are other examples where OPS is efficient by using particular OPE methods.

Original: For example,
Revised: For example,

Original: Liu et al.
Revised: Liu et al.

Original: (2021) consider environments where the state contains exogenous variables; the agent has limited impact on those exogenous variables; and the agent has the knowledge about the endogenous dynamic.
Revised: (2021) consider environments where the state contains exogenous variables; the agent has limited impact on those exogenous variables; and the agent has the knowledge about the endogenous dynamic.

Original: In such environments, they provide a sound OPE method to select hyperparameters.
Revised: In such environments, they provide a sound OPE method to select hyperparameters.

Original: Another example is when an accurate simulator of the environment is available.
Revised: Another example is when an accurate simulator of the environment is available.

Original: The simulation lemma (Kearns & Singh, 2002) shows that we can evaluate any policy accurately and thus perform sound OPS.
Revised: The simulation lemma (Kearns & Singh, 2002) shows that we can evaluate any policy accurately and thus perform sound OPS.

Original: All these specialized strategies indicate that we can perform OPS efficiently, under much stronger assumptions.
Revised: All these specialized strategies indicate that we can perform OPS efficiently, under much stronger assumptions.

Original:   O FFLINE POLICY SELECTIONUSING F ITTED Q- EVALUATION
Revised:  C ONDITIONS FOR S AMPLE E FFICIENT O FFLINE POLICY SELECTION

Revised: In the following sections, we discuss the conditions under which OPS using Fitted Q-iteration and

Original: FQE is a commonly-used OPE method that has been shown to be effective for  policy selection in benchmark datasets (Paine et al., 2020). In this section, we study when FQE is efficient for 
Revised: Bellman Error are efficient for offline policy selection. These results are based on existing theoretical analysis of FQE and BE, for example, Chen & Jiang (2019); Le et al.

Original: OPS.
Revised: ; Xie et al. (2021); Duan et al. 4. O FFLINE POLICY SELECTION USING F ITTED Q- EVALUATION Fitted Q-evaluation (FQE) is a commonly-used OPE method that has been shown to be effective for policy selection in benchmark datasets (Paine et al., 2020).

Original: FQE applies the empirical Bellman evaluation operator on the value estimate iteratively:  we initialize q H − 1 = 0 , and, for h
Revised: FQE applies the empirical Bellman evaluation operator on the value estimate iteratively: given a function class F , we initialize q H − 1 = 0 , and, for h

Original: = H − 2 , . .
Revised: = H − 2 , . .

Original: . , 0 , q h = arg min
Revised: . , 0 , q h = arg min

Original: f ∈F ˆ l h ( f, q h +1 )
Revised: f ∈F ˆ l h ( f, q h +1 )

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and D h is the dataset at horizon h with distribution µ h .
Revised: and D h is the dataset at horizon h with distribution µ h .

Original: The FQE estimate for J ( π ) is ˆ J ( π ) =
Revised: The FQE estimate for J ( π ) is ˆ J ( π ) =

Original: E a ∼π ( ·| s 0 )
Revised: E a ∼

Original: [ q 0 ( s 0 , a )] .
Revised: [Equation] [Equation]

Original: Intuitively, we should be able to evaluate the value of all policies covered by the data distribution.
Revised: Intuitively, we should be able to evaluate the value of all policies covered by the data distribution.

Original: If the candidate policy set is a subset of the set of policies covered by data, then we can do policy selection well.
Revised: If the candidate policy set is a subset of the set of policies covered by data, then we can do policy selection well.

Original: To define the notion of well-covered policies, we use the measure of distribution shift introduced in Xie et al.
Revised: To define the notion of well-covered policies, we use the measure of distribution shift introduced in Xie et al.

Original: Definition 1 (Well-covered policies) .

Original: Given a non-negative real value C , we define Π( C, µ, F ) be the set of policies covered by the data distribution µ , that is, for any policy π ∈ Π( C, µ, F ) ,
Revised: Given a non-negative real value C , we define Π( C, µ, F ) be the set of policies covered by the data distribution µ , that is, for any policy π ∈ Π( C, µ, F ) ,

Original: [Equation]
Revised: [Equation]

Original: ≤ C holds for any h ∈
Revised: C holds for any h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Liu et al.

Original: (2020) also consider well-covered policies in the sense that max s,a,h d πh ( s,a ) µ h ( s,a ) is bounded, but Xie et al.

Original: (2021) note that max f,f ′ ∈F ∥

Original: f −T π f ′ ∥ dπh ∥ f −T π f ′ ∥

Original: µh ≤ max s,a d πh ( s,a ) µ h ( s,a ) , that is, the condition we use is weaker than the one used in Liu et al.

Original: The theorem below provides a theoretical guarantee for OPS using FQE.
Revised: The theorem below provides a theoretical guarantee for OPS using FQE.

Original: The proof can be found in
Revised: The proof can be found in

Original: Appendix A.4.
Revised: Appendix A.4.

Original: Theorem 4 (Offline policy selection for well-covered policies) .
Revised: Theorem 4 (Offline policy selection for well-covered policies) .

Original: Suppose we have a function class
Revised: Suppose we have a function class

Original: F with Rademacher complexity R µn ( F ) , an approximation error ε apx , and an optimization error ε opt (see Appendix A.4 for the definitions).
Revised: F with Rademacher complexity R µn ( F ) , an approximation error ε apx , and an optimization error ε opt (see Appendix A.4 for the definitions).

Original: If there exists a non-negative value C such that Π ⊂ Π( C, µ, F ) , then the OPS algorithm L that selects the policy with the highest FQE estimate satisfies
Revised: If there exists a non-negative value C such that Π ⊂ Π( C, µ, F ) , then the OPS algorithm L that selects the policy with the highest FQE estimate satisfies

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: for some constant c 0 .
Revised: for some constant c 0 .

Original: If we have a good function class and an optimizer, then FQE only requires a polynomial number of episodes for OPS under the assumption that all candidate policies are well-covered by the data.
Revised: If we have a good function class and an optimizer, then FQE only requires a polynomial number of episodes for OPS under the assumption that all candidate policies are well-covered by the data.

Original: However, in practice, we need to choose the function class and optimization hyperparameters.
Revised: However, in practice, we need to choose the function class and optimization hyperparameters.

Original: It is easy to choose optimization hyperparameters such as the learning rate.
Revised: It is easy to choose optimization hyperparameters such as the learning rate.

Original: We can do so by choosing the optimizer with the lowest ε opt .
Revised: We can do so by choosing the optimizer with the lowest ε opt .

Original: This can be done by selecting the optimization hyperparameters that result in the smallest TD errors (cid:80) h ˆ l h ( q h , q h +1 ) on a validation dataset.
Revised: This can be done by selecting the optimization hyperparameters that result in the smallest TD errors (cid:80) h ˆ l h ( q h , q h +1 ) on a validation dataset.

Original: However, selecting a function class is nontrivial, because we need to balance the approximation error, the complexity measure and how well the data covers the candidate policies.
Revised: However, selecting a function class is nontrivial, because we need to balance the approximation error, the complexity measure and how well the data covers the candidate policies.

Original: In our experiments, we fix the function class as a neural network model with the same architecture and tune the optimization hyperparameters.
Revised: In our experiments, we fix the function class as a neural network model with the same architecture and tune the optimization hyperparameters.

Original: Theorem 4 presents a positive result that OPS can be sample efficient when selecting amongst wellcovered policies.
Revised: Theorem 4 presents a positive result that OPS can be sample efficient when selecting amongst wellcovered policies.

Original: However, if one of the candidate policies is not well-covered, then the FQE estimate may overestimate the value of the uncovered policy and resulting in poor OPS.
Revised: However, if one of the candidate policies is not well-covered, then the FQE estimate may overestimate the value of the uncovered policy and resulting in poor OPS.

Original: It is known that
Revised: It is known that

Original: FQE can even diverge (Sutton & Barto, 2018), due to the fact that it combines off-policy learning with bootstrapping and function approximation, known as the deadly triad.
Revised: FQE can even diverge (Sutton & Barto, 2018), due to the fact that it combines off-policy learning with bootstrapping and function approximation, known as the deadly triad.

Original: One way to circumvent the issue of uncovered policies is to find a way to assign low values for uncovered policies.
Revised: One way to circumvent the issue of uncovered policies is to find a way to assign low values for uncovered policies.

Original: One heuristic way is to early stop when FQE diverges and assign lower value estimates to policies for which FQE diverges since the policy is likely to be not covered by the data.
Revised: One heuristic way is to early stop when FQE diverges and assign lower value estimates to policies for which FQE diverges since the policy is likely to be not covered by the data.

Original: Another heuristic is to use a pessimistic version of FQE that penalizes policies whose data distribution is dissimilar from the behavior distribution.
Revised: Another heuristic is to use a pessimistic version of FQE that penalizes policies whose data distribution is dissimilar from the behavior distribution.

Original: This approach is similar to CQL, but it introduces an additional hyperparameter to control the level of pessimism, which again is not easy to tune in the offline setting.
Revised: This approach is similar to CQL, but it introduces an additional hyperparameter to control the level of pessimism, which again is not easy to tune in the offline setting.

Original: Practical insight.
Revised: Practical insight.

Original: Our result explains the empirical findings in Paine et al.
Revised: Our result explains the empirical findings in Paine et al.

Original: (2020) that FQE is sufficient for selecting hyperparameters for conservative algorithms and imitation learning algorithms.
Revised: (2020) that FQE is sufficient for selecting hyperparameters for conservative algorithms and imitation learning algorithms.

Original: A recent paper (Kumar et al., 2022) discusses when we should use pessimistic offline RL or behavioral cloning under different conditions.
Revised: A recent paper (Kumar et al., 2022) discusses when we should use pessimistic offline RL or behavioral cloning under different conditions.

Original: Our result shows that we can in fact run both algorithms and choose the one with highest FQE estimate.
Revised: Our result shows that we can in fact run both algorithms and choose the one with highest FQE estimate.

Original: With high probability, we can choose the best policy among offline RL and behavioral cloning algorithms without considering specific conditions.
Revised: With high probability, we can choose the best policy among offline RL and behavioral cloning algorithms without considering specific conditions.

Original:  O FFLINE POLICY SELECTION USING THE B ELLMAN ERROR
Revised: 4. O FFLINE POLICY SELECTION USING THE B ELLMAN ERROR

Original: In this section, we investigate whether OPS using the estimated Bellman error has any advantage over FQE.
Revised: In this section, we investigate whether OPS using the estimated Bellman error has any advantage over FQE.

Original: Most value-based methods output value functions and we often perform policy selection over value functions by considering the greedy policies with respect to these value functions.
Revised: Most value-based methods output value functions and we often perform policy selection over value functions by considering the greedy policies with respect to these value functions.

Original: Suppose we are given a candidate set of value functions { q i } Ki =1 , a natural criterion is to select the value function that is closest to the optimal value function, that is, ∥ q
Revised: Suppose we are given a candidate set of value functions { q i } Ki =1 , a natural criterion is to select the value function that is closest to the optimal value function, that is, ∥ q

Original: i − q ∗ ∥ ∞ is the smallest.
Revised: i − q ∗ ∥ ∞ is the smallest.

Original: A justification of choosing the smallest ∥ q i − q ∗ ∥ ∞ is that the value error gives us a lower bound on J ( π i ) where π i is the greedy policy with respect to q i .
Revised: A justification of choosing the smallest ∥ q i − q ∗ ∥ ∞ is that the value error gives us a lower bound on J ( π i ) where π i is the greedy policy with respect to q i .

Original: That is, J ( π i ) ≥ J ( π ∗ ) − 2 H
Revised: That is, J ( π i ) ≥ J ( π ∗ ) − 2 H

Original: ∥
Revised: ∥

Original: q
Revised: q

Original: i − q ∗ ∥ ∞
Revised: i − q ∗ ∥ ∞

Original: (Singh &
Revised: (Singh &

Original: Yee, 1994).
Revised: Yee, 1994).

Original: Therefore, choosing the smallest ∥ q i − q ∗ ∥ ∞ is equivalent to choosing the largest lower bound on the policy value.
Revised: Therefore, choosing the smallest ∥ q i − q ∗ ∥ ∞ is equivalent to choosing the largest lower bound on the policy value.

Original: While we do not know q ∗ , it might still be possible to derive a similar lower bound on the policy value, possibly by estimating the Bellman error (BE).
Revised: While we do not know q ∗ , it might still be possible to derive a similar lower bound on the policy value, possibly by estimating the Bellman error (BE).

Original: A potential use case of the BE is that if the data distribution covers the entire state-action space well, then the BE is a good surrogate for the policy value.
Revised: A potential use case of the BE is that if the data distribution covers the entire state-action space well, then the BE is a good surrogate for the policy value.

Original: More formally, the concentration coefficient C is defined as the smallest value such that max s,a,h d πh ( s,a ) µ h ( s,a ) ≤ C for all policies π .
Revised: More formally, the concentration coefficient C is defined as the smallest value such that max s,a,h d πh ( s,a ) µ h ( s,a ) ≤ C for all policies π .

Original: By the performance difference lemma (Kakade, 2003), we can show that
Revised: By the performance difference lemma (Kakade, 2003), we can show that

Original: [Equation]
Revised: [Equation]

Original: That is, the Bellman error evaluated on the dataset can be a useful indicator of the quality of the greedy policy, if C is small.
Revised: That is, the Bellman error evaluated on the dataset can be a useful indicator of the quality of the greedy policy, if C is small.

Original: Note that a small concentration coefficient is a stronger assumption compared to the assumption of FQE, since the measure of distribution shift is bounded for all policies.
Revised: Note that a small concentration coefficient is a stronger assumption compared to the assumption of FQE. To be more specific, the condition max s,a,h d πh ( s,a ) µ h ( s,a ) ≤ C for all policies π ∈ Π implies max [Equation] [Equation]

Original: For FQE, we just need coverage for all policies in the candidate set.
Revised: for all policies π ∈ Π . That is, we only need coverage for all policies in the candidate set for sample efficient OPS with FQE. However, for BE, the measure of distribution shift needs to be bounded for all policies.

Original: Finding the value function with the lowest Bellman error in deterministic environments is sample efficient.
Revised: Finding the value function with the lowest Bellman error in deterministic environments is sample efficient.

Original: To see why this is the case, let k be the output index of the selection algorithm that selects the index of the value function with the smallest empirical Bellman error, then with probability at least 1 − δ , for some constant c 0 ,
Revised: To see why this is the case, let k be the output index of the selection algorithm that selects the index of the value function with the smallest empirical Bellman error, then with probability at least 1 − δ , for some constant c 0 ,

Original: [Equation]
Revised: [Equation]

Original: n where we denote ∥ q i −T q i ∥ 22 ,µ = 1 H (cid:80) h
Revised: n where we denote ∥ q i −T q i ∥ 22 ,µ = 1 H (cid:80) h

Original: ∥ q i,h −T q i,h +1 ∥ 22 ,µ h for simplicity.
Revised: ∥ q i,h −T q i,h +1 ∥ 22 ,µ h for simplicity.

Original: Therefore, the sample size n only needs to scale with V 2 max .
Revised: Therefore, the sample size n only needs to scale with V 2 max .

Original: In contrast, the sample size of FQE depends on the complexity of the function class R µn ( F ) .
Revised: In contrast, the sample size of FQE depends on the complexity of the function class R µn ( F ) .

Original: If we use a function class with high complexity to perform FQE, then we expect to see that BE is a more sample efficient method for selection compared to FQE when the data covers the state-action space well, that is, when C is small.
Revised: If we use a function class with high complexity to perform FQE, then we expect to see that BE is a more sample efficient method for selection compared to FQE when the data covers the state-action space well, that is, when C is small.

Original: If the concentration coefficient is large or infinite, then the Bellman error can be a poor indicator of the greedy policy’s performance.
Revised: If the concentration coefficient is large or infinite, then the Bellman error can be a poor indicator of the greedy policy’s performance.

Original: For example, Fujimoto et al.
Revised: For example, Fujimoto et al.

Original: (2022) provide several examples where BE is poor surrogate for value error even when the environment is deterministic.
Revised: (2022) provide several examples where BE is poor surrogate for value error even when the environment is deterministic.

Original: Moreover, several empirical works have shown that OPS using the Bellman error or value estimate from offline RL algorithms underperform using OPE estimates (Tang & Wiens, 2021).
Revised: Moreover, several empirical works have shown that OPS using the Bellman error or value estimate from offline RL algorithms underperform using OPE estimates (Tang & Wiens, 2021).

Original: Our result provides theoretical reasoning for these experimental observations.
Revised: Our result provides theoretical reasoning for these experimental observations.

Original: Since, in most practical scenarios the concentration coefficient can be very large or infinite, OPS using the BE can be poor.
Revised: Since, in most practical scenarios the concentration coefficient can be very large or infinite, OPS using the BE can be poor.

Original: Estimating Bellman error in stochastic environments.
Revised: Estimating Bellman error in stochastic environments.

Original: Estimating Bellman error in stochastic environments potentially requires more samples.
Revised: Estimating Bellman error in stochastic environments potentially requires more samples.

Original: This is because doing so often involves fitting a function (Antos et al., 2008; Farahmand & Szepesv ´ ari, 2011), and resultingly the sample complexity of estimating the BE accurately depends on the complexity of the function class.
Revised: This is because doing so often involves fitting a function (Antos et al., 2008; Farahmand & Szepesv ´ ari, 2011), and resultingly the sample complexity of estimating the BE accurately depends on the complexity of the function class.

Original: We hypothesize that it makes the sample complexity similar to FQE.
Revised: We hypothesize that it makes the sample complexity similar to FQE.

Original: If the sample complexity is similar to FQE and policy selection using BE requires stronger assumptions, then it is natural to use FQE in stochastic environments even if the data coverage is good.
Revised: If the sample complexity is similar to FQE and policy selection using BE requires stronger assumptions, then it is natural to use FQE in stochastic environments even if the data coverage is good.

Original: We leave the exact analysis of the sample complexity of OPS using estimated Bellman Error in stochastic environments for future work.
Revised: We leave the exact analysis of the sample complexity of OPS using estimated Bellman Error in stochastic environments for future work.

Original:  E MPIRICAL COMPARISON BETWEEN FQE AND BE
Revised:  E MPIRICAL COMPARISON BETWEEN FQE AND BE

Original: In this section, we aim to validate our results experimentally by answering the following questions: (1) How do FQE and BE perform when (a) all candidate policies are well-covered, (b) the data contains diverse trajectories or (c) the data neither covers all candidate policies nor provides diverse trajectories?
Revised: In this section, we aim to validate our results experimentally by answering the following questions: (1) How do FQE and BE perform when (a) all candidate policies are well-covered, (b) the data contains diverse trajectories or (c) the data neither covers all candidate policies nor provides diverse trajectories?

Original: (2) Can BE be more sample efficient than FQE for OPS?
Revised: (2) Can BE be more sample efficient than FQE for OPS?

Original: We conduct experiments on two standard RL environments: Acrobot and Cartpole.
Revised: We conduct experiments on two standard RL environments: Acrobot and Cartpole.

Original: We first generate a set of candidate policy-value pairs { π i , q i } Ki =1 by running CQL (Kumar et al., 2020) with different hyperparameters on a batch of data collected by an optimal policy with random actions taken 40% of the time.
Revised: We first generate a set of candidate policy-value pairs { π i , q i } Ki =1 by running CQL (Kumar et al., 2020) with different hyperparameters on a batch of data collected by an optimal policy with random actions taken 40% of the time.

Original: We then use either FQE or BE to rank the candidate policies, and select the top- k policies.
Revised: We then use either FQE or BE to rank the candidate policies, and select the top- k policies.

Original: The OPS problem we described in this paper is a special case with k = 1 .
Revised: The OPS problem we described in this paper is a special case with k = 1 .

Revised: We also included a random selection baseline where the we randomly choose k policies from the candidate set.

Original: To evaluate the performance of top- k policy selection, we consider the top- k regret which is used in the existing literature (Zhang & Jiang, 2021; Paine et al., 2020).
Revised: To evaluate the performance of top- k policy selection, we consider the top- k regret which is used in the existing literature (Zhang & Jiang, 2021; Paine et al., 2020).

Original: Top-
Revised: Top-

Original: k regret is the gap between the best policy within the top-k policies, and the best among all candidate policies.
Revised: k regret is the gap between the best policy within the top-k policies, and the best among all candidate policies.

Original: More experimental details can be found in Appendix B.
Revised: More experimental details can be found in Appendix B.

Original: In the first set of experiments, we generate three different datasets: (a) well-covered data is generated such that all candidate policies are well-covered, (b) diverse data includes more diverse trajectories to obtain a lower concentration coefficient, and (c) expert data is collected by a deterministic and optimal policy.
Revised: In the first set of experiments, we generate three different datasets: (a) well-covered data is generated such that all candidate policies are well-covered, (b) diverse data includes more diverse trajectories to obtain a lower concentration coefficient, and (c) expert data is collected by a deterministic and optimal policy.

Original: Figure 1 shows the top- k regret with 900 episodes of data.
Revised: Figure 1 shows the top- k regret with 900 episodes of data.

Original: FQE performs very well with a small regret on well-covered and diverse data.
Revised: FQE performs very well with a small regret on well-covered and diverse data.

Original: BE does not perform well with wellcovered data.
Revised: BE does not perform well with wellcovered data.

Original: However, with diverse data, BE performs much better.
Revised: However, with diverse data, BE performs much better.

Original: Surprisingly, for expert data,
Revised: Surprisingly, for expert data,

Original: BE performs better than FQE even though the data distribution has an extremely large concentration 
Revised: BE performs better than FQE even though the data distribution has an extremely large concentration k

Original: [Figure]
Revised: R e g r e t Well-covered Data   k Diverse Data   k Expert Data

Original: BE
Revised: BE

Original: FQE k
Revised: FQE random

Original: [Figure]
Revised: Acrobot   k R e g r e t Well-covered Data   k

Original: Diverse Data   k
Revised: Diverse Data   k

Original: Expert Data
Revised: Expert Data

Original: Acrobot
Revised: BE

Original: [Figure]
Revised: FQE random

Original: [Figure] coefficient.
Revised: Cartpole coefficient.

Original: In Acrobot, BE tends to perform better with a larger k even with good data coverage, while FQE can have a small top- 1 regret as long as the data covers the candidate policy.
Revised: In Acrobot, BE tends to perform better with a larger k even with good data coverage, while FQE can have a small top- 1 regret as long as the data covers the candidate policy.

Original: In the second set of experiments [Figure] we use diverse data.
Revised: In the second set of experiments, we use diverse data.

Original: Figure 2 shows the results with varying numbers of episodes for top- k regret with k ∈ { 1 , 5 , 10 } .
Revised: Figure 2 shows the results with varying numbers of episodes for top- k regret with k ∈ { 1 , 5 , 10 } .

Original: BE performs better than FQE when the sample size is small (less than 100 episodes) in most cases.
Revised: BE performs better than FQE when the sample size is small (less than 100 episodes) in most cases.

Revised: The random selection baseline is expected to perform independent of the sample size.

Original: We expect to see a larger gap when the environment has a high-dimensional state spacee so a function class with large complexity is needed for FQE.
Revised: We expect to see a larger gap when the environment has a high-dimensional state spacee so a function class with large complexity is needed for FQE.

Original: In summary, these experimental results validate our theoretical results.
Revised: In summary, these experimental results validate our theoretical results.

Original: We show (1) FQE performs very well when the candidate policies are well-covered; (2) BE performs well under a diverse data coverage assumption; and (3) neither perform well without their corresponding data coverage conditions.
Revised: We show (1) FQE performs very well when the candidate policies are well-covered; (2) BE performs well under a diverse data coverage assumption; and (3) neither perform well without their corresponding data coverage conditions.

Original: Moreover, BE can be more sample efficient than FQE under good data coverage.
Revised: Moreover, BE can be more sample efficient than FQE under good data coverage.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In this section we provide a more comprehensive survey of prior work on model selection for reinforcement learning.
Revised: In this section we provide a more comprehensive survey of prior work on model selection for reinforcement learning.

Original: In the online setting, model selection has been studied extensively across contextual bandits (Foster et al., 2019) to reinforcement learning (Lee et al., 2021).
Revised: In the online setting, model selection has been studied extensively across contextual bandits (Foster et al., 2019) to reinforcement learning (Lee et al., 2021).

Original: In the online setting, the goal is to select model classes while balancing exploration and exploitation to achieve low regret, which is very different from the offline setting where no exploration is performed.
Revised: In the online setting, the goal is to select model classes while balancing exploration and exploitation to achieve low regret, which is very different from the offline setting where no exploration is performed.

Original: In the offline setting, besides using OPE for model selection, Farahmand & Szepesv´ari (2011) and
Revised: In the offline setting, besides using OPE for model selection, Farahmand & Szepesv´ari (2011) and

Original: Zhang & Jiang (2021) consider selecting a value function that has the smallest Bellman error or is the closest to the optimal value function.
Revised: Zhang & Jiang (2021) consider selecting a value function that has the smallest Bellman error or is the closest to the optimal value function.

Original: Farahmand & Szepesv´ari (2011) consider selecting a value function among a set of candidate value functions in stochastic environments such that with high probability, ∥ q k −T q k ∥ 22 ,µ ≤ min i ∥
Revised: Farahmand & Szepesv´ari (2011) consider selecting a value function among a set of candidate value functions in stochastic environments such that with high probability, ∥ q k −T q k ∥ 22 ,µ ≤ min i ∥

Original: q i −T q i ∥ 22 ,µ + ε where k is the output of a selection algorithm.
Revised: q i −T q i ∥ 22 ,µ + ε where k is the output of a selection algorithm.

Original: To estimate the Bellman error, they propose to fit a regression model
Revised: To estimate the Bellman error, they propose to fit a regression model

Original: ˜ q i to predict T q i and bound the Bellman error by ∥ q
Revised: ˜ q i to predict T q i and bound the Bellman error by ∥ q

Original: i − ˜ q i ∥ 2 ,µ + b i where b i is an upper bound on the estimation error of the regression.
Revised: i − ˜ q i ∥ 2 ,µ + b i where b i is an upper bound on the estimation error of the regression.

Original: However, as we show in this paper, even in a deterministic environment where selecting the smallest Bellman error is easy, it does not imply effective OPS unless the data coverage is good.
Revised: However, as we show in this paper, even in a deterministic environment where selecting the smallest Bellman error is easy, it does not imply effective OPS unless the data coverage is good.

Original: Zhang & Jiang (2021) consider the projected Bellman error with a piecewise constant function class.
Revised: Zhang & Jiang (2021) consider the projected Bellman error with a piecewise constant function class.

Original: They show that if q ∗ is in the candidate set and a stronger data assumption is satisfied, then they can choose the optimal value function by selecting the value function with the smallest Batch ValueFunction Tournament loss.
Revised: They show that if q ∗ is in the candidate set and a stronger data assumption is satisfied, then they can choose the optimal value function by selecting the value function with the smallest Batch [Figure] FQE [Figure]

Revised: Acrobot

Revised: [Figure]

Original:  In practice, their method is computationally expensive since it scales with O ( | Π | 2 ) instead of O ( | Π | ) , making the method impractical when the candidate set is not small.
Revised: Value-Function Tournament loss (BVFT). Xie & Jiang (2021) consider BVFT with approximation error. In practice, their method is computationally expensive since it scales with O ( | Π | 2 ) instead of O ( | Π | ) , making the method impractical when the candidate set is not small.

Original: Other work on model selection in RL is in other settings: selecting models and selecting amongst
Revised: Other work on model selection in RL is in other settings: selecting models and selecting amongst

Original: OPE estimators.
Revised: OPE estimators.

Original: Hallak et al.
Revised: Hallak et al.

Original: (2013) consider model selection for model-based RL algorithms with batch data.
Revised: (2013) consider model selection for model-based RL algorithms with batch data.

Original: They focus on selecting the most suitable model that generates the observed data, based on the maximum likelihood framework.
Revised: They focus on selecting the most suitable model that generates the observed data, based on the maximum likelihood framework.

Original: Su et al.
Revised: Su et al.

Original: (2020) consider estimator selection for OPE when the estimators can be ordered with monotonically increasing biases and decreasing confidence intervals.
Revised: (2020) consider estimator selection for OPE when the estimators can be ordered with monotonically increasing biases and decreasing confidence intervals.

Original: They also show that their method achieves an oracle inequality for estimator selection.
Revised: They also show that their method achieves an oracle inequality for estimator selection.

Original: To the best of our knowledge, there is no previous work on understanding the fundamental limits for the OPS problem in RL.
Revised: To the best of our knowledge, there is no previous work on understanding the fundamental limits for the OPS problem in RL.

Original: There is one related work in the batch contextual bandit setting, studying the selection of a linear model (Lee et al., 2022).
Revised: There is one related work in the batch contextual bandit setting, studying the selection of a linear model (Lee et al., 2022).

Original: They provide a hardness result suggesting it is impossible to achieve an oracle inequality that balances the approximation error, the complexity of the function class, and data coverage.
Revised: They provide a hardness result suggesting it is impossible to achieve an oracle inequality that balances the approximation error, the complexity of the function class, and data coverage.

Original: However, their results are restricted only to the contextual bandit setting.
Revised: However, their results are restricted only to the contextual bandit setting.

Original: In this paper, we consider the more general problem, selecting a policy from a set of policies, in the RL setting.
Revised: In this paper, we consider the more general problem, selecting a policy from a set of policies, in the RL setting.

Original:  C ONCLUSION & D ISCUSSION
Revised:  C ONCLUSION & D ISCUSSION

Original: In this paper, we have made progress towards understanding when OPS is feasible for RL.
Revised: In this paper, we have made progress towards understanding when OPS is feasible for RL.

Original: Our main result that the sample complexity of OPS is lower-bounded by the sample complexity of OPE, is perhaps expected.
Revised: Our main result that the sample complexity of OPS is lower-bounded by the sample complexity of OPE, is perhaps expected.

Original: However, to our knowledge, this has never been formally shown.
Revised: However, to our knowledge, this has never been formally shown.

Original: This result implies that without conditions to make OPE feasible, we cannot do policy selection efficiently.
Revised: This result implies that without conditions to make OPE feasible, we cannot do policy selection efficiently.

Original: We provided theoretical statements with experimental evidence demonstrating that FQE is sample efficient when all candidate policies are well-covered and that BE can be more sample efficient under stronger data coverage.
Revised: We provided theoretical statements with experimental evidence demonstrating that FQE is sample efficient when all candidate policies are well-covered and that BE can be more sample efficient under stronger data coverage.

Original: This paper highlights the importance of carefully considering hyperparameter selection when developing new offline RL algorithms.
Revised: This paper highlights the importance of carefully considering hyperparameter selection when developing new offline RL algorithms.

Original: Since selecting the best-performing policy is not possible in the absence of a sound OPE algorithm, we may need to use heuristics that exploit other properties of hyperparameters.
Revised: Since selecting the best-performing policy is not possible in the absence of a sound OPE algorithm, we may need to use heuristics that exploit other properties of hyperparameters.

Original: For example, Kumar et al.
Revised: For example, Kumar et al.

Original: (2021) use the TD error trend as an indicator to select the number of training steps for CQL.
Revised: (2021) use the TD error trend as an indicator to select the number of training steps for CQL.

Original: Their work is purely heuristic, but exploring this direction from a theoretical vantage point is a promising line of research  to avoid the fundamental limits of
Revised: Their work is purely heuristic, but exploring this direction from a theoretical vantage point is a potential research direction to avoid the fundamental limits of

Original: OPS.
Revised: OPS.

Original: Our goal with this work is to provide useful insights into the nature of offline RL, potentiallyleading to the development of sound hyperparameter selection procedures for real-world offline RL applications.
Revised: Our goal with this work is to provide useful insights into the nature of offline RL, potentially

Original: A T ECHNICAL D ETAILS A. P ROOF OF T HEOREM
Revised: A T ECHNICAL D ETAILS A. P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: (Upper bound on sample complexity of OPS)
Revised: (Upper bound on sample complexity of OPS)

Original: Given an MDP M , a data distribution d b , and a set of policies Π , suppose that, for any pair ( ε, δ ) , there exists an ( ε, δ ) -sound
Revised: Given an MDP M , a data distribution d b , and a set of policies Π , suppose that, for any pair ( ε, δ ) , there exists an ( ε, δ ) -sound

Original: OPE algorithm L on any OPE instance I ∈ { ( M, d b , π ) : π ∈ Π } with a sample size at most
Revised: OPE algorithm L on any OPE instance I ∈ { ( M, d b , π ) : π ∈ Π } with a sample size at most

Original: O ( N OPE ( S, A, H, ε, 1 /δ )) .
Revised: O ( N OPE ( S, A, H, ε, 1 /δ )) .

Original: Then there exists an ( ε, δ ) -sound
Revised: Then there exists an ( ε, δ ) -sound

Original: OPS algorithm for the OPS problem instance ( M, d b , Π) which requires at most O ( N OPE ( S, A, H, ε, | Π | /δ )) episodes.
Revised: OPS algorithm for the OPS problem instance ( M, d b , Π) which requires at most O ( N OPE ( S, A, H, ε, | Π | /δ )) episodes.

Original: Proof.
Revised: Proof.

Original: The OPS algorithm L ( D, Π) for a given ( ε, δ ) works as follows: we query an ( ε ′ , δ ′ ) -sound
Revised: The OPS algorithm L ( D, Π) for a given ( ε, δ ) works as follows: we query an ( ε ′ , δ ′ ) -sound

Original: OPE algorithm for each policy in Π and select the policy with the highest estimated value.
Revised: OPE algorithm for each policy in Π and select the policy with the highest estimated value.

Original: That is,
Revised: That is,

Original: L ( D, Π) outputs the policy ¯ π := argmax π ∈ Π ˆ J ( π ) , where ˆ J ( π ) is the value estimate for policy π by the ( ε ′ , δ ′ ) -sound
Revised: L ( D, Π) outputs the policy ¯ π := argmax π ∈ Π ˆ J ( π ) , where ˆ J ( π ) is the value estimate for policy π by the ( ε ′ , δ ′ ) -sound

Original: OPE algorithm with data D .
Revised: OPE algorithm with data D .

Original: By definition of an ( ε ′ , δ ′ ) -sound
Revised: By definition of an ( ε ′ , δ ′ ) -sound

Original: OPE algorithm
Revised: OPE algorithm

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Applying the union bound, we have
Revised: Applying the union bound, we have

Original: [Equation]
Revised: [Equation]

Original: Let π † denote the best policy in the candidate set Π , that is, π † := argmax π ∈ Π J ( π ) .
Revised: Let π † denote the best policy in the candidate set Π , that is, π † := argmax π ∈ Π J ( π ) .

Original: With probability 1 − δ ′
Revised: With probability 1 − δ ′

Original: | Π | , we have
Revised: | Π | , we have

Original: [Equation]
Revised: [Equation]

Original: The first and third inequalities follow from equation 1, and the second inequality follows from the definition of ¯ π .
Revised: The first and third inequalities follow from equation 1, and the second inequality follows from the definition of ¯ π .

Original: Finally, by setting δ ′ =
Revised: Finally, by setting δ ′ =

Original: δ/
Revised: δ/

Original: | Π | and ε ′ =
Revised: | Π | and ε ′ =

Original: ε/ 2 , we get
Revised: ε/ 2 , we get

Original: [Equation]
Revised: [Equation]

Original: That is, L is an ( ε, δ ) -sound
Revised: That is, L is an ( ε, δ ) -sound

Original: OPS algorithm.
Revised: OPS algorithm.

Original: This requires at most O ( N OPE ( S, A, H, ε/ 2 , | Π | /δ )) samples.
Revised: This requires at most O ( N OPE ( S, A, H, ε/ 2 , | Π | /δ )) samples.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Theorem 2 (Lower bound on sample complexity of OPS) .
Revised: Theorem 2 (Lower bound on sample complexity of OPS) .

Original: Suppose for any data distribution d b and any pair ( ε, δ ) with ε ∈ (0 , V max / 2) and δ ∈ (0 , 1) , there exists an MDP M and a policy π such that any ( ε, δ ) -sound
Revised: Suppose for any data distribution d b and any pair ( ε, δ ) with ε ∈ (0 , V max / 2) and δ ∈ (0 , 1) , there exists an MDP M and a policy π such that any ( ε, δ ) -sound

Original: OPE algorithm requires at least Ω( N OPE ( S, A, H, ε, 1 /δ )) episodes.
Revised: OPE algorithm requires at least Ω( N OPE ( S, A, H, ε, 1 /δ )) episodes.

Original: Then there exists an MDP M ′ with S ′ =
Revised: Then there exists an MDP M ′ with S ′ =

Original: S + 2 , H ′ =
Revised: S + 2 , H ′ =

Original: H + 1 , and a set of candidate policies such that for any pair ( ε, δ ) with ε ∈ (0 , V max / 3) and δ ∈ (0 , 1 /m ) where m :=
Revised: H + 1 , and a set of candidate policies such that for any pair ( ε, δ ) with ε ∈ (0 , V max / 3) and δ ∈ (0 , 1 /m ) where m :=

Original: ⌈ log( V max /ε )
Revised: ⌈ log( V max /ε )

Original: ⌉ ≥ 1 , any ( ε, δ ) -sound
Revised: ⌉ ≥ 1 , any ( ε, δ ) -sound

Original: OPS algorithm also requires at least Ω( N OPE ( S, A, H, ε, 1 / ( mδ ))))
Revised: OPS algorithm also requires at least Ω( N OPE ( S, A, H, ε, 1 / ( mδ ))))

Original: episodes.
Revised: episodes.

Original: Proof.
Revised: Proof.

Original: Our goal is to construct an ( ε, δ ) -sound
Revised: Our goal is to construct an ( ε, δ ) -sound

Original: OPE algorithm with δ ∈ (0 , 1) and ε ∈ [0 , V max / 2] .
Revised: OPE algorithm with δ ∈ (0 , 1) and ε ∈ [0 , V max / 2] .

Original: To evaluate any policy π in M with dataset D sampled from d b , we first construct a new MDP M r with two additional states: an initial state s 0 and a terminal state s 1 .
Revised: To evaluate any policy π in M with dataset D sampled from d b , we first construct a new MDP M r with two additional states: an initial state s 0 and a terminal state s 1 .

Original: Taking a 1 at s 0 transitions to s 1 with reward r .
Revised: Taking a 1 at s 0 transitions to s 1 with reward r .

Original: Taking a 2 at s 0 transitions to the initial state in the original MDP M .
Revised: Taking a 2 at s 0 transitions to the initial state in the original MDP M .

Original: Let Π =
Revised: Let Π =

Original: π 1 , π 2 } be the candidate set in M r where π 1 ( s 0 )
Revised: π 1 , π 2 } be the candidate set in M r where π 1 ( s 0 )

Original: = a 1 and π 2 ( s 0 )
Revised: = a 1 and π 2 ( s 0 )

Original: = a 2 and π 2 ( a | s ) =
Revised: = a 2 and π 2 ( a | s ) =

Original: π ( a | s ) for all ( s, a ) ∈ S ×A .
Revised: π ( a | s ) for all ( s, a ) ∈ S ×A .

Original: Since π 1 always transitions to s 1 , it never transitions to states in MDP
Revised: Since π 1 always transitions to s 1 , it never transitions to states in MDP

Original: M .
Revised: M .

Original: Therefore, π 1 can be arbitrary for all ( s, a ) ∈ S × A .
Revised: Therefore, π 1 can be arbitrary for all ( s, a ) ∈ S × A .

Original: We can add any number of transitions ( s 0 , a 1 , r, s ) and ( s 0 , a 2 , 0 , s ) in D to construct the dataset D r with distribution d b,r arbitrarily.
Revised: We can add any number of transitions ( s 0 , a 1 , r, s ) and ( s 0 , a 2 , 0 , s ) in D to construct the dataset D r with distribution d b,r arbitrarily.

Original: Suppose we have an ( ε ′ , δ ′ ) -sound
Revised: Suppose we have an ( ε ′ , δ ′ ) -sound

Original: OPS algorithm, where we set ε ′ =
Revised: OPS algorithm, where we set ε ′ =

Original: 2 ε/ 3 , δ ′ =
Revised: 2 ε/ 3 , δ ′ =

Original: δ/m and m :=
Revised: δ/m and m :=

Original: ⌈ log( V max /ε ′ )
Revised: ⌈ log( V max /ε ′ )

Original: Note that if this assumption does not hold, then it directly implies that the sample complexity of OPS is larger than Ω( N OPE ( S, A, H, ε, 1 /δ )) .
Revised: Note that if this assumption does not hold, then it directly implies that the sample complexity of OPS is larger than Ω( N OPE ( S, A, H, ε, 1 /δ )) .

Original: Our strategy will be to iteratively set the reward r of MDP M r and run our sound OPS algorithm on Π and using bisection search to estimate a precise interval for J ( π ) .
Revised: Our strategy will be to iteratively set the reward r of MDP M r and run our sound OPS algorithm on Π and using bisection search to estimate a precise interval for J ( π ) .

Original: The process is as follows.
Revised: The process is as follows.

Original: By construction, our OPS algorithm will output either π 1 , which has value
Revised: By construction, our OPS algorithm will output either π 1 , which has value

Original: J M r ( π 1 ) = r , or output π 2 , which has value J M r ( π 2 ) =
Revised: J M r ( π 1 ) = r , or output π 2 , which has value J M r ( π 2 ) =

Original: J M ( π ) .
Revised: J M ( π ) .

Original: That is, it has the same value as π in the original MDP.
Revised: That is, it has the same value as π in the original MDP.

Original: Let us consider the following two cases.
Revised: Let us consider the following two cases.

Original: Let π † be the best policy in Π for
Revised: Let π † be the best policy in Π for

Original: MDP M r .
Revised: MDP M r .

Original: Case 1: the OPS algorithm selects π 1 .
Revised: Case 1: the OPS algorithm selects π 1 .

Original: We know, by definition of a sound OPS algorithm, that
Revised: We know, by definition of a sound OPS algorithm, that

Original: [Equation]
Revised: [Equation]

Original: Case 2: the OPS algorithm selects π 2 .
Revised: Case 2: the OPS algorithm selects π 2 .

Original: [Equation]
Revised: [Equation]

Original: Given this information, we describe the iterative process by which we produce the estimate ˆ J ( π ) .
Revised: Given this information, we describe the iterative process by which we produce the estimate ˆ J ( π ) .

Original: We first set U
Revised: We first set U

Original: =
Revised: =

Original: V max , L = 0 and r =
Revised: V max , L = 0 and r =

Original: U + Land run the sound OPS algorithm with input D r of sample size n r and the candidate set Π .
Revised: U + Land run the sound OPS algorithm with input D r of sample size n r and the candidate set Π .

Original: Then if the selected policy is π 1 , then we conclude the desired event J ( π ) ≤ r + ε ′ occurs with probability at least 1 − δ ′ , and set U equal to r .
Revised: Then if the selected policy is π 1 , then we conclude the desired event J ( π ) ≤ r + ε ′ occurs with probability at least 1 − δ ′ , and set U equal to r .

Original: If the selected policy is π 2 , then we know the desired event J ( π ) ≥ r − ε ′ occurs with probability at least 1 − δ ′ , and set L equal to r .
Revised: If the selected policy is π 2 , then we know the desired event J ( π ) ≥ r − ε ′ occurs with probability at least 1 − δ ′ , and set L equal to r .

Original: We can continue the bisection search until the accuracy is less than ε ′ , that is, U − L ≤ ε ′ , and the output value estimate is ˆ J ( π )
Revised: We can continue the bisection search until the accuracy is less than ε ′ , that is, U − L ≤ ε ′ , and the output value estimate is ˆ J ( π )

Original: =
Revised: =

Original: U + L 2 .
Revised: U + L 2 .

Original: If all desired events at each call occur, then we conclude that L − ε ′ ≤ J ( π ) ≤ U + ε ′ and thus | J ( π ) − ˆ J
Revised: If all desired events at each call occur, then we conclude that L − ε ′ ≤ J ( π ) ≤ U + ε ′ and thus | J ( π ) − ˆ J

Original: ( π ) | ≤ ε .
Revised: ( π ) | ≤ ε .

Original: The total number of OPS calls is at most m .
Revised: The total number of OPS calls is at most m .

Original: Setting δ ′
Revised: Setting δ ′

Original: =
Revised: =

Original: δ/m and applying a union bound, we can conclude that with probability at least 1 − δ , | J ( π ) − ˆ J ( π ) | ≤ ε .
Revised: δ/m and applying a union bound, we can conclude that with probability at least 1 − δ , | J ( π ) − ˆ J ( π ) | ≤ ε .

Original: Finally, since any ( ε, δ ) -sound OPE algorithm on the instance ( M, d b , π ) needs at least Ω( N OPE ( S, A, H, ε, 1 /δ ))
Revised: Finally, since any ( ε, δ ) -sound OPE algorithm on the instance ( M, d b , π ) needs at least Ω( N OPE ( S, A, H, ε, 1 /δ ))

Original: samples, the ( ε ′ , δ ′ )
Revised: samples, the ( ε ′ , δ ′ )

Original: -sound
Revised: -sound

Original: OPS algorithm needs at least Ω( N OPE ( S, A, H, ε, 1 /δ )) , or Ω( N OPE ( S, A, H, 3 ε/ 2 , 1 /mδ ′ ))
Revised: OPS algorithm needs at least Ω( N OPE ( S, A, H, ε, 1 /δ )) , or Ω( N OPE ( S, A, H, 3 ε/ 2 , 1 /mδ ′ ))

Original: samples for at least one of the instances ( M r , d b,r , Π) .
Revised: samples for at least one of the instances ( M r , d b,r , Π) .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Theorem 3 (OPS using importance sampling) .
Revised: Theorem 3 (OPS using importance sampling) .

Original: Suppose the data collection policy is uniformly random, that is, π b ( a | s ) = 1 /A for all ( s, a ) ∈ S × A , and | G i | ≤ V max almost surely.
Revised: Suppose the data collection policy is uniformly random, that is, π b ( a | s ) = 1 /A for all ( s, a ) ∈ S × A , and | G i | ≤ V max almost surely.

Original: Then the selection algorithm L that selects the policy with the highest IS estimate is ( ε, δ ) -sound with
Revised: Then the selection algorithm L that selects the policy with the highest IS estimate is ( ε, δ ) -sound with

Original: O ( A H V max ln ( | Π | /δ ) /ε 2 ) episodes.
Revised: O ( A H V max ln ( | Π | /δ ) /ε 2 ) episodes.

Original: Proof.
Revised: Proof.

Original: Since the policy is uniform random, we know | W i G i | <
Revised: Since the policy is uniform random, we know | W i G i | <

Original: A H V max almost surely.
Revised: A H V max almost surely.

Original: Moreover, the importance sampling estimator is unbiased, that is, E [ W i G i ]
Revised: Moreover, the importance sampling estimator is unbiased, that is, E [ W i G i ]

Original: =
Revised: =

Original: J ( π ) .
Revised: J ( π ) .

Original: Using the Bernstein’s inequality, we can show that the IS estimator satisfies
Revised: Using the Bernstein’s inequality, we can show that the IS estimator satisfies

Original: [Equation]
Revised: [Equation]

Original: for one candidate policy π k .
Revised: for one candidate policy π k .

Original: Using the union bound [Equation] we have
Revised: Using the union bound, we have

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: That is,
Revised: That is,

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: {+s 0 ss H  
+}
Revised: {+s 0 ss H  
+}

Revised: s H  1 s H  2 s H s o 1 s os oH  1 s oH s 1 ss oH  2 s oH … a 0 aa h 2 a h

Revised: Bernoulli reward

Original: For the variance term,
Revised: For the variance term,

Original: [Equation]
Revised: [Equation]

Original: The second inequality follows from the Cauchy-Schwarz inequality.
Revised: The second inequality follows from the Cauchy-Schwarz inequality.

Original: Therefore, if n > 32 A H V max ln (2 | Π | /δ ) /ε 2 , L is ( ε, δ ) -sound.
Revised: Therefore, if n > [Equation]

Original: Theorem 5 (Exponential lower bound on the sample complexity of OPE) .
Revised: Theorem 5 (Exponential lower bound on the sample complexity of OPE) .

Original: For any positive integers
Revised: For any positive integers

Original: S, A, H with S > 2 H and a pair ( ε, δ ) with 0 < ε ≤ (cid:112) 1 / 8 , δ ∈ (0 , 1) , any ( ε, δ ) -sound
Revised: S, A, H with S > 2 H and a pair ( ε, δ ) with 0 < ε ≤ (cid:112) 1 / 8 , δ ∈ (0 , 1) , any ( ε, δ ) -sound

Original: OPE algorithm needs at least Ω( A H ln (1 /δ )
Revised: OPE algorithm needs at least Ω( A H ln (1 /δ )

Original: /ε 2 ) episodes.
Revised: /ε 2 ) episodes.

Original: Proof.
Revised: Proof.

Original: We provide a proof which uses the construction from Xiao et al.
Revised: We provide a proof which uses the construction from Xiao et al.

Original: They provide the result for the offline RL problem with Gaussian rewards.
Revised: They provide the result for the offline RL problem with Gaussian rewards.

Original: Here we provide the result for OPE problem with Bernoulli rewards since we assume rewards are bounded to match Theorem 3.
Revised: Here we provide the result for OPE problem with Bernoulli rewards since we assume rewards are bounded to match Theorem 3.

Original: We can construct an MDP with S states, A actions and 2 H states.
Revised: We can construct an MDP with S states, A actions and 2 H states in Figure 3.

Original: Given any behavior policy π b , let a h = arg min a π b ( a | s h ) be the action that leads to the next state s h +1 from state s h , and all other actions lead to an absorbing state so h .
Revised: Given any behavior policy π b , let a h = arg min a π b ( a | s h ) be the action that leads to the next state s h +1 from state s h , and all other actions lead to an absorbing state s

Original:  Once we reach an absorbing state, the agent gets zero reward for all actions for the remainder of the episode.
Revised: oh . Once we reach an absorbing state, the agent gets zero reward for all actions for the remainder of the episode.

Original: The only nonzero reward is in the last state s H − 1 .
Revised: The only nonzero reward is in the last state s H − 1 .

Original: Consider a target policy that chooses a h for state s h for all h = 0 , . .
Revised: Consider a target policy that chooses a h for state s h for all h = 0 , . .

Original: , H − 1 , and two MDPs where the only difference between them is the reward distribution in s H − 1 .
Revised: , H − 1 , and two MDPs where the only difference between them is the reward distribution in s H − 1 .

Original: MDP 1 has Bernoulli distribution with mean 1 / 2 and MDP 2 has Bernoulli distribution with mean 1 / 2 − 2 ε .
Revised: MDP 1 has Bernoulli distribution with mean 1 / 2 and MDP 2 has Bernoulli distribution with mean 1 / 2 − 2 ε .

Original: Let P 1 denote the probability measure with respect to MDP 1 and P 2 denote the probability measure with respect to MDP 2.
Revised: Let P 1 denote the probability measure with respect to MDP 1 and P 2 denote the probability measure with respect to MDP 2.

Original: Let ˆ r denote the OPE estimate by an algorithm L .
Revised: Let ˆ r denote the OPE estimate by an algorithm L .

Original: Define an event E = { ˆ r < 12 − ε } .
Revised: Define an event E = { ˆ r < 12 − ε } .

Original: Then L is not ( ε, δ ) -sound
Revised: Then L is not ( ε, δ ) -sound

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: This is because L is not ( ε, δ ) -sound if either P 1 (ˆ r < 12 − ε ) ≥ δ or P 2 (ˆ r > 12 − ε ) ≥ δ .
Revised: This is because L is not ( ε, δ ) -sound if either P 1 (ˆ r < 12 − ε ) ≥ δ or P 2 (ˆ r > 12 − ε ) ≥ δ .

Original: Using the Bretagnolle–Huber inequality (See Theorem 14.2 of Lattimore & Szepesv ´ ari (2020)), we know
Revised: Using the Bretagnolle–Huber inequality (See Theorem 14.2 of Lattimore & Szepesv ´ ari (2020)), we know

Original: [Equation]
Revised: [Equation]

Original: By the chain rule for KL-divergence and the fact that P 1 and P 2 only differ in the reward for ( s H − 1 , a H − 1 ) , we have[Equation] A
Revised: By the chain rule for KL-divergence and the fact that P 1 and P 2 only differ in the reward for ( s H − 1 , a H − 1 ) , we have

Original: [Equation]
Revised: [Equation]

Original: The last inequality follows from − log (1 − 4 ε 2 ) ≤ 8 ε 2 if 4 ε 2 ≤ 1 / 

Revised: The last inequality follows from − log (1 − 4 ε 2 ) ≤ 8 ε 2 if 4 ε 2 ≤ 1 /

Original: (Krishnamurthy et al.,2016) and P 1 ( S ( i ) H − 1 = s H − 1 , A ( i ) H − 1 = a H − 1 ) < 1 /A H from the construction of the MDPs.
Revised: (Krishnamurthy et al.,

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original: which is larger than δ if n ≤ A H ln(1 / 4 δ ) / 8 ε 2 .
Revised: which is larger than δ if n ≤ A H ln(1 / 4 δ ) / 8 ε 2 .

Original: As a result, we need at least Ω( A H ln (1 /δ ) /ε 2 ) episodes.
Revised: As a result, we need at least Ω( A H ln (1 /δ ) /ε 2 ) episodes.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Assumption 1 (Approximation error) .
Revised: Assumption 1 (Approximation error) .

Original: For any policy π ∈ Π and h ∈
Revised: For any policy π ∈ Π and h ∈

Original: [ H ] , we assume the approximation error is bounded by ε apx , that is,
Revised: [ H ] , we assume the approximation error is bounded by ε apx , that is,

Original: [Equation]
Revised: [Equation]

Original: Assumption 2 (Optimization error) .
Revised: Assumption 2 (Optimization error) .

Original: Given a target value function g , we want to find the empirical risk minimizer
Revised: Given a target value function g , we want to find the empirical risk minimizer

Original: [Equation]
Revised: [Equation]

Original: We assume we have an optimization oracel that can return a value function ˆ f such that the optimization error is bounded, that is,
Revised: We assume we have an optimization oracel that can return a value function ˆ f such that the optimization error is bounded, that is,

Original: [Equation]
Revised: [Equation]

Original: Definition 2 (Rademacher complexity) .
Revised: Definition 1 (Rademacher complexity) .

Original: Given a function class F , let X = { x 1 , . . .
Revised: Given a function class F , let X = { x 1 , . . .

Original: , x n } denotes n fixed data points at horizon h following the distribution
Revised: , x n } denotes n fixed data points at horizon h following the distribution

Original: µ h , the empirical Rademacher complexity is defined as
Revised: µ h , the empirical Rademacher complexity is defined as

Original: [Equation]
Revised: [Equation]

Original: where the expectation is with respect to the Rademacher random variables σ i .
Revised: where the expectation is with respect to the Rademacher random variables σ i .

Original: The Rademacher complexity is defined as R µ h n ( F ) =
Revised: The Rademacher complexity is defined as R µ h n ( F ) =

Original: E [ R X ( F )] where the expectation is with respect to the n data points.
Revised: E [ R X ( F )] where the expectation is with respect to the n data points.

Original: Finally, to simply the notation, we define R µn ( F ) = max h ∈
Revised: Finally, to simply the notation, we define R µn ( F ) = max h ∈

Original: [ H ] R µ h n ( F ) as the maximum
Revised: [ H ] R µ h n ( F ) as the maximum

Original: Rademacher complexity over all horizons.
Revised: Rademacher complexity over all horizons.

Original: Lemma 1 (Excess risk bound, modification from Theorem 5.2 of Duan et al. (2021)) .
Revised: Lemma 1 (Excess risk bound, modification from Theorem 5.2 of Duan et al. (2021)) .

Original: Let q = ( q 0 , . .
Revised: Let q = ( q 0 , . .

Original: , q H − 1 ) be the output of FQE with n sample drawn from the data distribution µ h at each horizon h .
Revised: , q H − 1 ) be the output of FQE with n sample drawn from the data distribution µ h at each horizon h .

Original: Then we have, with probability 1 − δ , the following holds for all h = 0 , . . .
Revised: Then we have, with probability 1 − δ , the following holds for all h = 0 , . . .

Original: , H −
Revised: , H −

Original: [Equation]
Revised: [Equation]

Original: n for some constant c o > 0 .
Revised: n for some constant c o > 0 .

Original: Proof.
Revised: Proof.

Original: For each horizon h ∈ [ H ] , let q h be the output value function such that ˆ l ( q h , q h +1 ) ≤
Revised: For each horizon h ∈ [ H ] , let q h be the output value function such that ˆ l ( q h , q h +1 ) ≤

Original: ˆ l (ˆ q h , q h +1 ) + ε opt where ˆ q h is the empirical minimizer, that is, ˆ q h = arg min
Revised: ˆ l (ˆ q h , q h +1 ) + ε opt where ˆ q h is the empirical minimizer, that is, ˆ q h = arg min

Original: f ∈F l h ( f, q h +1 ) .
Revised: f ∈F l h ( f, q h +1 ) .

Original: It follows ˆ l ( q h , ˆ f h +1 ) ≤ ˆ l ( q † h , q h +1 ) + ε opt where q † h is the population minimizer, that is,
Revised: It follows ˆ l ( q h , ˆ f h +1 ) ≤ ˆ l ( q † h , q h +1 ) + ε opt where q † h is the population minimizer, that is,

Original: q † h = arg min f ∈F ∥ f − T q h +1 ∥ 2 ,µ h .
Revised: q † h = arg min f ∈F ∥ f − T q h +1 ∥ 2 ,µ h .

Original: Then follow the proof of Theorem 5.2 in Duan et al.
Revised: Then follow the proof of Theorem 5.2 in Duan et al.

Original: by takeing f h
Revised: by takeing f h

Original: = q h , we have
Revised: = q h , we have

Original: [Equation]
Revised: [Equation]

Original:  holds for all h = 0 , . .
Revised: n holds for all h = 0 , . .

Original: , H − 1 with probability at least 1 − δ .
Revised: , H − 1 with probability at least 1 − δ .

Original: Theorem 4 (Offline policy selection for well-covered policies) .
Revised: Theorem 4 (Offline policy selection for well-covered policies) .

Original: Suppose we have a function class
Revised: Suppose we have a function class

Original: F with Rademacher complexity R µn ( F ) , an approximation error ε apx , and an optimization error ε opt (see Appendix A.4 for the definitions).
Revised: F with Rademacher complexity R µn ( F ) , an approximation error ε apx , and an optimization error ε opt (see Appendix A.4 for the definitions).

Original: If there exists a non-negative value C such that Π ⊂ Π( C, µ, F ) , then the OPS algorithm L that selects the policy with the highest FQE estimate satisfiesPr
Revised: If there exists a non-negative value C such that Π ⊂ Π( C, µ, F ) , then the OPS algorithm L that selects the policy with the highest FQE estimate satisfies

Original: [Equation]
Revised: [Equation]

Original: for some constant c 0 .

Original: Proof.
Revised: Proof.

Original: First fix a policy π ∈ Π .
Revised: First fix a policy π ∈ Π .

Original: Let q = ( q 0 , . .
Revised: Let q = ( q 0 , . .

Original: , q H − 1 ) be the output of FQE with n sample drawn from the data distribution µ h at each horizon h , then ∥ q h − T
Revised: , q H − 1 ) be the output of FQE with n sample drawn from the data distribution µ h at each horizon h , then ∥ q h − T

Original: π q h +1 ∥ 2 d b ≤ ε opt + ε apx + c 0 HR µ h n ( F ) + c 0 H 2 (cid:113) log( H/δ ) n from Lemma 1 for some constant c 0 .
Revised: π q h +1 ∥ 2 d b ≤ ε opt + ε apx + c 0 HR µ h n ( F ) + c 0 H 2 (cid:113) log( H/δ ) n from Lemma 1 for some constant c 0 .

Original: By the definition of Π( C, µ, F ) , we know ∥ qh
Revised: By the definition of Π( C, µ, F ) , we know ∥ q

Original: [Equation]
Revised: [Equation]

Original: for any f ∈ F .
Revised: for any f ∈ F .

Original: Therefore, we have ∥
Revised: Therefore, we have ∥

Original: [Equation]
Revised: [Equation]

Original: We know
Revised: We know

Original: [Equation]
Revised: [Equation]

Original: Apply the same inequality recursively, we have
Revised: Apply the same inequality recursively, we have

Original: [Equation]
Revised: [Equation]

Original: The last inequality follows from the Jensen’s inequality.
Revised: The last inequality follows from the Jensen’s inequality.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: with probability 1 − δ .
Revised: with probability 1 − δ .

Original: The bound on FQE estimate holds for one policy in the candidate set.
Revised: The bound on FQE estimate holds for one policy in the candidate set.

Original: Using the union bound, the bound can hold for all policies.
Revised: Using the union bound, the bound can hold for all policies.

Original: Therefore, the OPS algorithm has error up to
Revised: Therefore, the OPS algorithm has error up to

Original: [Equation]
Revised: [Equation]

Original: ) with probability at least 1 − δ .
Revised: ) with probability at least 1 − δ .

Original: A. D ETAILS OF S ECTION  
Revised: A. D ETAILS OF S ECTION 4.

Original: By the performance difference lemma,
Revised: By the performance difference lemma,

Original: [Equation]
Revised: [Equation]

Original: where d π h is the state-action distribution at horizon h induced by policy π .
Revised: where d π h is the state-action distribution at horizon h induced by policy π .

Original: The first inequality follows because π h is greedy with respect to q h .
Revised: The first inequality follows because π h is greedy with respect to q h .

Original: We consider a state-action distribution β 0 that is induced by some policy, then
Revised: We consider a state-action distribution β 0 that is induced by some policy, then

Original: [Equation]
Revised: [Equation]

Original: where β 1 ( s ′ , a ′ ) = (cid:80) s,a β 0 ( s, a ) P ( s, a, s ′ )
Revised: where β 1 ( s ′ , a ′ ) = (cid:80) s,a β 0 ( s, a ) P ( s, a, s ′ )

Original: I { a ′ =
Revised: I { a ′ =

Original: arg max a ′′ ∈A [Equation] q ∗
Revised: arg max a ′′ ∈A ( q ∗

Original: [Equation]
Revised: [Equation]

Original: s ′ , a

Original: [Equation] ) − q [Equation] s ′ , a ′′ )) 2 } is also induced by some policy.
Revised: s ′ , a ′′ ) − q 1 ( s ′ , a ′′ )) 2 } is also induced by some policy.

Original: The first inequality follows by the fact that q ∗ is the fixed point of the operator T .
Revised: The first inequality follows by the fact that q ∗ is the fixed point of the operator T .

Original: We can recursively apply the same process for ∥ q ∗ h − q h ∥ 2 ,β h , h > 0 , and we can get
Revised: We can recursively apply the same process for ∥ q ∗ h − q h ∥ 2 ,β h , h > 0 , and we can get

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Now we show that
Revised: Now we show that

Original: [Equation]
Revised: [Equation]

Original: n holds with probability at least 1 − δ for some constant c 0 .
Revised: n holds with probability at least 1 − δ for some constant c 0 .

Original: Proof.
Revised: Proof.

Original: Using Hoeffding’s inequality and the union bound, with probability at least 1 − δ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)n
Revised: Using Hoeffding’s inequality and the union bound, with probability at least 1 − δ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)n

Original: [Equation]
Revised: [Equation]

Original: n for all candidate value function q i =
Revised: n for all candidate value function q i =

Original: ( q i, 1 , . . .
Revised: ( q i, 1 , . . .

Original: , q i,H − 1 ) , i = 1 , . . .
Revised: , q i,H − 1 ) , i = 1 , . . .

Original: , | Π | and h ∈ [ H ] .
Revised: , | Π | and h ∈ [ H ] .

Original: Let k be the index with the lowest empirical Bellman error, that is,
Revised: Let k be the index with the lowest empirical Bellman error, that is,

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: We provide the experimental details in this section.
Revised: We provide the experimental details in this section.

Original: Generating candidate policies.
Revised: Generating candidate policies.

Original: To generate a set of candidate policies, we run CQL with different hyperparameter configurations on a batch of data with 300 episodes collected with an ε -greedy policy with respect to the optimal policy where ε = 0 .
Revised: To generate a set of candidate policies, we run CQL with different hyperparameter configurations on a batch of data with 300 episodes collected with an ε -greedy policy with respect to the optimal policy where ε = 0 .

Original: The hyperparameter configuration includes:
Revised: The hyperparameter configuration includes:

Original: • Learning rate ∈ { 0 . 001 , 0 . 0003 , 0 . 

Revised: • Learning rate ∈ { 0 . 001 , 0 . 0003 , 0 .

Original: } • Network hidden layer size ∈ { 128 , 256 , 512 } • Regularization coefficient ∈ { 1 . 0 , 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0 } • Iterations of CQL ∈ { 100 , 200 }
Revised: } • Network hidden layer size ∈ { 128 , 256 , 512 } • Regularization coefficient ∈ { 1 . 0 , 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0 } • Iterations of CQL ∈ { 100 , 200 }

Original: That is, we have 90 candidate policies for Acrobot.
Revised: That is, we have 90 candidate policies for Acrobot.

Original: For Cartpole, CQL with many hyperparameter configurations can generate an optimal policy (reaching a return of 200 ) so the selection is sufficiently easy that OPS algorithms using both FQE and BE achieve zero regret.
Revised: For Cartpole, CQL with many hyperparameter configurations can generate an optimal policy (reaching a return of 200 ) so the selection is sufficiently easy that OPS algorithms using both FQE and BE achieve zero regret.

Original: In order to make the result more meaningful, we remove all the policies that are optimal from the candidate set.
Revised: In order to make the result more meaningful, we remove all the policies that are optimal from the candidate set.

Original: That results in 67 candidate policies for Cartpole.
Revised: That results in 67 candidate policies for Cartpole.

Original: Generating offline data for OPS.
Revised: Generating offline data for OPS.

Original: To generate data for offline policy selection, we use three different data distributions: (a) a data distribution collected by running the mixture of all candidate policies.
Revised: To generate data for offline policy selection, we use three different data distributions: (a) a data distribution collected by running the mixture of all candidate policies.

Original: As a result, the data distribution covers all candidate policies well; (b) a data distribution collected by running the mixture of all candidate policies and an ε -greedy optimal policy that provides more diverse trajectories than (a); (c) a data distribution collected by a deterministic optimal policy.
Revised: As a result, the data distribution covers all candidate policies well; (b) a data distribution collected by running the mixture of all candidate policies and an ε -greedy optimal policy that provides more diverse trajectories than (a); (c) a data distribution collected by a deterministic optimal policy.

Original: To perform experiment with multiple runs.
Revised: To perform experiment with multiple runs.

Original: To perform experiments with multiple runs, we fix the offline data and the candidate policies and only resample the offline data for OPS.
Revised: To perform experiments with multiple runs, we fix the offline data and the candidate policies and only resample the offline data for OPS.

Original: This better reflects the theoretical result that the randomness is from resampling the data for an OPS algorithm.
Revised: This better reflects the theoretical result that the randomness is from resampling the data for an OPS algorithm.

Original: In our experiments, we use 5 runs and report the average regret with one standard error.
Revised: In our experiments, we use 5 runs and report the average regret with one standard error.

Revised: Since the variability across runs is not large, we find using 5 runs is enough.

Revised: Random selection baseline We include a random selection baseline that randomly chooses k policies given k .

Revised: Since the random selection algorithm has very high variance within one run, we first compute the expected regret of random selection by performing 2000 times random selection within each run, and report the average regret over runs.

Original: Practical considerations for FQE implementation.
Revised: Practical considerations for FQE implementation.

Original: As mentioned in the paper, we fix the function class as a two layer neural network model with hidden size 256 and tune the optimization hyperparameters.
Revised: As mentioned in the paper, we fix the function class as a two layer neural network model with hidden size 256 and tune the optimization hyperparameters.

Original: We use the Adam optimizer with learning rate selected from the set { 0 . 001 , 0 . 0003 , 0 . 0001 , 0 . 00003 } for 200 epochs.
Revised: We use the Adam optimizer with learning rate selected from the set { 0 . 001 , 0 . 0003 , 0 . 0001 , 0 . 00003 } for 200 epochs.

Original: We selected the hyperparameter configuration that resulted in a value function with the smallest RMSTD error evaluated on a separate validation dataset.
Revised: We selected the hyperparameter configuration that resulted in a value function with the smallest RMSTD error evaluated on a separate validation dataset.

Original: To avoid selecting an uncovered policy, we assign a low estimate to policies when FQE diverges to a large value.
Revised: To avoid selecting an uncovered policy, we assign a low estimate to policies when FQE diverges to a large value.