Original: Robust On-Policy Sampling for Data-Efﬁcient Policy Evaluation 
Revised: Robust On-Policy Sampling for Data-Efﬁcient Policy Evaluation in Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Reinforcement learning ( RL ) algorithms are often categorized as either on-policyor off-policy depending on whether they use data from a target policy of interestor from a different behavior policy.
Revised: Reinforcement learning ( RL ) algorithms are often categorized as either on-policy or off-policy depending on whether they use data from a target policy of interest or from a different behavior policy.

Original: In this paper, we study a subtle distinction between on-policy data and on-policy sampling in the context of the RL sub-problemof policy evaluation.
Revised: In this paper, we study a subtle distinction between on-policy data and on-policy sampling in the context of the RL sub-problem of policy evaluation.

Original: We observe that on-policy sampling may fail to match theexpected distribution of on-policy data after observing only a ﬁnite number of trajectories and this failure hinders data-efﬁcient policy evaluation.
Revised: We observe that on-policy sampling may fail to match the expected distribution of on-policy data after observing only a ﬁnite number of trajectories and this failure hinders data-efﬁcient policy evaluation.

Original: Towards improveddata-efﬁciency, we show how non-i.i.d., off-policy sampling can produce data thatmore closely matches the expected on-policy data distribution and consequentlyincreases the accuracy of the Monte Carlo estimator for policy evaluation.
Revised: Towards improved data-efﬁciency, we show how non-i.i.d., off-policy sampling can produce data that more closely matches the expected on-policy data distribution and consequently increases the accuracy of the Monte Carlo estimator for policy evaluation.

Original: We introduce a method called Robust On-policy Sampling and demonstrate theoreticallyand empirically that it produces data that converges faster to the expected on-policydistribution compared to on-policy sampling.
Revised: We introduce a method called Robust On-Policy Sampling and demonstrate theoretically and empirically that it produces data that converges faster to the expected on-policy distribution compared to on-policy sampling.

Original: Empirically, we show that this fasterconvergence leads to lower mean squared error policy value estimates.
Revised: Empirically, we show that this faster convergence leads to lower mean squared error policy value estimates.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning ( RL ) algorithms are often categorized using the dichotomy of on-policyversus off-policy.
Revised: Reinforcement learning ( RL ) algorithms are often categorized using the dichotomy of on-policy versus off-policy.

Original: On-policy algorithms learn about a particular target policy using data collectedby behaving according to the target policy.
Revised: On-policy algorithms learn about a particular target policy using data collected by behaving according to the target policy.

Original: Off-policy algorithms use data collected by behavingaccording to a different behavior policy.
Revised: Off-policy algorithms use data collected by behaving according to a different behavior policy.

Original: We study a subtle distinction between on-policy data versuson-policy sampling as a step towards more data-efﬁcient RL algorithms.
Revised: We study a subtle distinction between on-policy data versus on-policy sampling as a step towards more data-efﬁcient RL algorithms.

Original: To better understand thisdistinction, consider a simple example.
Revised: To better understand this distinction, consider a simple example.

Original: In this example, a certain target policy repeatedly visits a statein which it takes action A with probability 0 .
Revised: In this example, a certain target policy repeatedly visits a state in which it takes action A with probability 0 .

Original: 2 and action B with probability 0 .
Revised: 2 and action B with probability 0 .

Original: Under on-policysampling, after ﬁve visits to this state, we might actually observe action A 2 times and action B3 times instead of the expected 1 and 4 times.
Revised: Under on-policy sampling, after ﬁve visits to this state, we might actually observe action A 2 times and action B 3 times instead of the expected 1 and 4 times.

Original: Alternatively, we could collect data off-policy bydeterministically tracking the expected target policy action proportions resulting in  the exact expectedfrequencies.
Revised: Alternatively, we could collect data off-policy by deterministically tracking the expected target policy action proportions; doing so results in observing the exact expected action frequencies.

Original: Though the latter case uses off-policy sampling, it produces data that is arguably moreon-policy than the data produced by on-policy sampling.
Revised: Though the latter case uses off-policy sampling, it produces data that is arguably more on-policy than the data produced by on-policy sampling.

Original: In this paper, we study the distinction between on-policy sampling and on-policy data in the contextof the RL sub-problem of policy evaluation [Zinkevich et al., 2006].
Revised: In this paper, we study the distinction between on-policy sampling and on-policy data in the context of the RL sub-problem of policy evaluation [Zinkevich et al., 2006].

Original: In policy evaluation, we are givenan evaluation policy and asked to estimate the expected return that would be accrued when runningthe evaluation policy on a task of interest.
Revised: In policy evaluation, we are given an evaluation policy and asked to estimate the expected return that would be accrued when running the evaluation policy on a task of interest.

Original: This problem is important for high conﬁdence deploymentof RL -trained policies.
Revised: This problem is important for high conﬁdence deployment of RL -trained policies.

Original: In RL applications, such as robotics, data-efﬁcient policy evaluation is ofthe utmost importance – we desire the most accurate estimate with minimal collected data.
Revised: In RL applications, such as robotics, data-efﬁcient policy evaluation is of the utmost importance – we desire the most accurate estimate with minimal collected data.

Original: Whilemuch research has gone into how to most efﬁciently use a set of already collected data, i.e., theoff-policy policy evaluation problem, an implicit assumption in the RL community is that on-policydata is preferred to off-policy data when available.
Revised: While much research has gone into how to most efﬁciently use a set of already collected data, i.e., the off-policy policy evaluation problem [Jiang and Li, 2016, Thomas and Brunskill, 2016], an implicit assumption in the RL community is that on-policy data is preferred to off-policy data when available.

Original: When data can be collected on-policy, we can usethe Monte Carlo estimator which computes a mean return estimate using trajectories sampled i.i.d.by running the evaluation policy.
Revised: When data can be collected on-policy, we can use the Monte Carlo estimator which computes a mean return estimate using trajectories sampled i.i.d. by running the evaluation policy.

Original: In the limit, with inﬁnite trajectories, the empirical proportion of each trajectory will converge to its true probability under the evaluation policy and the estimate willconverge to the true expected return.
Revised: In the limit, with inﬁnite trajectories, the empirical proportion of each trajectory will converge to its true probability under the evaluation policy and the estimate will converge to the true expected return.

Original: However, for any ﬁnite sample-size, the empirical proportion ofeach trajectory will likely fail to match the true probability and the estimate will have error.
Revised: However, for any ﬁnite sample-size, the empirical proportion of each trajectory will likely fail to match the true probability and the estimate will have error.

Original: Suchsampling error is an inevitable feature of i.i.d.
Revised: Such sampling error is an inevitable feature of i.i.d.

Original: sampling.
Revised: sampling.

Original: The probability of each new trajectory isunaffected by the trajectories occurring in the past and thus the only way to ensure the empiricaldistribution matches the true probability is to sample a large enough data set.
Revised: The probability of each new trajectory is unaffected by the trajectories occurring in the past and thus the only way to ensure the empirical distribution matches the true probability is to sample a large enough data set.

Original: That is, it is only in thelimit that on-policy sampling produces exactly on-policy data.
Revised: That is, it is only in the limit that on-policy sampling produces exactly on-policy data.

Original: The observations made so far raise the question: “can non-i.i.d., off-policy trajectory sampling causethe empirical distribution of trajectories to converge to the expected on-policy distribution faster?"
Revised: The observations made so far raise the question: “can non-i.i.d., off-policy trajectory sampling cause the empirical distribution of trajectories to converge to the expected on-policy distribution faster?

Original: Weanswer this question afﬁrmatively by introducing a method that adapts the data collecting behaviorpolicy to consider what data has already been collected when selecting future actions.
Revised: " We answer this question afﬁrmatively by introducing a method that adapts the data collecting behavior policy to consider what data has already been collected when selecting future actions.

Original: We callthis method Robust On-policy Sampling ( ROS )  since the empirical distribution of data it producesconverges faster to the expected on-policy trajectory distribution compared to standard on-policysampling.
Revised: We call this method Robust On-Policy Sampling ( ROS ) 1 since the empirical distribution of data it produces converges faster to the expected on-policy trajectory distribution compared to standard on-policy sampling.

Original: We give a theoretical result supporting this claim and then conﬁrm our theory with policyevaluation experiments in ﬁnite and continuous-valued state- and action-space domains showing 1)
Revised: We give a theoretical result supporting this claim and then conﬁrm our theory with policy evaluation experiments in ﬁnite and continuous-valued state- and action-space domains showing 1)

Original: ROS reduces sampling error in ﬁnite datasets and 2) consequently lowers the MSE of policy valueestimates compared to i.i.d.
Revised: ROS reduces sampling error in ﬁnite datasets and 2) consequently lowers the MSE of policy value estimates compared to i.i.d.

Original: on-policy sampling.
Revised: on-policy sampling.

Original: Our paper contributes to the ﬁeld of RL on two fronts.
Revised: Our paper contributes to the ﬁeld of RL on two fronts.

Original: On one front, we introduce a practical methodfor data collection and demonstrate empirically that it leads to more accurate policy evaluationcompared to on-policy sampling.
Revised: On one front, we introduce a practical method for data collection and demonstrate empirically that it leads to more accurate policy evaluation compared to on-policy sampling.

Original: Simultaneously, our work examines nuance in the on-policy versusoff-policy dichotomy.
Revised: Simultaneously, our work examines nuance in the on-policy versus off-policy dichotomy.

Original: A better understanding of this nuance opens up the possibility of designingnew data collection procedures to improve the data efﬁciency of any RL algorithm that relies uponon-policy data.
Revised: A better understanding of this nuance opens up the possibility of designing new data collection procedures to improve the data efﬁciency of any RL algorithm that relies upon on-policy data.

Original:  Related Work
Revised:  Related Work

Original: Data collection is a fundamental part of the RL problem.
Revised: Data collection is a fundamental part of the RL problem.

Original: The most widely studied data collectionproblem is the question of how an agent should explore its environment to learn an optimal policy [Ostrovski et al., 2017, Tang et al., 2017].
Revised: The most widely studied data collection problem is the question of how an agent should explore its environment to learn an optimal policy [Schäfer et al., 2022, Ostrovski et al., 2017, Tang et al., 2017].

Original: In contrast to these approaches, our work focuses on thequestion of how an agent should collect data to evaluate a ﬁxed policy.
Revised: In contrast to these approaches, our work focuses on the question of how an agent should collect data to evaluate a ﬁxed policy.

Original: When given a choice of howto collect data for policy evaluation, on-policy data collection is generally preferable to off-policydata collection [Sutton and Barto, 1998].
Revised: When given a choice of how to collect data for policy evaluation, on-policy data collection is generally preferable to off-policy data collection [Sutton and Barto, 1998].

Original: Notable exceptions are adaptive importance sampling ( AIS )methods [Hanna et al., 2017, Oosterhuis and de Rijke, 2020,  Ciosek and Whiteson, 2017, Bouchardet al., 2016, Frank et al., 2008] and quasi-Monte Carlo methods [Arnold et al., 2022].
Revised: Notable exceptions are adaptive importance sampling ( AIS ) methods [Oosterhuis and de Rijke, 2020, Hanna et al., 2017, Ciosek and Whiteson, 2017, Bouchard et al., 2016, Frank et al., 2008] and quasi-Monte Carlo methods [Arnold et al., 2022].

Original: Both these AIS methods and the Quasi-Monte Carlo method of Arnold et al.
Revised: Both these AIS methods and the Quasi-Monte Carlo method of Arnold et al.

Original: [2022] lower variance in estimatescomputed with future samples while our method lowers the total error in the estimate computed fromboth past and future samples.
Revised: [2022] lower variance in estimates computed with future samples while our method lowers the total error in the estimate computed from both past and future samples.

Original: In one of our experiments, we consider a setting where we already have some data (collected offpolicy) and must decide how to collect additional data for policy evaluation.
Revised: In one of our experiments, we consider a setting where we already have some data (collected offpolicy) and must decide how to collect additional data for policy evaluation.

Original: This problem has beenpreviously studied in the bandit literature [Tucker and Joachims, 2022] or when there are only a ﬁnitenumber of policies that could be ran [Konyushkova et al., 2021].
Revised: This problem has been previously studied in the bandit literature [Tucker and Joachims, 2022] or when there are only a ﬁnite number of policies that could be ran [Konyushova et al., 2021].

Original: These prior works also show thaton-policy data collection is a sub-optimal choice.
Revised: These prior works also show that on-policy data collection is a sub-optimal choice.

Original: They differ from (and are complementary to) ourwork in that they still use i.i.d.
Revised: They differ from (and are complementary to) our work in that they still use i.i.d.

Original: sampling for data collection whereas we show how non-independentsampling can be used to produce data that more closely matches a desired distribution.
Revised: sampling for data collection whereas we show how non-independent sampling can be used to produce data that more closely matches a desired distribution.

Original: The method we introduce in this paper is motivated by the idea of decreasing sampling error in allcollected data.
Revised: The method we introduce in this paper is motivated by the idea of decreasing sampling error in all collected data.

Original: Previous work has considered how sampling error can be reduced after data collectionby re-weighting the obtained samples.
Revised: Previous work has considered how sampling error can be reduced after data collection by re-weighting the obtained samples.

Original: For example, Hanna et al.
Revised: For example, Hanna et al.

Original: [2021] show how importancesampling with an estimated behavior policy can lower sampling error and lead to more accurate policyevaluation.
Revised: [2021] show how importance sampling with an estimated behavior policy can lower sampling error and lead to more accurate policy evaluation.

Original: Similar methods have also been studied for policy evaluation in multi-armed bandits [Liet al., 2015, Narita et al., 2019] and temporal-difference learning [Pavse et al., 2020].
Revised: Similar methods have also been studied for policy evaluation in multi-armed bandits [Narita et al., 2019, Li et al., 2015] and temporal-difference learning [Pavse et al., 2020].

Original: These priorworks assume data is available a priori and ignore the question of how to collect it when unavailable.
Revised: These prior works assume data is available a priori and ignore the question of how to collect it when unavailable.

Original: Finally, the idea of adapting the sampling distribution , (i.e., behavior policy) has analogs outsideof policy evaluation in Markov decision processes.
Revised: Finally, the idea of adapting the sampling distribution , (i.e., behavior policy) has analogs outside of policy evaluation in Markov decision processes.

Original: O’Hagan [1987] identiﬁes ﬂaws with i.i.d.sampling for Monte Carlo estimation that motivate taking past samples into account.
Revised: O’Hagan [1987] identiﬁes ﬂaws with i.i.d. sampling for Monte Carlo estimation that motivate taking past samples into account.

Original: Rasmussen and
Revised: Rasmussen and

Original: Ghahramani [2003] use Gaussian processes to represent uncertainty in an expectation to be evaluatedand use this uncertainty to guide future sample generation.
Revised: Ghahramani [2003] use Gaussian processes to represent uncertainty in an expectation to be evaluated and use this uncertainty to guide future sample generation.

Revised: Concurrent to this work, Mukherjee et al.

Original: We are unaware of any adaptations ofthese ideas for  policy evaluation in RL .
Revised: [2022] introduced an uncertainty aware method for data collection in policy evaluation that can be seen as an adaptation of these ideas to the RL .

Original:  Preliminaries
Revised:  Preliminaries

Original: In this section, we introduce notation, formalize the policy evaluation problem, and introduce the
Revised: In this section, we introduce notation, formalize the policy evaluation problem, and introduce the

Original: Monte Carlo estimator for policy evaluation.
Revised: Monte Carlo estimator for policy evaluation.

Original: 3.1 Notation
Revised: 3.1 Notation

Original: We assume the environment is a ﬁnite-horizon, episodic Markov decision process ( MDP ) with stateset S , action set A , transition function, P : S ×
Revised: We assume the environment is a ﬁnite-horizon, episodic Markov decision process ( MDP ) with state set S , action set A , transition function, P : S ×

Original: A × S → [0 , 1] , reward function R : S ×
Revised: A × S → [0 , 1] , reward function R : S ×

Original: A → R ,discount factor γ , maximum horizon l , and initial state distribution d 0 [Puterman, 2014].
Revised: A → R , discount factor γ , maximum horizon l , and initial state distribution d 0 [Puterman, 2014].

Original: We assumethat S and A are ﬁnite though our empirical analysis considers both settings.
Revised: We assume that S and A are ﬁnite though our empirical analysis considers both settings.

Original: We assume that thetransition and reward functions are unknown.
Revised: We assume that the transition and reward functions are unknown.

Original: A policy, π : S ×
Revised: A policy, π : S ×

Original: A → [0 , 1] , is a function mappingstates and actions to probabilities.
Revised: A → [0 , 1] , is a function mapping states and actions to probabilities.

Original: We use π ( a | s ) :
Revised: We use π ( a | s ) :

Original: = π ( s, a ) to denote the conditional probability ofaction a given state s and P ( s (cid:48) | s, a ) :
Revised: = π ( s, a ) to denote the conditional probability of action a given state s and P ( s (cid:48) | s, a ) :

Original: = P ( s, a, s (cid:48) ) to denote the conditional probability of state s (cid:48)given state s and action a .
Revised: = P ( s, a, s (cid:48) ) to denote the conditional probability of state s (cid:48) given state s and action a .

Original: Since we assume a ﬁnite-horizon, we assume the state deﬁnition implicitlyincludes temporal information [Agarwal et al.].
Revised: Since we assume a ﬁnite-horizon, we assume the state deﬁnition implicitly includes temporal information [Agarwal et al., 2022].

Original: Let h :
Revised: Let h :

Original: =
Revised: =

Original: ( s 0 , a 0 , r 0 , s 1 , . .
Revised: ( s 0 , a 0 , r 0 , s 1 , . .

Original: , s l − 1 , a l − 1 , r l − 1 ) be a trajectory and g ( h ) :
Revised: , s l − 1 , a l − 1 , r l − 1 ) be a trajectory and g ( h ) :

Original: = (cid:80) l − 1 t =

Revised: = (cid:80) l − 1 t =

Original: γ t r t be the discounted return of h .
Revised: γ t r t be the discounted return of h .

Original: Any policy induces a distribution over trajectories, Pr( h | π ) .
Revised: Any policy induces a distribution over trajectories, Pr( h | π ) .

Original: We deﬁne thevalue of a policy, v ( π ) , as the expected discounted return when sampling a trajectory by followingpolicy π : v ( π ) :=
Revised: We deﬁne the value of a policy, v ( π ) , as the expected discounted return when sampling a trajectory by following policy π : v ( π ) :

Original:  E [ g ( H ) | H ∼ π ] = (cid:80)
Revised: = E [ g ( H ) | H ∼ π ] = (cid:80)

Original: h Pr( h | π ) g ( h ) where H is a random variable representinga trajectory and H ∼ π denotes sampling H by running π in the given environment.
Revised: h Pr( h | π ) g ( h ) where H is a random variable representing a trajectory and H ∼ π denotes sampling H by running π in the given environment.

Original: 3.2 Policy Evaluation
Revised: 3.2 Policy Evaluation

Original: In the policy evaluation problem, we are given an evaluation policy , π e , for which we would like toestimate v ( π e ) .
Revised: In the policy evaluation problem, we are given an evaluation policy , π e , for which we would like to estimate v ( π e ) .

Original: Conceptually, algorithms for policy evaluation involve two steps: collecting data (orreceiving previously collected data) and computing an estimate from that data.
Revised: Conceptually, algorithms for policy evaluation involve two steps: collecting data (or receiving previously collected data) and computing an estimate from that data.

Original: We assume that datais collected by running a policy which we call the behavior policy .
Revised: We assume that data is collected by running a policy which we call the behavior policy .

Original: If the behavior policy is the sameas the evaluation policy data collection is on-policy ; otherwise it is off-policy .
Revised: If the behavior policy is the same as the evaluation policy data collection is on-policy ; otherwise it is off-policy .

Original: Whether on-policy oroff-policy, we assume the data collection process produces a set of trajectories, D :=
Revised: Whether on-policy or off-policy, we assume the data collection process produces a set of trajectories, D :=

Original: { H i } ni =1 andwrite D ∼ π to denote collecting these trajectories by running policy π .
Revised: { H i } ni =1 and write D ∼ π to denote collecting these trajectories by running policy π .

Original: The ﬁnal value estimate isthen computed by a policy evaluation estimator ( PE ) that maps the set of trajectories to a scalar-valuedestimate of v ( π e ) .
Revised: The ﬁnal value estimate is then computed by a policy evaluation estimator ( PE ) that maps the set of trajectories to a scalar-valued estimate of v ( π e ) .

Original: Following earlier work in policy evaluation (e.g., [Thomas and Brunskill, 2016,
Revised: Following earlier work in policy evaluation (e.g., [Thomas and Brunskill, 2016,

Original: Jiang and Li, 2016]), we set our goal to be policy evaluation with low mean squared error ( MSE ):
Revised: Jiang and Li, 2016]), we set our goal to be policy evaluation with low mean squared error ( MSE ):

Original: [Equation]
Revised: [Equation]

Original: (1) where π b is the behavior policy that is run to collect D and PE is a generic policy evaluation estimator.
Revised: (1) where π b is the behavior policy that is run to collect D and PE is a generic policy evaluation estimator.

Original: 3.3 Monte Carlo Policy Evaluation
Revised: 3.3 Monte Carlo Policy Evaluation

Original: Perhaps the most fundamental, model-free policy evaluation method is the Monte-Carlo ( MC )estimator.
Revised: Perhaps the most fundamental, model-free policy evaluation method is the Monte-Carlo ( MC ) estimator.

Original: Given a data set, D , of n trajectories, the Monte Carlo estimate, MC( D ) , is the meanreturn over D :
Revised: Given a data set, D , of n trajectories, the Monte Carlo estimate, MC( D ) , is the mean return over D :

Original: [Equation]
Revised: [Equation]

Original: where Pr( h | D ) denotes the empirical probability of h , i.e. how often h appears in D .
Revised: where Pr( h | D ) denotes the empirical probability of h , i.e. how often h appears in D .

Original: If trajectories in D are collected i.i.d.
Revised: If trajectories in D are collected i.i.d.

Original: by running π e (i.e., on-policy sampling), the Monte Carloestimator is unbiased and consistent assuming g ( h ) is bounded [Sen and Singer, 1993].
Revised: by running π e (i.e., on-policy sampling), the Monte Carlo estimator is unbiased and consistent assuming g ( h ) is bounded [Sen and Singer, 1993].

Original: However,this method can have high variance as on-policy sampling may require many trajectories for theempirical trajectory distribution
Revised: However, this method can have high variance as on-policy sampling may require many trajectories for the empirical trajectory distribution

Original: Pr( h | D ) to accurately approximate Pr( h | π e ) .
Revised: Pr( h | D ) to accurately approximate Pr( h | π e ) .

Original: Since on-policysampling collects each trajectory i.i.d.
Revised: Since on-policy sampling collects each trajectory i.i.d.

Original: , it relies on the law of large numbers for an accurate weightingon each possible return.
Revised: , it relies on the law of large numbers for an accurate weighting on each possible return.

Original: We call error between Pr( h | D ) and Pr( h | π e ) sampling error .
Revised: We call error between Pr( h | D ) and Pr( h | π e ) sampling error .

Original:  Data-Conditioned Monte Carlo Estimates
Revised:  Data-Conditioned Monte Carlo Estimates

Original: In this section, we motivate how an estimator that uses on-policy data can beneﬁt from off-policysampling.
Revised: In this section, we motivate how an estimator that uses on-policy data can beneﬁt from off-policy sampling.

Original: Speciﬁcally, we consider the Monte Carlo estimator and suppose that we have already collected a data set, D 1 , of trajectories.
Revised: Speciﬁcally, we consider the Monte Carlo estimator and suppose that we have already collected a data set, D 1 , of trajectories.

Original: We now wish to collect an additional set of trajectories, D 2 ,and compute the Monte Carlo estimate with the set D 1 ∪ D 2 .
Revised: We now wish to collect an additional set of trajectories, D 2 , and compute the Monte Carlo estimate with the set D 1 ∪ D 2 .

Original: Note that D 1 is a ﬁxed set (the trajectoriesalready observed) while D 2 is a random variable (the trajectories yet to be observed).
Revised: Note that D 1 is a ﬁxed set (the trajectories already observed) while D 2 is a random variable (the trajectories yet to be observed).

Original: How should
Revised: How should

Original: D 2 be collected for minimal MSE policy evaluation with the Monte Carlo estimator?
Revised: D 2 be collected for minimal MSE policy evaluation with the Monte Carlo estimator?

Original: Our analysisin this section suggests that i.i.d.
Revised: Our analysis in this section suggests that i.i.d.

Original: sampling of trajectories with π e may be a sub-optimal choice.
Revised: sampling of trajectories with π e may be a sub-optimal choice.

Original: In this setting, the Monte Carlo estimator using D 1 ∪ D 2 can be written as:
Revised: In this setting, the Monte Carlo estimator using D 1 ∪ D 2 can be written as:

Original: [Equation]
Revised: [Equation]

Original: where n D 1 and n D 2 are the number of trajectories in D 1 and D 2 , respectively and n = n D 1 + n D 2 .
Revised: where n D 1 and n D 2 are the number of trajectories in D 1 and D 2 , respectively and n = n D 1 + n D 2 .

Original: 144 We refer to (3) as the data-conditioned Monte Carlo estimator .
Revised: We refer to (3) as the data-conditioned Monte Carlo estimator .

Original: Viewing the Monte Carlo estimator as a sum between a ﬁxed quantity and a random quantity changeshow we view the statistical properties of the estimator.
Revised: Viewing the Monte Carlo estimator as a sum between a ﬁxed quantity and a random quantity changes how we view the statistical properties of the estimator.

Original: For instance, while the Monte Carlo estimatoris known to be unbiased under on-policy sampling, its data-conditioned estimate is biased as shownin the following proposition.
Revised: For instance, while the Monte Carlo estimator is known to be unbiased under on-policy sampling, its data-conditioned estimate is biased as shown in the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: The data conditioned Monte Carlo estimator is biased under on-policy sampling of
Revised: The data conditioned Monte Carlo estimator is biased under on-policy sampling of

Original: D 2 unless MC( D 1 ) = v ( π e ) or D 1 = ∅ .
Revised: D 2 unless MC( D 1 ) = v ( π e ) or D 1 = ∅ .

Original: That is:
Revised: That is:

Original: [Equation]
Revised: [Equation]

Original: = v ( π e ) .

Original: Proof.
Revised: Proof.

Original: See Appendix A.
Revised: See Appendix A.

Original: Remark 1.
Revised: Remark 1.

Original: Proposition 1 holds even if D 1 was collected under on-policy sampling as well.
Revised: Proposition 1 holds even if D 1 was collected under on-policy sampling as well.

Original: When
Revised: When

Original: D 1 was collected under on-policy sampling then the Monte Carlo estimator is unbiased consideringall possible realizations of D 1 .
Revised: D 1 was collected under on-policy sampling then the Monte Carlo estimator is unbiased considering all possible realizations of D 1 .

Original: However, once the trajectories in D 1 are ﬁxed, it no longer matterswhat others values they could have taken.
Revised: However, once the trajectories in D 1 are ﬁxed, it no longer matters what others values they could have taken.

Original: Can we reduce the bias of the data-conditioned Monte Carlo estimator by collecting D 2 with apolicy that is different than π e ?
Revised: Can we reduce the bias of the data-conditioned Monte Carlo estimator by collecting D 2 with a policy that is different than π e ?

Original: We conclude this section with an example showing that we can.
Revised: We conclude this section with an example showing that we can.

Original: Consider a one-step MDP with one state, s , and two actions, a 0 and a 1 .
Revised: Consider a one-step MDP with one state, s , and two actions, a 0 and a 1 .

Original: The return following a 0 is2 and the return following a 1 is 4 .
Revised: The return following a 0 is 2 and the return following a 1 is 4 .

Original: The evaluation policy is π e ( a 0 | s ) =
Revised: The evaluation policy is π e ( a 0 | s ) =

Original: π e ( a 1 | s ) = 0 .
Revised: π e ( a 1 | s ) = 0 .

Original: Supposethat, after sampling 3 trajectories, D 1 contains two of { s, a 0 , 2 } and one occurrence of { s, a 1 , 4 } .
Revised: Suppose that, after sampling 3 trajectories, D 1 contains two of { s, a 0 , 2 } and one occurrence of { s, a 1 , 4 } .

Original: Note that action a 0 is over-sampled relative to its true probability in s and a 1 is under-sampled.
Revised: Note that action a 0 is over-sampled relative to its true probability in s and a 1 is under-sampled.

Original: If we collect an additional trajectory with π e the expected value of the Monte Carlo estimate is:14 (2 + 2 + 4 + 2 π e ( a 0 ) + 4 π e ( a 1 ))
Revised: If we collect an additional trajectory with π e the expected value of the Monte Carlo estimate is: [Equation] 0 ) + 4 π e ( a 1 ))

Original: =
Revised: =

Original: 114 = 2 .
Revised: 114 = 2 .

Original: The true value, v ( π e ) = 3 and thus, conditionedon prior data , the Monte Carlo estimate is biased in expectation as shown in Proposition 1 .
Revised: The true value, v ( π e ) = 3 and thus, conditioned on prior data , the Monte Carlo estimate is biased in expectation as shown in Proposition 1.

Original: If insteadwe choose the behavior policy such that π b ( a 1 ) 
Revised: If instead we choose the behavior policy such that π b ( a 1 ) =

Original: = 1 then neither action is over- or under-sampled andthe expected value of the Monte Carlo estimate is the exact true value: 1 4 (2 + 2 + 4 + 4) = 12 4 = 3 .
Revised: 1 then neither action is over- or under-sampled and the expected value of the Monte Carlo estimate is the exact true value: 1 4 (2 + 2 + 4 + 4) = 124 = 3 .

Original: This example highlights that adapting the behavior policy to consider previously collected data canlower the expected ﬁnite-sample error of policy evaluation.
Revised: This example highlights that adapting the behavior policy to consider previously collected data can lower the expected ﬁnite-sample error of policy evaluation.

Original: In the next section, we introduce anadaptive data collection method that adjusts the behavior policy based on what data has already beenobserved so as to lower the MSE of a Monte Carlo estimate using all observed data.
Revised: In the next section, we introduce an adaptive data collection method that adjusts the behavior policy based on what data has already been observed so as to lower the MSE of a Monte Carlo estimate using all observed data.

Original:  Robust On-Policy Data Collection
Revised:  Robust On-Policy Data Collection

Original: In this section, we introduce a method that adapts the data-collecting behavior policy online tominimize sampling error in the data used by the Monte Carlo estimator.
Revised: In this section, we introduce a method that adapts the data-collecting behavior policy online to minimize sampling error in the data used by the Monte Carlo estimator.

Original: Speciﬁcally, let D t denoteall trajectories observed up to time-step t of the current trajectory (including the partial currenttrajectory).
Revised: Speciﬁcally, let D t denote all trajectories observed up to time-step t of the current trajectory (including the partial current trajectory).

Original: At time-step t , our method sets the behavior policy so as to reduce the current samplingerror, i.e., divergence between Pr( h | π e ) and Pr( h |D t ) .
Revised: At time-step t , our method sets the behavior policy so as to reduce the current sampling error, i.e., divergence between Pr( h | π e ) and Pr( h |D t ) .

Original: Our method can be run starting with D t = ∅or already containing trajectories in a setting like that described in the preceding section.
Revised: Our method can be run starting with D t = ∅ or already containing trajectories in a setting like that described in the preceding section.

Original: To reduce sampling error when collecting future trajectories, we want to adjust the behavior policy toincrease the probability of under-sampled trajectories, i.e., h for which Pr( h |D t ) <
Revised: To reduce sampling error when collecting future trajectories, we want to adjust the behavior policy to increase the probability of under-sampled trajectories, i.e., h for which Pr( h |D t ) <

Original: Pr( h | π e ) .
Revised: Pr( h | π e ) .

Original: Unfortunately, the trajectory distributions are unknown because the transition function, P , is also unknown.
Revised: Unfortunately, the trajectory distributions are unknown because the transition function, P , is also unknown.

Original: Instead, we will increase the probability of under-sampled actions.
Revised: Instead, we will increase the probability of under-sampled actions.

Original: Let π D : S ×
Revised: Let π D : S ×

Original: A → [0 , 1] denotethe empirical policy which gives the proportion of times that each action was taken in each statein D t .
Revised: A → [0 , 1] denote the empirical policy which gives the proportion of times that each action was taken in each state in D t .

Original: If π D ( a | s ) <
Revised: If π D ( a | s ) <

Original: π e ( a | s ) , then a has appeared less often in the data than it would in expectationunder π e .
Revised: π e ( a | s ) , then a has appeared less often in the data than it would in expectation under π e .

Original: Thus, we should increase the probability of a in s for future data collection.
Revised: Thus, we should increase the probability of a in s for future data collection.

Original: When the state and action spaces are ﬁnite, π D can be computed exactly as the maximum likelihoodpolicy under D t :
Revised: When the state and action spaces are ﬁnite, π D can be computed exactly as the maximum likelihood policy under D t :

Original: [Equation]
Revised: [Equation]

Original: where the argmax is taken with respect to all policies.
Revised: where the argmax is taken with respect to all policies.

Original: In larger MDP s, we require function approximation which may make π D hard to compute and update online as new data is collected.
Revised: In larger MDP s, we require function approximation which may make π D hard to compute and update online as new data is collected.

Original: Fortunately,with an additional assumption we can determine the direction to adjust action probabilities withoutexplicitly computing π D .
Revised: Fortunately, with an additional assumption we can determine the direction to adjust action probabilities without explicitly computing π D .

Original: This assumption is that π e belongs to a class of differentiable, parameterizedpolicies and is parameterized by vector θ ∈ R d .
Revised: This assumption is that π e belongs to a class of differentiable, parameterized policies and is parameterized by vector θ ∈ R d .

Original: This assumption is mild for many RL applicationsas it permits tabular, linear, and neural network policy representations.
Revised: This assumption is mild for many RL applications as it permits tabular, linear, and neural network policy representations.

Original: We use θ e to represent theparameter values for π e .
Revised: We use θ e to represent the parameter values for π e .

Original: We show in the next subsection that the gradient of the log-likelihood atθ e , ∇ θ L ( π θ )
Revised: We show in the next subsection that the gradient of the log-likelihood at θ e , ∇ θ L ( π θ )

Original: | θ = θ e , can be used to make sampling-error-reducing changes to the behavior policy.
Revised: | θ = θ e , can be used to make sampling-error-reducing changes to the behavior policy.

Original: 5.1 Robust On-policy Sampling
Revised: 5.1 Robust On-Policy Sampling

Original: Our primary algorithmic contribution – R obust O n-Policy S ampling ( ROS ) – reduces sampling errorby adapting the behavior policy with a single step of gradient descent on the log-likelihood at eachtime-step.
Revised: Our primary algorithmic contribution – R obust O n-Policy S ampling ( ROS ) – reduces sampling error by adapting the behavior policy with a single step of gradient descent on the log-likelihood at each time-step.

Original: From here on, we use ∇ θ L to denote the gradient of the log-likelihood evaluated at θ e .
Revised: From here on, we use ∇ θ L to denote the gradient of the log-likelihood evaluated at θ e .

Original: Observe that ∇ θ L provides a direction to adjust θ e to increase the probability of actions that wereover-sampled relative to their probability under π e .
Revised: Observe that ∇ θ L provides a direction to adjust θ e to increase the probability of actions that were over-sampled relative to their probability under π e .

Original: Thus, −∇ θ L provides a direction to adjustθ e to decrease the probability of over-sampled actions for which π D ( a | s ) >
Revised: Thus, −∇ θ L provides a direction to adjust θ e to decrease the probability of over-sampled actions for which π D ( a | s ) >

Original: π e ( a | s ) .
Revised: π e ( a | s ) .

Original: With thisinsight, ROS is able to adapt θ e so that π D tracks π e without ever computing π D .
Revised: With this insight, ROS is able to adapt θ e so that π D tracks π e without ever computing π D .

Original: At each time-step,
Revised: At each time-step,

Original: ROS computes ∇ θ L with all state-action pairs previously observed and then changes the evaluationpolicy parameters with a single step of gradient descent so that under-sampled actions have greaterprobability than they would have under π e .
Revised: ROS computes ∇ θ L with all state-action pairs previously observed and then changes the evaluation policy parameters with a single step of gradient descent so that under-sampled actions have greater probability than they would have under π e .

Original: Pseudocode for ROS is given in Algorithm 1.
Revised: Pseudocode for ROS is given in Algorithm 1.

Original: ROS ﬁrst computes ∇ θ L with previously collectedtrajectories if any are provided (Line 4).
Revised: ROS ﬁrst computes ∇ θ L with previously collected trajectories if any are provided (Line 4).

Original: ROS then collects n additional trajectories by interactingwith the given MDP (Lines 6-14).
Revised: ROS then collects n additional trajectories by interacting with the given MDP (Lines 6-14).

Original: For each action selection, ROS sets the behavior policy parametersas θ e − α ∇ θ L ( π θ ) | θ = θ e
Revised: For each action selection, ROS sets the behavior policy parameters as θ e − α ∇ θ L ( π θ ) | θ = θ e

Original: (Lines 9 and 10).
Revised: (Lines 9 and 10).

Original: It then computes ∇ θ log π θ
Revised: It then computes ∇ θ log π θ

Original: ( A | s ) | θ = θ e and updates211 ∇ θ L ( π θ )| θ = θ e
Revised: ( A | s ) | θ = θ e and updates ∇ θ L ( π θ )

Original:  (Lines 11 and 12).
Revised: | θ = θ e (Lines 11 and 12).

Original: Finally, the chosen action is executed in the environment, a rewardreceived, and the agent moves to the next state (Line 13).
Revised: Finally, the chosen action is executed in the environment, a reward received, and the agent moves to the next state (Line 13).

Original: Importantly, note that updating ∇ θ Lrequires per-timestep computation that is linear in the number of policy parameters and remainsconstant as the size of D grows.
Revised: Importantly, note that updating ∇ θ L requires per-timestep computation that is linear in the number of policy parameters and remains constant as the size of D grows.

Original: 5.2 ROS Convergence
Revised: 5.2 ROS Convergence

Original: This section develops our theoretical understanding of ROS .
Revised: This section develops our theoretical understanding of ROS .

Original: Due to space constraints, we defer allproofs to Appendix B. First, we show that ROS converges to the expected state visitation frequenciesunder π e .
Revised: Due to space constraints, we defer all proofs to Appendix B. First, we show that ROS converges to the expected state visitation frequencies under π e .

Original: Second, we show that, for a ﬁxed state, π D ( ·| s ) converges to π e ( ·| s ) faster under ROScompared to on-policy sampling. 
Revised: Second, we show that, for a ﬁxed state, π D ( ·| s ) converges to π e ( ·| s ) faster under ROS compared to on-policy sampling. Finally, we introduce an upper bound on the squared error between θ θ + 1 k + θ π θ t θ

Original: Finally, we introduce an upper bound on thesquared error betweenthe Monte Carlo estimate and v ( π e ) in terms of sampling error which shows how ROS ’s fasterconvergence affects the MSE of policy evaluation.
Revised: = θ e the Monte Carlo estimate and v ( π e ) in terms of sampling error which shows how ROS ’s faster convergence affects the MSE of policy evaluation.

Original: These results use the following assumption:
Revised: These results use the following assumption:

Original: Assumption 1.
Revised: Assumption 1.

Revised: The discrete state-space of the MDP has a directed acyclic graph ( DAG ) structure.

Revised: Speciﬁcally, states in S can be partitioned into l disjoint sets S t indexed by episode step.

Revised: The transition function is such that P ( s (cid:48) | s, a ) > 0 implies that s ∈ S t and s (cid:48)

Revised: ∈ S t +1 .

Revised: Note that Assumption 1 is mild as any ﬁnite-horizon MDP can be made a DAG by including the current time-step as part of the state (as we have already assumed in Section 3.1).

Original:  ROS uses a step-size of α → ∞ and the behavior policy is parameterized as asoftmax function, i.e., π θ ( a | s ) ∝
Revised: Assumption 2. ROS uses a step-size of α → ∞ and the behavior policy is parameterized as a softmax function, i.e., π θ ( a | s ) ∝

Original: e θ s,a , where for each state, s , and action, a , we have a parameterθ s,a .
Revised: e θ s,a , where for each state, s , and action, a , we have a parameter θ s,a .

Original: As we formally show in Appendix B, this assumption implies that ROS always takes the mostunder-sampled action in each state.
Revised: As we formally show in Appendix B, this assumption implies that ROS always takes the most under-sampled action in each state.

Original: We also introduce the notation of d tπ ( s ) as the probability of visiting state s at episode time t whilefollowing policy π and d tn ( s ) as the empirical frequency of visitations to state s at episode time tafter observing n trajectories.
Revised: We also introduce the notation of d tπ ( s ) as the probability of visiting state s at episode time t while following policy π and d tn ( s ) as the empirical frequency of visitations to state s at episode time t after observing n trajectories.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under Assumption 1 and  ROS action selection, d tn ( s ) converges to d tπ ( s ) with probability1 for all s ∈ S and 0 < t < l :
Revised: Under Assumptions 1 and 2 and ROS action selection, d tn ( s ) converges to d tπ ( s ) with probability 1 for all s ∈ S and 0 < t < l :

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: Let s be a particular state that is visited m times during data collection and assumethat |A| ≥ 2 .
Revised: Let s be a particular state that is visited m times during data collection and assume that |A| ≥ 2 .

Original: Under Assumption 1, D KL ( π D ( ·| s )
Revised: Under Assumption 2, D KL ( π D ( ·| s )

Original: ||
Revised: ||

Original: π ( ·| s ))
Revised: π ( ·| s ))

Original: =
Revised: =

Original: O p ( 1 m 2 ) under ROS sampling while
Revised: O p ( 1 m 2 ) under ROS sampling while

Original: D KL ( π D ( ·| s ) || π ( ·| s ))
Revised: D KL ( π D ( ·| s ) || π ( ·| s ))

Original: =
Revised: =

Original: O p ( 1 m ) under on-policy sampling, where O p denotes stochastic boundedness.
Revised: O p ( 1 m ) under on-policy sampling, where O p denotes stochastic boundedness.

Original: Theorem 3.
Revised: Theorem 3.

Original: Assume ∀ s ∈ S , a ∈ A that R ( s, a ) ≤ R max .
Revised: Assume ∀ s ∈ S , a ∈ A that R ( s, a ) ≤ R max .

Original: The squared error in the Monte Carloestimate using D can be upper-bounded by:
Revised: The squared error in the Monte Carlo estimate using D can be upper-bounded by:

Original: [Equation]
Revised: [Equation]

Original: = γ 2 t R 2 max (cid:113) 2 D KL ( d tn ||

Original: d

Original: tπ e )

Original: + 2 E S ∼ d tn

Original: [ D KL ( π D ( ·| S ) ||

Original: π e ( ·| S )] .

Original: Remark 2.
Revised: Remark 2.

Original: The second term in the bound in Theorem 3 is the KL-divergence between π D andπ e which Theorem 2 tells us will decrease faster under ROS action selection.
Revised: The second term in the bound in Theorem 3 is the KL-divergence between π D and π e which Theorem 2 tells us will decrease faster under ROS action selection.

Original: The ﬁrst term is the
Revised: The ﬁrst term is the

Original: KL-divergence between the empirical and true state distributions which depends both on samplingerror in action selection as well as sampling error in the transition and initial state distributions.
Revised: KL-divergence between the empirical and true state distributions which depends both on sampling [Figure] error in action selection as well as sampling error in the transition and initial state distributions.

Original: While the former decreases faster under ROS , the latter will decrease the same for both ROS andon-policy sampling.
Revised: While the former decreases faster under ROS , the latter will decrease the same for both ROS and on-policy sampling.

Original: Hence, the theoretical faster rate of ROS for reducing sampling error in actionselection may be muted by high environment stochasticity.
Revised: Hence, the theoretical faster rate of ROS for reducing sampling error in action selection may be muted by high environment stochasticity.

Original: The experimental results given in Figure12(a) complement this theoretical observation.
Revised: The experimental results given in Figure 12(a) complement this theoretical observation.

Original:  Empirical Study
Revised:  Empirical Study

Original: We next conduct an empirical study of ROS in policy evaluation problems.
Revised: We next conduct an empirical study of ROS in policy evaluation problems.

Original: Our primary goal is toanswer the following questions:
Revised: Our primary goal is to answer the following questions:

Original: Does ROS reduce sampling error compared to on-policy sampling?
Revised: Does ROS reduce sampling error compared to on-policy sampling?

Original: Does ROS lower policy evaluation MSE when starting with and without off-policy data?
Revised: Does ROS lower policy evaluation MSE when starting with and without off-policy data?

Original: We conduct policy evaluation experiments in four domains covering discrete and continuous stateand action spaces: a multi-armed bandit problem [Sutton and Barto, 1998], Gridworld [Thomas and
Revised: We conduct policy evaluation experiments in four domains covering discrete and continuous state and action spaces: a multi-armed bandit problem [Sutton and Barto, 1998], Gridworld [Thomas and

Original: Brunskill, 2016], CartPole, and Continuous CartPole [Brockman et al., 2016].
Revised: Brunskill, 2016], CartPole, and Continuous CartPole [Brockman et al., 2016].

Original: Since these domainsare widely used, we defer their descriptions to Appendix C.
Revised: Since these domains are widely used, we defer their descriptions to Appendix C.

Original: Our primary baseline for comparison ison-policy sampling ( OS ) of i.i.d.
Revised: Our primary baseline for comparison is on-policy sampling ( OS ) of i.i.d.

Original: trajectories with the Monte Carlo estimator used to compute the ﬁnalpolicy value estimate (denoted OS - MC ).
Revised: trajectories with the Monte Carlo estimator used to compute the ﬁnal policy value estimate (denoted OS - MC ).

Original: We also compare to BPG which ﬁnds a minimum variancebehavior policy for the ordinary importance sampling ( OIS ) policy value estimator [Hanna et al.
Revised: We also compare to BPG which ﬁnds a minimum variance behavior policy for the ordinary importance sampling ( OIS ) policy value estimator [Hanna et al., 2017]

Original: ,2017] (denoted BPG - OIS ).
Revised: (denoted BPG - OIS ).

Original: We provide full experimental details concerning how π e and v ( π e ) weredetermined in Appendix C.2.[Figure] [Figure]
Revised: We provide full experimental details concerning how π e and v ( π e ) were determined in Appendix C.2.

Original: 6.1 Policy Evaluation without Initial Data
Revised: 6.1 Policy Evaluation without Initial Data

Original: We ﬁrst run experiments in a setting without initial data , in which all data is collected from scratch.
Revised: We ﬁrst run experiments in a setting without initial data , in which all data is collected from scratch.

Original: Letting T denote the average length of a trajectory, in each domain, we collect a total of 2 13 Tenvironment steps with each method and compute metrics every 2 1 , 2 2 , ..., 2 13 trajectories.
Revised: Letting T denote the average length of a trajectory, in each domain, we collect a total of 2 13 T environment steps with each method and compute metrics every 2 1 , 2 2 , ..., 2 13 trajectories.

Revised: Note that we specify the number of environment steps rather than number of trajectories in our empirical results.

Revised: For Bandit T = 1 , for GridWorld T = 7 . 43 , CartPole T = 48 . 48 , and for CartPoleContinuous

Original: Thehyper-parameter settings for these experiments are presented in Appendix E.
Revised: T = 49 . The hyper-parameter settings for all experiments are presented in Appendix E.

Original: We ﬁrst verify that ROS reduces sampling error compared to on-policy sampling.
Revised: We ﬁrst verify that ROS reduces sampling error compared to on-policy sampling.

Original: We measuresampling error with the KL -divergence ( KL ) between π e and a parametric maximum likelihoodestimate of π D from the observe data.
Revised: We measure sampling error with the KL -divergence ( KL ) between π e and a parametric maximum likelihood estimate of π D from the observe data.

Original: In Appendix D, we give a complete deﬁnition of the measureas well as an alternative measure that leads to qualitatively similar results.
Revised: In Appendix D, we give a complete deﬁnition of the measure as well as an alternative measure that leads to qualitatively similar results.

Original: Due to space constraints,we only show this result for the GridWorld domain (Figure 1(a)); results for other domains arequalitatively the same and can be found in Appendix D.1.
Revised: Due to space constraints, we only show this result for the GridWorld domain (Figure 1(a)); results for other domains are qualitatively the same and can be found in Appendix D.1.

Original: Figure 1(a) shows that with ROS samplingerror decreases faster than OS .
Revised: Figure 1(a) shows that with ROS sampling error decreases faster than OS .

Original: Unsurprisingly, BPG increases sampling error as it is an off-policymethod which adapts the behavior policy away from π e . 
Revised: Unsurprisingly, BPG increases sampling error as it is an off-policy method which adapts the behavior policy away from π e . These results answer our ﬁrst empirical [Figure] [Figure]

Original: These results answer our ﬁrst empiricalquestion and conﬁrm our theoretical claim that non-i.i.d.
Revised: Axes and conﬁdence intervals are the same as in Figure 2. question and conﬁrm our theoretical claim that non- i.i.d.

Original: off-policy sampling can cause the empiricaldistribution of data to converge to the expected on-policy distribution faster.
Revised: off-policy sampling can cause the empirical distribution of data to converge to the expected on-policy distribution faster.

Original: Ultimately, this paper focuses on reducing sampling error for lower MSE policy evaluation.
Revised: Ultimately, this paper focuses on reducing sampling error for lower MSE policy evaluation.

Original: Figure 2shows that ROS lowers MSE compared to both OS and BPG across all domains.
Revised: Figureshows that ROS lowers MSE compared to both OS and BPG across all domains.

Original: 1 These results addressour second empirical question and support the claim that reducing sampling error decreases the MSEof the Monte Carlo estimator for policy evaluation.
Revised: These results address our second empirical question and support the claim that reducing sampling error decreases the MSE of the Monte Carlo estimator for policy evaluation.

Original: 6.2 Policy Evaluation with Initial Data
Revised: 6.2 Policy Evaluation with Initial Data

Original: Our next set of experiments considers a setting with initial data , in which a set of 100 trajectories arealready available and we wish to use these trajectories in our policy value estimate.
Revised: Our next set of experiments considers a setting with initial data , in which a set of 100 trajectories are already available and we wish to use these trajectories in our policy value estimate.

Original: These trajectoriesare collected via i.i.d.
Revised: These trajectories are collected via i.i.d.

Original: off-policy sampling with a behavior policy that is slightly different than π e .
Revised: off-policy sampling with a behavior policy that is slightly different than π e .

Original: This setting is intended to represent a setting where π e has just been updated from an older policy[Figure] and we would like to still use the off-policy data already collected from the older policy combinedwith the data to be collected.
Revised: This setting is intended to represent a setting where π e has just been updated from an older policy and we would like to still use the off-policy data already collected from the older policy combined with the data to be collected.

Original: In addition to the off-policy data ( OPD ), we collect an additional 2 13 Tsteps of environment interaction with each method.
Revised: In addition to the off-policy data ( OPD ), we collect an additional 2 13 T steps of environment interaction with each method.

Revised: We do not count the initial 100 trajectories towards the total collected data.

Original: For ROS , we use the OPD to initialize ∇ θ L .
Revised: For ROS , we use the OPD to initialize ∇ θ L .

Original: We expect to see that ROS will collect data to combinewith the OPD such that the aggregate data set looks as if it had been collected with π e to begin with.
Revised: We expect to see that ROS will collect data to combine with the OPD such that the aggregate data set looks as if it had been collected with π e to begin with.

Original: We compare ROS to the following baseline methods.
Revised: We compare ROS to the following baseline methods.

Original: ( OPD + OS )- MC collects additional data with OSand uses the Monte Carlo estimator with the total data-set.
Revised: ( OPD + OS )- MC collects additional data with OS and uses the Monte Carlo estimator with the total data-set.

Original: ( OPD + OS )-( WIS + MC ) uses weightedimportance sampling ( WIS ) to compute an estimate from the OPD combines the WIS estimate with a
Revised: ( OPD + OS )-( WIS + MC ) uses weighted importance sampling ( WIS ) to compute an estimate from the OPD combines the WIS estimate with a

Original: Monte Carlo estimate using on-policy data.
Revised: Monte Carlo estimate using on-policy data.

Original: ( OPD + BPG )- OIS collects additional data with BPG anduses ordinary importance sampling as the estimator with all data.
Revised: ( OPD + BPG )- OIS collects additional data with BPG and uses ordinary importance sampling as the estimator with all data.

Original: Finally, ( OS - MC ) replaces the 100initial trajectories with trajectories from OS , then collects the remaining data with OS and uses the
Revised: Finally, ( OS - MC ) replaces the initial trajectories with trajectories from OS , then collects the remaining data with OS and uses the

Original: Monte Carlo estimator.
Revised: Monte Carlo estimator.

Revised: In the case of ( OS - MC ) , the 100 initial trajectories are counted towards the total data collected.

Original: Figure 1(b) shows that sampling error decreases fastest for ROS as the additional data is collected.
Revised: Figure 1(b) shows that sampling error decreases fastest for ROS as the additional data is collected.

Original: For policy evaluation, we show MSE for varying amounts of data in Figure 3 and provide numericalvalues for the ﬁnal MSE in Appendix F. These ﬁgures show thatROS can collect additional data thatreduces sampling error in the aggregate data set and produce lower MSE estimates compared to otherdata collection methods.
Revised: For policy evaluation, we show MSE for varying amounts of data in Figure 3 and provide numerical values for the ﬁnal MSE in Appendix F. We observe that the initial data provides an immediate reduction in MSE at the expense of injecting bias into the estimates. ( OPD + OS )- MC struggles to reduce this bias while ( OPD + ROS )- MC is able to through data collection.

Revised: Overall, this result highlights that ROS can collect additional data that reduces sampling error in the aggregate data set and produce lower MSE estimates compared to other data collection methods.

Original: Intuitively, OS requires many more samples to dilute the bias brought onby using OPD in the Monte Carlo estimator, while ROS is able to correct the empirical off-policydistribution to the expected on-policy distribution and use the Monte Carlo estimator without anyoff-policy corrections. 
Revised: Intuitively, OS requires many more samples to dilute the bias brought on by using OPD in the Monte Carlo estimator, while ROS is able to correct the empirical off-policy distribution to the expected on-policy distribution and use the Monte Carlo estimator without any off-policy corrections. The comparison to OS - MC demonstrates the potential of ROS for correcting an off-policy empirical distribution to the expected on-policy distribution. As noted above, OS - MC has 100 fewer trajectories than the other baselines. However – even when including the initial 100 off-policy trajectories in the data total for all methods – ROS eventually obtains lower MSE compared to OS - MC .

Revised: In this sense,

Revised: ROS has taken an initially biased dataset and collected the right trajectories to make it look as-if the evaluation policy had collected all trajectories in the ﬁrst place.

Original: 6.3 Sensitivity Study
Revised: 6.3 Sensitivity Study

Original: Finally, we evaluate the sensitivity of ROS to hyper-parameter, environment, and policy settings.
Revised: Finally, we evaluate the sensitivity of ROS to hyper-parameter, environment, and policy settings.

Original: ROSrequires setting a step size, α , which controls how much ROS updates the behavior policy away fromπ e .
Revised: ROS requires setting a step size, α , which controls how much ROS updates the behavior policy away from π e .

Original: We show MSE curves for ROS with different step size α on GridWorld and CartPole in Figures 4( α = 0 corresponds to OS ).
Revised: We show MSE curves for ROS with different step size α on GridWorld and CartPole in Figures( α = 0 corresponds to OS ).

Original: Figure 4(a) shows that, in GridWorld, ROS with any tested step-sizeproduces lower MSE policy evaluation than OS for any data set size.
Revised: Figure 4(a) shows that, in GridWorld, ROS with any tested step-size produces lower MSE policy evaluation than OS for any data set size.

Original: As it collects more data, ROS withlarger α enables lower MSE because the norm of ∇ θ L decreases as sampling error decreases, and thusa larger α is required to make signiﬁcant updates.
Revised: As it collects more data, ROS with larger α enables lower MSE because the norm of ∇ θ L decreases as sampling error decreases, and thus a larger α is required to make signiﬁcant updates.

Original: A larger α value is also in line with our theoreticalresults which prescribe α → ∞ .
Revised: A larger α value is also in line with our theoretical results which prescribe α → ∞ .

Original: However, in CartPole, (Figure 4(b)), ROS with the largest testedα (1000) diverges and the second largest ( α = 100 ) requires many steps before it improves upon OS .
Revised: However, in CartPole, (Figure 4(b)), ROS with the largest tested α (1000) diverges and the second largest ( α = 100 ) requires many steps before it improves upon OS .

Original: Thus, in domains with continuous state-spaces, more conservative α values may be preferred.
Revised: Thus, in domains with continuous state-spaces, more conservative α values may be preferred.

Original: Our ﬁnal set of experiments considers how the stochasticity of a domain and entropy of π e affectthe relative improvement that ROS offers.
Revised: Our ﬁnal set of experiments considers how the stochasticity of a domain and entropy of π e affect the relative improvement that ROS offers.

Original: In this sub-section, we study these settings in the Banditdomain for its simplicty; similar experimental results in GridWorld can be found in Appendix H. Wechoose α = 1000 for the following experiments.
Revised: In this sub-section, we study these settings in the Bandit domain for its simplicty; similar experimental results in GridWorld can be found in Appendix H. We choose α = 1000 for the following experiments.

Original: To study domain stochasticity, we ﬁrst create variants of the Bandit environment by multiplying eitherthe mean or scale of the reward distribution of each action by a varying factor.
Revised: To study domain stochasticity, we ﬁrst create variants of the Bandit environment by multiplying either the mean or scale of the reward distribution of each action by a varying factor.

Original: In each experimentaltrial, we use ROS to collect 1000 T steps for the Monte Carlo estimator and compute the relative
Revised: In each experimental trial, we use ROS to collect 1000 T steps for the Monte Carlo estimator and compute the relative

Original: MSE compared to the Monte Carlo estimator using OS with the same number of steps.
Revised: MSE compared to the Monte Carlo estimator using OS with the same number of steps.

Original: Figure 4(c)shows that as the factor on the mean increases, ROS provides a greater reduction in MSE as even smallamounts of sampling error translate into large MSE when the reward means are large.
Revised: Figure 4(c) shows that as the factor on the mean increases, ROS provides a greater reduction in MSE as even small amounts of sampling error translate into large MSE when the reward means are large.

Original: On the otherhand, as the scale factor increases, the MSE is dominated by reward noise and the relative beneﬁt ofreducing sampling error disappears.
Revised: On the other hand, as the scale factor increases, the MSE is dominated by reward noise and the relative beneﬁt of reducing sampling error disappears.

Original: [Figure]

Original: We also evaluate the relative improvement of ROS as a function of the entropy of π e .
Revised: We also evaluate the relative improvement of ROS as a function of the entropy of π e .

Original: For π e , we use(cid:15) -greedy policies which select the optimal action in a state with probability 1 − (cid:15) and otherwise selectan action uniformly at random.
Revised: For π e , we use (cid:15) -greedy policies which select the optimal action in a state with probability 1 − (cid:15) and otherwise select an action uniformly at random.

Original: Relative improvement in MSE is shown in Figure 4(d).
Revised: Relative improvement in MSE is shown in Figure 4(d).

Original: For all (cid:15) , ROSimproves upon the MSE of OS .
Revised: For all (cid:15) , ROS improves upon the MSE of OS .

Original: The improvement is generally larger for more stochastic π e whensampling error in action selection will be highest.
Revised: The improvement is generally larger for more stochastic π e when sampling error in action selection will be highest.

Original:  Discussion and Future Work
Revised:  Discussion and Future Work

Original: This work has shown that off-policy non-i.i.d.
Revised: This work has shown that off-policy non-i.i.d.

Original: sampling can produce data sets that more closelyapproximate the on-policy data distribution than on-policy i.i.d.
Revised: sampling can produce data sets that more closely approximate the on-policy data distribution than on-policy i.i.d.

Original: sampling.
Revised: sampling.

Original: We considered the problemof policy evaluation and showed that more closely approximating the on-policy data distribution leadsto more data efﬁcient policy evaluation across several domains.
Revised: We considered the problem of policy evaluation and showed that more closely approximating the on-policy data distribution leads to more data efﬁcient policy evaluation across several domains.

Original: As far as we know, ROS is the ﬁrstdata collection method for policy evaluation that uses off-policy sampling to produce more closelyon-policy data than the data produced by on-policy sampling.
Revised: As far as we know, ROS is the ﬁrst [Figure] data collection method for policy evaluation that uses off-policy sampling to produce more closely on-policy data than the data produced by on- sampling

Original: While ROS is a ﬁrst step towards off-policy algorithms that produce data matching a target distribution,we highlight a few limitations of the algorithm and our study.
Revised: While ROS is a ﬁrst step towards off-policy algorithms that produce data matching a target distribution, we highlight a few limitations of the algorithm and our study.

Original: In our view, the main limitations ofthe ROS algorithm are the need to set a step-size parameter (in contrast to parameter-free on-policysampling) and the need to update ∇ θ L at each action step.
Revised: In our view, the main limitations of the ROS algorithm are the need to set a step-size parameter (in contrast to parameter-free on-policy sampling) and the need to update ∇ θ L at each action step.

Original: For the former, future work shouldinvestigate robust methods for setting the step-size, particularly in settings where π θ generalizesacross the state-space.
Revised: For the former, future work should investigate robust methods for setting the step-size, particularly in settings where π θ generalizes across the state-space.

Original: For the latter limitation, a future study could consider only updating ∇ θ L atthe end of each episode instead of after each action choice (assuming more computation can be donebetween episodes).
Revised: For the latter limitation, a future study could consider only updating ∇ θ L at the end of each episode instead of after each action choice (assuming more computation can be done between episodes).

Original: In terms of our study, for this paper we chose to study many different facets of
Revised: In terms of our study, for this paper we chose to study many different facets of

Original: ROS on a suite of simpler domains (see the appendices for additional ablations and extensions); afuture study should assess the scalability of ROS with more complex function approximators.
Revised: ROS on a suite of simpler domains (see the appendices for additional ablations and extensions); a future study should assess the scalability of ROS with more complex function approximators.

Original: Finally,our theoretical results were conducted in the tabular setting; an important open question is at whatrate ROS converges when π θ uses a function approximator that must generalize across states.
Revised: Finally, our theoretical results were conducted in the tabular setting; an important open question is at what rate ROS converges when π θ uses a function approximator that must generalize across states.

Original: Beyondthese minor technical limitations, our paper addresses fundamental research questions in RL and thuswe do not see obvious negative societal impacts that are unique to this work in comparison to otherwork in RL and policy evaluation.
Revised: Beyond these minor technical limitations, our paper addresses fundamental research questions in RL and thus we do not see obvious negative societal impacts that are unique to this work in comparison to other work in RL and policy evaluation.

Original: While we evaluated ROS for policy evaluation, the long-term importance of this work may be inexploring the distinction between on-policy sampling and on-policy data.
Revised: While we evaluated ROS for policy evaluation, the long-term importance of this work may be in exploring the distinction between on-policy sampling and on-policy data.

Original: On-policy RL algorithmsrequire on-policy data and our work suggests that adaptive off-policy sampling can produce on-policydata more efﬁciently than on-policy sampling.
Revised: On-policy RL algorithms require on-policy data and our work suggests that adaptive off-policy sampling can produce on-policy data more efﬁciently than on-policy sampling.

Original: In the future, we wish to study these insights foron-policy policy improvement algorithms (e.g., policy gradient methods [Williams, 1992, Schulmanet al., 2017]) and to extend our convergence results to non-tabular settings.
Revised: In the future, we wish to study these insights for on-policy policy improvement algorithms (e.g., policy gradient methods [Williams, 1992, Schulman et al., 2017]) and to extend our convergence results to non-tabular settings.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have introduced a novel data collection method for policy evaluation in reinforcementlearning environments.
Revised: In this paper, we have introduced a novel data collection method for policy evaluation in reinforcement learning environments.

Original: Our method – Robust On-Policy Sampling ( ROS ) – considers previouslycollected data when selecting actions to reduce sampling error in the entire collected data set.
Revised: Our method – Robust On-Policy Sampling ( ROS ) – considers previously collected data when selecting actions to reduce sampling error in the entire collected data set.

Original: We showboth in theory and in practice that data from ROS converges faster to the on-policy data distributioncompared to on-policy sampling.
Revised: We show both in theory and in practice that data from ROS converges faster to the on-policy data distribution compared to on-policy sampling.

Original: Empirically, we ﬁnd that faster convergence to the on-policy datadistributions lowers the MSE of policy evaluation.
Revised: Empirically, we ﬁnd that faster convergence to the on-policy data distributions lowers the MSE of policy evaluation.

Revised: Acknowledgments and Disclosure of Funding

Revised: We thank Ishan Durugkar, Brahma Pavse, Subhojyoti Mukherjee, Elliot Fosong, and Filippos

Revised: Christianos for their feedback which greatly strengthened the paper.

Revised: We also wish to acknowledge the anonymous reviewers for their comments and constructive criticisms.

Original: A Proof of Proposition 1
Revised: Support for this research was provided by the Ofﬁce of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin — Madison with funding from the Wisconsin Alumni Research Foundation. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] See Section 7.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] See Section 7.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] See Section 5.2 and Appendix B. (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] See Appendices A and B. 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a U RL )?

Revised: [Yes] Included in supplementary material.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] See Section 6 and Appendices C.2 and E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [No] Experiments use RL domains and algorithms that can be ran on a typical personal computer.

Revised: Minimal compute resources required to reproduce any experiment in the paper.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] (b) Did you mention the license of the assets?

Revised: [No] (c) Did you include any new assets either in the supplemental material or as a U RL ?

Revised: [No] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original:  In this appendix we prove Proposition 1 from Section 4.
Revised: [N/A] A Proof of Proposition In this appendix we prove Proposition 1 from Section 4.

Original: Proposition 1.
Revised: Proposition 1.

Original: The data conditioned Monte Carlo estimator is biased under on-policy sampling of
Revised: The data conditioned Monte Carlo estimator is biased under on-policy sampling of

Original: D 2 unless MC( D 1 ) = v ( π e ) or D 1 = ∅ .
Revised: D 2 unless MC( D 1 ) = v ( π e ) or D 1 = ∅ .

Original: That is:
Revised: That is:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the following, we use E π e [ · ] as shorthand for E [ ·| D ∼ π e ] .
Revised: In the following, we use E π e [ · ] as shorthand for E [ ·| D ∼ π e ] .

Original: Let ˆ v 1 := 1 n D 1 (cid:80) n D 1 i =1 g ( h i ) =
Revised: Let ˆ v 1 := 1 n D 1 (cid:80) n D 1 i =1 g ( h i ) =

Original: MC( D 1 ) .
Revised: MC( D 1 ) .

Original: [Equation]
Revised: [Equation]

Original: where (a) uses the fact that no random variables appear inside the ﬁrst expectation, (b) uses thedeﬁnition of ˆ v , and (c) uses the fact that MC( D 2 ) is an unbiased estimator of v ( π e ) .
Revised: where (a) uses the fact that no random variables appear inside the ﬁrst expectation, (b) uses the deﬁnition of ˆ v , and (c) uses the fact that MC( D 2 ) is an unbiased estimator of v ( π e ) .

Original: The propositionfollows by observing that equation 11 is equal to v ( π e ) if and only if n D 1 = 

Revised: The proposition follows by observing that equation 11 is equal to v ( π e ) if and only if n D 1 =

Original: (i.e., D 1 = ∅ ) or ifˆ v = v ( π e ) .
Revised: (i.e., D 1 = ∅ ) or if ˆ v = v ( π e ) .

Original: Note that the latter case in which the data-conditioned Monte Carlo estimator is unbiased only occurswhen the Monte Carlo estimate only using D 1 has non-zero error.
Revised: Note that the latter case in which the data-conditioned Monte Carlo estimator is unbiased only occurs when the Monte Carlo estimate only using D 1 has non-zero error.

Original: If g ( H ) has non-zero varianceunder on-policy sampling then the probability that MC( D 1 ) is exactly equal to v ( π e ) with a ﬁnite-size
Revised: If g ( H ) has non-zero variance under on-policy sampling then the probability that MC( D 1 ) is exactly equal to v ( π e ) with a ﬁnite-size

Original: D 2 is low.
Revised: D 2 is low.

Original: B Proofs of ROS Properties
Revised: B Proofs of ROS Properties

Original: Before giving the proofs for Theorems 1 and 2 we re-state our key assumption.
Revised: Before giving the proofs for Theorems 1 and 2 we re-state our key assumption.

Original: Assumption 1.
Revised: Assumption 2.

Original: ROS uses a step-size of α → ∞ and the behavior policy is parameterized as asoftmax function, i.e., π θ ( a | s ) ∝
Revised: ROS uses a step-size of α → ∞ and the behavior policy is parameterized as a softmax function, i.e., π θ ( a | s ) ∝

Original: e θ s,a , where for each state, s , and action, a , we have a parameterθ s,a .
Revised: e θ s,a , where for each state, s , and action, a , we have a parameter θ s,a .

Original: As we formally show in Appendix B, this assumption implies that ROS always takes the mostunder-sampled action in each state.
Revised: As we formally show in Appendix B, this assumption implies that ROS always takes the most under-sampled action in each state.

Original: We next derive two lemmas that will be used in the proofs of our theorems.
Revised: We next derive two lemmas that will be used in the proofs of our theorems.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let S m ( a ) denote the number of times that action a was taken after visiting a particularstate, s , m times in D and let k := |A| .
Revised: Let S m ( a ) denote the number of times that action a was taken after visiting a particular state, s , m times in D and let k := |A| .

Original: Under Assumption 1 and ROS collection of D , we have that:
Revised: Under Assumption 2 and ROS collection of D , we have that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst formally show that Assumption 1 implies that ROS always taking the most undersampled action at each time-step.
Revised: We ﬁrst formally show that Assumption 2 implies that ROS always taking the most undersampled action at each time-step.

Original: Because we only consider a single state, we suppress dependencieson the state in this proof, e.g., we write π e ( a ) instead of π e ( a | s ) and θ a instead of θ s,a for thesoftmax parameters.
Revised: Because we only consider a single state, we suppress dependencies on the state in this proof, e.g., we write π e ( a ) instead of π e ( a | s ) and θ a instead of θ s,a for the softmax parameters.

Original: We have L ( π θ ) =
Revised: We have L ( π θ ) =

Original: (cid:80) a S m ( a ) θ a − m log( (cid:80) b ∈A e θ b ) and ∇ θ a L ( π θ ) | π θ =
Revised: (cid:80) a S m ( a ) θ a − m log( (cid:80) b ∈A e θ b ) and ∇ θ a L ( π θ ) | π θ =

Original: π e =
Revised: π e =

Original: S m ( a )(1 − π e ( a )) − π e ( a )( m − S m ( a )) =
Revised: S m ( a )(1 − π e ( a )) − π e ( a )( m − S m ( a )) =

Original: S m ( a ) − mπ e ( a ) , which is exactly the number of timesthat action a was over-sampled.
Revised: S m ( a ) − mπ e ( a ) , which is exactly the number of times that action a was over-sampled.

Original: Using α → ∞ , the softmax parameter for action a after the ROSupdate is given by:
Revised: The softmax parameter for action a after the ROS update is given by:

Original: [Equation]
Revised: [Equation]

Original: where θ e,a is the softmax parameter for action a under the evaluation policy.
Revised: where θ e,a is the softmax parameter for action a under the evaluation policy.

Original: Taking the limit as535 α → ∞ , we see that the θ e,a term is dominated by the α ( S m ( a ) − π e ( a ) m ) term.
Revised: Taking the limit as α → ∞ , we see that the θ e,a term is dominated by the α ( S m ( a ) − π e ( a ) m ) term.

Original: Thus the parameterfor each action is:
Revised: Thus the parameter for each action is:

Original: [Equation]
Revised: [Equation]

Original: Interpreting α as the inverse of the softmax temperature, we can see that α → ∞ corresponds to asoftmax temperature → 0 which in turn corresponds to a hard max over the ( mπ e ( a ) − S m ( a )) values.
Revised: Interpreting α as the inverse of the softmax temperature, we can see that α → ∞ corresponds to a softmax temperature → 0 which in turn corresponds to a hard max over the ( mπ e ( a ) − S m ( a )) values.

Original: Thus, the updated behavior policy puts all probability mass on the action for which π e ( a ) m − S
Revised: Thus, the updated behavior policy puts all probability mass on the action for which π e ( a ) m − S

Original: m
Revised: m

Original: ( a )is largest.
Revised: ( a ) is largest.

Original: Hence we select the most under-sampled action if we take α → ∞ in Algorithm 1.
Revised: Hence we select the most under-sampled action if we take α → ∞ in Algorithm 1.

Original: Now we show that under ROS action selection that the amount of over- or under-sampling is bounded.
Revised: Now we show that under ROS action selection that the amount of over- or under-sampling is bounded.

Original: First, we observe that,(cid:88)
Revised: First, we observe that,

Original: [Equation]
Revised: [Equation]

Original: and we claim that,
Revised: and we claim that,

Original: [Equation]
Revised: [Equation]

Original: for any a ∈ A .
Revised: for any a ∈ A .

Original: We prove this claim by contradiction.
Revised: We prove this claim by contradiction.

Original: Assume not: S m ( j ) − mπ e ( j ) > 1 for somej ∈ A .
Revised: Assume not: S m ( j ) − mπ e ( j ) > 1 for some j ∈ A .

Original: Let X m ( j ) :
Revised: Let X m ( j ) :

Original: = 1 if action j was taken at step m and 0 otherwise.
Revised: = 1 if action j was taken at step m and 0 otherwise.

Original: If X m ( j ) = 1 , we have that:
Revised: If X m ( j ) = 1 , we have that:

Original: [Equation]
Revised: [Equation]

Original: However, this results in a contradiction since we have that action j was over-sampled at the previousstep but action j would not be selected by ROS if it was over-sampled.
Revised: However, this results in a contradiction since we have that action j was over-sampled at the previous step but action j would not be selected by ROS if it was over-sampled.

Original: So, in order for S m ( j ) −mπ e ( j ) > 1 , we must have that X m ( j ) = 0 .
Revised: So, in order for S m ( j ) − mπ e ( j ) > 1 , we must have that X m ( j ) = 0 .

Original: Combining the fact that X m ( j ) = 0 with ourassumption that S m ( j ) − mπ e ( j ) > 1 tells us that:
Revised: Combining the fact that X m ( j ) = 0 with our assumption that S m ( j ) − mπ e ( j ) > 1 tells us that:

Original: [Equation]
Revised: [Equation]

Original: where (a) is because S m − 1 ( j ) must be equal to S m ( j ) if X m ( j )
Revised: where (a) is because S m − 1 ( j ) must be equal to S m ( j ) if X m ( j )

Original: = 0 .
Revised: = 0 .

Original: By the same logic we getthat X m − 1 ( j ) =
Revised: By the same logic we get that X m − 1 ( j ) =

Original: x m − 2 ( j ) = · · · = X 1 ( j ) = 0 which implies that S 1 ( j ) − π e ( j ) = 0 − π e > 552which is a contradiction.
Revised: x m − 2 ( j ) = · · · = X 1 ( j ) = 0 which implies that S 1 ( j ) − π e ( j ) = 0 − π e > 1 which is a contradiction.

Original: Thus, we conclude that S m ( j ) − mπ e ( j ) ≤ 1 for any j , i.e., anyaction can be over-sampled by at most 1.
Revised: Thus, we conclude that S m ( j ) − mπ e ( j ) ≤ 1 for any j , i.e., any action can be over-sampled by at most 1.

Original: Combining this conclusion with the observation that554 (cid:80) [-a ∈A S m ( a ) − mπ e ( a ) = m − m = 
-]
Revised: Combining this conclusion with the observation that (cid:80) [-a ∈A S m ( a ) − mπ e ( a ) = m − m = 
-][Equation]

Original: tells us that any action can be under-sampled by at mostk − 1 .
Revised: tells us that any action can be under-sampled by at most k − 1 .

Original: Thus, the absolute difference | S m ( a ) − mπ e ( a ) | is at most k − 1 for any action a ∈ A whichcompletes the proof.
Revised: Thus, the absolute difference | S m ( a ) − mπ e ( a ) | is at most k − 1 for any action a ∈ A which completes the proof.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let s be a state that we visit m times.
Revised: Let s be a state that we visit m times.

Original: Under ROS sampling, we have ∀ a ∈ A that:
Revised: Under ROS sampling, we have ∀ a ∈ A that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof follows from Lemma 1.
Revised: The proof follows from Lemma 1.

Original: As in the proof of Lemma 1, we suppress the dependencyon s in our notation since we are only concerned with a ﬁxed state.
Revised: As in the proof of Lemma 1, we suppress the dependency on s in our notation since we are only concerned with a ﬁxed state.

Original: Using the notation from the proof of Lemma 1, we have that π D ( a ) =
Revised: Using the notation from the proof of Lemma 1, we have that π D ( a ) =

Original: S m ( a ) m .
Revised: S m ( a ) m .

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: Under Assumption 1 and  ROS action selection, d tn ( s ) converges to d tπ ( s ) with probability1 for all s ∈ S and 0 < t < l :
Revised: Under Assumptions 1 and 2 and ROS action selection, d tn ( s ) converges to d tπ ( s ) with probability 1 for all s ∈ S and 0 < t < l :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is by induction.
Revised: The proof is by induction.

Original: For the base case, note that d 0 π ( s ) =
Revised: For the base case, note that d 0 π ( s ) =

Original: d 0 ( s ) and thuslim n →∞ d 0 n ( s ) =
Revised: d 0 ( s ) and thus lim n →∞ d 0 n ( s ) =

Original: d 0 π ( s ) ∀ s with probability 1 by the strong law of large numbers.
Revised: d 0 π ( s ) ∀ s with probability 1 by the strong law of large numbers.

Original: For the induction step, we assume lim n →∞ d tn ( s ) =
Revised: For the induction step, we assume lim n →∞ d tn ( s ) =

Original: d tπ ( s ) ∀ s with probability 1 for some episodestep t < l .
Revised: d tπ ( s ) ∀ s with probability 1 for some episode step t < l .

Original: We want to show that this implies that lim n →∞ d t +1 n ( s ) =
Revised: We want to show that this implies that lim n →∞ d t +1 n ( s ) =

Original: d t +1 π ( s ) with probability 1for all s .
Revised: d t +1 π ( s ) with probabilityfor all s .

Original: Let P n denote the empirical state transition function, i.e., P n ( s (cid:48) | s, a ) :
Revised: Let P n denote the empirical state transition function, i.e., P n ( s (cid:48) | s, a ) :

Original: =
Revised: =

Original: c n ( s,a,s (cid:48) )
Revised: c n ( s,a,s (cid:48) )

Original: c n ( s,a ) wherec n ( s, a, s (cid:48) ) is the number of times that ( s, a, s (cid:48) ) occurred in D and similarly for c n ( s, a ) .
Revised: c n ( s,a ) where c n ( s, a, s (cid:48) ) is the number of times that ( s, a, s (cid:48) ) occurred in D and similarly for c n ( s, a ) .

Original: Note thatd t +1 n ( s ) = (cid:80) ˜ s (cid:80) a d tn (˜ s ) π D ( a | ˜ s ) P n ( s | ˜ s, a ) and d t[-+
-]
Revised: Note that d t[-+
-]

Original: π ( s ) =
Revised: [Equation]

Original:  (cid:80) ˜ s (cid:80) a d tπ ˜ s ) π ( a | ˜ s ) P ( s | ˜ s, a ) .
Revised: | ˜ s ) P n ( s | ˜ s, a ) and d t +1 π ( s ) = (cid:80) ˜ s (cid:80) a d tπ (˜ s ) π ( a | ˜ s ) P ( s | ˜ s, a ) .

Original: Theformer claim follows as a consequence of the ﬁnite-horizon MDP setting in which the state implicitlymust depend on the current time-step.
Revised: The former claim follows as a consequence of the ﬁnite-horizon MDP setting in which the state implicitly must depend on the current time-step.

Original: In this setting P n ( s (cid:48) | s, a ) and π D ( a | s ) are only computed withsamples from a particular time-step (or pair of subsequent time-steps in the case of P n ).
Revised: In this setting P n ( s (cid:48) | s, a ) and π D ( a | s ) are only computed with samples from a particular time-step (or pair of subsequent time-steps in the case of P n ).

Original: Then wehave that:
Revised: Then we have that:

Original: [Equation]
Revised: [Equation]

Original: where (a) follows from the induction hypothesis, Lemma 2, and lim n →∞ P [Equation] s 
Revised: where (a) follows from the induction hypothesis, Lemma 2, and lim n →∞ P n ( s (cid:48) | s, a )

Original: [Equation]

Original: | s, a ) = P

Original: [Equation]

Original: s

Original: [Equation] | s [Equation] a )follows from the strong law of large numbers.
Revised: = P ( s (cid:48) | s, a ) follows from the strong law of large numbers.

Original: This completes the proof.
Revised: This completes the proof.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let s be a particular state that is visited m times during data collection and assumethat |A| ≥ 2 .
Revised: Let s be a particular state that is visited m times during data collection and assume that |A| ≥ 2 .

Original: Under Assumption 1, D KL ( π D ( ·| s )
Revised: Under Assumption 2, D KL ( π D ( ·| s )

Original: ||
Revised: ||

Original: π ( ·| s ))
Revised: π ( ·| s ))

Original: =
Revised: =

Original: O p ( 1 m 2 ) under ROS sampling while
Revised: O p ( 1 m 2 ) under ROS sampling while

Original: D KL ( π D ( ·| s ) || π ( ·| s ))
Revised: D KL ( π D ( ·| s ) || π ( ·| s ))

Original: =
Revised: =

Original: O p ( 1 m ) under on-policy sampling, where O p denotes stochastic boundedness.
Revised: O p ( 1 m ) under on-policy sampling, where O p denotes stochastic boundedness.

Original: Proof.
Revised: Proof.

Original: This proof has two parts.
Revised: This proof has two parts.

Original: Since we only consider a particular state, we suppress the statedependency in policies throughout this proof, i.e., we write π ( a ) instead of π ( a | s ) .
Revised: Since we only consider a particular state, we suppress the statedependency in policies throughout this proof, i.e., we write π ( a ) instead of π ( a | s ) .

Original: The on-policy sampling rate is adapted from Theorem 1 of Mardia et al.
Revised: The on-policy sampling rate is adapted from Theorem 1 of Mardia et al.

Original: [2019] which implies that, the KLdivergence between the empirical distribution of a discrete distribution and that discrete distributionitself is O p ( 1 m ) .
Revised: [2019] which implies that, the KLdivergence between the empirical distribution of a discrete distribution and that discrete distribution itself is O p ( 1 m ) .

Original: In our case, π e is the discrete distribution and π D is the empirical distribution.
Revised: In our case, π e is the discrete distribution and π D is the empirical distribution.

Original: Then we have from Mardia et al.
Revised: Then we have from Mardia et al.

Original: that 2 mD KL ( π D || π e ) d −→ χ 2 k − 1 where k is the number ofactions.
Revised: that 2 mD KL ( π D || π e ) d −→ χ 2 k − 1 where k is the number of actions.

Original: So we have that D KL ( π D || π e )
Revised: So we have that D KL ( π D || π e )

Original: =
Revised: =

Original: O p ( 1 m ) from Lemma 5.3 of Alexanderian [2009] sinceconvergence in distribution implies bounded in probability.
Revised: O p ( 1 m ) from Lemma 5.3 of Alexanderian [2009] since convergence in distribution implies bounded in probability.

Original: Second, we obtain the ROS rate.
Revised: Second, we obtain the ROS rate.

Original: Deﬁne the vectors ˜ π e :=
Revised: Deﬁne the vectors ˜ π e :=

Original: ( π e (1) , · · · , π e ( k − 1)) and ˜ π D :=( π D (1) , · · · , π D ( k − 1)) .
Revised: ( π e (1) , · · · , π e ( k − 1)) and ˜ π D := ( π D (1) , · · · , π D ( k − 1)) .

Original: We write the KL-divergence between π e and π D as a function of these twovectors:
Revised: We write the KL-divergence between π e and π D as a function of these two vectors:

Original: [Equation]
Revised: [Equation]

Original: Let g denote the gradient of D KL (˜ π D || ˜ π e ) with respect to ˜ π D , evaluated at point ˜
Revised: Let g denote the gradient of D KL (˜ π D || ˜ π e ) with respect to ˜ π D , evaluated at point ˜

Original: π e
Revised: π e

Original: and H denotesthe Hessian matrix of D KL ( π D||
Revised: and H denotes the Hessian matrix of D KL ( π D

Original:  π e ) evaluated at point ˜ π e .
Revised: || π e ) evaluated at point ˜ π e .

Original: Clearly, g = 0 because setting π D to π ewill minimize D KL ( π D || π e ) .
Revised: Clearly, g = 0 because setting π D to π e will minimize D KL ( π D || π e ) .

Original: We take the Hessian from Mardia et al.
Revised: We take the Hessian from Mardia et al.

Original: [Equation]
Revised: [Equation]

Original:  Alternatively, we can write H = 1 π e ( k ) 11 (cid:62) + diag (˜ π − 1 e ) where diag (˜ π − 1 e ) is the ( k − 1 × k − 1) matrix with 1˜ π e on its diagonal.
Revised: i = j Alternatively, we can write H = 1 π e ( k ) 11 (cid:62) + diag (˜ π − 1 e ) where diag (˜ π − 1 e ) is the ( k − 1 × k − 1) matrix with 1˜ π e on its diagonal.

Original: Using a Taylor Series expansion for D KL , we have:
Revised: Using a Taylor Series expansion for D KL , we have:

Original: [Equation]
Revised: [Equation]

Original: where Q m consists of the higher order terms.
Revised: where Q m consists of the higher order terms.

Original: Note that since D KL ( π e || π e ) = 0 , the ﬁrst two terms areboth 

Revised: Note that since D KL ( π e || π e ) = 0 , the ﬁrst two terms are both

Original: so we only need to bound the quadratic term and higher terms.
Revised: so we only need to bound the quadratic term and higher terms.

Original: By Lemma 1 every componentof m (˜ π D − ˜ π e ) is bounded by k − 1 , so the quadratic term is bounded by 12 (cid:80) i,j H ( i, j )( k − 1) 2 ,which is of order O (1) since H is a constant when evaluated at ˜ π e .
Revised: By Lemma 1 every component of m (˜ π D − ˜ π e ) is bounded by k − 1 , so the quadratic term is bounded by 12 (cid:80) i,j H ( i, j )( k − 1) 2 , which is of order O (1) since H is a constant when evaluated at ˜ π e .

Original: The higher order terms Q m containhigher powers of (˜ π D − ˜ π e ) multiplied by m 2 .
Revised: The higher order terms Q m contain higher powers of (˜ π D − ˜ π e ) multiplied by m 2 .

Original: Using Lemma 1, we can bound all components of anym (˜ π D − ˜ π e ) by k − 1 to replace each m with a constant (trivially, all components of (˜ π D − ˜ π e ) canalso be bounded by k − 1 ) and higher-order derivatives of D KL evaluated at π e are also constant withrespect to m .
Revised: Using Lemma 1, we can bound all components of any m (˜ π D − ˜ π e ) by k − 1 to replace each m with a constant (trivially, all components of (˜ π D − ˜ π e ) can also be bounded by k − 1 ) and higher-order derivatives of D KL evaluated at π e are also constant with respect to m .

Original: Thus higher order terms are also O (1) which gives us that m 2 D KL ( π D
Revised: Thus higher order terms are also O (1) which gives us that m 2 D KL ( π D

Original: ||
Revised: ||

Original: π e )
Revised: π e )

Original: =
Revised: =

Original: O (1)and thus that D KL ( π D || π e ) =
Revised: O (1) and thus that D KL ( π D || π e ) =

Original: O ( 1 m 2 ) .
Revised: O ( 1 m 2 ) .

Original: Deterministic convergence to zero implies probabilisticconvergence and thus, under ROS action selection, D KL ( π D || π e )
Revised: Deterministic convergence to zero implies probabilistic convergence and thus, under ROS action selection, D KL ( π D || π e )

Original: =
Revised: =

Original: O p ( 1 m 2 ) .
Revised: O p ( 1 m 2 ) .

Original: Theorem 3.
Revised: Theorem 3.

Original: Assume ∀ s ∈ S , a ∈ A that R ( s, a ) ≤ R max .
Revised: Assume ∀ s ∈ S , a ∈ A that R ( s, a ) ≤ R max .

Original: The squared error in the Monte Carloestimate using D can be upper-bounded by:
Revised: The squared error in the Monte Carlo estimate using D can be upper-bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we introduce some additional notation.
Revised: First, we introduce some additional notation.

Original: We deﬁne µ t n ( s, a ) :
Revised: We deﬁne µ t n ( s, a ) :

Original: = n t ( s,a ) n as the empiricaldistribution of a state-action pair at time t , where n t ( s, a ) is the number of times that state s andaction a occur jointly at time t across trajectories in D .
Revised: = n t ( s,a ) n as the empirical distribution of a state-action pair at time t , where n t ( s, a ) is the number of times that state s and action a occur jointly at time t across trajectories in D .

Original: Let µ tπ e ( s, a ) be the probability of state sand action a occurring jointly at time t while following π e .
Revised: Let µ tπ e ( s, a ) be the probability of state s and action a occurring jointly at time t while following π e .

Original: Note that µ tn ( s, a ) =
Revised: Note that µ tn ( s, a ) =

Original: d tn ( s ) π D ( a | s ) andµ
Revised: d tn ( s ) π D ( a | s ) and µ tπ e ( s, a )

Original: tπ e ( s, a ) =
Revised: =

Original: d tπ e ( s ) π e ( a | s ) .
Revised: d tπ e ( s ) π e ( a | s ) .

Original: Finally, we deﬁne n ( s 0 , a 0 , . . .
Revised: Finally, we deﬁne n ( s 0 , a 0 , . . .

Original: , s l − 1 , a l − 1 ) as the number of times thatstate-action trajectory s 0 , a 0 , . . .
Revised: , s l − 1 , a l − 1 ) as the number of times that state-action trajectory s 0 , a 0 , . . .

Original: , s l − 1 , a l − 1 occurs in D .
Revised: , s l − 1 , a l − 1 occurs in D .

Original: Observe that the Monte Carlo estimate can be re-written as:
Revised: Observe that the Monte Carlo estimate can be re-written as:

Original: [Equation]
Revised: [Equation]

Original: Similarly, the true value of the evaluation policy can be re-written in terms of the state-actiondistribution at each time step under policy π e :
Revised: Similarly [Equation] the true value of the evaluation policy can be re-written in terms of the state-action distribution at each time step under policy π e :

Original: [Equation]
Revised: [Equation]

Original: Now, we use these alternative formulations to bound the squared error between the true value and the
Revised: Now [Equation] we use these alternative formulations to bound the squared error between the true value and the

Original: Monte Carlo estimate computed with a ﬁxed data-set D .
Revised: Monte Carlo estimate computed with a ﬁxed data-set D .

Original: [Equation]
Revised: [Equation]

Original: where (a) uses Jensen’s inequality, (b) replaces R ( s, a ) with the constant r max , (c) notes that − 1 ≥µ tn ( s, a ) − µ tπ e ( s, a ) ≤ 1 so | µ tn ( s, a ) − µ tπ e ( s, a ) | ≥ ( µ tn ( s, a ) − µ tπ e ( s, a )) 2 , and (d) uses Pinsker’sinequality.
Revised: where (a) uses Jensen’s inequality, (b) replaces R ( s, a ) with the constant r max , (c) notes that − 1 ≥ µ tn ( s, a ) − µ tπ e ( s, a ) ≤ 1 so | µ tn ( s, a ) − µ tπ e ( s, a ) | ≥ ( µ tn ( s, a ) − µ tπ e ( s, a )) 2 , and (d) uses Pinsker’s inequality.

Original: All that remains is to use the deﬁnition of the KL-divergence and properties of logarithms and expectations to obtain the ﬁnal form of the bound: [Equation] − 1 (cid:88)
Revised: All that remains is to use the deﬁnition of the KL-divergence and properties of logarithms and expectations to obtain the ﬁnal form of the bound: l − 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: C Experiment Domains
Revised: C Experiment Domains

Original: [Figure]

Original: This appendix provides additional details on our experimental set-up.
Revised: This appendix provides additional details on our experimental set-up.

Original: C.1 Extended Domain Descriptions
Revised: C.1 Extended Domain Descriptions

Original: [Figure]

Original: This section describes the domains used in our empirical evaluation.
Revised: This section describes the domains used in our empirical evaluation.

Original: Figure 5 illustrates each domain.
Revised: Figure 5 illustrates each domain.

Original: [Figure]
Revised: [Figure]

Original: Our ﬁrst domain, Bandit, is a 30-armed bandit problem modelled after the 10-armed bandit in Suttonand Barto [1998, Chapter 2].
Revised: Our ﬁrst domain, Bandit, is a 30-armed bandit problem modelled after the 10-armed bandit in Sutton and Barto [1998, Chapter 2].

Original: This domain has a single state, 30 actions, and episodes terminate afterthe ﬁrst action is taken.
Revised: This domain has a single state, 30 actions, and episodes terminate after the ﬁrst action is taken.

Original: The reward following each action is normally distributed with a randomlygenerated mean and scale parameter.
Revised: The reward following each action is normally distributed with a randomly generated mean and scale parameter.

Original: The reward distribution parameters are sampled from a uniformdistribution on [0 , 1] at the start of each experimental trial.
Revised: The reward distribution parameters are sampled from a uniform distribution on [0 , 1] at the start of each experimental trial.

Original: Our second domain, GridWorld, is a discrete state and action domain that has been used in priorpolicy evaluation research (e.g., [Thomas and Brunskill, 2016, Farajtabar et al., 2018]).
Revised: Our second domain, GridWorld, is a discrete state and action domain that has been used in prior policy evaluation research (e.g., [Thomas and Brunskill, 2016, Farajtabar et al., 2018]).

Original: The domain(shown in Figure 5(b)) has 4 × 4 states.
Revised: The domain (shown in Figure 5(b)) has 4 × 4 states.

Original: The agent starts from (0 , 0) and has the action space {left,right, up, down}.
Revised: The agent starts from (0 , 0) and has the action space {left, right, up, down}.

Original: The reward is − 1 in all non-terminal states except (1 , 1) where the reward is − 10and (1 , 3) where the reward is +1 .
Revised: The reward is − 1 in all non-terminal states except (1 , 1) where the reward is −and (1 , 3) where the reward is +1 .

Original: The agent receives +10 in terminal state (3 , 3) .
Revised: The agent receives +10 in terminal state (3 , 3) .

Original: The maximumnumber of steps is 100 and γ = 1 in this domain.
Revised: The maximum number of steps is 100 and γ = 1 in this domain.

Original: Our last two domains are based on the CartPole problem from OpenAI Gym [Brockman et al., 2016].
Revised: Our last two domains are based on the CartPole problem from OpenAI Gym [Brockman et al., 2016].

Original: In CartPole, the agent tries to balance a pole, mounted on a cart, by moving the cart left and right.
Revised: In CartPole, the agent tries to balance a pole, mounted on a cart, by moving the cart left and right.

Original: States are given as vectors that describe the cart position and velocity and pole angle and angular velocity.
Revised: States are given as vectors that describe the cart position and velocity and pole angle and angular velocity.

Original: Our third domain is the standard variant in which the agent controls the cart by choosingeither a constant leftward force of − 1 , no force, or a constant rightward force of +1 .
Revised: Our third domain is the standard variant in which the agent controls the cart by choosing either a constant leftward force of − 1 , no force, or a constant rightward force of +1 .

Original: Our fourthdomain is a continuous control variant in which the agent can control the exact force ranging from− 1 to 1 .
Revised: Our fourth domain is a continuous control variant in which the agent can control the exact force ranging from − 1 to 1 .

Original: In either case, the agent receives a reward of +1 for each time-step until termination whenthe pole falls or the cart moves out of bounds.
Revised: In either case, the agent receives a reward of +1 for each time-step until termination when the pole falls or the cart moves out of bounds.

Original: The maximum number of steps is 200 and γ = 0 .
Revised: The maximum number of steps is 200 and γ = 0 .

Original: 99 inthis domain.
Revised: 99 in this domain.

Original: C.2 Creation of Pre-trained Evaluation Policy
Revised: C.2 Creation of Pre-trained Evaluation Policy

Original: Each domain requires creation of an evaluation policy to serve as π e .
Revised: Each domain requires creation of an evaluation policy to serve as π e .

Original: In the three domains with adiscrete action space we use softmax policies of the form:
Revised: In the three domains with a discrete action space we use softmax policies of the form:

Original: [Equation]
Revised: [Equation]

Original: (13) where φ is a one-hot encoding operator for domains with discrete state space (Bandit and GridWorld)and a feed-forward neural network for continuous state spaces (CartPole).
Revised: (13) where φ is a one-hot encoding operator for domains with discrete state space (Bandit and GridWorld) and a feed-forward neural network for continuous state spaces (CartPole).

Original: For ContinuousCartPole,
Revised: For ContinuousCartPole,

Original: [Equation]
Revised: [Equation]

Original: (14) where φ is a function of the state given by a feed-forward neural network.
Revised: (14) where φ is a function of the state given by a feed-forward neural network.

Original: The policy parameters, θ ,denote all policy parameters.
Revised: The policy parameters, θ , denote all policy parameters.

Original: For Bandit and GridWorld, this is just the vectors w a and for CartPoleand ContinuousCartPole, θ also includes the neural network weights and biases.
Revised: For Bandit and GridWorld, this is just the vectors w a and for CartPole and ContinuousCartPole, θ also includes the neural network weights and biases.

Original: When φ is a neuralnetwork, it is constructed with one batch normalization layer as the ﬁrst layer, followed by two hiddenlayers, both of which have 64 hidden states and use ReLU as the activation function.
Revised: When φ is a neural network, it is constructed with one batch normalization layer as the ﬁrst layer, followed by two hidden layers, both of which have 64 hidden states and use ReLU as the activation function.

Original: We use PyTorchfor neural network implementations [Paszke et al., 2019] and NumPy for linear algebra computations[Harris et al., 2020].
Revised: We use PyTorch for neural network implementations [Paszke et al., 2019] and NumPy for linear algebra computations [Harris et al., 2020].

Original: For all domains, we use REINFORCE
Revised: For all domains, we use REINFORCE

Original: [Williams, 1992] to train the policy model, and choose a policysnapshot during training as the evaluation policy, which has higher returns than the uniformly randompolicy, but is still far from convergence.
Revised: [Williams, 1992] to train the policy model, and choose a policy snapshot during training as the evaluation policy, which has higher returns than the uniformly random policy, but is still far from convergence.

Original: To obtain v ( π e ) , we use on-policy sampling to collect 10 6trajectories and compute the Monte Carlo estimate of v ( π e ) .
Revised: To obtain v ( π e ) , we use on-policy sampling to collect 10trajectories and compute the Monte Carlo estimate of v ( π e ) .

Original: C.3 Off-policy Data for With Initial Data Experiments
Revised: C.3 Off-policy Data for With Initial Data Experiments

Original: To create an initial data set of slightly off-policy data for each domain, we create the behavior policy π b based on the evaluation policy π e .
Revised: To create an initial data set of slightly off-policy data for each domain, we create the behavior policy π b based on the evaluation policy π e .

Original: For domains with discrete action space, the off-policy behavior policy is built as:
Revised: For domains with discrete action space, the off-policy behavior policy is built as:

Original: [Equation]
Revised: [Equation]

Original: where δ ∈ (0 , 1] controls the probability of randomly choosing an action from the action space, and otherwise sampling an action from the evaluation policy π e .
Revised: where δ ∈ (0 , 1] controls the probability of randomly choosing an action from the action space, and otherwise sampling an action from the evaluation policy π e .

Original: For ContinuousCartPole, the behavior policy is built as:
Revised: For ContinuousCartPole, the behavior policy is built as:

Original: [Equation]
Revised: [Equation]

Original: where µ e ( s ) and σ e ( s ) are the output mean and standard deviation of the evaluation policy π e in states , and δ (for δ > 0 ) increases the standard deviation.
Revised: where µ e ( s ) and σ e ( s ) are the output mean and standard deviation of the evaluation policy π e in state s , and δ (for δ > 0 ) increases the standard deviation.

Original: In all domains, we use δ = 0 .
Revised: In all domains, we use δ = 0 .

Original: 1 and collect 100trajectories to create the initial off-policy data set.
Revised: 1 and collecttrajectories to create the initial off-policy data set.

Original: D Measuring Sampling Error
Revised: D Measuring Sampling Error

Original: One of our central claims is that ROS reduces sampling error and reducing sampling error correspondsto a reduction in MSE for policy evaluation.
Revised: One of our central claims is that ROS reduces sampling error and reducing sampling error corresponds to a reduction in MSE for policy evaluation.

Original: To evaluate this claim, we must deﬁne a metric formeasuring sampling error.
Revised: To evaluate this claim, we must deﬁne a metric for measuring sampling error.

Original: In this appendix, we describe two possible metrics and demonstrate howthese metrics change as OS and ROS collect data for policy evaluation.
Revised: In this appendix, we describe two possible metrics and demonstrate how these metrics change as OS and ROS collect data for policy evaluation.

Original: D.1 KL-divergence of Data Collection
Revised: D.1 KL-divergence of Data Collection

Original: Our ﬁrst metric (also used in the main paper), is to measure sampling error in the collected data D byusing KL-divergence of the empirical policy π D and the evaluation policy π e .
Revised: Our ﬁrst metric (also used in the main paper), is to measure sampling error in the collected data D by using KL-divergence of the empirical policy π D and the evaluation policy π e .

Original: The KL-divergence between π D and π e in a particular state s is deﬁned as:
Revised: The KL-divergence between π D and π e in a particular state s is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: To obtain an explicit representation of π D , we use a parametric estimate by maximizing the loglikelihood function over a parametric policy class.
Revised: To obtain an explicit representation of π D , we use a parametric estimate by maximizing the loglikelihood function over a parametric policy class.

Original: Speciﬁcally, we use:
Revised: Speciﬁcally, we use:

Original: [Equation]
Revised: [Equation]

Original: for a policy class parameterized by θ .
Revised: for a policy class parameterized by θ .

Original: We then use π ˆ θ in place of π D when computing the KLdivergence.
Revised: We then use π ˆ θ in place of π D when computing the KLdivergence.

Original: Our ﬁnal sampling error metric for a data set D 1 is:
Revised: Our ﬁnal sampling error metric for a data set D 1 is:

Original: [Equation]
Revised: [Equation]

Original: The sampling error curves of data collection without and with initial data are shown in Figure 6and 7, respectively.
Revised: The sampling error curves of data collection without and with initial data are shown in Figureand 7, respectively.

Original: We observe from these experiments that ROS can collect data with lower samplingerror than OS and BPG .
Revised: We observe from these experiments that ROS can collect data with lower sampling error than OS and BPG .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D.2 l1-Norm of the Log-likelihood Gradient
Revised: D.2 l1-Norm of the Log-likelihood Gradient

Original: In this section, we propose an alternative sampling error measure based on the norm of the loglikelihood gradient evaluated at π e .
Revised: In this section [Figure] we propose an alternative sampling error measure based on the norm of the loglikelihood gradient evaluated at π e .

Original: If π e is the maximum likelihood policy under D then the norm ofthe gradient evaluated at θ e will be zero when sampling error is zero.
Revised: If π e is the maximum likelihood policy under D then the norm of the gradient evaluated at θ e will be zero when sampling error is zero.

Original: A non-zero norm indicates theparameters must change from θ e to maximize the log-likelihood under the observed data.
Revised: A non-zero norm indicates the parameters must change from θ e to maximize the log-likelihood under the observed data.

Original: We showempirically that this measure of sampling error roughly corresponds to using the KL-divergence bycomputing the gradient norm of log-likelihood with respect to the data collection without and withinitial data, shown in Figure 8 and 9, respectively.
Revised: We show empirically that this measure of sampling error roughly corresponds to using the KL-divergence by computing the gradient norm of log-likelihood with respect to the data collection without and with initial data, shown in Figure 8 and 9, respectively.

Original: These curves are generally consistent with theircorresponding sampling error curves in Appendix D.1.
Revised: These curves are generally consistent with their corresponding sampling error curves in Appendix D.1.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E Hyper-parameter Conﬁgurations
Revised: E Hyper-parameter Conﬁgurations

Original: This appendix gives the hyper-parameter settings for the policy evaluation experiments.
Revised: This appendix gives the hyper-parameter settings for the policy evaluation experiments.

Original: Settingsfor the without initial data experiments are given in Table 1 and settings for the with initial dataexperiments are given in Table2.
Revised: Settings for the without initial data experiments are given in Table 1 and settings for the with initial data experiments are given in Table

Original: For BPG , k denotes the batch-size and α the step-size.

Original: Domain

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F Numerical Results of Policy Evaluation with Initial Data
Revised: F Numerical Results of Policy Evaluation with Initial Data

Original: This appendix provides the numerical ﬁnal values for the MSE of policy evaluation with each datacollection method.
Revised: This appendix provides the numerical ﬁnal values for the MSE of policy evaluation with each data collection method.

Original: Table 3 gives these values for the without initial data experiments and Table 4gives these values for the with initial data experiments.
Revised: Table 3 gives these values for the without initial data experiments and Tablegives these values for the with initial data experiments.

Original: These tables provide the numerical valuecorresponding to the ﬁnal MSE value for each method-domain pair shown in Figures 2 and3.
Revised: These tables provide the numerical value corresponding to the ﬁnal MSE value for each method-domain pair shown in Figures 2 and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G Median and Inter-quartile Range of Policy Evaluation
Revised: G Median and Inter-quartile Range of Policy Evaluation

Original: In this appendix, we present the computed median and interquartile range for the squared error ofpolicy evaluation both with and without initial data across all four domains.
Revised: In this appendix [Table] we present the computed median and interquartile range for the squared error of policy evaluation both with and without initial data across all four domains.

Original: In the main paper wepresent the mean squared error and standard error as is typical in the policy evaluation literature.
Revised: In the main paper we present the mean squared error and standard error as is typical in the policy evaluation literature.

Original: Here, we include the median and interquartile ranges as they are more robust statistics.
Revised: Here, we include the median and interquartile ranges as they are more robust statistics.

Original: We give theseresults for completeness; qualitatively, they leave the conclusions from the main paper unchanged.
Revised: We give these results for completeness; qualitatively, they leave the conclusions from the main paper unchanged.

Original: Figure 10 shows the median and interquartile ranges for policy evaluation without initial data .
Revised: Figure 10 shows the median and interquartile ranges for policy evaluation without initial data .

Original: Figure 11 gives the same for policy evaluation with initial data .
Revised: Figure 11 gives the same for policy evaluation with initial data .

Original: From these ﬁgures, we can observethat all data collection methords have a similar inter-quartile range and ROS lowers the median of thesquared error compared to OS .
Revised: From these ﬁgures, we can observe that all data collection methords have a similar inter-quartile range and ROS lowers the median of the squared error compared to OS .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H Environment and Policy Sensitivity in GridWorld
Revised: H Environment and Policy Sensitivity in GridWorld

Original: In the main paper, we evaluated the relative MSE of ROS compared to OS in the Bandit environmentunder different settings of the reward scale and variance and the stochasticity of π e .
Revised: In the main paper, we evaluated the relative MSE of ROS compared to OS in the Bandit environment under different settings of the reward scale and variance and the stochasticity of π e .

Original: In this appendix,we repeat the same study in the GridWorld environment.
Revised: In this appendix, we repeat the same study in the GridWorld environment.

Original: The original GridWorld domain has ﬁxedrewards for each state.
Revised: The original GridWorld domain has ﬁxed rewards for each state.

Original: We vary these by either 1) multiplying by a ﬁxed factor (referred to as the meanfactor) or 2) replacing the deterministic reward with a reward sampled uniformly from [ − f scale , f scale ]where f scale determines the reward variance.
Revised: We vary these by either 1) multiplying by a ﬁxed factor (referred to as the mean factor) or 2) replacing the deterministic reward with a reward sampled uniformly from [ − f scale , f scale ] where f scale determines the reward variance.

Original: We then follow OS or ROS ( α = 1000 ) to collect 1000trajectories, and perform Monte Carlo estimation.
Revised: We then follow OS or ROS ( α = 1000 ) to collecttrajectories, and perform Monte Carlo estimation.

Original: The relative MSE s between OS and ROS are shownin Figure 12(a).
Revised: The relative MSE s between OS and ROS are shown in Figure 12(a).

Original: Results show an identical trend to the Bandit environment: a larger reward scaleincreases the amount of improvement because small amounts of sampling error can lead to largeramounts of error; larger reward variance decreases the amount of improvement because the MSEbecomes dominated by variance in the reward.
Revised: Results show an identical trend to the Bandit environment: a larger reward scale increases the amount of improvement because small amounts of sampling error can lead to larger amounts of error; larger reward variance decreases the amount of improvement because the MSE becomes dominated by variance in the reward.

Original: We also create different evaluation policies using (cid:15) -greedy policies with (cid:15) ranging from 0 to 1 .
Revised: We also create different evaluation policies using (cid:15) -greedy policies with (cid:15) ranging from 0 to 1 .

Original: We thenperform the same policy evaluation as above and compute the relative MSE s, shown in Figure 12(b).
Revised: We then perform the same policy evaluation as above and compute the relative MSE s, shown in Figure 12(b).

Original: As with the Bandit domain, we see that greater entropy in π e generally leads to a wider margin ofimprovement between ROS and OS .
Revised: As with the Bandit domain, we see that greater entropy in π e generally leads to a wider margin of improvement between ROS and OS .

Original: [Figure]
Revised: [Figure]

Original: I Mean Squared Error of Policy Evaluation with WRIS and FQE
Revised: I Mean Squared Error of Policy Evaluation with WRIS and FQE

Original: This appendix presents preliminary results on combining ROS with estimators speciﬁcally designedfor the off-policy setting instead of the Monte Carlo estimator.
Revised: This appendix presents preliminary results on combining ROS with estimators speciﬁcally designed for the off-policy setting instead of the Monte Carlo estimator.

Original: Speciﬁcally, we used weightedregression importance sampling ( WRIS )
Revised: Speciﬁcally, we used weighted regression importance sampling ( WRIS )

Original: [Hanna et al., 2021] and ﬁtted q-evaluation ( FQE )
Revised: [Hanna et al., 2021] and ﬁtted q-evaluation ( FQE )

Original: [Le et al.
Revised: [Le et al., 2019].

Original: We study these estimators in the without initial data setting and show their MSE with varyingamounts of data.
Revised: We study these estimators in the without initial data setting and show their MSE with varying amounts of data.

Original: Figure 13 shows the MSE of different data collection methods for WRIS across thedifferent domains.
Revised: Figure 13 shows the MSE of different data collection methods for WRIS across the different domains.

Original: Figure 14 shows the same except with FQE as the estimator.
Revised: Figure 14 shows the same except with FQE as the estimator.

Original: We observe in tabular domains that OS cannot obtain the same level MSE as ROS , even with the helpof WRIS , which is designed to correct sampling error during the value estimation stage.
Revised: We observe in tabular domains that OS cannot obtain the same level MSE as ROS , even with the help of WRIS , which is designed to correct sampling error during the value estimation stage.

Original: This showsthe importance of reducing sampling error during the data collection stage.
Revised: This shows the importance of reducing sampling error during the data collection stage.

Original: The same pattern canalso be observed when using FQE .
Revised: The same pattern can also be observed when using FQE .

Original: In non-tabular domains, the performances of off-policy evaluationmethods with different data collection methods are very similar.
Revised: In non-tabular domains, the performances of off-policy evaluation methods with different data collection methods are very similar.

Original: However, WRIS usually requireslarger sizes of data to make accurate estimation, and can only achieve similar MSE as ROS whencollecting a large amount of data.
Revised: However, WRIS usually requires larger sizes of data to make accurate estimation, and can only achieve similar MSE as ROS when collecting a large amount of data.

Original: When using FQE in non-tabular domains, data from ROS cangenerally enable lower MSE than OS , although this improvement is very small.
Revised: When using FQE in non-tabular domains, data from ROS can generally enable lower MSE than OS , although this improvement is very small.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]