Original: Robust On-Policy Sampling for Data-Efﬁcient Policy Evaluation
Revised: Robust On-Policy Sampling for Data-Efﬁcient Policy Evaluation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Reinforcement learning ( RL ) algorithms are often categorized as either on-policyor off-policy depending on whether they use data from a target policy of interestor from a different behavior policy.
Revised: Reinforcement learning ( RL ) algorithms are often categorized as either on-policyor off-policy depending on whether they use data from a target policy of interestor from a different behavior policy.

Original: In this paper, we study a subtle distinction between on-policy data and on-policy sampling in the context of the RL sub-problemof policy evaluation.
Revised: In this paper, we study a subtle distinction between on-policy data and on-policy sampling in the context of the RL sub-problemof policy evaluation.

Original: We observe that on-policy sampling may fail to match theexpected distribution of on-policy data after observing only a ﬁnite number of trajectories and this failure hinders data-efﬁcient policy evaluation.
Revised: We observe that on-policy sampling may fail to match theexpected distribution of on-policy data after observing only a ﬁnite number of trajectories and this failure hinders data-efﬁcient policy evaluation.

Original: Towards improveddata-efﬁciency, we show how non-i.i.d., off-policy sampling can produce data thatmore closely matches the expected on-policy data distribution and consequentlyincreases the accuracy of the Monte Carlo estimator for policy evaluation.
Revised: Towards improveddata-efﬁciency, we show how non-i.i.d., off-policy sampling can produce data thatmore closely matches the expected on-policy data distribution and consequentlyincreases the accuracy of the Monte Carlo estimator for policy evaluation.

Original: We introduce a method called Robust On-policy Sampling and demonstrate theoreticallyand empirically that it produces data that converges faster to the expected on-policydistribution compared to on-policy sampling.
Revised: We introduce a method called Robust On-policy Sampling and demonstrate theoreticallyand empirically that it produces data that converges faster to the expected on-policydistribution compared to on-policy sampling.

Original: Empirically, we show that this fasterconvergence leads to lower mean squared error policy value estimates.
Revised: Empirically, we show that this fasterconvergence leads to lower mean squared error policy value estimates.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning ( RL ) algorithms are often categorized using the dichotomy of on-policyversus off-policy.
Revised: Reinforcement learning ( RL ) algorithms are often categorized using the dichotomy of on-policyversus off-policy.

Original: On-policy algorithms learn about a particular target policy using data collectedby behaving according to the target policy.
Revised: On-policy algorithms learn about a particular target policy using data collectedby behaving according to the target policy.

Original: Off-policy algorithms use data collected by behavingaccording to a different behavior policy.
Revised: Off-policy algorithms use data collected by behavingaccording to a different behavior policy.

Original: We study a subtle distinction between on-policy data versuson-policy sampling as a step towards more data-efﬁcient RL algorithms.
Revised: We study a subtle distinction between on-policy data versuson-policy sampling as a step towards more data-efﬁcient RL algorithms.

Original: To better understand thisdistinction, consider a simple example.
Revised: To better understand thisdistinction, consider a simple example.

Original: In this example, a certain target policy repeatedly visits a statein which it takes action A with probability 0 .
Revised: In this example, a certain target policy repeatedly visits a statein which it takes action A with probability 0 .

Original: 2 and action B with probability 0 .
Revised: 2 and action B with probability 0 .

Original: Under on-policysampling, after ﬁve visits to this state, we might actually observe action A 2 times and action B3 times instead of the expected 1 and 4 times.
Revised: Under on-policysampling, after ﬁve visits to this state, we might actually observe action A 2 times and action B3 times instead of the expected 1 and 4 times.

Original: Alternatively, we could collect data off-policy bydeterministically tracking the expected target policy action proportions resulting in the exact expectedfrequencies.
Revised: Alternatively, we could collect data off-policy bydeterministically tracking the expected target policy action proportions resulting in the exact expectedfrequencies.

Original: Though the latter case uses off-policy sampling, it produces data that is arguably moreon-policy than the data produced by on-policy sampling.
Revised: Though the latter case uses off-policy sampling, it produces data that is arguably moreon-policy than the data produced by on-policy sampling.

Original: In this paper, we study the distinction between on-policy sampling and on-policy data in the contextof the RL sub-problem of policy evaluation [Zinkevich et al., 2006].
Revised: In this paper, we study the distinction between on-policy sampling and on-policy data in the contextof the RL sub-problem of policy evaluation [Zinkevich et al., 2006].

Original: In policy evaluation, we are givenan evaluation policy and asked to estimate the expected return that would be accrued when runningthe evaluation policy on a task of interest.
Revised: In policy evaluation, we are givenan evaluation policy and asked to estimate the expected return that would be accrued when runningthe evaluation policy on a task of interest.

Original: This problem is important for high conﬁdence deploymentof RL -trained policies.
Revised: This problem is important for high conﬁdence deploymentof RL -trained policies.

Original: In RL applications, such as robotics, data-efﬁcient policy evaluation is ofthe utmost importance – we desire the most accurate estimate with minimal collected data.
Revised: In RL applications, such as robotics, data-efﬁcient policy evaluation is ofthe upmost importance – we desire the most accurate estimate with minimal collected data.

Original: Whilemuch research has gone into how to most efﬁciently use a set of already collected data, i.e., theoff-policy policy evaluation problem, an implicit assumption in the RL community is that on-policydata is preferred to off-policy data when available.
Revised: Whilemuch research has gone into how to most efﬁciently use a set of already collected data, i.e., theoff-policy policy evaluation problem, an implicit assumption in the RL community is that on-policydata is preferred to off-policy data when available.

Original: When data can be collected on-policy, we can usethe Monte Carlo estimator which computes a mean return estimate using trajectories sampled i.i.d.by running the evaluation policy.
Revised: When data can be collected on-policy, we can usethe Monte Carlo estimator which computes a mean return estimate using trajectories sampled i.i.d.by running the evaluation policy.

Original: In the limit, with inﬁnite trajectories, the empirical proportion of each trajectory will converge to its true probability under the evaluation policy and the estimate willconverge to the true expected return.
Revised: In the limit, with inﬁnite trajectories, the empirical proportion of each trajectory will converge to its true probability under the evaluation policy and the estimate willconverge to the true expected return.

Original: However, for any ﬁnite sample-size, the empirical proportion ofeach trajectory will likely fail to match the true probability and the estimate will have error.
Revised: However, for any ﬁnite sample-size, the empirical proportion ofeach trajectory will likely fail to match the true probability and the estimate will have error.

Original: Suchsampling error is an inevitable feature of i.i.d.
Revised: Suchsampling error is an inevitable feature of i.i.d.

Original: sampling.
Revised: sampling.

Original: The probability of each new trajectory isunaffected by the trajectories occurring in the past and thus the only way to ensure the empiricaldistribution matches the true probability is to sample a large enough data set.
Revised: The probability of each new trajectory isunaffected by the trajectories occurring in the past and thus the only way to ensure the empiricaldistribution matches the true probability is to sample a large enough data set.

Original: That is, it is only in thelimit that on-policy sampling produces exactly on-policy data.
Revised: That is, it is only in thelimit that on-policy sampling produces exactly on-policy data.

Original: The observations made so far raise the question: “can non-i.i.d., off-policy trajectory sampling causethe empirical distribution of trajectories to converge to the expected on-policy distribution faster?"
Revised: The observations made so far raise the question: “can non-i.i.d., off-policy trajectory sampling causethe empirical distribution of trajectories to converge to the expected on-policy distribution faster?"

Original: Weanswer this question afﬁrmatively by introducing a method that adapts the data collecting behaviorpolicy to consider what data has already been collected when selecting future actions.
Revised: Weanswer this question afﬁrmatively by introducing a method that adapts the data collecting behaviorpolicy to consider what data has already been collected when selecting future actions.

Original: We callthis method Robust On-policy Sampling ( ROS ) since the empirical distribution of data it producesconverges faster to the expected on-policy trajectory distribution compared to standard on-policysampling.
Revised: We callthis method Robust On-policy Sampling ( ROS ) since the empirical distribution of data it producesconverges faster to the expected on-policy trajectory distribution compared to standard on-policysampling.

Original: We give a theoretical result supporting this claim and then conﬁrm our theory with policyevaluation experiments in ﬁnite and continuous-valued state- and action-space domains showing 1)
Revised: We give a theoretical result supporting this claim and then conﬁrm our theory with policyevaluation experiments in ﬁnite and continuous-valued state- and action-space domains showing 1)

Original: ROS reduces sampling error in ﬁnite datasets and 2) consequently lowers the MSE of policy valueestimates compared to i.i.d.
Revised: ROS reduces sampling error in ﬁnite datasets and 2) consequently lowers the MSE of policy valueestimates compared to i.i.d.

Original: on-policy sampling.
Revised: on-policy sampling.

Original: Our paper contributes to the ﬁeld of RL on two fronts.
Revised: Our paper contributes to the ﬁeld of RL on two fronts.

Original: On one front, we introduce a practical methodfor data collection and demonstrate empirically that it leads to more accurate policy evaluationcompared to on-policy sampling.
Revised: On one front, we introduce a practical methodfor data collection and demonstrate empirically that it leads to more accurate policy evaluationcompared to on-policy sampling.

Original: Simultaneously, our work examines nuance in the on-policy versusoff-policy dichotomy.
Revised: Simultaneously, our work examines nuance in the on-policy versusoff-policy dichotomy.

Original: A better understanding of this nuance opens up the possibility of designingnew data collection procedures to improve the data efﬁciency of any RL algorithm that relies uponon-policy data.
Revised: A better understanding of this nuance opens up the possibility of designingnew data collection procedures to improve the data efﬁciency of any RL algorithm that relies uponon-policy data.

Original:  Related Work
Revised:  Related Work

Original: Data collection is a fundamental part of the RL problem.
Revised: Data collection is a fundamental part of the RL problem.

Original: The most widely studied data collectionproblem is the question of how an agent should explore its environment to learn an optimal policy [Ostrovski et al., 2017, Tang et al., 2017].
Revised: The most widely studied data collectionproblem is the question of how an agent should explore its environment to learn an optimal policy [Ostrovski et al., 2017, Tang et al., 2017].

Original: In contrast to these approaches, our work focuses on thequestion of how an agent should collect data to evaluate a ﬁxed policy.
Revised: In contrast to these approaches, our work focuses on thequestion of how an agent should collect data to evaluate a ﬁxed policy.

Original: When given a choice of howto collect data for policy evaluation, on-policy data collection is generally preferable to off-policydata collection [Sutton and Barto, 1998].
Revised: When given a choice of howto collect data for policy evaluation, on-policy data collection is generally preferable to off-policydata collection [Sutton and Barto, 1998].

Original: Notable exceptions are adaptive importance sampling ( AIS )methods [Hanna et al., 2017, Oosterhuis and de Rijke, 2020, Ciosek and Whiteson, 2017, Bouchardet al., 2016, Frank et al., 2008] and quasi-Monte Carlo methods [Arnold et al., 2022].
Revised: Notable exceptions are adaptive importance sampling ( AIS )methods [Hanna et al., 2017, Oosterhuis and de Rijke, 2020, Ciosek and Whiteson, 2017, Bouchardet al., 2016, Frank et al., 2008] and quasi-Monte Carlo methods [Arnold et al., 2022].

Original: Both these
Revised: Both these

Original: AIS methods and the Quasi-Monte Carlo method of Arnold et al.
Revised: AIS methods and the Quasi-Monte Carlo method of Arnold et al.

Original: [2022] lower variance in estimatescomputed with future samples while our method lowers the total error in the estimate computed fromboth past and future samples.
Revised: [2022] lower variance in estimatescomputed with future samples while our method lowers the total error in the estimate computed fromboth past and future samples.

Original: In one of our experiments, we consider a setting where we already have some data (collected offpolicy) and must decide how to collect additional data for policy evaluation.
Revised: In one of our experiments, we consider a setting where we already have some data (collected offpolicy) and must decide how to collect additional data for policy evaluation.

Original: This problem has beenpreviously studied in the bandit literature [Tucker and Joachims, 2022] or when there are only a ﬁnitenumber of policies that could be ran [Konyushkova et al., 2021].
Revised: This problem has beenpreviously studied in the bandit literature [Tucker and Joachims, 2022] or when there are only a ﬁnitenumber of policies that could be ran [Konyushkova et al., 2021].

Original: These prior works also show thaton-policy data collection is a sub-optimal choice.
Revised: These prior works also show thaton-policy data collection is a sub-optimal choice.

Original: They differ from (and are complementary to) ourwork in that they still use i.i.d.
Revised: They differ from (and are complementary to) ourwork in that they still use i.i.d.

Original: sampling for data collection whereas we show how non-independentsampling can be used to produce data that more closely matches a desired distribution.
Revised: sampling for data collection whereas we show how non-independentsampling can be used to produce data that more closely matches a desired distribution.

Original: The method we introduce in this paper is motivated by the idea of decreasing sampling error in allcollected data.
Revised: The method we introduce in this paper is motivated by the idea of decreasing sampling error in allcollected data.

Original: Previous work has considered how sampling error can be reduced after data collectionby re-weighting the obtained samples.
Revised: Previous work has considered how sampling error can be reduced after data collectionby re-weighting the obtained samples.

Original: For example, Hanna et al.
Revised: For example, Hanna et al.

Original: [2021] show how importancesampling with an estimated behavior policy can lower sampling error and lead to more accurate policyevaluation.
Revised: [2021] show how importancesampling with an estimated behavior policy can lower sampling error and lead to more accurate policyevaluation.

Original: Similar methods have also been studied for policy evaluation in multi-armed bandits [Liet al., 2015, Narita et al., 2019] and temporal-difference learning [Pavse et al., 2020].
Revised: Similar methods have also been studied for policy evaluation in multi-armed bandits [Liet al., 2015, Narita et al., 2019] and temporal-difference learning [Pavse et al., 2020].

Original: These priorworks assume data is available a priori and ignore the question of how to collect it when unavailable.
Revised: These priorworks assume data is available a priori and ignore the question of how to collect it when unavailable.

Original: Finally, the idea of adapting the sampling distribution , (i.e., behavior policy) has analogs outsideof policy evaluation in Markov decision processes.
Revised: Finally, the idea of adapting the sampling distribution , (i.e., behavior policy) has analogs outsideof policy evaluation in Markov decision processes.

Original: O’Hagan [1987] identiﬁes ﬂaws with i.i.d.sampling for Monte Carlo estimation that motivate taking past samples into account.
Revised: O’Hagan [1987] identiﬁes ﬂaws with i.i.d.sampling for Monte Carlo estimation that motivate taking past samples into account.

Original: Rasmussen and
Revised: Rasmussen and

Original: Ghahramani [2003] use Gaussian processes to represent uncertainty in an expectation to be evaluatedand use this uncertainty to guide future sample generation.
Revised: Ghahramani [2003] use Gaussian processes to represent uncertainty in an expectation to be evaluatedand use this uncertainty to guide future sample generation.

Original: We are unaware of any adaptations ofthese ideas for policy evaluation in RL .
Revised: We are unaware of any adaptations ofthese ideas for policy evaluation in RL .

Original:  Preliminaries
Revised:  Preliminaries

Original: In this section, we introduce notation, formalize the policy evaluation problem, and introduce the
Revised: In this section, we introduce notation, formalize the policy evaluation problem, and introduce the

Original: Monte Carlo estimator for policy evaluation.
Revised: Monte Carlo estimator for policy evaluation.

Original: 3.1 Notation
Revised: 3.1 Notation

Original: We assume the environment is a ﬁnite-horizon, episodic Markov decision process ( MDP ) with stateset S , action set A , transition function, P : S ⇥
Revised: We assume the environment is a ﬁnite-horizon, episodic Markov decision process ( MDP ) with stateset S , action set A , transition function, P : S ⇥

Original: A ⇥ S !
Revised: A ⇥ S !

Original: [0 , 1] , reward function R : S ⇥ A !
Revised: [0 , 1] , reward function R : S ⇥ A !

Original: R , 99 discount factor  , maximum horizon l , and initial state distribution d 0 [Puterman, 2014].
Revised: R , 99 discount factor  , maximum horizon l , and initial state distribution d 0 [Puterman, 2014].

Original: We assumethat S and A are ﬁnite though our empirical analysis considers both settings.
Revised: We assumethat S and A are ﬁnite though our empirical analysis considers both settings.

Original: We assume that the 101 transition and reward functions are unknown.
Revised: We assume that the 101 transition and reward functions are unknown.

Original: A policy, ⇡ :
Revised: A policy, ⇡ :

Original: S ⇥ A !
Revised: S ⇥ A !

Original: [0 , 1] , is a function mappingstates and actions to probabilities.
Revised: [0 , 1] , is a function mappingstates and actions to probabilities.

Original: We use ⇡ ( a | s ) :
Revised: We use ⇡ ( a | s ) :

Original: = ⇡ ( s, a ) to denote the conditional probability ofaction a given state s and P ( s 0 | s, a ) :=
Revised: = ⇡ ( s, a ) to denote the conditional probability ofaction a given state s and P ( s 0 | s, a ) :=

Original: P ( s, a, s 0 ) to denote the conditional probability of state s 0given state s and action a .
Revised: P ( s, a, s 0 ) to denote the conditional probability of state s 0given state s and action a .

Original: Since we assume a ﬁnite-horizon, we assume the state deﬁnition implicitlyincludes temporal information [Agarwal et al.].

Original: Let h :
Revised: Let h :

Original: =
Revised: =

Original: ( s 0 , a 0 , r 0 , s 1 , . .
Revised: ( s 0 , a 0 , r 0 , s 1 , . .

Original: , s l  1 , a l  1 , r l  1 ) be a trajectory and g ( h ) :=
Revised: , s l  1 , a l  1 , r l  1 ) be a trajectory and g ( h ) :=

Original: P l  1 t =

Revised: P l  1 t =

Original:  t r t be the discounted return of h .
Revised:  t r t be the discounted return of h .

Original: Any policy induces a distribution over trajectories, Pr( h | ⇡ ) .
Revised: Any policy induces a distribution over trajectories, Pr( h | ⇡ ) .

Original: We deﬁne thevalue of a policy, v ( ⇡ ) , as the expected discounted return when sampling a trajectory by followingpolicy ⇡ : v ( ⇡ ) :
Revised: We deﬁne thevalue of a policy, v ( ⇡ ) , as the expected discounted return when sampling a trajectory by followingpolicy ⇡ : v ( ⇡ ) :

Original: = E [ g ( H ) | H ⇠ ⇡ ] =
Revised: = E [ g ( H ) | H ⇠ ⇡ ] =

Original: P h
Revised: P h

Original: Pr( h | ⇡ ) g ( h ) where H is a random variable representinga trajectory and H ⇠ ⇡ denotes sampling H by running ⇡ in the given environment.
Revised: Pr( h | ⇡ ) g ( h ) where H is a random variable representinga trajectory and H ⇠ ⇡ denotes sampling H by running ⇡ in the given environment.

Original: 3.2 Policy Evaluation
Revised: 3.2 Policy Evaluation

Original: In the policy evaluation problem, we are given an evaluation policy , ⇡ e , for which we would like toestimate v ( ⇡ e ) .
Revised: In the policy evaluation problem, we are given an evaluation policy , ⇡ e , for which we would like toestimate v ( ⇡ e ) .

Original: Conceptually, algorithms for policy evaluation involve two steps: collecting data (orreceiving previously collected data) and computing an estimate from that data.
Revised: Conceptually, algorithms for policy evaluation involve two steps: collecting data (orreceiving previously collected data) and computing an estimate from that data.

Original: We assume that datais collected by running a policy which we call the behavior policy .
Revised: We assume that datais collected by running a policy which we call the behavior policy .

Original: If the behavior policy is the sameas the evaluation policy data collection is on-policy ; otherwise it is off-policy .
Revised: If the behavior policy is the sameas the evaluation policy data collection is on-policy ; otherwise it is off-policy .

Original: Whether on-policy oroff-policy, we assume the data collection process produces a set of trajectories, D :=
Revised: Whether on-policy oroff-policy, we assume the data collection process produces a set of trajectories, D :=

Original: { H i } ni =1 and 118 write D ⇠ ⇡ to denote collecting these trajectories by running policy ⇡ .
Revised: { H i } ni =1 and 117 write D ⇠ ⇡ to denote collecting these trajectories by running policy ⇡ .

Original: The ﬁnal value estimate isthen computed by a policy evaluation estimator ( PE ) that maps the set of trajectories to a scalar-valuedestimate of v ( ⇡ e ) .
Revised: The ﬁnal value estimate isthen computed by a policy evaluation method ( PE ) that maps the set of trajectories to a scalar-valuedestimate of v ( ⇡ e ) .

Original: Following earlier work in policy evaluation (e.g., [Thomas and Brunskill, 2016,
Revised: Following earlier work in policy evaluation (e.g., [Thomas and Brunskill, 2016,

Original: Jiang and Li, 2016]), we set our goal to be policy evaluation with low mean squared error ( MSE ):
Revised: Jiang and Li, 2016]), we set our goal to be policy evaluation with low mean squared error ( MSE ):

Original: [Equation]
Revised: [Equation]

Original: where ⇡ b is the behavior policy that is run to collect D and PE is a generic policy evaluation estimator.
Revised: where ⇡ b is the behavior policy that is ran to collect D .

Original: 3.3 Monte Carlo Policy Evaluation
Revised: 3.3 Monte Carlo Policy Evaluation

Original: Perhaps the most fundamental, model-free policy evaluation method is the Monte-Carlo ( MC )estimator.
Revised: Perhaps the most fundamental, model-free policy evaluation method is the Monte-Carlo ( MC )estimator.

Original: Given a data set, D , of n trajectories, the Monte Carlo estimate, MC( D ) , is the meanreturn over D :
Revised: Given a data set, D , of n trajectories, the Monte Carlo estimate is the mean return over D :

Original: [Equation]
Revised: [Equation]

Original: where Pr( h | D ) denotes the empirical probability of h , i.e. how often h appears in D .
Revised: where Pr( h | D ) denotes the empirical probability of h , i.e. how often h appears in D .

Original: If trajectories in D are collected i.i.d.
Revised: If trajectories in D are collected i.i.d.

Original: by running ⇡ e (i.e., on-policy sampling), the Monte Carloestimator is unbiased and consistent assuming g ( h ) is bounded [Sen and Singer, 1993].
Revised: by running ⇡ e (i.e., on-policy sampling), the Monte Carloestimator is unbiased and consistent assuming g ( h ) is bounded [Sen and Singer, 1993].

Original: However,this method can have high variance as on-policy sampling may require many trajectories for theempirical trajectory distribution
Revised: However,this method can have high variance as on-policy sampling may require many trajectories for theempirical trajectory distribution

Original: Pr( h | D ) to accurately approximate Pr( h | ⇡ e ) .
Revised: Pr( h | D ) to accurately approximate Pr( h | ⇡ e ) .

Original: Since on-policy 132 sampling collects each trajectory i.i.d.
Revised: Since on-policysampling collects each trajectory i.i.d.

Original: , it relies on the law of large numbers for an accurate weightingon each possible return.
Revised: , it relies on the law of large numbers for an accurate weightingon each possible return.

Original: We call error between Pr( h | D ) and Pr( h | ⇡ e ) sampling error .
Revised: We call error between Pr( h | D ) and Pr( h | ⇡ e ) sampling error .

Original:  Data-Conditioned Monte Carlo Estimates
Revised:  Data-Conditioned Monte Carlo Estimates

Original: In this section, we motivate how an estimator that uses on-policy data can beneﬁt from off-policysampling.
Revised: In this section, we motivate how an estimator that uses on-policy data can beneﬁt from off-policysampling.

Original: Speciﬁcally, we consider the Monte Carlo estimator and suppose that we have already collected a data set, D 1 , of trajectories.
Revised: Speciﬁcally, we consider the Monte Carlo estimator and suppose that we have already collected a data set, D 1 , of trajectories.

Original: We now wish to collect an additional set of trajectories, D 2 ,and compute the Monte Carlo estimate with the set D 

Revised: We now wish to collect an additional set of trajectories, D 2 ,and compute the Monte Carlo estimate with the set D

Original: [ D 2 .
Revised: [ D 2 .

Original: Note that D 1 is a ﬁxed set (the trajectories 139 already observed) while D 2 is a random variable (the trajectories yet to be observed).
Revised: Note that D 1 is a ﬁxed set (the trajectories 137 already observed) while D 2 is a random variable (the trajectories yet to be observed).

Original: How should
Revised: How should

Original: D 2 be collected for minimal MSE policy evaluation with the Monte Carlo estimator?
Revised: D 2 be collected for minimal MSE policy evaluation with the Monte Carlo estimator?

Original: Our analysisin this section suggests that i.i.d.
Revised: Our analysisin this section suggests that i.i.d.

Original: sampling of trajectories with ⇡ e may be a sub-optimal choice.
Revised: sampling of trajectories with ⇡ e may be a sub-optimal choice.

Original: In this setting, the Monte Carlo estimator using D 1 [ D 2 can be written as:
Revised: In this setting, the Monte Carlo estimator using D 1 [ D 2 can be written as:

Original: [Equation]
Revised: [Equation]

Original: where n D 1 and n D 2 are the number of trajectories in D 1 and D 2 , respectively and n = n D 1 + n D 2 .
Revised: where n D 1 and n D 2 are the number of trajectories in D 1 and D 2 , respectively and n = n D 1 + n D 2 .

Original: 144 We refer to (3) as the data-conditioned Monte Carlo estimator .
Revised: 142 We refer to (3) as the data-conditioned Monte Carlo estimator .

Original: Viewing the Monte Carlo estimator as a sum between a ﬁxed quantity and a random quantity changeshow we view the statistical properties of the estimator.
Revised: Viewing the Monte Carlo estimator as a sum between a ﬁxed quantity and a random quantity changeshow we view the statistical properties of the estimator.

Original: For instance, while the Monte Carlo estimatoris known to be unbiased under on-policy sampling, its data-conditioned estimate is biased as shownin the following proposition.
Revised: For instance, while the Monte Carlo estimatoris known to be unbiased under on-policy sampling, its data-conditioned estimate is biased as shownin the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: The data conditioned Monte Carlo estimator is biased under on-policy sampling of
Revised: The data conditioned Monte Carlo estimator is biased under on-policy sampling of

Original: D 2 unless MC( D 1 ) = v ( ⇡ e ) or D 1 = ; .
Revised: D 2 unless MC( D 1 ) = v ( ⇡ e ) or D 1 = ; .

Original: That is:
Revised: That is:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix A.
Revised: See Appendix A.

Original: Remark 1.
Revised: Remark 1.

Original: Proposition 1 holds even if D 1 was collected under on-policy sampling as well.
Revised: Proposition 1 holds even if D 1 was collected under on-policy sampling as well.

Original: When
Revised: When

Original: D 1 was collected under on-policy sampling then the Monte Carlo estimator is unbiased considering 155 all possible realizations of D 1 .
Revised: D 1 was collected under on-policy sampling then the Monte Carlo estimator is unbiased consideringall possible realizations of D 1 .

Original: However, once the trajectories in D 1 are ﬁxed, it no longer matters 156 what others values they could have taken.
Revised: However, once the trajectories in D 1 are ﬁxed, it no longer matters 154 what others values they could have taken.

Original: Can we reduce the bias of the data-conditioned Monte Carlo estimator by collecting D 2 with apolicy that is different than ⇡ e ?
Revised: Can we reduce the bias of the data-conditioned Monte Carlo estimator by collecting D 2 with apolicy that is different than ⇡ e ?

Original: We conclude this section with an example showing that we can.
Revised: We conclude this section with an example showing that we can.

Original: Consider a one-step MDP with one state, s , and two actions, a 0 and a 1 .
Revised: Consider a one-step MDP with one state, s , and two actions, a 0 and a 1 .

Original: The return following a 0 is2 and the return following a 1 is 4 .
Revised: The return following a 0 is2 and the return following a 1 is 4 .

Original: The evaluation policy is ⇡ e ( a 0 | s ) =
Revised: The evaluation policy is ⇡ e ( a 0 | s ) =

Original: ⇡ e ( a 1 | s ) = 0 . 5 .
Revised: ⇡ e ( a 1 | s ) = 0 . 5 .

Original: Supposethat, after sampling 3 trajectories, D 1 contains two of { s, a 0 , 2 } and one occurrence of { s, a 1 , 4 } .
Revised: Supposethat, after sampling 3 trajectories, D 1 contains two of { s, a 0 , 2 } and one occurrence of { s, a 1 , 4 } .

Original: 162 Note that action a 0 is over-sampled relative to its true probability in s and a 1 is under-sampled.
Revised: 160 Note that action a 0 is over-sampled relative to its true probability in s and a 1 is under-sampled.

Original: If we collect an additional trajectory with ⇡ e the expected value of the Monte Carlo estimate is:14 (2 + 2 + 4 + 2 ⇡ e ( a 0 ) + 4 ⇡ e ( a 1 ))
Revised: If we collect an additional trajectory with ⇡ e the expected value of the Monte Carlo estimate is:14 (2 + 2 + 4 + 2 ⇡ e ( a 0 ) + 4 ⇡ e ( a 1 ))

Original: =
Revised: =

Original: 114 = 2 .
Revised: 114 = 2 .

Original: The true value, v ( ⇡ e ) = 3 and thus, conditionedon prior data , the Monte Carlo estimate is biased in expectation as shown in Proposition 1 .
Revised: The true value, v ( ⇡ e ) = 3 and thus, conditionedon prior data , the Monte Carlo estimate is biased in expectation as shown in Proposition 1 .

Original: If insteadwe choose the behavior policy such that ⇡ b ( a 1 )
Revised: If insteadwe choose the behavior policy such that ⇡ b ( a 1 )

Original: = 1 then neither action is over- or under-sampled andthe expected value of the Monte Carlo estimate is the exact true value: 14 (2 + 2 + 4 + 4) = 124 = 3 .
Revised: = 1 then neither action is over- or under-sampled andthe expected value of the Monte Carlo estimate is the exact true value: 14 (2 + 2 + 4 + 4) = 124 = 3 .

Original: This example highlights that adapting the behavior policy to consider previously collected data canlower the expected ﬁnite-sample error of policy evaluation.
Revised: This example highlights that adapting the behavior policy to consider previously collected data canlower the expected ﬁnite-sample error of policy evaluation.

Original: In the next section, we introduce anadaptive data collection method that adjusts the behavior policy based on what data has already beenobserved so as to lower the MSE of a Monte Carlo estimate using all observed data.
Revised: In the next section, we introduce anadaptive data collection method that adjusts the behavior policy based on what data has already beenobserved so as to lower the MSE of a Monte Carlo estimate using all observed data.

Original:  Robust On-Policy Data Collection
Revised:  Robust On-Policy Data Collection

Original: In this section, we introduce a method that adapts the data-collecting behavior policy online tominimize sampling error in the data used by the Monte Carlo estimator.
Revised: In this section, we introduce a method that adapts the data-collecting behavior policy online tominimize sampling error in the data used by the Monte Carlo estimator.

Original: Speciﬁcally, let D t denoteall trajectories observed up to time-step t of the current trajectory (including the partial currenttrajectory).
Revised: Speciﬁcally, let D t denoteall trajectories observed up to time-step t of the current trajectory (including the partial currenttrajectory).

Original: At time-step t , our method sets the behavior policy so as to reduce the current samplingerror, i.e., divergence between Pr( h | ⇡ e ) and Pr( h |D t ) .
Revised: At time-step t , our method sets the behavior policy so as to reduce the current samplingerror, i.e., divergence between Pr( h | ⇡ e ) and Pr( h |D t ) .

Original: Our method can be run starting with D t = ; 178 or already containing trajectories in a setting like that described in the preceding section.
Revised: Our method can be ran starting with D t = ; 176 or already containing trajectories in a setting like that described in the preceding section.

Original: To reduce sampling error when collecting future trajectories, we want to adjust the behavior policy toincrease the probability of under-sampled trajectories, i.e., h for which Pr( h |D t ) <
Revised: To reduce sampling error when collecting future trajectories, we want to adjust the behavior policy toincrease the probability of under-sampled trajectories, i.e., h for which Pr( h |D t ) <

Original: Pr( h | ⇡ e ) .
Revised: Pr( h | ⇡ e ) .

Original: Unfor- 181 tunately, the trajectory distributions are unknown because the transition function, P , is also unknown.
Revised: Unfor- 179 tunately, the trajectory distributions are unknown because the transition function, P , is also unknown.

Original: Instead, we will increase the probability of under-sampled actions.
Revised: Instead, we will increase the probability of under-sampled actions.

Original: Let ⇡ D : S ⇥ A !
Revised: Let ⇡ D : S ⇥ A !

Original: [0 , 1] denotethe empirical policy which gives the proportion of times that each action was taken in each statein D t .
Revised: [0 , 1] denotethe empirical policy which gives the proportion of times that each action was taken in each statein D t .

Original: If ⇡ D ( a | s ) < ⇡ e ( a | s ) , then a has appeared less often in the data than it would in expectation 185 under ⇡ e .
Revised: If ⇡ D ( a | s ) < ⇡ e ( a | s ) , then a has appeared less often in the data than it would in expectation 183 under ⇡ e .

Original: Thus, we should increase the probability of a in s for future data collection.
Revised: Thus, we should increase the probability of a in s for future data collection.

Original: When the state and action spaces are ﬁnite, ⇡ D can be computed exactly as the maximum likelihoodpolicy under D t :
Revised: When the state and action spaces are ﬁnite, ⇡ D can be computed exactly as the maximum likelihoodpolicy under D t :

Original: [Equation]
Revised: [Equation]

Original: where the argmax is taken with respect to all policies.
Revised: where the argmax is taken with respect to all policies.

Original: In larger MDP s, we require function approximation which may make ⇡ D hard to compute and update online as new data is collected.
Revised: In larger MDP s, we require function approximation which may make ⇡ D hard to compute and update online as new data is collected.

Original: Fortunately,with an additional assumption we can determine the direction to adjust action probabilities withoutexplicitly computing ⇡ D .
Revised: Fortunately,with an additional assumption we can determine the direction to adjust action probabilities withoutexplicitly computing ⇡ D .

Original: This assumption is that ⇡ e belongs to a class of differentiable, parameterizedpolicies and is parameterized by vector ✓ 2 R d .
Revised: This assumption is that ⇡ e belongs to a class of differentiable, parameterizedpolicies and is parameterized by vector ✓ ✓ R d .

Original: This assumption is mild for many RL applicationsas it permits tabular, linear, and neural network policy representations.
Revised: This assumption is mild for many RL applicationsas it permits tabular, linear, and neural network policy representations.

Original: We use ✓ e to represent theparameter values for ⇡ e .
Revised: We use ✓ e to represent theparameter values for ⇡ e .

Original: We show in the next subsection that the gradient of the log-likelihood at✓ e , r ✓ L ( ⇡ ✓ ) | ✓ = ✓ e , can be used to make sampling-error-reducing changes to the behavior policy.
Revised: We show in the next subsection that the gradient of the log-likelihood at✓ e , r ✓ L ( ⇡ ✓ ) | ✓ = ✓ e , can be used to make sampling-error-reducing changes to the behavior policy.

Original: 5.1 Robust On-policy Sampling
Revised: 5.1 Robust On-policy Sampling

Original: Our primary algorithmic contribution – R obust O n-Policy S ampling ( ROS ) – reduces sampling errorby adapting the behavior policy with a single step of gradient descent on the log-likelihood at eachtime-step.
Revised: Our primary algorithmic contribution – R obust O n-Policy S ampling ( ROS ) – reduces sampling errorby adapting the behavior policy with a single step of gradient descent on the log-likelihood at eachtime-step.

Original: From here on, we use r ✓ L to denote the gradient of the log-likelihood evaluated at ✓ e . 200 Observe that r ✓ L provides a direction to adjust ✓ e to increase the probability of actions that were 201 over-sampled relative to their probability under ⇡ e .
Revised: From here on, we use r ✓ L to denote the gradient of the log-likelihood evaluated at ✓ e . 198 Observe that r ✓ L provides a direction to adjust ✓ e to increase the probability of actions that were 199 over-sampled relative to their probability under ⇡ e .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: L provides a direction to adjust 202 ✓ e to decrease the probability of over-sampled actions for which ⇡ D ( a | s ) > ⇡ e ( a | s ) .
Revised: provides a direction to adjust 200 ✓ e to decrease the probability of over-sampled actions for which ⇡ D ( a | s ) > ⇡ e ( a | s ) .

Original: With thisinsight, ROS is able to adapt ✓ e so that ⇡ D tracks ⇡ e without ever computing ⇡ D .
Revised: With thisinsight, ROS is able to adapt ✓ e so that ⇡ D tracks ⇡ e without ever computing ⇡ D .

Original: At each time-step,
Revised: At each time-step,

Original: ROS computes r ✓ L with all state-action pairs previously observed and then changes the evaluation 205 policy parameters with a single step of gradient descent so that under-sampled actions have greaterprobability than they would have under ⇡ e .
Revised: ROS computes r ✓ L with all state-action pairs previously observed and then changes the evaluation 203 policy parameters with a single step of gradient descent so that under-sampled actions have greaterprobability than they would have under ⇡ e .

Original: Pseudocode for ROS is given in Algorithm 1.
Revised: Pseudocode for ROS is given in Algorithm 1.

Original: ROS ﬁrst computes r ✓ L with previously collected 208 trajectories if any are provided (Line 4).
Revised: ROS ﬁrst computes r ✓ L with previously collected 206 trajectories if any are provided (Line 4).

Original: ROS then collects n additional trajectories by interactingwith the given MDP (Lines 6-14).
Revised: ROS then collects n additional trajectories by interactingwith the given MDP (Lines 6-14).

Original: For each action selection, ROS sets the behavior policy parametersas ✓ e  ↵ r ✓ L ( ⇡ ✓ ) | ✓ = ✓ e (Lines 9 and 10).
Revised: For each action selection, ROS sets the behavior policy parametersas ✓ e  ↵ r ✓ L ( ⇡ ✓ ) | ✓ = ✓ e (Lines 9 and 10).

Original: It then computes r ✓ log ⇡ ✓ ( A | s ) | ✓ = ✓ e and updatesr ✓ L ( ⇡ ✓ ) | ✓ = ✓ e (Lines 11 and 12).
Revised: It then computes r ✓ log ⇡ ✓ ( A | s ) | ✓ = ✓ e and updatesr ✓ L ( ⇡ ✓ ) | ✓ = ✓ e (Lines 11 and 12).

Original: Finally, the chosen action is executed in the environment, a reward 212 received, and the agent moves to the next state (Line 13).
Revised: Finally, the chosen action is executed in the environment, a reward 210 received, and the agent moves to the next state (Line 13).

Original: Importantly, note that updating r ✓ L 213 requires per-timestep computation that is linear in the number of policy parameters and remainsconstant as the size of D grows.
Revised: Importantly, note that updating r ✓ L 211 requires per-timestep computation that is linear in the number of policy parameters and remainsconstant as the size of D grows.

Original: 5.2 ROS Convergence
Revised: 5.2 ROS Convergence

Original: This section develops our theoretical understanding of ROS .
Revised: This section develops our theoretical understanding of ROS .

Original: Due to space constraints, we defer allproofs to Appendix B. First, we show that  ROS converges to the expected state visitation frequenciesunder ⇡ e .
Revised: First, we show that if the MDP state-spacehas a DAG structure then ROS converges to the expected state visitation frequencies under ⇡ e .

Original: Second, we show that, for a ﬁxed state,  ⇡ D ( ·| s ) converges to ⇡ e ( ·| s ) faster under ROS219 compared to on-policy  sampling.
Revised: Second,we show, for a ﬁxed state, that ⇡ D ( ·| s ) converges to ⇡ e ( ·| s ) faster under ROS compared to on-policy 217 sampling.

Original: Finally, we introduce an upper bound on the squared error betweenthe Monte Carlo estimate and v ( ⇡ e ) in terms of sampling error which shows how ROS ’s fasterconvergence affects the MSE of policy evaluation.

Original: These results use the following assumption:
Revised: To prove these results, we make the following assumptions:

Original: Assumption 1.
Revised: Assumption 1.

Revised: The discrete state-space of the MDP has a directed acyclic graph ( DAG ) structure.

Revised: Speciﬁcally, states in S can be partitioned into l disjoint sets S t indexed by episode step.

Revised: The 220 transition function is such that P ( s| s, a ) > 0 implies that s 2 S t and s 0 2 S t +1 .

Revised: Note that the DAG assumption is mild as any ﬁnite-horizon MDP can be made a DAG by including thecurrent time-step as part of the state.

Revised: Assumption 2.

Original: ROS uses a step-size of ↵ ! 1 and the behavior policy is parameterized as a 223 softmax function, i.e., ⇡ ✓ ( a | s ) /
Revised: ROS uses a step-size of ↵ ! 1 and the behavior policy is parameterized as a 224 softmax function, i.e., ⇡ ✓ ( a | s ) /

Original: e ✓ s,a , where for each state, s , and action, a , we have a parameter✓ s,a .
Revised: e ✓ s,a , where for each state, s , and action, a , we have a parameter✓ s,a .

Original: As we formally show in Appendix B, this assumption implies that ROS always takes the mostunder-sampled action in each state.
Revised: As we formally show in Appendix B, this assumption implies that ROS always takes the mostunder-sampled action in each state.

Original: We also introduce the notation of d t ⇡ ( s ) as the probability of visiting state s at episode time t whilefollowing policy ⇡ and d t n ( s ) as the empirical frequency of visitations to state s at episode time tafter observing n trajectories.

Original: Theorem 1.
Revised: Theorem 1.

Revised: Let d t ⇡ ( s ) be the probability of visiting state s at episode time t while following policy⇡ .

Revised: Let d tn ( s ) be the empirical frequency of visitations to state s at episode time t after observing n trajectories.

Original: Under Assumption 1 and ROS action selection, d tn ( s ) converges to d t ⇡ ( s ) with probability1 for all s 2 S and 0 < t < l :
Revised: Then, under Assumptions 1, 2, and ROS action selection, d tn ( s ) converges to d t ⇡ ( s ) withprobability 1 for all s 2 S and 0 < t < l :

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: Let s be a particular state that is visited m times during data collection and assumethat |A|  2 .
Revised: Let s be a particular state that is visited m times during data collection and assumethat |A|  2 .

Original: Under Assumption 1, D KL ( ⇡ D ( ·| s ) || ⇡ ( ·| s ))
Revised: Under Assumption 2, D KL ( ⇡ D ( ·| s ) || ⇡ ( ·| s ))

Original: =
Revised: =

Original: O p ( 1 m 2 ) under ROS sampling while
Revised: O p ( 1 m 2 ) under ROS sampling while

Original: D KL ( ⇡ D ( ·| s ) || ⇡ ( ·| s ))
Revised: D KL ( ⇡ D ( ·| s ) || ⇡ ( ·| s ))

Original: =
Revised: =

Original: O p ( 1 m ) under on-policy sampling, where O p denotes stochastic boundedness.
Revised: O p ( 1 m ) under on-policy sampling, where O p denotes stochastic boundedness.

Original: Theorem 3. Assume 8 s 2 S , a 2 A that R ( s, a )  R max . The squared error in the Monte Carlo 237 estimate using
Revised: Due to space constraints, we defer all proofs to Appendix B.

Original: D can be upper-bounded by:
Revised: In addition to the DAG structure used by

Original: [Equation] 2 D KL ( d tn || d t ⇡ e ) + 2 E S ⇠ d tn [ D KL ( ⇡ D ( ·| S ) ||
Revised: Theorem 1, both Theorems 1 and 2 require a ﬁnite number of states and actions.

Original: ⇡ e ( ·| S )] . Remark 2. The second term in the bound in Theorem 3 is the KL-divergence between ⇡ D and⇡ e which Theorem 2 tells us will decrease faster under ROS action selection. The ﬁrst term is the 
Revised: In the next section,we complement our theoretical results with an empirical study that considers non-

Original: KL-divergence between the empirical and true statedistributions which depends both on samplingerror in action selection as well as sampling error in the transition and initial state distributions. While the former decreases faster under ROS , the latter will decrease the same for both ROS andon-policy sampling. Hence, the theoretical faster rate of ROS for reducing sampling error in actionselection may be muted by high environment stochasticity. The experimental results given in Figure12(a) complement this theoretical observation.
Revised: DAG MDP s andcontinuous state and action spaces.

Original:  Empirical Study
Revised:  Empirical Study

Original: We next conduct an empirical study of ROS in policy evaluation problems.
Revised: We next conduct an empirical study of ROS in policy evaluation problems.

Original: Our primary goal is toanswer the following questions:
Revised: Our primary goal is toanswer the following questions:

Original: Does ROS reduce sampling error compared to on-policy sampling?
Revised: Does ROS reduce sampling error compared to on-policy sampling?

Original: Does ROS lower policy evaluation MSE when starting with and without off-policy data?
Revised: Does ROS lower policy evaluation MSE when starting with and without off-policy data?

Original: We conduct policy evaluation experiments in four domains covering discrete and continuous stateand action spaces: a multi-armed bandit problem [Sutton and Barto, 1998], Gridworld [Thomas and
Revised: We conduct policy evaluation experiments in four domains covering discrete and continuous stateand action spaces: a multi-armed bandit problem [Sutton and Barto, 1998], Gridworld [Thomas and

Original: Brunskill, 2016], CartPole, and Continuous CartPole [Brockman et al., 2016].
Revised: Brunskill, 2016], CartPole, and Continuous CartPole [Brockman et al., 2016].

Original: Since these domainsare widely used, we defer their descriptions to Appendix C.
Revised: Since these domainsare widely used, we defer their descriptions to Appendix C.

Original: Our primary baseline for comparison ison-policy sampling ( OS ) of i.i.d.
Revised: Our primary baseline for comparison ison-policy sampling ( OS ) of i.i.d.

Original: trajectories with the Monte Carlo estimator used to compute the ﬁnalpolicy value estimate (denoted OS - MC ).
Revised: trajectories with the Monte Carlo estimator used to compute the ﬁnalpolicy value estimate (denoted OS - MC ).

Original: We also compare to BPG which ﬁnds a minimum variancebehavior policy for the ordinary importance sampling ( OIS ) policy value estimator [Hanna et al.
Revised: We also compare to BPG which ﬁnds a minimum variancebehavior policy for the ordinary importance sampling ( OIS ) policy value estimator [Hanna et al.

Original: ,2017] (denoted BPG - OIS ).
Revised: ,2017] (denoted BPG - OIS ).

Original: We provide full experimental details concerning how ⇡ e and v ( ⇡ e ) weredetermined in Appendix C.2. [Figure] [Figure]
Revised: We provide full experimental details concerning how ⇡ e and v ( ⇡ e ) weredetermined in Appendix C.1.

Original: 6.1 Policy Evaluation without Initial Data
Revised: 6.1 Policy Evaluation without Initial Data

Original: We ﬁrst run experiments in a setting without initial data , in which all data is collected from scratch.
Revised: We ﬁrst run experiments in a setting without initial data , in which all data is collected from scratch.

Original: Letting T denote the average length of a trajectory, in each domain, we collect a total of 2 13 Tenvironment steps with each method and compute metrics every 2 1 , 2 2 , ..., 2 13 trajectories.
Revised: Letting T denote the average length of a trajectory, in each domain, we collect a total of 2 13 T [Figure] [Figure] environment steps with each method and compute metrics every 2 1 , 2 2 , ..., 2 13 trajectories.

Original: Thehyper-parameter settings for these experiments are presented in Appendix E.
Revised: Thehyper-parameter settings for these experiments are presented in Appendix E.

Original: We ﬁrst verify that ROS reduces sampling error compared to on-policy sampling.
Revised: We ﬁrst verify that ROS reduces sampling error compared to on-policy sampling.

Original: We measuresampling error with the KL -divergence ( KL ) between ⇡ e and a parametric maximum likelihoodestimate of ⇡ D from the observe data.
Revised: We measuresampling error with the KL -divergence ( KL ) between ⇡ e and a parametric maximum likelihoodestimate of ⇡ D from the observe data.

Original: In Appendix D, we give a complete deﬁnition of the measureas well as an alternative measure that leads to qualitatively similar results.
Revised: In Appendix D, we give a complete deﬁnition of the measureas well as an alternative measure that leads to qualitatively similar results.

Original: Due to space constraints,we only show this result for the GridWorld domain (Figure 1(a)); results for other domains arequalitatively the same and can be found in Appendix D.1.
Revised: Due to space constraints,we only show this result for the GridWorld domain (Figure 1(a)); results for other domains arequalitatively the same and can be found in Appendix D.1.

Original: Figure 1(a) shows that with ROS samplingerror decreases faster than OS .
Revised: Figure 1(a) shows that with ROS samplingerror decreases faster than OS .

Original: Unsurprisingly, BPG increases sampling error as it is an off-policymethod which adapts the behavior policy away from ⇡ e .
Revised: Unsurprisingly, BPG increases sampling error as it is an off-policymethod which adapts the behavior policy away from ⇡ e .

Original: These results answer our ﬁrst empiricalquestion and conﬁrm our theoretical claim that non-i.i.d.
Revised: These results answer our ﬁrst empiricalquestion and conﬁrm our theoretical claim that non-i.i.d.

Original: off-policy sampling can cause the empiricaldistribution of data to converge to the expected on-policy distribution faster.
Revised: off-policy sampling can cause the empiricaldistribution of data to converge to the expected on-policy distribution faster.

Original: Ultimately, this paper focuses on reducing sampling error for lower MSE policy evaluation.
Revised: Ultimately, this paper focuses on reducing sampling error for lower MSE policy evaluation.

Original: Figure 2shows that ROS lowers MSE compared to both OS and BPG across all domains.
Revised: Figure 2shows that ROS lowers MSE compared to both OS and BPG across all domains.

Original: 1 These results addressour second empirical question and support the claim that reducing sampling error decreases the MSEof the Monte Carlo estimator for policy evaluation.
Revised: 1 These results addressour second empirical question and support the claim that reducing sampling error decreases the MSEof the Monte Carlo estimator for policy evaluation.

Original: 6.2 Policy Evaluation with Initial Data
Revised: 6.2 Policy Evaluation with Initial Data

Original: Our next set of experiments considers a setting with initial data , in which a set of 100 trajectories arealready available and we wish to use these trajectories in our policy value estimate.
Revised: Our next set of experiments considers a setting with initial data , in which a set of 100 trajectories arealready available and we wish to use these trajectories in our policy value estimate.

Original: These trajectories [Figure] are collected via i.i.d.
Revised: These trajectoriesare collected via i.i.d.

Original: off-policy sampling with a behavior policy that is slightly different than ⇡ e .
Revised: off-policy sampling with a behavior policy that is slightly different than ⇡ e .

Original: This setting is intended to represent a setting where ⇡ e has just been updated from an older policyand we would like to still use the off-policy data already collected from the older policy combinedwith the data to be collected.
Revised: This setting is intended to represent a setting where ⇡ e has just been updated from an older policyand we would like to still use the off-policy data already collected from the older policy combined [Figure] with the data to be collected.

Original: In addition to the off-policy data ( OPD ), we collect an additional 2 13 Tsteps of environment interaction with each method.
Revised: In addition to the off-policy data ( OPD ), we collect an additional 2 13 Tsteps of environment interaction with each method.

Original: For ROS , we use the OPD to initialize r ✓ L .
Revised: For ROS , we use the OPD to initialize r ✓ L .

Original: We expect to see that ROS will collect data to combine 289 with the OPD such that the aggregate data set looks as if it had been collected with ⇡ e to begin with.
Revised: We expect to see that ROS will collect data to combine 282 with the OPD such that the aggregate data set looks as if it had been collected with ⇡ e to begin with.

Original: We compare ROS to the following baseline methods.
Revised: We compare ROS to the following baseline methods.

Original: ( OPD + OS )- MC collects additional data with OSand uses the Monte Carlo estimator with the total data-set.
Revised: ( OPD + OS )- MC collects additional data with OSand uses the Monte Carlo estimator with the total data-set.

Original: ( OPD + OS )-( WIS + MC ) uses weightedimportance sampling ( WIS ) to compute an estimate from the OPD combines the WIS estimate with a
Revised: ( OPD + OS )-( WIS + MC ) uses weightedimportance sampling ( WIS ) to compute an estimate from the OPD combines the WIS estimate with a

Original: Monte Carlo estimate using on-policy data.
Revised: Monte Carlo estimate using on-policy data.

Original: ( OPD + BPG )- OIS collects additional data with BPG anduses ordinary importance sampling as the estimator with all data.
Revised: ( OPD + BPG )- OIS collects additional data with BPG anduses ordinary importance sampling as the estimator with all data.

Original: Finally, ( OS - MC ) replaces the 100initial trajectories with trajectories from OS , then collects the remaining data with OS and uses the
Revised: Finally, ( OS - MC ) replaces the 100initial trajectories with trajectories from OS , then collects the remaining data with OS and uses the

Original: Monte Carlo estimator.
Revised: Monte Carlo estimator.

Original: Figure 1(b) shows that sampling error decreases fastest for ROS as the additional data is collected.
Revised: Figure 1(b) shows that sampling error decreases fastest for ROS as the additional data is collected.

Original: For policy evaluation, we show MSE for varying amounts of data in Figure 3 and provide numericalvalues for the ﬁnal MSE in Appendix F. These ﬁgures show that ROS can collect additional data thatreduces sampling error in the aggregate data set and produce lower MSE estimates compared to otherdata collection methods.
Revised: For policy evaluation, we show MSE for varying amounts of data in Figure 3 and provide numericalvalues for the ﬁnal MSE in Appendix F. These ﬁgures show that ROS can collect additional data thatreduces sampling error in the aggregate data set and produce lower MSE estimates compared to otherdata collection methods.

Original: Intuitively, OS requires many more samples to dilute the bias brought onby using OPD in the Monte Carlo estimator, while ROS is able to correct the empirical off-policydistribution to the expected on-policy distribution and use the Monte Carlo estimator without anyoff-policy corrections.
Revised: Intuitively, OS requires many more samples to dilute the bias brought onby using OPD in the Monte Carlo estimator, while ROS is able to correct the empirical off-policydistribution to the expected on-policy distribution and use the Monte Carlo estimator without anyoff-policy corrections.

Original: 6.3 Sensitivity Study
Revised: 6.3 Sensitivity Study

Original: Finally, we evaluate the sensitivity of ROS to hyper-parameter, environment, and policy settings.
Revised: Finally, we evaluate the sensitivity of ROS to hyper-parameter, environment, and policy settings.

Original: ROSrequires setting a step size, ↵ , which controls how much ROS updates the behavior policy away from⇡ e .
Revised: ROSrequires setting a step size, ↵ , which controls how much ROS updates the behavior policy away from⇡ e .

Original: We show MSE curves for ROS with different step size ↵ on GridWorld and CartPole in Figures 4( ↵ = 0 corresponds to OS ).
Revised: We show MSE curves for ROS with different step size ↵ on GridWorld and CartPole in Figures 4( ↵ = 0 corresponds to OS ).

Original: Figure 4(a) shows that, in GridWorld, ROS with any tested step-sizeproduces lower MSE policy evaluation than OS for any data set size.
Revised: Figure 4(a) shows that, in GridWorld, ROS with any tested step-sizeproduces lower MSE policy evaluation than OS for any data set size.

Original: As it collects more data, ROS withlarger ↵ enables lower MSE because the norm of r ✓ L decreases as sampling error decreases, and thus 312 a larger ↵ is required to make signiﬁcant updates.
Revised: As it collects more data, ROS withlarger ↵ enables lower MSE because the norm of r ✓ L decreases as sampling error decreases, and thus 305 a larger ↵ is required to make signiﬁcant updates.

Original: A larger ↵ value is also in line with our theoreticalresults which prescribe ↵ ! 1 .
Revised: A larger ↵ value is also in line with our theoreticalresults which prescribe ↵ ! 1 .

Original: However, in CartPole, (Figure 4(b)), ROS with the largest tested 314 ↵ (1000) diverges and the second largest ( ↵ = 100 ) requires many steps before it improves upon OS .
Revised: However, in CartPole, (Figure 4(b)), ROS with the largest tested 307 ↵ (1000) diverges and the second largest ( ↵ = 100 ) requires many steps before it improves upon OS .

Original: Thus, in domains with continuous state-spaces, more conservative ↵ values may be preferred.
Revised: Thus, in domains with continuous state-spaces, more conservative ↵ values may be preferred.

Original: Our ﬁnal set of experiments considers how the stochasticity of a domain and entropy of ⇡ e affectthe relative improvement that ROS offers.
Revised: Our ﬁnal set of experiments considers how the stochasticity of a domain and entropy of ⇡ e affectthe relative improvement that ROS offers.

Original: In this sub-section, we study these settings in the Banditdomain for its simplicty; similar experimental results in GridWorld can be found in Appendix H. Wechoose ↵ = 1000 for the following experiments.
Revised: In this sub-section, we study these settings in the Banditdomain for its simplicty; similar experimental results in GridWorld can be found in Appendix H. Wechoose ↵ = 1000 for the following experiments.

Original: To study domain stochasticity, we ﬁrst create variants of the Bandit environment by multiplying eitherthe mean or scale of the reward distribution of each action by a varying factor.
Revised: To study domain stochasticity, we ﬁrst create variants of the Bandit environment by multiplying eitherthe mean or scale of the reward distribution of each action by a varying factor.

Original: In each experimentaltrial, we use ROS to collect 1000 T steps for the Monte Carlo estimator and compute the relative
Revised: In each experimentaltrial, we use ROS to collect 1000 T steps for the Monte Carlo estimator and compute the relative

Original: MSE compared to the Monte Carlo estimator using OS with the same number of steps.
Revised: MSE compared to the Monte Carlo estimator using OS with the same number of steps.

Original: Figure 4(c)shows that as the factor on the mean increases, ROS provides a greater reduction in MSE as even smallamounts of sampling error translate into large MSE when the reward means are large.
Revised: Figure 4(c)shows that as the factor on the mean increases, ROS provides a greater reduction in MSE as even smallamounts of sampling error translate into large MSE when the reward means are large.

Original: On the other
Revised: On the otherhand, as the scale factor increases, the MSE is dominated by reward noise and the relative beneﬁt ofreducing sampling error disappears.

Original: [Figure]
Revised: [Figure]

Original: hand, as the scale factor increases, the MSE is dominated by reward noise and the relative beneﬁt ofreducing sampling error disappears.

Original: We also evaluate the relative improvement of ROS as a function of the entropy of ⇡ e .
Revised: We also evaluate the relative improvement of ROS as a function of the entropy of ⇡ e .

Original: For ⇡ e , we use✏ -greedy policies which select the optimal action in a state with probability 1  ✏ and otherwise selectan action uniformly at random.
Revised: For ⇡ e , we use✏ -greedy policies which select the optimal action in a state with probability 1  ✏ and otherwise selectan action uniformly at random.

Original: Relative improvement in MSE is shown in Figure 4(d).
Revised: Relative improvement in MSE is shown in Figure 4(d).

Original: For all ✏ , ROSimproves upon the MSE of OS .
Revised: For all ✏ , ROSimproves upon the MSE of OS .

Original: The improvement is generally larger for more stochastic ⇡ e whensampling error in action selection will be highest.
Revised: The improvement is generally larger for more stochastic ⇡ e whensampling error in action selection will be highest.

Original:  Discussion and Future Work
Revised:  Discussion and Future Work

Original: This work has shown that off-policy non-i.i.d.
Revised: This work has shown that off-policy non-i.i.d.

Original: sampling can produce data sets that more closelyapproximate the on-policy data distribution than on-policy i.i.d.
Revised: sampling can produce data sets that more closelyapproximate the on-policy data distribution than on-policy i.i.d.

Original: sampling.
Revised: sampling.

Original: We considered the problemof policy evaluation and showed that more closely approximating the on-policy data distribution leadsto more data efﬁcient policy evaluation across several domains.
Revised: We considered the problemof policy evaluation and showed that more closely approximating the on-policy data distribution leadsto more data efﬁcient policy evaluation across several domains.

Original: As far as we know, ROS is the ﬁrstdata collection method for policy evaluation that uses off-policy sampling to produce more closelyon-policy data than the data produced by on-policy sampling.
Revised: As far as we know, ROS is the ﬁrstdata collection method for policy evaluation that uses off-policy sampling to produce more closelyon-policy data than the data produced by on-policy sampling.

Original: While ROS is a ﬁrst step towards off-policy algorithms that produce data matching a target distribution,we highlight a few limitations of the algorithm and our study.
Revised: While ROS is a ﬁrst step towards off-policy algorithms that produce data matching a target distribution,we highlight a few limitations of the algorithm and our study.

Original: In our view, the main limitations ofthe ROS algorithm are the need to set a step-size parameter (in contrast to parameter-free on-policysampling) and the need to update r ✓ L at each action step.
Revised: In our view, the main limitations ofthe ROS algorithm are the need to set a step-size parameter (in contrast to parameter-free on-policysampling) and the need to update r ✓ L at each action step.

Original: For the former, future work should 344 investigate robust methods for setting the step-size, particularly in settings where ⇡ ✓ generalizesacross the state-space.
Revised: For the former, future work should 337 investigate robust methods for setting the step-size, particularly in settings where ⇡ ✓ generalizesacross the state-space.

Original: For the latter limitation, a future study could consider only updating r ✓ L at 346 the end of each episode instead of after each action choice (assuming more computation can be donebetween episodes).
Revised: For the latter limitation, a future study could consider only updating r ✓ L at 339 the end of each episode instead of after each action choice (assuming more computation can be donebetween episodes).

Original: In terms of our study, for this paper we chose to study many different facets of
Revised: In terms of our study, for this paper we chose to study many different facets of

Original: ROS on a suite of simpler domains (see the appendices for additional ablations and extensions); afuture study should assess the scalability of ROS with more complex function approximators.
Revised: ROS on a suite of simpler domains (see the appendices for additional ablations and extensions); afuture study should assess the scalability of ROS with more complex function approximators.

Original: Finally,our theoretical results were conducted in the tabular setting; an important open question is at whatrate ROS converges when ⇡ ✓ uses a function approximator that must generalize across states.
Revised: Finally,our theoretical results were conducted in the tabular setting; an important open question is at whatrate ROS converges when ⇡ ✓ uses a function approximator that must generalize across states.

Original: Beyondthese minor technical limitations, our paper addresses fundamental research questions in RL and thuswe do not see obvious negative societal impacts that are unique to this work in comparison to otherwork in RL and policy evaluation.
Revised: Beyondthese minor technical limitations, our paper addresses fundamental research questions in RL and thuswe do not see obvious negative societal impacts that are unique to this work in comparison to otherwork in RL and policy evaluation.

Original: While we evaluated ROS for policy evaluation, the long-term importance of this work may be inexploring the distinction between on-policy sampling and on-policy data.
Revised: While we evaluated ROS for policy evaluation, the long-term importance of this work may be inexploring the distinction between on-policy sampling and on-policy data.

Original: On-policy RL algorithmsrequire on-policy data and our work suggests that adaptive off-policy sampling can produce on-policydata more efﬁciently than on-policy sampling.
Revised: On-policy RL algorithmsrequire on-policy data and our work suggests that adaptive off-policy sampling can produce on-policydata more efﬁciently than on-policy sampling.

Original: In the future, we wish to study these insights foron-policy policy improvement algorithms (e.g., policy gradient methods [Williams, 1992, Schulmanet al., 2017]) and to extend our convergence results to non-tabular settings.
Revised: In the future, we wish to study these insights foron-policy policy improvement algorithms (e.g., policy gradient methods [Williams, 1992, Schulmanet al., 2017]) and to extend our convergence results to non-tabular settings that require generalizationacross states.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have introduced a novel data collection method for policy evaluation in reinforcementlearning environments.
Revised: In this paper, we have introduced a novel data collection method for policy evaluation in reinforcementlearning environments.

Original: Our method – Robust On-Policy Sampling ( ROS ) – considers previouslycollected data when selecting actions to reduce sampling error in the entire collected data set.
Revised: Our method – Robust On-Policy Sampling ( ROS ) – considers previouslycollected data when selecting actions to reduce sampling error in the entire collected data set.

Original: We showboth in theory and in practice that data from ROS converges faster to the on-policy data distributioncompared to on-policy sampling.
Revised: We showboth in theory and in practice that data from ROS converges faster to the on-policy data distributioncompared to on-policy sampling.

Original: Empirically, we ﬁnd that faster convergence to the on-policy datadistributions lowers the MSE of policy evaluation. 
Revised: Empirically, we ﬁnd that faster convergence to the on-policy datadistributions lowers the MSE of policy evaluation. Checklist For all authors... (a) Do the

Revised: main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Revised: [Yes]

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] SeeSection 7.(d) Have you read the ethics review guidelines and ensured that your paper conforms to

Revised: them?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] See Section5.2 and Appendix B.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] See Appendices Aand B.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Revised: imental results (either in the supplemental material or as a U RL )?

Revised: [Yes] Included in

Revised: supplementary material.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Revised: [Yes] See Section 6 and Appendices C.1 and E.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes]

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [No] Experiments use RL domains andalgorithms that can be ran on a typical personal computer.

Revised: Minimal compute resources

Revised: required to reproduce any experiment in the paper.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes]

Revised: (b) Did you mention the license of the assets?

Revised: [No]

Revised: (c) Did you include any new assets either in the supplemental material or as a U RL ?

Revised: [No]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: information or offensive content?

Revised: [N/A]

Revised: [Table]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: [Table]

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: [Table]

Revised: applicable?

Revised: [N/A]

Revised: [Table]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: [Table]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: [Table]

Revised: spent on participant compensation?

Revised: [N/A]

Revised: [Table]