Original: I MPORTANCE R ESAMPLING FOR O FF - POLICY P OLICY E VALUATION
Revised: I MPORTANCE R ESAMPLING FOR O FF - POLICY P OLICY E VALUATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Importance sampling is a common approach to off-policy learning in reinforcement learning.
Revised: Importance sampling is a common approach to off-policy learning in reinforcement learning.

Original: While it is consistent and unbiased, it can result in high variance updates to the parameters for the value function
Revised: While it is consistent and unbiased, it can result in high variance updates to the parameters for the value function

Original: Weighted importance sampling (WIS) has been explored to reduce variance for off-policy policy evaluation, but only for linear value function approximation.
Revised: Weighted importance sampling (WIS) has been explored to reduce variance for off-policy policy evaluation, but only for linear value function approximation.

Original: In this work, we explore a resampling strategy to reduce variance, rather than a reweighting strategy.
Revised: In this work, we explore a resampling strategy to reduce variance, rather than a reweighting strategy.

Original: We propose
Revised: We propose

Original: Importance Resampling (IR) for off-policy learning, that resamples experience from the replay buffer and applies a standard on-policy update.
Revised: Importance Resampling (IR) for off-policy learning, that resamples experience from the replay buffer and applies a standard on-policy update.

Original: The approach avoids using importance sampling ratios directly in the update instead correcting the distribution over transitions before the update.
Revised: The approach avoids using importance sampling ratios directly in the update instead correcting the distribution over transitions before the update.

Original: We characterize the bias and consistency of the our estimator, particularly compared to WIS.
Revised: We characterize the bias and consistency of the our estimator, particularly compared to WIS.

Original: We then demonstrate in several toy domains that IR has improved sample efﬁciency and parameter sensitivity, as compared to several baseline WIS estimators and to IS.
Revised: We then demonstrate in several toy domains that IR has improved sample efﬁciency and parameter sensitivity, as compared to several baseline WIS estimators and to IS.

Original: We conclude with a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.
Revised: We conclude with a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: An important component of many learning systems is learning value functions for many policies.
Revised: An important component of many learning systems is learning value functions for many policies.

Original: Some examples of such systems are the Horde architecture composed of General Value Functions (GVFs) (Sutton et al., 2011;
Revised: Some examples of such systems are the Horde architecture composed of General Value Functions (GVFs) (Sutton et al., 2011;

Original: Modayil et al., 2014), systems that use options (Sutton et al., 1999;
Revised: Modayil et al., 2014), systems that use options (Sutton et al., 1999;

Original: Schaul et al., 2015a), predictive representation approaches (Sutton et al., 2005; Schaul and Ring, 2013; Silver et al., 2017) and systems with auxiliary tasks (Jaderberg et al., 2017).
Revised: Schaul et al., 2015a), predictive representation approaches (Sutton et al., 2005; Schaul and Ring, 2013; Silver et al., 2017) and systems with auxiliary tasks (Jaderberg et al., 2017).

Original: For each target policy, the value function returns the expected return from a state, and can provide useful information about (long-term) outcomes under different behaviors.
Revised: For each target policy, the value function returns the expected return from a state, and can provide useful information about (long-term) outcomes under different behaviors.

Original: Off-policy learning is critical for learning many value functions with different policies at scale, because it enables data to be generated from one behavior policy to update the values for each target policy in parallel.
Revised: Off-policy learning is critical for learning many value functions with different policies at scale, because it enables data to be generated from one behavior policy to update the values for each target policy in parallel.

Original: The typical strategy for off-policy learning is to use importance sampling (IS).
Revised: The typical strategy for off-policy learning is to use importance sampling (IS).

Original: For a given state s , with action a selected according to behaviour µ , the importance sampling ratio is the ratio between the probability of the action under the target policy π and the behaviour: π ( a | s ) µ ( a | s ) .
Revised: For a given state s , with action a selected according to behaviour µ , the importance sampling ratio is the ratio between the probability of the action under the target policy π and the behaviour: π ( a | s ) µ ( a | s ) .

Original: The update is multiplied by this ratio, adjusting the action probabilities so that the expectation of the update is as if the actions were sampled according to the target policy π .
Revised: The update is multiplied by this ratio, adjusting the action probabilities so that the expectation of the update is as if the actions were sampled according to the target policy π .

Original: Though the IS estimator is unbiased and consistent (Kahn and Marshall, 1953; Rubinstein and Kroese, 2016), it can suffer from high or even inﬁnite variance due to large magnitude IS ratios, in theory (Andradottir et al., 1995) and in practice (Precup et al., 2001; Mahmood et al., 2014; 2017).
Revised: Though the IS estimator is unbiased and consistent (Kahn and Marshall, 1953; Rubinstein and Kroese, 2016), it can suffer from high or even inﬁnite variance due to large magnitude IS ratios, in theory (Andradottir et al., 1995) and in practice (Precup et al., 2001; Mahmood et al., 2014; 2017).

Original: There have been some attempts to modify policy evaluation algorithms to mitigate this variance.
Revised: There have been some attempts to modify policy evaluation algorithms to mitigate this variance.

Original: Weighted IS (WIS) algorithms have been introduced (Precup et al., 2001; Mahmood et al., 2014;
Revised: Weighted IS (WIS) algorithms have been introduced (Precup et al., 2001; Mahmood et al., 2014;

Original: Mahmood and Sutton, 2015), which normalize each update by the sample average of the ratios.
Revised: Mahmood and Sutton, 2015), which normalize each update by the sample average of the ratios.

Original: These algorithms did improve learning over standard IS strategies, but are not straightforward to extend to nonlinear function approximation.
Revised: These algorithms did improve learning over standard IS strategies, but are not straightforward to extend to nonlinear function approximation.

Original: In the ofﬂine setting, a reweighting scheme, called importance sampling with unequal support (Thomas and Brunskill, 2017), was introduced to account for samples where the ratio is zero, in some cases signiﬁcantly reducing variance.
Revised: In the ofﬂine setting, a reweighting scheme, called importance sampling with unequal support (Thomas and Brunskill, 2017), was introduced to account for samples where the ratio is zero, in some cases signiﬁcantly reducing variance.

Original: Another strategy has been to use rescaling or truncation of IS ratios, such as in V-trace (Espeholt et al., 2018).
Revised: Another strategy has been to use rescaling or truncation of IS ratios, such as in V-trace (Espeholt et al., 2018).

Original: Several other methods have introduced bias similarly for algorithms with eligibility traces by truncating or scaling IS ratios to maintain stability of the eligibility trace vector, including Tree-Backup (Precup et al., 2000), Retrace (Munos et al., 2016) and ABQ (Mahmood et al., 2017).
Revised: Several other methods have introduced bias similarly for algorithms with eligibility traces by truncating or scaling IS ratios to maintain stability of the eligibility trace vector, including Tree-Backup (Precup et al., 2000), Retrace (Munos et al., 2016) and ABQ (Mahmood et al., 2017).

Original: Truncation of IS-ratios in V-trace can incur signiﬁcant bias, and this additional truncation parameter does need to be tuned.
Revised: Truncation of IS-ratios in V-trace can incur signiﬁcant bias, and this additional truncation parameter does need to be tuned.

Original: An alternative to reweighting updates is to instead correct the distribution before updating the estimator using weighted bootstrap sampling: resampling a new set of data from the previously generated samples (Smith et al., 1992; Arulampalam et al., 2002).
Revised: An alternative to reweighting updates is to instead correct the distribution before updating the estimator using weighted bootstrap sampling: resampling a new set of data from the previously generated samples (Smith et al., 1992; Arulampalam et al., 2002).

Original: Consider a setting where a buffer of data is stored, generated by a behavior policy.
Revised: Consider a setting where a buffer of data is stored, generated by a behavior policy.

Original: Samples for policy π can be obtained by resampling from this buffer, proportionally to π ( a | s ) µ ( a | s ) for state-action pairs ( s, a ) in the buffer.
Revised: Samples for policy π can be obtained by resampling from this buffer, proportionally to π ( a | s ) µ ( a | s ) for state-action pairs ( s, a ) in the buffer.

Original: In the sampling literature, this strategy has been proposed under the name Sampling Importance Resampling (SIR) (Rubin, 1988; Smith et al., 1992; Gordon et al., 1993), and has been particularly successful for Sequential
Revised: In the sampling literature, this strategy has been proposed under the name Sampling Importance Resampling (SIR) (Rubin, 1988; Smith et al., 1992; Gordon et al., 1993), and has been particularly successful for Sequential

Original: Monte Carlo sampling (Gordon et al., 1993; Skare et al., 2003).
Revised: Monte Carlo sampling (Gordon et al., 1993; Skare et al., 2003).

Original: Such resampling strategies have also been popular in classiﬁcation, with over-sampling or under-sampling typically being preferred to weighted (cost-sensitive) updates (Lopez et al., 2013).
Revised: Such resampling strategies have also been popular in classiﬁcation, with over-sampling or under-sampling typically being preferred to weighted (cost-sensitive) updates (Lopez et al., 2013).

Original: Such a resampling strategy, however, has yet to be proposed for policy evaluation, though there are several potential beneﬁts.
Revised: Such a resampling strategy, however, has yet to be proposed for policy evaluation, though there are several potential beneﬁts.

Original: By correcting the distribution before updating, standard on-policy updates can be used, without needing to modify or re-derive with IS ratios.
Revised: By correcting the distribution before updating, standard on-policy updates can be used, without needing to modify or re-derive with IS ratios.

Original: This simpliﬁes application of different optimizers, such as those with momentum terms.
Revised: This simpliﬁes application of different optimizers, such as those with momentum terms.

Original: Another beneﬁt is that the magnitude of the updates will vary less—because updates are not multiplied by very small or very large importance sampling ratios—potentially reducing variance of stochastic updates and simplifying stepsize selection.
Revised: Another beneﬁt is that the magnitude of the updates will vary less—because updates are not multiplied by very small or very large importance sampling ratios—potentially reducing variance of stochastic updates and simplifying stepsize selection.

Original: Resampling should have larger beneﬁts for learning approaches, as compared to averaging or numerical integration problems, because updates accumulate in the weight vector and change the optimization trajectory of the weights.
Revised: Resampling should have larger beneﬁts for learning approaches, as compared to averaging or numerical integration problems, because updates accumulate in the weight vector and change the optimization trajectory of the weights.

Original: For example, very large importance sampling ratios could destabilize the weights.
Revised: For example, very large importance sampling ratios could destabilize the weights.

Original: Such a problem does not occur for resampling, as instead the same transition will simply be resampled multiple times, spreading out the large magnitude update across multiple updates.
Revised: Such a problem does not occur for resampling, as instead the same transition will simply be resampled multiple times, spreading out the large magnitude update across multiple updates.

Original: On the other extreme, with small ratios, IS will waste updates on transitions with very small IS ratios.
Revised: On the other extreme, with small ratios, IS will waste updates on transitions with very small IS ratios.

Original: Resampling, therefore, should have better sample efﬁciency.
Revised: Resampling, therefore, should have better sample efﬁciency.

Original: Two important questions, therefore, are if these hypothesized advantages manifest in practice in off-policy learning, and how SIR can be extended for use in policy evaluation.
Revised: Two important questions, therefore, are if these hypothesized advantages manifest in practice in off-policy learning, and how SIR can be extended for use in policy evaluation.

Original: In this work, we investigate the use of SIR for online off-policy policy evaluation.
Revised: In this work, we investigate the use of SIR for online off-policy policy evaluation.

Original: We ﬁrst introduce
Revised: We ﬁrst introduce

Original: Importance Resampling (IR), which uses an SIR strategy to sample transitions from a buffer of (recent) transitions.
Revised: Importance Resampling (IR), which uses an SIR strategy to sample transitions from a buffer of (recent) transitions.

Original: These sampled transitions are then used for on-policy updates.
Revised: These sampled transitions are then used for on-policy updates.

Original: We show that
Revised: We show that

Original: IR, with a sliding-window buffer, is a consistent estimator of the one-step on-policy updates, with the same bias as WIS.
Revised: IR, with a sliding-window buffer, is a consistent estimator of the one-step on-policy updates, with the same bias as WIS.

Original: We then empirically investigate IR on three toy domains and a racing car simulation learning from images.
Revised: We then empirically investigate IR on three toy domains and a racing car simulation learning from images.

Original: We ﬁnd that IR is more sample efﬁcient—learning more quickly in terms of number of updates—and has reduced sensitivity in terms of learning rate parameters, for both ﬁxed parameters and within the RMSProp optimizer.
Revised: We ﬁnd that IR is more sample efﬁcient—learning more quickly in terms of number of updates—and has reduced sensitivity in terms of learning rate parameters, for both ﬁxed parameters and within the RMSProp optimizer.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We consider the problem of learning General Value Functions (GVFs) (Sutton et al., 2011).
Revised: We consider the problem of learning General Value Functions (GVFs) (Sutton et al., 2011).

Original: The agent interacts in an environment deﬁned by a set of states S , a set of actions A and Markov transition dynamics, with probability P( s (cid:48) | s, a ) of transitions to state s (cid:48) when taking action a in state s .
Revised: The agent interacts in an environment deﬁned by a set of states S , a set of actions A and Markov transition dynamics, with probability P( s (cid:48) | s, a ) of transitions to state s (cid:48) when taking action a in state s .

Original: A
Revised: A

Original: GVF is deﬁned for policy π : S×A→ [0 , 1] , cumulant c : S×A×S → R and continuation function γ : S×A×S → [0 , 1] , with c t +1 def =
Revised: GVF is deﬁned for policy π : S×A→ [0 , 1] , cumulant c : S×A×S → R and continuation function γ : S×A×S → [0 , 1] , with c t +1 def =

Original: c ( S t , A t , S t +1 ) and γ t +1 def = γ ( S t , A t , S t +1 ) , with values
Revised: c ( S t , A t , S t +1 ) and γ t +1 def = γ ( S t , A t , S t +1 ) , with values

Original: V ( s ) def =
Revised: V ( s ) def =

Original: E π [ c t +1 + γ t +1 c t +2 + γ t +1 γ t
Revised: E π [ c t +1 + γ t +1 c t +2 + γ t +1 γ t

Original: +2 c t +3 + . . .
Revised: +2 c t +3 + . . .

Original: | S t = s ]
Revised: | S t = s ]

Original: [Equation]
Revised: [Equation]

Original: E π (cid:104)
Revised: E π (cid:104)

Original: [Equation]
Revised: [Equation]

Original: The operator E π indicates the actions are selected according to policy π for the expectation.
Revised: The operator E π indicates the actions are selected according to policy π for the expectation.

Original: GVFs encompass standard deﬁnitions of value functions, where the cumulant is a reward and the continuation function is a constant.
Revised: GVFs encompass standard deﬁnitions of value functions, where the cumulant is a reward and the continuation function is a constant.

Original: Otherwise, they specify a broader set of value functions, that enable predictions about discounted sums of others signals into the future, when following a target policy π .
Revised: Otherwise, they specify a broader set of value functions, that enable predictions about discounted sums of others signals into the future, when following a target policy π .

Original: These values are typically estimated using parametric function approximation, with parameters θ ∈ R
Revised: These values are typically estimated using parametric function approximation, with parameters θ ∈ R

Original: d deﬁning approximate values V θ
Revised: d deﬁning approximate values V θ

Original: ( s ) .
Revised: ( s ) .

Original: In off-policy learning, transitions are sampled according to behaviour policy, rather than the target policy.
Revised: In off-policy learning, transitions are sampled according to behaviour policy, rather than the target policy.

Original: To get an unbiased sample of an update to the parameters, the action probabilities need to be adjusted.
Revised: To get an unbiased sample of an update to the parameters, the action probabilities need to be adjusted.

Original: Consider on-policy temporal difference (TD) learning, with update α t δ t ∇ θ V θ ( s ) for a given S t = s , for stepsize α t ∈ R + and TD-error δ t def = C t +1 + γ t +1 V θ
Revised: Consider on-policy temporal difference (TD) learning, with update α t δ t ∇ θ V θ ( s ) for a given S t = s , for stepsize α t ∈ R + and TD-error δ t def = C t +1 + γ t +1 V θ

Original: ( S t +1 )
Revised: ( S t +1 )

Original: − V θ ( s ) .
Revised: − V θ ( s ) .

Original: If actions are instead sampled according to a behaviour policy µ : S ×
Revised: If actions are instead sampled according to a behaviour policy µ : S ×

Original: A → [0 , 1] , then we can use importance sampling (IS) to modify the update, giving the off-policy TD update α t ρ t δ t ∇ θ V θ ( s ) for IS ratio ρ t def =
Revised: A → [0 , 1] , then we can use importance sampling (IS) to modify the update, giving the off-policy TD update α t ρ t δ t ∇ θ V θ ( s ) for IS ratio ρ t def =

Original: π ( A t | S t ) µ ( A t | S t ) .
Revised: π ( A t | S t ) µ ( A t | S t ) .

Original: Given state S t
Revised: Given state S t

Original: = s , if µ ( a | s ) > 0 when π ( a | s ) > 0 , then the expected value of these two updates are equal.
Revised: = s , if µ ( a | s ) > 0 when π ( a | s ) > 0 , then the expected value of these two updates are equal.

Original: To see why, notice that
Revised: To see why, notice that

Original: [Equation]
Revised: [Equation]

Original: and we have
Revised: and we have

Original: [Equation]
Revised: [Equation]

Original: Other on-policy updates can also be modiﬁed with IS ratios to adjust these action probabilities.
Revised: Other on-policy updates can also be modiﬁed with IS ratios to adjust these action probabilities.

Original: Though unbiased, IS can be high-variance, and so weighted IS ratios are typically preferred.
Revised: Though unbiased, IS can be high-variance, and so weighted IS ratios are typically preferred.

Original: For a batch consisting of transitions { ( s i , a i , s i +1 ,
Revised: For a batch consisting of transitions { ( s i , a i , s i +1 ,

Original: c i +1 , ρ i ) } ni
Revised: c i +1 , ρ i ) } ni

Original: =1 , batch WIS uses a normalized estimate for the update.
Revised: =1 , batch WIS uses a normalized estimate for the update.

Original: For example, an ofﬂine batch WIS TD algorithm would use update α t ρ t δ t ∇ θ V θ ( s ) (cid:80) ni
Revised: For example, an ofﬂine batch WIS TD algorithm would use update α t ρ t δ t ∇ θ V θ ( s ) (cid:80) ni

Original: =

Revised: =

Original: ρ i .
Revised: ρ i .

Original: When learning online, an efﬁcient WIS update is less straightforward, and has resulted in algorithms specialized to the tabular setting (Precup et al., 2001) or linear functions (Mahmood et al., 2014;
Revised: When learning online, an efﬁcient WIS update is less straightforward, and has resulted in algorithms specialized to the tabular setting (Precup et al., 2001) or linear functions (Mahmood et al., 2014;

Original: Mahmood and Sutton, 2015).
Revised: Mahmood and Sutton, 2015).

Original: We nonetheless use WIS as baseline, in the experiments and theory.
Revised: We nonetheless use WIS as baseline, in the experiments and theory.

Original:  R ESAMPLING STRATEGIES FOR OFF - POLICY POLICY EVALUATION
Revised:  R ESAMPLING STRATEGIES FOR OFF - POLICY POLICY EVALUATION

Original: In this section, we introduce resampling, as an alternative to importance sampling for off-policy learning.
Revised: In this section, we introduce resampling, as an alternative to importance sampling for off-policy learning.

Original: We ﬁrst introduce the algorithm, Importance Resampling (IR).
Revised: We ﬁrst introduce the algorithm, Importance Resampling (IR).

Original: We then prove consistency and characterize the bias.
Revised: We then prove consistency and characterize the bias.

Original: We conclude with a discussion about its variance properties.
Revised: We conclude with a discussion about its variance properties.

Original: Importance Resampling requires access to a buffer of samples, from which we can resample.
Revised: Importance Resampling requires access to a buffer of samples, from which we can resample.

Original: Replaying experience from a buffer was introduced as a biologically plausible mechanism to reuse old experience (Lin, 1992; 1993), and has since become common for improving sample efﬁciency, particularly for control (Mnih et al., 2015; Schaul et al., 2015b).
Revised: Replaying experience from a buffer was introduced as a biologically plausible mechanism to reuse old experience (Lin, 1992; 1993), and has since become common for improving sample efﬁciency, particularly for control (Mnih et al., 2015; Schaul et al., 2015b).

Original: In the simplest case—which we assume here—the buffer is a sliding window of the most recent n samples, { ( s i , a i , s i +1 , c i +1 , ρ i ) } ti = t − n , at time step t > n .
Revised: In the simplest case—which we assume here—the buffer is a sliding window of the most recent n samples, { ( s i , a i , s i +1 , c i +1 , ρ i ) } ti = t − n , at time step t > n .

Original: These samples are generated by taking actions according to behaviour µ , and so the tuples are generated with probability d µ ( s ) µ ( a | s )P( s (cid:48) | s, a ) , where d µ : S → [0 , 1] is the stationary distribution for policy µ .
Revised: These samples are generated by taking actions according to behaviour µ , and so the tuples are generated with probability d µ ( s ) µ ( a | s )P( s (cid:48) | s, a ) , where d µ : S → [0 , 1] is the stationary distribution for policy µ .

Original: The goal is to obtain samples instead according to d µ ( s ) π ( a | s )P( s (cid:48) | s, a ) , as if we had taken actions according to policy π from state s ∼ d µ .
Revised: The goal is to obtain samples instead according to d µ ( s ) π ( a | s )P( s (cid:48) | s, a ) , as if we had taken actions according to policy π from state s ∼ d µ .

Original: This assumption—that states are still sampled from d µ —underlies most off-policy learning algorithms; very few attempt to use IS to adjust probabilities d µ to d π (Precup et al., 2001).
Revised: This assumption—that states are still sampled from d µ —underlies most off-policy learning algorithms; very few attempt to use IS to adjust probabilities d µ to d π (Precup et al., 2001).

Original: The IR algorithm is simple: resample a mini-batch of size k on each step t from the buffer of size n , proportionally to ρ i in the buffer.
Revised: The IR algorithm is simple: resample a mini-batch of size k on each step t from the buffer of size n , proportionally to ρ i in the buffer.

Original: Standard on-policy updates, such as on-policy TD or on-policy gradient TD, are then used on this resample.
Revised: Standard on-policy updates, such as on-policy TD or on-policy gradient TD, are then used on this resample.

Original: The key difference to IS and WIS is that the sampling distribution itself is corrected (see Theorem C.1), before the update, whereas IS and WIS correct the update itself.
Revised: The key difference to IS and WIS is that the sampling distribution itself is corrected (see Theorem C.1), before the update, whereas IS and WIS correct the update itself.

Original: This small difference, however, can have larger ramiﬁcations practically, particularly with updates that accumulate in the parameters.
Revised: This small difference, however, can have larger ramiﬁcations practically, particularly with updates that accumulate in the parameters.

Original: We consider two variants of IR: with and without bias correction.
Revised: We consider two variants of IR: with and without bias correction.

Original: For point t j sampled from the buffer, let ∆ t
Revised: For point t j sampled from the buffer, let ∆ t

Original: j be the on-policy update for that transition.
Revised: j be the on-policy update for that transition.

Original: For example, for TD, ∆ t j = δ t j ∇ θ V θ ( s t j ) .
Revised: For example, for TD, ∆ t j = δ t j ∇ θ V θ ( s t j ) .

Original: The ﬁrst step for either variant is to sample a mini-batch of size k from the buffer, proportionally to ρ i , as described above.
Revised: The ﬁrst step for either variant is to sample a mini-batch of size k from the buffer, proportionally to ρ i , as described above.

Original: The standard IR update simply uses a mini-batch, whereas Bias-Corrected
Revised: The standard IR update simply uses a mini-batch, whereas Bias-Corrected

Original: IR (BC-IR) pre-multiplies with the average ratio in the buffer ¯ ρ t = 1 n (cid:80) ni
Revised: IR (BC-IR) pre-multiplies with the average ratio in the buffer ¯ ρ t = 1 n (cid:80) ni

Original: =

Revised: =

Original: ρ i ,
Revised: ρ i ,

Original: IR
Revised: IR

Original: [Equation]
Revised: [Equation]

Original: BC-IR negates bias introduced by the average ratio in the buffer deviating signiﬁcantly from the true mean.
Revised: BC-IR negates bias introduced by the average ratio in the buffer deviating signiﬁcantly from the true mean.

Original: For reasonably large buffers, ¯ ρ t will be close to 1 making IR and BC-IR have nearidentical updates.
Revised: For reasonably large buffers, ¯ ρ t will be close to 1 making IR and BC-IR have nearidentical updates.

Original: In practice, we ﬁnd the two variants of IR perform similarly.
Revised: In practice, we ﬁnd the two variants of IR perform similarly.

Original: Nonetheless, they do have different theoretical properties, particularly for small buffer sizes n , so we characterize both.
Revised: Nonetheless, they do have different theoretical properties, particularly for small buffer sizes n , so we characterize both.

Original: Though BC-IR has better bias properties, IR is simpler—not requiring any modiﬁcation to the updates—which may be more important than the small amount of bias introduced by the IR without bias correction.
Revised: Though BC-IR has better bias properties, IR is simpler—not requiring any modiﬁcation to the updates—which may be more important than the small amount of bias introduced by the IR without bias correction.

Original: For this reason, we advocate for and analyze both variants.
Revised: For this reason, we advocate for and analyze both variants.

Original: Across all results, we make the following assumption.
Revised: Across all results, we make the following assumption.

Original: Assumption 1.
Revised: Assumption 1.

Original: Transition tuples
Revised: Transition tuples

Original: X i =
Revised: X i =

Original: ( S i , A
Revised: ( S i , A

Original: i , S i +1 ) are sampled i.i.d.
Revised: i , S i +1 ) are sampled i.i.d.

Original: according to the distribution p ( x = ( s, a, s (cid:48) ))
Revised: according to the distribution p ( x = ( s, a, s (cid:48) ))

Original: =
Revised: =

Original: d µ ( s ) µ ( a | s )P( s (cid:48) | s, a ) , for i = 1 , 2 , 3 , . . . .
Revised: d µ ( s ) µ ( a | s )P( s (cid:48) | s, a ) , for i = 1 , 2 , 3 , . . . .

Original: To distinguish expectations under p ( x )
Revised: To distinguish expectations under p ( x )

Original: =
Revised: =

Original: d µ ( s ) µ ( a | s )P( s (cid:48) | s, a ) and q ( x ) =
Revised: d µ ( s ) µ ( a | s )P( s (cid:48) | s, a ) and q ( x ) =

Original: d µ ( s ) π ( a | s )P( s (cid:48) | s, a ) , we overload the notation from above, using operators E µ and E π respectively.
Revised: d µ ( s ) π ( a | s )P( s (cid:48) | s, a ) , we overload the notation from above, using operators E µ and E π respectively.

Original: To reduce clutter, we will typically write E to mean E µ , because most expectations are under the sampling distribution.
Revised: To reduce clutter, we will typically write E to mean E µ , because most expectations are under the sampling distribution.

Original: 3. B IAS OF IR
Revised: 3. B IAS OF IR

Original: We ﬁrst show that IR is biased, and that its bias is actually equal to batch WIS (in Theorem 3.1).
Revised: We ﬁrst show that IR is biased, and that its bias is actually equal to batch WIS (in Theorem 3.1).

Original: This bias is small for reasonably large n , because it is proportional to 1 /n .
Revised: This bias is small for reasonably large n , because it is proportional to 1 /n .

Original: In terms of mean-squared error, composed of squared bias and variance, this term is 1 /n 2 and is relatively negligible compared to the variance.
Revised: In terms of mean-squared error, composed of squared bias and variance, this term is 1 /n 2 and is relatively negligible compared to the variance.

Original: Nonetheless, for smaller buffers, such bias could have an impact.
Revised: Nonetheless, for smaller buffers, such bias could have an impact.

Original: We show that with a simple modiﬁcation, in BC-IR, we obtain an unbiased estimate of the update (Corollary 3.1.1).
Revised: We show that with a simple modiﬁcation, in BC-IR, we obtain an unbiased estimate of the update (Corollary 3.1.1).

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: [Bias for a ﬁxed buffer of size n ] Assume a buffer B of n transitions is sampled i.i.d., according to d µ ( s ) µ ( a | s )P( s (cid:48) | s, a ) .
Revised: [Bias for a ﬁxed buffer of size n ] Assume a buffer B of n transitions is sampled i.i.d., according to d µ ( s ) µ ( a | s )P( s (cid:48) | s, a ) .

Original: Let X IR def = 1 k (cid:80) kj =1 ∆ i j for transitions i 1 , . .
Revised: Let X IR def = 1 k (cid:80) kj =1 ∆ i j for transitions i 1 , . .

Original: , i k sampled randomly from the buffer proportionally to ρ i .
Revised: , i k sampled randomly from the buffer proportionally to ρ i .

Original: Let X WIS ∗ def = (cid:80) ni
Revised: Let X WIS ∗ def = (cid:80) ni

Original: = ρ
Revised: = ρ

Original: i (cid:80) nj =1 ρ j ∆ i be the batch WIS estimator of the update, with ρ i def =
Revised: i (cid:80) nj =1 ρ j ∆ i be the batch WIS estimator of the update, with ρ i def =

Original: π ( A i | S i ) /µ
Revised: π ( A i | S i ) /µ

Original: ( A i | S i ) .
Revised: ( A i | S i ) .

Original: Then, E [ X IR ]
Revised: Then, E [ X IR ]

Original: =
Revised: =

Original: E [ X WIS ∗ ] , and so the bias of X IR is proportional to
Revised: E [ X WIS ∗ ] , and so the bias of X IR is proportional to

Original: [Equation]
Revised: [Equation]

Original: where E π [∆] is the expected update across all transitions, with actions from S taken by the target policy π ; σ 2 ρ =
Revised: where E π [∆] is the expected update across all transitions, with actions from S taken by the target policy π ; σ 2 ρ =

Original: Var( 1 n (cid:80) nj =1 ρ j ) ; σ 2∆ =
Revised: Var( 1 n (cid:80) nj =1 ρ j ) ; σ 2∆ =

Original: Var( 1 n (cid:80) ni
Revised: Var( 1 n (cid:80) ni

Original: =

Revised: =

Original: ρ i ∆ i ) ; and covariance σ ( ρ, ∆) =
Revised: ρ i ∆ i ) ; and covariance σ ( ρ, ∆) =

Original: Cov( 1 n (cid:80) nj =

Revised: Cov( 1 n (cid:80) nj =

Original: ρ j , 1 n (cid:80) ni
Revised: ρ j , 1 n (cid:80) ni

Original: =

Revised: =

Original: ρ i ∆ i ) .
Revised: ρ i ∆ i ) .

Original: Proof.
Revised: Proof.

Original: Notice ﬁrst that when we weight with ρ i , this is equivalent to weighting with d µ ( S i ) π
Revised: Notice ﬁrst that when we weight with ρ i , this is equivalent to weighting with d µ ( S i ) π

Original: ( A i | S i )P( S i +1 | S i ,A i )
Revised: ( A i | S i )P( S i +1 | S i ,A i )

Original: d µ ( S i ) µ ( A i | S i )P
Revised: d µ ( S i ) µ ( A i | S i )P

Original: ( S i +1 |
Revised: ( S i +1 |

Original: S i ,A i ) , and so we are applying the correct IS ratio for the transition.
Revised: S i ,A i ) , and so we are applying the correct IS ratio for the transition.

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: (cid:46)
Revised: (cid:46)

Original: because E [∆ i j | B ] = n (cid:88)
Revised: because E [∆ i j | B ] = n (cid:88)

Original: i =
Revised: i =

Original: ρ i (cid:80) nj =1 ρ j ∆ i
Revised: ρ i (cid:80) nj =1 ρ j ∆ i

Original: This bias of X IR is therefore the same as batch WIS, which is characterized in (Owen, 2013, Section 2.7), completing the proof.
Revised: This bias of X IR is therefore the same as batch WIS, which is characterized in (Owen, 2013, Section 2.7), completing the proof.

Original: This bias of IR will be small for reasonably large n , both because it is proportional to 1 /n and because larger n will result in lower variance of the average ratios and average update for the buffer.
Revised: This bias of IR will be small for reasonably large n , both because it is proportional to 1 /n and because larger n will result in lower variance of the average ratios and average update for the buffer.

Original: In particular, as n grows, these variances decay proportionally to n .
Revised: In particular, as n grows, these variances decay proportionally to n .

Original: Corollary 3.1.1.
Revised: Corollary 3.1.1.

Original: Bias-corrected IR, with estimator X BC def = ¯ ρk (cid:80) kj =1 ∆ i j for ¯ ρ = 1 n (cid:80) nj =1 ρ j , is unbiased: E [ X BC ]
Revised: Bias-corrected IR, with estimator X BC def = ¯ ρk (cid:80) kj =1 ∆ i j for ¯ ρ = 1 n (cid:80) nj =1 ρ j , is unbiased: E [ X BC ]

Original: =
Revised: =

Original: E π [∆] .
Revised: E π [∆] .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: 3. C ONSISTENCY OF IR
Revised: 3. C ONSISTENCY OF IR

Original: Consistency of IR in terms of an increasing buffer, with n → ∞ , is a relatively straightforward extension on results for SIR, with or without bias correction (see Theorem C.1 in Appendix C).
Revised: Consistency of IR in terms of an increasing buffer, with n → ∞ , is a relatively straightforward extension on results for SIR, with or without bias correction (see Theorem C.1 in Appendix C).

Original: More interesting is consistency in terms of increasing interactions with the environment, t → ∞ , with a ﬁxed length buffer, as will be the case in practice.
Revised: More interesting is consistency in terms of increasing interactions with the environment, t → ∞ , with a ﬁxed length buffer, as will be the case in practice.

Original: IR, without bias correction, is asymptotically biased in this case; in fact, its asymptotic bias is the one characterized above for a ﬁxed length buffer in Theorem 3.1.
Revised: IR, without bias correction, is asymptotically biased in this case; in fact, its asymptotic bias is the one characterized above for a ﬁxed length buffer in Theorem 3.1.

Original: This asymptotic bias, though, is proportional to 1 /n , which is negligible for typical buffer sizes.
Revised: This asymptotic bias, though, is proportional to 1 /n , which is negligible for typical buffer sizes.

Original: BC-IR, on the other hand, is consistent, even with a sliding window, as we show in the following theorem.
Revised: BC-IR, on the other hand, is consistent, even with a sliding window, as we show in the following theorem.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Let B i =
Revised: Let B i =

Original: { X i − n +1 , ..., X i } be the buffer of the most recent n transitions sampled by time i , i.i.d.
Revised: { X i − n +1 , ..., X i } be the buffer of the most recent n transitions sampled by time i , i.i.d.

Original: as speciﬁed in Assumption 1.
Revised: as speciﬁed in Assumption 1.

Original: Let X ( i )BC be the bias-corrected IR estimator, with k samples from buffer B i .
Revised: Let X ( i )BC be the bias-corrected IR estimator, with k samples from buffer B i .

Original: Deﬁne the sliding-window estimator X t def = 1 t (cid:80) ti =1 X ( i )BC .
Revised: Deﬁne the sliding-window estimator X t def = 1 t (cid:80) ti =1 X ( i )BC .

Original: Assume there exists a c > 0 such that Var( X ( i )BC ) ≤ c ∀ i .
Revised: Assume there exists a c > 0 such that Var( X ( i )BC ) ≤ c ∀ i .

Original: Then, as t → ∞ , X t converges in probability to E π [∆] .
Revised: Then, as t → ∞ , X t converges in probability to E π [∆] .

Original: Proof.
Revised: Proof.

Original: Notice ﬁrst that X ( i ) BC is random because B i is random and because transitions are sampled from B i .
Revised: Notice ﬁrst that X ( i ) BC is random because B i is random and because transitions are sampled from B i .

Original: Therefore, given B i , X ( i )BC is independent of other random variables B j and X ( j ) BC for j (cid:54) =
Revised: Therefore, given B i , X ( i )BC is independent of other random variables B j and X ( j ) BC for j (cid:54) =

Original: i .
Revised: i .

Original: Now, using Corollary 3.1.1, we can show that BC-IR is unbiased for a sliding window
Revised: Now, using Corollary 3.1.1, we can show that BC-IR is unbiased for a sliding window

Original: [Equation]
Revised: [Equation]

Original: π [∆] .
Revised: π [∆] .

Original: Next, we show that lim
Revised: Next, we show that lim

Original: | i − j |→∞ Cov
Revised: | i − j |→∞ Cov

Original: ( X ( i )BC , X ( j ) BC )
Revised: ( X ( i )BC , X ( j ) BC )

Original: = 0 .
Revised: = 0 .

Original: For | i − j | ≥ m , B i and B j are independent, because they are disjoint sets of i.i.d.
Revised: For | i − j | ≥ m , B i and B j are independent, because they are disjoint sets of i.i.d.

Original: random variables.
Revised: random variables.

Original: Correspondingly, X ( i )BC is independent of X ( j ) BC .
Revised: Correspondingly, X ( i )BC is independent of X ( j ) BC .

Original: Explicitly, using the law of total covariance, we get that Cov( X ( i )BC , X ( j ) BC ) =
Revised: Explicitly, using the law of total covariance, we get that Cov( X ( i )BC , X ( j ) BC ) =

Original: Cov( X ( i )BC , X ( j ) BC | B i , B j ) + Cov( E [ X ( i )BC | B i ] , E [ X ( j ) BC | B j ]) = 0 .
Revised: Cov( X ( i )BC , X ( j ) BC | B i , B j ) + Cov( E [ X ( i )BC | B i ] , E [ X ( j ) BC | B j ]) = 0 .

Original: The ﬁrst term is zero because X ( i )BC is independent of X ( j ) BC given B i , and the second term is zero because B i and B j are independent.
Revised: The ﬁrst term is zero because X ( i )BC is independent of X ( j ) BC given B i , and the second term is zero because B i and B j are independent.

Original: Therefore, lim | i − j |→∞ Cov( X ( i )BC , X ( j ) BC )
Revised: Therefore, lim | i − j |→∞ Cov( X ( i )BC , X ( j ) BC )

Original: = 0 .
Revised: = 0 .

Original: Using the assumption on the variance, we can apply Lemma C.3 to X t to get the desired result.
Revised: Using the assumption on the variance, we can apply Lemma C.3 to X t to get the desired result.

Original: 3. V ARIANCE AND E FFECTIVE SAMPLE SIZE
Revised: 3. V ARIANCE AND E FFECTIVE SAMPLE SIZE

Original: In this section, we provide some intuition on the variance properties of the discussed off-policy estimators.
Revised: In this section, we provide some intuition on the variance properties of the discussed off-policy estimators.

Original: Similarly to bias, we can characterize the variance of the IR estimator relative to batch
Revised: Similarly to bias, we can characterize the variance of the IR estimator relative to batch

Original: WIS.
Revised: WIS.

Original: X WIS ∗ is able to use a batch update on all the data in the buffer, which should result in a lowvariance estimate but is an unrealistic algorithm to use in practice.
Revised: X WIS ∗ is able to use a batch update on all the data in the buffer, which should result in a lowvariance estimate but is an unrealistic algorithm to use in practice.

Original: Instead, it provides a benchmark, where the goal is to obtain similar variance to X WIS ∗ , but within realistic computational restrictions.
Revised: Instead, it provides a benchmark, where the goal is to obtain similar variance to X WIS ∗ , but within realistic computational restrictions.

Original: Because of the clear relationship between IR and WIS, as used in Theorem 3.1, we can easily characterize the variance of X IR relative to X WIS ∗ using the law of total covariance:
Revised: Because of the clear relationship between IR and WIS, as used in Theorem 3.1, we can easily characterize the variance of X IR relative to X WIS ∗ using the law of total covariance:

Original: [Equation]
Revised: [Equation]

Original: where the variability is due to having randomly sampled buffers B and random sampling from B .
Revised: where the variability is due to having randomly sampled buffers B and random sampling from B .

Original: The second term corresponds to the noise introduced by sampling a mini-batch of k transitions from the buffer B , instead of using the entire buffer like WIS.
Revised: The second term corresponds to the noise introduced by sampling a mini-batch of k transitions from the buffer B , instead of using the entire buffer like WIS.

Original: For more insight, we can expand this second term, E [ V [ X IR | B ]]
Revised: For more insight, we can expand this second term, E [ V [ X IR | B ]]

Original: = E (cid:104) ( 1 k (cid:80) kj =1 ∆ i j − 1 n (cid:80) ni
Revised: = E (cid:104) ( 1 k (cid:80) kj =1 ∆ i j − 1 n (cid:80) ni

Original: =1 ∆ i ) 2 | B (cid:105) , where we consider the variance independently for each element of ∆ i and so apply the square element-wise.
Revised: =1 ∆ i ) 2 | B (cid:105) , where we consider the variance independently for each element of ∆ i and so apply the square element-wise.

Original: The variability is not due to IS ratios, and instead arises from variability in the updates themselves.
Revised: The variability is not due to IS ratios, and instead arises from variability in the updates themselves.

Original: Therefore, the variance of IR corresponds to the variance of WIS, with some additional variance due to this variability around the average update in the buffer.
Revised: Therefore, the variance of IR corresponds to the variance of WIS, with some additional variance due to this variability around the average update in the buffer.

Original: This variance contrasts the variance of the corresponding mini-batch IS estimator:
Revised: This variance contrasts the variance of the corresponding mini-batch IS estimator:

Original: [Equation]
Revised: [Equation]

Original: For large importance sampling ratios, this mini-batch can deviate signiﬁcantly around the mean update.
Revised: For large importance sampling ratios, this mini-batch can deviate signiﬁcantly around the mean update.

Original: Further, this deviation around the mean update is across all transitions, unlike E [ V [ X IR | B ]] which reﬂects the expected deviation for a buffer of size n .
Revised: Further, this deviation around the mean update is across all transitions, unlike E [ V [ X IR | B ]] which reﬂects the expected deviation for a buffer of size n .

Original: The IS estimator is unbiased, as opposed to IR and WIS which both introduce some bias.
Revised: The IS estimator is unbiased, as opposed to IR and WIS which both introduce some bias.

Original: A more fair comparison is to BC-IR, which is unbiased, with variance
Revised: A more fair comparison is to BC-IR, which is unbiased, with variance

Original: [Equation]
Revised: [Equation]

Original: It is difﬁcult to state generally that the variability of the IS estimator will be greater than X BC , because it depends on the properties of the update vector ∆ i j itself.
Revised: It is difﬁcult to state generally that the variability of the IS estimator will be greater than X BC , because it depends on the properties of the update vector ∆ i j itself.

Original: However, the key difference between them is that X BC multiplies all the updates by the averaged ratio, whereas IS includes individual (potentially high magnitude) ratios inside the sum.
Revised: However, the key difference between them is that X BC multiplies all the updates by the averaged ratio, whereas IS includes individual (potentially high magnitude) ratios inside the sum.

Original: Finally, the variance of IR will be affected by what is known as the effective sample size .
Revised: Finally, the variance of IR will be affected by what is known as the effective sample size .

Original: For data with several high magnitude ratios, and many small ratios, the IS estimator will likely suffer from high-variance updates.
Revised: For data with several high magnitude ratios, and many small ratios, the IS estimator will likely suffer from high-variance updates.

Original: IR, however, will not be completely robust to this setting either: it will prevent high magnitude updates, but will be sampling from an effectively smaller dataset.
Revised: IR, however, will not be completely robust to this setting either: it will prevent high magnitude updates, but will be sampling from an effectively smaller dataset.

Original: This effective size is smaller because IR will repeatedly sample the same transitions, and potentially never sample some of the transitions with small IS ratios.
Revised: This effective size is smaller because IR will repeatedly sample the same transitions, and potentially never sample some of the transitions with small IS ratios.

Original: With less data, we typically incur more variance.
Revised: With less data, we typically incur more variance.

Original: One common estimator of effective sample size is (
Revised: One common estimator of effective sample size is (

Original: [Equation]
Revised: [Equation]

Original: (Kong et al., 1994; Martino et al., 2017).
Revised: (Kong et al., 1994; Martino et al., 2017).

Original: This estimate lies between 1 and n .
Revised: This estimate lies between 1 and n .

Original: When the effective sample size is low, this indicates that most of the probability is concentrated on a few samples, which could be problematic.
Revised: When the effective sample size is low, this indicates that most of the probability is concentrated on a few samples, which could be problematic.

Original: An important next step is to better understand theoretically the implications of effective sample size.
Revised: An important next step is to better understand theoretically the implications of effective sample size.

Original: Here, we now turn to experiments to gain more insight into the relative efﬁcacy of IR to IS, in settings including such skewed ratios.
Revised: Here, we now turn to experiments to gain more insight into the relative efﬁcacy of IR to IS, in settings including such skewed ratios.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present results for predictions made with difﬁcult polices in three domains.
Revised: In this section, we present results for predictions made with difﬁcult polices in three domains.

Original: We compare IR and BC-IR 2 to an importance sampling approach using uniformly sampled experiences from the replay buffer (ER+IS), and three variants of weighted importance sampling (WIS).
Revised: We compare IR and BC-IR 2 to an importance sampling approach using uniformly sampled experiences from the replay buffer (ER+IS), and three variants of weighted importance sampling (WIS).

Original: The three variants of WIS considered are WIS-Batch, WIS-Buffer, and WIS-Optimal discussed further in the appendix 3 4 .
Revised: The three variants of WIS considered are WIS-Batch, WIS-Buffer, and WIS-Optimal discussed further in the appendix 3 4 .

Original: For tabular domains, we don’t average over the mini-batch updates, although doing so won’t change results signiﬁcantly.
Revised: For tabular domains, we don’t average over the mini-batch updates, although doing so won’t change results signiﬁcantly.

Original: We report parameter studies and learning curves, excluding buffer size studies, on a single buffer size shared between all the methods.
Revised: We report parameter studies and learning curves, excluding buffer size studies, on a single buffer size shared between all the methods.

Original: Although results from other buffer sizes (not shown here) have similar conclusions.
Revised: Although results from other buffer sizes (not shown here) have similar conclusions.

Original: We show 95% conﬁdence intervals on all results unless otherwise speciﬁed.
Revised: We show 95% conﬁdence intervals on all results unless otherwise speciﬁed.

Original: 4. S ETTINGS
Revised: 4. S ETTINGS

Original: Random Walk Markov Chain
Revised: Random Walk Markov Chain

Original: We use a typically constructed random walk Markov chain (Sutton and Barto, 2018) with 8 non-terminating states and 2 terminating, with a reward of 1 on the transition to the right-most terminal state and 0 everywhere else.
Revised: We use a typically constructed random walk Markov chain (Sutton and Barto, 2018) with 8 non-terminating states and 2 terminating, with a reward of 1 on the transition to the right-most terminal state and 0 everywhere else.

Original: The agent follows a policy µ and learns the value function according to a target policy π .
Revised: The agent follows a policy µ and learns the value function according to a target policy π .

Original: We compute the root mean squared value error (RMSVE) on every training step with a value function found using dynamic programming with threshold 10 − 15 . 
Revised: We compute the root mean squared value error (RMSVE) on every training step with a value function found using dynamic programming with threshold 10 − 15 . [Figure] RMSVE [Figure]

Original: [Figure]
Revised: [Figure]

Revised: RMSVE

Original: [Figure]
Revised: [Figure]

Original: Four Rooms Environment
Revised: Four Rooms Environment

Original: The four rooms domain is a well known hard domain used for training options (Stolle and Precup, 2002).
Revised: The four rooms domain is a well known hard domain used for training options (Stolle and Precup, 2002).

Original: The behavior policy followed by the agent is equiprobable everywhere except for 25 randomly selected states which take the action down with probability 0.05 with remaining probability split equally amongst the other actions.
Revised: The behavior policy followed by the agent is equiprobable everywhere except for 25 randomly selected states which take the action down with probability 0.05 with remaining probability split equally amongst the other actions.

Original: The 25 random states are the same for all runs, but we also evaluated different states for each run (see appendix).
Revised: The 25 random states are the same for all runs, but we also evaluated different states for each run (see appendix).

Original: The target policy of interest is to take the down action deterministically, inducing highly variant IS ratios.
Revised: The target policy of interest is to take the down action deterministically, inducing highly variant IS ratios.

Original: The cumulant for the value function is 1 when the agent hits a wall and 0 otherwise.
Revised: The cumulant for the value function is 1 when the agent hits a wall and 0 otherwise.

Original: The continuation function is γ = 0 .
Revised: The continuation function is γ = 0 .

Original: 9 terminating when the agent hits a wall.
Revised: 9 terminating when the agent hits a wall.

Original: We calculate the RMSVE similarly to the Markov
Revised: We calculate the RMSVE similarly to the Markov

Original: Chain.
Revised: Chain.

Original: Simulated Car Domain We use the TORCs race car simulator to perform scaling experiments using neural networks.
Revised: Simulated Car Domain We use the TORCs race car simulator to perform scaling experiments using neural networks.

Original: We set up the simulator to produce 64x128 cropped grayscale images.
Revised: We set up the simulator to produce 64x128 cropped grayscale images.

Original: We have an underlying deterministic steering controller that produces steering actions a det ∈ [ − 1 , +1] and take an action with probability deﬁned by a Gaussian a N ( a det , 0 . 1) .
Revised: We have an underlying deterministic steering controller that produces steering actions a det ∈ [ − 1 , +1] and take an action with probability deﬁned by a Gaussian a N ( a det , 0 . 1) .

Original: We show a demonstration learning a single GVF with cumulant 1 when the car is near the center of the road (signal provided by Torcs), continuation function with 0.9 everywhere with terminating condition the same as the cumulant, and a target policy modeled as a gaussian N (0 . 15 , 0 . 0075) , which corresponds to steering left.
Revised: We show a demonstration learning a single GVF with cumulant 1 when the car is near the center of the road (signal provided by Torcs), continuation function with 0.9 everywhere with terminating condition the same as the cumulant, and a target policy modeled as a gaussian N (0 . 15 , 0 . 0075) , which corresponds to steering left.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: In each of the domains tested we see signiﬁcant improvements in the range of effective learning rates.
Revised: In each of the domains tested we see signiﬁcant improvements in the range of effective learning rates.

Original: The most stark example is seen in the right plot of ﬁgure 1.
Revised: The most stark example is seen in the right plot of ﬁgure 1.

Original: Here the behaviour policy induces importance sampling ratios as high as 99 while also having very few effective samples from which to train.
Revised: Here the behaviour policy induces importance sampling ratios as high as 99 while also having very few effective samples from which to train.

Original: After 1000 training epochs only a single learning rate seemed to not diverge using ER+IS and WIS-Buffer.
Revised: After 1000 training epochs only a single learning rate seemed to not diverge using ER+IS and WIS-Buffer.

Original: In this same setting, IR performs very similar to the value function trained with on policy data.
Revised: In this same setting, IR performs very similar to the value function trained with on policy data.

Original: We also see lower sensitivity to learning rate in the four rooms environment, rightmost ﬁgure 2.
Revised: We also see lower sensitivity to learning rate in the four rooms environment, rightmost ﬁgure 2.

Original: Another beneﬁt of IR is the gains in sample efﬁciency, and the focus on potentially rare samples following the target policy.
Revised: Another beneﬁt of IR is the gains in sample efﬁciency, and the focus on potentially rare samples following the target policy.

Original: We show this with the learning curves found in ﬁgure 3.
Revised: We show this with the learning curves found in ﬁgure 3.

Original: In the random
Revised: In the random

Original: [Figure]
Revised: [Figure]

Original: walk and four rooms domains we see WIS-Buffer and WIS-Batch perform approximately the same as ER+IS with IR outperforming the competitors.
Revised: walk and four rooms domains we see WIS-Buffer and WIS-Batch perform approximately the same as ER+IS with IR outperforming the competitors.

Original: This can be attributed to IR’s resampling scheme, where we see more samples important to training.
Revised: This can be attributed to IR’s resampling scheme, where we see more samples important to training.

Original: This is especially apparent in the four rooms experiments, where we may only get a few chances to train from certain hard to reach states.
Revised: This is especially apparent in the four rooms experiments, where we may only get a few chances to train from certain hard to reach states.

Original: The uniform sampling methods will more likely miss out on rare examples, or only see them once making learning slow.
Revised: The uniform sampling methods will more likely miss out on rare examples, or only see them once making learning slow.

Original: We also note that WIS-Batch is unable to learn in the hardest of settings for the
Revised: We also note that WIS-Batch is unable to learn in the hardest of settings for the

Original: Markov chain, potentially due to the bias incurred from only performing WIS on a subsample of the entire data.
Revised: Markov chain, potentially due to the bias incurred from only performing WIS on a subsample of the entire data.

Original: One interesting observation is in the Baird’s Star Problem where IR results in better weights and lower value error (see appendix).
Revised: One interesting observation is in the Baird’s Star Problem where IR results in better weights and lower value error (see appendix).

Original: The experiments in the Torcs domain show faster learning using a deep learning system.
Revised: The experiments in the Torcs domain show faster learning using a deep learning system.

Original: While the results are promising, they are not as stark as we would have expected given the prior experiments and the variance in ﬁnal performance is much larger for BC-IR and IR.
Revised: While the results are promising, they are not as stark as we would have expected given the prior experiments and the variance in ﬁnal performance is much larger for BC-IR and IR.

Original: There are several contributing factors to learning off-policy using IR and IS that need to be considered.
Revised: There are several contributing factors to learning off-policy using IR and IS that need to be considered.

Original: First by using RMSProp or other adaptive learning rate algorithms we are potentially gaining WIS like beneﬁts, reducing the variance of the updates considerably.
Revised: First by using RMSProp or other adaptive learning rate algorithms we are potentially gaining WIS like beneﬁts, reducing the variance of the updates considerably.

Original: IR may improve performance when the behaviour and target policies cause even more variant importance ratios, but with an adaptive learning rate algorithm this becomes less problematic when using IS.
Revised: IR may improve performance when the behaviour and target policies cause even more variant importance ratios, but with an adaptive learning rate algorithm this becomes less problematic when using IS.

Original: More needs to be understood about the interactions between adaptive algorithms and off-policy learning with importance sampling ratios.
Revised: More needs to be understood about the interactions between adaptive algorithms and off-policy learning with importance sampling ratios.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Resampling for off-policy learning has been unconsidered, to our knowledge, up until now.
Revised: Resampling for off-policy learning has been unconsidered, to our knowledge, up until now.

Original: This may be due to a focus on learning from only the most recent experiences and throwing away transitions once used.
Revised: This may be due to a focus on learning from only the most recent experiences and throwing away transitions once used.

Original: The resampling approach is now viable because of the increased prominence of the experience replay buffer in deep reinforcement learning.
Revised: The resampling approach is now viable because of the increased prominence of the experience replay buffer in deep reinforcement learning.

Original: Previous approaches have exploited the experience replay buffer for orthogonal purposes including Prioritized Experience Replay (Schaul et al., 2015b), which prioritizes training examples according to the temporal difference error w i = |
Revised: Previous approaches have exploited the experience replay buffer for orthogonal purposes including Prioritized Experience Replay (Schaul et al., 2015b), which prioritizes training examples according to the temporal difference error w i = |

Original: δ i | + (cid:15) .
Revised: δ i | + (cid:15) .

Original: A possible extension to IR is to sample from an intermediate sampling distribution which eases high variant importance sampling ratios (see Appendix B.4).
Revised: A possible extension to IR is to sample from an intermediate sampling distribution which eases high variant importance sampling ratios (see Appendix B.5).

Original: In this paper we introduced a new approach to off-policy learning: resampling.
Revised: In this paper we introduced a new approach to off-policy learning: resampling.

Original: We explored the theoretical implications of importance resampling, including a correction term to guarantee consistency in the moving window setting.
Revised: We explored the theoretical implications of importance resampling, including a correction term to guarantee consistency in the moving window setting.

Original: We provided a number of empirical studies in hard off-policy learning settings outperforming the realistic competitors often by a wide margin, while also being less sensitive to learning rate in the mini-batch stochastic gradient update.
Revised: We provided a number of empirical studies in hard off-policy learning settings outperforming the realistic competitors often by a wide margin, while also being less sensitive to learning rate in the mini-batch stochastic gradient update.

Original: We found IR to outperform IS in cases with potentially high importance sampling ratios suggesting a possible reduction in variance.
Revised: We found IR to outperform IS in cases with potentially high importance sampling ratios suggesting a possible reduction in variance.

Original: Finally, we show improvements in learning rate performance for IR and BC-IR methods using deep learning within a challenging racing car simulator environment.
Revised: Finally, we show improvements in learning rate performance for IR and BC-IR methods using deep learning within a challenging racing car simulator environment.

Original: There remains a number of both theoretical and empirical questions surrounding the beneﬁts of the resampling approach to off-policy learning worth exploring in future work.
Revised: There remains a number of both theoretical and empirical questions surrounding the beneﬁts of the resampling approach to off-policy learning worth exploring in future work.

Original: A W EIGHTED I MPORTANCE S AMPLING
Revised: A W EIGHTED I MPORTANCE S AMPLING

Original: We consider three weighted importance sampling updates as competitors to IR.
Revised: We consider three weighted importance sampling updates as competitors to IR.

Original: N is the size of the experience replay buffer, b is the size of a single batch.
Revised: N is the size of the experience replay buffer, b is the size of a single batch.

Original: [Equation]
Revised: [Equation]

Original: WIS-Batch
Revised: WIS-Batch

Original: [Equation]
Revised: [Equation]

Revised: θ ) (cid:80) Nj

Original: WIS-Buffer
Revised: [Equation] WIS- [Figure] [Equation]

Revised: =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:80) Nj ρ 
Revised: (cid:80) Nj ρ j

Original: [Equation] WIS-Optimal
Revised: WIS- [Figure]

Original: B M ORE E XPERIMENTAL R ESULTSB. B AIRD ’ S S TAR P ROBLEM
Revised: B M ORE E XPERIMENTAL R ESULTS

Revised: B. M ARKOV C HAIN

Revised: [Figure]

Revised: We also include results in the random walk markov chain for V-Trace and Sarsa.

Revised: V-Trace exhibited the same issues as importance sampling, where we can see a clear disadvantage in the harder policies.

Revised: Also, we can clearly see a bias-variance trade off in V-Trace as the clipping parameter ¯ ρ became more aggressive.

Revised: Sarsa makes clear gains over ER+IS (and IR in the easiest setting), but fails to learn the true value function for the hardest settings.

Original: We use a well known variant of the Star Problem (Baird, 1995; Sutton et al., 2009) proposed as a counter example to the convergence of semi-gradient temporal difference learning in the off-policy setting.
Revised: We use a well known variant of the Star Problem (Baird, 1995; Sutton et al., 2009) proposed as a counter example to the convergence of semi-gradient temporal difference learning in the off-policy setting.

Original: We train using a batch version of TDC (Sutton et al., 2009).
Revised: We train using a batch version of TDC (Sutton et al., 2009).

Original: We calculate the RMSVE for each state on every training step.[Figure] B [Figure]
Revised: We calculate the RMSVE for each state on every training step.

Original:  F OUR R OOMS D OMAIN
Revised: B. F OUR R OOMS D OMAIN

Original: We also sampled new random states in which to act unfavorably for every run.
Revised: We also sampled new random states in which to act unfavorably for every run.

Original: We found the behavior used in the experiments presented in the main text to be harder than many of the other policies sampled randomly. 
Revised: We found the behavior used in the experiments presented in the main text to be harder than many of the other policies sampled randomly. [Figure]

Original: [Figure]
Revised: [Figure]

Original: B. M OUNTAIN C AR
Revised: B. M OUNTAIN C AR

Original: We use the standard mountain car domain described in (Sutton and Barto, 2018).
Revised: We use the standard mountain car domain described in (Sutton and Barto, 2018).

Original: To make the simulations more realistic we collect experience from behavior policy learned through Q-learning (Sutton and Barto, 2018) with a (cid:15) -greedy exploration strategy.
Revised: To make the simulations more realistic we collect experience from behavior policy learned through Q-learning (Sutton and Barto, 2018) with a (cid:15) -greedy exploration strategy.

Original: We code a GVF to predict whether the agent will hit the back wall within a horizon of γ = 0 .
Revised: We code a GVF to predict whether the agent will hit the back wall within a horizon of γ = 0 .

Original: 9 while following the the persistent policy accelerating forward.
Revised: 9 while following the the persistent policy accelerating forward.

Original: We use two exploration parameters (cid:15) = { 0 . 1 , 0 . 5 } with maximum importance sampling ratio values at ρ max = { 6 , 30 } respectively.
Revised: We use two exploration parameters (cid:15) = { 0 . 1 , 0 . 5 } with maximum importance sampling ratio values at ρ max = { 6 , 30 } respectively.

Original: We show comparisons for both a static step size SGD and with the Adam optimizer.
Revised: We show comparisons for both a static step size SGD and with the Adam optimizer.

Original: These experiments perform as expected, except for one conﬁguration of the behavior policy using the Adam optimizer.
Revised: These experiments perform as expected [Figure] except for one conﬁguration of the behavior policy using the Adam optimizer.

Original: It is possible that the Adam optimizer is getting some beneﬁt similar to WIS with the recency averages of updates accounting for the high variance of the update.
Revised: It is possible that the Adam optimizer is getting some beneﬁt similar to WIS with the recency averages of updates accounting for the high variance of the update

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: B. S AMPLING ACCORDING TO INTERMEDIATE POLICIES
Revised: B. S AMPLING ACCORDING TO INTERMEDIATE POLICIES

Original: While intuitively it might seem sampling according to the target policy will produce the best value functions, when making multiple predictions with the same training network it would be more convenient to sample from the experience replay buffer with an intermediate policy.
Revised: While intuitively it might seem sampling according to the target policy will produce the best value functions, when making multiple predictions with the same training network it would be more convenient to sample from the experience replay buffer with an intermediate policy.

Original: We get the beneﬁts of less variant importance sampling ratios with the ability to use one IR buffer for multiple policies. 
Revised: We get the beneﬁts of less variant importance sampling ratios with the ability to use one IR buffer for multiple policies. [Figure]

Revised: [Figure]

Original: To combine the IR and importance sampling with a uniform experience replay we sample from the buffer according to the PMF
Revised: To combine the IR and importance sampling with a uniform experience replay we sample from the buffer according to the PMF

Original: [Equation]
Revised: [Equation]

Original: We would then us off-policy temporal difference methods with the effective importance sampling ratio used as ρ effective = 
Revised: We would then us off-policy temporal difference methods with the effective importance sampling ratio used as ρ effective = π (

Original: [Equation]
Revised: [Equation]

Revised: |

Original:  We show initial results for this extensionin the markov chain random walk in ﬁgure 9.
Revised: [Equation] [Equation] sample ( a t | s t ) . We show initial results for this extension

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: /OuhJJcUNR1mRbvCAEfZc4RxI9xfKb9ihnF0iZZcCMFvy39Js1YN/GpwulupH47jmCMbZJNsk4Dskzo5JiekQTi5Iw/kiTx7996j9+K9frVOeOOZdfID3vsnGY+jrg==</latexit> <latexit sha1_base64="k/5sEVX3Zoirhl8GGrPRPLY7vOc=">AAACHnicbVDLSgMxFM34tr6qLt0Ei+DGMlMUXRbduFSwrdCpQya9o8FMMiR3hDrtl7jxV9y4UERwpX9jWrvwdSDJ4dx7k5wTZ1JY9P0Pb2Jyanpmdm6+tLC4tLxSXl1rWp0bDg2upTbnMbMghYIGCpRwnhlgaSyhFV8fDeutGzBWaHWGvQw6KbtUIhGcoZOi8l4Ya9m1vdQdRb8fZoLuULdHRZhojUojWHEL1LI0kzDo9y9qUW0QlSt+1R+B/iXBmFTIGCdR+S3sap6noJBLZm078DPsFMyg4O7aUphbyBi/ZpfQdlSxFGynGNkb0C2ndGmijVsK6Uj9PlGw1A4duM6U4ZX9XRuK/9XaOSYHnUKoLEdQ/OuhJJcUNR1mRbvCAEfZc4RxI9xfKb9ihnF0iZZcCMFvy39Js1YN/GpwulupH47jmCMbZJNsk4Dskzo5JiekQTi5Iw/kiTx7996j9+K9frVOeOOZdfID3vsnGY+jrg==</latexit
Revised: [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: +tLC4tLxSXl1rWp0bDg2upTbnMbMghYIGCpRwnhlgaSyhFV8fDeutGzBWaHWGvQw6KbtUIhGcoZOi8l4Ya9m1vdQdRb8fZoLuULdHRZhojUojWHEL1LI0kzDo9y9qUW0QlSt+1R+B/iXBmFTIGCdR+S3sap6noJBLZm078DPsFMyg4O7aUphbyBi/ZpfQdlSxFGynGNkb0C2ndGmijVsK6Uj9PlGw1A4duM6U4ZX9XRuK/9XaOSYHnUKoLEdQ/OuhJJcUNR1mRbvCAEfZc4RxI9xfKb9ihnF0iZZcCMFvy39Js1YN/GpwulupH47jmCMbZJNsk4Dskzo5JiekQTi5Iw/kiTx7996j9+K9frVOeOOZdfID3vsnGY+jrg==</latexit>

Revised: [Figure]

Revised: AverageRMSVE

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: 9XaOSYHnUKoLEdQ/OuhJJcUNR1mRbvCAEfZc4RxI9xfKb9ihnF0iZZcCMFvy39Js1YN/GpwulupH47jmCMbZJNsk4Dskzo5JiekQTi5Iw/kiTx7996j9+K9frVOeOOZdfID3vsnGY+jrg==</latexit> <latexit sha1_base64="k/5sEVX3Zoirhl8GGrPRPLY7vOc

Revised: [Figure]

Original: /GpwulupH47jmCMbZJNsk4Dskzo5JiekQTi5Iw/kiTx7996j9+K9frVOeOOZdfID3vsnGY+jrg==</latexit>
Revised: OuhJJcUNR1mRbvCAEfZc4RxI9xfKb9ihnF0iZZcCMFvy39Js1YN [Figure] kiTx7996j9+K9frVOeOOZdfID3vsnGY+jrg==</latexit>

Original: [Figure]
Revised: [Figure]

Original: C M ORE T HEORETICAL R ESULTS C.C ONSISTENCY OF THE IR ESTIMATOR WITH GROWING BUFFER SIZE
Revised: C M ORE T HEORETICAL R ESULTS C.

Revised: C ONSISTENCY OF THE IR ESTIMATOR WITH GROWING BUFFER SIZE

Original: We show the consistency of the IR estimator with n → ∞ for convenience, but our approach closely follows that of (Smith et al., 1992).
Revised: We show the consistency of the IR estimator with n → ∞ for convenience, but our approach closely follows that of (Smith et al., 1992).

Original: Theorem C.1.
Revised: Theorem C.1.

Original: Let B = { x 1 , x 2 , ..., x n } be a buffer of data sampled i.i.d.
Revised: Let B = { x 1 , x 2 , ..., x n } be a buffer of data sampled i.i.d.

Original: according to proposal distribution p ( x ) .
Revised: according to proposal distribution p ( x ) .

Original: Let q ( x ) be some distribution of interest and assume the proposal distribution samples everywhere where q ( x ) is non-zero.
Revised: Let q ( x ) be some distribution of interest and assume the proposal distribution samples everywhere where q ( x ) is non-zero.

Original: Also, let Y be a discrete random variable taking values x
Revised: Also, let Y be a discrete random variable taking values x

Original: i with probability ∝ q ( x i ) p ( x i ) .
Revised: i with probability ∝ q ( x i ) p ( x i ) .

Original: Then, Y converges in distribution to X ∼ Q as n → ∞ .
Revised: Then, Y converges in distribution to X ∼ Q as n → ∞ .

Original: Proof.
Revised: Proof.

Original: Let ρ i
Revised: Let ρ i

Original: = q ( x i ) p ( x i ) .
Revised: = q ( x i ) p ( x i ) .

Original: From the probability mass function of Y , we have that:
Revised: From the probability mass function of Y , we have that:

Original: [Equation]
Revised: [Equation]

Original: The above shows consistency.
Revised: The above shows consistency.

Original: If we have a large enough buffer, then the importance resampling will closely approximate sampling from the target distribution Q .
Revised: If we have a large enough buffer, then the importance resampling will closely approximate sampling from the target distribution Q .

Original: In particular, any expectation we want to estimate under Q can also be a well-approximated by using the deﬁned sampling scheme.
Revised: In particular, any expectation we want to estimate under Q can also be a well-approximated by using the deﬁned sampling scheme.

Original: C. C HANGING BEHAVIOUR POLICIES
Revised: C. C HANGING BEHAVIOUR POLICIES

Original: We consider the importance sampling estimator under conditions of a changing policy.
Revised: We consider the importance sampling estimator under conditions of a changing policy.

Original: Theorem C.2.
Revised: Theorem C.2.

Original: Let P be a target distribution with density p and { x i } ni =1 be a dataset where each x i is sampled from P i independently.
Revised: Let P be a target distribution with density p and { x i } ni =1 be a dataset where each x i is sampled from P i independently.

Original: Then, the weighted importance sampling estimator ˆ µ = (cid:80)ni
Revised: Then, the weighted importance sampling estimator ˆ µ = (cid:80)

Original: = ρ
Revised: [Equation]

Original: i (cid:80) nj =1 ρ j g ( x i ) is consistent for E
Revised: x i ) is consistent for E

Original: f [ g ( X )] where g ( . ) is a function of interest and ρ i =
Revised: f [ g ( X )] where g ( . ) is a function of interest and ρ i =

Original: q ( x i ) p
Revised: q ( x i ) p

Original: i ( x i ) .
Revised: i ( x i ) .

Original: Proof.
Revised: Proof.

Original: First, we rewrite ˆ µ = 1 n (cid:80) ni
Revised: First, we rewrite ˆ µ = 1 n (cid:80) ni

Original: =

Revised: =

Original: ρ i g ( x i )n (cid:80) nj =
Revised: ρ i g ( x i )n (cid:80) nj =

Original: ρ j and let Y i
Revised: ρ j and let Y i

Original: =
Revised: =

Original: ρ i g ( x i ) .
Revised: ρ i g ( x i ) .

Original: Next, we ﬁnd the expectations of Y i and ρ i .
Revised: Next, we ﬁnd the expectations of Y i and ρ i .

Original: [Equation]
Revised: [Equation]

Original: Finally, since all Y i and all ρ i have the same expectation and are independent, we apply the law of large numbers to obtain that
Revised: Finally, since all Y i and all ρ i have the same expectation and are independent, we apply the law of large numbers to obtain that

Original: [Equation]
Revised: [Equation]

Original: as desired.
Revised: as desired.

Original: Theorem C.2 implies that, even if the behaviour policy is changing over time, as long as we use an importance ratio corresponding to the policy that was used to select an action, we will retain an unbiased estimate of the expected update.
Revised: Theorem C.2 implies that, even if the behaviour policy is changing over time, as long as we use an importance ratio corresponding to the policy that was used to select an action, we will retain an unbiased estimate of the expected update.

Original: C. C ONSISTENCY UNDER A SLIDING WINDOW DATASET
Revised: C. C ONSISTENCY UNDER A SLIDING WINDOW DATASET

Original: Lemma C.3.
Revised: Lemma C.3.

Original: Let Z 1 , ...
Revised: Let Z 1 , ...

Original: Z n be random variables with mean µ .
Revised: Z n be random variables with mean µ .

Original: Suppose there exists a c > 

Revised: Suppose there exists a c >

Original: such that V ( Z i ) ≤ c ∀ i and that lim |
Revised: such that V ( Z i ) ≤ c ∀ i and that lim |

Original: i − j |→∞ Cov( X i , X j )
Revised: i − j |→∞ Cov( X i , X j )

Original: = 0 .
Revised: = 0 .

Original: Then, as N → ∞ , 1 N (cid:80) Ni
Revised: Then, as N → ∞ , 1 N (cid:80) Ni

Original: =

Revised: =

Original: Z i converges in probability to µ .
Revised: Z i converges in probability to µ .

Original: Proof.
Revised: Proof.

Original: Let S N = (cid:80) Ni
Revised: Let S N = (cid:80) Ni

Original: =1 Z i .
Revised: =1 Z i .

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is bounded by cN from our assumption on the variance.
Revised: The ﬁrst term is bounded by cN from our assumption on the variance.

Original: Now, to bound the second term.
Revised: Now, to bound the second term.

Original: Fix δ > 0 and choose M such that ∀| i − j | > M , | Cov( Z i , Z j ) | < δ (such an M must exist since lim | i − j |→∞ Cov( X i , X j ) = 0 ).
Revised: Fix δ > 0 and choose M such that ∀| i − j | > M , | Cov( Z i , Z j ) | < δ (such an M must exist since lim | i − j |→∞ Cov( X i , X j ) = 0 ).

Original: Assuming that N > M , we can decompose the second term into
Revised: Assuming that N > M , we can decompose the second term into

Original: N (cid:88)
Revised: N (cid:88)

Revised: i

Revised: [Equation]

Revised: N (cid:88)

Revised: j = i +

Revised: [Equation]

Revised: ( Z i

Revised: [Equation]

Revised: Z j

Revised: [Equation]

Revised: N (cid:88)

Original: i
Revised: i

Revised: [Equation]

Revised: (cid:88)

Revised: j

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: N (cid:88)

Revised: i

Revised: [Equation]

Revised: (cid:88)

Revised: j = i + M

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Revised: N (cid:88) i =

Revised: N (cid:88)

Revised: j = i +

Revised: Cov

Revised: [Equation]

Revised: i , Z j

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤

Revised: N (cid:88) i

Revised: =

Revised: i + M (cid:88)

Revised: j = i + | Cov( Z i , Z j ) | +

Revised: N (cid:88) i =

Revised: N (cid:88)

Original:  By the Cauchy-Schwarz inequality and our variance assumption, | Cov( Z i , Z j ) | ≤ c .
Revised: j = i + M + | Cov ( Z i , Z j ) | By the Cauchy-Schwarz inequality and our variance assumption, | Cov( Z i , Z j ) | ≤ c .

Original: So, we get (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: So, we get (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Revised: N (cid:88) i =

Original: N (cid:88)
Revised: N (cid:88)

Original: [Equation]
Revised: j = i +

Original:  N 
Revised: Cov( Z i , Z j ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ N (cid:88) i = i + M (cid:88)

Original: [Equation]
Revised: j = i + c + N (cid:88) i = N (cid:88) j = i + M + δ ≤ NMc + N 2 δ

Original: Altogether, our upper bound is
Revised: Altogether, our upper bound is

Revised: V (cid:18) S N

Revised: N (cid:19) ≤ c

Revised: N +

Revised: Mc

Revised: N

Original: [Equation]
Revised: [Equation]

Revised: δ

Original: Finally, we apply Chebyshev’s inequality.
Revised: Finally, we apply Chebyshev’s inequality.

Original: For a ﬁxed (cid:15) > 0 ,
Revised: For a ﬁxed (cid:15) > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Since we can choose δ to be arbitrarily small (sayδ = 1 N ), the right-hand side goes to 0 as N → ∞ , concluding the proof.
Revised: Since we can choose δ to be arbitrarily small (say