Original: E XPLAINING O FF -P OLICY A CTOR -C RITIC
Revised: E XPLAINING O FF -P OLICY A CTOR -C RITIC

Original: F ROM
Revised: F ROM

Original: A
Revised: A

Original: B IAS -V ARIANCE P ERSPECTIVE
Revised: B IAS -V ARIANCE P ERSPECTIVE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations.
Revised: Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations.

Original: To this end, we show its policy evaluation error on the distribution of transitions decomposes into: a Bellman error, a bias from policy mismatch, and a variance term from sampling.
Revised: To this end, we show its policy evaluation error on the distribution of transitions decomposes into: a Bellman error, a bias from policy mismatch, and a variance term from sampling.

Original: By comparing the magnitude of bias and variance, we explain the success of the Emphasizing
Revised: By comparing the magnitude of bias and variance, we explain the success of the Emphasizing

Original: Recent Experience sampling and 1/age weighted sampling.
Revised: Recent Experience sampling and 1/age weighted sampling.

Original: Both sampling strategies yield smaller bias and variance and are hence preferable to uniform sampling.
Revised: Both sampling strategies yield smaller bias and variance and are hence preferable to uniform sampling.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A practical reinforcement learning (RL) algorithm is often in an actor-critic setting (Lin, 1992;
Revised: A practical reinforcement learning (RL) algorithm is often in an actor-critic setting (Lin, 1992;

Original: Precup et al., 2000) where the policy (actor) generates actions and the Q/value function (critic) evaluates the policy’s performance.
Revised: Precup et al., 2000) where the policy (actor) generates actions and the Q/value function (critic) evaluates the policy’s performance.

Original: Under this setting, off-policy RL uses transitions sampled from a replay buffer to perform Q function updates, yielding a new policy π .
Revised: Under this setting, off-policy RL uses transitions sampled from a replay buffer to perform Q function updates, yielding a new policy π .

Original: Then, a ﬁnite-length trajectory under π is added to the buffer, and the process repeats.
Revised: Then, a ﬁnite-length trajectory under π is added to the buffer, and the process repeats.

Original: Notice that sampling from a replay buffer is an ofﬂine operation and that the growth of replay buffer is an online operation.
Revised: Notice that sampling from a replay buffer is an ofﬂine operation and that the growth of replay buffer is an online operation.

Original: This implies off-policy actor-critic RL lies between ofﬂine RL (Yu et al., 2020; Levine et al., 2020) and onpolicy RL (Schulman et al., 2015; 2017).
Revised: This implies off-policy actor-critic RL lies between ofﬂine RL (Yu et al., 2020; Levine et al., 2020) and onpolicy RL (Schulman et al., 2015; 2017).

Original: From a bias-variance perspective, ofﬂine RL experiences large policy mismatch bias but low sampling variance, while on-policy RL has a low bias but high variance.
Revised: From a bias-variance perspective, ofﬂine RL experiences large policy mismatch bias but low sampling variance, while on-policy RL has a low bias but high variance.

Original: Hence, with a careful choice of the sampling from its replay buffer, off-policy actor-critic
Revised: Hence, with a careful choice of the sampling from its replay buffer, off-policy actor-critic

Original: RL may achieve a better bias-variance trade-off.
Revised: RL may achieve a better bias-variance trade-off.

Original: This is the direction we explore this work.
Revised: This is the direction we explore this work.

Original: To reduce policy mismatch bias, off-policy RL employs importance sampling with the weight given by the probability ratio of the current to behavior policy (the policy that samples the trajectories)
Revised: To reduce policy mismatch bias, off-policy RL employs importance sampling with the weight given by the probability ratio of the current to behavior policy (the policy that samples the trajectories)

Original: (Precup et al., 2000; Xie et al., 2019;
Revised: (Precup et al., 2000; Xie et al., 2019;

Original: Schmitt et al., 2020).
Revised: Schmitt et al., 2020).

Original: However, because the behavior policy is usually not given in practice, one can either estimate the probability ratio from the data (Lazic et al., 2020; Yang et al., 2020; Sinha et al., 2020) or use other reasonable quantities, such as the Bellman error (Schaul et al., 2016), as the sampling weight.
Revised: However, because the behavior policy is usually not given in practice, one can either estimate the probability ratio from the data (Lazic et al., 2020; Yang et al., 2020; Sinha et al., 2020) or use other reasonable quantities, such as the Bellman error (Schaul et al., 2016), as the sampling weight.

Original: Even using a naive uniform sampling from the replay buffer, some off-policy actor-critic algorithms can achieve a nontrivial performance (Haarnoja et al., 2018; Fujimoto et al., 2018).
Revised: Even using a naive uniform sampling from the replay buffer, some off-policy actor-critic algorithms can achieve a nontrivial performance (Haarnoja et al., 2018; Fujimoto et al., 2018).

Original: These observations suggest we need to better understand the success of off-policy actor-critic algorithms, especially in practical situations where a ﬁxed behavior policy is unavailable.
Revised: These observations suggest we need to better understand the success of off-policy actor-critic algorithms, especially in practical situations where a ﬁxed behavior policy is unavailable.

Original: Our contributions are as follows.
Revised: Our contributions are as follows.

Original: To understand the actor-critic setting without a ﬁxed behavior policy, we construct a non-stationary policy that generates the averaged occupancy measure.
Revised: To understand the actor-critic setting without a ﬁxed behavior policy, we construct a non-stationary policy that generates the averaged occupancy measure.

Original: We use this policy as a reference and show the policy evaluation error in an off-policy actor-critic setting decomposes into the Bellman error, the policy mismatch bias, and the variance from sampling.
Revised: We use this policy as a reference and show the policy evaluation error in an off-policy actor-critic setting decomposes into the Bellman error, the policy mismatch bias, and the variance from sampling.

Original: Since supervised learning during the Q function update only controls the Bellman error, we need careful sampling strategies to mitigate bias and variance.
Revised: Since supervised learning during the Q function update only controls the Bellman error, we need careful sampling strategies to mitigate bias and variance.

Original: We show that the 1/age weighting or its variants like the Emphasizing Recent Experience (ERE) strategy (Wang & Ross, 2019) are preferable because both their biases and variances are smaller than that of uniform weighting.
Revised: We show that the 1/age weighting or its variants like the Emphasizing Recent Experience (ERE) strategy (Wang & Ross, 2019) are preferable because both their biases and variances are smaller than that of uniform weighting.

Original: To ensure the applicability of our explanation to practical off-policy actor-critic algorithms, we adopt weak but veriﬁable assumptions such as Lipschitz Q functions, bounded rewards, and a bounded action space.
Revised: To ensure the applicability of our explanation to practical off-policy actor-critic algorithms, we adopt weak but veriﬁable assumptions such as Lipschitz Q functions, bounded rewards, and a bounded action space.

Original: We avoid strong assumptions such as a ﬁxed well-explored behavior policy, concentration coefﬁcients, bounded probability ratios (e.g., ratios of current to behavior policy), and tabular or linear function approximation.
Revised: We avoid strong assumptions such as a ﬁxed well-explored behavior policy, concentration coefﬁcients, bounded probability ratios (e.g., ratios of current to behavior policy), and tabular or linear function approximation.

Original: Hence our analysis is more applicable to practical settings.
Revised: Hence our analysis is more applicable to practical settings.

Original: In addition, we explain the success of the ERE strategy (Wang & Ross, 2019) from a bias-variance perspective and show that  ERE is essentially equivalent to 1/age weighting. Our simulations also support this conclusion.
Revised: In addition, our analysis suggests that 1/age-based samplings (e.g., ERE, ERE apx, 1/age) have advantages in bias and variance. Our experiments verify that SAC (Haarnoja et al., 2018) with 1/age-based samplings outperform the prior work.

Original: Our results thus provide theoretical foundations for practical off-policy actor-critic RL algorithms and allow us to reduce the algorithm to a simpler form.
Revised: Thus, our results not only provide theoretical foundations for practical off-policy actor-critic RL algorithms but also achieve better performances.

Original:  P RELIMINARIES 2. R EINFORCEMENT LEARNING
Revised:  P RELIMINARIES 2. R EINFORCEMENT LEARNING

Original: Consider an inﬁnite-horizon Markov Decision Process (MDP) (cid:104)S , A , T, r, γ (cid:105) , where S , A are ﬁnite-dimensional continuous state and action spaces, r ( s, a ) is a deterministic reward function, γ ∈ (0 , 1) is the discount factor, and T ( s (cid:48) | s, a ) is the state transition density; i.e., the density of the next state s (cid:48) given the current state and action ( s, a ) .
Revised: Consider an inﬁnite-horizon Markov Decision Process (MDP) (cid:104)S , A , T, r, γ (cid:105) , where S , A are ﬁnite-dimensional continuous state and action spaces, r ( s, a ) is a deterministic reward function, γ ∈ (0 , 1) is the discount factor, and T ( s (cid:48) | s, a ) is the state transition density; i.e., the density of the next state s (cid:48) given the current state and action ( s, a ) .

Original: Given an initial state distribution ρ 0 , the objective of RL is to ﬁnd a policy π that maximizes the γ -discounted cumulative reward when the actions along the trajectory follow π :
Revised: Given an initial state distribution ρ 0 , the objective of RL is to ﬁnd a policy π that maximizes the γ -discounted cumulative reward when the actions along the trajectory follow π :

Original: [Equation]
Revised: [Equation]

Original: Let ρ i ( s | ρ 0 , π, T ) be the state distribution under π at trajectory step i .
Revised: Let ρ i ( s | ρ 0 , π, T ) be the state distribution under π at trajectory step i .

Original: Deﬁne the normalized state occupancy measure by
Revised: Deﬁne the normalized state occupancy measure by

Original: [Equation]
Revised: [Equation]

Original: Ideally, the maximization in (1) is achieved using a parameterized policy π θ and policy gradient updates with (Sutton et al., 1999; Silver et al., 2014):
Revised: Ideally, the maximization in (1) is achieved using a parameterized policy π θ and policy gradient updates with (Sutton et al., 1999; Silver et al., 2014):

Original: [Equation]
Revised: [Equation]

Original: Here ρ π θ ρ [Equation] s, a ) =
Revised: Here ρ π θ ρ 0 ( s, a ) =

Original: ρ π θ ρ 0 ( s ) π θ ( a | s ) is given in (2), and Q π θ is the Q function under policy π θ :
Revised: ρ π θ ρ 0 ( s ) π θ ( a | s ) is given in (2), and Q π θ is the Q function under policy π θ :

Original: [Equation]
Revised: [Equation]

Original: Off-policy RL estimates Q π by approximating the solution of the Bellman ﬁxed-point equation:
Revised: Off-policy RL estimates Q π by approximating the solution of the Bellman ﬁxed-point equation:

Original: [Equation]
Revised: [Equation]

Original: It is well-known that Q π is the unique ﬁxed point of the Bellman operator B π .
Revised: It is well-known that Q π is the unique ﬁxed point of the Bellman operator B π .

Original: Hence if ( B π ˆ Q )( s, a ) ≈ ˆ Q ( s, a ) , then ˆ Q may be a “good” estimate of Q π .
Revised: Hence if ( B π ˆ Q )( s, a ) ≈ ˆ Q ( s, a ) , then ˆ Q may be a “good” estimate of Q π .

Original: In the next subsection, we will see that an off-policy actor-critic algorithm encourages this to hold for the replay buffer.
Revised: In the next subsection, we will see that an off-policy actor-critic algorithm encourages this to hold for the replay buffer.

Original: 2. O FF - POLICY ACTOR - CRITIC ALGORITHM
Revised: 2. O FF - POLICY ACTOR - CRITIC ALGORITHM

Original: We study an off-policy actor-critic algorithm of the form shown in Alg. 1.
Revised: We study an off-policy actor-critic algorithm of the form shown in Alg. 1.

Original: In line 2, for episode

Original: Algorithm 1 Off-policy Actor-critic Algorithm

Original: Require:

Original: Parameterized policy π e = π θ e .

Original: Learning rate α .

Original: 1: for episode e = 1 , 2 , ... do 2:

Original: Sample

Original: a

Original: length

Original: - L

Original: trajectory { ( s ei , a ei ) }

Original: L

Original: [-− i =
-]
Revised: [-− i =
-]

Original: under initial distribution ρ 0 ( s ) , policy π e ( a | s ) and state transition T ( ·| s, a ) .

Original: Train a

Original: [Table] [Table] [Table] index e , π e samples one trajectory of length L.
Revised: In line 2, for episode index e , π e samples one trajectory of length L.

Original: The transitions in this trajectory are then added to the replay buffer.
Revised: The transitions in this trajectory are then added to the replay buffer.

Original: Since the policies for different episodes are distinct, the collection of transitions in the replay buffer are generally inconsistent with the current policy.
Revised: Since the policies for different episodes are distinct, the collection of transitions in the replay buffer are generally inconsistent with the current policy.

Revised: Notice the trajectory is simply collected by π e , not a perturbed version of π e (e.g., gaussian corrupted version of pi e ).

Original: Line 3 is a supervised learning that minimizes the Bellman error of ˆ Q , making ( B π e ˆ Q )( s, a ) ≈ ˆ Q ( s, a ) for ( s, a ) in the replay buffer.
Revised: Line 3 is a supervised learning that minimizes the Bellman error of ˆ Q using gradient descents, making ( B π e ˆ Q )( s, a ) ≈ ˆ Q ( s, a ) for ( s, a ) in the replay buffer.

Original: When this holds, we will prove that the “distance” between ˆ Q and Q π e become smaller over the replay buffer.
Revised: When this holds, we will prove that the “distance” between ˆ Q and Q π e become smaller over the replay buffer.

Original: This is a crucial step for line 4 to be truly useful.
Revised: This is a crucial step for line 4 to be truly useful.

Original: In line 4, Q π e is replaced by ˆ Q , and the policy is updated accordingly.
Revised: In line 4, Q π e is replaced by ˆ Q , and the policy is updated accordingly.

Original: In practice, line 3 is replaced bygradient-descent-based mini-batch updates (Fujimoto et al., 2018; Haarnoja et al., 2018). For example, φ ← φ − α ∇ J ( ˆ Q φ ) with φ and J ( ˆ Q φ ) being the parameter and the Bellman error of ˆ Q, respectively.
Revised: In practice, line 3 is replaced by mini-batch updates (Fujimoto et al., 2018; Haarnoja et al., 2018) where the summation of J ( ˆ Q φ ) is approximated by a sum over the mini-batches.

Original: In section 5.2, we show that a uniform-weighted mini-batch sampling biases learning towards older samples; this motivates the need for a countermeasure. 
Revised: In section 5.2, we show that a uniform-weighted mini-batch sampling biases learning towards older samples; this motivates the need for a countermeasure. Evaluate

Revised: Bellman ji ji

Original: Note that Alg. 1 seems to be inconsistent in the horizon because the Q function, Eq.
Revised: Note that Alg. 1 seems to be inconsistent in the horizon because the Q function, Eq.

Original: (4), is deﬁned in the inﬁnite horizon but the trajectories are ﬁnite-length.
Revised: (4), is deﬁned in the inﬁnite horizon but the trajectories are ﬁnite-length.

Original: In Corollary 1, we will address this inconsistency by approximating the Q function using ﬁnite-length trajectories.
Revised: In Corollary 1, we will address this inconsistency by approximating the Q function using ﬁnite-length trajectories.

Original: 2. T HE CONSTRUCTION OF OUR BEHAVIOR POLICY
Revised: 2. T HE CONSTRUCTION OF OUR BEHAVIOR POLICY

Original: Although Alg. 1 doesn’t have a ﬁxed behavior policy, in Lemma 1, we construct a non-stationary policy that generates the averaged occupancy measure at every step.
Revised: Although Alg. 1 doesn’t have a ﬁxed behavior policy, in Lemma 1, we construct a non-stationary policy that generates the averaged occupancy measure at every step.

Original: This describes the averagedover-episode behavior at every trajectory step i of the historical trajectory { ( s ei , a ei ) } L − 1 ,N i,e =0 , 1 .
Revised: This describes the averagedover-episode behavior at every trajectory step i of the historical trajectory { ( s ei , a ei ) } L − 1 ,N i,e =0 , 1 .

Original: We hence deﬁne it to be our behavior policy and use it as a reference when analyzing Alg. 1.
Revised: We hence deﬁne it to be our behavior policy and use it as a reference when analyzing Alg. 1.

Original: The construction is as follows.
Revised: The construction is as follows.

Original: Denote the state distribution at trajectory step i in episode e as ρ ei ( s ) =
Revised: Denote the state distribution at trajectory step i in episode e as ρ ei ( s ) =

Original: ρ i ( s | ρ 0 , π e , T ) .
Revised: ρ i ( s | ρ 0 , π e , T ) .

Original: By Eq.
Revised: By Eq.

Original: ρ π e ρ 0 is the state occupancy measure generated by ( ρ 0 , π e , T ) .
Revised: ρ π e ρ 0 is the state occupancy measure generated by ( ρ 0 , π e , T ) .

Original: Intuitively, ρ π e ρ 0 describes the discounted state distribution starting at trajectory step 0 in episode e .
Revised: Intuitively, ρ π e ρ 0 describes the discounted state distribution starting at trajectory step 0 in episode e .

Original: More generally, deﬁne ρ π e ρ ei as the state occupancy measure generated by ( ρ ei , π e , T ) .
Revised: More generally, deﬁne ρ π e ρ ei as the state occupancy measure generated by ( ρ ei , π e , T ) .

Original: Then, ρ π e ρ ei describes the discounted state distribution starting at trajectory step i in episode e .
Revised: Then, ρ π e ρ ei describes the discounted state distribution starting at trajectory step i in episode e .

Original: [Equation]
Revised: [Equation]

Revised: Behavior policy through averaging.

Original: Since Alg. 1 considers trajectories from all episodes, the average-over-episodes distribution 
Revised: Since Alg. 1 considers trajectories from all episodes, the average-over-episodes distribution (cid:37) ( i ) =

Original: N [Equation] [Equation] e ρ π e ρ ei will be of interest.
Revised: N − 1 (cid:80) e ρ π e ρ ei will be of interest.

Original: It describes the averaged-discounted state distribution at step i over all episodes. Behavior policy through averaging. Let ρ oi be the average-over-episodes of { ρ π e ρ ei } e ; i.e., ρ oi is the averaged occupancy measure if initialized at step i .
Revised: Namely, (cid:37) ( i ) is the average of all occupancy measures starting at step i .

Original: To describe the average-over-episodes behavior at step i , we want a policy that generates ρ oi , and Eq.
Revised: To describe the average-over-episodes behavior at step i , we want a policy that generates (cid:37) ( i ) , and Eq.

Original: (5) helps construct such a policy.
Revised: (5) helps construct such a policy.

Original: Concretely, let ρ i , π Di ( a | s ) be the averaged state distribution and the averaged policy at step i , respectively.
Revised: Concretely, let ρ i , π D i ( a | s ) be the averaged state distribution and the averaged policy at step i , respectively.

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 shows π Di in Eq.
Revised: Lemma 1 shows π Di in Eq.

Original: (6) is a notion of behavior policy in the sense that ( ρ i , π Di , T ) generates ρ oi ; i.e., π Di generates the averaged occupancy measures when the initial state follows ρ i .
Revised: (6) is a notion of behavior policy in the sense that ( ρ i , π Di , T ) generates (cid:37) ( i ) ; i.e., π Di generates the averaged occupancy measures when the initial state follows ρ i .

Original: Since π Di describes the averaged discounted behavior starting at step i , we deﬁne it to be our behavior policy.
Revised: Since π Di describes the averaged discounted behavior starting at step i , we deﬁne it to be our behavior policy.

Original: This is a key to analyzing the policy evaluation error in Alg. 1.
Revised: This is a key to analyzing the policy evaluation error in Alg. 1.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let ρ π Di
Revised: Let ρ π Di

Original: ρ i ( s ) be the normalized state occupancy measure generated by ( ρ i , π Di , T ) .
Revised: ρ i ( s ) be the normalized state occupancy measure generated by ( ρ i , π Di , T ) .

Original: Then ρ oi ( s ) =
Revised: Then (cid:37) ( i ) ( s ) =

Original: ρ π Di ρ i ( s ) a.e. and hence from Eq.
Revised: ρ π Di ρ i ( s ) a.e. and hence from Eq.

Original: (6) , N − 1 (cid:80) N e =

Revised: (6) , N − 1 (cid:80) Ne =

Original: ρ π e ρ e i ( s ) π e ( a | s ) =
Revised: ρ π e ρ ei ( s ) π e ( a | s ) =

Original: ρ π Di ρ i ( s ) π Di ( a | s ) a.e.
Revised: ρ π Di ρ i ( s ) π Di ( a | s ) a.e.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: There are two main approaches to off-policy RL: importance sampling (IS) (Tokdar & Kass, 2010) and regression-based approach.
Revised: There are two main approaches to off-policy RL: importance sampling (IS) (Tokdar & Kass, 2010) and regression-based approach.

Original: IS has a low bias but high variance, while the opposite holds for the regression-based approach.
Revised: IS has a low bias but high variance, while the opposite holds for the regression-based approach.

Original: Prior work also suggests that combining IS and the regression-based yields robust results (Dud´ık et al., 2011; Jiang & Li, 2016; Thomas & Brunskill, 2016;

Original: Kallus &

Original: Uehara, 2020).

Original: Below, we brieﬂy review these techniques.
Revised: Below, we brieﬂy review these techniques.

Original: Importance Sampling.
Revised: Importance Sampling.

Original: Standard IS uses behavior policy to form an importance weight from the probability ratio of the current to the behavior policy.
Revised: Standard IS uses behavior policy to form an importance weight from the probability ratio of the current to the behavior policy.

Original: For a fully accessible behavior policy, examples of this approach include: the naive IS weight (Precup et al., 2000), importance weight clipping (Schmitt et al., 2020) and the marginalized importance weight (Xie et al., 2019; Yin & Wang, 2020;
Revised: For a fully accessible behavior policy, examples of this approach include: the naive IS weight (Precup et al., 2000), importance weight clipping (Schmitt et al., 2020) and the marginalized importance weight (Xie et al., 2019; Yin & Wang, 2020;

Original: Yin et al., 2021).
Revised: Yin et al., 2021).

Original: Alternatively, one can use the density ratio of the occupancy measures of the current to the behavior policies (Liu et al., 2018).
Revised: Alternatively, one can use the density ratio of the occupancy measures of the current to the behavior policies (Liu et al., 2018).

Original: This is estimated using a maximum entropy approach (Lazic et al., 2020), a Lagrangian approach (Yang et al., 2020), or a variational approach (Sinha et al., 2020).
Revised: This is estimated using a maximum entropy approach (Lazic et al., 2020), a Lagrangian approach (Yang et al., 2020), or a variational approach (Sinha et al., 2020).

Original: A distinct approach emphasizes experience without considering probability ratios.
Revised: A distinct approach emphasizes experience without considering probability ratios.

Original: Examples include emphasizing samples with a higher TD error (Schaul et al., 2016; Horgan et al., 2018), emphasizing recent experience (Wang & Ross, 2019) or updating the policy towards the past and discarding distant experience (Novati & Koumoutsakos, 2019).
Revised: Examples include emphasizing samples with a higher TD error (Schaul et al., 2016; Horgan et al., 2018), emphasizing recent experience (Wang & Ross, 2019) or updating the policy towards the past and discarding distant experience (Novati & Koumoutsakos, 2019).

Revised: It is also shown that IS on replay buffer is equivalent to weighting on loss functions (Fujimoto et al., 2020).

Original: Regression-based.
Revised: Regression-based.

Original: A regression-based approach can achieve strong experimental results using properpolicy exploration andgood function approximation (Fujimoto et al., 2018; Haarnoja et al., 2018).
Revised: A regression-based approach can achieve strong experimental results using proper exploration and function approximation (Fujimoto et al., 2018; Haarnoja et al., 2018).

Original: It also admits strong theoretical results such as generalization error using a concentration coefﬁcient (Le et al., 2019), policy evaluation error using bounded probability ratios (Agarwal et al., 2019)[Chap 5.1], minimax optimal bound under linear function approximation Duan et al.
Revised: It also admits strong theoretical results such as generalization error using a concentration coefﬁcient (Le et al., 2019), policy evaluation error using bounded probability ratios (Agarwal et al., 2019)[Chap 5.1], minimax optimal bound under linear function approximation Duan et al.

Original: (2020), conﬁdence bounds constructed by kernel Bellman loss Feng et al.
Revised: (2020), conﬁdence bounds constructed by kernel Bellman loss Feng et al.

Original: (2020), and sharp sample complexity in asynchronous setting (Li et al., 2020).

Original: However, these settings require a ﬁxed behavior policy and bounded probability ratios (i.e., the behavior policy should be sufﬁciently exploratory). Both requirements rarely hold in practice.
Revised: However, these settings require a ﬁxed behavior policy and bounded probability ratios, which rarely hold in practice.

Original: To avoid this issue, we construct a non-stationary behavior policythat generates the occupancy measure and use it to analyze the policy evaluation error.
Revised: We hence construct a non-stationary behavior policy to avoid this issue. Combined.

Original: This allows us to avoid assumptions and  to work with continuous state and action spaces.
Revised: Prior work also suggests that combining IS and the regression-based yields robust results (Dud ´ ık et al., 2011; Jiang & Li, 2016; Thomas & Brunskill, 2016; Kallus & Uehara, 2020). It is also common to consider some reﬁned contractions to improve stability. Examples include nstep Q-learning (Hessel et al., 2018), Retrace (Munos et al., 2016), Peng’s Q (Kozuno et al., 2021), and other correction techniques (Harutyunyan et al., 2016; Tang et al., 2020; Rowland et al., 2020).

Original:  P OLICY EVALUATION ERROR OF OFF - POLICY ACTOR - CRITIC ALGORITHMS
Revised:  P OLICY EVALUATION ERROR OF OFF - POLICY ACTOR - CRITIC ALGORITHMS

Original: Let Q ∗ , Q π N be the Q function of the optimal policy and π N , respectively.
Revised: Let Q ∗ , Q π N be the Q function of the optimal policy and π N , respectively.

Original: Let ˆ Q be the estimated
Revised: Let ˆ Q be the estimated

Original: Q function.
Revised: Q function.

Original: The performance error | ˆ Q − Q ∗ | decomposes into | ˆ Q − Q π N | + | Q π N − Q ∗ | .
Revised: The performance error | ˆ Q − Q ∗ | decomposes into | ˆ Q − Q π N | + | Q π N − Q ∗ | .

Original: The ﬁrst term | ˆ Q − Q π N | is the policy evaluation error and is the focus in the off-policy evaluation literature (Duan et al., 2020).
Revised: The ﬁrst term | ˆ Q − Q π N | is the policy evaluation error and is the focus in the off-policy evaluation literature (Duan et al., 2020).

Original: The second term is the policy’s optimality gap and to bound this term currently requires strong assumptions such as tabular or linear MDPs (Jin et al., 2018; 2020).
Revised: The second term is the policy’s optimality gap and to bound this term currently requires strong assumptions such as tabular or linear MDPs (Jin et al., 2018; 2020).

Original: In an off-policy actor-critic setting, the policy evaluation error has not been analyzed adequately since most analysis requires a ﬁxed behavior policy.
Revised: In an off-policy actor-critic setting, the policy evaluation error has not been analyzed adequately since most analysis requires a ﬁxed behavior policy.

Original: This is the focus of our analysis.
Revised: This is the focus of our analysis.

Original: Suppose we are given the trajectories sampled in the past episodes (the replay buffer).
Revised: Suppose we are given the trajectories sampled in the past episodes (the replay buffer).

Original: We analyze the policy evaluation error over the expected distribution of transitions.
Revised: We analyze the policy evaluation error over the expected distribution of transitions.

Original: We express this error in terms of the Bellman error of ˆ Q , the bias term in 1-Wasserstein distance between the policies ( π N , π D i ) , and the variance term in the number of trajectories N .
Revised: We express this error in terms of the Bellman error of ˆ Q , the bias term in 1-Wasserstein distance between the policies ( π N , π D i ) , and the variance term in the number of trajectories N .

Original: Note π D
Revised: Note π D

Original: i is the behavior policy at trajectory step i deﬁned in Eq.
Revised: i is the behavior policy at trajectory step i deﬁned in Eq.

Original: The use of 1-Wasserstein distance (Villani, 2008) makes the results applicable to both stochastic and deterministic policies.
Revised: The use of 1-Wasserstein distance (Villani, 2008) makes the results applicable to both stochastic and deterministic policies.

Original: Since supervised learning only makes the Bellman error small, we need good sampling strategies to mitigate the bias and variance terms.
Revised: Since supervised learning only makes the Bellman error small, we need good sampling strategies to mitigate the bias and variance terms.

Original: We hence investigate sampling techniques from a bias-variance perspective in the next section.
Revised: We hence investigate sampling techniques from a bias-variance perspective in the next section.

Original: 4. P ROBLEM SETUP
Revised: 4. P ROBLEM SETUP

Original: Notation.
Revised: Notation.

Original: In episode e, a length- L trajectory { ( s ei , a ei ) } L − 1 i =0 following policy π e is sampled (Alg. 1, line 2).
Revised: In episode e, a length- L trajectory { ( s ei , a ei ) } L − 1 i =0 following policy π e is sampled (Alg. 1, line 2).

Original: Then, ˆ Q is ﬁtted over the replay buffer (line 3).
Revised: Then, ˆ Q is ﬁtted over the replay buffer (line 3).

Original: Because the error of ˆ Q at step i depends on the states sampled at steps i, i + 1 , . .
Revised: Because the error of ˆ Q at step i depends on the states sampled at steps i, i + 1 , . .

Original: . , the importance of these samples ( s, a ) depends on the trajectory step i .
Revised: . , the importance of these samples ( s, a ) depends on the trajectory step i .

Original: Also, due to the discount factor, the importance of step j > i is discounted by γ j − i relative to step i .
Revised: Also, due to the discount factor, the importance of step j > i is discounted by γ j − i relative to step i .

Original: Hence, we will use a Bellman error and a policy mismatch error that reﬂects the dependency on the trajectory step and the discount factor.
Revised: Hence, we will use a Bellman error and a policy mismatch error that reﬂects the dependency on the trajectory step and the discount factor.

Original: For f : S ×
Revised: For f : S ×

Original: A → R and g : S → R , deﬁne an averaging-discounted operator over the replay buffer in N episodes:
Revised: A → R and g : S → R , deﬁne an averaging-discounted operator over the replay buffer in N episodes:

Original: [Equation]
Revised: [Equation]

Original: Using ˜ E Li , the Bellman error of ˆ Q and the distance between the policies ( π N , π Di ) on the replay buffer are written as
Revised: Using ˜ E Li , the Bellman error of ˆ Q and the distance between the policies ( π N , π Di ) on the replay buffer are written as

Original: [Equation]
Revised: [Equation]

Original: W 1 ( π N || π Di )( s ) =
Revised: W 1 ( π N || π Di )( s ) =

Original: W 1 ( π [Equation]
Revised: W 1 ( π N ( ·| s ) ||

Original: s ) || π Di ( ·| s )) is the 1-Wasserstein distance between two policies at state s , which can be viewed as a function of ( s, a ) .
Revised: π Di ( ·| s )) is the 1-Wasserstein distance between two policies at state s , which can be viewed as a function of ( s, a ) .

Original: Both the Bellman error (cid:15) i,L ˆ Q and the policy mismatch error
Revised: Both the Bellman error (cid:15) i,L ˆ Q and the policy mismatch error

Original: W
Revised: W

Original: i,Ldepend on trajectory step i and are both discounted by γ .
Revised: i,Ldepend on trajectory step i and are both discounted by γ .

Original: Assumptions.
Revised: Assumptions.

Original: We now relate the Bellman error and the policy mismatch error deﬁned in Eq.
Revised: We now relate the Bellman error and the policy mismatch error deﬁned in Eq.

Original: (7) to the policy evaluation error | ˆ Q − Q π N | .
Revised: (7) to the policy evaluation error | ˆ Q − Q π N | .

Original: To do so requires some assumptions.

Original: First, to control the error of ˆ Q by policy mismatch error in W 1 distance, we assume that for every state, ˆ Q and Q π N are L A -Lipschitz over actions.
Revised: First, to control the error of ˆ Q by policy mismatch error in W 1 distance, we assume that for every state, ˆ Q and Q π N are L A -Lipschitz over actions.

Original: We provide reasoning for this assumption in the later discussion.
Revised: We provide reasoning for this assumption in the later discussion.

Original: Next, observe that both quantities in Eq.
Revised: Next, observe that both quantities in Eq.

Original: (7) are random with sources of randomness from initial states, policies at different episodes, and the state transitions.
Revised: (7) are random with sources of randomness from initial states, policies at different episodes, and the state transitions.

Original: To control this randomness, we need assumptions on the Q functions and apply a concentration inequality.
Revised: To control this randomness, we need assumptions on the Q functions and apply a concentration inequality.

Original: Because Alg. 1 only samples one trajectory in each episode, higher-order quantities (e.g., variance) is unavailable.
Revised: Because Alg. 1 only samples one trajectory in each episode, higherorder quantities (e.g., variance) is unavailable.

Original: This motivates us to use ﬁrst-order quantities (e.g., a uniform bound on the Q functions) and apply Hoeffding’s inequality.
Revised: This motivates us to use ﬁrst-order quantities (e.g., a uniform bound on the Q functions) and apply Hoeffding’s inequality.

Original: Hence, we assume that ˆ Q ( s, a ) and Q π N ( s, a ) are bounded in the interval [0 , r max / (1 − γ )] and that the action space is bounded with the diameter diam A .
Revised: Hence, we assume that ˆ Q ( s, a ) and Q π N ( s, a ) are bounded in the interval [0 , r max / (1 − γ )] and that the action space is bounded with the diameter diam A .

Original: A justiﬁcation of these assumptions is provided in Appendix A.2.
Revised: A justiﬁcation of these assumptions is provided in Appendix A.2.

Original: 4. P OLICY EVALUATION ERROR
Revised: 4. P OLICY EVALUATION ERROR

Original: Observe that the Q function, Eq.
Revised: Observe that the Q function, Eq.

Original: (4), is deﬁned on inﬁnite-length trajectories while the errors, Eq.
Revised: (4), is deﬁned on inﬁnite-length trajectories while the errors, Eq.

Original: (7), are evaluated on length- L trajectories.
Revised: (7), are evaluated on length- L trajectories.

Original: Discounting makes it possible to approximate the Q functions using ﬁnite-length L .
Revised: Discounting makes it possible to approximate the Q functions using ﬁnite-length L .

Original: To approximate the Q function at trajectory step i up to a constant error, we need samples at least to step i + Ω((1 − γ ) − 1 ) .
Revised: To approximate the Q function at trajectory step i up to a constant error, we need samples at least to step i + Ω((1 − γ ) − 1 ) .

Original: Hence, we ﬁrst prove the main theorem with i =and L = ∞ .
Revised: Hence, we ﬁrst prove the main theorem with i =and L = ∞ .

Original: Then, we generalize the result to i ≥ 0 and ﬁnite L in Corollary 1.
Revised: Then, we generalize the result to i ≥ 0 and ﬁnite L in Corollary 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let N be the number of episodes.
Revised: Let N be the number of episodes.

Original: For f : S ×
Revised: For f : S ×

Original: A → R , deﬁne the operator ˜ E ∞ 

Revised: A → R , deﬁne the operator ˜ E ∞

Original: f (cid:44)
Revised: f (cid:44)

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: i f

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: s

Original: [Equation]
Revised: [Equation]

Original: f

Original: [Equation]
Revised: [Equation]

Original: s ei , a ei ) .
Revised: ei ) .

Original: Denote the policy mismatch error and the Bellman error asW
Revised: Denote the policy mismatch error and the Bellman error as

Original: [Equation]
Revised: [Equation]

Original: and (cid:15)

Original: [Equation]

Original: [Equation]

Original: [Equation] − [Equation] ˆ Q )( · , · ) | , respectively.
Revised: , respectively.

Original: Assume
Revised: Assume

Original: (1) For each s ∈ S , ˆ Q ( s, a ) and Q π N ( s, a ) are L A -Lipschitz in a. (2) ˆ Q ( s, a ) and Q π N ( s, a ) are bounded and take values in [0 , r max / (1 − γ )] .
Revised: (1) For each s ∈ S , ˆ Q ( s, a ) and Q π N ( s, a ) are L A -Lipschitz in a. (2) ˆ Q ( s, a ) and Q π N ( s, a ) are bounded and take values in [0 , r max / (1 − γ )] .

Original: The action space is bounded with diameter diam A < ∞ .
Revised: The action space is bounded with diameter diam A < ∞ .

Original: Then, with probability at least 1 − δ ,
Revised: Then, with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: ≤ [Equation] Theorem 1 expresses the error of ˆ Q as the sum of Bellman error, bias, and variance terms.
Revised: Theorem 1 expresses the error of ˆ Q as the sum of Bellman error, bias, and variance terms.

Original: To be more speciﬁc, the ﬁrst two terms are understood as the “variance from sampling” because these decrease in the number of episodes N .
Revised: To be more speciﬁc, the ﬁrst two terms are understood as the “variance from sampling” because these decrease in the number of episodes N .

Original: On the other hand, W 0 , ∞is the “policy mismatch bias” w.r.t.
Revised: On the other hand, W 0 , ∞is the “policy mismatch bias” w.r.t.

Original: π N .
Revised: π N .

Original: Because the behavior policy π D 0 is a mixture of the historical policies { π e } Ne =1 , Eq.
Revised: Because the behavior policy π D 0 is a mixture of the historical policies { π e } Ne =1 , Eq.

Original: (6), we expect it to increase in N until π N begins to converge.
Revised: (6), we expect it to increase in N until π N begins to converge.

Original: Theorem 1 only indicates the difference between ˆ Q and Q π N at i = 0 and inﬁnite-length trajectories.
Revised: Theorem 1 only indicates the difference between ˆ Q and Q π N at i = 0 and inﬁnite-length trajectories.

Original: We can generalize it to i ≥ 0 and ﬁnite-length trajectories as follows.
Revised: We can generalize it to i ≥ 0 and ﬁnite-length trajectories as follows.

Original: Recall from Eq.
Revised: Recall from Eq.

Original: (6) and
Revised: (6) and

Original: Lemma 1, the average state distribution at the i -th step, ρ i , and the behavior policy at the i -th step, π Di , generate the average state occupancy measure, i.e., ρ π Di ρ i = N − 1 (cid:80) Ne =

Revised: Lemma 1, the average state distribution at the i -th step, ρ i , and the behavior policy at the i -th step, π Di , generate the average state occupancy measure, i.e., ρ π Di ρ i = N − 1 (cid:80) Ne =

Original: ρ π e ρ ei a.e.
Revised: ρ π e ρ ei a.e.

Original: Therefore, by restricting attention to the states sampled at time steps i, i + 1 , . . .
Revised: Therefore, by restricting attention to the states sampled at time steps i, i + 1 , . . .

Original: the “initial state distribution” and the behavior policy become ρ i and π Di , which generalizes Theorem 1 from i = 

Revised: the “initial state distribution” and the behavior policy become ρ i and π Di , which generalizes Theorem 1 from i =

Original: to i ≥ 0 .
Revised: to i ≥ 0 .

Original: In addition, due to γ -discounting, we may use length- L trajectories to approximate inﬁnite-length ones, provided that L ≥ i + Ω((1 − γ ) − 1 ) .
Revised: In addition, due to γ -discounting, we may use length- L trajectories to approximate inﬁnite-length ones, provided that L ≥ i + Ω((1 − γ ) − 1 ) .

Original: These observations lead to the following corollary.
Revised: These observations lead to the following corollary.

Original: Corollary 1.
Revised: Corollary 1.

Original: Fix assumptions (1) (2) (3) of Theorem 1.
Revised: Fix assumptions (1) (2) (3) of Theorem 1.

Original: Rewrite the policy mismatch error and the
Revised: Rewrite the policy mismatch error and the

Original: Bellman error as W i,L= ˜ E
Revised: Bellman error as W i,L= ˜ E

Original: Li W 1 ( π N || π Di )( · ) and (cid:15) i,L ˆ Q = ˜
Revised: Li W 1 ( π N || π Di )( · ) and (cid:15) i,L ˆ Q = ˜

Original: E Li | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively.
Revised: E Li | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively.

Original: Note ˜
Revised: Note ˜

Original: E Li is deﬁned in Eq.
Revised: E Li is deﬁned in Eq.

Original: Then, with probability at least 1 − δ ,
Revised: Then, with probability at least 1 − δ ,

Original: E ( s i ,a i ) ∼ ρ i ( s ) π Di ( a | s ) (cid:12)(cid:12)(cid:12) ˆ Q

Original: [Equation]
Revised: [Equation]

Original: s

Original: [Equation]

Original: −

Original: [Equation] ( s [Equation] ) (cid:12)(cid:12)(cid:12) ≤ r max
Revised: r max

Original: [Equation]
Revised: [Equation]

Original: Moreover, if i ≤ L − log (cid:15) log γ with 0 < (cid:15) < 1 and the constants are normalized as L A =
Revised: Moreover, if i ≤ L − log (cid:15) log γ with 0 < (cid:15) < 1 and the constants are normalized as L A =

Original: c/ (1 − γ ) , (cid:15) i,L ˆ Q = ξ i,L ˆ Q / (1 − γ ) , then, with probability at least 1 − δ ,
Revised: c/ (1 − γ ) , (cid:15) i,L ˆ Q = ξ i,L ˆ Q / (1 − γ ) , then, with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: where ˜ O ( · ) is a variant of O ( · ) that ignores logarithmic factors.
Revised: where ˜ O ( · ) is a variant of O ( · ) that ignores logarithmic factors.

Original: Note that if i = 0 and L = ∞ , Corollary 1 is identical to Theorem 1.
Revised: Note that if i = 0 and L = ∞ , Corollary 1 is identical to Theorem 1.

Original: Because the Bellman error of ˆ Q and the bias of the policy are both evaluated using the averaging-discounted operator ˜ E Li ,
Revised: Because the Bellman error of ˆ Q and the bias of the policy are both evaluated using the averaging-discounted operator ˜ E Li ,

Original: Corollary 1 implies the difference of ˆ Q and Q π N at trajectory step i mainly depends on states at trajectory steps ≥ i .
Revised: Corollary 1 implies the difference of ˆ Q and Q π N at trajectory step i mainly depends on states at trajectory steps ≥ i .

Original: Since the Q function is a discounted sum of rewards from the current step to the future, the error at step i should mainly depend on the steps ≥ i .
Revised: Since the Q function is a discounted sum of rewards from the current step to the future, the error at step i should mainly depend on the steps ≥ i .

Original: Normalization.
Revised: Normalization.

Original: Although the ﬁrst conclusion in Corollary 1 gives a bound on the error of ˆ Q , the constants may implicitly depend on the expected horizon (1 − γ ) − 1 .
Revised: Although the ﬁrst conclusion in Corollary 1 gives a bound on the error of ˆ Q , the constants may implicitly depend on the expected horizon (1 − γ ) − 1 .

Original: Hence its interpretation requires care.
Revised: Hence its interpretation requires care.

Original: For instance, L A , the Lipschitz constant of the Q functions w.r.t.
Revised: For instance, L A , the Lipschitz constant of the Q functions w.r.t.

Original: actions, is probably the most tricky constant.
Revised: actions, is probably the most tricky constant.

Original: While it is used extensively in the prior work (Luo et al., 2019; Xiao et al., 2019; Ni et al., 2019; Yu et al., 2020), its magnitude is never properly addressed in the literature.
Revised: While it is used extensively in the prior work (Luo et al., 2019; Xiao et al., 2019; Ni et al., 2019; Yu et al., 2020), its magnitude is never properly addressed in the literature.

Original: Intuitively, if a policy π is good enough, it should quickly correct some disturbance on actions.
Revised: Intuitively, if a policy π is good enough, it should quickly correct some disturbance on actions.

Original: In this case, the rewards after the disturbance only differ in a few trajectory steps, so the Lipschitzness of Q π in actions is sublinear in (1 − γ ) − 1 .
Revised: In this case, the rewards after the disturbance only differ in a few trajectory steps, so the Lipschitzness of Q π in actions is sublinear in (1 − γ ) − 1 .

Original: On the other hand, if the policy π fails to correct a disturbance δ , due to error propagation, the error δ propagates to every future step, leading to a linear error dependency to the horizon O ( δH ) .
Revised: On the other hand, if the policy π fails to correct a disturbance δ , due to error propagation, the error δ propagates to every future step, leading to a linear error dependency to the horizon O ( δH ) .

Original: Therefore, the Lipschitzness of Q π in actions can be as large as O ((1 − γ ) − 1 ) .
Revised: Therefore, the Lipschitzness of Q π in actions can be as large as O ((1 − γ ) − 1 ) .

Original: In addition to L A , the Bellman error (cid:15) i,L ˆ Q should scale linearly in (1 − γ ) −because the Q function represents the discounted cumulative reward, Eq.
Revised: In addition to L A , the Bellman error (cid:15) i,L ˆ Q should scale linearly in (1 − γ ) −because the Q function represents the discounted cumulative reward, Eq.

Original: (4), which scales linearly in (1 − γ ) − 1 .
Revised: (4), which scales linearly in (1 − γ ) − 1 .

Original: These observations suggest that (cid:15) i,L ˆ Q and L A in Corollary 1 are either linear in the horizon or lie between sublinear and linear.
Revised: These observations suggest that (cid:15) i,L ˆ Q and L A in Corollary 1 are either linear in the horizon or lie between sublinear and linear.

Original: To better capture the dependency on the horizon, we normalize the constants and get the second conclusion.
Revised: To better capture the dependency on the horizon, we normalize the constants and get the second conclusion.

Original: Interpretation.
Revised: Interpretation.

Original: The second conclusion shows the approximation error from inﬁnite to ﬁnitelength trajectories is bounded by a constant for i ≤ L − log (cid:15) log γ Taylor ≈ L − Ω((1 − γ ) − 1 ) , and will become harder to control for the higher i due to the lack of samples.
Revised: The second conclusion shows the approximation error from inﬁnite to ﬁnitelength trajectories is bounded by a constant for i ≤ L − log (cid:15) log γ Taylor ≈ L − Ω((1 − γ ) − 1 ) , and will become harder to control for the higher i due to the lack of samples.

Original: Besides, the variance from sampling ˜ O

Original: (1 / √ N ) and the bias from policy mismatch W i,Lcorrespond to the same order of the expected horizon (1 − γ ) − 1 .

Original: This means that the relative magnitude between them is irrelevant to the expected horizon and may largely depend on N .

Original: The variance term dominates when N is small, while the bias term dominates when N is large.
Revised: Besides, the variance term dominates when N is small, while the bias term dominates at large N .

Original: Therefore, one may imagine that the training of Alg. 1 has two phases.
Revised: Therefore, one may imagine that the training of Alg. 1 has two phases.

Original: At phase 1, the variance term dominates and decreases in N , so the learning improves quickly as more trajectories are collected.
Revised: At phase 1, the variance term dominates and decreases in N , so the learning improves quickly as more trajectories are collected.

Original: At phase 2, the bias term dominates, so π N becomes harder to improve and  tends to converge.The tendency of convergence makes the bias smaller, meaning that the error of ˆ Q becomes smaller, too. Therefore, the policy can still slowly improve in N at phase 2. This explains why experiments (Figure 2) usually show a quick performance improvement at the beginning followed by a drastic slowdown.
Revised: At phase 2, the bias term dominates, so the policy evaluation error becomes harder to improve and π N tends to converge.

Original:  P RACTICAL SAMPLING STRATEGIES
Revised:  P RACTICAL SAMPLING STRATEGIES

Original: As previously mentioned, the supervised learning during the Q function update fails to control the bias and variance.
Revised: As previously mentioned, the supervised learning during the Q function update fails to control the bias and variance.

Original: We need careful sampling techniques during the sampling from the replay buffer to mitigate the policy evaluation error.
Revised: We need careful sampling techniques during the sampling from the replay buffer to mitigate the policy evaluation error.

Original: In particular, Wang & Ross (2019) proposes to emphasize recent experience (ERE) because the recent policies are closer to the latest policy.
Revised: In particular, Wang & Ross (2019) proposes to emphasize recent experience (ERE) because the recent policies are closer to the latest policy.

Original: We show below that the ERE strategy is a reﬁnement of 1/age weighting and that both methods help balance the expected selection number of each training transition ( s, a ) .
Revised: We show below that the ERE strategy is a reﬁnement of 1/age weighting and that both methods help balance the expected selection number of each training transition ( s, a ) .

Original: Balanced selection numbers reduce both the policy mismatch bias and sampling variance.
Revised: Balanced selection numbers reduce both the policy mismatch bias and sampling variance.

Original: Hence, this suggests the potential usefulness of ERE and 1/age, which we verify through experiments in the last subsection.
Revised: Hence, this suggests the potential usefulness of ERE and 1/age, which we verify through experiments in the last subsection.

Original: 5. E MPHASIZING RECENT EXPERIENCE
Revised: 5. E MPHASIZING RECENT EXPERIENCE

Original: In Wang & Ross (2019), the authors use a length- K trajectory ( K may differ across episodes) and perform K updates.
Revised: In Wang & Ross (2019), the authors use a length- K trajectory ( K may differ across episodes) and perform K updates.

Original: In the k -th ( 1 ≤ k ≤ K ) update, a mini-batch is sampled uniformly from the most recent c k = max( N 0 η k L 0 K , c min ) samples, where N 0 is the current size of the replay buffer,
Revised: In the k -th ( 1 ≤ k ≤ K ) update, a mini-batch is sampled uniformly from the most recent c k = max( N 0 η k L 0 K , c min ) samples, where N 0 is the current size of the replay buffer,

Original: L 0 is the maximum horizon of the environment, η is the decay parameter, and c min is the minimum coverage of the sampling.
Revised: L 0 is the maximum horizon of the environment, η is the decay parameter, and c min is the minimum coverage of the sampling.

Original: For MuJoCo (Todorov et al., 2012) environments, the paper suggests the values: ( L 0 , η, c min ) =
Revised: For MuJoCo (Todorov et al., 2012) environments, the paper suggests the values: ( L 0 , η, c min ) =

Original: One can see that η = 1 does a uniform weighting over the replay buffer, and the emphasis on the recent data becomes larger as η becomes smaller.
Revised: One can see that η = 1 does a uniform weighting over the replay buffer, and the emphasis on the recent data becomes larger as η becomes smaller.

Original: To see how the ERE strategy affects the mini-batch sampling, we prove the following result.
Revised: To see how the ERE strategy affects the mini-batch sampling, we prove the following result.

Original: Proposition 1.
Revised: Proposition 1.

Original: ERE is approximately equivalent (Taylor Approx.)
Revised: ERE is approximately equivalent (Taylor Approx.)

Original: to the non-uniform weighting:
Revised: to the non-uniform weighting:

Original: [Equation]
Revised: [Equation]

Original: where t is the age of a data point relative to the newest time step; i.e., w 1 is the newest sample.
Revised: where t is the age of a data point relative to the newest time step; i.e., w 1 is the newest sample.

Original: Note that Prop. 1 holds for η (cid:54)
Revised: Note that Prop. 1 holds for η (cid:54)

Original: = 1 because it is derived from the geometric series formula: (1 − η n ) / (1 − η ) , which is valid when η (cid:54) = 1 .
Revised: = 1 because it is derived from the geometric series formula: (1 − η n ) / (1 − η ) , which is valid when η (cid:54) = 1 .

Original: Despite this discontinuity, we may still claim that the
Revised: Despite this discontinuity, we may still claim that the

Original: ERE strategy performs a uniform weighting when η is close to 1.
Revised: ERE strategy performs a uniform weighting when η is close to 1.

Original: This is because when η ≈ 1 ,
Revised: This is because when η ≈ 1 ,

Original: Eq.
Revised: Eq.

Original: (8) suggests w t is proportional to 1 / ( N 0 η L 0 ) − 1 /N 0 for all 1 ≤ t ≤ N 0 , which is a uniform weighting.
Revised: (8) suggests w t is proportional to 1 / ( N 0 η L 0 ) − 1 /N 0 for all 1 ≤ t ≤ N 0 , which is a uniform weighting.

Original: The emphasis on the recent experience (indicated by 1 ( t ≤ c min ) ) is also evident from
Revised: The emphasis on the recent experience (indicated by 1 ( t ≤ c min ) ) is also evident from

Original: Eq.
Revised: Eq.

Original: Precisely, the second term increases logarithmically ln 1 η when η becomes smaller, so the smaller η indeed gives more weight on the recent experience. 
Revised: Precisely, the second term increases logarithmically ln 1 η when η becomes smaller, so the smaller η indeed gives more weight on the recent experience. [Figure]

Original: Discussion.
Revised: Discussion.

Original: A key distinction between the original ERE and Prop. 1 is that the original ERE considers the trajectory length K while Prop. 1 doesn’t.
Revised: A key distinction between the original ERE and Prop. 1 is that the original ERE considers the trajectory length K while Prop.doesn’t.

Original: Intuitively, the disappearance of K ’s dependency results from the aggregation of all effects in 1 ≤ k ≤ K updates.
Revised: Intuitively, the disappearance of K ’s dependency results from the aggregation of all effects in 1 ≤ k ≤ K updates.

Original: We verify that Eq.
Revised: We verify that Eq.

Original: (8) tracks the original ERE well in the next subsection.
Revised: (8) tracks the original ERE well in the next subsection.

Original: Another feature of Prop. 1 is an implicit 1/age weighting in Eq.
Revised: Anotherfeature of Prop. 1 is an implicit 1/age weighting in Eq.

Original: Although the original ERE samples uniformly from the recent c k points , the aggregate effect of all 1 ≤ k ≤ K updates appears to be well approximated by a 1/age weighting.
Revised: Although the original ERE samples uniformly from the recent c k points , the aggregate effect of all 1 ≤ k ≤ K updates appears to be well approximated by a 1/age weighting.

Original: To understand the effect of 1/age, recall from section 2.2 that in practice, ˆ Q is updated using mini-batch samples.
Revised: To understand the effect of 1/age, recall from section 2.2 that in practice, ˆ Q is updated using minibatch samples.

Original: Deﬁne a point ( s e i , a ei ) ’s time step as i + 1 + L · ( e − 1) .
Revised: Deﬁne a point ( s ei , a ei ) ’s time step as i + 1 + L · ( e − 1) .

Original: Then the expected number of times in all batch samples that a point at a certain time step is selected (the expected selection number) gives the aggregate weight over the time steps.
Revised: Then the expected number of times in all batch samples that a point at a certain time step is selected (the expected selection number) gives the aggregate weight over the time steps.

Original: As shown in Figure.
Revised: As shown in Figure.

Original: 1, 1/age weighting and ERE apx, Eq.
Revised: 1, 1/age weighting and ERE apx, Eq.

Original: (8), give almost uniform expected selection numbers across time steps while uniform weighting is signiﬁcantly biased toward the old samples.
Revised: (8), give almost uniform expected selection numbers across time steps while uniform weighting is signiﬁcantly biased toward the old samples.

Original: Therefore, 1/age weighting and its variants help balance the expected selection number.
Revised: Therefore, 1/age weighting and its variants help balance the expected selection number.

Original: [Figure]

Original: E x p e c t e d s e l e c t i o n nu m b e r / b a t

Original: c h s i z e [Figure] 5. M ERIT OF BALANCED SELECTION NUMBER
Revised: 5. P OLICY EVALUATION ERROR UNDER NON - UNIFORM WEIGHTS

Original: Recall the expected selection numbers are the aggregate weights over time.
Revised: Recall the expected selection numbers are the aggregate weights over time.

Original: To understand the merit of balanced selection numbers, we develop an error analysis under non-uniform aggregate weights in the Appendix (Theorem 2 & Corollary 2). Similar to  the uniform-weight error in section 4, the nonuniform-weight error is controlled by bias from policy mismatch and variance from sampling.
Revised: To understand the merit of balanced selection numbers, we develop an error analysis under non-uniform aggregate weights in the Appendix A.5. In particular, Corollary 2 generalizes Corollary 1 to non-uniform weights { w i > 0 } such that the policy evaluation error is bounded by three terms: Hoefﬁding error ˜ O

Original: In the following, we discuss these biases and variances in ERE, 1/age weighting, and uniform weighting.
Revised: ( (cid:112)(cid:80) i w 2 i / (cid:80) i w i ) (weighted variance), weighted bias W i,L 1 ,w , and normalized weighted bellman error ξ i,L ˆ Q,w . Since the weighted bellman error is well-controlled by supervised learning, in the following, we discuss the weighted biases and variances in ERE, 1/age weighting, and uniform sampling.

Original: We conclude that the ERE and 1/age weighting are better because both of their biases and variances are smaller than that of uniform weighting.
Revised: We conclude that the ERE and 1/age weighting are better because both of their biases and variances are smaller than that of uniform sampling.

Original: For the bias from policy mismatch, Figure 1 shows the uniform weighting (for each sampling from the replay buffer) makes the aggregate weights (expected selection number) bias toward old samples.
Revised: For the bias from policy mismatch, Figure 1 shows the uniform sampling (for each sampling from the replay buffer) makes the aggregate weights (expected selection number) bias toward old samples.

Original: Because the old policies tend to be distant from the current policy, the uniform weighting has a larger policy mismatch bias  than ERE and 1/age weighting.
Revised: Because the old policies tend to be distant from the current policy, the uniform sampling induces a larger weighted bias W i,L 1 ,w than ERE and 1/age do.

Original: As for the variance from sampling, we generalize Corollary 1’s uniform case: ˜ O (1 / √ N ) to Corollary 

Revised: As for the variance from sampling, we generalize Corollary 1’s uniform case: ˜ O (1 / √ N ) to Corollary

Original: ’s weighted case: ˜ O
Revised: ’s weighted case: ˜ O

Original: ( (cid:112)(cid:80) i
Revised: ( (cid:112)(cid:80) i

Original: w 2 i / (cid:80) i w i ) in the Appendix.
Revised: w 2 i / (cid:80) i w i ) .

Original: To see this, let { X i } Ni =1 be a martingale difference sequence with 0 ≤ X i ≤ ∆ .

Original: Consider a weighted sequence { w i X i } Ni =1 with weights { w i > 0 } .

Original: The Azuma-Hoeffding inequality suggests that with probability ≥ 1 − δ , (cid:80) i w

Original: [Equation] which means that a general non-uniform weight admits a Hoeffding error
Revised: That is, the variance under non-uniform weight is bounded by the Hoeffding error

Original: ˜ O
Revised: ˜ O

Original: ( (cid:112)(cid:80) i
Revised: ( (cid:112)(cid:80) i

Original: w 2 i / (cid:80) i w i ) and is reduced to ˜ O
Revised: w 2 i / (cid:80) i w i ) and is reduced to ˜ O

Original: (1 / √ N ) when the weights are equal.
Revised: (1 / √ N ) when the weights are equal.

Original: Furthermore, one can prove that the Hoeffding error is minimized under uniform weights:
Revised: Furthermore, one can prove that the Hoeffding error is minimized under uniform weights:

Original: Proposition 2.
Revised: Proposition 2.

Original: Let { w t > 0 } Nt
Revised: Let { w t > 0 } Nt

Original: =1 be the weights of the data indexed by t .
Revised: =1 be the weights of the data indexed by t .

Original: Then the Hoeffding error (cid:113)(cid:80)
Revised: Then the Hoeffding error (cid:113)(cid:80)

Original: Nt =1 w 2 t / (cid:80)
Revised: Nt =1 w 2 t / (cid:80)

Original: Nt =1 w t is minimized when the weights are equal: w t = c > 0 , ∀ t .
Revised: Nt =1 w t is minimized when the weights are equal: w t = c > 0 , ∀ t .

Original: Therefore, the variance from sampling is large under non-uniform aggregate weights and is minimized by uniform aggregate weights.
Revised: Therefore, the variance from sampling is large under non-uniform aggregate weights and is minimized by uniform aggregate weights.

Original: Since the uniform weighting leads to more non-uniform aggregate weights than ERE and 1/age weighting, its variance is also larger.
Revised: Since the uniform sampling leads to more non-uniform aggregate weights than ERE and 1/age weighting, its variance is also larger.

Original: Because the ERE and 1/age weighting have balanced selection numbers (i.e., balanced aggregate weights), their biases and variance are smaller.
Revised: Because the ERE and 1/age weighting have balanced selection numbers (i.e., balanced aggregate weights), their biases and variance are smaller.

Original: They should perform better than uniform weighting for off-policy actor-critic RL.
Revised: They should perform better than uniform weighting for off-policy actor-critic RL.

Original: We will verify this in the next subsection.
Revised: We will verify this in the next subsection.

Original: 5. E XPERIMENTAL VERIFICATION
Revised: 5. E XPERIMENTAL VERIFICATION

Original: Since we’ve established a theoretical foundation of ERE and 1/age from a bias-variance perspective, we will explore two main propositions from the preceding subsections: (1) Are ERE and 1/age weighting better than uniform weighting?
Revised: Since we’ve established a theoretical explanation for 1/age-based samplings (ERE, ERE apx, 1/age), we will explore two main propositions from the preceding subsections: (1) Are 1/age-based samplings better than uniform weighting?

Original: (2) Does the approximated ERE proposed in Eq.
Revised: (2) Does the approximated ERE proposed in Eq.

Original: (8) track the original ERE well?
Revised: (8) track the original ERE well?

Original: In addition, since prioritized experience replay (Schaul et al., 2016) (PER) is a popular sampling method, a natural questionto ask is (3) Do ERE and 1/age outperform PER?
Revised: In addition, since prioritized experience replay (Schaul et al., 2016) (PER) is a popular sampling method, a natural question is (3) Do 1/age-based samplings outperform PER?

Original: We evaluate ﬁve sampling methods (ERE, ERE apx, 1/age weighting, uniform, PER) on ﬁve
Revised: We evaluate ﬁve sampling methods (ERE, ERE apx, 1/age weighting, uniform, PER) on ﬁve

Original: MuJoCo continuous-control environments (Todorov et al., 2012):
Revised: MuJoCo continuous-control environments (Todorov et al., 2012):

Original: Humanoid, Ant, Walker2d,
Revised: Humanoid, Ant, Walker2d,

Original: HalfCheetah, and Hopper.
Revised: HalfCheetah, and Hopper.

Original: All tasks have a maximum horizon of 1000 and are trained using a Pytorch Soft-Actor-Critic (Haarnoja et al., 2018) implementation on Github (Tandon, 2018).
Revised: All tasks have a maximum horizon of 1000 and are trained using a Pytorch Soft-Actor-Critic (Haarnoja et al., 2018) implementation on Github (Tandon, 2018).

Revised: Because the standard SAC implementation uses the uniform sampling, by comparing uniform sampling with the other four methods, we can deduce ways to boost SAC from a sampling’s perspective.

Original: Most hyper-parameters of the SAC algorithm are the same as that in Tandon (2018) except for the batch size, where we ﬁnd a batch size of 512 tends to give more stable results.
Revised: Most hyper-parameters of the SAC algorithm are the same as that in Tandon (2018) except for the batch size, where we ﬁnd a batch size of 512 tends to give more stable results.

Original: Our code is available at https://github.com/sunfex/weighted-sac .
Revised: Our code is available at https://github.com/sunfex/weighted-sac .

Original: The SAC implementation and the MuJoCo environment are licensed under the MIT license and the personal student license, respectively.
Revised: The SAC implementation and the MuJoCo environment are licensed under the MIT license and the personal student license, respectively.

Original: The experiment is run on a server with an Intel i7-6850K CPU and Nvidia GTX 1080 Ti GPUs.
Revised: The experiment is run on a server with an Intel i7-6850K CPU and Nvidia GTX 1080 Ti GPUs.

Original: [Figure] In Figure 2, the ERE and 1/age weighting strategies perform better than the uniform weighting does in all tasks.
Revised: In Figure 2, 1/age-based samplings (ERE, ERE apx, 1/age) perform better than the uniform weighting does in all tasks.

Original: This veriﬁes our preceding assertion that the ERE and 1/age weighting are superior because their biases and variances of the estimated Q function are smaller.
Revised: This veriﬁes our preceding assertion that 1/age-based samplings are superior because their biases and variances of the estimated Q function are smaller.

Original: Moreover, ERE and
Revised: Moreover, ERE and

Original: ERE apx mostly coincide with each other, so Eq.
Revised: ERE apx mostly coincide with each other, so Eq.

Original: (8) is indeed a good approximation of the ERE strategy.
Revised: (8) is indeed a good approximation of the ERE strategy.

Original: This also explains the implicit connection between ERE and 1/age weighting strategies:
Revised: This also explains the implicit connection between ERE and 1/age weighting strategies:

Original: ERE is almost equivalent to ERE apx and 1/age is the main factor in ERE apx, so ERE and 1/age weighting should produce similar results.
Revised: ERE is almost equivalent to ERE apx and 1/age is the main factor in ERE apx, so ERE and 1/age [Figure] produce similar results.

Original: Finally, ERE and 1/age generally outperform PER, so the 1/age-based samplings that we study achieve nontrivial performance improvements.
Revised: Finally, ERE and 1/age generally outperform PER, so the 1/age-based samplings that we study achieve nontrivial performance improvements.

Revised: Finally, we provide performance comparison with the prior work Fujimoto et al.

Revised: ; Kozuno et al.

Revised: (2021) in Table 1).

Revised: Table 1 shows the SAC with 1/age-based samplings (ERE, ERE apx, 1/age) outperform the prior work.

Revised: This suggests 1/age-based samplings not only possess simplicity and theoretical explanations but also achieve better performances.

Original:   C ONCLUSION
Revised: [Table]  C ONCLUSION

Original: To understand off-policy actor-critic algorithms, we show the policy evaluation error on the expected distribution of transitions decomposes into the Bellman error, the bias from policy mismatch, and the variance from sampling.
Revised: To understand off-policy actor-critic algorithms, we show the policy evaluation error on the expected distribution of transitions decomposes into the Bellman error, the bias from policy mismatch, and the variance from sampling.

Original: We use this to explain that a successful off-policy actor-critic algorithm should have a careful sampling strategy that controls its bias and variance.
Revised: We use this to explain that a successful off-policy actor-critic algorithm should have a careful sampling strategy that controls its bias and variance.

Original: Motivated by the empirical success of Emphasizing Recent Experience (ERE), we prove that ERE is a variant of the 1/age weighting.
Revised: Motivated by the empirical success of Emphasizing Recent Experience (ERE), we prove that ERE is a variant of the 1/age weighting.

Original: We then explainfrom a bias-variance perspective that  ERE  and1/age weighting are preferable over uniform weighting because the resulting biases and variances are smaller.
Revised: We then explain that 1/age-based samplings (e.g., ERE, ERE apx, 1/age) have smaller bias and variance and are preferable over uniform sampling. Our experiments verify that soft actorcritic with 1/age-based samplings outperforms the prior work. We hence conclude that a simple but careful design in the sampling of off-policy actor-critic RL can lead to better performances.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our work explains off-policy actor-critic RL algorithms and designs simple but effective training techniques.
Revised: Our work explains off-policy actor-critic RL algorithms and designs simple but effective training techniques.

Original: The results apply to real domains where past trajectories are abundant while new sam- ples are costly, e.g., robotics, recommender systems, and power grid management.
Revised: The results apply to real domains where past trajectories are abundant while new samples are costly, e g., robotics, recommender systems, and power grid management.

Original: RL algorithms may have positive economic effects by boosting efﬁciency and lowering risk.
Revised: RL algorithms may have positive economic effects by boosting efﬁciency and lowering risk.

Original: But inappropriate use can have negative societal impacts.
Revised: But inappropriate use can have negative societal impacts.

Original: These include job loss due to automation, the ethical challenges of delegating important decisions to a machine, and data privacy issues (e g., recommendation agents).
Revised: These include job loss due to automation, the ethical challenges of delegating important decisions to a machine, and data privacy issues (e g., recommendation agents).

Original: These implications apply to most control and RL studies and are not associated with any speciﬁc work.
Revised: These implications apply to most control and RL studies and are not associated with any speciﬁc work.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: To ensure reproducibility of the experiments, we’ve provided hyper-parameters in Appendix A. and the source code at an anonymous Github page https://github.com/sunfex/ weighted-sac .
Revised: To ensure reproducibility of the experiments, we’ve provided hyper-parameters in Appendix A. and the source code at an anonymous Github page https://github.com/sunfex/ weighted-sac .

Original: Discussions on the assumptions are in section 4.1 and justiﬁcations are in Appendix A.2.
Revised: Discussions on the assumptions are in section 4.1 and justiﬁcations are in Appendix A.2.

Original: A complete proof of the theorems and lemmas can be found in Appendix.
Revised: A complete proof of the theorems and lemmas can be found in Appendix

Original: A A PPENDIX A. H YPER - PARAMETERS
Revised: A A PPENDIX A. H YPER - PARAMETERS

Original: To train the SAC agents, we use deep neural networks to parameterize the policy and Q functions.
Revised: To train the SAC agents we use deep neural networks to parameterize the policy and Q functions.

Original: Both networks consist of dense layers with the same widths.
Revised: Both networks consist of dense layers with the same widths.

Original: Table 1 presents the suggested hyperparameters.
Revised: Table 2 presents the suggested hyperparameters.

Original: As mentioned in the experiment section, the hyper-parameters are similar as the implementation in Tandon (2018).[Table] [Table]
Revised: As mentioned in the experiment section, the hyper-parameters are similar as the implementation in Tandon (2018).

Original: A. J USTIFICATION OF ASSUMPTIONS
Revised: A. J USTIFICATION OF ASSUMPTIONS

Original: In section 4.1, we introduce three main assumptions in this work.
Revised: In section 4. we introduce three main assumptions in this work.

Original: Below is a validation for each.
Revised: Below is a validation for each

Original:  For each s ∈ S , ˆ Q ( s, a ) and Q π N ( s, a ) are L A -Lipschitz in a. As mentioned in the paragraph ”Normalization” of section 4.2, the Lipschitzness of Q π N is sublinear or linear in the horizon, which quantiﬁes the magnitude of Q π N ’s Lipschitz constant.
Revised: [Table] [Table] For each s ∈ S , ˆ Q ( s, a ) and Q π N ( s, a ) are L A -Lipschitz in a. As mentioned in the paragraph ”Normalization” of section 4.2, the Lipschitzness of Q π N is sublinear or linear in the horizon, which quantiﬁes the magnitude of Q π N ’s Lipschitz constant.

Original: Since ˆ Q approximates
Revised: Since ˆ Q approximates

Original: Q π N , ˆ Q should have a similar property as long as the training error is well controlled.
Revised: Q π N , ˆ Q should have a similar property as long as the training error is well controlled.

Original: The practitioner can also enforce the Lipschitzness of ˆ Q using gradient penalty (Gulrajani et al., 2017) or spectral normalization (Miyato et al., 2018).
Revised: The practitioner can also enforce the Lipschitzness of ˆ Q using gradient penalty (Gulrajani et al., 2017) or spectral normalization (Miyato et al., 2018).

Original: ˆ Q ( s, a ) and Q π N ( s, a ) are bounded and take values in [0 , r max / (1 − γ )] .
Revised: ˆ Q ( s, a ) and Q π N ( s, a ) are bounded and take values in [0 , r max / (1 − γ )] .

Original: This is a standard assumption in the RL literature.
Revised: This is a standard assumption in the RL literature.

Original: If the bound is violated, one can either clip, translate, or rescale to obtain new Q functions that satisfy the constraint.
Revised: If the bound is violated, one can either clip, translate, or rescale to obtain new Q functions that satisfy the constraint.

Original: Note a bounded reward r ( s, a ) ∈ [0 , r max ] has implied Q π N ∈ [0 , r max / (1 − γ )] .
Revised: Note a bounded reward r ( s, a ) ∈ [0 , r max ] has implied Q π N ∈ [0 , r max / (1 − γ )] .

Original: The action space is bounded with diameter diam A < ∞ .
Revised: The action space is bounded with diameter diam A < ∞ .

Original: This is a standard assumption in continuous-control environments and is usually satisﬁed in practice.
Revised: This is a standard assumption in continuous-control environments and is usually satisﬁed in practice.

Original: It is also common to use clipping to ensure the bounds of the actions generated by the policy (Fujita & Maeda, 2018).
Revised: It is also common to use clipping to ensure the bounds of the actions generated by the policy (Fujita & Maeda, 2018).

Original: A. T HE CONSTRUCTION OF OUR BEHAVIOR POLICY
Revised: A. T HE CONSTRUCTION OF OUR BEHAVIOR POLICY

Original: We ﬁrst discuss some important relations between state occupancy measures and Bellman ﬂow operator.
Revised: We ﬁrst discuss some important relations between state occupancy measures and Bellman ﬂow operator.

Original: Similar results about Fact 1 and 2 be found in Liu et al.
Revised: Similar results about Fact 1 and 2 be found in Liu et al.

Original: (2018)[Lemma 3].
Revised: (2018)[Lemma 3].

Original: Deﬁnition 1 (Bellman ﬂow operator) .
Revised: Deﬁnition 1 (Bellman ﬂow operator) .

Original: The Bellman ﬂow operator B ρ 0 ,π,T ( · ) generated by ( ρ 0 , π, T ) with discount factor γ is deﬁned as
Revised: The Bellman ﬂow operator B ρ 0 ,π,T ( · ) generated by ( ρ 0 , π, T ) with discount factor γ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Fact 1.
Revised: Fact 1.

Original: B ρ 0 ,π,T is a γ -contraction w.r.t.
Revised: B ρ 0 ,π,T is a γ -contraction w.r

Revised: [Equation]

Original:  total variational distance.
Revised: t. total variational distance.

Original: Proof.
Revised: Proof.

Original: Let p 1 ( s ) , p 2 ( s ) be the density functions of some state distributions.
Revised: Let p 1 ( s ) , p 2 ( s ) be the density functions of some state distributions

Original: [Equation]
Revised: [Equation]

Original: Fact 2.
Revised: Fact 2.

Original: The normalized state occupancy measure ρ
Revised: The normalized state occupancy measure ρ

Original: πρ 0 generated by ( ρ 0 , π, T ) with discount factor γ is a ﬁxed point of B ρ 0 ,π,T ; i.e., B ρ 0 ,π,T ( ρ πρ 0 )
Revised: πρ 0 generated by ( ρ 0 , π, T ) with discount factor γ is a ﬁxed point of B ρ 0 ,π,T ; i.e., B ρ 0 ,π,T ( ρ πρ 0 )

Original: ( s ) =
Revised: ( s ) =

Original: ρ πρ 0 ( s ) .
Revised: ρ πρ 0 ( s ) .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Thus, the Bellman ﬂow operator is useful to analyze the state occupancy measures, and we have the following lemma to construct the behavior policy π Di .
Revised: Thus, the Bellman ﬂow operator is useful to analyze the state occupancy measures, and we have the following lemma to construct the behavior policy π Di .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let ρ ei ( s ) the state distribution at trajectory step i in episode e .
Revised: Let ρ ei ( s ) the state distribution at trajectory step i in episode e .

Original: Let ρ π e ρ ei ( s ) be the normalized occupancy measure starting at trajectory step i in episode e .
Revised: Let ρ π e ρ ei ( s ) be the normalized occupancy measure starting at trajectory step i in episode e .

Original: Then 1 N (cid:80) Ne =

Revised: Then 1 N (cid:80) Ne =

Original: ρ π e ρ ei ( s ) =
Revised: ρ π e ρ ei ( s ) =

Original: ρ π Di ρ i ( s ) a.e. , where ρ π Di ρ
Revised: ρ π Di ρ i ( s ) a.e. , where ρ π Di ρ

Original: i is the normalized state occupancy measure is generated by ( ρ i , π Di , T ) .
Revised: i is the normalized state occupancy measure is generated by ( ρ i , π Di , T ) .

Original: Moreover, we have 1 N (cid:80) Ne =

Revised: Moreover, we have 1 N (cid:80) Ne =

Original: ρ π e ρ ei ( s ) π e ( a | s ) =
Revised: ρ π e ρ ei ( s ) π e ( a | s ) =

Original: ρ π Di ρ i ( s ) π Di ( a | s ) a.e.
Revised: ρ π Di ρ i ( s ) π Di ( a | s ) a.e.

Original: Proof.
Revised: Proof.

Original: Precisely, ρ e i ( s ) =
Revised: Precisely, ρ ei ( s ) =

Original: ρ i ( s | ρ 0 , π e , T ) is the state distribution at trajectory step i following the laws of ( ρ 0 , π e , T ) .
Revised: ρ i ( s | ρ 0 , π e , T ) is the state distribution at trajectory step i following the laws of ( ρ 0 , π e , T ) .

Original: Since ρ π e ρ ei is the normalized occupancy measure generated by ( ρ ei , π e , T ) , each ρ π e ρ ei is the ﬁxed-point of the Bellman ﬂow equation:
Revised: Since ρ π e ρ ei is the normalized occupancy measure generated by ( ρ ei , π e , T ) , each ρ π e ρ ei is the ﬁxed-point of the Bellman ﬂow equation:

Original: [Equation]
Revised: [Equation]

Original: This implies the average normalized occupancy measure is the ﬁxed point of the Bellman ﬂow equation characterized by ( ρ i , π Di , T ) :
Revised: This implies the average normalized occupancy measure is the ﬁxed point of the Bellman ﬂow equation characterized by ( ρ i , π Di , T ) :

Original: [Equation]
Revised: [Equation]

Original: where π [Equation] a [Equation]
Revised: where π Di ( a | s ) (cid:44) (cid:80)

Original:  s [Equation]
Revised: Ne =1 π e ( a | s ) ρ πeρei ( s ) (cid:80)

Original: e =

Revised: Ne =

Original: ρ πe ρei ( s ) is the average behavior policy at step i .
Revised: ρ πeρei ( s ) is the average behavior policy at step i .

Original: Since the Bellman ﬂow operator is a γ -contraction in TV distance and hence has a unique ﬁxed point in TV distance, denoted as ρ π Di
Revised: Since the Bellman ﬂow operator is a γ -contraction in TV distance and hence has a unique ﬁxed point in TV distance, denoted as ρ π Di

Original: ρ i ( s ) , we arrive at 1 N (cid:80) Ne =

Revised: ρ i ( s ) , we arrive at 1 N (cid:80) Ne =

Original: ρ π e ρ ei ( s ) =
Revised: ρ π e ρ ei ( s ) =

Original: ρ π Di ρ i ( s ) almost everywhere.
Revised: ρ π Di ρ i ( s ) almost everywhere.

Original: Also, by construction, we have
Revised: Also, by construction, we have

Original: [Equation]
Revised: [Equation]

Original: A. P OLICY E VALUATION E RROR
Revised: A. [Equation] OLICY [Equation]

Revised: VALUATION

Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: If Q ( s, a ) is L A -Lipschitz in a for any s , then, for any state distribution ρ ,
Revised: If Q ( s, a ) is L A -Lipschitz in a for any s , then, for any state distribution ρ ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any ﬁxed s , we have
Revised: For any ﬁxed s , we have

Original: [Equation]
Revised: [Equation]

Original: where the second line follows from Kantorovich-Rubinstein duality (Villani, 2008).
Revised: where the second line follows from Kantorovich-Rubinstein duality (Villani, 2008).

Original: Since the 1Wasserstein distance W 1 is symmetric, we can interchange the roles of π 1 and π 2 , yielding
Revised: Since the 1Wasserstein distance W 1 is symmetric, we can interchange the roles of π 1 and π 2 , yielding

Original: [Equation]
Revised: [Equation]

Original: Taking the expectation E s ∼ ρ on both sides completes the proof.
Revised: Taking the expectation E s ∼ ρ on both sides completes the proof.

Original: Lemma 3.
Revised: Lemma 3.

Original: If Q π ( s, a ) is L A -Lipschitz in a for any s , then
Revised: If Q π ( s, a ) is L A -Lipschitz in a for any s , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any ( s, a )
Revised: For any ( s, a )

Original: [Table]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last line follows from Lemma 2.
Revised: where the last line follows from Lemma 2.

Original: Let ρ π D 

Revised: Let ρ π D

Original: i be the state distribution at step i following the laws of (ˆ ρ 0 , π D 0 , T ) .
Revised: i be the state distribution at step i following the laws of (ˆ ρ 0 , π D 0 , T ) .

Original: Take expectation over ˆ ρ 0 and expand the recursive relation.
Revised: Take expectation over ˆ ρ 0 and expand the recursive relation.

Original: We arrive at
Revised: We arrive at

Original: E

Original: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the second line follows fromρ π
Revised: where the second line follows from

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: If Q ( s, a ) is L A -Lipschitz in a for any s , then
Revised: If Q ( s, a ) is L A -Lipschitz in a for any s , then

Original: [Equation]
Revised: [Equation]

Original: 1 − γ .

Original: Proof.
Revised: Proof.

Original: For any ( s, a ) ∈ S × A , we have a recursive relation:
Revised: For any ( s, a ) ∈ S × A , we have a recursive relation:

Original: [Equation]
Revised: [Equation]

Original: Expand the recursive relation.
Revised: Expand the recursive relation.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: The last line uses a fact for the normalized occupancy measure :

Original: [Equation]
Revised: [Equation]

Original: We are almost done once the B π D 0 in Eq.
Revised: We are almost done once the B π D 0 in Eq.

Original: (9) is replaced by B π .
Revised: (9) is replaced by B π .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Revised: Lem

Revised: [Equation]

Original:  The second-last line follows from that the distribution of s (cid:48) is ρ 
Revised: [Equation] The second-last line follows from that the distribution of s (cid:48) is ρ π [Equation] ˆ

Original: [Equation]
Revised: [Equation]

Original: which satisﬁes the identity
Revised: which satisﬁes the identity

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (9) and Eq.
Revised: (9) and Eq.

Original: (10), the result follows.
Revised: (10), the result follows.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let 0 ≤ f ( s, a ) ≤ ∆ be a bounded function for ( s, a ) ∈ S × A .
Revised: Let 0 ≤ f ( s, a ) ≤ ∆ be a bounded function for ( s, a ) ∈ S × A .

Original: Let ˜ E ∞ 0 be the averaging-discounted operator with inﬁnite-length trajectories in N episodes; i.e., ˜ E ∞ 
f = 1 N (cid:80) Ne = [Equation] (cid:80) ∞ i =
Revised: Let ˜ E ∞ 0 be the averaging-discounted operator with inﬁnite-length trajectories in N episodes; i.e., ˜ E ∞

Original: [Equation] 1 − γ [Equation] γ i f 
Revised: f =

Original: [Equation]
Revised: [Equation]

Original: s

Original: [Equation]

Original: , a ei ) .

Original: Then, with probability greather than 1 − δ ,
Revised: Then, with probability greather than 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let δ s e 0 ( s ) be the delta measure at the initial state in episode e .
Revised: Let δ s e 0 ( s ) be the delta measure at the initial state in episode e .

Original: Because the empirical distribution ˆ ρ 0 is composed of the realization of the trajectories’s initial states, we know ˆ ρ 0 = 1 N (cid:80) Ne =

Revised: Because the empirical distribution ˆ ρ 0 is composed of the realization of the trajectories’s initial states, we know ˆ ρ 0 = 1 N (cid:80) Ne =

Original: δ s e 0 .
Revised: δ s e 0 .

Original: Then, Lemma 1 implies ρ π D 

Revised: Then, Lemma 1 implies ρ π D

Original: ˆ ρ 0 is an average of the normalized occupancy measures in N episodes.
Revised: ˆ ρ 0 is an average of the normalized occupancy measures in N episodes.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ρ ei is the state density at trajectory step i in episode e .
Revised: where ρ ei is the state density at trajectory step i in episode e .

Original: Since ( s ei , a ei ) ∼ ρ ei ( s | δ s e 0 , π e , T ) π e ( a | s ) , we have
Revised: Since ( s ei , a ei ) ∼ ρ ei ( s | δ s e 0 , π e , T ) π e ( a | s ) , we have

Original: [Equation]
Revised: [Equation]

Original: We claim that { M e } Ne =1 deﬁned as M e = (cid:80) ∞ i =

Revised: We claim that { M e } Ne =1 deﬁned as M e = (cid:80) ∞ i =

Original: (1 − γ ) γ i (cid:2) f
Revised: (1 − γ ) γ i (cid:2) f

Original: ( s ei , a ei ) − E
Revised: ( s ei , a ei ) − E

Original: [ f ( s ei , a ei ) | s e 0 ] (cid:3) is a martingale difference sequence.
Revised: [ f ( s ei , a ei ) | s e 0 ] (cid:3) is a martingale difference sequence.

Original: To see this, let F e be the ﬁltration of all randomness from episode 1 to e , with F 0 being a null set.
Revised: To see this, let F e be the ﬁltration of all randomness from episode 1 to e , with F 0 being a null set.

Original: Clearly, we have M e ∈ F e , E
Revised: Clearly, we have M e ∈ F e , E

Original: [ M e |F e − 1 ] = 0 and M e ∈ [ − ∆ , ∆] since f ( s, a ) ∈ [0 , ∆] by assumption, which proves { M e } Ne =1 is a martingale difference sequence.
Revised: [ M e |F e − 1 ] = 0 and M e ∈ [ − ∆ , ∆] since f ( s, a ) ∈ [0 , ∆] by assumption, which proves { M e } Ne =1 is a martingale difference sequence.

Original: Finally, since M e is bounded in [ − ∆ , ∆] , by Azuma-Hoeffding inequality, we conclude that with probability greater than 1 − δ ,
Revised: Finally, since M e is bounded in [ − ∆ , ∆] , by Azuma-Hoeffding inequality, we conclude that with probability greater than 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: Let N be the number of episodes.
Revised: Let N be the number of episodes.

Original: For f : S ×
Revised: For f : S ×

Original: A → R , deﬁne the operator ˜ E ∞ 

Revised: A → R , deﬁne the operator ˜ E ∞

Original: f (cid:44) 1 N (cid:80) Ne =1 (cid:80) ∞
Revised: f (cid:44) 1 N (cid:80) Ne =1 (cid:80) ∞

Original: i =0 (1 − γ ) γ i f ( s ei , a ei ) .
Revised: i =0 (1 − γ ) γ i f ( s ei , a ei ) .

Original: Denote the policy mismatch error and the Bellman error as
Revised: Denote the policy mismatch error and the Bellman error as

Original: W 0 , ∞= ˜ E ∞ 

Revised: W 0 , ∞= ˜ E ∞

Original: W 1 ( π N || π D 0 )( · ) and (cid:15) 0 , ∞ ˆ Q= ˜
Revised: W 1 ( π N || π D 0 )( · ) and (cid:15) 0 , ∞ ˆ Q

Original: [-E ∞ 
-]
Revised: [-E ∞ 
-][Equation]

Original: | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively.
Revised: · ) | , respectively.

Original: Assume
Revised: Assume

Original: (1) For each s ∈ S , ˆ Q ( s, a ) and Q π N ( s, a ) are L A -Lipschitz in a. 2) ˆ Q ( s, a ) and Q π N ( s a ) 
Revised: (1) For each s ∈ S , ˆ Q ( s, a ) and Q π N ( s, a ) are L A -Lipschitz in a. (2) ˆ Q ( s, a ) and Q π N ( s, a ) are bounded and take values in [0 , r max / (1 − γ )] .

Original: are bounded

Original: and

Original: take values

Original: in [0 , r max / (1 − γ (3) The action space is bounded with diameter diam A < ∞ 
Revised: The action space is bounded with diameter diam A < ∞ .

Original: Note W 1 ( π N || π Di )( s ) means W 1 ( π N ( ·| s )
Revised: Note W 1 ( π N || π Di )( s ) means W 1 ( π N ( ·| s )

Original: || π Di ( ·| s )) , which is a function of s .
Revised: || π Di ( ·| s )) , which is a function of s .

Original: Then, with probability greater than 1 − δ , we have
Revised: Then, with probability greater than 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof basically combines Lemma 3, 4 and 5.
Revised: The proof basically combines Lemma 3, 4 and 5.

Original: To start with [Equation] the objective is decomposed as:
Revised: To start with, the objective is decomposed as:

Original: [Equation]
Revised: [Equation]

Original: Because | ˆ Q ( s

Original: [Equation]

Original: a ) − Q π N ( s

Original: [Equation]

Original: a ) | ∈ [

Original: [Equation] r max 1 − γ ] [Equation] we know E a ∼ π D 0 ( ·| s ) | ˆ Q ( s, a ) − Q π N ( s, a ) | ∈
Revised: Because | ˆ Q ( s, a ) − Q π N ( s, a ) | ∈ [0 , r max 1 − γ ] , we know E a ∼ π D 0 ( ·| s ) | ˆ Q ( s, a ) − Q π N ( s, a ) | ∈

Original: [0 , r max 1 − γ ] , too.
Revised: [0 , r max 1 − γ ] , too.

Original: Suppose ˆ ρ 0 have N independent samples.
Revised: Suppose ˆ ρ 0 have N independent samples.

Original: By Hoeffding’s inequality, with probability ≥ 1 − δ ,
Revised: By Hoeffding’s inequality, with probability ≥ 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: ) (cid:12)(cid:12) [Equation] r max [Equation]
Revised: δ .

Original: [Equation]

Original: [Equation]

Original: Also [Equation] let f ( s, a ) =
Revised: Also, let f ( s, a ) =

Original: (cid:12)(cid:12) ˆ Q ( s, a ) − ( B [Equation] N [Equation]
Revised: (cid:12)(cid:12) ˆ Q ( s, a ) − ( B π N ˆ Q )( s, a ) (cid:12)(cid:12) +

Original: Q )( s, a ) (cid:12)(cid:12) +

Original: [Equation]
Revised: [Equation]

Original: L A W 1 ( π N ( ·| s ) ||
Revised: L A W 1 ( π N ( ·| s ) ||

Original: π D 0 ( ·| s )) .
Revised: π D 0 ( ·| s )) .

Original: Then f ( s, a ) is bounded in [0 , r max 1 − γ + 2 L A diam A ] .
Revised: Then f ( s, a ) is bounded in [0 , r max 1 − γ + 2 L A diam A ] .

Original: Thereby, Lemma 5 implies that with probability greater than 1 − δ ,
Revised: Thereby, Lemma 5 implies that with probability greater than 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (13), (14) and (15), a union bound implies with probability greater than 1 − 2 δ ,
Revised: (13), (14) and (15), a union bound implies with probability greater than 1 − 2 δ ,

Original: E [Equation] max
Revised: E ( s 0 ,a 0 ) ∼ ρ 0 ( s ) π D 0 ( a | s )

Original: [Equation]
Revised: [Equation]

Original: (cid:17)

Original: Finally, rescaling δ to δ/ 2 ﬁnishes the proof.
Revised: Finally, rescaling δ to δ/ 2 ﬁnishes the proof.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let ˜ E Li be the operator deﬁned as ˜
Revised: Let ˜ E Li be the operator deﬁned as ˜

Original: E Li f
Revised: E Li f

Original: = 1 N (cid:80) N e =1 (cid:80) L − 1 j = i (1 − γ ) γ j − i f
Revised: = 1 N (cid:80) N e =1 (cid:80) L − 1 j = i (1 − γ ) γ j − i f

Original: ( s e j , a e j ) .
Revised: ( s ej , a ej ) .

Original: Fix assumptions (1) (2) (3) of Theorem 1.
Revised: Fix assumptions (1) (2) (3) of Theorem 1.

Original: Rewrite the policy mismatch error and the Bellman error as W i,
Revised: Rewrite the policy mismatch error and the Bellman error as W i,

Original: L= ˜ E
Revised: L= ˜ E

Original: Li W 1 ( π N || π Di )( · ) and (cid:15) i,L ˆ Q = ˜
Revised: Li W 1 ( π N || π Di )( · ) and (cid:15) i,L ˆ Q = ˜

Original: E Li | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively, whereW [Equation] π
Revised: E Li | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively, where

Original: [Equation] || [Equation] Di )( s ) =
Revised: W 1 ( π N || π Di )( s ) =

Original: W 1 ( π N ( ·| s ) ||
Revised: W 1 ( π N ( ·| s ) ||

Original: π Di ( ·| s )) .
Revised: π Di ( ·| s )) .

Original: Let ρ i ( s ) be the average state density at trajectory step i over all N episodes.
Revised: Let ρ i ( s ) be the average state density at trajectory step i over all N episodes.

Original: Then, with probability greater than 1 − δ , we have
Revised: Then, with probability greater than 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, if i ≤ L − log (cid:15) log γ and the constants are normalized as L A =
Revised: Moreover, if i ≤ L − log (cid:15) log γ and the constants are normalized as L A =

Original: c/ (1 − γ ) , (cid:15) i,L ˆ Q = ξ i,L ˆ Q / (1 − γ ) , then, with probability greater than 1 − δ , we have
Revised: c/ (1 − γ ) , (cid:15) i,L ˆ Q = ξ i,L ˆ Q / [Equation] 1 − γ ) , then, with probability greater than 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Notice that Theorem 1 is the situation when i = 0 and L = ∞ .
Revised: Notice that Theorem 1 is the situation when i = 0 and L = ∞ .

Original: To push this to i ≥and L = ∞ , recall that Lemma 1 deﬁnes the behavior policy π Di , the average state distribution
Revised: To push this to i ≥and L = ∞ , recall that Lemma 1 deﬁnes the behavior policy π Di , the average state distribution

Original: ρ i and the state occupancy measure ρ π Di
Revised: ρ i and the state occupancy measure ρ π Di

Original: ρ i at step i .
Revised: ρ i at step i .

Original: Also, since the trajectories in each episode are initialized using the same initial state distribution, we have ρ 0 = ρ 0 .
Revised: Also, since the trajectories in each episode are initialized using the same initial state distribution [Equation] we have ρ 0 = ρ 0 .

Original: Therefore, the objective of
Revised: Therefore, the objective of

Original: Theorem 1 is actually E ( s 0 ,a 0 ) ∼ ρ 0 ( s ) π D 0 ( a | s ) (cid:12)(cid:12)(cid:12) ˆ Q ( s 0 , a 0 ) − Q π N ( s 0 , a 0 ) (cid:12)(cid:12)(cid:12) .
Revised: Theorem 1 is actually E ( [Equation] [Equation]

Original: This is generalized to i ≥using substitutions: ( ρ 0 , π D 0 , ˜ E ∞ 0 )
Revised: This is generalized to i ≥using substitutions: ( ρ 0 , π D 0 , ˜ E ∞ 0 )

Original: → ( ρ i , π Di , ˜ E ∞ i ), yielding
Revised: → ( ρ i , π Di , ˜ E ∞ i )

Original: E ( s i ,a i ) ∼ ρ i ( s ) π Di ( a | s ) (cid:12)(cid:12)(cid:12) ˆ Q ( s i , a i ) − Q π N ( s i , a i ) (cid:12)(cid:12)(cid:12) ≤ r max 1 − γ (cid:114)

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: yielding

Original: [Equation]
Revised: [Equation]

Original:  observe that for any bounded f :
Revised: Finally, observe that for any bounded f :

Original: [Equation]
Revised: [Equation]

Original: We arrive at
Revised: We arrive at

Original: [Equation]
Revised: [Equation]

Revised: , a i ) (cid:12)(cid:12)(cid:12) ≤ r max 1 − γ (cid:114)2 N log δ + (cid:16) r max (1 − γ ) 2 + 2 L A diam A 1 − γ (cid:17)(cid:16)(cid:114)

Revised: N log δ + γ L − i (cid:17) + 1 − γ (cid:16) (cid:15) i,L ˆ Q + 2 L A W i,L 1 (cid:17) .

Original: As for the second conclusion, with the substitutions in the statement, we know γ L − i ≤ (cid:15) .
Revised: As for the second conclusion, with the substitutions in the statement, we know γ L − i ≤ (cid:15) .

Original: Hence,
Revised: Hence,

Original: E {+( s i ,a i ) ∼ ρ i ( s ) π Di ( a | s ) (cid:12)(cid:12)(cid:12) ˆ Q ( s i , a i ) − Q π N ( s i , a i ) (cid:12)(cid:12)(cid:12) ≤ r max 1 − γ (cid:114)2 N log δ + (cid:16) r max (1 − γ ) 
 + 2 c · diam A (1 − γ )(cid:17)(cid:16)(cid:114)+}
Revised: E {+( s i ,a i ) ∼ ρ i ( s ) π Di ( a | s ) (cid:12)(cid:12)(cid:12) ˆ Q ( s i , a i ) − Q π N ( s i , a i ) (cid:12)(cid:12)(cid:12) ≤ r max 1 − γ (cid:114)2 N log δ + (cid:16) r max (1 − γ ) 
 + 2 c · diam A (1 − γ )(cid:17)(cid:16)(cid:114)+}

Original: [Equation] i ,a i ) [Equation]
Revised: N log δ + (cid:15) (cid:17) + (1 − γ )(cid:16) ξ i,L ˆ Q + 2 c · W i,L 1 (cid:17) =

Original: , a
Revised: ˜ O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: A. P OLICY E VALUATION E RROR UNDER N ON - UNIFORM W EIGHTS
Revised: A. P OLICY E VALUATION E RROR UNDER N ON - UNIFORM W EIGHTS

Original: Lemma 6.
Revised: Lemma 6.

Original: Following from Lemma [Equation] let w = { w e > 0 } Ne =1 be the (unnormalized) weights for the episodes.
Revised: Following from Lemma 1, let w = { w e > 0 } Ne =1 be the (unnormalized) weights for the episodes.

Original: We have for the weighted case,
Revised: We have for the weighted case,

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: ρ ei
Revised: ei

Original: [Equation]
Revised: [Equation]

Original: s )
Revised: s )

Original: =
Revised: =

Original: ρ π Di
Revised: ρ π Di,w

Original: [Equation]
Revised: [Equation]

Original: w ρ i,w ( s ) a.e. , where ρ π Di,wρ
Revised: i,w ( s ) a.e. , where ρ π Di,w

Revised: [Equation]

Original: i,w ( s ) is the normalized state occupancy measure is generated by ( ρ i,w , π D i,w , T ) .
Revised: i,w ( s ) is the normalized state occupancy measure is generated by ( ρ i,w , π Di,w , T ) .

Original: Also,
Revised: Also,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is basically a generalization of Lemma 1.
Revised: The proof is basically a generalization of Lemma 1.

Original: Since ρ π e ρ ei is the normalized occupancy measure generated by ( ρ ei , π e , T ) [Equation] we know each ρ π e ρ ei is the ﬁxed-point of the Bellman ﬂow equation:
Revised: Since ρ π e ρ ei is the normalized occupancy measure generated by ( ρ ei , π e , T ) , we know each ρ π e ρ ei is the ﬁxed-point of the Bellman ﬂow equation:

Original: [Equation]
Revised: [Equation]

Original: Taking a weighted average

Original: [Equation]
Revised: [Table]

Original: we get

Original: [Equation]
Revised: [Equation]

Original: where π Di ( a | s ) (cid:44) (cid:80) e w e π e ( a | s ) ρ πeρei ( s ) (cid:80) e w e ρ πeρei ( s ) is the weighted behavior policy at step i .
Revised: where π Di ( a | s ) (cid:44) (cid:80) e w e π e ( a | s ) ρ πeρei ( s ) (cid:80) e w e ρ πeρei ( s ) is the weighted behavior policy at step i .

Original: Since the Bellman ﬂow operator is a γ -contraction in TV distance and hence has a unique (up to difference in some measure zero set) ﬁxed point, denoted as ρ π Di,w ρ i,w ( s ) , we arrive at
Revised: Since the Bellman ﬂow operator is a γ -contraction in TV distance and hence has a unique (up to difference in some measure zero set) ﬁxed point, denoted as ρ π Di,w ρ i,w ( s ) , we arrive at

Original: [Equation]
Revised: [Equation]

Original: ρ π Di,w ρ i,w ( s ) a.e.
Revised: ρ π Di,w ρ i,w ( s ) a.e.

Original: Finally, by deﬁnition of π Di,w ( a | s ) , we conclude that
Revised: Finally, by deﬁnition of π Di,w ( a | s ) , we conclude that

Original: [Equation]
Revised: [Equation]

Original: ρ π Di,w ρ i,w ( s ) π Di

Original: [Equation]
Revised: [Equation]

Original: ) a.e.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let w = { w e > 0 } Ne =1 be the (unnormalized) weights for the episodes.
Revised: Let w = { w e > 0 } Ne =1 be the (unnormalized) weights for the episodes.

Original: Let ˜ E ∞ 0 ,w be the operator deﬁned as ˜ E ∞ 0 ,w f =
Revised: Let ˜ E ∞ 0 ,w be the operator deﬁned as ˜ E ∞ 0 ,w f =

Original: [Equation]
Revised: [Equation]

Original: [-(cid:80) Ne =1 (cid:80) ∞ i =
-]
Revised: [-(cid:80) Ne =1 (cid:80) ∞ i =
-]

Original: (1 − γ ) γ i w e f ( s ei , a ei ) .
Revised: f [Equation] a [Equation]

Original: Fix assumptions (1) (2) (3) of Theorem 1.
Revised: Fix assumptions (1) (2) (3) of Theorem 1.

Original: Rewrite the policy mismatch error and the Bellman error as
Revised: Rewrite the policy mismatch error and the Bellman error as

Original: W 0 , ∞ 1 ,w = ˜
Revised: W 0 , ∞ 1 ,w = ˜

Original: E ∞ 0 ,w W 1 ( π N || π D 0 ,w )( · ) and (cid:15) 0 , ∞ ˆ Q,w = ˜
Revised: E ∞ 0 ,w W 1 ( π N || π D 0 ,w )( · ) and (cid:15) 0 , ∞ ˆ Q,w = ˜

Original: E ∞ 0 ,w | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively, where
Revised: E ∞ 0 ,w | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively, where

Original: W 1 ( π N || π D 0 ,w )( s ) = W 1 ( π N ( ·| s ) ||
Revised: W [Equation]

Original:  π D 0 ,w ( ·| s )) .
Revised: || π D 0 ,w ( ·| s )) .

Original: With probability greater than 1 − δ , we have
Revised: With probability greater than 1 − δ , we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: a
Revised: s

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof basically follows from Theorem 
Revised: The proof basically follows from Theorem 1.

Original: [Equation]

Original: Decompose the objective using Lemma 3,as:
Revised: Decompose the objective using Lemma 3,as:

Original: [Equation]
Revised: [Equation]

Original: 16) where ˆ [Equation]
Revised: 16) where ˆ ρ 0 ,w ( s ) =

Original:  is the weighted empirical initial state distribution
Revised: (cid:80) e w e δ ( s − s e 0 ) (cid:80) e w e is the weighted empirical initial state distribution.

Original: [Equation] Because 
Revised: Because | ˆ Q ( s, a ) − Q π N ( s, a ) | ∈ [0 , r max 1 − γ ] , we know E a ∼ π D 0 ,w ( ·| s ) | ˆ Q ( s, a ) − Q π N ( s, a ) | ∈

Original: [Equation] γ ] , too.
Revised: [0 , r max 1 − γ ] , too.

Original: ByHoeffding’s inequality, with probability greater than 1 − δ ,
Revised: By

Original: [Equation]
Revised: [Equation]

Original: Also, let f ( s, a ) =
Revised: Also, let f ( s, a ) =

Original: (cid:12)(cid:12) ˆ Q ( s, a ) − ( B π N ˆ Q )( s, a ) (cid:12)(cid:12) +
Revised: (cid:12)(cid:12) ˆ Q ( s, a ) − ( B π N ˆ Q )( s, a ) (cid:12)(cid:12) +

Original: 2 L A W 1 ( π N ( ·| s )
Revised: 2 L A W 1 ( π N ( ·| s )

Original: ||
Revised: ||

Original: π D 0 ,w ( ·| s )) .
Revised: π D 0 ,w ( ·| s )) .

Original: Then f ( s, a ) is bounded in [0 , r max 1 − γ + 2 L A diam A ] .
Revised: Then f ( s, a ) is bounded in [0 , r max 1 − γ + 2 L A diam A ] .

Original: Using a weighted version of Azuma-Hoeffding in Lemma 5, we have that with probability greater than 1 − δ ,
Revised: Using a weighted version of Azuma-Hoeffding in Lemma 5, we have that with probability greater than 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (16), (17) and (18), a union bound implies with probability greater than 1 − 2 δ ,
Revised: (16), (17) and (18), a union bound implies with probability greater than 1 − 2 δ ,

Original: E ( s 0 ,a 0 ) ∼ ρ 0 ( s ) π D 0 ,w ( a | s )

Original: [Equation]
Revised: [Equation]

Original: Finally, rescaling δ to δ/ 2 ﬁnishes the proof.
Revised: Finally, rescaling δ to δ/ 2 ﬁnishes the proof.

Original: Corollary 2.
Revised: Corollary 2.

Original: Let ˜ E Li,w be the operator deﬁned as
Revised: Let ˜ E Li,w be the operator deﬁned as

Original: [Equation]
Revised: [Equation]

Original: γ [Equation] f [Equation]
Revised: γ ) γ j − i w e f ( s ej , a ej ) .

Original: s ej

Original: [Equation]

Original: a ej ) .

Original: Fix assumptions (1) (2) (3) of Theorem 1.
Revised: Fix assumptions (1) (2) (3) of Theorem 1.

Original: Rewrite the policy mismatch error and the Bellman error as W i,L 1 ,w = ˜
Revised: Rewrite the policy mismatch error and the Bellman error as W i,L 1 ,w = ˜

Original: E Li,w W 1 ( π N || π Di,w )( · ) and (cid:15) i,L ˆ Q,w = ˜
Revised: E Li,w W 1 ( π N || π Di,w )( · ) and (cid:15) i,L ˆ Q,w = ˜

Original: E Li,w | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively, where W 1 ( π N || π Di,w )( s ) =
Revised: E Li,w | ˆ Q ( · , · ) − ( B π N ˆ Q )( · , · ) | , respectively, where W 1 ( π N || π Di,w )( s ) =

Original: W 1 ( π N ( ·| s ) ||
Revised: W 1 ( π N ( ·| s ) ||

Original: π Di,w ( ·| s )) .
Revised: π Di,w ( ·| s )) .

Original: Let ρ i,w ( s ) be the average weighted state density at trajectory step i .
Revised: Let ρ i,w ( s ) be the average weighted state density at trajectory step i .

Original: Then, with probability greater than 1 − δ , we have
Revised: Then, with probability greater than 1 − δ , we have

Revised: E ( s i ,a i ) ∼ ρ

Revised: i,w ( s ) π Di,w ( a | s )

Original: [Equation]
Revised: [Equation]

Revised: Q

Revised: [Equation]

Revised: s i , a

Revised: i

Original:  Moreover, if i ≤ L − log (cid:15) log γ and the constants are normalized as L A =
Revised: [Equation] s i , [Equation] Moreover, if i ≤ L − log (cid:15) log γ and the constants are normalized as L A =

Original: c/ (1 − γ ) , (cid:15) i,L ˆ Q,w = ξ i,L ˆ Q,w / (1 − γ ) , then, with probability greater than 1 − δ , we have
Revised: c/ (1 − γ ) , (cid:15) i,L ˆ Q,w = ξ i,L ˆ Q,w / (1 − γ ) , then, with probability greater than 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Revised: ≤ ˜ O

Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Observe that for any bounded f :
Revised: Observe that for any bounded f :

Original: [Equation]
Revised: [Equation]

Original: Thus, we can start from Theorem 2 and prove with the same argument in Corollary 1.
Revised: Thus, we can start from Theorem 2 and prove with the same argument in Corollary 1.

Original: A. P ROOF OF E MPHAZING R ECENT E XPERIENCE
Revised: A. P ROOF OF E MPHAZING R ECENT E XPERIENCE

Original: Proposition 1.
Revised: Proposition 1.

Original: The ERE strategy in Wang & Ross (2019) is equivalent to a non-uniform sampling with weight w t :
Revised: The ERE strategy in Wang & Ross (2019) is equivalent to a non-uniform sampling with weight w t :

Original: [Equation]
Revised: [Equation]

Original: where t is the age of a data point relative to the newest time step; i.e., w 0 is the newest sample.
Revised: where t is the age of a data point relative to the newest time step; i.e., w 0 is the newest sample.

Original: N 0 is the size of the experience replay.
Revised: N 0 is the size of the experience replay.

Original: L 

Revised: L

Original: = 1000 is the maximum horizon of the environment.
Revised: = 1000 is the maximum horizon of the environment.

Original: K is the length of the recent trajectory.
Revised: K is the length of the recent trajectory.

Original: η
Revised: η

Original: ≈ 0 .
Revised: ≈ 0 .

Original: 996 is the decay parameter.
Revised: 996 is the decay parameter.

Original: c min ≈ 5000 is the minimum coverage of the sampling.
Revised: c min ≈ 5000 is the minimum coverage of the sampling.

Original: Moreover, the ERE strategy can be approximated (by Taylor
Revised: Moreover, the ERE strategy can be approximated (by Taylor

Original: Approximation) as
Revised: Approximation) as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that Wang & Ross (2019) assume a situation of doing K updates in each episode.
Revised: Recall that Wang & Ross (2019) assume a situation of doing K updates in each episode.

Original: In the k th update, the data is sampled uniformly from the most recent c k = max( N 0 η k L 0 K , c min ) points.
Revised: In the k th update, the data is sampled uniformly from the most recent c k = max [Equation] N 0 η k L [Equation]

Original:  To compute the aggregrated weight w t over these K updates, observe that a data point of age t is in the most recent c k points if c k ≥ t and that the weight in each uniform sample is 1 /c k .
Revised: K , c min ) points. To compute the aggregrated weight w t over these K updates, observe that a data point of age t is in the most recent c k points if c k ≥ t and that the weight in each uniform sample is 1 /c k .

Original: Therefore, w t should be proportional to
Revised: Therefore, w t should be proportional to

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Because c k is designed to be lower bounded by c min , we shall discuss Eq.
Revised: Because c k is designed to be lower bounded by c min , we shall discuss Eq.

Original: (19) by cases.
Revised: (19) by cases.

Original: (1) When t > c min , we know c k > c min because c k ≥ t is a constraint in the sum.
Revised: (1) When t > c min , we know c k > c min because c k ≥ t is a constraint in the sum.

Original: This means c k
Revised: This means c k

Original: =
Revised: =

Original: N 0 η kL 0 /K
Revised: N 0 η kL 0 /K

Original: and hence c k ≥ t is equivalent to k ≤ ln tN 0 /
Revised: and hence c k ≥ t is equivalent to k ≤ ln tN 0 /

Original: ln η L 0 /K .
Revised: ln η L 0 /K .

Original: Eq.
Revised: Eq.

Original: (19) becomes (cid:88)
Revised: (19) becomes (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where (*) is a substitution: ξ = η L 0 /K .
Revised: where (*) is a substitution: ξ = η L 0 /K .

Original: (2) When t ≤ c min , we know c k ≥ t for all k because c k ≥ c min by deﬁnition.
Revised: (2) When t ≤ c min , we know c k ≥ t for all k because c k ≥ c min by deﬁnition.

Original: Eq.
Revised: Eq.

Original: (19) becomes (cid:88)
Revised: (19) becomes (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where (*) does a substitution:
Revised: where (*) does a substitution:

Original: ξ = η L 0 /K and split the sum.
Revised: ξ = η L 0 /K and split the sum.

Original: (**) reuses the analysis in case (1).
Revised: (**) reuses the analysis in case (1).

Original: Combining cases (1) and (2), we arrive at the ﬁrst conclusion.
Revised: Combining cases (1) and (2), we arrive at the ﬁrst conclusion.

Original: As for the approximation, since η ≈ 0 . 996 , let η =
Revised: As for the approximation, since η ≈ 0 . 996 , let η =

Original: − κ .
Revised: − κ .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Thus the ﬁrst term in the conclusion of Prop. 1 is proportional to K .
Revised: Thus the ﬁrst term in the conclusion of Prop. 1 is proportional to K .

Original: The second term is also proportional to K .
Revised: The second term is also proportional to K .

Original: Since w t is only made to be proportional to the RHS and both terms on the RHS become proportional to K , we can remove K on the RHS:
Revised: Since w t is only made to be proportional to the RHS and both terms on the RHS become proportional to K , we can remove K on the RHS:

Original: [Equation]
Revised: [Equation]

Original: Finally, because 0 < η < 1 , − L 0 ln η is a positive number, the above expression can be further simpliﬁed by timing − L 0 ln η on the RHS, yielding the result.
Revised: Finally, because 0 < η < 1 , − L 0 ln η is a positive number, the above expression can be further simpliﬁed by timing − L 0 ln η on the RHS, yielding the result.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let { w t > 0 } Nt
Revised: Let { w t > 0 } Nt

Original: =1 be the weights of the data indexed by t .
Revised: =1 be the weights of the data indexed by t .

Original: Then the Hoeffding error (cid:113) (cid:80)
Revised: Then the Hoeffding error (cid:113)(cid:80)

Original: Nt =1 w 2 t / (cid:80)
Revised: Nt =1 w 2 t / (cid:80)

Original: Nt =1 w t is minimized when the weights are equal: w t = c > 0 , ∀ t .
Revised: Nt =1 w t is minimized when the weights are equal: w t = c > 0 , ∀ t .

Original: Proof.
Revised: Proof.

Original: Let w =
Revised: Let w =

Original: [ w 1 , ..., w N ] (cid:62) be the weight vector and f ( w ) = √ w (cid:62) w/ ( 1 (cid:62) w ) be the Hoeffding error.
Revised: [ w 1 , ..., w N ] (cid:62) be the weight vector and f ( w ) = √ w (cid:62) w/ ( 1 (cid:62) w ) be the Hoeffding error.

Original: Observe that f ( w ) is of the form:
Revised: Observe that f ( w ) is of the form:

Original: [Equation]
Revised: [Equation]

Original:  where (cid:107) w (cid:107) = √ w (cid:62) w is the 2-norm of w .
Revised: / ( 1 (cid:62) w [Equation] = (cid:107) w/ ( 1 (cid:62) w ) (cid:107) , where (cid:107) w (cid:107) = √ w (cid:62) w is the 2-norm of w .

Original: Thereby, let z = w/ ( 1 (cid:62) w ) be the normalized vector.
Revised: Thereby, let z = w/ ( 1 (cid:62) w ) be the normalized vector.

Original: That f ( w ) is minimized is equivalent to that g ( z ) = (cid:107) z (cid:107) is minimized for 1 (cid:62) z = 1 .
Revised: That f ( w ) is minimized is equivalent to that g ( z ) = (cid:107) z (cid:107) is minimized for 1 (cid:62) z = 1 .

Original: By the lagrange multiplier, this happens when
Revised: By the lagrange multiplier, this happens when

Original: [Equation]
Revised: [Equation]

Original: This can be achieved by z t = c for some c > 0 .
Revised: This can be achieved by z t = c for some c > 0 .

Original: Therefore, we know f is minimized when
Revised: Therefore, we know f is minimized when

Original: [Equation]
Revised: [Equation]

Original: Since 1 (cid:62) w does not depend on t , we conclude that the minimizer happens at w t = c > 0 , ∀ t.
Revised: Since 1 (cid:62) w does not depend on t , we conclude that the minimizer happens at w t = c > 0 , ∀ t.