Original: B REGMAN G RADIENT P OLICY O PTIMIZATION
Revised: B REGMAN G RADIENT P OLICY O PTIMIZATION

Original: A BSTRACT
Revised: A BSTRACT

Original: In the paper, we design a novel Bregman gradient policy optimization framework for reinforcement learning based on Bregman divergences and momentum techniques.
Revised: In the paper, we design a novel Bregman gradient policy optimization framework for reinforcement learning based on Bregman divergences and momentum techniques.

Original: Speciﬁcally, we propose a Bregman gradient policy optimization (BGPO) algorithm based on the basic momentum technique and mirror descent iteration.
Revised: Speciﬁcally, we propose a Bregman gradient policy optimization (BGPO) algorithm based on the basic momentum technique and mirror descent iteration.

Original: Meanwhile, we further propose an accelerated Bregman gradient policy optimization (VR-BGPO) algorithm based on the variance reduced technique.
Revised: Meanwhile, we further propose an accelerated Bregman gradient policy optimization (VR-BGPO) algorithm based on the variance reduced technique.

Original: Moreover, we provide a convergence analysis framework for our Bregman gradient policy optimization under the nonconvex setting.
Revised: Moreover, we provide a convergence analysis framework for our Bregman gradient policy optimization under the nonconvex setting.

Original: We prove that our BGPO achieves a sample complexity of O ( (cid:15) − 4 ) for ﬁnding (cid:15) -stationary policy only requiring one trajectory at each iteration, and our VR-BGPO reaches the best known sample complexity of O ( (cid:15) − 3 ) , which also only requires one trajectory at each iteration.
Revised: We prove that our BGPO achieves a sample complexity of O ( (cid:15) − 4 ) for ﬁnding (cid:15) -stationary policy only requiring one trajectory at each iteration, and our VR-BGPO reaches the best known sample complexity of O ( (cid:15) − 3 ) , which also only requires one trajectory at each iteration.

Original: In particular, by using different Bregman divergences, our BGPO framework uniﬁes many existing policy optimization algorithms such as the existing (variance reduced) policy gradient algorithms such as natural policy gradient algorithm.
Revised: In particular, by using different Bregman divergences, our BGPO framework uniﬁes many existing policy optimization algorithms such as the existing (variance reduced) policy gradient algorithms such as natural policy gradient algorithm.

Original: Extensive experimental results on multiple reinforcement learning tasks demonstrate the efﬁciency of our new algorithms.
Revised: Extensive experimental results on multiple reinforcement learning tasks demonstrate the efﬁciency of our new algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Policy Gradient (PG) methods are a class of popular policy optimization methods for Reinforcement Learning (RL), and have achieved signiﬁcant successes in many challenging applications (Li, 2017) such as robot manipulation (Deisenroth et al., 2013), the Go game (Silver et al., 2017) and autonomous driving (Shalev-Shwartz et al., 2016).
Revised: Policy Gradient (PG) methods are a class of popular policy optimization methods for Reinforcement Learning (RL), and have achieved signiﬁcant successes in many challenging applications (Li, 2017) such as robot manipulation (Deisenroth et al., 2013), the Go game (Silver et al., 2017) and autonomous driving (Shalev-Shwartz et al., 2016).

Original: In general, PG methods directly search for the optimal policy by maximizing the expected total reward of Markov Decision Processes (MDPs) involved in RL, where an agent takes action dictated by a policy in an unknown dynamic environment over a sequence of time steps.
Revised: In general, PG methods directly search for the optimal policy by maximizing the expected total reward of Markov Decision Processes (MDPs) involved in RL, where an agent takes action dictated by a policy in an unknown dynamic environment over a sequence of time steps.

Original: Since the PGs are generally estimated by Monte-Carlo sampling, such vanilla PG methods usually suffer from very high variances resulted in slow convergence rate and destabilization.
Revised: Since the PGs are generally estimated by Monte-Carlo sampling, such vanilla PG methods usually suffer from very high variances resulted in slow convergence rate and destabilization.

Original: Thus, recently many fast PG methods have been proposed to reduce variances in vanilla stochastic PGs.
Revised: Thus, recently many fast PG methods have been proposed to reduce variances in vanilla stochastic PGs.

Original: For example, Sutton et al.
Revised: For example, Sutton et al.

Original: (2000) introduced a baseline to reduce variances of the stochastic PG.
Revised: (2000) introduced a baseline to reduce variances of the stochastic PG.

Original: Konda & Tsitsiklis (2000) proposed an efﬁcient actor-critic algorithm by estimating the value function to reduce effects of large variances.
Revised: Konda & Tsitsiklis (2000) proposed an efﬁcient actor-critic algorithm by estimating the value function to reduce effects of large variances.

Original: (Schulman et al., 2016) proposed the generalized advantage estimation (GAE) to control both the bias and variance in policy gradient.
Revised: (Schulman et al., 2016) proposed the generalized advantage estimation (GAE) to control both the bias and variance in policy gradient.

Original: More recently, some faster variance-reduced PG methods (Papini et al., 2018; Xu et al., 2019a; Shen et al., 2019;
Revised: More recently, some faster variance-reduced PG methods (Papini et al., 2018; Xu et al., 2019a; Shen et al., 2019;

Original: Liu et al., 2020; Huang et al., 2020) have been developed based on the variance-reduction techniques in stochastic optimization.
Revised: Liu et al., 2020; Huang et al., 2020) have been developed based on the variance-reduction techniques in stochastic optimization.

Original: Alternatively, some successful PG algorithms (Schulman et al., 2015; 2017) improve convergence rate and robustness of vanilla PG methods by using some penalties such as Kullback-Leibler (KL) divergence penalty.
Revised: Alternatively, some successful PG algorithms (Schulman et al., 2015; 2017) improve convergence rate and robustness of vanilla PG methods by using some penalties such as Kullback-Leibler (KL) divergence penalty.

Original: For example, trust-region policy optimization (TRPO) (Schulman et al., 2015) ensures that the new selected policy is near to the old one by using KL-divergence constraint, while proximal policy optimization (PPO)
Revised: For example, trust-region policy optimization (TRPO) (Schulman et al., 2015) ensures that the new selected policy is near to the old one by using KL-divergence constraint, while proximal policy optimization (PPO)

Original: (Schulman et al., 2017) clips the weighted likelihood ratio to implicitly reach this goal.
Revised: (Schulman et al., 2017) clips the weighted likelihood ratio to implicitly reach this goal.

Original: Subsequently, Shani et al.
Revised: Subsequently, Shani et al.

Original: (2020) have analyzed the global convergence properties of TRPO in tabular RL based on the convex mirror descent algorithm.
Revised: (2020) have analyzed the global convergence properties of TRPO in tabular RL based on the convex mirror descent algorithm.

Original: Liu et al.
Revised: Liu et al.

Original: [Table]
Revised: [Table]

Original: (2019) have also studied the global convergence properties of PPO and TRPO equipped with overparametrized neural networks based on mirror descent iterations.
Revised: (2019) have also studied the global convergence properties of PPO and TRPO equipped with overparametrized neural networks based on mirror descent iterations.

Original: At the same time, Yang et al.
Revised: At the same time, Yang et al.

Original: (2019) tried to propose the PG methods based on the mirror descent algorithm.
Revised: (2019) tried to propose the PG methods based on the mirror descent algorithm.

Original: More recently, mirror descent policy optimization (MDPO) (Tomar et al., 2020) iteratively updates the policy beyond the tabular RL by approximately solving a trust region problem based on convex mirror descent algorithm.
Revised: More recently, mirror descent policy optimization (MDPO) (Tomar et al., 2020) iteratively updates the policy beyond the tabular RL by approximately solving a trust region problem based on convex mirror descent algorithm.

Original: In addition, Agarwal et al.
Revised: In addition, Agarwal et al.

Original: ; Cen et al.
Revised: ; Cen et al.

Original: (2020) have studied the natural PG methods for regularized RL.
Revised: (2020) have studied the natural PG methods for regularized RL.

Original: However, Agarwal et al.
Revised: However, Agarwal et al.

Original: (2019) mainly focuses on tabular policy and log-linear, neural policy classes.
Revised: (2019) mainly focuses on tabular policy and log-linear, neural policy classes.

Original: Cen et al.
Revised: Cen et al.

Original: (2020) mainly focuses on softmax policy class.
Revised: (2020) mainly focuses on softmax policy class.

Original: Although these speciﬁc PG methods based on mirror descent iteration have been recently studied, which are scattered in empirical and theoretical aspects respectively, it lacks a universal framework for these PG methods without relying on some speciﬁc RL tasks.
Revised: Although these speciﬁc PG methods based on mirror descent iteration have been recently studied, which are scattered in empirical and theoretical aspects respectively, it lacks a universal framework for these PG methods without relying on some speciﬁc RL tasks.

Original: In particular, there still does not exist the convergence analysis of PG methods based on the mirror descent algorithm under the nonconvex setting.
Revised: In particular, there still does not exist the convergence analysis of PG methods based on the mirror descent algorithm under the nonconvex setting.

Original: Since mirror descent iteration adjusts gradient updates to ﬁt problem geometry, and is useful in regularized RL (Geist et al., 2019), there exists an important problem to be addressed:
Revised: Since mirror descent iteration adjusts gradient updates to ﬁt problem geometry, and is useful in regularized RL (Geist et al., 2019), there exists an important problem to be addressed:

Original: Could we design a universal policy optimization framework based on the mirror descent algorithm, and provide its convergence guarantee under the non-convex setting ?
Revised: Could we design a universal policy optimization framework based on the mirror descent algorithm, and provide its convergence guarantee under the non-convex setting ?

Original: In the paper, we ﬁrmly answer the above challenging question with positive solutions and propose an efﬁcient Bregman gradient policy optimization framework based on Bregman divergences and momentum techniques.
Revised: In the paper, we ﬁrmly answer the above challenging question with positive solutions and propose an efﬁcient Bregman gradient policy optimization framework based on Bregman divergences and momentum techniques.

Original: In particular, we provide a convergence analysis framework of the PG methods based on mirror descent iteration under the nonconvex setting.
Revised: In particular, we provide a convergence analysis framework of the PG methods based on mirror descent iteration under the nonconvex setting.

Original: In summary, our main contributions are provided as follows:
Revised: In summary, our main contributions are provided as follows:

Original: a) We propose an effective Bregman gradient policy optimization (BGPO) algorithm based on the basic momentum technique, which achieves the sample complexity of O ( (cid:15) − 4 ) for ﬁnding (cid:15) -stationary policy only requiring one trajectory at each iteration.
Revised: a) We propose an effective Bregman gradient policy optimization (BGPO) algorithm based on the basic momentum technique, which achieves the sample complexity of O ( (cid:15) − 4 ) for ﬁnding (cid:15) -stationary policy only requiring one trajectory at each iteration.

Original: b) We propose an accelerated Bregman gradient policy optimization (VR-BGPO) algorithm based on the variance-reduced technique of STORM (Cutkosky & Orabona, 2019).
Revised: b) We propose an accelerated Bregman gradient policy optimization (VR-BGPO) algorithm based on the variance-reduced technique of STORM (Cutkosky & Orabona, 2019).

Original: Moreover, we prove that the VR-BGPO reaches the best known sample complexity of O ( (cid:15) − 3 ) .
Revised: Moreover, we prove that the VR-BGPO reaches the best known sample complexity of O ( (cid:15) − 3 ) .

Original: c)
Revised: c)

Original: We design a uniﬁed policy optimization framework based on mirror descent iteration and momentum techniques, and provide its convergence analysis under nonconvex setting.
Revised: We design a uniﬁed policy optimization framework based on mirror descent iteration and momentum techniques, and provide its convergence analysis under nonconvex setting.

Original: In Table 1 shows that sample complexities of the representative PG algorithms based on mirror descent algorithm.
Revised: In Table 1 shows that sample complexities of the representative PG algorithms based on mirror descent algorithm.

Original: Shani et al.
Revised: Shani et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2019) have established global convergence of a mirror descent variant of PG under some pre-speciﬁed setting such as over-parameterized networks (Liu et al., 2019) by exploiting these speciﬁc problems’ hidden convex nature.
Revised: (2019) have established global convergence of a mirror descent variant of PG under some pre-speciﬁed setting such as over-parameterized networks (Liu et al., 2019) by exploiting these speciﬁc problems’ hidden convex nature.

Original: Without these special structures, global convergence of these methods cannot be achieved.
Revised: Without these special structures, global convergence of these methods cannot be achieved.

Original: However, our framework does not rely on any speciﬁc policy classes, and our convergence analysis only builds on the general nonconvex setting.
Revised: However, our framework does not rely on any speciﬁc policy classes, and our convergence analysis only builds on the general nonconvex setting.

Original: Thus, we only prove that our methods convergence to stationary points.
Revised: Thus, we only prove that our methods convergence to stationary points.

Original: Geist et al.
Revised: Geist et al.

Original: ; Jin & Sidford (2020); Lan (2021); Zhan et al.
Revised: ; Jin & Sidford (2020); Lan (2021); Zhan et al.

Original: (2021) studied a general theory of regularized MDPs based on policy space such as a discrete probability space that generally is discontinuous.
Revised: (2021) studied a general theory of regularized MDPs based on policy space such as a discrete probability space that generally is discontinuous.

Original: Since both the state and action spaces S and A generally are very large in practice, the policy space is large.
Revised: Since both the state and action spaces S and A generally are very large in practice, the policy space is large.

Original: While our methods build on policy’ parameter space that is generally continuous Euclidean space and relatively small.
Revised: While our methods build on policy’ parameter space that is generally continuous Euclidean space and relatively small.

Original: Clearly, our methods and theoretical results are more practical than the results in (Geist et al., 2019; Jin & Sidford, 2020; Lan, 2021; Zhan et al., 2021).
Revised: Clearly, our methods and theoretical results are more practical than the results in (Geist et al., 2019; Jin & Sidford, 2020; Lan, 2021; Zhan et al., 2021).

Original: (Tomar et al., 2020) also proposes mirror descent PG framework based on policy parameter space, but it does not provide any theoretical results and only focuses on Bregman divergence taking form of KL divergence.
Revised: (Tomar et al., 2020) also proposes mirror descent PG framework based on policy parameter space, but it does not provide any theoretical results and only focuses on Bregman divergence taking form of KL divergence.

Original: While our framework can collaborate with any Bregman divergence forms.
Revised: While our framework can collaborate with any Bregman divergence forms.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: In this section, we review some related works about mirror descent-based algorithms in RL and variance-reduced PG methods, respectively.
Revised: In this section, we review some related works about mirror descent-based algorithms in RL and variance-reduced PG methods, respectively.

Original: 2. M IRROR D ESCENT A LGORITHM IN RL
Revised: 2. M IRROR D ESCENT A LGORITHM IN RL

Original: Due to easily deal with the regularization terms, mirror descent (a.k.a., Bregman gradient) algorithm (Censor & Zenios, 1992; Beck & Teboulle, 2003) has shown signiﬁcant successes in regularized RL, which is ﬁrst proposed in (Censor & Zenios, 1992) based on Bregman distance (divergence)
Revised: Due to easily deal with the regularization terms, mirror descent (a.k.a., Bregman gradient) algorithm (Censor & Zenios, 1992; Beck & Teboulle, 2003) has shown signiﬁcant successes in regularized RL, which is ﬁrst proposed in (Censor & Zenios, 1992) based on Bregman distance (divergence)

Original: (Bregman, 1967; Censor & Lent, 1981).
Revised: (Bregman, 1967; Censor & Lent, 1981).

Original: For example, Neu et al.
Revised: For example, Neu et al.

Original: (2017) have shown both the dynamic policy programming (Azar et al., 2012) and TRPO (Schulman et al., 2015) algorithms are approximate variants of mirror descent algorithm.
Revised: (2017) have shown both the dynamic policy programming (Azar et al., 2012) and TRPO (Schulman et al., 2015) algorithms are approximate variants of mirror descent algorithm.

Original: Subsequently, Geist et al.
Revised: Subsequently, Geist et al.

Original: (2019) have introduced a general theory of regularized MDPs based on the convex mirror descent algorithm.
Revised: (2019) have introduced a general theory of regularized MDPs based on the convex mirror descent algorithm.

Original: More recently,
Revised: More recently,

Original: Liu et al.
Revised: Liu et al.

Original: (2019) have studied the global convergence properties of PPO and TRPO equipped with overparametrized neural networks based on mirror descent iterations.
Revised: (2019) have studied the global convergence properties of PPO and TRPO equipped with overparametrized neural networks based on mirror descent iterations.

Original: At the same time, Shani et al.
Revised: At the same time, Shani et al.

Original: (2020) have analyzed the global convergence properties of TRPO in tabular policy based on the convex mirror descent algorithm.
Revised: (2020) have analyzed the global convergence properties of TRPO in tabular policy based on the convex mirror descent algorithm.

Original: Wang et al.
Revised: Wang et al.

Original: (2019) have proposed divergence augmented policy optimization for off-policy learning based on mirror descent algorithm.
Revised: (2019) have proposed divergence augmented policy optimization for off-policy learning based on mirror descent algorithm.

Original: MDPO (Tomar et al., 2020) iteratively updates the policy beyond the tabular RL by approximately solving a trust region problem based on convex mirror descent algorithm.
Revised: MDPO (Tomar et al., 2020) iteratively updates the policy beyond the tabular RL by approximately solving a trust region problem based on convex mirror descent algorithm.

Original: 2. (V ARIANCE -R EDUCED ) PG M ETHODS
Revised: 2. (V ARIANCE -R EDUCED ) PG M ETHODS

Original: PG methods have been widely studied due to their stability and incremental nature in policy optimization.
Revised: PG methods have been widely studied due to their stability and incremental nature in policy optimization.

Original: For example, the global convergence properties of vanilla policy gradient method in inﬁnite-horizon MDPs have been recently studied in (Zhang et al., 2019).
Revised: For example, the global convergence properties of vanilla policy gradient method in inﬁnite-horizon MDPs have been recently studied in (Zhang et al., 2019).

Original: Subsequently, Zhang et al.
Revised: Subsequently, Zhang et al.

Original: (2020) have studied asymptotically global convergence properties of the REINFORCE (Williams, 1992), whose policy gradient is approximated by using a single trajectory or a ﬁxed size mini-batch of trajectories under soft-max parametrization and log-barrier regularization.
Revised: (2020) have studied asymptotically global convergence properties of the REINFORCE (Williams, 1992), whose policy gradient is approximated by using a single trajectory or a ﬁxed size mini-batch of trajectories under soft-max parametrization and log-barrier regularization.

Original: To accelerate these vanilla PG methods, some faster variance-reduced PG methods have been proposed based on the variance-reduction techniques of SVRG (Johnson & Zhang, 2013), SPIDER (Fang et al., 2018) and
Revised: To accelerate these vanilla PG methods, some faster variance-reduced PG methods have been proposed based on the variance-reduction techniques of SVRG (Johnson & Zhang, 2013), SPIDER (Fang et al., 2018) and

Original: STORM (Cutkosky & Orabona, 2019) in stochastic optimization.
Revised: STORM (Cutkosky & Orabona, 2019) in stochastic optimization.

Original: For example, fast SVRPG (Papini et al., 2018; Xu et al., 2019a) algorithm have been proposed based on SVRG.
Revised: For example, fast SVRPG (Papini et al., 2018; Xu et al., 2019a) algorithm have been proposed based on SVRG.

Original: Fast HAPG (Shen et al., 2019) and SRVR-PG (Xu et al., 2019a) algorithms have been presented by using SPIDER technique.
Revised: Fast HAPG (Shen et al., 2019) and SRVR-PG (Xu et al., 2019a) algorithms have been presented by using SPIDER technique.

Original: Subsequently, the momentum-based PG methods, i.e., ProxHSPGA (Pham et al., 2020) and IS-MBPG (Huang et al., 2020), have been developed based on variance-reduced technique of
Revised: Subsequently, the momentum-based PG methods, i.e., ProxHSPGA (Pham et al., 2020) and IS-MBPG (Huang et al., 2020), have been developed based on variance-reduced technique of

Original: STORM/Hybrid-SGD (Cutkosky & Orabona, 2019; Tran-Dinh et al., 2019).
Revised: STORM/Hybrid-SGD (Cutkosky & Orabona, 2019; Tran-Dinh et al., 2019).

Original: More recently, (Ding et al., 2021) studied the global convergence of momentum-based policy gradient methods.
Revised: More recently, (Ding et al., 2021) studied the global convergence of momentum-based policy gradient methods.

Original: (Zhang et al., 2021) proposed a truncated stochastic incremental variance-reduced policy gradient (TSIVRPG) method to relieve the uncheckable importance weight assumption in above variance-reduced
Revised: (Zhang et al., 2021) proposed a truncated stochastic incremental variance-reduced policy gradient (TSIVRPG) method to relieve the uncheckable importance weight assumption in above variance-reduced

Original: PG methods and provided the global convergence of the TSIVR-PG under overparameterizaiton of policy assumption.
Revised: PG methods and provided the global convergence of the TSIVR-PG under overparameterizaiton of policy assumption.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In the section, we will review some preliminaries of Markov decision process and policy gradients.
Revised: In the section, we will review some preliminaries of Markov decision process and policy gradients.

Original: 3. N OTATIONS
Revised: 3. N OTATIONS

Original: Let [ n ] = { 1 , 2 , · · · , n } for all n ∈ N + .
Revised: Let [ n ] = { 1 , 2 , · · · , n } for all n ∈ N + .

Original: For a vector x ∈ R d , let (cid:107) x (cid:107) denote the (cid:96) 2 norm of x , and (cid:107) x (cid:107) p
Revised: For a vector x ∈ R d , let (cid:107) x (cid:107) denote the (cid:96) 2 norm of x , and (cid:107) x (cid:107) p

Original: = (cid:0) (cid:80) d i =

Revised: = (cid:0) (cid:80) d i =

Original: | x i | p (cid:1) 1 /p ( p ≥ 1) denotes the p -norm of x .
Revised: | x i | p (cid:1) 1 /p ( p ≥ 1) denotes the p -norm of x .

Original: For two sequences { a k } and { b k } , we denote a k
Revised: For two sequences { a k } and { b k } , we denote a k

Original: = O ( b k ) if a k ≤ Cb k for some constant C > 0 .
Revised: = O ( b k ) if a k ≤ Cb k for some constant C > 0 .

Original: E [ X ] and V [ X ] denote the expectation and variance of random variable X , respectively.
Revised: E [ X ] and V [ X ] denote the expectation and variance of random variable X , respectively.

Original: 3. M ARKOV D ECISION P ROCESS
Revised: 3. M ARKOV D ECISION P ROCESS

Original: Reinforcement learning generally involves a discrete time discounted Markov Decision Process (MDP) deﬁned by a tuple {S , A , P , r, γ, ρ 0 } .
Revised: Reinforcement learning generally involves a discrete time discounted Markov Decision Process (MDP) deﬁned by a tuple {S , A , P , r, γ, ρ 0 } .

Original: S and A denote the state and action spaces of the agent, respectively.
Revised: S and A denote the state and action spaces of the agent, respectively.

Original: P ( s (cid:48) | s, a ) : S ×
Revised: P ( s (cid:48) | s, a ) : S ×

Original: A → (cid:52) ( S ) is the Markov kernel that determines the transition probability from the state s to s (cid:48) under taking an action a ∈ A .
Revised: A → (cid:52) ( S ) is the Markov kernel that determines the transition probability from the state s to s (cid:48) under taking an action a ∈ A .

Original: r ( s, a ) :
Revised: r ( s, a ) :

Original: S ×A → [ − R, R ] ( R > 0) is the reward function of s and a , and ρ 0 = p ( s 0 ) denotes the initial state distribution.
Revised: S ×A → [ − R, R ] ( R > 0) is the reward function of s and a , and ρ 0 = p ( s 0 ) denotes the initial state distribution.

Original: γ ∈ (0 , 1) is the discount factor.
Revised: γ ∈ (0 , 1) is the discount factor.

Original: Let π : S → (cid:52) ( A ) be a stationary policy, where (cid:52) ( A ) is the set of probability distributions on A .
Revised: Let π : S → (cid:52) ( A ) be a stationary policy, where (cid:52) ( A ) is the set of probability distributions on A .

Original: Given the current state s t ∈ S , the agent executes an action a t ∈
Revised: Given the current state s t ∈ S , the agent executes an action a t ∈

Original: A following a conditional probability distribution π ( a t | s t ) , and then the agent obtains a reward r t = r
Revised: A following a conditional probability distribution π ( a t | s t ) , and then the agent obtains a reward r t = r

Original: ( s t , a t ) .
Revised: ( s t , a t ) .

Original: At each time t , we can deﬁne the state-action value function Q π ( s t , a t ) and state value function V π ( s t ) as follows:
Revised: At each time t , we can deﬁne the state-action value function Q π ( s t , a t ) and state value function V π ( s t ) as follows:

Original: [Equation]
Revised: [Equation]

Original: We also deﬁne the advantage function A π ( s t , a t ) =
Revised: We also deﬁne the advantage function A π ( s t , a t ) =

Original: Q π ( s t , a t ) − V π
Revised: Q π ( s t , a t ) − V π

Original: ( s t ) .
Revised: ( s t ) .

Original: The goal of the agent is to ﬁnd the optimal policy by maximizing the expected discounted reward
Revised: The goal of the agent is to ﬁnd the optimal policy by maximizing the expected discounted reward

Original: [Equation]
Revised: [Equation]

Original: Given a time horizon H , the agent collects a trajectory τ = { s t , a t } H − 1 t =0 under any stationary policy.
Revised: Given a time horizon H , the agent collects a trajectory τ = { s t , a t } H − 1 t =0 under any stationary policy.

Original: Then the agent obtains a cumulative discounted reward r ( τ ) =
Revised: Then the agent obtains a cumulative discounted reward r ( τ ) =

Original: (cid:80) H − 1 t =

Revised: (cid:80) H − 1 t =

Original: γ t r ( s t , a t ) .
Revised: γ t r ( s t , a t ) .

Original: Since the state and action spaces S and A are generally very large, directly solving the problem (2) is difﬁcult.
Revised: Since the state and action spaces S and A are generally very large, directly solving the problem (2) is difﬁcult.

Original: Thus, we let the policy π be parametrized as π θ for the parameter θ ∈ Θ ⊆ R d .
Revised: Thus, we let the policy π be parametrized as π θ for the parameter θ ∈ Θ ⊆ R d .

Original: Given the initial distribution ρ 0 = p ( s 0 ) , the probability distribution over trajectory τ can be obtained
Revised: Given the initial distribution ρ 0 = p ( s 0 ) , the probability distribution over trajectory τ can be obtained

Original: [Equation]
Revised: [Equation]

Original: Thus, the problem (2) will be equivalent to maximize the expected discounted trajectory reward:
Revised: Thus, the problem (2) will be equivalent to maximize the expected discounted trajectory reward:

Original: [Equation]
Revised: [Equation]

Original: In fact, the above objective function J ( θ ) has a truncation error of O ( γ H 1 − γ ) compared to the original inﬁnite-horizon MDP.
Revised: In fact, the above objective function J ( θ ) has a truncation error of O ( γ H 1 − γ ) compared to the original inﬁnite-horizon MDP.

Original: 3. P OLICY G RADIENTS
Revised: 3. P OLICY G RADIENTS

Original: The policy gradient methods (Williams, 1992; Sutton et al., 2000) are a class of effective policybased methods to solve the above RL problem (4).
Revised: The policy gradient methods (Williams, 1992; Sutton et al., 2000) are a class of effective policybased methods to solve the above RL problem (4).

Original: Speciﬁcally, the gradient of J ( θ ) with respect to θ is given as follows:
Revised: Speciﬁcally, the gradient of J ( θ ) with respect to θ is given as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Given a mini-batch trajectories B = { τ i } ni =1 sampled from the distribution p ( τ | θ ) , the standard stochastic policy gradient ascent update at ( k + 1) -th step, deﬁned as
Revised: Given a mini-batch trajectories B = { τ i } ni =1 sampled from the distribution p ( τ | θ ) , the standard stochastic policy gradient ascent update at ( k + 1) -th step, deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (6) where η > 0 is learning rate, and ∇ J B ( θ k ) = 1 n (cid:80) n i
Revised: (6) where η > 0 is learning rate, and ∇ J B ( θ k ) = 1 n (cid:80) n i

Original: =1 g ( τ i | θ k ) is stochastic policy gradient.
Revised: =1 g ( τ i | θ k ) is stochastic policy gradient.

Original: Given
Revised: Given

Original: H =
Revised: H =

Original: O ( 11 − γ ) as in (Zhang et al., 2019; Shani et al., 2020), g ( τ | θ ) is the unbiased stochastic policy gradient of J ( θ ) , i.e. , E [ g ( τ | θ )]
Revised: O ( 11 − γ ) as in (Zhang et al., 2019; Shani et al., 2020), g ( τ | θ ) is the unbiased stochastic policy gradient of J ( θ ) , i.e. , E [ g ( τ | θ )]

Original: = ∇ J ( θ ) , where
Revised: = ∇ J ( θ ) , where

Original: [Equation]
Revised: [Equation]

Original: Based on the gradient estimator in (7), we can obtain the existing well-known policy gradient estimators such as REINFORCE (Williams, 1992), policy gradient theorem (PGT (Sutton et al., 2000)) estimator .
Revised: Based on the gradient estimator in (7), we can obtain the existing well-known policy gradient estimators such as REINFORCE (Williams, 1992), policy gradient theorem (PGT (Sutton et al., 2000)) estimator .

Original: Speciﬁcally, the REINFORCE obtains a policy gradient estimator by adding a baseline b , deﬁned as
Revised: Speciﬁcally, the REINFORCE obtains a policy gradient estimator by adding a baseline b , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The PGT is a version of the REINFORCE, deﬁned as
Revised: The PGT is a version of the REINFORCE, deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  B REGMAN G RADIENT P OLICY O PTIMIZATION
Revised:  B REGMAN G RADIENT P OLICY O PTIMIZATION

Original: In this section, we propose a novel Bregman gradient policy optimization framework based on Bregman divergences and momentum techniques.
Revised: In this section, we propose a novel Bregman gradient policy optimization framework based on Bregman divergences and momentum techniques.

Original: We ﬁrst let f ( θ ) =
Revised: We ﬁrst let f ( θ ) =

Original: − J ( θ ) , the goal of policy-based
Revised: − J ( θ ) , the goal of policy-based

Original: RL is to solve the problem: max θ ∈ Θ J ( θ ) ⇐⇒ min θ ∈ Θ f ( θ ) , so we have ∇ f ( θ ) =
Revised: RL is to solve the problem: max θ ∈ Θ J ( θ ) ⇐⇒ min θ ∈ Θ f ( θ ) , so we have ∇ f ( θ ) =

Original: −∇ J ( θ ) .
Revised: −∇ J ( θ ) .

Original: Assume ψ ( x ) is a continuously-differentiable and ν -strongly convex function, i.e., (cid:104) x − y, ∇ ψ ( x ) − ∇ ψ ( y ) (cid:105) ≥ ν (cid:107) x − y (cid:107) , ν > 0 , we deﬁne a Bregman distance,
Revised: Assume ψ ( x ) is a continuously-differentiable and ν -strongly convex function, i.e., (cid:104) x − y, ∇ ψ ( x ) − ∇ ψ ( y ) (cid:105) ≥ ν (cid:107) x − y (cid:107) , ν > 0 , we deﬁne a Bregman distance:

Original: D ψ ( y, x ) =
Revised: D ψ ( y, x ) =

Original: [Equation]
Revised: [Equation]

Original: Then given a function h ( x ) deﬁned on a closed convex set X , we deﬁne a proximal operator (a.k.a., mirror descent),
Revised: Then given a function h ( x ) deﬁned on a closed convex set X , we deﬁne a proximal operator (a.k.a., mirror descent):

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 .
Revised: where λ > 0 .

Original: Based on this proximal operator P ψλ,h as in (Ghadimi et al., 2016; Zhang & He, 2018), we can deﬁne a Bregman gradient of function h ( x ) as follows:
Revised: Based on this proximal operator P ψλ,h as in (Ghadimi et al., 2016; Zhang & He, 2018), we can deﬁne a Bregman gradient of function h ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: If ψ ( x ) = 12 (cid:107) x (cid:107) 2 and X = R d , x ∗ is a stationary point of h ( x ) if and only if B ψλ,h ( x ∗ )
Revised: If ψ ( x ) = 12 (cid:107) x (cid:107) 2 and X = R d , x ∗ is a stationary point of h ( x ) if and only if B ψλ,h ( x ∗ )

Original: =
Revised: =

Original: ∇ h
Revised: ∇ h

Original: ( x ∗ )
Revised: ( x ∗ )

Original: = 0 .
Revised: = 0 .

Original: Thus, this Bregman gradient can be regarded as a generalized gradient.
Revised: Thus, this Bregman gradient can be regarded as a generalized gradient.

Original: 4. BGPO A LGORITHM
Revised: 4. BGPO A LGORITHM

Original: In the subsection, we propose a Bregman gradient policy optimization (BGPO) algorithm based on the basic momentum technique.
Revised: In the subsection, we propose a Bregman gradient policy optimization (BGPO) algorithm based on the basic momentum technique.

Original: The pseudo code of BGPO Algorithm is provided in Algorithm 1.
Revised: The pseudo code of BGPO Algorithm is provided in Algorithm 1.

Original: In Algorithm 1, the step 4 uses the stochastic Bregman gradient descent (a.k.a., stochastic mirror descent) to update the parameter θ .
Revised: In Algorithm 1, the step 4 uses the stochastic Bregman gradient descent (a.k.a., stochastic mirror descent) to update the parameter θ .

Original: Let h ( θ ) =
Revised: Let h ( θ ) =

Original: (cid:104) θ, u k (cid:105) be the ﬁrst-order approximation of function f ( θ ) at θ k , where u k is an approximated gradient of function f
Revised: (cid:104) θ, u k (cid:105) be the ﬁrst-order approximation of function f ( θ ) at θ k , where u k is an approximated gradient of function f

Original: ( θ ) at θ k .
Revised: ( θ ) at θ k .

Original: By the step 4 of Algorithm 1 and the above equality (10), we have
Revised: By the step 4 of Algorithm 1 and the above equality (10), we have

Original: [Equation]
Revised: [Equation]

Original: (11) where λ > 0 .
Revised: (11) where λ > 0 .

Original: Then by the step 5 of Algorithm 1, we have
Revised: Then by the step 5 of Algorithm 1, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where 0 < η k ≤ 1 .
Revised: where 0 < η k ≤ 1 .

Original: Due to the convexity of set Θ ⊆ R d and θ 1 ∈ Θ , we choose the parameter η k ∈ (0 , 1] to ensure the updated sequence { θ k } Kk =1 in Θ .
Revised: Due to the convexity of set Θ ⊆ R d and θ 1 ∈ Θ , we choose the parameter η k ∈ (0 , 1] to ensure the updated sequence { θ k } Kk =1 in Θ .

Original: In fact, our BGPO algorithm uniﬁes many popular policy optimization algorithms.
Revised: In fact, our BGPO algorithm uniﬁes many popular policy optimization algorithms.

Original: When the mirror mappings ψ k ( θ ) = 12 (cid:107) θ (cid:107) 2 for ∀ k ≥ 1 , then the update (12) is reduced to classic policy gradient algorithms (Sutton et al., 2000; Zhang et al., 2019; 2020).
Revised: When the mirror mappings ψ k ( θ ) = 12 (cid:107) θ (cid:107) 2 for ∀ k ≥ 1 , then the update (12) is reduced to classic policy gradient algorithms (Sutton et al., 2000; Zhang et al., 2019; 2020).

Original: In the other words, given u k = − g
Revised: In the other words, given u k = − g

Original: ( τ k | θ k ) and ψ k ( θ ) = 12 (cid:107) θ (cid:107) 2 , we have B ψ k λ,h
Revised: ( τ k | θ k ) and ψ k ( θ ) = 12 (cid:107) θ (cid:107) 2 , we have B ψ k λ,h

Original: ( θ k )
Revised: ( θ k )

Original: = − g ( τ k | θ k ) and
Revised: = − g ( τ k | θ k ) and

Original: [Equation]
Revised: [Equation]

Original: When the mirror mappings ψ k ( θ ) = 12 θ T F ( θ k ) θ with F ( θ k ) =
Revised: When the mirror mappings ψ k ( θ ) = 12 θ T F ( θ k ) θ with F ( θ k ) =

Original: E (cid:2) ∇ θ π θ k
Revised: E (cid:2) ∇ θ π θ k

Original: ( s a ) (cid:0) ∇ θ π θ k
Revised: ( s a ) (cid:0) ∇ θ π θ k

Original: ( s, a ) (cid:1) T (cid:3) , the update (12) is reduced to natural policy gradient algorithms (Kakade, 2001; Liu et al., 2020).
Revised: ( s, a ) (cid:1) T (cid:3) , the update (12) is reduced to natural policy gradient algorithms (Kakade, 2001; Liu et al., 2020).

Original: Similarly, given u k = − g ( τ k | θ k ) and ψ k ( θ ) = 12 θ T F ( θ k ) θ , we have B ψ k λ,h ( θ k )
Revised: Similarly, given u k = − g ( τ k | θ k ) and ψ k ( θ ) = 12 θ T F ( θ k ) θ , we have B ψ k λ,h ( θ k )

Original: = − F ( θ k )
Revised: = − F ( θ k )

Original: + g ( τ k | θ k ) and
Revised: + g ( τ k | θ k ) and

Original: [Equation]
Revised: [Equation]

Original: (14) where F ( θ k ) + denotes the Moore-Penrose pseudoinverse of the Fisher information matrix F ( θ k ) .
Revised: (14) where F ( θ k ) + denotes the Moore-Penrose pseudoinverse of the Fisher information matrix F ( θ k ) .

Original: When given the mirror mapping ψ k ( θ ) = (cid:80) s ∈S π θ ( s ) log( π θ ( s )) ,
Revised: When given the mirror mapping ψ k ( θ ) = (cid:80) s ∈S π θ ( s ) log( π θ ( s )) ,

Original: i.e., Boltzmann-Shannon entropy function (Shannon, 1948) and Θ = (cid:8) θ
Revised: i.e., Boltzmann-Shannon entropy function (Shannon, 1948) and Θ = (cid:8) θ

Original: ∈ R
Revised: ∈ R

Original: d | (cid:80) s ∈S π θ ( s )
Revised: d | (cid:80) s ∈S π θ ( s )

Original: = 1 (cid:9) , we have D ψ k
Revised: = 1 (cid:9) , we have D ψ k

Original: ( θ, θ k ) =
Revised: ( θ, θ k ) =

Original: KL (cid:0) π θ ( s ) , π θ k ( s ) (cid:1) =
Revised: KL (cid:0) π θ ( s ) , π θ k ( s ) (cid:1) =

Original: (cid:80) s ∈S π θ ( s ) log (cid:0)
Revised: (cid:80) s ∈S π θ ( s ) log (cid:0)

Original: π θ ( s ) π θk ( s ) (cid:1) , which is the KL divergence.
Revised: π θ ( s ) π θk ( s ) (cid:1) , which is the KL divergence.

Original: Thus, the update (12) is reduced to the mirror descent policy optimization (MDPO) (Tomar et al., 2020).
Revised: Thus, the update (12) is reduced to the mirror descent policy optimization (MDPO) (Tomar et al., 2020).

Original: 4. VR-BGPO A LGORITHM
Revised: 4. VR-BGPO A LGORITHM

Original: In the subsection, we propose a faster variance-reduced Bregman gradient policy optimization (VRBGPO) algorithm based on a variance-reduced technique.
Revised: In the subsection, we propose a faster variance-reduced Bregman gradient policy optimization (VRBGPO) algorithm based on a variance-reduced technique.

Original: The pseudo code of VR-BGPO algorithm is provided in Algorithm 2.
Revised: The pseudo code of VR-BGPO algorithm is provided in Algorithm 2.

Original: Consider the problem (4) is non-oblivious that the distribution p ( τ | θ ) depends on the variable θ varying through the whole optimization procedure, we apply the importance sampling weight (Papini et al., 2018; Xu et al., 2019a) in estimating our policy gradient u k +1 , deﬁned as
Revised: Consider the problem (4) is non-oblivious that the distribution p ( τ | θ ) depends on the variable θ varying through the whole optimization procedure, we apply the importance sampling weight (Papini et al., 2018; Xu et al., 2019a) in estimating our policy gradient u k +1 , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Except for different stochastic policy gradients { u k } and tuning parameters { η k , β k } using in Algorithms 1 and 2, the steps 4 and 5 in these algorithms for updating parameter θ are the same.
Revised: Except for different stochastic policy gradients { u k } and tuning parameters { η k , β k } using in Algorithms 1 and 2, the steps 4 and 5 in these algorithms for updating parameter θ are the same.

Original: Interestingly, when choosing mirror mapping ψ k ( θ ) = 12 (cid:107) θ (cid:107) 2 , our VR-BGPO algorithm will reduce to a non-adaptive version of IS-MBPG algorithm (Huang et al., 2020).
Revised: Interestingly, when choosing mirror mapping ψ k ( θ ) = 12 (cid:107) θ (cid:107) 2 , our VR-BGPO algorithm will reduce to a non-adaptive version of IS-MBPG algorithm (Huang et al., 2020).

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In this section, we will analyze the convergence properties of our BGPO and VR-BGPO algorithms.
Revised: In this section, we will analyze the convergence properties of our BGPO and VR-BGPO algorithms.

Original: All related proofs are provided in the Appendix A.
Revised: All related proofs are provided in the Appendix A.

Original: Here we use the standard convergence metric (cid:107)B ψ k λ, (cid:104) θ, ∇ f ( θ k ) (cid:105) ( θ ) (cid:107) used in (Zhang & He, 2018; Yang et al., 2019) to evaluate the convergence Bregman gradient-based (a.k.a., mirror descent) algorithms.
Revised: Here we use the standard convergence metric (cid:107)B ψ k λ, (cid:104) θ, ∇ f ( θ k ) (cid:105) ( θ ) (cid:107) used in (Zhang & He, 2018; Yang et al., 2019) to evaluate the convergence Bregman gradient-based (a.k.a., mirror descent) algorithms.

Original: To give the convergence analysis, we ﬁrst give some standard assumptions.
Revised: To give the convergence analysis, we ﬁrst give some standard assumptions.

Original: Assumption 1.
Revised: Assumption 1.

Original: For function log π θ ( a | s ) , its gradient and Hessian matrix are bounded, i.e., there exist constants C g , C h > 

Revised: For function log π θ ( a | s ) , its gradient and Hessian matrix are bounded, i.e., there exist constants C g , C h >

Original: such that (cid:107)∇ θ log π θ ( a | s ) (cid:107) ≤ C g , (cid:107)∇ 2 θ log π θ ( a | s ) (cid:107) ≤ C h .
Revised: such that (cid:107)∇ θ log π θ ( a | s ) (cid:107) ≤ C g , (cid:107)∇ 2 θ log π θ ( a | s ) (cid:107) ≤ C h .

Original: Assumption 2.
Revised: Assumption 2.

Original: Variance of stochastic gradient g ( τ | θ ) is bounded, i.e., there exists a constant σ > 0 , for all π θ such that V ( g ( τ | θ ))
Revised: Variance of stochastic gradient g ( τ | θ ) is bounded, i.e., there exists a constant σ > 0 , for all π θ such that V ( g ( τ | θ ))

Original: =
Revised: =

Original: E (cid:107) g ( τ | θ ) − ∇ J ( θ ) (cid:107) 2 ≤ σ 2 .
Revised: E (cid:107) g ( τ | θ ) − ∇ J ( θ ) (cid:107) 2 ≤ σ 2 .

Original: Assumption 3.
Revised: Assumption 3.

Original: For importance sampling weight w ( τ | θ 1 , θ 2 )
Revised: For importance sampling weight w ( τ | θ 1 , θ 2 )

Original: =
Revised: =

Original: p ( τ | θ 1 ) /p
Revised: p ( τ | θ 1 ) /p

Original: ( τ | θ 2 ) , its variance is bounded, i.e., there exists a constant W > 0 , it follows V ( w ( τ | θ 1 , θ 2 )) ≤ W for any θ 1 , θ 2 ∈ R d and τ ∼ p ( τ | θ 2 ) .
Revised: ( τ | θ 2 ) , its variance is bounded, i.e., there exists a constant W > 0 , it follows V ( w ( τ | θ 1 , θ 2 )) ≤ W for any θ 1 , θ 2 ∈ R d and τ ∼ p ( τ | θ 2 ) .

Original: Assumption 4.
Revised: Assumption 4.

Original: The function J ( θ ) has an upper bound in Θ , i.e., J ∗ = sup θ ∈ Θ J ( θ ) <
Revised: The function J ( θ ) has an upper bound in Θ , i.e., J ∗ = sup θ ∈ Θ J ( θ ) <

Original: + ∞ .
Revised: + ∞ .

Original: Assumptions 1 and 2 are commonly used in the PG algorithms (Papini et al., 2018;
Revised: Assumptions 1 and 2 are commonly used in the PG algorithms (Papini et al., 2018;

Original: Xu et al., 2019a;b).
Revised: Xu et al., 2019a;b).

Original: Assumption 3 is widely used in the study of variance reduced PG algorithms (Papini et al., 2018;
Revised: Assumption 3 is widely used in the study of variance reduced PG algorithms (Papini et al., 2018;

Original: Xu et al., 2019a).
Revised: Xu et al., 2019a).

Original: In fact, the bounded importance sampling weight might be violated in some cases such as using neural networks as the policy.
Revised: In fact, the bounded importance sampling weight might be violated in some cases such as using neural networks as the policy.

Original: Thus, we can clip this importance sampling weights to guarantee the effectiveness of our algorithms as in (Papini et al., 2018).
Revised: Thus, we can clip this importance sampling weights to guarantee the effectiveness of our algorithms as in (Papini et al., 2018).

Original: At the same time, the importance weights actually also have some nice properties, e.g., in soft-max policy it is bounded by e c (cid:107) θ 1 − θ 2 (cid:107) 2 for all θ 1 , θ 2 ∈ Θ .
Revised: At the same time, the importance weights actually also have some nice properties, e.g., in soft-max policy it is bounded by e c (cid:107) θ 1 − θ 2 (cid:107) 2 for all θ 1 , θ 2 ∈ Θ .

Original: More recently, (Zhang et al., 2021) used a simple truncated update to relieve this uncheckable importance weight assumption.
Revised: More recently, (Zhang et al., 2021) used a simple truncated update to relieve this uncheckable importance weight assumption.

Original: Assumption 4 guarantees the feasibility of the problem (4).
Revised: Assumption 4 guarantees the feasibility of the problem (4).

Original: Note that Assumptions 2 and 4 are satisﬁed automatically given Assumptionand the fact that all the rewards are bounded, i.e., | r ( s, a ) | ≤ R for any s ∈ S and a ∈ A .
Revised: Note that Assumptions 2 and 4 are satisﬁed automatically given Assumptionand the fact that all the rewards are bounded, i.e., | r ( s, a ) | ≤ R for any s ∈ S and a ∈ A .

Original: For example, due to | r ( s, a ) | ≤ R , we have | J ( θ ) | ≤ R 1 − γ .
Revised: For example, due to | r ( s, a ) | ≤ R , we have | J ( θ ) | ≤ R 1 − γ .

Original: So we have J ∗ =
Revised: So we have J ∗ =

Original: R 1 − γ .
Revised: R 1 − γ .

Original: 5. C ONVERGENCE A NALYSIS OF BGPO A LGORITHM
Revised: 5. C ONVERGENCE A NALYSIS OF BGPO A LGORITHM

Original: In the subsection, we provide convergence properties of the BGPO algorithm.
Revised: In the subsection, we provide convergence properties of the BGPO algorithm.

Original: The detailed proof is provided in Appendix A.1.
Revised: The detailed proof is provided in Appendix A.1.

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume the sequence { θ k } Kk =1 be generated from Algorithm 1.
Revised: Assume the sequence { θ k } Kk =1 be generated from Algorithm 1.

Original: Let η k
Revised: Let η k

Original: =
Revised: =

Original: b
Revised: b

Original: ( m + k ) 1 / 2 for all k ≥ 1 , 0 < λ ≤ νm 1 / 2 9 Lb , b > 0 , 8 Lλν ≤ c ≤ m 1 /
Revised: ( m + k ) 1 / 2 for all k ≥ 1 , 0 < λ ≤ νm 1 / 2 9 Lb , b > 0 , 8 Lλν ≤ c ≤ m 1 /

Original: 2 b , and m ≥ max { b 2 , ( cb ) 2 } , we have
Revised: 2 b , and m ≥ max { b 2 , ( cb ) 2 } , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where M = J ∗ − J ( θ 1 ) νλb + σ
Revised: where M = J ∗ − J ( θ 1 ) νλb + σ

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ( m + K ) .
Revised: ( m + K ) .

Original: Remark 1.
Revised: Remark 1.

Original: Without loss of generality, let b
Revised: Without loss of generality, let b

Original: = O (1) , m =
Revised: = O (1) , m =

Original: O (1) and λ
Revised: O (1) and λ

Original: = O (1) , we have M =
Revised: = O (1) , we have M =

Original: O (ln( m + K ))
Revised: O (ln( m + K ))

Original: =
Revised: =

Original: ˜ O (1) .
Revised: ˜ O (1) .

Original: Theorem 1 shows that the BGPO algorithm has a convergence rate of ˜ O
Revised: Theorem 1 shows that the BGPO algorithm has a convergence rate of ˜ O

Original: ( 1 K 1 / 4 ) .
Revised: ( 1 K 1 / 4 ) .

Original: Let K − 14 ≤ (cid:15) , we have K =
Revised: Let K − 14 ≤ (cid:15) , we have K =

Original: ˜ O
Revised: ˜ O

Original: ( (cid:15) − 4 ) .
Revised: ( (cid:15) − 4 ) .

Original: Since the BGPO algorithm only needs one trajectory to estimate the stochastic policy gradient at each iteration and runs K iterations, it has the sample complexity of 1 · K
Revised: Since the BGPO algorithm only needs one trajectory to estimate the stochastic policy gradient at each iteration and runs K iterations, it has the sample complexity of 1 · K

Original: = O ( (cid:15) − 4 ) for ﬁnding an (cid:15) -stationary point.
Revised: = O ( (cid:15) − 4 ) for ﬁnding an (cid:15) -stationary point.

Original: 5. C ONVERGENCE A NALYSIS OF VR-BGPO A LGORITHM
Revised: 5. C ONVERGENCE A NALYSIS OF VR-BGPO A LGORITHM

Original: In the subsection, we give convergence properties of the VR-BGPO algorithm.
Revised: In the subsection, we give convergence properties of the VR-BGPO algorithm.

Original: The detailed proof is provided in Appendix A.2.
Revised: The detailed proof is provided in Appendix A.2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose the sequence { θ k } Kk =1 be generated from Algorithm 2.
Revised: Suppose the sequence { θ k } Kk =1 be generated from Algorithm 2.

Original: Let η k
Revised: Let η k

Original: =
Revised: =

Original: b
Revised: b

Original: ( m + k ) 1 / 3 for all k ≥ 0 , 0 < λ ≤ νm 1 / 3 5ˆ
Revised: ( m + k ) 1 / 3 for all k ≥ 0 , 0 < λ ≤ νm 1 / 3 5ˆ

Original: Lb , b > 0 , c ∈ (cid:2) 

Revised: Lb , b > 0 , c ∈ (cid:2)

Original: b 3 + 20ˆ L 2 λ 2 ν, m 2 / 3 b 2 (cid:3) and m ≥ max (cid:0) 2 , b 3 , ( cb ) 3 , ( 56 b ) 2 / 3 (cid:1) , we have
Revised: b 3 + 20ˆ L 2 λ 2 ν, m 2 / 3 b 2 (cid:3) and m ≥ max (cid:0) 2 , b 3 , ( cb ) 3 , ( 56 b ) 2 / 3 (cid:1) , we have

Original: [Equation]
Revised: [Equation]

Original: ) where M (cid:48)
Revised: ) where M (cid:48)

Original: =
Revised: =

Original: J ∗ − J ( θ 1 ) bνλ
Revised: J ∗ − J ( θ 1 ) bνλ

Original: + m 1 / 3 σ 2 16 b 2 ˆ L 2 λ 2 + c 2 σ 2 b 2 8ˆ L 2 λ 2 , ˆ L 2 =
Revised: + m 1 / 3 σ 2 16 b 2 ˆ L 2 λ 2 + c 2 σ 2 b 2 8ˆ L 2 λ 2 , ˆ L 2 =

Original: L 2 + 2 G 2 C 2 w , G
Revised: L 2 + 2 G 2 C 2 w , G

Original: =
Revised: =

Original: C g
Revised: C g

Original: R/ (1 − γ ) 2 and C w = (cid:113) H (2 HC 2 g + C h )( W + 1) .
Revised: R/ (1 − γ ) 2 and C w = (cid:113) H (2 HC 2 g + C h )( W + 1) .

Original: Remark 2.
Revised: Remark 2.

Original: Without loss of generality, let b
Revised: Without loss of generality, let b

Original: = O (1) , m =
Revised: = O (1) , m =

Original: O (1) and λ
Revised: O (1) and λ

Original: = O (1) , we have M =
Revised: = O (1) , we have M =

Original: O (ln( m + K ))
Revised: O (ln( m + K ))

Original: =
Revised: =

Original: ˜ O (1) .
Revised: ˜ O (1) .

Original: Theorem 2 shows that the VR-BGPO algorithm has a convergence rate of ˜ O
Revised: Theorem 2 shows that the VR-BGPO algorithm has a convergence rate of ˜ O

Original: ( 1 K 1 / 3 ) .
Revised: ( 1 K 1 / 3 ) .

Original: Let K − 13 ≤ (cid:15) , we have K =
Revised: Let K − 13 ≤ (cid:15) , we have K =

Original: ( (cid:15) − 3 ) .
Revised: ( (cid:15) − 3 ) .

Original: Since the VR-BGPO algorithm only needs one trajectory to estimate the stochastic policy gradient at each iteration and runs K iterations, it reaches a lower sample complexity of 1 · K = ˜
Revised: Since the VR-BGPO algorithm only needs one trajectory to estimate the stochastic policy gradient at each iteration and runs K iterations, it reaches a lower sample complexity of 1 · K = ˜

Original: O ( (cid:15) − 3 ) for ﬁnding an (cid:15) -stationary point.
Revised: O ( (cid:15) − 3 ) for ﬁnding an (cid:15) -stationary point.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct some RL tasks to verify the effectiveness of our methods.
Revised: In this section, we conduct some RL tasks to verify the effectiveness of our methods.

Original: We ﬁrst study the effect of different choices of Bregman divergences with our algorithms (BGPO and VRBGPO), and then we compare our VR-BGPO algorithm with other state-of-the-art methods such as TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017), ProxHSPGA (Pham et al., 2020),
Revised: We ﬁrst study the effect of different choices of Bregman divergences with our algorithms (BGPO and VRBGPO), and then we compare our VR-BGPO algorithm with other state-of-the-art methods such as TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017), ProxHSPGA (Pham et al., 2020),

Original: VRMPO (Yang et al., 2019), and MDPO (Tomar et al., 2020).
Revised: VRMPO (Yang et al., 2019), and MDPO (Tomar et al., 2020).

Original: Our code is available at https: //github.com/gaosh/BGPO .
Revised: Our code is available at https: //github.com/gaosh/BGPO .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 6. E FFECTS OF B REGMAN D IVERGENCES
Revised: 6. E FFECTS OF B REGMAN D IVERGENCES

Original: In the subsection, we examine how different Bregman divergences affect the performance of our algorithms.
Revised: In the subsection, we examine how different Bregman divergences affect the performance of our algorithms.

Original: In the ﬁrst setting, we let mirror mapping ψ k ( x ) = (cid:107) x (cid:107) p
Revised: In the ﬁrst setting, we let mirror mapping ψ k ( x ) = (cid:107) x (cid:107) p

Original: ( p ≥ 1) with different p to test the performance our algorithms.
Revised: ( p ≥ 1) with different p to test the performance our algorithms.

Original: Let ψ ∗ k
Revised: Let ψ ∗ k

Original: ( y )
Revised: ( y )

Original: =
Revised: =

Original: ( (cid:80) di =1 | y i | q ) 1 q be the conjugate mapping of ψ k ( x ) , where p − 1 + q − 1 = 1 , p, q > 1 .
Revised: ( (cid:80) di =1 | y i | q ) 1 q be the conjugate mapping of ψ k ( x ) , where p − 1 + q − 1 = 1 , p, q > 1 .

Original: According to (Beck & Teboulle, 2003), when Θ = R d , the update of ˜ θ k +1 in our algorithms can be calculated by ˜ θ k +

Revised: According to (Beck & Teboulle, 2003), when Θ = R d , the update of ˜ θ k +1 in our algorithms can be calculated by ˜ θ k +

Original: =
Revised: =

Original: ∇ ψ ∗ k ( ∇ ψ k ( θ k )
Revised: ∇ ψ ∗ k ( ∇ ψ k ( θ k )

Original: + λu k ) , where ∇ ψ k ( x j ) and ∇ ψ ∗ k
Revised: + λu k ) , where ∇ ψ k ( x j ) and ∇ ψ ∗ k

Original: ( y j ) are p -norm link functions, and ∇ ψ k ( x j ) =
Revised: ( y j ) are p -norm link functions, and ∇ ψ k ( x j ) =

Original: sign ( x j ) | x j | p − 1 (cid:107) x (cid:107) p − 2 p , ∇ ψ ∗ k ( y j ) =
Revised: sign ( x j ) | x j | p − 1 (cid:107) x (cid:107) p − 2 p , ∇ ψ ∗ k ( y j ) =

Original: sign ( y j ) | y j | q −(cid:107) y (cid:107)
Revised: sign ( y j ) | y j | q −(cid:107) y (cid:107)

Original: q − 2 q , and j is the coordinate index of x and y .
Revised: q − 2 q , and j is the coordinate index of x and y .

Original: In the second setting, we apply diagonal term on the mirror mapping ψ k ( x ) = 12 x T M k x , where M k is a diagonal matrix with positive values.
Revised: In the second setting, we apply diagonal term on the mirror mapping ψ k ( x ) = 12 x T M k x , where M k is a diagonal matrix with positive values.

Original: In the experiments, we generate H k = diag ( √ v k + α ) , v k = βv
Revised: In the experiments, we generate H k = diag ( √ v k + α ) , v k = βv

Original: k − 1 + (1 − β ) u 2 k , and α > 0 , β ∈ (0 , 1) , as in Super-Adam algorithm (Kingma & Ba, 2014; Huang et al., 2021).
Revised: k − 1 + (1 − β ) u 2 k , and α > 0 , β ∈ (0 , 1) , as in Super-Adam algorithm (Kingma & Ba, 2014; Huang et al., 2021).

Original: Then we have D ψ k ( y, x ) =
Revised: Then we have D ψ k ( y, x ) =

Original: ( y − x ) T H k ( y − x ) .
Revised: ( y − x ) T H k ( y − x ) .

Original: Under this setting, the update of ˜ θ k +1 can also be analytically solved ˜ θ k +

Revised: Under this setting, the update of ˜ θ k +1 can also be analytically solved ˜ θ k +

Original: = θ k − λH − 1 k u k .
Revised: = θ k − λH − 1 k u k .

Original: To test the effectiveness of two different Bregman divergences, we evaluate them on three classic control environments from gym Brockman et al.
Revised: To test the effectiveness of two different Bregman divergences, we evaluate them on three classic control environments from gym Brockman et al.

Original: (2016): CartPole-v1, Acrobat-v1, and
Revised: (2016): CartPole-v1, Acrobat-v1, and

Original: MountainCarContinuous-v0.
Revised: MountainCarContinuous-v0.

Original: In the experiment, categorical policy is used for CartPole and Acrobot environments, and Gaussian policy is used for MountainCar.
Revised: In the experiment, categorical policy is used for CartPole and Acrobot environments, and Gaussian policy is used for MountainCar.

Original: Gaussian value functions are used in all settings.
Revised: Gaussian value functions are used in all settings.

Original: All policies and value functions are parameterized by multilayer perceptrons (MLPs).
Revised: All policies and value functions are parameterized by multilayer perceptrons (MLPs).

Original: For a fair comparison, all settings use the same initialization for policies.
Revised: For a fair comparison, all settings use the same initialization for policies.

Original: We run each setting ﬁve times and plot the mean and variance of average returns.
Revised: We run each setting ﬁve times and plot the mean and variance of average returns.

Original: For l p -norm mapping, we test three different values of p = (1 . 50 , 2 . 0 , 3 . 0) .
Revised: For l p -norm mapping, we test three different values of p = (1 . 50 , 2 . 0 , 3 . 0) .

Original: For diagonal mapping, we set β = 0 .
Revised: For diagonal mapping, we set β = 0 .

Original: 999 and α = 10 − 8 .
Revised: 999 and α = 10 − 8 .

Original: We set hyperparameters { b, m, c } to be the same.
Revised: We set hyperparameters { b, m, c } to be the same.

Original: λ still needs to be tuned for different p to achieve relatively good performance.
Revised: λ still needs to be tuned for different p to achieve relatively good performance.

Original: For simplicity, we use BGPO-Diag to represent BGPO with diagonal mapping, and we use BGPO- l p to represent BGPO with l p -norm mapping.
Revised: For simplicity, we use BGPO-Diag to represent BGPO with diagonal mapping, and we use BGPO- l p to represent BGPO with l p -norm mapping.

Original: Details about the setup of environments and hyperparameters are provided in the Appendix C.
Revised: Details about the setup of environments and hyperparameters are provided in the Appendix C.

Original: From Fig.
Revised: From Fig.

Original: 1, we can ﬁnd that BGPO-Diag largely outperforms BGPO-
Revised: 1, we can ﬁnd that BGPO-Diag largely outperforms BGPO-

Original: l p with different choices of p .
Revised: l p with different choices of p .

Original: The parameter tuning of BGPO- l p is much more difﬁcult than BGPO-Diag because each p requires an individual λ to achieve the desired performance.
Revised: The parameter tuning of BGPO- l p is much more difﬁcult than BGPO-Diag because each p requires an individual λ to achieve the desired performance.

Original: 6. C OMPARISON BETWEEN BGPO AND VR-BGPO
Revised: 6. C OMPARISON BETWEEN BGPO AND VR-BGPO

Original: To understand the effectiveness of variance reduced technique used in our VR-BGPO algorithm, we compare BGPO and VR-BGPO using the same settings introduced in section.
Revised: To understand the effectiveness of variance reduced technique used in our VR-BGPO algorithm, we compare BGPO and VR-BGPO using the same settings introduced in section.

Original: Both algorithms use the diagonal mapping for ψ , since it performs much better than l p -norm.
Revised: Both algorithms use the diagonal mapping for ψ , since it performs much better than l p -norm.

Original: From Fig.
Revised: From Fig.

Original: 2 given in the Appendix C, we can see that VR-BGPO can outperform BGPO in all three environments.
Revised: 2 given in the Appendix C, we can see that VR-BGPO can outperform BGPO in all three environments.

Original: [Figure]
Revised: [Figure]

Original: In CartPole, both algorithms converge very fast and have similar performance, and VR-BGPO is more stable than BGPO.
Revised: In CartPole, both algorithms converge very fast and have similar performance, and VR-BGPO is more stable than BGPO.

Original: The advantage of VR-BGPO becomes large in Acrobot and MountainCar environments, probably because the task is more difﬁcult compared to CartPole.
Revised: The advantage of VR-BGPO becomes large in Acrobot and MountainCar environments, probably because the task is more difﬁcult compared to CartPole.

Original: 6. C OMPARE TO OTHER M ETHODS
Revised: 6. C OMPARE TO OTHER M ETHODS

Original: In this subsection, we apply our BGPO and VR-BGPO algorithms to compare with the other methods.
Revised: In this subsection, we apply our BGPO and VR-BGPO algorithms to compare with the other methods.

Original: For our BGPO and VR-BGPO, we use diagonal mapping for ψ .
Revised: For our BGPO and VR-BGPO, we use diagonal mapping for ψ .

Original: For VRMPO, we follow their implementation and use l p -norm for ψ .
Revised: For VRMPO, we follow their implementation and use l p -norm for ψ .

Original: For MDPO, ψ is the negative Shannon entropy, and the
Revised: For MDPO, ψ is the negative Shannon entropy, and the

Original: Bregman divergence becomes KL-divergence.
Revised: Bregman divergence becomes KL-divergence.

Original: To evaluate the performance of these algorithms, we test them on six gym (Brockman et al., 2016) environments with continuous control tasks: Inverted-DoublePendulum-v2, Walker2d-v2, Reacherv2, Swimmer-v2, Inverted-Pendulum-v2 and HalfCheetah-v2.
Revised: To evaluate the performance of these algorithms, we test them on six gym (Brockman et al., 2016) environments with continuous control tasks: Inverted-DoublePendulum-v2, Walker2d-v2, Reacherv2, Swimmer-v2, Inverted-Pendulum-v2 and HalfCheetah-v2.

Original: We use Gaussian policies and Gaussian value functions for all environments, and both of them are parameterized by MLPs.
Revised: We use Gaussian policies and Gaussian value functions for all environments, and both of them are parameterized by MLPs.

Original: To ensure a fair comparison, all policies use the same initialization.
Revised: To ensure a fair comparison, all policies use the same initialization.

Original: For TRPO and PPO, we use the implementations provided by garage (garage contributors, 2019).
Revised: For TRPO and PPO, we use the implementations provided by garage (garage contributors, 2019).

Original: We carefully implement MDPO and VRMPO following the description provided by the original papers.
Revised: We carefully implement MDPO and VRMPO following the description provided by the original papers.

Original: All methods include our method, are implemented with garage (garage contributors, 2019) and pytorch (Paszke et al., 2019).
Revised: All methods include our method, are implemented with garage (garage contributors, 2019) and pytorch (Paszke et al., 2019).

Original: We run all algorithms ten times on each environment and report the mean and variance of average returns.
Revised: We run all algorithms ten times on each environment and report the mean and variance of average returns.

Original: Details about the setup of environments and hyperparameters are also provided in the Appendix C.
Revised: Details about the setup of environments and hyperparameters are also provided in the Appendix C.

Original: From Fig.
Revised: From Fig.

Original: 3, we can ﬁnd that our VR-BGPO method consistently outperforms all the other methods.
Revised: 3, we can ﬁnd that our VR-BGPO method consistently outperforms all the other methods.

Original: Our BGPO basically reaches the second best performances.
Revised: Our BGPO basically reaches the second best performances.

Original: From the results of our BGPO, we can ﬁnd that using a proper Bregman (mirror) distance can improve performances of the PG methods.
Revised: From the results of our BGPO, we can ﬁnd that using a proper Bregman (mirror) distance can improve performances of the PG methods.

Original: From the results of our VR-BGPO, we can ﬁnd that using a proper variance-reduced technique can further improve performances of the BGPO.
Revised: From the results of our VR-BGPO, we can ﬁnd that using a proper variance-reduced technique can further improve performances of the BGPO.

Original: ProxHSPGA can reach some relatively good performances by using the variance reduced technique.
Revised: ProxHSPGA can reach some relatively good performances by using the variance reduced technique.

Original: MDPO can achieve good results in some environments, but it can not outperform PPO or TRPO in Swimmer and InvertedDoublePendulum.
Revised: MDPO can achieve good results in some environments, but it can not outperform PPO or TRPO in Swimmer and InvertedDoublePendulum.

Original: VRMPO only outperforms PPO and TRPO in Reacher and InvertedDoublePendulum.
Revised: VRMPO only outperforms PPO and TRPO in Reacher and InvertedDoublePendulum.

Original: The undesirable performance of VRMPO is probably because it uses l p norm for ψ , which requires careful tuning of learning rate.
Revised: The undesirable performance of VRMPO is probably because it uses l p norm for ψ , which requires careful tuning of learning rate.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In the paper, we proposed a novel Bregman gradient policy optimization framework for reinforcement learning based on Bregman divergences and momentum techniques.
Revised: In the paper, we proposed a novel Bregman gradient policy optimization framework for reinforcement learning based on Bregman divergences and momentum techniques.

Original: Moreover, we studied convergence properties of the proposed methods under the nonconvex setting.
Revised: Moreover, we studied convergence properties of the proposed methods under the nonconvex setting.

Original: A CKNOWLEDGMENT
Revised: A CKNOWLEDGMENT

Original: This work was partially supported by NSF IIS 1845666, 1852606, 1838627, 1837956, 1956002,
Revised: This work was partially supported by NSF IIS 1845666, 1852606, 1838627, 1837956, 1956002,

Original: OIA 2040588.
Revised: OIA 2040588.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In this section, we study the convergence properties of our algorithms.
Revised: In this section, we study the convergence properties of our algorithms.

Original: We ﬁrst provide some useful lemmas.
Revised: We ﬁrst provide some useful lemmas.

Original: Lemma 1.
Revised: Lemma 1.

Original: (Proposition 4.2 in Xu et al. (2019b))
Revised: (Proposition 4.2 in Xu et al. (2019b))

Original: Suppose g ( τ | θ ) is the PGT estimator.
Revised: Suppose g ( τ | θ ) is the PGT estimator.

Original: Under
Revised: Under

Original: Assumption 1, we have
Revised: Assumption 1, we have

Original: 1) g ( τ | θ ) is L -Lipschitz differential, i.e., (cid:107) g
Revised: 1) g ( τ | θ ) is L -Lipschitz differential, i.e., (cid:107) g

Original: ( τ |
Revised: ( τ |

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) (cid:107) ≤ L (cid:107)
Revised: ) (cid:107) ≤ L (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:107) for all θ 1 , θ 2 ∈ Θ , where L = C h
Revised: 2 (cid:107) for all θ 1 , θ 2 ∈ Θ , where L = C h

Original: R/ (1 − γ ) 2 ; 2)
Revised: R/ (1 − γ ) 2 ; 2)

Original: J ( θ ) is L -smooth, i.e., (cid:107)∇ 2 J ( θ ) (cid:107) ≤ L ; 3) g
Revised: J ( θ ) is L -smooth, i.e., (cid:107)∇ 2 J ( θ ) (cid:107) ≤ L ; 3) g

Original: ( τ | θ ) is bounded, i
Revised: ( τ | θ ) is bounded, i

Original: e., (cid:107) g ( τ | θ ) (cid:107) ≤ G for all θ ∈ Θ with G = C g
Revised: e., (cid:107) g ( τ | θ ) (cid:107) ≤ G for all θ ∈ Θ with G = C g

Original: R/
Revised: R/

Original: (1 − γ ) 2 .
Revised: (1 − γ ) 2 .

Original: Lemma 2.
Revised: Lemma 2.

Original: (Lemma 6.1 in Xu et al. (2019a))
Revised: (Lemma 6.1 in Xu et al. (2019a))

Original: Under Assumptions 1 and 3, let w ( τ | θ k − 1 , θ k ) =
Revised: Under Assumptions 1 and 3, let w ( τ | θ k − 1 , θ k ) =

Original: g ( τ | θ k − 1 ) /g
Revised: g ( τ | θ k − 1 ) /g

Original: ( τ | θ k ) , we have
Revised: ( τ | θ k ) , we have

Original: [Equation]
Revised: [Equation]

Original: where C w = (cid:113) H (2 HC 2 g + C h )( W + 1) .
Revised: where C w = (cid:113) H (2 HC 2 g + C h )( W + 1) .

Original: Lemma 3.
Revised: Lemma 3.

Original: (Lemma 1 in (Ghadimi et al., 2016))
Revised: (Lemma 1 in (Ghadimi et al., 2016))

Original: Let X
Revised: Let X

Original: ⊆
Revised: ⊆

Original: R d be a closed convex set, and φ : X → R be a convex function but possibly nonsmooth, and D ψ : X × X
Revised: R d be a closed convex set, and φ : X → R be a convex function but possibly nonsmooth, and D ψ : X × X

Original: → R is Bregman divergence related to the ν -strongly convex function ψ .
Revised: → R is Bregman divergence related to the ν -strongly convex function ψ .

Original: Then we deﬁne
Revised: Then we deﬁne

Original: [Equation]
Revised: [Equation]

Original: where g ∈ R d , λ > 0 and D ψ ( z, x ) = ψ ( z ) − (cid:0) ψ
Revised: where g ∈ R d , λ > 0 and D ψ ( z, x ) = ψ ( z ) − (cid:0) ψ

Original: ( x ) + (cid:104)∇ ψ ( x ) , z − x (cid:105) (cid:1) .
Revised: ( x ) + (cid:104)∇ ψ ( x ) , z − x (cid:105) (cid:1) .

Original: Then the following statement holds
Revised: Then the following statement holds

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: (Proposition 1 in Ghadimi et al.
Revised: (Proposition 1 in Ghadimi et al.

Original: Let x +1 and x +2 be given in (17) with g replaced by g 1 and g 2 respectively.
Revised: Let x +1 and x +2 be given in (17) with g replaced by g 1 and g 2 respectively.

Original: Then let P X ( x, g 1 , λ ) and P X ( x, g 2 , λ ) be deﬁned in (18) with x + replaced by x +1 and x +2 respectively.
Revised: Then let P X ( x, g 1 , λ ) and P X ( x, g 2 , λ ) be deﬁned in (18) with x + replaced by x +1 and x +2 respectively.

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 5.
Revised: Lemma 5.

Original: (Lemma 1 in (Cortes et al., 2010)) Let w ( x ) =
Revised: (Lemma 1 in (Cortes et al., 2010)) Let w ( x ) =

Original: P ( x ) Q ( x ) be the importance weight for distributions P and Q .
Revised: P ( x ) Q ( x ) be the importance weight for distributions P and Q .

Original: The following identities hold for the expectation, second moment, and variance of w ( x )
Revised: The following identities hold for the expectation, second moment, and variance of w ( x )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where d 2 ( P || Q ) = 2 D ( P || Q ) , and D ( P || Q ) is R ´ enyi divergence between distributions P and Q .
Revised: where d 2 ( P || Q ) = 2 D ( P || Q ) , and D ( P || Q ) is R ´ enyi divergence between distributions P and Q .

Original: Lemma 6.
Revised: Lemma 6.

Original: Suppose that the sequence { θ k } Kk =1 be generated from Algorithms 1 or 2.
Revised: Suppose that the sequence { θ k } Kk =1 be generated from Algorithms 1 or 2.

Original: Let 0 <
Revised: Let 0 <

Original: η k ≤and 0 < λ ≤ ν 2 Lη k , then we have
Revised: η k ≤and 0 < λ ≤ ν 2 Lη k , then we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to Assumption 1 and Lemma
Revised: According to Assumption 1 and Lemma

Original: [Equation]
Revised: [Equation]

Original: the function f ( θ ) is L -smooth.
Revised: the function f ( θ ) is L -smooth.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where the second equality is due to θ k
Revised: where the second equality is due to θ k

Original: +

Revised: +

Original: =
Revised: =

Original: θ k + η k
Revised: θ k + η k

Original: (˜ θ k +1 − θ k ) .
Revised: (˜ θ k +1 − θ k ) .

Original: By the step 4 of Algorithm 1 or 2, we have ˜ θ k +1 = arg
Revised: By the step 4 of Algorithm 1 or 2, we have ˜ θ k +1 = arg

Original: [Equation]
Revised: [Equation]

Original: Θ
Revised: Θ

Original: [Equation]
Revised: [Equation]

Original: u k , θ (cid:105) +
Revised: u k , θ (cid:105) +

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: ψ k
Revised: ψ k

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: θ k ) (cid:9) .
Revised: θ k ) (cid:9) .

Original: By using Lemma 3 with φ ( · ) = 0 , we have
Revised: By using Lemma 3 with φ ( · ) = 0 , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: According to the Cauchy-Schwarz inequality and Young’s inequality
Revised: According to the Cauchy-Schwarz inequality and Young’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Combining the inequalities (23), (25) with (26), we obtain f
Revised: Combining the inequalities (23), (25) with (26), we obtain f

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to 0 < λ ≤
Revised: where the last inequality is due to 0 < λ ≤

Original: [Equation]
Revised: [Equation]

Original: A. C ONVERGENCE A NALYSIS OF BGPO A LGORITHM
Revised: A. C ONVERGENCE A NALYSIS OF BGPO A LGORITHM

Original: In this subsection
Revised: In this subsection

Original: [Equation]
Revised: [Equation]

Original: we analyze the convergence properties of BGPO algorithm.
Revised: we analyze the convergence properties of BGPO algorithm.

Original: Lemma 7.
Revised: Lemma 7.

Original: Assume the stochastic policy gradient u k +1 be generated from Algorithm
Revised: Assume the stochastic policy gradient u k +1 be generated from Algorithm

Original: [Equation]
Revised: [Equation]

Original: given 0 < β k ≤ 1 , we have
Revised: given 0 < β k ≤ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the deﬁnition of u k +1 in Algorithm
Revised: By the deﬁnition of u k +1 in Algorithm

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since ∇ f ( θ k ) = − J ( θ k ) for all k ≥ 1 , we have
Revised: Since ∇ f ( θ k ) = − J ( θ k ) for all k ≥ 1 , we have

Original: E (cid:107)∇ f ( θ k +1 )
Revised: E (cid:107)∇ f ( θ k +1 )

Original: − u k +1 (cid:107)= E (cid:107) − ∇ J
Revised: − u k +1 (cid:107)= E (cid:107) − ∇ J

Original: ( θ k ) − u k − ∇ J ( θ k +1 )
Revised: ( θ k ) − u k − ∇ J ( θ k +1 )

Original: + ∇ J ( θ k ) − (
Revised: + ∇ J ( θ k ) − (

Original: u k +1 − u k ) (cid:107)= E (cid:107) − ∇ J ( θ k ) − u k − ∇ J ( θ k +1 )
Revised: u k +1 − u k ) (cid:107)= E (cid:107) − ∇ J ( θ k ) − u k − ∇ J ( θ k +1 )

Original: + ∇ J ( θ k )
Revised: + ∇ J ( θ k )

Original: +
Revised: +

Original: β k +

Revised: β k +

Original: u k + β k +1 g
Revised: u k + β k +1 g

Original: ( τ k +1 | θ k +1 )
Revised: ( τ k +1 | θ k +1 )

Original: (cid:107)= E (cid:107) (1 − β k +1 )
Revised: (cid:107)= E (cid:107) (1 − β k +1 )

Original: ( −∇ J ( θ k ) − u k ) + β k +1 ( −∇ J ( θ k +1 )
Revised: ( −∇ J ( θ k ) − u k ) + β k +1 ( −∇ J ( θ k +1 )

Original: + g
Revised: + g

Original: ( τ k +1 | θ k +1 ))
Revised: ( τ k +1 | θ k +1 ))

Original: + (1 − β k +1 ) (cid:0) − ∇ J ( θ k +1 )
Revised: + (1 − β k +1 ) (cid:0) − ∇ J ( θ k +1 )

Original: + ∇ J ( θ k ) (cid:1) (cid:107)= (1 − β k +1 ) 2 E (cid:107)∇ J
Revised: + ∇ J ( θ k ) (cid:1) (cid:107)= (1 − β k +1 ) 2 E (cid:107)∇ J

Original: ( θ k )
Revised: ( θ k )

Original: + u k
Revised: + u k

Original: + ∇ J
Revised: + ∇ J

Original: ( θ k +1 )
Revised: ( θ k +1 )

Original: − ∇ J ( θ k ) (cid:107) 2 + β 2 k +1 E (cid:107)∇ J ( θ k +1 )
Revised: − ∇ J ( θ k ) (cid:107) 2 + β 2 k +1 E (cid:107)∇ J ( θ k +1 )

Original: − g ( τ k +1 | θ k +1 )
Revised: − g ( τ k +1 | θ k +1 )

Original: (cid:107)≤ (1 − β k +1 ) 2 (1 + β k +1 ) E (cid:107)∇ J
Revised: (cid:107)≤ (1 − β k +1 ) 2 (1 + β k +1 ) E (cid:107)∇ J

Original: ( θ k ) + u k (cid:107) 

Revised: ( θ k ) + u k (cid:107)

Original: + (1 − β k +1 ) 

Revised: + (1 − β k +1 )

Original: (1 + β k +1 ) E (cid:107)∇ J ( θ k +1 )
Revised: (1 + β k +1 ) E (cid:107)∇ J ( θ k +1 )

Original: − ∇ J ( θ k ) (cid:107)+ β 2 k +1 E (cid:107)∇ J ( θ k +1 )
Revised: − ∇ J ( θ k ) (cid:107)+ β 2 k +1 E (cid:107)∇ J ( θ k +1 )

Original: − g ( τ k +1 | θ k +1 )
Revised: − g ( τ k +1 | θ k +1 )

Original: (cid:107)≤ (1 − β k +1 ) E (cid:107)∇ J ( θ k ) + u k (cid:107) 

Revised: (cid:107)≤ (1 − β k +1 ) E (cid:107)∇ J ( θ k ) + u k (cid:107)

Original: + β k
Revised: + β k

Original: +
Revised: +

Original: E (cid:107)∇ J ( θ k +1 )
Revised: E (cid:107)∇ J ( θ k +1 )

Original: − ∇ J ( θ
Revised: − ∇ J ( θ

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ≤ (1 − β
Revised: ≤ (1 − β

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ≤ (1 − β k +1 )
Revised: ≤ (1 − β k +1 )

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: f ( θ k
Revised: f ( θ k

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: where the fourth equality holds by E τ k +1 ∼ p ( τ | θ k +1 )
Revised: where the fourth equality holds by E τ k +1 ∼ p ( τ | θ k +1 )

Original: [ g ( τ k +1 | θ k +1 )]
Revised: [ g ( τ k +1 | θ k +1 )]

Original: = ∇ J ( θ k +1 ) ; the ﬁrst inequality holds by Young’s inequality; the second inequality is due to 0 < β k +1 ≤ 1 such that (1 − β k +1 ) 

Revised: = ∇ J ( θ k +1 ) ; the ﬁrst inequality holds by Young’s inequality; the second inequality is due to 0 < β k +1 ≤ 1 such that (1 − β k +1 )

Original: (1+ β k +1 )
Revised: (1+ β k +1 )

Original: = 1 − β k +1 − β 

Revised: = 1 − β k +1 − β

Original: k
Revised: k

Original: +

Revised: +

Original: + β 3 k +1 ≤ 1 −
Revised: + β 3 k +1 ≤ 1 −

Original: β k +

Revised: β k +

Original: and (1 − β k +1 ) 2 (1 + 

Revised: and (1 − β k +1 ) 2 (1 +

Original: β k +1 ) ≤ 1 + 

Revised: β k +1 ) ≤ 1 +

Original: β k +1 ≤ 2 β k
Revised: β k +1 ≤ 2 β k

Original: +1 ; the last inequality holds by Assumption 2.
Revised: +1 ; the last inequality holds by Assumption 2.

Original: Theorem 3.
Revised: Theorem 3.

Original: Assume the sequence { θ k } Kk =1 be generated from Algorithm 1.
Revised: Assume the sequence { θ k } Kk =1 be generated from Algorithm 1.

Original: Let η k
Revised: Let η k

Original: =
Revised: =

Original: b
Revised: b

Original: ( m + k ) 1 / 2 for all k ≥ 1 , 0 < λ ≤ νm 1 / 2 9 Lb , b > 0 , 8 Lλν ≤ c ≤ m 1 /
Revised: ( m + k ) 1 / 2 for all k ≥ 1 , 0 < λ ≤ νm 1 / 2 9 Lb , b > 0 , 8 Lλν ≤ c ≤ m 1 /

Original: 2 b , and m ≥ max { b 2 , ( cb ) 2 } , we have
Revised: 2 b , and m ≥ max { b 2 , ( cb ) 2 } , we have

Original: [Equation]
Revised: [Equation]

Original: where M = J ∗ − J ( θ 1 ) νλb
Revised: where M = J ∗ − J ( θ 1 ) νλb

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since η k = b ( m + k ) 1 / 2 is decreasing on k , we have η k ≤ η 0 = bm 1 / 2 ≤ 1 for all k ≥ 0 .
Revised: Since η k = b ( m + k ) 1 / 2 is decreasing on k , we have η k ≤ η 0 = bm 1 / 2 ≤ 1 for all k ≥ 0 .

Original: At the same time, let m ≥ ( cb ) 2 , we have β k +

Revised: At the same time, let m ≥ ( cb ) 2 , we have β k +

Original: =
Revised: =

Original: cη k ≤ cη 0 =
Revised: cη k ≤ cη 0 =

Original: cbm 1 / 2 ≤ 1 .
Revised: cbm 1 / 2 ≤ 1 .

Original: Consider m ≥ ( cb ) 2 , we have c ≤ m 1 / 2 b .
Revised: Consider m ≥ ( cb ) 2 , we have c ≤ m 1 / 2 b .

Original: Since 0 < λ ≤ νm 1 / 2 9 Lb , we have λ ≤ νm 1 / 2 9 Lb ≤ νm 1 / 2 2 Lb
Revised: Since 0 < λ ≤ νm 1 / 2 9 Lb , we have λ ≤ νm 1 / 2 9 Lb ≤ νm 1 / 2 2 Lb

Original: [Equation]
Revised: [Equation]

Original: for all k
Revised: for all k

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 7, we have
Revised: According to Lemma 7, we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality is due to β k
Revised: where the ﬁrst equality is due to β k

Original: +1 =
Revised: +1 =

Original: cη k and the last equality holds by 8 Lλν ≤ c ≤ m 1 / 2 b .
Revised: cη k and the last equality holds by 8 Lλν ≤ c ≤ m 1 / 2 b .

Original: Next we deﬁne a Lyapunov function Φ k = E (cid:2) f
Revised: Next we deﬁne a Lyapunov function Φ k = E (cid:2) f

Original: [Equation]
Revised: [Equation]

Original: θ k ) + 1 L (cid:107)∇ f
Revised: θ k ) + 1 L (cid:107)∇ f

Original: [Equation]
Revised: [Equation]

Original: θ k ) − u k (cid:107) 2 (cid:3) for any t ≥ 1 .
Revised: θ k ) − u k (cid:107) 2 (cid:3) for any t ≥ 1 .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows by the Lemma 6 and the above inequality (30).
Revised: where the ﬁrst inequality follows by the Lemma 6 and the above inequality (30).

Original: Summing the above inequality (31) over k from 1 to K , we can obtain
Revised: Summing the above inequality (31) over k from 1 to K , we can obtain

Original: K (cid:88)
Revised: K (cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last second inequality holds by Assumptions 2 and
Revised: where the last second inequality holds by Assumptions 2 and

Original: [Equation]
Revised: [Equation]

Original: Since η k is decreasing
Revised: Since η k is decreasing

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Let M =
Revised: Let M =

Original: J ∗ − J
Revised: J ∗ − J

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ln( m + K )
Revised: ln( m + K )

Original: , the above inequality (33) reduces to
Revised: , the above inequality (33) reduces to

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: According to Jensen’s inequality
Revised: According to Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to the inequality ( a + b ) 1 / 4 ≤ a 1 / 4 + b 1 / 4 for all a, b ≥ 0 .
Revised: where the last inequality is due to the inequality ( a + b ) 1 / 4 ≤ a 1 / 4 + b 1 / 4 for all a, b ≥ 0 .

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: By the step 4 of Algorithm
Revised: By the step 4 of Algorithm

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: At the same time, as in Ghadimi et al.
Revised: At the same time, as in Ghadimi et al.

Original: (2016), we deﬁne
Revised: (2016), we deﬁne

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: According to the above Lemma
Revised: According to the above Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: By the above inequalities (36) and (40), we have
Revised: By the above inequalities (36) and (40), we have

Original: [Equation]
Revised: [Equation]

Original: A. C ONVERGENCE A NALYSIS OF VR-BGPO ALGORITHM
Revised: A. C ONVERGENCE A NALYSIS OF VR-BGPO ALGORITHM

Original: In this subsection, we will analyze convergence properties of the VR-BGPO algorithm.
Revised: In this subsection, we will analyze convergence properties of the VR-BGPO algorithm.

Original: Lemma 8.
Revised: Lemma 8.

Original: Assume that the stochastic policy gradient u k +1 be generated from Algorithm 2, given 0 < β k ≤ 1 , we have
Revised: Assume that the stochastic policy gradient u k +1 be generated from Algorithm 2, given 0 < β k ≤ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: where ˆ L 2 =
Revised: where ˆ L 2 =

Original: L 2 + 2 G 2 C 2 w and C w =
Revised: L 2 + 2 G 2 C 2 w and C w =

Original: (cid:113) H (2 HC 2 g + C h )( W + 1) .
Revised: (cid:113) H (2 HC 2 g + C h )( W + 1) .

Original: Proof.
Revised: Proof.

Original: By the deﬁnition of u k +1 in Algorithm 2, we have u k
Revised: By the deﬁnition of u k +1 in Algorithm 2, we have u k

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: Since ∇ f ( θ k +1 ) = −∇ J ( θ k +1 ) , we have
Revised: Since ∇ f ( θ k +1 ) = −∇ J ( θ k +1 ) , we have

Original: [Equation]
Revised: [Equation]

Original: (42) =
Revised: (42) =

Original: E (cid:107)∇ J
Revised: E (cid:107)∇ J

Original: ( θ k )
Revised: ( θ k )

Original: + u k
Revised: + u k

Original: + ∇ J
Revised: + ∇ J

Original: ( θ k +1 )
Revised: ( θ k +1 )

Original: − ∇ J ( θ k ) − β k +1 u k − β k +1 g
Revised: − ∇ J ( θ k ) − β k +1 u k − β k +1 g

Original: ( τ k +1 | θ k +1 )
Revised: ( τ k +1 | θ k +1 )

Original: + (1 − β k +1 ) (cid:0) − g ( τ k +

Revised: + (1 − β k +1 ) (cid:0) − g ( τ k +

Original: | θ k +1 )
Revised: | θ k +1 )

Original: + w ( τ k +

Revised: + w ( τ k +

Original: | θ k , θ k +1 )
Revised: | θ k , θ k +1 )

Original: g ( τ k +1 | θ k ) (cid:1) (cid:107)= E (cid:107) (1 − β k +1 )( ∇ J ( θ k ) + u k ) + β k +1 ( ∇ J ( θ k +1 )
Revised: g ( τ k +1 | θ k ) (cid:1) (cid:107)= E (cid:107) (1 − β k +1 )( ∇ J ( θ k ) + u k ) + β k +1 ( ∇ J ( θ k +1 )

Original: − g ( τ k +1 | θ k +1 )) − (1 − β k +1 ) (cid:0) g
Revised: − g ( τ k +1 | θ k +1 )) − (1 − β k +1 ) (cid:0) g

Original: ( τ k +1 | θ k +1 ) − w ( τ k +1 | θ k , θ k +1 )
Revised: ( τ k +1 | θ k +1 ) − w ( τ k +1 | θ k , θ k +1 )

Original: g ( τ k +1 | θ k ) − ( ∇ J ( θ k +1 )
Revised: g ( τ k +1 | θ k ) − ( ∇ J ( θ k +1 )

Original: − ∇ J ( θ k )) (cid:1) (cid:107)= (1 − β k +1 ) 2 E (cid:107)∇ J ( θ k ) + u k (cid:107) 2 + E (cid:107) β k +1 ( ∇ J ( θ k +1 )
Revised: − ∇ J ( θ k )) (cid:1) (cid:107)= (1 − β k +1 ) 2 E (cid:107)∇ J ( θ k ) + u k (cid:107) 2 + E (cid:107) β k +1 ( ∇ J ( θ k +1 )

Original: − g ( τ k +1 | θ k +1 )) − (1 − β k +1 ) (cid:0) g
Revised: − g ( τ k +1 | θ k +1 )) − (1 − β k +1 ) (cid:0) g

Original: ( τ k +1 | θ k +1 ) − w ( τ k +1 | θ k , θ k +1 )
Revised: ( τ k +1 | θ k +1 ) − w ( τ k +1 | θ k , θ k +1 )

Original: g ( τ k +1 | θ k ) − ( ∇ J ( θ k +1 )
Revised: g ( τ k +1 | θ k ) − ( ∇ J ( θ k +1 )

Original: − ∇ J ( θ k )) (cid:1) (cid:107)≤ (1 − β k +1 ) 2 E (cid:107)∇ J ( θ k ) + u k (cid:107) 2 + 2 β 2 k +1 E (cid:107)∇ J ( θ k +1 )
Revised: − ∇ J ( θ k )) (cid:1) (cid:107)≤ (1 − β k +1 ) 2 E (cid:107)∇ J ( θ k ) + u k (cid:107) 2 + 2 β 2 k +1 E (cid:107)∇ J ( θ k +1 )

Original: − g ( τ k +1 | θ k +1 )
Revised: − g ( τ k +1 | θ k +1 )

Original: (cid:107)+ 2(1 − β k +1 ) 2 E (cid:107) g ( τ k +1 | θ k +1 ) − w ( τ k +1 | θ k , θ k +1 )
Revised: (cid:107)+ 2(1 − β k +1 ) 2 E (cid:107) g ( τ k +1 | θ k +1 ) − w ( τ k +1 | θ k , θ k +1 )

Original: g ( τ k +1 | θ k ) − ( ∇ J ( θ k +1 )
Revised: g ( τ k +1 | θ k ) − ( ∇ J ( θ k +1 )

Original: − ∇ J ( θ k ))
Revised: − ∇ J ( θ k ))

Original: (cid:107)≤ (1 − β k +1 ) E (cid:107)∇ f ( θ k ) − u k (cid:107) 2 + 2 β 2 k +

Revised: (cid:107)≤ (1 − β k +1 ) E (cid:107)∇ f ( θ k ) − u k (cid:107) 2 + 2 β 2 k +

Original: σ 2 + 2 E (cid:107) g
Revised: σ 2 + 2 E (cid:107) g

Original: ( τ k +1 | θ k +1 ) − w ( τ k +1 | θ k , θ k +1 )
Revised: ( τ k +1 | θ k +1 ) − w ( τ k +1 | θ k , θ k +1 )

Original: g ( τ k +1 | θ k ) (cid:107)(cid:124) (cid:123)(cid:122) (cid:125) =
Revised: g ( τ k +1 | θ k ) (cid:107)(cid:124) (cid:123)(cid:122) (cid:125) =

Original: T, where the forth equality holds by E τ k +1 ∼ p ( τ | θ k +1 )
Revised: T, where the forth equality holds by E τ k +1 ∼ p ( τ | θ k +1 )

Original: [ g ( τ k +1 | θ k +1 )]
Revised: [ g ( τ k +1 | θ k +1 )]

Original: = ∇ J ( θ k +1 ) and E
Revised: = ∇ J ( θ k +1 ) and E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ] = ∇
Revised: ] = ∇

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: +1 ) − ∇ J ( θ k ) ; the sec- ond last inequality follows by Young’s inequality; and the last inequality holds by Assumption 2, and the inequality E (cid:107) ζ − E [ ζ ] (cid:107) 

Revised: +1 ) − ∇ J ( θ k ) ; the sec- ond last inequality follows by Young’s inequality; and the last inequality holds by Assumption 2, and the inequality E (cid:107) ζ − E [ ζ ] (cid:107)

Original: =
Revised: =

Original: E (cid:107) ζ (cid:107) 2 − ( E [ ζ ]) 2 ≤ E (cid:107) ζ (cid:107) 2 , and 0 < β k +1 ≤ 1 .
Revised: E (cid:107) ζ (cid:107) 2 − ( E [ ζ ]) 2 ≤ E (cid:107) ζ (cid:107) 2 , and 0 < β k +1 ≤ 1 .

Original: Next, we give an upper bound of the term T 1 as follows:
Revised: Next, we give an upper bound of the term T 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds by Lemma
Revised: where the second inequality holds by Lemma

Original: [Equation]
Revised: [Equation]

Original: and the third equality holds by Lemma 5, and the last inequality follows by Lemma
Revised: and the third equality holds by Lemma 5, and the last inequality follows by Lemma

Original: [Equation]
Revised: [Equation]

Original: Combining the inequalities (42) with (43)
Revised: Combining the inequalities (42) with (43)

Original: [Equation]
Revised: [Equation]

Original: let ˆ L 2 =
Revised: let ˆ L 2 =

Original: L 2 + 2 G 2 C 2 w , we have
Revised: L 2 + 2 G 2 C 2 w , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose the sequence { θ k } Kk =1 be generated from Algorithm 2.
Revised: Suppose the sequence { θ k } Kk =1 be generated from Algorithm 2.

Original: Let η k
Revised: Let η k

Original: =
Revised: =

Original: b
Revised: b

Original: ( m + k ) 1 / 3 for all k ≥ 0 , 0 < λ ≤ νm 1 / 3 5ˆ
Revised: ( m + k ) 1 / 3 for all k ≥ 0 , 0 < λ ≤ νm 1 / 3 5ˆ

Original: Lb , b > 0 , 23 b 3 + 20ˆ L 2 λ 2 ν≤ c ≤ m
Revised: Lb , b > 0 , 23 b 3 + 20ˆ L 2 λ 2 ν≤ c ≤ m

Original: [Equation]
Revised: [Equation]

Original: 3 b 2 and m ≥ max (cid:0) 2 , b 3 , ( cb ) 3 , ( 56 b ) 2 / 3 (cid:1) , we have
Revised: 3 b 2 and m ≥ max (cid:0) 2 , b 3 , ( cb ) 3 , ( 56 b ) 2 / 3 (cid:1) , we have

Original: [Equation]
Revised: [Equation]

Original: where M (cid:48)
Revised: where M (cid:48)

Original: =
Revised: =

Original: J ∗ − J ( θ 1 ) bνλ
Revised: J ∗ − J ( θ 1 ) bνλ

Original: + m 1 / 3 σ 2 16 b 2 ˆ L 2 λ 2 + c 2 σ 2 b 2 8ˆ L 2 λ 2 and ˆ L 2 =
Revised: + m 1 / 3 σ 2 16 b 2 ˆ L 2 λ 2 + c 2 σ 2 b 2 8ˆ L 2 λ 2 and ˆ L 2 =

Original: L 2 + 2 G 2 C 2 w .
Revised: L 2 + 2 G 2 C 2 w .

Original: Proof.
Revised: Proof.

Original: Since η k = b ( m + k ) 1 / 3 on k is decreasing and m ≥ b 3 , we have η k ≤ η 0 = bm 1 / 3 ≤ 1 .
Revised: Since η k = b ( m + k ) 1 / 3 on k is decreasing and m ≥ b 3 , we have η k ≤ η 0 = bm 1 / 3 ≤ 1 .

Original: Due to ˆ L = (cid:112) L 2 + 2 G 2 C 2 w ≥ L , we have 0 < λ ≤ νm 1 / 3 5ˆ
Revised: Due to ˆ L = (cid:112) L 2 + 2 G 2 C 2 w ≥ L , we have 0 < λ ≤ νm 1 / 3 5ˆ

Original: Lb ≤ νm 1 / 3 2 Lb = ν 2 Lη 0 ≤ ν 2 Lη k for any k ≥ 0 .
Revised: Lb ≤ νm 1 / 3 2 Lb = ν 2 Lη 0 ≤ ν 2 Lη k for any k ≥ 0 .

Original: Consider 0 < η k ≤ 1 and m ≥ ( cb ) 3 , we have β k +

Revised: Consider 0 < η k ≤ 1 and m ≥ ( cb ) 3 , we have β k +

Original: =
Revised: =

Original: cη 2 k ≤ cb 2 m 2 / 3 ≤ 1 .
Revised: cη 2 k ≤ cb 2 m 2 / 3 ≤ 1 .

Original: At the same time, we have c ≤ m 2 / 3 b 2 .
Revised: At the same time, we have c ≤ m 2 / 3 b 2 .

Original: According to Lemma 8, we have
Revised: According to Lemma 8, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds by the following inequality
Revised: where the last inequality holds by the following inequality

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality holds by the concavity of function f ( x )
Revised: where the ﬁrst inequality holds by the concavity of function f ( x )

Original: = x 1 / 3 , i.e. , ( x + y ) 1 / 3 ≤ x 1 / 3 + y 3 x 2 / 3 ; the second inequality is due to m ≥ 2 , and the last inequality is due to 0 < η k ≤ 1 .
Revised: = x 1 / 3 , i.e. , ( x + y ) 1 / 3 ≤ x 1 / 3 + y 3 x 2 / 3 ; the second inequality is due to m ≥ 2 , and the last inequality is due to 0 < η k ≤ 1 .

Original: Let c ≥ 

Revised: Let c ≥

Original: b 3 + 20ˆ L 2 λ 2 ν, we have
Revised: b 3 + 20ˆ L 2 λ 2 ν, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we simultaneously consider c ≥ 23 b 3 + 20ˆ L 2 λ 2 ν, c ≤ m 2 / 3 b 2 and 0 < λ ≤ νm 1 / 3 5ˆ
Revised: Here we simultaneously consider c ≥ 23 b 3 + 20ˆ L 2 λ 2 ν, c ≤ m 2 / 3 b 2 and 0 < λ ≤ νm 1 / 3 5ˆ

Original: Lb , we have
Revised: Lb , we have

Original: [Equation]
Revised: [Equation]

Original: Then we have m ≥ ( 5 6 b ) 2 / 3 .
Revised: Then we have m ≥ ( 5 6 b ) 2 / 3 .

Original: Next we deﬁne a Lyapunov function Ω k = E (cid:2)
Revised: Next we deﬁne a Lyapunov function Ω k = E (cid:2)

Original: f ( θ k )
Revised: f ( θ k )

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: f ( θ k ) − u k (cid:107) 2 (cid:3) for any k ≥ 1 .
Revised: f ( θ k ) − u k (cid:107) 2 (cid:3) for any k ≥ 1 .

Original: According to Lemma 6, we have Ω
Revised: According to Lemma 6, we have Ω

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to the above inequality (47)
Revised: where the ﬁrst inequality is due to the above inequality (47)

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation]
Revised: [Equation]

Original: Taking average over k =
Revised: Taking average over k =

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: K on both sides of (50)
Revised: K on both sides of (50)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: T
Revised: T

Original: K (cid:88)
Revised: K (cid:88)

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is due to u 1 = − g ( τ 1 | θ 1 ) , ∇ f ( θ 1 ) =
Revised: where the second inequality is due to u 1 = − g ( τ 1 | θ 1 ) , ∇ f ( θ 1 ) =

Original: −∇ J ( θ 1 ) and Assumption 1, and the last inequality holds by Assumption 2.
Revised: −∇ J ( θ 1 ) and Assumption 1, and the last inequality holds by Assumption 2.

Original: Since η k is decreasing, i.e., η − 1 K ≥ η − 1 k for any
Revised: Since η k is decreasing, i.e., η − 1 K ≥ η − 1 k for any

Original: 0 < k ≤ K , we have
Revised: 0 < k ≤ K , we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds by (cid:80)
Revised: where the second inequality holds by (cid:80)

Original: Kk =1 η 3 k dk ≤
Revised: Kk =1 η 3 k dk ≤

Original: [Equation]
Revised: [Equation]

Original: m + k ) − 1 dk .
Revised: m + k ) − 1 dk .

Original: Let M (cid:48)
Revised: Let M (cid:48)

Original: =
Revised: =

Original: J ∗ − J ( θ 1 ) bνλ
Revised: J ∗ − J ( θ 1 ) bνλ

Original: + m 1 /
Revised: + m 1 /

Original: [Equation]
Revised: [Equation]

Original: 2 16 b 2 ˆ L 2 λ 2 + c 2 σ 2 b 2 8ˆ
Revised: 2 16 b 2 ˆ L 2 λ 2 + c 2 σ 2 b 2 8ˆ

Original: L 2 λ 2 , the above inequality (52) reduces to
Revised: L 2 λ 2 , the above inequality (52) reduces to

Original: [Equation]
Revised: [Equation]

Original: According to Jensen’s inequality
Revised: According to Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last inequality is due to the inequality ( a + b ) 1 / 6 ≤ a 1 / 6 + b 1 / 6 for all a, b ≥ 1 .
Revised: where the last inequality is due to the inequality ( a + b ) 1 / 6 ≤ a 1 / 6 + b 1 / 6 for all a, b ≥ 1 .

Original: Thus we
Revised: Thus we

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then by using the above inequality (40)
Revised: Then by using the above inequality (40)

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: K
Revised: K

Original: K (cid:88)
Revised: K (cid:88)

Original: [Equation]
Revised: [Equation]

Original: B A CTOR -C RITIC S TYLE BGPO AND VR-BGPO A LGORITHMS
Revised: B A CTOR -C RITIC S TYLE BGPO AND VR-BGPO A LGORITHMS

Original: In the experiments, we use the advantage-based policy gradient estimator:
Revised: In the experiments, we use the advantage-based policy gradient estimator:

Original: [Equation]
Revised: [Equation]

Original: network 5:
Revised: network 5:

Original: Update ˜ θ k +1 =
Revised: Update ˜ θ k +1 =

Original: arg min θ ∈ Θ (cid:8) (cid:104)
Revised: arg min θ ∈ Θ (cid:8) (cid:104)

Original: u k , θ (cid:105)
Revised: u k , θ (cid:105)

Original: + 1 λ D ψ k
Revised: + 1 λ D ψ k

Original: ( θ, θ k ) (cid:9) ; 6:
Revised: ( θ, θ k ) (cid:9) ; 6:

Original: Update θ k +

Revised: Update θ k +

Original: =
Revised: =

Original: θ k + η k
Revised: θ k + η k

Original: (˜ θ k +1 − θ k ) with η k = b ( m + k ) 1 / 3 ; 7: # Update the value network 8:
Revised: (˜ θ k +1 − θ k ) with η k = b ( m + k ) 1 / 3 ; 7: # Update the value network 8:

Original: Update θ vk +1 by solving the subproblem (58); 9: # Sample a new trajectory and compute policy gradients 10:
Revised: Update θ vk +1 by solving the subproblem (58); 9: # Sample a new trajectory and compute policy gradients 10:

Original: Sample a trajectory τ k +1 from p ( τ | θ k +1 ) , and compute u k +1 =
Revised: Sample a trajectory τ k +1 from p ( τ | θ k +1 ) , and compute u k +1 =

Original: − β k +1 g ( τ k +

Revised: − β k +1 g ( τ k +

Original: | θ k +1 )
Revised: | θ k +1 )

Original: + (1 − β k +1 ) (cid:2)
Revised: + (1 − β k +1 ) (cid:2)

Original: u k − g
Revised: u k − g

Original: ( τ k +1 | θ k +1 )
Revised: ( τ k +1 | θ k +1 )

Original: + w ( τ k +

Revised: + w ( τ k +

Original: | θ k , θ k +1 )
Revised: | θ k , θ k +1 )

Original: g ( τ k +1 | θ k ) (cid:3) with β k +

Revised: g ( τ k +1 | θ k ) (cid:3) with β k +

Original: =
Revised: =

Original: cη 2 k ; 11: end for 12: Output: θ ζ chosen uniformly random from { θ k } Kk =1 .
Revised: cη 2 k ; 11: end for 12: Output: θ ζ chosen uniformly random from { θ k } Kk =1 .

Original: where θ ( ∈ Θ ⊆ R d ) denotes parameters of the policy network, and ˆ A π θ ( s, a ) is an estimator of the advantage function A π θ ( s, a ) .
Revised: where θ ( ∈ Θ ⊆ R d ) denotes parameters of the policy network, and ˆ A π θ ( s, a ) is an estimator of the advantage function A π θ ( s, a ) .

Original: In using advantage-based policy gradient, we also need the statevalue function V π θ ( s ) .
Revised: In using advantage-based policy gradient, we also need the statevalue function V π θ ( s ) .

Original: Here, we use a value network V θ v ( s ) to approximate the state-value function
Revised: Here, we use a value network V θ v ( s ) to approximate the state-value function

Original: V π θ ( s ) .
Revised: V π θ ( s ) .

Original: Speciﬁcally, we solve the following problem to obtain the value network:
Revised: Speciﬁcally, we solve the following problem to obtain the value network:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where θ v ( ∈ Θ v ⊆ R d v ) denotes parameters of the value network, and ˆ V π θ ( s ) is an estimator of the state-value function V π θ ( s ) , which is obtained by the GAE Schulman et al.
Revised: where θ v ( ∈ Θ v ⊆ R d v ) denotes parameters of the value network, and ˆ V π θ ( s ) is an estimator of the state-value function V π θ ( s ) , which is obtained by the GAE Schulman et al.

Original: Then we use the GAE to estimate ˆ A π θ based on value network V θ v .
Revised: Then we use the GAE to estimate ˆ A π θ based on value network V θ v .

Original: We describe the actor-critic style BGPO and
Revised: We describe the actor-critic style BGPO and

Original: VR-BGPO algorithms in Algorithm 3 and Algorithm 4, respectively.
Revised: VR-BGPO algorithms in Algorithm 3 and Algorithm 4, respectively.

Original: C D ETAILED S ETUP OF E XPERIMENTAL E NVIRONMENTS AND H YPER - PARAMETERS
Revised: C D ETAILED S ETUP OF E XPERIMENTAL E NVIRONMENTS AND H YPER - PARAMETERS

Original: In this section, we provide the detailed setup of experimental environments and hyper-parameters.
Revised: In this section, we provide the detailed setup of experimental environments and hyper-parameters.

Original: We ﬁrst provide the detailed setup of our experiments in Tab. 2 and Tab. 3.
Revised: We ﬁrst provide the detailed setup of our experiments in Tab. 2 and Tab. 3.

Original: We use ADAM optimizer to optimize value functions for all methods and settings, which is a common practice.
Revised: We use ADAM optimizer to optimize value functions for all methods and settings, which is a common practice.

Original: The impor-
Revised: The impor-

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Table] [Figure] [Table]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: tance sampling weight used for VR-BGPO algorithm is clipped within [0 . 5 , 1 . 5] .
Revised: tance sampling weight used for VR-BGPO algorithm is clipped within [0 . 5 , 1 . 5] .

Original: The momentum term β k is set to be less or equal than one ( β k = min( β k , 1 . 0) ) through the whole training process.
Revised: The momentum term β k is set to be less or equal than one ( β k = min( β k , 1 . 0) ) through the whole training process.

Original: BGPO and VR-BGPO algorithms involve 4 hyper-parameters { λ, b, m, c } , which may bring additional efforts for hyper-parameter tuning.
Revised: BGPO and VR-BGPO algorithms involve 4 hyper-parameters { λ, b, m, c } , which may bring additional efforts for hyper-parameter tuning.

Original: However, the actual hyper-parameter tuning is not so hard, and we only use one set of { b, m, c } for 9 environments.
Revised: However, the actual hyper-parameter tuning is not so hard, and we only use one set of { b, m, c } for 9 environments.

Original: The strategy of hyper-parameter tuning is to separate the four hyper-parameters into two parts.
Revised: The strategy of hyper-parameter tuning is to separate the four hyper-parameters into two parts.

Original: The ﬁrst part is { b, m, c } , which mainly decide when the momentum term β k actually affects ( β k < 1 . 0 ) updates.
Revised: The ﬁrst part is { b, m, c } , which mainly decide when the momentum term β k actually affects ( β k < 1 . 0 ) updates.

Original: The second part, λ , only affects how fast the policy is learning.
Revised: The second part, λ , only affects how fast the policy is learning.

Original: To further reduce the complexity of hyper-parameter tuning, we always set m = 2 .
Revised: To further reduce the complexity of hyper-parameter tuning, we always set m = 2 .

Original: By grouping hyper-parameters, we only consider λ and how β k changes, which largely simpliﬁes the process of hyper-parameter tuning.
Revised: By grouping hyper-parameters, we only consider λ and how β k changes, which largely simpliﬁes the process of hyper-parameter tuning.