Original: E RROR C ONTROLLED A CTOR -C RITIC M ETHOD TO R EINFORCEMENT L EARNING
Revised: E RROR C ONTROLLED A CTOR -C RITIC M ETHOD TO R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In the reinforcement learning (RL) algorithms which incorporate function approximation methods, the approximation error of value function inevitably cause overestimation phenomenon and have a negative impact on the convergence of the algorithms.
Revised: In the reinforcement learning (RL) algorithms which incorporate function approximation methods the approximation error of value function inevitably causes overestimation phenomenon and impacts algorithm performances

Original: To mitigate the negative effects of approximation error, we propose a new actor-critic algorithm called Error Controlled Actor-critic which ensures conﬁning the approximation error in value function.
Revised: To mitigate the negative effects caused by approximation error , we propose a new actor- critic algorithm called Error Controlled Actor-critic which ensures conﬁning the ap - proximation error in value function.

Original: In this paper, we ﬁrstly present an analysis of how the approximation error can hinder the optimization process of actor-critic methods.
Revised: In this paper

Original: Then, we derive an upper boundary ofthe approximation error of Q function approximator, and found that the error can be lowered by placing restrictions on the KL-divergence between every two consecutive policies during the training phase of the policy.
Revised: , we derive an upper boundary of approximation error for Q function approximator in actor-critic methods and ﬁnd that the error can be lowered by keep new policy close to the previous one during the training phase of the policy

Original: The results of experiments on a range of continuous control tasks from OpenAI gym suite demonstrate that the proposed actor-critic algorithm apparently reduces the approximation error and signiﬁcantly outperforms other model-free RL algorithms.
Revised: The results of experiments on a range of continuous control tasks from OpenAI gym suite demonstrate that the proposed actor -critic algorithm apparently reduces the approximation error and signiﬁcantly outperforms other model free RL algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In pratice, reinforcement learning (RL)  are combined with function approximation methods to adapt to the application scenarios whose state spaces are combinatorial, large, or even continuous.
Revised: In practice, reinforcement learning (RL) algorithms are combined with function approximation methods to adapt to the application scenarios whose state spaces are combinatorial, large, or even continuous.

Original: Many function approximation methods RL methods, including the Fourier basis(Konidaris et al., 2011), kernel regression(Xu, 2006; Barreto et al., 2011;
Revised: Many function approximation methods RL methods, including the Fourier basis(Konidaris et al., 2011), kernel regression(Xu, 2006; Barreto et al., 2011;

Original: Bhat et al., 2012), and neural neworks(Barto et al., 1982; Tesauro, 1992; Boyan et al., 1992; Gullapalli, 1992) have been used to learn value functions.
Revised: Bhat et al., 2012), and neural neworks(Barto et al., 1982; Tesauro, 1992; Boyan et al., 1992; Gullapalli, 1992) have been used to learn value functions.

Original: In recent years, many deep reinforcement learning (DRL) methods were implemented by incorporating deep learning into RL methods.
Revised: In recent years, many deep reinforcement learning (DRL) methods were implemented by incorporating deep learning into RL methods.

Original: Deep Q-learning Network (DQN)(Mnih et al., 2013) reported by Mnih in 2013 is a typical work that uses a deep convolutional neural network (CNN) to represent a suitable action value function estimating future rewards (returns); it successfully learned end-toend control policies for seven Atari 2600 games directly from large state spaces.
Revised: Deep Q-learning Network (DQN)(Mnih et al., 2013) reported by Mnih in 2013 is a typical work that uses a deep convolutional neural network (CNN) to represent a suitable action value function estimating future rewards (returns); it successfully learned end-to-end control policies for seven Atari 2600 games directly from large state spaces.

Original: Thereafter, deep RL methods, such as Deep Deterministic Policy Gradient (DDPG)(Lillicrap et al., 2016), Proximal 
Revised: Thereafter, deep RL methods, such as Deep Deterministic Policy Gradient (DDPG)(Lillicrap et al., 2016), Proximal Policy Optimization (PPO)(Schulman et al., 2017), Twin

Original: Policy Optimization (PPO)(Schulman et al., 2017), Twin Delayed Deep Deterministic policy gradient (TD3)(Fujimoto et al., 2018), and Soft Actor-Critic (SAC)(Haarnoja et al., 2018), started to become mainstream in the ﬁeld of RL.
Revised: Delayed Deep Deterministic policy gradient (TD3)(Fujimoto et al., 2018), and Soft Actor-Critic (SAC)(Haarnoja et al., 2018), started to become mainstream in the ﬁeld of RL.

Original: Althouth function approximation methods have assisted reinforcement learning (RL) algorithms to perform well in complex problems by providing great representation power; however, they also cause an issue called overestimation phenomenon that jeopardize the optimization process of RL algorithms.
Revised: Althouth function approximation methods have assisted reinforcement learning (RL) algorithms to perform well in complex problems by providing great representation power; however, they also cause an issue called overestimation phenomenon that jeopardize the optimization process of RL algorithms.

Original: Thrun & Schwartz (1993) presented a theoretical analysis of this systematic overestimation phenomenon in Q-learning methods that use function approximation methods.
Revised: Thrun & Schwartz (1993) presented a theoretical analysis of this systematic overestimation phenomenon in Q-learning methods that use function approximation methods.

Original: Similar problem persists in the actor-critic methods employed function approximation methods.
Revised: Similar problem persists in the actor-critic methods employed function approximation methods.

Original: Thomas (2014) reported that several natural actor-critic algorithms use biased estimates of policy gradient to update parameters when using function approximation to approximate the action value function.
Revised: Thomas (2014) reported that several natural actor-critic algorithms use biased estimates of policy gradient to update parameters when using function approximation to approximate the action value function.

Original: Fujimoto et al.
Revised: Fujimoto et al.

Original: (2018) proved that the value estimation in the deterministic policy gradient method also lead to overestimation problem.
Revised: (2018) proved that the value estimation in the deterministic policy gradient method also lead to overestimation problem.

Original: In brief, the approximation errors of value functions caused the inaccuracy of estimated values, and such inaccuracy induced the overestimation on value function; so that poor performances might be assigned to high reward values.
Revised: In brief, the approximation errors of value functions caused the inaccuracy of estimated values, and such inaccuracy induced the overestimation on value function; so that poor performances might be assigned to high reward values.

Original: As a result, policies with poor performance might be obtained.
Revised: As a result, policies with poor performance might be obtained.

Original: Previous works attempted to ﬁnd direct strategies to effectively reduce the overestimation.
Revised: Previous works attempted to ﬁnd direct strategies to effectively reduce the overestimation.

Original: Hasselt (2010) proposed Double Q-learning, in which the samples are divided into two sets to train two independent Q-function estimators.
Revised: Hasselt (2010) proposed Double Q-learning, in which the samples are divided into two sets to train two independent Q-function estimators.

Original: To diminish the overestimation, one Q-function estimator is used to select actions, and the other one is applied to estimate its value.
Revised: To diminish the overestimation, one Q-function estimator is used to select actions, and the other one is applied to estimate its value.

Original: Fujimoto et al.
Revised: Fujimoto et al.

Original: (2018) proposed mechanisms, including clipped double Q-learning and delayed policy updates, to minimize the overestimation.
Revised: (2018) proposed mechanisms, including clipped double Q-learning and delayed policy updates, to minimize the overestimation.

Original: In contrast to these methods, we focus on actor-critic setting and manage to diminish the approximation error of value function, which is the source of the overestimation, in an indirect but effective way.
Revised: In contrast to these methods, we focus on actor-critic setting and manage to reduce the approximation error of value function, which is the source of the overestimation, in an indirect but effective way.

Original: Before introducing our method, we ﬁrst prove convergence of actor-critic methods under certain conditions basing on the work reported by Kumar, and conclude that it is difﬁcult to meet the conditions when using function approximators.

Original: Then, inspired by the work (Ben-David et al., 2010) on domain adaptation, we use the concepts of domain adaptation  to derive an upper boundary of the approximation error of Q function approximator.
Revised: We use the concepts of domain adaptation (Ben-David et al., 2010) to derive an upper boundary of the approximation error in Q function approximator.

Original: Further, we ﬁnd that the least upper bound of this error can be obtained by minimizing the Kullback-Leibler divergence (KL divergence) between the polices before and after each training iteration.
Revised: Then, we ﬁnd that the least upper bound of this error can be obtained by minimizing the Kullback-Leibler divergence (KL divergence) between new policy and its previous one.

Original: This means minimizing the KL divergence when traning policy can indirectly conﬁne the approximation error of value function.
Revised: This means minimizing the KL divergence when traning policy can stabilize the critic and then conﬁne the approximation error in Q function.

Revised: Interestingly, we arrive at similar conclusion as two literatures Geist et al.

Original: Furthermore, this indirect operation can work together with the mentioned direct strategies for reducing overestimation, for example, clipped double Q-learning.
Revised: Vieillard et al. (2020) by a somewhat different route. In their works, the authors directly studied the effect of KL and entropy regularization in RL and proved that a KL regularization indeed leads to averaging errors made at each iteration of value function update. While our idea is very different from theirs: It is impracticable to minimize the approximation error directly, so instead we try to minimize an upper bound of approximation error.

Original: Then, a new actor-critic method called Error Controlled
Revised: This is similar to Expectation-Maximization Algorithm (Bishop, 2006) which maximize a lower bound of the log-likelihood instead of the log-likelihood directly. We derive an upper boundary of approximation error for Q function approximatorin actor-critic methods, and arrive at a more general conclusion: approximation error can be reduced by keep new policy close to the previous one. Note that KL penalty is a effective way, but not the only way. Furthermore, the mentioned indirect operation (i.e. the KL penalty) can work together with the mentioned direct strategies for reducing overestimation, for example, clipped double Q-learning.

Original:  Actor-critic (ECAC) is established by adopting an effective operation that minimizes the KL divergence to keep the upper bound as low as possible.
Revised: Then, a new actor-critic method called Error Controlled Actor-critic (ECAC) is established by adopting an effective operation that minimizes the KL divergence to keep the upper bound as low as possible.

Original: In other words, this method ensures the similarity between every two consecutive polices in training process and reduces the optimization difﬁculty of value function, so that the error in value function approximators can be decreased.
Revised: In other words, this method ensures the similarity between every two consecutive polices in training process and reduces the optimization difﬁculty of value function, so that the error in Q function approximators can be decreased.

Original: Ablation studies were performed to examine the effectiveness of our proposed strategy for decreasing the approximation error, and comparative evaluations were conducted to verify that our method can outperform other mainstream RL algorithms.
Revised: Ablation studies were performed to examine the effectiveness of our proposed strategy for decreasing the approximation error, and comparative evaluations were conducted to verify that our method can outperform other mainstream RL algorithms.

Original: The main contributions of this paper are summarized as follow: (1) We present a convergence analysis for actor-critic methods by Banach’s Fixed Point Theorem, and ﬁnd a key condition ofconvergence, which is hard to meet when function approximators are used to approximate value functions. (2) We analyze thesource of approximation error in value function of actor-critic methods, and prove that restricting the KL divergence between every two consecutive policies during training process can decrease the approximation error.
Revised: The main contributions of this paper are summarized as follow: (1) We presented an upper boundary of the approximation error in Q function approximator; (2) We proposed a practical actor-critic method—ECAC which decreases the approximation error by restricting the KL divergence between every two consecutive policies and adopt a mechanism to automatically adjust the coefﬁcient of KL term.

Original:  P RELIMINARIES 2. R EINFORCEMENT L EARNING
Revised:  P RELIMINARIES 2. R EINFORCEMENT L EARNING

Original: Reinforcement learning (RL) algorithms are modeled as a mathematical framework called Markov
Revised: Reinforcement learning (RL) algorithms are modeled as a mathematical framework called Markov

Original: Decision Process (MDP).
Revised: Decision Process (MDP).

Original: In each time-step of MDP, an agent generates an action based on current state of its environment, then receives a reward and a new state from the environment.
Revised: In each time-step of MDP, an agent generates an action based on current state of its environment, then receives a reward and a new state from the environment.

Original: Environmental state and agent’s action at time t are denoted as s t ∈ S and a t ∈ A , respectively; S and A denote the state and action spaces respectively, which may be either discrete or continuous.
Revised: Environmental state and agent’s action at time t are denoted as s t ∈ S and a t ∈ A , respectively; S and A denote the state and action spaces respectively, which may be either discrete or continuous.

Original: Environment is described by a reward function, r ( s t , a t ) , and a transition probability distribution, P ( s t
Revised: Environment is described by a reward function, r ( s t , a t ) , and a transition probability distribution, P ( s t

Original: +

Revised: +

Original: =
Revised: =

Original: s (cid:48) | s t = s , a t = a ) .
Revised: s (cid:48) | s t = s , a t = a ) .

Original: Transition probability distribution speciﬁes the probability that the environment will transition to next state.
Revised: Transition probability distribution speciﬁes the probability that the environment will transition to next state.

Original: Initial state distribution is denoted as P 0 ( s ) .
Revised: Initial state distribution is denoted as P 0 ( s ) .

Original: The goal of RL is to ﬁnd a policy, π , that maximizes the expected discounted rewards:
Revised: The goal of RL is to ﬁnd a policy, π , that maximizes the expected discounted rewards:

Original: [Equation]
Revised: [Equation]

Original: where γ denotes a discount rate and 0 ≤ γ ≤ 1 .
Revised: where γ denotes a discount rate and 0 ≤ γ ≤ 1 .

Original: Different interpretations of this optimization goal result in different routes to the their solutions.
Revised: Different interpretations of this optimization goal result in different routes to the their solutions.

Original: RL methods can be roughly divided into three categories: 1) value-based, 2) policy-based, and 3) actor-critic methods.
Revised: RL methods can be roughly divided into three categories: 1) value-based, 2) policy-based, and 3) actor-critic methods.

Original: Value-based methods have the advantage of fast convergence while policy-based methods have better convergence guarantees.
Revised: Value-based methods have the advantage of fast convergence while policy- based methods have better convergence guarantees.

Original: Actor-critic framework tries to trade off a reduction of learning speed for a better convergence guarantee by incorporating value-based methods into policy-based methods.
Revised: Actor-critic framework tries to trade off a reduction of learning speed for a better convergence guarantee by incorporating value-based methods into policy-based methods.

Original: An actor-critic method consists of two components: actor and critic modules.
Revised: An actor-critic method consists of two components: actor and critic modules.

Original: In n -th iteration, actor module tries to ﬁnd a better policy, π ∗ ,n , than the current policy, π n , with the help of value functions (state-value or action-state-value function or both); and the critic module learns value functions, V π n or Q π n , of current policy π n , which outperforms V π n − 1 or Q π n − 1 , by means of π ∗ ,n − 1 and learning algorithms, e.g. temporal-difference (TD) learning.
Revised: In n -th iteration, actor module tries to ﬁnd a better policy, π ∗ ,n , than the current policy, π n , with the help of value functions (state-value or action-state-value function or both); and the critic module learns value functions, V π n or Q π n , of current policy π n , which outperforms V π n − 1 or Q π n − 1 , by means of π ∗ ,n − 1 and learning algorithms, e.g. temporal-difference (TD) learning.

Original: Fig 1 illustrates the update process of an actor-critic method from a perspective similar to policy iteration which is a alternating process of value function and policy updates.
Revised: Fig 1 illustrates the update process of an actor-critic method from a perspective similar to policy iteration which is a alternating process of value function and policy updates.

Original: [Figure]
Revised: [Figure]

Original: Critic

Original: Fit

Original: Critic

Original: Fit

Original: Critic

Original: 2. D OMAIN A DAPTATION
Revised: 2. D OMAIN A DAPTATION

Original: Domain adaptation is the task which aims at adapting a well performing model from a source domain to a different target domain.
Revised: Domain adaptation is the task which aims at adapting a well performing model from a source domain to a different target domain.

Original: Domain is deﬁned as a speciﬁc pair consisting of a distribution, D , on an input space, X , and a labeling function, f : X → R .
Revised: Domain is deﬁned as a speciﬁc pair consisting of a distribution, D , on an input space, X , and a labeling function, f : X → R .

Original: In domain adaption, source and target domains are denoted as < D S , f S > and < D T , f T > , respectively.
Revised: In domain adaption, source and target domains are denoted as < D S , f S > and < D T , f T > , respectively.

Original: A function, h : X → R , is deﬁned as a hypothesis.
Revised: A function, h : X → R , is deﬁned as a hypothesis.

Original: And the difference between a hypothesis, h ( x ) , and a labeling function, f ( x ) , on a source distribution is denoted as follow:[Equation]
Revised: The difference between a hypothesis, h ( x ) , and a labeling function, f ( x ) , on a source distribution is denoted as follow:

Original: P ROBLEM D EFINITION –H IDDEN D ANGERS IN A CTOR - CRITIC

Original: In this section, we investigate the continuous state and action space, which are the most complex setting, to deﬁne an underlying problem in actor-critic methods.

Original: First, we prove convergence of the actor-critic methods that utilize action value function to improve policies.

Original: Then, we ﬁnd a hidden danger in the optimization process of actor-critic methods when use the generalizing approximation methods.

Original: 3. A CTOR - CRITIC C ONVERGENCE T HEOREM

Original: We now consider a simple actor-critic setting, as described below.

Original: There are a Q function to be estimated and a policy to be learned.

Original: At each iteration of optimization, the actor module attempts to ﬁnd a better policy than the current one with the help of the Q function.

Original: We deﬁne this attempting operations by:

Original: [Equation]

Original: where the mapping, f Actor , satisﬁes the following condition:

Original: [Equation]

Original: where ρ π denotes state marginals of the trajectory distribution induced by the policy π ( a | s ) .

Original: The critic module tries to estimate the Q function by using the mentioned bootstrapping operation, for example, the following operation in a recursive form:

Original: [Equation]

Original: where p ( ·| s, a ) denotes the transition probability distribution.

Original: We provide the convergence theorem for this actor-critic setting, and its proof is provided in Appendix A.

Original: Theorem 1 (Actor-critic Convergence Theorem) .

Original: Given the following two conditions: (1) Actor operation maps an actor value function, Q π n , into a policy, π ∗ ,n , that satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Critic operation updates actor

Original: value

Original: function

Original: via Bellman equation (or bootstrapping

Original: operation

Original: Then repeated application of actor-critic operation from any π ∈ Π converges to a policy π ∗ .

Original: 3. H IDDEN D ANGERS IN A CTOR - CRITIC

Original: Although actor-critic methods are theoretically guaranteed to converge to an optimal policy, when using function approximation, it is practically difﬁcult to strictly meet the ﬁrst condition of Theorem 1, i.e. it is practically hard to ﬁnd a better policy, π ∗ ,n ( ·| s ) , by using a noised Q function.

Original: In addition, because of the error of value function approximator, there is an overestimation on the value function as presented in the previous works (Thrun & Schwartz, 1993; Thomas, 2014; Fujimoto et al., 2018).

Original: Further, the overestimated Q values result in the bias of gradient estimation, which will also affect the system’s convergence.

Original:  E RROR C ONTROLLED A CTOR - CRITIC
Revised:  E RROR C ONTROLLED A CTOR - CRITIC

Original: To solve the problem described in Section 3—the approximation error of value function can lead actor-critic algorithms to fail to converge, we propose a newoff-policy actor-critic algorithm called error controlled actor-critic (ECAC) to reduce the impact of the approximation error.
Revised: To reduce the impact of the approximation error, we propose a new actor-critic algorithm called error controlled actor-critic (ECAC).

Original: We derive an upper boundary of the error of Q function approximator, and ﬁnd that the error can be reduced by limiting the KL divergence between every two consecutive policies.
Revised: We derive an upper boundary of the error of Q function approximator, and ﬁnd that the error can be reduced by limiting the KL divergence between every two consecutive policies.

Original: Although this operation is conducted when training the policy, it can indirectly reduce the optimization difﬁculty of Q function.
Revised: Although this operation is conducted when training the policy, it can indirectly reduce the optimization difﬁculty of Q function.

Original: Moreover, this indirect operation can work together with the strategies for diminishing overestimation phenomenon.
Revised: Moreover, this indirect operation can work together with the strategies for diminishing overestimation phenomenon.

Original: We incorporate clipped double-Q learning into the critic module.
Revised: We incorporate clipped double-Q learning into the critic module.

Original: In this section, we present the details of a version of ECAC for continous control tasks, and, more importantly, explain the rationale for conﬁning the KL divergence.
Revised: In this section, we present the details of a version of ECAC for continous control tasks, and, more importantly, explain the rationale for conﬁning the KL divergence.

Original: 4. C RITIC MODULE – LEARNING ACTION VALUE FUNCTIONS
Revised: 3. C RITIC MODULE – LEARNING ACTION VALUE FUNCTIONS

Original: The learning task of critic module is to approximate Q functions.
Revised: The learning task of critic module is to approximate Q functions.

Original: In the critic module of ECAC, two Q functions are approximated by two neural networks with weight φ 1 and φ 2 , respectively.
Revised: In the critic module of ECAC, two Q functions are approximated by two neural networks with weight φ 1 and φ 2 , respectively.

Original: As noted previously, we adopt clipped double-Q strategy to directly reduce overestimation.
Revised: As noted previously, we adopt clipped double-Q strategy to directly reduce overestimation.

Original: Furthermore, we adopt experience replay mechanism (Lin, 1992)—agent’s experiences at each timestep, ( s t , a t , r t +1 , s t +1 ) , are stored in a replay buffer, D ; and training samples are uniformly drawn from this buffer.
Revised: Furthermore, we adopt experience replay mechanism (Lin, 1992)—agent’s experiences at each timestep, ( s t , a t , r t +1 , s t +1 ) , are stored in a replay buffer, D ; and training samples are uniformly drawn from this buffer.

Original: The two Q networks, Q φ 1 and Q φ 2 , are trained by using temporal-difference learning.
Revised: The two Q networks, Q φ 1 and Q φ 2 , are trained by using temporal-difference learning.

Original: Notice that the bootstrapping operation in this setting—uses function approximation—means to minimize the following TD error of Q funtion:
Revised: Notice that the bootstrapping operation in this setting—uses function approximation—means to minimize the following TD error of Q funtion:

Original: [Equation]
Revised: [Equation]

Original: Moreover, clipped double-Q learning uses the smaller of the two Q values to form the TD error.
Revised: Moreover, clipped double-Q learning uses the smaller of the two Q values to form the TD error.

Original: The two Q networks are respectively trained by minimizing the following two loss functions:
Revised: The two Q networks are respectively trained by minimizing the following two loss functions:

Original: [Equation]
Revised: [Equation]

Original: where D denotes replay buffer, p ( ·| s , a ) denotes the transition probability distribution, and π denotes the target policy.
Revised: where D denotes replay buffer, p ( ·| s , a ) denotes the transition probability distribution, and π denotes the target policy.

Original: 4. T HE UPPER BOUND OF THE APPROXIMATION ERROR OF Q FUNCTION
Revised: 3. T HE UPPER BOUND OF THE APPROXIMATION ERROR OF Q FUNCTION

Original: For convenience, we analyze the setting with only one Q function.
Revised: For convenience, we analyze the setting with only one Q function.

Original: Here, the concept of domain adaptation from transfer learning is used to describe the task of critic module and to derive the upper bound of the approximation error of Q function.
Revised: Here, the concept of domain adaptation from transfer learning is used to describe the task of critic module and to derive the upper bound of the approximation error of Q function.

Original: At the ( n + 1) -th iteration, the learning task of critic module is a subtask that can be views adapting the learned Q -network, Q φ n , to the next one, Q φ n +1 .
Revised: At the ( n + 1) -th iteration, the learning task of critic module is a subtask that can be viewed as adapting the learned Q -network, Q φ n , to the next one, Q φ n +1 .

Original: The distribution on s and a is denoted as D n +1 , and the target for training the Q-network, y π n +1 ( s , a ) , is deﬁned by:
Revised: The distribution on s and a is denoted as D n +1 , and the target for training the Q-network, y π n +1 ( s , a ) , is deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: where p ( ·| s , a ) denotes the transition probability distribution, and π ∗ ,n denotes the policy updated in actor module by using Q φ n .
Revised: where p ( ·| s , a ) denotes the transition probability distribution, and π ∗ ,n denotes the policy updated in actor module by using Q φ n .

Original: Furthermore, the difference between the Q -network, Q φ ( s , a ) , and its target at the n -th iteration, y π n , on a source distribution is given by
Revised: Furthermore, the difference between the Q -network, Q φ ( s , a ) , and its target at the n -th iteration, y π n , on a source distribution is given by

Original: [Equation]
Revised: [Equation]

Original: The difference between the Q -network, Q φ ( s , a ) , and its target at ( n + 1) -th iteration, y π n +1 , on a target distribution is given by
Revised: The difference between the Q -network, Q φ ( s , a ) , and its target at ( n + 1) -th iteration, y π n +1 , on a target distribution is given by

Original: [Equation]
Revised: [Equation]

Original: The difference between the Q -network, Q ( s , a ) [Equation] and its target at ( n + 1) -th iteration, y π n +1 , on a source distribution is given by
Revised: The difference between the Q -network, Q ( s , a ) , and its target at ( n + 1) -th iteration, y π n +1 , on a source distribution is given by

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst derive the following inequality:
Revised: We ﬁrst derive the following inequality:

Original: [Equation]
Revised: [Equation]

Original: where the third term in the third line, | (cid:15) T ( Q φ ) − (cid:15) S ( Q φ , y π n +1 ) | , is transformed further as:
Revised: where the third term in the third line, | (cid:15) T ( Q φ ) − (cid:15) S ( Q φ , y π n +1 ) | , is transformed further as:

Original: [Equation]
Revised: [Equation]

Original: Recall that D is actually the replay buffer; and the number of samples in D n +1 is only a little more than in D n [Equation] we ignore the difference between D  and D n +1 .
Revised: Recall that D is actually the replay buffer; and the number of samples in D n +1 is only a little more than in D n , we ignore the difference between D n and D n +1 .

Original: Finally, the upper bound of error in
Revised: Finally, the upper bound of error in

Original: Q-network is determined by
Revised: Q-network is determined by

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: It is noticeable that the third term in the upper bound will be smaller if the more similar the two consecutive policies, π ∗ ,n and π ∗ ,n +1 are.
Revised: It is noticeable that the third term in the upper bound will be smaller if the more similar the two consecutive policies, π ∗ ,n and π ∗ ,n +1 are.

Original: Hence, we can conclude that conﬁning the KL divergence between every two consecutive policies can help limit the approximation error during the optimization process of actor-critic.
Revised: Hence, we can conclude that conﬁning the KL divergence between every two consecutive policies can help limit the approximation error during the optimization process of actor-critic.

Original: This conclusion is used to design the learning method of the policy.
Revised: This conclusion is used to design the learning method of the policy.

Original: 4. A CTOR MODULE – LEARNING A POLICY
Revised: 3. A CTOR MODULE – LEARNING A POLICY

Original: The learning task of the actor module is to learn a parameterized stochastic policy, π θ ( a | s ) .In
Revised: The learning task of the actor module is to learn a parameterized stochastic policy, π θ ( a | s ) .In

Original: order to lower the upper bound of approximation error of Q function (or to reduce the optimization difﬁculty of the Q functions), the goal of the policy is converted from maximizing expect discounted return two parts: maximizing the estimated Q values—the minimun of the two Q approximators— and, concurrently, minimizing the KL divergence between two successive policies.
Revised: order to lower the upper bound of approximation error of Q function (or to reduce the optimization difﬁculty of the Q functions), the goal of the policy is converted from maximizing expect discounted return two parts: maximizing the estimated Q values—the minimum of the two Q approximators— and, concurrently, minimizing the KL divergence between two successive policies.

Original: The optimization objective is speciﬁed by
Revised: The optimization objective is speciﬁed by

Original: [Equation]
Revised: [Equation]

Original: (17) where D is the replay buffer; θ is the parameters of the policy network; θ old is the parameters of the policy updated in the last iteration; (cid:101) a θ ( s ) is the samples drawn from the target stochastic policy π θ ( a | s ) .
Revised: (13) where D is the replay buffer; θ is the parameters of the policy network; θ old is the parameters of the policy updated in the last iteration; (cid:101) a θ ( s ) is the samples drawn from the target stochastic policy π θ ( a | s ) .

Original: Note that, in order to back-propagate the error through this sampling operation, we use a squashed Gaussian policy and the reparameterization trick, i.e. samples are obtained according to
Revised: Note that, in order to back-propagate the error through this sampling operation, we use a Diagonal Gaussian policy and the reparameterization trick, i.e. samples are obtained according to

Original: [Equation]
Revised: [Equation]

Original: where µ θ ( s ) and σ θ ( s ) are the output of the policy network, and denotes the mean and covariance matrix’s diagonal elements, respectively.
Revised: where µ θ ( s ) and σ θ ( s ) are the output of the policy network, and denotes the mean and covariance matrix’s diagonal elements, respectively.

Original: Moreover, the KL divergence between two distributions, for example p ( x ) and q ( x ) , can be thought of the difference between the cross entropy and entropy, which is speciﬁed by
Revised: Moreover, the KL divergence between two distributions, for example p ( x ) and q ( x ) , can be thought of the difference between the cross entropy and entropy, which is speciﬁed by

Original: [Equation]
Revised: [Equation]

Original: where H ( p, q ) denotes the cross entropy between p ( x ) and q ( x ) , and H ( p ) denotes the entropy of p ( x ) .
Revised: where H ( p, q ) denotes the cross entropy between p ( x ) and q ( x ) , and H ( p ) denotes the entropy of p ( x ) .

Original: In practice, we ﬁnd it is more effective to minimize the cross entropy between two successive policies and to maximize the entropy of current policy, separately, than to minimize the KL divergence directly.
Revised: In practice, we ﬁnd it is more effective to minimize the cross entropy between two successive policies and to maximize the entropy of current policy, separately, than to minimize the KL divergence directly.

Original: Hence, we expand the original objective Eq.
Revised: Hence, we expand the original objective Eq.

Original: 17 into the following one:
Revised: into the following one:

Original: [Equation]
Revised: [Equation]

Original:  where α and β denote the coefﬁcients of the cross entropy and the entropy, respectively.
Revised: (16) where α and β denote the coefﬁcients of the cross entropy and the entropy, respectively.

Original: Moreover, we adopt a mechanism to automatically adjust α and β .
Revised: Moreover, we adopt a mechanism to automatically adjust α and β .

Original: To do this, α is adjusted by making the current cross entropy close to a target value.
Revised: To do this, α is adjusted by keeping the current cross entropy close to a target value.

Original: This mechanism is speciﬁed by the following optimization objective:
Revised: This mechanism is speciﬁed by the following optimization objective:

Original: [Equation]
Revised: [Equation]

Original: where δ KL denotes the target KL value and δ entropy denotes the target entropy.
Revised: where δ KL denotes the target KL value and δ entropy denotes the target entropy.

Original: Note that log( · ) is use to ensure that α is greater than 0 .
Revised: Note that log( · ) is use to ensure that α is greater than 0 .

Original: β is adjusted in the same way:
Revised: β is adjusted in the same way:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The overall training process is summarized in Appendix B, and code can be found on our GitHub https://github.com/SingerGao/ECAC .
Revised: The overall training process is summarized in Appendix A, and code can be found on our GitHub https://github.com/SingerGao/ECAC .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The experiments aim to evaluate the effectiveness of the proposed strategy to lower the approximation error and to verify that the proposed method can outperform other mainstream RL algorithms.
Revised: The experiments aim to evaluate the effectiveness of the proposed strategy to lower the approximation error and to verify that the proposed method can outperform other mainstream RL algorithms.

Original: The experiments for ablation study and comparative evaluation are conducted on a few challenging continuous control tasks from the OpenAI Gym(Brockman et al., 2016) environments, which includes Mujoco(Todorov et al., 2012) and Pybullet (Coumans & Bai, 2016–2019) versions.
Revised: The experiments for ablation study and comparative evaluation are conducted on a few challenging continuous control tasks from the OpenAI Gym(Brockman et al., 2016) environments, which includes Mujoco(Todorov et al., 2012) and Pybullet (Coumans & Bai, 2016–2019) versions.

Original: Implementation details and hyperparameters of ECAC are presented in Appendix C.
Revised: Implementation details and hyperparameters of ECAC are presented in Appendix B.

Original: 5. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Ablation studies are performed to verify the contribution of the operation of KL limitation.
Revised: Ablation studies are performed to verify the contribution of the operation of KL limitation.

Original: We compared the performance of ECAC with the method removing KL limitation from ECAC.
Revised: We compared the performance of ECAC with the method removing KL limitation from ECAC.

Original: [Figure]
Revised: [Figure]

Original: Figure 2 compares ﬁve different instances for both methods with and without KL limition using different random seeds; and each instance performs ﬁve evaluation episodes every 1 , 000 environment steps.
Revised: Figure 2 compares ﬁve different instances for both methods with and without KL limition using different random seeds; and each instance performs ﬁve evaluation episodes every 1 , 000 environment steps.

Original: The solid curves corresponds to the mean and the shaded region to the minimum and maximum returns over the ﬁve runs.
Revised: The solid curves corresponds to the mean and the shaded region to the minimum and maximum returns over the ﬁve runs.

Original: The experimental result shows that our method with KL limitation performs better than the one without KL limitation.
Revised: The experimental result shows that our method with KL limitation performs better than the one without KL limitation.

Original: Figure 3 demonstrates that by using ECAC the
Revised: Figure 3 demonstrates that by using ECAC the

Original: KL divergence remains comfortably low during all the training process.[Figure]
Revised: KL divergence remains comfortably low during all the training process.

Original: [Figure] [Figure] Furthermore, to verify that conﬁning the KL divergence can decrease the approximation error in
Revised: Furthermore, to verify that conﬁning the KL divergence can decrease the approximation error in

Original: Q function, we measured the normalized approximation error in 100 random states every 10 ,environment steps.
Revised: Q function, we measured the normalized approximation error in 100 random states every 10 ,environment steps.

Original: The normalized approximation error is speciﬁed by
Revised: The normalized approximation error is speciﬁed by

Original: [Equation]
Revised: [Equation]

Original:  where Q approx is the approximate Q value given by the current Q network, and Q ture is the true discounted return.
Revised: 19) where Q approx is the approximate Q value given by the current Q network, and Q ture is the true discounted return.

Original: The true value is estimated using the average discounted return over 100 episodes  following the current policy, starting from states sampled from the replay buffer.
Revised: The true value is estimated using the average discounted return over 100 episodes [Figure] [Figure] following the current policy, starting from states sampled from the replay buffer.

Original: Figure 4 shows that the method with KL limitation have lower error in the Q function.
Revised: Figure 4 shows that the method with KL limitation has lower error in the Q function.

Original: The results of all the ablation studies indicates that the approximation error of Q functioncan be decreased and the performance of the RL algorithm can be improved by placing restrictions on KL divergence between every two consecutive policies.
Revised: The results of all the ablation studies indicates that the approximation error of Q functioncan be decreased and the performance of the RL algorithm can be improved by placing restrictions on KL divergence between every two consecutive policies.

Original: 5. C OMPARATIVE E VALUATION
Revised: 4. C OMPARATIVE E VALUATION

Original: Comparative evaluation are conducted to verify that our method can outperform other traditional RL methods including A2C, PPO, TD3, and SAC.
Revised: Comparative evaluation are conducted to verify that our method can outperform other traditional RL methods including A2C, PPO, TD3, and SAC.

Original: Five individual runs of each algorithm with different random seeds are done; and each run performs ﬁve evaluation episodes every 1 , 000 environment steps.
Revised: Five individual runs of each algorithm with different random seeds are done; and each run performs ﬁve evaluation episodes every 1 , 000 environment steps.

Original: Figure 5 demonstrates the results of comparative evaluation on the Mujoco version of continuous control tasks.
Revised: Figure 5 demonstrates the results of comparative evaluation on the Mujoco version of continuous control tasks.

Original: ECAC outperforms all other algorithms on the tasks except Hopper-v3 and are only next to TD3 on Hopper-v3.
Revised: ECAC outperforms all other algorithms on the tasks except Hopper-v3 and are only next to TD3 on Hopper-v3.

Original: The results of comparative evaluation on thePybullet version of continuous control tasks  are presented in Appendix D.
Revised: Figure 6 shows that ECAC outperforms all other algorithms on the tasks except HumanoidBulletEnv-v0 and are only next to TD3 on HumanoidBulletEnv-v0.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presented a model-free actor-critic method based on a ﬁnding that the approximation error in value function of RL methods can be decreased by placing restrictions on KL-divergence between every two consecutive policies.
Revised: This paper presented a model-free actor-critic method based on a ﬁnding that the approximation error in value function of RL methods can be decreased by placing restrictions on KL-divergence between every two consecutive policies.

Original: Our method increases the similarity between every two consecutive polices in the training process and therefore reduces the optimization difﬁculty of value function.
Revised: Our method increases the similarity between every two consecutive polices in the training process and therefore reduces the optimization difﬁculty of value function.

Original: In the ablation studies, we compare the approximation error in Q function, KL divergence, and performance of the methods with and without KL limitation.
Revised: In the ablation studies, we compare the approximation error in Q function, KL divergence, and performance of the methods with and without KL limitation.

Original: Theexperimental results  show that the proposed method can decrease the approximation error and improved the performance.
Revised: The results of ablation study show that the proposed method can decrease the approximation error and improved the performance.

Original: Moreover, the results of comparative evaluation demonstrate that ECAC outperforms other model-free deep RL algorithm including A2C, PPO, TD3, and SAC.APPENDICES
Revised: Moreover, the results of comparative evaluation demonstrate that ECAC outperforms other modelfree deep RL algorithm including A2C, PPO, TD3, and SAC.

Original: A P ROOF OF CONVERGENCE OF ACTOR - CRITIC METHODS

Original: We prove the convergence of actor-critic methods by using Banach’s ﬁxed point theorem.

Original: Deﬁnition 1 (Contraction) .

Original: Let X be a metric space equipped with a distance d .

Original: f : X → X is considered as a contraction mapping, if there is a constant, 0 ≤ λ < 1 , thus:

Original: [Equation]
Revised: [Figure]

Original: Lemma 1 (Banach’s Fixed Point Theorem) .

Original: Let f be a contraction on a complete metric space X .

Original: Then f has a unique ﬁxed point x ∗ ∈ X .

Original: Proof of Theorem 1.

Original: For convenience, the mentioned two actor-critic operations, i.e. equation 3 and 5, is combined into the following recursive expression:

Original: [Equation]
Revised: [Figure]

Original: Let ( X, d ) denotes the metric space of action value function.

Original: The set X = { Q ( s , a ) } is the hypothesis space of value functions; d is a metric measuring the distance between two action value functions, which is deﬁned as follow:

Original: [Equation]

Original: Let Q 1 and Q 2 be two action value functions, then we have:

Original: [Equation]

Original: [Equation]

Original: In Deﬁnition

Original: [Equation]

Original: we determine that f AC is a contraction mapping for the complete metric space ( X, d ) ; in addition, since the results of operations of actor-critic always stay in hypothesis space of value functions, the space, f AC , is complete. Then the condition of lemma 1 holds. According to the Banach’s ﬁxed point theorem, the repeated application of f AC will converge to a unique optimal actor value function; and an optimal policy can be derived by using the Banach’s ﬁxed point theorem [Equation]
Revised: APPENDICES

Original: B P SEUDO CODE OF ECAC
Revised: A P SEUDO CODE OF ECAC

Original: Require: initial policy patameters, θ ; Q function parameters, φ 1 and φ 2 ; discount rate, γ ; the coefﬁcient of KL term, β ; empty replay buffer D ; the number of episodes, M ; the maximum number of steps in each episode, T ; minibatch size, N .
Revised: Require: initial policy patameters, θ ; Q function parameters, φ 1 and φ 2 ; discount rate, γ ; the coefﬁcient of KL term, β ; empty replay buffer D ; the number of episodes, M ; the maximum number of steps in each episode, T ; minibatch size, N .

Original: Ensure: optimal policy parameters θ ∗ .
Revised: Ensure: optimal policy parameters θ ∗ .

Original: 1: for episode = 1 , M do 2:
Revised: 1: for episode = 1 , M do 2:

Original: Reset environment.
Revised: Reset environment.

Original: 3: for t = 1 , T do 4:
Revised: 3: for t = 1 , T do 4:

Original: Obeserve state s and select action a ∼ π ( ·| s ) .
Revised: Obeserve state s and select action a ∼ π ( ·| s ) .

Original: Execute a in the enviroment.
Revised: Execute a in the enviroment.

Original: Observe next state s (cid:48) and reward r .
Revised: Observe next state s (cid:48) and reward r .

Original: Store transition ( s , a , r, s (cid:48) ) in replay buffer D .
Revised: Store transition ( s , a , r, s (cid:48) ) in replay buffer D .

Original: Randomly sample a minibatch of N transitions, B = { ( s , a , r, s (cid:48) ) } from D .
Revised: Randomly sample a minibatch of N transitions, B = { ( s , a , r, s (cid:48) ) } from D .

Original: Compute targets for the Q functions:
Revised: Compute targets for the Q functions:

Original: [Equation]
Revised: [Equation]

Original: Update Q functions by one step of gradient descent using
Revised: Update Q functions by one step of gradient descent using

Original: [Equation]
Revised: [Equation]

Original: Backup old policy, θ old ← θ .
Revised: Backup old policy, θ old ← θ .

Original: Update α and β by one step of gradient ascent using
Revised: Update α and β by one step of gradient ascent using

Original: [Equation]
Revised: [Equation]

Original: s ))
Revised: s ))

Original: − δ entropy )] , where δ KL and δ entropy denote target KL divergence and target entropy, respectively.
Revised: − δ entropy )] , where δ KL and δ entropy denote target KL divergence and target entropy, respectively.

Original: Update policy by one step of gradient ascent using
Revised: Update policy by one step of gradient ascent using

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) a θ ( s ) is a sample from π θ ( ·| s ) , which is differentiable with respect to θ via the reparameterization trick.
Revised: where (cid:101) a θ ( s ) is a sample from π θ ( ·| s ) , which is differentiable with respect to θ via the reparameterization trick.

Original: 14: end for 15: end for
Revised: 14: end for 15: end for

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C I MPLEMENTATION DETAILS AND HYPERPARAMETERS OF ECAC
Revised: B I MPLEMENTATION DETAILS AND HYPERPARAMETERS OF ECAC

Original: For the implementation of ECAC
Revised: For the implementation of ECAC

Original: [Table]
Revised: [Table]

Original: a two layer feedforward neural network of 256 hidden units, with rectiﬁed linear units (ReLU) between each layer are used to build the two Q functions an the policy.
Revised: a two layer feedforward neural network of 256 hidden units [Table] with rectiﬁed linear units (ReLU) between each layer are used to build the two Q functions an the policy.

Original: The parameters of the neural networks and the two coefﬁcients (i
Revised: The parameters of the neural networks and the two coefﬁcients (i

Original: [Table]
Revised: [Table]

Original: e α and β ) are optimized by using
Revised: e α and β ) are optimized by using

Original: Adam(Kingma & Ba, 2014).
Revised: Adam(Kingma & Ba, 2014).

Original: The hyperparameters of ECAC are listed in Table 1.
Revised: The hyperparameters of ECAC are listed in Table 1.

Original: Moreover, we adopt the target policy smoothing regularization technique in ECAC, which is proposed in  Fujimoto et al.
Revised: Moreover, we adopt the target network technique in ECAC, which is common in previous works (Lillicrap et al., 2016; Fujimoto et al., 2018).

Original: We also adopt reward scale trick which is presented in Haarnoja et al.
Revised: We also adopt reward scale trick which is presented in Haarnoja et al.

Original: (2018); and the reward scale parameter is listed in Table2.
Revised: (2018); and the reward scale parameter is listed in Table

Original: [Figure]
Revised: [Table]

Original: D A DDITIONAL R ESULTS OF C OMPARATIVE E VALUATION

Original: Figure 6 shows that ECAC outperforms all other algorithms on the tasks except

Original: HumanoidBulletEnv-v0 and are only next to TD3 on HumanoidBulletEnv-v0.