Original: E RROR C ONTROLLED A CTOR -C RITIC M ETHOD TO R EINFORCEMENT L EARNING
Revised: E RROR C ONTROLLED A CTOR -C RITIC M ETHOD TO R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In the reinforcement learning (RL) algorithms which incorporate function approximation methods, the approximation error of value function inevitably causes overestimation phenomenon and impacts algorithm performances.
Revised: In the reinforcement learning (RL) algorithms which incorporate function approximation methods, the approximation error of value function inevitably causes overestimation phenomenon and impacts algorithm performances.

Original: To mitigate the negative effects caused by approximation error, we propose a new actor-critic algorithm called Error Controlled Actor-critic which ensures conﬁning the approximation error in value function.
Revised: To mitigate the negative effects caused by approximation error, we propose a new actor-critic algorithm called Error Controlled Actor-critic which ensures conﬁning the approximation error in value function.

Original: In this paper, we derive an upper boundary of approximation error for Q function approximator in actor-critic methods, and ﬁnd that the error can be lowered by keep new policy close to the previous one during the training phase of the policy.
Revised: In this paper, we derive an upper boundary of approximation error for Q function approximator in actor-critic methods, and ﬁnd that the error can be lowered by keep new policy close to the previous one during the training phase of the policy.

Original: The results of experiments on a range of continuous control tasks from OpenAI gym suite demonstrate that the proposed actor-critic algorithm apparently reduces the approximation error and signiﬁcantly outperforms other model-free RL algorithms.
Revised: The results of experiments on a range of continuous control tasks from OpenAI gym suite demonstrate that the proposed actor-critic algorithm apparently reduces the approximation error and signiﬁcantly outperforms other model-free RL algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) algorithms are combined with function approximation methods to adapt to the application scenarios whose state spaces are combinatorial, large, or even continuous.
Revised: Reinforcement learning (RL) algorithms are combined with function approximation methods to adapt to the application scenarios whose state spaces are combinatorial, large, or even continuous.

Original: Many function approximation methods RL methods, including the Fourier basis (Konidaris et al., 2011), kernel regression (Xu, 2006; Barreto et al., 2011;
Revised: Many function approximation methods RL methods, including the Fourier basis (Konidaris et al., 2011), kernel regression (Xu, 2006; Barreto et al., 2011;

Original: Bhat et al., 2012), and neural neworks (Barto et al., 1982; Tesauro, 1992; Boyan et al., 1992; Gullapalli, 1992) have been used to learn value functions.
Revised: Bhat et al., 2012), and neural neworks (Barto et al., 1982; Tesauro, 1992; Boyan et al., 1992; Gullapalli, 1992) have been used to learn value functions.

Original: In recent years, many deep reinforcement learning (DRL) methods were implemented by incorporating deep learning into RL methods.
Revised: In recent years, many deep reinforcement learning (DRL) methods were implemented by incorporating deep learning into RL methods.

Original: Deep Q-learning Network (DQN) (Mnih et al., 2013) reported by Mnih in 2013 is a typical work that uses a deep convolutional neural network (CNN) to represent a suitable action value function estimating future rewards (returns); it successfully learned end-to-end control policies for seven Atari 2600 games directly from large state spaces.
Revised: Deep Q-learning Network (DQN) (Mnih et al., 2013) reported by Mnih in 2013 is a typical work that uses a deep convolutional neural network (CNN) to represent a suitable action value function estimating future rewards (returns); it successfully learned end-to-end control policies for seven Atari 2600 games directly from large state spaces.

Original: Thereafter, deep RL methods, such as Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016),
Revised: Thereafter, deep RL methods, such as Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016),

Original: Proximal Policy Optimization (PPO) (Schulman et al., 2017), Twin Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., 2018), and Soft Actor-Critic (SAC) (Haarnoja et al., 2018), started to become mainstream in the ﬁeld of RL.
Revised: Proximal Policy Optimization (PPO) (Schulman et al., 2017), Twin Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., 2018), and Soft Actor-Critic (SAC) (Haarnoja et al., 2018), started to become mainstream in the ﬁeld of RL.

Original: Althouth function approximation methods have assisted reinforcement learning (RL) algorithms to perform well in complex problems by providing great representation power; however, they also cause an issue called overestimation phenomenon that jeopardize the optimization process of RL algorithms.
Revised: Althouth function approximation methods have assisted reinforcement learning (RL) algorithms to perform well in complex problems by providing great representation power; however, they also cause an issue called overestimation phenomenon that jeopardize the optimization process of RL algorithms.

Original: Thrun & Schwartz (1993) presented a theoretical analysis of this systematic overestimation phenomenon in Q-learning methods that use function approximation methods.
Revised: Thrun & Schwartz (1993) presented a theoretical analysis of this systematic overestimation phenomenon in Q-learning methods that use function approximation methods.

Original: Similar problem persists in the actor-critic methods employed function approximation methods.
Revised: Similar problem persists in the actor-critic methods employed function approximation methods.

Original: Thomas (2014) reported that several natural actor-critic algorithms use biased estimates of policy gradient to update parameters when using function approximation to approximate the action value function.
Revised: Thomas (2014) reported that several natural actor-critic algorithms use biased estimates of policy gradient to update parameters when using function approximation to approximate the action value function.

Original: Fujimoto et al.
Revised: Fujimoto et al.

Original: (2018) proved that the value estimation in the deterministic policy gradient method also lead to overestimation problem.
Revised: (2018) proved that the value estimation in the deterministic policy gradient method also lead to overestimation problem.

Original: In brief, the approximation errors of value functions caused the inaccuracy of estimated values, and such inaccuracy induced the overestimation on value function; so that poor performances might be assigned to high reward values.
Revised: In brief, the approximation errors of value functions caused the inaccuracy of estimated values, and such inaccuracy induced the overestimation on value function; so that poor performances might be assigned to high reward values.

Original: As a result, policies with poor performance might be obtained.
Revised: As a result, policies with poor performance might be obtained.

Original: Previous works attempted to ﬁnd direct strategies to effectively reduce the overestimation.
Revised: Previous works attempted to ﬁnd direct strategies to effectively reduce the overestimation.

Original: Hasselt (2010) proposed Double Q-learning, in which the samples are divided into two sets to train two independent Q-function estimators.
Revised: Hasselt (2010) proposed Double Q-learning, in which the samples are divided into two sets to train two independent Q-function estimators.

Original: To diminish the overestimation, one Q-function estimator is used to select actions, and the other one is applied to estimate its value.
Revised: To diminish the overestimation, one Q-function estimator is used to select actions, and the other one is applied to estimate its value.

Original: Fujimoto et al.
Revised: Fujimoto et al.

Original: (2018) proposed mechanisms, including clipped double Q-learning and delayed policy updates, to minimize the overestimation.
Revised: (2018) proposed mechanisms, including clipped double Q-learning and delayed policy updates, to minimize the overestimation.

Original: In contrast to these methods, we focus on actor-critic setting and manage to reduce the approximation error of value function, which is the source of the overestimation, in an indirect but effective way.
Revised: In contrast to these methods, we focus on actor-critic setting and manage to reduce the approximation error of value function, which is the source of the overestimation, in an indirect but effective way.

Original: We use the concepts of domain adaptation (Ben-David et al., 2010) to derive an upper boundary of the approximation error in Q function approximator.
Revised: We use the concepts of domain adaptation (Ben-David et al., 2010) to derive an upper boundary of the approximation error in Q function approximator.

Original: Then, we ﬁnd that the least upper bound of this error can be obtained by minimizing the Kullback-Leibler divergence (KL divergence) between new policy and its previous one.
Revised: Then, we ﬁnd that the least upper bound of this error can be obtained by minimizing the Kullback-Leibler divergence (KL divergence) between new policy and its previous one.

Original: This means minimizing the KL divergence when traning policy can stabilize the critic and then conﬁne the approximation error in Q function.
Revised: This means minimizing the KL divergence when traning policy can stabilize the critic and then conﬁne the approximation error in Q function.

Original: Interestingly, we arrive at similar conclusion as two literatures Geist et al.
Revised: Interestingly, we arrive at similar conclusion as two literatures Geist et al.

Original: Vieillard et al.
Revised: Vieillard et al.

Original: (2020) by a somewhat different route.
Revised: (2020) by a somewhat different route.

Original: In their works, the authors directly studied the effect of KL and entropy regularization in RL and proved that a KL regularization indeed leads to averaging errors made at each iteration of value function update.
Revised: In their works, the authors directly studied the effect of KL and entropy regularization in RL and proved that a KL regularization indeed leads to averaging errors made at each iteration of value function update.

Original: While our idea is very different from theirs: It is impracticable to minimize the approximation error directly, so instead we try to minimize an upper bound of approximation error.
Revised: While our idea is very different from theirs: It is impracticable to minimize the approximation error directly, so instead we try to minimize an upper bound of approximation error.

Original: This is similar to Expectation-Maximization Algorithm (Bishop, 2006) which maximize a lower bound of log-likelihood instead of log-likelihood directly.
Revised: This is similar to Expectation-Maximization Algorithm (Bishop, 2006) which maximize a lower bound of log-likelihood instead of log-likelihood directly.

Original: We derive an upper boundary of approximation error for Q function approximatorin actor-critic methods, and arrive at a more general conclusion: approximation error can be reduced by keep new policy close to the previous one.
Revised: We derive an upper boundary of approximation error for Q function approximatorin actor-critic methods, and arrive at a more general conclusion: approximation error can be reduced by keep new policy close to the previous one.

Original: Note that KL penalty is a effective way, but not the only way.
Revised: Note that KL penalty is a effective way, but not the only way.

Original: Furthermore, the mentioned indirect operation (i.e. the KL penalty) can work together with the mentioned direct strategies for reducing overestimation, for example, clipped double Q-learning.
Revised: Furthermore, the mentioned indirect operation (i.e. the KL penalty) can work together with the mentioned direct strategies for reducing overestimation, for example, clipped double Q-learning.

Original: Then, a new actor-critic method called Error Controlled Actor-critic (ECAC) is established by adopting an effective operation that minimizes the KL divergence to keep the upper bound as low as possible.
Revised: Then, a new actor-critic method called Error Controlled Actor-critic (ECAC) is established by adopting an effective operation that minimizes the KL divergence to keep the upper bound as low as possible.

Original: In other words, this method ensures the similarity between every two consecutive polices in training process and reduces the optimization difﬁculty of value function, so that the error in Q function approximators can be decreased.
Revised: In other words, this method ensures the similarity between every two consecutive polices in training process and reduces the optimization difﬁculty of value function, so that the error in Q function approximators can be decreased.

Original: Ablation studies were performed to examine the effectiveness of our proposed strategy for decreasing the approximation error, and comparative evaluations were conducted to verify that our method can outperform other mainstream RL algorithms.
Revised: Ablation studies were performed to examine the effectiveness of our proposed strategy for decreasing the approximation error, and comparative evaluations were conducted to verify that our method can outperform other mainstream RL algorithms.

Original: The main contributions of this paper are summarized as follow: (1) We presented an upper boundary of the approximation error in Q function approximator; (2) We proposed a practical actor-critic method—ECAC which decreases the approximation error by restricting the KL divergence between every two consecutive policies and adopt a mechanism to automatically adjust the coefﬁcient of KL term.
Revised: The main contributions of this paper are summarized as follow: (1) We presented an upper boundary of the approximation error in Q function approximator; (2) We proposed a practical actor-critic method—ECAC which decreases the approximation error by restricting the KL divergence between every two consecutive policies and adopt a mechanism to automatically adjust the coefﬁcient of KL term.

Original:  P RELIMINARIES 2. R EINFORCEMENT L EARNING
Revised:  P RELIMINARIES 2. R EINFORCEMENT L EARNING

Original: Reinforcement learning (RL) algorithms are modeled as a mathematical framework called Markov
Revised: Reinforcement learning (RL) algorithms are modeled as a mathematical framework called Markov

Original: Decision Process (MDP).
Revised: Decision Process (MDP).

Original: In each time-step of MDP, an agent generates an action based on current state of its environment, then receives a reward and a new state from the environment.
Revised: In each time-step of MDP, an agent generates an action based on current state of its environment, then receives a reward and a new state from the environment.

Original: Environmental state and agent’s action at time t are denoted as s t ∈ S and a t ∈ A , respectively; S and A denote the state and action spaces respectively, which may be either discrete or continuous.
Revised: Environmental state and agent’s action at time t are denoted as s t ∈ S and a t ∈ A , respectively; S and A denote the state and action spaces respectively, which may be either discrete or continuous.

Original: Environment is described by a reward function , r ( s t , a t ) , and a transition probability distribution , Pr ( s t +1 = s (cid:48) | s t = s , a t = a ) .
Revised: Environment is described by a reward function , r ( s t , a t ) , and a transition probability distribution , Pr ( s t +1 = s (cid:48) | s t = s , a t = a ) .

Original: Transition probability distribution speciﬁes the probability that the environment will transition to next state.
Revised: Transition probability distribution speciﬁes the probability that the environment will transition to next state.

Original: Initial state distribution is denoted as Pr 0 ( s ) .
Revised: Initial state distribution is denoted as Pr 0 ( s ) .

Original: Let π denotes a policy, η ( π ) denotes its expected discounted rewards:
Revised: Let π denotes a policy, η ( π ) denotes its expected discounted rewards:

Original: [Equation]
Revised: [Equation]

Original: where γ denotes a discount rate and 0 ≤ γ ≤ 1 .
Revised: where γ denotes a discount rate and 0 ≤ γ ≤ 1 .

Original: The goal of RL is to ﬁnd a policy, π ∗ , that maximizes a performance function over policy, J ( π ) , which measures the performance of policy:
Revised: The goal of RL is to ﬁnd a policy, π ∗ , that maximizes a performance function over policy, J ( π ) , which measures the performance of policy:

Original: [Equation]
Revised: [Equation]

Original: A natural form of J ( π ) is η ( π ) .
Revised: A natural form of J ( π ) is η ( π ) .

Original: Different interpretations of this optimization goal lead to different routes to the their solutions.
Revised: Different interpretations of this optimization goal lead to different routes to the their solutions.

Original: Almost all reinforcement learning algorithms involve estimating value functions, including statevalue and action-value functions.
Revised: Almost all reinforcement learning algorithms involve estimating value functions, including statevalue and action-value functions.

Original: State-value function, V π ( s ) , gives the expected sum of discounted reward when starting in s and following a given policy, π .
Revised: State-value function, V π ( s ) , gives the expected sum of discounted reward when starting in s and following a given policy, π .

Original: V π ( s ) speciﬁed by:
Revised: V π ( s ) speciﬁed by:

Original: [Equation]
Revised: [Equation]

Original: Similarly, action-value function, Q π ( s , a ) , is given by:
Revised: Similarly, action-value function, Q π ( s , a ) , is given by:

Original: [Equation]
Revised: [Equation]

Original: 2. A CTOR - CRITIC A RCHITECTURE
Revised: 2. A CTOR - CRITIC A RCHITECTURE

Original: To avoid confusion, by default, we discuss only RL methods with function approximation in this section.
Revised: To avoid confusion, by default, we discuss only RL methods with function approximation in this section.

Original: RL methods can be roughly divided into three categories: 1) value-based , 2) policy-based , and 3) actor-critic methods.
Revised: RL methods can be roughly divided into three categories: 1) value-based , 2) policy-based , and 3) actor-critic methods.

Original: Value-based method only learn value functions (state-value or action-value functions), and have the advantage of fast convergence.
Revised: Value-based method only learn value functions (state-value or action-value functions), and have the advantage of fast convergence.

Original: Policy-based methods primarily learn parameterized policies.
Revised: Policy-based methods primarily learn parameterized policies.

Original: A parameterized policy (with parameter vector, θ ) is either a distribution over actions given a state, π θ ( a | s ) , or a deterministic function, a = π θ ( s ) .
Revised: A parameterized policy (with parameter vector, θ ) is either a distribution over actions given a state, π θ ( a | s ) , or a deterministic function, a = π θ ( s ) .

Original: Their basic update is θ n +

Revised: Their basic update is θ n +

Original: = θ n + α ∇ J ( θ n ) where is learning rate.
Revised: = θ n + α ∇ J ( θ n ) where is learning rate.

Original: Policy based methods show better convergence guarantees but have high variance in gradient estimates.
Revised: Policy based methods show better convergence guarantees but have high variance in gradient estimates.

Original: Actor-critic methods learn both value functions and policies and use value functions to improve policies.
Revised: Actor-critic methods learn both value functions and policies and use value functions to improve policies.

Original: In this way, they trade off small bias in gradient estimates to low variance in gradient estimates.
Revised: In this way, they trade off small bias in gradient estimates to low variance in gradient estimates.

Original: Actor-critic architecture (Peters & Schaal, 2008; Degris et al., 2013; Sutton & Barto, 2018) consists of two components: actor and critic modules.
Revised: Actor-critic architecture (Peters & Schaal, 2008; Degris et al., 2013; Sutton & Barto, 2018) consists of two components: actor and critic modules.

Original: Critic module learns learns state-value function,
Revised: Critic module learns learns state-value function,

Original: V φ ( s ) , or action-value function, Q φ ( s , a ) or both of them, usually by temporal-difference (TD) methods.
Revised: V φ ( s ) , or action-value function, Q φ ( s , a ) or both of them, usually by temporal-difference (TD) methods.

Original: Actor module learns a stochastic policy, π θ ( a | s ) , or a deterministic policy, a = π θ ( s ) , and utilizes value function to improve the policy.
Revised: Actor module learns a stochastic policy, π θ ( a | s ) , or a deterministic policy, a = π θ ( s ) , and utilizes value function to improve the policy.

Original: For example, in actor module of DDPG (Lillicrap et al., 2016), the policy is updated by using the following performance function
Revised: For example, in actor module of DDPG (Lillicrap et al., 2016), the policy is updated by using the following performance function

Original: [Equation]
Revised: [Equation]

Original: where π θ ( s t ) is a deterministic policy.
Revised: where π θ ( s t ) is a deterministic policy.

Original: 2. D OMAIN A DAPTATION
Revised: 2. D OMAIN A DAPTATION

Original: Domain adaptation is a task which aims at adapting a well performing model from a source domain to a different target domain.
Revised: Domain adaptation is a task which aims at adapting a well performing model from a source domain to a different target domain.

Original: It is used to describe the task of critic module in section 3.2.
Revised: It is used to describe the task of critic module in section 3.2.

Original: The learning task of critic module is viewed as adapting a learned Q function approximator to next one, and the target error equates to the approximation error at current iteration of critic update.
Revised: The learning task of critic module is viewed as adapting a learned Q function approximator to next one, and the target error equates to the approximation error at current iteration of critic update.

Original: Here, we present some concepts in domain adaptation, including domain, source error, and target error.
Revised: Here, we present some concepts in domain adaptation, including domain, source error, and target error.

Original: Domain is deﬁned as a speciﬁc pair consisting of a distribution, P , on an input space, X , and a labeling function, f : X → R .
Revised: Domain is deﬁned as a speciﬁc pair consisting of a distribution, P , on an input space, X , and a labeling function, f : X → R .

Original: In domain adaption, source and target domains are denoted as (cid:104) P S , f S (cid:105) and (cid:104) P T , f T (cid:105) , respectively.
Revised: In domain adaption, source and target domains are denoted as (cid:104) P S , f S (cid:105) and (cid:104) P T , f T (cid:105) , respectively.

Original: A function, h : X → R , is deﬁned as a hypothesis.
Revised: A function, h : X → R , is deﬁned as a hypothesis.

Original: Source error is the difference between a hypothesis, h ( x ) , and a labeling function of source domain, f S ( x ) , on a source distribution which is denoted as follow:
Revised: Source error is the difference between a hypothesis, h ( x ) , and a labeling function of source domain, f S ( x ) , on a source distribution which is denoted as follow:

Original: [Equation]
Revised: [Equation]

Original: Target error is the difference between a hypothesis, h ( x ) , and a labeling function of target domain, f T ( x ) , on a target distribution which is denoted as follow:
Revised: Target error is the difference between a hypothesis, h ( x ) , and a labeling function of target domain, f T ( x ) , on a target distribution which is denoted as follow:

Original: [Equation]
Revised: [Equation]

Original: | ] .
Revised: | ] .

Original:  E RROR C ONTROLLED A CTOR - CRITIC
Revised:  E RROR C ONTROLLED A CTOR - CRITIC

Original: To reduce the impact of the approximation error, we propose a new actor-critic algorithm called error controlled actor-critic (ECAC).
Revised: To reduce the impact of the approximation error, we propose a new actor-critic algorithm called error controlled actor-critic (ECAC).

Original: We present the details of a version of ECAC for continous control tasks, and, more importantly, explain the rationale for conﬁning the KL divergence.
Revised: We present the details of a version of ECAC for continous control tasks, and, more importantly, explain the rationale for conﬁning the KL divergence.

Original: In section 3.2, we will show that, at each iteration of critic update, the target error equals to the approximation error of Q function approximator.
Revised: In section 3.2, we will show that, at each iteration of critic update, the target error equals to the approximation error of Q function approximator.

Original: Then, we derive an upper boundary of the error, and ﬁnd that the error can be reduced by limiting the KL divergence between every two consecutive policies.
Revised: Then, we derive an upper boundary of the error, and ﬁnd that the error can be reduced by limiting the KL divergence between every two consecutive policies.

Original: Although this operation is conducted when training the policy, it can indirectly reduce the optimization difﬁculty of
Revised: Although this operation is conducted when training the policy, it can indirectly reduce the optimization difﬁculty of

Original: Q function.
Revised: Q function.

Original: Moreover, this indirect operation can work together with the strategies for diminishing overestimation phenomenon.
Revised: Moreover, this indirect operation can work together with the strategies for diminishing overestimation phenomenon.

Original: We incorporate clipped double-Q learning into the critic module.
Revised: We incorporate clipped double-Q learning into the critic module.

Original: 3. C RITIC MODULE – LEARNING ACTION VALUE FUNCTIONS
Revised: 3. C RITIC MODULE – LEARNING ACTION VALUE FUNCTIONS

Original: The learning task of critic module is to approximate Q functions.
Revised: The learning task of critic module is to approximate Q functions.

Original: In the critic module of ECAC, two Q functions are approximated by two neural networks with weight φ (1) and φ (2) , respectively.
Revised: In the critic module of ECAC, two Q functions are approximated by two neural networks with weight φ (1) and φ (2) , respectively.

Original: As noted previously, we adopt clipped double-Q strategy to directly reduce overestimation.
Revised: As noted previously, we adopt clipped double-Q strategy to directly reduce overestimation.

Original: Furthermore, we adopt experience replay mechanism (Lin, 1992)—agent’s experiences at each timestep, ( s t , a t , r t +1 , s t +1 ) , are stored in a replay buffer, D ; and training samples are uniformly drawn from this buffer.
Revised: Furthermore, we adopt experience replay mechanism (Lin, 1992)—agent’s experiences at each timestep, ( s t , a t , r t +1 , s t +1 ) , are stored in a replay buffer, D ; and training samples are uniformly drawn from this buffer.

Original: The two Q networks, Q φ (1) and Q φ (2) , are trained by using temporal-difference learning.
Revised: The two Q networks, Q φ (1) and Q φ (2) , are trained by using temporal-difference learning.

Original: Notice that the bootstrapping operation in this setting—uses function approximation—means to minimize the following two TD errors of Q funtion:
Revised: Notice that the bootstrapping operation in this setting—uses function approximation—means to minimize the following two TD errors of Q funtion:

Original: [Equation]
Revised: [Equation]

Original: Moreover, clipped double-Q learning uses the smaller of the two Q values to form the TD error.
Revised: Moreover, clipped double-Q learning uses the smaller of the two Q values to form the TD error.

Original: The two Q networks are respectively trained by minimizing the following two loss functions:
Revised: The two Q networks are respectively trained by minimizing the following two loss functions:

Original: [Equation]
Revised: [Equation]

Original: where D denotes replay buffer, P D denotes the distribution that describes the likelihoods of samples drawn from D uniformly, Pr ( ·| s , a ) denotes the transition probability distribution, and π denotes the target policy.
Revised: where D denotes replay buffer, P D denotes the distribution that describes the likelihoods of samples drawn from D uniformly, Pr ( ·| s , a ) denotes the transition probability distribution, and π denotes the target policy.

Original: 3. A N UPPER BOUND OF THE APPROXIMATION ERROR OF Q FUNCTION
Revised: 3. A N UPPER BOUND OF THE APPROXIMATION ERROR OF Q FUNCTION

Original: For convenience, we analyze the setting with only one Q function.
Revised: For convenience, we analyze the setting with only one Q function.

Original: The concept of domain adaptation is used to describe the task of critic module.
Revised: The concept of domain adaptation is used to describe the task of critic module.

Original: The learning task of critic module can be viewed as adapting the learned Q-network to a new Q function for newly learned policy.
Revised: The learning task of critic module can be viewed as adapting the learned Q-network to a new Q function for newly learned policy.

Original: Thus, naturally, target error, i.e. Eq.
Revised: Thus, naturally, target error, i.e. Eq.

Original: (7), equates to the approximation error at current iteration of critic update.
Revised: (7), equates to the approximation error at current iteration of critic update.

Original: In this section, we derive an upper bound of the approximation error of Q function and ﬁnd that the upper bound of approximation error will be smaller if the more similar the two consecutive policies.
Revised: In this section, we derive an upper bound of the approximation error of Q function and ﬁnd that the upper bound of approximation error will be smaller if the more similar the two consecutive policies.

Original: Fig 1 illustrates the training process of an actor-critic method which is a alternating process of value function.
Revised: Fig 1 illustrates the training process of an actor-critic method which is a alternating process of value function.

Original: At ( n + 1) -th iteration, the critic module tries to ﬁt the value function, Q π θn , by means of π θ n .
Revised: At ( n + 1) -th iteration, the critic module tries to ﬁt the value function, Q π θn , by means of π θ n .

Original: But because of appriximation error, the actually obtained Q-network (approximator), Q φ n +1 , is not equal to Q π θn .
Revised: But because of appriximation error, the actually obtained Q-network (approximator), Q φ n +1 , is not equal to Q π θn .

Original: This can be expressed by the following equation:
Revised: This can be expressed by the following equation:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) s , a n +1 denotes the appriximation error in Q function given state, s , and action, a .
Revised: where (cid:15) s , a n +1 denotes the appriximation error in Q function given state, s , and action, a .

Original: This can be viewed as adapting the learned Q-network, Q φ n , to the value function, Q π θn .
Revised: This can be viewed as adapting the learned Q-network, Q φ n , to the value function, Q π θn .

Original: Hence, the source distribution is P D n , and the target distribution is P D n +1 .
Revised: Hence, the source distribution is P D n , and the target distribution is P D n +1 .

Original: As mentioned at the argument following
Revised: As mentioned at the argument following

Original: Eq.
Revised: Eq.

Original: (9) and (10), P D denotes the distribution that describes the likelihoods of samples drawn from replay buffer D uniformly.
Revised: (9) and (10), P D denotes the distribution that describes the likelihoods of samples drawn from replay buffer D uniformly.

Original: D n and D n +1 are replay buffers at n -th and ( n + 1) -th iteration, respectively.
Revised: D n and D n +1 are replay buffers at n -th and ( n + 1) -th iteration, respectively.

Original: Q φ n is the labeling function .
Revised: Q φ n is the labeling function .

Original: Clearly, the target error here equals to the approximation error at current iteration of critic update .
Revised: Clearly, the target error here equals to the approximation error at current iteration of critic update .

Original: This can be expressed by the following equation:
Revised: This can be expressed by the following equation:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: In addition, because TD method is use to estimate Q φ n , the real labeling function in target domain is actually the following one:
Revised: In addition, because TD method is use to estimate Q φ n , the real labeling function in target domain is actually the following one:

Original: [Equation]
Revised: [Equation]

Original: (13) where Pr ( ·| s , a ) denotes the transition probability distribution, and π θ n denotes the policy updated in actor module by using Q φ n .
Revised: (13) where Pr ( ·| s , a ) denotes the transition probability distribution, and π θ n denotes the policy updated in actor module by using Q φ n .

Original: Hence, the target error here means the difference between the
Revised: Hence, the target error here means the difference between the

Original: Q-network, Q φ ( s , a ) , and its target (labeling function) at ( n + 1) -th iteration, y π θn , on a target distribution.
Revised: Q-network, Q φ ( s , a ) , and its target (labeling function) at ( n + 1) -th iteration, y π θn , on a target distribution.

Original: The actual target error is given by
Revised: The actual target error is given by

Original: [Equation]
Revised: [Equation]

Original: Furthermore, two types of error are used to derive upper bound of appximation error, including source error e S ( Q φ ) and error e S ( Q φ , y π θn ) .
Revised: Furthermore, two types of error are used to derive upper bound of appximation error, including source error e S ( Q φ ) and error e S ( Q φ , y π θn ) .

Original: Source error error here is the difference between the
Revised: Source error error here is the difference between the

Original: Q-network, Q φ ( s , a ) , and its target (labeling function) at the n -th iteration, y π θn − 1 , on a source distribution.
Revised: Q-network, Q φ ( s , a ) , and its target (labeling function) at the n -th iteration, y π θn − 1 , on a source distribution.

Original: Hence, the approximation error is given by
Revised: Hence, the approximation error is given by

Original: [Equation]
Revised: [Equation]

Original: Error e S ( Q φ , y π θn ) is the difference between the Q-network, Q ( s , a )
Revised: Error e S ( Q φ , y π θn ) is the difference between the Q-network, Q ( s , a )

Original: [Equation]
Revised: [Equation]

Original: and its target at ( n + 1) -th iteration, y π θn , on a source distribution, which is given by
Revised: and its target at ( n + 1) -th iteration, y π θn , on a source distribution, which is given by

Original: [Equation]
Revised: [Equation]

Original: Target error can be derived into the following inequality:
Revised: Target error can be derived into the following inequality:

Original: [Equation]
Revised: [Equation]

Original: where the third term in the third line, | e T ( Q φ ) − e S ( Q φ , y π θn ) |
Revised: where the third term in the third line, | e T ( Q φ ) − e S ( Q φ , y π θn ) |

Original: [Equation]
Revised: [Equation]

Original: is transformed further as:
Revised: is transformed further as:

Original: [Equation]
Revised: [Equation]

Original: Recall that D is actually the replay buffer; and P D denotes the distribution that describes the likelihoods of samples drawn from replay buffer D uniformly.
Revised: Recall that D is actually the replay buffer; and P D denotes the distribution that describes the likelihoods of samples drawn from replay buffer D uniformly.

Original: Because, in experience replay mechanism (Lin, 1992), the number of samples in D n +1 is only a little more than in D n , the difference between
Revised: Because, in experience replay mechanism (Lin, 1992), the number of samples in D n +1 is only a little more than in D n , the difference between

Original: P D n and P D n +1 can be ignored.
Revised: P D n and P D n +1 can be ignored.

Original: Finally, the upper bound of error in Q-network is determined by
Revised: Finally, the upper bound of error in Q-network is determined by

Original: [Equation]
Revised: [Equation]

Original: It is noticeable that the third term in the upper bound will be smaller if the more similar the two consecutive policies, π θ n − 1 and π θ n are.
Revised: It is noticeable that the third term in the upper bound will be smaller if the more similar the two consecutive policies, π θ n − 1 and π θ n are.

Original: Hence, we can conclude that conﬁning the KL divergence between every two consecutive policies can help limit the approximation error during the optimization process of actor-critic.
Revised: Hence, we can conclude that conﬁning the KL divergence between every two consecutive policies can help limit the approximation error during the optimization process of actor-critic.

Original: This conclusion is used to design the learning method of the policy.
Revised: This conclusion is used to design the learning method of the policy.

Original: 3. A CTOR MODULE – LEARNING A POLICY
Revised: 3. A CTOR MODULE – LEARNING A POLICY

Original: The learning task of the actor module is to learn a parameterized stochastic policy, π θ ( a | s ) .In
Revised: The learning task of the actor module is to learn a parameterized stochastic policy, π θ ( a | s ) .In

Original: order to lower the upper bound of approximation error of Q function (or to reduce the optimization difﬁculty of the Q functions), the goal of the policy is converted from maximizing expect discounted return two parts: maximizing the estimated Q values—the minimum of the two Q approximators— and, concurrently, minimizing the KL divergence between two successive policies.
Revised: order to lower the upper bound of approximation error of Q function (or to reduce the optimization difﬁculty of the Q functions), the goal of the policy is converted from maximizing expect discounted return two parts: maximizing the estimated Q values—the minimum of the two Q approximators— and, concurrently, minimizing the KL divergence between two successive policies.

Original: The optimization objective is speciﬁed by
Revised: The optimization objective is speciﬁed by

Original: [Equation]
Revised: [Equation]

Original: where D is the replay buffer; P D denotes the distribution that describes the likelihoods of samples drawn from D uniformly; θ is the parameters of the policy network; θ old is the parameters of the policy updated in the last iteration; (cid:101) a θ ( s ) is the samples drawn from the target stochastic policy π θ ( a | s ) .
Revised: where D is the replay buffer; P D denotes the distribution that describes the likelihoods of samples drawn from D uniformly; θ is the parameters of the policy network; θ old is the parameters of the policy updated in the last iteration; (cid:101) a θ ( s ) is the samples drawn from the target stochastic policy π θ ( a | s ) .

Original: Note that, in order to back-propagate the error through this sampling operation, we use a Diagonal Gaussian policy and the reparameterization trick, i.e. samples are obtained according to
Revised: Note that, in order to back-propagate the error through this sampling operation, we use a Diagonal Gaussian policy and the reparameterization trick, i.e. samples are obtained according to

Original: [Equation]
Revised: [Equation]

Original: where µ θ ( s ) and σ θ ( s ) are the output of the policy network, and denotes the mean and covariance matrix’s diagonal elements, respectively.
Revised: where µ θ ( s ) and σ θ ( s ) are the output of the policy network, and denotes the mean and covariance matrix’s diagonal elements, respectively.

Original: The KL divergence between two distributions, for example p ( x ) and q ( x ) , can be thought of the difference between the cross entropy and entropy, which is speciﬁed by
Revised: The KL divergence between two distributions, for example p ( x ) and q ( x ) , can be thought of the difference between the cross entropy and entropy, which is speciﬁed by

Original: [Equation]
Revised: [Equation]

Original: (22) where H ( p, q ) denotes the cross entropy between p ( x ) and q ( x ) , and H ( p ) denotes the entropy of p ( x ) .
Revised: (22) where H ( p, q ) denotes the cross entropy between p ( x ) and q ( x ) , and H ( p ) denotes the entropy of p ( x ) .

Original: In practice, we ﬁnd it is more effective to minimize the cross entropy between two successive policies and to maximize the entropy of current policy, separately, than to minimize the KL divergence directly.
Revised: In practice, we ﬁnd it is more effective to minimize the cross entropy between two successive policies and to maximize the entropy of current policy, separately, than to minimize the KL divergence directly.

Original: Hence, we expand the original objective Eq.
Revised: Hence, we expand the original objective Eq.

Original: 20 into the following one:
Revised: 20 into the following one:

Original: [Equation]
Revised: [Equation]

Original: where α and β denote the coefﬁcients of the cross entropy and the entropy, respectively.
Revised: where α and β denote the coefﬁcients of the cross entropy and the entropy, respectively.

Original: Moreover, we adopt a mechanism to automatically adjust α and β .
Revised: Moreover, we adopt a mechanism to automatically adjust α and β .

Original: To do this, α is adjusted by keeping the current cross entropy close to a target value.
Revised: To do this, α is adjusted by keeping the current cross entropy close to a target value.

Original: This mechanism is speciﬁed by the following optimization objective:
Revised: This mechanism is speciﬁed by the following optimization objective:

Original: [Equation]
Revised: [Equation]

Original: where δ KL denotes the target KL value and δ entropy denotes the target entropy.
Revised: where δ KL denotes the target KL value and δ entropy denotes the target entropy.

Original: Note that log( · ) is use to ensure that α is greater than 0 .
Revised: Note that log( · ) is use to ensure that α is greater than 0 .

Original: β is adjusted in the same way:
Revised: β is adjusted in the same way:

Original: [Equation]
Revised: [Equation]

Original: The overall training process is summarized in Appendix A, and code can be found on our GitHub https://github.com/SingerGao/ECAC .
Revised: The overall training process is summarized in Appendix A, and code can be found on our GitHub https://github.com/SingerGao/ECAC .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The experiments aim to evaluate the effectiveness of the proposed strategy to lower the approximation error and to verify that the proposed method can outperform other mainstream RL algorithms.
Revised: The experiments aim to evaluate the effectiveness of the proposed strategy to lower the approximation error and to verify that the proposed method can outperform other mainstream RL algorithms.

Original: The experiments for ablation study and comparative evaluation are conducted on a few challenging continuous control tasks from the OpenAI Gym(Brockman et al., 2016) environments, which includes Mujoco(Todorov et al., 2012) and Pybullet (Coumans & Bai, 2016–2019) versions.
Revised: The experiments for ablation study and comparative evaluation are conducted on a few challenging continuous control tasks from the OpenAI Gym(Brockman et al., 2016) environments, which includes Mujoco(Todorov et al., 2012) and Pybullet (Coumans & Bai, 2016–2019) versions.

Original: Implementation details and hyperparameters of ECAC are presented in Appendix B.
Revised: Implementation details and hyperparameters of ECAC are presented in Appendix B.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Ablation studies are performed to verify the contribution of the operation of KL limitation.
Revised: Ablation studies are performed to verify the contribution of the operation of KL limitation.

Original: We compared the performance of ECAC with the method removing KL limitation from ECAC.
Revised: We compared the performance of ECAC with the method removing KL limitation from ECAC.

Original: [Figure]
Revised: [Figure]

Original: Figure 2 compares ﬁve different instances for both methods with and without KL limition using different random seeds; and each instance performs ﬁve evaluation episodes every 1 , 000 environment steps.
Revised: Figure 2 compares ﬁve different instances for both methods with and without KL limition using different random seeds; and each instance performs ﬁve evaluation episodes every 1 , 000 environment steps.

Original: The solid curves corresponds to the mean and the shaded region to the minimum and maximum returns over the ﬁve runs.
Revised: The solid curves corresponds to the mean and the shaded region to the minimum and maximum returns over the ﬁve runs.

Original: The experimental result shows that our method with KL limitation performs better than the one without KL limitation.
Revised: The experimental result shows that our method with KL limitation performs better than the one without KL limitation.

Original: Figure 3 demonstrates that by using ECAC the
Revised: Figure 3 demonstrates that by using ECAC the

Original: KL divergence remains comfortably low during all the training process.
Revised: KL divergence remains comfortably low during all the training process.

Original: [Figure]
Revised: [Figure]

Original: Furthermore, to verify that conﬁning the KL divergence can decrease the approximation error in
Revised: Furthermore, to verify that conﬁning the KL divergence can decrease the approximation error in

Original: Q function, we measured the normalized approximation error in 100 random states every 10 ,environment steps.
Revised: Q function, we measured the normalized approximation error in 100 random states every 10 ,environment steps.

Original: The normalized approximation error is speciﬁed by
Revised: The normalized approximation error is speciﬁed by

Original: [Equation]
Revised: [Equation]

Original: where Q approx is the approximate Q value given by the current Q network, and Q ture is the true discounted return.
Revised: where Q approx is the approximate Q value given by the current Q network, and Q ture is the true discounted return.

Original: The true value is estimated using the average discounted return over 100 episodes following the current policy, starting from states sampled from the replay buffer.
Revised: The true value is estimated using the average discounted return over 100 episodes following the current policy, starting from states sampled from the replay buffer.

Original: Figure 4 shows that the method with KL limitation has lower error in the Q function.
Revised: Figure 4 shows that the method with KL limitation has lower error in the Q function.

Original: [Figure]
Revised: [Figure]

Original: The results of all the ablation studies indicates that the approximation error of Q functioncan be decreased and the performance of the RL algorithm can be improved by placing restrictions on KL divergence between every two consecutive policies.
Revised: The results of all the ablation studies indicates that the approximation error of Q functioncan be decreased and the performance of the RL algorithm can be improved by placing restrictions on KL divergence between every two consecutive policies.

Original: 4. C OMPARATIVE E VALUATION
Revised: 4. C OMPARATIVE E VALUATION

Original: Comparative evaluation are conducted to verify that our method can outperform other traditional RL methods including A2C, PPO, TD3, and SAC.
Revised: Comparative evaluation are conducted to verify that our method can outperform other traditional RL methods including A2C, PPO, TD3, and SAC.

Original: Five individual runs of each algorithm with different random seeds are done; and each run performs ﬁve evaluation episodes every 1 , 000 environment steps.
Revised: Five individual runs of each algorithm with different random seeds are done; and each run performs ﬁve evaluation episodes every 1 , 000 environment steps.

Original: Our results are reported ﬁve random seeds (one random seed for one individual run) of the
Revised: Our results are reported ﬁve random seeds (one random seed for one individual run) of the

Original: Gym simulator, the network initialization, and sampling actions from policy during the training.
Revised: Gym simulator, the network initialization, and sampling actions from policy during the training.

Original: The results of max average return over ﬁve runs on all the 10 tasks are presented in Table 1.
Revised: The results of max average return over ﬁve runs on all the 10 tasks are presented in Table 1.

Original: ECAC outperforms all other algorithms on the tasks except Hopper-v3 and HumanoidBulletEnv-v0, and are only next to TD3 on  and HumanoidBulletEnv-v0.
Revised: ECAC outperforms all other algorithms on the tasks except Hopper-v3 and HumanoidBulletEnv-v0 are only next to TD3 on Hopper-v3 and HumanoidBulletEnv-v0.

Original: Figure 5 and Figure 6 demonstrates learning curves of comparative evaluation on the 10 continuous control  (Mujoco and PyBullet version, respectively).
Revised: Figure 5 and Figure 6 demonstrates learning curves of comparative evaluation on the 10 continuous control tasks (Mujoco and PyBullet version, respectively).

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presented a model-free actor-critic method based on a ﬁnding that the approximation error in value function of RL methods can be decreased by placing restrictions on KL-divergence between every two consecutive policies.
Revised: This paper presented a model-free actor-critic method based on a ﬁnding that the approximation error in value function of RL methods can be decreased by placing restrictions on KL-divergence between every two consecutive policies.

Original: Our method increases the similarity between every two consecutive polices in the training process and therefore reduces the optimization difﬁculty of value function.
Revised: Our method increases the similarity between every two consecutive polices in the training process and therefore reduces the optimization difﬁculty of value function.

Original: In the ablation studies, we compare the approximation error in Q function, KL divergence, and performance of the methods with and without KL limitation.
Revised: In the ablation studies, we compare the approximation error in Q function, KL divergence, and performance of the methods with and without KL limitation.

Original: The results of ablation study show that the proposed method can decrease the approximation error and improved the performance.
Revised: The results of ablation study show that the proposed method can decrease the approximation error and improved the performance.

Original: Moreover, the results of comparative evaluation demonstrate that ECAC outperforms other modelfree deep RL algorithm including A2C, PPO, TD3, and SAC.
Revised: Moreover, the results of comparative evaluation demonstrate that ECAC outperforms other modelfree deep RL algorithm including A2C, PPO, TD3, and SAC.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: APPENDICES A P SEUDO CODE OF ECAC
Revised: APPENDICES A P SEUDO CODE OF ECAC

Original: Require: initial policy patameters, θ ; Q function parameters, φ 1 and φ 2 ; discount rate, γ ; the coefﬁcient of KL term, β ; empty replay buffer D ; the number of episodes, M ; the maximum number of steps in each episode, T ; minibatch size, N .
Revised: Require: initial policy patameters, θ ; Q function parameters, φ 1 and φ 2 ; discount rate, γ ; the coefﬁcient of KL term, β ; empty replay buffer D ; the number of episodes, M ; the maximum number of steps in each episode, T ; minibatch size, N .

Original: Ensure: optimal policy parameters θ ∗ .
Revised: Ensure: optimal policy parameters θ ∗ .

Original: 1: for episode = 1 , M do 2:
Revised: 1: for episode = 1 , M do 2:

Original: Reset environment.
Revised: Reset environment.

Original: 3: for t = 1 , T do 4:
Revised: 3: for t = 1 , T do 4:

Original: Obeserve state s and select action a ∼ π ( ·| s ) .
Revised: Obeserve state s and select action a ∼ π ( ·| s ) .

Original: Execute a in the enviroment.
Revised: Execute a in the enviroment.

Original: Observe next state s (cid:48) and reward r .
Revised: Observe next state s (cid:48) and reward r .

Original: Store transition ( s , a , r, s (cid:48) ) in replay buffer D .
Revised: Store transition ( s , a , r, s (cid:48) ) in replay buffer D .

Original: Randomly sample a minibatch of N transitions, B = { ( s , a , r, s (cid:48) ) } from D .
Revised: Randomly sample a minibatch of N transitions, B = { ( s , a , r, s (cid:48) ) } from D .

Original: Compute targets for the Q functions:
Revised: Compute targets for the Q functions:

Original: [Equation]
Revised: [Equation]

Original: Update Q functions by one step of gradient descent using
Revised: Update Q functions by one step of gradient descent using

Original: [Equation]
Revised: [Equation]

Original: Backup old policy, θ old ← θ .
Revised: Backup old policy, θ old ← θ .

Original: Update α and β by one step of gradient ascent using
Revised: Update α and β by one step of gradient ascent using

Original: [Equation]
Revised: [Equation]

Original: θ ( ·| s ))
Revised: θ ( ·| s ))

Original: − δ entropy )] , where δ KL and δ entropy denote target KL divergence and target entropy, respectively.
Revised: − δ entropy )] , where δ KL and δ entropy denote target KL divergence and target entropy, respectively.

Original: Update policy by one step of gradient ascent using
Revised: Update policy by one step of gradient ascent using

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) a θ ( s ) is a sample from π θ ( ·| s ) , which is differentiable with respect to θ via the reparameterization trick.
Revised: where (cid:101) a θ ( s ) is a sample from π θ ( ·| s ) , which is differentiable with respect to θ via the reparameterization trick.

Original: 14: end for 15: end for
Revised: 14: end for 15: end for

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B I MPLEMENTATION DETAILS AND HYPERPARAMETERS OF ECAC
Revised: B I MPLEMENTATION DETAILS AND HYPERPARAMETERS OF ECAC

Original: For the implementation of ECAC
Revised: For the implementation of ECAC

Original: [Table]
Revised: [Table]

Original: a two layer feedforward neural network of 256 hidden units, with rectiﬁed linear units (ReLU) between each layer are used to build the two Q functions an the policy.
Revised: a two layer feedforward neural network of 256 hidden units, with rectiﬁed linear units (ReLU) between each layer are used to build the two Q functions an the policy.

Original: The parameters of the neural networks and the two coefﬁcients (i
Revised: The parameters of the neural networks and the two coefﬁcients (i

Original: [Table]
Revised: [Table]

Original: e α and β ) are optimized by using
Revised: e α and β ) are optimized by using

Original: Adam(Kingma & Ba, 2014).
Revised: Adam(Kingma & Ba, 2014).

Original: The hyperparameters of ECAC are listed in Table 2.
Revised: The hyperparameters of ECAC are listed in Table 2.

Original: Moreover
Revised: Moreover

Original: [Table]
Revised: [Table]

Original: we adopt the target network technique in ECAC, which is common in previous works (Lillicrap et al., 2016; Fujimoto et al., 2018).
Revised: we adopt the target network technique in ECAC, which is common in previous works (Lillicrap et al., 2016; Fujimoto et al., 2018).

Original: We also adopt reward scale trick which is presented in Haarnoja et al.
Revised: We also adopt reward scale trick which is presented in Haarnoja et al.

Original: (2018); and the reward scale parameter is listed in Table
Revised: (2018); and the reward scale parameter is listed in Table

Original: [Table]
Revised: [Table]