I NTERACTIVE P ARALLEL E XPLORATION FOR R EINFORCEMENT L EARNING IN C ONTINUOUS A CTION
S PACES
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, a new interactive parallel learning scheme is proposed to enhance the performance of off-policy continuous-action reinforcement learning.
In the proposed interactive parallel learning scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information.
The information of the best policy is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search space by the multiple learners.
The guidance by the previous best policy and the enlarged search space by the proposed interactive parallel learning scheme enable faster and better policy search in the policy parameter space.
Working algorithms are constructed by applying the proposed interactive parallel learning scheme to several off-policy reinforcement learning algorithms such as the twin delayed deep deterministic (TD3) policy gradient algorithm and the soft actor-critic (SAC) algorithm, and numerical results show that the constructed IPE-enhanced algorithms outperform most of the current state-of-the-art reinforcement learning algorithms for continuous action control.
I NTRODUCTION
Reinforcement learning (RL) for continuous action control is an active research ﬁeld.
In RL, an agent learns a policy through interaction with the environment to maximize the cumulative reward.
One of the key issues in RL is the trade-off between exploitation and exploration.
Exploitation is to make a best decision based on the already collected information, whereas exploration is to collect more new information about the environment.
The balance between the two is important for good RL algorithms.
For example, DQN (Mnih et al.
(2015)) balances exploitation and exploration by taking actions based on the ǫ -greedy approach.
Deep deterministic policy gradient (DDPG) (Lillicrap et al.
and twin delayed deep deterministic (TD3) (Fujimoto et al.
policy gradient algorithms promote exploration by adding Ornstein-Uhlenbeck noise and Gaussian noise to the best decision action, respectively.
Soft actor-critic (SAC) (Haarnoja et al.
(2018)) performs balancing by using a maximum entropy objective.
However, most of the previous works focus on exploration to obtain unobserved states or actions.
In this paper, we consider exploration in the policy parameter space by using parallel identical learners for the same environment.
By having multiple identical learners for the same environment, we can have increased search capability for a better policy.
Parallelism in learning has been investigated widely in distributed RL (Nair et al.
(2015), Mnih et al.
, Horgan et al.
(2018), Barth-Maron et al.
(2018), Espeholt et al.
evolutional strategies (Salimans et al.
(2017), Choromanski et al.
(2018)), and recently in population based training (PBT) (Jaderberg et al. (2017), Jaderberg et al. (2018), Conti et al. (2017)) for faster and better search for parameters and/or hyperparameters.
In this paper, we also apply parallelism to RL in order to enhance the learning performance but in a slightly different way as compared to the previous methods.
The proposed algorithm is intended for any off-policy RL algorithms and is composed of a chief,
N environment copies of the same environment, and N identical learners with a shared common experience replay buffer and a common base algorithm, as shown in Fig.
Each learner has its own value function(s) and policy, and trains its own policy by interacting with its own environment copy with some additional interaction with the chief, as shown in Fig.
At each time step, each learner takes an action to its environment copy by using its own policy, stores its experience to the shared common experience replay buffer.
Then, each learner updates its value function parameter and policy parameter by drawing mini-batches from the shared common replay buffer by minimizing its own value loss function and policy loss function, respectively.
One way to implement parallel learning under the above setup is to run N fully independent parallel learning without interaction among the learners except sharing their experiences until the end of time steps and to choose the policy from the learner with the maximum accumulated reward at the end for future use. We will refer to this method as the experience-sharing-only method.
However, this method ignores possible beneﬁt from mutual interaction among the learners during training.
In order to harness the beneﬁt of mutual interaction among the learners in parallel learning, we exploit the information from the best learner among all learners periodically during training like in PBT (Jaderberg et al. (2017), Jaderberg et al.
Suppose that the value and policy parameters of each learner are initialized and learning is performed as described above for M time steps.
At the end of M time steps, we can determine who is the best learner based on the average of the most recent E r episodic rewards for each learner.
Then, the policy parameter information of the best learner can be used to enhance the learning of other learners for the next M time steps.
This information can help learners stuck in local minima escape from the local minima and guide other learners for better direction.
One simple way to exploit this best policy parameter information is that we reset the policy parameter of each learner with the policy parameter of the best learner at the beginning of the next M time steps, make each learner perform learning from this initial point in the policy parameter space for the next M time steps, select the best learner again at the end of the next M time steps, and repeat this procedure every M time steps in a similar way that PBT (Jaderberg et al. (2017)) copies the best learner’s parameters and hyperparameters to other learners. We will refer to this method as the reloading method in this paper.
However, this reloading method has the problem that the search area covered by all N searching policies collapses to one point at the time of parameter copying and thus the search area can be narrow around the previous best policy point.
In order to overcome such disadvantage, instead of resetting the policy parameter with the best policy parameter every M time steps, we here propose using the policy parameter information of the best learner in a soft manner to enhance the performance of the overall parallel learning.
In the proposed scheme, the shared best policy information is used only to guide the policies of other learners.
The policy of each learner is updated by improving the performance around a certain distance from the shared guiding policy.
The chief periodically determines the best policy among the policies of all learners and distributes the best policy parameter to all learners so that the learners search for better policies around the previous best policy.
The chief also enforces that the N searching policies are spread in the policy parameter space with a given distance from the previous best policy point so that the search area in the policy space by all N learners maintains a wide area and does not collapse into a narrow region.
The proposed interactive parallel exploration (IPE) learning method can be applied to any off-policy
RL algorithms and implementation is easy.
Furthermore, the proposed method can be extended directly to distributed or multi-agent RL systems.
We apply our IPE scheme to the TD3 algorithm and the SAC algorithm, which are state-of-the-art off-policy algorithms, as our base algorithms, and the new algorithms are named IPE-TD3 and IPE-SAC algorithms, respectively. Numerical result shows that the IPE-enhanced algorithms outperform the baseline algorithms both in the speed of convergence and in the ﬁnal steady-state performance. The gain by IPE
Distributed RL is an efﬁcient way that takes advantage of parallelism to achieve fast training for large complex tasks (Nair et al. Most of the works in distributed RL assume a common structure composed of multiple actors interacting with multiple copies of the same environment and a central system which stores and optimizes the common Q-function parameter or the policy parameter shared by all actors. The focus of distributed RL is to optimize the Q-function parameter or the policy parameter fast by generating more samples for the same wall clock time with multiple actors.
In order to achieve this goal, researchers investigated various techniques for distributed RL, e.g., [Table] [Figure] asynchronous update of parameters (Mnih et al.
(2016), Babaeizadeh et al.
(2017)), sharing an experience replay buffer (Horgan et al.
(2018)), GPU-based parallel computation (Babaeizadeh et al.
(2017), Clemente et al.
, GPU-based simulation (Liang et al.
and V-trace in case of on-policy algorithms (Espeholt et al.
Distributed RL yields signiﬁcant performance improvement in terms of the wall clock time but it does not consider the possible enhancement by interaction among multiple learners like in IPE and PBT.
The proposed IPE uses a similar structure to that in (Nair et al.
, Espeholt et al.
(2018)): that is, IPE is composed of multiple learners and a central system called chief.
The difference is that each learner has its own Q or value function parameter and policy parameter and optimizes the parameters in parallel with interactions.
2. P OPULATION B ASED T RAINING
Parallelism is also exploited in ﬁnding optimal parameters and hyperparameters of training algorithms for neural networks in PBT (Jaderberg et al.
(2017), Jaderberg et al.
(2018), Conti et al.
PBT (Jaderberg et al.
ﬁrst chooses multiple sets of hyperparameters and parameters for a common base algorithm, and runs the base algorithm separatively in parallel at multiple learners to train their neural networks using those parameters and hyperparameters.
Each learner updates the neural network parameters by perturbing the assigned hyperparameters.
During training, in principle PBT evaluates the performance of multiple learners periodically, and selects the best performing hyperparameters, and then distributes the best performing hyperparameters and the corresponding parameters to other learners, although implementation details can be changed.
Recently, PBT is applied to competitive multi-agent RL (Jaderberg et al.
and novelty search algorithms (Conti et al.
Although PBT is mainly developed to tune hyperparamters, the philosophy of PBT can be applied to ﬁnd optimal parameters for given hyperparameters by multiple learners.
In this case, multiple learners update their parameters in parallel, their performance is measured periodically, the parameters of the best performing learner are copied to other learners, other learners independently update their parameters from the copied parameters as their new initialization, and this process is repeated.
The proposed IPE is similar to PBT in the sense that it exploits the parameters of the best performing learner among multiple parallel learners.
However, IPE is different from the PBT-derived method in the way how IPE uses the parameters of the best learner.
In the PBT-derived method, the parameters of the best learner are copied to other learners and other learners’ parameters are reset to the parameters of the best performing learner.
Then, the parameters of each learner are updated by stochastic gradient descent (SGD).
However, in IPE the parameters of the best performing learner are not copied but used in a soft manner as a guiding direction.
Copying means that the parameters of all learners collapse to a single point in the parameter space.
Furthermore, unlike PBT, IPE uses a common experience replay buffer to store all experiences from multiple learners with different parameters to exploit the diverse experiences of multiple learners with different parameters.
As mentioned in Section 1, we refer to the PBT-derived method with a common experience replay buffer as the reloading method of which performance will be given in ablation study in Section 4.
Although IPE is considered only for parallel parameter search in this paper, combining the soft way of using the parameters of the best performing learner with hyperparameter search is an interesting future work.
2. G UIDED P OLICY S EARCH
Our IPE method is also related to guided policy search (Levine & Koltun (2013), Levine et al.
(2016), Teh et al.
(2017), Ghosh et al.
Recently, Teh et al.
(2017) proposed a guided policy search method for joint training of multiple tasks in which a common policy is used to guide local policies and the common policy is distilled from the local policies.
Here, the local policies’ parameters are updated to maximize the performance and minimize the KL divergence between the local policies and the common distilled policy.
The proposed IPE is related to guided policy search in the sense that multiple policies are guided by a common policy.
However, the difference is that the goal of IPE is not learning multiple tasks but learning optimal parameters for a common task as in
PBT and hence the guiding policy is not distilled from multiple local policies but chosen as the best performing policy among multiple learners.
2. E XPLORATION
Improving exploration has been one of the key issues in RL and many different ways have been developed to improve exploration through maximum entropy objectives (Haarnoja et al.
(2017; 2018)), noise in networks (Fortunato et al.
; Plappert et al.
(2018)), and intrinsically motivated approaches (Bellemare et al.
; Ostrovski et al.
; Pathak et al.
; Achiam & Sastry (2017); Zheng et al.
The proposed IPE also enhances exploration.
Speciﬁcally, IPE uses exploitation for exploration.
Exploitation for exploration has been considered in the previous works (White & Sofge (1992), Oh et al. In particular, Oh et al. (2018) exploited past good experiences to explore the sample space, whereas IPE exploit the current good policy among multiple policies to explore the policy space.
The considered parallel learning setting consists of the environment E and N policies { π 1 , · · · , π N } . The environment E is described as a Markov decision process (MDP) deﬁned by the tuple hS , A , T , r i , where S is the state space,
A is the action space, T : S ×
A × S → [0 , 1] is the state transition probability, and r : S ×
A → R is the reward function.
There exist N copies {E 1 , · · · , E N } of the environment E , i.e., E 1 = · · · = E N = E , and the N environment copies may have different random initial seeds.
The policy π i interacts with its corresponding environment copy
E i and builds up its trajectory { ( s i t , a i t , r i t ) , t = 1 , 2 , · · · } for each i = 1 , · · · , N .
At time step t , the environment copy E i has a state s i t ∈ S .
The policy π i interacts with the environment copy E i by taking an action a it according to π i given the current state s it .
Then, the environment copy E i yields the reward r it =
r ( s it , a it ) and makes transition to the next state s it +
according to T .
In this paper, in order to account for the actual amount of interaction with the environment, we deﬁne environment steps as the total number of interactions by all N parallel policies with all N environment copies.
Suppose that all N policies generate their trajectories simultaneously in parallel, and suppose that M time steps have elapsed.
Then, although the number of elapsed time steps is M , the number of environment steps is NM .
I NTERACTIVE P ARALLEL P OLICY E XPLORATION
We now present the proposed IPE scheme with the parallel environment learning setting described in Section 2.5, and the overall structure is described in Fig.
We have N identical parallel learners with a shared common experience replay buffer D , and all N identical learners employ a common base algorithm, which can be any off-policy RL algorithm.
The execution is in parallel.
The i -th learner has its own environment E i , which is a copy of the common environment E , and has its own value function (e.g., Q-function) parameter θ i and policy parameter φ i .
The i -th learner interacts with the environment copy E i with some additional interaction with the chief, as shown in Fig.
At each time step, the i -th learner performs an action a i t to its environment copy E i by using its own policy π φ i , stores its experience ( s it , a it , r it , s i t +1 ) to the shared common experience replay buffer D for all
i = 1 , 2 , · · · , N .
Note that one time step corresponds to N environment steps.
Then, at each time step, each learner updates its value function parameter and policy parameter for N times by drawing N mini-batches of size B from the shared common replay buffer D by minimizing its own value loss function and policy loss function, respectively.
The N time updates for each learner for each time step is to exploit the samples provided by other N −learners stored in the shared common replay buffer.
In order to harness the beneﬁt of mutual interaction among the learners in parallel learning, we exploit the information from the best learner periodically during training like in PBT (Jaderberg et al.
Suppose that the Q-function parameter and the policy parameter of each learner are initialized and learning is performed as described above for M time steps.
At the end of the M time steps, we determine who is the best learner based on the average of the most recent E r episodic rewards for each learner.
Let the index of the best learner be b .
Then, the policy parameter information φ b of the best learner can be used to enhance the learning of other learners for the next M time steps.
Here, instead of copying φ b to other learners, we propose using the information of φ b in a soft manner to enhance the performance of the overall parallel learning.
That is, during the next M time steps, whereas we set the loss function
˜ L ( θ i ) for the Q-function to be the same as the loss L ( θ i ) of the base algorithm, we set the loss function ˜ L ( φ i ) for the policy parameter φ i of the i -th learner as the following augmented version:
[Equation]
where L ( φ i ) is the policy loss function of the base algorithm, 1 {·} denotes the indicator function, β ( > 0) is a weighting factor, D ( π, π ′ ) be some distance measure between two policies π and π ′ , and ˆ E s ∼D denotes the sample expectation based on mini-batch drawn randomly from the experience replay buffer D .
The augmented loss function ˜ L ( φ i ) in (1) is composed of two terms L ( φ i ) and 1 { i 6 = [Equation] } β ˆ E s ∼D
[Equation]
Thus, for the non-best learners in the previous M time steps, the gradient of ˜ L ( φ i ) is the mixture of two directions: one is to maximize the return by itself and the other is to follow the previously best learner’s policy.
The second term in the right-hand side (RHS) of (1) guides non-best learners towards a good direction in addition to each leaner’s self search.
3. D ESIGN OF THE W EIGHTING F ACTOR β
In (1), the weighting factor β is common to all N learners and should be determined judiciously to balance between improving its performance by each learner itself and going towards the previous best policy among the N learners.
We adopt an adaptive method to determine the value of β as follows:
[Equation] [Equation]
where ˆ D best is the estimated distance between π φ i and π φ b averaged over all N − 1 non-best learners, and ˆ D bchange is the estimated distance between π φ bupdated (i.e., the policy of the current best learner at the end of the current M time steps) and π φ b (i.e, the policy of the current best learner at the end of the previous M time steps), given respectively by [Equation] and ˆ D bchange =
ˆ [Equation]
D
[Equation]
π φ bupdated , π φ b
[Equation] i . Here, I − b = { 1 , . . .
, N } \ { b } , and d search and ρ are predetermined hyperparameters.
This adaptation method is similar to that used in proximal policy optimization (PPO)
(Schulman et al.
The update of β is done every M time steps and the updated β is used for the next M time steps.
First, suppose that we do not have the ﬁrst term ρ ˆ D bchange in the maximum of the condition in (2).
Then, when the estimated average distance ˆ D best from the best policy to the remaining policies is smaller than d search / 1 . 5 , the parameter β is decreased by half.
Hence, the movement in the gradient direction of the second term in the RHS of (1) is diminished and the independent movement into the optimization direction for L ( φ i ) becomes more important.
So, each policy gradually diverges from the previous best policy serving as the reference point due to internal exploration mechanism such as added noise in action.
On the other hand, when ˆ D best is larger than 1 . 5 d search , the parameter β increases by factor two and the movement towards the previous best policy becomes more important.
As time steps elapse, β is settled down so that ˆ D best is around d search .
Hence, the proposed IPE scheme searches a wide area with rough radius d search around the best policy in the policy parameter space, as shown in Fig. 2(a).
Furthermore, with the ﬁrst term ρ ˆ D bchange in the maximum of the condition in (2), we can control the speed of tracking the best policy. ˆ D bchange measures the speed of change in the best policy parameter.
When the best policy parameter change scaled by ρ , i.e., ρ ˆ D b change is less than d search , the term is invisible by the maximum operation in (2). On the other hand, when ρ ˆ D bchange > d search , the term is active and it means that the best policy parameter changes fast. Thus, the tracking speed should be controlled. If ˆ D best > ρ ˆ D b change , i.e., the distance from π φ i to π φ b is larger than ρ times the distance from π φ b updated to π φ b , then this means that the speed of tracking the best policy is slow.
Hence, we increase β by factor two.
Otherwise, we decrease β by half.
When the search for the current M time steps is ﬁnished, the new best learner is selected and a new search for a wide area around the new best learner’s policy π φ b is performed, as illustrated in Fig. 2(b). The policy parameter information φ b of the best learner can be changed before the next best learner selection.
[Equation] [Figure]
Now, the overall procedure for the proposed IPE scheme is explained with the diagram in Fig.
The value function parameter and policy parameter of each learner are initialized.
The chief distributes the parameter β and the reference policy parameter φ b , which is the policy information of the best learner over the previous M time steps, to all learners.
At each time step, each learner interacts with its own environment copy by taking its action and receiving the reward and the next state, and stores its experience to the shared common replay buffer D .
Then, the i -th learner updates its value function parameter θ i by minimizing its own value loss function ˜
L ( θ i ) which is the same as that of the base algorithm, and updates the policy parameter φ i by minimizing the augmented loss function ˜ L ( φ i ) in (1) for N times by drawing N mini-batches from the shared common replay buffer D .
Whenever an episode ends for a learner, the learner reports the episodic reward to the chief.
The i -th learner reports ˆ E s ∼D (cid:2)
D ( π φ i , π φ b ) (cid:3) to the chief for computation of ˆ D best in (3).
At every M time steps, the chief updates β according to (2), determines the best learner over the most recent M time steps based on the collected episodic rewards from each learner. Once the best learner is determined, the chief obtains the policy parameter information φ b from the determined best learner, and distributes the new β and the reference policy parameter φ b to all N learners.
This procedure repeats until the time steps reaches the predeﬁned maximum.
When the parallel learning based IPE reaches a steady state, we can choose any of the N learners’ policies and use the chosen policy for the environment E in future since it is expected that at the steady-state the performance of all N policies is more or less similar due to their distance property.
3. IPE-E NHANCED A LGORITHMS
The proposed IPE method can be applied to any off-policy RL algorithms regardless of whether the base RL algorithms have continuous actions or discrete actions.
Here, we consider the application of IPE to the TD3 algorithm as the base algorithm and the constructed algorithm is named the IPETD3 algorithm.
The details of baseline TD3 are explained in Appendix A. With TD3 as the base algorithm, each learner has its own parameters θ i 1 , θ i 2 , and φ i for its two Q-functions and policy.
Furthermore, it has ( θ i 1 ) ′ , ( θ i 2 ) ′ , and ( φ i ) ′ which are the parameters of the corresponding target networks.
For the distance measure between two policies, we use the mean square difference given by
[Equation]
For the i -th learner, as in TD3, the parameters θ i j , j = 1 , 2 are updated every time step by minimizing
[Equation]
where y = r + γ min j =1 , 2 Q ( θ ij )
( s ′ , π ( φ i ) ′
( s ′ )
+ ǫ )
, ǫ ∼ clip ( N (0 , ˜ σ 2 ) , − c, c ) .
The parameter φ i is updated every d time steps by minimizing the following augmented loss function:
[Equation]
For the ﬁrst T initial timesteps for initial exploration we use a random policy and do not update all policies over the initial exploration period.
With these loss functions, the reference policy, and the initial exploration policy, all procedure is the same as the general IPE procedure described previously.
The pseudocode of the IPE-TD3 algorithm is provided in Appendix B. The application of IPE to other algorithms such as SAC and DQN is also provided in Appendices.
E XPERIMENTS
In this section, we provide the numerical results on the performance of the proposed IPE-TD3 and current state-of-the-art on-policy and off-policy baseline algorithms on several MuJoCo environments (Todorov et al. The baseline algorithms are Proximal Policy Optimization (PPO) (Schulman et al. Actor Critic using Kronecker-Factored Trust Region (ACKTR) (Wu et al.
(2017)), Soft Q-learning (SQL) (Haarnoja et al. Soft Actor-Critic (SAC) (Haarnoja et al. (2018)), and TD3 (Fujimoto et al. More numerical result on IPE applied to SAC is provided in Appendices.
4. P ARAMETER S ETTING
All hyperparameters we used for evaluation are the same as those in the original papers (Schulman et al.
(2017); Haarnoja et al.
; Fujimoto et al.
Here, we provide the hyperparameters of TD3 and IPE-TD3 only. TD3 The networks for two Q-functions and the policy have 2 hidden layers.
The ﬁrst and second layers have sizes 400 and 300, respectively.
The non-linearity function of the hidden layers is ReLU, and the activation functions of the last layers of the Q-functions and the policy are linear and hyperbolic tangent, respectively.
We used the Adam optimizer with learning rate 10 − 3 , discount factor γ = 0 .
99 , target smoothing factor τ = 5 × 10 − 3 , the period d = 2 for updating the policy.
The experience replay buffer size is 10 6 , and the mini-batch size B is 100 .
The standard deviation for exploration noise σ and target noise ˜ σ are 0 .
1 and 0 . 2 , respectively, and the noise clipping factor c is 0 .
IPE-TD
In addition to the parameters for TD3, we used N = 4 learners, the period M =of updating the best policy and β , the number of recent episodes E r = 10 for determining the best
[Figure]
learner b .
The parameters d search and ρ for the exploration range are 0.04 and 2, respectively.
The timesteps for initial exploration T initial is set as 250 for Hopper-v1 and Walker2d-v1 and asfor HalfCheetah-v1 and Ant-v1.
4. C OMPARISON TO BASELINES
In order to have sample-wise fair comparison among the considered algorithms, we obtain the performance with respect to environment steps (not time steps), which is deﬁned as the total number of interactions with the environment by the agent.
This comparison makes sense because the performance at the same environment steps means that all algorithms use the same number of samples obtained from the environment.
The performance is obtained through the evaluation method that is similar to those in (Haarnoja et al.
; Fujimoto et al.
Evaluation of the policies are conducted every R eval = 4000 environment steps for all algorithms.
At each evaluation instant, the agent (or learner) ﬁxes its policy as the one at the evaluation instant, and interacts with the same environment separate for the evaluation purpose with the ﬁxed policy to obtain 10 episodic rewards.
The average of these 10 episodic rewards is the performance at the evaluation instant.
In the case of
IPE-TD3 and other parallel learning schemes, each of the N learners ﬁxes its policy as the one at the evaluation instant, and interacts with the environment with the ﬁxed policy to obtain 10 episodic rewards.
First, the 10 episodic rewards are averaged for each learner and then the maximum of the 10-episode-average rewards of the N learners is taken as the performance at that evaluation instant.
We performed this operation for ﬁve different random seeds, and the mean and variance of the learning curve are obtained from these ﬁve simulations.
The policies used for evaluation are stochastic for PPO and deterministic for the others.
Fig.
3 shows the learning curves over one million environment steps for several MuJoCo tasks:
Hopper-v1, Walker2d-v1, HalfCheetah-v1, and Ant-v1.
First, it is observed that the performance of
TD3 here is similar to that in the original TD3 paper (Fujimoto et al.
(2018)), and the performance of other baseline algorithms is also similar to that in the original papers (Schulman et al.
Haarnoja et al.
It is seen that the IPE-TD3 algorithm outperforms the state-of-the-art RL algorithms in terms of both the speed of convergence with respect to environment steps and the ﬁnal steady-state performance (except in Walker2d-v1, the initial convergence is a bit slower than TD3.) Especially, in the cases of Hopper-v1 and Ant-v1, TD3 has large variance and this means that the performance of TD3 is quite dependent on the initial condition of the environment and it is not easy for TD3 to escape out of bad local minima in certain environments. However, it is seen that IPE-TD yields much less variance as compared to TD3.
This implies that the wide area search by IPE in the policy parameter space helps the learners escape out of bad local optima.
It is seen that the wide area search around the previous best policy point in the policy parameter space by IPE yields faster and better policy search.
[Figure]
A BLATION STUDY
IPE-TD3 has several components to improve the performance based on parallelism: 1) sharing experiences from multiple policies, 2) using the best policy information, and 3) fusing the best policy information in a soft manner based on the augmented loss function. Thus, we investigated the impact of each component on the performance improvement.
For comparison we considered the following parallel policy exploration methods gradually incorporating more techniques:
TD3 The original TD3 with one learner 2. Distributed RL TD3 (DRL-TD3) N actors obtain samples from N environment copies. The policy and the experience replay buffer are shared by all N actors. Experience-Sharing-Only TD3 (ESO-TD3) N learners interact with N environment copies and update their own policies using experiences drawn from the shared experience replay buffer.
Reloading TD3 (Re-TD3)
At every M ′ timesteps, the best policy is determined and all policies are initialized as the best policy,
i
e., the best learner’s policy parameter is copied to all other learners.
The rest of the procedure is the same as experience-sharing-only TD3. 5.
{+IPE-TD
+}
At every M timesteps, the best policy information is determined and this policy is used in a soft manner based on the augmented loss function.
Note that Re-TD3 exploits the best policy information from N learners.
The main difference between IPE-TD3 and Re-TD3 is the way how the best learner’s policy parameter is used.
Re-TD initializes all policies with the best policy parameter every M ′ timesteps like in PBT (Jaderberg et al.
(2017)), whereas IPE-TD3 uses the best learner’s policy parameter information determined every M timesteps to construct an augmented loss function.
For fair comparison, M and M ′ are determined
[Table]
independently and optimally for IPE-TD3 and Re-TD3, respectively, since the optimal period can be different for the two methods.
Thus, M ′ = 5000 is determined for Re-TD3 by tuning, whereas
M = 250 is used for IPE-TD3.
Since all N policies collapse as one point in Re-TD3 at the beginning of each period, we expect that a larger period is required for Re-TD3 to have sufﬁciently spread policies at the end of the best policy selection period. Fig. 4 shows the learning curves of the considered parallel exploration methods for the Ant-v task and Table 1 shows the ﬁnal (steady-state) performance of the considered parallel exploration methods for four MuJoCo tasks, respectively. It is seen that IPE-TD3 outperforms other parallel methods: DRL-TD3, ESO-TD3 and Re-TD3 except the case that ESO-TD3 outperforms all other parallel schemes in Hopper-v1.
Both Re-TD3 and IPE-TD3 have better ﬁnal (steady-state) performance than TD3 and ESO-TD3 for all tasks except Hopper-v1 for which ESO-TD3 performs best. Note that ESO-TD3 obtains most diverse experiences since the N learners shares the experience replay buffer but there is no interaction among the N learners until the end of training. So, it seems that this diverse experience is beneﬁcial to Hopper-v1. The ﬁnal performances of Re-TD3 and IPE-TD are more or less the same for HalfCheetah-v1 but the ﬁnal performance of IPE-TD3 is noticeably better than that of Re-TD3 in other cases.
In this paper, we have proposed a new interactive parallel learning scheme, IPE, to enhance the performance of off-policy RL systems. In the proposed IPE scheme, multiple identical learners with their own value-functions and policies sharing a common experience replay buffer search a good policy with the guidance of the best policy information in the previous search interval. The information of the best policy parameter of the previous search interval is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search space by the multiple learners.
The guidance by the previous best policy and the enlarged search space by IPE enables faster and better policy search in the policy parameter space. The IPE-enhanced algorithms constructed by applying the proposed IPE scheme to TD3 or SAC outperforms most of the current state-of-the-art continuous-action RL algorithms. Although we mainly considered continuous-action off-policy algorithms in this paper, the proposed IPE method can also be applied to RL with discrete actions, as seen in Appendix E. In the case of continuous action control, the gain by IPE can be beneﬁcial for recent trend of fast computer-based prototyping of complex robotics systems or autonomous cars, whereas in the discrete-action case better policy parameters can be searched for more challenging tasks by IPE.
A PPENDIX A. T HE T WIN D ELAYED D EEP D ETERMINISTIC P OLICY G RADIENT A LGORITHM AND T HE S OFT A CTOR -C RITIC A LGORITHM A.1 T HE TWIN DELAYED DEEP DETERMINISTIC POLICY GRADIENT (TD3) ALGORITHM
The TD3 algorithm is a current state-of-the-art off-policy algorithm and is a variant of the deep deterministic policy gradient (DDPG) algorithm (Lillicrap et al. The TD3 algorithm tries to resolve two problems in typical actor-critic algorithms: 1) overestimation bias and 2) high variance in the approximation of the Q-function. In order to reduce the bias, the TD3 considers two Q-functions and uses the minimum of the two Q-function values to compute the target value, while in order to reduce the variance in the gradient, the policy is updated less frequently than the Qfunctions.
Speciﬁcally, let Q θ 1 , Q θ 2 and π φ be two current Q-functions and the current deterministic policy, respectively, and let Q θ ′ 1 , Q θ ′ 2 and π φ ′ be the target networks of Q θ 1 , Q θ 2 and π φ , respectively.
The target networks are initialized by the same networks as the current networks.
At time step t , the TD3 algorithm takes an action a t with exploration noise ǫ : a t = π φ ( s t ) + ǫ , where ǫ is zero-mean Gaussian noise with variance σ 2 , i.e., ǫ ∼ N (0 , σ 2 ) .
Then, the environment returns reward r t and the state is switched to s t +1 .
The TD3 algorithm stores the experience ( s t , a t , r t , s t +1 ) at the experience replay buffer D .
After storing the experience, the Q-function parameters θ 1 and θare updated by gradient descent of the following loss functions:
[Equation]
where ˆ E ( s,a,r,s ′ )
∼D denotes the sample expectation with an uniform random mini-batch of size B drawn from the replay buffer D , and the target value y is given by
[Equation]
Here, for the computation of the target value, the minimum of the two target Q-functions is used to reduce the bias.
The procedure of action taking and gradient descent for θ 1 and θ 2 are repeated for d times ( d = 2 ), and then the policy and target networks are updated.
The policy parameter φ is updated by gradient descent by minimizing the loss function for φ :
[Equation]
and the target network parameters θ ′ j and φ ′ are updated as
[Equation]
The networks are trained until the number of time steps reaches a predeﬁned maximum.
A.2 T HE SOFT ACTOR - CRITIC (SAC) ALGORITHM
The SAC algorithm is an off-policy algorithm comparable to TD3 and yields good performance especially in environments with high dimensional action spaces.
The SAC algorithm is a maximum entropy RL which is based on the discounted sum of reward and the entropy of the current policy given by
[Equation]
(11) where α is a weighting factor that balances between the reward and the entropy of the policy.
This objective function stimulates the algorithm to explore more diverse experiences so as to ﬁnd a better policy.
The SAC algorithm has one value function V ψ ( s ) , two Q-functions Q θ j ( s, a ) , j = 1 , 2 , and one stochastic policy π φ ( ·| s ) , which are parameterized by parameters ψ , θ j , and φ , respectively.
It also has a target value function V ψ ′ ( s ) for stable convergence.
After initialization, at each time step t the algorithm obtains experience ( s t , a t , r t , s t +1 ) by interacting with the environment and stores the experience to the experience replay buffer D .
Then, it updates the parameters ψ , θ j , and φ by gradient descent of the following loss functions:
[Equation]
where ¯ Q ( s, a ) = min { Q θ 1 ( s, a ) , Q θ 2 ( s, a ) } , and ˆ E ( s,a,r,s ′ )
∼D is the sample expectation with an uniform random mini-batch of size B drawn from the replay buffer D .
After updating the parameters, the target value function parameter ψ ′ is updated as
[Equation]
In order to obtain diverse experience in the initial stage of learning, it uses a uniform policy for initial T initial time steps and the current policy π φ ( ·| s ) for the rest of learning.
A PPENDIX B. P SEUDOCODE OF THE IPE-TD3 A LGORITHM
Require: N : number of learners, T initial : initial exploration time steps, T : maximum time steps,
M : the best policy period
B : mini batch d : policy networks.
1: Initialize φ 1 = · · · = φ N = φ b , θ 1 j = · · · = θ Nj , j = 1 , 2 , randomly.
2: Initialize β
= 1 , t =3: while t < T do 4: t ← t + 1 (one time step) 5: for i = 1 , 2 , · · · , N in parallel do 6: if t < T initial then 7:
Take a uniform random action a i t to environment copy E i 8: else 9:
Take an action a it
=
π i (cid:0) s it (cid:1) + ǫ , ǫ ∼ N (0 , σ 2 ) to environment copy E i 10: end if 11:
Store experience ( s i t , a i t , r i t , s i t +1 ) to the shared common experience replay D 12: end for 13: if t < T initial then 14: continue (i.e., go to the beginning of the while loop) 15: end if 16: for i
= 1 , 2 , · · · , N in parallel do 17: for k = 1 , 2 , · · · , N do 18:
Sample a mini-batch B = { ( s t l , a t l , r t l , s t l +1 ) } l =1 ,...,B from D 19:
Update θ ij , j = 1 , 2 , by gradient descent for minimizing ˜ L ( θ ij ) in (5) with B 20: if k ≡ 0( mod d ) then 21:
Update φ i by gradient descent for minimizing ˜ L ( φ i ) in (6) with B 22:
Update the target networks: ( ij ) ij ) ′
τθ ij , ( φ i ) ′
← ( τ ) i ) ′
τφ i 23: end if 24: end for 25: end for 26: if t ≡ 0( mod M ) then 27:
Select the best learner b 28:
Adapt β with (2) 29: end if 30: end while
A PPENDIX C. P SEUDOCODE OF THE IPE-SAC A LGORITHM
Require: N : number of learners, T initial : initial exploration time steps, T : maximum time steps,
M : the best policy update period, B : size mini-batch 1: Initialize ψ 1 = · · · = ψ N , φ 1 = · · · = φ N = φ b , θ 1 j = · · · = θ N j , j = 1 , 2 , randomly.
2: Initialize β
= 1 , t =3: while t < T do 4: t ← t + 1 (one time step) 5: for i = 1 , 2 , · · · , N in parallel do 6: if t < T initial then 7:
Take a uniform random action a it to environment copy E i 8: else 9:
Take an action a i t ∼ π i (cid:0) ·| s i t (cid:1) to environment copy E i 10: end if 11:
Store experience ( s i t , a i t , r it , s i t +1 ) to the shared common experience replay D 12: end for 13: if t < T initial then 14: continue (i.e., go to the beginning of the while loop) 15: end if 16: for i
= 1 , 2 , · · · , N in parallel do 17: for k = 1 , 2 , · · · , N do 18:
Sample a mini-batch B = { ( s t l , a t l , r t l , s t l +1 ) } l =1 ,...,B from D 19:
Update ψ i , θ ij , and φ i by gradient descent for minimizing (16), (17), and (18) with B , respectively.
Update the target parameters: ( ψ i ) ′
← (1 − τ )( ψ i ) ′
+ τψ i 21: end for 22: end for 23: if t ≡ 0( mod M ) then 24:
Select the best learner b 25:
Update the best policy parameter φ b 26:
Adapt β with (2) 27: end if 28: end while
In IPE-SAC, each learner has its own parameters ψ i , θ i 1 , θ i 2 , and φ i for its value function, two Qfunctions, and policy.
Each learner also has ( ψ i ) ′
which is the parameter of the target value function.
For the distance measure between two policies, we use the mean square difference of the mean action of Gaussian policy, given by D ( π ( ·| s ) , ˜ π ( ·| s ))
=
12 k mean { π ( ·| s ) } − mean { ˜ π ( ·| s ) }k 22 .
The i -th learner updates the parameters ψ i , θ
i 1 , θ i 2 , and φ i every timestep by minimizing
[Equation]
mean
[Equation]
mean
[Equation]
where ¯ Q i ( s, a ) = min
[Equation]
o .
After updating these parameters, each learner updates its target value function parameters.
With these loss functions, all procedure is the same as the general IPE procedure described in Section 3.
The pseudocode of the IPE-SAC algorithm is shown above.
A PPENDIX D. R ESULT OF IPE-SAC ON H UMANOID ( RLLAB )
As mentioned already, IPE is general in that it can be applied to other off-policy algorithms.
Here, we provide numerical results on IPE-SAC, shown in Appendix C, constructed by combining IPE with SAC.
Experiment was perform on the task of Humanoid (rllab)
(Duan et al.
that needs more exploration.
We compared IPE-SAC with SAC and multi-learner reloading SAC (Re-SAC) which copies the parameter of the best learner to other learners periodically.
D.1 P ARAMETER S ETTING SAC
The networks for the state-value function, two Q-functions, and the policy had 2 hidden layers of size 256.
The activation functions for the hidden layers and the last layers were ReLU and linear, respectively.
We used the Adam optimizer with learning rate 3 × 10 − 4 , discount factor γ = 0 . 99 , and target smoothing factor τ = 5 × 10 − 3 .
The algorithm was trained by random mini-batches of size
B = 256 from the experience replay buffer of the maximum size 10 6 .
The reward scale for updating
Q-functions was 10 for the Humanoid (rllab) environment.
The initial exploration timesteps T initial was set to 1000.
IPE-SAC Additional parameters for IPE-SAC are as follows.
We used N = 4 learners, the period
M = 500 of updating the best policy and β , the number of recent episodes E r = 10 for determining the best learner b .
The parameters d search and ρ for the exploration range were 0.01 and 2, respectively.
We used the initial exploration timesteps T initial = 250 .
D.2 P ERFORMANCE ON H UMANOID ( RLLAB )
The learning curve on Humanoid (rllab) is shown in Figure 5 1 .
It is seen that IPE-SAC outperforms the original SAC and Re-SAC.
This result shows the promising aspect of IPE that it can be useful for tasks requiring more exploration.
A PPENDIX E. P SEUDOCODE OF THE IPE-DQN A LGORITHM
Require:
M
1: Initialize θ 1 = · · · = θ N = θ b randomly.
2: Initialize β
= 1 , t =3: while t < T do 4: t ← t + 1 (one time step) 5: for i = 1 , 2 , · · · , N in parallel do 6: if t < T initial then 7:
Take a uniform random action a it to environment copy E i 8: else 9:
Take an action a it
= arg max a ∈A (cid:8)
Q θ i ( s it , a ) (cid:9) w.p. 1 − ε or a uniform random action a it w.p. ε to environment copy E i 10: end if 11:
Store experience ( s it , a it , r it , s it +1 ) to the shared common experience replay D 12: end for 13: if t < T initial then 14: continue (i.e., go to the beginning of the while loop) 15: end if 16: for i
= 1 , 2 , · · · , N in parallel do 17: for k = 1 , 2 , · · · , N do 18: if k ≡ 0( mod f ) then 19:
Sample a mini-batch B = { ( s t l , a t l , r t l , s t l +1 ) } l =1 ,...,B from D 20:
Update θ i by gradient descent minimizing (19) with B .
21: end if 22: end for 23: end for 24: if t ≡ 0( mod d ) then 25: for i = 1 , 2 , · · · , N in parallel do 26:
Update ( θ i ) ′
← θ i 27: end for 28: end if 29: if t ≡ 0( mod M ) then 30:
Select the best learner b 31:
Update the best policy parameter θ b 32:
Adapt β with (2) 33: end if 34: end while
IPE can also be applied to off-policy algorithms with discrete action spaces as well as continuous action spaces.
Thus, we applied IPE to DQN to construct IPE-DQN.
In IPE-DQN, each learner has its own Q-function parameters θ i and target Q-function parameters ( θ i ) ′ .
We deﬁne the distance for two Q-functions Q ( s, a ) and ˜ Q ( s, a ) as D ( Q ( s, · ) , ˜ Q ( s, · )) =
KL
[Equation]
[Equation]
softmax
[Equation]
[Equation]
(cid:17)(cid:17) .
We used the Q-function parameter θ b of the best learner as the reference parameter, which was originally φ b in (1) and (3).
The i -th learner updates the parameters θ i every f timesteps by minimizing ˜ L ( θ i )
=
ˆ E ( s,a,r,s ′ )
∼D
[Equation]
(19) where
y = r + γQ ( θ i ) ′
( s ′ , arg max a ′ ∈A { Q θ i ( s ′ , a ′ ) } ) .
With the loss function and the reference policy, all procedure is the same as the general IPE procedure described in Section 3.
The pseudocode of the IPE-DQN algorithm is shown above.