Experience Replay More When It’s a Key Transition in Deep Reinforcement Learning
Anonymous authors
Paper under double-blind review
Abstract
We proposed an experience replay mechanism in Deep Reinforcement Learning based on Add Noise to Noise (AN2N), which requires agent to replay more experience containing key state, abbreviated as Experience Replay More (ERM).
In the AN2N algorithm, we refer to the states explored more as the key states.
We found that how the transitions containing the key state participates in updating the policy and Q networks has a significant impact on the performance improvement of the deep reinforcement learning agent, and the problem of catastrophic forgetting in neural networks is further magnified in the AN2N algorithm.
Therefore, We sample the transition used for experience replay according to whether the transition contains key states and whether it is the most recently generated, rather than sampling uniformly, which is the core idea of the ERM algorithm.
The experimental results show that this algorithm improves the performance of the agent by a wide margin.
We combine the ERM algorithm with Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic policy gradient (TD3) and Soft Actor-Critic (SAC), and evaluate algorithm on the suite of OpenAI gym tasks, SAC with ERM achieves a new state of the art, and DDPG with ERM can even exceed the average performance of SAC under certain random seeds, which is incredible.
Introduction
Deep reinforcement learning (RL) has shown its promising future for decision-making in various computer games, such as atari (Mnih et al., 2013; 2015), go (Schrittwieser et al., 2020) and starcraft (Vinyals et al., 2019).
However, most successes have been exclusively in simulation due to poor sample eﬀiciency of typical Deep RL algorithm and other challenges.
Reinforcement learning can be divided into model-based RL and model-free RL in the light of its data eﬀiciency.
Model-free RL is usually subdivided into off-policy RL and on-policy RL.
Although model-based RL requires less sampled data, it needs to build a world model and predict the next state based on a lot of prior work, demonstrated in Hafner et al.
Although the on-policy RL algorithm do without establishing a world model and has good stability, the performance improves slowly due to limiting the step of updating the policy, a large number of sampled trajectory data are required in the training process (Schulman et al., 2015; 2017).
Off-policy RL is between model-based RL and on-policy RL in terms of sampling eﬀiciency, and the research in this field is enduring (Watkins & Dayan, 1992;
Hessel et al., 2018; Barth-Maron et al., 2018) on account of its relatively high data eﬀiciency and world model free.
Experience Replay (Lin, 1992) is an important part of improving the data eﬀiciency of Offpolicy RL, which stores experience in a replay buffer and break the temporal correlations by mixing data, therefore, the experience can be used multiple times to update the networks.
However, most of the current work is to uniformly sample transitions from the buffer, such as Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), Soft Actor-Critic (SAC) (Haarnoja et al., 2018), Twin Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., 2018) and many other algorithms (Van Hasselt et al., 2016; Mnih et al., 2016;
Andrychowicz et al., 2017; Dabney et al., 2018; Liu et al., 2020).
However, these approaches replay experience transitions at the same frequency, regardless of their meaning.
Schaul et al.
(2016) develops a framework for prioritizing experience to replay important transitions more frequently, therefore agent learns more eﬀiciently.
Prioritized experience replay (PER) method samples transitions with the magnitude of their temporal-difference (TD) error.
However, prioritization introduces bias, which needs to be corrected with importance sampling.
Meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience (Rothfuss et al., 2018;
Mishra et al., 2018), Rakelly et al.
(2019) develops an off-policy meta-RL algorithm that disentangles the inference and control of the task, but its performance is still far behind the mainstream off-policy RL algorithms.
In this paper, We first analyzed how the state changes from beginning of agent interacting with the environment to the policy converged, and found that the state of the agent is different at different stages, agent will rarely transfer to those states where the policy is far from converging when the agent’s policy already converged.
If a large number of initialstates 1 are used to train Q-value and policy networks, the network weights that have been learned well will be gradually updated.
Therefore, the sampling proportion of the most recently generated transitions should be appropriately increased, so that the agent pays more attention to learning the recent transitions, In this way, the state of the agent is gradually transferred from a poor state to a better state, similar to updating from a poor policy to a better policy stabily in Trust region policy optimization (TRPO) (Schulman et al., 2015) or Proximal policy optimization (PPO) (Schulman et al., 2017).
Inspired by the Add Noise to Noise (AN2N)
Algorithm (Guo & Gao, 2021), We divide the states into two categories.
These states explored with added noise are called key states 2 , and the rest are called non-key states.
For the sake of improving the performance in key states in time, we have increased the probability of sampling new key states, making them more likely to participate in updating policy, we call this process as ERM (Experience Replay
More), and combine it with commonly used off-policy RL algorithms in continuous control tasks, such as SAC, which obtained faster convergence and state-of-the-art performance.
Preliminaries
We consider a reinforcement learning setup consisting of agent learning policies to maximize the expected reward when interacting with the environment (Sutton & Barto, 2018).
At each timestep t , the agent receives an observation o t ∈ O , selects action a t ∈ A with respect to its policy π : O → A .
After taking the action a t in environment E , agent receives a reward r t and the next observation o t +1 .
The practical problem is usually a partial Markov decision process (POMDP), only part of the observation information could be obtained.
To simplify the problem, we assumed the environment is fully-observed, so s t
= o t , S
= O .
In reinforcement learning, the action-value function Q π ( s, a ) is uesed to approximate the expected sum reward of the ation a in state s , defined as following:
[Equation]
Where γ ∈ [0 , 1] is the discount factor, E s t ∼ p π ,a t ∼ π is the expectation over the distribution of the trajectories ( s 0 , a 0 , s 1 , a 1 , . . . ) .
The mean value of Q π in the same state s called the value function V π , defined as
V π ( s ) =
E a ∼ π ( ·| s )
[ Q π ( s, a )] .
We express the action-value function Q π in the form of Bellman equation (Bellman & Kalaba, 1965):
[Equation]
In this paper, we need to be familiar with DDPG, TD3 and SAC algorithms, here, we mainly introduce DDPG as the basis.
DDPG applied two different fully connected neural networks to approximate the action-value function Q ( s, a | θ Q ) and policy function µ ( s | θ µ ) ,
DDPG introduces action-value target network θ Q ′ and policy target network θ µ ′ , so as to
Stable the policy update.
Consequently, gradient descent is used to optimize the network weights by minimizing the loss:
[Equation]
Where
[Equation]
Where equation 4 derived from equation 2, the weights of target networks are updated periodically to slowly track the learned networks: θ ′ ← τθ
+ (1 − τ ) θ ′ with τ ≪ 1 , which alleviates the fluctuation in the agent’s learning process.
The policy is updated by equation 5, following the chain rule to the expected sum return Q ( s, a | θ Q ) with respect to parameters θ µ .
TD3 addresses the problem that DDPG is prone to overestimating the Q function.
An additional Q-function is added, and the predicted smaller Q value is used as the calculation of TD-error.
At the same time, which also reduces the update frequency of the policy function.
The most significant difference between SAC and DDPG is the introduction of policy entropy H ( π ( ·| s t )) , so the objective function equation 1 is rewritten as:
[Equation]
Where α is temperature parameter, which adjusts the optimization target agent pays more attention to exploration if increase the coeﬀicient α .
[Figure]
Experience Replay More When It’s a Key Transition
The addition of experience replay improves the sample eﬀiciency of off-policy RL.
Many off-policy algorithms usually sample transitions uniformly from the experience buffer, which potentially considers the transitions generated at different times to be of the same importance, we will discuss this in section 3.1.
Prioritized Experience Replay (PER) is an optimization of the uniform sampling method based on the TD-error value of transitions so as to learn the samples more eﬀiciently, however, PER essentially still does not consider whether the transitions generated at different times are equally important to the current agent’s policy.
3.1 States are Different at Different Stages
Taking the HalfCheetah-v2 simulation environment as an example, we recorded the state information of an agent at different stages based on DDPG algorithm, collecting attitude information at the specific position and the corresponding Q-value starting from 2e4 to 5.6e5, and then we uniformly sample 100 sets of data from the collected data for drawing, as shown in the Fig.
The legend in the upper left corner represents the Q value, which is discretized to reasonably reduce the number of Legends.
Subgraph (a) is discreted in units of 20, and subgraph b is discreted in units of 100.
It can be found when the agent interacts with the environment, not only the policy is gradually improving 3 , but the state is also changing accordingly.
After policy converges and the agent is in a state set different from the previous one, more experience similar to the agent’s recent states should be collected to train the neural network to strengthen the current policy.
If the sampling experience for training policy network and the action state network are quite different from the recently generated states, it will not only help improve the policy to a small extent, but will also even forget the newly learned network weights due to the catastrophic forgetting of the neural networks.
Therefore, to further improving the sample eﬀiciency, experience replay should be switched from uniform sampling to targeted sampling.
3.1.1 Replaying More Key Transitions
The AN2N algorithm will record the dilemma state s d during the evaluation process, so when agent interacts with the environment, if the current state s c and s d are considered to be highly similar, an additional exploration noise N a will be added.
In this paper, the state s c with additional noise is called the key state s k , and the transition containing s k is called the key transition tran k .
The agent’s policy is relatively fragile in s k , if agent has not learned how to get out of this dilemma before, agent is more likely to fall into a series of bad states after s k , which greatly reduces the agent’s overall performance.
Therefore, whether the agent can learn a good policy in s k is very important for improving the performance.
It can be seen from equation 2 that reward of the key state s k will affect the Q value of the state at the previous moment through the iteration of the dynamic equation, nevertheless, if we uniformly sample experience for training the networks, there will be the following three problems:
• Since most of the experience generated by AN2N are non-key states, the probability of sampling key states s k in the experience buffer is small.
The key state s k is time-sensitive.
The new key state needs to participate in training the network as quickly as possible.
Otherwise, if agent’s state undergoes a relatively change, its role in improving the policy will be declined.
• Since the agent’s state is gradually changing, more recent the sampled experience, more similar its distribution is to the distribution of the latest experience generated currently, and the more it satisfies the assumption of independent and identical distribution (iid).
In response to the first question, we designed two experience buffers to store key transition and non-key transition respectively, and used min( Prt AN 2 N , K t ) to adjust the proportion of sampling key transition, where Prt AN 2 N is the proportion of the number of key transitions generated by AN2N, K t is linearly related to the simulation times of the agent.
The above ensures that key transitions can be sampled strictly according to the proportion from the experience buffer.
For the second and third questions, we will linearly increase the probability of new transition being sampled.
Two sampling functions will sample 100 samples in [0 , 20] to compare the difference of the linear distribution sampling function and the uniform distribution more vividly, the result shown in Fig.
The pseudo code of ERM algorithm is shown in algorithm 1.
Input: Sampling ratio Prt AN 2 N , K t , Replay buffer R non − key , R key , batch size bs 1 , bs 2 , bs sum and AN2N parameters
Randomly initialize critic network Q ( s, a | θ Q ) and actor µ ( s | θ µ ) with weights θ Q and θ µ
Initialize target network Q ′ and µ ′ with weights θ Q ′ ← θ Q , θ µ ′ ← θ µ for episode e ∈ { 1,...,M } do
Initialize a random process N for action exploration
Receive initial observation state s 1 for t ∈ {
T } do
Execute AN2N action a t and observe reward r t and observe new state s t + if (AN2N exploring more) then
Store key transitio elseStore transition end
Sample trainingRunDDPG or
Algorithm end end
Experiments
In this section, we test the performance of the combinations of Experience Replay More algorithm (ERM) with different off-policy RL algorithms acrossing a variety of continuous control tasks.
As is shown in Fig.
3, consistent with the benchmark, we use the mainstream Mujoco physics engine (Todorov et al., 2012) as the simulation environment to test the performance of the algorithm in the HalfCheetah-v2, Swimmer-v2, Walker2d-v2, and
Hopper-v2 tasks, as Mujoco offers a unique combination of speed, accuracy and modeling power, and it is also the first full-featured simulator designed from the ground up for the purpose of motion control.
For the specific introduction of the task environment is shown in Table 2 in Appendix A.
We present the ERM by classifying key transitions and sampling them with linear distribution functions for the purpose of increasing the stability and performance with sampling eﬀiciency, described in section 3.
In each task, we run our algorithm and test it without exploration noise.
The goal of our experimental evaluation is to understand how good the sample eﬀiciency and stability of our method are, compared with prior off-policy RL algorithms.
In all tasks, we run experiments for five times, which fix random seeds in 0, 5, 10, 15, 20 respectively.
The results of ERM combined with different off policy RL algorithms are analyzed below.
4.1 DDPG with ERM
For the implementation of DDPG with ERM, we use a two layer fully connected network consists of 256 × 256 hidden nodes respectively, with rectified linear units (ReLU) between each layer for both the policy and action state networks, and a final tanh unit following the output of the policy network for limiting amplitude.
After a certain number of steps, the networks are trained with a mini-batch of 100 transitions repeatedly, sampled from a replay buffer containing the entire history of the agent.
See Appendix A for more experimental details.
[Figure]
DDPG with ERM uses two actor-networks and critic-networks respectively to approximate the policy and action-state value.
In the test stage, we record the reward of each state of the agent and then calculate all the action state values of the trajectory when the episode finished, save those transitions whose total rewards are minimal.
When the agent interacts with the environment, it uses the policy superimposed a small disturbance noise, if the current state is similar to key states, add a disturbance noise on small noise, otherwise, only use the small noise.
Besides, we store the key tansitions and non-key transitions separatly, which is convenient for sampling two kinds of transitions in a appropriate proportion with linear distribution sampling function.
The pseudo code of DDPG with ERM is shown in algorithm 2 in Appendix B.
We compare the DDPG with ERM algorithm with DDPG baseline, illustrated in Fig.
4, among the four tasks, the performance of DDPG with ERM is higher.
In the HalfCheetah task, the method combined with ERM has significantly lower variance while maintaining higher performance, indicating that the algorithm we proposed with DDPG has better stability and sample eﬀiciency.
4.2 TD3 with ERM
TD3 is an optimized version on the basis of DDPG, and the structure is very similar to the DDPG algorithm.
The main three improvements are for the problems of DDPG in engineering practice: 1.Clipped Double-Q Learning.
TD3 uses two Q networks with the same structure to predict the state action value at the same time, but when calculating TD-error, only the Q value with the smallest prediction participates in the calculation, so as to alleviate the overestimation of DDPG.
2.Delayed Policy Updates.
Since updating the policy frequently is prone to make the policy unstable, therefore, TD3 updates the policy with a lower frequency.
3.Target Policy Smoothing.
When calculating target Q, a small range of noise is added to the policy to make the calculated target Q value more robust.
Therefore, TD3 with ERM can maintain the same network structure and hyper-parameters with DDPG with ERM.
See Appendix A for more details.
[Figure]
In the same tasks with above, we tested the performance of TD3 and TD3 with ERM.
As illustrated in Fig.
In all the test tasks, the performance improvement of TD3 with ERM is very obvious compared to TD3, especially in HalfCheetah and Walker2d tasks.
At the same time, the method of combining TD3 and ERM has significantly lower variance while maintaining higher performance in the HalfCheetah and Hopper tasks, indicating that the algorithm we proposed is very stable and sample eﬀicient on TD3.
4.3 SAC with ERM
Compared with TD3, SAC is more different from DDPG, but it is still an Actor-Critic structure.
The main differences are as follows:
1.The policy entropy is added to the objective function, which can more fully explore the action space, but there is also an additional temperature coeﬀicient that adjusts the entropy proportion of the policy.
2.SAC does not directly outputting the deterministic policy, but a mean value and variance of a policy’s distribution, so it is necessary to adjust the range of the variance to achieve a noise-like addition.
See Appendix A for the specific hyper-parameter settings of SAC.
The SAC algorithm is currently one of the most commonly used off-policy RL methods in academic and industry owing to its good performance, stability and sample eﬀiciency.
We tested the performance of SAC and SAC with ERM in four tasks, as shown in Fig.
6, we found that in half of the tasks, the performance of SAC with ERM still has a performance improvement compared to sac, while the performance of the remaining tasks was flat, indicating that the algorithm we proposed also owns a better stability and sample eﬀiciency on
SAC.
[Figure]
In addition, we summarize the results of the above several algorithms in the HalfCheetah simulation environment.
As shown in the sub-graph (a) in Fig.
We run agent with each algorithm for 1 million time steps with evaluations every 4000 time steps, where each evaluation reports the average reward over 10 episodes with no exploration noise.
Our results are reported over 5 random seeds of the Gym simulator.
It can be seen that the performance of SAC with ERM is the best, surpassing state of the art (sac), and its convergence speed is also the fastest, indicating that it has higher sample eﬀiciency compared with other off-policy RL algorithms.
[Figure]
We select the result in DDPG with ERM with the random seed 5 in HalfCheetah-v2 task, we compare it with the average performance of SAC, as shown in the subgraph (b) in Fig.
It can be found that although the convergence speed of ddpg with ERM is slower in the early stage, its performance has approached or even exceeded the average performance of SAC in the later stage.
This improvement is very incredible for DDPG.
We display the statistical data of all the experimental results in Table 1.
The first column indicates the algorithm or policy, among which random indicates that the agent uses a random policy to interact with the environment.
The numbers in the table represent the average cumulative rewards obtained by the corresponding algorithm or policy in the environment in a episode with 4000 steps.
The numbers in bold are the highest performance scores.
It can be seen that the SAC with ERM algorithm has the highest score, followed by the DDPG with ERM algorithm, and ERM has the greatest effect on improving the performance of TD3.
The statistical average cumulative reward results show that ERM is very helpful to improve the performance of DDPG, TD3 and SAC.
[Table]
In all the experiments, the most time-consuming part of the algorithm is calculating the similarity between states.
To solve this problem, we accelerate the process by using matrix operation, which expands the dimension of current state S c to be consistent with the dimension of key states buffer, so we calculate the similarity with the matrix S k composed of all key states quickly.
Therefore, in the case of a small increase in time consumption, the performance of the algorithm is significantly improved, especially on the HalfCheetah task.
Conclusion
This work divides the states of the agent into key states and non-key states, and analyzes the reasons why the recent generated key states need to be sampled and trained as soos as possible.
For the purpose of sampling key transitions more accurately, we introduce two experiences buffers to store the key transitions and non-key transitions respectively, and set a adjustable coeﬀicient to determine the proportion of transitions sampled from two experience buffers.
Besides, we analyze the advatages of sampling the key transitions if we make use of linear distribution function from two perspectives: 1.The distribution of transitions sampled recently are more similar to the distribution of latest experience generated, making transitions more satisfie the assumption of independent and identical distribution (iid);
2.The recent generated experience can be used for the training of the Q value network and policy network of the agent more quickly, so as to make up for the lack of the policy in time.
Finally, on the basis of AN2N, the combination of ERM method and
DDPG, TD3 or SAC algorithm has a very obvious performance improvement on tasks such as HalfCheetah.
The performance of the DDPGERM algorithm with some random seeds can even exceed the average performance of SAC, and SACERM has also outperforms the current state of the art.
Acknowledgments
We would like to thank Feng Pan, Weixing Li, Xiaoxue Feng, Yan Gao and many others at Institute of Pattern Recognition and Intelligent System of BIT for insightful discussions and feedback.
A Experiment Details
The experiment uses a unified parameter configuration for all algorithms, the two hidden layers are composed of 256 nodes and the neural networks use the rectified non-linearity (Glorot et al., 2011) for all hidden layers.
The activation function of the actor output layer uses a tanh for bounding the actions.
Adam (Kingma & Ba, 2014) optimizes the weights of the neural networks with a learning rate of 10 3 for the both of actor and critic networks, discount factor set to 0 .
For the soft target updates, we set τ = 0 .
At the beginning, the agent uses a random policy to generate 10 4 transitions for the initial training of the network, the proportion of sampling key transitions is fixed by min( Prt AN 2 N , K t ) , where
Prt AN 2 N = 0 .
4 − 0 .
2 ∗ t totaltime , K t = 50 ∗ t totaltime .
We train the networks with minibatch sizes of 100 for all of the tasks, and use a replay buffer size of 10 6 .
For DDPG or TD3 with
ERM, we set two kind noise N big = 0 .
4 and N small = 0 .
For SAC with ERM, different from the policy generation method of DDPG and TD3, SAC generates the mean and variance of the policy, and then sample a specific action.
Therefore, we set the expansion and reduction coeﬀicients of the variance to 1.5 and 0.5.
We test our algorithm in the Mujoco task environment.
The state dimension, observation dimension, and action dimension of the tasks are shown in Table 2, and the detailed description of the tasks are shown in Table 3.
[Table] [Table]
The agent should move forward as quickly as possible with a cheetah like body that is constrained to the plane (Wawrzyński & Tanwani, 2013).
[Table]
This task involves a 3-link swimming robot in a viscous fluid, where the goal is to make it swim forward as fast as possible (Coulom, 2002).
[Table]
The agent’s goal is making a two-dimensional bipedal robot walk forward as fast as possible (Erez et al., 2011).
[Table]
The agent’s goal is making a two-dimensional one-legged robot hop forward as fast as possible (Erez et al., 2011).
B Algorithm
Appendix B includes two parts: the first part is the overall structure diagram 8 of the Experience Replay More (ERM) algorithm.
The core of the entire algorithm is to classify and store the data stream, and sample the experience with a linear distribution function.
The second part is the pseudo code of the algorithm: algorithm pseudo code of DDPG with ERM in Algorithm 2, algorithm pseudo code of TD3 with ERM in Algorithm 3 and algorithm pseudo code of SAC with ERM in Algorithm 4.
Input: Sampling ratio Prt AN 2 N , K t , Replay buffer R non − key , R key , batch size bs 1 , bs 2 , bs sum and AN2N parameters
Randomly initialize critic network Q ( s, a | θ Q ) and actor µ ( s | θ µ ) with weights θ Q and θ µ
Initialize target network Q ′ and µ ′ with weights θ Q ′ ← θ Q , θ µ ′ ← θ µ for episode e ∈ { 1,...,M } do
Initialize a random process N for action exploration
Receive initial observation state s 1 for t ∈ { 1,...,T } do
Execute AN2N action a t and observe reward r t and observe new state s t + if (AN2N exploring more) then
Store key transitio ( s t , a t , r t , s t +1 ) in R key elseStore transition ( s t , a t , r t , s t +1 ) in R non − key end bs 1 = min( Prt AN 2 N , K t ) bs 2 = bs sum − bs 1 Sample bs 1 and bs 2 transitions with linear distribution in R key and R non − key for training if t mod u then
Sample a random minibatch of N transitions ( s i , a i , r i , s i +1 ) from R
Set y i
= r i + γQ ′
( s i +1 , µ ′ ( s i +1 | θ µ ′ )
| θ Q ′ )
Update critic by minimizing the loss:
L = 1 N ∑ i ( y i − Q
( s i , a i | θ Q ))
Update the actor policy using the sampled policy gradient: ∇ θ µ J ≈ 1 N ∑ i ∇ a Q ( s, a | θ Q ) | s = s i ,a = µ ( s i ) ∇ θ µ µ ( s | θ µ ) | s i
Update the target networks: θ Q ′ ← τθ Q
+ (1 − τ ) θ Q ′ θ µ ′ ← τθ µ + (1 − τ ) θ Q µ ′ end end end
Input: Sampling ratio Prt AN 2 N , K t , Replay buffer R non − key , R key , batch size bs 1 , bs 2 , bs sum and AN2N parameters
Randomly initialize critic networks Q ( s, a | θ Q 1 ) , Q ( s, a | θ Q 2 ) , and actor µ ( s | θ µ ) with weights θ Q 1 , θ Q 2 and θ µ
Initialize target network Q ′ and µ ′ with weights θ Q ′ 1 ← θ Q 1 , θ Q ′ 2 ← θ Q 2 , θ µ ′ ← θ µ for episode e ∈ { 1,...,M } do
Initialize a random process N for action exploration
Receive initial observation state sfor t ∈ { 1,...,T } do
Execute AN2N action a t and observe reward r t and observe new state s t + if (AN2N exploring more) then
Store key transitio ( s t , a t , r t , s t +1 ) in R key elseStore transition ( s t , a t , r t , s t +1 ) in R non − key end bs 1 = min( Prt AN 2 N , K t ) bs 2 = bs sum − bs
Sample bs 1 and bs 2 transitions with linear distribution in R key and R non − key for training if t mod u then
Sample a random minibatch of N transitions ( s i , a i , r i , s i +1 ) from R
Get ˜ a ← µ ( s | θ µ ′ )
+ ϵ, ϵ ∼ clip ( N (0 , ˜ σ ) , − c, c )
Set y i
= r i + γ min j =1 , 2 Q ′
( s i +1 , µ ′ ( s i +1 | ˜ a ) | θ
Q ′ j )
Update critic by minimizing the loss:
L = min θ j 1 N ∑ i ( y i − Q
( s i , a i | θ Qj ))
if t mod ( d × u ) then
Update the actor policy µ ( s | θ µ ) using the sampled policy gradient: ∇ θ µ J ≈ 1 N ∑ i ∇ a Q ( s, a
[Equation]
Update the target networks:
[Equation]
end end end end
Input: Sampling ratio Prt AN 2 N , K t , Replay buffer R non − key , R key , batch size bs 1 , bs 2 , bs sum and AN2N parameters, Temperature parameter α
Randomly initialize critic networks Q ( s, a | θ Q 1 ) , Q ( s, a | θ Q 2 ) , and actor µ ( s | θ µ ) with weights θ Q 1 , θ Q 2 and θ µ
Initialize target network Q ′ and µ ′ with weights θ Q ′ 1 ← θ Q 1 , θ Q ′ 2 ← θ Qfor episode e ∈ { 1,...,M } do
Initialize a random process N for action exploration
Receive initial observation state s 1 for t ∈ { 1,...,T } do
Execute AN2N action a t and observe reward r t and observe new state s t + if (AN2N exploring more) then
Store key transitio ( s t , a t , r t , s t +1 ) in R key elseStore transition ( s t , a t , r t , s t +1 ) in R non − key end bs 1 = min( Prt AN 2 N ,
[Equation]
) bs 2 = bs sum − bs 1 Sample bs 1 and bs 2 transitions with linear distribution in R key and R non − key for training if t mod u then
Sample a random minibatch of N transitions ( s i , a i , r i , s i +1 ) from R
Set y i
=
r i + γ (min
[Equation]
=
[Equation]
Update critic (soft Q-function) by minimizing the loss:
[Equation]
Update the actor policy using the sampled policy gradient:
[Equation]
−∇
[Equation]
( s t | θ µ )) − ∇ θ µ log µ ( s t | θ µ ) | s i )
Update the target networks:
[Equation]
end end end