D ROPOUT Q-F UNCTIONS FOR D OUBLY E FFICIENT R EINFORCEMENT L EARNING
A BSTRACT
Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efﬁciency on continuous-action reinforcement learning benchmarks.
This superior sample efﬁciency is made possible by using a large Q-function ensemble.
However, REDQ is much less computationally efﬁcient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et al., 2018a).
To make REDQ more computationally efﬁcient, we propose a method of improving computational efﬁciency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions.
Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization.
Despite its simplicity of implementation, our experimental results indicate that DroQ is doubly (sample and computationally) efﬁcient.
It achieved comparable sample efﬁciency with REDQ, much better computational efﬁciency than REDQ, and comparable computational efﬁciency with that of SAC.
I NTRODUCTION
In the reinforcement learning (RL) community, improving sample efﬁciency of RL methods has been important.
Traditional RL methods have been shown to be promising for solving complex control tasks such as dexterous in-hand manipulation (OpenAI et al., 2018).
However, RL methods generally require millions of training samples to solve a task (Mendonca et al., 2019).
This poor sample efﬁciency of RL methods is a severe obstacle to practical RL applications (e.g., applications on limited computational resources or in real-world environments without simulators).
Motivated by these issues, many RL methods have been proposed to achieve higher sample efﬁciency.
For example, Haarnoja et al.
(2018a;b) proposed Soft Actor-Critic (SAC), which achieved higher sample efﬁciency than the previous state-of-the-art RL methods (Lillicrap et al., 2015; Fujimoto et al., 2018;
Schulman et al., 2017).
Since 2019, RL methods that use a high update-to-data (UTD) ratio to achieve high sample efﬁciency have emerged.
The UTD ratio is the number of updates taken by the agent compared to the number of actual interactions with the environment.
A high UTD ratio promotes sufﬁciently training Q-functions within a few interactions, which leads to sample-efﬁcient learning.
Model-Based
Policy Optimization (MBPO) (Janner et al., 2019) is a seminal RL method that uses a high UTD ratio of 20–40 and achieves signiﬁcantly higher sample efﬁciency than SAC, which uses a UTD ratio of 1.
Encouraged by the success of MBPO, many RL methods with high UTD ratios have been proposed (Shen et al., 2020; Lai et al., 2020).
With such methods, randomized ensembled double Q-learning (REDQ) proposed by Chen et al.
(2021b) is currently the most sample-efﬁcient method for the MuJoCo benchmark.
REDQ uses a high UTD ratio and large ensemble of Q-functions.
The use of a high UTD ratio increases an estimation bias in policy evaluation, which degrades sample-efﬁcient learning.
REDQ uses an ensemble of Q-functions to suppress the estimation bias and improve sample efﬁciency.
Chen et al.
(2021b) demonstrated that the sample efﬁciency of REDQ is equal to or even better than
However, REDQ leaves room for improvement in terms of computational efﬁciency .
REDQ runs 1.1 to 1.4 times faster than MBPO
(Chen et al., 2021b)
but is still less computationally efﬁcient than non-ensemble-based RL methods (e.g., SAC) due to the use of large ensembles.
In Section 4.2, we show that REDQ requires more than twice as much computation time per update, and much larger memory.
Computational efﬁciency is important in several scenarios, e.g., in RL applications with much lighter on-device computation (e.g. mobile phones or other light-weight edge devices)
(Chen et al., 2021a), or in situations in which rapid trials and errors are required for developing RL agents (e.g., hyperparameter tuning or proof of concept for RL applications with limited time resources).
Therefore, RL methods that are superior not only in terms of sample efﬁciency but also in computational efﬁciency are preferable.
We propose a method of improving computational efﬁciency.
Our method is called DroQ and is a REDQ variant that uses a small ensemble of dropout Q-functions, in which dropout (Srivastava et al., 2014) and layer normalization (Ba et al., 2016) are used (Section 3).
We experimentally show that DroQ is doubly (computationally and sample) efﬁcient: (i) DroQ signiﬁcantly improves computational efﬁciency over REDQ (Section 4.2) by more than two times and (ii) achieves sample efﬁciency comparable to REDQ (Section 4.1).
Although our primary contribution is proposing a doubly efﬁcient RL method , we also make three signiﬁcant contributions from other perspectives:
Simplicity of implementation.
DroQ can be implemented by basically adding a few lines of readily available functions (dropout and layer normalization) to Q-functions in REDQ (and SAC).
This simplicity enables one to easily replicate and extend it.
First successful demonstration of the usefulness of dropout in high UTD ratio settings.
Previous studies incorporated dropout into RL (Gal & Ghahramani, 2016; Harrigan, 2016; Moerland et al., 2017; Gal et al., 2016; 2017; Kahn et al., 2017; Jaques et al., 2019;
He et al., 2021) (see Section 5 for details).
However, these studies focused on low UTD ratio settings (i.e., UTD ratio ≤ 1 ).
Dropout approaches generally do not work as well as ensemble approaches (Osband et al., 2016; Ovadia et al., 2019; Lakshminarayanan et al., 2017; Durasov et al., 2021).
For this reason, instead of dropout approaches, ensemble approaches have been used in RL with high UTD ratio settings (Chen et al., 2021b; Janner et al., 2019;
Shen et al., 2020;
Hiraoka et al., 2021; Lai et al., 2020).
In Section 4, we argue that DroQ achieves almost the same or better bias reduction ability and sample/computationally efﬁciency compared with ensemble-based RL methods in high UTD ratio settings.
This sheds light on dropout approaches once again and promotes their use as a reasonable alternative (or complement) to ensemble approaches in high UTD ratio settings.
Discovery of engineering insights to effectively apply dropout to RL.
Speciﬁcally, we discovered that the following two engineering practices are effective in reducing bias and improving sample efﬁciency: (i) using the dropout and ensemble approaches together for constructing Q-functions (i.e., using multiple dropout Q-functions) (Section 4.1 and Appendix A.3); and (ii) introducing layer normalization into dropout Q-functions (Section 4.3 and Appendices D, E, and F).
These engineering insights were not revealed in previous RL studies and would be useful to practitioners who attempt to apply dropout to RL.
P RELIMINARIES 2. M AXIMUM E NTROPY R EINFORCEMENT L EARNING ( MAXIMUM ENTROPY RL)
RL addresses the problem of an agent learning to act in an environment.
At each discrete time step t , the environment provides the agent with a state s t , the agent responds by selecting an action a t , and then the environment provides the next reward r t and state s t +1 .
For convenience, as needed, we use the simpler notations of r , s , a , s ′ , and a ′ to refer to a reward, state, action, next state, and next action, respectively.
We focus on maximum entropy RL, in which an agent aims to ﬁnd its policy that maximizes the expected return with an entropy bonus: arg max π
P Tt =
γ t E s t ,a t [ r t + α H
( π ( ·| s t ))] .
Here, π : S ×
A → [0 , ∞ ) is a policy and H is entropy.
Temperature α balances exploitation and exploration and affects the stochasticity of the policy.
Compute y =
r + γ (cid:18) min i ∈M Q ¯
ϕ i ( s ′ , a ′ )
− α log π θ ( a ′ | s ′ )
(cid:19) , a ′ ∼ π θ ( ·| s ′ )
∇ ϕ 1 |B|
X ( s,a,r,s ′ ) ∈B ( Q ϕ i ( s, a ) − y )10:
Update target networks i ← ρ ¯ ϕ i + (1 − ρ ϕ
i
Update gradient using ∇ θ 1 |B|
X s ∈B
N
N X
i =
Q ϕ i ( s, a ) − α log π θ ( a | s ) ! , a ∼ π θ ( ·| s )
R ANDOMIZED E NSEMBLED D OUBLE Q-L EARNING (REDQ)
REDQ (Chen et al., 2021b) is a sample-efﬁcient model-free method for solving maximum-entropy
RL problems (Algorithm 1).
It has two primary components to achieve high sample efﬁciency.
High UTD ratio: It uses a high UTD ratio G , which is the number of updates (lines 4–10) taken by the agent compared to the number of actual interactions with the environment (line 3). The high UTD ratio promotes sufﬁcient training of Q-functions within a few interactions.
However, this also increases an overestimation bias in the Q-function training, which degrades sample-efﬁcient learning (Chen et al., 2021b).
Ensemble of Q-functions: To reduce the overestimation bias, it uses an ensemble of N Qfunctions for the target to be minimized (lines 6–7).
Speciﬁcally, a random subset M of the ensemble is selected (line 6) then used for calculating the target (line 7).
The size of the subset M is kept ﬁxed and is denoted as M .
In addition, each Q-function in the ensemble is randomly and independently initialized but updated with the same target (lines 8–9).
Chen et al.
(2021b) showed that using a large ensemble ( N = 10 ) and small subset ( M = 2 ) successfully reduces the bias.
Although using a large ensemble of Q-functions is beneﬁcial for reducing bias and improving sample efﬁciency, this makes REDQ computationally intensive.
In the next section, we discuss reducing the ensemble size.
I NJECTING M ODEL U NCERTAINTY INTO T ARGET WITH D ROPOUT Q- FUNCTIONS
In this section, we discuss replacing the large ensemble of Q-functions in REDQ with a small ensemble of dropout Q-functions.
We start our discussion by reviewing what the ensemble in REDQ does from the viewpoint of model uncertainty injection.
Then, we propose to use dropout for model uncertainty injection instead of the large ensemble.
Speciﬁcally, we propose (i) the dropout Q-function that is a Q-function equipped with dropout and layer normalization, and (ii) DroQ, a variant of REDQ that uses a small ensemble of dropout Q-functions.
Finally, we explain that the size of the ensemble can be smaller in DroQ than in REDQ.
We ﬁrst explain our insight that, in REDQ, model (Q-function parameters’) uncertainty is injected into the target.
In REDQ, the subset M of N Q-functions is used to compute the target value min i ∈M Q ¯ ϕ
i ( s ′ , a ′ ) (lines 6–7 in Algorithm 1).
This can be interpreted as an approximation for
E ¯ ϕ 1 ,..., ¯ ϕ M
[Equation]
min [Equation] [Equation] (cid:3) , the expected target value with respect to model uncertainty:
[Equation]
used in line
[Equation] Algorithm [Equation] On the left hand side in the second line, the model distribution p (¯ ϕ i ) is replaced with a proposal distribution q en , which is based on resampling from the ensemble in line 6 in Algorithm 1 1 .
On the right hand side (RHS) in the second line, the expected target value is further approximated by one sample average on the basis of q en .
The resulting approximation is used in line 7 in Algorithm 1.
In the remainder of this section, we discuss another means to approximate the expected target value.
We use M dropout Q-functions Q Dr for the target value approximation (Figure 1).
Here, Q Dr is a
Q-function that has dropout connections (Srivastava et al., 2014).
The left part of Figure 1 shows
Q Dr implementation by adding dropout layers to the Q-function implementation used by Chen et al.
(2021b).
Layer normalization (Ba et al., 2016) is applied after dropout for more effectively using dropout as with Vaswani et al.
; Xu et al.
By using Q Dr , the target value is approximated as
[Equation]
Instead of q en , a proposal distribution q dr based on the dropout is used on the RHS in the ﬁrst line.
The expected target value is further approximated by one sample average on the basis of q dr in the second line.
We use the resulting approximation min i =1 ,...,M Q Dr , ¯ ϕ
i ( s ′ , a ′ ) for injecting model uncertainty into the target value (the right part of Figure 1).
For calculating min i =1 ,...,M Q Dr , ¯ ϕ i ( s ′ , a ′ ) , we use
M dropout Q-functions, which have independently initialized and trained parameters ¯ ϕ 1 , ..., ¯ ϕ M .
Using M dropout Q-functions improves the performance of DroQ (our RL method described in the next paragraph), compared with that using a single dropout Q-function (further details are given in
A.3).
We now explain DroQ, in which Q Dr is used for considering model uncertainty.
The algorithmic description of DroQ is shown in Algorithm 2.
DroQ is a variant of REDQ, and the modiﬁed parts y =
[Equation]
+ γ (cid:18) min i =1 ,...,M Q Dr ,
¯ ϕ i ( s ′ , a ′ )
− α log π θ ( a ′ | s ′ )
(cid:19) , a ′ ∼ π θ ( ·| s ′ )
∇ ϕ 1 |B|
X ( s,a,r,s ′ ) ∈B ( Q Dr ,ϕ i ( s, a ) − y ) ∇ θ 1 |B| X [Equation]
[Equation]
from REDQ are highlighted in red in the algorithm.
In line 6, Q Dr is used to inject model uncertainty into the target, as we discussed in the previous paragraph.
In lines 8 and 10, M dropout Q-functions are used instead of N Q-functions ( N ≥ M ) to make DroQ more computationally efﬁcient.
The ensemble size of the dropout Q-functions for DroQ (i.e., M ) should be smaller than that of
Q-functions for REDQ (i.e., N ).
M for DroQ is equal to the subset size for REDQ, which is not greater than N .
In practice, M is much smaller than N (e.g., M = 2 and N = 10 in Chen et al.
(2021b)).
This reduction in the number of Q-functions makes DroQ more computationally efﬁcient.
Speciﬁcally, DroQ is faster due to the reduction in the number of Q-function updates (line 8 in
Algorithm 2).
DroQ also requires less memory for holding Q-function parameters.
In Section 4.2, we show that DroQ is computationally faster and less memory intensive than REDQ.
E XPERIMENTS
We conducted experiments to evaluate and analyse DroQ.
In Section 4.1, we evaluate DroQ’s performance (sample efﬁciency and bias-reduction ability).
In Section 4.2, we evaluate the computational efﬁciency of DroQ. In Section 4.3, we present an ablation study for DroQ.
4. S AMPLE EFFICIENCY AND BIAS - REDUCTION ABILITY OF D RO Q
To evaluate the performances of DroQ, we compared DroQ with three baseline methods in MuJoCo benchmark environments (Todorov et al., 2012; Brockman et al., 2016).
Following Chen et al.
(2021b)
; Janner et al.
(2019), we prepared the following environments: Hopper , Walker2d , Ant , and Humanoid .
In these environments, we compared the following four methods:
REDQ: Baseline method that follows the REDQ algorithm (Chen et al., 2021b)
(Algorithm 1).
SAC: Baseline method that follows the SAC algorithm (Haarnoja et al., 2018a;b).
To improve the sample efﬁciency of this method, delayed policy update and high UTD ratio G
= 20 were used, as suggested by Chen et al.
(2021b).
DroQ: Proposed method that follows Algorithm 2.
Double uncertainty value network (DUVN): Baseline method.
This method is a DroQ variant that injects model uncertainty into a target in the same manner as the original DUVN (Moerland et al., 2017).
The original DUVN uses the output of a single dropout Q-function Q Dr , ¯ ϕ 1 ( s ′ , a ′ ) without
[Table]
layer normalization for target as y = r + γQ Dr , ¯ ϕ 1 ( s ′ , a ′ ) 2 .
We introduce the original DUVN to
DroQ by modifying target on line 6 in Algorithm 2 as y = r
+ γ (cid:0)
[Table]
Here, layer normalization is not applied in Q Dr , ¯ ϕ 1 ( s ′ , a ′ ) .
We use this method to investigate the performance of the existing single dropout Q-function (non-ensemble) method.
Following Chen et al.
(2021b), we set the hyperparameters as G = 20 , N = 10 , and M = 2 for all methods except DUVN ( M = 1 ).
More detailed hyperparameter settings are given in Appendix H.
The methods were compared on the basis of average return and estimation bias.
Average return: An average return over episodes.
We regarded 1000 environment steps in Hopper and 3000 environment steps in the other environments as one epoch, respectively.
After every epoch, we ran ten test episodes with the current policy and recorded the average return.
Average/std. bias: Average and standard deviation of the normalized estimation error (bias) of
Q-functions Q ϕ i (Chen et al., 2021b).
The error represents how signiﬁcantly the Q-value estimate differs from the true one.
Formally, the error is deﬁned as | Q π ( s, a ) − ˆ Q ( s, a ) | / E ¯ s, ¯ a ∼ π [ Q π (¯ s, ¯ a )] , where Q π ( s, a ) is the true Q-value under the current policy π and ˆ Q ( s, a ) is its estimate.
In our experiment, Q π ( s, a ) was approximated by the discounted Monte Carlo return obtained with π in the test episodes.
In addition, ˆ Q ( s, a ) was evaluated as ˆ Q ( s, a ) = 1 /N P Ni =1 Q ϕ
i ( s, a ) for SAC and REDQ and as ˆ Q ( s, a ) = 1 /M
P Mi =1 Q Dr ,ϕ i ( s, a ) for DroQ and DUVN, respectively.
The comparison results (Figure 2) indicate that DroQ achieved almost the same level of performance as REDQ.
Regarding the average return, REDQ and DroQ achieved almost the same sample efﬁciency overall.
In Walker2d and Ant, their learning curves highly overlapped, and there was no signiﬁcant difference between them.
In Humanoid, REDQ was slightly better than DroQ. In Hopper, DroQ was better than REDQ.
In all environments except Hopper, REDQ and DroQ improved their average return signiﬁcantly earlier than SAC and DUVN.
Regarding the bias, REDQ and DroQ consistently kept the value-estimation bias closer to zero than SAC and DUVN in all environments.
4. C OMPUTATIONAL EFFICIENCY OF D RO Q
We next evaluated the computational efﬁciency of DroQ. We compared DroQ with the baseline methods on the basis of the following criteria: (i) Process time required for executing methods; (ii) Number of parameters of each method; (iii) Bottleneck memory consumption suggested from the Pytorch proﬁler 3 .
Bottleneck memory consumption is the maximum memory consumption recorded when running the methods.
For evaluation, we ran each method on a machine equipped with two Intel(R) Xeon(R) CPU E5-2667 v4 and one NVIDIA Tesla K80.
Process times per update (numbers in parentheses in Table 1) indicate that DroQ runs more than two times faster than REDQ.
DroQ (and SAC) requires process times in the 800–900-msec range.
REDQ requires process times in the 2200–2300-msec range.
Process times also show that learning
Q-functions is dominant in an overall loop.
This suggests that using compact (e.g., small numbers of) Q-functions is important for improving overall process times.
The number of parameters and bottleneck memory consumption of each method indicate that DroQ is more memory efﬁcient than REDQ.
Regarding the numbers of parameters (Table 2), we can see that those of DroQ (and SAC and DUVN) are about one-ﬁfth those of REDQ.
Note that the number of parameters of DroQ is equal to that of SAC since DroQ and SAC use the same number (two) of Q-functions.
Regarding the bottleneck memory consumption (Table 3), we can see that that for DroQ (SAC and DUVN) is about one-third that for REDQ.
We can also see that the bottleneck [Table] memory consumption is almost independent of the environment.
This is because one of the most memory-intensive parts is the ReLU activation at the hidden layers in Q-functions 4 .
[Table]
4. A BLATION STUDY
As an ablation study of DroQ, we investigated the performance of DroQ variants that do not use either dropout, or layer normalization, or both.
We refer to the one without dropout as -DO , that without layer normalization as -LN , and that without both as -DO-LN .
The results (Figure 3) indicate that the synergic effect of using dropout and layer normalization is high, especially in complex environments (Ant and Humanoid).
In these environments, DroQ signiﬁcantly outperformed its ablation variants (-DO, -LN and -DO-LN) in terms of both average return and bias reduction.
[Table]
R ELATED W ORK
In this section, we review related studies and compare them with ours (Table 4).
Ensemble Q-functions: Ensembles of Q-functions have been used in RL to consider model uncertainty (Faußer & Schwenker, 2015;
Osband et al., 2016; Anschel et al., 2017; Osband et al., 2018;
Agarwal et al., 2020; Lee et al., 2021;
Lan et al., 2020; Chen et al., 2021b).
Ensemble transition models: Ensembles of transition (and reward) models have been introduced to model-based RL, e.g., (Chua et al., 2018;
Kurutach et al., 2018; Janner et al., 2019;
Shen et al., 2020; Yu et al., 2020;
Lee et al., 2020; Hiraoka et al., 2021; Abraham et al., 2020).
The methods proposed in the above studies use a large ensemble of Q-functions or transition models, thus are computationally intensive.
DroQ does not use a large ensemble of Q-functions, and thus is computationally lighter.
Dropout transition models: Gal et al.
(2016; 2017); Kahn et al.
(2017) introduced dropout and its modiﬁed variant to transition models of model-based RL methods.
Dropout Q-functions: Gal & Ghahramani (2016) introduced dropout to a Q-function at action selection for considering model uncertainty in exploration.
Harrigan (2016); Moerland et al.
; Jaques et al.
; He et al.
(2021) introduced dropout to policy evaluation in the same vein as us.
However, there are three main differences between these studies and ours.
(i) Target calculation: they introduced dropout to a single Q-function and used it for a target value, whereas we introduce dropout to multiple Q-functions and use the minimum of their outputs for the target value.
(ii) Use of engineering to stabilize learning: their methods do not use engineering to stabilize the learning of dropout Q-functions, whereas DroQ uses layer normalization to stabilize the learning.
(iii) RL setting: they focused on a low UTD ratio setting, whereas we focused on a high UTD ratio setting.
A high UTD ratio setting increases a high estimation bias, and thus is a more challenging setting than a low UTD ratio setting.
In Sections 4. and A.3, we showed that their methods do not perform successfully in a high UTD setting.
Normalization in RL: Normalization (e.g., batch normalization (Ioffe & Szegedy, 2015) or layer normalization (Ba et al., 2016)) has been introduced into RL.
Batch normalization and its variant are introduced in deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015) and twin delayed
DDPG (Fujimoto et al., 2018) (Bhatt et al., 2020).
Layer normalization is introduced in the implementation of maximum a posteriori policy optimisation (Abdolmaleki et al., 2018; Hoffman et al., 2020).
It is also introduced in SAC extensions (Ma et al., 2020; Zhang et al., 2021).
Unlike our study, the above studies did not introduce dropout to consider model uncertainty.
In Section 4.3, we showed that using layer normalization alone is not sufﬁcient in high UTD settings.
In Appendix E, we also show that using batch normalization does not contribute to performance improvement in high UTD settings.
C ONCLUSION
We proposed, DroQ, an RL method based on a small ensemble of Q-functions that are equipped with dropout connection and layer normalization.
We experimentally demonstrated that DroQ signiﬁcantly improves computational and memory efﬁciencies over REDQ while achieving sample efﬁciency comparable with REDQ.
In the ablation study, we found that using both dropout and layer normalization had a synergistic effect on improving sample efﬁciency, especially in complex environments (e.g., Humanoid).
E FFECT OF DROPOUT RATE ON D RO Q AND ITS VARIANTS
[Figure]
[Figure]
A. S IN -D RO Q: D RO Q VARIANT USING A SINGLE DROPOUT Q- FUNCTION
DroQ (Algorithm 2) uses an ensemble of multiple ( M ) dropout Q-functions.
This raises the question: “Why not use a single dropout Q-function for DroQ?”
To answer this question, we compared DroQ with Sin-DroQ , a variant of DroQ that uses a single dropout Q-function.
Speciﬁcally, with SinDroQ, the target in line 6 in Algorithm 2 is calculated by evaluating Q Dr ¯ ϕ 1 ( s ′ , a ′ ) , a single dropout
Q-function, M times:
[Equation]
Note that, the output of Q Dr , ¯ ϕ 1 ( s ′ , a ′ ) can differ in each evaluation due to the use of a dropout connection.
It should be also noted that this target is the same as the one proposed in Jaques et al.
The remaining part of Sin-DroQ is the same as DroQ.
From the comparison results of DroQ and Sin-DroQ (Figure 6), we can see that the average return of
Sin-DroQ was lower than that of DroQ.
This result indicates that using multiple dropout Q-functions is preferable to using a single dropout Q-function with DroQ.
B REDQ WITH DIFFERENT ENSEMBLE SIZE N
In Section 4, we discussed comparing DroQ with REDQ, which uses an ensemble size of 10 (i.e.,
N =10).
To make a more detailed comparison, we compared DroQ and REDQ by varying the ensemble size for REDQ.
We denote REDQ that uses an ensemble size of N as “REDQ N ” (e.g., REDQ for REDQ with an ensemble size of ﬁve).
Regarding average return (left part of Figure 7), overall, DroQ was superior to REDQ2–5.
DroQ was comparable with REDQ2 and REDQ3 in Hopper but superior in more complex environments (Walker, Ant, and Humanoid).
In addition, DroQ was somewhat better than REDQ5 in all environments.
Regarding estimation bias (middle and right parts of Figure 7), overall, DroQ was signiﬁcantly better than REDQ2 and comparable with REDQ3–10.
Regarding the processing speed (Table 5), DroQ ran as fast as REDQ3 and faster than REDQ5 by 1.4 to 1.5 times.
Regarding memory efﬁciency (Tables 6 and 7), DroQ was less memory intensive than REDQ3–10.
[Table] [Table]
[Table]
[Table] [Table]
[Table]
REDQ 94 / 64 /94 / 64 /94 / 64 /94 / 65 /
REDQ 136 / 106 /136 / 106 /136 / 106 /136 / 107 /
REDQ 241 / 211 /241 / 211 /241 / 211 /241 / 212 /
C A DDITIONAL ABLATION STUDY OF D RO Q
Dropout is introduced into three parts of the algorithm for DroQ (i.e., lines 6, 8 and 10 of Algorithmin Section 3).
In this section, we conducted an ablation study to answer the question “which dropout introduction contributes to overall performance improvement of DroQ?”
We compared DroQ with its following variants:
-DO@TargetQ:
A DroQ variant that does not use dropout in line 6.
Speciﬁcally
dropout is not used in Q Dr , ¯ ϕ i ( s ′ , a ′ ) in the following part in line 6.
[Equation]
-DO@CurrentQ:
A DroQ variant that does not use dropout in line 8.
Speciﬁcally, dropout is not used in Q Dr ,ϕ i ( s, a ) in the following part in line 8.
[Equation]
-DO@PolicyOpt:
A DroQ variant that does not use dropout in line 10.
Speciﬁcally, dropout is not used in Q Dr ,ϕ i ( s, a ) in the following part in line 10.
[Equation]
-DO:
A DroQ variant that does not use dropout in lines 6, 8, and 10.
In this ablation study, we compared the methods on the basis of average return and estimation bias.
The comparison results (Figure 8) indicate that using dropout for both current and target Q-functions (i.e., Q Dr , ¯ ϕ
i ( s ′ , a ′ ) and Q Dr ,ϕ i ( s, a ) in lines 6 and 8) is effective.
Regarding average return, the DroQ variants that do not use dropout for either target Q-functions in line 6 or current Q-functions in line 8 perform signiﬁcantly worse than that of DroQ. For example, in Ant, the variants that do not use dropout for target Q-functions (-DO@TargetQ and -DO) perform much worse than DroQ.
In addition, in Humanoid, the variants that do not use dropout for current Q-functions (-DO@CurrentQ and -DO) perform much worse than DroQ.
Regarding estimation bias, that of the variants that do not use dropout for target Q-functions (-DO@TargetQ and -DO) is signiﬁcantly worse than that of DroQ in all environments.
These results complement the results presented in He et al.
They show that using dropout for both current and target Q-functions improves the performance of the maximum entropy RL method (SAC) in low UTD settings.
Our results presented in the previous paragraph implies that their insight (i.e., “using dropout for both current and target Q-functions improves the performance”) is valid, even in high UTD settings. Q-function loss is calculated as 1 M P M i = 1 L i , where L i :=
[Equation] [Equation] [Equation] D W HY IS THE COMBINATION OF DROPOUT AND LAYER NORMALIZATION IMPORTANT ?
Figure 3 in Section 4.3 showed that combining dropout and layer normalization improves overall performance, especially in complex environments (Ant and Humanoid).
The main reason for this performance improvement could be that layer normalization suppresses the learning instability caused by dropout.
Using dropout destabilizes the Q-functions learning.
Figure 9 shows that (i) the Q-function loss of the method using dropout (-LN) oscillates more signiﬁcantly than that of the method not using dropout (-DO-LN) and that (ii) the variance of the gradient loss with respect to the Q-function parameters also oscillates accordingly.
Layer normalization suppresses such learning instabilities.
The right part of Figure 9 shows that the method using both dropout and layer normalization (DroQ) suppresses the oscillations of the gradient more signiﬁcantly than the -LN.
This stabilization of learning could enable better Q-value propagation and consequently improve overall performance.
E E FFECTS OF OTHER NORMALIZATION METHODS
In Section 4.3 and Appendix D, we demonstrated that combining dropout and layer normalization is effective in improving performance.
In this section, we examine the effectiveness of normalization methods other than layer normalization.
We examine the effects of (i) batch normalization (Ioffe & Szegedy, 2015), (ii) group normalization (Wu & He, 2018), and (iii) layer normalization without variance re-scaling.
Batch normalization is an off-the-shelf and popular normalization method, and has been introduced into RL in low
UTD settings (Bhatt et al., 2020).
We introduce this popular method to verify whether it is effective in high UTD settings.
Group normalization is a method of normalization using the average and variance over the elements of a target input, which is similar to layer normalization 5 .
We introduce this method to verify whether a method equipped with a similar mechanism to layer normalization works well in high UTD settings.
Layer normalization without variance re-scaling is the layer normalization variant that does not re-scale the target input’s variance.
Xu et al.
(2019) pointed out that variance re-scaling is an important component for layer normalization to work well.
We introduce this method to verify whether variance re-scaling is an important component also in our settings.
We compare six DroQ variants that use these three normalization methods.
We denote -DO 6 variants using batch normalization, group normalization, and layer normalization without variance rescaling, instead of layer normalization, as +BN, +GN, and +LNwoVR, respectively.
We also denote
DroQ variants using batch normalization, group normalization, and layer normalization without variance re-scaling, instead of layer normalization, as +DO+BN, +DO+GN, and +DO+LNwoVR, respectively.
We compare these six methods (+BN, +GN, +LNwoVR, +DO+BN, +DO+GN, and +DO+LNwoVR).
From the experimental results of the six methods (Figures 10 and 11), we get the following insights:
Batch normalization does not work well.
Figure 10 shows that the methods using batch normalization (+BN and +DO+BN) do not signiﬁcantly improve the average return and estimation bias in Humanoid and Ant.
Further, Figure 11 shows that the Q-function learning with these methods is very unstable.
Group normalization, which has a similar mechanism to layer normalization, works well when combined with dropout.
Figure 10 shows that the method using the group normalization and dropout (+DO+GN) successfully improves the average return and estimation bias in Humanoid and Ant.
The ﬁgure also shows that the method that uses only group normalization (+GN) fails in improving the average return and estimation bias.
Layer normalization without variance re-scaling does not work well.
Figure 10 shows that the methods that introduce this layer normalization variant (+LNwoVR and +DO+LNwoVR) do not signiﬁcantly improve the average return and estimation bias well especially in Humanoid.
Further, Figure 11 shows that the Q-function learning is unstable compared to +DO+GN (and DroQ in Figure 9) in both Ant and Humanoid.
This result implies that one of the primal factors for the synergistic effect of dropout and layer normalization is the variance re-scaling.
1: Initialize policy parameters θ , N Q-function parameters ϕ
i , i = 1 , ..., N , and empty replay buffer D .
Set target parameters ¯ ϕ i ← ϕ i , for i = 1 , ..., N .
2: repeat 3:
Take action a t ∼ π θ ( ·| s t ) .
Observe reward r t , next state s t +1 ; D ← D S ( s t , a t , r t , s t +1 ) .
4: for G updates do 5:
Sample a mini-batch B = { ( s, a, r, s ′ ) } from D .
Sample a set M of M distinct indices from { 1 , 2 , ..., N } .
Compute the Q target y for the dropout Q-functions:
y = r
+ γ (cid:18) min i ∈M Q Dr , ¯ ϕ i ( s ′ , a ′ )
− α log π θ ( a ′ | s ′ )
(cid:19) , a ′ ∼ π θ ( ·| s ′ )
8: for i = 1 , ..., N do 9:
Update ϕ
i with gradient descent using ∇ ϕ 1 |B|
X ( s,a,r,s ′ ) ∈B ( Q Dr ,ϕ i ( s, a ) − y )10:
Update target networks with ¯ ϕ
i ← ρ ¯ ϕ i
+ (1 − ρ ) ϕ i .
Update θ with gradient ascent using [Equation] | [Equation]
X [Equation]
N
N X
i =
[Equation] [Equation] [Equation] [Equation]
F R ELATION BETWEEN (1) ENSEMBLE SIZE AND (2) THE EFFECT OF LAYER NORMALIZATION AND DROPOUT ON OVERALL PERFORMANCE
In Section 4.3 and Appendix D, we showed that using layer normalization and dropout improves overall performance.
Further, in Sections 4.1 and A.3, we showed that using multiple (two) dropout
Q-functions improves overall performance.
These results raise two new questions: (i) what happens if we use more than three dropout Q-functions?
and (ii) what the effect of layer normalization and dropout would be in such cases?
In this section, we conduct an additional experiment to answer questions (i) and (ii).
For our additional experiment, we introduce the DroQ with ensemble of N dropout Q-functions (DroQ N ) algorithm
(Algorithm 3).
This algorithm is a variant of DroQ algorithm (Algorithm 2) which uses bootstrapping of a subset M of N dropout Q-functions (line 6).
This algorithm can also be regarded as a variant of REDQ algorithm
(Algorithm 1) which uses N dropout Q-functions.
With
DroQ N algorithm, the ensemble size can be changed by varying the value of N .
We compare the method based on the DroQ N algorithm and its ablation variants:
DroQ N -LN: the DroQ N variant that does not use layer normalization for the dropout Q-function Q Dr ,ϕ i .
DroQ N -DO-LN: the DroQ N variant that does not use layer normalization and dropout for Q Dr ,ϕ i .
Note that DroQ N -DO-LN is identical with REDQ N presented in Appendix B.
We evaluate the performance of these methods for the case of N = 2 , 3 , 5 , 10 .
Learning curves of the methods are shown in Figure 12.
For the question (i), in complex environments (Ant and Humanoid), increasing the size of the ensemble improves the sample efﬁciency.
In Ant, DroQ5–10 are more sample efﬁcient than DroQ2–3.
In Humanoid, the sample efﬁciency is monotonically improved when the ensemble size of DroQ N is increased as N = 2 , 3 , 5 , 10 .
For the question (ii), in the complex environments, the synergy of layer normalization and dropout is signiﬁcant especially when the value of N is small ( N = 2 , 3 ).
In Ant and Humanoid, DroQ2– 3 signiﬁcantly improves the sample efﬁciency compared to DroQ2–3-DO.
On the other hand, the effect of layer normalization itself becomes dominant when the value of N is large ( N ≥ 5 ).
For
Ant and Humanoid, we can see that DroQ5–10-DO signiﬁcantly improves the sample efﬁciency compared to DroQ5–10-DO-LN.
However, DroQ5–10 does not signiﬁcantly improve the sample efﬁciency compared to DroQ5–10-DO.
Based on the above result, we suggest (1) using layer normalization and dropout together for small ensemble cases, and (2) using layer normalization alone for large ensemble cases.
G E FFECT OF DROPOUT Q- FUNCTIONS ON SAC
SAC+DO is a SAC variant using dropout, SAC+LN is a SAC variant using layer normalization, and SAC+DO+LN is a SAC variant using dropout and layer normalization.
Note that the only difference between DroQ ( M = 2 ) and SAC+DO+LN is the policy update part (line 10 in Algorithm 2).
DroQ ( M = 2 ) updates policy as ∇ θ [Equation] B | P
[Equation]
[Equation]
i = [Equation] Q [Equation]
[Equation]
( s, a )
[Equation] log [Equation] a | s ) (cid:17) .
On the other hand, SAC+DO+LN updates policy as ∇ θ 1 | B |
P [Equation]
H H YPERPARAMETER SETTINGS
The hyperparameter settings for each method in the experiments discussed in Section 4 are listed in
Table 8.
Parameter values, except for (i) dropout rate for DroQ and DUVN and (ii) M for DUVN, were set according to Chen et al.
(2021b).
The dropout rate (i) was set through line search, and M for DUVN (ii) was set according to Harrigan (2016); Moerland et al.
[Table]
I E XPERIMENTS ON THE ORIGINAL REDQ CODEBASE
Experiments presented in the main content of this paper have been implemented on top of the
SAC codebase ( https://github.com/ku2482/soft-actor-critic.pytorch ).
In this section, we report the results of our experiments on the original REDQ codebase ( https: //github.com/watchernyu/REDQ ).
Our experiments are to replicate experiments in the main content of this paper.
To evaluate the computational efﬁciency, we plot the wallclock time required to complete a certain number of interactions with the environment, instead of the standard deviation of bias.
In addition, dropout rate values are re-tuned for each environment (Table 9).
Figure 14 shows the replica of the results presented in Section 4.1.
Figure 15 shows the replica of the results provided in Section 4.3.
Figure 16 shows the replica of the results provided in Section A.1.
Figure 17 shows the replica of the results provided in Section D. Figures 18 and 19 show the replica of the results provided in Section E. Figure 20 shows the replica of the results provided in Section F.
[Table]
Q-function loss is calculated as
[Equation]
L i , where L i :=
[Equation]
and its gradient is calculated as
[Equation]
J O UR SOURCE CODE
Our source code is available at https://github.com/TakuyaHiraoka/
Dropout-Q-Functions-for-Doubly-Efficient-Reinforcement-Learning