Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
Abstract
Recent advances in model-free deep reinforcement learning (DRL) show that simple model-free methods can be highly effective in challenging high-dimensional continuous control tasks.
In particular, Truncated Quantile Critics (TQC) achieves state-of-the-art asymptotic training performance on the MuJoCo benchmark with a distributional representation of critics; and Randomized Ensemble Double QLearning (REDQ) achieves high sample efficiency that is competitive with stateof-the-art model-based methods using a high update-to-data ratio and target randomization.
In this paper, we propose a novel model-free algorithm, Aggressive
Q-Learning with Ensembles (AQE), which improves the sample-efficiency performance of REDQ and the asymptotic performance of TQC, thereby providing overall state-of-the-art performance during all stages of training.
Moreover, AQE is very simple, requiring neither distributional representation of critics nor target randomization.
The effectiveness of AQE is further supported by our extensive experiments, ablations, and theoretical results.
Introduction
Off-policy Deep Reinforcement Learning algorithms aim to improve sample efficiency by reusing past experience.
A number of off-policy Deep RL algorithms have been proposed for control tasks with continuous state and action spaces, including Deep Deterministic Policy Gradient (DDPG),
Twin Delayed DDPG (TD3) and Soft Actor Critic (SAC) (Lillicrap et al., 2016; Fujimoto et al., 2018; Haarnoja et al., 2018a,b).
TD3 introduced clipped double-Q learning, and was shown to be significantly more sample efficient than popular on-policy methods for a wide range of MuJoCo benchmarks.
Soft Actor Critic (SAC) has similar off-policy structures with clipped double-Q learning, but it also employs maximum entropy reinforcement learning.
SAC was shown to provide excellent sample efficiency and asymptotic performance in a wide-range of MuJoCo environments, including the high-dimensional Humanoid environment for which both DDPG and TD3 perform poorly.
More recently, Kuznetsov et al.
proposed Truncated Quantile Critics (TQC), a model-free algorithm which includes distributional representations of critics, truncation of critics prediction, and ensembling of multiple critics.
Instead of the usual modeling of the Q-function of the expectation of return, TQC approximates the distribution of the return random variable conditioned on the state and action.
By dropping several of the top-most “atoms” and varying the number of dropped atoms of the return distribution approximation, TQC can control the over-estimation bias.
TQC’s asymptotic performance (that is after a long period of training) was shown to be better than that of SAC on the continuous control MuJoCo benchmark suite, including a 25% improvement on the most challenging
Humanoid environment.
However, TQC is not sample efficient in that it generally requires a large number of samples to reach even moderate performance levels.
Chen et al.
(2021) proposed Randomized Ensembled Double Q-learning(REDQ), a model-free algorithm which includes a high Update-To-Data (UTD) ratio, an ensemble of Q functions, and in-target minimization across a random subset of Q functions from the ensemble.
Using a UTD ratio much larger than one, meaning that several gradient steps are taken for each environment interaction, improves sample efficiency, while the ensemble and in-target minimization allows the algorithm to maintain stable and near-uniform bias under the high UTD ratio.
The algorithm was shown to attain much better performance than SAC at the early stage of training, and to match or improve the sampleefficiency of the state-of-the-art model-based algorithms for the MuJoCo benchmarks.
However, although REDQ is highly sample efficient for early-stage training, its asymptotic performance is significantly below that of TQC.
Is it possible to design a simple, streamlined model-free algorithm which can achieve REDQ’s high sample efficiency in early-stage training and also achieve TQC’s high asymptotic performance in late stage training?
In this paper, we achieve this goal with a new model-free algorithm, Aggressive
Q-Learning with Ensembles (AQE).
Like TQC and REDQ, AQE uses an ensemble of Q-functions, and like REDQ it uses a UTD ratio > 1 .
However AQE is very simple, requiring neither distributional representation of critics as in TQC nor target randomization and double-Q learning as in REDQ.
AQE controls overestimation bias and the standard deviation of the bias by varying the number of ensemble members
N and the number of ensembles K ≤ N that are kept when calculating the targets
Through carefully designed experiments, we provide a detailed analysis of AQE.
We perform extensive and comprehensive experiments for both MuJoCo and DeepMind Control Suite (DMC) environments.
We first show that for the five most challenging MuJoCo benchmark, AQE provides state-of-the-art performance, surpassing the performance of SAC, REDQ, and TQC at all stages of training.
When averaged across the five MuJoCo environments, AQE’s early stage performance is 2.9 times better than SAC, 1.6 times better than TQC and 1.1 times better than REDQ.
AQE’s asymptotic performance is 26%, 22%, and 6% higher than SAC, REDQ and TQC, respectively
Then we provide additional experimental results for the nine most challenging DeepMind Control
Suite (DMC) environments, which TQC and REDQ did not consider.
We show that AQE also provides state-of-the-art performance at both early stage and late-stage of training.
When averaged over nine environments, AQE’s early stage performance is 13.71 times better than SAC, 7.59 times better than TQC and 1.02 times better than REDQ.
AQE’s asymptotic performance is 37% better than SAC, 3% better than REDQ, and 8% better than TQC.
We also perform an ablation study, and show that AQE is robust to choices of hyperparameters: AQE can work well with small ensembles consisting of 10-20 ensemble members, and performance does not vary significantly with small changes in the keep parameter K .
We show that that AQE performs better than several variations, including using the median of all ensemble members and removing the most extreme minimum and maximum outlier in the targets.
In order to improve computational time, we also consider different multi-head architectures for the ensemble of critics: consistent with the supervised convolutional network literature, we find that a two-head architecture not only reduces computational time but can actually improve performance for some environments.
Additionally, we show that AQE continues to out-perform SAC and TQC even when these algorithms are made aggressive with a UTD ≫
To ensure a fair comparison and to obtain reliable and reproducible results (Henderson et al., 2018;
Islam et al., 2017; Duan et al., 2016), we provide open source code 2 .
For all algorithmic comparisons, we use the the authors’ code.
Additional Related Work
Overestimation bias due to in target maximization in Q-learning can significantly slow down learning (Thrun and Schwartz, 1993).
For tabular Q-learning, van Hasselt (2010) introduced Double Q- with-
Learning, and showed that it removes the overestimation basis and in general leads to an underestimation bias.
Hasselt et al.
(2016) showed that adding Double Q-learning to deep-Q networks can have a similar effect, leading to a major performance boost for the Atari games benchmark.
As stated in the Introduction, for continuous-action spaces, TD3 and SAC address the overestimation bias using clipped-double Q-learning, which brings significant performance improvements (Fujimoto et al., 2018; Haarnoja et al., 2018a,b).
As mentioned in the Introduction, Kuznetsov et al.
(2020) control the over-estimation bias by estimating the distribution of the return random variable, and then by dropping several of the topmost “atoms” from the estimated distribution.
The distribution estimate is based on a methodology developed in Bellemare et al.
; Dabney et al.
(2018a,b), which employs an asymmetric Huber loss function to minimize the Wasserstein distance between the neural network output distribution and the target distribution.
In this paper, in order to counter over-estimation bias, we also drop the top-most estimates, although we do so solely with an ensemble of Q-function mean estimators rather than with an ensemble of the more complex distributional models employed in (Kuznetsov et al., 2020).
It is well-known that using ensembles can improve the performance of DRL algorithms (Faußer and
Schwenker, 2015; Osband et al., 2016; Lee et al., 2021).
For Q-learning based methods, Anschel et al.
(2017) use the average of multiple Q estimates to reduce variance.
Lan et al.
(2020) introduced
Maxmin Q-learning, which uses the minimum of all the Q-estimates rather than the average.
Agarwal et al.
(2020) use Random Ensemble Mixture (REM), which employs a random convex combination of multiple Q estimates.
Model-based methods often attain high sample efficiency by using a high UTD ratio.
In particular,
Model-Based Policy Optimization (MBPO)
(Janner et al., 2019) uses a large UTD ratio of 20-40.
Compared to Soft-Actor-Critic (SAC), which is model-free and uses a UTD of 1, MBPO achieves much higher sample efficiency in the OpenAI MuJoCo benchmark (Todorov et al., 2012; Brockman et al., 2016).
REDQ (Chen et al., 2021), a model-free algorithm, also successfully employs a high
UTD ratio to achieve high sample efficiency.
Algorithm
We propose Aggressive Q-learning with Ensembles (AQE), a simple model-free algorithm which provides state-of-the-art performance for the MuJoCo benchmark for both early and late stage of training.
The pseudocode can be found in Algorithm 1.
As is the case with most off-policy continuous-control algorithms, AQE has a single actor (policy network) and multiple critics (Qfunction networks), and employs Polyak averaging of the target parameters to enhance stability.
Building on this algorithmic base, it also employs an update-to-data ratio G > 1 , an ensemble of
N ≥ 3 Q-functions (rather than just two as in TD3 and SAC), and targets that average all the Qfunctions excluding the Q-functions with the highest N − K values.
In the Appendix, we demonstrate theoretically in the tabular case of the algorithm that we can control over-estimation through adjusting
K and N .
More concretely, we can bring the bias term from above zero (i.e. overestimation) to under zero (i.e. underestimation) by decreasing K and/or increasing N .
For comparison, REDQ employs two randomly chosen ensemble members when calculating the target, the bias does not depend on the number of ensemble models N (Chen et al., 2021).
As discussed in Appendix, with M = 2 fixed, increasing the size of ensemble N with the multi-head architecture does not necessarily improve the performance of REDQ.
Unlike REDQ, AQE can control the bias term through both the number of ensemble models used in the average calculation K and the total number of ensembles N , allowing for more flexibility.
One other drawback for REDQ is that it ignores the estimates of all other ensemble estimates except for the minimal one in the randomly chosen set, which diminishes the power of the multiple ensemble sets.
In contrast, AQE utilizes most of the ensemble models when calculating the target.
The resulting algorithm is not only simple and streamlined, but also provides state-of-the art performance.
For exploration, it uses entropy maximization as in SAC, although it could easily incorporate alternative exploration schemes.
AQE has three key hyperparameters, G , N , and K .
If we set N = 2 , K = 1 and G = 1 , AQE is simply the underlying off-policy algorithm such as SAC.
When N > 2 , K = 1 and G = 1 , then
AQE becomes similar to, but not equivalent to, Maxmin Q-learning (Lan et al., 2020).
Initialize: Initial policy parameters θ , N Q-function parameters ϕ
i , i = 1 , ... , N , empty replay buffer D .
Set target parameters ϕ targ
,i ← ϕ
i for i = 1 , 2 , ... , N.
1: repeat 2:
Take one action a t ∼ π θ ( ·| s t ) .
Observe reward r t , new state s t +1 .
Add data to replay buffer: D ← D ∪ { ( s t , a t , r t , s t +1 ) } 4: for G updates do 5:
Randomly sample a mini-batch B = { ( s, a, r, s ′ ) } from D .
6: for each ( s, a, r, s ′ )
∈ B do 7:
Sample ˜ a ′ ∼ π θ ( ·| s ′ ) .
Determine the K indices from i = 1 , . .
, N that minimize Q target ,i ( s ′ , ˜
a ′ ) .
Compute the Q target y : 10: y ( s, a ) =
r + γ (cid:18)
K (cid:88) i ∈ K
Q ϕ targ ,i ( s ′ , ˜ a ′ )
− α log π θ
(˜ a ′ | s ′ )
(cid:19) 11: for i = 1 , . .
. , N do 12:
Update ϕ
i with gradient descent using 13: ∇ ϕ i 1 | B | (cid:88) ( s,a,r,s ′ )
∈ B ( Q ϕ i ( s, a )
− y ( s, a ))14:
Update target networks with ϕ targ ,i ← ρϕ targ ,i + (1 − ρ ) ϕ i 15:
Update policy parameters θ with gradient ascent using 16: ∇ θ 1 | B | (cid:88) s ∈ B (cid:18)
N
N (cid:88) i =
Q ϕ i ( s, ˜ a θ ( s )) − α log π θ (˜ a θ ( s ) | s ) (cid:19)
˜ a θ ( s ) ∼ π θ ( ·
[Table]
AQE uses an ensemble of Q networks (as does REDQ and TQC).
Employing multiple networks, one for each Q-function output, can be expensive in terms of computation and memory.
In order to reduce the computation and memory requirements, we combine network ensemble with multi-head architectures to generate multiple Q-function outputs.
We consider N separate Q networks each with h heads, providing a total of h · N estimates.
The h heads from one network share all of the layers except the final fully-connected layer.
In practice, we found h = 2 heads works well for AQE, consistent with work in ensembles of convolutional neural networks for computer vision tasks (Lee et al., 2015).
When properly sharing low-level weights, multi-headed networks may not only retain the performance of full ensembles but can sometimes outperform them.
In the next section, we discuss our experimental results.
Experimental Results
We perform extensive and comprehensive experiments for two sets of popular benchmarks.
First we provide experimental results for AQE, TQC, REDQ and SAC for the five most challenging
MuJoCo environments, namely, Hopper, Walker2d, HalfCheetah, Ant and Humanoid.
Then we provide additional experimental results for the nine most challenging DeepMind Control Suite (DMC) environments, namely, Cheetah-run, Fish-swim, Hopper-hop, Humanoid-stand, Humanoid-walk,
Humanoid-run, Quadruped-walk, Quadruped-run and Walker-run.
To make a fair comparison, the
TQC, REDQ and SAC results are reproduced using the authors’ open source code, and use the same network sizes and hyperparameters reported in their papers.
In particular, for the MuJoCo environments, TQC employs 5 critic networks with 25 distributional samples for a total of 125 atoms.
TQC drops 5 atoms per critic for Hopper, 0 atoms per critic for Half Cheetah, and 2 atoms per critic for Walker, Ant, and Humanoid.
For REDQ, we also use the authors’ suggested values of N =and M = 2, where M is the number of ensemble members used in the target calculation.
The REDQ paper uses G = 20 for the update-to-data ratio, and provides results for up to 300K environment interactions.
Using such a high value for G is computationally infeasible in our experimental setting, since we use 3 million environment interactions for Ant and Humanoid and over 4 million environment interactions for Humanoid-run in order to investigate asymptotic performance as well early-stage sample efficiency.
In the experiments reported here, we use a value of G = 5 for both REDQ and AQE.
For AQE, we use 10 Q-networks each with 2 heads, producing 20 Q-values for each input.
The AQE networks are the same size as those in the REDQ paper.
For MuJoCo benchmark, AQE keeps 10 out of 20 values for Hopper, all 20 values for half-Cheetah, and 16 out of 20 values for Walker, Ant and
Humanoid.
[Figure]
Figure 1 shows the training curves for AQE, TQC, REDQ, and SAC for the five MuJoCo environments.
For each algorithm, we plot the average return of 5 independent trials as the solid curve, and plot the standard deviation across 5 seeds as the shaded region.
For each environment, we train each algorithm for exactly the same number of environment interactions as done in the SAC paper.
We use the same evaluation protocol as in the TQC paper.
Specifically, after every epoch, we run ten test episodes with the current policy, record the undiscounted sum of all the rewards in each episode and take the average of the sums as the performance.
A more detailed discussion on hyperparameters and implementation details is given in the Appendix.
We see from Figure 1 that AQE is the only algorithm that beats SAC in all five environments during all stages of training.
Moreover, it typically beats SAC by a very wide margin.
Table 1 shows that, when averaged across the five environments, AQE achieves SAC asymptotic performance approximately 3x faster than SAC and 2x faster than REDQ and TQC.
As seen from Figure 1 and Table 2, in the early stages of training, AQE matches the excellent performance of REDQ in all five environments, and both algorithms are much more sample efficient than SAC and TQC.
As seen from Figureand Table 3, in late-stage training, AQE always matches or beats all other algorithms, except for
Humanoid, where TQC is about 10% better.
Table 3 shows that, when averaged across all five environments, AQE’s asymptotic performance is 26%, 22%, and 6% higher than SAC, REDQ, and
TQC, respectively.
4.1 Fixed Hyperparameters across Environments
Following the TQC paper, in Figure 1 we used different drop atoms for TQC for the different MuJoCo environments.
To make the comparison fair, we also used different keep values K for AQE for the different environments.
We repeat the experiment on the five MuJoCo environments, but now use the same hyperparameter values across environments for TQC (drop two atoms per network) and AQE ( K = 16).
These choices of fixed hyperparameters appear to give the best overall performance for the
[Table] [Table]
two algorithms.
The training curves for AQE, TQC, REDQ, and SAC and detailed early-stage and late-stage performance comparisons of all algorithms for this experiment are shown in the Appendix.
We can see from the results that with fixed hyperparamters, the conclusions for AQE remain largely unchanged, except for Hopper, where REDQ becomes the strongest algorithm.
When averaged across environments, AQE still matches the high sample efficiency of REDQ during the early stages of training.
Furthermore, on average, AQE’s asymptotic performance is still 16%, 11% and 9% higher than SAC, REDQ and TQC, respectively.
4.2 DeepMind Control Suite Results
In this section, we provide detailed experimental results for AQE, TQC, REDQ and SAC for the nine most challenging DeepMind Control Suite (DMC) environments.
The TQC and REDQ papers do not consider DMC benchmark, so we employ the same hyperparameters for TQC and REDQ as for the
MuJoCo environments.
For TQC, we employ 5 critic networks with 25 distributional samples and drop 2 atoms per critic across environments.
For REDQ, we keep using N = 10 and M = 2 .
To make the comparison fair, we also use the same hyperparameter values across environments for AQE.
We present the learning curves in Figure 2.
Similar to MuJoCo benchmark, for each algorithm, we plot the average return of 5 independent trials as the solid curve, and plot the standard deviation across 5 seeds as the shaded region.
We run all algorithms to 1 million environment interactions except for the most challenging environment, Humanoid-run, where we run up to 4.5 million environment interactions.
We use the same evaluation protocol as for the MuJoCo environments.
Figure 2 shows that in DMC environments with fixed hyperparameters, AQE continues to outperform
TQC except for the Humanoid-run environment, where TQC performs better than AQE in the final stage of training.
AQE and REDQ have comparable results in some of the DMC environments during traning, however, AQE usually outperforms REDQ in the more challenging environments, such as Hopper-hop, Humanoid-run, and Quadruped-run.
We report detailed early-stage and late-stage performance comparisons of all algorithms in Appendix.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
In summary, in the early stage of training (100K data), AQE performs 13.71x better than SAC, 7.59x better than TQC and matches the excellent performance of REDQ in nine environments.
In the late-stage training (1M data), AQE always matches or outperforms all other algorithms, except for
Humanoid-run, where TQC performs the best.
On average, AQE performs 37% better than SAC, 8% better than TQC, and 3% better than REDQ.
Using the same hyperparameters and averaged across nine DMC environments, AQE achieves the asymptotic performance of SAC approximately 3x faster than SAC, 1.57x faster than TQC, and 1.05x faster than REDQ.
4.3 Ablations
In this section, we use ablations to provide further insight into AQE.
We focus on the Ant environment, and run the experiments up to 1M time steps.
(In the Appendix we provide ablations for the other four environments.)
As in the REDQ paper, we consider not only performance but normalized bias and standard deviation of normalized bias as defined by the REDQ authors.
We first look at how the ensemble size N affects AQE.
The first row in Figure 3 shows AQE with N equal to 2, 5,and 15, with two heads for each Q network, and the percentage of kept Q-values unchanged.
As the ensemble size N increases, we generally obtain a more stable average bias, a lower std of bias, and stronger performance.
When trained with high UTD value, a relatively small ensemble size, for example, N = 5 , can greatly reduce bias accumulation, resulting in much stronger performance.
This experimental finding is consistent with the results in Theorem 1 in Appendix G.
[Figure]
The second row in Figure 3 shows how the keep parameter can affect the algorithm’s performance: under the same high UTD value, as K decreases, the average normalized Q bias goes from overestimation to under-estimation.
Consistent with the theoretical result in Theorem 1, by decreasing K we lower the average bias.
When K becomes too small, the Q estimate becomes too conservative and starts to have negative bias, which makes learning difficult.
We see that K = 16 has an average bias closest to 0 and also a consistently small std of bias.
These results are similar for the other four environments, as shown in the Appendix.
The third row in Figure 3 shows results for variants of the target computation methods.
The Median curve uses the median value of all the Q estimates in the ensemble to compute the Q target.
The
RemoveMinMax curve drops the minimum and maximum values of all the Q estimates in the ensemble to compute the Q target.
We see that these two variants give larger positive Q bias values.
We also considered different combinations of ensemble size N and the number of multi-heads h while keeping the total number of Q-function estimates N · h fixed.
We performed these experiments for all five environments.
In terms of performance, we found the two best combinations to be N = 20 , h =and N = 10 , h
= 2 , with the former being about 50% slower than the latter in terms of computation time.
We also consider endowing REDQ with the same multi-head ensemble architecture as AQE and find that it does not improve REDQ substantially (additional details in Appendix F).
We also consider comparing AQE with SAC and TQC, all using a more aggressive UTD ratio of
G = 5 .
Although SAC becomes more sample efficient with G = 5 , AQE continues to outperform both algorithms except for Humanoid, where once again TQC performs somewhat better than AQE for the final stage of training (Additional details in appendix E).
Conclusion
Perhaps the most important takeaway from this study is that a simple model-free algorithm can do surprisingly well, providing state-of-art performance at all stages of training.
There is no need for a model, distributional representation of the return, or in-target randomization to achieve high sample efficiency and asymptotic performance.
With extensive experiments and ablations, we show that AQE is both performant and robust.
In both
OpenAI Gym and DMControl, AQE is able to achieve superior performance in all stages of training with the same hyperparameters, and it can be further improved with per-task finetuning.
Our ablations show that AQE is robust to small changes in the hyperparameters.
Our theoretical results complement the experimental results, showing that the estimation bias can be controlled by either varying the ensemble size N or the keep parameter K .
AQE along with prior works show that a high update-to-data ratio combined with refined bias control can lead to very significant performance gain.
An interesting future work direction is to investigate whether there are other critical factors (in addition to bias control) that can allow us to further benefit from a high update-to-data ratio, and achieve even better sample efficiency with simple model-free methods.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
Supplementary Material A Hyperparameters and implementation details
Table 4 gives a list of hyperparameters used in the experiments.
Most of AQE’s hyperparameters are made the same as in the REDQ paper to ensure fairness and consistency in comparisons, except that AQE has 2-head critic networks.
As compared with AQE and REDQ, TQC uses a larger critic network with 3 layers of 512 units per layer.
In table 5, we report the dropped atoms d for TQC and the number of Q values we keep in the ensemble to calculate the target in AQE.
In algorithm 1, we provide detailed pseudo-code.
[Table] [Table]
B Additional Results for AQE, TQC, REDQ and SAC in MuJoCo Benchamrk with Fixed Hyper-parameters
We present the experiment on the five MuJoCo environments with the same hyperparameter values across environments for TQC (drop two atoms per network) and AQE (K = 16) in Figure 4.
[Figure]
AQE uses K = 16 and TQC uses atoms
= 2 per critic.
[Table] [Table]
C Additional Results for AQE, TQC, REDQ and SAC in DeepMind Control Suite Benchmark
Figure 2 presents the performance of AQE, TQC, REDQ and SAC for 9 DeepMind Control Suite (DMC) environments.
We can see that AQE continues to outperform TQC except for Humanoid run environment, where TQC performs better than AQE in the final stage training.
AQE and REDQ have comparable results in some of the DMC environments, however, AQE usually outperforms REDQ in the more challenging environments, such as Hopper-hop, Humanoid-run and Quadruped-run.
We report detailed early-stage and late-stage performance comparisons of all algorithms in Table 8 and
Table 9.
On average, in the early stage of training, AQE performs 13.71 times better than SAC, 7.
times better than TQC and 1.02 times better than REDQ.
In the late-stage training, on average, AQE performs 1.37 times better than SAC, 1.08 times better than TQC and 1.03 times better than REDQ.
[Table]
[Table] [Table]
D Additional Results for AQE, SAC-5 and TQC-
Figure 5 presents the performance of AQE, SAC-5 and TQC-5 for all the environments.
SAC-5 and
TQC-5 uses UTD ratio G = 5 for SAC and TQC, respectively.
We can see that AQE continues to outperform both algorithms except for Humanoid, where TQC performs somewhat better than AQE in the final stage training.
SAC becomes more sample efficient with G = 5 ; however, AQE still beats
SAC-5 by a large margin.
[Figure]
E Additional Results for parameter K
Due to lack of space
[Figure]
Figure 3 only compares different AQE keep numbers K for Ant.
Figure 6 shows the performance, average estimation bias and standard deviation for all five environments.
Consistent with the theoretical result in Theorem 1, by decreasing K , we lower the average bias.
[Figure]
F Additional Results for Multi-head Architecture
Due to lack of space, Figure 3 only compares the different size of the ensemble N and the number of heads h for Ant.
Figure 7 shows the results for all five environments.
We can see that the combination of N = 10 , h = 2 and N = 20 , h = 1 have comparable performance.
However, N = 10 and h =is faster in terms of computation time.
[Figure]
Will the performance of REDQ match that of AQE if we also provide REDQ a multi-head architecture?
Figure 8 examines the performance of REDQ when it is endowed with the same multi-head architecture as AQE.
We see that the performance of REDQ does not substantially improve.
[Figure]
G Theoretical Results
In this section, we characterize how changing the size of the ensemble N and the keep parameter K affects the estimation bias term in the AQE algorithm.
We will restrict our analysis to the tabular version of AQE shown in algorithm 2.
Our analysis will follow similar lines of reasoning as Lan et al.
and
Chen et al.
(2021) which extends upon the theoretical framework introduced in Thrun and Schwartz (1993).
For each a ∈ A , let E K,N ( s, a ) be the ensemble members in { 1 , . .
. , N } with the K lowest values of Q j ( s, a ) , j = 1 , . .
N .
In the tabular case, the target for the Q networks take the form:
[Equation]
Define the post-update estimation bias as
[Equation]
Under this definition, if E [ Z K,N ] > 0 , then the expected post-update estimation bias is positive and there is a tendency for the positive bias to accumulate during updates.
Similarly, if E [ Z K,N ] < 0 , then the expected post-update estimation bias is negative and there is a tendency for the negative bias to accumulate during updates.
Ideally, we would like E [ Z K,N ] ≈
Also let
[Equation]
where e j ( s, a ) is an independent and identically distributed error term across all j ’s and all a ’s for each fixed s .
We further assume that E [ e j ( s, a )] = 0 .
Note that with this assumption
[Equation]
that is the pre-update estimation bias is zero.
The following theorem shows how the expected estimation bias changes with N and K :
Theorem 1.
The following results hold for E [ Z K,N ] :
E [ Z N,N ] ≥ 0 for all N ≥ 1 . 2.
E [ Z K − 1 ,N ] ≤ E
[ Z K,N ] for all K ≤ N . 3.
E [ Z K,N +1 ] ≤ E [ Z K,N ] .
Suppose that e jsa ≤ c for some c > 0 for all s, a and j .
Then there exists an N sufficiently large and K < N such that E [ Z K,N ] < 0 .
Proof Sketch.
Part 1 is a result of Jensen’s Inequality, and Parts 2 and 3 can be shown by analyzing how the average of the K smallest ensembles changes when adding an extra ensemble model.
Given the first three parts, we only need to show that E [ Z 1 ,N ] < 0 to show that there exists a K for a sufficiently large N where the expected bias is negative.
See the next section for full proof.
Theorem 1 shows that we can control the expected post-update bias E [ Z K,N ] through adjusting K and N .
More concretely, we can bring the bias term from above zero (i.e. over estimation) to under zero (i.e. under estimation) by decreasing K and/or increasing N .
We note also that similar to Chen et al.
(2021), we make very few assumptions on the error term e s,a .
This is in contrary to Thrun and Schwartz (1993) and Lan et al.
(2020), both of whom assume that the error term is uniformly distributed.
[Equation]
(cid:88)
[Equation] [Equation]
Q
[Equation] [Equation]
a
[Equation]
H Proofs
We first present the following lemma:
Lemma 1 (Chen et al., 2021) .
Let X 1 , X 2 , . .
. be an infinite sequence of i.i.d.
random variables with cdf F ( x ) and let τ = inf x : F ( x ) > 0 .
Also let Y N = min { X 1 , X 2 , . . .
, X N } .
Then Y 1 , Y 2 , . .
converges to τ almost surely.
Proof.
See Appendix A.2 of Chen et al.
Theorem 1.
The following results hold for E [ Z K,N ] :
E [ Z N,N ] ≥ 0 for all N ≥ 1 . 2.
E [ Z K − 1 ,N ] ≤ E
[ Z K,N ] for all K ≤ N . 3.
E [ Z K,N +1 ] ≤ E [ Z K,N ] .
Suppose that e jsa ≤ c for some c > 0 for all s, a and j .
Then there exists an N sufficiently large and K < N such that E [ Z K,N ] < 0 .
Proof.
By definition,
[Equation]
Let
[Equation]
Since for any state s , max a ¯ Q K +1 ,N ( s, a ) ≥ max a
[Equation]
K
[Equation]
Comparing E [ Z K,N ] and E [ Z K,N +1 ] is equivalent to comparing ¯ Q K,N ( s, a ) and ¯
Q K,N +1 ( s, a ) .
Since e j ( s, a ) is i.i.d., by extension Q j ( s, a ) is also i.i.d.
for j = 1 , 2 , · · · .
Suppose Q j ( s, a ) is drawn from some probability distribution F , then given ¯ Q K,N ( s, a ) , ¯
Q K,N +1 ( s, a ) can be calculated by generating an additional Q i ( s, a ) from F .
The new sample Q i ( s, a ) affects the calculation of ¯ Q K,N +1 ( s, a ) under the following two cases
: • If Q i ( s, a ) > max j ∈ E K,N Q j ( s, a ) , then the lowest K values remain unchanged hence ¯ Q K,N ( s, a ) =
¯ Q K,N +1 ( s, a ) .
• Else if Q i ( s, a ) ≤ max j ∈ E K,N Q j ( s, a ) , then max j ∈ E K,N Q j ( s, a ) would be removed from and Q i ( s, a ) would be added to the set of lowest K values, therefore ¯ Q K,N +1 ( s, a ) ≤ ¯ Q K,N ( s, a ) .
Combining the two cases ¯ Q K,N +1 ( s, a ) ≤ ¯ Q K,N ( s, a ) , therefore E [ Z K,N +1 ] ≤ E [ Z K,N ]
Since E [ Z N,N ] ≥ 0 , E [ Z K,N ] ≤ E [ Z K +1 ,N ] and E [ Z K,N +1 ] ≤ E [ Z K,N ] .
It is suffice to show that E [ Z 1 ,N ] < 0 for some N .
The rest of the proof largely follows Theorem 1 of
Chen et al.
Let τ = inf { x : F a ( x ) > 0 } where F a ( x ) is the cdf of Q j ( s, a ) , j = 1 , 2 , . . . .
By Lemma 1, ¯ Q 1 ,N ( s, a ) =
min 1 ≤ j ≤ N Q
j
( s, a ) converges almost surely to to τ a for each a .
Since the action space is finite, it then follows that max a ¯ Q 1 ,N ( s, a ) converges almost surely to to τ = max a τ a .
Due to our assumption that e j ( s, a ) ≤ c and that Q π ( s, a ) is finite, it then follows that max a ¯ Q 1 ,N ( s, a ) is also bounded above.
By Part 3 of the theorem, ¯ Q 1 ,N ( s, a ) is monotonoically decreasing w.r.t.
N .
and since max a ¯ Q 1 ,N ( s, a ) is also bounded above and converges almost surely to τ , we have
[Equation]
where the last equality follows from the assumption that the error e j ( s, a ) is non-trivial, and hence τ a < max a Q π ( s, a ) for all a .
Therefore for a sufficiently large N , there exists a 1 ≤ K ≤ N such that E K,N < 0 .
I Computing Infrastructure
[Table]