M AXIMIZING E NSEMBLE D IVERSITY IN D EEP R EINFORCEMENT L EARNING
A BSTRACT
Modern deep reinforcement learning (DRL) has been successful in solving a range of challenging sequential decision-making problems.
Most of these algorithms use an ensemble of neural networks as their backbone structure and beneﬁt from the diversity among the neural networks to achieve optimal results.
Unfortunately, the members of the ensemble can converge to the same point either the parametric space or representation space during the training phase, therefore, losing all the leverage of an ensemble.
In this paper, we describe Maximize Ensemble Diversity in
Reinforcement Learning (MED-RL), a set of regularization methods inspired from the economics and consensus optimization to improve diversity in the ensemblebased deep reinforcement learning methods by encouraging inequality between the networks during training.
We integrated MED-RL in ﬁve of the most common ensemble-based deep RL algorithms for both continuous and discrete control tasks and evaluated on six Mujoco environments and six Atari games.
Our results show that MED-RL augmented algorithms outperform their un-regularized counterparts signiﬁcantly and in some cases achieved more than 300 % in performance gains
I NTRODUCTION
Reinforcement learning (RL) agents trained with high capacity function approximators such a deep neural networks have shown to solve complex sequential decision-making problems, including the board games of Chess, GO and Shogi (Silver et al., 2016; 2017; 2018), achieving super-human performance in video games (Mnih et al., 2015; Vinyals et al., 2019) and solving robotic manipulation tasks (Liu et al., 2021).
Despite achieving these tremendous goals, modern deep reinforcement learning (DRL) algorithms have plethora of limitations.
For example, it is well-known that DRL algorithms are sample-inefﬁcient and require stupendous amount of environment interactions to learn an optimal policy (Łukasz Kaiser et al., 2020).
Additional problems encountered and exacerbates during training a DRL agent includes the overestimation bias that occurs while estimating the target values for Q-learning (Fujimoto et al., 2018; Lan et al., 2020;
Hado van Hasselt et al., 2016), error propagation during Bellman backup (Kumar et al., 2019) and trade-off between exploration and exploitation (Chen et al., 2017).
Recently, the use of ensemble has been a popular choice to address the above mentioned issues.
These methods combine multiple neural networks to model the value functions or (and) the policy (Osband et al., 2016; Chen et al., 2017; Lan et al., 2020; Lee et al., 2020).
For example, TD3 (Fujimoto et al., 2018) used two critics to address the overestimation bias problem in continuous control problems while MaxminDQN (Lan et al., 2020) provided a mechanism to use the cardinality of the ensemble to use as a knob to tune between over and under estimation in deep Q-learning.
Similarly, Bootstrapped
DQN (Osband et al., 2016; Chen et al., 2017) used ensemble for effective exploration.
The primary insight of this paper is that the performance of ensemble based methods is contingent on maintaining sufﬁcient diversity between the neural networks of the ensemble.
If the neural networks in the ensembles converge to a common representation (we will show that this is the case in many scenarios), the performance of these approaches signiﬁcantly degrades.
We note that even with different representations, the Q-values will still converge towards a shared optimum, but they are statistically less likely to follow the same learning trajectory elsewhere.
In this paper, we propose M aximize E nsemble D iversity in
R einforcement
L earning (MED-RL), a set of regularization methods inspired from the economics and consensus optimization to improve diversity and to prevent the collapse of the representations in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training.
The objective of the regularizers is solely to keep the representations different, while still allowing the models to converge to the optimal Q-value.
The motivation for the regularizers came from topic of income distribution in economic theory that provides a rich source of mathematical formulations that measure inequality.
While in economics, high inequality is seen as a negative, in our case we used the inequality metrics to encourage diversity between the neural networks.
To summarize, our contributions are following:
We empirically show that high representation similarity between neural network based Q-functions leads to degradation in performance in ensemble based Q-learning methods.
To mitigate this, we propose ﬁve regularizers based on inequality measures from economics theory and consensus optimization that maximize diversity between the neural networks in ensemble based reinforcement learning methods.
We integrated MED-RL in TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al., 2018) and REDQ (Chen et al., 2021) for continuous control tasks and in MaxminDQN (Lan et al., 2020) and EnsembleDQN (Anschel et al., 2017) for discrete control tasks and evaluated on six Mujoco environments and six Atari games.
Our results show that MED-RL augmented algorithms outperform their un-regularized counterparts signiﬁcantly and in some cases achieved more than 300 % in performance gains and are up to 75% more sample-efﬁcient.
We also show that MED-RL augmented SAC is more sample-efﬁcient than REDQ, an ensemble based method speciﬁcally designed for sample-efﬁciency, and can achieve similar performance to REDQ up to 50 times faster on wall-clock time.
R ELATED W ORK
Ensembles in Deep RL: Use of an ensemble of neural networks in Deep RL has been studied in several recent studies for different purposes.
In (Fujimoto et al., 2018; Anschel et al., 2017; Lan et al., 2020) have used an ensemble to address the overestimation bias in deep Q-learning based methods for both continuous and discrete control tasks.
Similarly, Bootstrapped DQN and extensions (Osband et al., 2016; Chen et al., 2017) have leveraged ensemble of neural networks for efﬁcient exploration.
The problem of error propagation in Bellman backup was addressed in (Kumar et al., 2019) using an ensemble of neural networks.
Sample efﬁciency, a notorious problem in RL has taken advantage from an ensemble (Chen et al., 2021).
Recently, SUNRISE (Lee et al., 2020) proposed a uniﬁed framework for ensemble-based deep reinforcement learning.
Diversity in Ensembles: Diversity in neural network ensembles has been studied years before the resurgence of deep learning (Brown, 2004).
Even though diversity is an important topic in neural networks, most of the studies in this topic revolve around addressing problems in supervised learning settings.
More recently there has been a number of studies that have diversity in ensembles to measure and improve model uncertainty.
Jain et al.
(2020) have proposed a diversity regularizer to improve the uncertainty estimates in out-of-data distribution.
Lee et al.
(2015) have used Multiple choice Learning to learn diverse Convolutional Neural Networks for image recognition.
Regularization in Reinforcement Learning: Regularization in reinforcement learning has been used to perform effective exploration and learning generalized policies.
For instance, (Grau-Moya et al., 2019) uses mutual-information regularization to optimize a prior action distribution for better performance and exploration, (Cheng et al., 2019) regularizes the policy π ( a | s ) using a control prior, (Galashov et al., 2019) uses temporal difference error regularization to reduce variance in
Generalized Advantage Estimation (Schulman et al., 2016).
Generalization in reinforcement learning refers to the performance of the policy on different environment compared to the training environment.
For example, (Farebrother et al., 2018) studied the effect of L 2 norm on DQN on generalization, (Tobin et al., 2017) studied generalization between simulations vs. the real world, (Pattanaik et al., 2018) studied parameter variations and (Zhang et al., 2018) studied the effect of different random seeds in environment generation.
Diversity in Reinforcement Learning: Diversity in reinforcement learning is active area of research.
(Pacchiano et al., 2020) uses Determinantal Point Processes to promote behavioral diversity, Lupu et al.
(2021) have used policy diversity to improve zero-shot coordination in multi-agent setting.
(Tang et al., 2021) uses reward randomization for discovering diverse strategic policies in complex multi-agent games.
In (Li et al., 2021) proposed CDS that uses information-theoretical objective to maximize the mutual information between agents’ identities and trajectories and encourage diversity.
More recently (An et al., 2021) have used diversiﬁed Q-ensembles to address overestimation in ofﬂine reinforcement learning.
Representation Similarity:
Measuring similarity between the representations learned by different neural networks is an active area of research.
For instance, (Raghu et al., 2017) used Canonical
Correlation Analysis (CCA) to measure the representation similarity.
CCA ﬁnd two basis matrices
such that when original matrices are projected on these bases, the correlation is maximized.
(Raghu et al., 2017; Mroueh et al., 2015) used truncated singular value decomposition on the activations to make it robust for perturbations.
Other work such as (Li et al., 2015) and (Wang et al., 2018) studied the correlation between the neurons in the neural networks.
B ACKGROUND
Reinforcement learning: We consider an agent as a Markov Decision Process (MDP) deﬁned as a ﬁve element tuple ( S , A , P, r, γ ) , where S is the state space, A is the action space, P : S ×
A × S → [0 , 1] are the state-action transition probabilities, r : S ×
A × S → R is the reward mapping and γ → [0 , 1] is the discount factor.
At each time step t the agent observes the state of the environment s t ∈ S and selects an action a t ∈ A .
The effect of the action triggers a transition to a new state s t +1 ∈ S according to the transition probabilities P , while the agent receives a scalar reward
R t = r ( s t , a t , s t +1 ) .
The goal of the agent is to learn a policy π that maximizes the expectation of the discounted sum of future rewards.
Representation Similarity Measure: Let X ∈ R n × p 1 denote a matrix of activations of p 1 neurons for n examples and Y ∈ R n × p 2 denote a matrix of activations of p 2 neurons for the same n examples.
Furthermore, we consider K ij = k
( x i , x j ) and L ij =
l ( y i , y j ) where k and l are two kernels.
Centered Kernel Alignment (CKA) (Kornblith et al., 2019; Cortes et al., 2012; Cristianini et al., 2002) is a method for comparing representations of neural networks, and identifying correspondences between layers, not only in the same network but also on different neural network architectures.
CKA is a normalized form of Hilbert-Schmidt Independence Criterion (HSIC)
(Gretton et al., 2005).
Formally, CKA is deﬁned as:
[Equation]
HSIC is a test statistic for determining whether two sets of variables are independent.
The empirical estimator of HSIC is deﬁned as:
[Equation]
where H is the centering matrix H n
= I n − 1 n 11 T .
[Table]
u r n
[Table]
[Figure]
M AXIMIZE E NSEMBLE D IVERSITY IN R EINFORCEMENT L EARNING
In this section, we propose MED-RL: M aximize E nsemble D iversity in R einforcement
L earning, a set of regularizers inspired from the Economics and consensus optimization to improve diversity and to prevent the collapse of the representations in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training.
This section is organized as follows:
We empirically show that high representation similarity between neural network based Q-functions leads to degradation in performance in ensemble based Q-learning methods.
we present the Economics theory and consensus optimization inspired regularizers with their mathematical formulation.
E MPIRICAL E VIDENCE TO C ORRELATE P ERFORMANCE AND R EPRESENTATION S IMILARITY
The work in this paper starts from the conjecture that high representation similarity between neural networks in an ensemble-based Q-learning technique correlates to poor performance.
To empirically verify our hypothesis, we trained a MaxminDQN (Lan et al., 2020) agent with two neural networks on the Catcher environment (Qingfeng, 2019) for about 3000 episodes ( 5 × 10 6 training steps) and calculated the CKA similarity with a linear kernel after every 500 episodes.
The training graph along with the CKA similarity heatmaps are shown in Figure 1.
Notably at episode 500 (heatmap A) and episode 2000 (heatmap C), the representation similarity between neural networks is low but the average return is relatively high.
In contrast, at episode 1000 (heatmap B) and episode 3000 (heatmap
D) the representation similarity is highest but the average return is lowest.
4. R EGULARIZATION FOR M AXIMIZING E NSEMBLE D IVERSITY
In order to maximize the ensemble diversity, we propose to regularize the training algorithm with an additional criteria that favors diversity between the ensembles.
In the following, N is the number of neural networks in the ensemble, (cid:96) i is the L 2 norm of the i -th neural network’s parameters, ¯ (cid:96) is the mean of all the L 2 norms and (cid:96) is the list of all the L 2 norms.
The ﬁrst four metrics we consider are based on inequality measures from economic theory.
While in economics, inequality is usually considered something to be avoided, in our case we aim to increase inequality (and thus, ensemble diversity).
The Atkinson Index (Atkinson et al., 1970) measures income inequality and is useful in identifying the end of the distribution that contributes the most towards the observed inequality.
Formally, it is deﬁned as
[Equation]
where (cid:15) at is the inequality aversion parameter used to tune the sensitivity of the measured change.
When (cid:15) at = 0 , the index is more sensitive to the changes at the upper end of the distribution, while it becomes sensitive towards the change at the lower end of the distribution when (cid:15) at approaches 1.
The Gini coefﬁcient (Allison, 1978) is a statistical measure of the wealth distribution or income inequality among a population and deﬁned as the half of the relative mean absolute difference:
[Equation]
The Gini coefﬁcient is more sensitive to deviation around the middle of the distribution than at the upper or lower part of the distribution.
The Theil index (Johnston, 1969) measures redundancy, lack of diversity, isolation, segregation and income inequality among a population.
Using the Theil index is identical to measuring the redundancy in information theory, deﬁned as the maximum possible entropy of the data minus the observed entropy:
[Equation]
The variance of logarithms (Ok & Foster, 1997) is a widely used measure of dispersion with natural links to wage distribution models.
Formally, it is deﬁned as:
[Equation]
where g( (cid:96) ) is the geometric mean of (cid:96) deﬁned as ( (cid:81)
Ni =1 (cid:96) i ) 1 /N .
The ﬁnal regularization method we use is inspired from consensus optimization.
In a consensus method (Boyd et al., 2011), a number of models are independently optimized with their own taskspeciﬁc parameters, and the tasks communicate via a penalty that encourages all the individual solutions to converge around a common value.
Formally, it is deﬁned as
[Equation]
[Equation]
Where ¯ θ is the mean of the parameters of all the neural networks and θ i represents the parameters of the
i − th neural network.
We will refer this regularizer as MeanVector throughout this paper.
For completeness, the algorithm shown in Algorithm 1.
Notice that the regularization term appears with a negative sign, as the regularizers are essentially inequality metrics that we want to maximize.
4. T RAINING A LGORITHM
Using the regularization functions deﬁned above, we can develop diversity-regularized variants of the the ensemble based algorithms.
The training technique is identical to the algorithms described in (Lan et al., 2020; Anschel et al., 2017; Fujimoto et al., 2018; Haarnoja et al., 2018; Chen et al., 2021), with a regularization term added to the loss of the Q-functions.
The loss term for i -th Q-function with parameters ψ i is:
[Equation]
where Y is the target value depending on the algorithm, I is the regularizer of choice from the list above and λ is the regularization weight.
Notice that the regularization term appears with a negative sign, as the regularizers are essentially inequality metrics that we want to maximize.
As a reference the modiﬁed algorithm for MaxminDQN is shown in Algorithm 1.
E XPERIMENTS 5. I SN ’ T RESAMPLING AND DIFFERENT INITIALIZATION OF WEIGHTS ENOUGH ?
The most common question that comes to mind to address the diversity issue is why not initialize the neural networks with different weights and train each network with a different sample from the buffer?
This approach has been thoroughly discussed in (Brown, 2004) and have been shown to be ineffective.
To re-iterate the ﬁndings in (Brown, 2004), we performed a regression experiment in which we learnt a sine function using two different three layered fully connected neural networks with 64 and 32 neurons in each hidden layer with ReLU.
The neural networks were initialized using different weights and were trained using different batch sizes (512 , 128) and learning rates (1 e − 4 , 1 e − 3 ) .
The Figure 2a shows the learnt functions while Figure 2b represents their CKA similarity heatmap before and after training.
The odd numbered layers represent pre-ReLU activations while the even numbered layers represent post-ReLU activations.
It can be seen that before training, the CKA similarity between the two neural networks from layer 4 and onward is relatively low and the output being 0% similar while after training, the trained networks have learnt highly similar representation while their output being 98% similar.
[Figure]
This example shows that neural networks can learn similar representation while trained on different batches.
This observation is important because in MaxminDQN and EnsembleDQN training, each neural network is trained on a separate batch from the replay buffer but still learns similar representations [Figure] [Figure]
5. E XPERIMENTAL S ETUP
Continuous control tasks:
We evaluated MED-RL augmented continuous control algorithms such as TD3, SAC and REDQ on the Mujoco continuous control benchmark environments.
We compared the results of MED-RL with un-regularized counterparts.
We report the mean and standard deviation across ﬁve runs after 1M timesteps on six complex environments: Cheetah, Walker, Hopper, Ant,
Humanoid and Humanoid-Standup.
For REDQ, we evaluated on Cheetah, Walker, Hopper and Ant for 300K timesteps only.
5. C OMPARATIVE EVALUATION
Continuous control tasks:
Tables 1 to 3 show the average returns of evaluation roll-outs for all the continuous control methods.
MED-RL consistently improves the performance of SAC, TD3 and
REDQ in all the experiments.
Even though the focus of this work is to maximize the average return, we ﬁnd that MED-RL augmented algorithms are more sample-efﬁcient than their un-regularized counterparts (see Figures 3, 4 and 5).
For example, it can be seen Figure 3b that baseline SAC reaches the average return of 10K in about 1M environment interactions while all MED-RL variants reach the same average return in approximately 250K environment interaction, therefore, being approximately 75 % more sample efﬁcient than the baseline SAC.
This improvement in sample-efﬁciency can be noted in all nearly all the experiments except SAC-Humanoid and TD3-Walker experiments.
The training plots for REDQ are shown in Appendix.
[Table] [Table]
[Table]
5. S AMPLE E FFICIENCY AND C OMPUTE T IME
Tables 1 to 3 show that MED-RL augmented continuous control algorithms outperform the baseline versions signiﬁcantly and a visual inspection of Figures 3, 4 and 5 show that MED-RL augmented algorithms are more sample-efﬁcient as well.
But are they more sample-efﬁcient than algorithms that are speciﬁcally designed for sample-efﬁciency such as REDQ?
To answer this question, we took the bolded results from Table 1, referred as MED-RL in this section, and evaluated the number of environment interactions and wall-clock time it took for MED-RL to reach similar performance as that of baseline REDQ.
As shown in Table 4, MED-RL achieves similar performance to REDQ in 50% and 20% few environment interactions on Ant and HalfCheetah environment respectively and have signiﬁcantly surpassed REDQ on 300 K environment interactions.
MED-RL does not only improve sample-efﬁciency but signiﬁcantly improves compute time.
As shown in Table 4, MED-RL achieves similar performance to REDQ up to 50 times faster on wall-clock time.
Note it can be argued that REDQ can be parallelized to achieve faster wall-clock time but here we are only comparing standard sequential implementations but that will not address the sample-efﬁciency issue.
[Table]
Discrete control tasks:
We evaluated MED-RL augmented discrete control algorithms such as
MaxminDQN and EnsembleDQN on the PyGames (Qingfeng, 2019) and MinAtar (Young & Tian, 2019).
We chose these environments to have a fair comparison since we used the source code provided by MaxminDQN authors.
We reused all the hyperparameter settings from (Lan et al., 2020) except the number of neural networks, which we limited to four and trained each solution for ﬁve ﬁxed seeds.
The results on the discrete control tasks are shown in the Appendix.
C ONCLUSION
In this paper, we proposed Maximize Ensemble Diversity in Reinforcement Learning (MED-RL), a set of regularization methods inspired from the economics and consensus optimization to improve diversity in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training.
We also empirically showed that high representation similarity between the networks of the ensemble could cause degradation in the performance.
Our experiments have shown that MED-RL not only improves the average return of ensemble based reinforcement learning algorithms but can increase their sample-efﬁciency by approximately 75% when compared to their un-regularized counterparts.
Additionally we have shown the SAC when augmented with
MED-RL can outperform REDQ, an algorithm speciﬁcally designed for sample-efﬁciency, in both sample-efﬁciency and compute time.
Acknowledgement: This work had been supported in part by the National Science Foundation under grant number CNS-
A T RAINING P LOTS OF REDQ
[Figure]
B R ESULTS ON D ISCRETE C ONTROL T ASKS
AXMIN DQN
[Figure]
n
[Figure]
v e r a g e R e t u r n
[Figure]
v e r a g e R e
[Figure]
u r n
[Figure] [Figure]
Breakout
[Figure]
B. R ESULTS IN T ABULAR F ORM
[Table]
[Table] [Table]
Figure 6 shows the training curves for the all the six environments and Tables 5 to 7 represent the results in the tabular, similar to our observations for the continuous control experiments, MED-RL augmented MaxminDQN outperformed un-regularized MaxminDQN signiﬁcantly on average return and sample-efﬁciency metrics.
Notably, in Seaquest and SpaceInvader environments where MED-RL
MaxminDQN with two neural networks achieved 400% and 300% increase in performance.
Note that the baselines shown in Figure 6 are strong baselines.
For example, in the MaxminDQN paper, the best performance of baseline MaxminDQN on Asterix and SpaceInvader environments is around 20 and 50 respectively while we have achieved an average reward of 35 and 135 respectively.
C R ESULTS ON D ISCRETE C ONTROL T ASKS U SING E NSEMBLE DQN
[Figure]
Catcher
[Figure] [Figure]
C. R ESULTS IN T ABULAR F ORM
[Table]
[Table] [Table]
Figure 7 shows the training curves for the all the six environments and Tables 8 to 10 represent the results in the tabular, similar to our observations for the continuous control experiments, MED-RL augmented EnsembleDQN outperformed un-regularized EnsembleDQN signiﬁcantly on average return and sample-efﬁciency metrics.
Notably, in Copter, SeaQuest and SpaceInvader environments where MED-RL EnsembleDQN with two neural networks achieved 200% , 350% and 250% increase in performance.
D T -SNE V ISUALIZATIONS
To visualize the impact of the regularization, Figures 8 and 9 shows t-SNE (van der Maaten & Hinton, 2008) visualization of the activations of the last layer of the trained networks.
Figure 8a show the network trained for the Catcher environment, while Figures 8b and 9, the network trained for the
PixelCopter environment.
The upper row of the ﬁgure shows the original, unregularized models, while the lower row a regularized version.
For all combinations, we ﬁnd that the activations from the original MaxminDQN and EnsembleDQN versions do not show any obvious pattern, while the regularized ones show distinct clusters.
An additional beneﬁt of t-SNE visualizations over CKA similarity heatmaps is that the CKA similarity heatmaps are useful to show representation similarity between two neural networks, but they become counter intuitive as the number of neural networks increases.
[Figure]
[Figure]
E P LOTTING THE G INI I NEQUALITY
We measured the L 2 norm inequality of the baseline MaxminDQN and EnsembleDQN along with their regularized versions.
We trained baseline MaxminDQN and EnsembleDQN with two neural networks along with their Gini index versions with regularization weight of 10 − 8 on the PixelCopter environment on a ﬁxed seed .
Figure 10 represents the L 2 norm inequality of the experiments along their average return during training.
Notably, despite each neural network being trained on a different batch, the L 2 norm of the baseline MaxminDQN and EnsembleDQN are quite similar while the Lnorm of the regularized MaxminDQN and EnsembleDQN have high inequality
[Figure]
F I MPLEMENTATION D ETAILS AND H YPERPARAMETERS
For our implementation of MaxminDQN and EnsembleDQN, we used the code provided by the MaxminDQN authors that has implementations of different DQN based methods github.com/qlan3/Explorer).
For the baseline experiments, we used most of the hyperparameter settings provided in the conﬁguration ﬁles by the authors except the number of ensembles which we limited to four.
[Table] [Table]
F. C OMPUTING I NFRASTRUCTURE
All the experiments were performed on a Kubernetes managed cluster with Nvidia V100 GPUs and
Intel Skylake CPUs.
Each experiment was run as an individual Kubernetes job with 11 CPUs, 16GB of RAM and 1 GPU (if needed). This conﬁguration allowed us to run experiments without any interference from other applications which was important to accurately measure the wall-clock time. Future Work: Even though the focus of this paper was on empirical testing, there are several different research questions that needs to be addressed, for example how do we select the best regularizer for a particular environment or an algorithm with some N numbers of networks in the ensemble.
Another interesting line of research is to study how different inequality distributions effect the diversity of the ensembles.