SANE: S PECIALIZATION - AWARE NEURAL NETWORK ENSEMBLE
Anonymous authors
Paper under double-blind review
A BSTRACT
Real-world data is often generated by some complex distribution, which can be approximated by a composition of multiple simpler distributions.
Thus, it is intuitive to divide complex model learning into training several simpler models, each of which specializes in one simple distribution, called model specialization in this paper.
Ensemble learning is one way to realize specialization, and has been widely used in practical machine learning scenarios.
Many ensemble methods propose to increase diversity of base models, which could potentially result in model specialization.
However, our studies show that without explicitly enforcing specialization, pursuing diversity may not be enough to achieve satisfactory ensemble performance.
In this paper, we propose SANE — an end-to-end ensemble learning method that actively enforces model specialization, where base models are trained to specialize in sub-regions of a latent space representing the simple distribution composition, and aggregated based on their specialties.
Experiments in several prediction tasks on both image datasets and tabular datasets demonstrate the superior performance of our proposed method over state-of-the-art ensemble methods.
I NTRODUCTION
Real-world data distribution could be complex in most cases and people usually approximate it by a composition of several simpler distributions (Xie et al., 2016;
Yang et al., 2016; Tsai et al., 2020).
Intuitively, to ﬁt the complex data distribution conveniently, we can divide the complex model learning process into training several simpler models, each of which specializes in one simple distribution.
In this paper, we call this idea model specialization.
Ensemble learning is one of the most effective methods to leverage model specialization.
Originated from decades ago (Hansen & Salamon, 1990), ensemble method (Schapire, 1990; Breiman, 1996;
Zhou et al., 2002) has been proven effective for practical machine learning tasks in various scenarios including computer vision (Huang et al., 2017), natural language processing (Shazeer et al., 2017) and tabular data mining (Liu et al., 2020).
Many works conduct ensemble learning through improving diversity across base models, which has been shown to reduce the variance of the combined predictions and improve generalization (Lee et al., 2015; Zhou et al., 2018).
The recent methodologies pursuing model diversity mainly through incorporating randomization in data sampling (Breiman, 1996; Ho, 1995) or model training (Srivastava et al., 2014; Lakshminarayanan et al., 2016).
Some other works propose sequential optimization such as boosting (Freund, 1995; Chen & Guestrin, 2016; Ke et al., 2017) and snapshot ensemble (Huang et al., 2017).
Other methods, e.g., model decorrelation (Zhou et al., 2018) and diversity enhancement (Zhang et al., 2020), explicitly encourage model diversity through incorporating additional objectives.
However, overvaluing diversity may hurt the ensemble effectiveness and the trade-off between the performance and diversity of base models still remains an open problem (Rame & Cord, 2021; Fort et al., 2019; Masegosa, 2019).
Moreover, without explicitly enforcing model specialization, pursuing model diversity does not provide guarantees of each model’s specialization on different simple distribution in the composition.
The last two columns in Figure 1 visualize the specialty of each model by plotting the data regions where the base model makes correct predictions.
It shows that neither the ensemble of (b) randomly initialized base models nor (c) diversiﬁed base models produce models with different specialties.
Speciﬁcally, although the method in (c) enforces diversity and shows low correlation among base models’ outputs, their data regions with correct predictions are very similar as shown in the last two columns in Figure 1, resulting in inferior performance after ensemble.
Besides, simply putting specialized base models together does not necessarily bring promising ensemble performance.
We need to know each base model’s specialty over different samples and aggregate their outputs based on their specialty.
As shown in Figure 1 (d), although the overall accuracy of the two base models are low, the ensemble model performs surprisingly well (100% accuracy in (d)) as long as 1) each base model specializes in unique sub-regions of the latent space and 2) the ensemble model knows which sub-regions each base model specializes in, demonstrating the effectiveness of specialization in ensemble learning.
More detailed analyses of Figure 1 are presented in Section 2.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
In this paper, we propose an end-to-end ensemble learning method that actively enforces model specialization by training models to specialize in speciﬁc simple distributions, and aggregates them based on their specialties.
Firstly, we represent the composition of simple distributions by mapping them into a uniﬁed latent space, where samples from each data distribution form a locally clustered sub-region.
This paper proposes to enforce each base model to specialize in one sub-region, and we call this region the base model’s corresponding specialty region .
The base models’ specialty regions are deﬁned by a set of learnable anchor points, each of which corresponds to the center of the specialty region, called specialty anchor .
During training, base models’ parameters and their corresponding specialty anchors are learned simultaneously.
Each base model’s specialty is enforced by encouraging model to focus more on samples in the vicinity of its specialty anchor.
Finally, during prediction, based on the correlation between models’ specialty anchors and the samples in the latent space, our method is able to automatically estimate each base model’s specialty over testing samples, and the outputs of the base models are adaptively aggregated based on their specialty over different testing samples.
To demonstrate the effectiveness of the proposed method, we conduct extensive experiments in prediction tasks on two tabular datasets and two image classiﬁcation datasets.
The comprehensive empirical studies show that 1) coupling specialized base models and specializationaware aggregation can beneﬁt ensemble learning and 2) the proposed specialization-aware neural network ensemble method can achieve superior performance compared with the state-of-the-art ensemble methods in various prediction tasks.
P RELIMINARIES AND SYNTHETIC ANALYSIS 2. P RELIMINARIES
In general, ensemble learning consists of two tasks: training multiple base models and aggregating their predictions.
We use {M k } Kk =1 to denote a collection of K base models.
For a given instance x n ∈ R d and its label y n ∈ [1 , C ] , each base model learns a mapping M k : R d (cid:55)−→ R C , which produces logits for C classes.
Base models’ predictions are then aggregated in a weighted manner.
The likelihood is written as p ( y n | x n ) =
(cid:80)
Kk =1 w k p ( y n | x n ; θ k ) s.t.
(cid:80)
Kk
=
w k = 1 , where M k is parameterized by θ k and w k denotes the weight for aggregating k -th model prediction.
Objectives of ensemble learning.
To maximize the likelihood, most previous studies attempt to promote (cid:80)
Kk =1 p ( y n | x n ; θ k ) (true conﬁdence of M k , for brevity), with ﬁxed model weights on all samples.
To this end, some works train models based on random initialization to reduce prediction variance and generalization errors, while others explicitly quantify diversity to reduce the correlation of errors between models.
However, there is always a problem about the trade-off between diversity and accuracy in this scenario, since the weighted majority correctness among selected models determines the correctness of the ensemble.
One way to avoid the conﬂict between adopting diverse models and boosting the overall performance is to use sample-speciﬁc model weights to aggregate models’ predictions.
Intuitively, if a base model performs much better than any other base models over a speciﬁc sub-region of the dataset containing one or more samples (i.e., a base model specializes in a sub-set of the dataset), it should be assigned with higher weight in this sub-region.
As a result, we deﬁne the specialized model as follows.
Deﬁnition 2.1.
(Specialized model)
We deﬁne the specialized model M k of a sample ( x n , y n ) with two aspects: (1) k = arg max k (cid:48) p ( y n | x n ; θ k (cid:48) ) , and (2) y n = arg max c p ( c | x n ; θ k ) where c ∈ [1 , C ] denotes the class label.
If no model satisﬁes the above conditions, then x n is not specialized by any model.
That is, the specialized model is the one that works correctly and has the highest prediction conﬁdence on a given sample among all the base models.
2. S YNTHETIC ANALYSIS
In this section, we conduct the experiment with three ensemble solutions including a random initialization based method and a diversity-based ensemble method, and then we illustrate how specialization can beneﬁt ensemble.
A 5 × 5 checkerboard synthetic dataset containing 25 Gaussian blobs shown in Figure 1(a) is used for analysis.
Two widely used ensemble methods are selected for this experiment.
Firstly, following the classical random ensemble, base models with different weight initialization are trained independently and aggregated averagely.
On the basis of the random ensemble, we further implement a diversitydriven ensemble by incorporating a diversity loss (Li et al., 2012) into the error function of each base model to encourage base models to perform differently on the same sample.
To illustrate how different base models and their ensembles make predictions, we draw their decision boundaries and graphs showing their individual specialty regions, which for a single model is where it makes the correct prediction.
A multi-layer perceptron (MLP) with one hidden layer and 20 neurons serves as the base model of ensembles through this section.
For simplicity of visualization and analysis, we maintain two base models for each ensemble method.
Random ensemble v.s. diversity-driven ensemble.
The results of these two methods are, respectively, shown in Figure 1(b) and (c).
From random ensemble results in Figure 1(b), the base models are similar in performance and decision boundary thus resulting in their limited ability for ﬁtting the complex distribution with ensemble.
One question may be raised: will enforcing the base learners to predict differently on the same samples by incorporating diversity loss be sufﬁcient to solve the above issue?
From Figure 1(c), base models’ specialty regions for the diversity-driven ensemble are similar to that for the random ensemble, despite the fact that the prediction conﬁdence of the base model diverges signiﬁcantly across different sample regions.
This suggests that diversity may not be sufﬁcient to encourage the base model towards specialized on more samples, thus failing to improve ensemble performance.
We can also observe that ensemble fails on some samples correctly predicted by a base model, for such a model we deﬁne as an unexpressed specialist .
Deﬁnition 2.2.
(Unexpressed specialist)
Given the specialized model M k on a sample ( x n , y n ) , we deﬁne M k as an unexpressed specialist if y n (cid:54) =
arg max c (cid:80) Ki
=
w i p ( c | x n ; θ i ) where c ∈ [1 , C ] .
Specialization-aware ensemble.
In terms of diversity-driven results, the pursuit of diversity does not lead to model specialization.
To verify whether a specialization-driven approach is better than the diversity-driven approach, we further use true conﬁdence of model prediction on a sample (i.e., p ( c = y n | x n ; θ k ) ) to explicitly quantify its specialization and conduct specialization-aware ensemble learning.
As for training, we train the two base models while weighting the classiﬁcation losses of each base model with the normalized true conﬁdence.
As a result, for model inference and ensemble, averaging their predictions only yields an accuracy of 71.0%, whereas an ensemble weighted by the true conﬁdence of the base models yields an accuracy of 100.0%.
It clearly illustrates that, as long as the specialization has been cautiously tackled, specialization-aware ensemble over weak base models will signiﬁcantly improve the ensemble performance.
As shown in Figure 1(d), base models’ specialty regions overlap less, with more samples having a specialized model compared with the ﬁrst two methods.
Both specialization-aware training and output aggregation based on model specialty could contribute to the result that improves the accuracy from 71.0% to 100.0%.
We verify that both factors are equally important by conducting an ablation study on the diversitydriven ensemble, where we assign the ensemble weights in (c) by the true conﬁdences of each base model as we do in (d).
However, the test accuracy is not improved (still 81%) by such a design.
Therefore, ideally, our target is to have an expressed specialist for each sample by training specialized models and aggregating based on their specialty accordingly.
To this end, a problem with the current specialization-driven ensemble is that the base models tend to have overconﬁdence in the samples they are not specialized in (outside their known distributions) (Hein et al., 2019).
These overconﬁdent yet incorrect predictions tend to induce the presence of unexpressed specialists.
Thus, not only do we need the models to be specialized in different specialty regions, but we also want these specialists to have a global perception to avoid overconﬁdence.
As opposed to specialized, we also try to jointly train the sum of base models by directly minimizing the cross entropy between ground truth and the ﬁnal ensemble result 1 K (cid:80)
Kk =1 M k ( · ) .
Surprisingly, from our experiments in Section 4.3 and evidences in other study (Allen-Zhu & Li, 2020), the performance of the trained large model is even worse than that of the single model.
It shows that in this way we can actually give more loss weights in back-propagation for models with higher true conﬁdence, i.e., it leads to specialization of models.
However, without specialization-aware aggregation, the averaged ensemble result turns out to be inferior.
S PECIALIZATION - AWARE N EURAL N ETWORK E NSEMBLE
We propose
Specialization-Aware Neural Network
Ensemble (SANE), an adaptive approach that learns to enforce the base models to specialize in sub-regions of the latent space representing the complex data distribution.
In contrast to previous work on trading-off diversity and performance, our goal is to help the base models develop their own specializations and then aggregate their predictions based on their specialty.
In this section, we will ﬁrst state the solution structure and then present the details of our design.
3. O VERALL FRAMEWORK
We aim to train an ensemble of base models {M k } Kk =1 , each of which is parameterized by θ k and gives its own prediction for the labels y n given input x n .
To aggregate the predictions of the base models, an ensemble network E : R d (cid:55)−→ R K parameterized by ψ predicts the weights of the K base models given the current input.
A =
[ a 1 , . .
. , a K ] is a set of learnable latent parameters of the ensemble network, and (cid:80)
Kk =1 p ( k | x n ; A , ψ ) = 1 .
As a result, the likelihood is written as:
[Equation]
Here, p ( k | x n ; A , ψ ) can be regarded as the predictive conﬁdence of M k being a proper model for x n .
For model M k , its prediction conﬁdence of grounded label p ( y n | x n , k ; θ k ) can be seen as a measure of appropriateness.
3. D ESIGN
The whole network consists of two parts: a pool of base models {M k } Kk =1 and an ensemble network
E .
In this paper, A in ensemble network is deﬁned as a set of anchors in a latent space of the dataset distribution, which will guide the base models towards their own specialty regions in the latent space.
During training, base models are enforced to be more specialized on the vicinity of their corresponding specialty anchors, and are encouraged to be better at what they might already be skilled in.
Ultimately, the whole sample space can be covered by specialty regions of base models, whose specialties are well captured by the specialty anchors, enabling a specializationaware aggregation.
In this section, we detail in turn the design of training diversiﬁed base models, the evolution of model specialization, and specialization-aware model aggregation.
3.2. T RAINING DIVERSIFIED BASE MODELS
For many tasks, it is difﬁcult to train a model to be excellent over the whole sample space.
Therefore, we would like to partition the entire sample space into multiple local regions and train each base model to be a specialist in one region.
Thereafter they can be aggregated together to produce a feasible solution.
In ensemble learning, it is common to partition samples before feeding them into different base models, i.e., to partition them randomly (Breiman, 1996) or according to their original feature space (Zhou et al., 2018).
Either of these two ways would lead to a reduction in the number of samples for training each base model, which affects the performance of the model (especially for deep neural networks).
Moreover, since neural networks can be overconﬁdent in samples outside their known distributions, reducing samples will have a further inﬂuence on the ensemble performance, as discussed in Section 2.
Besides, it is possible that segmenting samples through the raw feature space will not well capture valid information between model specialization and sample characteristics.
In contrast, we propose to locate base models’ specialty regions in the latent space rather than in the raw feature space, since the latent space can provide meaningful representations (Gelada et al., 2019).
In addition, the specialty regions of the models should be explored and learned in an adaptive manner to avoid inappropriate locking (Zhou et al., 2018).
Through exploration, these base models not only focus on local regions, but also gain global perception.
For these reasons, we introduce specialty anchors to guide the traversal of the latent space.
Specialty anchors are learnable representatives of simpler distributions in the latent space, the composition of which can derive a complex distribution to characterize the complexity of real-world data.
Then, we can learn the base models, each of which is specialized in a specialty region represented by the corresponding specialty anchor.
However, one challenge arises when relying on the specialty anchors for ensemble learning: how to capture the relationship between each sample and the specialty anchors to obtain the ensemble weights?
To address this issue, we propose an attention-based method to guide the ensemble network to adaptively weigh the base models.
In addition, we expect the specialty regions of the base models to cover the entire sample space, which may place a high requirement on exploration guided by these specialty anchors.
Therefore, we further resort to the help of explicit specialization measures, which will be detailed in Section 3.2.2.
Explicitly, the guidance is achieved by attentional weighting the model loss w.r.t.
the specialty anchors on each sample.
Given x n , an encoder F computes a nonlinear mapping of x n as e n with the same embedding dimension as A .
To capture the relations between x n and anchors { a k } Kk =1 , we compute the pairwise attention as { Softmax ( Q ( a k )( K ( e n )) (cid:62) √ d h ) V ( e n ) } Kk =1 , where Q , K , V are three projection functions each of which is implemented as one layer MLP and d h is the attention head number.
These attention results are further fed into the scale-transfer layer, and then these K values are processed by softmax operation Softmax ( z ) k
= e z k / (cid:80) Kj =
e z j to produce the weights w =
[ w 1 , . .
. , w K ] ∈ R K corresponding to the K models.
That is, in contrast to hard assignment where one sample is associated with a single model or a subset of models, in our ensemble framework, one sample is associated with all models simultaneously.
The loss of the base models can be written as
[Equation]
where τ b is the temperature parameter, T is the number of iterations, and H is the cross-entropy loss.
3.2. T HE EVOLUTION OF MODEL SPECIALIZATION
Given each base model, we propose a learning-based method to know its specialty region.
As mentioned above, w k is the learned conﬁdence for leveraging the k -th base model to make predictions for each sample.
Also, during training, we can observe the true conﬁdence of each base model for each sample, denoted as o k = p ( y n | x n , k ; θ k ) .
Thus, w k should be close to o k .
In other words, for a pair ( x n , y n ) , the weight on the model with higher true conﬁdence should be greater than the weight on the model with lower true conﬁdence.
To facilitate the training of the ensemble network, we introduce the conﬁdence loss as follows:
[Equation]
This work is not the ﬁrst to favor model specialization.
Zhou et al.
(2018) attempted to assign a subset of samples to each base model based on their current expertise.
However, such design may limit the performance of the base models for two reasons.
First, the specialization of each base model is limited to a set of assigned samples, which prevents each base model from exploring other specialty regions.
Second, it will lead to overﬁtting and overconﬁdent issues, as discussed in
Section 2.
By incorporating specialty anchors and L c , we expect the ensemble network to gradually master predicting whether a base model is potentially specialized in certain samples.
Therefore, in addition to focusing on the current specialty regions of each base model, specialty anchors and L c also help to explore other regions where each base model may potentially be specialized in.
3.2. SPECIALIZATION - AWARE MODEL ENSEMBLE
While the L c only regularizes the normalized true likelihood of base models, ensemble can still fail if all base models’ true conﬁdence is low.
Thus, we further include an ensemble loss to optimize the likelihood of ﬁnal target as follows:
[Equation]
where τ e is the temperate.
It is worth noting that L e does not update the parameters of M k , only the ensemble network.
As mentioned in Section 2, ensemble loss gives more weight to models with higher conﬁdence in back-propagation.
This collides with the effect of L c , which is intended to slowly approach the true conﬁdence.
In Section 4.3, we will perform an ablation study on it.
All three losses L b , L c and L e are combined to accomplish our goal.
For each loss, we assign an independent loss weights s b , s c and s e as hyper-parameters, so that the total loss is given by
[Equation]
E XPERIMENTS
In this section, we seek to answer research questions as follows.
( RQ1 ) Is SANE superior to stateof-the-art ensemble methods, especially compared with diversity-driven methods?
( RQ2 ) Is it better to encourage models to actively specialize at speciﬁc regions than statically assigning sample subsets with diversity regularization?
( RQ3 ) How does each proposed module in SANE affect the ﬁnal performance?
( RQ4 ) Can SANE encourage models to be specialized over different sample regions?
4. D ATASETS AND BASELINES
We brieﬂy describe the benchmark datasets, baselines and the implementation in this section.
For all classiﬁcation tasks we use accuracy to evaluate, and in addition Area under ROC Curve ( AUC ) has also been adopted as the evaluation metric for binary classiﬁcation.
A detailed description of the experimental setup can be found in Appendix A.1.
4.1. T ABULAR DATASETS
Higgs Boson (Baldi et al., 2014) and Blastchar (Kaggle, 2019) are two binary classiﬁcation benchmarks.
Higgs Boson contains 10.5M training samples and 0.5M test samples.
Blastchar has 7,  samples, from which we randomly sample 20.0% of the dataset as the test data.
We split 12.5% of the training samples from both datasets for validation.
Baselines.
Gradient boosting decision trees (GBDT) are widely used in tabular data.
Therefore, we compare two state-of-the-art algorithms, XGBoost (Chen & Guestrin, 2016) and LightGBM (Ke et al., 2017).
We also conduct a comparison with the deep neural network based model TabNet (Arık & Pﬁster, 2020) which is proven to achieve state-of-the-art results on tabular data.
Additionally, we benchmark with random ensemble, in which the base models are trained independently with random initialization and the outputs are averaged for inference
[Table]
[Table] [Table]
Base models and hyper-parameters.
We use MLP, whose implementation is in Appendix A.1.1, as the base model.
For fair comparison, we keep the same hyper-parameters of the base models for random ensemble and our method.
4.1. I MAGE DATASETS
CIFAR.
CIFAR-10 and CIFAR-100 are the standard image classiﬁcation benchmarks (Krizhevsky et al., 2009), containing 50,000 and 10,000 samples, labeled as 10 and 100 classes, respectively.
From the training set, we randomly select 2,500 samples to compose the validation set and apply the standard data augmentation detailed in He et al.
(2016b)
[Table]
Baselines.
We compare SANE with several state-of-the-art ensemble algorithms applied in image datasets, including Snapshot
Ensemble (Huang et al., 2017), Fast Ensemble (Garipov et al., 2018), Multi-Model Ensemble via Adversarial Learning ( MEAL )
(Shen et al., 2019), Diversiﬁed Ensemble (Zhang et al., 2020), and Diverse ensemble evolution ( Div 2 )
(Zhou et al., 2018).
Additionally, we also implement mixture-of-experts ( MOE method ) with reference to Shazeer et al.
(2017) for a soft gating network, see the Appendix A.1.3 for implementation details.
Base models and hyper-parameters.
We use
VGGNet-19 (Simonyan & Zisserman, 2014),
ResNet-18, and ResNet-32 (He et al., 2016a) as the backbone networks for the CIFAR datasets, and follow the standard hyper-parameters.
4. E VALUATION ANALYSIS
We assess the performance of SANE and baselines on tabular and image datasets.
The results are shown in Table 1 and Table 2.
Tabular dataset results.
SANE outperforms all the tree-based methods and the state-of-the-art
TabNet method on both datasets ( RQ1 ) which illustrates the superior performance of our method in tabular datasets with various scales.
Image dataset results.
For the CIFAR dataset, experiments are conducted on several backbone networks.
We can see that SANE is consistently superior to all the baselines ( RQ1 ).
Among all methods using VGGNet-19 on CIFAR-100 data, SANE outperforms the second one (Diverisﬁed
Ensemble) by 2.70%.
That is to say, specialization-driven SANE is better than implicitly and explicitly diversity-driven methods.
Recall that Div 2 assigns a subset of samples to the base models with static expertise.
By comparing its result with that of SANE, we can see that SANE with active specialization on speciﬁc regions in latent space works better ( RQ2 ).
Additionally, SANE brings higher improvement on CIFAR-100 than CIFAR-10, which reveals that it may perform better when the task is relatively more difﬁcult.
The MOE-based approach performs worse than a single model in most cases.
We speculate the reason that MOE uses ensemble loss to update base models and the gating network, resulting in diverse base models while aggregation does not achieve the effect of specialized aggregation.
This result can also correspond to our ablation setting 7 of SANE only using L e , in Section 4.3.
This suggests that L e is not the main component that makes SANE work.
For comparison with papers without publicly available code, here we only present results with the number of base models as K = 3 .
Results for other K values can be found in Appendix A.2.
4. A BLATION STUDIES
[Figure]
To investigate the implications of different components in our model, we design ablation experiments with 8 different settings, which are performed in CIFAR-100 using
VGGNet-19.
Recall that, we adopt three losses L b , L c and
L e that effectively serve our ensemble mechanism, thus we ﬁrst conduct ablation study over them.
Moreover, as mentioned in Section 3.2.3, we also empirically demonstrate the effect of detaching the update of base model parameters { θ k } Kk =1 from L e ’s computation graph (referred to as detach, otherwise attach) on the ensemble performance.
The results are presented in Figure 2, from which it is clear that each loss plays a part ( RQ3 ).
It seems that L b in Eq.
(2) is the most necessary one, as the two settings without it perform worse than the single model.
After discarding L c and
L e , the performance is respectively degraded by 1.04% and 0.33% while still outperforming the baselines.
Additionally, we can compare the performance the base model detachment operation in two different settings (with or without L c ), as 1 4 and 3 6 in Figure 2.
The ensemble loss L e naturally assigns the weights based on the model’s prediction conﬁdence over the true label class in back-propagation.
From the results, the detachment affects the performance in opposite ways for the two different settings.
In setting 4 , the current specialization of the base model is encouraged by L c and L e , but no better ﬁnal results are achieved than that in setting 1 .
In contrast, in the absence of L c , the loss of L e in the ensemble makes the base model more specialized when trained under setting 3 , which brings some performance improvement.
This suggests that ensembles can beneﬁt from specialized base models.
Notably, in addition to assigning weights based on the model’s current specialization, L c encourages the model to explore potential specializations.
By comparing the results of L e -driven model specialization (setting 3 ) with that of our adaptive specialization (setting 1 ), it illustrates that our approach yields superior performance ( RQ2 ).
4. F URTHER ANALYSIS
Analysis of base models’ specialization By design, base models need to specialize in different regions.
It is worth noting that the regions (distributions) in which a model specializes do not correspond to data classes.
Here we only provide an intuitive visualization to answer RQ4 .
Since inter class distribution differences are generally more signiﬁcant than those of intra class, we attempt to analyze base models’ specializations on CIFAR-100 superclasses, by ﬁnding expressed specialists of samples.
Ideally, samples of the same class would be dominantly specialized by one model.
Figure 3 shows the amount of samples (proportional to the connection width) for which the model specialized in that class.
The 3 base models (A, B, and C illustrated on the right side) trained by SANE show divergent specialization on different data classes (on the left side).
The number of parameters added for ensemble network.
The ensemble network can be viewed as a model-agnostic framework.
For different tasks, only the feature dimension and the number of base models will affect the number of parameters.
When using ResNet-18 as the base model, the ensemble network uses only 4.0% of the number of parameters of the single model to control 3-model ensemble, which improves the performance of CIFAR-100 by 5.1%.  R ELATED W ORK
[Figure]
Ensemble method has been proven effective for practical machine learning scenarios (Hansen & Salamon, 1990;
Schapire, 1990; Zhou et al., 2002).
There are mainly two phases in ensemble methods, i.e., base model training and model aggregation, and many works are studying either or both of them with emphasizing diversity.
In the ﬁrst phase, for example, bagging (Breiman, 1996) proposed to randomly select the subsets from the whole training data for training various base models.
However, literature (Lee et al., 2015) shows that the base models in bagging cannot make full use of data yet suppresses the performance of deep neural networks.
The similar issue also occurs in other sample-model matching methods such as
Zhou et al.
Liu et al.
Other approaches relying on random initialization (Zhang et al., 2020; Lakshminarayanan et al., 2016) may derive similar base models lacking diversity, as shown in this paper.
Our method is in contrast to Bayesian neural networks which estimate uncertainty through prior assumption (Gal & Ghahramani, 2016) and rely on randomization.
There also exist some other issues in the second phase of ensemble learning or the joint optimization methods.
Most works adopt averaging over the outputs from all the base models, which has not demonstrated promising performance after ensemble operation (Lee et al., 2015).
Some works combine the two phases and propose to directly learn all the baseline models under the loss upon ensemble (Zhang et al., 2020;
Zhou et al., 2018).
However, Lee et al. (2015) has pointed out that ensemble-aware loss through averaging base model predictions incorporates instability.
Other methods incorporate boosting (Schapire, 1990; Freund, 1995; Moghimi et al., 2016) which focuses on weighted prediction combination through a sequentially training paradigm for deriving a series of base models where each of them is optimized based on the condition derived from the previous ones.
Other sequential optimization methods also include snapshot ensemble (Huang et al., 2017), however, either boosting or snapshot ensemble is hard to parallelize and can lead to long training time when applied to deep neural networks (Zhou et al., 2018).
Our work is also related to mixture-of-experts methods, which have been proposed to tackle the problems of combining some specialized experts to achieve promising ensemble performance (Jacobs et al., 1991; Jordan & Jacobs, 1994;
Shazeer et al., 2017).
However, the existing MOE methods neither consider local patterns explicitly, nor pay enough attention to effective model aggregation.
Our method learns the specialized base models and specialization-aware model aggregation simultaneously, which can achieve superior performance as demonstrated in the experiments.
C ONCLUSION
Real-world data is often generated by some complex distribution, which can be approximated by a composition of several simpler distributions.
Existing ensemble methods that diversify base models are not able to explicitly enforce model specialization.
Therefore, this paper proposes an end-toend ensemble learning method that actively enforces model specialization, where base models are trained to specialize in sub-regions of a latent space representing the simple distribution composition, and aggregated based on their specialty.
Experiments on both image datasets and tabular datasets demonstrate the superior performance of our proposed method.
E THICS STATEMENT
Ensemble learning is a general method for machine learning.
To the best of our knowledge, there are no intentional ethical issues created by the proposed method.
R EPRODUCIBILITY STATEMENT
The datasets used in this paper are all publicly available, and all implementation details are introduced either in the main ﬁle or in the appendix.
In addition, we will release the source code of all the experiments upon paper acceptance
A A PPENDIX A. I MPLEMENTATION DETAILS
In this section, we present the details of the base model design, hyper-parameter setup, and implementation of the MOE method.
For all tasks, we ﬁrst tune the hyper-parameters of the base model and then ﬁx them.
The learning rate of the base model is kept by hyper-parameter tuning of the base model, thus, the loss weight s b in Eq.
(5) is set to 1.
For the ensemble network, we tune its learning rate, s c , and s e .
Additionally, the embedding dimension of specialty anchors is also a hyper-parameter.
Table 3 lists the hyper-parameters to tune and their search space.
For all tasks, we run a hyper-parameter tuning with 100 search steps.
In the ensemble network, the projections Q , K , and V and the scale-transfer layer are implemented as fully connected layers, which are the same for both image and tabular datasets.
The temperature parameters τ b and τ e are set as 0.12 and 0.01, respectively.
[Table]
From the results, SANE works on larger datasets using larger models as well and consistently outperforms baselines.
A.1. I MPLEMENTATION DETAILS FOR TABULAR DATASETS
Base models.
For tabular datasets, we use an MLP as the base model.
For each layer of the MLP, the fully connected layer is followed by a batch normalization layer and a residual connection, using the gated linear unit (GLU) as the activation function.
The feature encoder F in the ensemble network is implemented as MLP of the same structure as the base model, and we take the output of the last fully connected layer as the feature embedding.
For the Higgs Boson dataset, we follow the network structure used by Baldi et al.
(2014), which is a 5-layer MLP with 300 hidden units in each layer.
For Blastchar, we use a 2-layer MLP with 32 hidden units in each layer.
Hyper-parameter setup.
For the Higgs Boson dataset, we follow most of the hyper-parameters in Baldi et al.
(2014), except that the learning rate is modiﬁed to 0.01.
For Blastchar, we use Adam with a learning rate of 0.001 and a weight decay of 0.001.
A.1. I MPLEMENTATION DETAILS FOR IMAGE DATASETS
Base models.
For image datasets, we use several backbones, including VGGNet-19, ResNet-18, and ResNet-32.
Implementations of these backbones are standard.
We use a pre-trained backbone corresponding to the base model and freeze it as the feature encoder F in the ensemble network.
The pre-trained VGGNet-19 and ResNet-18 are taken from TorchVision, an open-source machine vision package for Torch.
Since there is no available pre-trained ResNet-32 available in TorchVision, we use VGGNet-19 as a replacement.
All of these pre-trained model outputs a 1,000-dimensional feature embedding.
Hyper-parameter setup.
On the CIFAR datasets, the standard parameters were applied for these commonly used backbones.
For VGGNet-19 and ResNet-18, we follow the hyper-parameters used in DeVries & Taylor (2017).
For ResNet-32, we use the same hyper-parameters from Chen et al.
To be speciﬁc, for VGGNet-19 and Resnet-18, the initial learning rate was 0.1, and the learning rate was divided by 5 for the 60th, 120th, and 160th epochs, with 200 epochs of training, a batch size of 128, a weight decay of 5e-4, and a Nesterov momentum of 0.9.
For ResNet32, we trained for 300 epochs, with all other parameters unchanged, while the learning rate was divided by 10 at the 150th and 225th epochs.
A.1. I MPLEMENTATION DETAILS OF M IXTURE - OF - EXPERT METHOD
In similar fashion to our framework, MOE consists of a set of expert networks { E i } ni =1 and a gating network G .
For x n , the output of MOE is (cid:80) ni
=1 G ( x n ) i E i ( x n ) .
In Shazeer et al.
(2017), the gating network is realized as G ( x ) =
Softmax ( x · W g ) where W g is a trainable weight matrix.
The training of expert networks and the gating network is done by back-propagation of cross entropy loss between its outputs and ground truth, i.e., analogously to our L e in Eq.
To present a fair comparison with ours, we further strength the gating network as G ( x ) =
Softmax (W( F ( x ))) , where F is the feature encoder and W is an MLP with 2 hidden layers.
A. N UMBER OF BASE MODELS
[Figure]
Figure 4 presents the accuracy of CIFAR-10/100 with different number of base models K .
Here we compare with snapshot ensembles and fast ensembles, which perform worse than our method at any
K value.
The best performance for CIFAR-10 and -100 is obtained when K is equal to 6 and 7, respectively, as seen in the ﬁgures.
B E XPERIMENTAL RESULTS ON T INY I MAGE N ET
In addition to the CIFAR dataset, we also conduct experiments on Tiny ImageNet with more complex data distribution.
Base models.
While previous papers focused on the CIFAR dataset, Snapshot Ensemble conducted experiments on Tiny ImageNet using ResNet-110 as the backbone network, which we followed.
In addition, experiment on EfﬁcientNet-B0, a promising choice on the dataset, is performed using the same hyper-parameters of (Luo et al., 2019).
We compared with Random Ensemble, Snapshot Ensemble
[Table]
and Fast Ensemble
[Table]
and results are as follows.
[Table]
C R ESULTS ON THE CIFAR DATASET COMPARED TO THE SUPPLEMENTAL BASELINES
Apart from baselines listed in Section 4.1.2, Stochastic Weight Averaging (Izmailov et al., 2018) and Hyper-parameter Ensemble (Wenzel et al., 2020) are also STOA methods in ensemble learning.
Due to space limitation
[Table]
we present the results of the comparison experiments in this section.
For Stochastic Weight Averaging
[Table]
we conduct its experiments on CIFAR-10/100 with VGGNetusing settings consistent with our experiments to ensure fair comparison.
For the Hyper-parameter
Ensemble, we re-experiment SANE on ResNet-20 to compare with the results presented in their paper.
The comparison results are as follows:
[Table]
[Table]
From the results, SANE outperforms Stochastic Weight Averaging and Hyper-parameter Ensemble in a fair comparison setting.
D E NSEMBLE OF DIVERSE BASE MODELS
In previous experiments, we focus on the ensemble of base models with the same architecture, since evaluating the effectiveness and generalization of the framework on different data types is the main target.
That is, the only type of randomness that exists in SANE is the different initialization of base models.
Gontijo-Lopes et al.
(2021) claimed that models across different hyper-parameters, architectures, frameworks, and datasets could be specialized for sub-domains of data, leading to higher ensemble performance.
It conﬁrmed the power of specialization in ensemble learning, although they propose to implicitly achieve specialization rather than explicitly as in our method.
To verify the validity of pursuing specialization explicitly, we compare the performance of specialization-aware ensemble on diverse base models with that of specialization achieved by implicit manner, i.e., a random ensemble of diverse architectures.
Moreover, we measure and visualize the specialization of the models trained by the two methods respectively.
[Figure]
Base models and hyper-parameters.
We use three different architectures, i.e., VGGNet-19,
ResNet-18, and DenseNet-40.
All the hyper-parameter setup is the same as in Appendix A.1.2.
From the results, we ﬁng that the SANE-trained ensemble (78.37%) outperform the random ensemble (76.83%) by 1.54%.
It indicates that, under the same network architectures and model size setting, even though architectural diversity can beneﬁt the ensemble by bringing implicit specialization, explicitly encouraging specialization will beneﬁt the ensemble further.
In Gontijo-Lopes et al.
(2021), they deﬁne a specialization metric θ as the angular distance θ in the conﬁdence-conﬁdence graph, as shown in Figure 5(a), and visualize it.
Inspired by that, we show the specialization θ of the two base models (ResNet-18 and DenseNet-40) in Figure 5(b) when only model 1/2 makes a correct prediction, respectively.
We compare with the random ensemble with architecture and initialization diversity and no explicit specialization guidance, and the results are shown in Figure 5(c).
In the ﬁgure, we label the average θ in both cases.
As can be seen, while both ensembles present that ‘when only one model makes the correct prediction, it tends to be more conﬁdent’, SANE’s base model exhibits a higher level of specialization compared to the baseline.
Additionally, though effective, introducing randomness in hyper-parameters, architectures, frameworks, and datasets will require a lot of expert knowledge and manual efforts.
In contrast, our method is a generic framework with the ability to learn to achieve ensemble and explicitly enforce specialization.