Hyperparameter Sensitivity in Deep Outlier Detection Analysis and a Scalable Hyper-Ensemble Solution
Abstract
Outlier detection (OD) literature exhibits numerous algorithms as it applies to diverse domains.
However, given a new detection task, it is unclear how to choose an algorithm to use, nor how to set its hyperparameter(s) (HPs) in unsupervised settings.
HP tuning is an ever-growing problem with the arrival of many new detectors based on deep learning, which usually come with a long list of HPs.
Surprisingly, the issue of model selection in the outlier mining literature has been “the elephant in the room”; a signiﬁcant factor in unlocking the utmost potential of deep methods, yet little said or done to systematically tackle the issue.
In the ﬁrst part of this paper, we conduct the ﬁrst large-scale analysis on the HP sensitivity of deep OD methods, and through more than 35,000 trained models, quantitatively demonstrate that model selection is inevitable.
Next, we design a HP-robust and scalable deep hyper-ensemble model called R OB OD that assembles models with varying HP conﬁgurations, bypassing the choice paralysis.
Importantly, we introduce novel strategies to speed up ensemble training, such as parameter sharing, batch/simultaneous training, and data subsampling, that allow us to train fewer models with fewer parameters.
Extensive experiments on both image and tabular datasets show that R OB OD achieves and retains robust, state-of-the-art detection performance as compared to its modern counterparts, while taking only 2 - 10 % of the time by the naïve hyper-ensemble with independent training.
Introduction
Outlier detection (OD) ﬁnds numerous real-world applications in ﬁnance, security, healthcare, to name a few.
Thanks to this popularity, the literature has grown to offer a large catalog of detection algorithms [1].
With the recent advances in deep learning, the literature has been booming with the addition of many more OD models based on deep neural networks (NNs).
(See surveys [9, 31, 36].)
While there is no shortage of OD methods today, given a new task, it is unclear how to choose which algorithm or model to use, nor how to conﬁgure its hyperparameter(s) (HPs) in unsupervised settings.
That is, the fundamental problem of outlier model selection remains vastly understudied.
Several evaluation studies have illustrated the sensitivity to HPs for traditional (i.e. non-deep) OD methods [2, 8, 17].
Most surprisingly, however, the issue of HP tuning/model selection for the newly burgeoning deep OD models has been “the elephant in the room”; a well-known problem that no one seems to want to bring up, which is exactly the focus of this paper.
Deep OD models are promising thanks to appealing properties such as task-driven representation learning and end-to-end optimization.
On the other hand, while their traditional counterparts had only 1-2 HPs 1 , deep OD models come with a long list of HPs: ( i ) architecture HPs (e.g. depth, width), ( ii ) regularization HPs (e.g. dropout, weight decay rates), and ( iii ) optimization HPs (e.g. learning rate, epochs).
These are inherited ones from regular deep NNs while most also exhibit their model-speciﬁc/specialized HPs.
It would not be a freak occurrence to assume that their performance is heavily dependent on these HP settings.
However, our closer analysis of the experiment testbeds in recent literature on deep OD models falls far from systematically addressing the issue.
(See Appx.
A.1 Table 8 for a preview.)
We ﬁnd hardly any discussion on model selection, with only a few work empirically studying sensitivity but to model-speciﬁc HPs only.
Majority of work report results for a single “recommended” (how, unclear) conﬁguration used for all datasets, or tune only a subset of the
HPs on labeled validation and sometimes even test (!) data.
To the best of our knowledge, there is no existing work that attempts (unsupervised) model selection for deep OD models.
In this work, our research goals are two-fold.
First , through extensive experiments, we quantitatively demonstrate that deep OD models from various families are all sensitive to their HP settings.
Our analysis shows that model selection is inevitable and is key to truly unlock the utmost potential of deep OD models.
Second , motivated by our analysis, we propose a scalable deep hyper-ensemble called R OB OD that obviates HP selection through assembly of deep autoencoders with varying HP conﬁgurations.
To speed up ensemble training, we introduce novel architectural and training strategies, and train fewer models, with fewer parameters, on smaller subsamples of data; by leveraging parameter sharing and joint/simultaneous training.
The main contributions of our work are as follows.
• First large-scale study on HP sensitivity of deep OD models: We build a large testbed to systematically measure the performance variability of deep OD models under varying HP settings.
Our study involves models from four different families, on both image and vector data, under both “clean” (i
e. inlier only) as well as “polluted” (i
e. outlier-contaminated) training data, over 3 random initializations, 80-800+ different HP conﬁgurations per model across 4-8 unique HPs.
Overall, our analysis involves more than 35,000 runs.
(Sec. 3) • R OB OD , a new deep hyper-ensemble OD model:
Motivated by our empirical study, we propose a hyper-ensemble model called R OB OD which combines scores from a collection of models, each trained with a different HP conﬁguration.
Rather than trying to choose, R OB OD fully bypasses the choice of and hence sensitivity to HP settings, and achieves robust (i
e. stable) performance across different initializations.
(Sec. 4.1) • Design strategies to speed up hyper-ensemble training: We propose speed-up strategies to efﬁciently hyper-ensemble model depth and width.
We use an autoencoder (AE) with skip-connections to simultaneously train multiple AEs with different depths.
In addition, we employ batch training of multiple models and use zero-masking on shared parameters to get different widths.
Together, these provide a 90-98 % savings in running time.
(Sec. 4.2) • Extensive experiments: Besides our large-scale measurement study, we also perform experiments on additional benchmark datasets, comparing R OB OD to baseline deep OD models as well as a traditional tree-ensemble.
R OB OD achieves competitive or often better performance which, importantly, exhibits low variance by random initialization.
(Sec. 5)
We expect that our work will increase awareness and help shift the community’s focus (at least to some extent) from building the next yet-another deep OD model toward the fundamental issue of unsupervised model selection and hyperparameter-robust model design.
To foster future research, we open source all code and datasets at https://github.com/xyvivian/ROBOD .
Related Work
Unsupervised Outlier Detection (OD).
There exists a large pool of what-is-now-called traditional, i.e. not deep learning based, OD methods [1, 10].
These methods work with the original features or subspaces thereof, and typically exhibit just one or two hyperparameters (HPs).
1 With recent advances in deep learning, there has been a boom in deep OD models, as those can learn new task-dependent feature representations and directly optimize an OD objective.
Despite their short history, multiple surveys have been published that aim to cover this fast-growing literature [9, 31, 36].
While deep OD models have been shown to outperform their traditional counterparts, they exhibit a much longer list of (typically 4-8) HPs (e.g. depth, width, dropout, weight decay, learning rate, epochs, etc.
besides other model-speciﬁc HPs) that makes them very challenging to tune in unsupervised settings.
Model Selection in Unsupervised OD: Prior (Black) Art.
At large, unsupervised outlier model selection remains to be a vastly understudied, yet extremely important area.
Various evaluation studies have reported traditional detectors to be quite sensitive to their HP choices [2, 8, 17], raising concern for the fair evaluation and comparison of different models.
Earlier work on automatically selecting
HPs are limited to one-class models [14, 40, 41].
More recently, general-purpose internal (i.e., unsupervised) model evaluation heuristics have been proposed [15, 29, 30], which solely rely on the input data (without labels) and the output (i.e., outlier scores).
MetaOD [47] employs meta-learning to transfer information from similar historical tasks to a new task for model selection, which has only been tested on traditional OD models.
Different from those that aim to select a single model, ensemble models have also been employed for OD [3], including those that combine models from the same family [26] as well as heterogeneous detectors from different families [33].
Regarding deep OD methods, we have surveyed a large collection of recent papers and their experimental testbed and HP settings, a summary of which is given in Appx.
A.1 Table 8.
To our surprise, we found hardly any discussion on model selection , with only a few work presenting sensitivity analysis with respect to not all but some, model-speciﬁc HPs.
While some work reserve labeled validation/hold-out data to tune a subset of the HPs [6, 18, 23], majority of them ﬁx the HP values and call them “recommended”/default settings [5, 11, 37, 38, 46, 49].
Moreover, a non-negligible number of existing work choose some critical HPs empirically on test data (!) to yield optimum results [4, 35, 48] (See Table 8, last column).
Some work that builds on previous models (e.g., deep
SVDD-based methods [35] vs. multi-sphere extension [13], transformation-based methods [16] for images vs. their extension to vector data [5], AnoGAN [37] and the follow-up EGBAD [46]) use the same architecture and HP settings as the prior work for consistent/“fair” comparison.
However it is unlikely that the same HP values would work comparably for different models.
Admittedly, it is challenging to tune (a long list of) HPs in the absence of labels, yet, the opacity in the deep OD literature warrants careful investigation on the stability of model performance under varying
HP settings, and ultimately on the fair comparison between these and traditional OD methods.
Deep Model Ensembles.
Recently, deep NN predictions have been found to be often poorly calibrated [20].
As Bayesian learning does not offer straightforward training, deep ensemble models have been proposed as a simple alternative [25] to improve predictive uncertainty, as well as efﬁcient ways of training deep NN ensembles [19, 42].
In this work, we leverage ensemble modeling toward a different goal: to improve the stability and robustness of unsupervised OD models to HP settings, combining predictions from models with different HPs into an OD hyper -ensemble.
The closest to our work is Wenzel et al.
’s
deep hyper-ensemble [43], which, different from ours, considers supervised problems, to further foster diversity in the ensemble and thereby achieve better uncertainty estimation.
Hyperparameter-Sensitivity Analysis of Deep OD 3.1 Testbed Setup
Models.
We study HP sensitivity of ﬁve deep OD methods of four different types: a basic deep autoencoder VanillaAE trained with reconstruction loss, robust deep autoencoder RDA [48], one-class classiﬁcation based DeepSVDD [35], adversarial training based GANomaly [4], and an (AE) ensemble model RandNet [11].
These exhibit 4 to 8 HPs, as listed in Table 1.
(See Appx.
A.2 for descriptions.)
(Note that RandNet is not a hyper -ensemble: members use the same HP conﬁgs except for NN sparsity.)
We deﬁne a grid of 2-3 different values for each HP, including the author-recommended values when available (See details in Appx.
Table 9), and train each deep OD method with all combinations; yielding 81-864 different models.
(Note that 192 RandNet models each consists of an ensemble of 50 or 200 AEs.)
We repeat each experiment 3 times with different initializations.
(See Appx.
A.2 for HP descriptions and Appx.
Table 9 for list of grid values per HP.)
[Table] [Table]
Train/Test settings.
In their original papers, DeepSVDD and GANomaly are trained on what we refer to as Clean (inlier only) data, and tested on a disjoint test dataset.
In contrast, RDA and RandNet consider the transductive setting where the train data is the same as the test data, containing inliers as well as outliers, which we refer to as Polluted.
It is often understood to be more challenging than the
Clean setting for unsupervised OD.
For completeness, we evaluate these methods under both settings.
Datasets.
For evaluation we consider both image point datasets.
As in the original papers [4, 35], we use MNIST and CIFAR10 to construct OD tasks, except for RandNet which uses fully-connected rather than convolutional layers and is originally tested only on point-cloud benchmarks.
Metrics.
Across all HP settings of a deep OD method, we report the mean AUROC, standard deviation (stdev), as well as minimum and maximum.
Mean corresponds to the expected performance when a HP conﬁg is randomly picked from our (multi-dimensional) grid.
We also report the average mean and its stdev over 3 repeated runs.
For datasets that are directly comparable to those in the original papers, we contrast the mean model performance that we obtain to that reported value for author-recommended HPs.
We compare to a simple model (i.e. HP) selection heuristic, which selects the (one) model with the lowest loss/objective value.
In addition, instead of selecting one, we average the scores from all conﬁgurations and report the AUROC of this hyper-ensemble.
Over ﬁve deep OD methods, tens to hundreds of HP conﬁgs, multiple initializations, Clean and
Polluted settings, and various datasets, we have trained a total of more than 35,000 models.
As such, our study constitutes the ﬁrst large-scale HP sensitivity analysis in the deep outlier mining literature.
3.2 Results and Observations
Fig. 1(left) provides the AUROC performances 2 for VanillaAE , RDA , DeepSVDD and GANomaly across all HP conﬁgurations (circles), for the Polluted setting (See Appx.
A.3 Fig.
6 for Clean setting.)
on MNIST-4 dataset where digit ‘4’ images are designated as inliers and the rest nine classes are down-sampled at 10% as outliers, as in the DeepSVDD paper [35].
Horizontal bars mark the 1st, 2nd (i.e. median), and 3rd quartiles, square the mean AUROC across HPs, triangle the selected model by lowest loss, and star the AUROC of the hyper-ensemble.
(Corresponding plots over 3 runs per method are in Appx.
A.3 Fig.
Table 2 provides summary statistics as well as the hyper-ensemble performance for comparison for both the (left) Clean and (right) Polluted settings.
First, in Fig.
1 we observe that all methods exhibit notable variability in performance across HPs, with many worse-than-average conﬁgurations; e.g. for GANomaly , as well as many better-than-average models; e.g. for RDA .
Mean performance is considerably lower than the best model’s, illustrating the opportunity or room for improvement.
As one would expect, the performances in Table 2 are lower in the Polluted setting as compared to Clean (less so for the robust RDA ), while sensitivity (i.e. stdev) is comparable or only slightly higher; showing that HP choice is critical under both settings.
We also ﬁnd that selecting a model by the value of loss ( O ), despite requiring to train and choose from all models, often is much worse than the mean, i.e. random picking a (single) model ( ⇤ ); proving this simple heuristic ineffective.
On the other hand, the hyper-ensemble ( 9 ) outperforms the mean in all cases, with quite small standard deviation across runs (i.e. low sensitivity to initialization).
[Table] [Table]
The reported performance ( 0 . 949 ± 0 . 008 ) of DeepSVDD on MNIST-4 (under Clean) [35] is similar yet somewhat optimistic over the mean value we obtain.
MNIST-4 is an easy task for DeepSVDD since mean AUROC is already around 0 .
Thus, we set up MNIST-5 (with digit ‘5’ as inliers) with reported AUROC 0 .
885 ± 0 .
009 , as well as CIFAR10-auto (with class ‘automobile’ as inliers) (in both cases rest of the classes are subsampled at 10% each as outliers) with reported AUROC 0 .
659 ± 0 . 021 .
We run DeepSVDD on both datasets under all 864 conﬁgurations.
As shown in Fig.(right), our mean AUROCs are 0 .
857 ± 0 .
037 for MNIST-5, and 0 .
605 ± 0 .
024 for CIFAR10-auto.
(See Appx.
A.3 Fig.
8 for all 3 runs.)
The reported performances for the “recommended” HPs 3 in
DeepSVDD (dashed red lines) appear to be optimistic over the mean, i.e. what one would expect by random choice (in the absence of any labels or other strategies).
The optimistic reporting trend holds for GANomaly and RandNet as well.
Following their original paper [4], and different from [35], we set up MNIST-4out dataset to contain digit ‘4’ images this time as the outliers and the rest nine classes as inliers.
GANomaly performances across 162 HP conﬁgs are shown in Fig.
1 (right), where the reported result (red line) in [4] of AUROC 0 .
795 appears notably higher than the mean 0 .
668 ± 0 .
105 that we obtain.
(See Appx.
A.3 Fig.
9 for all 3 runs.)
For RandNet , we ﬁrst ﬁnd and verify a third-party implementation, as the authors could not publicly share theirs, by replicating similar performances to those reported in [11] using the authorrecommended HP settings on all 8 datasets.
Then for Cardio and Thyroid datasets, we train
RandNet ensembles with varying HPs under Polluted as in the original paper.
Results from one run are shown in Fig.
1 (right).
(See Appx.
A.3 Fig.
10 for all 3 runs.)
On Cardio, AUROC mean is 0 .
894 ± 0 .
045 vs. 0 .
929 (reported), and Thyroid mean is 0 .
822 ± 0 .
In summary, the take-aways from our analysis are as follows.
First, it is clear that deep OD models are sensitive to their HP settings, showing that model/HP selection is inevitable.
The mean/expected value (of random choice) can be quite away from the best model, motivating this line of research.
Second, the recommended settings in recent deep OD papers are arguably optimistic; otherwise more transparency into their selection mechanism is warranted.
Finally, hyper-ensemble performance is superior to the mean, possibly owing to different HPs implicitly imposing diversity among constituent models which helps improve detection.
Besides performance improvement, hyper-ensembling obviates model selection by joining all models rather than choosing one and is not much sensitive to initialization – setting the stage for our proposed HP-robust OD method R OB OD.
R OB OD : A Deep Hyper-ensemble for Hyperparameter-Robust OD 4.1 Motivation and Overview
The main research question we consider is RQ1) how to design an unsupervised deep OD model that is robust to its HPs , i.e. an OD method that has stable, low-variance predictions under varying HPs.
Motivated by our sensitivity analysis in Sec. 3, we propose a deep autoencoder (AE) hyper-ensemble model that combines scores from AE models with different HP conﬁgurations.
Deﬁnition 4.1 (Hyper-ensemble)
Given a model family M with H HPs, let  2 R H denote a speciﬁc setting of the HPs.
A hyper-ensemble averages the output (outlier scores) from a ﬁnite number of base models with m different conﬁg.s, i.e. 1 m
P mi =
[Equation]
M ( x ;  i ) for point x .
Hyper-ensembles exhibit multiple advantages.
First, and to our end goal, they ease the model/HP selection burden, bypassing the “choice paralysis”.
They are less sensitive to random initializations,
[Table] [Table]
H y p
e r-
en s .
d e p t h w / A
E - S
[Figure]
[Table]
/gD6/MHnx+VKA==</latexit> = W
[Figure]
[Table]
[Figure]
training with zero-masks : Multiple AE-S models with shared parameters are trained simultaneously, having various widths thanks to zero-masking; e.g. at layer 2, W is shared by all three different-width AE-S models in the batch (denoted f j , f k , f l ), where hidden sizes follow | h j | > | h k | > | h l | through varying size zero-masks on s k and s l .
( sr T depicts an outer product, and  denotes Hadamard/element-wise matrix multiplication.) with lower variance in performance.
Moreover, they can even boost detection performance thanks to the diversity offered by different HPs and ensemble prediction.
The caveat is that deep ensembles are computationally expensive to train.
As such, the second research task we tackle is RQ2) how to speed up hyper-ensemble training .
To that end, we propose novel architectural and training strategies.
In a nutshell, these strategies involve three main ideas: (1) we design a multi-layer AE architecture with skip connections, denoted AE-S , that helps hyper-ensemble, under a single model, varying-depth AEs with shared parameters; (2) we employ batch ensemble training [42] of multiple AE-S models using varying size zero-masking that helps hyper-ensemble varying-width AEs all trained simultaneously with shared parameters; and (3) we train each AE-S on a subsample and use out-of-sample scoring.
In effect, these strategies allow us to build fewer models, with fewer total number of parameters, on less training data—taking only 2 - 10 % of the time that the naïve ensemble training would take where each model is trained independently.
Fig.
2 illustrates the main design elements of R OB OD pictorially.
We present details of the architecture and the training as follows.
4.2 Design Strategies for Speeding up Ensemble Training 4.2.1 Hyper-ensembling depth
One model for multiple
depths
Skip or shortcut connections are applied in NNs for various purposes; e.g., in ResNet [21] for solving the depth degradation problem, in JK-Net [45] for representation learning with varying localities, in
U-Net [34] and stochastic depth NNs [24] for training models with adaptive depth, and so on.
Due to the “hourglass” structure of the AE, skipping one layer in an AE would cause dimensionality mismatch for the next layer.
Instead, we create long shortcuts by skipping the middle layers while keeping the outer encoder-decoder pairs symmetrically, and refer to this architecture as AE-S .
An illustration of hyper-ensembling depth can be found in Fig.
2; see e.g. the AE-S denoted by f j .
Given a 2 L -layer AE-S ( L encoder and L decoder layers), there are 2 L trainable weight matrices.
For input D , we generate L outputs by allowing the input to pass through the outermost encoder-decoder pair (which we denote as AE - 2 ), the two outermost encoder-decoder pairs ( AE - 4 ), and so on, until the full structure ( AE - 2 L ) is traversed by the input.
In effect, this trains L different AEs under one
AE-S model.
Then, the overall loss L AE-S of the depth-hyper-ensembling AE-S is the summation of the member AEs’ reconstruction errors (denoted  Err ).
Eq.
(1) gives an example of the loss function for a 6 -layer AE-S as shown in Fig.
[Equation]
AE-S is an implicit hyper-ensemble which allows simultaneous/joint training of AEs with various depths.
It is computationally efﬁcient thanks to parameter sharing, as the outer layers’ weights are reused and tuned among different members.
Each ensemble member can also play a regularization effect and prevent an AE from producing low scores for outliers due to overﬁtting.
(See Appx.
A.4).
4.2. Hyper
ensembling width: Zero-masked joint training
BatchEnsemble [42] (BE) is a state-of-the-art parameter-efﬁcient deep ensemble training approach.
However, off-the-shelf, it does not allow for hyper- ensebling, that is, all models in the batch are trained with the same HP conﬁguration, and share equal size parameters.
Our approach builds on BE, adapting it to simultaneously train varying-width AE-S with shared parameters.
First, we brieﬂy review the BE architecture with K AEs.
For a speciﬁc layer in BE, weight
W 2 R m ⇥ r is shared across all K individual members, while each member i maintains two trainable rank-1 vectors: s i 2 R m and r i 2 R r .
The outer product ( · ) of r i and s Ti creates a “mask” (a matrix in R m ⇥ r ) onto the shared weight W and generates individual weight W i .
For the input mini-batch X i 2 R n ⇥ m , the forward propagation computation is
[Equation]
where  is the activation function,  is the element-wise product, and r i and s i are broadcasted row-wise or column-wise depending on the shapes at play.
To vectorize the calculation for all K members in the ensemble, two matrices S 2 R K ⇥ m and R 2 R K ⇥ r are constructed with rows of s i and r i , respectively.
Thus, for the input X 2 R Km ⇥ n , the BE layer computes the next layer as
[Equation]
Since X is composed of K mini-batching X
i ’s tiling up, each member can utilize a different minibatch of input.
The K members are training simultaneously in parallel using one forward pass, thus relieving the memory and computational burden of traditional ensemble methods.
While BE itself is not a hyper-ensemble, we can leverage this architecture to aggregate networks of different widths, also in an efﬁcient way.
The widths of a neural network layer correspond to the rows in the weight matrix.
For a speciﬁc layer, instead of directly operating on W , we instead initialize a zero-one masking vector ↵ i 2 { 0 , 1 } r .
↵ i is to element-wise multiply with the r i 2 R r , such that the operation W  s i · ( r Ti  ↵ Ti ) will create a masking matrix of size ( m ⇥ r ) and sparsify the individual weight W i by allowing zero’ed-out rows.
In vectorized notation, the matrix A 2 { 0 , 1 } K ⇥ r is composed of K distinct masking vectors.
Then, the zero-masked BE forward propagation becomes
[Equation]
Sparsifying neural networks has been shown to decrease storage and improve training efﬁciency, with many algorithms built to wisely prune the neural network [22].
While zero-masked BE is similar to creating neural networks of varying density, our main goal is to obtain models with different widths (i.e. hidden sizes), such that when trained under BE, creates a width-hyper-ensemble.
Speciﬁcally, we construct the zero-masked BE layer with the maximum width and specify the zeros in respective ↵ vectors, such that the masked-out individual weights correspond to varying-width models.
The zero-masked BE is efﬁcient as A is ﬁxed throughout training.
The element-wise product of the masking incurs little extra time during forward and backward propagation, while all the (rank-1) vectors are cheap to store compared to separate weight matrices as in traditional ensemble training.
4.2. R OB OD : The overall hyper-ensemble
Let H denote the total number of HPs for an AE-S , where we deﬁne a grid of values for each HP; e.g. depth =[4 , 6 , 8] , width =[64 , 128 , 256] , lrn_rate =[1 e  3 , 5 e  4 ] , drop_out =[0 .
2] , etc.
For the two HPs, depth and width , we specify the AE-S with the largest depth value (following
Sec.
4.2.1, say 2 L ) in the grid and leverage the skip connections to obtain the smaller-depth AEs.
Similarly, we specify each AE-S in the zero-masked BE with the largest width value (following Sec.
4.2.2, say K ) in the grid and leverage the zero-masking to obtain other, smaller-width AEs.
Then, a zero-masked BE trains in parallel K varying-width AE-S models, each being an ensemble of
L varying-depth AEs.
Outlier score for a point x is averaged across all KL AE models as
[Equation]
x
[Equation]
x
[Equation]
where f i is the i ’th AE-S member, f ( AE - 2 d ) i denotes the AE associated with depth 2 d within f i , and  is a vector depicting a speciﬁc conﬁguration of all the remaining ( H  2) HPs; e.g. [ lrn_rate =1 e  3 , drop_out
=0 .
2 , etc. ] .
Denoting the total number such conﬁgurations by B , R OB OD averages scores, i.e. the s ( x ) values, from B different zero-masked BEs as the ﬁnal outlier score of x .
4.2.4 Further speed up by subsampling
As shown in Eq. 2, BE allows mini-batching, where each data point can be used by one or several different ensemble members.
This makes BE a natural ﬁt to subsampling, which further expedites the training procedure.
To this end, we create { X ini , X outi } Ki =1 splits of the training data, where for each AE-S member i , we divide the training data into X ini and X outi =
X i / X ini and solely train on mini-batches from X ini .
We then compute the out-of-sample 4 outlier score of point x as s ( x ) = 1 K 0 L
[Equation]
, where K 0 =
[Equation]
Experiments 5.1 Experimental Setup
Baselines.
We compare to SOTA deep OD methods VanillaAE , RDA [48], DeepSVDD [35] and
RandNet [11], the last of which is a deep AE ensemble.
We also include the tree-ensemble Isolation
Forest [28] ( IF ) which stands as the SOTA among traditional detectors [12].
Besides R OB OD without subsampling, we experiment with two subsampling versions, denoted R OB OD-  , for sampling rates  2 { 0 . 1 , 0 . 5 } .
We also compare to the naïve R OB OD with independent training, denoted i-R OB OD.
Conﬁgurations.
The baselines exhibit 2-8 HPs, per which we deﬁne a small grid of values (See Appx.
A.5 Table 10).
We report the expected AUROC performance, i.e., averaged across all conﬁgurations in the grid, along with the standard deviation.
For R OB OD (and variants) we set L =6 , and K =8 ; the other HP conﬁg.s are listed in Appx.
Table 11.
To measure sensitivity to random initialization, we average performance across 3 runs.
All models are trained on a NVIDIA RTX A6000 GPUs server.
[Table]
Datasets.
We conduct experiments on 5 image datasets from MNIST and CIFAR10, as well as 3 tabular datasets from the ODDS repository.
5 MNIST and CIFAR10 are multi-class, where we pick one class as the inliers and subsample the rest at 10% each to constitute outliers.
Datasets have varying outlier % and size, with images having high dimensionality.
(See Table 3; MNIST-4, -5, -8 depict respective digits as inliers.
CIFAR10-0 and -1 refer to airplane and auto class as inliers, respectively.)
Details on dataset description and preparation can be found in Appx.
A.5.2.
The experiments are all conducted under the Polluted (i.e., transductive) setting.
5.2 Results
With main results shown in Table 4, we want to answer the following questions: Q1.
How does
R OB OD compare to state-of-the-art (SOTA) OD methods?
R OB OD achieves superior performance to all deep OD baselines on MNIST and tabular datasets, and competitive performance on
CIFAR10 datasets against the overall runner-up RandNet .
Notably, R OB OD performs similarly or even better than i-R OB OD, the latter potentially owing to the guarding effect of parameter sharing against overﬁtting.
Moreover, similar performance can be retained under subsampling, where we train each hyper-ensemble member with 50% or even 10% of the data.
Q2.
How much does R OB OD ’s performance vary by initialization?
The standard deviation (stdev) of the deep OD baselines is notably large; it is slightly smaller for the ensemble model RandNet , whereas R OB OD has signiﬁcantly smaller stdev, with near-zero sensitivity to random initialization.
The sensitivity of deep baselines suggests that with an arbitrary choice of HPs in the absence of any other guidance, one may acquire much less satisfactory outcomes than the average AUROC.
[Table]
An interesting observation is regarding the traditional IF baseline, which excels on (low-dim.) tabular datasets, and remains competitive on MNIST-4 and CIFAR10-auto.
It also shows relatively small variance w.r.t.
its (two) HPs.
However, it is signiﬁcantly inferior on other high-dim. image OD tasks, which may be attributed to its lack of representation learning.
Nevertheless, the competitiveness of this simple baseline suggests that traditional OD methods cannot be ignored in the ‘horse-race’ of developing new deep OD methods, not only for their competitiveness but also for their robustness to only-a-few HPs that makes them easy to employ by practitioners.
In Fig.
3 we show the running time (in sec.s) versus the performance for all OD methods (colored symbols) on one image and one tabular dataset for brevity.
(See Appx.
A.6 Fig.
12 for remaining datasets.)
R OB OD achieves higher performance, with lower, near-zero variability compared to deep baselines.
Traditional
IF , based on randomized trees, is very fast, yet it often underperforms on high dimensional tasks such as with images.
Q3. What are the savings in running time compared to naïve hyperensemble w/ independent training?
We study the savings that R OB OD offers compared to the naïve hyper-ensembling.
As shown in Table 5, it runs 3-10 ⇥ faster across datasets.
With subsampling at 10%, R OB OD-0.1 runs in about 1 .
6 - 10 % of what it takes by i-R OB OD.
Fig.
4 (on MNIST-5) shows that while i-R OB OD time increases by the number of models, i.e. larger L and K (ﬁxing other HPs), R OB OD takes near-constant time thanks to the batch/simultaneous training of varying-depth and -width members.
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
Q4.
How sensitive hyper-ensembles are to their own HPs, which include (1) the selected
HP ranges, and (2) the number of sub-models?
The two HP terms are directly related, since expanding/shrinking the HP ranges result in more/fewer sub-models constituting the ensemble.
We ﬁrst answer how HP value ranges affect the accuracy of hyper-ensembles.
With MNIST-4, we expand, shrink or shift the HP ranges for the VanillaAE , which are consitituents of i-R OB OD.
The experimental details are provided in Appx.
A.7.1.
As shown in Table 6, i-R OB OD achieves more stable results across all settings.
Moreover, i-R OB OD produces lower variance with respect to its own HPs than that of the individual model results.
We also study how number of sub-models impact our model.
For the same MNIST-4, DeepSVDD and VanillaAE are selected as the base model for hyper-ensembles, with details in Appx.
A.7.2.
Fig.
[Table]
shows the AUROC corresponding to different number of sub-models have both
Polluted and Clean settings, and for VanillaAE we have Polluted right) setting only).
Observe that AUROC quickly stabilizes and the variance shrinks, as the number of sub-models grows beyond a certain size.
The larger number of sub-models is, the more stable is the performance.
Finally, we conduct the sensitivity analysis to both HP value ranges and the number of sub-models (since they are correlated) for our proposed R OB OD.
We evaluate on the Cardio dataset and provide the details in Appx.
A.7.3.
Table 7 shows that R OB OD yields smaller variance to its own HPs and provides more competitive results than other benchmarked models.
To summarize, our experiments show that larger number of sub-models and expanded HP ranges under ﬁne grids can relax an hyper-ensemble model’s dependency on its own HPs.
[Table]
Conclusion
In this work, we have provided a thorough analysis on the hyperparameter (HP) sensitivitiy of several state-of-the-art deep outlier detection (OD) methods.
Our ﬁndings quantitatively conﬁrm that model selection is vital and advocate efforts in this line of research.
To this end, we introduce R OB OD, a scalable hyper-ensemble OD method that remedies the “choice paralysis” by assembling various autoencoder models of different HP conﬁgurations, hence obviating HP/model selection.
We speed up ensemble training through novel strategies that simultaneously train varying depth and width models under parameter sharing.
Extensive experiments on image and point-cloud datasets show the competitiveness of R OB OD compared to existing OD baselines, while providing consistent results across random initializations.
We hope that our work increases awareness to the unsupervised model selection challenge for the newly booming deep OD literature and motivates future work on hyperparameter-robust model design.
Acknowledgments
This work is sponsored by NSF CAREER 1452425.
We also thank PwC Risk and Regulatory Services
Innovation Center at Carnegie Mellon University.
Any conclusions expressed in this material are those of the author and do not necessarily reﬂect the views, expressed or implied, of the funding parties.