H ETEROGENEOUS L OSS F UNCTION WITH A GGRES -
SIVE
R EJECTION FOR C ONTAMINATED DATA IN ANOMALY DETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
A clean training dataset, which consists of only normal data, is crucial for detecting anomalous data.
However, a clean dataset is challenging to produce in practice.
Here, a heterogeneous loss function with aggressive rejection is proposed, which strengthens robustness against contamination.
Aggressive rejection constrains training on the potential anomalies which is the intersection of normal and abnormal distributions.
Heterogeneous loss function applies a mini-batch stochastic choice of an asymptotic polynomial to a generalized robust loss function, which dynamically optimizes the gradient for the intersection further.
Through the proposed method, mean square error based models can outperform various robust loss functions and generate comparable performance with robust models for contaminated data.
I NTRODUCTION
Identifying outliers or abnormalities in data is known as anomaly detection (AD) (Chandola et al., 2009).
AD assumes that a model is trained on a clean dataset that consists of only normal data so that the output contains normal features.
Autoencoder (AE) (Hinton et al., 2006; Bergmann et al., 2018) is a representative model whose output is generated by an encoder and a decoder.
The encoder converts the input into a latent vector, which is then reconstructed by the decoder as the original one.
Due to the clean dataset, the output of AE takes on normal features.
As a result, the reconstruction error (the difference between input and output) should be close to zero for normal input, whereas high for abnormal input.
However, it is challenging to produce clean datasets because of the ambiguity between normal and abnormal data.
Moreover, the ground truth depends on the individual.
Contaminated data is produced when training data are labeled as normal, although data include both normal and abnormal.
It is essential to make the model robust to contamination because it impairs performance.
To address contamination, a few anomalies were postulated and excluded from the training data (Beggel et al., 2019).
However, it is challenging to access and eliminate overall anomalies due to the normal-like ones.
Both classification and regression have related methods for the problem.
Classification approaches utilized ground truth (Xu et al., 2019; Pleiss et al., 2020) or combined cross-entropy loss function and mean absolute error (MAE)
(Englesson & Azizpour, 2021; Zhang & Sabuncu, 2018).
In the case of regression, robust loss functions were used, such as pseudo-Huber loss (Huber, 1992).
The classification approaches, which rely on ground truth or cross-entropy loss, cannot be extended to the field of AD.
AD can utilize the regression approaches, but they are ineffective since the training data still contains anomalies enough to interrupt the assumption.
In this paper, a novel robust loss function, heterogeneous loss function with aggressive rejection, is proposed to make detectors robust against contaminated data.
Aggressive rejection removes a large amount of data to handle the anomalies as much as possible.
Heterogeneous loss function is based on a general and adaptive robust loss function (GA) (Barron, 2019) that generalizes loss function from mean square error (MSE) to Welsch loss (Dennis Jr & Welsch, 1978).
The proposed loss function dynamically adjusts the gradient to utilize the normal samples as much as possible while impacting less for abnormal samples.
The experiments showed that heterogeneous loss function with aggressive rejection outperformed the existing robust loss functions and models.
In addition, the heterogeneous loss function can be extended to MSE-based AD models, as demonstrated in section 5.3.
R ELATED W ORK
In the field of AD, deep learning has improved performance significantly.
Initially, reconstruction error based models such as AE, variational autoencoder (VAE) (Kingma & Welling, 2013), and adversarial autoencoder (AAE) (Makhzani et al., 2015) were used.
Since AD assumes a clean dataset, the normal input results in a low reconstruction error, whereas the anomaly results in a high reconstruction error.
Memory-augmented autoencoder (MemAE)
(Gong et al., 2019) added a memory module to convert the input latent vector to the most relevant latent vector in memory.
Deep support vector data description (DSVDD) (Ruff et al., 2018) makes the latent vector of the training data close to the center vector.
The score function is measured by the distance between the center vector and the input latent vector.
The performance of AD has been increased by using contrastive loss (Tack et al., 2020; Reiss & Hoshen, 2021), outlier exposure (Hendrycks et al., 2018), and a few ground truths (Ruff et al., 2019).
These methods are sensitive to contaminated data due to the assumption.
Contaminated data is produced by difficulty or mistake in labeling.
The contaminated data makes
AD models difficult to detect anomalies by training both normal and abnormal features.
The problem has already been addressed in the field of classification and regression.
For classification, the mislabeled data are prevented by removing them during or after training or robust loss function.
Pleiss et al.
(2020) identified the mislabeled data based on the entire predictive results generated in the training process.
Englesson & Azizpour (2021) and Zhang & Sabuncu (2018) proposed a loss function that combines MAE, a slow but noise robust loss function, and the opposite crossentropy loss function.
For regression, loss functions such as pseudo-Huber (Huber, 1992), GemanMcClure (Ganan & McClure, 1985), and Cauchy (Black & Anandan, 1996) were employed to reduce the influence of the outliers.
A general robust loss function (Barron, 2019) was proposed, generalizing various robust loss functions.
The classification methods require ground truth or crossentropy, and the regression methods still have anomalies in training data.
Thus, they are not appropriate for solving the contamination problem.
The previous approaches designed robust models or loss functions with data refinement.
Robust variational autoencoder with attention based feature adaptation (RVAE-ABFA) (Gao et al., 2020) is based on deep autoencoding gaussian mixture model (DAGMM)
(Zong et al., 2018).
They achieved robustness by replacing AE with VAE and adjusting the weight between latent vector and reconstruction error via attention based feature adaption.
The strategies for data refinement eliminate samples identified as anomalies (Yoon et al., 2021; G¨ornitz et al., 2014; Xia et al., 2015).
Iterative training set refinement (ITSR) (Beggel et al., 2019) applied one-class support vector machine (Sch ¨ olkopf et al., 1999) to a latent vector of AAE for refinement during training.
Normality-calibrated autoencoder (NCAE) (Yu et al., 2021) generated high-confident normal samples in the low entropy space and utilized them for predicting anomalies.
Latent outlier exposure (LOE) (Qiu et al., 2022) defined normal and abnormal loss functions.
The normal loss function is used for the data discriminated as normal, and the abnormal loss function, such as the reciprocal of the normal loss function, is used for others.
They assumed that high differences between normal and abnormal loss functions are anomalies, and 10% of data are treated as anomalies.
Pseudo-Huber loss was employed by Liznerski et al.
Since AD takes a clean dataset or a few contaminated data, the refinement strategies set a contamination ratio around 10%, which keeps the normal samples and only removes the high-confident anomalies.  M OTIVATION
The previous robust loss functions try to inhibit training on anomalies.
For instance, pseudo-Huber loss (Liznerski et al., 2020) reduces the gradient of high loss, while maintaining the same gradient to MSE for low loss. LOE (Qiu et al., 2022) assumes 10% anomalies in datasets, and trains to maximize the difference between normal and potential anomalies. However, pseudo-Huber loss does not eliminate the potential anomalies and LOE assumes a low contamination ratio. The assumption of a 10% contamination ratio, as shown in Figure 1a, which depicts the distribution of anomaly scores
[Figure]
cannot handle the entire anomalies due to the similar feature between normal and abnormal.
Therefore we introduce a rejection method that suppresses training by setting large potential anomalies, and a heterogeneous loss function to address the problem of the rejection that ignore the degree of intersection through gradient adjustment.
P ROPOSED M ETHOD
In this section heterogeneous loss function and aggressive rejection are described.
The ambiguous data which intersects the normal and abnormal distributions are rejected.
In addition, various loss functions are used for each normal and abnormal sample.
4. R EJECTION
Aggressive Rejection.
The previous approaches roughly set 10% contamination ratio (Beggel et al., 2019; Qiu et al., 2022), thus they process 10% data as anomalies.
However, they cannot handle all the anomalies since abnormal distribution overlaps normal distribution as Figure 1a.
Moreover, the assumption of contamination ratio limits the performance when the contamination ratio is over 10%.
Figure 1b shows that the anomaly ratio decreases as more data are removed.
Based on this concept, aggressive rejection removes the anomalies at the expense of a significant amount of normal samples.
The formulation of aggressive rejection is given as follows:
[Equation]
where x i is the training data, w i is the weight for aggressive rejection, s i is the i -th anomaly score, s q is its q -quantile, and f ( · ) is a model such as AE.
Aggressive rejection removes the data in which an anomaly score (defined for each model, e.g., reconstruction error) is higher than s q .
Since q increases monotonically at 0.5 in Figure 1b, q is set to 0.5.
The experiments on the q setting can be seen in Appendix A.
Soft Rejection.
Although aggressive rejection removes the potential anomalies, it also removes a large amount of normal data, which causes performance degradation, especially on a clean dataset.
To address this problem, the rejection weight w i is adjusted partially by a hyperparameter t s .
As in equation 2, w i depends on t s , where t s = [0 , 1] .
“Hard rejection” in which t s = 0 excludes the rejection target completely, whereas “soft rejection” in which t s = (0 , 1] is trained partially.
It is important to set an appropriate value because a low t s lowers performance on a clean dataset, whereas a high one reduces robustness.
Models achieve minimal loss on a clean dataset and robustness on a contaminated dataset when ts is set to 0.1.
(see Appendix B).
[Equation]
4. H ETEROGENEOUS L OSS
The previous loss functions, such as LOE (Qiu et al., 2022) and reconstruction learning (Xia et al., 2015), minimized the normal loss function and the abnormal loss function to promote the distinction between normal and abnormal distribution.
They assumed a 10% contamination ratio.
Therefore, it affects performance less under a clean dataset even if they minimize the abnormal loss function.
MSE, normal loss function in LOE, can be used for aggressive rejection.
However, the abnormal loss function is not suitable due to the abundance of normal samples in the rejection target.
The loss function should satisfy two conditions to utilize aggressive rejection.
First, it should not minimize the abnormal loss function.
When the models are trained to minimize the loss, the normal samples in the rejection target disturb the training.
Second, the normal samples should be trained with fast convergence, whereas the anomalies should be trained less.
A novel loss function called heterogeneous loss function is proposed, which maximizes the difference between normal and abnormal distribution by modifying the gradient.
The number of normal samples increases as the anomaly score decreases (see Figure 1a).
It is preferable to alter the gradient based on the anomaly score to train more normal and less abnormal samples.
As mentioned before, only the potential anomalies larger than q -quantile score are handled since the others are set to normal and employ MSE loss.
When the anomaly score is close to the 0.5-quantile, heterogeneous loss function turns into MSE.
In contrast, the opposite case has a lower gradient.
The dynamic gradient loss function is applied using GA loss (Barron, 2019).
GA loss is a generalized loss function that covers from MSE to Welsch loss by a parameter α =
[ −∞ , ∞ ] as in equation equation 3 (the formulation is rewritten slightly since the reconstruction error is used as input).
c is the point where various loss functions have similar gradients.
[Equation]
To project the anomaly score to gradient parameter α , modified z-score (Rousseeuw & Croux, 1993) is utilized.
The formulation of the modified z-score is given as follows:
[Equation]
The modified z-score is robust against outliers since it is based on the median.
In addition, the modified z-score generates a normal distribution and has a relative distance.
The modified z-score z is normalized by a maximum between 3.5 and max ( | z | ) , where 3.5 is the outlier threshold.
When m is defined only by max ( | z | ) , the normalized score ranges from 0 to 1, even if the variance is low.
3.5 is used instead of the low max ( | z | ) to make the normalized z close to zero when the variance is low.
As the boundary value, 3.5 is utilized to increase the convergence.
The normalized z is converted to α by the equation 5, where s i is the anomaly score,
z i is the modified z-score, and α r is the parameter for the lowest gradient loss.
The minimum of z i is matched to MSE (normal loss function), and the maximum of z i is closed to the robust loss function, GA loss with α = α r .
Thus, α ranges from α r to 2.
Since α determines the lower boundary of the gradient in the loss function, it is critical to use proper values.
In this paper, alpha is set to 1.5, the median between
MSE and pseudo-Huber (see Appendix C). z is applied in the form of a quadratic function because the anomaly ratio increases as a quadratic function (see Figure 1b).
[Equation]
otherwise
[Equation]
-10c -5c -c0c 5c 10c x 0.
L GA (x 2 ,2,c) 0.1*L
GA (x 2 ,2,c) 0.1*L
GA (x 2 ,1.5,c) 0.1*L GA (x 2 ,1,c) 0.1*L
GA (x 2 ,0,c) 0.1*L
GA (x 2 ,-2,c)
[Figure]
-10c -5c -c0c 5c 10c x 0.
Normal - MSE Low Abnormality - 0.1*
[Figure]
(x 2 ,2,
[Figure]
Figure 2 illustrates the heterogeneous loss function with soft rejection with t s = 0 . 1 and c = √ 0 . The gradient in Figure 2a decreases as the α gets smaller. Figure 2b is an example of heterogeneous loss function with t s = 0 .
1 and α r = 1 . The potential anomalies use the loss function between MSE and pseudo-Huber loss as in the gray area of Figure 2b. The lowest abnormality, where z is close to 0 (0.5-quantile), uses MSE with soft rejection, whereas the largest abnormality, which is at the tail of the z distribution, uses pseudo-Huber loss with soft rejection.
Algorithm 1 summarizes heterogeneous loss function with aggressive rejection.
The samples are discriminated as potential anomalies when the anomaly score of model s is larger than s q , which is q -quantile of s .
For the normal samples, w i is set to 1, whereas the others are set to t s .
The anomaly score in the mini-batch determines the gradient parameters α .
The anomaly score is converted to α based on the modified z-score and input α r .
Finally, the model parameters are updated with the loss generated by multiplying soft rejection weight w and loss function L GA .
Algorithm 1 Training with heterogeneous loss function and aggressive rejection
Input: Sample X , model f , hyperparamter q , t s , α r , c 1: foreach Epoch do 2: foreach Mini - batch x ⊆
X do 3: s = ||
x − f ( x ) ||4: s q = q -quantile of s 5: w = (cid:26) t s if s i >
s q otherwise ▷ Equation6:
z = Modified z-score ( s )
▷ Equation7: m = max (3 . 5 , max ( | z | )) 8: α = (cid:26) 2 − (2 − α r ) ∗
( z i /m )
if z i > otherwise ▷ Equation9:
L = w ∗ L GA ( s , α , c )
▷ Equation10:
Update model parameters with L 11: end for 12: end for
E VALUATION
This section compares existing robust models and loss functions to heterogeneous loss function with aggressive rejection.
Three fundamental image datasets-MNIST (LeCun et al., 2010), FashionMNIST (F-MNIST)
(Xiao et al., 2017), and CIFAR-10 (Krizhevsky, 2009)-are used to evaluate the methods.
MNIST and F-MNIST consist of 10 classes and 28 × 28 gray scale images. CIFAR- [Table]
Hetero 0. is made up of 32 × 32 color images with 10 classes.
For general applicability, the experiments on tabular datasets (Rayana, 2016; Dua & Graff, 2017) are reported in Appendix F.
5. D ATASETS AND SETUPS
One vs. rest setup which set one class as normal and the other classes as abnormal was used in the experiments.
For normal data, training data is twice as much as test data, and 10% of the original training data was used as validation.
γ ct / (1 − γ ct ) ∗ N number of abnormal data were added, where γ ct was the contamination ratio and N was the number of normal data.
30% of the test data consisted of anomalies.
The results without the constraint (using the entire training and test data) are reported in Appendix E. The model with the lowest validation loss was taken as the test model.
The validation loss for the proposed method is measured by aggressive and hard rejection.
The Area Under Receiver Operating Characteristic (AUROC) is used as a metric.
The experiments set each class as normal and measured the average AUROC with three different seeds.
5. COMPARISON METHOD
ITSR (Beggel et al., 2019), RVAE-ABFA (Gao et al., 2020), and NCAE (Yu et al., 2021) are employed as the robust models.
ITSR utilized OC-SVM and AAE for refinement.
RVAE-ABFA developed DAGMM (Zong et al., 2018) by adopting VAE and attention based feature adaption.
NCAE utilizes normal samples generated from generative adversarial model (Goodfellow et al., 2020) to refine the dataset.
The loss functions are evaluated based on three conventional models-AE (Bergmann et al., 2018),
MemAE (Gong et al., 2019), and DSVDD (Ruff et al., 2018).
AE is employed since it is a conventional model.
DSVDD utilizes MSE but is different from reconstruction error.
MemAE is compared due to the additional loss function for memory augmented loss.
MSE is substituted by pseudo-Huber (Liznerski et al., 2020), GA loss (Barron, 2019), and LOE (Qiu et al., 2022).
GA loss was used to demonstrate how much GA loss affects robustness since it utilizes negative loglikelihood to determine α while the proposed loss function utilizes z distribution.
Heterogeneous loss function with aggressive rejection is denoted as Hetero.
The details of experiments such as neural network architectures and batch size are described in Appendix D.
5. E VALUATION WITH ROBUST LOSS FUNCTION
The robust loss functions are used to evaluate the robustness of Hetero loss.
Table 1 shows the
AUROC on clean datasets ( γ ct = 0 %) and contaminated datasets ( γ ct = 0 . 2 )
(the highest AUROC is in bold).
The comparison methods on clean datasets show minimum AUROC loss compared to
MSE except for AE on MNIST.
Hetero loss on clean datasets shows comparable performance to [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
A v e r a g e
A U R O C
AE 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Contamination Ratio 0.
A v e r a g e
A U R O C
MemAE (c) CIFAR0.00 0.05 0.10 0.15 0.20 0.25 0.30 Contamination Ratio 0.
A v e r a g e
A U R O C DSVDD MSE within 0.01. The performance of AE with pseudo-Huber, GA, and LOE does not significantly improve robustness.
However, AE with hetero loss achieves over 0.045 robustness.
When DSVDD on F-MNIST is trained with robust loss functions, it shows the most effective results.
LOE generates a similar performance to Hetero by minimizing the abnormal loss function.
Since CIFAR-10 is a hard dataset compared to MNIST or F-MNIST, the AUROC on the clean dataset is low.
Therefore, robustness has not improved significantly.
Hetero loss surpasses 0.002 0.084 more robustness compared to the overall robust loss functions on 20% contaminated data.
Figure 3 visualizes the AUROC of robust loss functions depending on contamination ratio γ ct .
Hetero loss achieves the most robustness in the case of AE and MemAE.
In the case of DSVDD on
F-MNIST, LOE outperforms hetero when the contamination ratio is under 10%, but hetero loss outperforms LOE when the contamination ratio is over 20%.
According to Qiu et al.
, LOE performs best when the assumed γ ct is equal to the actual γ ct .
LOE achieves robustness by minimizing the abnormal loss function, but the γ ct = 0 .
1 assumption limits the robustness.
Since DSVDD is not a reconstruction error based model, MSE and GA loss show robustness on CIFAR-10 when the contamination ratio is under 15%.
However, hetero loss outperforms the other loss functions when the contamination ratio is 20% and shows a large difference as the contamination ratio increases.
Pseudo-Huber and GA shows insufficient robustness because they simply mitigate the effects of large losses rather than assuming the anomalies and avoiding the training of anomalies.
LOE assumes the 10% contamination ratio and minimizing the abnormal loss function, which improves the robustness, but loses the performance on clean dataset.
It can exceed the hetero loss in some cases like CIFAR-10 with DSVDD, but it can also cause a significant performance decline on clean datasets such as AE on MNIST.
Moreover, when the contamination ratio is above 10%, LOE performs worse.
Hetero loss outperforms the other robust loss functions by handling a lot of anomalies
[Table]
A v
[Table]
r a
[Table]
A U R O
[Figure]
through aggressive rejection and a mini-batch distribution based gradient adjustment.
The experiment demonstrates that hetero loss can be used with a various MSE-based AD models to generate high robustness on contaminated datasets and minimal AUROC loss on clean datasets.
5. E VALUATION WITH ROBUST MODELS
The standard models with Hetero loss are compared to robust models.
The AUROC of comparison on different contamination ratios is shown in Table 2.
NCAE has a low decrease in the AUROC of contamination datasets compared to clean datasets but lacks performance.
ITSR and RVAEABFA perform more robustness than AE and MemAE.
However, hetero loss function improves the robustness of AE and MemAE by about 0.05 compared to ITSR.
DSVDD with Hetero loss shows comparable results toward RVAE-ABFA, which shows the most robust results on MNIST.
Although
RVAE-ABFA shows the highest AUROC compare to the other methods on F-MNIST, DSVDD with hetero loss shows 0.012 higher robustness.
Hetero loss achieves the highest robustness on CIFAR10.
It demonstrates that models with hetero loss can achieve comparable results to robust models.
Figure 4 illustrates the robustness of models on varying contamination ratios.
On MNIST dataset, RVAE-ABFA has the most robustness, but hetero exhibits the most comparable outcomes.
When the contamination ratio is above 20% on F-MNIST dataset, DSVDD with hetero loss outperforms RVAE-ABFA.
In the case of CIFAR-10, DSVDD outperforms the RVAE-ABFA, and hetero makes MemAE comparable to RVAE-ABFA.
Compared to the existing robust models, Hetero loss achieves high robustness without increasing inference time or modifying the architecture.
5. A BLATION STUDY
In this experiment, each component was removed to measure its effect.
The performance on MNIST with clean and contaminated datasets is shown in Table 3 and Figure 5.
q and t s are the key compo- [Table] 0.20 Contamination Ratio 0.
A v e r a g e A U R O C
[Figure]
nents that determine the effect of soft or hard rejection.
The loss function and values are denoted as
L ( q, t s ) , where L is the loss function, q is the q -quantile, and t s is the degree of rejection.
Effect of aggressive rejection.
Hard rejection-MSE(0.5,0) is more robust than the standard lossMSE(1,1) on the contaminated data.
However, hard rejection on a clean dataset shows significant performance degradation of AE (0.115), MemAE (0.134), and DSVDD (0.045).
Since it excludes many normal samples, it has an adverse effect under clean datasets.
Soft rejection-MSE(0.5,0.1) complements the decline.
It shows comparable AUROC to MSE on clean datasets.
Since the normal samples in the exclusion are used for training in part, soft rejection improves robustness for AE,
MemAE, and DSVDD by 0.045, 0.07, and 0.044, respectively.
Effect of heterogeneous loss.
Heterogeneous loss function without aggressive rejection is referred to as Hetero(1,1).
It performs better than the baseline MSE loss and demonstrates greater robustness than MemAE with MSE(0.5,0.1).
Moreover, Hetero(0.5, 0.1), hetero loss with aggressive rejection, shows 0.008 and 0.15 improvements for AE and MemAE compared to MSE(0.5, 0.1).
The experiment demonstrates that the difference between normal and abnormal data can be produced by the gradient adaptation based on the mini-batch distribution.
C ONCLUSION
In this paper, heterogeneous loss function with aggressive rejection is introduced.
Numerous normal data are involved in the rejection target to handle the overall anomalies and excluded partially via soft rejection.
In addition, the mini-batch distribution is transformed into the gradient of general robust loss to suppress the convergence of suspicious outliers.
The loss function outperforms the previous robust loss function and shows comparable results to robust models.
The experiments proved that the MSE-based models could strengthen robustness against contaminated data by the proposed loss function.
In the future, we will utilize the minimization of abnormal loss function to promote the difference between normal and abnormal samples.
To complement the minimization, we will apply the dynamic q function based on the abnormality in distribution.
A Q UANTILE IN AGGRESSIVE REJECTION
Table 4 and Figure 6 show the effect of q -quantile on MNIST dataset. Since hetero loss is based on the z-score and is optimum when q = 0 .
5 , MSE with soft rejection ( t s = 0 . 1 ) is employed instead of hetero loss.
25 -quantile performs better than 0 .
75 -quantile except for AE.
Although q = 0 .includes more anomalies in the exclusion, 0.5-quantile rejection performs the best. The tendency is maintained even the contamination ratio increases (see Figure 6).
It indicates that over-elimination is not always ensured robustness
[Table]
[Table] [Figure] B S TUDIES OF SOFT REJECTION
Sensitivity Figure 5 illustrates the sensitivity of MSE and hetero loss with soft rejection.
As t s increases, their performance decreases inversely.
However, hetero loss reduces the sensitivity of t s .
When t s = 1 , the soft rejection becomes a non-robust loss function MSE.
Therefore, soft rejection becomes less effective as t s increases.
In contrast to MSE, hetero loss which is still robust due to gradient adaptation exhibits less difference between t s = 0 . 1 and t s = 0 . 5 on 20% contaminated data. [Figure] U R O
C
MSE(γ ct =0)
MSE(γ ct =0.1)
MSE(γ ct =0.2)
Hetero(γ ct =0)
Hetero(γ ct =0.1)
Hetero(γ ct =0.2)
Optimizing AE, MemAE, and DSVDD with hetero loss are trained on clean MNIST ( γ ct = 0 ) to optimize the t s .
Except for t s , the parameters are utilized as previously ( q = 0 .
5 , α r = 1 . 5 and c = √ 0 . 5 ).
The dashed line means the standard model with only MSE.
AUROC is saturated from 0.05 for AE and 0.1 for MemAE.
DSVDD is unstable but similar to MSE when t s is 0.1.
As a result, t s is set to 0.1 generally, considering minimum performance loss and lowest value.
[Figure]
A v e r a g e
A U R O
[Figure]
C O PTIMIZING A P ARAMETER OF H ETEROGENEOUS L OSS
A variety of robust loss functions can be used as the lowest gradient loss function in hetero loss.
The pseudo-Huber ( α = 1 ), Cauchy ( α = 0 ), and Geman-McClure ( α = − 2 ) are the representative loss functions.
Figure 9 illustrates the effect of α r in hetero loss.
The loss function should enhance robustness on contaminated data while minimizing performance degradation on a clean dataset.
When α is adjusted below 1, hetero loss degrades on the clean dataset.
α = 1 shows insufficient performance on the clean dataset, whereas α = 2 shows lower robustness.
Therefore, the intermediate point α = 1 . 5 is used since it achieves comparable performance on clean and contaminated data.
0.20 Contamination Ratio 0.
A v e r a g e A U R
O C
MSE Hetero ( α r =2) Hetero ( α r =1.5)
Hetero ( α r =1) Hetero ( α r =0)
Hetero ( α r = −2) (
[Figure]
0.20 Contamination Ratio 0.
A v e r a g e A U R
O C
MSE Hetero ( α r =1.5) Hetero ( α r =1)
Hetero ( α r =0) Hetero ( α r = −2)
[Figure]
D D ETAILS ON E XPERIMENTS
Setup Batch size and epochs are set to 100 and 300, respectively, except for ITSR, NCAE, and
DSVDD.
ITSR refines data for every 10 epochs after the first 100 epochs and is trained on refined data with 100 epochs.
NCAE uses 150 epochs and DSVDD uses 150 epochs for pre-training and 100 epochs for the rest.
The parameters are updated by Adam optimizer (Kingma & Ba, 2014) with 0.0001 learning rate and 10 − 6 weight decay.
As mentioned before, the parameter q , α r , and c in
Hetero loss are set to 0.5, 1.5, and √ 0 .
AE/MemAE/ITSR
The architectures of AE are based on the report in Gong et al.
On
MNIST and FashionMNIST datasets, the encoder consist of three convolution modules that consists of convolution, batch normalization (Ioffe & Szegedy, 2015), and leaky ReLU activation (Xu et al., 2015) with 16-32-64 filters that kernel and stride size are 3 and 2.
On CIFAR-10 dataset, the encoder consists of four convolution modules, 64-128-128-256 filters that kernel and stride size are 3 and 2.
DSVDD/RVAE-ABFA/NCAE
The architectures of autoencoder in DSVDD, RVAE-ABFA, and
NCAE are based on the report in Ruff et al.
On MNIST and FashionMNIST datasets, the encoder consists of two convolutions (8×5×5-filters and 4×5×5-filters) and a final fully-connected layers of 32 units.
The batch normalization, leaky ReLU, and (2×2)-max-pooling are followed by the convolutions.
On CIFAR-10 dataset, the encoder consists of three convolutions (32×5×5-filters, 64×5×5-filters, and 128×5×5-filters) and a final fully-connected layer of 128 units (except for RVAEABFA).
RVAE-ABFA uses only 32 dimensions due to the computation error.
For RVAE-ABFA, the encoder has an additional fully-connected layer for mean and variance of the latent distribution.
The bias of layers in DSVDD and NCAE is eliminated due to the trivial solutions as reported in Ruff et al.
The decoder is symmetric to the encoder, in which convolution is substituted by deconvolutions, and max-pooling is substituted by up-sampling.
The last deconvolution has no additional operations such as batch normalization.
The results on MNIST and F-MNIST with overall data can be seen in Table 5.
All normal data in training data are used.
90% of them are used for training and the remainder for validation.
As mentioned previously, γ ct / (1 − γ ct ) ∗ N number of anomalies are added, where γ ct is the contamination ratio and N is the number of normal data.
The evaluation makes use of all 10,000 test data.
For NCAE, both the reported results by Yu et al.
and the results reproduced using the author’s code are shown in the table.
The loss function with the highest AUROC is underlined, and the highest AUROC in the comparison is in bold on every dataset.
With the exception of MemAE on MNIST, the existing robust loss functions are insufficient to improve the robustness of AE and MemAE.
Comparing DSVDD with hetero loss to MSE, the robustness is increased by 0.043 and 0.048 on each dataset.
In the case of DSVDD, LOE decreases performance on clean MNIST and exhibits robustness values that are higher than hetero loss on FMNIST.
RVAE-ABFA and reproduced NCAE achieve greater robustness than in section 5.4 with the entire MNIST data usage.
RVAE-ABFA exhibits the best robustness on MNIST (0.918), whereas
NCAE, according to reports, exhibits the highest AUROC (0.889) on the 20% contaminated FMNIST.
Among AE, MemeAE, and DSVDD with robust loss functions, DSVDD with hetero loss generally performs best.
DSVDD with hetero loss has a large difference (0.06) for MNIST and a low difference (0.028) for F-MNIST when compared to RVAE-ABFA and NCAE, which have the best performance on contaminated data.
Since LOE performs better on F-MNIST, the minimization of the abnormal loss function can improve hetero loss when it does not impair the proposed loss function. [Table] [Table] O
[Table] [Table]
Four tabular datasets are used for general applicability.
The information about the datasets is shown in Table 6.
The following provides configuration details.
Arrhythmia Arrhythmia dataset can be obtained from Rayana (2016).
Minority classes (3, 4, 5, 7, 8, 9, 14, and 15) are defined as abnormal class, whereas the others are defined as normal class.
Three fully connected layers with 128-64-32 units are employed for autoencoder architecture.
The layers except for the final layer in encoder and decoder are followed by batch normalization and leaky ReLU activation.
The batch size is set to 64.
Thyroid Thyroid dataset also can be obtained from Rayana (2016).
There are three classes and a minority class (hyperfunction) is defined as an abnormal class.
The architecture for Thyroid has a similar architecture that was used on Arrhythmia dataset but has 32-16-4 units.
The batch size is set to 512.
KDDCUP KDDCUP99 10 percent dataset is taken from Dua & Graff (2017).
Originally, there were 41 dimensions, where 34 of which were continuous and the rest were categorical.
Since the categorical features are transformed by one-hot encoding, the final dimensions become 120.
Although 20% of the data is labeled as “normal”, the “normal” class is defined as anomalies since the normal class is the minority class.
The architecture for KDDCUP has three fully connected layers (32-16-8 units) with batch normalization and ReLU activation.
The batch size is set to 4096, and AE and MemAE are trained with 100 epochs.
KDDCUP-Rev KDDCUP-Rev is the reverse version of KDDCUP.
“normal” class is defined as normal and the others are defined as anomalies.
The dataset contains all of the normal samples, and anomalies are randomly sampled with a 4:1 ratio of normal to abnormal samples.
The configurations of KDDCUP-Rev were the same as those of KDDCUP.
Half of the normal samples are used for training.
Artificial anomalies for contaminated training data are constructed by adding zero-mean Gaussian noise to test anomalies, as in Shenkar & Wolf (2021);
Qiu et al.
(2022) since the datasets have a few anomalies.
The standard deviations are derived from test anomalies.
The last trained models are used in the evaluation.
MSE, pseudo-Huber, GA loss, and LOE are employed to evaluate hetero loss.
The rest of the configurations such as learning rate are the same as those used on image datasets.
F. E VALUATION
Figure 10 reports the average AUROC with 10 different seeds on each dataset.
Although hetero loss is less robust than the other losses on the Arrhythmia and KDDCUP datasets, it outperforms when the contamination ratio is greater than 20%.
In the case of Thyroid dataset, AE and DSVDD had a similar tendency to Arrhythmia and KDDCUP.
Regardless of the contamination ratio, MemAE delivers the best performance.
Hetero loss resulted in the best performance on KDDCUP-Rev except for DSVDD with a 30% contamination ratio.
With a low contamination ratio, heterogeneous loss has a comparable AUROC to other robust losses, which describes that the loss can effectively make a difference in training speed between normal and abnormal.
With a high contamination ratio, hetero loss achieves significant improvements through aggressive rejection which half of the training data handled as potential anomalies.
The experiments show that hetero loss can be employed broadly and in other domains.
[Table]
A v e r a g e
A U R O C
AE 0.
0.4 Contamination Ratio 0.86 0.
A v e r a g e
A U R O C
MemAE (c) KDDCUP 0.
0.4 Contamination Ratio 0.
A v e r a g e
A U R O C
DSVDD 0.
0.4 Contamination Ratio 0.
A v e r a g e
A U R O C
AE 0.
0.4 Contamination Ratio 0.
A v e r a g e
A U R O C
MemAE (d) KDDCUP-Rev 0.
0.4 Contamination Ratio 0.
A v e r a g e
A U R O C
DSVDD