H ETEROGENEOUS L OSS F UNCTION WITH A GGRES - SIVE R EJECTION FOR C ONTAMINATED DATA IN ANOMALY DETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
A training clean dataset, which consists of only normal data, is crucial for detecting anomalous data.
However, a clean dataset is challenging to produce in practice.
Here, heterogeneous loss function with aggressive rejection is proposed, which strengthens robustness against contamination.
Aggressive rejection constrains training on the intersection of normal and abnormal distributions to handle the potential anomalies.
Heterogeneous loss function utilizes an adaptive minibatch stochastic choice of an order of asymptotic polynomial of GA loss, which dynamically optimizes the gradient for the intersection further.
Through the proposed method, mean square error based models can outperform various robust loss functions and generate comparable performance with robust models for contaminated data on three image datasets.
I NTRODUCTION
Identifying outliers or abnormalities in data is known as anomaly detection (AD) (Chandola et al., 2009).
AD assumes that a model is trained on a clean dataset that consists of only normal data so that the output contains normal features.
Autoencoder (AE) (Hinton et al., 2006; Bergmann et al., 2018) is a representative model whose output is generated by an encoder and a decoder.
The encoder converts the input into a latent vector, which is then reconstructed by the decoder as the original one.
Due to the clean dataset, the output of AE takes on normal features.
As a result, the reconstruction error (the difference between input and output) should be close to zero for normal input, whereas high for abnormal input.
However, it is challenging to produce clean datasets because of the ambiguity between normal and abnormal data.
Moreover, the ground truth depends on the individual. Contaminated data is produced when training data are labeled as normal, although data include both normal and abnormal.
It is essential to make the model robust to contamination because it impairs performance.
To address contamination, a few anomalies were postulated and excluded in the training data (Beggel et al., 2019).
However, it is challenging to access and eliminate overall anomalies due to the normallike ones.
Both classification and regression have related methods for the contamination problem.
Classification approaches utilized ground truth (Xu et al., 2019; Pleiss et al., 2020) or combined cross-entropy loss function and mean absolute error (MAE) (Englesson & Azizpour, 2021; Zhang & Sabuncu, 2018).
In the case of regression, robust loss functions were used, such as pseudo-Huber loss (Huber, 1992).
The classification approaches, which rely on ground truth or cross-entropy loss, cannot be extended to the field of AD.
AD can utilize the regression approaches, but they are ineffective since the training data still contains anomalies enough to interrupt the assumption.
In this paper, a novel robust loss function, heterogeneous loss function with aggressive rejection, is proposed to make detectors robust against contaminated data.
Aggressive rejection removes a large amount of data to handle the anomalies as much as possible.
Heterogeneous loss function is based on
GA loss function (Barron, 2019), generalized loss function from mean square error (MSE) to Welsch loss (Dennis Jr & Welsch, 1978). The proposed loss function dynamically adjusts the gradient to utilize the normal samples as much as possible while impacting less for abnormal samples.
The experiments showed significant improvement that heterogeneous loss function with aggressive rejection outperformed the existing robust loss functions and models.
In addition, the heterogeneous loss function can be extended to MSE-based AD models, as demonstrated in section 4.3.
R ELATED W ORK
In the field of AD, deep learning has improved performance significantly.
Initially, reconstruction error based models such as AE, variational autoencoder (VAE) (Kingma & Welling, 2013), and adversarial autoencoder (AAE) (Makhzani et al., 2015) were used.
Since AD assumes a clean dataset, the normal input results in a low reconstruction error, whereas the anomaly results in a high reconstruction error.
Memory-augmented autoencoder (MemAE) (Gong et al., 2019) added a memory module to convert the input latent vector to the most relevant latent vector in memory.
Deep support vector data description (DSVDD) (Ruff et al., 2018) makes the latent vector of the training data close to the center vector.
The score function is measured by the distance between the center vector and the input latent vector.
The performance of AD has been increased by using contrastive loss (Tack et al., 2020; Reiss & Hoshen, 2021), outlier exposure (Hendrycks et al., 2018), and a few ground truths (Ruff et al., 2019).
These methods are sensitive to contaminated data due to the assumption.
Contaminated data is produced by difficulty or mistake in labeling.
The contaminated data makes AD models difficult to detect anomalies by training both normal and abnormal features.
The problem has already been addressed in the field of classification and regression.
For classification, the mislabeled data are prevented by removing them during or after training or robust loss function.
Pleiss et al.
(2020) identified the mislabeled data based on the entire predictive results generated in the training process.
Englesson & Azizpour (2021) and Zhang & Sabuncu (2018) proposed a loss function that combines MAE, a slow but noise robust loss function, and the opposite crossentropy loss function.
For regression, loss functions such as pseudo-Huber (Huber, 1992), GemanMcClure (Ganan & McClure, 1985), and Cauchy (Black & Anandan, 1996) were employed to reduce the influence of the outliers.
A general robust loss function (Barron, 2019) was proposed, generalizing various robust loss functions.
The classification methods require ground truth or crossentropy, and the regression methods still have anomalies in training data.
Thus, they are not appropriate for solving the contamination problem.
The previous approaches designed robust models or loss functions with data refinement to address the contamination problem.
Robust variational autoencoder with attention based feature adaptation (RVAE-ABFA) (Gao et al., 2020) is a robust AD model based on deep autoencoding gaussian mixture model (DAGMM) (Zong et al., 2018).
They achieved robustness through replacing AE with VAE and adjusting the weight between latent vector and reconstruction error via attention based feature adaption.
The strategies for data refinement eliminate samples identified as anomalies (Yoon et al., 2021; G ¨ ornitz et al., 2014; Xia et al., 2015).
Iterative training set refinement (ITSR) (Beggel et al., 2019) applied one-class support vector machine (Sch ¨ olkopf et al., 1999) to a latent vector of AAE for refinement during training.
Normality-calibrated autoencoder (NCAE) (Yu et al., 2021) generated high-confident normal samples in the low entropy space and utilized them for predicting anomalies.
Latent outlier exposure (LOE) (Qiu et al., 2022) defined normal and abnormal loss functions. The normal loss function is used for the data discriminated as normal, and the abnormal loss function, such as the reciprocal of the normal loss function, is used for others.
They assumed that high differences between normal and abnormal loss functions are anomalies and maximized the abnormal loss function to encourage the difference between the normal and abnormal distribution.
Pseudo-Huber loss was employed by Liznerski et al.
Since AD takes a clean dataset or a few contaminated data, the refinement strategies set a contamination ratio around 10%, which keeps the normal samples and only removes the high-confident anomalies.
P ROPOSED M ETHOD
In this section, heterogeneous loss function and aggressive rejection are described.
The ambiguous data which intersects the normal and abnormal distributions are rejected.
In addition, various loss functions are used for each normal and abnormal sample. [Figure] Although the samples that are over 0. quantile are more normal than the 0.9-quantile
, its anomaly ratio is about 0.034 (3.4%).
3. R EJECTION
Aggressive Rejection.
The previous approaches set the contamination ratio as roughly 10% (Beggel et al., 2019; Qiu et al., 2022), thus they process 10% data as anomalies.
However, they cannot handle all the anomalies since abnormal distribution overlaps normal distribution. Moreover, the assumption of contamination ratio limits the performance when the contamination ratio is over 10%.
Based on this concept, aggressive rejection removes the anomalies at the expense of a significant amount of normal samples.
Figure 1 shows the anomaly score distribution of 10% contaminated data and anomaly ratio for different quantiles. Figure 1a illustrates that the normal and abnormal distributions are overlapped.
Therefore, the contamination ratio decreases as more data are removed like figure 1b.
[Equation]
where x i is the training data, w i is the weight for aggressive rejection, s i is the i -th anomaly score, s q is its q -quantile, and f ( · ) is a model such as AE.
Aggressive rejection removes the data in which an anomaly score (defined for each model, e.g., reconstruction error) is higher than s q .
Since q increases monotonically at 0.5 in figure 1b, q is set to 0.5. The experiments on the q setting can be seen in Appendix A.
Soft Rejection.
Although aggressive rejection reduces the contamination ratio, it degrades performance by eliminating a large amount of normal data.
It results in the most degradation on a clean dataset.
To address this problem, the rejection weight w i is adjusted partially by a hyperparameter t s .
As in equation 2, w i depends on t s , where t s = [0 , 1] .
“Hard rejection” in which t s = 0 excludes the rejection target completely, whereas “soft rejection” in which t s = (0 , 1] is trained partially with the samples in the rejection target.
It is important to set an appropriate value because a low t s lowers performance on a clean dataset, whereas a high one reduces robustness.
Models achieves minimal loss on a clean dataset and robustness on a contaminated dataset when ts is set to 0.1. (see
Appendix B).
[Equation]
3. H ETEROGENEOUS L OSS
The previous loss functions, such as LOE (Qiu et al., 2022) and reconstruction learning (Xia et al., 2015), minimized the normal loss function and maximized the abnormal loss function to promote the distinction between normal and abnormal distribution.
They assumed a 10% contamination ratio. Therefore, it affects performance less under a clean dataset even if they maximize the abnormal loss function.
MSE, normal loss function in LOE, can be used for aggressive rejection.
However, the abnormal loss function is not suitable due to the abundance of normal samples in the rejection target.
The loss function should satisfy two conditions to utilize aggressive rejection.
First, it should not maximize the abnormal loss function.
When the models are trained to maximize the loss, the normal samples in the rejection target disturb the training. Second, the normal samples should be trained with fast convergence, whereas the anomalies should be trained less.
A novel loss function called heterogeneous loss function is proposed, which maximizes the difference between normal and abnormal distribution by modifying the gradient.
The number of normal samples increases as the anomaly score decreases (see Figure 1a).
It is preferable to alter the gradient based on the anomaly score to train more normal and less abnormal samples.
As mentioned before, only the potential anomalies larger than q -quantile score are handled since the others are set to normal and employ MSE loss. When the anomaly score is close to the 0.5-quantile, heterogeneous loss function turns into MSE.
In contrast, the opposite case has a lower gradient. The dynamic gradient loss function is applied using GA loss (Barron, 2019).
GA loss is a generalized loss function that covers from MSE to Welsch loss by a parameter α = [ −∞ , ∞ ] as in equation equation 3 (the formulation is rewritten slightly since the reconstruction error is used as input).
c is the point where various loss functions have similar gradients.
[Equation]
To project the anomaly score to gradient parameter α , modified z-score (Rousseeuw & Croux, 1993) is utilized.
The formulation of the modified z-score is given as follows:
[Equation]
The modified z-score is robust against outliers since it is based on the median.
In addition, the modified z-score generates a normal distribution and has relative distance.
The modified z-score z is normalized by a maximum between 3.5 and max ( | z | ) , where 3.5 is the outlier threshold. When m is defined only by max ( | z | ) , the normalized score ranges from 0 to 1, even if the variance is low. 3. is used instead of the low max ( | z | ) to make the normalized z close to zero when the variance is low.
As the boundary value, 3.5 is utilized to increase the convergence. The normalized z is converted to α by the equation 5, where s i is the anomaly score,
z i is modified z-score, and α r is the parameter for the lowest gradient loss.
The minimum of z i is matched to MSE (normal loss function), and the maximum of z i is closed to the robust loss function, GA loss with α
= α r .
Thus, α ranges from α r to 2. Since the anomaly ratio increases as a quadratic function (see Figure 1b), z is applied in the form of a quadratic function. [Equation] otherwise
[Equation]
Figure 2 illustrates the heterogeneous loss function with soft rejection with t s = 0 . 1 and c = √ 0 . The gradient in figure 2a decreases as the α gets smaller. Figure 2b is an example of heterogeneous loss function with t s = 0 .
1 and α r = 1 .
The potential anomalies use the loss function between MSE and pseudo-Huber loss as in the gray area of figure 2b.
The lowest abnormality, where z is close to 0 (0.5-quantile), uses MSE with soft rejection, whereas the largest abnormality, which is at the tail of the z distribution, uses pseudo-Huber loss with soft rejection.
-10c -5c -c0c 5c 10c x 0.
L GA (x 2 ,2,c) 0.1*L
GA (x 2 ,2,c) 0.1*L
GA (x 2 ,1.5,c) 0.1*L GA (x 2 ,1,c) 0.1*L
GA (x 2 ,0,c) 0.1*L
GA (x 2 ,-2,c)
[Figure] -10c -5c -c0c 5c 10c x 0. Normal - MSE Low Abnormality - 0.1*L GA (x 2 ,2,c) High Abnormalty - 0.1*L GA (x
[Figure]
[Figure]
c) [Figure] Algorithm 1 summarizes heterogeneous loss function with aggressive rejection. The samples are discriminated as potential anomalies when the anomaly score of model s is larger than s q , which is q -quantile of s .
For the normal samples, w i is set to 1, whereas the others are set to t s .
The anomaly score in the mini-batch determines the gradient parameters α .
The anomaly score is converted to α based on the modified z-score and input α r .
Finally, the model parameters are updated with the loss generated by multiplying soft rejection weight w and loss function L GA . Input: Sample X , model f , hyperparamter q , t s , α r , c 1: foreach Epoch do 2: foreach Mini - batch X do 3: s = || x − f ( x ) ||4: s q = q -quantile of s 5: w = (cid:26) t s if s i >
s q otherwise ▷ Equation6:
z = Modified z-score ( s )
▷ Equation7: m = max (3 . 5 , max ( | z | )) 8: α = (cid:26) 2 − (2 − α r ) ∗ ( z i /m ) if z i > otherwise ▷ Equation9: L = w ∗ L GA ( s , α , c )
▷ Equation10: Update model parameters with L 11: end for 12: end for  E VALUATION This section compares existing robust models and loss functions to heterogeneous loss function with aggressive rejection.
Three fundamental image datasets-MNIST (LeCun et al., 2010), FashionMNIST (F-MNIST)
(Xiao et al., 2017), and CIFAR-10 (Krizhevsky, 2009)-are used to evaluate the methods.
MNIST and F-MNIST consist of 10 classes and 28 × 28 gray scale images.
CIFAR-10 is made up of 32 × 32 color images with 10 classes.
4. D ATASETS AND SETUPS
One vs. rest setup which set one class as normal and the other classes as abnormal was used in the experiments.
For normal data, the training data is twice as much as test data, and 10% of the original training data was used as validation.
γ ct / (1 − γ ct ) ∗ N number of abnormal data were added, where γ ct was the contamination ratio and N was the number of normal data.
30% of the test data consisted of anomalies.
The result using the entire training data is reported in Appendix D. The model with the lowest validation loss was taken as the test model.
The validation loss for the proposed method is measured by aggressive and hard rejection.
The Area Under Receiver Operating Characteristic (AUROC) is used as a metric.
The experiment set each class as normal and measured the average AUROC with three different seeds.
4. COMPARISON METHOD
ITSR (Beggel et al., 2019), RVAE-ABFA (Gao et al., 2020), and NCAE (Yu et al., 2021) are employed as the robust models.
ITSR utilized OC-SVM and AAE for refinement.
RVAE-ABFA developed DAGMM (Zong et al., 2018) by adopting VAE and attention based feature adaption.
NCAE utilizes normal samples generated from generative adversarial model (Goodfellow et al., 2020) to refine the dataset.
The loss functions are evaluated based on three conventional models-AE (Bergmann et al., 2018), MemAE (Gong et al., 2019), and DSVDD (Ruff et al., 2018).
AE is employed since it is a conventional model.
DSVDD utilizes MSE but is different from reconstruction error.
MemAE is compared due to the additional loss function for memory augmented loss.
MSE is substituted by pseudo-Huber (Liznerski et al., 2020), GA loss (Barron, 2019), and LOE Qiu et al.
GA loss was used to demonstrate how much GA loss affects robustness since it utilizes negative loglikelihood to determine α while the proposed loss function utilizes z distribution.
Heterogeneous loss function with aggressive rejection is denoted as Hetero.
The architecture of AE, MemAE, and ITSR is described in Gong et al.
For MNIST and
F-MNIST, the encoder has three convolutions, and the decoder has three deconvolutions, in which batch normalization (Ioffe & Szegedy, 2015) and leaky ReLU activation (Xu et al., 2015) are followed except for the last deconvolution. For CIFAR-10 data, the encoder and decoder have four layers similar in configuration to MNIST with different kernel and stride sizes. For the rest of the models, LeNet based convolution neural network (as described in Ruff et al. (2018)) is used with 32 and 128 representation dimensions (except for RVAE-ABFA).
RVAE-ABFA uses only 32 dimensions due to the computation error.
Batch size and epochs are set to 100 and 300, respectively, except for ITSR, NCAE, and DSVDD.
ITSR refines data for every 10 epochs after the first 100 epochs and is trained on refined data with 100 epochs.
NCAE uses 150 epochs and DSVDD uses 150 epochs for pre-training and 100 epochs for the rest.
The parameters are updated by Adam optimizer (Kingma & Ba, 2014) with 0.0001 learning rate and 10 − 6 weight decay.
As mentioned before, the parameter q , α r and c in Hetero loss are set to 0.5, 1.5 and √ 0 .
4. E VALUATION WITH ROBUST LOSS FUNCTION
The robust loss functions are used to evaluate the robustness of Hetero loss.
Table 1 shows the
AUROC on clean datasets ( γ ct = 0 %) and contaminated datasets ( γ ct = 0 . 2 )
(the highest AUROC is in bold). The comparison methods on clean datasets show minimum AUROC loss compared to MSE except for AE on MNIST. Hetero loss on clean datasets shows comparable performance to
MSE within 0.01.
The performance of AE with pseudo-Huber, GA, and LOE does not significantly improve robustness.
However, AE with hetero loss achieves over 0.045 robustness.
When DSVDD on F-MNIST is trained with robust loss functions, it shows the most effective results.
LOE generates similar performance to Hetero by maximizing the abnormal loss function.
Since CIFAR-10 is a hard dataset compared to MNIST or F-MNIST, the AUROC on the clean dataset is low.
Therefore, robustness has not improved significantly.
Hetero loss surpasses 0.002 0.084 more robustness compared to the overall robust loss functions on 20% contaminated data.
Figure 3 visualizes the AUROC of robust loss functions depending on contamination ratio γ ct .
Hetero loss achieves the most robustness in the case of AE and MemAE.
In the case of DSVDD on
F-MNIST, LOE outperforms hetero when the contamination ratio is under 10%, but hetero loss outperforms LOE when γ ct = 0 . 2 .
According to Qiu et al.
, LOE performs best when the assumed γ ct is equal to the actual γ ct .
LOE achieves robustness by maximizing the abnormal loss function, but the γ ct = 0 .
1 assumption limits the robustness.
Since DSVDD is not a reconstruction error based model, MSE and GA loss show robustness on CIFAR-10 except for γ ct = 0 .
The
[Table]
Hetero 0.
[Table]
[Table]
[Table]
Contamination Ratio 0.
A v e r a g e A U
R O
C
MSE pseudo Huber
GA
LOE
Hetero
[Table]
[Table]
[Table]
[Table]
Ratio
[Table]
[Table]
A v e r a g e
A U R O C
AE
[Table]
Ratio
[Table]
A v e r a
[Table]
e
A U R
[Table]
C
MemAE
[Table]
a) MNIST
[Table]
Ratio
[Table]
[Table]
[Table]
A v e r a g e
A U R O C
DSVDD
[Table]
Ratio
[Table]
A v e r a g e
A U R O C
AE
[Table]
Ratio
[Table]
[Table]
A v e r a g e
A U R O C
MemAE (b) F-MNIST
[Table]
Ratio
[Table]
[Table]
A v e r a g e
A U R O C
DSVDD 0.
0.20 Contamination Ratio 0.
A v e r a g e
A U R O C
AE 0.
0.20 Contamination Ratio 0.
A v e r a g e
A U R O C
MemAE (c) CIFAR0.
0.20 Contamination Ratio 0.
A v e r a g e
A U R O C
DSVDD experiment demonstrates that hetero loss can be applied to different MSE-based AD models and make high robustness and minimal AUROC loss on clean datasets.
[Table]
A v e r a g e
A U R O C
[Figure]
[Figure]
[Figure]
Ratio 0.775 0.
[Figure]
A v e r a g e
A U R O C
[Figure]
0.20 Contamination Ratio 0.54 0.
A v e r a g e
A U R O C
[Figure]
4. E VALUATION WITH ROBUST MODELS
The standard models with Hetero loss are compared to robust models.
The AUROC of comparison on different contamination ratios is shown in Table 2.
NCAE has a low decrease in the AUROC of contamination datasets compared to clean datasets but lacks performance.
ITSR and RVAEABFA perform more robustness than AE and MemAE.
However, hetero loss function improves the robustness of AE and MemAE by about 0.05 compared to ITSR. DSVDD with Hetero loss shows comparable results toward RVAE-ABFA, which shows the most robust results on MNIST. Although RVAE-ABFA shows the highest AUROC compare to the other methods on F-MNIST, DSVDD with hetero loss shows 0.012 higher robustness.
Hetero loss achieves the highest robustness on CIFAR10.
It demonstrates that models with hetero loss can achieve comparable results to robust models.
Figure 4 illustrates the robustness of models on varying contamination ratios.
On MNIST and FMNIST datasets, RVAE-ABFA has the most robustness, but hetero exhibits the most comparable outcomes.
When the contamination ratio rises, the difference between RVAE-ABFA and DSVDD with hetero loss gets smaller.
In the case of CIFAR-10, DSVDD outperforms the RVAE-ABFA, and hetero makes MemAE comparable to RVAE-ABFA.
Compared to the existing robust models, Hetero loss achieves high robustness without increasing inference time or modifying the architecture.
4. A BLATION STUDY
In this experiment, each component was removed to measure its effect.
The performance on MNIST with clean and contaminated datasets is shown in Table 3. q and t s are the key components that determine the effect of aggressive rejection with soft or hard rejection. The loss function and values are denoted as L ( q, t s ) , where L is the loss function, q is the q -quantile, and t s is the degree of rejection. AE
[Table]
0.5 t s 0.
A v e r a g e A
U R O
C
MSE(γ ct =0)
MSE(γ ct =0.1)
MSE(γ ct =0.2)
Hetero(γ ct =0)
Hetero(γ ct =0.1)
Hetero(γ ct =0.2)
Effect of aggressive rejection.
Hard rejection-MSE(0.5,0) is more robust than the standard lossMSE(1,1) on the contaminated data.
However, hard rejection on a clean dataset shows significant performance degradation of AE (0.115), MemAE (0.134), and DSVDD (0.045).
Since it excludes many normal samples, it has an adverse effect under clean datasets.
Soft rejection-MSE(0.5,0.1) complements the decline.
It shows comparable AUROC to MSE on clean datasets.
Since the normal samples in the exclusion are used for training in part, soft rejection improves robustness for AE,
MemAE, and DSVDD by 0.045, 0.07, and 0.044, respectively.
Effect of heterogeneous loss.
Heterogeneous loss function without aggressive rejection is referred to as Hetero(1,1).
It performs better than the baseline MSE loss and demonstrates greater robustness than MemAE with MSE(0.5,0.1).
Moreover, Hetero(0.5, 0.1), hetero loss with aggressive rejection, shows 0.008 and 0.15 improvements for AE and MemAE compared to MSE (0.5, 0.1).
The experiment demonstrates that the difference between normal and abnormal data can be produced by the gradient adaptation based on the mini-batch distribution.
Hetero loss makes soft rejection less sensitive to t s .
In contrast to hetero loss, which is still robust due to gradient adaptation, the soft section turns into a non-robust loss function MSE when t s = 1 .
The sensitivity of MSE and hetero loss with soft rejection is shown in Figure 5.
Soft rejection becomes less effective as t s increases.
However, hetero loss, which is less sensitive than MSE, exhibits less difference between t s = 0 .
1 and t s = 0 .
5 on 20% contaminated data.
C ONCLUSION
In this paper, heterogeneous loss function with aggressive rejection is introduced.
Numerous normal data are involved in the rejection target to handle the overall anomalies and excluded partially via soft rejection.
In addition, the mini-batch distribution is transformed into the gradient of general robust loss to suppress the convergence of suspicious outliers.
The loss function outperforms the previous robust loss function and shows comparable results to robust models in three image datasets.
The experiments proved that the MSE-based models could strengthen robustness against contaminated data by the proposed loss function.
In the future, we will utilize the maximization of abnormal loss function to promote the difference between normal and abnormal samples.
To complement the maximization, we will apply the dynamic q function based on the abnormality in distribution.
A Q UANTILE IN AGGRESSIVE REJECTION
Table 4 shows the effect of q -quantile.
The experiment is tested on 20% contaminated MNIST.
MSE with soft rejection ( t s = 0 . 1 ) is utilized for AE, MemAE, and DSVDD. 25 -quantile performs better than 0 . 75 -quantile except for AE. Although there are more anomalies in the exclusion for q = 0 .
25 , 0.5-quantile rejection performs the best.
It indicates that robustness is not always ensured by over-elimination.
[Table]
B O PTIMIZING A PARAMETER OF SOFT REJECTION
AE, MemAE, and DSVDD with hetero loss are trained on clean MNIST ( γ ct = 0 ) to optimize the t s .
Except for t s , the parameters are utilized as previously ( q = 0 .
5 , α r = 1 . 5 and c = √ 0 . 5 ).
The dashed line means the standard model with only MSE.
AUROC is saturated from 0.05 for AE and 0.1 for MemAE.
DSVDD is unstable but similar to MSE when t s is 0.1.
As a result, t s is set to 0.
generally, considering minimum performance loss and lowest value.
[Table]
[Figure]
C O PTIMIZING A P ARAMETER OF H ETEROGENEOUS L OSS
A variety of robust loss functions can be used as the lowest gradient loss function in hetero loss.
The pseudo-Huber ( α = 1 ), Cauchy ( α = 0 ), and Geman-McClure ( α = − 2 ) are the representative loss functions.
Figure 7 illustrates the effect of α r in hetero loss.
The loss function should enhance robustness on contaminated data while minimizing performance degradation on a clean dataset.
When α is adjusted below 1, hetero loss degrades on the clean dataset.
α = 1 shows insufficient performance on the clean dataset, whereas α = 2 shows lower robustness.
Therefore, the intermediate point α = 1 . 5 is used since it achieves comparable performance on clean and contaminated data.
0.20 Contamination Ratio 0.
A v e r a g e A U R
O C
MSE Hetero ( α r =2) Hetero ( α r =1.5)
Hetero ( α r =1) Hetero ( α r =0)
Hetero ( α r = −2) (
[Figure]
0.20 Contamination Ratio 0.
A v e r a g e A U R
O C
MSE Hetero ( α r =1.5) Hetero ( α r =1)
Hetero ( α r =0) Hetero ( α r = −2)
[Figure]
D O VERALL DATA RESULTS
The results on MNIST and F-MNIST with overall data can be seen in Table 5.
All normal data in training data are used.
90% of them are used for training and the remainder for validation.
As mentioned previously, γ ct / (1 − γ ct ) ∗ N number of anomalies are added, where γ ct is the contamination ratio and N is the number of normal data.
The evaluation makes use of all 10,000 test data.
For NCAE, both the reported results by Yu et al.
and the results reproduced using the author’s code are shown in the table.
The loss function with the highest AUROC is underlined, and the highest AUROC in the comparison is in bold on every dataset.
With the exception of MemAE on MNIST, the existing robust loss functions are insufficient to improve the robustness for AE and MemAE.
Comparing DSVDD with hetero loss to MSE, the robustness is increased by 0.043 and 0.048 on each dataset.
In the case of DSVDD, LOE decreases performance on clean MNIST and exhibits robustness values that are higher than hetero loss on FMNIST.
RVAE-ABFA and reproduced NCAE achieve greater robustness than in section 4.4 with the entire MNIST data usage.
RVAE-ABFA exhibits the best robustness on MNIST (0.918), whereas
NCAE, according to reports, exhibits the highest AUROC (0.889) on the 20% contaminated FMNIST.
Among AE, MemeAE, and DSVDD with robust loss functions, DSVDD with hetero loss generally performs best.
DSVDD with hetero loss has a large difference (0.06) for MNIST and a low difference (0.028) for F-MNIST when compared to RVAE-ABFA and NCAE, which have the best performance on contaminated data.
Since LOE performs better on F-MNIST, the maximization of abnormal loss function can improve hetero loss when it does not impair the proposed loss function.
[Table]