L EARNING AND E VALUATING R EPRESENTATIONS FOR D EEP O NE - CLASS C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a two-stage framework for deep one-class classiﬁcation.
We ﬁrst learn self-supervised representations from one-class data, and then build classiﬁers using generative or discriminative models on learned representations.
In particular, we present a novel distribution-augmented contrastive learning that extends training distributions via data augmentation to obstruct the uniformity of vanilla contrastive representations, yielding more suitable representations for one-class classiﬁcation.
Moreover, we argue that classiﬁers inspired by the statistical perspective in generative or discriminative models are more effective than existing approaches, such as an average of normality scores from a surrogate classiﬁer.
In experiments we demonstrate state-of-the-art performance on visual domain one-class classiﬁcation benchmarks.
The framework does not only learn a better representation but it also permits building one-class classiﬁers that are more faithful to the target task.
Finally, we present visual explanations, conﬁrming that the decision-making process of our deep one-class classiﬁer is intuitive to humans.
I NTRODUCTION
One-class classiﬁcation aims to identify if an example belongs to the same distribution as the training data.
There are several applications of one-class classiﬁcation, such as anomaly detection or outlier detection, where we learn a classiﬁer that distinguishes the anomaly/outlier data without access to them from the normal/inlier data accessible at training.
The data could come from various domains, such as industrial manufacturing (defect detection), ﬁnance (fraudulent transaction detection), etc.
Generative models, such as kernel density estimation (KDE), have been popular for one-class classiﬁcation [1, 2] as they model the distribution by assigning high density to the training data.
At test time, low density examples are determined as outliers.
Unfortunately, the curse of dimensionality hinders estimating the density of the high-dimensional data accurately [3].
Deep generative models (e.g. [4, 5, 6]), have demonstrated success in modeling high-dimensional data (e.g., images) and have been applied to anomaly detection [7, 8, 9, 10, 11].
However, learning deep generative models on raw inputs remains as challenging as they appear to assign high density to background pixels [10] or learn local pixel correlations [12].
A good representation might still be beneﬁcial to those models.
Alternately, discriminative models like one-class SVM (OC-SVM)
[13] or support vector data description (SVDD)
[14] learn classiﬁers describing the support of one-class distributions to distinguish them from outliers.
Speciﬁcally, these methods become more powerful with non-linear kernel functions that implicitly transform the raw data into a high-dimensional feature space.
However, the performance of discriminative models is still limited by the quality of data representations.
The fundamental limitation of one-class classiﬁcation centers on learning good high-level data representations.
Following the success of deep learning [15], deep one-class classiﬁcations [16, 17, 18], which extend the discriminative one-class classiﬁcation using trainable deep neural networks, have shown promising results compared to their kernel counterparts.
However, a naive training of deep one-class classiﬁers leads to a degenerate solution that maps all data into a single representation, also known as “hypersphere collapse” [16].
Previous works circumvent such issues by constraining network architectures [16], autoencoder pretraining [16, 17], training with a surrogate multi-class classiﬁcation loss by introducing simulated outliers [19, 20, 21, 22] or injecting noise [18].
In this work, we present a two-stage framework for building deep one-class classiﬁers.
As shown in
Figure 1, in the ﬁrst stage, we train a deep neural network to obtain a high-level data representation.
[Figure]
Then, in the second
, we build a one-class classiﬁer, such as OC-SVM or KDE, using representations from the ﬁrst stage.
Comparing to those using surrogate losses [20, 21], our framework allows to build a classiﬁer that is more faithful to one-class classiﬁcation.
Decoupling representation learning from classiﬁer construction further opens up opportunities of using state-of-the-art representation learning methods, such as self-supervised contrastive learning [23].
While vanilla contrastive representations are less compatible with one-class classiﬁcation as they are uniformly distributed on the hypersphere [24], we show that, with proper ﬁxes, it provides representations achieving competitive one-class classiﬁcation performance to previous state-of-the-arts.
Furthermore, we propose a distribution-augmented contrastive learning, a novel variant of contrastive learning with distribution augmentation [25].
This is particularly effective in learning representations for one-class classiﬁcation, as it reduces the class collision between examples from the same class [26] and uniformity [24].
Lastly, although representations are not optimized for one-class classiﬁcation as in end-to-end trainable deep one-class classiﬁers [16], we demonstrate state-of-the-art performance on visual one-class classiﬁcation benchmarks.
We summarize our contributions as follows:
• We present a two-stage framework for building deep one-class classiﬁers using unsupervised and self-supervised representations followed by shallow one-class classiﬁers.
• We systematically study representation learning methods for one-class classiﬁcation, including augmentation prediction, contrastive learning, and the proposed distribution-augmented contrastive learning method that extends training data distributions via data augmentation.
• We show that, with a good representation, both discriminative (OC-SVM) and generative (KDE) classiﬁers, while being competitive with each other, are better than surrogate classiﬁers based on the simulated outliers [20, 21].
• We achieve strong performance on visual one-class classiﬁcation benchmarks, such as CIFAR10/100 [27], Fashion MNIST [28], Cat-vs-Dog [29] and CelebA [30].
In addition to comprehensive analysis on one-class contrastive learning, we present visual explanations of self-supervised deep one-class classiﬁers to better understand their decision making processes
A T WO - STAGE F RAMEWORK FOR D EEP O NE -C LASS C LASSIFICATION
In Section 2.1, we review self-supervised representation learning algorithms, discuss how they connect to existing one-class classiﬁcation methods, raise issues of state-of-the-art contrastive representation learning [23] for one-class classiﬁcation, and propose ways to resolve these issues.
Then, in
Section 2.2, we study how to leverage the learned representations for one-class classiﬁcation.
2. L EARNING R EPRESENTATIONS FOR O NE -C LASS C LASSIFICATION
Let A be the stochastic data augmentation process, which is composed of resize and crop, horizontal ﬂip, color jittering, gray-scale and gaussian blur, following [23], for image data.
As in Figure 1, selfsupervised learning methods consist of a feature extractor f parameterized by deep neural networks and the proxy loss L .
Below, we discuss details of different self-supervised learning methods.
2.1. A UGMENTATION P REDICTION
One popular method for learning representations learns by discriminating the type of augmentation applied to the data.
Rotation prediction [31], which is one of the most representative of kinds, learns deep representations by predicting the degree of rotation augmentations.
The training objective of the rotation prediction proxy task is written as follows:
[Equation]
[Table]
where y ∈ { 0 , 1 , 2 , 3 } is a prediction target representing the rotation degree, and rot90( x, y ) rotates an input x by 90 degree y times.
We represent the classiﬁer p q ◦ f
( y | x ) as p q ( y | f ( x ))
∝ exp( q ◦ f ( x ))
[ y ] which contains the representation f and a linear layer q with 4 output units for rotation degrees.
The learned network f yields a good representation for classiﬁcation tasks [31].
Application to One-Class Classiﬁcation.
While proposed for learning representations, augmentation prediction has been successfully adopted to learn deep one-class classiﬁers [20, 21, 22].
Although it is not trained to do so, the likelihood of learned rotation classiﬁers 1 p q ( y = 0 | f
( x )) is shown to well approximate normality score.
Plausible explanation is via outlier exposure [19], where the classiﬁer learns a decision boundary distinguishing original images from simulated outliers created by image rotation.
However, it assumes inlier images are not rotated.
Moreover, the classiﬁer may not generalize to one-class classiﬁcation task if it overﬁts to the proxy task (e.g., rotation prediction).
2.1. C ONTRASTIVE L EARNING
Unlike augmentation prediction that learns discriminative representations to data augmentation, contrastive learning [23] learns representation by distinguishing different views (e.g., augmentations) of itself from other data instances.
Let φ ( x ) = normalize( f ( x )) , i.e., (cid:107) φ ( x ) (cid:107) = 1 .
Following the formulation of [32], the proxy task loss of contrastive learning is written as:
[Equation]
where A and A (cid:48) are identical but independent stochastic augmentation processes for two different views of x .
L clr regularizes representations of the same instance with different views ( A ( x ) , A (cid:48)
( x ) ) to be similar, while those of different instances ( A ( x ) , A (cid:48)
( x (cid:48) ) ) to be unlike.
Class Collision and Uniformity for One-Class Classiﬁcation.
While contrastive representations have achieved state-of-the-art performance on visual recognition tasks [23, 24, 33, 34] and have been theoretically proved to be effective for multi-class classiﬁcation [26, 35], we argue that it could be problematic for one-class classiﬁcation.
First, a class collision [26].
The contrastive loss in Equation (2) is minimized by maximizing the distance between representations of negative pairs ( x, x i ) , x (cid:54) =
x i , even though they are from the same class when applied to the one-class classiﬁcation.
This seems to contradict to the idea of deep one-class classiﬁcation [16], which learns representations by minimizing the distance between representations with respect to the center: min f E x (cid:107) f
( x ) − c (cid:107) 2 .
Second, a uniformity of representations [24].
It is proved that the optimal solution for the denominator of Equation (2) is perfect uniformity as M → ∞ [24], meaning that φ ( x ) follows a uniform distribution on the hypersphere.
This is problematic since one can always ﬁnd an inlier x ∈ X in the proximity to any outlier x (cid:48) /
∈ X on the hypersphere, as shown in Figure 2a.
In contrast, with reduced uniformity as in Figure 2b, it is easier to isolate outliers from inliers.
One-Class Contrastive Learning.
First, to reduce the uniformity of representations, we propose to use a moderate M (batch size).
This is in contrast with previous suggestions to train with large M for contrastive representations to be most effective on multi-class classiﬁcation tasks [23, 34, 36].
The impact of batch size M for one-class classiﬁcation will be discussed in Section 5.1.
[Figure] [Figure]
In addition, we propose distribution augmentation 2 for one-class contrastive learning.
The idea is that, instead of modeling the training data distribution P X , we model the union of augmented training distribution P (cid:83) a a ( X ) , where a ( X ) =
{ a ( x ) | x ∈ X} .
Note that augmentation a for augmenting distribution is disjoint from those for data augmentation A that generates views.
Inspired by Golan and El-Yaniv [20], we employ geometric transformations, such as rotation or horizontal ﬂip, for distribution augmentation.
For example, as in Figure 3, x and rot90( x ) (German shepherds in the top row) are considered as two separate instances and therefore are encouraged to be distant in the representation space.
Not only increasing the number of data instances to train on (e.g., distribution augmentation by rotating 90 ◦ , 180 ◦ , 270 ◦ increases the dataset by 4 times), but it also eases the uniformity of representations on the resulted hypersphere.
A pictorial example is in Figure 2c, where thanks to augmented distribution, the inlier distribution may become more compact.
2.1. E XTRACTING R ICHER R EPRESENTATION BY L EARNING WITH P ROJECTION H EAD
Self-supervised representations are prone to overﬁtting to the proxy task.
Gidaris et al.
[31] show that f is not the best representation for downstream tasks as f becomes overly discriminative to augmentation prediction while losing useful information of the data.
Lower-layer representations are suggested for downstream tasks.
While the output of f trained by contrastive loss has been theoretically proven to be useful for multi-class classiﬁcation [26, 35, 37], the efﬁcacy of projection head parameterized by MLP has been conﬁrmed in practice [23, 38].
The effect of projection heads may be explained from the information theory perspective.
To illustrate, we extend the network structure of augmentation prediction as q ◦ g ◦ f , where q is the linear classiﬁcation head, g is the projection head, and f outputs self-supervised representations used for downstream tasks.
We note that using an identity head g ( x ) = x recovers the network structure of previous works [20, 21, 31].
The data processing inequality [39] I ( g ◦ f ( x ); x ) ≤ I ( f ( x ); x ) tells that f can retain more information than g , thus more suitable for downstream tasks that are not necessarily correlated with the proxy tasks.
The effectiveness of deep MLP projection head is empirically shown for both augmentation prediction and contrastive learning in Section 4.
2. B UILDING D EEP O NE -C LASS C LASSIFIERS WITH L EARNED R EPRESENTATIONS
We present a two-stage framework for deep one-class classiﬁcation that builds one-class classiﬁers on learned representations as in Figure 1.
Compared to end-to-end training [16, 20, 21, 22], our framework provides ﬂexibility in using various representations as the classiﬁer is not bound to representation learning.
It also allows the classiﬁer consistent with one-class classiﬁcation objective.
To construct a classiﬁer, we revisit an old wisdom which considers the full spectrum of the distribution of the learned data representation.
For generative approaches, we propose to use nonparametric kernel density estimation (KDE) to estimate densities from learned representations.
For discriminative approaches, we train one-class SVMs [13].
Both methods work as a black-box and in experiment we use the default training setting except the kernel width where we reduce by 10 times than default.
We provide detailed classiﬁer formulations in Appendix A.1.
2.2. G RADIENT - BASED E XPLANATION OF D EEP O NE -C LASS C LASSIFIER
Explaining the decision making process helps users to trust deep learning models.
There have been efforts to visually explain the reason for model decisions of multi-class classiﬁers [40, 41, 42, 43, 44, 45, 46] using the gradients computed from the classiﬁer.
In this work, we introduce a gradient-based visual explanation of one-class classiﬁcation that works for any deep representations.
To construct an end-to-end differentiable decision function, we employ a KDE detector, whose formulation is in Appendix A.1, built on top of any differentiable deep representations: ∂ KDE
[Equation]
R ELATED W ORK
One-class classiﬁcation [47] has broad use cases, including fraud detection [48], spam ﬁltering [49], medical diagnosis [50], manufacturing defect detection [51], to name a few.
Due to the lack of granular semantic information for one-class data, methods for learning from unlabeled data have been employed for one-class classiﬁcation.
Generative models, which model the density of training data distribution, are able to determine outlier when the sample shows low density [8, 50, 52].
These include simple methods such as kernel density estimation or mixture models [53], as well as advanced ones [4, 5, 6, 54, 55, 56, 57].
However, the density from generative models for highdimensional data could be misleading [9, 12, 58, 59].
New detection mechanisms based on the typicality [60] or likelihood ratios [10] have been proposed to improve out-of-distribution detection.
Self-supervised learning is commonly used for learning representations from unlabeled data by solving proxy tasks, such as jigsaw puzzle [61], rotation prediction [31], clustering [62], instance discrimination [63] and contrastive learning [23, 33, 64].
The learned representations are then used for multi-class classiﬁcation, or transfer learning, all of which require labeled data for downstream tasks.
They have also been extended to one-class classiﬁcation.
For example, contrastive learning is adopted to improve the out-of-distribution detection under multi-class setting [65], whereas our work focuses on learning from a single class of examples, leading to propose a novel distributionaugmented contrastive learning.
Notably, learning to predict geometric transformations [20, 21, 22] extends the rotation prediction [31] to using more geometric transformations as prediction targets.
Unlike typical applications of self-supervised learning where the classiﬁer or projection head [23] are discarded after training, the geometric transformation classiﬁer is used as a surrogate for oneclass classiﬁcation.
As in Section 4.1, however, the surrogate classiﬁer optimized for the selfsupervised proxy task is suboptimal for one-class classiﬁcation.
We show that replacing it with simple one-class classiﬁers consistently improve the performance.
Furthermore, we propose strategies for better representation learning for both augmentation prediction and contrastive learning.
Distribution-augmented contrastive learning is concurrently developed in [66] as a part of their multi-task ensemble model.
While sharing a similar technical formulation, we motivate from ﬁxing the uniformity of contrastive representations.
We note that our study not only focuses on representation learning, but also on the importance of detection algorithms, which is under explored before.
E XPERIMENTS
Following Golan and El-Yaniv [20], we evaluate on one-class classiﬁcation benchmarks, including
CIFAR-10, CIFAR-100 3 [27], Fashion-MNIST [28], and Cat-vs-Dog [29].
Images from one class are given as inlier and those from remaining classes are given as outlier.
We further propose a new protocol using CelebA eyeglasses dataset [30], where face images with eyeglasses are denoted as an outlier.
It is more challenging since the difference between in and outlier samples is ﬁner-grained.
We resize images into 64 × 64 for cat-vs-dog and CelebA datasets and 32 × 32 for the rest.
[Table] [Table]
denotes evaluation methods using test time data augmentation.
Our methods are both more accurate and computationally efﬁcient.
We evaluate the performance of (1) representations trained with unsupervised and self-supervised learning methods, including denoising autoencoder [67], rotation prediction [31], contrastive learning [23, 63]
[Table]
and (2) using different one-class classiﬁers, such as OC-SVM or KDE.
We use rotation augmentations for distribution-augmented contrastive learning, denoted as Contrastive (DA).
We train a ResNet-18 [68] for feature extractor f and a stack of linear, batch normalization, and ReLU, for MLP projection head g .
More experiment details can be found in Appendix A.3.
4. M AIN R ESULTS
We report the mean and standard deviation of AUCs averaged over classes over 5 runs in Table 2.
The mean of 5 datasets is weighted by the number of classes for each dataset.
Besides those using self-supervised representations, we provide results using ImageNet pretrained ResNet-50 to highlight the importance of learning representations from in-domain distribution.
ImageNet pretrained ResNet-50 achieves the best performance of 84 .
0 mean AUC over 5 datasets.
Compared to representations learned with denoising objective, it works particularly well on datasets such as CIFAR-100, cat-vs-dog, and CelebA, which we attribute it to the subset of ImaegNet classes is closely related to the classes of these datasets.
Similar to the ﬁndings from [20, 21], we observe signiﬁcant performance gains with self-supervised learning.
Moreover, while RotNet [20], an end-to-end trained classiﬁer using rotation prediction, achieves 83 .
1 AUC, the RotNet representation evaluated with the KDE detector achieves 86 .
6 , emphasizing the importance of a proper detector in the second stage.
Finally, the quality of RotNet representation improves when trained with the MLP projection head, resulting in 88 .
2 AUC.
The representations learned with vanilla contrastive loss achieve 86 .
9 AUC with OC-SVM, which under-perform those trained with rotation prediction loss ( 88 . 2 ).
The distribution-augmented contrastive loss achieves the highest mean AUC of 89 . 9 among all methods, demonstrating performance gains on all datasets by a large margin upon its vanilla counterparts.
Comparison to Previous Works.
We make comparisons to previous works in Table 3.
While some comparisons may not be apples-to-apples as different works use different implementations of networks, optimizers, etc., we note that our implementation is based on the common choices of network (e.g., ResNet-18) and optimizer (e.g., momentum SGD) for image classiﬁcation.
We advance the previous state-of-the-art on one-class classiﬁcation benchmarks by a large margin without test-time augmentation nor ensemble of models.
We further improve the performance with model ensemble, which we report in Appendix A.2.2.
[Figure]
A NALYSIS AND A BLATION S TUDY
In Section 5.1, we analyze behaviors of one-class contrastive representations and in Section 5.3, we report visual explanations of various deep one-class classiﬁers.
Due to a space constraint, more studies, including an in-depth study on distribution-augmented contrastive representations and data efﬁciency of self-supervised learning for one-class classiﬁcation, are in Appendix A.2.
5. U NIFORMITY , B ATCH S IZE AND D ISTRIBUTION A UGMENTATION
[23, 33, 34, 36] have shown substantial improvement on contrastive representations evaluated on multi-class classiﬁcation using very large batch sizes, which results in uniformity.
However, uniformity [24] and class collision [26] can be an issue for one-class classiﬁcation as discussed in
Section 2.1.2.
Here we investigate the relations between performance and uniformity, and how we can resolve the issue via batch size, MLP head and distribution-augmented contrastive learning.
We measure the uniformity via MMD distance [69] between the learned representation and samples from uniform distributions on hyperspheres.
Smaller MMD distance implies the distribution of representation is closer to uniform distributions.
We train models with various batch sizes { 2 3 , . .
We report in Figure 4 the average and standard deviation over 5 runs on CIFAR-10 validation set.
Distribution Augmentation.
In Figure 4a, the representations from the projection head g ◦ f trained by standard contrastive learning (Contrastive g ◦ f ) are closer to be uniformly distributed when we increase the batch size as proven by [24].
Therefore, in one-class classiﬁcation, the nearly uniformly distributed representation from contrastive learning is only slightly better than random guess (
AUC) as in Figure 4b.
In contrast, with distribution augmentations, the learned representations (Contrastive (DA) g ◦ f ) are less uniformly distributed and result in a signiﬁcant performance gain.
Batch Size.
Following the discussion, large batch sizes result in nearly uniformly distributed representations ( g ◦ f ), which is harmful to the one-class classiﬁcation.
On the other hand, small batch sizes ( ≤ 16 ), though less uniform, hinders us learning useful representations via information maximization [70].
As in Figure 4b, there is a trade-off of batch size for one-class classiﬁcation, and we ﬁnd batch size of 32 results in the best one-class classiﬁcation performance.
MLP Projection Head.
As discussed in Section 2.1.3, f , an input to projection head, contains no less information than the output g ◦ f , but overﬁts less to the proxy task.
As in Figure 4b, we observe signiﬁcantly better performance with f than g ◦ f .
Lastly, we emphasize that all these ﬁxes contribute to an improvement of contrastive representations for one-class classiﬁcation.
As in Figure 4 AUC drops when any of these components are missing.
5. A NALYSIS ON D IFFERENT D ISTRIBUTION A UGMENTATIONS
The choice of distributions affects the performance.
The ablation study using horizontal ﬂip ( hﬂip ) and rotation augmentations is reported in Figure 6.
Note that hﬂip is used only to augment distribution in this experiment.
Interestingly, simply adding hﬂip improves the AUC to 90 .
This suggests a different insight from [66] who augments distribution as a means to outlier exposure [19].
Although we report the numbers with rotation augmentations in Table 2, with hﬂip , rot90 , rot90+hﬂip as
[Figure] [Figure]
augmented distributions, we achieve the best mean AUC, 93 . 7 , on CIFAR-10, without any test-time augmentation.
Additional study on distribution augmentation follows in Appendix A
V ISUAL E XPLANATION OF D EEP O NE -C LASS C LASSIFIERS
We investigate the decision making process of our deep one-class classiﬁers using the tools described in Section 2.2.1.
Speciﬁcally, we inspect by highlighting the most inﬂuential regions using two popular visual explanation algorithms, namely, integrated gradients (IG) [44] and GradCAM [43], for distribution-augmented contrastive representation as well as RotNet and DAE on images from cat-vs-dog and CelebA eyeglasses datasets.
For RotNet, we test using both a rotation classiﬁer and
KDE (RotNet ∗ in Figure 5) to compute gradients.
As in Figure 5, the proposed visual explanation method based on the KDE one-class classiﬁer permits highlighting human-intuitive, meaningful regions in the images, such as dog faces or eyeglasses instead of spurious background regions (Figure 5b).
On the other hand, even though the classiﬁcation AUC is not too worse ( 86 .
1 AUC on cat-vs-dog as opposed to 89 .
6 for ours)
, the visual explanation based on the rotation classiﬁer (Figure 5c) suggests that the decision may be made sometimes based on the spurious features (e.g., human face in the background).
We present more examples for visual explanation in Appendix A.5.
C ONCLUSION
Inspired by an old wisdom of learning representations followed by building classiﬁers, we present a two-stage framework for deep one-class classiﬁcation.
We emphasize the importance of decoupling building classiﬁers from learning representations, which allows classiﬁer to be consistent with the target task, one-class classiﬁcation.
Moreover, it permits applications of various self-supervised representation learning methods, including contrastive learning [23] with proper ﬁxes, for one-class classiﬁcation, achieving strong performance on visual one-class classiﬁcation benchmarks.
Finally, we exhibit visual explanation capability of our two-stage self-supervised deep one-class classiﬁers.
A A PPENDIX A. F ORMULATIONS OF O NE -C LASS C LASSIFIERS
For completeness, we provide formulations of one-class classiﬁers, such as one-class support vector machine (OC-SVM) [13] and kernel density estimator, used in our work.
One-Class Support Vector Machine solves the following optimization problem to ﬁnd support vectors describing the boundary of one-class distribution:
[Equation]
(3) subject to w (cid:62)
f i ≥ ρ − ξ i , ξ i ≥ 0 , ∀ i
= 1 , · · ·
[Equation]
n (4) where f i = f
( x i ) is a feature map.
The decision score is given as follows:
[Equation]
(5) with coefﬁcients α i > 0 for support vectors.
Linear or RBF ( k γ ( x, y ) =
exp (cid:0) − γ (cid:107) x − y (cid:107) 2 (cid:1) ) kernels are used for experiments.
Kernel Density Estimation is a nonparametric density estimation algorithm.
The normality score of KDE with RBF kernel parameter γ is written as follows:
[Equation]
A. A DDITIONAL A BLATION S TUDY A.2. W ELL B EHAVED C ONTRASTIVE R EPRESENTATION
Linear Separability.
A common belief is the good representations are linearly separable with respect to some underlying labels, which is supported by several empirical success [23].
The linear classiﬁers are proven to be nearly optimal [35] when we learn representations on data with all possible labels along with some additional assumptions.
In one-class classiﬁcation, although we violate the assumptions that we only train on one class of data, we are interested in the linear separability of the data to understand the characteristics of the learned embedding.
To this end, in addition to training RBF kernel OC-SVM, we train OC-SVM with linear kernels on the learned embedding.
We note that it is atypical to use linear OC-SVM, which is usually considered to be suboptimal for describing decision boundaries of one-class classiﬁcation problems.
However, with good representations, linear classiﬁers show good performance on one-class classiﬁcation as shown in Table 4, and are better than existing works [20] in Table 2.
Lastly, we note that OC-SVM with nonlinear RBF kernel is still better than linear models in Table 4.
Similar observations are found in other problems such as regression [71] and generative models [72].
Parametric Models.
In addition to studying from the perspective of discriminative model, we also dive into the generative model side.
Instead of using the nonparametric KDE for density estimation, we use a parametric model, single multivariate Gaussian, whose density is deﬁned as
[Equation]
A shown in Table 4, the Gaussian density estimation (GDE) model shows competitive performance as KDE.
It suggests the learned representations from distribution-augmented contrastive learning is compact as our expectation, which can be well approximated by a simple parametric model with single Gaussian.
Compared with nonparametric methods, although parametric models have strong assumptions on the underlying data distributions, using parametric model is more data efﬁcient if the assumption holds 4 .
In addition, parametric models have huge advantage in computation during testing.
Therefore, there is trade-off between assumptions of the data as well as model ﬂexibility and computational efﬁciency.
We note that the single Gaussian parametric model may not be a good alternative of KDE universally.
A candidate with good trade-off is Gaussian Mixture Models, which can be treated as a middle ground of nonparametric KDE and parametric single Gaussian model.
We leave the study for future works.
[Table]
A.2. A NALYSIS ON D ISTRIBUTION A UGMENTED C ONTRASTIVE R EPRESENTATIONS
Relation to Outlier Exposure [19].
Distribution augmentation shares a similarity to outlier exposure [19] in that both methods introduce new data distributions for training.
However, outlier exposure requires a stronger assumption on the data distribution that introduced outlier should not overlap with inlier distribution, while such an assumption is not required for distribution augmentation.
For example, let’s consider rotation prediction as an instance of outlier exposure.
When training the model on randomly-rotated CIFAR-10, where we randomly rotate images of CIFAR-10, the performance of rotation prediction representation drops to 60 .
7 AUC, which is slightly better than a random guess.
On the other hand, rotation-augmented contrastive representation is not affected by random rotation and achieves 92 .
4 AUC.
Ensemble of Contrastive Representations.
We note that most previous methods have demonstrated improved performance via extensive test-time data augmentation [20, 21, 22, 66].
While already achieving state-of-the-art one-class classiﬁcation performance without test-time data augmentation, we observe marginal improvement using test time data augmentation.
Instead, we ﬁnd that an ensemble of classiﬁers built on different representations signiﬁcantly improve the performance.
In Table 5, we report the performance of distribution-augmented contrastive representations trained with different sets of augmented distributions on CIFAR-10.
We observe that ensemble ofmodels trained with different seeds consistently improves the performance.
Moreover, not only we improve one-class classiﬁcation AUCs when aggregating scores across models trained with different distribution augmentations, we also observe lower standard deviation across different seeds.
Finally, when ensemble over 5 × 5 = 25 models, we achieve 94 .
6 AUC.
[Table]
) and that of ensemble model whose ensemble score is aggregated from 5 models trained with different random seeds (“ensemble of 5 models”).
“Ensemble of 5” aggregates score from 5 models with different augmentation distributions.
Correlation between Uniformity and One-class Classiﬁcation.- We show scatter plots showing the positive correlation between the uniformity ( log(MMD) ) and the one-class classiﬁcation AUCs in Figure 7. Speciﬁcally, we compute the MMD using g ◦ f , an output of MLP projection head that are used to optimize the contrastive loss at training. We evaluate AUCs using both g ◦ f (left of
Figure 7) and f (right of Figure 7).
Data points in the plots are obtained from vanilla and DistAug contrastive models trained with various batch sizes, including 32, 64, 128, 256 and 512.
For each conﬁguration, we train 5 models with different random seeds.
From the left plot of Figure 7, we observe a strong positive correlation between MMD and AUC, suggesting that the more uniformly distributed (i.e., lower MMD), the worse the one-class classi-
[Figure]
[Figure]
[Figure]
U
[Figure]
( g f )
Pearson corr: 0.914 (p-value: 0.000)
[Figure]
2.5 log MMD ( g f )
A U C ( f )
Pearson corr: 0.774 (p-value: 0.000)
Vanilla DistAug ﬁcation performance is (i.e., low AUCs).
We also compute the Pearson’s correlation coefﬁcient, which results in 0 . 914 .
We also investigate the correlation between the MMD using g ◦ f and the
AUC using f , as shown in the right plot of Figure 7.
We observe highly positive correlation of 0.
Pearsons correlation coefﬁcient between the MMD using g ◦ f and the AUC using f .
Overall, our results suggest that reducing uniformity of g ◦ f improves the one-class classiﬁcation accuracy tested with both g ◦ f and f representations.
A.2. D ATA E FFICIENCY OF S ELF -S UPERVISED R EPRESENTATION L EARNING
While previous works on self-supervised learning have demonstrated the effectiveness on learning from large-scale unlabeled data, not much has shown for the data efﬁciency of these methods.
Unlike multi-class classiﬁcation tasks where the amount of data could scale multiplicative with the number of classes, data efﬁciency of representation learning becomes of particular interest for one-class classiﬁcation as it is hard to collect large-scale data even without an annotation.
We present one-class classiﬁcation AUCs of representations with various training data sizes, along with two baseline representations such as ResNet-18 using random weights or ImageNet-pretrained
ResNet-50 in Figure 8.
Note that we vary the training set sizes at representation learning phase only, but use a ﬁxed amount ( 5000 ) to train classiﬁers for fair evaluation of the representation quality.
We ﬁnd that even with 50 examples, classiﬁers beneﬁt from self-supervised learning when comparing against raw or deep features with random weights.
The proposed distribution-augmented contrastive loss could match the performance of ImageNet-pretrained ResNet-50 with only 100 examples, while rotation prediction loss and vanilla contrastive loss require 250 and 1000 examples, respectively.
A. D ETAILS OF E XPERIMENTAL S ETTING
Unless otherwise stated, models are trained for 2048 epochs with momentum ( 0 . 9 ) SGD and a single cycle of cosine learning rate decay [73].
L2 weight regularization with coefﬁcient of 0 .is applied.
We use scikit-learn [74] implementation of OC-SVMs with default value of ν .
We use γ = 10 / | f |· Var( f ) , which is 10 times larger than the default value for kernel OC-SVM.
Same value of γ is used for KDE.
We use scikit-learn implementation of Gaussian mixture model using a single component for Gaussian density estimator (GDE).
No hyperparameters are tuned for GDE.
Finally, all experiments are conducted using TensorFlow [75].
Other hyperparameters, such as learning rate or the MLP projection head depth, are cross-validated using small labeled data.
While this may violate the assumption of one-class classiﬁcation, supervised model selection is inevitable for deep learning models as their behaviors may be largely dependent on hyperparameters.
To this end, we use 10% of inlier ( 500 ) and the same number of outlier examples of CIFAR-10 for hyperparameter selection, and use the same set of hyperparameters [Figure] [Figure] [Table]
to test methods on other datasets, which could demonstrate the algorithm robustness.
Learning rate ∈{ 0 . 1 , 0 . 03 , 0 . 01 , 0 . 003 , 0 . 001 } and the depth ∈{ 0 , · · · , 8 } of an MLP projection head are tuned for all methods.
In addition, the temperature τ ∈{ 1 , 0 . 5 , 0 . 2 , 0 . 1 } and the batch size ∈{ 2 n , n = 3 , ..., 9 } of contrastive loss are tuned.
To this end, we train all models across all datasets using the same hyperparameter conﬁgurations, such as learning rate of 0 .
01 , projection head of depth 8 ( [512 × 8 , 128] ), temperature τ of 0 . 2 , or batch size of 32 .
A. P ER -C LASS AUC S
[Table]
[Table]
A. M ORE E XAMPLES FOR V ISUAL E XPLANATION OF D EEP O NE - CLASS CLASSIFIERS
[Table]
[Table]
[Figure] [Figure] [Figure]
[Figure]
B E XPERIMENTS ON MVT EC A NOMALY D ETECTION D ATASET B. D ATASET D ESCRIPTION
There are 15 different categories in MVTec anomaly detection dataset [51], where 10 are object (e.g., bottle, cable, transistor) and 5 are texture (e.g., grid, leather) categories.
Each category comes with the training set containing ∼ 241 images per category from normal data distribution and the test set containing ∼ 115 images per category from both normal and defective data distributions.
The dataset also provides pixel-accurate annotation for defective regions, allowing to measure the defect localization performance.
Images are of high-resolution, whose side length is as long as 1024 .
B. E XPERIMENTAL S ETTING
First, we resize all images into 256 × 256 in our experiments.
One challenge of learning representations from scratch on MVTec dataset is that the amount of training set is too small ( ∼ 241 images per category).
As in Figure 8, self-supervised representation learning methods beneﬁt from the large amount of training data and the performance degrades when the amount of training data is limited.
Instead of learning the holistic image representation, we learn a patch representation of size 32 × 32 via the proposed self-supervised representation learning methods.
Similarly to our previous experiments, we train ResNet-18 from scratch.
For evaluation, we compute both image-level detection and localization AUCs.
We densely extract embeddings f from image patches with the stride of 4 and compute anomaly scores with KDE at each patch location, which results in a ( 256 − 2244 + 1) ×
( 256 − 2244 + 1) =
× 57 anomaly score map.
For image-level detection, we apply spatial max-pooling to obtain a single score.
For localization, we upsample 57 × 57 into 256 × 256 with Gaussian kernel 5 [76] and compare with the annotation.
B. E XPERIMENTAL R ESULTS
[Table]
We report quantitative results in Table 11.
Speciﬁcally, we report detection and localization AUCs averaged over object, texture
[Table]
and all categories.
Likewise, we run experiments 5 times with different random seeds and report the mean and standard deviation.
We report the performance of
RotNet (using rotation classiﬁer, as in [20, 21]), the same representation but with KDE detectors as proposed, and RotNet trained with an MLP head and evaluated with KDE detector as proposed.
In addition, we evaluate the performance of contrastive representations without (Vanilla) and with distribution augmentation (DistAug) as proposed.
For comparison, we include the results of CAVGA
R u [77], which is also evaluated under the same setting.
Below we highlight some points from our results:
The proposed two-stage framework, which trains a RotNet and constructs an one-class classiﬁer, clearly outperforms an end-to-end framework, which trains the RotNet and uses a built-in rotation classiﬁer [20, 21].
The proposed modiﬁcation of RotNet with MLP projection head further improves the representation of RotNet for one-class classiﬁcation.
The proposed distribution augmentation (rotation) improves the performance of contrastive representations.
We improve the localization AUC upon [77] both in detection and localization, both on object and texture categories.
Both RotNet and contrastive representations are effective for “object” categories, but not as much for “texture” categories.
As reviewed by [78], semantic and texture anomaly detection
problems
are different and may require different tools to solve.
The methods relying on geo-
metric transformations
are effective in learning
representations of visual objects [
] and thus more effective for semantic anomaly detection, but less effective for texture anomaly
detection
B. Q UALITATIVE R ESULTS ON L OCALIZATION
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]