R ESTRICTED G ENERATIVE P ROJECTION FOR O NEC LASS C LASSIFICATION AND A NOMALY D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a novel framework for one-class classification and anomaly detection.
The core idea is to learn a mapping to transform the unknown distribution of training (normal) data to a known distribution that is supposed to be different from the transformed distribution of unknown abnormal data.
Crucially, the target distribution of training data should be sufficiently simple, compact, and informative.
The simplicity is to ensure that we can sample from the distribution easily, the compactness is to ensure that the decision boundary between normal data and abnormal data is clear and reliable, and the informativeness is to ensure that the transformed data preserve the important information of the original data.
Therefore, we propose to use truncated Gaussian, uniform in hyperball, uniform on hypersphere, or uniform between hyperspheres, as the target distribution.
We then minimize the distance between the transformed data distribution and the target distribution while keeping the reconstruction error for the original data small enough.
Our model is simple and easy to train especially compared with those based on generative models.
Comparative studies on a few benchmark datasets verify the effectiveness of our method in comparison to baselines.
I NTRODUCTION
Anomaly detection (AD) aims to distinguish normal data and abnormal data using a model trained on only normal data without using any information of abnormal data (Chandola et al., 2009;
Pang et al., 2021; Ruff et al., 2021).
AD is useful in numerous real problems such as intrusion detection for video surveillance, fraud detection in finance, and fault detection for sensors.
Many AD methods have been proposed in the past decades (Sch ¨ olkopf et al., 1999; 2001; Tax & Duin, 2004; Liu et al., 2008).
For instance, Sch ¨ olkopf et al.
(2001) proposed the one-class support vector machine (OCSVM) that finds, in a high-dimensional kernel feature space, a hyperplane yielding a large distance between the normal training data and the origin.
Tax & Duin (2004) presented the support vector data description (SVDD), which obtains a spherically shaped boundary (with minimum volume) around the normal training data to identify abnormal samples.
There are also many deep learning based AD methods (Erfani et al., 2016; Ruff et al., 2018; Golan & El-Yaniv, 2018; Abati et al., 2019;
Pidhorskyi et al., 2018; Zong et al., 2018;
Wang et al., 2019; Qiu et al., 2021; Raghuram et al., 2021;
Wang et al., 2021).
Deep learning based AD methods may be organized into three categories.
The first category is based on compression and reconstruction.
These methods usually use autoencoder (Hinton & Salakhutdinov, 2006; Kingma & Welling, 2013) to learn a low-dimensional representation to reconstruct the high-dimensional data (Vincent et al., 2008; Wang et al., 2021).
It is expected that the learned autoencoder on the normal training data has a much higher reconstruction error on unknown abnormal data than on normal data.
The second category is based on the combination of classical one-class classification (Tax & Duin, 2004) and deep learning (Ruff et al., 2018; 2019; Perera &
Patel, 2019; Bhattacharya et al., 2021; Shenkar & Wolf, 2022; Chen et al., 2022).
For instance, Ruff et al.
(2018) proposed a method called deep one-class SVDD.
The main idea is to use deep learning to construct a minimum-radius hypersphere to include all the training data, while the unknown abnormal data are expected to fall outside.
The last category is based on generative learning or adversarial learning (Malhotra et al., 2016;
Deecke et al., 2018; Pidhorskyi et al., 2018; Nguyen et al., 2019; Perera et al., 2019; Goyal et al., 2020; Raghuram et al., 2021; Yan et al., 2021).
For example, Perera et al.
(2019) proposed to use the generative adversarial network (GAN) (Goodfellow et al., 2014) with constrained latent representation to detect anomalies for image data.
Goyal et al.
(2020) presented a method called deep robust one-class classification (DROCC).
The method aims to find a low-dimensional manifold to accommodate the normal data via an adversarial optimization approach.
Although deep learning AD methods have shown promising performance on various datasets, they still have limitations.
For instance, the one-class classification methods such as Deep SVDD (Ruff et al., 2018) only ensure that the normal data could be included by a hypersphere but cannot guarantee that the normal data are distributed evenly in the hypersphere, which may lead to large empty regions in the hypersphere and hence yield incorrect decision boundary.
The adversarial learning methods such as (Nguyen et al., 2019; Perera et al., 2019; Goyal et al., 2020) may suffer from high computational cost and instability in optimization.
In this work, we present a restricted generative projection framework for one-class classification and anomaly detection.
The model of the framework is efficient to train and able to provide reliable decision boundaries for precise anomaly detection.
Our main idea is to train a deep neural network to convert the distribution of normal training data to a target distribution that is simple, compact, and informative, which will provide a reliable decision boundary to identify abnormal data from normal data.
There are many choices for the target distribution, such as truncated Gaussian and uniform on hypersphere.
Our contributions are three-fold.
• We present a novel framework for one-class classification and anomaly detection.
It aims to transform the data distribution to some target distributions that are easy to be violated by unknown abnormal data.
• We present a few simple, compact, and informative target distributions and propose to minimize the distances between the converted data distribution and these target distributions via minimizing the maximum mean discrepancy.
• We conduct extensive experiments to compare the performance of different target distributions and compare our method with state-of-the-art competitors.
The numerical results on five benchmark datasets verify the effectiveness of our methods.
M ETHODOLOGY
Suppose we have a set of m -dimensional training data X
= { x 1 , x 2 , . . .
, x n } drawn from an unknown bounded distribution D x and any samples drawn from D x are normal data.
We want to train a model M on X to determine whether a test data x new is drawn from D x or not.
One may consider to estimate the density function (denoted by p x ) of D x using some techniques such as kernel density estimation (Rosenblatt, 1956).
Suppose the estimation ˆ p x is good enough, then one can determine whether x new is normal or not according to the value of ˆ p x ( x new ) : if ˆ p x ( x new ) is zero or close to zero, x new is an abnormal data point; otherwise, x new is a normal data point 1 .
However, the dimensionality of the data is often high and hence it is very difficult to obtain a good estimation ˆ p x .
We propose to learn a mapping T : R m → R d to transform the unknown bounded distribution D x to a known distribution D z while there still exists a mapping T ′ :
R d → R m that can recover D x from D z approximately.
Let p z be the density function of D z .
Then we can determine whether x new is normal or not according to the value of p z ( T ( x new )) .
To be more precisely, we want to solve the following problem
[Equation]
where M ( · , · ) denotes some distance metric between two distributions and λ is a trade-off parameter for the two terms.
Note that if λ = 0 , T may convert any distribution to D z and lose the ability of distinguishing normal data and abnormal data.
Based on the universal approximation theorems (Pinkus, 1999; Lu et al., 2017) and substantial success of neural networks, we use deep neural networks (DNN) to model T and T ′ respectively.
Let f θ and g ϕ be two DNNs with parameters θ and ϕ respectively.
We solve the following problem
[Equation]
where f θ and g ϕ serve as encoder and decoder respectively.
However, problem (2) is intractable because D x is unknown and D f θ ( x ) , D g ϕ
( f θ ( x )) cannot be computed analytically.
Note that the samples of D x and D g ϕ ( f θ ( x )) are given and paired.
Then the second term in the objective of (2) can be replaced by sample reconstruction error such as 1 n (cid:80) ni
=1 ∥ x i − g ϕ
( f θ ( x i ))
∥ 2 .
On the other hand, we can also sample from D f θ ( x ) and D z easily but their samples are not paired.
Hence, the metric M in the first term of the objective of (2) should be able to measure the distance between two distributions using their finite samples.
To this end, we propose to use the kernel maximum mean discrepancy (MMD) (Gretton et al., 2012) to measure the distance between D f θ ( x ) and D z .
In statistics, MMD is often used for Two-Sample test and its principle is to find a function that assumes different expectations on two different distributions:
[Equation]
where p, q are probability distributions, F is a class of functions f : X → R and H denotes a reproducing kernel Hilbert space.
Its empirical estimate can be given by
[Equation]
where X = { x 1 , .
, x m } and Y = { y 1 , . . .
, y n } are samples consisting of i.i.d observations drawn from p and q , respectively.
k ( · , · ) denotes a kernel function, e.g., k ( x , y )
=
exp( − γ ∥ x − y ∥ 2 ) , a
Gaussian kernel.
Based on the above analysis, we reformulate (2) as
[Equation]
where Z θ = { f θ ( x 1 ) , f θ ( x 2 ) , . .
, f θ ( x n ) } and Z T =
{ z i : z i ∼ D z , i = 1 , . . . , n } .
The first term of the objective function in (5) makes f θ learn the mapping T from data distribution D x to target distribution D z and the second term ensures that f θ can preserve the main information of observations provided that λ is sufficiently large.
Figure 1 shows the paradigm of our method.
[Equation]
Now we introduce four examples of simple and compact D z for (5).
The four distributions are
Gaussian in Hypersphere ( GiHS ), Uniform in Hypersphere ( UiHS ), Uniform between Hyperspheres ( UbHS ), and Uniform on Hypersphere ( UoHS ).
Their 2-dimensional examples are visualized in
Figure 2.
[Figure]
GiHS (Figure 2.a) is actually a truncated Gaussian.
Suppose we want to draw n samples from
GiHS.
A simple approach is drawing (1+ ρ ) n samples from a standard d -dimensional Gaussian and discarding the ρn samples with larger ℓ 2 norms.
The maximum ℓ 2 norm of the remaining n points is the radius of the hypersphere.
One may also use inverse transform method (Marsaglia, 1963).
We have the following proposition (a simple proof can be found in the appendix).
Proposition 2.1.
Suppose z 1 , z 2 , . .
z n are sampled from N ( 0 , I d ) independently.
Then for any r > (cid:112) d/ 2 , the following inequality holds
[Equation]
In the proposition, using union bound, we have
[Equation]
It means a hypersphere of radius r can include all the n samples with probability at least 1 − nρ , where ρ = exp( − ( r 2 + (cid:112) d (2 r 2 − d )) / 2) .
On the other hand, according to (6), if we sample n/ρ samples from N ( 0 , I d ) , the expected number of samples falling into a hypersphere of radius r is at least n .
Thus, suppose we sample n ′ samples and remove the n ′ − n samples with larger ℓ 2 norms, the expected radius of the smallest hypersphere encasing the remaining n samples is at most r 0 , where r 0 is the solution of the function log( n ′ /n )
=
r 2 + (cid:112) d (2 r 2 − d )) / 2 w.r.t r .
UiHS (Figure 2.b) is a hyperball in which all the samples are distributed uniformly.
To sample from
UiHS, we first need to sample from U ( − 1 , 1) d .
Then we discard all the data points outsides the radius-1 hyperball centered at the origin.
The following proposition shows some probability result of sampling from a d -dimensional uniform distribution.
Proposition 2.2.
Suppose z 1 , z 2 , . .
z n are sampled from U ( − 1 , 1) d independently.
Then for any r > 0 , the following inequality holds
[Equation]
Using union bound for (8), we obtain
[Equation]
It means a hypersphere of radius r can include all the n samples with probability at least 1 − nρ , where ρ = 0 .
8 d − 1 (3 r 2 − d ) 2 .
On the other hand, (8) indicates that if we draw n/ρ samples from
U ( − 1 , 1) d , the expected number of samples falling into a hypersphere of radius r is at least n .
Actually, sampling from UiHS is closely related to the Curse of Dimensionality and we need to sample a large number of points from U ( − 1 , 1)
d if d is large because only a small volume of the hypercube is inside the hyperball.
To be more precisely, letting V hypercube be the volume of a hypercube with length 2 r and V hyperball be the volume of a hyperball with radius r , we have
[Equation]
η
[Equation]
where Γ is the gamma function.
Therefore, we need to draw n/η samples from U ( − 1 , 1) d to ensure that the expected number of samples included in the hyperball is n , where η is small if d is large.
UbHS (Figure 2.c) can be obtained via UiHS.
We first sample from UiHS and then remove all samples included by a smaller hypersphere.
Since the volume ratio of two hyperballs with radius rand r 2 is ( r 1 r 2 ) d , we need to draw n/ (1 − ( r 2 /r 1 ) d ) samples from UiHS to ensure that the expected number of samples between the two hyperspheres is n .
UoHS (Figure 2.d) can be easily obtained via sampling from N ( 0 , I d ) .
Specifically, for every z i drawn from N ( 0 , I d ) , we normalize it as z i ← r z i / ∥
z i ∥ , where r is the predefined radius of the hypersphere.
In testing stage, we only use the trained f ∗ θ to calculate anomaly scores.
For a given test sample x new , we define anomaly score s for each target distribution by
[Equation]
for UoHS
[Equation]
for GiHB or UiHS
[Equation]
for UbHS
[Equation] [Equation]
We call our method Restricted Generative Projection (RGP), which has four variants, denoted by
RGP-GiHS , RGP-UiHS , RGP-UbHS , and RGP-UoHS respectively, though any target bounded distribution applies.
C ONNECTION WITH P REVIOUS W ORK
Our method has a connection with the variational autoencoder (VAE) (Kingma & Welling, 2013).
Both methods are autoencoders.
The latent distribution in VAE is often Gaussian and not bounded while the latent distribution in our method is more general and bounded.
The optimizations of VAE and our method are also different: VAE involves KL-divergence while our method involves MMD.
Our method is closely related to Deep SVDD (Ruff et al., 2018).
Both our method and Deep SVDD aim to project the normal training data into some space such that a decision boundary between normal data and unknown abnormal data can be found easily.
Deep SVDD minimizes the sum of the squared distances between the projected data and a predefined center to find a hypersphere to include the normal training data, which cannot ensure that the data in the hypersphere are evenly distributed or close to Gaussian and hence may lead to an ineffective decision boundary.
Compared with the autoencoder based anomaly detection method NAE (Yoon et al., 2021) that uses reconstruction error to normalize autoencoder, our method pays more attention to learning a mapping that can transform the unknown data distribution into a simple and compact target distribution.
Similar to our method, Perera et al.
(2019) also considered bounded latent distribution in autoencoder for anomaly detection.
They proposed to train a denoising autoencoder with a hyper-cube (multi-dimensional uniform) supported latent space, via adversarial training.
Obviously, the latent distribution and optimization of our method are different from theirs.
In addition, the latent distributions of our method are more compact than the multi-dimensional uniform latent distribution of their method.
E XPERIMENTS 4. D ATASETS AND B ASELINES
In this section, we compare the proposed method with several state-of-the-art anomaly detection methods on three tabular datasets and two widely used image datasets for one-class classification.
The datasets are detailed as follows
Abalone (Dua, 2017) is a dataset of physical measurements of abalone to predict the age.
It contains 1,920 instances with 8 attributes.
• Arrhythmia (Rayana, 2016) is an ECG dataset.
It was used to identify arrhythmic samples in five classes and contains 452 instances with 279 attributes.
• Thyroid (Rayana, 2016) is a hypothyroid disease dataset that contains 3,772 instances with 6 attributes.
• Fashion-MNIST
(Xiao et al., 2017) contains 70,000 grey-scale images with 10 classes.
• CIFAR-10 (Krizhevsky et al., 2009) is a widely-used benchmark for image anomaly detection.
It contains 60,000 color images with 10 classes.
We compare our method with three classic shallow models, four deep autoencoder based methods, three deep generative model based methods and some latest anomaly detection methods.
• Classic shallow models : local outlier factor(LOF)
(Breunig et al., 2000), one-class support vector machine(OC-SVM)
(Sch¨olkopf et al., 2001), isolation forest (IF) (Liu et al., 2008).
• Deep autoencoder based methods : denoising auto-encoder (DAE) (Vincent et al., 2008), DCAE (Seeb¨ock et al., 2016), E2E-AE and DAGMM (Zong et al., 2018), DCN (Caron et al., 2018).
• Deep generative model based methods : AnoGAN(Schlegl et al., 2017), ADGAN(Deecke et al., 2018), OCGAN (Perera et al., 2019), • Some latest anomaly detection methods : DeepSVDD(Ruff et al., 2018), GOAD (Bergman & Hoshen, 2020), DROCC (Goyal et al., 2020), HRN (Hu et al., 2020), SCADN (Yan et al., 2021), NeuTraL AD (Qiu et al., 2021), GOCC (Shenkar & Wolf, 2022).
4. I MPLEMENTATION D ETAILS AND E VALUATION M ETRICS
In this section, we introduce the implementation details of the proposed method RGP and describe experimental settings for image and tabular datasets.
Note that all the compared methods do not utilize any pre-trained feature extractors
For the three tabular datasets (Abalone, Arrhythmia, and Thyroid), in our method, f θ , g ϕ are both
MLPs.
We follow the dataset preparation of (Zong et al., 2018) to preprocess the three tabular datasets for one-class classification task.
The hyper-parameter λ is set to 1.0 for the three datasets.
For the two image datasets (Fashion-MNIST and CIFAR-10), in our method, f θ , g ϕ are both CNNs.
Since both image datasets contain 10 different classes, we conduct 10 independent one-class classification tasks on both datasets.
In each task on CIFAR-10, there are 5,000 training samples and 10,000 testing samples.
In each task on Fashion-MNIST, there are 6,000 training samples and 10, testing samples.
The hyper-parameter λ is chosen from { 0 . 05 , 0 . 2 , 0 . 1 , 1 . 0 } and varies for different classes.
In our method, regarding the restricted target distributions GiHS, we first generate a large number (denoted by N ) of samples from Gaussian or uniform, sort the samples according to their ℓ 2 norms, and set r to be the pN -th smallest ℓ 2 norm, where p = 0 . 9 .
In each iteration (mini-batch) of the optimization, we resample Z T according to r .
The sampling for UiHS and UbHS are similar to GiHS and hence omitted here for simplicity.
For UoHS, we draw samples from Gaussian and normalize them to have unit ℓ 2 norm, then they lie on a unit hypersphere uniformly.
The procedure is repeated in each iteration (mini-batch) of the optimization.
We use Adam (Kingma & Ba, 2014) as the optimizer in our method.
For Fashion-MNIST, CIFAR-10, and Arrhythmia, the learning rate is set to 0 . 0001 .
For Abalone and Thyroid, the learning rate is set to 0 .
The details of our network settings are provided in the supplementary material.
All experiments were run on AMD EPYC CPU with 64 cores and with NVIDIA Tesla A100 GPU, CUDA 11.6.
To evaluate the performance of all methods, we follow the previous works such as (Ruff et al., 2018) and (Zong et al., 2018) to use AUC (Area Under the ROC curve) for image datasets and F1-score for tabular datasets.
4. R ESULTS ON I MAGE D ATASETS
Table 1 and Table 2 compare RGP under four different settings with classic shallow methods, deep autoencoder based methods, generative model based methods, and some latest state-of-the-art methods.
We have the following observations.
• Firstly, in contrast to classic shallow methods such as OC-SVM(Sch¨olkopf et al., 2001) and IF (Liu et al., 2008), our RGP has significantly higher AUC scores on all classes of CIFAR-10 and most classes of Fashion-MNIST.
An interesting phenomenon is that most deep learning based methods including ours have inferior performance compared to IF (Liu et al., 2008) on class ’Sandal’ of Fashion-MNIST.
• Our methods outperformed the deep autoencoder based methods and generative model based methods in most cases and have competitive performance compared to the stateof-the-art in all cases.
• RGP has superior performance on most classes of Fashion-MNIST and CIFAR-10 datasets under the setting of UoHS(uniform distribution on hypersphere).
Furthermore, our four settings have relatively close performance on Fashion-MNIST.
On CIFAR-10, UoHS outperformed other settings consistently. [Table] [Table]
4. R ESULTS ON T ABULAR D ATASETS
In Table [Table] we report the F1-scores of our methods in comparison to ten baselines on Arrhythmia,
Thyroid [Table] and Abalone.
Our four methods significantly outperform all baseline methods in all cases [Table]
Particularly, RGP UoHS has 18 . 70% , and 16 . 96% improvements on the three datasets in terms of F1-score compared to the runner-up, respectively.
Compared with the performance improvement of RGP on Fashion-MNIST and CIFAR-10, RGPs on the three tabular datasets are more significant.
In addition to the quantitative results [Table] we choose Thyroid (with 6 attributes) as an example and transform the data distribution to 2-dimensional target distributions, which are visualized in Figure 3.
We see that RGP is effective to transform the data distribution to the restricted target distributions.
Therefore, RGPs can learn clear boundaries for normal data and hence have satisfactory performance in anomaly detection
[Table]
Methods
[Table]
[Figure]
4. A BLATION S TUDY
There is one hyperparameter in our method, namely λ in problem (5).
Now we show the influence of λ on the performance of our method.
Figure 4 shows F1-scores of our methods with λ varying from 0 to 100, on the three tabular datasets.
Too small or too large λ can lower the performance of
RGP.
When λ is very tiny, the reconstruction term of (5) makes less impact on the training target and f θ can easily transform the training data to the target distribution but ignores the importance of original data distribution (see Figure 5).
On the other hand, when λ is very large, the MMD term of optimization objective becomes trivial for the whole training target and f θ under the constraint of reconstruction term more concentrates on the original data distribution yet can not learn a good mapping from data distribution to target distribution.
Figure 5 illustrates the influence of hyperparameter λ on training set of Thyroid dataset.
It can be clearly observed that f θ transform training data to target distribution better with the decrease of the λ . [Figure] λ
= 0.01 λ = 0.
λ = 0. λ = 0.01 λ = 0. λ
= 0.
λ = 0.01 λ = 0.
λ
= 0.
λ = 0.01 λ = 0.
λ
= 0.
[Figure]
C ONCLUSION
We have presented a novel and simple framework for one-class classification and anomaly detection.
Our method RGP aims to convert the data distribution to a simple, compact and informative target distribution that can be easily violated by abnormal data.
We presented four target distributions and the numerical results showed that uniform on hypersphere is more effective than other distributions in anomaly detection.
The reason is that the uniform on hypersphere is much more compact than other distributions and hence provides smaller room for abnormal data points to fall into.
A M ORE NUMERICAL RESULTS
In addition to the results of Tables [Table] we here report the average AUC of all classes in Figure 6.
We see that on Fashion-MNIST, our methods have competitive performance as IF and HRN.
On CIFAR-10, our RGP-UoHS outperformed all other methods [Table]
[Figure]
We also conduct one-class anomaly detection on MNIST, and report experimental results in Table 4. [Figure] [Table] In addition to comparison of evaluation metrics, we also measure the time cost of different types of anomaly detection methods on training phase with GUP acceleration or not.
We run the experiments of one-class classification on MNIST, Fashion-MNIST, CIFAT10 and report the time cost in Table 5, 6, 7. Table 5 compares training time of Deep SVDD (Ruff et al., 2018), DROCC (Goyal et al., 2020), OCGAN (Perera et al., 2019), RGP on MNIST with the same batch size and training epochs, and without GPU acceleration. Table 6 and Table 7 show the training time of Deep SVDD (Ruff et al., 2018), DROCC (Goyal et al., 2020), RGP on Fashion-MNIST, CIFAR10 with accelerating of NVIDIA RTX2080 GPU(1x). Table 6 and Table 7 indicate that the three methods have the same magnitude on time cost with GPU acceleration and our method has a few advantage on both FashionMNIST and CIFAR10 when ensuring the convergence of model.
The training hyperparameters of Deep SVDD (Ruff et al., 2018) and DROCC (Goyal et al., 2020) in table 6 and table 7 are both from the official settings and listed in Table 8. We do not obtain the time cost of OCGAN (Perera et al., 2019) with GPU acceleration since the official code occurs system error when using GPU. [Table]
[Table] [Table] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
B P ROOF FOR P ROPOSITION 2.
Proof.
Letting z 1 , z 2 , . .
z d be i [Figure] i.d Gaussian variables with mean 0 and variance 1.
According to Lemma 1 of (Laurent & Massart, 2000) (in which letting a 1 = · · · = a D = 1 ), the following inequality holds for any t
[Equation]
Letting z
=
( z 1 , z 2 , . . .
, z d ) ⊤ and r 2 =
d + 2 √ dt + 2 t , we have
[Equation]
where d < 2 r 2 .
This finished the proof.
C P ROOF FOR P ROPOSITION 2.
Proof.
Letting z 1 , z 2 , . .
z d be i.i.d variables of U ( − 1 , 1) .
Then E ( z i ) = 13 and Var ( z i )
= 445 , i = 1 , 2 , . . . , n .
Using Chebyshev’s Inequality, for any t > 0 , we obtain
[Equation]
It follows that
[Equation]
[Equation]
Letting r
= d 3 + t , we have
[Equation]
[Equation]
Note that one may obtain tighter tail bound than (16).
D D ETAILED NEURAL NETWORKS ARCHITECTURE
For the two image datasets (Fashion-MNIST and CIFAR-10), in our method, f θ , g ϕ are both CNNs.
The detailed neural network architecture is showed in Table 9.
f θ in channels=1, mid dim=128, if Fashion-MNIST; in channels=3, mid dim=1024, if CIFAR-10.
Basic-Block(in channels, out channels=64)
Basic-Block(in channels=64, out channels=128) Basic-Block(in channels=128, out channels=256)
Basic-Block(in channels=256, out channels=512), AveragePool2d
Linear(mid dim, bias=False) g ϕ
Basic-Block(in channels, out channels=512)
Upsample(scale factor=2, mode=’nearest’)
Basic-Block(in channels=512, out channels=256)
Upsample(scale factor=2, mode=’nearest’)
Basic-Block(in channels=256, out channels=128)
Upsample(scale factor=2, mode=’nearest’)
Basic-Block(in channels=128, out channels=64)
Basic-Block(in channels=64, out channels=32)
Conv2d(in channels=32, out channels=16, kernel size=3, padding=1, bias=False)
BatchNorm2d(16), LeakyReLU
Conv2d(in channels=32, out channels=in channels, kernel size=3, padding=1, bias=False)
Basic-Block(in channels, out channels)
Conv2d(in channels, out channels, kernel size=3, padding=1, bias=False)
BatchNorm2d(out channels), LeakyReLU
Conv2d(out channels, out channels, kernel size=3, padding=1, bias=False)
BatchNorm2d(out channels)
For the three tabular datasets (Abalone, Arrhythmia, Thyroid), in our method, f θ , g ϕ are both MLPs.
The detailed neural network architecture is showed in Table
[Table]