T HE T ILTED V ARIATIONAL A UTOENCODER : I MPROVING O UT - OF -D ISTRIBUTION D ETECTION
A BSTRACT
A problem with using the Gaussian distribution as a prior for a variational autoencoder (VAE) is that the set on which Gaussians have high probability density is small as the latent dimension increases.
This is an issue because VAEs aim to achieve both a high likelihood with respect to a prior distribution and at the same time, separation between points for better reconstruction.
Therefore, a small volume in the high-density region of the prior is problematic because it restricts the separation of latent points.
To address this, we propose a simple generalization of the Gaussian distribution, the tilted Gaussian, whose maximum probability density occurs on a sphere instead of a single point.
The tilted Gaussian has exponentially more volume in high-density regions than the standard Gaussian as a function of the distribution dimension.
We empirically demonstrate that this simple change in the prior distribution improves VAE performance on the task of detecting unsupervised out-of-distribution (OOD) samples.
We also introduce a new OOD testing procedure, called the Will-It-Move test, where the tilted Gaussian achieves remarkable OOD performance.
I NTRODUCTION
Due to its simplicity, the Gaussian distribution is a common prior for the variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014).
One drawback it has is that the region of high probability density becomes relatively smaller as the latent dimension increases.
To see why this is an issue, consider the objective of the VAE.
It tries to encode points such that they are close to the prior and can reconstruct points into their original form.
Given a limited capacity of an encoder/decoder model, points in the latent space must be separated to have significant differences in their reconstructed points.
With a sufficiently complex data set, it would be required to have a large volume in the high density region of the prior distribution to accommodate each of the latent points, while allowing for sufficient separation.
We argue that the Gaussian distribution’s volume under regions of high probability density is not large enough to accommodate real data sets.
To this end, we show that many of the points encoded by Gaussian prior VAEs exist in low-density regions, and that the high-density region remains relatively empty.
In support, Nalisnick et al.
(2019a) report that the latent point at the highest density of a Gaussian VAE trained on MINST was an all-black image.
To deal with these issues, we propose a simple generalization of the Gaussian distribution called the tilted Gaussian distribution.
We create this distribution by “exponentially tilting” the ordinary multivariate Gaussian distribution by its norm.
The operation of exponential tilting is a common procedure in such diverse fields as statistical mechanics, large deviations or importance sampling, but we believe using it for VAEs as we do here is a novel contribution.
The tilted Gaussian has a maximum probability density lying on the surface of a sphere rather than at a single point.
A single parameter corresponds to the sphere’s radius, allowing for control of the volume under the high-density region of the distribution.
We show that the tilted Gaussian has exponentially more volume that the standard Gaussian as as function of the latent dimension, allowing for a far greater proportion of points from a dataset to exist in regions of high probability density.
We investigate a simpler method of increasing the prior volume by using a
Gaussian with large variance, however the effect on performance was minimal D.2.
To demonstrate the benefits of the tilted Gaussian as a prior for the VAE, we focus on the task of
OOD detection.
It has been noted that somewhat surprisingly, VAEs assign high likelihood to OOD points, despite being optimized on a lower bound of the log-likelihood (Nalisnick et al., 2019a;
Choi et al., 2019).
A possible contributing factor to this poor performance is that the high-density region of the latent space is not densely populated by in-distribution (ID) points due to the volume considerations previously detailed.
We show that VAEs using the tilted Gaussian as a prior (which we call the tilted VAE), have a far greater percentage of points in high density regions (see Figurefor an illustration and Section 3.5 for detailed numbers) and perform significantly better on the OOD task (See Table 2).
While the improvement is a step towards robust OOD detection with VAEs, we show that a prior distribution alone cannot achieve the desired level of performance on this task.
Thus, we propose a new test, called the Will-It-Move test, for the OOD problem.
Combined with the tilted Gaussian as a prior, it consistently improves the performance of current methods to perform
OOD detection with VAEs.
(See Section 4.2 for a description and Table 2 for results)
R ELATED W ORK 2. E XTENSIONS OF V ARIATIONAL A UTOENCODERS
In this section, we give a non-exhaustive list of extensions proposed for VAEs.
The majority of approaches aim to increase the flexibility of the prior.
For example, a mixture of Gaussians can be used as an alternative (Dilokthanakul et al., 2016) to the standard Gaussian prior.
The VampPrior attempts to improve upon the mixture of Gaussians by using a mixture of variational posteriors (Tomczak & Welling, 2018).
Another proposal for the prior distribution is the hyperspherical VAE (Davidson et al., 2018), which uses a von-Mises-Fisher (VMF) distribution.
In contrast, our tilted prior is concentrated around the hypersphere as a “soft” constraint, which allows the ordinary normal distribution instead of the VMF to be used.
Other proposed methods use the Dirichlet process (Nalisnick & Smyth, 2017), the Chinese restaurant process (Goyal et al., 2017), and the Gaussian process (Casale et al., 2018).
Hierarchical constructions of priors can be used to construct prior distributions as in Sønderby et al.
(2016), Maaløe et al. (2016), and Maaløe et al.
(2019), using various methods to construct the latent variable hierarchy.
A different approach is to enforce a lower bound on the Kullback–Leibler divergence (KLD) term of the VAE, taken by the delta-VAE (Razavi et al., 2019).
A powerful way to achieve greater flexibility is to use normalizing flows, a class of invertible transformations that can be used to construct more complex posteriors.
2. O UT - OF -D ISTRIBUTION D ETECTION M ETHODS FOR V ARIATIONAL A UTOENCODERS
Arguably the simplest approach to perform OOD detection with VAEs is by using a one-sided test based on the likelihood that VAEs assign to data points (Bishop, 1994).
Given that this method performs surprisingly poorly on the OOD task Nalisnick et al. (2019a), a number of alternative scores have been provided.
Likelihood Ratios (Ren et al., 2019) proposes to use the ratio between two different types of models, one capturing the semantic content of data, the other capturing background information.
Likelihood Regret (Xiao et al., 2020) uses a similar principle, but uses the ratio between an model that is optimized for the training dataset and another optimized for an individual sample.
ROSE (Choi et al., 2021) uses a method to compute how much a sample would update a model’s parameters.
Input complexity (Serr`a et al., 2020) use an estimate of the Kolmogorov complexity as well as the log-likelihood estimate assigned by the VAE.
Density of States Estimation (Morningstar et al., 2021) uses an approach based on statistical physics and directly measures the typicality of different model statistics to classify OOD samples.
Nalisnick et al.
(2019b) uses a typicality test and
Song et al.
(2019) uses batch normalization statistics.
Ran et al.
(2021) takes a different approach by adding Gaussian noise to images and employs a noise contrastive prior to the VAE architecture.
T HE T ILTED G AUSSIAN D ISTRIBUTION AND T HE T ILTED VAE 3. R EVIEW OF THE V ARIATIONAL B OUND USED FOR T RAINING VAE S
Consider a dataset X
= { x i } Ni =1 consisting of N i.i.d.
samples of some distribution in R d x .
We model the data by a two-step process: first, an unobserved latent variable z ∈ R d z is sampled according to a prior distribution p Z ( z ) ; second, a value x i is produced conditionally on the latent z by a parameterized generator (“decoder”)
p θ ( x | z ) according to some unknown parameter θ = θ ∗ .
To attempt to recover θ ∗ , we maximize the marginal log-likelihood of the data (cid:80) Ni =1 log p θ ( x i ) = [Equation]
= [Equation]
log p
[Equation] x i | z ) p Z [Equation] z ) dz .
As the integral over the latent variables is often intractable, the variational lower bound is used to introduce a parameterized inference model (“encoder”)
q ϕ ( z | x ) to approximate the true posterior p θ ( z | x ) .
Given an encoder model q ϕ , the variational lower bound of the log-likelihood is
[Equation]
where D KL ( · ) is the Kullback-Leibler divergence (KLD).
The KLD term in the equation above can be interpreted as fitting the aggregated posterior q ϕ (cid:0)
z | x i (cid:1) to the pre-determined prior p Z ( z ) .
Typically, the standard Gaussian distribution p Z ( z ) ∼ N (0 , I ) is used as the prior and the encoder distribution is q ϕ ( z | x ) ∼ N (cid:0) µ ( x ) , diag ( σ 2 ( x ) (cid:1) ) with parameterized functions µ ( x ) , σ 2 ( x ) ∈ R d z for the encoder mean and variance.
In this case, the KLD term appearing in (1) can be evaluated explicitly as
[Equation]
this formula makes the lower bound of (1) explicit, and can be optimized using stochastic gradient descent.
Gaussian noise is added to each point during training so that the latent point z i assigned to the data point x i is z i = µ ( x i ) + σ ( x i ) N (0 , I ) , see e.g. (Kingma & Welling, 2014).
3. I MPROVING THE VAE
The primary technical contribution of this paper is the proposal of an alternative prior distribution p Z ( z ) , the exponentially tilted Gaussian prior.
As the standard Gaussian prior has maximum density at the origin, the KLD term in the log-likelihood bound (2) forces all latent points into the same location, µ i = 0 and σ i = 1 .
This leads to “crowding” around the origin, making it difficult to differentiate between encoded data samples (Hoffman & Johnson, 2016;
Alemi et al., 2018).
Unlike the standard Gaussian, the tilted prior does not have maximum density at a single point.
Instead, the maximum density occurs at all the points on the hyper-sphere of radius τ as illustrated in Figure 2.
This allows the model to spread out latent points while still optimizing the marginal log-likelihood bound (1).
Additionally, the radii ∥
z ∥ of points drawn from this distribution are near τ with high probability.
Therefore, the norm of an encoded point ∥ z ∥ drawn from z ∼ q ϕ ( z | x ) can form a simple statistic which is an effective test for OOD points (see (7)).
We also obtain a new variational bound, the analogue of equation 2, for the KLD of the tilted Gaussian distribution.
This allows training the tilted VAE by a simple modification to the standard Gaussian.
This new prior can substitute for the standard Gaussian with minimal changes to existing code.
In practice, the only change is replacing the term 12 ∥ µ ∥ 2 in the KLD from equation (2) to 12 ( ∥ µ ∥ − ∥ µ ∗ ∥ ) 2 where ∥ µ ∗ ∥ is a fixed constant depending on the tilting parameter τ .
This corresponds to the KLD of the new tilted model as in equation (6).
3. E XPONENTIALLY T ILTED G AUSSIAN D ISTRIBUTION AND THE T ILTED VAE
Definition 3.1.
For a tilting parameter τ ≥ 0 , the exponentially tilted Gaussian distribution , denoted
N τ (0 , I ) , is the random variable on R d z with probability density ρ τ ( z ) defined by:
[Equation]
Compared to the standard Gaussian, which corresponds to τ = 0 , the tilting term e τ ∥ z ∥ pushes the distribution towards values greater than ∥ z ∥ .
By completing the square, one sees the density is proportional to e − 12 ( || z ||− τ ) 2 , meaning that the density is radially symmetric and has a maximum value at ∥ z ∥ = τ .
An illustration is provided in Figure 2.
The tilted VAE uses the tilted Gaussian as the latent prior p Z ( z ) .
An important feature of the tilted
VAE is that the encoder distribution (i.e. the distribution assigned in latent space to a single input x i ) is not exponentially tilted.
We take this distribution to be a simple Gaussian of the form z ∼ N ( µ, I ) where µ = µ ( x ) is determined by the encoder parameters.
For the sake of simplicity and to reduce computation we have fixed the covariance matrix Σ
= I , whereas for the Gaussian VAE the variance Σ = diag ( σ ) is allowed to be chosen by the encoder.
We believe that it would be possible to also allow this extra flexibility for the tilted VAE but we do not do so here.
Intuitively, the tilted prior allows the encoder distributions to choose µ anywhere on the surface of a hyper-sphere with a minimum KLD, rather than located at the single point µ = 0 , as would be the case when the prior is an standard Gaussian.
Note also that since a Gaussian cannot be perfectly fit to the distribution N τ (0 , I ) , there will always be a non-zero minimum KLD between the encoder and prior distributions, i.e. δ ( τ ) :=
inf µ ∈ R dz D KL ( N ( µ, I ) ∥
N τ (0 , I )) > 0 when τ > 0 .
This can be interpreted as the minimum average amount of information in nats that each sample contains after being encoded.
The decoder is then able to use this information to differentiate distributions in the latent space.
This minimum value δ ( τ ) is referred to as the “committed rate” in Razavi et al.
3. N ORMALIZATION C ONSTANT AND THE T ILTED KLD B OUND
The calculation of the distribution’s normalization constant Z τ and D KL ( N ( µ, I ) ∥
N τ (0 , I )) is summarized below with proofs deferred to Appendix A.
The normalizing constant Z τ satisfies
[Equation]
[Equation]
where M is the Kummer confluent hypergeometric function M ( a, b, z )
= (cid:80) ∞ n =
a ( n ) z n b ( n ) n ! and a ( n ) is the rising factorial a ( n ) = a ( a + 1) . . .
( a + n − 1) .
The KLD can then be written as
[Equation]
where L is the generalized Laguerre polynomial, L ( α ) n ( x ) = (cid:0) n + αn (cid:1) M ( − n, α + 1 , x ) .
During training, instead of computing D KL ( N ( µ, I ) ∥ N τ (0 , I )) exactly (involving the difficult-to-compute
Laguerre polynomial in (5)), we use the following simple quadratic approximation of the KLD, vastly simplifying the computation while preserving the bound (1):
[Equation]
(6) where µ ⋆ ( τ ) = argmin µ D KL ( N ( µ, I )
∥
N τ (0 , I )) and C ∗ ( τ ) =
min µ D KL ( N ( µ, I ) ∥
N τ (0 , I )) is the minimum such value 3 .
This bound allows easy training of the tilted VAE just as (2) allowed training of the standard Gaussian VAE.
Note that the term C ∗ ( τ ) is constant and can therefore be omitted during training.
The proof of (6) is deferred to Appendix A.3.
Figure 5 illustrates the difference between the exact vs approximate KLD in (6) and shows how ∥ µ ⋆ ( τ ) ∥ depends on d z .
3. C OMPARISON B ETWEEN T ILTED G AUSSIAN AND R EGULAR G AUSSIAN
We now highlight some important differences between the tilted and regular Gaussian distribution as the dimension of the space grows.
First, the volume of the space under regions of high probability density is much larger for the tilted Gaussian.
Second, regions of high probability density have a much larger contribution to the total probability for the tilted Gaussian over the regular Gaussian.
To demonstrate these facts, consider a given density function ρ ( z ) and c ∈ [0 , 1] .
We are interested in the set of points that have a density at least c times the maximum possible density.
This set is denoted
R c :=
{ z | ρ ( z ) ≥ c max w ∈ R d z ρ ( w ) } .
In Appendix B, we show that the volume of R c for the tilted
Gaussian grows exponentially faster than the regular Gaussian with respect to the dimension of the space.
The contribution to the total probability by the set R c is given by P ρ [ R c ] :
= (cid:82)
z ∈ R c
ρ ( z )d z .
VAE
For both the tilted and regular Gaussian distribution, the integral cannot be solved in closed-form.
Appendix B shows a comparison between the distributions when c
= 0 .
Empirically, we observe that when using the tilted VAE, a far greater proportion of latent points are embedded in regions of high probability density.
Figure 1 shows an example of the differences in latent spaces when using the two different priors.
[Table]
[Equation] [Table]
[Equation]
[Table]
illustrated in Figure notions of are compared as ”: The of the data set the VAE in Figure 1 has in the region probability P [ z ∈ R c z is chosen according to the latent distribution “ V volume of
O UT - OF -D ISTRIBUTION D ETECTION 4. U SING THE L IKEHOOD AS AN OOD S CORE
We now focus on a particular application of VAEs to demonstrate the advantages of using the tilted
Gaussian as a prior.
The task of OOD detection can be described by considering a training distribution P .
Samples that have a high probability density under the training distribution are considered to be ID, whereas samples that have a low probability density under the training distribution are considered to be OOD.
In practice it is common to use data from a different distribution Q as examples of OOD data.
As we only have access to in-distribution training data, the OOD task we described is a case of positive unlabelled (PU) learning Jaskie & Spanias (2022)
Given that the VAE is optimized to get a lower bound on the marginal log-likelihood, (1) is a natural statistic to determine if a sample is OOD.
Samples with a higher marginal log-likelihood would be considered to be more likely to be ID.
In this work we assume that the reconstruction or likelihood term of (1) takes the form of a
Gaussian likelihood and we approximate the expectation over a single sample q ϕ ( z | x i ) .
This allows the marginal log-likelihood bound to be used as an “OOD score” namely:
[Equation]
(7) where z ∼ q ϕ ( z | x ) is a sample of the latent space image of the point x , and ˆ x is the reconstructed data point from the image z , i.e. ˆ x ∼ p θ ( x | z ) .
We note that value of C ⋆ ( τ ) in the KLD approximation (6) is not required as it is constant for all inputs.
In practice, some value of S OOD ( x ) must be used as a threshold to determine if a point is OOD or not.
By sweeping through all possible threshold values we obtain the full ROC curve.
We conduct an experiment comparing the tilted Gaussian vs the standard Gaussian as a prior for the VAE on two OOD detection tasks.
To give a quantitative measure of OOD classification, we examine the Receiver Operating Characteristics (ROC) curve looking at the relationship between the false positive and true positive rates as the threshold value for (7) changes.
The Area Under the Curve-Receiver Operating Characteristics (AUCROC) (Fawcett, 2006) is used to give a single number measuring OOD detection performance.
Details regarding the full experimental settings can be found in Appendix C and the results are shown in Figure 3 4 .
OOD detection on Fashion-MNIST
[Figure]
i t i v
[Figure]
t
[Figure] 4. T HE W ILL - IT -M OVE T ASK FOR O UT - OF -D ISTRIBUTION D ETECTION 4.2. M OTIVATION : E NTROPIC I SSUES WITH V ARIATIONAL A UTOENCODER As detailed in Caterini & Loaiza-Ganem (2022)
[Figure]
using the marginal likelihood from the VAE as an OOD score has fundamental issues for low-entropy data distributions. To describe this issue theoretically, imagine we are given samples from a “true” distribution P to distinguish from an “outside” distribution Q . We train a parameterized model p θ to maximize the log-likelihood of points from P , i.e. we maximize L P ( θ ) =
E X ∼ P [log p θ ( X )] 5 .
Once p θ is trained, the lower bound of log p θ ( x ) is used as an OOD score; points with low log-likelihood are to be labelled from Q rather than P because one would hope that L P ( θ ) >
L Q ( θ ) for our choice of parameters.
However, this simple log-likelihood comparison can fail when the entropy of the “outside” distribution Q is low compared to that of P .
One can decompose the log-likelihood into two terms:
[Equation]
where H [ P ] is the entropy of the distribution P .
This decomposition shows that comparing the expected log-likelihood of P to that of Q gives
[Equation]
Even if our model θ is perfectly chosen so that D KL ( P ∥ P θ ))
= 0 , it is possible to observe a higher average log-likelihood for points from the “outsider” distribution Q when H [ P ] > H [ Q ] +
D KL ( Q ∥ P θ ) .
This phenomenon means that distributions Q with low entropy, (i.e. H [ Q ] small) can sometimes be hard to detect using a VAE model.
Indeed, experiments show that detecting a constant image as out of distribution can counter-intuitively be quite difficult (see “Constant” row in Table 2).
To deal with this problem, we propose the Will-It-Move testing procedure.
4.2. D ESCRIPTION OF THE W ILL -I T -M OVE T EST
The Will-It-Move (WIM) test works by fine-tuning the parameters of the VAE to try to “push” OOD latent points away from the model prior, while keeping known ID points close to their original prior.
Points that are truly OOD should be easily moved, while points from the original training [Figure] (Note that the true identity of the test points is unknown to the algorithm.)
Originally, the VAE trained only on the training point maps all points to the latent distribution well.
During parameter fine-tuning in the WIM test, only the truly OOD points are easily moved away to the alternative latent distribution (in this case, a standard Gaussian) and can therefore be easily identified.
distribution should be hard to move away from the original prior.
In other words, the answer to “Will-It-Move?” is “Yes” for OOD points and “No” for ID points during the test.
After the WIMfine-tuning is completed, the same score (7) is then used to label unknown points as either OOD or
ID.
By moving points away form the model prior, this allows the KLD term in the likelihood bound to become very large and overcome the entropy problem previously discussed.
To precisely describe the WIM task, consider a training data set X drawn iid from a distribution
P .
We suppose that we have already used X to train a VAE by maximizing the log-likelihood w.r.t.
some given latent distribution Z X .
(e.g. using the tilted Gaussian Z X
=
N τ (0 , I ) ).
The WIM test fine-tunes the parameters of the VAE to label the identity of points from an “unknown” data set U that has samples from both the training and the OOD distribution i.e. U
= { x i } ni =1 ∪{ ˜ x i } mj =1 where x i ∼ P and ˜ x
j ∼ Q .
Note that the identity of the points in U is not known apriori.
To do this, we choose a different latent space distribution for the data U , denoted by Z U .
In the
WIM test, the model fine-tunes the parameters so that points from X stay close to Z X and points from U move to Z U in latent space.
Specifically, we maximize by gradient descent the objective function that is the weighted sum of their log-likelihoods for some α ∈ R :
[Equation]
where L ( Z X , X ) :
= (cid:80) x i ∈ X E q ϕ ( z | x i ) (cid:2) log p θ (cid:0) x
i | z (cid:1)(cid:3) − D KL ( q ϕ ( z | x i ) ∥
Z X )) and analogously
L ( Z U , U ) is the log-likelihood for the data set of unknown points U w.r.t.
the alternative latent distribution Z U .
The first term of (8) is the original training objective of the VAE, and is intended to keep the data set
X in it’s original configuration.
The effect of the second term is to push the points of U toward Z U in latent space.
The hope is that OOD points from Q can easily be moved to Z U , whereas points from P are similar enough to the original data set so that the first term in (8) keeps these points around Z X despite the pull from the second term.
The points that we observe moving during this fine-tuning are therefore identified as the OOD points (hence the name “Will-It-Move?”).
In practice, we use Z X as a tilted Gaussian and Z U as a standard Gaussian.
Implementation details of the WIM test are discussed in D.1.
As an ablation test, we perform the WIM test with both
Z X
and Z U as Gaussian distributions with different location parameters and observed significantly better performance with the tilted Gaussian.
Results from this experiment can be found in D.
R ESULTS AND C OMPARISON TO O THER OOD M ETHODS
[Table] To further validate the performance of our proposed prior, we compare the tilted prior to a variety of methods that achieve top performance in OOD detection with VAEs.
The methods we consider are
Input complexity ( IC (png) ), ( IC (JPEG2000) )
(Serr ` a et al., 2020), likelihood ratios ( Ratio ) (Ren et al., 2019), and likelihood regret ( Regret ) (Xiao et al., 2020).
A VAE with a standard Gaussian prior and log-likelihood OOD score ( Gauss ) is also used as a benchmark.
The tilted VAE and WIM test are denoted Tilt and WIM respectively.
Table 2 is a summary of the results from this analysis.
From this experiment, we observe that the tilted VAE significantly outperforms the Gaussian VAE across a variety of datasets.
Furthermore, the tilted VAE alone has competitive performance with the OOD based scores we compare against.
When the WIM test is used with the tilted prior we observe that this method matches or improves upon the compared methods in all tests considered.
For difficult cases such as when CIFAR-10 is the training distribution and the OOD datasets are SVHN,
LSUN and CELEBA the WIM test achieves significant performance increases over all compared methods.
When looking at cases where the OOD dataset is much simpler that the training distribution, for example the constant dataset, we observe methods that use the log-likelihood estimate as an OOD score ( Gauss and Tilt ) perform poorly.
C ONCLUSION
We propose a generalization of the Gaussian distribution called the tilted Gaussian, and show that it can be implemented in the same way as the standard Gaussian VAE with a simple change to the KLD term.
It’s use as a prior distribution for the VAE was motivated by showing that when the standard Guassian is used as a prior, a large percentage of data points are encoded into regions of low probability density.
We then prove that the tilted Gaussian has exponentially more volume in high probability density regions that the standard Gaussian as a function of the distribution dimension.
Empirically, we show that the tilted VAE encodes a far greater percentage of points on regions of high-probability density.
We introduce a new OOD score for the VAE and empirically demonstrate that it is a consistent improvement over the performance of recent OOD scores for the VAE.
Finally, we perform an ablation study and show that the WIM test performs better when using the tilted
Gaussian prior over the standard Gaussian prior.
While this work investigated the tilted Gaussian distribution in the context of OOD detection with
VAEs, we believe that there are many applications where the tilted Gaussian is a viable drop-in replacement for the standard Gaussian.
We encourage researchers to experiment with this distribution in other situations.
A CKNOWLEDGEMENTS
We thank 3 anonymous reviewers for their thorough reading of the paper, and for the many suggestions which improved the paper.
A F ACTS ABOUT THE T ILTED G AUSSIAN A. D ERIVATION OF THE N ORMALIZATION C ONSTANT Z τ
[Equation]
n ! where µ 1 = E [ χ ] = √ 2Γ
[Equation]
d
[Equation]
M
[Equation]
where we have used the moments of the χ ( d ) distribution and we let M ( a, b, z )
= (cid:80) ∞ n =
a ( n ) b ( n ) z n n ! is the Kummer confluent hypergeometric function and a ( n ) = a ( a + 1) . . .
( a + n − 1) is the rising factorial.
A. E XACT FORMULA FOR THE KLD USED IN THE VAE
Define the shorthand f τ ( µ ) to be the KLD defined below:
[Equation]
where we have used E z ∼N ( µ,I )
[ ∥ z ∥ ] = E [ χ µ ] =
L ( d/ 2 − 1)(cid:16) − ∥ µ ∥ 2 2 (cid:17) and L ( α ) n is the generalized
Laguerre polynomial.
A. A PPROXIMATION OF THE KLD
Let f τ ( µ ) ∈
R be the KLD as in (9).
Note that f τ depends only on the magnitude of µ , not the direction.
With x = ∥ µ ∥ , we can therefore define g τ : R → R by
[Equation]
where ⃗e 1 is the unit vector ⃗e
= (1 , 0 , 0 , . .
. , 0) T .
Proposition A.1.
The function g satisfies
[Equation]
and in particular for τ > 0 ,
[Equation]
Proof.
The formula for g ′′ τ goes by taking the derivative of the definition g τ ( x ) from (10) using d 2 d x 2 12 x 2 = 1 and the elementary fact
[Equation]
Since this is non-negative, it is immediate that g ′′ τ ( x ) ≤ 1 as claimed.
Remark A.2 .
Note that the random variable appearing in the formula for g τ is related to the noncentral Beta distribution
[Equation]
Corollary A.3.
Let µ ∗ ( τ ) be the minimum of f τ so that
f τ ( µ ∗ ( τ )) ≤ f τ ( µ ) for all µ ∈ R d and g τ ( ∥ µ ∗ ( τ ) ∥ ) ≤
g τ ( x ) for all x ∈ R .
Then:
[Equation]
or equivalently:
[Equation]
Proof.
Since g ′′ τ ( x ) ≤ 1 and g is differentiable, by Fermat’s theorem, g ′ τ ( ∥ µ ∗ ( τ ) ∥ ) = 0 .
We can now integrate to find that for x > ∥ µ ∗ ( τ ) ∥
[Equation]
and finally integrating again gives the desired inequality for x > ∥ µ ∗ ( τ ) ∥ :
[Equation]
The same inequality holds for x <
∥ µ ∗ ( τ ) ∥ by integrating from (cid:82) ∥ µ ∗ ( τ ) ∥ x .
The final inequality for f follows by setting x = ∥ µ ∥ .
B R EGIONS OF H IGH P ROBABILITY D ENSITY FOR G AUSSIAN VS T ILTED
To compute the “size” of regions where the probability density function is at least c ∈ (0 , 1) times its maximum, R c = { ρ ( z ) >
c max w ρ ( w ) } , we compute for the standard Gaussian that:
[Equation]
This shows that this region is always a sphere for the standard Gaussian, R c = B √ 2ln
( c − 1 ) (0) .
In contrast, for the tilted Gaussian we have:
[Equation]
This shows that the region of high probability is a “shell”, the difference of two spheres R c =
B τ + √ [Equation] [Equation] 0) \ B
[Equation]
2ln
[Equation]
The formula for the volume of the high dimensional sphere
V d ( r ) =
π d/ 2 Γ(
[Equation]
2 +1) r d from which we can compute volumes:
[Equation]
V ol Tilted ( R c ) =
π d/Γ( d/ 2 + 1) (cid:18)(cid:16) τ
+ (cid:112) 2 ln(1 /c ) (cid:17) d − (cid:16) τ − (cid:112) 2 ln(1 /c )
(cid:17) d (cid:19)
By the “curse/blessing of dimensionality”, the region of high probability density is exponentially larger for the tilted prior as a function of d .
We compute the probability of landing in the region
R 50% by numerically integrating the probability density on these regions below.
[Figure]
= =
= =
=
= =
C E XPERIMENTAL S ETTINGS C. M ODEL S TRUCTURE AND O PTIMIZATION P ROCEDURE
The parameters used for the tilted prior are τ = 30 for the Fashion-MNIST test and τ = 25 for the
CIFAR-10 test.
The WIM test uses α = 0 .
1 for all of experiments.
For all tests we train the VAE for 250 epochs with a batch size of 64.
We use the ADAM optimizer with a learning rate of 10 − 4 and clip gradients that are greater than 100.
The encoder and decoder of the VAE is based on the work done by (Choi et al., 2019) and consists of 5 convolutions layers with a fully connected layer for both µ and σ .
The decoder uses 6 deconvolutional layers and ends with a single convolutional layer.
The use of a fully connected encoder layers for the tilted prior is essential as the KLD is a function of ∥ µ ∥ which is impractical to optimize with a fully convolutional model.
We initialize all convolutional weights from the distribution N (0 , 0 . 2) .
The model details can be viewing in C.1.
On the FashionMNIST test we use latent dimensions d z = 10 and for the CIFAR-10 test we used d z = 100 .
Both
Likelihood Regret and Likelihood Ratio are intended to be run with categorical cross entropy loss, thus we employ this as the reconstruction loss function for all comparison models.
C. I MPLEMENTING M ETHODS
For all comparison methods we use IWAE with 200 samples to derive a lower bound on the loglikelihood.
For likelihood regret, we use gradient descent on all model parameters for 100 steps in each of the tests.
We use the ADAM optimizer for this process at a learning rate of 1e-4, same as the
[Table] [Table]
[Table]
[Table]
training procedure.
We train the background model in Likelihood Ratio by setting the perturbation ratio parameter µ to be 0.2 for all tests.
Besides this, the background model is trained in an identical format to the regular models.
For input complexity we use the OpenCV implementations of the
PNG and JP2 compression algorithms.
C. D ATASETS
The datasets that are used in our experiments are MNIST, Fashion-MNIST, KMNIST, CIFAR-10,
SVHN, CelebA and LSUN.
We also use two synthetic datasets that we call Noise and Constant.
Images from the Noise dataset are created by sampling from the uniform distribution in the range of [0 , 255] for each pixel in an image, where the constant dataset is created by sampling from the uniform distribution in the range of [0 , 255] where all pixels in the image have the same value.
All images are resized to shape 32 x 32.
Color images are converted to gray-scale for the FashionMNIST test by taking the first channel and discarding the other two.
Grayscale images are converted to color images by taking 3 copies of the first channel.
When testing with the CelebA and LSUN datasets, we use 50000 random samples from each due to the large dataset sizes.
D W ILL -I T -M OVE T EST D ETAILS D. I MPLEMENTATION D ETAILS OF THE WIM T EST
The WIM test was implemented by using batches that had equal parts training data, in-distribution test data, and OOD data.
(We used 256 images of each type.)
Our experiments used the full indistribution test-set, which was equivalent to approximately 25% of the OOD dataset.
We trainedepochs on each batch, then tested both datasets for OOD points.
This means that the model enjoys better performance on OOD points that it hasn’t be tuned on, and shows effective generalization.
The gradient of the training data was weighted more strongly by choosing α = 0 .
In time-sensitive applications, it could be infeasible to run the 5 epoch WIM updates on every data point given that the backward pass through the model is far slower than a forward pass.
Instead, one could fine tune once only once on some data, and then run the fine tuned model for OOD detection.
In table 4 we compare the relative number of images per second of the methods used in this paper.
For the WIM test we include the time for the model to do 5 epochs of fine tuning with a final forward pass as well as the time for a single forward pass, which could be performed after the model has already been fine tuned.
D. A BLATION S TUDY
We perform an ablation of the WIM test, by replacing the tilted Gaussian prior with a standard
Gaussian prior, meaning that both Z X and Z U are Gaussian distributions.
To keep the distributions separate in the latent space, we set each element of the location parameter for Z U to be 3.
We observe that the tilted prior consistently improves the performance of the WIM test over the standard
Gaussian prior.
[Table]
[Table]
E C OMPARISON TO L ARGE V ARIANCE G AUSSIAN
We investigate the result of using a standard VAE with a large variance Gaussian prior.
The form of the prior is N (0 , aI ) where a ∈ R + , and we vary a to understand the effect of on OOD detection.
This larger variance also increases the region of high density, but effectively increases the “scale” of the latent space.
This means separating points by adding N (0 , Id ) as part of the encoder becomes less effective (i.e. Increasing the variance of the posterior a is equivalent to decreasing the variance added to each point during encoding.)
It is not surprising therefore that changing the variance does not improve the ordinary VAE.
Results can be found in Table 6.
F O THER DETAILS F. I SORADIAL PROJECTION
The isoradial projection used in Figure 1 and Figure 4 map 10 -dimensional space to 2 -dimensional space, f isoradial : R 10 → R 2 so that the norms of the vectors in the domain R 10 are preserved as the radius in the range R 2 , i.e. R =
||
f isoradial ( z ) ||
2 D = ||
z
|| 10 D for all z ∈ R 10 .
This requirement determines the radius for every point z , so the only thing left to determine is the angle θ in 2D polar coordinates.
In Figure 1 and Figure 4 we chose the angle according to the first two principle components (PCA) of the given data set.
That is we chose θ to be the angle exactly that of the two dimensional
[Table]
[Equation]
vector ( z PCA 1 , z PCA 2 ) of the first two principle components of z , i.e. θ = arctan
( z PCA 2 z PCA 1 ) where z PCA 1 is the first component of the PCA and z PCA 2 is the second component.
These components are linear combinations of the original components determined from the dataset.
The projection f can hence be written as
[Equation]
F. D ISCUSSION OF CLASSIFIERS WITH AUROC < 0 .
For any classifier with an AUROC < 0 .
5 , a better classifier can be constructed by “flipping” the classifier (i.e. when the original classifier outputs “OOD”, the new classifier will label this as indistribution and vice versa).
The AUROC of the new classifier is simply 1 − the AUROC of the original classifier.
This is a simple way to “fix” many of the poor classifiers listed in Table 2.
For clarity, we chose to only report the AUROC of the original classifier.
This makes interpreting
Table 2 straightforward since the exact same classifier is used in each column (as opposed to a mixture of flipped/un-flipped classifiers).
Additionally, since the choice of whether or not to flip is evidently very data dependent, in practice one would not know apriori whether flipping improves the classification score (e.g. if classifying only a single data point, there would be no way to know).
OOD classifiers that consistently output high AUROC scores across many different data sets (such as the WIM method) are therefore much more useful than classifiers that sometimes have to be flipped.
In the setting of OOD for VAEs, this kind of poor performance leading to AUROC < 0 .
5 has been widely reported when the OOD distribution images are much “simpler” than the in-distribution images, see e.g. Nalisnick et al. (2019a).
One possible way to understand this is from an information theory point of view, which is described in Section 4.2.1.