Out-of-Distribution Detection with An Adaptive Likelihood Ratio on Informative Hierarchical VAE
Abstract
Unsupervised out-of-distribution (OOD) detection is essential for the reliabilityof machine learning.
In the literature, existing work has shown that higher-levelsemantics captured by hierarchical VAEs can be used to detect OOD instances.
However, we empirically show that, the inheirt “ posterior collapse ” of hierarchical
VAEs would seriously limit their capacity for OOD detection.
Based on a thoroughanalysis, we propose an informative hierarchical VAE to alleviate this issue throughenhancing the connections between the data sample and its multi-layer stochasticlatent representations during training.
Furthermore, we propose a novel scorefunction for unsupervised OOD detection, referred to as Adaptive Likelihood Ratio,which can selectively aggregate the semantic information on multiple hidden layersof hierarchical VAEs, leading to a strong separability between in-distribution and
OOD samples.
Experimental results demonstrate that our method can significantlyoutperform existing state-of-the-art unsupervised OOD detection approaches.
Introduction
Despite achieving great success in real-world applications recently, existing machine learning (ML)systems are still designed to be tested on the in-distribution dataset, whose statistics are similar tothose of the training set [1].
However, when applied to deal with the dataset consisting of out-ofdistribution (OOD) samples, whose statistics are extremely different from those of the training set,these ML systems would produce a series of incorrect judgments [2, 3].
Considering the fact that
OOD data is very common in real-world applications, pre-execution OOD detection is increasinglyattractive to make sure the reliability and safety of ML systems.
Although several supervised methods[4, 5, 6, 7, 8, 9, 10, 11, 12] have achieved great success in OOD detection, the unsupervised ones aremore practical since the category labels of in-distribution samples are often missing in real-worldapplications, which brings more challenges for OOD detection and is also the focus of this work.
Without labels, likelihood-based models could be a promising way for unsupervised OOD detection,such as flow-based models [13], auto-regressive models [14, 15], and variational autoencoders (VAEs)[16, 17].
Unfortunately, some recent studies have shown that, in some cases, these generative modelstend to achieve higher likelihoods on certain types of OOD samples [1, 18, 19, 20, 21], whichmakes the OOD detection methods based on thresholding likelihood scores problematic.
To addressthis issue, based on the prior knowledge collected from OOD samples, some improvements havebeen made for unsupervised OOD detection, i.e. , Ren et al.
[1] take additional datasets as the OODvalidation sets for choosing the best hyperparameters; Hendrycks et al.
[22] introduce an auxiliarydataset to teach the network to learn more expressive representations for OOD detection; However,the behaviour of borrowing the prior knowledge of OOD data is usually unreasonable in practice,because we will never know the statistic information of OOD samples to be dealt with.
To conduct purely unsupervised OOD detection without the labels or prior assumptions, deepensemble method named WAIC [18] is developed by making full use of the difference between the density estimations of multiple independent models trained on the in-distribution data.
Recently,through capturing the semantic information with multi-layer latent variables, Maaløe et al.
[23]and Havtorn et al.
[21] develop various score functions based on Likelihood-Ratio, which helphierarchical VAEs achieve competitive performance in unsupervised OOD detection.
However, in ourimplementation, we find that the phenomenon of “ posterior collapse ” in hierarchical VAEs still limitstheir performance on OOD detection, where the main reason could be that “ posterior collapse ” willmake the high-level latent variables meaningless and cannot provide faithful summaries for the data.
With this insight in hand, in this paper, we start from rethinking the cause of “ posterior collapse ”in hierarchical VAEs, and then theoretically explain why “ posterior collapse ” will limit the OODdetection performance of these Likelihood-Ratio based methods.
Further, we develop an informativehierarchical VAE to alleviate “ posterior collapse ” and a novel Adaptive Likelihood Ratio scorefunction for unsupervised OOD detection.
The major contributions of this work include:
• With a thorough analysis of “ posterior collapse ” in hierarchical VAEs, we enhance the
connections between the data sample and its multiple latent representations in the expectedlog-likelihood term of evidence lower bound (ELBO) for training, and develop a novel
informative hierarchical VAE to extract more expressive hierarchical latent representations.
• We theoretically explain why alleviating “ posterior collapse ” in hierarchical VAEs can help
the performance of Likelihood Ratio on OOD detection, and then develop a novel score
function for fully unsupervised OOD detection, termed Adaptive Likelihood Ratio, which
owns fewer hyperparameters to be tuned and can make full use of the semantic divergences
between in-distribution and OOD samples across all hidden layers of hierarchical VAEs.
• Combing the informative hierarchical VAE with the Adaptive Likelihood Ratio, we demon-
strate that our method can achieve state-of-the-art OOD detection performance across a
wide range of benchmarks in an unsupervised manner
Background and Related Works2.1 Hierarchical Variational AutoencoderPreliminary
: Extending the basic VAE [16], a hierarchical VAE [24, 25] is defined by the observationx that depends on a hierarchy of stochastic latent variables z
= z 1 , ..., z L , where the generative modelis defined with a top-down structure, formulated as p θ ( x , z ) =
p θ ( x | z 1 ) (cid:81)
L − 1 l =1 p θ ( z l | z l +1 )
p θ ( z L ) ;the inference model is designed to approximate the posterior over these latent variables, commonlyfactorized with a top-down structure as q ϕ ( z | x ) =
(cid:81)
L − 1 l =1 q ϕ
( z l | z l +1 )
q ϕ
( z L | x ) or a bottom-upstructure as q ϕ ( z | x ) = (cid:81)
L − 1 l =1 q ϕ
( z l +1 | z l ) q ϕ
( z 1 | x ) .
The demonstration of these structures could beseen in Fig.
The parameters of the generative and infernce models, denoted as θ and ϕ respectively,can be jointly optimized by maximizing the evidence lower bound (ELBO) expressed as
[Equation]
where L x is denoted as
[Equation]
where D KL ( ·||· ) denotes the KL divergence and maximizing ELBO is equivalent to minimize thedivergence between the varaitional distribution q ϕ ( z | x ) and true posterior p θ ( z | x ) .
Related Works: While the hierarchy of latent stochastic variables can improve the generationcapability of standard VAEs, in practice, the posterior of higher-level stochastic latent variables havea tendency to collapse into the prior, called “ posterior collapse ”.
To address this issue, Sønderbyet al.
[24] propose a Ladder VAE (LVAE) to change the bottom up inference process into a top-downone; Vahdat and Kautz [25] develop a variant of the LVAE, which carefully designs a sophisticatednetwork architecture to achieve better generation quality; Maaløe et al.
[23] combine the bottomup inference with the top-down inference by proposing a bidirectional inference scheme.
Despiteobtaining performance improvements with more flexible inference networks, these hierarchical VAEsstill lack a theoretical guide to alleviate the phenomenon of “ posterior collapse ”.
2.2 Out-of-distribution Detection with Variational Autoencdoer
Problem Formulation: Suppose that there are a set of N training samples { x i } Ni =1 drawn fromthe data distribution x i ∼ p ( x ) , after training a VAE to model the generation of these data samples, 𝒉𝒉𝒉𝒛𝒙 𝒛𝒛𝒙
Top-downInference
Generation 𝒛𝒛𝒛𝒙 𝒛𝒛𝒛
Bottom-up Inference 𝒙 𝒛 𝑳 𝒛 𝒌+ 𝒛
Partial Generation 𝑝 𝜃 (𝑥|𝑧 >𝑘 ) the generative model p θ ( x ) is supposed to detect whether a testing sample x is an outlier, wherethe outlier should have a low density estimation under the true data distribution p ( x ) .
However,on the contrary, former likelihood-based methods found the generative model always assign higherp θ ( x ) for OOD data than in-distribution data [1, 18, 19, 20].
Luckily, although the likelihood basedmethods with VAE are rarely investigated, they have revealed the potential to better address thisproblem without the help of labels or assumptions about the prior.
Related Works: A pioneering VAE-based OOD detection method is Likelihood Regret (LRe)
[26]obtained by iteratively finetuning the decoder parameters of VAE, which is time-consuming butachieves competitive performance in an unsupervised manner.
Maaløe et al.
[23] reveal the potentialof hierarchical VAE for OOD detection and Havtorn et al.
[21] further propose a score function toimprove the model performance, which needs to specify the backbone model.
The backbone modelsin likelihood-based methods [1, 18, 27] can be directly replaced with VAEs, but these methods usuallyunderperform flow-based models like Glow [13] or autoregressive models like PixelCNN [15].
From Informative Hierarchical VAE to Adaptive Likelihood Ratio
3.1 Rethinking of “ posterior collapse ” in Hierarchical VAEs
Firstly, let’s understand the cause of “ posterior collapse ” in hierarchical VAEs theoretically.
Takingan L -layer hierarchical VAE with a top-down inference network as an example, the set of latentvariables can be separated as the lower-level variables z ≤ k = { z 1 , ..., z k } and the higher-level onesz >k = { z k +1 , ..., z L } , where k ∈ { 0 , ..., L − 1 } , then the ELBO in Eq.
(1) can be reformulated as
[Equation]
where q ϕ ( z L | z L +1 ) :
= q ϕ ( z L | x ) , p θ ( z L | z L +1 ) :
= p θ ( z L ) , and the main contribution to theexpected log-likelihood term is coming from the lower-level latent variables z ≤ k before the k th hiddenlayer [23].
Once the generation capacity of the generative model p θ ( x | z ≤ k ) is powerful enough toreconstruct the observation x well, the variational posteriors of higher-level latent variables z >kwill be optimized to be close to their priors, i.e., q ϕ ( z >k | x )
≈ p
θ ( z >k ) , leading the representationslearned by VAE at higher layers to be meaningless and cannot provide faithful summaries for x , whichis well-known as the phenomenon of “ posterior collapse ” or “ latent variable collapse ” [24, 28].
To find the potential solutions to alleviating “ posterior collapse ”, in the following, we reinterpret thisphenomenon from the perspective of information theory [29] by extending the findings in [28] to ahierarchical VAE scenario.
For ease of undertanding, we define the mutual information between thedata x and the higher-level latent variables z >k as
[Equation]
q ϕ ( z >k | x ) − E q ϕ
[Equation]
z >k ) log q ϕ ( z >k ) , which is induced by the variational posterior q ϕ ( z >k | x ) .
Then KL term in Eq.
(3) can be rewritten as
[Equation]
𝑝 𝜃 (𝑥) 𝑝 𝜃 (𝑥|𝑧 >0 ) 𝑝 𝜃 (𝑥|𝑧 >1 ) 𝑝 𝜃 (𝑥|𝑧 >2 ) 𝑝 𝜃 (𝑥|𝑧 >3 ) 𝑝 𝜃 (𝑥|𝑧 >4 ) (a) FashionMNIST (in) 𝑝 𝜃 (𝑥) 𝑝 𝜃 (𝑥|𝑧 >0 ) 𝑝 𝜃 (𝑥|𝑧 >1 ) 𝑝 𝜃 (𝑥|𝑧 >2 ) 𝑝 𝜃 (𝑥|𝑧 >3 ) 𝑝 𝜃 (𝑥|𝑧 >4 ) (b) MNIST (out) where q ϕ ( z >k ) =
E p ( x ) [ q ϕ ( z >k | x )] and the detailed derivation can be found in Appendix A. Bysubstituting Eq.
(4) into Eq.
(3), due to the non-negativity of mutual information and KL divergence,we can find that maximizing the ELBO is opposite to maximizing the mutual information I q ( x , z >k ) .
When I q ( x , z >k ) is minimized to zero, the variational posterior q ϕ ( z >k | x ) will be independent ofthe data x , which leads to the phenomenon of “ posterior collapse ”.
3.2 Why “ posterior collapse ” limits Likelihood-Ratio for OOD Detection
OOD detection has become one of the most important applications of VAEs, which can be appliedto filter OOD samples by setting a threshold on the score of log-likelihood term.
However, somerecent studies have shown that, in some cases, VAEs tend to achieve higher likelihoods on certaintypes of OOD samples [30], which makes the OOD detection rules based on likelihood thresholdproblematic.
Recently, Havtorn et al.
[21] reinterpreted this problematic behavior by providingevidence that the low-level features learned by VAEs generalize well across datasets and dominate theestimated likelihoods.
Inspired by the alternative log-likelihood lower bound [23] that partly replacesthe inference network with the generative model to highlight high-level features, formulated as
[Equation]
Havtorn et al.
[21] considered to subtract L >k x from L x to cancel out the data distribution log p ( x ) ,resulting in a likelihood-ratio score for unsupervised OOD detection as
[Equation]
which discards the likelihood term to prevent the low-level features from dominating and measuresdivergence in the latent space to ensure that data should be in-distribution across all feature levels.
To intuitively understand the nature of success in the likelihood-ratio score and illustrate whyalleviating “posterior collapse” in hierarchical VAEs can improve its performance on OOD detection,we provide an insightful analysis on LLR >k in Eq.
(6) by reformulating it as follows:
[Equation]
where the detailed derivations can be found in Appendix B. Eq.
(7) shows that, when the inferencenetwork q ϕ ( z ≤ k | z >k ) can approximate the true posterior p θ ( z ≤ k | z >k , x ) very well, thorough equivalent replacement, LLR >k will approach the expected KL divergence between the prior p θ ( z ≤ k | z >k )and variational posterior q ϕ ( z ≤ k | z >k ) .
More specifically, conditioned on the expectation of z >kdrawn from its variational posterior q ϕ ( z >k | x ) , LLR >k is developed to calculate the summation ofk KL divergence terms, measuring the distance between the lower-level variables z ≤ k drawn fromthe generative model p θ ( z ≤ k | z >k ) and those from the variational inference network q ϕ ( z ≤ k | z >k ) .
The premise of applying LLR >k to OOD detection is that, after training a hierarahical VAE onin-distribution samples, for each OOD sample, the latent variables z ≤ k generated from the generativemodel p θ ( z ≤ k | z >k ) will be clearly distinct from those drawn from the variational inference networkq ϕ
( z ≤ k | z >k ) .
For ease of understanding the principle, after training a 5-layer hierarchical VAEon FashionMNIST, for each hidden layer, we exhibit the reconstructions of both in-distribution(FashionMNIST) and OOD (MNIST) data samples with the partial generative model p θ ( x | z >k ) conditioned on the latent variables z >k drawn from q ϕ ( z >k | x ) as shown in Fig.
From the results,we can find that, when setting k = 2 , the reconstructions of MNIST samples tend to reflect thehigh-level semantic structures learned from FashionMNIST, indicating that the generation mechanismof p θ ( x | z > 2 ) seems to prevent accurate reconstruction of out-of-distribution data, which impliesthat a score function based on the distance between p θ ( z ≤ 2 | z > 2 ) and q ϕ ( z ≤ 2 | z > 2 ) , like LLR > 2 in
Eq.
(7), could be a promising metric for OOD detection.
However, when the phenomenon of “ posterior collapse ” occurs, the variational posterior q ϕ ( z >k | x )will be independent of the data x , resulting in q ϕ ( z >k | x ) ≈
p θ ( z >k ) , and the reconstructions ofin-distribution and OOD samples, which are generated from the partial generative model p θ ( x | z >k ) ,will be almost the same, such as the visualization examples shown in Fig.
2 by setting k = 3 ork = 4 .
In that case, for each in-distribution sample, the latent variables z ≤ k generated p θ ( z ≤ k | z >k )will be clearly distinct from those drawn from q ϕ ( z ≤ k | z >k ) , and similar conclusions can also beachieved by OOD samples, which will reduce the variance of LLR >k scores between in-distributionand OOD samples and further bring troubles for OOD detection with LLR >k .
3.3 Informative Hierarchical VAE to Alleviate “ posterior collapse ”
Recall to the conflict between the ELBO objective and I q ( x , z >k ) as discussed in Sec. 3.1, whichcauses “ posterior collapse ” in hierarchical VAEs, there could be two main approaches to alleviatethis phenomenon, including: 1) downweight the KL term, like applying a warm-up scheme on it[31], which is the most common heuristic in practice but still cannot essentially address this issue; 2)enhance the connections between the observation and its multi-layer stochastic latent representationsin the expected log-likelihood term, like modifying the generative process described by p θ ( x | z ) [23].
In this paper, focused on exploring the potential of alleviating “ posterior collapse ” with the secondapproach, we try to introduce skip-connection-liked structures into expected log-likelihood term toenhance the connections between the data x and the latent variables z
= z 1 , ..., z L .
However, constrained by the layer-by-layer generation process of hierarchical VAE, there remains a great challengeto introduce physical skip connections into the generative model p θ ( x | z ) , because arbitrarily addingor concatenating the stochastic hidden layers at different semantic levels will hurt the hierarchy ofthese multi-layer latent representations.
We emphasize that the skip connections between the singlestochastic layer and multiple deterministic layers [28] cannot be extended for hierarchical VAEs withmultiple stochastic hidden layers, but our developed method below can be applied to any existinghierarchical VAE, which is one of the main contributions of this paper.
Generally speaking, moving beyond downweighting I q ( x , z >k ) included in the KL term or modifyingthe structure of generative model p θ ( x | z ) , our main idea is to upweight the mutual informationbetween the data x and the higher-level variables z >k , which is denoted as
[Equation]
in the objective function of hierarchical VAEs.
In Eq.
(8), the first item can be approximated by
H p,q ( x | z >k ) =
E p ( x ) q ϕ ( z >k | x ) log p θ ( x | z >k ) and H p ( x ) is a constant, leading to the optimizationdirection of I p ( x , z >k ) is consistent with H p,q ( x | z >k ) .
Thus, targeted at directly maximizingmultiple H p,q ( x | z >k ) , we develop an informative loss for training hierarchical VAEs, denoted as
[Equation]
) where p θ ( x | z >k ) =
E p θ ( z ≤ k | z >k )
[ p θ ( x | z ≤ k )] describes a partial generative model to reconstructthe observation x taking z >k drawn from the variational inference network q ϕ ( z >k | x ) as input; theweight 1 /L before each expected term is introduced to keep numerical stability.
The developed L innot only inherits the terms of ELBO in Eq.
(1), which helps preserve the original model properties of
VAE, but also introduces virtual skip-connection-liked structures with partial generative models toenhance the connections between x and z >k , contributing to alleviating “ posterior collapse ”.
To avoid directly calculating p θ ( x | z >k ) in practice, inspired by [21, 23], the informative loss in
Eq.
(9) can be optimized by maximizing its lower bound ˆ L in , expressed as
[Equation]
where the expected log-likelihood term is the summation of L components and each component denoted as LL >k
= E p θ ( z ≤ k | z >k ) q ϕ ( z >k | x ) [log p θ ( x | z ≤ k )] can be obtained by replacing the inferencenetwork q ϕ ( z ≤ k | z >k ) in the original log-likelihood term as described in Eq.
(3) with the generativemodel p θ ( z ≤ k | z >k ) .
Intuitively, for each expected log-likelihood term LL >k , after sampling thetop variables z >k from the variational posterior q ϕ
( z >k | x ) , these variables z >k will be forcedto reconstruct the observation x with the partial generative model p θ ( x | z >k ) , which builds thestraightforward connections between x and z >k to alleviate “ posterior collapse ”.
We refer to the hierarchical VAE trained with the lower bound of informative loss in Eq.
(10) asinformative hierarchical VAE.
We note that this method is applicable for hierarchical VAEs witheither top-down or bottom-up inference network to explicitly utilize the generative hierarchy of themulti-layered stochastic variables during training, and can be flexibly extended in future works.
3.4 Adaptive Likelihood Ratio for OOD detection
Besides “ posterior collapse ”, an inappropriate choice of k in the likelihood-ratio score function,denoted as LLR >k in Eq.
(6), will also bring negative impact on the performance of applyinghierarchical VAEs for OOD detection.
Recall to the visualization examples in Fig.
when settingk = 0 or k = 1 , the reconstruction quality of either in-distribution or OOD samples is surprisinglyhigh, leading to the LLR >k scores of both in-distribution and OOD samples are relatively small andfurther making it difficult to distinguish whether the data sample is OOD or not.
Moving beyondcherry picking the hyperparameter k on testing OOD samples, which is unreasonable for unsupervised
OOD detection, we develop a novel adaptive likelihood-ratio score function LLR ada , described as
[Equation]
where R ( x , z >k ) is designed to measure the relevance between the data sample x and its latentvariables z >k sampled from the variational posterior q ϕ ( z >k | x ) , specifically defining LLR > − 1 :=
0and R ( x , z > − 1 ) :
= R ( x , z > 0 ) .
More specifically, there are many choices for the definition of
R ( x , z >k ) , but in the following experiments, we only use the log-likelihood score for brevity, byspecifically defining R ( x , z >k ) :
= 1 / log p θ ( x | z >k ) .
The intuition of designing LLR ada is to move beyond the choose of k but adaptively enhancethe importance of some discriminative terms, like LLR > 2 , in the overall score function for OODdetection.
With R ( x , z >k ) to measure the relevance between x and z >k , we find that the adaptiveweight R ( x , z >k − 1 ) R ( x , z >k ) will be relatively large when the data information drop rapidly at the currenthidden layer, like k = 2 in Fig.
2, which can be naturally applied as the importance weights toenlarge the gap between the metric scores of in-distribution and OOD samples.
Compared to theprevious score functions for OOD detection [21, 26], the developed LLR ada in Eq.
(11) owns lesshyperparameters to be tuned, making its performance more stable on various benchmarks.
Morediscussions about LLR ada can be found in Appendix C.
Experiments
4.1 Experimental setup
Datasets: Following [1, 21, 26], we compare our method with previous works on two dataset pairs,including: FashionMNIST [32] (in) /
MNIST [33] (out) and CIFAR10 [34] (in) / SVHN [35] (out),where the suffix “in” and “out” denote the in-distribution dataset and OOD dataset, respectively.
Tobetter evaluate the generalization ability of these methods, we introduce additional OOD datasets: for
FashionMNIST/MNIST pair, we add KMNIST [36], notMNIST [37], Omniglot [38] and SmallNORB[39] datasets; for CIFAR10/SVHN pair, we add CelebA [40], Places365 [41], Flower102 [42] and
LFWPeople [43] datasets.
More details about datasets can be found in Appendix D.
Evaluation and Metrics: We follow the evaluation procedure in Havtorn et al.
[21], where allmethods are trained on the training split of the in-distribution dataset, and their OOD detectionperformance is evaluated on both the testing split of the in-distribution dataset and OOD dataset.
Following previous works’ evaluation approaches [4, 5, 22], we adopt two popular threshold-independentevaluation metrics, including Area Under the Receiver Operator Characteristic (AUROC ↑ ) and Area
[Table] [Table]
Under the Precision Recall Curve (AUPRC ↑ ), and another metric False Positive Rate at 80% truepositive rate (FPR80 ↓ ), where the arrow indicates the direction of improvement.
Baselines: The comparisons in our experiments mainly include two aspects: i ) the comparisons withprevious OOD detection methods to see whether our method can achieve competitive performance;ii ) the comparisons with several hierarchical VAEs to see whether the new training objective of ourmethod can lead to better performance.
For the comparisons in i , the baselines can be divided intothree categories: “ Labels ”: methods using in-distribution data labels [4, 5, 7, 8, 44, 45]; “ Prior ”:methods using the prior knowledge collected from OOD data [1, 22, 27]; and “ Unsupervised ”:methods without any OOD-specific assumptions [18, 21, 26].
For the comparisons in ii , we compareour method with a normal bottom-up inference hierarchical VAE (HVAE)
[16], which is also thebackbone of our method, and its two major variants: a top-down inference hierarchical VAE named
Ladder VAE (LVAE)
[24] and a bidirectional inference hierarchical VAE (BIVA) [23].
More detailsof these baselines and the categories they belong to can be found in Appendix E.
Implementation Details: For the comparisons on FashionMNIST(in)/MNIST(out), we set thenetwork structure of hierarchical VAEs as [16, 8, 4] and [32, 24, 16, 8, 4] from shallow to deep,respectively.
For CIFAR10(in)/SVHN(out), we set the network structure as [128, 64, 32] and [128, 64,32, 28, 24], respectively.
For optimization, we adopt the same Adam optimizer [46] with a learningrate of 3e-4.
We train all models in comparison by setting the batch size as 128 and the max epoch as1000.
All experiments are performed on a PC with an NVIDIA RTX 3090 GPU and the our code isimplemented with PyTorch [47].
More implementation details can be found in Appendix F.
4.2 Quantitative Comparisons
Overall Comparisons: Following the experimental settings in Sec. 4.1, we exhibit the experimentalresults in in Tab. 1.
From the results, we can find that our method with the LLR ada score function iscomparable with those non-ensemble completely unsupervised methods in FashionMNIST/MNIST,and significantly outperform them in CIFAR10/SVHN.
We emphasize that, without utilizing thelabels of in-distribution samples [4, 5, 7, 8, 44, 45] or the prior knowledge collected from OODsamples [1, 22, 27], our method can still achieve competitive performance with these methods.
Effectiveness of LLR ada : Focused on the comparison between LLR >k and LLR ada exhibited in
Tab. 2 and Fig. 3(c), when the performance of LLR >k is sensitive to the selection of k , we can findthat the performance of LLR ada can approach the best performance achieved by LLR >k with theoptimal k , as shown in the right part of Tab.
(CIFAR/SVHN).
Furthermore, when the performance of
LLR >k is stable, the developed LLR ada can still achieve comparable OOD detection performance,as shown in the left part of Tab. 2 (FashionMNIST/MNIST) and Fig. 3(c).
The experimental resultsabove demonstrate the adaptability of our developed LLR ada .
[Table]
in) /
MNIST (out)
Ours >2 BIVA >2 LVAE >
HVAE >
M1:97.4 M1:96.9 M1:92.
M1:97.
(a) 0.
1.0 False Positive Rate 0.
T r u e P o s i
t
i v e R a t e
CIFAR10 (in) / SVHN (out)
Ours >2 BIVA >2 LVAE >
HVAE >
M1:86.3M1:93.0 M1:77.
M1:75.
(b) 0.
1.0 False Positive Rate 0.
T r u e P o s i
t
i v e R a t e
CIFAR10 (in) / SVHN (out)
Ours >1 Ours >2 Ours >
Ours >
Ours ada M1:94.2 M1:93.8 M1:95.
M1:95.8 M1:92.
(c)
Effectiveness of Informative Hierarchical VAE: Taking the same LLR ada as the score function,we compare the performance of informative hierarchical VAE with other hierarchical VAEs in Tab. 3.
From the results, we can find that our model outperforms others, indicating the effectiveness ofalleviating “ posterior collapse ” with informative hierarchical VAE for OOD detection.
To evaluate the degree of “ posterior collapse ”, we use the ROC curves in Fig.
3 and the empirical densities in Fig.
4 to compare the performance of LLR > 2 scores based on the top-level latent variables z 2of these VAEs.
The ROC results in Fig.
3 demonstrate the superiority of the informative hierarchical
VAE, confirming that our model can provide more expressive higher-level latent representations for
OOD detection.
The empirical densities in Fig.
4 show that z 2 learned by our model on the CIFAR
[Figure]
in)/SVHN (out) pairs owns better separability than those learned by BIVA.
[Figure]
4.3 Qualitative Analysis
Meaningful Semantic Space Learned by Informative Hierarchical VAE:
Following the sameprocedure as Fig.
2, we visualize more reconstructed samples on various benchmarks in Fig.
Fromthe visualized results, for both in-distribution and OOD samples, we can find that the quality of thereconstructions generated from p θ ( x
| z > 0 ) is surprisedly high, indicating that these reconstructionsare almost dominated by the low-level features, which potentially explains the previous problematicphenomenon that these methods based on single-layer likelihood will assign higher likelihood scoresfor OOD samples and fail on OOD detection.
Focused on the reconstructions generated by p θ ( x | z > 2 ) ,we can find that the developed informative hierarchical VAE can provide more realistic and clearreconstructed samples for both in-distribution and OOD inputs, indicating that our model can learna more meaningful high-level latent semantic space than other models.
Furthermore, based onproviding higher-quality reconstruction for OOD samples, the gap between the metric scores of
[Figure]
x z >z >z >z >z >(a) Ours/FashionMNIST x z >z >z >z >z >(
[Figure]
/MNIST x z >z >z >z >z >(
[Figure]
HVAE
[Figure]
FashionMNIST x z >z >z >z >z >(
[Figure]
HVAE/MNIST 𝑥 𝑧 >0 𝑧 > 𝑧 > (
[Figure]
/CIFAR10 (in
[Figure]
𝑥 𝑧 >0 𝑧 > 𝑧 > (
[Figure] [Figure]
/SVHN (
[Figure]
𝑥 𝑧 >0 𝑧 > 𝑧 > (
[Figure]
[Figure]
in-distribution and OOD samples in informative hierarchical VAE tend to be larger than models,leading to performance as shown in
Alleviating “ Posterior Collapse ” with Informative Hierarchical VAE:
As discussed in Sec. 3.1,“ posterior collapse ” will cause the higher-level latent variables to become uninformative.
Consideringthe developed informative hierarchical VAE shares the same network structure with the basic HVAE,in this part, we focus on evaluating whether our model can alleviate “ posterior collapse ” in higherlayers when the network depth becomes deeper.
As shown in Fig.
6, for both in-distribution data and
OOD data samples, the overall quality of the reconstructions generated by our model is significantlyhigher than those generated by the basic HVAE.
Specifically, for FashionMNIST (in)/MNIST (out),the reconstructions generated by p θ ( x | z > 3 ) and p θ ( x | z > 4 ) of HVAE are almost the same, indicatingthat its posterior described by q ϕ ( z > 3 | x ) or q ϕ ( z > 4 | x ) tend to collapse to a prior distribution about
T-shirts.
On the contrary, the reconstructions generated by the our model are still realistic, wherethe reconstructions of MINST (out) samples generated by q ϕ ( z > 3 | x ) or q ϕ ( z > 4 | x ) preserve thesemantic structural information learned form FashionMNIST (in), explaining the underlying reasonwhy our model can achieve better OOD detection performance.
Similar conclusions can be achievedby the experimental results on CIFAR10 (in)/SVHN (out) as shown in Fig. 6.
Conclusion
In this paper, after presenting a thorough analysis of “ posterior collapse ”, we develop a novelinformative hierarchical VAE to extract more expressive hierarchical latent representations by alleviating “ posterior collapse ”.
Then we theoretically explain why “ posterior collapse ” will limit theperformance of existing hierarchical VAEs, and develop a novel Adaptive Likelihood Ratio scorefunction for unsupervised OOD detection.
Experiments demonstrate the effectiveness of our method,whose main thought can be borrowed other hierarchical VAEs to improving their performance ondownstream tasks relied on the hierarchy of latent representations.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See Abstract
(b) Did you describe the limitations of your work?
[Yes] See Appendix H
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A andAppendix B
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tal material
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.1 and Appendix F(c)
Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Appendix G
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1 and Appendix F
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplemental
material
(b) Did you mention the license of the assets?
[Yes] See supplemental material
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix
Due to the page limitation, we put more technical details here for ease of understanding.
We willappreciate it so much if the audience can have a careful reading of the following Appendix.
A More discussions about “Posterior Collapse”
Here, we present a detailed reinterpretation for intuitively understanding the conflict of the optimization between the ELBO L and the mutual information I q ( x , z >k ) .
Then, we provide additionalexperiments to meausre the “posterior collapse” in higher layer of the latent variables z >k .
A.1 Detailed Derivations for Section
First, we present a detailed formulation of the aggregated posterior q ϕ ( z >k ) as
[Equation]
Under the setting of top-down inference structure, with the aggregated posterior q ϕ ( z >k ) , whereq ϕ ( z L | z L +1 ) :
= q ϕ ( z L | x ) , and p θ ( z L | z L +1 ) :
= p θ ( z L ) , the KL term in Eq.
(3) can be rewrittenas follows:
[Equation]
) (cid:21) + E p ( x ) [
L (cid:88) l =
[Table]
+
E q ϕ ( z l | z l +1 ) log q ϕ ( z l | z l +1 ) p
θ ( z l | z l +1 )]
=
E p ( x ) (cid:20)(cid:88) k l =
[Table]
D KL ( q ϕ ( z l | z l +1 )
||
p θ ( z l | z l +1 ))
(cid:21) + E p
x ) [
L (cid:88) l =
[Table]
+
E q ϕ ( z >
[Table]
| x )
[Table] [Table]
| z
[Table]
1 ) p θ (
[Table]
|
[Table]
1 )] =
[Table]
p ( x ) (cid:20)(cid:88) k
[Table]
=
[Table]
D
[Table] [Table]
[Table]
| z
[Table]
|| p
[Table] [Table]
)) (cid:21) +
[Table]
p
[Table]
[ E
[Table]
>
[Table]
log
L
[Table]
l = k +
[Table]
( z l | z l +1 ) p
[Table]
( z l | z
[Table]
=
E p
[Table]
) (cid:20)(cid:88)
[Table]
=
[Table]
D
[Table] [Table]
( z l | z
[Table]
1 ) || p
[Table] [Table]
l | z
[Table]
1 )) (cid:21) + E p
x ) [ D KL (
[Table]
ϕ ( z >k | x ) || p θ (
[Table]
>k ))]
=
[Table]
p
[Table]
x
[Table]
(cid:20)(cid:88)
[Table]
=
[Table]
D
[Table] [Table]
[Table]
|
[Table]
|| p
[Table] [Table]
)) (cid:21) +
[Table]
p
[Table]
=
[Table]
(cid:20)(cid:88)
[Table]
z l | z l +1 ))
(cid:21) + E p ( x ) q ϕ ( z >k | x ) log q ϕ ( z >k | x ) − E p ( x ) q ( z >k | x ) log p θ ( z >k ) =
E p ( x ) (cid:20)(cid:88) k
l =1 D KL ( q ϕ ( z l | z l +1 )
||
p θ ( z l | z l +1 ))
(cid:21) + E p ( x ) q ϕ ( z >k | x ) log q ϕ ( z >k | x ) − E q ϕ
( z >k ) log q θ ( z >k )
+ E q ϕ
( z >k ) log q θ ( z >k ) − E p ( x ) q ϕ ( z >k | x ) log p θ ( z >k ) =
E p ( x ) (cid:20)(cid:88) k
l =1 D KL ( q ϕ ( z l | z l +1 )
||
p θ ( z l | z l +1 ))
(cid:21) + E p ( x ) q ϕ ( z >k | x ) log q ϕ ( z >k | x ) − E q ϕ
( z >k ) log q θ ( z >k )
+ E q ϕ
( z >k ) log q θ ( z >k ) − E q ϕ
( z >k ) log p θ ( z >k ) =
E p ( x ) (cid:20)(cid:88) k
l =1 D KL ( q ϕ ( z l | z l +1 )
||
p θ ( z l | z l +1 ))
(cid:21) + E p ( x ) q ϕ ( z >k | x ) log q ϕ ( z >k | x ) − E q ϕ
( z >K ) log q ϕ ( z >k )
+ D KL ( q ϕ ( z >k )
||
p θ ( z >k ))
=
E p ( x ) (cid:20)(cid:88) k l =1 D KL ( q ϕ ( z l | z l +1 ) ||
p θ ( z l | z l +1 ))
[Equation]
[Equation]
Thus, the ELBO in Eq.
(3) can be rewritten as
[Equation]
Therefore, maximizing the ELBO will be opposite to maximizing the I q ( x , z >k ) , which leads thehigher-layers posterior latent variables z >k to be independent of the input data x and collapse to theuninformative p θ ( z >k ) .
A.2 Quantitative experiments on “Posterior Collapse”
[Table]
Since the “posterior collapse” of the in-distribution data would lead to a larger Likelihood Ratio
LLR >k in high layers, which is harmful for OOD detection, we add an additional experiment here totestify the “posterior collapse” with the metric average bits per dim as shown in Table 4.
The L >L − 1 xis the ELBO for the partial generative model, which could be used to evaluate the reconstructionquality of z L , and more details can be found in Eq.
The average bits per dim is calculated in the testing split of the in-distribution dataset and the OOD detection performance is tested with LLR >L − 1 , where L = 5 in the FashionMNIST/MNIST pair and L = 3 in the CIFAR10/SVHN pair.
Note that, LLR >L − 1 =
L x − L >L − 1 x .
FashionMNIST(in)/MNIST(out)
[Table]
Metthod
[Equation]
[Table]
As the results shown in Table 4, although the baselines for comparison (HVAE, LVAE, and BIVA)can obtain better reconstruction performance on L x , they still suffer from a large shrink in L >L − 1 x ,which is mainly caused by the “posterior collapse”.
On the contrary, the developed informative
HVAE can acquire stable performance from L x to L >L − 1 x , resulting in a smaller Likelihood Ratio
LLR >L − 1 , which illustrates why our method can achieve much better performance on unsupervised
OOD detection.
B Derivations for analyzing the LLR >k in Section 3.
For ease of understanding of the Eq.
(6), we give a detailed derivation below, which is mostly basedon the Havtorn et al.
First, we define a looser ELBO for each observation x as below of the partial generative modelp θ ( x | z >k ) =
E p θ ( z ≤ k | z >k )
[ p θ ( x | z ≤ k )] , which reconstructs the observation x by taking z >k drawnfrom the variational inference network q ϕ
( z >k | x ) ,
[Equation]
recall to the common ELBO for each observation x as
[Equation]
then, the LLR >k is defined as
[Equation]
Further
[Equation]
the detailed derivation for Eq.
(7) is as follows:
[Equation]
k
[Equation]
(18) when the approximated posterior q ϕ
( z ≤ k | z >k ) is closer to the true posterior p θ ( z ≤ k | z >k , x ) aftertraining, i.e. , D KL ( q ϕ ( z ≤ k | z >k )
||
p θ ( z ≤ k | z >k , x ))
≈
and D KL ( p θ ( z ≤ k | z >k )
||
p θ ( z ≤ k | z >k , x ))
≈ D KL ( p θ ( z ≤ k | z >k ) ||
q ϕ ( z ≤ k | z >k )) .
C
More discussions about LLR ada
Recall to the visualization exhibited in Fig.
2, when setting k = 0 or k = 1 , we can find that thequality of the reconstructions generated from p θ ( x | z > 0 ) is surprisedly high, indicating that
the
KL-divergence between p θ ( z ≤ k | z >k ) and q ϕ ( z ≤ k | z >k ) is small for both in-distribution and OODsamples, which makes it problematic for OOD detection with single-layer likelihood LLR > 0 or
LLR > 1 ; when setting k = 2 , the KL divergence between p θ ( z ≤ k | z >k ) and q ϕ ( z ≤ k | z >k ) will berelative small for in-distribution samples, but large for OOD samples, which is the main reason forthe success of LLR > 2 ; however, when setting k = 3 or k = 4 , the latent variables z ≤ k generatedp θ ( z ≤ k | z >k ) will be clearly distinct from those drawn from q θ ( z ≤ k | z >k , x ) for both in-distributionand OOD samples, resulting in that the performance of OOD detection with LLR > 3 or LLR > 4 willbe worse than LLR > 2 .
The reason why OOD detection based on LLR > 3 or LLR > 4 can outperform
OOD detection based on LLR > 0 or LLR > 1 is that the generative model p θ ( z ≤ k | z >k ) can still learnthe generation mechanism of in-distribution samples at higher hidden layers, even when “ posteriorcollapse ” occurs.
Based on these findings, the intuition of designing LLR ada is to move beyond the choose of k butenhance the importance of the score over some discriminative layers, like LLR > 2 , in the overallmetric for OOD detection.
To achieve these goals, we use R ( x , z >k ) to measure the relevancebetween x and z >k , and the adaptive weight R ( x , z >k − 1 ) R
( x , z >k ) in LLR ada will be relatively large whenthe data information drop rapidly at the current hidden layer, like k = 2 .
We admit that “ posteriorcollapse ” will still hurt the performance of LLR ada , leading to worse performance than LLR >kwith the optimal k , but it can avoid the unreasonable hyper-parameter adjustment based on OODsamples.
Moreover, with the informative hierarchical VAE to alleviate “ posterior collapse ”, theperformance of LLR ada will be even better than LLR > 2 on some datasets.
D More Details of the Datasets
We use additional datasets to evaluate the OOD detection performance.
For FashionMNIST/MNIST pair, we add KMNIST [36], notMNIST [37], Omniglot [38] and SmallNORB [39] datasets.
KMNIST is a dataset, adapted from Kuzushiji Dataset, as a drop-in replacementfor MNIST dataset, which contains 70,000 28 × 28 grayscale images.
notMNIST is a dataset madeby 547,838 28 × 28 grayscale images of extracted glyphs from some publicly available fonts withletters A-J taken from different fonts.
Omniglot contains 32,460 28 × 28 grayscale images of 1623different handwritten characters from 50 different alphabets.
SmallNORB contains 97,200 28 × 28grayscale images of 50 toys belonging to 5 generic categories: four-legged animals, human figures,airplanes, trucks, and cars.
For CIFAR10/SVHN pair, we add CelebA [40], Places365 [41], Flower102 [42] and LFWPeople[43] datasets.
CelebA is a large-scale face attributes dataset with more than 200K celebrity images,each with 40 attribute annotations.
Places365 contains 1.8 million train images from 365 scenecategories, 50 images per category in the validation set, and 900 images per category in the testingset.
Flowers102 is an image classification dataset consisting of 102 flower categories, where flowerswere chosen to be flowers commonly occurring in the United Kingdom and each class consists ofbetween 40 and 258 images.
LFWPeople contains more than 13,000 images of faces collected fromthe web.
All these datasets’ images would be randomly cropped into the dimension of 32 × 32 × 3before sending into the models.
E Details of the Baselines
Due to the space limitation, we use the abbreviation for each baseline in Table 1.
Here, we give adetailed description for each baseline of the three categories:
• “Labels” (Methods using in-distribution data labels y ): maximum softmax classification
probability (CP) method [4] and its variants, denoted as "CP", "CP(OOD)" with OOD
as noise class, "CP(Cal)" with calibration on OOD and "CP(Ent)" with entropy of soft-
max classification probability p ( y | x ) , and Mahalanobis distance (MD) method [8], latent
Mahalanobis distance (LMD) method [44], ODIN method [7], VIB method [5] and deep
ensembles (DE) method [45] with 20 classifiers;
• “Prior” (Methods using prior knowledge assumption of OOD): Likelihood Ratio (LR)
method [1] with different backbones, denoted as "LR(PC)" with backbone PixcelCNN,
"LR(VAE)" with VAE and "LR(BC)" with binary classifier), Outlier exposure (OE) method
[22] and Input complexity (IC) method [27] with different backbones, denoted as "IC(PC)"
with backbone PixcelCNN, "IC(Glow)" with backbone Glow and "IC(HVAE)" with back-
bone HVAE;
• “Unsupervised” (Methods with no OOD-specific assumptions): Ensemble methods: WAIC
method [18] with different backbones, denoted as "WAIC(5Glow)" with 5 Glow models,
"WAIC(5VAE)" with 5 VAE models and "WAIC(5PC)" with 5 PixcelCNN models; Not
ensembles methods: Likelihood regret method [26] and its variant "Likelihood regret(z)",
Log-Likelihood Ratio ( LLR ) method [21], which achieved the best performance with
" LLR > 1 (HVAE)" (hyperparameter k = 1 and backbone method 3-layer HVAE trained on
binarized data) for FashionMNIST(in)/MNIST(out) pair and " LLR > 2 (BIVA)"
(hyperparam-
eter k
=
and backbone method 10-layer BIVA) for CIFAR10(in)/SVHN(out) pair.
For this
LLR method, we denote their best combinations’ result in Tab. 1 as "HVK" (Hierarchical
VAEs Know what they don’t know).
F Details of the Implementation
To make sure the new training objective in Eq.
(10) can really lead to an informative hierarchical
VAE, we do not use the warm-up trick or the free bits trick.
However, we apply the warm-up trick(200 epochs for the Warmup anneal period) and free bits trick (2 nats per z i and 400 epochs forthe free bits period) to the other three hierarchical VAEs (HVAE, LVAE, and BIVA) to empiricallyalleviate the posterior collapse, which is proven in Havtorn et al.
[21] and we follow their procedureto train these three hierarchical VAEs.
[Table]
Following Havtorn et al.
[21], the VAE-based methods’ results are computed with 1000 importancesamples.
However, our method’s results only get slight improvement after sampling, considering thecomputation burden brought by it, we report the results of our method without importance sampling.
But we also use the importance sampling for other baseline hierarchical VAEs (HVAE, LVAE, and
BIVA).
G Error Bar
We randomly run 5 seeds for our method in experiments and report the error bar as below.
[Table]
H Limitation
The developed informative hierarchical VAE can alleviate “posterior collapse” to a certain degree,but still cannot completely avoid the appearing of this phenomenon in VAEs.
Then, the developed
LLR ada is not the optimal choice of score function of unsupervised OOD detection, which needs tobe investigated in the future work.
I Broader Impact
The developed method in this paper can be straightforwardly applied to real-word applications basedon hierarchical VAEs, and alleviate their “posterior collapse” to achieve better model performance.
The adaptive score function can be used for purely unsupervised OOD detection, which can boost thereliability and safety of recent machine learning (ML) systems.
J Comparison with other methods to alleviate “posterior collapse”
We provide more comparisons with the methods designed for alleviating the “posterior collapse”,including "Warm-up", "OverSmooth", "BIVA", and Our method "Informative".
"Warm-up" [24]gradually increases the weight of the KL-divergence term in the learning objective from 0 to 1and we set the warm-up epochs as 200 with a total training epoch as 1000.
"OverSmooth" [48]sets the σ x as a one-dimensional parameter and updated according to the training objective, wherethe reconstruction likelihood function is p θ ( x | z ) =
N ( x | µ x ( z ) , σ 2 x I ) .
"BIVA" [23] introduces a bidirectional inference and generative network architecture, but it changes the original structure ofvanilla hierarchical VAE and may hurt the hierarchy of the latent variables.
To provide an intuitivecomparison of these methods’s effect on alleviating “posterior collapse”, we introduce the OODdetection performance on a vanilla hierachical VAE, termd as "Vanilla", as an additional baseline.
Then, we testify these methods’ performance under 3 different score methods for OOD detection,where L x represents the evidence lower bound (ELBO) of the VAE, LLR >L − 1 represents the gapbetween partial ELBO of the highest-level latent variables and the the ELBO of lowest-level latentvariables, and LLR ada is an adaptive score that evalutes the whole hierarchy of the latent variables.
As shown in Table 6 and Table 7, the developed informative hierarchical VAE, termed as "Informative",outperforms other methods significantly especially under the score LLR >L − 1 and LLR ada .
HVAE: FashionMNIST (in) / MNIST (out))
Score + Methods
AUROC
[Table]
L x + Vanilla [16] 15.
L x + Warm-up [24] 26.
L x + OverSmooth [48] 45.
L x + BIVA [23] 26.
L x + Informative(ours) 49.
LLR >L − 1 + Vanilla 33.
LLR >L − 1 + Warm-up 47.
LLR >L − 1 + OverSmooth 81.
LLR >L − 1 + BIVA 35.
LLR >L − 1 + Informative(ours) 98.
LLR ada +
[Table] [Table]
K Comparison with non-VAE methods
Likelihood-based methods are promising to detect the OOD data in an unsupervised manner, sincethey could give an estimation of the data x ’s likelihood p ( x ) under the learned distribution p ofin-distribution data.
Among likelihood-based methods, Flow-based models [13], auto-regressivemodels [14], and variational auto-encoder (VAE) models are popular for OOD detection tasks.
HVK[21] achieves the state-of-the-art of OOD detection under the unsupervised setting.
Though ourmethod could outperform HVK, it is still interesting to have an comparison with other two type ofmodels (Flow-based and auto-regressive models) to see whether our methods could also outperformthem.
We add 3 non-VAE methods as our baselines for comparison: "Glow", "Flow+Group", and "PixcelCNN++".
"Glow" [19] try to do OOD detection with Glow model and find the counterfactualbehaviour of assigning high likelihood to OOD data specially in the model family of Flow models."FLow+Group
[49] is an SOTA flow-based OOD detection method but is not initially designed forour setting, but for group OOD detection, where their model needs to justify whether a batch ofsample { x 1 , x 2 , ..., x n } , ( n > 1) is an OOD batch.
Note that, the batch size n is set as 5, 10, and 20in their paper.
Luckily, "FLow+Group" also modify their method via data augmentaion to the point
OOD detection situation, i.e., n = 1 , which is the same as our setting, and we directly report their
OOD detection results in their Appendix F. "PixcelCNN++" [1] propose to use an auto-regressivemodel (PixcelCNN++) for OOD detection with the help of additional OOD datasets like NotMNISTdataset, and we report the results of this model under our setting (no additional datasets to helptraining).
As shown in Table 8 and Table
[Table]
Our method could also outperform other non-VAE methods.
[Table] [Table]
[Table]
L Comparisons of Score Functions
OOD detection methods need to assign a score for each data sample and detect the OOD data outaccording to this score.
However, since the score method LLR >k proposed by HVK [21] needsto select the optimal hyperparameter k to achieve the best OOD detection performance, which isunreasonable under the unsupervised setting, we design a novel score function named LLR ada thatdoes not need to select the k .
Thanks for the Reviewer 9EQd’s awesome suggestion, there could beanother way for automatically selecting the k , where the k is selected based on the largest R -ratioin Eq.
To make a further investigation, we provide a comparison between this score function,termed " LLR opt k ", and our LLR ada .
As shown in Table 10 and Table 11, LLR opt k could achieve a promising OOD detection performancebut still underperform our developed LLR ada .
A more intuitive and numerical analysis about these methods has been provided in Appendix M.
[Table] [Table]
M Measure LLR ada on vanilla HVAE without Informative loss
It would be interesting to see whether the LLR ada will be effective on a vanilla hierarchical VAEtrained without the informative loss.
Take a 5-layer HVAE trained on FashionMNIST for example,we give a comparison with different score functions for OOD detection as shown in Table 12.
To better undertand the underlying mechanism of these score methods, we compute the mean negativelog-likelihood − log p θ ( x | z >k ) for reconstruction and log-likelihood ratio ( LLR >k ) for each layerof a 5-layer vanilla HVAE in Table 13.
Since the values LLR >k of in-distribution data is closer or larger than OOD data, it is not surprisingthat the LLR >k cannot achieve promising OOD detection performance.
However, the performance could be significantly improved with the score LLR opt k and LLR ada .
Specifically, for score LLR opt k , it would highly possible to assign LLR > 1 (1.65 × 10 3 ) for indistribution data and assign LLR > 2 (2.90 × 10 3 ) for OOD data, which makes it easier to detect OODdata.
For score LLR ada , its average score for in-distribution data is 1 .
65 + 4 .
50 ∗ (2 . 97 − 1 . 65)
+7 .
87 (5 . 90 − 2 . 97) + 7 . 81 7 .
81 ∗ (5 . 90 − 5 . 90) = 8 .
173(10 3 ) , but for OOD data, the average score is0 .
74 + 4 .
(2 . 9 − 0 .
74) + 6 . 89 4 .
− 2 . 9) + 6 . 89 6 . 89 (5 .
46 − 5 . 46)
= 11 .
Since the averagescore LLR ada of OOD data is much larger than in-distribution data, our developed LLR ada couldbe a more promising score function to achieve better OOD detection performance.
N Numerical analysis to illustrate “posterior collapse” in Fig. 2
It is interesting to investigate the numerical changes when the “posterior collapse” occurs, such as thedata samples visualized in Fig.
As shown in Table 14, from shallow to deep (Layer 1 to Layer4), we can find that the KL-divergence of HVAE gradually reduces to 0, which indicates that theposteriors of 4-th and 5-th hidden layers collapse to their priors, resulting in that the high-level latentvariables z 4 and z 5 sampled from the posterior q ϕ ( z k | z >k ) have no information of input data x .
[Table]
[Table] [Table]
Further
[Table]
we use t-SNE method to visualize the learned latent data representations in Fig.
Note that,different colors in Fig.
7 indicates different classes of data samples.
As shown in Fig.
7, we can findthat the latent space of HVAE gradually collapses to a non-informative prior distribution, while thelearned latent space of our method is still informative at higher layers.
To see more reconstructed data samples of the partial generative models p ( x | z > 3 ) and p ( x | z > 4 ) ,
To intuitively demonstrate that the posterior of our method does not collapse to a single point, wevisualize the data samples generated from p θ ( x | z >k ) by taking the latent variables
z k sampledfrom the posterior q ϕ
( z k | z >k , x ) as input, where x is a fixed data point.
As shown in Fig.
8, thediversity of the generated samples demonstrate that the posterior q ϕ ( z k | z >k , x ) collapses to its priordistribution p θ ( z k | z >k ) rather than a single point.
KL-divergence in different layers
[Table]
O Comparisons of Reconstruction and Generation quality
The reconstruction and generation capability are two important model properties of VAE, and herewe compare our method with the vanilla HVAE on both these two aspects.
Firstly, we quantitatively evaluate the reconstruction capability of the partial generative modelsp θ ( x | z >k ) conditioned on latent variables z >k at different hidden layers, and report the average bit
[Table]
per dim results in Table 15.
From the results, we can see that our method can achieve a comparablereconstruction quality with other baselines on p θ ( x | z > 0 ) and significantly outperform them on thereconstruction conditioned on higher-layer latent variables.
For the generation capability, we qualitatively visualize the data samples generated from artialgenerative models p θ ( x | z >k ) conditioned on z >k drawn from the prior distribution p θ ( z L ) =
N (0 , I ) .
From the results shown in in Fig.
9, we can find that the quality and diversity of datasamples generated by our model significantly outperform those generated by HVAE, indicating thebenefits of alleviating “posterior collapse.”
Thus, the aforementioned experimental results demonstrate that our method can perverse the versatilityof VAE.
P Comparison on more natural images
To investigate the effectiveness of our method on more natural images, we provide additionalcomparisons on the other datasets, including LFWPeople [43] (people’s faces in the wild), Flower102[42] (102 types of flowers), Food101 [50] (101 types of food), Places365 [41] (365 scene categories),and Tiny-ImageNet [51] (containing 200 categories of images).
With the same score function
LLR ada , we train the VAE-based model on each of these datasets, where each image is resized as32 ×
32 × 3 before training, and then it for OOD detection on SVHN dataset.
The unified networkstructure of these 3-layer models are [64, 32, 16] from shallow to deep.
As shown in Table 16, LVAE, BIVA, and our method can generally outperform the vanilla HVAE,while our method could still significantly achieve the best performance on these dataset pairs, whichindicates the generality of our method on unsupervised OOD detection.
[Table]
[Figure]
[Figure] [Figure]