Out-of-Distribution Detection with An Adaptive Likelihood Ratio on Informative Hierarchical VAE
Abstract
Unsupervised out-of-distribution (OOD) detection is essential for the reliabilityof machine learning.
In the literature, existing work has shown that higher-levelsemantics captured by hierarchical VAEs can be used to detect OOD instances.
However, we empirically show that, the inheirt “ posterior collapse ” of hierarchical VAEs would seriously limit their capacity for OOD detection.
Based on a thoroughanalysis, we propose an informative hierarchical VAE to alleviate this issue throughenhancing the connections between the data sample and its multi-layer stochasticlatent representations during training.
Furthermore, we propose a novel scorefunction for unsupervised OOD detection, referred to as Adaptive Likelihood Ratio,which can selectively aggregate the semantic information on multiple hidden layersof hierarchical VAEs, leading to a strong separability between in-distribution and OOD samples.
Experimental results demonstrate that our method can significantlyoutperform existing state-of-the-art unsupervised OOD detection approaches.
Introduction
Despite achieving great success in real-world applications recently, existing machine learning (ML)systems are still designed to be tested on the in-distribution dataset, whose statistics are similar tothose of the training set [1].
However, when applied to deal with the dataset consisting of out-ofdistribution (OOD) samples, whose statistics are extremely different from those of the training set,these ML systems would produce a series of incorrect judgments [2, 3].
Considering the fact that OOD data is very common in real-world applications, pre-execution OOD detection is increasinglyattractive to make sure the reliability and safety of ML systems.
Although several supervised methods[4, 5, 6, 7, 8, 9, 10, 11, 12] have achieved great success in OOD detection, the unsupervised ones aremore practical since the category labels of in-distribution samples are often missing in real-worldapplications, which brings more challenges for OOD detection and is also the focus of this work.
Without labels, likelihood-based models could be a promising way for unsupervised OOD detection,such as flow-based models [13], auto-regressive models [14, 15], and variational autoencoders (VAEs)[16, 17].
Unfortunately, some recent studies have shown that, in some cases, these generative modelstend to achieve higher likelihoods on certain types of OOD samples [1, 18, 19, 20, 21], whichmakes the OOD detection methods based on thresholding likelihood scores problematic.
To addressthis issue, based on the prior knowledge collected from OOD samples, some improvements havebeen made for unsupervised OOD detection, i.e. , Ren et al.
[1] take additional datasets as the OODvalidation sets for choosing the best hyperparameters; Hendrycks et al.
[22] introduce an auxiliarydataset to teach the network to learn more expressive representations for OOD detection; However,the behaviour of borrowing the prior knowledge of OOD data is usually unreasonable in practice,because we will never know the statistic information of OOD samples to be dealt with.
To conduct purely unsupervised OOD detection without the labels or prior assumptions, deepensemble method named WAIC [18] is developed by making full use of the difference between the density estimations of multiple independent models trained on the in-distribution data.
Recently,through capturing the semantic information with multi-layer latent variables, Maaløe et al.
[23]and Havtorn et al.
[21] develop various score functions based on Likelihood-Ratio, which helphierarchical VAEs achieve competitive performance in unsupervised OOD detection.
However, in ourimplementation, we find that the phenomenon of “ posterior collapse ” in hierarchical VAEs still limitstheir performance on OOD detection, where the main reason could be that “ posterior collapse ” willmake the high-level latent variables meaningless and cannot provide faithful summaries for the data.
With this insight in hand, in this paper, we start from rethinking the cause of “ posterior collapse ”in hierarchical VAEs, and then theoretically explain why “ posterior collapse ” will limit the OODdetection performance of these Likelihood-Ratio based methods.
Further, we develop an informativehierarchical VAE to alleviate “ posterior collapse ” and a novel Adaptive Likelihood Ratio scorefunction for unsupervised OOD detection.
The major contributions of this work include:
• With a thorough analysis of “ posterior collapse ” in hierarchical VAEs, we enhance the
connections between the data sample and its multiple latent representations in the expectedlog-likelihood term of evidence lower bound (ELBO) for training, and develop a novel
informative hierarchical VAE to extract more expressive hierarchical latent representations.
• We theoretically explain why alleviating “ posterior collapse ” in hierarchical VAEs can help
the performance of Likelihood Ratio on OOD detection, and then develop a novel score
function for fully unsupervised OOD detection, termed Adaptive Likelihood Ratio, which
owns fewer hyperparameters to be tuned and can make full use of the semantic divergences
between in-distribution and OOD samples across all hidden layers of hierarchical VAEs.
• Combing the informative hierarchical VAE with the Adaptive Likelihood Ratio, we demon-
strate that our method can achieve state-of-the-art OOD detection performance across a
wide range of benchmarks in an unsupervised manner
Background and Related Works2.1 Hierarchical Variational AutoencoderPreliminary
: Extending the basic VAE [16], a hierarchical VAE [24, 25] is defined by the observationx that depends on a hierarchy of stochastic latent variables z
= z 1 , ..., z L , where the generative modelis defined with a top-down structure, formulated as p θ ( x , z ) =
p θ ( x | z 1 ) (cid:81)
L − 1 l =1 p θ ( z l | z l +1 )
p θ ( z L ) ;the inference model is designed to approximate the posterior over these latent variables, commonlyfactorized with a top-down structure as q ϕ ( z | x ) =
(cid:81)
L − 1 l =1 q ϕ
( z l | z l +1 )
q ϕ
( z L | x ) or a bottom-upstructure as q ϕ ( z | x ) = (cid:81)
L − 1 l =1 q ϕ
( z l +1 | z l ) q ϕ
( z 1 | x ) .
The demonstration of these structures could beseen in Fig.
The parameters of the generative and infernce models, denoted as θ and ϕ respectively,can be jointly optimized by maximizing the evidence lower bound (ELBO) expressed as
[Equation]
where L x is denoted as
[Equation]
where D KL ( ·||· ) denotes the KL divergence and maximizing ELBO is equivalent to minimize thedivergence between the varaitional distribution q ϕ ( z | x ) and true posterior p θ ( z | x ) .
Related Works: While the hierarchy of latent stochastic variables can improve the generationcapability of standard VAEs, in practice, the posterior of higher-level stochastic latent variables havea tendency to collapse into the prior, called “ posterior collapse ”.
To address this issue, Sønderbyet al.
[24] propose a Ladder VAE (LVAE) to change the bottom up inference process into a top-downone; Vahdat and Kautz [25] develop a variant of the LVAE, which carefully designs a sophisticatednetwork architecture to achieve better generation quality; Maaløe et al.
[23] combine the bottomup inference with the top-down inference by proposing a bidirectional inference scheme.
Despiteobtaining performance improvements with more flexible inference networks, these hierarchical VAEsstill lack a theoretical guide to alleviate the phenomenon of “ posterior collapse ”.
2.2 Out-of-distribution Detection with Variational Autoencdoer
Problem Formulation: Suppose that there are a set of N training samples { x i } Ni =1 drawn fromthe data distribution x i ∼ p ( x ) , after training a VAE to model the generation of these data samples, 𝒉𝒉𝒉𝒛𝒙 𝒛𝒛𝒙
Top-downInference
Generation 𝒛𝒛𝒛𝒙 𝒛𝒛𝒛
Bottom-up Inference 𝒙 𝒛 𝑳 𝒛 𝒌+ 𝒛
Partial Generation 𝑝 𝜃 (𝑥|𝑧 >𝑘 ) the generative model p θ ( x ) is supposed to detect whether a testing sample x is an outlier, wherethe outlier should have a low density estimation under the true data distribution p ( x ) .
However,on the contrary, former likelihood-based methods found the generative model always assign higherp θ ( x ) for OOD data than in-distribution data [1, 18, 19, 20].
Luckily, although the likelihood basedmethods with VAE are rarely investigated, they have revealed the potential to better address thisproblem without the help of labels or assumptions about the prior.
Related Works: A pioneering VAE-based OOD detection method is Likelihood Regret (LRe)
[26]obtained by iteratively finetuning the decoder parameters of VAE, which is time-consuming butachieves competitive performance in an unsupervised manner.
Maaløe et al.
[23] reveal the potentialof hierarchical VAE for OOD detection and Havtorn et al.
[21] further propose a score function toimprove the model performance, which needs to specify the backbone model.
The backbone modelsin likelihood-based methods [1, 18, 27] can be directly replaced with VAEs, but these methods usuallyunderperform flow-based models like Glow [13] or autoregressive models like PixelCNN [15].
From Informative Hierarchical VAE to Adaptive Likelihood Ratio
3.1 Rethinking of “ posterior collapse ” in Hierarchical VAEs
Firstly, let’s understand the cause of “ posterior collapse ” in hierarchical VAEs theoretically.
Takingan L -layer hierarchical VAE with a top-down inference network as an example, the set of latentvariables can be separated as the lower-level variables z ≤ k = { z 1 , ..., z k } and the higher-level onesz >k = { z k +1 , ..., z L } , where k ∈ { 0 , ..., L − 1 } , then the ELBO in Eq.
(1) can be reformulated as
[Equation]
where q ϕ ( z L | z L +1 ) : = q ϕ ( z L | x ) , p θ ( z L | z L +1 ) : = p θ ( z L ) , and the main contribution to theexpected log-likelihood term is coming from the lower-level latent variables z ≤ k before the k th hiddenlayer [23].
Once the generation capacity of the generative model p θ ( x | z ≤ k ) is powerful enough toreconstruct the observation x well, the variational posteriors of higher-level latent variables z >kwill be optimized to be close to their priors, i.e., q ϕ ( z >k | x )
≈ p
θ ( z >k ) , leading the representationslearned by VAE at higher layers to be meaningless and cannot provide faithful summaries for x , whichis well-known as the phenomenon of “ posterior collapse ” or “ latent variable collapse ” [24, 28].
To find the potential solutions to alleviating “ posterior collapse ”, in the following, we reinterpret thisphenomenon from the perspective of information theory [29] by extending the findings in [28] to ahierarchical VAE scenario.
For ease of undertanding, we define the mutual information between thedata x and the higher-level latent variables z >k as
[Equation]
q ϕ ( z >k | x ) − E q ϕ [Equation] z >k ) log q ϕ ( z >k ) , which is induced by the variational posterior q ϕ ( z >k | x ) .
Then KL term in Eq.
(3) can be rewritten as
[Equation]
𝑝 𝜃 (𝑥) 𝑝 𝜃 (𝑥|𝑧 >0 ) 𝑝 𝜃 (𝑥|𝑧 >1 ) 𝑝 𝜃 (𝑥|𝑧 >2 ) 𝑝 𝜃 (𝑥|𝑧 >3 ) 𝑝 𝜃 (𝑥|𝑧 >4 ) (a) FashionMNIST (in) 𝑝 𝜃 (𝑥) 𝑝 𝜃 (𝑥|𝑧 >0 ) 𝑝 𝜃 (𝑥|𝑧 >1 ) 𝑝 𝜃 (𝑥|𝑧 >2 ) 𝑝 𝜃 (𝑥|𝑧 >3 ) 𝑝 𝜃 (𝑥|𝑧 >4 ) (b) MNIST (out) where q ϕ ( z >k ) =
E p ( x ) [ q ϕ ( z >k | x )] and the detailed derivation can be found in Appendix A. Bysubstituting Eq.
(4) into Eq.
(3), due to the non-negativity of mutual information and KL divergence,we can find that maximizing the ELBO is opposite to maximizing the mutual information I q ( x , z >k ) .
When I q ( x , z >k ) is minimized to zero, the variational posterior q ϕ ( z >k | x ) will be independent ofthe data x , which leads to the phenomenon of “ posterior collapse ”.
3.2 Why “ posterior collapse ” limits Likelihood-Ratio for OOD Detection
OOD detection has become one of the most important applications of VAEs, which can be appliedto filter OOD samples by setting a threshold on the score of log-likelihood term.
However, somerecent studies have shown that, in some cases, VAEs tend to achieve higher likelihoods on certaintypes of OOD samples [30], which makes the OOD detection rules based on likelihood thresholdproblematic.
Recently, Havtorn et al.
[21] reinterpreted this problematic behavior by providingevidence that the low-level features learned by VAEs generalize well across datasets and dominate theestimated likelihoods. Inspired by the alternative log-likelihood lower bound [23] that partly replacesthe inference network with the generative model to highlight high-level features, formulated as
[Equation]
Havtorn et al. [21] considered to subtract L >k x from L x to cancel out the data distribution log p ( x ) ,resulting in a likelihood-ratio score for unsupervised OOD detection as [Equation] which discards the likelihood term to prevent the low-level features from dominating and measuresdivergence in the latent space to ensure that data should be in-distribution across all feature levels.
To intuitively understand the nature of success in the likelihood-ratio score and illustrate whyalleviating “posterior collapse” in hierarchical VAEs can improve its performance on OOD detection,we provide an insightful analysis on LLR >k in Eq.
(6) by reformulating it as follows:
[Equation]
where the detailed derivations can be found in Appendix B. Eq.
(7) shows that, when the inferencenetwork q ϕ ( z ≤ k | z >k ) can approximate the true posterior p θ ( z ≤ k | z >k , x ) very well, thorough equivalent replacement, LLR >k will approach the expected KL divergence between the prior p θ ( z ≤ k | z >k )and variational posterior q ϕ ( z ≤ k | z >k ) .
More specifically, conditioned on the expectation of z >kdrawn from its variational posterior q ϕ ( z >k | x ) , LLR >k is developed to calculate the summation ofk KL divergence terms, measuring the distance between the lower-level variables z ≤ k drawn fromthe generative model p θ ( z ≤ k | z >k ) and those from the variational inference network q ϕ ( z ≤ k | z >k ) .
The premise of applying LLR >k to OOD detection is that, after training a hierarahical VAE onin-distribution samples, for each OOD sample, the latent variables z ≤ k generated from the generativemodel p θ ( z ≤ k | z >k ) will be clearly distinct from those drawn from the variational inference networkq ϕ
( z ≤ k | z >k ) .
For ease of understanding the principle, after training a 5-layer hierarchical VAEon FashionMNIST, for each hidden layer, we exhibit the reconstructions of both in-distribution(FashionMNIST) and OOD (MNIST) data samples with the partial generative model p θ ( x | z >k ) conditioned on the latent variables z >k drawn from q ϕ ( z >k | x ) as shown in Fig.
From the results,we can find that, when setting k = 2 , the reconstructions of MNIST samples tend to reflect thehigh-level semantic structures learned from FashionMNIST, indicating that the generation mechanismof p θ ( x | z > 2 ) seems to prevent accurate reconstruction of out-of-distribution data, which impliesthat a score function based on the distance between p θ ( z ≤ 2 | z > 2 ) and q ϕ ( z ≤ 2 | z > 2 ) , like LLR > 2 in
Eq.
(7), could be a promising metric for OOD detection.
However, when the phenomenon of “ posterior collapse ” occurs, the variational posterior q ϕ ( z >k | x )will be independent of the data x , resulting in q ϕ ( z >k | x ) ≈
p θ ( z >k ) , and the reconstructions ofin-distribution and OOD samples, which are generated from the partial generative model p θ ( x | z >k ) ,will be almost the same, such as the visualization examples shown in Fig.
2 by setting k = 3 ork = 4 .
In that case, for each in-distribution sample, the latent variables z ≤ k generated p θ ( z ≤ k | z >k )will be clearly distinct from those drawn from q ϕ ( z ≤ k | z >k ) , and similar conclusions can also beachieved by OOD samples, which will reduce the variance of LLR >k scores between in-distributionand OOD samples and further bring troubles for OOD detection with LLR >k .
3.3 Informative Hierarchical VAE to Alleviate “ posterior collapse ”
Recall to the conflict between the ELBO objective and I q ( x , z >k ) as discussed in Sec. 3.1, whichcauses “ posterior collapse ” in hierarchical VAEs, there could be two main approaches to alleviatethis phenomenon, including: 1) downweight the KL term, like applying a warm-up scheme on it[31], which is the most common heuristic in practice but still cannot essentially address this issue; 2)enhance the connections between the observation and its multi-layer stochastic latent representationsin the expected log-likelihood term, like modifying the generative process described by p θ ( x | z ) [23].
In this paper, focused on exploring the potential of alleviating “ posterior collapse ” with the secondapproach, we try to introduce skip-connection-liked structures into expected log-likelihood term toenhance the connections between the data x and the latent variables z
= z 1 , ..., z L .
However, constrained by the layer-by-layer generation process of hierarchical VAE, there remains a great challengeto introduce physical skip connections into the generative model p θ ( x | z ) , because arbitrarily addingor concatenating the stochastic hidden layers at different semantic levels will hurt the hierarchy ofthese multi-layer latent representations.
We emphasize that the skip connections between the singlestochastic layer and multiple deterministic layers [28] cannot be extended for hierarchical VAEs withmultiple stochastic hidden layers, but our developed method below can be applied to any existinghierarchical VAE, which is one of the main contributions of this paper.
Generally speaking, moving beyond downweighting I q ( x , z >k ) included in the KL term or modifyingthe structure of generative model p θ ( x | z ) , our main idea is to upweight the mutual informationbetween the data x and the higher-level variables z >k , which is denoted as
[Equation]
in the objective function of hierarchical VAEs.
In Eq.
(8), the first item can be approximated by
H p,q ( x | z >k ) =
E p ( x ) q ϕ ( z >k | x ) log p θ ( x | z >k ) and the second item H p ( x ) is a positive constant,leading to I p ( x , z >k ) ≥ H p,q ( x | z >k ) .
Thus, targeted at directly maximizing multiple H p,q ( x | z >k )in the objective function, we develop an informative loss for training hierarchical VAEs, denoted as
[Equation]
where p θ ( x | z >k ) =
E p θ ( z ≤ k | z >k )
[ p θ ( x | z ≤ k )] describes a partial generative model to reconstructthe observation x taking z >k drawn from the variational inference network q ϕ ( z >k | x ) as input; theweight 1 /L before each expected term is introduced to keep numerical stability.
The developed L innot only inherits the terms of ELBO in Eq.
(1), which helps preserve the original model properties of
VAE, but also introduces virtual skip-connection-liked structures with partial generative models toenhance the connections between x and z >k , contributing to alleviating “ posterior collapse ”.
To avoid directly calculating p θ ( x | z >k ) in practice, inspired by [21, 23], the informative loss in Eq.
(9) can be optimized by maximizing its lower bound ˆ L in , expressed as
[Equation]
where the expected log-likelihood term is the summation of L components and each component denoted as LL >k = E p θ ( z ≤ k | z >k ) q ϕ ( z >k | x ) [log p θ ( x | z ≤ k )] can be obtained by replacing the inferencenetwork q ϕ ( z ≤ k | z >k ) in the original log-likelihood term as described in Eq.
(3) with the generativemodel p θ ( z ≤ k | z >k ) .
Intuitively, for each expected log-likelihood term LL >k , after sampling thetop variables z >k from the variational posterior q ϕ ( z >k | x ) , these variables z >k will be forcedto reconstruct the observation x with the partial generative model p θ ( x | z >k ) , which builds thestraightforward connections between x and z >k to alleviate “ posterior collapse ”.
We refer to the hierarchical VAE trained with the lower bound of informative loss in Eq.
(10) asinformative hierarchical VAE.
We note that this method is applicable for hierarchical VAEs witheither top-down or bottom-up inference network to explicitly utilize the generative hierarchy of themulti-layered stochastic variables during training, and can be flexibly extended in future works.
3.4 Adaptive Likelihood Ratio for OOD detection
Besides “ posterior collapse ”, an inappropriate choice of k in the likelihood-ratio score function,denoted as LLR >k in Eq.
(6), will also bring negative impact on the performance of applyinghierarchical VAEs for OOD detection.
Recall to the visualization examples in Fig.
when settingk = 0 or k = 1 , the reconstruction quality of either in-distribution or OOD samples is surprisinglyhigh, leading to the LLR >k scores of both in-distribution and OOD samples are relatively small andfurther making it difficult to distinguish whether the data sample is OOD or not.
Moving beyondcherry picking the hyperparameter k on testing OOD samples, which is unreasonable for unsupervised
OOD detection, we develop a novel adaptive likelihood-ratio score function LLR ada , described as
[Equation]
where R ( x , z >k ) is designed to measure the relevance between the data sample x and its latentvariables z >k sampled from the variational posterior q ϕ ( z >k | x ) , specifically defining LLR > − 1 :=
0and R ( x , z > − 1 ) :
= R ( x , z > 0 ) .
More specifically, there are many choices for the definition of
R ( x , z >k ) , but in the following experiments, we only use the log-likelihood score for brevity, byspecifically defining R ( x , z >k ) : = 1 / log p θ ( x | z >k ) .
The intuition of designing LLR ada is to move beyond the choose of k but adaptively enhancethe importance of some discriminative terms, like LLR > 2 , in the overall score function for OODdetection.
With R ( x , z >k ) to measure the relevance between x and z >k , we find that the adaptiveweight R ( x , z >k − 1 ) R ( x , z >k ) will be relatively large when the data information drop rapidly at the currenthidden layer, like k = 2 in Fig.
2, which can be naturally applied as the importance weights toenlarge the gap between the metric scores of in-distribution and OOD samples.
Compared to theprevious score functions for OOD detection [21, 26], the developed LLR ada in Eq.
(11) owns lesshyperparameters to be tuned, making its performance more stable on various benchmarks.
Morediscussions about LLR ada can be found in Appendix C.
Experiments
4.1 Experimental setup
Datasets: Following [1, 21, 26], we compare our method with previous works on two dataset pairs,including: FashionMNIST [32] (in) /
MNIST [33] (out) and CIFAR10 [34] (in) / SVHN [35] (out),where the suffix “in” and “out” denote the in-distribution dataset and OOD dataset, respectively.
Tobetter evaluate the generalization ability of these methods, we introduce additional OOD datasets: for
FashionMNIST/MNIST pair, we add KMNIST [36], notMNIST [37], Omniglot [38] and SmallNORB[39] datasets; for CIFAR10/SVHN pair, we add CelebA [40], Places365 [41], Flower102 [42] and
LFWPeople [43] datasets.
More details about datasets can be found in Appendix D.
Evaluation and Metrics: We follow the evaluation procedure in Havtorn et al.
[21], where allmethods are trained on the training split of the in-distribution dataset, and their OOD detectionperformance is evaluated on both the testing split of the in-distribution dataset and OOD dataset.
Following previous works’ evaluation approaches [4, 5, 22], we adopt two popular threshold-independentevaluation metrics, including Area Under the Receiver Operator Characteristic (AUROC ↑ ) and Area [Table]
[Table] Under the Precision Recall Curve (AUPRC ↑ ), and another metric False Positive Rate at 80% truepositive rate (FPR80 ↓ ), where the arrow indicates the direction of improvement.
Baselines: The comparisons in our experiments mainly include two aspects: i ) the comparisons withprevious OOD detection methods to see whether our method can achieve competitive performance;ii ) the comparisons with several hierarchical VAEs to see whether the new training objective of ourmethod can lead to better performance.
For the comparisons in i , the baselines can be divided intothree categories: “ Labels ”: methods using in-distribution data labels [4, 5, 7, 8, 44, 45]; “ Prior ”:methods using the prior knowledge collected from OOD data [1, 22, 27]; and “ Unsupervised ”:methods without any OOD-specific assumptions [18, 21, 26].
For the comparisons in ii , we compareour method with a normal bottom-up inference hierarchical VAE (HVAE)
[16], which is also thebackbone of our method, and its two major variants: a top-down inference hierarchical VAE named
Ladder VAE (LVAE) [24] and a bidirectional inference hierarchical VAE (BIVA) [23].
More detailsof these baselines and the categories they belong to can be found in Appendix E.
Implementation Details: For the comparisons on FashionMNIST(in)/MNIST(out), we set thenetwork structure of hierarchical VAEs as [16, 8, 4] and [32, 24, 16, 8, 4] from shallow to deep,respectively.
For CIFAR10(in)/SVHN(out), we set the network structure as [128, 64, 32] and [128, 64,32, 28, 24], respectively.
For optimization, we adopt the same Adam optimizer [46] with a learningrate of 3e-4.
We train all models in comparison by setting the batch size as 128 and the max epoch as1000.
All experiments are performed on a PC with an NVIDIA RTX 3090 GPU and the our code isimplemented with PyTorch [47].
More implementation details can be found in Appendix F.
4.2 Quantitative Comparisons
Overall Comparisons: Following the experimental settings in Sec. 4.1, we exhibit the experimentalresults in in Tab. 1.
From the results, we can find that our method with the LLR ada score function iscomparable with those non-ensemble completely unsupervised methods in FashionMNIST/MNIST,and significantly outperform them in CIFAR10/SVHN.
We emphasize that, without utilizing thelabels of in-distribution samples [4, 5, 7, 8, 44, 45] or the prior knowledge collected from OODsamples [1, 22, 27], our method can still achieve competitive performance with these methods.
Effectiveness of LLR ada : Focused on the comparison between LLR >k and LLR ada exhibited in
Tab. 2 and Fig. 3(c), when the performance of LLR >k is sensitive to the selection of k , we can findthat the performance of LLR ada can approach the best performance achieved by LLR >k with theoptimal k , as shown in the right part of Tab.
(CIFAR/SVHN).
Furthermore, when the performance of
LLR >k is stable, the developed LLR ada can still achieve comparable OOD detection performance,as shown in the left part of Tab. 2 (FashionMNIST/MNIST) and Fig. 3(c).
The experimental resultsabove demonstrate the adaptability of our developed LLR ada .
[Table]
in) / MNIST (out)
Ours >2 BIVA >2 LVAE >
HVAE >
M1:97.4 M1:96.9 M1:92. M1:97. (a) 0.
1.0 False Positive Rate 0.
T r u e P o s i
t
i v e R a t e
CIFAR10 (in) / SVHN (out)
Ours >2 BIVA >2 LVAE > HVAE >
M1:86.3M1:93.0 M1:77.
M1:75. (b) 0.
1.0 False Positive Rate 0.
T r u e P o s i
t
i v e R a t e
CIFAR10 (in) / SVHN (out)
{+Ours >1 Ours >2 Ours > Ours >
 Ours ada M1:94.2 M1:93.8 M1:95.+}
M1:95.8 M1:92. (c) Effectiveness of Informative Hierarchical VAE: Taking the same LLR ada as the score function,we compare the performance of informative hierarchical VAE with other hierarchical VAEs in Tab. 3. From the results, we can find that our model outperforms others, indicating the effectiveness ofalleviating “ posterior collapse ” with informative hierarchical VAE for OOD detection.
To evaluate the degree of “ posterior collapse ”, we use the ROC curves in Fig.
3 and the empirical densities in Fig.
4 to compare the performance of LLR > 2 scores based on the top-level latent variables z 2of these VAEs.
The ROC results in Fig.
3 demonstrate the superiority of the informative hierarchical
VAE, confirming that our model can provide more expressive higher-level latent representations for
OOD detection.
The empirical densities in Fig.
4 show that z 2 learned by our model on the {+CIFAR
+}
[Figure]
in)/SVHN (out) pairs owns better separability than those learned by BIVA.
[Figure]
4.3 Qualitative Analysis
Meaningful Semantic Space Learned by Informative Hierarchical VAE:
Following the sameprocedure as Fig.
2, we visualize more reconstructed samples on various benchmarks in Fig.
Fromthe visualized results, for both in-distribution and OOD samples, we can find that the quality of thereconstructions generated from p θ ( x
| z > 0 ) is surprisedly high, indicating that these reconstructionsare almost dominated by the low-level features, which potentially explains the previous problematicphenomenon that these methods based on single-layer likelihood will assign higher likelihood scoresfor OOD samples and fail on OOD detection.
Focused on the reconstructions generated by p θ ( x | z > 2 ) ,we can find that the developed informative hierarchical VAE can provide more realistic and clearreconstructed samples for both in-distribution and OOD inputs, indicating that our model can learna more meaningful high-level latent semantic space than other models.
Furthermore, based onproviding higher-quality reconstruction for OOD samples, the gap between the metric scores of
[Figure]
x z >z >z >z >z >(a) Ours/FashionMNIST x z >z >z >z >z >(
[Figure]
/MNIST x z >z >z >z >z >(
[Figure]
HVAE
[Figure]
FashionMNIST x z >z >z >z >z >(
[Figure]
HVAE/MNIST 𝑥 𝑧 >0 𝑧 > 𝑧 > (
[Figure]
/CIFAR10 (in
[Figure]
𝑥 𝑧 >0 𝑧 > 𝑧 > (
[Figure]
[Figure]
/SVHN (
[Figure]
𝑥 𝑧 >0 𝑧 > 𝑧 > (
[Figure]
[Figure]
in-distribution and OOD samples in informative hierarchical VAE tend to be larger than models,leading to performance as shown in
Alleviating “ Posterior Collapse ” with Informative Hierarchical VAE:
As discussed in Sec. 3.1,“ posterior collapse ” will cause the higher-level latent variables to become uninformative.
Consideringthe developed informative hierarchical VAE shares the same network structure with the basic HVAE,in this part, we focus on evaluating whether our model can alleviate “ posterior collapse ” in higherlayers when the network depth becomes deeper.
As shown in Fig.
6, for both in-distribution data and
OOD data samples, the overall quality of the reconstructions generated by our model is significantlyhigher than those generated by the basic HVAE.
Specifically, for FashionMNIST (in)/MNIST (out),the reconstructions generated by p θ ( x | z > 3 ) and p θ ( x | z > 4 ) of HVAE are almost the same, indicatingthat its posterior described by q ϕ ( z > 3 | x ) or q ϕ ( z > 4 | x ) tend to collapse to a prior distribution about T-shirts.
On the contrary, the reconstructions generated by the our model are still realistic, wherethe reconstructions of MINST (out) samples generated by q ϕ ( z > 3 | x ) or q ϕ ( z > 4 | x ) preserve thesemantic structural information learned form FashionMNIST (in), explaining the underlying reasonwhy our model can achieve better OOD detection performance.
Similar conclusions can be achievedby the experimental results on CIFAR10 (in)/SVHN (out) as shown in Fig. 6.
Conclusion
In this paper, after presenting a thorough analysis of “ posterior collapse ”, we develop a novelinformative hierarchical VAE to extract more expressive hierarchical latent representations by alleviating “ posterior collapse ”.
Then we theoretically explain why “ posterior collapse ” will limit theperformance of existing hierarchical VAEs, and develop a novel Adaptive Likelihood Ratio scorefunction for unsupervised OOD detection.
Experiments demonstrate the effectiveness of our method,whose main thought can be borrowed other hierarchical VAEs to improving their performance ondownstream tasks relied on the hierarchy of latent representations.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Abstract
(b) Did you describe the limitations of your work?
[Yes] See Appendix H
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A and Appendix B
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tal material
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.1 and Appendix F
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Appendix G
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1 and Appendix F
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplemental
material
(b) Did you mention the license of the assets?
[Yes] See supplemental material
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]