Out-of-Distribution Detection with An Adaptive Likelihood Ratio on Informative Hierarchical VAE
Abstract
Unsupervised out-of-distribution (OOD) detection is essential for the reliability of machine learning.
In the literature, existing work has shown that higher-level semantics captured by hierarchical VAEs can be used to detect OOD instances.
However, we empirically show that, the inheirt “ posterior collapse ” of hierarchical
VAEs would seriously limit their capacity for OOD detection.
Based on a thorough analysis, we propose an informative hierarchical VAE to alleviate this issue through enhancing the connections between the data sample and its multi-layer stochastic latent representations during training.
Furthermore, we propose a novel score function for unsupervised OOD detection, referred to as Adaptive Likelihood Ratio, which can selectively aggregate the semantic information on multiple hidden layers of hierarchical VAEs, leading to a strong separability between in-distribution and
OOD samples.
Experimental results demonstrate that our method can significantly outperform existing state-of-the-art unsupervised OOD detection approaches.
Introduction
Despite achieving great success in real-world applications recently, existing machine learning (ML) systems are still designed to be tested on the in-distribution dataset, whose statistics are similar to those of the training set [1].
However, when applied to deal with the dataset consisting of outof-distribution (OOD) samples, whose statistics are extremely different from those of the training set these ML systems would produce a series of incorrect judgments [2, 3, 4].
Considering the fact that OOD data is very common in real-world applications, pre-execution OOD detection is increasingly attractive to make sure the reliability and safety of ML systems.
Although several supervised methods [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] have achieved great success in OOD detection, the unsupervised ones are more practical since the category labels of in-distribution samples are often missing in real-world applications, which brings more challenges for OOD detection and is also the focus of this work.
Without labels, likelihood-based models could be a promising way for unsupervised OOD detection, such as flow-based models [17], auto-regressive models [18, 19], and variational autoencoders (VAEs) [20, 21, 22, 21, 23].
Unfortunately, some recent studies have shown that, in some cases, these generative models tend to achieve higher likelihoods on certain types of OOD samples [1, 24, 25, 26, 27], which makes the OOD detection methods based on thresholding likelihood scores problematic.
To address this issue, based on the prior knowledge collected from OOD samples, some improvements have been made for unsupervised OOD detection, i.e. , Ren et al.
[1] take additional datasets as the OOD validation sets for choosing the best hyperparameters; Hendrycks et al.
[28] introduce an auxiliary dataset to teach the network to learn more expressive representations for OOD detection; However, the behaviour of borrowing the prior knowledge of OOD data is usually unreasonable in practice, because we will never know the statistic information of OOD samples to be dealt with.
To conduct purely unsupervised OOD detection without the labels or prior assumptions, deep ensemble method named WAIC [24] is developed by making full use of the difference between the density estimations of multiple independent models trained on the in-distribution data.
Recently, through capturing the semantic information with multi-layer latent variables, Maaløe et al.
[29] and Havtorn et al.
[27] develop various score functions based on Likelihood-Ratio, which help hierarchical VAEs achieve competitive performance in unsupervised OOD detection.
However, in our implementation, we find that the phenomenon of “ posterior collapse ” in hierarchical VAEs still limits their performance on OOD detection, where the main reason could be that “ posterior collapse ” will make the high-level latent variables meaningless and cannot provide faithful summaries for the data.
With this insight in hand, in this paper, we start from rethinking the cause of “ posterior collapse ” in hierarchical VAEs, and then theoretically explain why “ posterior collapse ” will limit the OOD detection performance of these Likelihood-Ratio based methods.
Further, we develop an informative hierarchical VAE to alleviate “ posterior collapse ” and a novel Adaptive Likelihood Ratio score function for unsupervised OOD detection.
The major contributions of this work include:
• With a thorough analysis of “ posterior collapse ” in hierarchical VAEs, we enhance the connections between the data sample and its multiple latent representations in the expected log-likelihood term of evidence lower bound (ELBO) for training
and develop a novel informative hierarchical VAE to extract more expressive hierarchical latent representations.
• We theoretically explain why alleviating “ posterior collapse ” in hierarchical VAEs can help the performance of Likelihood Ratio on OOD detection, and then develop a novel score function for fully unsupervised OOD detection, termed Adaptive Likelihood Ratio, which owns fewer hyperparameters to be tuned and can make full use of the semantic divergences between in-distribution and OOD samples across all hidden layers of hierarchical VAEs.
• Combing the informative hierarchical VAE with the Adaptive Likelihood Ratio, we demonstrate that our method can achieve state-of-the-art OOD detection
performance across a wide range of benchmarks in an unsupervised manner.
Background and Related Works 2.1 Hierarchical Variational Autoencoder
Preliminary: Extending the basic VAE [20], a hierarchical VAE [30, 31] is defined by the observation x that depends on a hierarchy of stochastic latent variables
z = z 1 , ..., z L , where the generative model is defined with a top-down structure, formulated as p θ ( x , z ) =
p θ ( x | z 1 ) (cid:81)
L − 1 l =1 p θ ( z l | z l +1 )
p θ ( z L ) ; the inference model is designed to approximate the posterior over these latent variables, commonly factorized with a top-down structure as q ϕ ( z | x ) =
(cid:81)
L − 1 l =1 q ϕ
( z l | z l +1 )
q ϕ
( z L | x ) or a bottom-up structure as q ϕ ( z | x ) = (cid:81)
L − 1 l =1 q ϕ
( z l +1 | z l ) q ϕ
( z 1 | x ) .
The demonstration of these structures could be seen in Fig.
The parameters of the generative and infernce models, denoted as θ and ϕ respectively, can be jointly optimized by maximizing the evidence lower bound (ELBO) expressed as
[Equation]
where L x is denoted as
[Equation]
where D KL ( ·||· ) denotes the KL divergence and maximizing ELBO is equivalent to minimize the divergence between the varaitional distribution q ϕ ( z | x ) and true posterior p θ ( z | x ) .
Related Works: While the hierarchy of latent stochastic variables can improve the generation capability of standard VAEs, in practice, the posterior of higher-level stochastic latent variables have a tendency to collapse into the prior, called “ posterior collapse ”.
To address this issue, Sønderby et al.
[30] propose a Ladder VAE (LVAE) to change the bottom up inference process into a top-down one; Vahdat and Kautz [31] develop a variant of the LVAE, which carefully designs a sophisticated network architecture to achieve better generation quality; Maaløe et al.
[29] combine the bottomup inference with the top-down inference by proposing a bidirectional inference scheme.
Despite obtaining performance improvements with more flexible inference networks, these hierarchical VAEs still lack a theoretical guide to alleviate the phenomenon of “ posterior collapse ”.
𝒉𝒉𝒉𝒛𝒙 𝒛𝒛𝒙
Top-downInference
Generation 𝒛𝒛𝒛𝒙 𝒛𝒛𝒛
Bottom-up Inference 𝒙 𝒛 𝑳 𝒛 𝒌+ 𝒛
Partial Generation 𝑝 𝜃 (𝑥|𝑧 >𝑘 ) 2.2 Out-of-distribution Detection with Variational Autoencdoer Problem Formulation: Suppose that there are a set of N training samples { x i } N i =1 drawn from the data distribution x i ∼ p ( x ) , after training a VAE to model the generation of these data samples, the generative model p θ ( x ) is supposed to detect whether a testing sample x is an outlier, where the outlier should have a low density estimation under the true data distribution p ( x ) .
However, on the contrary, former likelihood-based methods found the generative model always assign higher p θ ( x ) for OOD data than in-distribution data [1, 24, 25, 26].
Luckily, although the likelihood based methods with VAE are rarely investigated, they have revealed the potential to better address this problem without the help of labels or assumptions about the prior.
Related Works: A pioneering VAE-based OOD detection method is Likelihood Regret (LRe)
[32] obtained by iteratively finetuning the decoder parameters of VAE, which is time-consuming but achieves competitive performance in an unsupervised manner.
Maaløe et al.
[29] reveal the potential of hierarchical VAE for OOD detection and Havtorn et al.
[27] further propose a score function to improve the model performance, which needs to specify the backbone model.
The backbone models in likelihood-based methods [1, 24, 33] can be directly replaced with VAEs, but these methods usually underperform flow-based models like Glow [17] or autoregressive models like PixelCNN [19].
From Informative Hierarchical VAE to Adaptive Likelihood Ratio 3.1 Rethinking of “ posterior collapse ” in Hierarchical VAEs
Firstly, let’s understand the cause of “ posterior collapse ” in hierarchical VAEs theoretically.
Taking an L -layer hierarchical VAE with a top-down inference network as an example, the set of latent variables can be separated as the lower-level variables z ≤ k = { z 1 , ..., z k } and the higher-level ones z >k = { z k +1 , ..., z L } , where k ∈ { 0 , ..., L − 1 } , then the ELBO in Eq.
(1) can be reformulated as
[Equation]
where q ϕ ( z L | z L +1 ) :
= q ϕ ( z L | x ) , p θ ( z L | z L +1 ) :
= p θ ( z L ) , and the main contribution to the expected log-likelihood term is coming from the lower-level latent variables z ≤ k before the k th hidden layer [29].
Once the generation capacity of the generative model p θ ( x | z ≤ k ) is powerful enough to reconstruct the observation x well, the variational posteriors of higher-level latent variables z >k will be optimized to be close to their priors, i.e., q ϕ ( z >k | x )
≈ p
θ ( z >k ) , leading the representations learned by VAE at higher layers to be meaningless and cannot provide faithful summaries for x , which is well-known as the phenomenon of “ posterior collapse ” or “ latent variable collapse ” [30, 34].
To find the potential solutions to alleviating “ posterior collapse ”, in the following, we reinterpret this phenomenon from the perspective of information theory [35] by extending the findings in [34] to a hierarchical VAE scenario.
For ease of undertanding, we define the mutual information between the data x and the higher-level latent variables z >k as
[Equation]
[Equation]
[Equation]
𝑝 𝜃 (𝑥) 𝑝 𝜃 (𝑥|𝑧 >0 ) 𝑝 𝜃 (𝑥|𝑧 >1 ) 𝑝 𝜃 (𝑥|𝑧 >2 ) 𝑝 𝜃 (𝑥|𝑧 >3 ) 𝑝 𝜃 (𝑥|𝑧 >4 ) (a) FashionMNIST (in) 𝑝 𝜃 (𝑥) 𝑝 𝜃 (𝑥|𝑧 >0 ) 𝑝 𝜃 (𝑥|𝑧 >1 ) 𝑝 𝜃 (𝑥|𝑧 >2 ) 𝑝 𝜃 (𝑥|𝑧 >3 ) 𝑝 𝜃 (𝑥|𝑧 >4 ) (b) MNIST (out) which is induced by the variational posterior q ϕ ( z >k | x ) .
Then KL term in Eq.
(3) can be rewritten as
[Equation]
4) where q ϕ ( z >k ) =
E p ( x ) [ q ϕ ( z >k | x )] and the detailed derivation can be found in Appendix A. By substituting Eq.
(4) into Eq.
(3), due to the non-negativity of mutual information and KL divergence, we can find that maximizing the ELBO is opposite to maximizing the mutual information I q ( x , z >k ) .
When I q ( x , z >k ) is minimized to zero, the variational posterior q ϕ ( z >k | x ) will be independent of the data x , which leads to the phenomenon of “ posterior collapse ”.
3.2 Why “ posterior collapse ” limits Likelihood-Ratio for OOD Detection
OOD detection has become one of the most important applications of VAEs, which can be applied to filter OOD samples by setting a threshold on the score of log-likelihood term.
However, some recent studies have shown that, in some cases, VAEs tend to achieve higher likelihoods on certain types of OOD samples [36], which makes the OOD detection rules based on likelihood threshold problematic.
Recently, Havtorn et al.
[27] reinterpreted this problematic behavior by providing evidence that the low-level features learned by VAEs generalize well across datasets and dominate the estimated likelihoods.
Inspired by the alternative log-likelihood lower bound [29] that partly replaces the inference network with the generative model to highlight high-level features, formulated as
[Equation]
Havtorn et al.
[27] considered to subtract L >k x from L x to cancel out the data distribution log p ( x ) , resulting in a likelihood-ratio score for unsupervised OOD detection as
[Equation]
6) which discards the likelihood term to prevent the low-level features from dominating and measures divergence in the latent space to ensure that data should be in-distribution across all feature levels.
To intuitively understand the nature of success in the likelihood-ratio score and illustrate why alleviating “posterior collapse” in hierarchical VAEs can improve its performance on OOD detection, we provide an insightful analysis on LLR >k in Eq.
(6) by reformulating it as follows:
[Equation]
where the detailed derivations can be found in Appendix B. Eq.
(7) shows that, when the inference network q ϕ ( z ≤ k | z >k ) can approximate the true posterior p θ ( z ≤ k | z >k , x ) very well, thorough equivalent replacement, LLR >k will approach the expected KL divergence between the prior p θ ( z ≤ k | z >k ) and variational posterior q ϕ ( z ≤ k | z >k ) .
More specifically, conditioned on the expectation of z >k drawn from its variational posterior q ϕ ( z >k | x ) , LLR >k is developed to calculate the summation of k KL divergence terms, measuring the distance between the lower-level variables z ≤ k drawn from the generative model p θ ( z ≤ k | z >k ) and those from the variational inference network q ϕ ( z ≤ k | z >k ) .
The premise of applying LLR >k to OOD detection is that, after training a hierarahical VAE on in-distribution samples, for each OOD sample, the latent variables z ≤ k generated from the generative model p θ ( z ≤ k | z >k ) will be clearly distinct from those drawn from the variational inference network q ϕ ( z ≤ k | z >k ) .
For ease of understanding the principle, after training a 5-layer hierarchical VAE on FashionMNIST, for each hidden layer, we exhibit the reconstructions of both in-distribution (FashionMNIST) and OOD (MNIST) data samples with the partial generative model p θ ( x | z >k ) conditioned on the latent variables z >k drawn from q ϕ ( z >k | x ) as shown in Fig.
From the results, we can find that, when setting k = 2 , the reconstructions of MNIST samples tend to reflect the high-level semantic structures learned from FashionMNIST, indicating that the generation mechanism of p θ ( x | z > 2 ) seems to prevent accurate reconstruction of out-of-distribution data, which implies that a score function based on the distance between p θ ( z ≤ 2 | z > 2 ) and q ϕ ( z ≤ 2 | z > 2 ) , like LLR > 2 in
Eq.
(7), could be a promising metric for OOD detection.
However, when the phenomenon of “ posterior collapse ” occurs, the variational posterior q ϕ ( z >k | x ) will be independent of the data x , resulting in q ϕ ( z >k | x ) ≈
p θ ( z >k ) , and the reconstructions of in-distribution and OOD samples, which are generated from the partial generative model p θ ( x | z >k ) , will be almost the same, such as the visualization examples shown in Fig.
2 by setting k = 3 or k = 4 .
In that case, for each in-distribution sample, the latent variables z ≤ k generated p θ ( z ≤ k | z >k ) will be clearly distinct from those drawn from q ϕ ( z ≤ k | z >k ) , and similar conclusions can also be achieved by OOD samples, which will reduce the variance of LLR >k scores between in-distribution and OOD samples and further bring troubles for OOD detection with LLR >k .
3.3 Informative Hierarchical VAE to Alleviate “ posterior collapse ”
Recall to the conflict between the ELBO objective and I q ( x , z >k ) as discussed in Sec. 3.1, which causes “ posterior collapse ” in hierarchical VAEs, there could be two main approaches to alleviate this phenomenon, including: 1) downweight the KL term, like applying a warm-up scheme on it [37], which is the most common heuristic in practice but still cannot essentially address this issue; 2) enhance the connections between the observation and its multi-layer stochastic latent representations in the expected log-likelihood term, like modifying the generative process described by p θ ( x | z ) [29].
In this paper, focused on exploring the potential of alleviating “ posterior collapse ” with the second approach, we try to introduce skip-connection-liked structures into expected log-likelihood term to enhance the connections between the data x and the latent variables z
= z 1 , ..., z L .
However, constrained by the layer-by-layer generation process of hierarchical VAE, there remains a great challenge to introduce physical skip connections into the generative model p θ ( x | z ) , because arbitrarily adding or concatenating the stochastic hidden layers at different semantic levels will hurt the hierarchy of these multi-layer latent representations.
We emphasize that the skip connections between the single stochastic layer and multiple deterministic layers [34] cannot be extended for hierarchical VAEs with multiple stochastic hidden layers, but our developed method below can be applied to any existing hierarchical VAE, which is one of the main contributions of this paper.
Generally speaking, moving beyond downweighting I q ( x , z >k ) included in the KL term or modifying the structure of generative model p θ ( x | z ) , our main idea is to upweight the mutual information between the data x and the higher-level variables z >k , which is denoted as
[Equation]
in the objective function of hierarchical VAEs.
In Eq.
(8), the first item can be approximated by
H p,q ( x | z >k ) =
E p ( x ) q ϕ ( z >k | x ) log p θ ( x | z >k ) and H p ( x ) is a constant, leading to the optimization direction of I p ( x , z >k ) is consistent with H p,q ( x | z >k ) .
Thus, targeted at directly maximizing multiple H p,q ( x | z >k ) , we develop an informative loss for training hierarchical VAEs, denoted as
[Equation]
where p θ ( x | z >k ) =
E p θ ( z ≤ k | z >k )
[ p θ ( x | z ≤ k )] describes a partial generative model to reconstruct the observation x taking z >k drawn from the variational inference network q ϕ ( z >k | x ) as input; the weight 1 /L before each expected term is introduced to keep numerical stability.
The developed L in not only inherits the terms of ELBO in Eq.
(1), which helps preserve the original model properties of
VAE, but also introduces virtual skip-connection-liked structures with partial generative models to enhance the connections between x and z >k , contributing to alleviating “ posterior collapse ”.
To avoid directly calculating p θ ( x | z >k ) in practice, inspired by [27, 29], the informative loss in
Eq.
(9) can be optimized by maximizing its lower bound ˆ L in , expressed as
[Equation]
where the expected log-likelihood term is the summation of L components and each component denoted as LL >k
= E p θ ( z ≤ k | z >k ) q ϕ ( z >k | x ) [log p θ ( x | z ≤ k )] can be obtained by replacing the inference network q ϕ ( z ≤ k | z >k ) in the original log-likelihood term as described in Eq.
(3) with the generative model p θ ( z ≤ k | z >k ) .
Intuitively, for each expected log-likelihood term LL >k , after sampling the top variables z >k from the variational posterior q ϕ ( z >k | x ) , these variables z >k will be forced to reconstruct the observation x with the partial generative model p θ ( x | z >k ) , which builds the straightforward connections between x and z >k to alleviate “ posterior collapse ”.
We refer to the hierarchical VAE trained with the lower bound of informative loss in Eq.
(10) as informative hierarchical VAE.
We note that this method is applicable for hierarchical VAEs with either top-down or bottom-up inference network to explicitly utilize the generative hierarchy of the multi-layered stochastic variables during training, and can be flexibly extended in future works.
3.4 Adaptive Likelihood Ratio for OOD detection
Besides “ posterior collapse ”, an inappropriate choice of k in the likelihood-ratio score function, denoted as LLR >k in Eq.
(6), will also bring negative impact on the performance of applying hierarchical VAEs for OOD detection.
Recall to the visualization examples in Fig.
2, when setting k = 0 or k = 1 , the reconstruction quality of either in-distribution or OOD samples is surprisingly high, leading to the LLR >k scores of both in-distribution and OOD samples are relatively small and further making it difficult to distinguish whether the data sample is OOD or not.
Moving beyond cherry picking the hyperparameter k on testing OOD samples, which is unreasonable for unsupervised
OOD detection, we develop a novel adaptive likelihood-ratio score function LLR ada , described as
[Equation]
(11) where R ( x , z >k ) is designed to measure the relevance between the data sample x and its latent variables z >k sampled from the variational posterior q ϕ ( z >k | x ) , specifically defining LLR > − 1 :=
and R ( x , z > − 1 ) :
= R ( x , z > 0 ) .
More specifically, there are many choices for the definition of
R ( x , z >k ) , but in the following experiments, we only use the log-likelihood score for brevity, by specifically defining R ( x , z >k ) :
= 1 / log p θ ( x | z >k ) .
The intuition of designing LLR ada is to move beyond the choose of k but adaptively enhance the importance of some discriminative terms, like LLR > 2 , in the overall score function for OOD detection.
With R ( x , z >k ) to measure the relevance between x and z >k , we find that the adaptive weight R ( x , z >k − 1 ) R ( x , z >k ) will be relatively large when the data information drop rapidly at the current hidden layer, like k = 2 in Fig.
2, which can be naturally applied as the importance weights to enlarge the gap between the metric scores of in-distribution and OOD samples.
Compared to the previous score functions for OOD detection [27, 32], the developed LLR ada in Eq.
(11) owns less hyperparameters to be tuned, making its performance more stable on various benchmarks.
More discussions about LLR ada can be found in Appendix C.
[Table]
[Table]
Experiments 4.1 Experimental setup
Datasets: Following [1, 27, 32], we compare our method with previous works on two dataset pairs, including: FashionMNIST [39] (in) /
MNIST [40] (out) and CIFAR10 [41] (in) / SVHN [42] (out), where the suffix “in” and “out” denote the in-distribution dataset and OOD dataset, respectively.
To better evaluate the generalization ability of these methods, we introduce additional OOD datasets: for
FashionMNIST/MNIST pair, we add KMNIST [43], notMNIST [44], Omniglot [45] and SmallNORB [46] datasets; for CIFAR10/SVHN pair, we add CelebA [47], Places365 [48], Flower102 [49] and
LFWPeople [50] datasets.
More details about datasets can be found in Appendix D.
Evaluation and Metrics: We follow the evaluation procedure in Havtorn et al.
[27], where all methods are trained on the training split of the in-distribution dataset, and their OOD detection performance is evaluated on both the testing split of the in-distribution dataset and OOD dataset.
Following previous works’ evaluation approaches [5, 6, 28], we adopt two popular threshold-independent evaluation metrics, including Area Under the Receiver Operator Characteristic (AUROC ↑ ) and Area
Under the Precision Recall Curve (AUPRC ↑ ), and another metric False Positive Rate at 80% true positive rate (FPR80 ↓ ), where the arrow indicates the direction of improvement.
Baselines: The comparisons in our experiments mainly include two aspects: i ) the comparisons with previous OOD detection methods to see whether our method can achieve competitive performance; ii ) the comparisons with several hierarchical VAEs to see whether the new training objective of our method can lead to better performance.
For the comparisons in i , the baselines can be divided into three categories: “ Labels ”: methods using in-distribution data labels [5, 6, 8, 9, 38, 51]; “ Prior ”: methods using the prior knowledge collected from OOD data [1, 28, 33]; and “ Unsupervised ”: methods without any OOD-specific assumptions [24, 27, 32].
For the comparisons in ii , we compare our method with a normal bottom-up inference hierarchical VAE (HVAE)
[20], which is also the backbone of our method, and its two major variants: a top-down inference hierarchical VAE named
Ladder VAE (LVAE) [30] and a bidirectional inference hierarchical VAE (BIVA) [29].
More details of these baselines and the categories they belong to can be found in Appendix E.
Implementation Details: For the comparisons on FashionMNIST(in)/MNIST(out), we set the network structure of hierarchical VAEs as [16, 8, 4] and [32, 24, 16, 8, 4] from shallow to deep, respectively.
For CIFAR10(in)/SVHN(out), we set the network structure as [128, 64, 32] and [128, 64, 32, 28, 24], respectively.
For optimization, we adopt the same Adam optimizer [52] with a learning rate of 3e-4.
We train all models in comparison by setting the batch size as 128 and the max epoch as
[-Tab. 2 and Fig. 3(c), when the performance of LLR >k is sensitive to the selection of k , we can findthat the performance of LLR ada can approach the best performance achieved by LLR >k with theoptimal k , as shown in the right part of Tab. 
-]
[Table]
in) /
MNIST (out)
Ours >2 BIVA >2 LVAE >
HVAE >
M1:97.4 M1:96.9 M1:92.
M1:97.
(a) 0.
1.0 False Positive Rate 0.
T r u e P o s i
t
i v e R a t e
CIFAR10 (in) / SVHN (out)
Ours >2 BIVA >2 LVAE >
HVAE >
M1:86.3M1:93.0 M1:77.
M1:75.
(b) 0.
1.0 False Positive Rate 0.
T r u e P o s i
t
i v e R a t e
CIFAR10 (in) / SVHN (out)
Ours >1 Ours >2 Ours >
Ours >
Ours ada M1:94.2 M1:93.8 M1:95.
M1:95.8 M1:92.
(c)
All experiments are performed on a PC with an NVIDIA RTX 3090 GPU and the our code is implemented with PyTorch [53].
More implementation details can be found in Appendix F. 4.2 Quantitative Comparisons Overall Comparisons: Following the experimental settings in Sec. 4.1, we exhibit the experimental results in in Tab. 1. From the results, we can find that our method with the LLR ada score function is comparable with those non-ensemble completely unsupervised methods in FashionMNIST/MNIST, and significantly outperform them in CIFAR10/SVHN.
We emphasize that, without utilizing the labels of in-distribution samples [5, 6, 8, 9, 38, 51] or the prior knowledge collected from OOD samples [1, 28, 33], our method can still achieve competitive performance with these methods.
Effectiveness of LLR ada : Focused on the comparison between LLR >k and LLR ada exhibited in
{+Tab. 2 and Fig. 3(c), when the performance of LLR >k is sensitive to the selection of k , we can find that the performance of LLR ada can approach the best performance achieved by LLR >k with the optimal k , as shown in the right part of Tab. 
+}
(CIFAR/SVHN).
Furthermore, when the performance of
LLR >k is stable, the developed LLR ada can still achieve comparable OOD detection performance, as shown in the left part of Tab. 2 (FashionMNIST/MNIST) and Fig. 3(c).
The experimental results above demonstrate the adaptability of our developed LLR ada .
[Figure]
{+1.0 Normalized LLR >
+}
D e n s i t y
Trained on CIFAR10 CIFAR10 test (in) SVHN test (out) (d) Ours
[Figure]
x z >z >z >z >z >(
[Figure]
Ours/FashionMNIST x z >z >z >z >z >
[Figure]
MNIST x z >z >z >z >z >
[Figure]
x z >z >z >z >z >(
[Figure]
HVAE
[Figure]
MNIST 𝑥 𝑧 >0 𝑧 > 𝑧 > (e)
[Figure]
/CIFAR10 (in
[Figure]
𝑥 𝑧 >0 𝑧 > 𝑧 > (
[Figure]
[Figure]
SVHN (out
[Figure]
𝑥 𝑧 >0 𝑧 > 𝑧 >
[Figure]
[Figure]
[Figure]
Effectiveness of Informative Hierarchical VAE: Taking the same LLR ada as the score function, we compare the performance of informative hierarchical VAE with other hierarchical VAEs in Tab. 3.
From the results, we can find that our model outperforms others, indicating the effectiveness of alleviating “ posterior collapse ” with informative hierarchical VAE for OOD detection.
To evaluate the degree of “ posterior collapse ”, we use the ROC curves in Fig.
3 and the empirical densities in Fig.
4 to compare the performance of LLR > 2 scores based on the top-level latent variables zof these VAEs.
The ROC results in Fig.
3 demonstrate the superiority of the informative hierarchical
VAE, confirming that our model can provide more expressive higher-level latent representations for
OOD detection.
The empirical densities in Fig.
4 show that z 2 learned by our model on the [-CIFAR
 [Figure] in)/SVHN-]CIFAR (in)/SVHN (out) pairs owns better separability than those learned by BIVA.
4.3 Qualitative Analysis
Meaningful Semantic Space Learned by Informative Hierarchical VAE:
Following the same procedure as Fig.
2, we visualize more reconstructed samples on various benchmarks in Fig.
From the visualized results, for both in-distribution and OOD samples, we can find that the quality of the reconstructions generated from p θ ( x | z > 0 ) is surprisedly high, indicating that these reconstructions are almost dominated by the low-level features, which potentially explains the previous problematic phenomenon that these methods based on single-layer likelihood will assign higher likelihood scores for OOD samples and fail on OOD detection.
Focused on the reconstructions generated by p θ ( x | z > 2 ) , we can find that the developed informative hierarchical VAE can provide more realistic and clear reconstructed samples for both in-distribution and OOD inputs, indicating that our model can learn a more meaningful high-level latent semantic space than other models.
Furthermore, based on providing higher-quality reconstruction for OOD samples, the gap between the metric scores of in-distribution and OOD samples in informative hierarchical VAE tend to be larger than other models, leading to a better OOD performance as shown in Tab. 3.
Alleviating “ Posterior Collapse ” with Informative Hierarchical VAE:
As discussed in Sec. 3.1, “ posterior collapse ” will cause the higher-level latent variables to become uninformative.
Considering the developed informative hierarchical VAE shares the same network structure with the basic HVAE, in this part, we focus on evaluating whether our model can alleviate “ posterior collapse ” in higher layers when the network depth becomes deeper.
As shown in Fig.
6, for both in-distribution data and
OOD data samples, the overall quality of the reconstructions generated by our model is significantly higher than those generated by the basic HVAE.
Specifically, for FashionMNIST (in)/MNIST (out), the reconstructions generated by p θ ( x | z > 3 ) and p θ ( x | z > 4 ) of HVAE are almost the same, indicating that its posterior described by q ϕ ( z > 3 | x ) or q ϕ ( z > 4 | x ) tend to collapse to a prior distribution about
T-shirts.
On the contrary, the reconstructions generated by the our model are still realistic, where the reconstructions of MINST (out) samples generated by q ϕ ( z > 3 | x ) or q ϕ ( z > 4 | x ) preserve the semantic structural information learned form FashionMNIST (in), explaining the underlying reason why our model can achieve better OOD detection performance.
Similar conclusions can be achieved by the experimental results on CIFAR10 (in)/SVHN (out) as shown in Fig. 6.
Conclusion
In this paper, after presenting a thorough analysis of “ posterior collapse ”, we develop a novel informative hierarchical VAE to extract more expressive hierarchical latent representations by alleviating “ posterior collapse ”.
Then we theoretically explain why “ posterior collapse ” will limit the performance of existing hierarchical VAEs, and develop a novel Adaptive Likelihood Ratio score function for unsupervised OOD detection.
Experiments demonstrate the effectiveness of our method, whose main thought can be borrowed other hierarchical VAEs to improving their performance on downstream tasks relied on the hierarchy of latent representations.
Acknowledgment
This research is supported by the National Research Foundation, Singapore under its Industry
Alignment Fund – Pre-positioning (IAF-PP)
Funding Initiative and Competitive Research Programme (Grant No. NRF-CRP23-2019-0006).
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research
Foundation, Singapore.
Additionally, Tongliang Liu is partially supported by Australian Research
Council Projects DP180103424, DE-190101473, IC-190100031, DP-220102121, and FT-220100318.
Xiaobo Xia is supported by Australian Research Council Projects DE-190101473.
Appendix
Due to the page limitation, we put more technical details here for ease of understanding.
We will appreciate it so much if the audience can have a careful reading of the following Appendix.
A More discussions about “Posterior Collapse”
Here, we present a detailed reinterpretation for intuitively understanding the conflict of the optimization between the ELBO L and the mutual information I q ( x , z >k ) .
Then, we provide additional experiments to meausre the “posterior collapse” in higher layer of the latent variables z >k .
A.1 Detailed Derivations for Section 3.
First, we present a detailed formulation of the aggregated posterior q ϕ ( z >k ) as
[Equation]
[Equation]
( z [Equation]
| x ) .
Under the setting of top-down inference structure, with the aggregated posterior q ϕ ( z >k ) , where q ϕ ( z L | z L +1 ) :
= q ϕ ( z L | x )
[Equation]
and p θ ( z L | z L +1 ) :
= p θ ( z L ) , the KL term in Eq.
(3) can be rewritten as follows:
[Equation]
[Table]
[Table]
[Table]
) q ϕ ( z >k | x ) log q ϕ ( z >k | x ) − E p ( x ) q ( z >k | x ) log p θ ( z >k ) =
E p ( x ) (cid:20)(cid:88) k
l =1 D KL ( q ϕ ( z l | z l +1 )
||
p θ ( z l | z l +1 ))
(cid:21) + E p ( x ) q ϕ ( z >k | x ) log q ϕ ( z >k | x ) − E q ϕ
( z >k ) log q θ ( z >k )
+ E q ϕ
( z >k ) log q θ ( z >k ) − E p ( x ) q ϕ ( z >k | x ) log p θ ( z >k ) =
E p ( x ) (cid:20)(cid:88) k
l =1 D KL ( q ϕ ( z l | z l +1 )
||
p θ ( z l | z l +1 ))
(cid:21) + E p ( x ) q ϕ ( z >k | x ) log q ϕ ( z >k | x ) − E q ϕ
( z >k ) log q θ ( z >k )
+ E q ϕ
( z >k ) log q θ ( z >k ) − E q ϕ
( z >k ) log p θ ( z >k ) =
E p ( x ) (cid:20)(cid:88) k
l =1 D KL ( q ϕ ( z l | z l +1 )
||
p θ ( z l | z l +1 ))
(cid:21) + E p ( x ) q ϕ ( z >k | x ) log q ϕ ( z >k | x ) − E q ϕ
( z >K ) log q ϕ ( z >k )
+ D KL ( q ϕ ( z >k )
||
p θ ( z >k ))
=
E p ( x ) (cid:20)(cid:88) k l =1 D KL ( q ϕ ( z l | z l +1 ) ||
p θ ( z l | z l +1 ))
(cid:21)
+ I q ( x , z >k ) + D KL ( q ϕ ( z >k ) || p θ ( z >k )) .
Thus, the ELBO in Eq.
(3) can be rewritten as
[Equation]
[Equation]
Therefore, maximizing the ELBO will be opposite to maximizing the I q ( x , z >k ) , which leads the higher-layers posterior latent variables z >k to be independent of the input data x and collapse to the uninformative p θ ( z >k ) .
A.2 Quantitative experiments on “Posterior Collapse”
Since the “posterior collapse” of the in-distribution data would lead to a larger Likelihood Ratio
LLR >k in high layers, which is harmful for OOD detection, we add an additional experiment here to testify the “posterior collapse” with the metric average bits per dim as shown in Table 4.
The L >L − 1 x is the ELBO for the partial generative model, which could be used to evaluate the reconstruction quality of z L , and more details can be found in Eq.
The average bits per dim is calculated in the testing split of the in-distribution dataset and the OOD detection performance is tested with LLR >L − 1 , where L = 5 in the FashionMNIST/MNIST pair and L = 3 in the CIFAR10/SVHN pair.
Note that, LLR >L − 1 =
L x − L >L − 1 x .
FashionMNIST(in)/MNIST(out)
[Table]
Metthod
L x L > 4 x AUROC ↑ AUPRC ↑ FPR80 ↓ HVAE(5) 2.
[Table]
As the results shown in Table 4, although the baselines for comparison (HVAE, LVAE, and BIVA) can obtain better reconstruction performance on L x , they still suffer from a large shrink in L >L − 1 x , which is mainly caused by the “posterior collapse”.
On the contrary, the developed informative
HVAE can acquire stable performance from L x to L >L − 1 x , resulting in a smaller Likelihood Ratio
LLR >L − 1 , which illustrates why our method can achieve much better performance on unsupervised
OOD detection.
B Derivations for analyzing the LLR >k in Section 3.
For ease of understanding of the Eq.
(6), we give a detailed derivation below, which is mostly based on the Havtorn et al.
First, we define a looser ELBO for each observation x as below of the partial generative model p θ ( x | z >k ) =
E p θ ( z ≤ k | z >k )
[ p θ ( x | z ≤ k )] , which reconstructs the observation x by taking z >k drawn from the variational inference network q ϕ ( z >k | x ) , L
[Equation]
recall to the common ELBO for each observation x as
[Equation]
then, the LLR >k is defined as
[Equation]
Further
[Equation]
the detailed derivation for Eq.
(7) is as follows:
[Equation]
] =
[Equation]
−
[Equation]
[Equation]
=
[Equation]
x ))]
≈ E q ϕ
( z >k | x ) [ D KL ( p θ ( z ≤ k | z >k ) ||
q
[Equation]
( z ≤ k | z >k ))] , (18) when the approximated posterior q ϕ
( z ≤ k | z >k ) is closer to the true posterior p θ ( z ≤ k | z >k , x ) after training, i.e. , D KL ( q ϕ ( z ≤ k | z >k )
||
p θ ( z ≤ k | z >k , x ))
≈
and D KL ( p θ ( z ≤ k | z >k )
||
p θ ( z ≤ k | z >k , x ))
≈ D KL ( p θ ( z ≤ k | z >k ) ||
q ϕ ( z ≤ k | z >k )) .
exhibited in Fig.
2, when setting k = 0 or k = 1 , we can find that the quality of the reconstructions generated from p θ ( x | z > 0 ) is surprisedly high, indicating that
the
KL-divergence between p θ ( z ≤ k | z >k ) and q ϕ ( z ≤ k | z >k ) is small for both in-distribution and OOD samples, which makes it problematic for OOD detection with single-layer likelihood LLR > 0 or
LLR > 1 ; when setting k = 2 , the KL divergence between p θ ( z ≤ k | z >k ) and q ϕ ( z ≤ k | z >k ) will be relative small for in-distribution samples, but large for OOD samples, which is the main reason for the success of LLR > 2 ; however, when setting k = 3 or k = 4 , the latent variables z ≤ k generated p θ ( z ≤ k | z >k ) will be clearly distinct from those drawn from q θ ( z ≤ k | z >k , x ) for both in-distribution and OOD samples, resulting in that the performance of OOD detection with LLR > 3 or LLR > 4 will be worse than LLR > 2 .
The reason why OOD detection based on LLR > 3 or LLR > 4 can outperform
OOD detection based on LLR > 0 or LLR > 1 is that the generative model p θ ( z ≤ k | z >k ) can still learn the generation mechanism of in-distribution samples at higher hidden layers, even when “ posterior collapse ” occurs.
Based on these findings, the intuition of designing LLR ada is to move beyond the choose of k but enhance the importance of the score over some discriminative layers, like LLR > 2 , in the overall metric for OOD detection.
To achieve these goals, we use R ( x , z >k ) to measure the relevance between x and z >k , and the adaptive weight R ( x , z >k − 1 ) R
( x , z >k ) in LLR ada will be relatively large when the data information drop rapidly at the current hidden layer, like k = 2 .
We admit that “ posterior collapse ” will still hurt the performance of LLR ada , leading to worse performance than LLR >k with the optimal k , but it can avoid the unreasonable hyper-parameter adjustment based on OOD samples.
Moreover, with the informative hierarchical VAE to alleviate “ posterior collapse ”, the performance of LLR ada will be even better than LLR > 2 on some datasets.
D More Details of the Datasets
We use additional datasets to evaluate the OOD detection performance.
For FashionMNIST/MNIST pair, we add KMNIST [43], notMNIST [44], Omniglot [45] and SmallNORB [46] datasets.
KMNIST is a dataset, adapted from Kuzushiji Dataset, as a drop-in replacement for MNIST dataset, which contains 70,000 28 × 28 grayscale images.
notMNIST is a dataset made by 547,838 28 × 28 grayscale images of extracted glyphs from some publicly available fonts with letters A-J taken from different fonts.
Omniglot contains 32,460 28 × 28 grayscale images ofdifferent handwritten characters from 50 different alphabets.
SmallNORB contains 97,200 28 ×grayscale images of 50 toys belonging to 5 generic categories: four-legged animals, human figures, airplanes, trucks, and cars.
For CIFAR10/SVHN pair, we add CelebA [47], Places365 [48], Flower102 [49] and LFWPeople [50] datasets.
CelebA is a large-scale face attributes dataset with more than 200K celebrity images, each with 40 attribute annotations.
Places365 contains 1.8 million train images from 365 scene categories, 50 images per category in the validation set, and 900 images per category in the testing set.
Flowers102 is an image classification dataset consisting of 102 flower categories, where flowers were chosen to be flowers commonly occurring in the United Kingdom and each class consists of between 40 and 258 images.
LFWPeople contains more than 13,000 images of faces collected from the web.
All these datasets’ images would be randomly cropped into the dimension of 32 × 32 ×before sending into the models.
E Details of the Baselines
Due to the space limitation, we use the abbreviation for each baseline in Table 1.
Here, we give a detailed description for each baseline of the three categories:
• “Labels” (Methods using in-distribution data labels y ): maximum softmax classification probability (CP) method [5] and its variants, denoted as "CP", "CP(OOD)" with OOD as noise class, "CP(Cal)" with calibration on OOD and "CP(Ent)" with entropy of softmax classification probability p ( y | x ) , and Mahalanobis distance (MD) method [9], latent Mahalanobis distance (LMD) method [38], ODIN method [8], VIB method [6] and deep ensembles (DE) method [51] with 20 classifiers; • “Prior” (Methods using prior knowledge assumption of OOD): Likelihood Ratio (LR) method [1] with different backbones, denoted as "LR(PC)" with backbone PixcelCNN, "LR(VAE)" with VAE and "LR(BC)" with binary classifier), Outlier exposure (OE) method [28] and Input complexity (IC) method [33] with different backbones, denoted as "IC(PC)" with backbone PixcelCNN, "IC(Glow)" with backbone Glow and "IC(HVAE)" with backbone HVAE; • “Unsupervised” (Methods with no OOD-specific assumptions): Ensemble methods: WAIC method [24] with different backbones, denoted as "WAIC(5Glow)"
with 5 Glow models, "WAIC(5VAE)" with 5 VAE models and "WAIC(5PC)" with 5 PixcelCNN models; Not ensembles methods: Likelihood regret method [32] and its variant "Likelihood regret(z)",
Log-Likelihood Ratio ( LLR ) method [27], which achieved the best performance with " LLR > 1 (HVAE)"
(hyperparameter k = 1 and backbone method 3-layer HVAE trained on binarized data) for FashionMNIST(in)/MNIST(out) pair and " LLR > 2 (BIVA)" (hyperparameter k = 2 and backbone method 10-layer BIVA) for CIFAR10(in)/SVHN(out) pair.
For this
LLR method, we denote their best combinations’ result in Tab. 1 as "HVK" (Hierarchical
VAEs Know what they don’t know).
F Details of the Implementation
To make sure the new training objective in Eq.
(10) can really lead to an informative hierarchical
VAE, we do not use the warm-up trick or the free bits trick.
However, we apply the warm-up trick (200 epochs for the Warmup anneal period) and free bits trick (2 nats per z i and 400 epochs for the free bits period) to the other three hierarchical VAEs (HVAE, LVAE, and BIVA) to empirically alleviate the posterior collapse, which is proven in Havtorn et al.
and we follow their procedure to train these three hierarchical VAEs.
Following Havtorn et al.
[27], the VAE-based methods’ results are computed with 1000 importance samples.
However, our method’s results only get slight improvement after sampling, considering the computation burden brought by it, we report the results of our method without importance sampling.
But we also use the importance sampling for other baseline hierarchical VAEs (HVAE, LVAE, and
BIVA).
G Error Bar
We randomly run 5 seeds for our method in experiments and report the error bar as below.
[Table]
H Limitation
The developed informative hierarchical VAE can alleviate “posterior collapse” to a certain degree, but still cannot completely avoid the appearing of this phenomenon in VAEs.
Then, the developed
LLR ada is not the optimal choice of score function of unsupervised OOD detection, which needs to be investigated in the future work.
Considering the computational footprint change, we take the vanilla VAE equipped with Likelihood Ratio as the baseline for analysis.
For the space complexity, our method doesn’t introduce any additional model parameters or memory cost.
For the time complexity, compared to the baseline, our method requires additional L − 1 times computation cost to calculate those expected log-likelihood terms in the loss function during training, specifically
L
( x | z k )] , where L denotes the number of layers and will be a relative small number in practice.
I Broader Impact
The developed method in this paper can be straightforwardly applied to real-word applications based on hierarchical VAEs, and alleviate their “posterior collapse” to achieve better model performance.
The adaptive score function can be used for purely unsupervised OOD detection, which can boost the reliability and safety of recent machine learning (ML) systems.
J Comparison with other methods to alleviate “ posterior collapse ”
We provide more comparisons with the methods designed for alleviating the “posterior collapse”, including "Warm-up", "OverSmooth", "BIVA", and Our method "Informative".
"Warm-up" [30] gradually increases the weight of the KL-divergence term in the learning objective from 0 toand we set the warm-up epochs as 200 with a total training epoch as 1000.
"OverSmooth" [54] sets the σ x as a one-dimensional parameter and updated according to the training objective, where the reconstruction likelihood function is p θ ( x | z ) =
N ( x | µ x ( z ) , σ 2 x I ) .
"BIVA" [29] introduces a bidirectional inference and generative network architecture, but it changes the original structure of vanilla hierarchical VAE and may hurt the hierarchy of the latent variables.
To provide an intuitive comparison of these methods’s effect on alleviating “posterior collapse”, we introduce the OOD detection performance on a vanilla hierachical VAE, termd as "Vanilla", as an additional baseline.
Then, we testify these methods’ performance under 3 different score methods for OOD detection, where L x represents the evidence lower bound (ELBO) of the VAE, LLR >L − 1 represents the gap between partial ELBO of the highest-level latent variables and the the ELBO of lowest-level latent variables, and LLR ada is an adaptive score that evalutes the whole hierarchy of the latent variables.
As shown in Table 6 and Table 7, the developed informative hierarchical VAE, termed as "Informative", outperforms other methods significantly especially under the score LLR >L − 1 and LLR ada .
[Table]
+ Vanilla [ [Table]
L
[Table]
+ Warm-
[Table]
[Table]
L x + OverSmooth [54]
[Table]
L x + BIVA
[Table]
K Comparison with non-VAE methods
Likelihood-based methods are promising to detect the OOD data in an unsupervised manner, since they could give an estimation of the data x ’s likelihood p ( x ) under the learned distribution p of in-distribution data.
Among likelihood-based methods, Flow-based models [17], auto-regressive models [18] [Table] and variational auto-encoder (VAE) models are popular for OOD detection tasks.
HVK [27] achieves the state-of-the-art of OOD detection under the unsupervised setting.
Though our method could outperform HVK, it is still interesting to have an comparison with other two type of models (Flow-based and auto-regressive models) to see whether our methods could also outperform them.
We add 3 non-VAE methods as our baselines for comparison: "Glow", "Flow+Group", and "PixcelCNN++".
"Glow" [25] try to do OOD detection with Glow model and find the counterfactual behaviour of assigning high likelihood to OOD data specially in the model family of Flow models. "FLow+Group" [55] is an SOTA flow-based OOD detection method but is not initially designed for HVAE: CIFAR10 (
/ SVHN [Table] [Table] our setting, but for group OOD detection, where their model needs to justify whether a batch of sample { x 1 , x 2 , ..., x n } , ( n > 1) is an OOD batch.
Note that, the batch size n is set as 5, 10, andin their paper.
Luckily, "FLow+Group" also modify their method via data augmentaion to the point
OOD detection situation, i.e., n = 1 , which is the same as our setting, and we directly report their
OOD detection results in their Appendix F. "PixcelCNN++" [1] propose to use an auto-regressive model (PixcelCNN++) for OOD detection with the help of additional OOD datasets like NotMNIST dataset, and we report the results of this model under our setting (no additional datasets to help training).
As shown in Table 8 and Table
[Table]
Our method could also outperform other non-VAE methods.
[Table]
[Table]
L Comparisons of Score Functions
OOD detection methods need to assign a score for each data sample and detect the OOD data out according to this score.
However, since the score method LLR >k proposed by HVK [27] needs to select the optimal hyperparameter k to achieve the best OOD detection performance, which is unreasonable under the unsupervised setting, we design a novel score function named LLR ada that does not need to select the k .
Thanks for the Reviewer 9EQd’s awesome suggestion, there could be another way for automatically selecting the k , where the k is selected based on the largest R -ratio in Eq.
To make a further investigation, we provide a comparison between this score function, termed " LLR opt k ", and our LLR ada .
As shown in Table 10 and Table [Table] LLR opt k could achieve a promising OOD detection performance but still underperform our developed LLR ada .
A more intuitive and numerical analysis about these methods has been provided in Appendix M
[Table] [Table]
based on a 3-layer HVAE trained with informative loss on in-distribution dataset CIFAR10.
M Measure LLR ada on vanilla HVAE without Informative loss
It would be interesting to see whether the LLR ada will be effective on a vanilla hierarchical VAE trained without the informative loss.
Take a 5-layer HVAE trained on FashionMNIST for example, we give a comparison with different score functions for OOD detection as shown in Table 12.
To better undertand the underlying mechanism of these score methods, we compute the mean negative log-likelihood − log p θ ( x | z >k ) for reconstruction and log-likelihood ratio ( LLR >k ) for each layer of a 5-layer vanilla HVAE in Table 13.
Since the values LLR >k of in-distribution data is closer or larger than OOD data, it is not surprising that the LLR >k cannot achieve promising OOD detection performance.
However, the performance could be significantly improved with the score LLR opt k and LLR ada .
Specifically, for score LLR opt k , it would highly possible to assign LLR > 1 (1.65 × 10 3 ) for indistribution data and assign LLR > 2 (2.90 × 10 3 ) for OOD data, which makes it easier to detect OOD data.
For score LLR ada , its average score for in-distribution data is 1 .
65 + 4 .
50 ∗ (2 . 97 − 1 . 65) + 7 .4 .
− 2 . 97) + 7 . 81 7 .
81 ∗ (5 . 90 − 5 . 90) = 8 .
173(10 3 ) , but for OOD data, the average score is 0 .
74 + 4 .
(2 . 9 − 0 .
74) + 6 . 89 4 .
− 2 . 9) + 6 . 89 6 . 89 (5 .
46 − 5 . 46)
= 11 .
Since the average score LLR ada of OOD data is much larger than in-distribution data, our developed LLR ada could be a more promising score function to achieve better OOD detection performance. [Table] [Table] HVAE.
N Numerical analysis to illustrate “ posterior collapse ” in Fig.
It is interesting to investigate the numerical changes when the “posterior collapse” occurs, such as the data samples visualized in Fig.
As shown in Table 14, from shallow to deep (Layer 1 to Layer 4), we can find that the KL-divergence of HVAE gradually reduces to 0, which indicates that the posteriors of 4-th and 5-th hidden layers collapse to their priors, resulting in that the high-level latent variables z 4 and z 5 sampled from the posterior q ϕ ( z k | z >k ) have no information of input data x .
Further, we use t-SNE method to visualize the learned latent data representations in Fig.
Note that, different colors in Fig.
7 indicates different classes of data samples.
As shown in Fig.
7, we can find that the latent space of HVAE gradually collapses to a non-informative prior distribution, while the learned latent space of our method is still informative at higher layers.
To see more reconstructed data samples of the partial generative models p ( x | z > 3 ) and p ( x | z > 4 ) ,
To intuitively demonstrate that the posterior of our method does not collapse to a single point, we visualize the data samples generated from p θ ( x | z >k ) by taking the latent variables z k sampled from the posterior q ϕ ( z k | z >k , x ) as input, where x is a fixed data point.
As shown in Fig.
8, the diversity of the generated samples demonstrate that the posterior q ϕ ( z k | z >k , x ) collapses to its prior distribution p θ ( z k | z >k ) rather than a single point.
O Comparisons of Reconstruction and Generation quality
The reconstruction and generation capability are two important model properties of VAE, and here we compare our method with the vanilla HVAE on both these two aspects.
[Table] [Table] Firstly
[Table]
we quantitatively evaluate the reconstruction capability of the partial generative models p θ ( x | z >k ) conditioned on latent variables z >k at different hidden layers, and report the average bit per dim results in Table 15.
From the results, we can see that our method can achieve a comparable reconstruction quality with other baselines on p θ ( x | z > 0 ) and significantly outperform them on the reconstruction conditioned on higher-layer latent variables.
For the generation capability [Table] we qualitatively visualize the data samples generated from artial generative models p θ ( x | z >k ) conditioned on z >k drawn from the prior distribution p θ ( z L ) =
N (0 , I ) .
From the results shown in in Fig.
[Table]
we can find that the quality and diversity of data samples generated by our model significantly outperform those generated by HVAE, indicating the benefits of alleviating “posterior collapse.”
Thus, the aforementioned experimental results demonstrate that our method can perverse the versatility of VAE.
P Comparison on more natural images
To investigate the effectiveness of our method on more natural images, we provide additional comparisons on the other datasets, including LFWPeople [50] (people’s faces in the wild), Flower [49] (102 types of flowers), Food101 [56] (101 types of food), Places365 [48] (365 scene categories), and Tiny-ImageNet [57] (containing 200 categories of images).
With the same score function
LLR ada , we train the VAE-based model on each of these datasets, where each image is resized as 32 × 32 × 3 before training, and then use it for OOD detection on SVHN dataset.
The unified network structure of these 3-layer models are [64, 32, 16] from shallow to deep.
As shown in Table 16, LVAE, BIVA, and our method can generally outperform the vanilla HVAE, while our method could still significantly achieve the best performance on these dataset pairs, which indicates the generality of our method on unsupervised OOD detection.
[Table]
[Figure]
[Figure] [Figure]