Paper: Rethinking Test-time Likelihood: The Likelihood Path Principle and Its Application to OOD Detection (ICLR-2024)

Reviewer comment (Theory):
I have the concern of whether the assumptions (essential separation concepts) are too strong, so that they can easily imply the theoretical guarantee. Also, I am not sure whether these "separations" are reasonable in realistic dataset.

Author response:
We thank you for pointing this confusion out. We would like to invite you to Main Responses 7 and 8, and further clarify below.

**[Essential separation is no stronger than prior literature]**

The assumptions of essential separation and distance are no stronger than any other prior works, theoretical or empirical. To our best knowledge, Essential separation and distance strictly include “near” OOD and “far” OOD settings considered in most if not all prior literature [1, 2, 3]. If they were too strong, prior informal and rigorous attempts would all become too strong. 

We remark that prior arts consider either “far OOD” (total separation) or “near OOD” (not much separation). Essential separation is separation with a specified level of probability, which incorporates both. We invite you to explore the generality of our definitions via examples from B.2 to B.5.


**[Our definitions work for any any metric including realistic datasets, e.g. SVHN v.s. CIFAR10]**

They are also developed for any abstract metric spaces, in theory including human-like perceptual distances (proxied by perceptual distance in [4]). Take SVHN vs. CIFAR10 as an example, they are clearly separable (no one confuses digits with cars), as a result they are by definition essentially separable.


**[Our proof mainly uses VAEs’ structure, not the essential separation]** 

As for whether these general assumptions can imply theoretical guarantees, we invite you to check our proof in the Appendix. First, our proof mostly uses VAEs’ unique structure, not the essential separation themselves. Second, since our definitions strictly include prior ones, prior assumptions are stronger. If the proof for VAEs simply follows from the definitions, prior works would have proved similar results. To our best knowledge, ours is the first provable work in the unsupervised OOD detection setting.

Reviewer comment (Writing):
The writing is unclear in the sense that some notation seems not to be defined, such as $p\_{\theta}$, $\mu\_z$. This makes me sometimes a little bit confusing.

Author response:
We thank you for improving our exposition. We assume familiarity to the VAEs literature where we recall Equation 1. $p_{\theta}$ is defined there. We also assumed familiarity with the minimal sufficient statistics of VAEs on page 4, where we defined the sufficient statistics $T$ including $\mu_z$. We will make these dependencies more explicit.

Reviewer comment (Theory):
The utilization of  $\mid || \mu_{z} (x_{{OOD}}) \|-r_0 \mid$ to approximate the distance between test and ID samples is questioned for its lack of a principled basis. If ID samples have a wide spread in $μ(x_{IID})$ in the latent space, the absence of a singular reference point makes the approximation meaningless. The reliance on VAEs' regularization of the posterior on $z$ towards a Gaussian (typically zero mean) implies the technique may not be generalizable to other generative models with distinct latent variable regularization.

Author response:
>The utilization of  |$\lvert \mu_{z}(x_{\text{OOD}}) - r_0 \rvert$| to approximate the distance between test and ID samples is questioned for its lack of a principled basis. If ID samples have a wide spread in in the latent space, the absence of a singular reference point makes the approximation meaningless. 

Thank you for pointing this our and improving our rigor. We invite you to Main Response 8 to see our now fully rigorous justification. In particular, the aforementioned approximations, as in Equations 13 and 14, enjoy similar provable guarantees (Proposition B 16 and Corollary B 17) as Theorem 3.8. The gap between Equation 10 (main theorem) and Equations 13 +14 (experimental version) depends on empirically verifiable concentration phenomenon (discussed in Section 3.2, illustrated on Figure 2. In light of these observations, we also strengthen this concentration via regularization (Section 3.2)).

For the DC-VAEs we used, even without regularization, we empirically observe that the latent code concentrates. You are correct that when IID samples have a wide spread, our experiments without regularization showed minor performance degradation. But the unregularized model still delivers strong performances.

Theoretically, this is related to our essential definitions. Many false statements in fact are true with high probability. The widespread latent code is an example: while the latent code norm can spread a wide range, most of them are concentrated near a narrow range. In sum, widespread latent codes might weaken an absolute logical statement to a high probability one. But it does not make it meaningless. These are experimentally verified and rigorously proved (Proposition B 16 and Corollary B 17).


>The reliance on VAEs' regularization of the posterior on towards a Gaussian (typically zero mean) implies the technique may not be generalizable to other generative models with distinct latent variable regularization.

You are correct that the regularization is VAEs specific, but we would like to invite you to visit Main Response 4 for why our VAEs model specific techniques are useful in the present unsupervised OOD detection we consider.
The no free lunch principle, stating no single algorithm works the best in all cases, also discussed in Footnote 3, suggests dependence on VAEs can also be a strength, making our dependence on VAEs a double-edged sword. Whether it is a weakness or not highly depends on the applications of interest.

When a few current SOTA sits below AUC 0.7, we are willing to trade off generality for empirical performances. Looking at Table 1, we believe taking advantage of VAEs’ structural uniqueness is exactly why our empirical performances match or exceed SOTA benchmarks which are based on arguably more sophisticated models (Glow, DDPM, etc.). This implicitly shows our method is more sophisticated, since our model is much smaller. The sophistication probably comes from our model specific choice. For the present application (IID v.s. OOD), while the benefits of richer latent structures remain unclear (geometrically, we only need to carve the latent or visible spaces into IID v.s. OOD, it is less clear richer latent structure surely helps), our methods’ performances are demonstrated.

Moreover, as discussed previously, our performances only degraded a little without such concentration regularizations. This is because the unregularized VAEs already exhibit concentration phenomenon when VAEs have high latent dimensions, which motivates us to further strengthen it.

Reviewer comment (Theory):
The assumptions for essential separation and Lipschitz conditions are too strong. The separation (defined by the $L_2$ norm) may not hold for real world problems. The Lipschitz conditions are not enforced during training of VAEs (or in other generative models) as well.

Author response:
> The assumptions for essential separation and Lipschitz conditions are too strong. The separation (defined by the L2 norm) may not hold for real world problems. 

We thank you for pointing out these expository improvements. 

For a discussion on essential separation, we would like to invite you to visit Main Response 7. In short, The assumptions of essential separation and distance are no stronger than any other prior works, theoretical or empirical. To our best knowledge, Essential separation and distance strictly include “near” OOD and “far” OOD considered in most if not all prior literature [1, 2, 3]. If they were too strong, prior informal and rigorous attempts would all become too strong. 


In our construction of essential separation, the probabilistic guarantee $\epsilon_{\text{IID}}$ and $\epsilon_\text{OOD}$ trades off with the margin $m_{\text{inter}}$. For a difficult problem (“near OOD” such as HFlip), if we want a big margin, the probabilities won’t be high. We invite you to Appendix B.1, Examples B.4 and B.5 for illustrations. 
In there, we show how essential separation and distance behave even when $P_{\text{IID}} = P_{\text{OOD}}$. In short, it would produce claims like, with probability at least greater than 0, $P_{\text{IID}} = P_{\text{OOD}}$ are separated by a fixed positive margin. However, when $P_{\text{IID}}$ and $ P_{\text{OOD}}$ begin to isolate a little, it would say, with probability greater than 0.1, $P_{\text{IID}} = P_{\text{OOD}}$ are separated by 0.01. The harder the problem, the more vacuous the guarantees become. The more separable between $P_{\text{IID}}$ and $ P_{\text{OOD}}$, the bigger their essential distance (far OOD problem such as SVHN v.s. CIFAR).


For the L2 norm, we invite you to Main Response 8. In short, our proof techniques do not rely on the L2 norm property; it holds for any metric spaces (e.g. $L^{\infty}$ popular in the adversarial robustness literature, perceptual distance [4], etc.).

We chose L2 norm for the paper because of VAEs' Gaussian decoder. We agree that the utility of Theorem 3.8 depends on which perceptual distance is used. However, designing a proper metric for perception is beyond the scope of the paper - we only prove that our theory likely includes it. We do not consider this as a weakness of our paper. We will restate and modify the proof of Theorem 3.8 for any abstract metric spaces.


> The Lipschitz conditions are not enforced during training of VAEs (or in other generative models) as well.

We thank you for this insightful comment. We have done some works on it, but we did not include it in the paper, due to the ~20 pages appendix. 

Lipschitz conditions can be enforced ([5], [6], [7], [8]]) during VAEs training, and we conducted preliminary experiments on enforcing the decoder. However we do not observe substantial differences on OOD detection performances. The problem with this constraint alone is that Theorem 3.8 also requires the encoder to be co-Lipschtiz (our new definition) as well. Since this involves the inverse image of the encoders, we are unaware of any literature that enforces co-Lipscchtizness.

We managed to link it to some better known concepts, i.e. quasi-isometry in geometric group theory (Co-Lipschitzness and Quasi-Isometric Embedding, Section B.2 in Appendix). We reformulated co-Lipschitzness in a way that is reminiscent of the adversarial robustness certification literature. We remark that it took the adversarial robustness community a few years before this became more mature, in the supervised classification setting. Enforcing both Lipschitz and co-Lipscthiz is highly non-trivial in our setting and is beyond the scope of the present paper.

Nevertheless, even if we are unable to enforce it, it does not mean DGMs are not Lipschitz or co-Lipschitz. In fact, empirical evidence suggests that they are ([9]), but the constants can be larger than ideal. We will add these observations to the Appendix.

Reviewer comment (Writing):
Some concepts are articulated in an imprecise, non-rigorous manner. For example, the phrase “break in the right way” from Section 2 lacks clarity.

Author response:
> The paper's presentation quality needs improvements. Some concepts are articulated in an imprecise, non-rigorous manner. For example, the phrase “break in the right way” from Section 2 lacks clarity.

We intended to make the paper more lively due to relatively more math in the paper. “Break in the right way” is detailed in Section B.4.2. We basically mean all the known VAEs likelihood estimation is broken, but how we pair two broken VAEs can lead to performance gain.

Reviewer comment (Presentation):
Several explanations are relegated to appendices, compromising the fluidity and comprehension of the main text. Definitions, like B.6 and B.7, are cited without main text elaboration.

Author response:
> Several explanations are relegated to appendices, compromising the fluidity and comprehension of the main text. Definitions, like B.6 and B.7, are cited without main text elaboration. The excessive use of bold text and protracted informal subtitles detract the reader.

While our paper develops some new theories, we also want it to be accessible to ML practitioners. This will create some exposition difficulty. We thank your suggestion and will try to improve the flow.

Reviewer comment (Presentation):
The excessive use of bold text and protracted informal subtitles detract the reader.

Author response:
> Several explanations are relegated to appendices, compromising the fluidity and comprehension of the main text. Definitions, like B.6 and B.7, are cited without main text elaboration. The excessive use of bold text and protracted informal subtitles detract the reader.

While our paper develops some new theories, we also want it to be accessible to ML practitioners. This will create some exposition difficulty. We thank your suggestion and will try to improve the flow.

Reviewer comment (Theory):
When the assumptions will hold and measure it empirically if possible to validate?

Author response:
> When the assumptions will hold and measure it empirically if possible to validate?

We thank you for this insightful comment. We have now included a new Section, Co-Lipschitzness and Quasi-Isometric Embedding, in the Appendix B.2. As discussed in our previous comment on enforcing Lipschtizness, we relate co-Lipschtizness and quasi-isometry in geometric group theory. This suggests ways of estimating the co-Lipschitz degrees, for example, by rethinking the adversarial robustness certification literature (both lower and upper bounds). We believe this is very interesting, but it is beyond the scope of the present paper.

To our best knowledge, this is the first provable work in the DGMs unsupervised OOD setting. While imperfect, we believe it adds significant value to the current works. Fully verifying it requires much more not-yet-developed machinery, which could require a few papers of work.

Reviewer comment (Theory):
Explain the use of $\mid || \mu_{z} (x_{{OOD}}) \|-r_0 \mid$, when does this serve as a good approximate? Is this limited to methods like VAE that regularize posterior distribution of $z$?

Author response:
> Explain the use of |$\lvert \mu_{z}(x_{\text{OOD}}) - r_0 \rvert$|  when does this serve as a good approximate? Is this limited to methods like VAE that regularize posterior distribution of ?

We believe this is already addressed in the above discussion (Proposition B 16 and Corollary B 17) where the approximation gap depends on the empirically verifiable concentration effect, but we reiterate that the no free lunch principle applies. A method that is “limited” to the standard VAEs also means this method is tailored for it. This inductive bias makes the method perform well in our applications of interest, namely OOD detection. If any OOD detection score function is applicable to all other models, it is unlikely to perform well in many detection problems. See footnote 3 in our paper or [10] for more details.

Reviewer comment (Reproducibility):
*Complexity of implementation*. While not explicitly mentioned, the introduction of new theoretical concepts might imply a more complex implementation and understanding, potentially limiting accessibility for practitioners. The author has not provided any valid implementations for reviewing.

Author response:
We did not provide implementations because there is dependency on private repos. We have attached a self-contained jupyter notebook that illustrates the pipeline on synthetic datasets. We would like to emphasize that although it takes some math to derive our algorithm, its implementation is simple, similar to DoSE [1] as we mentioned in the paper.

Reviewer comment (Presentation):
The intuition behind the theorem and the illustration in Figure 1 can be further improved. As the main figure of the paper which is introducing the idea, Figure 1 is not easy to follow. You need to read the paper all the way to the end of page 7 so you can understand the 4 cases and their connection to the idea presented in the paper.

Author response:
We thank you for pointing this out. We will work on a revised figure with modified caption to better illustrate the LPath principle.

Reviewer comment (Experiments):
As the authors have mentioned, Equation 10 is non-trivial to compute and an approximation is provided. The effect of the error of this approximation on the performance of the algorithm can be further studied in synthetic cases.

Author response:
We believe you are referring to the gap between Equation 10 and Equations 13+14. We have now provided a more detailed justification with rigorous proofs. Equation 13 and 14 now come with similar guarantees (Proposition B 16 and Corollary B 17). The gap between Equation 10 and Equations 13+14 depends on empirically verifiable concentration phenomenon (discussed in Section 3.2, illustrated on Figure 2). In light of these observations, we also strengthen this concentration via regularization (Section 3.2).

Reviewer comment (Evaluation):
The fact that the method outperforms other VAE baselines but doesn’t perform as well as more sophisticated baselines makes the practical usage of the method in safety critical domains less feasible.

Author response:
We would like to emphasize that while we do not perform better in all cases, we surpass or match all others except SVHN (IID) vs CIFAR 10/VFlip/HFlip (OOD). Among these mostly widely used benchmarks, our method is at least as good as all others overall. 

As for practical application in safety critical domains, we invite you to Main Response 2 for our advantage. In safety critical domains, theoretical guarantees are arguably more important in OOD detection, because in deployment, there is no way to control the streaming data. Offline validations such as the ones every paper performs are not guarantees to translate to the real world.

In contrast, as a provable method, even if we are not the best in the deployment environment, we can theoretically identify the factors that caused the degradation. The co-Lipschitz degrees (K, k) are the most difficult to estimate in practice, which is a separate topic itself. We now developed Lemma B.12 that suggests a way of estimating those, related to the adversarial robustness certification literature.

For the above reasons, we believe our method is more interpretable and reliable than other SOTA methods. We would like to hear from reviewer 6idB whether this discussion has addressed the aforementioned concern.

Reviewer comment (Theory):
How does the approximation error of Equation 12 affect the performance?

Author response:
Approximation occurs in Equation 13 and 14, where justification heuristically is provided in Footnote 10 and fully rigorously in Proposition B 16 and Corollary B 17 in the new version. They also come with provable guarantees.

Reviewer comment (Presentation):
The motivation behind the paired VAE idea in section 4 is unclear. The idea has been introduced in few lines and the reasoning behind it is deferred to the appendix. Can you either expand the motivation in the main text or move these few lines to the appendix?

Author response:
We thank you for improving our exposition. We propose adding the following to Section 4. The idea behind VAEs pairing is that Equation 10 in Theorem 3.8 prefers smaller K, while Equation 11 prefers bigger K. These two conditions cannot be satisfied by a single VAE, but can be achieved with a paired VAEs. More details and discussion are given in the Appendix B 4.2 and B 4.3.

We appreciate your additional thoughts on the change. Any further suggestions would be welcomed.

Reviewer comment (Reproducibility):
What are the hyperparameters that are needed for the OOD decision rule?

Author response:
We do not require any hyperparameter in our paper as we used the AUROC as the evaluation metric, commonly used in the literature. We may be missing something basic, could you clarify what decision criteria are being referred to? 

Our algorithm (Page 8) and Appendix C.3 VAE ARCHITECTURE AND TRAINING describe how we get a decision score for OOD detection. 

In stage 1, our VAEs training follows the widely cited open sourced implementation of DC-VAEs, and we train in a similar manner. The major difference is that we experiment with different latent dimensions guided by our theory.

In stage 2, from the open sourced implementation, https://github.com/winstonll/COPOD/blob/master/models/cod.py, since COPOD requires no hyper-parameter to fit the 2nd stage classical OOD algorithm, it is easy to obtain a decision score.