Paper:
Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) have been observed to assign lower uncertainty to the OOD than the target distribution. In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images. Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training. Our detector is simple, efficient at test time, and outperforms baseline OOD detectors in various domains. Further results show that SVD-RND learns better target distribution representation than the baseline RND algorithm. Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy on the CelebA dataset.

Review:
UPDATE: 
I acknowledge that I‘ve read the author responses as well as the other reviews. 
I appreciate the clarifications and improvements made to the paper. I‘ve updated my score to 6 Weak Accept. 
####################
This paper presents the idea to use blurred images as regularizing examples to improve out-of-distribution (OOD) detection performance based on Random Network Distillation (RND). The paper proposes to generate sets of such blurred images via Singular Value Decomposition (SVD) on the training images by pruning the lowest K non-zero singular values. The proposed method, SVD-RND, then extends the standard RND objective, which is to train a predictor network f to minimize the L2 loss to the output of some randomly initialized network over the (original) training data, with an additional regularization term that minimizes the L2 loss to the outputs of further multiple randomly initialized networks over the sets of blurred images. In OOD experiments on CIFAR-10, SVHN, TinyImageNet, LSUN, and CelebA, the proposed SVD-RND consistently outperforms baselines and recent competitors which are demonstrated to be vulnerable to blurred images.
I find it hard to make a definitive evaluation for this work at this point and would like to take the authors' responses into account for my final recommendation. The main idea of the paper to generate adversarial examples for training via blurring is rather simple and thus the novelty of this work is somewhat minor. I think the quality of the paper also suffers from many statements in the text that draw too general and too bold conclusions at this point in my mind. The presentation overall is rather unpolished (see comments below). However, I find the empirical results itself quite strong and convincing and think they would make a relevant and significant contribution to the community. I do have questions left open though that need clarification:
(i) I don’t see why different techniques for blurring (SVD, DCT, GB) should lead to such different results as the approach remains conceptually similar. Do you have a reason/intuition why SVD gives the best results? Might SVD just be the easiest to tune method?
(ii) What do you think is the key reason that SVD-RND also appears to generalize too non-blurry OOD samples? Could you elaborate more on the two reasons you give in the paper? (1. RND performance on samples orthogonal to the data; 2. Discrimination between data and its low-rank projection)
(iii) The generation and tuning of multiple sets of blurred images (how many samples per set?) may get quite extensive for large datasets. Could you be specific on the computational cost?
(iv) Might the deep generative models (e.g. GPND) fail to detect blurred images due to insufficient model capacity of the decoder which results in blurry reconstructions? Have you varied the network capacity or latent space dimensionality of such models?
(v) What is the idea behind choosing the log effective rank in such an equidistant manner as proposed?
####################
*Additional Feedback*
*Positive Highlights*
1. SVD-RND shows strong OOD detection performance in OOD experiments on a variety of image dataset combinations (CIFAR-10, SVHN, TinyImageNet, LSUN, and CelebA).
2. The related work includes major works from all the related lines of research (Deep anomaly detection; OOD detection using side information; Adversarial examples/training)
3. Useful hyperparameter selection criterion based on effective rank if no OOD validation data is available.
*Ideas for Improvement* 
4. I think the tone of the paper would greatly benefit from not drawing too general conclusions and too bold implications. Keep statements precise and evidence-based. Declare hypotheses as such.
5. I would appreciate a plot showing samples before and after blurring in the appendix to see the most effective degree of blurring. Maybe also compare the different blurring baselines here to see differences.
6. Report std. devs. with your performance results to infer statistical significance.
7. Compare to the specific geometric transforms method as proposed in the paper [3] and not only using those transformations within your RND approach.
8. Add missing deep anomaly detection related work [6, 2, 5, 1].
9. Expand the sensitivity analysis in Figure 3 (left) over a greater range of K. Especially, I would like to also see K = 0 (unblurred, original images) as a sanity check which may only improve over RND due to ensembling over multiple randomly initialized networks.
10. Consider the one-vs-rest anomaly detection evaluation benchmarks from Ruff et al. [4] or Golan and El-Yaniv [3] to further infer the generalization performance of the proposed method.
11. I think the paper spends too much time on introducing previous work. Section 2 Related Work and Section 3 Background might be combined into one section.
*Minor comments*
12. In the abstract: “... VAE or RND are known to assign lower uncertainty to the OOD data than the target distribution.” is a bit strong. Rather “have been observed” etc. This is a working hypothesis in the community, but there is recent work (https://openreview.net/forum?id=Skg7VAEKDS) indicating (at least for VAEs) those are effects of poor model design.
13. In the abstract: “... efficient in test time ...” » “... efficient at test time ...”
14. In the abstract: “... in CelebA domain.” » “... on the CelebA dataset.” or just “... on CelebA.”
15. Section 1: “However, such models show underwhelming performance on detecting OOD, such as detecting SVHN from CIFAR-10. Specifically, generative models assign a higher likelihood to the OOD data than the training data.”. I think those are way too general conclusions at the moment. Rather something like “OOD detection performance of deep generative models has been called into question” and “have been observed to assign ...”.
16. Section 1: “Such results clearly support the degeneracy of deep OOD detection schemes”. Again, I find this way too bold of a statement at this point in time.
17. Section 2: “... a recently proposed paper ...” » “... a recent paper ...”
18. Section 2: “... outlier data independent of OOD data.”. What would outlier data not being out-of- distribution be?
19. Section 2: “Golan et al. (2018) design geometrically transformed data and regularized the classifier ...”. Not regularized. They trained a classifier on labels identified with these transformations.
20. Section 2: “..., resulting in OOD detection in each labeled data” » “..., resulting in OOD detection on labeled data”
21. A subsection title directly following a section title is bad style. A major section should be introduced with a few sentences on what this section is about.
22. Figure 2, right plot: These loss curves are rather strange... Increasing, then sharply decreasing again. Is there a drop in learning rate at epoch 80?
23. Many axis labels are too small and hard to read.
24. In Section 3.3.: “.. in Section 7.2.” » “.. in Section 4.2.”?
25. The definition of the log effective rank in Eq.~2 is weird. It's the entropy over the singular value distribution, i.e.~$H\left(\sigma_1 / \sum_j \sigma_j, \ldots, \sigma_N / \sum_j \sigma_j \right)$. Also, the parameters/notation involved are not introduced.
26. Make clear you apply SVD on single images. The paper alters formulations between data matrix and images...
27. Several instances where citet is used instead of citep.
28. In Table 1: Separate the target column from the three OOD columns more clearly. (e.g. vertical separator, center OOD, target in bold, etc.)
29. In Section 5.1: “area of the region under the ... curve” » “area under the ... curve”.
30. Figure 3: Better explain the plots. Are the three curves the respective target classes? What is the OOD set?
31. Figure 4: Maybe use subfigures with individual titles/captions.
32. Section 6: “For evidence, we fine-tune the classifier ...” » “For evidence, we fine-tune a classifier ...”
33. Plots and Figures are somewhat scattered and not referenced chronologically.
34. Introduce the effective rank at the point where it is used (Section 6.2). Somewhat unclear why to introduce this in Section 3 already.
35. Visually speaking, I find the RND examples in Figure 4 actually more anomalous than the top-anomalous SVD-RND samples (flashy colors, weird angles, borders, high contrast, ...)
####################
*References*
[1] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019.
[2] H. Choi, E. Jang, and A. A. Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018.
[3] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.
[4] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
[5] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.
[6] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Proceedings International Conference on Information Processing in Medical Imaging, pages 146–157. Springer, 2017.