Paper:
Most unsupervised neural networks training methods concern generative models, deep clustering, pretraining or some form of representation learning. We rather deal in this work with unsupervised training of the final classification stage of a standard deep learning stack, with a focus on two types of methods: unsupervisedsupervised risk approximations and one-class models. We derive a new analytical solution for the former and identify and analyze its similarity with the latter. We apply and validate the proposed approach on multiple experimental conditions, in particular on four challenging recent Natural Language Processing tasks as well as on an anomaly detection task, where it improves over state-of-the-art models.

Review:
UPDATE:
I acknowledge that I’ve read the author responses as well as other reviews. 
After reading, I would keep my rating at 3 (Weak Reject), since the key reasons for my rating still hold.
####################
This work makes a connection between recently introduced one-class neural networks [8, 4] and the unsupervised approximation of the binary classifier risk under the hinge loss [1]. An explicit expression of this risk approximation is derived for the case that the prior class probabilities are known and that the class-conditional distributions of classification scores are Gaussian. This solution is then used to formulate an end-to-end differentiable loss for unsupervised binary classification which is combined with a posterior class probability regularizer to avoid trivial solutions. Finally, the paper presents an experimental evaluation on synthetic data, the Wisconsin Breast Cancer dataset, four NLP tasks, as well as on the anomaly detection task on MNIST where the proposed method slightly outperforms the two existing one-class networks [8, 4].
I think this paper makes an interesting, original connection between unsupervised-supervised risk estimation and one-class neural networks which provides a principled motivation for existing methods [8, 4] and also hints to potential flaws in their formulations, namely that OC-NN [4] and soft-boundary Deep SVDD [8] make no use of positive samples during learning (as illustrated in Figure 2). The paper is not yet ready for acceptance in my opinion, however, due to the following key reasons: 
(i) The experimental evaluation is not convincing and not sufficient to assess the significance of results; 
(ii) Though making this connection is interesting, the theoretical derivations presented in the paper are rather straightforward.
(i) I think the experimental evaluation is the weakest part of the paper at the moment which I find not convincing due to the lack of competitors, the use of rather simple datasets, and missing experimental details. The synthetic experiment only serves as a sanity check not giving any additional insights. As the proposed unsupervised method approximates the risk of a supervised binary classifier, I agree that it makes sense to compare to the supervised “gold standard” on binary classification tasks (Wisconsin and NLP sentiment tasks) to infer the unsupervised-supervised performance gap. However, there is no comparison to other unsupervised competitors (OC-SVM, GMM, Deep SVDD, OC-NN, etc.) to put the performance of the proposed method in these experiments into perspective (only K-Means does not establish a strong baseline). Those classification tasks further are not really a convincing use case in my mind since labels here are usually available. In contrast, I find anomaly detection to be an important application of this method, but an evaluation solely on MNIST that also lacks recent deep competitors [6] is not sufficient to assess the significance of the presented results. Moreover, from the text it seems that only the hyperparameters of the proposed method are tuned on some validation set that includes positive as well as negative samples which would be an unfair advantage and might explain the slight edge in performance. Finally, many experimental details are not reported: (ia) Are the networks used randomly initialized or pretrained? (ib) How are prior class probabilities set? (ic) What are the batch sizes (relevant for quantile estimation) (id) What score are the hyperparameters tuned on? (ie) Are negative samples in the validation set from all the anomaly classes?
(ii) The technical derivations in the paper (and the appendix) are correct but rather straightforward. The theoretical heavy-lifting is from Balasubramanian et al. [1] and this paper presents an explicit solution for the risk approximation under the assumption that the prior class probabilities are known and the class-conditional score distributions are Gaussian. I do not want to discount that making this connection and loss derivation may lead to significant results (which is left to be demonstrated experimentally), but I find the current theoretical contribution on its own not sufficient. Parts of the theoretical Section 3 could also be greatly cut in my opinion (no need to define a quantile or sample mean and variance etc.). Finally, one key property of the loss expressed in the paper is its differentiability and use with autograd, but this does not hold after adding the posterior regularization term which is based on the empirical p0-quantile, correct? (the gradient of the quantile is zero almost everywhere due to the argmin)
Apart from the two key points above, the presentation of the paper is unpolished (nested lists in the main text, etc.) and major deep anomaly detection related work [10, 6, 5, 7, 3] is missing.
####################
*Additional Feedback*
*Positive Highlights*
1. The paper makes an interesting connection between unsupervised-supervised risk approximation [1] and recently introduced one-class neural networks [8, 4] that provides a principled motivation and points to potential flaws in existing formulations.
2. I think the question how to learn neural classifiers in an unsupervised manner is original and interesting.
3. The technical derivations in the paper are rigorous and correct.
*Ideas for Improvement*
4. Make a comparison to state-of-the-art unsupervised deep anomaly detection (AD) methods.
5. Run AD experiments on more complex datasets like Fashion-MNIST, CIFAR-10, and the recently introduced MVTec [2].
6. Include major deep AD works [10, 6, 5, 7, 3] into the related work.
7. Compress Sections 1–3 (fewer lists; no need to give definitions of a quantile, sample mean and variance; etc.)
8. Motivate the non-AD experiments. Currently these appear rather constructed artificially.
9. In the NLP tasks, make a comparison to text-specific one-class classifiers [9].
10. Add a sensitivity analysis w.r.t. the prior class probability p0 to infer robustness to this parameter that seems crucial.
11. Provide guidance how to select p0 in a particular application.
12. Consistently report performance metrics with standard deviations in your tables to allow to infer statistical
significance.
13. What to do if there are no negative, but only normal samples as in fully unsupervised AD? Nevertheless make an assumption on the class prior?
*Minor comments*
14. Unordered lists should be used sparsely in a main text, stylistically speaking. Avoid nesting as in the introduction.
15. Section 2.1, first sentence (and elsewhere): “Let be given ...” is grammatically wrong. Correct would be either “Let f be a binary linear classifier ...” or “Given a binary linear classifier f ...”.
16. In Section 2.1, index the classifier $f$ with parameter $\theta$, i.e. $f_\theta$. Otherwise the risk optimization parameter $\theta$ does not even appear on the right hand side of the risk Eq. (1).
17. Combine Eqs. (1) and (2) into one equation.
18. Consistently enumerate equations throughout the paper or do not enumerate at all.
19. In Eq. (3) index $i$ misses in the sum.
20. A subsection title following a section title directly is bad style. A new major section should at least be introduced with a few sentences on what this section is about.
21. Mention that $erf$ is the Gaussian error function.
22. Center the equation in Section 3.2.
23. Plots is Figure 3 are rather poorly formatted: use thicker lines and more distinctive colors; place the legend legibly.
24. Show the average with confidence intervals over the 10 runs in Figure 3.
25. Put results, as in Section 4.2 on the Wisconsin Breast Cancer dataset rather in a table.
####################
*References*
[1] K. Balasubramanian, P. Donmez, and G. Lebanon. Unsupervised supervised learning ii: Margin-based classification without labels. Journal of Machine Learning Research, 12(Nov):3119–3145, 2011.
[2] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.
[3] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019.
[4] R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018.
[5] H. Choi, E. Jang, and A. A. Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018.
[6] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.
[7] D. Hendrycks, M. Mazeika, and T. G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
[8] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
[9] L. Ruff, Y. Zemlyanskiy, R. Vandermeulen, T. Schnake, and M. Kloft. Self-attentive, multi-context one-class classification for unsupervised anomaly detection on text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4061–4071, 2019.
[10] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Proceedings International Conference on Information Processing in Medical Imaging, pages 146–157. Springer, 2017.