Paper:
Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification
This paper studies the problem of deriving fast and accurate classification algorithms with uncertainty quantification. Gaussian process classification provides a principled approach, but the corresponding computational burden is hardly sustainable in large-scale problems and devising efficient alternatives is a challenge. In this work, we investigate if and how Gaussian process regression directly applied to classification labels can be used to tackle this question. While in this case training is remarkably faster, predictions need to be calibrated for classification and uncertainty estimation. To this aim, we propose a novel regression approach where the labels are obtained through the interpretation of classification labels as the coefficients of a degenerate Dirichlet distribution. Extensive experimental results show that the proposed approach provides essentially the same accuracy and uncertainty quantification as Gaussian process classification while requiring only a fraction of computational resources.

Review:
The authors of this paper introduce a novel approach to GP classification, called GPD. The authors use a GP to produce the parameters of a Dirichlet distribution, and use a categorical likelihood for multi-class classification problems. After applying a log-normal approximation to the Dirichlet distribution, inference for GPD is the same as exact-GP inference (i.e. does not require EP, Laplace approximation, etc.) The authors show that GPD has competitive accuracy, is well calibrated, and offers a speedup over existing GP-classificaiton methods.  Quality:  The method introduced by this paper is a clever probabilistic formulation of Bayesian classification. The method is well-grounded in Bayesian statistics and makes good comparisons to similar existing approaches. The one issue I have with this paper are the claims about the ``speedup'' afforded by this method (see detailed comments). The authors are not specific about the speedup (is it training time or inference time?) and I'm concerned they are missing an important baseline.  Clarity:  The paper is clearly written.  Originality:  To the best of my knowledge this is a novel approach to Bayesian classification.  Significance:  The idea is an interesting new approach to approximate inference for classification, though it does not perform better than other classification methods (in terms of accuracy). Nevertheless, it will be of interest to the Bayesian ML community.  Overall:  The proposed method is a novel approach to classification - and based on these merits I would be willing to accept the paper. However, the authors need to substantiate the speedup results with more detail and analysis if they wish to report this as a primary advantage.  Detailed comments:  My main concern with this paper is with regards to the ``speedup'' that the authors claim. There are many missing details.  The authors claim that their GPD method will be faster than GP classification because "GPC requires carrying out several matrix factorizations." (What matrix factorizations are the authors referring to?) The inducing-point GP-classification method in the results (SVGP) is asymptotically the same as the inducing-point method used for GPD (SGPR). The NMLL computations (used for training) and inference computations for SVGP and SGPR both require the Cholesky decomposition of an m x m matrix (where m is the number of inducing points). This is the primary computational bottleneck of both SVGP (for GP classification) and SGPR (for GPD). I'm assuming the authors claim that GPD is faster because GPD does not require learning variational parameters. The ``training'' of GPD - as far as I understand - only requires optimizing the hyperparameters, which presumably requires fewer optimization iterations. However, one key advantage of SVGP (for GP classification) is that it can be optimized with SGD. So even though SVGP requires more optimization iterations, each iteration will be fast since it only requires a minibatch of data. On the other hand, every optimization iteration of GPD will require the entire dataset. Therefore, there is not an obvious reason why GPD (and GP regression) should be that much faster than GP classification. The authors should elaborate with more detail in Sections 2 and 3 as to why GP-classification is slow.  While the authors do report some empirical speedups, the experiments are missing several important details. Firstly, what is the ``speedup'' measuring - training or inference? (I'm assuming that it is measuring training, but this must be clarified.) Secondly, was the author's GP-classification baseline trained with or without SGD? If it was trained without SGD, I would argue that the speed comparisons are unfair since stochastic optimization is key to making GP-classification with SVGP a fast method.  Small comments:  - 149: "For a classifier to be well-calibrated, it should accurately approximate fp." - This is not necessarily true. Assuming a balanced dataset without loss of generality, a latent function that outputs zero (in the case of CE) or 0.5 (in the case of least-squares classification) for all samples is well calibrated. - Figure 1: it is difficult to differentiate between the lines in the n=60 and n=80 plots - 241: do you mean the initial inducing points were chosen by k-means? SVGP and SGPR should optimize the inducing point locations.   ----------------------- Post author response:  While I would still vote to accept the paper, I will note that the author's response did not clarify my questions about the speedup. (Was it training or inference?) Additionally, the author's claim about speed still does not acknowledge many recent advances in scalable GP classification:  "Our claim in the paper about multiple matrix factorizations pertains to training times and it is specific to the case of standard implementations of EP and the Laplace approximation."  Though the authors make a statement about GPC speed based on the EP/Laplace approximations, yet they do not compare to these methods in their experiments! Regarding SGD-based GPC methods: there have been a number of recent works in scalable GPC methods that rely on SGD [1-3]. These papers report large empirical speedups when using SGD-based optimization over batch GD optimization. By not comparing against SGD-based methods, the authors are disregarding GPC advances from the past 4 years.  That being said, I still do believe that this method is novel, and I would vote for acceptance. I would encourage the authors to be much more rigorous about their speedup claims. There are some scenarios where a non-GPC-based approach would be clearly advantageous (e.g. in an active learning setting, when the kernel hyperparameters can be reused between data acquisition), and I would suggest the authors to focus on these specific scenarios rather than making broad claims about speed.  [1] Hensman, James, Alexander Matthews, and Zoubin Ghahramani. "Scalable Variational Gaussian Process Classification." Artificial Intelligence and Statistics. 2015. [2] Hernández-Lobato, Daniel, and José Miguel Hernández-Lobato. "Scalable Gaussian process classification via expectation propagation." Artificial Intelligence and Statistics. 2016. [3] Wilson, Andrew G., et al. "Stochastic variational deep kernel learning." Advances in Neural Information Processing Systems. 2016.