Gaussian Processes (GPs) are a generic modelling tool for supervised learning. While they have been successfully applied on large datasets, their use in safetycritical applications is hindered by the lack of good performance guarantees. To this end, we propose a method to learn GPs and their sparse approximations by directly optimizing a PAC-Bayesian bound on their generalization performance, instead of maximizing the marginal likelihood. Besides its theoretical appeal, we find in our evaluation that our learning method is robust and yields significantly better generalization guarantees than other common GP approaches on several regression benchmark datasets. Gaussian Processes (GPs) are a powerful modelling method due to their non-parametric nature [1] . Although GPs are probabilistic models and hence come equipped with an intrinsic measure of uncertainty, this uncertainty does not allow conclusions about their performance on previously unseen test data. For instance, one often observes overfitting if a large number of hyperparameters is adjusted using marginal likelihood optimization [2] . While a fully Bayesian approach, i.e. marginalizing out the hyperparameters, reduces this risk, it incurs a prohibitive runtime since the predictive distribution is no longer analytically tractable. Also, it does not entail out-of-the-box safety guarantees. In this work, we propose a novel training objective for GP models, which enables us to give rigorous and quantitatively good performance guarantees on future predictions. Such rigorous guarantees are developed within Statistical Learning Theory (e.g. [3] ). But as the classical uniform learning bounds are meaningless for expressive models like deep neural nets [4] (as e.g. the VC dimension exceeds the training size) and GPs or non-parametric methods in general, such guarantees cannot be employed for learning those models. Instead, common optimization schemes are (regularized) empirical risk minimization (ERM) [4, 3] , maximum likelihood (MLE) [1], or variational inference (VI) [5, 6] . On the other hand, better non-uniform learning guarantees have been developed within the PAC-Bayesian framework [7, 8, 9] (Sect. 2). They are specially adapted to probabilistic methods like GPs and can yield tight generalization bounds, as observed for GP classification [10], probabilistic SVMs [11, 12] , linear classifiers [13], or stochastic NNs [14]. Most previous works used PAC-Bayesian bounds merely for the final evaluation of the generalization performance, whereas learning by optimizing a PAC-Bayesian bound has been barely explored [13, 14] . This work, for the first time, explores the use of PAC-Bayesian bounds (a) for GP training and (b) in the regression setting. Specifically, we propose to learn full and sparse GP predictors Q directly by minimizing a PAC-Bayesian upper bound B(Q) from Eq. (5) on the true future risk R(Q) of the predictor, as a * https://www.bosch-ai.com
REF develop a new sGP algorithm by optimizing a PAC-Bayes bound.
Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Bounds
Advances in Neural Information Processing Systems 31 (Proceedings of the NeurIPS Conference 2018), https://papers.nips.cc/paper/7594-learning-gaussian-processes-by-minimizing-pac-bayesian-generalization-bounds
Computer Science Mathematics