  Gaussian processes (GPs) are non-parametric, flexible, models that work well
in many tasks. Combining GPs with deep learning methods via deep kernel
learning is especially compelling due to the strong expressive power induced by
the network. However, inference in GPs, whether with or without deep kernel
learning, can be computationally challenging on large datasets. Here, we
propose GP-Tree, a novel method for multi-class classification with Gaussian
processes and deep kernel learning. We develop a tree-based hierarchical model
in which each internal node of the tree fits a GP to the data using the
Polya-Gamma augmentation scheme. As a result, our method scales well with both
the number of classes and data size. We demonstrate our method effectiveness
against other Gaussian process training baselines, and we show how our general
GP approach is easily applied to incremental few-shot learning and reaches
state-of-the-art performance.
