  We examine the general problem of inter-domain Gaussian Processes (GPs):
problems where the GP realization and the noisy observations of that
realization lie on different domains. When the mapping between those domains is
linear, such as integration or differentiation, inference is still closed form.
However, many of the scaling and approximation techniques that our community
has developed do not apply to this setting. In this work, we introduce the
hierarchical inducing point GP (HIP-GP), a scalable inter-domain GP inference
method that enables us to improve the approximation accuracy by increasing the
number of inducing points to the millions. HIP-GP, which relies on inducing
points with grid structure and a stationary kernel assumption, is suitable for
low-dimensional problems. In developing HIP-GP, we introduce (1) a fast
whitening strategy, and (2) a novel preconditioner for conjugate gradients
which can be helpful in general GP settings.
