Marginalised Spectral Mixture Kernels with Nested Sampling
Abstract
Gaussian Process (GPs) models are a rich distribution over functions with inductive biases controlled by a kernel function.
Learning occurs through optimisation of the kernel hyperparameters using the marginal likelihood as the objective (ML-II).
This work analyses the beneﬁts of marginalising kernel hyperparameters using nested sampling (NS), a technique well-suited to sample from complex, multi-modal distributions.
We benchmark against Hamiltonian Monte Carlo (HMC) on time-series regression tasks and ﬁnd that a principled approach to quantifying hyperparameter uncertainty substantially improves the quality of prediction intervals.
1. Introduction
Gaussian processes (GPs) represent a powerful non-parametric and probabilistic framework for performing regression and classiﬁcation.
GPs are typically trained using ML-II, a procedure which denotes the maximisation of the GP marginal likelihood.
ML-II yields point estimates of hyperparameters for the user chosen covariance / kernel function.
The posterior predictive distribution is then evaluated at the ML-II point estimates.
The form of the kernel function inﬂuences the geometry of the marginal likelihood surface.
For instance, periodic kernels give rise to multiple local minima as functions with diﬀerent periodicities can be compatible with the data.
Expressive kernels which are derived by adding/multiplying together primitive kernels to encode diﬀerent types of inductive biases typically have many hyperparameters, exacerbating the local minima problem.
Concretely, the classical approach to training suﬀers from two issues we try and analyse in this work: 1) Point estimate hyperparameters give overconﬁdent predictions by not accounting for hyperparameter uncertainty; 2) Non-convexity of the marginal likelihood surface can lead to poor estimates located at local minima.
Further, the presence of multiple modes aﬀects the interpretability of kernel hyperparameters.
The spectral mixture (SM) kernel proposed in Wilson and Adams (2013) is an expressive class of kernels derived from the spectral density reparameterisation of the kernel using Bochner’s Therorem
Bochner (1959).
The SM kernel has prior support over all stationary kernels which means it can recover sophisticated structure provided suﬃcient spectral components are used.
Several previous works Kom Samo and Roberts (2015); Remes et al.
(2017, 2018) have attempted to further enhance the ﬂexibility of spectral mixture kernels, such as the introduction of a time-dependent spectrum.
However, we postulate that the key limitation in the SM kernel’s performance lies not its stationarity or expressivity, but in the optimisation procedure.
We show that the form of the SM kernel gives rise to multiple symmetric modes in the marginal likelihood making optimisation extremely unstable and prone to overﬁtting.
Our two main contributions are:
• Highlight the failure modes of ML-II training in the context of spectral mixture kernels.
We provide insights into the eﬀectiveness of ML-II training in weak and strong data regimes.
• Propose the relevance of Nested Sampling (NS) as a way to eﬀectively sample from the hyperparameter posterior (see also Faria et al., 2016; Aksulu et al., 2020), particularly in the presence of multiple modes.
2. Background 2.1. Marginalised Gaussian Processes
Given some input-output pairs ( X, y )
= { x i , y i } Ni =1 where y i are noisy realizations of latent function values f
i with Gaussian noise, y i
=
f i + (cid:15) i , (cid:15) i ∈ N (0 , σ 2 n ) , we seek to infer some as-yet unseen values y ∗ .
Let k θ ( x i , x j ) denote a positive deﬁnite kernel function parameterized with hyperparameters θ .
Following the prescription of Lalchand and Rasmussen (2020), the marginalised GP framework is given by
[Equation]
where K θ denotes the N × N covariance matrix, ( K θ )
i,j = k θ ( x i , x j ) .
The predictive distribution for unknown test inputs X (cid:63) integrates over the joint posterior 1 ,
[Equation]
where f has been integrated out analytically and θ is dealt with numerically, yielding the ﬁnal mixture of Gaussians form of the posterior predictive in marginalised framework.
Throughout this work we shall adopt a Gaussian likelihood, hence the only intractable integrand we need to consider is the hyperparameter posterior p ( θ | y )
∝ p ( y | θ ) p
( θ ) .
2.2. Spectral Mixture Kernels
Deﬁnition 1 (SM kernel)
A spectral mixture kernel is a positive deﬁnite stationary kernel given by,
[Equation]
where τ d , µ ( d ) q , σ ( d ) q are the d th components of the D dimensional vectors τ , µ q and σ q respectively.
The vector of kernel hyperparameters θ = { w q , µ q , σ q } Qq =1 is typically unknown, we account for this uncertainty by treating them as random variables with priors and sampling from their intractable posterior.
The SM kernel is derived by leveraging Bochner’s Therorem (Bochner, 1959) which provides an explicit relationship between the spectral density S ( ν ) and the kernel function k as the integral (cid:82)
S ( ν ) e 2 πiν T τ dν .
If the S ( ν ) is modelled as a weighted mixture of Q Gaussian pairs,
[Equation]
(4) and plugged into integral mentioned above, yields the SM kernel in eq.
Here the weight w q speciﬁes the variance contributed by the q th component while G ( ν, µ, σ ) denotes a Gaussian function with mean µ q and standard deviation σ q .
To avoid confusion with other standard deviations, and convey its physical signiﬁcance, we shall refer to σ q as the bandwidth .
Fig.
1 shows a two-dimensional slice of the 7-dimensional 3 marginal likelihood surface for a 2-component SM kernel, given two datasets of diﬀerent size.
One of the striking features of these surfaces lies in their symmetry due to the kernel’s invariance to the ordering of its components.
The marginal likelihood of a SM kernel with Q spectral components possesses Q !
identical regions of parameter space.
A naive attempt to explore the full posterior distribution of a spectral mixture kernel would try to quantify probability mass across these degenerate regions, a much more computationally intensive task than is necessary.
One solution is to only sample from one region, and ignore its symmetric counterparts.
To achieve this, we can adopt an approach known as forced identiﬁability Buscicchio et al.
(2019) to ensure that the components are deﬁned in sequential order with respect to their frequency µ .
2.3. Nested Sampling
The nested sampling algorithm was developed by Skilling (2004)
(see also Skilling et al.
as a means of estimating the model evidence Z = (cid:82) ψ ( θ ) dπ ( θ ) , where ψ denotes the likelihood 4 , and π ( θ ) the prior.
This may be recast as a one-dimensional integral over the unit interval, Z = (cid:82) 1 0 ψ ( X ) dX , irrespective of the dimensionality of θ .
Here X is the quantile function associated with the likelihood: it describes the volume of the prior lying below the likelihood value ψ .
The extreme values of the integrand, ψ ( X =0) and ψ ( X =1) , therefore correspond to the minimum and maximum likelihood values found under the support of the prior π ( θ ) .
Sampling proceeds in accordance with Algorithm(appendix 1).
While the sequence of samples provides an estimate of Z , an invaluable quantity in the context of Bayesian model selection, they also represent importance weighted samples of the posterior.
We employ the PolyChord algorithm (Handley et al., 2015; Handley et al., 2015), which performs slice sampling (Neal, 2003) at each iteration.
Unless otherwise stated, we use 100 live points, which are bounded in a set of ellipsoids (Feroz et al., 2009).
These bounding surfaces allow the macroscopic structure of the likelihood contours to be traced, enabling a much more eﬃcient sampling process.
This approach has proven particularly adept at navigating multi-modal likelihood surfaces (Allison and Dunkley, 2014).
These attractive properties have motivated numerous scientiﬁc applications, including the detection of gravitational waves (Veitch et al., 2015), the categorisation of cosmic rays (Cholis et al., 2015) and the imaging of a supermassive black hole (Akiyama et al., 2019).
The algorithm described in 1 is implemented as part of the dynesty package (Speagle, 2020).
We use the ‘rslice’ sampling option, along with the default number of ﬁve slices and adopt the default criterion for convergence.
This is deﬁned as the point at which the estimated posterior mass contained within the set of live points falls below 1% of the total
3. Experiments 3.1. Ground Truth Recovery
In the two left hand panels of Fig.2 we summarise the performance of ML-II, HMC and nested sampling inference in their ability to recover the true setting of the hyperparameters for a synthetic dataset with 100 datapoints, a noise amplitude of σ n = 0 .
1 and a signal-to-noise ratio of ≈ 3.2 on a ﬁxed domain [-1,1].
The magenta star (cid:63) denotes the true value and the red cross × denotes
the
ML-II estimate.
The HMC and NS sampling schemes are both better at recovering the ground truth hyperparameters than the point estimates.
Further, the nested sampling scheme is able to obtain samples from both modes inherent in the marginal likelihood.
3.2. Time series benchmarks
We evaluate the predictive performance of marginalised GPs with the SM kernel against thirteen time series, as used in Lloyd et al.
The time series are of variable length, with up to 1 ,data points in each.
Our ﬁducial kernel has seven components ( Q = 7) , yielding a 22-dimensional hyperparameter space to explore.
For reference, we also include results from the Neural Kernel Network (Sun et al., 2018) where a ﬂexible kernel is learnt incrementally through a weighted composition of primitive base kernels.
This is trained with the Adam optimiser for 100,000 iterations and a learning rate of 10 − 3 .
In Table 3.1 we report the negative log of predictive density (NLPD) across test data.
The evaluation was conducted with a 60/40 train/test split.
The quoted uncertainties are estimated by repeating each set of experiment three times with a diﬀerent random seed.
We ﬁnd that the spectral mixture kernel exhibits signiﬁcant performance gains when using a sampling-based inference method compared to the conventional ML-II approach.
HMC and NS oﬀer similar performance levels, though NS carries an advantage of faster evaluation times and an estimate of the model [Table] evidence.
More detailed results for various conﬁgurations of the nested sampler can be found in Table 3.
Fig.
3 depicts training times in wall clock seconds for our time-series experiments, all of which were run on a single Nvidia GTX1070 GPU.
For HMC this corresponds to a single chain with 500 warm-up iterations and 500 samples.
Nested sampling was performed with 100 live points, and the sampling continued until the convergence criterion was met.
Across these benchmarks, HMC was found to be slower by approximately a factor of two compared to nested sampling.
Both methods could enjoy signiﬁcant gains in speed with schemes such as those proposed in Loper et al.
3.2.1. Why do wider predictive intervals give better NLPD?
The right hand panel of Fig.
2 is a zoomed in plot of the ‘RADIO’ predictions.
It helps convey exactly why the NLPDs corresponding to the marginalised GP schemes are better.
The plot shows test predictions and respective 95% intervals.
The blue band corresponds to prediction intervals under the marginalised HMC scheme; the 95 % quantiles are estimated empirically by sampling from the Gaussian mixture distribution at each test input X ∗ .
The far right panel shows the mixture density for a single test point, the predictive mixture form places higher probability mass on the true test point than the narrower ML-II intervals.
3.3. 2d Pattern Extrapolation
To provide a demonstration of how the inference methods are readily adapted to higher dimensional kernels, we revisit the challenge presented in Sun et al [26].
The two-dimensional ground truth function is given by y =
(cos 2 x 1 × cos 2 x 2 ) (cid:112) | x 1 x 2 | .
We train with just 50 points (as opposed to 100 in the original reference), which are chosen at random across [ − 6 , +6] in the xy − domain.
The test points are deﬁned on a 20 × 20 grid.
In Figure 4 we visualise the mean values of the posterior predictive distribution from three diﬀerent inference methods: ML-II, HMC and nested sampling (NS).
Visually, the reconstruction under the marginalised GP schemes (HMC / NS) appears to be superior to ML-II.
Further, the 95 % conﬁdence intervals (not visualised) diﬀer markedly, as is evident from the NLPD values.
These are given by 216, 2.56, and 2.62 for ML-II, HMC and NS respectively ( lower is better).
For reference we also trained the neural kernel network (Sun et al [26]), which achieved an NLPD of 3.8.
This experiment marks a signiﬁcant increase in the dimensionality of our parameters space, since in two dimensions each spectral component has ﬁve degrees of freedom.
Yet the marginalised Gaussian processes comfortably outperform competing methods.
ML-II was trained with Adam (learning rate=0.05) with 10 restarts and 2,000 iterations.
4. Discussion
While a pathological marginal likelihood geometry can pose problems for both gradient based optimisation and sampling; sampling schemes are able to quantify them better if the practical diﬃculties of deploying them (like chains stuck in bad optima) are overcome.
The nested sampling scheme, which does not require gradient information, provides remarkably good approximations to gold-standard HMC samples at a fraction of the cost of running HMC.
Further, it is crucial to ask if marginalisation increases expressivity and representational power by accounting for a more diverse range of models than point estimates?
A marginalised GP with a standard SM kernel not only improved upon its conventional counterpart, but also surpassed the performance of the
Neural Kernel Network model (Sun et al., 2018) in the time series tasks, despite possessing far fewer hyperparameters.
This indicates there is merit in the question.
Appendix A. Further Experimental Results A.1. Synthetic data
As a ﬁrst demonstration of how the diﬀerent methods perform, we draw four samples from a twocomponent SM kernel with known weights, frequencies and bandwidths.
For each latent function sample we construct noisy training data across three noise levels [0 . 01 , 0 .
5] and three training sizes [10 , 30 , 100] , on a ﬁxed input domain [ − 1 , 1] .
In this way we seek to quantify the quality of predictions and prediction uncertainty under weakly identiﬁed regimes characterised by very few data points in a ﬁxed domain to strongly identiﬁed regimes characterized by a dense distribution of data in a ﬁxed domain.
Further, the intrinsic noise level σ 2 n of the data can also impact inference in weakly and strongly identiﬁed data regimes.
In order to analyse the impact of σ 2 n and training size we calculate the average performance across each of three diﬀerent noise levels for each training set size.
We train under each of the candidate inference methods (ML-II, HMC, Nested) for each of the 9 ×data sets created and report prediction performance in terms of the average negative log predictive density (NLPD) for test data in Figure 5.
ML-II uses ﬁve random restarts with a initialisation protocol tied to the training data.
Following protocols from Wilson and Adams (2013), the SM weights ( w i ) were initialised to the standard deviation of the targets y scaled by the number of components ( Q = 2) .
The SM bandwidths ( σ i ) were initialised to points randomly drawn from a truncated Gaussian |N (0 , max d ( x, x (cid:48) ) 2 ) | where max d ( x, x (cid:48) ) is the maximum distance between two training points and mean frequencies ( µ i ) were drawn from Unif (0 , ν N ) to bound against degenerate frequencies.
HMC used LogNormal (0 , 2) priors for all the hyperparameters.
The ML-II experiments used gpytorch while the HMC experiments used the NUTS sampler in pymc3 .
In Figure 5 we notice that ML-II struggles with small training data sets and catastrophically underestimates the noise level.
The two sampling methods, HMC and NS, perform comparably well, though it is notable that a single NS run correctly identiﬁes both modes in the likelihood surface, while a single HMC chain can only discover one.
Fig.6 shows the test mean squared error across the three inference schemes.
The sampling schemes largely dominate the ML-II method when the hyperparameters are well identiﬁed ( n _ train = 100 ).
The data generating conﬁgurations are the same as the ones described in the main paper.
A.2. Ground Truth Recovery
In this subsection we summarise the performance of ML-II, HMC and Nested sampling inference in recovering the true setting of the hyperparameters under two diﬀerent noise settings, for a ﬁxed training size n _ train = 100 on a ﬁxed domain [-1,1].
The top row of panels in Fig.7 indicate a low noise setting σ n = 0 .
and the bottom row indicates a higher noise setting of σ n = 0 .
The magenta star (cid:63) denotes the true value and the red cross × denotes the ML-II estimate.
The sampling schemes
HMC and Nested are both better at recovering the target than the point estimate.
While ML-II estimates the noise-level to a high precision when the noise is low (top-row, σ n = 0 . 01 ), it does not fare so well when noise level is raised by an order of magnitude (bottom row, σ n = 0 . 1 ).
In this case, the estimate of the intrinsic noise level is oﬀ by several orders of magnitude.
The sampling schemes prove to be far more robust in recovering the frequencies, bandwidths and noise level, especially when operating in the low signal-to-noise regime.
The true hyperparameters are [ µ 1 , µ 2 ] =
[3 . 14 , 12 . 56] and [ σ 1 , σ 2 ] =
For frequencies and bandwidths we note the symmetry i.e. the estimates can converge on [ µ 1 , µ 2 ] or [ µ 2 , µ 1 ] , and that the nested sampling algorithm successfully identiﬁes both.
A.3. Time Series
In table 2 we present further results from the benchmark time series experiments.
Instead of making full use of the data, we consider only the ﬁrst 100 points as training data, followed by testing with the subsequent 30 points.
As with the results from the full training set, signiﬁcant performance gains are found when marginalising the hyperparameters of the spectral mixture kernels.
However in this case, the nested sampling algorithm doesn’t oﬀer a performance advantage over HMC.
We speculate this may be due to the simpler likelihood surface associated with the smaller set of training data.
Fewer modes in the surface would facilitate exploration via HMC.
Table 3 shows how the conﬁguration of the nested sampler impacts its performance.
The ﬁducial conﬁguration is shown in the ﬁrst column, and the uncertainty in the derived mean NLPD is obtained by repeating this experiment for several diﬀerent random seeds.
The three subsequent columns correspond to changes in the number of live points, combining multiple runs, and using a linear prior to describe the frequency parameter µ .
The only change which makes a statistically signiﬁcant alteration to the performance is found to be the choice of the prior.
[Table]
[Table]
Appendix B. Candidate Inference Methods B.1. ML-II
A Gaussian noise setting facilitates an analytically tractable marginal likelihood,
[Equation]
using the deﬁnitions from (1).
ML-II estimates are obtained by maximizing L ( θ ) = log p ( y | θ ) over the kernel hyperparameters, θ (cid:63)
=
argmax θ L ( θ ) .
B.2. Hamiltonian Monte Carlo (HMC)
Hamiltonian Monte Carlo (Duane et al., 1987) is a fundamental tool for inference in intractable
Bayesian models.
HMC relies on using gradient information to suppress random walk behaviour inherent to samplers like Metropolis-Hastings and its variants.
The sampler operates in an augmented space ( θ , p ) of position variables and momentum variables where the position variables are the variable of interest.
The momentum variables are sampled from an independent N (0 , 1) and can safely be marginalised out from the joint samples to obtain just the samples of interest.
New proposals are generated by simulating Hamiltonian dynamics in the joint 2D phase space ( θ i , p i )
→ ( θ i +1 , p i +1 )
(where D is the dimensionality of θ ) for preset number of steps called the path length ( L ) .
Hamiltonian dynamics are simulated using the leap-frog symplectic integrator which relies on a step size ( (cid:15) ) .
Further, each iteration requires the gradients of the log marginal likelihood w.r.t the hyperparameters.
In the case of a GP, this means each iteration of the leap-frog integrator requires the inversion of
K θ , L times to simulate one proposal.
Refer to Neal et al.
(2011) for a detailed tutorial.
In the experiments we use a self-tuning variant of HMC called the No-U-Turn Sampler (NUTS) (Hoﬀman and Gelman, 2014) where the path length is adapted for every iteration.
NUTS is frequently shown to work as well as a hand-tuned HMC; hence in this way we avoid the compute overhead in tuning for good values of the step-size ( (cid:15) ) and path length ( L ) .
We use the version of NUTS available in the python package pymc3 .
B.3. Nested Sampling Algorithm Appendix C. Hyperpriors
In this section we outline a suitable set of priors for the three fundamental parameters of a single spectral component.
As deﬁned in (4), these are the mean frequency µ , bandwidth σ , and weight w .
For most of these hyperparameters, we ﬁnd it is suﬃcient to impose a weakly informative prior of the form { σ, w, σ 2 n } ∼ LogNormal (0 , 2) .
However the spectral component’s characteristic frequency µ deserves closer attention.
Two properties of the data strongly inﬂuence our perspective on which frequencies we can expect to observe: the fundamental frequency and the highest observable frequency.
The fundamental frequency ν F is the lowest frequency observable within the data, and is given by the inverse of the interval spanned by the observed x locations. Meanwhile the maximum frequency ν N represents the highest observable frequency. For gridded data, this is naturally determined by the Nyquist frequency, which is half the sampling frequency. It is crucial to bear in mind that the spectral density we wish to model is that of the underlying process, and not the spectral density of the data.
These two quantities are often very diﬀerent, due to the limited scope of the observations.
For example, the change in stock prices over the period of a single day cannot exhibit frequencies above 10 − 6 Hz.
Yet the process will have received contributions from long term ﬂuctuations, such as those due to macroeconomic factors, whose periods can span many years.
If we make no assumption regarding the relationship between the process we wish to model and the ﬁnite range over which it is observed, then a priori, some as-yet undiscovered frequency within the process ought to be considered equally likely to lie above or below the fundamental frequency.
Furthermore, given the large initial uncertainty in frequency µ , it is appropriate to adopt a prior which spans many orders of magnitude.
Initialisation: Draw n L ‘live’ points { θ } n L i =1 from the prior θ i ∼ π ( θ ) , set model evidence
Z = 0 .
while stopping criterion is unmet do
• Compute ψ i = min( ψ ( θ 1 ) , . . .
ψ ( θ N )) , the lowest marginal likelihood from the current set of live points.
• Sample a new live point θ (cid:48) subject to ψ ( θ (cid:48) ) >
ψ i • Remove the point θ i corresponding to the lowest marginal likelihood ψ i , moving it to a set of ‘saved’ points • Assign estimated prior mass at this step ˆ X i =
e − iN • Assign a weight for the saved point, V i
=
ˆ X i − 1 − ˆ X i • Accumulate evidence, Z = Z
+ ψ i V i •
Evaluate stopping criterion, if triggered then break;
end
Add ﬁnal n L points to the ‘saved’ list of K samples:
• Weight of each of these ﬁnal points is assigned to p i
=
ˆ X K /n
L ∀ i = K, . . . , n L + K // final slab of enclosed prior mass • Final evidence is given by, Z = (cid:80) n L
+ K i =
• Importance weights for each sample are given by, p i =
ψ i V i / Z return set of samples { θ i } n L
+ K i =
, along with importance weights { p
K and evidence estimate Z .
Towards very low frequencies, µ (cid:28) ν F , a sinusoid contributes very little variance to the observations - an annual modulation makes a tiny contribution to a minute’s worth of data.
As we consider frequencies much lower than the fundamental frequency, it therefore becomes less likely that we will detect their contributions.
We model this suppressed probability of observation with a broad
Gaussian in log frequency for the regime µ < ν F .
Meanwhile, at frequencies above the Nyquist frequency, µ > ν F , we encounter a degenerate behaviour: these sinusoids are indistinguishable from their counterparts at lower frequencies: they are said to be aliases of each other.
As a result of this aliasing behaviour, the likelihood surface is littered with degenerate modes with identical likelihood values.
From a computational perspective, it is advantageous to restrict our frequency parameter to a much narrower range than is permitted by our prior, while maintaining the same probability mass.
As illustrated in the supplementary, mapping these higher frequencies down to their corresponding alias at µ < ν N yields a uniform prior on µ .
[Equation]
C.1. Spectral Priors
Figure 8 shows the parameter space for the frequency and bandwidth of a single spectral component.
The likelihood surfaces adjacent to any of the dashed lines are mirror images of each other.
It is therefore preferable to avoid exploring multiple copies of these regions when performing nested sampling, as it will attempt to locate the duplicate modes, dispersing the live points.
To give a clearer picture of how the parameters of the spectral mixture kernel are inferred via nested sampling, we take as a example the radio experiment.
The posterior distribution in this case can be seen in Figure 9.
The corresponding joint posterior distribution of the 22 hyperparameters are displayed in Fig.10.
We make use of the dynesty package in order to perform nested sampling.
The full set of conﬁguration parameters used can be found in Table 4.
-up in dynesty, with the most signiﬁcant changes being in the sampling method, and a reduction in the number of live points.
[Table]