Generalized Bayesian Quadrature with Spectral Kernels
Abstract
Bayesian probabilistic integration, or Bayesian quadrature (BQ), has arisen as a popular means of numerical integral estimation with quantified uncertainty for problems where computational cost limits data availability.
BQ leverages flexible
Gaussian processes (GPs) to model an integrand which can be subsequently analytically integrated through properties of Gaussian distributions.
However, BQ is inherently limited by the fact that the method relies on the use of a strict set of kernels for use in the GP model of the integrand, reducing the flexibility of the method in modeling varied integrand types.
In this paper, we present spectral
Bayesian quadrature, a form of Bayesian quadrature that allows for the use of any shift-invariant kernel in the integrand GP model while still maintaining the analytical tractability of the integral posterior, increasing the flexibility of BQ methods to address varied problem settings.
Additionally our method enables integration with respect to a uniform expectation, effectively computing definite integrals of challenging integrands.
We derive the theory and error bounds for this model, as well as demonstrate GBQ’s improved accuracy, flexibility, and data efficiency, compared to traditional
BQ and other numerical integration methods, on a variety of quadrature problems.
INTRODUCTION
Methods for estimation of non-analytical integrals through numerical methods play a key role across a broad spectrum of scientific fields, but these methods are often computationally expensive in nature.
Methods such as finite-element or volumes, which are widely used in physical simulation to integrate partial differential equations, or Monte Carlo estimation, which is widely used in Bayesian statistics for estimation of posteriors, require a large number of function evaluations to reach a desired level of accuracy.
In addition, many numerical integration methods fail to provide uncertainty quantification on their estimates, which is crucial in the applied settings in which physical simulation is often used.
Bayesian quadrature (BQ) (Diaconis, 1988; O’Hagan, 1991) is a probabilistic method which can remedy these concerns by offering performance on computationally-limited small data while admitting robust uncertainty bounds.
BQ takes the form of a traditional quadrature rule:
[Equation]
for n evaluations of the function f , where weights w i ∼ p ( x ) are instead learned through manipulation of a Bayesian non-parametric Gaussian process (GP) (C. E. Rasmussen and Williams, 2006) model on observations of the integrand f ( x ) .
The use of such a Bayesian non-parametric model for learning weights leverages the ability for GPs to perform well under data-scarcity as well as quantify uncertainty in a principled manner.
In addition, the Gaussian nature of this model allows for the integral estimate of f to be a simple analytical integration of the GP prior on f using well-known characteristics of multi-variate Gaussian distributions.
Previous work (Ghahramani and C. Rasmussen, 2003; Kandasamy,
Schneider, and Póczos, 2015) has clearly demonstrated computational efficiency of BQ versus traditional methods such as Monte Carlo integration when the data dimensionality d < 10 .
A chief advantage of using GPs in any probabilistic learning setting is the flexibility of choice of the GP kernel function k , which allows for a practitioner to inject domain knowledge of the problem space into the GP model.
Characteristics such as as data smoothness or periodicity can easily be applied through choice or composition of specific kernel functions tailored to these settings.
However, the traditional BQ formulation hamstrings this flexibility by limiting the choice of kernel in the integrand
GP to only a small subset of kernels with known analytical kernel means, such as Gaussian or polynomial kernels.
For well-known kernels that may not be analytically tractable in the BQ setting, but nonetheless might better model an integrand, traditional numerical quadrature methods must be used, reducing the computational efficiency that BQ offers.
The question naturally arises of how practitioners might enable the full suite of kernel choices for use in the GP integrand prior while still maintaining the analytical tractability in the BQ setting, to most efficiently produce an accurate estimate to the integral of f .
In this paper, we expand on the literature of BQ and propose a solution to the problem of kernel choice with generalized
Bayesian quadrature (GBQ), a method derived from random
Fourier features (RFFs) by which any shift-invariant kernel can be used in the GP integrand prior while still allowing for analytical tractability in the BQ setting.
By allowing for both kernel flexibility and analytical integration, we expand upon the ability of traditional BQ to model a variety of integrand types while still maintaining the computational efficiency BQ offers.
We summarize our contributions here:
Contributions
• We propose generalized Bayesian quadrature (GBQ), a method of Bayesian quadrature that allows for the use of any shift-invariant kernel in the GP model of the integrand while still admitting an analytical estimate of the integral posterior mean and variance.
• We show that GBQ can directly be used to compute integrals over Gaussian and uniform measures within the same framework.
• We derive the upper-bounded error to this approximation as a function of data-availability.
• We outline the assumptions under which GBQ shares the computational complexity of traditional BQ.
• We demonstrate the accuracy and flexibility of this quadrature method versus traditional BQ, as well as data-efficiency versus typical Monte Carlo integration, on a selection of relevant domain problems.
RELATED WORKS
Quadrature methods of the type in equation 1 are wellstudied due to their importance to a variety of fields, and there is a deep literature dating back centuries on methods for numerically approximating integrals.
We will briefly review here relevant methods in relation to Bayesian quadrature.
Rather than deterministic quadrature weighting, various probabilistic quadrature approaches have been proposed (C. J. Oates and Sullivan, 2019) for integration when model observations are expensive, with one of the most popular methods being Bayesian quadrature.
Many extensions to vanilla BQ have been developed over the years to improve performance and provide theoretical guarantees (Acerbi, 2018; Belhadji, Bardenet, and Chainais, 2019; Briol, Chris
J. Oates, Girolami, and M. A. Osborne, 2015; Kennedy, 1998).
Other applications include use in multi-fidelity modeling (Gessner, Gonzalez, and Mahsereci, 2020), Bayesian posterior estimation (Gunter et al., 2014; M. Osborne, Garnett, Roberts, et al., 2012), Bayesian optimization (Nguyen et al., 2020), and model selection (Chai et al., 2019; M.
Osborne, Garnett, Ghahramani, et al., 2012).
The derivation of analytical forms, or empirical approximation, of kernel means, which is a significant component of the BQ formulation, is a problem that appears in numerous other fields.
Namely, kernel mean embedding (Muandet et al., 2017), deep Gaussian processes (Damianou and
Lawrence, 2013), and neural operators (Kovachki et al., 2021; Li et al., 2021) all attempt to do so through various means.
There also exist empirical methods for the estimation of kernel means using random Fourier features (Muandet et al., 2017), as well as strong theoretical connections between the very concept of kernel-based quadrature and random Fourier features (Bach, 2017).
In a related manner, methods have been proposed that seek to implement Fourier feature kernels through quadrature based methods (Mutny and Krause, 2018).
However, to our knowledge, no methods directly solve kernel integrals analytically in the BQ setting using RFFs, as we propose to do in this paper.
Similar to RFFs, spectral mixture kernels (SMKs) (Oliva et al., 2016; Wilson and Adams, 2013), also model shiftinvariant kernels as the spectral transform of a probability measure. In the SMK case, this measure is a Gaussian mixture model, which can be shown to asymptotically approximate any stationary kernel as the number of mixture components increases. While BQ has found applications in constructing hyper-kernels by marginalizing SMKs over mixture priors (Hamid et al., 2022), their use within BQ has been limited.
The method which shares the most overlap with this work is the Fourier neural operator (FNO) (Li et al., 2021), which, as a part of a larger deep neural network architecture, estimates the convolution of shift-invariant kernels with a probability measure using parameters in Fourier space.
While we take a similar approach to deriving kernel means using Fourier frequencies, the overall frameworks differ, with GBQ existing in the Gaussian process framework, thus offering uncertainty estimates for integral posteriors, while FNOs exist within a deterministic neural network architecture.
PRELIMINARIES 3.1 BAYESIAN QUADRATURE
We will now review various preliminary methods upon which GBQ is built, starting with Bayesian quadrature.
BQ assumes we have a function f that we are trying to integrate and a dataset D
= { x i , y i } ni =1 with n noisy observations of f , where x ∈ R d , y i = f
( x i ) + ϵ , and ϵ is i.i.d normal distributed noise.
Typically, f is computationally expensive to evaluate, implying a small n and highlighting the need for uncertainty estimation in the final integral approximation.
BQ does this by first placing a Gaussian process (C. E. Rasmussen and Williams, 2006) prior on f , which we will briefly review here.
Gaussian Processes Gaussian processes are a Bayesian non-parametric method which model the target data generation function f
we are attempting to learn as a joint multivariate Gaussian of the form:
[Equation]
where k θ is a positive semi-definite kernel function with hyper-parameters θ , and µ is a mean function.
In the above, we assume an additive and independent Gaussian noise observation model with, ϵ ∼ N (0 , σ 2 I ) , where y are noisy observations with standard deviation σ .
k is typically chosen a-priori to encode known characteristics of the data D such as periodicity and smoothness.
For inference, the posterior-predictive distribution of f ∗ for a new data point { x ∗ } , given the training data D
= { x i , y i } ni =1 , and Gram matrix K xx
= k θ ( x , x ′ ) , ∀ x , x ′ , is given by N ( µ ( f ∗ ) , Cov( f ∗ ))
where,
[Equation]
In BQ, by setting a GP prior on the integrand f we can leverage the ability of GPs flexibly and accurately model functions with uncertainty on small data, but it is also advantageous in that we can directly and analytically integrate the integrand GP prior.
This is performed using well-known characteristics of Gaussian distributions in order to form a posterior estimate ⟨ ¯ f ⟩ of the integral of f .
Formally, the mean of the BQ estimate of ⟨ ¯ f ⟩ is the expected value over measure p ( x ) of the posterior mean of the GP prior (4) on f :
[Equation]
where µ x ( X ) =
[ µ x ( x 1 ) . . .
µ x ( x n )] can be seen as the kernel mean over measure p ( x ) .
The variance of this estimate is:
[Equation]
which is notably independent of prior observations X .
The mean formulation mirrors that of standard quadrature methods shown in equation (1), differing in that weights µ x ( X ) T K − 1 are the result of probabilistic learning on observed data D and associated kernel choice, rather than decided a priori or by a heuristic.
Under a very limited selection of kernel and sampling measure choices, the mean (6) and variance (7) can be calculated analytically (Briol, Chris J. Oates, Girolami, M. A. Osborne, and Sejdinovic, 2019).
Most commonly, a Gaussian kernel and Gaussian distribution for the measure p ( x ) , as proposed by (O’Hagan, 1991), is one such case.
It is also prudent to note that the measure distribution can be fluid while retaining analytical tractability through use of importance sampling (Briol, Chris J. Oates, Cockayne, et al., 2017;
Ghahramani and C. Rasmussen, 2003), while the choice of kernel is more restricted.
In BQ, the limitation of the kernel to certain forms dependent on known closed-form analytical integration over the measure p ( x ) gives up one of the greatest advantages of the
GP prior: flexible selection of kernels for specific domains.
To alleviate this issue, GBQ introduces random Fourier features into the BQ formulation for parametrization of the GP kernel.
3.2 RANDOM FOURIER FEATURES
As we shall see in Section 4 Random Fourier features enable the use of any shift-invariant kernel in the BQ-GP prior without sacrificing the analytical tractability of the integral posterior.
This greatly increases the flexibility of the BQ to perform under a variety of problem conditions for which different kernels may be necessary.
Random Fourier features are obtained from the spectral representation of shift-invariant kernels given by Bochner’s theorem:
Theorem 1 (Bochner’s theorem (Rudin, 2011)) .
A shiftinvariant kernel k ( x , x ′ )
=
k ( x − x ′ ) is positive-definite if and only if it is the Fourier transform of a non-negative measure.
Theorem 1 is the building block upon which (Rahimi and
Recht, 2008) introduce random Fourier features (RFFs), which define a practical means by which Bochner’s theorem can be applied in practice to estimate kernel functions in finite dimensions.
Using the derivation from (Rahimi and
Recht, 2008), if the probability density p ( ω ) is the Fourier transform of k :
[Equation]
For brevity, equation (8) provides the formulation for the case that the kernel and data x are real-valued, but an alternative formulation exists for the case they are not.
It can be easily seen that the kernel function k is entirely defined by the choice of density p ( ω ) , and several common kernels have known associated densities.
For example, if p ( ω ) is multivariate isotropic Gaussian, then (8) represents the radial basis function (RBF) kernel.
By drawing from the associated p ( ω ) for our choice of kernel, RFFs approximate (8) with Monte Carlo by:
[Equation]
where R is the number of Monte Carlo samples or Fourier features .
Alternatively, we can directly parametrize these features ω as GP hyperparameters, which allows for optimal kernels to learned during training to best adapt to specific problem settings (Chang et al., 2017; Oliva et al., 2016;
Tompkins et al., 2019; Zhen et al., 2020).
GENERALIZED BAYESIAN QUADRATURE
We build upon these concepts to devise our method, generalized Bayesian quadrature, which enables flexible Bayesian quadrature for use with any arbitrary shift-invariant kernel while maintaining analytical tractability of the kernel mean µ x ( X ) .
We begin by showing that a Gaussian density can be approximated with RFFs, which will lead to analytical tractability for general shift-invariant kernels.
4.1 PROBABILITY DENSITY FUNCTIONS AS RFF KERNELS
Analytical tractability of the BQ mean in (6) for any kernel represented by RFFs can be achieved by reformulating the kernel mean measure p ( x ) as an RFF as well.
In general, we can turn any positive-definite probability density function p : X → [0 , ∞ ) on X ⊆ R d into a stationary kernel via the following construction:
[Equation]
It is easy to verify that a kernel defined as in equationis translation-invariant and positive-definite whenever p is.
As examples of distributions with positive-definite densities we have the Gaussian and the Student-T (Rossberg, 1995).
Given that many probability distributions can be approximated by these densities, or mixtures of them, kernel modeling of distributions as in (10) has a wide range of potential applicability.
RFF Representation of the Gaussian Given that an RBF kernel represents an un-normalized Gaussian, by sampling ρ from N ( 0 , I ) and using a multivariate Gaussian normalizing constant τ − 1 =
[(2 π ) d | Σ | ] − 1 / 2 , where Σ is the lengthscale matrix for features ρ , we can formulate an RFF kernel approximation of a Gaussian density function q ( x ) as lim R →∞ as:
[Equation]
This form allows for the use of simple trigonometric identities to form an analytically integrable kernel mean formulation (6) over a Gaussian measure, which we will shortly demonstrate.
4.2 GENERALIZED BAYESIAN QUADRATURE POSTERIOR
We now reformulate the BQ mean and variance by substituting the RFF formulations of both the kernel and measure in equations (9) and (11) into the BQ mean in equation (6).
[Equation]
The trigonometric form of both the kernel and measure distribution in this setting allow for the application of basic identities to rewrite the integrand as a linear function.
Using the identity cos( α ) cos( β )
=
cos( α + β ) / 2+cos( α − β ) / 2 , and simple properties regarding the anti-derivatives of trigonometric functions, we arrive at the following definition of GBQ over an approximated Gaussian measure q ( x ) .
Definition 1 (Generalized Bayesian Quadrature Over Gaussian Measures) .
Given n noisy observations { x i , y i } ni
= = X , y } of a function f where x i ∈ R d , a kernel function k parametrized through random Fourier frequencies ω ∈ R R ×
d sampled from density p ( ω ) , a Gaussian measure approximation q ( x ) parametrized by Fourier frequencies ρ ∈ R Z × d sampled from N ( 0 , I ) , and kernel matrix K =
[ k ( x i , x ′ j )] ni,j =1 ∈ R n × n , the GBQ estimate ⟨ ¯ f ⟩ of the mean of the integral of f over domain a ≤ x ≤ b is: [Equation]
[Equation]
where d is the dimensionality of x , and h d is the function at the d -th index of the repeating series h
= [sin , − cos , − sin , cos , sin , . . . ] .
The normalization constant L is defined as:
[Equation]
where Q ba = (cid:82) b a q ( x ) d x is an estimate to the CDF of the RFF-parametrized Gaussian q ( x ) , which is analytically calculable from (11) 1 .
See the supplement for full proof, variance derivation, and details of an algorithm for efficient implementation.
In addition, the supplement provides a GBQ formulation over uniform measures, which equates to direct integration of the GP integrand ¯ f .
Through definition 1 we obtain an analytical posterior for ⟨ ¯ f ⟩ and V ( ⟨ ¯ f ⟩ ) that allows for flexible kernel choice through the use of RFFs.
4.3 APPROXIMATION ERROR 4.3.1 Gaussian Process and Random Fourier Features Error Bounds
The approximation error of GBQ extends from well-known error bounds derived from the literature of RFFs and BQ respectively.
We present here an abbreviated form of this proof, the full version of which can be found in the supplement.
We begin with the following lemma outlining the error of the GP estimate ¯ f to the integrand f under the assumption f is a member of the Hilbert space H k defined by kernel k :
Lemma 1 (Durand, Maillard, and Pineau (2017, Theorem 1)) .
Assume f ∈ H k and that the observation noise ϵ is σ ϵ -sub-Gaussian.
Then the following holds with probability at least 1 − δ :
[Equation]
where µ n and σ 2 n denote the GP posterior mean and variance given n observations, according to (4) and 5, respectively, and
[Equation]
with
[Equation]
We follow with a lemma related to the error bounds on the
RFF approximation to a shift-invariant kernel k .
Lemma 2 (Sutherland and Schneider (2015, Proposition 1)) .
Let k : X ×X → R be a continuous shift-invariant positivedefinite kernel with k ( x , x ) = 1 and such that ∇ 2 k ( x , x ) exists, for all x ∈ X ⊂ R d .
Suppose X is compact with diameter ℓ X < ∞ .
Denote k ’s Fourier transform as P k , which is a probability measure, and let σ 2 k := E [ ∥ ω ∥ 22 ] for ω ∼ P k .
Let ˜ k
X ×X → R denote k ’s RFF approximation with R frequencies according to (9) .
Then the following holds for any 0 < ξ < σ k ℓ X :
[Equation]
sup [Equation] Therefore, for any δ ∈ (0 , 1) , we can achieve pointwise approximation error less than ξ with probability at least 1 − δ if:
[Equation]
4.3.2 Generalized Bayesian Quadrature Error
Next, we formulate an error bound on the RFF parametrization of the Gaussian (or any arbitrary) density shown in equation (11), as we build towards a final bound on GBQ.
Theorem 2 (Error of the RFF Density Approximation) .
Let p : X → R be a positive-definite probability density function defined on X ⊂ R d which is such that ∇ 2 p ( 0 ) exists.
Assume X is compact, and let b p > 0 be any constant such that b p ≥ max x ∈X p ( x ) .
Let ˜ k p denote an RFF approximation with Z ∈ N frequencies to k p as defined in (10) , and let ˜ p : x (cid:55)→ ˜ k p ( x , 0 ) , x ∈ X .
Then, for any ξ > 0 , the following holds:
[Equation]
[Figure]
where for the second statement we assume ξ ≤ σ k p ℓ X , and σ k p , ℓ X , α ξ and β ξ are the same as defined in Lemma 2 for k :
= 1 b p k p .
Finally, we combine these results to arrive at the upper bounded error for GBQ as a composition of the errors of
GP approximation, RFF approximation, and RFF measure density estimation.
Theorem 3 (Upper-Bounded Generalized Bayesian Quadrature Error) .
Let f ∈ H k , where k : X × X
→ R is a positive-definite, translation-invariant kernel on X ⊂ R d .
Assume that:
X is compact with diameter ℓ X < ∞ and volume v X := (cid:82) X d x < ∞ ; 2.
k ( 0 , 0 ) = 1 and ∇ 2 k ( 0 , 0 ) exists; 3. and p : X → [0 , ∞ ) is a positive-definite probability density function.
Then the following holds with probability at least 1 − δ :
[Equation]
≤ (cid:18) n λβ ϵ (cid:18) δ (cid:19) ξ k + β k (cid:18) δ (cid:19) max x ∈X σ n ( x ) (cid:19) × (1 + b p ξ p v X ) + ∥ f ∥ ∞
b p ξ p v X , (22) where β ϵ ( δ ) :=
∥ f ∥ ∞ + σ ϵ (cid:113) 2 log (cid:0) nδ (cid:1) , for an RFF approximation to k with R ≥ R (cid:0) ξ k , δ 4 , σ k (cid:1) frequencies and an RFF approximation to p with Z ≥ R (cid:0) ξ p , δ 4 , σ k p (cid:1) frequencies, given ξ k > 0 and ξ p > 0 .
We refer the reader to the supplementary for the full proof of theorems 2 and 3.
4.4 COMPLEXITY
We consider here the complexity of calculating the BQ and GBQ mean integral approximation ⟨ ¯ f ⟩ as in 6 and 14.
Traditional BQ over a Gaussian measure p ( x ) (6), under the assumption N > d , has a mean-calculation complexity that is dominated by the operation K − 1 , which scales in O ( N 3 ) . Comparatively, GBQ can be either dominated by the same term or via the complexity introduced through the novel method of estimation of the RFF kernel mean as in definition GBQ mean calculation with a Gaussian measure, over definite bounds of dimensionality d for all N , scales in O ( dNRZ ) with the number of Fourier features R used for kernel k approximation, and the number of Fourier features Z used for RFF approximation q ( x ) of Gaussian measure p ( x ) .
Over a uniform measure and definite bounds, which equates to direct integration of the integrand GP ¯ f , calculation of the GBQ mean scales in O ( dNR ) time.
In the case of GBQ over a Gaussian measure, if dRZ < N 2 ,
GBQ mean-calculation is also O ( N 3 ) as in traditional BQ.
For GBQ over a uniform measure, if dR < N 2 , we can assume the same.
EXPERIMENTS
We demonstrate here the empirical results of GBQ compared to traditional Monte Carlo quadrature methods and
BQ.
Specifically, we measure percent error versus the analytical integral solution, with baselines of Monte Carlo (MC) integration, quasi Monte Carlo (QMC) using Halton sequence sampling (Halton, 1960) over a uniform hypercube, and BQ with the RBF kernel and a Gaussian measure.
For GBQ, we present results in the form of GBQ-MeasureKernel, where the kernel is chosen from the RFF estimates to the RBF, Matérn 1/2 (M1/2), Matérn 3/2 (M3/2), and
Matérn 5/2 (M5/2), and the measure is either uniform (U) or Gaussian (G).
We hold static the number of integrand observations f ( x ) available across all baselines and GBQ models.
Additionally, we use the same GP kernel hyperpa- rameters θ in both BQ and GBQ, which are trained once per each experiment at each n and shared across all models and kernels.
For Fourier features ω and ρ in equations (9) and (11), we sample using Halton sequences as well to produce a smoother coverage of the sample space.
Finally, we implement these methods in Julia (Bezanson et al., 2015), and code has been made available 2 .
We note that while our experiments consider the employment of the Matérn family of kernels, any shift-invariant kernel can be used in the GBQ integrand prior to adapt to a wide array of problem settings.
While there are various analytical solutions to the Matérn family in traditional BQ, they require a kernel-specific integral to be calculated and implemented, and don’t exist over all measures p ( x ) .
We provide evidence to the flexibility of our method by showing that
Matérn kernels can be implemented without change of problem formulation by simply sampling features ω according to the appropriate frequency distribution.
For all experiments, at each training size n we report results as the average model-wise results over multiple runs under different random seeds, and include information on the error variance over runs.
While experiments were run for all kernel-measure combinations for GBQ, for brevity we include here only those models that performed best on a given experiment.
5.1 1D EXPERIMENTS
Our first experiment is a simple 1D polynomial to empirically verify our theoretical results of section 4 regarding the efficacy of the GBQ method in both recreating results of traditional BQ using the RBF kernel as well as demonstrate the flexibility of kernel choice that GBQ offers.
We model the integral of a polynomial of the form:
[Equation]
in the first case, and disjointed version of the polynomial f
[Equation]
[Equation]
in the second.
The choice of the disjoint polynomial is in order to assess the value of the flexibility of GBQ in enabling varied kernel choice in BQ, and in this case we leverage
Matérn kernels, which typically perform better than the RBF on non-smooth data.
We use 100 Fourier features in all GBQ models, and run each experiment 10 times under different seeds at each n and report the aggregated mean and 95% confidence bounds in figure 1.
In the first experiment, which represents a smoother polynomial, BQ and GBQ both outperform QMC in accuracy as a function of data scarcity.
We can see that GBQ-G-RBF is an excellent approximation to BQ, which similarly leverages an RBF kernel over a Gaussian measure, which helps to validate our theoretical results on both the accuracy of the
RFF-based integration of the RFF-RBF kernel over a Gaussian measure, as well as the ability for RFFs to parametrize
Gaussian distributions.
In the disjoint case, we see that at low n , GBQ has a slight advantage over BQ when using the Matérn kernel, but that results converge for all methods as training size increases.
While QMC achieves better error at some points, it generally displays more variance over n in this experiment than the
BQ and GBQ-based models.
5.2 2D EXPERIMENTS
We now move to a selection of 2D experiments, first of which is estimating the integral of a polynomial of the form f ( x, y )
= − 0 .
005 x 4 ∗ 0 . 1 x 3 + y 5 (0 . 02 x − 0 .
− 0 .
001 y 2 + 0 . 2 y
+ 0 .
5 (25) over the interval x ∈ [ − 4 , 4] , y ∈ [ − 2 . 5 , 2 . 5] , as well as a disjoint 2D function: f ( x, y ) = (cid:40)
[Equation]
over the unit cube.
We perform both experiments over a range of training data sizes from 10 to 1000 n , with 5 runs per n at different random seeds.
All GBQ models use 300 Fourier features.
Plots of these functions can be seen in figure 2, and the [Table] [Table] means and standard deviations of the results are reported in tables 1 and 2.
In both experiments, we see that GBQ methods have universally lower mean error than QMC and BQ.
The best performing kernel varies across n , but in several cases we see that the Matérn has the lowest error, supporting the case that flexibility of kernel choice is a valuable addition to the
BQ method when considering both different integrand types as well as available training data.
In the disjoint polynomial experiment, we intentionally include GBQ-G with the RBF (the BQ equivalent) in table 2, even though it was not high performing among the GBQ methods, to demonstrate the potential performance enhancement GBQ offers through kernel choice.
We see GBQ-GRBF track closely with BQ, while GBQ-U with the Matérn 1/2 and GBQ-U-RBF in combination perform better at all n , and frequently with implied worst-case error bounds well below the BQ mean error. 5.3 5D EXPERIMENTS
We use a 5D problem from a seminal BQ paper (Ghahramani and C. Rasmussen, 2003) to provide an initial evaluation of GBQ in higher dimensions. We model the equation: [Equation]
as well as a disjoint variant:
[Equation]
where observations y =
f
( x ) + ϵ and y ∗ =
f ∗ ( x ) + ϵ have added noise ϵ ∼ N (0 , 12 ) .
We perform integration methods over the 5D unit hypercube using 100 Fourier features.
Shortened results are provided in tables 3 and 4 as the average and standard deviation of integral approximation percent error versus the analytical solution across 10 random seeds.
Results across all n are available in the supplement.
In the non-disjoint setting, GBQ methods are the highest performing across all experiments with n > 50 . Notably, we choose to report MC as a baseline other than QMC, as across both experiments in 5D we see a degradation of QMC methods in favor of simple MC. In the disjoint setting,
MC is the highest performing at low N, with BQ and GBQ methods performing best at mid to high n .
An interesting experimental result was the importance of consistent methodology used for solving the kernel mean µ x ( X ) and producing the kernel matrix K , when applied in the BQ posterior mean formulation (6).
Anecdotally, we found that using the combination of a kernel mean derived from traditional BQ and a kernel that was estimated through RFFs, and vice-versa, produced significantly unstable posterior integral mean estimates.
These results suggest the benefit of using the full-stack GBQ method with RFF parametrization of both the kernel and measure distribution in order to achieve the best experimental results. [Table] [Table]
DISCUSSION
In this paper, we have introduced generalized Bayesian quadrature, a method for performing Bayesian quadrature using any shift-invariant kernel while maintaining posterior tractability.
We derive the upper bound on the error of this approximation, while also demonstrating the practical benefits on a selection of quadrature problems when compared to traditional numerical integration methods and baseline
BQ.
More broadly, we note the wider applicability of the methods proposed in this paper.
Our chief theoretical contribution comes within the framework of Bayesian quadrature, but in essence it is providing the analytical solution to a kernel mean when the kernel and measure distribution are approximated by RFFs.
However, kernel means have a wide array of use cases as discussed in 2, and represent fertile ground for future applications of our theoretical results.
Additionally, as part of the process of applying GBQ over closed-bounds in multiple dimensions, raising necessity for a truncation term composed of multivariate cumulative distribution functions, we devised a method to parametrize distributions using RFFs and analytically integrate this estimate in order to produce a CDF.
For many distributions which offer no closed-form multivariate CDF, this method might be of use.
Future research may look into these applications as well as extending the flexibility and computational aspects of the method.
Potential extensions include learning the RFF kernel through its spectral density, leveraging low-rank GP posteriors for computational efficiency improvements in kernel matrix inversion in the BQ mean, and composing multiple levels of GBQ together into deeper architectures for applications to highly nonlinear problems.
The introduction proposed in this paper has demonstrated both theoretical and empirical promise that will provide a solid launching point for these pursuits.
Acknowledgements
Rafael Oliveira was supported by the Medical Research
Future Fund for Applied Artificial Intelligence in Health
Care grant (MRFAI000097).