Combining Pseudo-Point and State Space Approximations for Sum-Separable Gaussian Processes
Abstract
Spatio-temporal Gaussian processes (GPs) are important probabilistic tools for inference and learning in climate science, epidemiology, or any time-driven general GP modelling problem.
The current gold-standard methods for scaling GPs to large data sets are various ﬂavours of pseudo-point methods.
These methods do not cope well with long or unbounded temporal observation horizons, which undermines their eﬃciency and eﬀectively turns the computational scaling back to cubic in the number of temporal observations.
On the other hand, if the temporal part in the GP prior admits a Markov form, the inference can be sped up to linear in the number of temporal observations by using state space models.
In this work we show how to combine the most widely used pseudo-point method, Titsias’ variational approach, with the state space approximation framework.
Our approach hinges on a surprising conditional independence property which applies to space–time separable
GPs.
By utilising pseudo-point approximations over space, and state space approximations through time, we are able to construct an approximation that is more scalable and widely applicable to spatio-temporal problems than either method on their own.
1. Introduction
Large spatio-temporal data containing millions of observations arise in various domains, such as climate science.
While Gaussian process (GP) models (Rasmussen and Williams, 2006) are eﬀective in such settings, their computational expense is prohibitive if they are employed naively.
Consequently, approximation is necessary.
In this work we combine the complementary strengths of pseudo-point (Qui˜nonero-Candela and Rasmussen, 2005; Bui et al., 2017) and state-space (S¨arkk¨a et al., 2013; S¨arkk¨a and Solin, 2019) approximations to tackle spatio-temporal problems.
At its core this work hinges on an observation made by O’Hagan (1998) that, if a GP’s kernel is separable, then it possesses a surprising kind of conditional independence property.
Interestingly, this observation appears to have gone largely unnoticed within the GP community.
This facilitates the combination of pseudo-point and state-space approximations, resulting in algorithms that scale linearly in time.
In particular, we show (i) how the conditional independence property can be exploited to signiﬁcantly accelerate the variational inference scheme of Titsias (2009) for GPs with separable kernels and sum-separable kernels, (ii) how this can be straightforwardly combined with the Markov property exploited by state-space approximations (S¨arkk¨a and Solin, 2019) to obtain an accurate approximate inference algorithm for sum-separable spatio-temporal GPs, that scales linearly in time, and (iii) how the earlier work of Hartikainen et al.
(2011) on this topic is more closely-related to Snelson and Ghahramani (2005) than previously realised.
2. Pseudo-Point Approximations
Consider a GP prior f ∼ GP ( m, κ ) and an observation (likelihood) model with N observations y ∈ R N made at locations x ∈ X N , with conditional distribution p ( y | f )
= (cid:81)
Nn =1 p ( y n | f n ), f n := f ( x n ).
The seminal work of Titsias (2009), later clariﬁed by de G. Matthews et al.
(2016), introduced the following approximation to the posterior distribution over f :
[Equation]
where u m := f ( z m ) are the pseudo-points for a collection of M pseudo-inputs z 1: M , and f (cid:54)
=
u :=
f \ u are all of the random variables in f except those used as pseudo-points.
We assume that q ( u ) is Gaussian with mean
ˆ m u and covariance matrix ˆ C u ; subject to the constraint imposed in Eq.
(1) this family contains the optimal choice for q ( u ) if each conditional p ( y n | f ( x n )) is Gaussian, and is otherwise the de-facto standard choice (Hensman et al., 2013).
This yields the approximate posterior predictive distribution at any collection of test points x ∗
[Equation]
where Λ u := C − 1 u is the inverse of the covariance matrix between all pseudo-points, C f ∗ u is the cross-covariance between the prediction points and pseudo-points under f , and m u and m f ∗ are the mean vectors at the pseudo points and prediction points respectively.
Supposing that the conditionals are Gaussian, which we denote by p ( y | f ) = N ( y ; f , S ) for some positive-deﬁnite diagonal matrix S ∈ R N × N , it is possible to optimise q ( u ) analytically and obtain a closed-form expression for the ELBO, known as the saturated bound :
[Equation]
Through the use of the matrix inversion and determinant lemmas, this quantity can be computed using only O (cid:0) NM 2 (cid:1) operations.
This is typically acceptable for regression tasks where the inputs are sampled i.i.d.
as the value of M required as N increases generally seems not to grow too fast—indeed Burt et al.
(2019) showed that if the inputs x n are sampled i.i.d.
from a Gaussian then the value of M required scales roughly logarithmically in N .
However, Bui and Turner (2014) noted that this is typically not the case for time series problems, where the interval in which the observations live typically grows linearly in N .
Moreover Tobar (2019) showed that to ensure the posterior approximation does not degrade, the density of the pseudo-points must not drop below a rate analogous to the
Nyquist-Shannon rate.
Consequently the number of pseudo-points M required to maintain a good approximation must grow linearly in N , so the cost of accurate approximate inference using pseudo-point methods is really O (cid:0) N 3 (cid:1) in this case.
3. State-Space Approximations to Sum-Separable Spatio-Temporal GPs
S¨arkk¨a et al. (2013) and S¨arkk¨a and Solin (2019) showed that a given separable spatiotemporal GP f ( τ, r ) can be approximated by another GP ¯
f
( τ, r , d ), where d ∈ { 1 , . . .
, D } picks one of D latent dimensions which render ¯ f Markov in τ , the ﬁrst of which approximates f .
This approximation can generally be made tight, and achieves equality in various interesting cases (e.g., Mat´ern-family GPs).
¯ f is speciﬁed implicitly through a linear stochastic diﬀerential equation, meaning that inference can be performed via eﬃcient ﬁltering / smoothing in a Linear-Gaussian State-Space Model (LGSSM).
Let ¯ f t be the collection of random variables in ¯ f at inputs given by the Cartesian product between the singleton { t } ,
N T arbitrary locations in space r 1: N T , and all of the latent dimensions { 1 , . . .
, D } .
Further, let the kernel of f be separable: κ (( r , τ ) , ( r (cid:48) , τ (cid:48) ))
=
κ r ( r , r (cid:48) )
κ τ ( τ, τ (cid:48) ).
Any collection of such ﬁnite dimensional marginals ¯ f :=
¯ f 1: T , each using the same r 1: N T , form an LGSSM with
N T D -dimensional state-space, and dynamics
[Equation]
where H = (cid:2) 1 0 . . . 0 (cid:3) ∈ R 1 × D ignores all but the ﬁrst dimension of ¯ f n,t , A t and Q t are functions of κ τ , and C rf is the covariance matrix associated with κ r and r 1: N T .
This formulation can be straightforwardly extended to sums of separable processes (App. B).
While this formulation truly scales linearly in T it has two clear limitations, (i) all locations of observations must lie on a rectilinear time-space grid if any computational gains are to be achieved; and (ii) inference scales cubically in N T , meaning that inference is rendered infeasible by time or memory constraints if a large number of spatial locations are observed.
4. Exploiting Separability to Obtain the Best of Both Worlds
We now combine the pseudo-point and state-space approximations, and show how a temporal conditional independence property means that the optimal approximate posterior is
Markov.
This in turn leads to a closed-form expression for the optimum under Gaussian observation models, and the existence of a simpliﬁed LGSSM in which exact inference yields optimal approximate inference in the original model.
Pseudo-Point Approximation of State-Space Augmentation
We perform approximate inference in a GP f by applying the standard variational pseudo-point approximation (Sec. 2) to its state-space augmentation (Sec. 3) ¯ f :
[Equation]
where the pseudo-points ¯ u = ¯ u 1: T form a rectilinear grid of points in time, space, and the latent dimensions with the same structure as ¯ f in Sec. 3, but replacing r 1: N T with a collection of M τ spatial pseudo-inputs, for a total of T M τ D pseudo-points.
p (¯ u ) is therefore
Markov-through-time, so has block-tridiagonal precision Λ ¯ u
(Grigorievskiy et al., 2017).
Crucially, we will now relax the assumption that f must associated with inputs on a rectilinear grid, requiring only that each observation is made at one of the T times at which we have placed pseudo-points.
We denote the number of observations at time t by N t .
Exploiting Conditional Independence
Due to O’Hagan (1998)’s conditional independence property, p (cid:0) ¯ f t (cid:12)(cid:12) ¯ u (cid:1) =
p (cid:0) ¯ f t (cid:12)(cid:12) ¯ u t (cid:1) ; see App.
A for details.
Consequently, the reconstruc- tion terms in the ELBO depend only on ¯ u t as opposed to the entirety of ¯ u :
[Equation]
This property yields substantial computational savings in both the separable and sumseparable case (App. B.1).
The Optimal Approximate Posterior is Markov
As an immediate consequence of
Eq.
(7), by the same argument as that made by Opper and Archambeau (2009)
the optimal approximate posterior precision has the form
[Equation]
G is block-diagonal with the same block-sizes as Λ ¯ u , so
ˆΛ ∗ ¯ u is also block-tridiagonal and the optimal approximate posterior is Markov; see App.
C for details.
Moreover, Ashman et al.
(2020) (App.
A) show that G t can be written as a sum of N t rank-1 matrices.
Optimal Approx. Posterior under Gaussian Observation Model
In the case that each p ( y n,t | f n,t ) = N ( y n,t ; f n,t , S n,t ), an explicit expression for the optimal parameters of the approximate posterior N (cid:16) ¯ u ; ˆ m ∗ ¯ u , [ˆΛ ∗ ] − 1 ¯ u (cid:17) can be obtained from Eq.
(8): ˆΛ ∗ ¯ u = Λ ¯ u + G , where G t :
=
H (cid:62) u
t
Λ u t
[Equation]
u
[Equation]
u
[Equation]
where H u t := I M τ ⊗ H , H u := I T ⊗ H u t , S t := blk-diag( S 1 ,t , . . .
, S N t ,t ), S :
= blk-diag( S 1 , . . .
, S T ), and Λ u C uf are block-diagonal; see App. A.3.
Approximate Inference via Exact Inference in an Approximate Model
This optimal approximate posterior over ¯ u is equal to the exact posterior under the simpler state-space model given by
[Equation]
that is, ˜ p
(¯ u | y ) =
N (cid:16) ¯ u ; ˆ m ∗ ¯ u , [ˆΛ ∗ ] − 1 ¯ u (cid:17) .
This is analogous to the relationship between the approximate model employed by the Deterministic Training Conditional (DTC) (Seeger et al., 2003), and the variational approximation (Titsias, 2009).
Indeed, this model is precisely that employed by DTC, and it can be exploited both to perform approximate inference and the saturated bound; see App.
D. This allows for approximate inference to be performed by running exact inference in an LGSSM, which scales linearly in time and can re-use existing code.
For example, it is clear that the parallelised inference procedure derived by S¨arkk¨a and Garc´ıa-Fern´andez (2020) could be utilised in this approximation.
Computational
Intensity
The total number of ﬂops required to compute the saturated
ELBO is roughly T M 3 τ + M 2 τ (cid:80)
Tt =
1 N t .
This is a great deal fewer when T is large than the
M 3 + M 2 N = M 3 τ T 3 + M 2 τ T 2 N required if the bound is computed naively.
Related Work
The conditional independence property exploited to develop the variational approximation in this section also shines new light on the work of Hartikainen et al.
Speciﬁcally, performing inference in the approximate model they introduce would yield FITC (Snelson and Ghahramani, 2005) exactly ; see App.
E.
The popular Kronecker-product methods for separable kernels explored by (Saat¸ci, 2012) are unable to handle heteroscedastic observation noise / missing data, scale cubically in time, and require observations to lie on a rectilinear grid.
Our approach suﬀers none of these drawbacks.
5. Experiments: Proof-of-Concept
We conducted two simple proof-of-concept experiments on synthetic data with a separable
GP to verify our approach to approximate inference, more detail in App.
G. Similar evaluation of a sum-separable kernel is in App. H; evaluation on real-world problems is on-going.
All timing experiments are conducted using a single thread on a 2019 MacBook Pro with 2.6 GHz CPU.
In both experiments we consider quite a large temporal extent, but only moderate spatial, since we expect the proposed method to perform well in such situations.
If the spatial extent of a data set is very large relative to the characteristic spatial variation, pseudo-point methods will struggle and, by extension, so will our method.
Arbitrary Spatial Locations
Fig.
1 (lhs) shows how inputs were arranged for this experiment; at each time 10 spatial locations were sampled uniformly between 0 and 10.
The spatial location of pseudo-inputs are regular between 0 and 10.
The right hand side shows that when using pseudo-points we are indeed able to achieve substantial performance improvements relative to exact inference by utilising the state-space methodology, while retaining a tight bound.
Approximating the log marginal likelihood well for T = 10 5 time points ( N = 10 6 observations total), takes just over 10s in this example.
Grid-with-Missings
Fig.
2 (lhs) shows how (pseudo-)inputs were arranged for this experiment for M τ = 10; the same 50 spatial locations are considered at each time point, but 5 of the observations are dropped at random, for a total of N t = 45 observations per time point – our largest case therefore involves N = 4 .
5 × 10 6 observations.
The right of the ﬁgure shows that we able to compute a good approximation to the log marginal likelihood using roughly a third of the computation required by the standard state-space approach to inference.
The extent of the improvement here will vary from setting-to-setting depending on how many pseudo-points are required at each point in time to account for the observations – for example, the gains relative to exact state-space inference will be greater if fewer pseudo-points-per-observation can be used.
6. Conclusion
This work shows that for GPs with sum-separable kernels, there exists a natural and eﬃcient manner which to combine pseudo-point and state-space approximations.
Preliminary
[Figure]
T
C o m pu t e
T i m e ( s )
M τ = M τ =
M τ =exact
[Figure]
M
[Figure]
synthetic experiments show that approach can yield substantial improvements to computational eﬃciency whilst retaining a high degree of accuracy.
The methods presented can be straightforwardly combined with approximations for non-Gaussian likelihoods, such as those discussed by Wilkinson et al.
(2020), Chang et al.
(2020), and Ashman et al.
Due to the Markov property there is a natural way to parametrise the approximate posterior (App. F) in such cases, that is analogous to parametrising the ﬁltering distributions.
This parametrisation might also be useful when dealing with problems in which mini-batching (Hensman et al., 2013) is desirable.
Acknowledgments
Will Tebbutt is supported by Deepmind.
Appendix A. Conditional Independence in Separable Gaussian models
This section splits cleanly into four subsections.
Everything in here follows from the deﬁnitions presented in this paper, a main result from O’Hagan (1998), the properties of the
Kronecker product, and basic linear algebra.
The second subsection contains the key result.
A.1. Kronecker Structure in the State-Space Prior
In the separable case, there is Kronecker structure that persists through the LGSSM prior.
Our starting point is an LGSSM whose transition dynamics are of the form
[Equation]
where t ∈ { 1 , ..., T } .
Here f might comprise the ﬁnite-dimensional marginals over e.g. f and its derivatives at both observation locations and pseudo-inputs.
Note that this is the standard form of the LGSSM that separable spatio-temporal GPs are converted to.
First consider the marginal covariance matrix of f t :
[Equation]
Adopting the inductive hypothesis V (cid:2) ¯ f t − 1 (cid:3)
=
C r ⊗ C τt − 1 , it is clear from Eq.
(14) that the base case holds:
[Equation]
Furthermore
[Equation]
so the Kronecker structure of the marginal covariance at time t is clear, as is the method by which it can be computed.
Now consider the covariance between two time points, t < t (cid:48)
w.l.o.g:
[Equation]
Assuming the inductive hypothesis
[Equation]
and inducting on t (cid:48) , the base case ( t (cid:48) =
t ) follows immediately from the result in Eq.
(17), and the induction step shows that
[Equation]
A.2. Conditional Independence given Current Pseudo-Points
Fortunately, under the assumption of separability it is very straightforward to compute the approximate posterior marginal distribution over ¯ f t,n given the approximate posterior marginal distribution over ¯ u t .
This is a consequence of property derived by O’Hagan (1998), which we generalise here to vector-valued RVs
[Equation]
First consider the prior joint distribution over ¯ f t,n , ¯ u t , and ¯ u t (cid:48) :
[Equation]
t
[Equation]
The upper triangle of the covariance matrix is given by symmetry.
Note that this covariance matrix follows from the results in the preceding section with
[Equation]
From the usual rules of conditioning for Gaussians
[Equation]
we deduce that the covariance matrix of ¯ f t,n , ¯ u t (cid:48) | ¯ u t is
[Equation]
where
[Equation]
Observe that the oﬀ-diagonal block is
[Equation]
from which it follows that
[Equation]
for any t (cid:48) .
Moreover
[Equation]
follows from this result and the marginalisation property of Gaussians.
Consider the covariance of the conditional distribution ¯ f t,n , ¯ u 1: T \ t | ¯ u t , assuming w
[Equation]
l
[Equation]
o
[Equation]
g. that t = T :
[Equation]
Observe that by the marginalisation property of Gaussians, any of C ¯ f T,n ¯ u 1 | ¯ u T , C ¯ f T,n ¯ u 2 | ¯ u T , etc being non-zero would contradict property Eq.
It must therefore hold that they are indeed zero, so property Eq.
(28) must hold if Eq.
(27) does.
A.3. Block-Diagonal Structure
Furthermore, this conditional independence property implies that
[Equation]
This is easily proven by considering that, were it not the case, then
[Equation]
31) contradicting property Eq.
From this it follows from repeated application of Eq.
(30) that the larger matrix C f ¯ u C − 1 ¯ u is block-diagonal, and is given by
[Equation]
A.4.
A Further Conditional Independence Property
Note the Kronecker structure present in Eq.
(17): this is the ﬁnite-dimensional manifestation of the conditional independence property of separable GPs.
It follows immediately from this that
[Equation]
[Equation]
where ¯ u t \ u t comprises the latent variables in ¯ u t .
Note this means that the conditional density
[Equation]
must be equal to
[Equation]
Recalling that u t = H u t ¯ u t
[Equation]
it follows that
[Equation]
Substituting this into the expression for the mean in Eq.
(34) yields the following:
[Equation]
which must hold for any value of ¯ u t .
We deduce that
[Equation]
Substituting this result into the expressions for the covariance in Eq.
(35) shows it to be consistent with the covariance in Eq.
[Equation]
Appendix B. Conditional Independence in Sum-Separable GPs A sum-separable kernel
[Equation]
is not itself separable.
However
[Equation]
we can interpret a zero-mean GP f with a sum-separable kernel as a sum over a collection of P independent GPs f p ∼ GP (0 , κ p ), where each κ p is separable.
The covariance between f and any f p is
[Equation]
which is separable.
Note that it is trivial to extend this to handle a non-zero mean GP f by adding the mean function to the sum over the f p s.
This structure persists in the state-space formulation.
Consider a set of P state space approximations ¯ f p , one for to each f p .
The ﬁnite dimensional marginals form an LGSSM of the form
[Equation]
Let ¯
[Equation]
n
[Equation]
 
[Equation]

[Equation]
vcat  
[Equation] [Equation]
[Equation]
  (cid:124) (cid:123)(cid:122) (cid:125) D
× P (42) where vcat transforms a matrix into a column vector by stacking its columns on top of each other, then f n,t =
H ¯ f n,t (43)
y
[Equation]
n
[Equation]
t
[Equation]
η n
[Equation]
∼
[Equation] [Equation]
n
[Equation]
B.1. Extending Conditional Independence Results to Sum-Separable Processes
To extend the results in App.
A.
ﬁrst note that each ¯ f p 1: T is independent of any other ¯ f q 1: T ( p (cid:54) = q ) a priori.
Consequently, we can apply the analysis in App.
A.1 to each ¯ f p 1: T separately and arrive at the associated results in precisely the same way.
Similarly by introducing pseudo-points ¯ u p 1: T for each p with the same structure as ¯ u 1: T we may apply the analysis in App.
A.2 to obtain
[Equation]
for any t (cid:48) , and
[Equation]
Letting ¯ u t be the concatenation of all ¯ u pt , we further wish to show that
[Equation]
To do this, we ﬁrst introduce some additional notation:
[Equation]
By property 46 we know that
[Equation] [Equation]
so it suﬃces to show that
[Equation]
to which the same marginal consistency argument can be made as was used to show property 28.
Since β p ⊥⊥ β q and γ p ⊥⊥ β q , p (cid:54) =
q , we have that
[Equation] [Equation]
as required.
The intuition here is that no additional information is gained by additionally conditioning f n,t on ¯ u q 1: T \ t , q (cid:54)
= p because they are independent a priori.
From propertythe other results in App.
A follow.
Appendix C. The Optimal Approximate Posterior Under Gaussian Likelihoods
We ﬁrst derive an expression for G t .
While the form of the reconstruction term presented in Eq.
(8) is useful in the general case, in the case that
[Equation]
it is most straightforwardly expressed in the vectorised form
[Equation]
where y t and f t the vectors formed by stacking the individual y n,t and f n,t , and S t is the diagonal matrix comprising the S n,t .
By the result in App.
A.4 we have that p ( f t | ¯ u t )
= p ( f t | u t
[Equation] [Equation]
Recall that p ( f t | u t ) =
[Equation]
) where Q t :=
C f
[Equation]
C f t u t Λ u t C u t f t , then expanding the inner expectation yields
[Equation]
[Equation]
Recall that by deﬁnition, u t =
H t ¯ u t , from which m u t =
H u t ¯ u t follows.
Let
[Equation]
C f t u t Λ u t H u t then expanding the remaining expectation yields
[Equation]
From this form it is clear that
[Equation]
as required.
Similar manipulations that also involve KL [ q (¯ u ) || p (¯ u )] produce the expression for the optimal approximate posterior mean
ˆ m ∗ ¯ u .
Appendix D. State-Space DTC
Eq.
(11) introduces the following LGSSM:
[Equation]
To see that ˜ p (¯ u | y )
=
N (cid:16) ¯ u ; ˆ m ∗ ¯ u , [ˆΛ ∗ ] − 1 ¯ u (cid:17) ﬁrst note that ˜ p ( y t | ¯ u t ) can be written as
[Equation]
Furthermore, it follows from this that the conditional distribution ˜ p ( y | ¯ u ) can be expressed succinctly as
[Equation]
which follows from recalling that C fu Λ u is a block-diagonal matrix comprising T blocks, the t th of which is C f t u t Λ u t .
From here we can simply apply the usual rules for linear-Gaussian systems (Murphy, 2012) to obtain the desired result.
D.1. Computing Approximate Posterior Marginals
Observe that, as with any f t from the training data, the marginal distribution over some f ∗ t under the approximate posterior only involves ¯ u t as f ∗ t ⊥⊥ ¯ u \ t | ¯ u t :
[Equation]
where
[Equation]
Performing smoothing in the approximate model provides ˆ m ¯ u t and [ˆΛ ∗ ¯ u t ] − 1 , from which the optimal approximate posterior marginals are straightforwardly obtained via the above.
D.2. Compute the Saturated Bound
Recall the standard saturated bound introduced by Titsias (2009), that is obtained at the optimal approximate posterior:
[Equation]
To see how this can be computed eﬃciently using the approximate model, ﬁrst note that the approximate model deﬁnition in 60 implies that
[Equation]
which is precisely the ﬁrst term of Eq.
The second equality follows by noticing that the action of the linear maps C fu Λ u H u and C f ¯ u Λ ¯ u are equal, which follows from the conditional independence property in App.
A.4 and the block-diagonal structure of C f ¯ u Λ ¯ u derived in
App.
A.3.
This means that to compute the ﬁrst term of the optimal ELBO is simply the log marginal likelihood of the approximate model, which can be computed eﬃciently via ﬁltering.
By application of the same two properties
[Equation]
the trace term can be written as
T (cid:88)
[Equation]
These quantities are all straightforward to compute by running the approximate model forwards through time and simply computing the marginal statistics
[Equation]
Appendix E. FITC
Consider the approximate model employed by FITC:
[Equation]
We know that in our separable setting, C f ¯ u Λ ¯ u is block-diagonal from App.
A.3.
This means that the t th block on the diagonal of the conditional covariance matrix is
[Equation] [Equation]
and the entire conditional distribution factorises as follows:
[Equation]
By comparing this with equation 5 of (Hartikainen et al., 2011), and letting the likelihood in that equation p ( y k | x k ) =
N ( y k ; [ I N ⊗ H ] x k , S t ), the correspondence is clear.
Appendix F. Inference Under Non-Gaussian Likelihoods
While the optimal approximate posterior over the pseudo-points is not Gaussian, in line with most other approximations (e.g. (Hensman et al., 2015)) we restrict it to be so.
As we have shown that the optimal approximate posterior covariance is block-tridiagonal regardless the likelihood, it follows that the optimal Gaussian approximation must be a Gauss-Markov model.
While in general such a model has a total of T ( DM τ + 2( DM τ ) 2 ) free (variational) parameters, in our case we know that the oﬀ-diagonal blocks of the precision are the same as in the prior, meaning that there are at most T ( M τ + M 2 τ ) free (variational) parameters – this is also clear from Eq.
While one could directly parametrise the precision, this might be inconvenient from the perspective of numerical stability and implementation (standard ﬁltering / smoothing algorithms do not work directly with the precision).
Consequently, it probably makes sense to set up a surrogate model as in line with that discussed by Khan and Lin (2017) and Ashman et al.
Alternatively one could parametrise the ﬁltering distributions directly, from which the posterior marginals could be obtained using standard smoothing algorithms.
Appendix G. Additional Experimental Details
The kernel of the GP used in all experiments is
[Equation]
(70) where κ r is an Exponentiated Quadratic kernel with length scale 0 .
9 and amplitude 0 .
92, and κ τ is a Matern-3/2 kernel with length scale 1 . 2.
The particular values of the length scales / amplitudes are of little importance to the proof-of-concept experiments presented in this work – they were chosen pseudo-randomly.
1010101010−−−−
T
E L
B O / T
M τ =
M τ =
M τ =exact 1010101010−−−−
T
E L
B O / T
M τ =
M τ =
M τ =exact (sde) exact
Appendix H. Sum-Separable Experiments
Similar experiments to those in section Sec. 5 were performed with the sum-separable kernel given by adding two separable kernels of the form in Eq.
(70), although with similar length-scales and amplitudes.
The result are broadly similar, although the state-space approximations and state-space + pseudo-point approximations take a bit longer to run as there are twice as many latent dimensions for a given number of pseudo-points than in the separable model.
As before, these experiments should be thought of purely as a proof of concept.
I f
[Figure]
i
[Figure]
1010101010−−−−−
T
E L
B O / T
M τ =
M τ =
M τ =exact 1010101010−−−−−
T
E L
B O / T
M τ =
M τ =
M τ =exact (sde) exact