A Nonstationary Designer Space-Time Kernel
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In spatial statistics, kriging models are often designed using a stationary covariancestructure; this translation-invariance produces models which have numerous favorable properties.
This assumption can be limiting, though, in circumstances wherethe dynamics of the model have a fundamental asymmetry, such as in modelingphenomena that evolve over time from a ﬁxed initial proﬁle.
We propose a newnonstationary kernel which is only deﬁned over the half-line to incorporate timemore naturally in the modeling process.
Introduction
Covariance kernels for Gaussian random ﬁelds can be deﬁned in a number of ways—see, e.g., [16] or[9].
In many spatial statistics applications in R d , kernels are often radial : the covariance betweenrandom ﬁeld values at two points x , z ∈ R d is deﬁned through a sense of distance between those twopoints [20].
Radial covariance kernels (also commonly referred to as radial basis functions) have numerousadvantageous properties, such as computational efﬁciency and Bochner’s theorem to prove positivedeﬁniteness [7, 18].
However, they also yield kriging models which are stationary, a proprerty whichmay not be ideal in all circumstances.
Some work has been done to try to build nonstationary modelsfrom stationary kernels (see, e.g., [15] or [13]).
Another strategy to permit anisotropic models is to instead use anisotropic kernels, e.g., polynomialkernels [19].
Strategies such as power series kernels [22], Green’s kernels [10] and kernels derivedfrom weighted Sobolev spaces [5] can frequently (though not necessarily) yield aniosotropic kernels.
Additionally, complicated (possibly nonstationary) kernels can be constructed from simpler (possiblystationary) kernels through addition or multiplication (see [2] for a thorough discussion and [14] forrecent applications).
One of the beneﬁts of designing special kernels through strategies such as weighted/generalized
Sobolev spaces is to create kernels which satisfy certain properties such as smoothness, locality orboundary behavior.
In this paper, we adapt a strategy for designing kernels in a bounded domain todesign a kernel on the half-line.
We explore the properties of this kernel as well as its viability forcontributing to modeling space-time phenomena.
Mercer series kernels
Given a compact domain Ω ⊆ R d , Mercer’s theorem says that all positive deﬁnite kernels (i.e.,covariance kernels) K : Ω × Ω → R have the series representation
[Equation]
where λ n , ϕ n are the eigenvalues and eigenfunctions of the integral eigenvalue problem with theassociated orthonormality condition
[Equation]
for some weight function ρ : Ω → R + such that (cid:82) Ω ρ ( x ) d x = 1 .
For K to be positive deﬁnite,λ n > 0 .
This kernel decomposition is often used theoretically to invoke the “kernel trick” for support vectormachines [18].
In [16], eigenfunctions for the popular squared exponential (Gaussian) kernel werediscussed.
Later, in [3] and [9, Chapter 3.9] new nonstationary kernels were designed using explicitlydeﬁned eigenfunctions identiﬁed through a differential equation and desired orthogonality properties,respectively.
A kernel deﬁned on the half-line
In this section, we design a kernel for which Ω = [0 , ∞ ) ; this will match the structure of mostmodeling problems in time, where there is a distinct starting time for observing data, but no distinctend.
Note the use of t in place of x because we will primarily apply this kernel to problems in time.
3.1 Eigenfunctions from the Laguerre polynomials
We adapt the concept from [9, Chapter 3.9], but we replace the Chebyshev with the Laguerrepolynomials and their associated orthogonality condition
[Equation]
for α > − 1 .
Analogously to the Gaussian eigenfunctions from [8], we choose the eigenfunctions to have the form
[Equation]
e
[Equation]
n ( t ) , for 0 < δ < 12 and a normalization constant γ n .
We seek to enforce the orthogonality in (2) utilizingthe identity in (3); this suggests the weight function
[Equation]
which satisﬁes (cid:82) ∞ 0 ρ ( t ) d t = 1 .
Using this we can identify γ n from(cid:90) ∞
[Equation]
3.2 Forming a kernel from the eigenfunctions
The Mercer series (1) requires both eigenfunctions ϕ n and eigenvalues λ n .
As was the case in [9,
Chapter 3.9], we have freedom in how we choose the eigenvalues; these choices will affect thesmoothness of the resulting kernel.
Here, we choose the eigenvalues
[Equation]
which satisﬁes the positive deﬁnite criteria that λ n > 0 and (cid:80) ∞ n =0 λ n = 1 < ∞ .
Because theseeigenvalues decay geometrically, we expect K ( · , s ) ∈ C ∞ (Ω) [17].
At this point, the kernel is fully speciﬁed by (1); however, it is possible to deﬁne a closed form forthese speciﬁc eigenvalues, so that the inﬁnite series need not be approximated.
We can utilize thegenerating function of the Laguerre polynomials to write the Hardy–Hille formula [21]∞
[Equation]
where I α is a modiﬁed Bessel function of the ﬁrst kind [6, Chapter 10.25].
The Mercer series has theform
[Equation]
so substituting in (4) gives the kernel in its closed form
[Equation]
3.3 Computing the kernel safely
The computation of the kernel in (5) may be complicated by the presence of indeterminate formsand competing terms of exponential magnitude.
The latter complication can be alleviated by insteadcomputing exp(log( K ( t, s ))) ; log( K ( t, s )) can be safely computed by potentially using Stirling’sapproximation [6, Chapter 5.11] to compute log(Γ( α + 1)) and leveraging the decomposition
I α ( u ) =
e u ˜ I
α ( u ) where ˜
I α ( u ) is bounded and can be computed safely [1].
We can resolve indeterminate forms by utilizing the asymptotic forms of I α from [6, Chapter 10.30],
I α
[Equation]
[Equation]
u
[Equation]
∼
[Equation]
πu as u → ∞ , to consider the behavior of K ( · , s ) for a ﬁxed s ∈ [0 , ∞ ) , [Equation] as t → 0 , (6a)
[Equation]
as t → ∞ .
(6b)
3.4 Free parameters and their interpretations/implications
As is the case with all positive deﬁnite kernels, this new kernel has free parameters which characterizeits behavior: α > − 1 , 0 < δ < 1 / 2 , and 0 < ω < 1 .
The interplay between these parameters isnontrivial.
We state here some initial analyses which we hope lay the foundation for future work.
The behavior of the kernel is easiest to analyze when the argument of I α is large and (6b) isappropriate.
This can be true for both t → ∞ or for any t with large enough s , i.e., when observeddata is sufﬁciently far from the origin.
Some insights follow below (encapsulated somewhat in Figure1).
• The exponential term guides the long-distance behavior.
Its impact varies based on the
relationship between δ and √ ω/ (1 + √ ω ) .
Regardless, of the value of δ , the ﬁrst term inthe exponential dominates the second for any ﬁxed s as t → ∞ .
implying that the kernelalways goes to 0 at ∞ .
• Similarly, the behavior of the tsω component depends on α being greater than/less than/equal to -1/2.
For δ (cid:54) = √ ω/ (1 + √ ω ) , the t → ∞ behavior of the kernel is dominated by the
exponential term and α plays only a scaling role.
• Setting α = − 1 / 2 and δ = √ ω/ (1 + √ ω ) yields a kernel which has only a √ t − √ scontribution; this implies that lim s →∞ max t ≥
[Equation]
[Equation]
• The ω term acts as a sort of inverse length scale (or shape parameter) for the kernel.
Theinterpretation of it as a length scale is complicated for s near the origin, but more obvious ass grows (for observed results far from the origin)
Example: Modeling west coast temperature data
The kernel in (5) must be supplemented by components meant to model spatial phenomena.
One facilitating this a product kernel of separate time and space
[Figure]
components (discussed in [4] and [11]).
In this section, we consider a tensor product between a
Gaussian K g in space (though any other kernel, such as a Matérn kernel, could be used instead)
andthe kernel (5), denoted here as K (cid:96) , in time
[Equation]
Our example comes from the GEFS Reforecast dataset [12]; in particular, we use daily measurementsfor eight days of surface temperature data across the western United States with 1 ◦ × 1 ◦ resolution.
After training on seven days, we compute the Gaussian random ﬁeld posterior mean on the eighthday.
We use this as a prediction at each of the 28 × 29 grid points and assess the root mean squarederror (RMSE) of that prediction.
This amounts to 5684 training points and 812 test points.
We explore the implications of varying the free parameters α , ω and δ in Figure 2.
To simplify theanalysis, the Gaussian kernel component has a single ﬁxed length scale in both dimensions of . 01 .
We also use a ﬁxed noise variance term (Tikhonov regularization parameter) of 10 − 8 , primarily toavoid ill-conditioning.
In principle, these quantities should be determined from the data.
[Figure]
For ω too small, ill-conditioning plays a confounding role that requires further consideration.
Figure 2 suggests that for many values of ω the predictive power is unaffected by a wide range of αvalues.
Similarly, though to a lesser extent, δ appears to have a limited and somewhat predictableeffect on performance provided ω is reasonably large.
Future Work
This initial analysis suggests that there is an opportunity for this new kernel to help in ﬁtting andpredicting space-time data.
The free parameters provide interpretable ﬂexibility in modeling, but thiscomes at the cost of choosing them appropriately.
Doing so will be a fundamental component offuture research on the viability of this tool.
Of particular interest will be understanding how the nonstationarity and asymmetry of the kernelmanifests itself in predictions at future times.
Initial experimentation, including some omitted forspace considerations, suggests a nontrivial interplay between δ and ω ; furthermore, this relationshipseems to vary as observed data (and, thus, the desired region of prediction) advances from s = 0 .
Onefuture goal is to identify how existing prediction strategies, including cross-validation and maximumlikelihood estimation, can be effectively adapted for this kernel.