Spectral Diffusion Processes
Abstract
Score-based generative modelling (SGM) has proven to be a very effective method for modelling densities on finite-dimensional spaces.
In this work we propose to extend this methodology to learn generative models over functional spaces.
To do so, we represent functional data in spectral space to dissociate the stochastic part of the processes from their space-time part.
Using dimensionality reduction techniques we then sample from their stochastic component using finite dimensional SGM.
We demonstrate our method’s effectiveness for modelling various multimodal datasets
Introduction
Score-based Generative Models (SGMs) (Song and Ermon, 2019; Song et al., 2021;
Ho et al., 2020;
Dhariwal and Nichol, 2021) are a powerful class of generative models.
Noise is progressively added to data following a Stochastic Differential Equation (SDE)—the forward noising process—until it is approximately Gaussian.
The generative model is given by an approximation of the associated time-reversed process called the backward denoising process.
This process is an SDE whose drift depends on the gradient of the logarithm of the densities of the forward process.
The drift is estimated by leveraging techniques from deep learning and score matching (Hyvärinen, 2005; Vincent, 2011).
Probabilistic modelling over functional spaces is of high importance in the physical sciences where objects of interest are often fields , mapping (space-time) inputs to tensors like scalars or vectors.
Aleatoric and epistemic uncertainties motivate for a probabilistic treatment of these phenomena, like weather forecasting (Ravuri et al., 2021) or electric potential prediction (Yang et al., 2020).
A well-established framework to model distributions over functions are Gaussian processes (GPs) (Rasmussen, 2003).
To alleviate their scalability and flexibility limitations, neural processes (NPs) have been introduced and successfully applied on a wide variety of probabilistic prediction tasks.
Yet, their Gaussian likelihood assumption limits their flexibility.
is performed in a spectral space
In this work, we circumvent this problem by casting the modelling problem in a spectral space.
Doing so, we decouple the stochastic part of the process from its space-time part.
This approach turns an uncountable dimensional problem into a countable dimensional one, which is turned into a finite dimensional one by truncation.
Finally, we deal with this last problem using standard SGM techniques.
Our methodology is illustrated in Fig.
1 and presented in more detail in Sec. 3.
Background
Score-based generative modelling.
We briefly recall the concepts behind SGMs on the Euclidean space R d .
Let ( Y t ) t ≥ 0 be a noising process defined by the following SDE
[Equation]
with ( B t ) t ≥ 0 a d -dimensional Brownian motion and p 0 the data distribution on R d .
This process converges towards the unit Gaussian distribution N(0 , Id) .
Under conditions on p 0 , the time-reversal ( ¯ Y t ) t ∈ [0 ,T ] =
( Y T − t ) t ∈ [0 ,T ] is given by (Cattiaux et al., 2021; Haussmann and Pardoux, 1986)
[Equation]
with p t the density of Y t .
Denoising diffusion models are defined to sample approximately from (1).
First, we replace p T by N(0 , Id) .
Then ∇ log p t is approximated by a parametric function s θ ( t, · ) so as to minimise ℓ t ( s ) =
E [ ∥ s θ ( t, Y t ) − ∇ y t log p t |
( Y t | Y 0 )
∥ 2 ] (Hyvärinen, 2005).
Finally, an
Euler–Maruyama discretisation of Eq.
(1) is considered using a discretisation step γ and N = T/γ ¯ Y n
[Equation]
=
[Equation]
Stochastic processes.
Here we are interested in modelling distributions over functions.
Given a compact input space X , a R d -valued stochastic process ( Y x ) x ∈X is a collection of random variables Y x 5 .
Most existing methods which model stochastic processes rely on the parametrisation of finite dimensional marginals { Y x
i : i ∈ { 1 , . . . , n } , x i ∈ X} for every n ∈ N .
However, it is not clear a priori that a collection of probability distributions S =
{ π x 1 ,...,x n : ( x 1 , . . .
, x n ) ∈
X n , n ∈ N } defines a distribution π ∈ P (( R d ) X ) 6 .
This is the focus of the Kolmogorov extension theorem (Charalambos and Aliprantis, 2013, Thm 15.26) which ensures the existence of such a distribution under a set of conditions.
The conditions required of the system are exchangeability and consistency.
A system S is exchangeable , if for any n ∈ N , n -permutation σ ∈ S n , x 1 , . . .
, x n ∈ X and continuous and bounded function f ∈ C b (( R d ) n ) , we have (cid:82) f ( y σ − 1 (1) , . . .
, y σ − 1 ( n ) )
d π x σ (1) ,...,x σ ( n ) ( y 1 , . . .
, y n )
= (cid:82) f
( y 1 , . . .
y n )d π x 1 ,...,x n ( y 1 , . .
y n ) .
A system S is consistent , if for any n 1 ≤ n 2 , x 1 , . . .
, x n 2 ∈ X and f ∈ C b (( R d ) n 1 ) we have (cid:82) f
( y 1 , . . .
, y n 1 )d π x 1 ,...,x n 1 ( y 1 , . .
, y n 1 )
= (cid:82) f
( y 1 , . . .
, y n 1 )d π x 1 ,...,x n 2 ( y 1 , . .
, y n 2 ) .
Existing generative modelling approaches often satisfy the exchangeability criterion but fall short on the consistency.
In this work we aim at bridging this gap by developing a spectral method which immediately yields the consistency of the obtained system.
Spectral Score-Based Processes
Karhunen-Loève theorem.
In order to derive a consistent method, our key insight is to dissociate the stochastic part of the process from its space-time component using a spectral decomposition.
This approach stems from the Karhunen-Loève theorem which we recall below.
Theorem 1.
Let X be a compact space and ( Y x ) x ∈X a continuous stochastic process such that for any x ∈ X , E
[ Y x ] = 0 and E
[ ∥ Y x ∥ 2 ] < + ∞ .
For any x 1 , x 2 ∈ X , we denote the covariance kernel as K Y ( x 1 , x 2 ) =
E [ ⟨ Y x 1 , Y x 2 ⟩ ] and define a L 2 ( X ) operator T K Y given for any f ∈ L 2 ( X ) by T K Y f ( x 2 ) =
(cid:82) X K Y ( x 1 , x 2 ) f ( x 1 )d x 1 .
Denoting by ( e m ) m ∈ N ∈
(L 2 ( X ))
N , ( λ m ) m ∈ N ∈ R N the eigenfunctions and eigenvalues of T K Y , we have
[Equation]
where for any m ∈ N , Z [Equation] = λ [Equation]
2 m [Equation] to Therefore, a natural approximation of the process ( Y x ) x ∈X 7 is given by F M ≜ (cid:80)
Mm =
λ m 1 /
Z m e m .
In the previous theorem, we dissociated the stochastic part of the process—embedded in { Z m } Mm =0 — from its space-time part—embedded in { e m } Mm =0 .
In fact, even though our approach is originally based on the Karhunen-Loève theorem, in practice we consider arbitrary kernels K (not necessarily based on the process Y ) using Mercer’s theorem (Ferreira and Menegatto, 2009, Thm 1.1).
Proposition 2.
We denote by π M the distribution of ( (cid:80) Mm =
λ m 1 /
Z m e m ( x )) x ∈X and S =
{ π Mx 1 ,...,x n : ( x 1 , . . . , x n ) ∈ X n , n ∈ N } .
Then we have that S is exchangeable and consistent .
In order to sample from ( (cid:80) Mm =
λ m 1 /
Z m e m ( x )) x ∈X one only needs to sample { Z m } Mm =0 , effectively approximating the infinite dimensional modelling problem by a finite-dimensional one.
We are now ready to present our method: Spectral Process Score-Based Generative Modelling (SP-SGM).
Spectral Process Score-Based Generative modelling.
SP-SGM works by first choosing a kernel K and computing the associated system of orthonormal eigenfunctions { e m } m ∈ N —see Appendix B.3.
We assume we have access to a dataset D
= { Y i } Li =1 of functions 8 from the target distribution π .
We set M ∈ N and compute a new M + 1 -dimensional dataset D M =
{{ Z im } Mm =0 : i ∈ { 1 , . . .
, L }} where Z im =
λ − 1 / 2 m (cid:82) X ⟨ Y ix , e m ( x ) ⟩ d x .
Then, we train a M + 1 -dimensional standard SGM using
D M .
The SGM model is initialised with { ¯ Z m, 0 } Mm =0 ∼ N(0 , Id) and outputs { ¯ Z m } Mm =0 which is approximately distributed as { Z m } Mm =0 .
Finally, SP-SGM outputs the functional model ¯ F M = (cid:80) Mm =
λ 1 / 2 m ¯ Z m e m .
We remark that standard SGMs would only be able to deal with finite input space.
Note that ¯ F M, 0 =
(cid:80)
Mm =
λ 1 / 2 m ¯ Z m,
e m is a Gaussian process (GP) and in particular—when K = K Y — it is the closest GP to the target distribution π in the following sense.
Proposition 3.
Let π 0 be the distribution of (cid:80) + ∞ m =
λ 1 / 2 m ¯ Z m, 0 e m and π the target distribution.
Denote GP( X ) the space of Gaussian processes on X and assume that K is the covariance kernel.
Then, π 0 ∈ arg min π GP ∈ GP( X ) KL ( π | π GP ) .
In addition, (cid:80) Mm =
λ 1 / 2 m ¯ Z m,
e m is the projection of (cid:80) + ∞ m =
λ 1 / 2 m ¯ Z m,
e m on the subspace of L 2 ( X ) spanned by { e m } Mm =0 .
Related work
Gaussian processes and the Neural processes family.
One standard and powerful framework to construct distributions over functional spaces are Gaussian processes (Rasmussen, 2003).
However, they scale badly with the number of datapoints and are restricted in their modelling capacity.
These problems can be partially alleviated by using neural processes (Kim et al., 2019; Garnelo et al., 2018b,a; Jha et al., 2022; Louizos et al., 2019; Singh et al., 2019), although they also assume a
Gaussian likelihood and only satisfy predictive consistency, meaning one must condition on a context set to achieve consistency.
Recently, denoising diffusion models have been combined with attention network architectures to build consistent models of stochastic processes (Dutordoir et al., 2022).
Finally, Dupont et al. (2022) model weights of implicit neural representation using SGM.
Spectral approaches.
Another possibility to circumvent the consistency problem is to consider spectral decompositions—such as the Karhunen-Loéve one— which shows that stochastic processes can be represented over a functional basis with random coefficients.
Some prior works have explored this avenue using variational auto-encoders to model these coefficients (Mishra et al., 2020) whereas Lim et al.
(2022) considered energy-based models.
Another related line of work is ‘inter-domain Gaussian processes’ (Lázaro-Gredilla and Figueiras-Vidal, 2009), which rely on inducing variables that are obtained via inducing functions such as elements of a Fourier basis (Hensman et al., 2018).
Spatial structure in score-based generative models A variety of approaches have also been proposed to incorporate spatial correlation in the noising process of finite-dimensional diffusion models leveraging the multiscale structure of data (Jing et al., 2022;
Guth et al., 2022; Ho et al., 2022;
Saharia et al., 2021; Hoogeboom and Salimans, 2022; Rissanen et al., 2022).
Our methodology can also be seen as a principled way to modify the forward dynamics in classical denoising diffusion models.
Indeed by applying the diffusion in the spectral space the destruction process blurs the samples instead of noising them, see Fig.
3 for an illustration.
Therefore, our contribution can be
[Table]
Statistically significant best result is understood in the light of recent advances in generative modelling on cold and soft denoising diffusion models (Daras et al., 2022; Bansal et al., 2022; Hoogeboom and Salimans, 2022).
Experimental results
First, we assess our method’s capacity to model bi-modality by qualitative comparison of samples on the synthetic Quadratic dataset.
Figure 2 shows that our method is capable of expressing bi-modality while NPs are not.
While SGMs alone exhibit multimodality, they only apply to finite input spaces, hence expressing multimodality in stochastic processes is indeed a feature of our approach.
To provide a quantitative assessment we compute the power of a two-sample hypothesis test for functional data (Wynne and Duncan, 2022) which tries to discriminate between samples from the model and the dataset.
Table 1 shows that our method outperforms GPs and NPs.
Then, we illustrate the behaviour of our proposed parametrisation of stochastic processes.
We plot the forward noising process of both SP-SGM (reconstructed in the original image space) and of a standard
SGM in Figure 3.
We see that by performing the diffusion in the spectral space, we capture and incorporate spatial correlations governed by the spatial component of the Karhunen-Loève theorem.
Finally we conduct ablation studies on the MNIST dataset.
We consider the effect of the truncation order M in Figure 5, where we find best performance for intermediate values of M .
We decompose this effect into the reconstruction between spectral and spatial domain, which improves with M , and the quality of the stochastic model in the spectral domain, which decreases with M .
These conflicting objectives are also visualised in Figure 5.
We also assess different kernels in Figure 4.
Qualitatively, eigenfunctions of the covariance kernel capture spatial correlations effectively and thus generates the best samples while the RBF kernel imposes a smoothness which tends to produce blurry samples.
[Equation]
Acknowledgements
We thank the anonymous reviewers for their insightful and constructive comments.
AP and MH are funded through the StatML CDT through grant EP/S023151/1.
AD acknowledges support of the UK Defence Science and Technology Laboratory (Dstl) and Engineering and Physical Research
Council (EPSRC) under grant EP/R013616/1.
This is part of the collaboration between US DOD,
UK MOD and UK EPSRC under the Multidisciplinary University Research Initiative.
AD is also partially supported by the EPSRC grant
{+EP/R034710/
+}
CoSines.
EM is supported by an EPSRC
Prosperity Partnership EP/T005386/1 between Microsoft Research and the University of Cambridge.
A Introduction to the supplementary material
We provide details on the implementation of SP-SGM in Appendix B.
The proof of Propositionis presented in Appendix D. The proof of Proposition 3 is presented in
Appendix E.
The likelihood computation is given in Appendix C. Experimental details are provided in Appendix F.
B SP-SGM in practice B.1 SP-SGM algorithm
Algorithm
Require: T, D , θ 0 , N iter , ε, K, { ( λ m , e m ) }
Mm =0 1: ///
TRAINING /// 2:
Get D M from D ▷ Dataset projection 3: for n ∈ { 0 , . .
, N iter − 1 } do 4:
Get { Y m, 0 } Mm =0 mini-batch from D M 5: t ∼ U ([ ε, T ])
▷ Uniform sampling between ε and T 6:
Y m,t = e − t Y m,
+ (1 − e − 2 t ) 1 / 2 G,
G ∼ N(0 , Id) ▷ Diffuse 7:
Get DSM loss ℓ ( θ n ) ▷ Compute score matching loss 8: θ n +
=
optimiser_update ( θ n , ℓ ( θ n ))
▷ ADAM optimiser step 9: end for 10: θ ⋆ = θ N iter 11: ///
SAMPLING ///
12: { ¯ Y m, 0 } Mm =
∼
N(0 , Id) ▷ Sample from Gaussian distribution 13:
b ⋆θ ( t, y ) =
s θ ⋆ ( T − t, ¯ y ) for any t ∈ [0 , T ] ,
¯ y ∈ R M + ▷ Reverse process drift 14: { ¯ Y m,n } M,Nm =0 ,n =0 Euler-Maruyama with drift b ⋆θ ▷ Approximate reverse diffusion 15: return θ ⋆ , x (cid:55)→ (cid:80) Mm =
λ 1 / 2 m ¯ Y m,N e m ( x )
B.2 Karhunen-Loève coefficients
In this section we show that the random coefficients from the KL basis expansion, given for any m ∈ N by
[Equation]
= ⟨ Y , λ − 1 / 2 m e m ⟩ L
[Equation]
X
[Equation]
=
[Equation]
X
[Equation]
Y
[Equation]
x
[Equation]
[-satisfy E [ Z m ] = 
-]x [Equation] [Equation] −
{+[Equation] [Equation] m e m ( x ) ⟩ d x, satisfy E [ Z m ] = 
+} and furthermore if K = K Y then E [ Z m Z m ′ ]
= δ m,m ′ .
First, we have that for any m ∈ N
E [ Z m ] = (cid:82) X ⟨ E
[ Y ( x ) − µ ( x )] , λ − 1 / 2 m e m ( x ) ⟩ d x = 0 .
Then, we have for any m, m ′ ∈ N E [ Z m Z m ′ ]
[Equation]
E [
[Equation]
X
[Equation]
B.3 Computation of eigenfunctions
We focus on the case where Y takes values in R but the computations can be extended to the case where Y takes values in R d with d ∈ N .
Recall from Theorem 1 the L 2 ( X ) operator T K associated with kernel K , given by T K f ( x 2 )
=
(cid:82) X K( x 1 , x 2 ) f
( x 1 )d x 1 .
In the following we describe how we approximate the orthonormal system of eigenfunctions { e m } m ∈ N of T K from a dataset
D =
{ Y i } Li =1 .
Firstly, we note that in practice we have access to the functions in our dataset at a finite set of query points, that is D
= {{ Y ix i,n } N i n =1 } Li =1 .
For ease of notation, in the following we denote { x s } Ss =1 = { x i,n : i ∈ { 1 , . . .
, L } , n ∈ { 1 , . . .
N i }} , with s = (cid:80)
Li =1 N i .
The eigensystem { e m } m ∈ N satisfies T K e m ( x ) = λ m e m ( x ) for all x ∈ X , which we approximate by
[Equation]
Evaluating the above at the entire set of evaluation points x = { x s } Ss =1 results in the matrix eigenproblem
[Equation]
where K is an S × S matrix and has entries K i,j = K( x i , x j ) , λ mat m ∈ R is the matrix eigenvalue and u m ∈ R S is the corresponding matrix eigenvector, normalised such that u ⊤ m u m = 1 .
By Baker’s theorem (Baker, 1979, Theorem 3.4), λ mat m → λ m as S → ∞ .
Note that if we take e m ( x s ) = ( u m ) s then we have
[Equation]
where p ( x ) is the density of observation points on X .
So, in order to have an orthonormal eigensystem (cid:82)
X e m ( x ) e m ′ ( x ) p ( x )d x = δ mm ′ we must rescale according to e m ( x s ) =
√ S ( u m ) s .
Finally, rearranging (3) we get
[Equation]
Having obtained an approximate eigensystem as above, it is simple to obtain the spectral dataset
D M = {{ Z i m } M m =0 : i ∈ { 1 , . . .
, L }} using
[Equation]
where µ ( x ) =
E [ Y x ] is the process mean.
B.4 Truncation order
For a given approximate eigensystem { (ˆ λ m , ˆ e m ) } Lm
=1 , we define the truncation order M as
[Equation]
where η ∈ [0 , 1] is a threshold parameter, fixed to η = 0 .
99 for our experimental results on the 1D datasets.
Alternatively, as we do on the MNIST dataset, one can treat the truncation order M as an hyperparameter.
B.5 Eigenvalue weighted objective
We propose the following weighted DSM loss to train the score network s θ : [0 , T ] × R M +
→ R M + :
[Equation]
with ˜ λ
= [ λ 0 / Λ , . . .
, λ M / Λ] ⊤ 9 , Λ = (cid:80) Mm =
λ m , the vector of normalised eigenvalues, α > 0 a tuneable hyperparameter, t ∼ U ([0 , T ]) , Z
= { λ − 1 / 2 m (cid:82) X ⟨ Y x , e m ( x [Equation] ⟩ d x } M m =0 and [Equation]
[Equation]
where t (cid:55)→ β t is an hyperparameter.
The motivation behind (4) is to put more importance on fitting well the lower frequency (i.e. higher eigenvalue) components of the decomposition (cid:80)
Mm =
λ m 1 /
Z m e m ( x ) since these matter the most in terms of quality of reconstruction as per
Theorem 1.
With α = 0 , we have that λ α = 1 , thus that all dimensions of the spectral space are weighted equally and we recover the standard DSM loss.
C Likelihood evaluation
Let’s denote the Karhunen-Loeve expansion with ϕ : R M +
→ R d such that ( Z 1 , . . .
, Z M ) (cid:55)→ µ ( · ) + (cid:80) Mm =
Z m √ λ m e m ( · ) .
We have that ∂ϕ∂Z m = √
λ m e m , hence with J z ϕ
= (cid:16) ∂ϕ∂z 1 , . .
. , ∂ϕ∂z N (cid:17) we get J z
ϕ
⊤ J z ϕ
= (cid:0) √ λ m √ λ n
⟨ e m , e n ⟩ (cid:1)
mn = (cid:0) √ λ m √ λ n δ mn (cid:1)
mn = diag ( λ m ) .
Hence, denoting a realisation F = ϕ ( Z ) , applying the change of variable formula we have
[Equation]
D Proof of Proposition
We recall Proposition 2.
Proposition 2.
We denote by π M the distribution of ( (cid:80) Mm =
λ m 1 /
Z m e m ( x )) x ∈X and S =
{ π Mx 1 ,...,x n : ( x 1 , . . . , x n ) ∈ X n , n ∈ N } .
Then we have that S is exchangeable and consistent .
Proof.
Note that this result is the trivial direction of the Kolmogorov extension theorem (Charalambos and Aliprantis, 2013, Theorem 15.26) since F M is by definition a stochastic process.
For completeness
[Equation]
we provide a proof of this result in our setting.
We denote Z =
{ Z m } Mm =0 and π the model distribution on Z .
For any n ∈ N and f ∈ C b (( R d ) n , R ) we have
[Equation]
Hence
[Equation]
for any n ∈ N
[Equation]
n -permutation σ ∈ S n [Equation] x [Equation]
[Equation]
x n ∈ X
[Equation]
and continuous and bounded function f ∈ C b (( R d ) n ) we have
[Equation]
, y
[Equation]
=
[Equation]
[Equation]
n
[Equation]
y
[Equation]
[Equation] , y n [Equation] Similarly, for any n 1 ≤ n 2 , x 1 , . . .
, x n 2 ∈ X and f ∈ C b (( R d ) n 1 ) we have
[Equation]
In this setting, the random variable Z is finite dimensional as it is supported in R M [Equation] but more generally this result is still true with an (infinite dimensional) process F .
E Proof of Proposition
In this section, we provide two proofs of Proposition 3.
E.1 Partition definition
We recall Proposition 3.
Proposition 4.
Let π 0 be the distribution of (cid:80) + ∞ m =
λ 1 / 2 m ¯ Z m, 0 e m and π the target distribution.
Denote GP( X ) the space of Gaussian processes on X and assume that K is the covariance kernel.
Then, π 0 ∈ arg min π GP ∈ GP( X ) KL ( π | π GP ) .
In addition, (cid:80) Mm =
λ 1 / 2 m ¯ Z m,
e m is the projection of (cid:80) + ∞ m =
λ 1 / 2 m ¯ Z m,
e m on the subspace of L 2 ( X ) spanned by { e m } Mm =0 .
Note that in this first approach we do not assume any regularity on the samples of the process.
This proof is based on (Sun et al., 2019, Theorem 1) which considered a modified definition of the
Kullback-Leibler divergence.
Namely, for any µ, ν probability measures over a probability space (Ω , F ) we define
[Equation]
where for any measurable finite partition P = { Ω 1 , . .
, Ω N } we define µ P = { µ (Ω 1 ) , . . .
, µ (Ω N ) } , similarly for ν P .
With that definition (Sun et al., 2019, Theorem 1) holds.
Namely, for any µ, ν ∈ P (( R d ) X ) 10 , we have
[Equation]
Let π be the target distribution of the stochastic process.
We define GP( X ) the space of Gaussian processes over the input space X .
We have that
[Equation]
In addition
[Equation]
we have
[Equation]
where we emphasise that the set GP( { x 1 , . .
, x n } ) is simply the set of nd -dimensional Gaussian probability measures.
Hence, for any { x 1 , . . .
, x n } ∈ X n and n ∈ N
[Equation]
we have that
[Equation]
) } , where π 0 ,⋆x 1 ,...,x n is the Gaussian measure with same mean and covariance matrix as π x 1 ,...,x n .
Using the Kolmogorov extension theorem (Charalambos and Aliprantis, 2013, Theorem 15.26), there exists π ⋆ ∈ P (( R d ) X ) such that for any { x 1 , . . .
, x n } ∈ X n and n ∈ N , π ⋆x 1 ,...,x n = π 0 ,⋆x 1 ,...,x n .
Therefore, we have that
[Equation]
Hence, using (6) we get that
[Equation]
Therefore [Equation] we have that
[Equation]
which implies using (6) that
[Equation]
The equality holds since π ⋆ ∈ GP( X ) .
Finally, since π ⋆ and π have the same covariance kernels, they have the same Karhunen-Loève eigensystems.
Therefore, there exists { Z m } m ∈ N such that
Y ⋆ = (cid:80) m ∈ N λ 1 / 2 m Z m e m has distribution π ⋆ , which concludes the proof.
E.2 Sample continuous
In our second approach, we restrict ourselves to the case of sample continuous processes.
Namely, we now longer consider µ ∈ P (( R d ) X ) but µ ∈ P (C( X , R d )) .
Proposition 5.
Assume that π ∈ C
( X , R d ) and that there exists ϕ : [0 , + ∞ )
such that for any x 1 , x 2 ∈ X ,
[Equation]
, such that (cid:82) + ∞ϕ (exp[ − t 2 ])d t < + ∞ .
Let π 0 be the distribution of (cid:80) + ∞ m =
λ 1 / 2 m ¯ Z m, 0 e m .
Denote GP( X ) the space of Gaussian processes on X and assume that K is the covariance kernel.
Then, π 0 ∈ arg min π GP ∈ GP( X ) KL ( π | π GP ) .
In addition, (cid:80) Mm =
λ 1 / 2 m ¯ Z m,
e m is the projection of (cid:80) + ∞ m =
λ 1 / 2 m ¯ Z m,
e m on the subspace of L 2 ( X ) spanned by { e m } Mm =0 .
In that case, we show that
[Equation]
More precisely, we have the following proposition.
Proposition 6.
Let µ, ν ∈ P (C( X , R d ))
[Equation]
| ν x 1 ,...,x n ) : ( x 1 , . . .
, x n ) ∈ X n , n ∈ N } .
Proof.
First, note that for any { x 1 , . . .
, x n } ∈ X n , n ∈ N we have using the data processing theorem (Ambrosio et al., 2008, Lemma 9.4.5)
[Equation]
Since X is compact, for any n ∈ N , there exists { x 1 , . . .
, x n } such that X ⊂ ∪ nk =1 B( x k , 1 / ( n + 1)) .
For any n ∈ N , let { φ k } nk =1 be a smooth partition of unity associated with { B( x k , 1 / ( n + 1)) } nk =1 .
For any C( X , R d ) -valued random variable Y and n ∈ N , denote Y n such that for any x ∈ X
[Equation]
Since Y is continuous and X is compact we have that Y is uniformly continuous.
Hence, for any ε > 0 , there exists n ∈ N such that for any x 1 , x 2 , ∥ x 1 − x 2 ∥ ≤ 1 / ( n + 1) , ∥ Y x 1 − Y x 2 ∥ ≤ ε .
Therefore, we have that
[Equation]
Therefore, we have that lim n → + ∞ sup x ∈X ∥ Y
nx − Y x ∥ = 0 .
Therefore, for any n ∈ N , denoting µ n the distribution of Y n , we get that ( µ n ) n ∈ N converges to µ in P (C( X , R d )) .
For any n ∈ N , let a = { a k } nk =1 ∈ ( R d ) n and f a n : X → R d such that for any x ∈ X ,
f a n ( x ) =
(cid:80) nk
=1 a k φ k ( x ) .
Denote C n ( X , R d )
= { f a n : a ∈ ( R d ) n } .
Define φ n : ( R d ) n → C n ( X , R d ) such that for any a ∈ ( R d ) n , φ n ( a ) =
f a n .
We have that φ n is a bijection.
In addition, we have that ( φ n ) # µ n = µ x 1 ,...,x n .
Therefore, using the data processing inequality we have that for any n ∈ N
[Equation]
In addition, since ( µ n , ν n ) → ( µ, ν ) we using that the Kullback-Leibler divergence is lower semicontinuous (Dupuis and Ellis, 2011, Lemma 1.4.3)
[Equation]
Therefore, we get that
[Equation]
Combining this result and (7), we conclude the proof.
The rest of the proof is similar to Appendix E.1, expect that one needs to check that the obtained
Gaussian process π ⋆ is sample continuous (up to a modification).
This is done using that there exists ϕ : [0 , + ∞ ) such that for any x 1 , x 2 ∈ X ,
[Equation]
such that (cid:82) + ∞ϕ (exp[ − t 2 ])d t < + ∞ , see Fernique (1975).
F Experimental details
In the following sections we provide details on our experimental procedures.
F.1 1D Datasets
Quadratic A synthetic dataset constructed to exhibit clear bi-modality.
Samples consist of function evaluations of f ( x ) :
= ax 2 + ϵ with a ∼ Unif ( {− 1 , 1 } ) and ϵ ∼ N (0 , 1) on a uniformly sampled grid of 100 evaluation points in [ − 10 , 10] .
We allow unlimited samples during training, validation and testing phases.
Melbourne A real dataset recording the number of pedestrians on streets in Melbourne, each sample being a period of 24 hours with readings taken hourly (giving a grid of 24 evaluation points per sample).
The dataset is sourced from http://www .
timeseriesclassification .
com/ description .
php?Dataset=
MelbournePedestrian .
We pre-processed the dataset by removing rows with unobserved values and rescaling the dataset to have unit variance.
The pre-processed dataset contains 2319 samples and we split the dataset in the ratio of [0 . 8 , 0 . 1 , 0 . 1] for training, validation and testing respectively.
Gridwatch A real dataset recording the energy demand of the UK grid, each sample being a period of 24 hours with readings taken every five minutes (giving a grid of 288 evaluation points per sample).
The dataset is sourced from https://www .
gridwatch .
templar .
co . uk/download .
php by selecting the ‘demand’ field.
We performed a number of pre-processing steps:
• Select days where the readings were taken exactly on the fifth minute, • Remove days where the the difference between two consecutive readings was beyond the 99.5 th percentile of differences, • Remove days where consecutive readings did not vary for half an hour or more, • Centered the dataset so each sample has zero mean, • Scaled the dataset so each sample has unit variance.
This resulted in 1013 samples which we split in the ratio of [0 . 8 , 0 . 1 , 0 . 1] for training, validation and testing respectively.
F.2 Baselines
Gaussian Processes We used a Matern-1/2 kernel with lengthscale and noise variance learnt by maximising the marginal likelihood.
Neural Processes We modified a PyTorch implementation of Neural Proccesses available online at https://github .
com/EmilienDupont/neural-processes .
Our modifications adapted the implementation to unconditional training/sampling by implementing Equation 7 of Garnelo et al. (2018b).
We set the dimension of the representation of context points and the dimension of the latent variable to be 512 (1024 for Gridwatch dataset).
The encoder and decoder were three-layered FCNs with 512 (1024 for Gridwatch dataset) units and ReLU activations, trained to 1000 epochs.
F.3 Functional MMD and two-sample test power
Our quantitative results are based off the functional MMD and kernel two-sample hypothesis test of
Wynne and Duncan (2022).
The hypothesis test considered uses the null hypothesis that samples from the model and dataset are from the same distribution, while the alternative states that they are not.
We know that the null is false as the distribution of samples from the model will never truly match the data distribution.
Therefore, we expect a perfectly powerful test to always reject the null hypothesis.
By choosing a specific test with finite power (for example by restricting the number of samples used when computing the test statistic), we can compare the quality of samples from different models by computing the power of the specific test on each of the sets of samples.
If the test exhibits a lower power on a set of samples, it indicates those samples are harder to distinguish from the data distribution and thus can be considered of higher quality.
The specific test we choose is the kernel two-sample test with the ID kernel computed on 10 samples.
We perform 1000 tests to estimate the power.
Confidence intervals are obtained by repeating the procedure on models trained from different seeds.
F.4 Uncurated samples
We visualise samples from all models on the 1D datasets and from SP-SGM on MNIST.
Dataset
SP-SGM
NP
GP
F.5 Further ablation studies
M =
M = M =