D ECOUPLED K ERNEL N EURAL P ROCESSES : N EURAL N ETWORK -P ARAMETERIZED S TOCHASTIC P ROCESSES USING E XPLICIT D ATA - DRIVEN K ERNEL
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural Processes (NPs) are a class of stochastic processes parametrized by neural networks.
Unlike traditional stochastic processes ( e.g. , Gaussian processes), which require specifying explicit kernel functions, NPs implicitly learn kernel functions appropriate for a given task through observed data.
While this data-driven learning of stochastic processes has been shown to model various types of data, the current
NPs’ implicit treatment of the mean and the covariance of the output variables limits its full potential when the underlying distribution of the given data is highly complex.
To address this, we introduce a new class of neural stochastic processes, Decoupled Kernel Neural Processes (DKNPs), which explicitly learn a separate mean and kernel function to directly model the covariance between output variables in a data-driven manner.
By estimating kernel functions with cross-attentive neural networks, DKNPs demonstrate improved uncertainty estimation in terms of conditional likelihood and diversity in generated samples in 1-D and 2-D regression tasks, compared to other concurrent NP variants.
Also, maintaining explicit kernel functions, a key component of stochastic processes, allows the model to reveal a deeper understanding of underlying distributions.
I NTRODUCTION
Neural processes (NPs) (Garnelo et al., 2018a;b) are a class of stochastic processes parametrized by neural networks.
By embracing statistical properties in stochastic processes, NPs can effectively estimate the uncertainty of underlying distributions of functions with a set of realizations and their data points.
Different from traditional stochastic processes ( e.g. , Gaussian processes (GP) (Rasmussen & Williams, 2006)), NPs learn data-driven stochastic processes without a need to specify or keep an explicit form of kernel functions.
As a result of their simplicity and flexibility, there have been numerous efforts to further develop improved variants of NPs (Kim et al., 2019; Lee et al., 2020;
Gordon et al., 2020) and apply them to various downstream tasks (Singh et al., 2019; Requeima et al., 2019).
Though significant progress has been made in NPs, the current architectures of NPs either fails to capture output dependencies as in Conditional NPs (Garnelo et al., 2018a; Gordon et al., 2020), or indirectly capture the full stochasticity present in the traditional stochastic processes.
For instance, different from GPs, conventional NPs reserve stochasticity in a global latent variable and output variables separately.
The output variables estimate point-wise uncertainty, which corresponds to the diagonal elements of a kernel matrix.
Similarly, the global latent variable takes charge of the functional uncertainty and diversity, represented by the full covariance matrix in GPs.
Due to this inductive bias of conventional NPs, the role of estimating functional stochasticity is mainly assigned to a fixed-length vector ( i.e. the global latent variable), and consequently, capturing the underlying distributions can be restricted in complex scenarios ( e.g. , variable relationships are periodic or abruptly changing at a certain point).
Although several approaches attempt to alleviate the problem by introducing attention (Kim et al., 2019) and bootstrapping (Lee et al., 2020) on top of conventional NPs, the problem still exists as the architectural limitation ( i.e. implicit modeling of the mean and covariance) has not been addressed directly.
Besides this, as NPs implicitly learn the kernel functions inside the model, the interpretability of kernels such as in GPs (Lloyd et al., 2014) is diminished.
[Figure]
To address this concern, we propose Decoupled Kernel Neural Processes (DKNPs), a new class of neural stochastic processes that explicitly learn a separate mean and kernel function to directly model the covariance between output variables in a data-driven manner.
Our experiments in 1-D and 2-D regression tasks reveal that the DKNP outperforms concurrent NP variants in terms of predictive likelihood, better global coherence of generated samples, and improved interpretability via explicitly learned kernels.
B ACKGROUND 2. N EURAL P ROCESS
Given a stochastic process sample consisting of n points, let us denote the input and output as
X = { x i } ni =1 and Y
=
{ y i } ni
=1 , respectively, where x i ∈ R
d x and y i ∈ R d y .
For a set of target input X T = { x i } i ∈ T ⊂ X , NPs model the conditional distribution of target outputs Y T conditioned on the context set ( X C , Y C )
=
{ ( x i , y i ) } i ∈ C using a factorized Gaussian distribution: [Equation] i
[Equation]
For obtaining the predictive distribution log p
( y i | x i , X C , Y C ) , NPs use an encoder-decoder architecture that ensures the permutation invariance of the predictions of the target points given the context set ( X C , Y C ) . Following Kim et al.
and Lee et al.
, we consider the NP encoder consisting of two separate paths, namely the deterministic path and the latent path.
For the deterministic path, f θ represents each context points in { ( x i , y i ) } i ∈ C as r i ∈ R
d r , i.e. , r i =
f
θ
( x i , y i ) .
Then, we aggregate the r i ’s by averaging them across all context points, r C = 1 n c (cid:80) i ∈ C r i where n c = | C
| .
This vector r C ∈ R d r is the summarized representation of context points, and it is permutation invariant over the order of ( x i , y i ) ∈ ( X C , Y C ) .
The latent path of the NP encoder operates in a similar fashion to the deterministic path, i.e. , e C = 1 n c (cid:80) i ∈
C e i where e i = g ϕ
([ x i ; y i ]) .
Unlike the deterministic path, however, the latent path uses stochastic layers for obtaining a distribution of the latent variable z ∈ R d z ; q ( z | e C ) =
N ( z ; µ z , σ 2 z ) where µ z , σ z are the output of the additional fully-connected layer applied to e C .
Finally, by concatenating these aggregated vectors r C , z with the target inputs x i ∈ X T , the decoder h ψ produces the predictive distribution p ( y t | x t , r C , z ) =
N ( y ; µ y , σ 2 y ) where ( µ y , σ y ) =
h ψ ( x t , r C , z ) .
Note that the σ y is used for capturing the point-wise uncertainty.
During the training phase, the parameters are learned by maximizing the evidence lower bound of log p
( Y T | X T , X C , Y C ) via the reparametrization trick (Kingma & Welling, 2014; Rezende et al., 2014) as follows:
𝑀𝐿𝑃 𝜃 𝒆 𝑐𝑀𝐿𝑃 𝜙 𝑀𝑢𝑙𝑡𝑖ℎ𝑒𝑎𝑑 𝐶𝑟𝑜𝑠𝑠 −𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝒉 𝑐𝑀𝐿𝑃 𝜙 𝒉 𝑐𝑀𝐿𝑃 𝜙 𝒉 𝑡𝑀𝐿𝑃 𝜙 𝒉 𝑡𝑀𝐿𝑃 𝜓 𝑀𝐿𝑃 𝜓 ෝ𝒚 𝑡ෝ𝒚 𝑡𝑀𝐿𝑃 𝜔 𝑀𝐿𝑃 𝜔 𝒘 𝑡𝒘 𝑡𝝁 𝒚 𝒙 𝑐𝒚 𝑐𝒙 𝑐𝒙 𝑐𝒙 𝑡𝒙 𝑡𝑀𝐿𝑃 𝜃 𝒆 𝑐 1 𝒙 𝑐𝒚 𝑐Σ 𝒚
𝑊 ⋯ 𝑊𝑊 ⊤ : Concat 𝒘 𝑡 1 𝒘 𝑡⋯ 𝒉 𝑡 1 𝒉 𝑐 1 𝒆 𝑐𝑄𝑢𝑒𝑟𝑦 𝐾𝑒𝑦 𝑉𝑎𝑙𝑢𝑒 𝒉 𝑡 2 𝒉 𝑐 2 𝒆 𝑐𝑀𝑢𝑙𝑡𝑖ℎ𝑒𝑎𝑑 𝐶𝑟𝑜𝑠𝑠 −𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝒉 𝑡 1 𝒉 𝑐𝑄𝑢𝑒𝑟𝑦 𝐾𝑒𝑦 𝑉𝑎𝑙𝑢𝑒 𝒉 𝑡 2 𝒉 𝑐𝒉 𝑐𝒉 𝑐
[Equation]
where q ( z | e T )
=
q ( e t | X T , Y T ) is a distribution of latent variable by encoding the set of target points ( X T , Y T )
= { ( x i , y i ) } i ∈ T .
This objective function consists of two parts: 1) the reconstruction loss for the target points and 2) the KL divergence term, minimizing the divergence between two distributions q ( z | X T , Y T ) and q ( z | X C , Y C ) .
Note that, in practice, we assume that X C ⊂ X T during the training phase.
The KL divergence term encourages two distributions inferred by the context sets and target sets to be similar, which is reasonable because two sets are generated from the same function.
Therefore, during the inference phase, the distribution of q ( z | X C , Y C ) captures the functional stochasticity which is demonstrated with the coherent sample generation.
It can be thought that z learns to capture the correlation of output variables of the stochastic processes.
2. A TTENTIVE N EURAL P ROCESSES Multihead Attention
Given n key-value pairs of matrices K ∈ R n × d model and V ∈ R n × d model , and m queries Q ∈ R m × d model , the scaled dot product attention is formulated as:
[Equation]
where K , V , and Q are projected by learnable linear maps W Ks , W Vs , and W Qt from the source S and target T .
The attention mechanism can be calculated from multiple subspaces, namely, multihead attention (MHA) (Vaswani et al., 2017).
Denoting a single attention head as Head i =
Attention ( Q i , K i , V i ) , the aggregate attention from multiple subspaces can be expressed as:
[Equation]
where W O is the learnable linear map for aggretating the subspaces.
Attentive NPs
Attentive NPs (ANPs) leverage attention to resolve the underfitting issue in NPs.
Unlike NPs that produce a single variable r C from the deterministic path, ANPs utilize a queryspecific variable r ∗ i by applying the attention score a i for each r i during the aggregation of the deterministic path, formulated as r ∗ i = (cid:80) i ∈
C a i · r i .
The attention-based aggregation of ANPs resembles how the GPs utilize the correlation to estimate the predictive distribution of the context and the target points.
D ECOUPLED K ERNEL N EURAL P ROCESSES
Decoupled Kernel Neural Processes use attention to explicitly learn a separate mean and kernel function so as to directly model covariances between output variables with related input variables in a data-driven fashion, which is contrary to conventional NPs that implicitly model the mean and kernel function through the latent variable z .
As shown in Figure 2, DKNPs estimate the predictive distribution as multivariate Gaussian N ( y ; µ y , Σ y ) by using two attention-based deterministic paths using Multihead Cross-Attention (MCA) to model the mean vector (upper path) and the full covariance matrix (lower path).
Here, attention modules are extensively utilized as Le et al.
; Kim et al.
(2019) have demonstrated attention was helpful in achieving low predictive uncertainty near the context points.
With the predictive distribution N ( y ; µ y , Σ y ) , DKNPs are trained and evaluated based on the function likelihood.
The core design of DKNPs is motivated by the predictive posterior distribution of GPs, where
X T , X C , Y C are used for deriving the posterior mean, 1 but only X T , X C for deriving the poste- rior covariance as follows:
[Equation]
where, in contrast to GPs, DKNPs’ covariance is learned via attention.
This decoupled process allows DKNPs to explicitly learn the prior of the given dataset, and thus act as a true generative process, ensuring the global consistency of all points in the given stochastic process samples.
Unlike
DKNPs, NPs pack all information ( X C , Y C , X T ) into latent variables to derive both mean and covariance, thus inherently becoming a conditional process that requires a sufficient amount of context points, unable to explicitly learn a prior.
Specifically, the DKNPs pass each context point { ( x i , y i ) } i ∈ C , the concatenation of x i and y i , to
MLP θ and represent it as e i .
Similarly, we produce the representation vector of x i , h i , using another MLP ϕ for all inputs { x i } i ∈ C ∪ T .
Then, e i and h i are passed to the MCA module to create the mean vector µ y .
All heads in the MCA module perform cross-attention, Q = { h i } i ∈ T , K = { h i } i ∈ C , and V = { e i } i ∈ C .
We adopted the architecture of MCA used in image transformers (Parmar et al., 2018), where the original query vectors are added to the output from the MCA through the residual path.
This allows to do inference the output distributions without context points, which can be considered as prior distributions of DKNPs—the learned prior of DKNP.
After the MCA, the last MLP ψ generates the predictive mean vector µ y for each data point.
Intuitively, this can be interpreted as predicting the target mean based on the context and the correlation, which resembles the estimation of the predictive distribution in GPs and ANPs.
Different from the NPs, DKNPs explicitly capture the correlation between the output variables using another multihead cross-attention (MCA), where Q = { h i } i ∈ T , K = { h i } i ∈ C , and V = { e i } i ∈ C .
Then MLP ω produces the representation vector
w i ∈ R dw for each position, which are combined to generate the covariance matrix Σ = WW ⊤ , where W i, : = w i and Σ ij = kernel ( x i , x j ) =
w ⊤ i w j .
One might consider using self-attention to let the model learn the correlation between all data points.
However, the self-attention module on only X as inputs receives no indication of context and target points and therefore fails to reduce the uncertainty near the points that have high confidence ( e.g. , context points).
Also, the interaction between the target points through self-attention does not guarantee consistency under the marginal distribution of target points when the context points are given.
Lastly, it is also important to note that the representation h is shared when modeling both the mean and the covariance. This motivation is drawn from Equation 5 that the calculation of the mean is also based on the kernel matrices, Σ ⊤ X C ,X T and Σ − 1 X C ,X C .
To train the DKNPs, the obtained mean vector µ and the covariance matrix Σ act as parameters of a predictive distribution N ( Y ; µ Y , Σ Y ) .
Instead of maximizing the lower bound of the log-likelihood as in most NP models, the training objective of DKNPs is to maximize the tractable log-likelihood of the Gaussian as follows:
[Equation] [Table]
[Figure]
[Table]
Although the proposed objective function is equivalent to CNP’s (Garnelo et al., 2018a), modeling the correlation between output variables for capturing functional stochasticity shares the same motivation of NPs, thus DKNP being one of NP variants.
E XPERIMENTS
We compare DKNP with diverse NP variants such as Conditional NP (CNP)(Garnelo et al., 2018a),
NP, Bootstrapping NP (BNP)(Lee et al., 2020), ANP, and Bootstrapping ANP (BANP) on both 1D and 2D regression tasks.
For a fair comparison, all NP variants use two paths for context encoding.
CNP, BNP, and BANP have two deterministic paths, and NP and ANP have one deterministic and one stochastic path.
Also, ANP and BANP have additional self-attention in both stochastic and deterministic paths.
Following Lee et al.
(2020), BNP and BANP were trained with 50 bootstrap context samples and tested with 4 samples.
Note that we mostly followed the same hyperparameter setup used in Lee et al.
and the details are described in Appendix A. Unlike the NP variants (except
CNP) which are trained with the lower bound of conditional log-likelihood log p
( Y T | X T , X C , Y C ) or its slight modification, DKNP can directly evaluate it as well as maximize it during training.
All results are also reported with the conditional log-likelihood to evaluate the methods’ ability to model the given stochastic processes.
Following (Le et al., 2018; Lee et al., 2020), we use importance weighting estimation (Burda et al., 2016) with 50 samples to evaluate the performance of NP variants that utilize z .
4. 1D R EGRESSION WITH G AUSSIAN P ROCESSES D ATA
We first conduct basic evaluation of DKNP in comparison to NP variants by testing its ability to model 1D stochastic process samples generated from Gaussian Processes of diverse kernels, namely
RBF, Periodic and Mat´ern 5/2.
In this task, we randomly sampled input x from [-2, 2] and generated y from GP kernels where the hyperparameters were also sampled both during training and testing.
The number of context points n c is sampled from Unif (3 , 25) and the target points is
[Figure]
[Figure]
n c + Unif (3 , 50 − n c ) .
Details about the data generation process (kernel hyperparameters), the training process, and the evaluation process are in Appendix C.1.
The results in Table 1 show that DKNP can better model the given stochastic processes compared to various NP variants in terms of the log-likelihood.
Note that the attention mechanism dramatically increases all methods’ ability to model GP samples, as can be seen by the clear divide between
CNP, NP, BNP and the rest.
The true merit of DKNP, however, is its capability to better estimate the uncertainty of the stochastic processes thanks to its explicit modeling of the full covariance matrix.
Figure 3, for example, clearly demonstrates the distinguishing feature of DKNP compared to previous NP variants, in this case ANP, which recorded the second highest likelihood in Table 1.
In all kernel types, ANP fails to contain the target points within one standard deviation range while DKNP successfully captures the uncertainty in all unobserved points.
Specifically, ANP is over-confident when modeling kernel types RBF and Mat´ern 5/2, where only a local structure exists.
For periodic kernels, where both local and global structure exists, ANP fails to capture the underlying periodicity, due to its architecture that tries to estimate functional stochasticity with a fixed-size global latent variable.
Unlike ANP, however, DKNP can correctly model the whole stochastic process thanks to its ability to explicitly model the correlations between all output variable.
Employing decoupled paths for modeling mean and covariance as in Figure 2, another advantage of
DKNP is improved interpretability, as we can explicitly check the learned prior by visualizing the covariance matrix as described in Section 3.
Such example is depicted in Figure 4, where we can readily compare the learned prior with the ground truth prior for all three kernel types.
Note that
DKNP not only model allows us to visually check the learned prior, but also demonstrate its ability to accurately learn the ground truth kernels for all kernel types.
4. 1D R EGRESSION WITH C HANGE P OINT
Next we test all methods in a more challenging setup, where the underlying dynamics of the stochastic process changes at midpoint.
Specifically, we employ three different GP configurations: Periodic-
P e r i od i c - P e r i od i c
R B F - P e r i o
d i c
[Figure]
[Table]
Periodic, Periodic-RBF, and RBF-RBF.
Each configuration uses one kernel type (with a fixed hyperparamter setup) up to point 0, then another kernel type afterwards.
As the correlations between output variables are more complex in this setup, we expect DKNP to demonstrate even more distinguishing performance than NP variants.
The training and evaluation processes are the same as the previous experiments in Section 4.1.
See Appendix C.1 for data generation details.
The results in Table 2, as expected, show wider gaps between DKNP and NP variants for all three configurations in terms of log-likelihood.
Interestingly, unlike the the previous results (Table 1) where DKNP marginally outperformed NP variants for RBF kernels (but beyond one standard deviation margin), the performance gap is more distinguished even for RBF-RBF configuration in this setup.
This observation indicates DKNP’s ability to better capture complex correlations derived from a combination of rather simpler kernels than NP variants.
Figure 5 demonstrates predictions by ANP and DKNP for all three change point configurations.
Compared to ANP which fails to correctly capture the changing dynamics of the given processes (especially for configurations including periodic kernels), DKNP shows its ability to correctly learn the underlying dynamics before and after the change point.
In Figure 6, we compare the learned priors as well as the generated samples from them to the ground truths priors and samples for two change point configurations, Periodic-Periodic and RBF-Periodic.
Note that DKNP can accurately learn two distinct kernels in both sides of the midpoint, ( i.e. low and high frequency in the PeriodicPeriodic case, smooth and periodic in the RBF-Periodic case) thus able to generate samples that are practically equivalent to the ground truth samples.
4. 2D I MAGE C OMPLETION
The purpose of the 2D image completion task is to test how well each model shows its functional flexibility to learn non-trivial kernels.
Assuming that image data is generated by a stochastic process, the task is to regress pixels that are missing based on the provided pixels ( i.e. context pixels) in the image.
Specifically, we use the CelebA dataset (Liu et al., 2015), which consists of 202,599 number of facial images of celebrities of diverse ethnicities, gender, and age. [Figure]
For both training and evaluation, the number of context points n c is sampled from Unif(3, 200) and the target points is n c + Unif(3, 200 - n c ).
Details about the data generation process, the training process, and the evaluation process are in Appendix C.2.
In Table 3, consistent with the previous results, DKNP shows superior ability in modeling even twodimensional stochastic processes compared to all NP variants.
This is somewhat predictable based on two previous results (Table 1 and Table 2) where the performance gap between DKNP and NP variants was more visible for stochastic processes with complex underlying dynamics (where both local and global structure exist), and the fact that the facial images are likely to follow non-trivial dynamics.
For example, the dynamics to generate the hair or the background, which are typically low frequency signals, would be quite different from the dynamics to generate the details of the face where eyes, nose, and mouth altogether compose higher frequency signals.
And unlike the previous change point data where the boundary between two different dynamics ( i.e. kernels) is a point in a 1D line, the boundary between different kernels in facial images is a (curved) line in a 2D plane ( e.g. hairline).
Therefore we can expect DKNP to significantly outperform all NP variants in estimating the correlations between output variables, as shown in Table 3.
Figure 7 illustrates the mean of the prior distribution learned by DKNP on the CelebA dataset and the predictive mean of the NP variants and DKNP when a single context point is given in the background.
The posterior mean of DKNP demonstrates distinct behavior for the face and background, such that the given context uniformly affects the background while keeping the face practically intact.
This shows that DKNP accurately captures the correlations between output variables by successfully learning kernel functions that behave differently for face and background.
BANP, ANP, and NP, on the other hand, demonstrate unpredictable behaviors where the context point in the background also affects the hair or the clothes, or the background is only partially affected by the given context point.
This behavior reveals that NPs, which implicitly learn the kernel function, have a harder time modeling the accurate correlation between output variables. [Table]
[Table]
We conclude this section with qualitative examples in
Figure 8 that highlight the DKNP’s capability to generate diverse samples thanks to its explicit modeling of the covariance.
Given 10% of the data points as context, ANP samples are generated by sampling the latent variable z , BANP samples are generated via bootstrapping, and
DKNP samples are generated with the predicted mean and covariance.
While ANP, BANP and DKNP all produce reasonable predictive means, ANP’s generated samples heavily resemble one another, whereas DKNP’s generated samples demonstrate considerable diversity such as different genders, facial expressions and hair styles.
BANP also demonstrates more varied samples than ANP thanks to its bootstrapping process, but they sometimes contain noisy pixels.
R ELATED W ORK
Many advances in NPs are made with the advent of neural network-based stochastic processes such as NPs (Garnelo et al., 2018b) and CNPs (Garnelo et al., 2018a).
ANP (Kim et al., 2019) show a dramatic performance gain by leveraging attention mechanisms in the aggregating operation.
Also, Le et al.
(2018) confirm through extensive empirical evaluation on the design of NPs such as architecture and objective functions that combining attention and NPs improve the predictive log-likelihood marginally.
Lee et al.
(2020) extends NPs using the bootstrap technique for estimating functional uncertainty without maintaining a latent variable in the NP architecture.
This notion of removing the latent variable part is inline with our motivation.
For modeling stationary stochastic processes, Gordon et al.
(2020) introduced Convolutional CNP (ConvCNP), where translation equivariance in the data is explicitly encoded in the model.
Convolutional NPs (ConvNP) Foong et al.
(2020) are a natural extension of ConvCNPs where a global latent variable was introduced to model dependencies of the predictive distribution.
Gaussian Neural Processes (GNP) (Bruinsma et al., 2021) further generalize these classes of models with translation equivariance by leveraging convolutional neural networks to capture the predictive correlations.
While GNPs share a similar motivation to our work, the way covariance matrix is parameterized is quite different (e.g., 1-D, 2-D convolution versus attention) from DKNPs, as their focus is mainly on translation equivariance.
Specifically, the covariance matrix of GNPs depends on Y T , while for DKNPs, motivated by posterior of GPs, the covariance only depends on the inputs X C , X T (see equation 6).
C ONCLUSION
We propose a new neural stochastic processes, Decoupled Kernel Neural Processes (DKNPs), that learn an explicit kernel function to better capture the correlation between output variables.
By leveraging cross- and mixed attention mechanisms to model an explicit kernel function, DKNPs outperform the concurrent NP variants in terms of predictive likelihood and better global coherence of generated samples.
By the novel model architecture of DKNPs, the learned prior can be accessible, which provides a deeper understanding of the underlying distributions of data.
As future work, one could consider developing a method to manipulate a learned kernel or to impose a constraint on the kernel learning process of DKNPs with prior knowledge.
[Reproducibility Statement] We utilized the datasets and the baselines that are all available on Github (https://github.com/juho-lee/bnp).
As for our proposed model and code, we will make a comment directed to the reviewer and area chairs once the discussion forums are open.
A A RCHITECTURES D ETAILS
We mostly followed the same model architectures in the paper (Lee et al., 2020).
num det path and num det path indicate the number of deterministic and stochastic paths.
A. 1D REGRESSION TASK
[Table]
A. I MAGE COMPLETION TASK
[Table]
B C OMPUTATIONAL C OMPLEXITY
[Table]
DKNP has attention mechanisms like ANP, therefore, it costs O(n(n+m)), which is still computationally much more efficient than GP even with the full covariance matrix.
C E XPERIMENTAL D ETAILS C. 1D R EGRESSION Data generation from a single kernel
We constructed training and test data generated from GPs with multiple kernels including RBF, periodic kernels, and Mat´ern 5/2.
For each task, we generated x ∼ Unif( − 2 , 2 ) and y from GP kernels where the GP hyperparameters are sampled from uniform distributions.
For RBF kernel, k ( x, x ′ )
= s 2 exp( −|| x − x ′ ||
2 / 2 l 2 ) , we sampled s ∼ Unif( 0 . 1 , 1 . 0 ), l ∼ Unif( 0 . 1 , 0 . 6 ), and output additive noise N (0 , 10 − 2 ) .
Given n , the size of the context C was drawn from Unif( 3 , 47 ) and the size of the target is sampled from Unif( 3 , 50 − n c ).
For Mat´ern 5/ kernel k ( x, x ′ )
=
s 2 (1 + √ 5 d/l + 5 d 2 / (3 l 2 ))
· exp( −√ 5 d/l ) , ( d = || x − x ′ || ) , we sampled from s ∼ Unif( 0 . 1 , 1 . 0 ) and l ∼ Unif( 0 . 1 , 0 . 6 ).
For periodic kernel, k ( x, x ′ )
=
s 2 exp( − 2 sin 2 (
π || x − x ′
|| 2 /p )
/l 2 ) , we sampled from s ∼ Unif( 0 . 3 , 1 . 0 ), l ∼ Unif( 0 . 6 , 1 . 0 ), and p ∼ Unif( 0 . 8 , 1 . 0 ).
We trained all models identically for 100,000 steps with training batch size of 100.
We used Adam optimizer with initial learning rate of 5 · 10 − 4 and decayed using cosine annealing scheme.
Data generation from two kernels Similar to a single kernel case, data points are sampled from a
GP but with two types of kernels, RBF and periodic.
The half of the points ranging from x = − 2 and x = 0 were sampled from one kernel and the rest from x = 0 to x = 2 were from another kernel.
We generated samples from three scenarios: Periodic - Periodic, RBF - Periodic, and RBF - RBF.
Note that we set the correlation between two kernels being zero.
The training details are identical to the single kernel case.
C. I MAGE C OMPLETION
CelebA32 Similar to 1-D regression tasks, we randomly sampled pixels of a given image at training as targets, and treat a subset of the points as contexts.
The size of the contexts and targets is sampled from Unif( 3 , 200 ) and n c +Unif( 0 , 200 − n c ).
For preprocessing, x is rescaled to [-1, 1] and y is rescaled to [-0.5, 0.5].
We trained all models identically for 200 steps with training batch size of 100.
We used Adam optimizer with initial learning rate of 5 · 10 − 4 with cosine annealing scheme.