Original: Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel
Revised: Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress
Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: It is challenging to guide neural network (NN) learning with prior knowledge.
Revised: It is challenging to guide neural network (NN) learning with prior knowledge.

Original: In contrast, many known properties, such as spatial smoothness or seasonality,are straightforward to model by choosing an appropriate kernel in a Gaussianprocess (GP).
Revised: In contrast, many known properties, such as spatial smoothness or seasonality,are straightforward to model by choosing an appropriate kernel in a Gaussianprocess (GP).

Original: Many deep learning applications could be enhanced by modelingsuch known properties.
Revised: Many deep learning applications could be enhanced by modelingsuch known properties.

Original: For example, convolutional neural networks (CNNs) arefrequently used in remote sensing, which is subject to strong seasonal effects.
Revised: For example, convolutional neural networks (CNNs) arefrequently used in remote sensing, which is subject to strong seasonal effects.

Original: Wepropose to blend the strengths of deep learning and the clear modeling capabilitiesof GPs by using a composite kernel that combines a kernel implicitly deﬁned by aneural network with a second kernel function chosen to model known properties(e.g., seasonality).
Revised: Wepropose to blend the strengths of deep learning and the clear modeling capabilitiesof GPs by using a composite kernel that combines a kernel implicitly deﬁned by aneural network with a second kernel function chosen to model known properties(e.g., seasonality).

Original: Then, we approximate the resultant GP by combining a deepnetwork and an efﬁcient mapping based on the Nyström approximation, which wecall Implicit Composite Kernel (ICK).
Revised: Then, we approximate the resultant GP by combining a deepnetwork and an efﬁcient mapping based on the Nyström approximation, which wecall Implicit Composite Kernel (ICK).

Original: ICK is ﬂexible and can be used to includeprior information in neural networks in many applications.
Revised: ICK is ﬂexible and can be used to includeprior information in neural networks in many applications.

Original: We demonstrate thestrength of our framework by showing its superior performance and ﬂexibilityon both synthetic and real-world data sets.
Revised: We demonstrate thestrength of our framework by showing its superior performance and ﬂexibilityon both synthetic and real-world data sets.

Original: The code is available at: https://anonymous.4open.science/r/ICK_NNGP-17C5/ .
Revised: The code is available at: https://anonymous.4open.science/r/ICK_NNGP-17C5/ .

Original:  Introduction
Revised:  Introduction

Original: In complex regression tasks, input data often contains multiple sources of information .
Revised: In complex regression tasks, input data often contains multiple sources of information .

Original: These sourcescan be presented in both high-dimensional (e g. images, audios, texts, etc.)
Revised: These sourcescan be presented in both high-dimensional (e g. images, audios, texts, etc.)

Original: and low-dimensional(e g. timestamps, spatial locations etc.) forms.
Revised: and low-dimensional(e g. timestamps, spatial locations etc.) forms.

Original: A common approach to learn from high-dimensional information is to use neural networks (NNs) [21, 33], as NNs are powerful enough to capture therelationship between complex high-dimensional data and target variables of interest.
Revised: A common approach to learn from high-dimensional information is to use neural networks (NNs) [21, 33], as NNs are powerful enough to capture therelationship between complex high-dimensional data and target variables of interest.

Original: In many areas,
Revised: In many areas,

Original: NNs are standard practice, such as the dominance of Convolutional Neural Networks (CNNs) forimage analysis [26, 61, 62].
Revised: NNs are standard practice, such as the dominance of Convolutional Neural Networks (CNNs) forimage analysis [26, 61, 62].

Original: In contrast, for low-dimensional information, we usually have someprior knowledge on how the information relates to the predictions.
Revised: In contrast, for low-dimensional information, we usually have someprior knowledge on how the information relates to the predictions.

Original: As a concrete example, considera remote sensing problem where we predict ground measurements from satellite imagery withassociated timestamps.
Revised: As a concrete example, considera remote sensing problem where we predict ground measurements from satellite imagery withassociated timestamps.

Original: A priori , we expect the ground measurements to vary periodically withrespect to time between summer and winter due to seasonal effects.
Revised: A priori , we expect the ground measurements to vary periodically withrespect to time between summer and winter due to seasonal effects.

Original: We would typically use a CNNto capture the complex relationship between the imagery and the ground measurements.
Revised: We would typically use a CNNto capture the complex relationship between the imagery and the ground measurements.

Original: In this case,we want to guide the learning of the CNN with our prior knowledge about the seasonality.
Revised: In this case,we want to guide the learning of the CNN with our prior knowledge about the seasonality.

Original: This ischallenging because knowledge represented in NNs pertains mainly to correlation between networkunits instead of quantiﬁable statements [36].
Revised: This ischallenging because knowledge represented in NNs pertains mainly to correlation between networkunits instead of quantiﬁable statements [36].

Original: Conversely, Gaussian processes (GPs) have been used historically to incorporate relevant prior beliefsby specifying the appropriate form of its kernel function (or covariance function) [2, 54].
Revised: Conversely, Gaussian processes (GPs) have been used historically to incorporate relevant prior beliefsby specifying the appropriate form of its kernel function (or covariance function) [2, 54].

Original: Oneapproach to modeling multiple sources of information is to assign a relevant kernel function to eachsource of information respectively and combine them through addition or multiplication, resulting in a composite kernel function [14].
Revised: Oneapproach to modeling multiple sources of information is to assign a relevant kernel function to eachsource of information respectively and combine them through addition or multiplication, resulting in a composite kernel function [14].

Original: This formulation means that specifying a kernel to match prior beliefson one source of information is straightforward.
Revised: This formulation means that specifying a kernel to match prior beliefson one source of information is straightforward.

Original: Such composite kernel learning techniques areextensively used in many application areas such as multi-media data [40], neuroimaging [60], spatialdata analysis, and environmental data analysis [28, 44].
Revised: Such composite kernel learning techniques areextensively used in many application areas such as multi-media data [40], neuroimaging [60], spatialdata analysis, and environmental data analysis [28, 44].

Original: In view of the clear modeling capabilities of
Revised: In view of the clear modeling capabilities of

Original: GP, it is desirable to examine how a NN could be imbued with the same modeling ease.
Revised: GP, it is desirable to examine how a NN could be imbued with the same modeling ease.

Original: In recent years, researchers have come up with a variety of methods to incorporate prior knowledgeinto NNs.
Revised: In recent years, researchers have come up with a variety of methods to incorporate prior knowledgeinto NNs.

Original: These efforts can be broken into many categories, such as those that add prior informationthrough loss terms like physics-informed NNs [32, 41].
Revised: These efforts can be broken into many categories, such as those that add prior informationthrough loss terms like physics-informed NNs [32, 41].

Original: Here, we focus on the major category of thosemethods that build integrated models of NNs and GPs with various structures [50, 57, 58].
Revised: Here, we focus on the major category of thosemethods that build integrated models of NNs and GPs with various structures [50, 57, 58].

Original: Relatedto our proposed methodology, Pearce et al.
Revised: Relatedto our proposed methodology, Pearce et al.

Original: [43] exploited the fact that a Bayesian neural network(BNN) approximates a GP to construct additive and multiplicative kernels, but they were limitedto speciﬁc predeﬁned kernels.
Revised: [43] exploited the fact that a Bayesian neural network(BNN) approximates a GP to construct additive and multiplicative kernels, but they were limitedto speciﬁc predeﬁned kernels.

Original: Matsubara et al.
Revised: Matsubara et al.

Original: [38] then resolved this limitation by constructingpriors of BNN parameters based on the ridgelet transform and its dual, but they did not explicitlyshow how their approach works for data with multiple sources of information.
Revised: [38] then resolved this limitation by constructingpriors of BNN parameters based on the ridgelet transform and its dual, but they did not explicitlyshow how their approach works for data with multiple sources of information.

Original: To our knowledge,none of these existing approaches allows a modeler to choose any appropriate kernel of known priorinformation from multiple sources.
Revised: To our knowledge,none of these existing approaches allows a modeler to choose any appropriate kernel of known priorinformation from multiple sources.

Original: We address this limitation by presenting a simple yet novel
Revised: We address this limitation by presenting a simple yet novel

Original: Implicit Composite Kernel (ICK) framework, which processes high-dimensional information using akernel implicitly deﬁned by a neural network and low-dimensional information using a chosen kernelfunction.
Revised: Implicit Composite Kernel (ICK) framework, which processes high-dimensional information using akernel implicitly deﬁned by a neural network and low-dimensional information using a chosen kernelfunction.

Original: The low-dimensional kernels are mapped into the neural network framework to create astraightforward and simple-to-learn implementation.
Revised: The low-dimensional kernels are mapped into the neural network framework to create astraightforward and simple-to-learn implementation.

Original: Our key results and contributions are:
Revised: Our key results and contributions are:

Original: • We analytically show our ICK framework, under reasonable assumptions, is approximately
Revised: • We analytically show our ICK framework, under reasonable assumptions, is approximately

Original: equivalent to a Gaussian process regression (GPR) model with a composite kernel a priori .
Revised: equivalent to a Gaussian process regression (GPR) model with a composite kernel a priori .

Original: • We demonstrate that our ICK framework yields better performance on both prediction and
Revised: • We demonstrate that our ICK framework yields better performance on both prediction and

Original: forecasting tasks even with very limited data.
Revised: forecasting tasks even with very limited data.

Original: • We compare to joint deep
Revised: • We compare to joint deep

Original: learning models, such as a neural network-random forest jointmodel, to show that ICK can ﬂexibly capture the patterns of the low-dimensional informationwithout deliberately designing a pre-processing procedure or complex NN structures.
Revised: learning models, such as a neural network-random forest jointmodel, to show that ICK can ﬂexibly capture the patterns of the low-dimensional informationwithout deliberately designing a pre-processing procedure or complex NN structures.

Original: Based on these contributions, we believe ICK will be useful in the context of learning from complexhybrid data with prior knowledge, especially in the ﬁeld of remote sensing and spatial statistics.
Revised: Based on these contributions, we believe ICK will be useful in the context of learning from complexhybrid data with prior knowledge, especially in the ﬁeld of remote sensing and spatial statistics.

Original:  Related Work
Revised:  Related Work

Original: Equivalence between NNs and GPs The equivalence between GPs and randomly initialized singlelayer NNs with inﬁnite width was ﬁrst shown by Neal [42].
Revised: Equivalence between NNs and GPs The equivalence between GPs and randomly initialized singlelayer NNs with inﬁnite width was ﬁrst shown by Neal [42].

Original: With the development of modern deeplearning, researchers further extended this relationship to deep networks [34, 39] and convolutionalneural networks (CNNs) [17].
Revised: With the development of modern deeplearning, researchers further extended this relationship to deep networks [34, 39] and convolutionalneural networks (CNNs) [17].

Original: This relationship is crucial for proving the resemblance between GPRand our ICK framework, which will be discussed in Section 4.1.
Revised: This relationship is crucial for proving the resemblance between GPRand our ICK framework, which will be discussed in Section 4.1.

Original: NNs with prior knowledge As mentioned before, one approach to equip NNs with prior knowledgeis to modify the loss function.
Revised: NNs with prior knowledge As mentioned before, one approach to equip NNs with prior knowledgeis to modify the loss function.

Original: For example, Lagaris et al.
Revised: For example, Lagaris et al.

Original: [32] solved differential equations (DEs)using NNs by setting the loss to be a function whose derivative satisﬁes the DE conditions.
Revised: [32] solved differential equations (DEs)using NNs by setting the loss to be a function whose derivative satisﬁes the DE conditions.

Original: Anotherapproach is to build integrated models of NNs and GPs.
Revised: Anotherapproach is to build integrated models of NNs and GPs.

Original: For example, Wilson et al.
Revised: For example, Wilson et al.

Original: [58] implementeda regression network with GP priors over latent variables and made inference by approximatingthe posterior using Variational Bayes or sampling from the posterior using Gibbs sampling scheme.
Revised: [58] implementeda regression network with GP priors over latent variables and made inference by approximatingthe posterior using Variational Bayes or sampling from the posterior using Gibbs sampling scheme.

Original: Garnelo et al.
Revised: Garnelo et al.

Original: [16] introduced a class of neural latent variable models called Neural Processes (NPs)which are capable of learning efﬁciently from the data and adapting rapidly to new observations.
Revised: [16] introduced a class of neural latent variable models called Neural Processes (NPs)which are capable of learning efﬁciently from the data and adapting rapidly to new observations.

Original: Zhuet al.
Revised: Zhuet al.

Original: [10] proposed NeuralEF which can accurately approximately kernel functions by using a seriesof objective functions parameterized by NNs under the principle of eigen-decomposition.
Revised: [10] proposed NeuralEF which can accurately approximately kernel functions by using a seriesof objective functions parameterized by NNs under the principle of eigen-decomposition.

Original: GP with composite kernels Composite kernel GPs are widely used in both machine learning[14, 54] and geostatistical modeling [9, 18].
Revised: GP with composite kernels Composite kernel GPs are widely used in both machine learning[14, 54] and geostatistical modeling [9, 18].

Original: GPR in geostatistical modeling is also known as kriging[27, 31], which serves as a surrogate model to replace expensive function evaluations.
Revised: GPR in geostatistical modeling is also known as kriging[27, 31], which serves as a surrogate model to replace expensive function evaluations.

Original: The inputs for are usually low dimensional e g spatial distance)
Revised: The inputs for are usually low dimensional e g spatial distance)

Original: GPs do not scale well the number of high-dimensional inputs overcome this issue
Revised: GPs do not scale well the number of high-dimensional inputs overcome this issue

Original: Pearce et al et al.
Revised: Pearce et al et al.

Original: [38] developed BNN analogue composite GPs
Revised: [38] developed BNN analogue composite GPs

Original: Similar to these studies our framework can also be viewed a simulation
Revised: Similar to these studies our framework can also be viewed a simulation

Original: Approximation methods for GP For large data sets, approximation methods are needed as exactkernel learning and inference scales O ( N 3 ) .
Revised: Approximation methods for GP For large data sets, approximation methods are needed as exactkernel learning and inference scales O ( N 3 ) .

Original: Nyström low-rank matrix approximation [12, 53] and
Revised: Nyström low-rank matrix approximation [12, 53] and

Original: Random Fourier Features [45, 46] are two of the most commonly used methods.
Revised: Random Fourier Features [45, 46] are two of the most commonly used methods.

Original: A common techniqueis to choose inducing points as pseudo-inputs to efﬁciently approximate the full kernel matrix [49, 23].
Revised: A common techniqueis to choose inducing points as pseudo-inputs to efﬁciently approximate the full kernel matrix [49, 23].

Original: Our work is inspired by these approximation techniques and we use them as transformation functionsto map the kernel matrix into latent space representations in Section 4.2.
Revised: Our work is inspired by these approximation techniques and we use them as transformation functionsto map the kernel matrix into latent space representations in Section 4.2.

Original:  Background
Revised:  Background

Original: Before elaborating on the details of our ICK framework, we introduce our notation, brieﬂy go overthe concepts of composite GPs, and describe the relationship between GPs and NNs.
Revised: Before elaborating on the details of our ICK framework, we introduce our notation, brieﬂy go overthe concepts of composite GPs, and describe the relationship between GPs and NNs.

Original: 3.1 Problem Setup
Revised: 3.1 Problem Setup

Original: To formalize the problem, we have a training data set which contains N data points X =[ x i ]
Revised: To formalize the problem, we have a training data set which contains N data points X =[ x i ]

Original: Ni =1 =
Revised: Ni =1 =

Original: [ x 1 , x 2 , ..., x N ] T and the corresponding labels of these data points are y =
Revised: [ x 1 , x 2 , ..., x N ] T and the corresponding labels of these data points are y =

Original: [ y i ]
Revised: [ y i ]

Original: Ni =1 =[
Revised: Ni =1 =[

Original: y 1 , y 2 , ..., y N ] T where y i ∈ R .
Revised: y 1 , y 2 , ..., y N ] T where y i ∈ R .

Original: Each data point x i = (cid:110) x
Revised: Each data point x i = (cid:110) x

Original: (1) i , x (2) i , ..., x ( M ) i (cid:111) is composed of information from M different sources where the m th source of information of the i th data point is denoted asx ( m ) i ∈
Revised: (1) i , x (2) i , ..., x ( M ) i (cid:111) is composed of information from M different sources where the m th source of information of the i th data point is denoted asx ( m ) i ∈

Original: R D m .
Revised: R D m .

Original: Our goal is to learn a function ˆ y i =
Revised: Our goal is to learn a function ˆ y i =

Original: f ( x i ) : (cid:8) R D 1 , R D 2 , ..., R D M (cid:9) →
Revised: f ( x i ) : (cid:8) R D 1 , R D 2 , ..., R D M (cid:9) →

Original: R which takesin a data point x i and outputs a predicted value ˆ y i .
Revised: R which takesin a data point x i and outputs a predicted value ˆ y i .

Original: 3.2 Composite GPs
Revised: 3.2 Composite GPs

Original: A Gaussian process (GP) describes a distribution over functions [54].
Revised: A Gaussian process (GP) describes a distribution over functions [54].

Original: A key property of GP is thatit can be completely deﬁned by a mean function µ ( x ) and a kernel function K ( x , x (cid:48) ) .
Revised: A key property of GP is thatit can be completely deﬁned by a mean function µ ( x ) and a kernel function K ( x , x (cid:48) ) .

Original: The meanfunction µ ( x ) is often assumed to be zero for simplicity.
Revised: The meanfunction µ ( x ) is often assumed to be zero for simplicity.

Original: In that case, the outcome function isf
Revised: In that case, the outcome function isf

Original: [Equation]
Revised: [Equation]

Original: Any ﬁnite subset of these random variables has a multivariate Gaussian distribution with mean 0and kernel matrix K whose entries can be calculated as K ij =
Revised: Any ﬁnite subset of these random variables has a multivariate Gaussian distribution with mean 0and kernel matrix K whose entries can be calculated as K ij =

Original: K ( x i , x j ) .
Revised: K ( x i , x j ) .

Original: In many situations, thefull kernel function is built by a composite kernel by combining simple kernels through addition
Revised: In many situations, thefull kernel function is built by a composite kernel by combining simple kernels through addition

Original: K comp ( x , x (cid:48) )
Revised: K comp ( x , x (cid:48) )

Original: =
Revised: =

Original: K 1 ( x , x (cid:48) )
Revised: K 1 ( x , x (cid:48) )

Original: + K 2 ( x , x (cid:48) )
Revised: + K 2 ( x , x (cid:48) )

Original: or multiplication K comp ( x , x (cid:48) )
Revised: or multiplication K comp ( x , x (cid:48) )

Original: =
Revised: =

Original: K 1 ( x , x (cid:48) )
Revised: K 1 ( x , x (cid:48) )

Original: K 2 ( x , x (cid:48) )
Revised: K 2 ( x , x (cid:48) )

Original: A useful property that ICK exploits is that K 1 and K 2 can take different subparts of x astheir inputs.
Revised: A useful property that ICK exploits is that K 1 and K 2 can take different subparts of x astheir inputs.

Original: For example, K comp ( x , x (cid:48) )
Revised: For example, K comp ( x , x (cid:48) )

Original: =
Revised: =

Original: K 1 (cid:16) x (1) , x (1) (cid:48) (cid:17) + K 2 (cid:16) x (2) , x (2) (cid:48) (cid:17) or K comp ( x , x (cid:48) )
Revised: K 1 (cid:16) x (1) , x (1) (cid:48) (cid:17) + K 2 (cid:16) x (2) , x (2) (cid:48) (cid:17) or K comp ( x , x (cid:48) )

Original: =
Revised: =

Original: K 1 (cid:16) x (1) , x (1) (cid:48) (cid:17) K 2 (cid:16) x (2) , x (2) (cid:48) (cid:17) .
Revised: K 1 (cid:16) x (1) , x (1) (cid:48) (cid:17) K 2 (cid:16) x (2) , x (2) (cid:48) (cid:17) .

Original: Other methods such as functional mapping are also valid if theresulting kernel matrix K is positive semideﬁnite (PSD) for all possible choices of data set X [47].
Revised: Other methods such as functional mapping are also valid if theresulting kernel matrix K is positive semideﬁnite (PSD) for all possible choices of data set X [47].

Original: 3.3 Correspondence between GPs and NNs
Revised: 3.3 Correspondence between GPs and NNs

Original: Neal [42] proved that a single-hidden layer network with inﬁnite width is exactly equivalent to a GPover data indices i = 1 , 2 , ..., N under the assumption that the weight and bias parameters of thehidden layer are i.i.d.
Revised: Neal [42] proved that a single-hidden layer network with inﬁnite width is exactly equivalent to a GPover data indices i = 1 , 2 , ..., N under the assumption that the weight and bias parameters of thehidden layer are i.i.d.

Original: Gaussian with zero mean.
Revised: Gaussian with zero mean.

Original: Lee et al.
Revised: Lee et al.

Original: [34] and Garriga et al.
Revised: [34] and Garriga et al.

Original: [17] then extendedthis statement to deep neural networks and convolutional neural networks (CNNs), respectively.
Revised: [17] then extendedthis statement to deep neural networks and convolutional neural networks (CNNs), respectively.

Original: However, if the width (or the number of channels) of a network is ﬁnite, then these results state thatthe network approximately converges to a GP with zero mean as in the following lemma.
Revised: However, if the width (or the number of channels) of a network is ﬁnite, then these results state thatthe network approximately converges to a GP with zero mean as in the following lemma.

Original: Lemma 1 Let z = f NN (cid:0) x
Revised: Lemma 1 Let z = f NN (cid:0) x

Original: (1) (cid:1) : R D 1 → R p be the latent representation extracted from x (1) wherep is the dimension of the extracted representation and f NN is a neural network with ﬁnite widthand zero-mean i.i.d.
Revised: (1) (cid:1) : R D 1 → R p be the latent representation extracted from x (1) wherep is the dimension of the extracted representation and f NN is a neural network with ﬁnite widthand zero-mean i.i.d.

Original: parameters.
Revised: parameters.

Original: The k th entry of this representation will approximately follow a
Revised: The k th entry of this representation will approximately follow a

Original: NN-implied GP
Revised: NN-implied GP

Original: [Equation]
Revised: [Equation]

Original: That is to say, the k th component z k of the representation extracted by the network has zero mean
Revised: That is to say, the k th component z k of the representation extracted by the network has zero mean

Original: E p ( θ (1) ) (cid:104) z (1) ik (cid:105) = 

Revised: E p ( θ (1) ) (cid:104) z (1) ik (cid:105) =

Original: for all i = 1 , 2 , ..., N where θ represents the network parameters.
Revised: for all i = 1 , 2 , ..., N where θ represents the network parameters.

Original: The covariance between z (1) ik and z (1) jk for different data indices i, j = 1 , 2 , ..., N can be approximated as 
Revised: The covariance between z (1) ik and z (1) jk for different data indices i, j = 1 , 2 , ..., N can be approximated as [Equation] [Equation]

Original:  Implicit Composite Kernel (ICK) Framework
Revised:  Implicit Composite Kernel (ICK) Framework

Original: We show the structure of a composite GPR model and our ICK framework in Figure 1.
Revised: We show the structure of a composite GPR model and our ICK framework in Figure 1.

Original: To makethe illustration clear, we limit ourselves to data with information from 2 different sources x = 135 (cid:8) x (1) , x (2) (cid:9) where x (1) is high-dimensional and x (2) is low-dimensional (i.e. D 1 (cid:29) D 2 ) with someknown relationship with the target y .
Revised: To makethe illustration clear, we limit ourselves to data with information from 2 different sources x = 135 (cid:8) x (1) , x (2) (cid:9) where x (1) is high-dimensional and x (2) is low-dimensional (i.e. D 1 (cid:29) D 2 ) with someknown relationship with the target y .

Original: We are inspired by composite GPR, which computes 2 differentkernel matrices K 1 and K 2 and then combines them into a single composite kernel matrix K comp .
Revised: We are inspired by composite GPR, which computes 2 differentkernel matrices K 1 and K 2 and then combines them into a single composite kernel matrix K comp .

Original: However, as discussed before, it is more suitable to use a NN to learn from the high dimensionalinformation x (1) .
Revised: However, as discussed before, it is more suitable to use a NN to learn from the high dimensionalinformation x (1) .

Original: In our ICK framework, we process x (1) with a NN f NN ( · ) : R D 1 → R p and x (2)with a mapping g ( · ) : R D 2 → R p which consists of a kernel function K 2 followed by a kernelto-latent-space transformation (described in Section 4.2), resulting in two latent representationsz (1) , z (2) ∈ R p .
Revised: In our ICK framework, we process x (1) with a NN f NN ( · ) : R D 1 → R p and x (2)with a mapping g ( · ) : R D 2 → R p which consists of a kernel function K 2 followed by a kernelto-latent-space transformation (described in Section 4.2), resulting in two latent representationsz (1) , z (2) ∈ R p .

Original: Then, we make a prediction ˆ
Revised: Then, we make a prediction ˆ

Original: y
Revised: y

Original: by doing an inner product between these tworepresentations
Revised: by doing an inner product between these tworepresentations

Original: ˆ y =
Revised: ˆ y =

Original: f NN (cid:0) x
Revised: f NN (cid:0) x

Original: (1) (cid:1) · g (cid:0) x (2) (cid:1) .
Revised: (1) (cid:1) · g (cid:0) x (2) (cid:1) .

Original: Finally, the parameters of both the NN and the kernelfunction are learned via gradient-based optimization methods [3, 30].
Revised: Finally, the parameters of both the NN and the kernelfunction are learned via gradient-based optimization methods [3, 30].

Original: In the sections below, we ﬁrst analytically show that our ICK framework is approximately equivalentto a composite GPR model a priori using a multiplicative kernel between the kernel implicitlydeﬁned by the NN on x (1) and the chosen kernel on x (2) .
Revised: In the sections below, we ﬁrst analytically show that our ICK framework is approximately equivalentto a composite GPR model a priori using a multiplicative kernel between the kernel implicitly deﬁnedby the NN on x (1) and the chosen kernel on x (2) .

Revised: This theory is used to motivate the model form.

Revised: The model will deviate from the GP solution after learning, but we note that recent work suggeststhat the predictions from a trained NN may not vary too much from its GP equivalent [34].

Original: We then show how we implement thekernel-to-latent-space transformation in detail.
Revised: We thenshow how we implement the kernel-to-latent-space transformation in detail.

Original: Here, we note that we apply ICK for multiplicativekernels but note that an additive kernel may be constructed using the methods of Pearce et al.
Revised: Here, we note that weapply ICK for multiplicative kernels but note that an additive kernel may be constructed using themethods of Pearce et al.

Original: 4.1 Resemblance between Composite GPR and ICK
Revised: 4.1 Resemblance between Composite GPR and ICK

Original: We will analytically prove the following theorem for data with information from 2 different sourcesx = (cid:8) x (1) , x (2) (cid:9) for clarity, and we note this theorem can be straightforwardly extended to M > 2 .
Revised: We will analytically prove the following theorem for data with information from 2 different sourcesx = (cid:8) x (1) , x (2) (cid:9) for clarity, and we note this theorem can be straightforwardly extended to M > 2 .

Original: Theorem 1 Let f NN : R D 1 → R p be a NN-implied function  and g : R D 2 → R p be a mappingfunction, and deﬁne an inner product ˆ y between the representations z (1) =
Revised: Theorem 1 Let f NN : R D 1 → R p be a NN function with random weights and g : R D 2 → R p be amapping function, and deﬁne an inner product ˆ y between the representations z (1) =

Original: f NN (cid:0) x
Revised: f NN (cid:0) x

Original: (1) (cid:1) and z (2) =
Revised: (1) (cid:1) andz (2) =

Original: g (cid:0) x (2) (cid:1) .
Revised: g (cid:0) x (2) (cid:1) .

Original: Then this inner product approximately follows a composite GPR model
Revised: Then this inner product approximately follows a composite GPR model

Original: [Equation]
Revised: [Equation]

Original: if g includes the following deterministic kernel-to-latent-space transformation
Revised: if g includes the following deterministic kernel-to-latent-space transformation

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where K (1) NN is a NN-implied kernel and K (2) is any valid kernel of our choice.
Revised: where K (1) NN is a NN-implied kernel and K (2) is any valid kernel of our choice.

Original: To prove Theorem 1, we ﬁrst make the following assumption.
Revised: To prove Theorem 1, we ﬁrst make the following assumption.

Original: Assumption 1 For latent representations z ( m ) i and z ( m ) j extracted from different data points x i andx j where i (cid:54) =
Revised: Assumption 1 For latent representations z ( m ) i and z ( m ) j extracted from different data points x i andx j where i (cid:54) =

Original: j and m ∈ { 1 , 2 } , the interactions between different entries of z ( m ) i and z ( m ) j can bereasonably ignored.
Revised: j and m ∈ { 1 , 2 } , the interactions between different entries of z ( m ) i and z ( m ) j can bereasonably ignored.

Original: In other words, let θ ( m ) be the parameters of the network or the kernel functionwhich takes in x ( m ) and outputs z ( m ) , we have E p ( θ ( m ) )
Revised: In other words, let θ ( m ) be the parameters of the network or the kernel functionwhich takes in x ( m ) and outputs z ( m ) , we have E p ( θ ( m ) )

Original: (cid:104) z ( m )
Revised: (cid:104) z ( m )

Original: ik z ( m ) jl (cid:105) = 

Revised: ik z ( m ) jl (cid:105) =

Original: for all k (cid:54)
Revised: for all k (cid:54)

Original: =
Revised: =

Original: l .
Revised: l .

Original: With Assumption 1 and Lemma 1, let Θ = (cid:8) θ (1) , θ (2) (cid:9) represent the parameters of the ICK framework, we can calculate the covariance between ˆ y i and ˆ y j for different data indices i (cid:54)
Revised: With Assumption 1 and Lemma 1, let Θ = (cid:8) θ (1) , θ (2) (cid:9) represent the parameters of the ICK framework, we can calculate the covariance between ˆ y i and ˆ y j for different data indices i (cid:54)

Original: =
Revised: =

Original: j as follows
Revised: j as follows

Original: [Equation]
Revised: [Equation]

Original: Here, from Equation 5 to Equation 6, we use the statement E p ( θ (1) )
Revised: Here, from Equation 5 to Equation 6, we use the statement E p ( θ (1) )

Original: (cid:104)
Revised: (cid:104)

Original: z (1) ik (cid:105) = 

Revised: z (1) ik (cid:105) =

Original: from Lemma 1 and theindependence between θ (1) and θ (2) , which leads to E p (Θ) [ˆ y i ] =
Revised: from Lemma 1 and theindependence between θ (1) and θ (2) , which leads to E p (Θ) [ˆ y i ] =

Original: E p (Θ) [ˆ y j ] = 0 .
Revised: E p (Θ) [ˆ y j ] = 0 .

Original: From Equation 7 to
Revised: From Equation 7 to

Original: Equation 8, we get rid of all the cross terms under Assumption 1.
Revised: Equation 8, we get rid of all the cross terms under Assumption 1.

Original: From Equation 8 to Equation 9, weagain make use of the independence between θ (1) and θ (2) .
Revised: From Equation 8 to Equation 9, weagain make use of the independence between θ (1) and θ (2) .

Original: From Equation 9 to Equation 10, we usethe statement E p ( θ(1) )
Revised: From Equation 9 to Equation 10, we usethe statement E p ( θ

Original: (cid:104)
Revised: [Equation]

Original: z (1) ik z (1) jk (cid:105)
Revised: [Equation] ik z (1) jk [Equation]

Revised: ≈ K NN

Original: ≈ K NN (cid:16) x (1) i , x (1) j (cid:17) from Lemma 1.
Revised: [Equation] x [Equation] 1) i , x (1) j (cid:17) from Lemma 1.

Original: If the kernel-to-latent-spacetransformation in g ( · ) is deterministic , we can remove the expectation sign from the summation termin Equation 10 and the covariance can be further expressed as
Revised: If the kernel-to-latent-spacetransformation in g ( · ) is deterministic , we can remove the expectation sign from the summation termin Equation 10 and the covariance can be further expressed as

Original: [Equation]
Revised: [Equation]

Original:  which means that ˆ y approximately follows a GP with composite kernel K comp ( x i , x j ) =
Revised: , (11) which means that ˆ y approximately follows a GP with composite kernel K comp ( x i , x j ) =

Original: K (1) NN (cid:16) x (1) i , x (1) j (cid:17) K (2) (cid:16) x (2) i , x (2) j (cid:17) a priori .
Revised: K (1) NN (cid:16) x (1) i , x (1) j (cid:17) K (2) (cid:16) x (2) i , x (2) j (cid:17) a priori .

Original: This completes our proof of Theorem 1.
Revised: This completes our proof of Theorem 1.

Original: 4.2 Kernel-to-latent-space Transformation
Revised: 4.2 Kernel-to-latent-space Transformation

Original: We now show how we can construct an appropriate mapping g ( · ) that approximately satisﬁes theassumed form of (4) and is used in the derivation of ICK from (10) to (11).
Revised: We now show how we can construct an appropriate mapping g ( · ) that approximately satisﬁes theassumed form of (4) and is used in the derivation of ICK from (10) to (11).

Original: Here we adopt twomethods, Nyström approximation and Random Fourier Features (RFF), to map the kernel matrix intothe latent space.
Revised: Here we adopt twomethods, Nyström approximation and Random Fourier Features (RFF), to map the kernel matrix intothe latent space.

Original: Below, we give the formulations and results for the Nyström method, and give themethods and results for RFF in Appendix B.
Revised: Below, we give the formulations and results for the Nyström method, and give themethods and results for RFF in Appendix B.

Original: According to Yang et al.
Revised: According to Yang et al.

Original: [59], the Nyström methodwill yield much better performance than RFF if there exists a large gap in the eigen-spectrum of thekernel matrix.
Revised: [59], the Nyström methodwill yield much better performance than RFF if there exists a large gap in the eigen-spectrum of thekernel matrix.

Original: In our applications, we also observe a large eigen-gap (see details in Appendix C)and Nyström method does generalize much better than RFF.
Revised: In our applications, we also observe a large eigen-gap (see details in Appendix C)and Nyström method does generalize much better than RFF.

Original: We name our framework with N y strömmethod and r andom Fourier Features ICK y and ICK r , respectively.
Revised: We name our framework with N y strömmethod and r andom Fourier Features ICK y and ICK r , respectively.

Original: 4.2.1 Nyström Method
Revised: 4.2.1 Nyström Method

Original: The main idea of Nyström method [53] is to approximate the kernel matrix K ∈ R N × N with a muchsmaller low-rank matrix K q ∈ R q × q where q (cid:28) N so both the computational and space complexityof kernel learning can be signiﬁcantly reduced
Revised: The main idea of Nyström method [53] is to approximate the kernel matrix K ∈ R N × N with a muchsmaller low-rank matrix K q ∈ R q × q where q (cid:28) N so both the computational and space complexityof kernel learning can be signiﬁcantly reduced

Original: [Equation]
Revised: [Equation]

Original: The entries of K q and K nq can be calculated as ( K q ) ij =
Revised: The entries of K q and K nq can be calculated as ( K q ) ij =

Original: K (ˆ x i , ˆ x j ) , i, j ∈ { 1 , 2 , ..., q } and( K nq ) ij =
Revised: K (ˆ x i , ˆ x j ) , i, j ∈ { 1 , 2 , ..., q } and( K nq ) ij =

Original: K
Revised: K

Original: ( x i , ˆ x j ) , i ∈ { 1 , 2 , ..., N } , j ∈ { 1 , 2 , ..., q } , respectively.
Revised: ( x i , ˆ x j ) , i ∈ { 1 , 2 , ..., N } , j ∈ { 1 , 2 , ..., q } , respectively.

Original: x represents the original data points and ˆ x represents pre-deﬁned inducing points (or pseudo-inputs [49]).
Revised: x represents the originaldata points and ˆ x represents pre-deﬁned inducing points (or pseudo-inputs [49]).

Original: In our study, theseinducing points are chosen by deﬁning an evenly spaced vector over the range of original data points.
Revised: In our study, theseinducing points are chosen by deﬁning an evenly spaced vector over the range of original data points.

Original: By performing Cholesky decomposition K − 1 q = U T U , where U ∈ R q × q , ˆ K is
Revised: By performing Cholesky decomposition K − 1 q = U T U , where U ∈ R q × q , ˆ K is

Original: [Equation]
Revised: [Equation]

Original: Therefore, if we set the number of inducing points to be q = p , then we can use z i (cid:44) U (cid:16) K Tnp (cid:17) : ,ias a kernel-to-latent-space transformation because each element in K approximately satisﬁes (4)as stated in Theorem 1: K
Revised: Therefore, if we set the number of inducing points to be q = p , then we can use z i (cid:44) U (cid:16) K Tnp (cid:17) : ,ias a kernel-to-latent-space transformation because each element in K approximately satisﬁes (4)as stated in Theorem 1: K

Original: ( x i , x j ) =
Revised: ( x i , x j ) =

Original: K ij ≈ ˆ K ij =
Revised: K ij ≈ ˆ K ij =

Original: z Ti z j .
Revised: z Ti z j .

Original: Conveniently, modern deep learningframeworks can propagate gradients through the Cholesky operation, making it straightforward toupdate the kernel parameters with gradient methods.
Revised: Conveniently, modern deep learningframeworks can propagate gradients through the Cholesky operation, making it straightforward toupdate the kernel parameters with gradient methods.

Original: Note that as we increase the number of inducingpoints p , the approximation error between K and ˆ K decreases.
Revised: Note that as we increase the number of inducingpoints p , the approximation error between K and ˆ K decreases.

Original: However, it is not recommended toset p very large as updating the Cholesky decomposition requires O ( p 3 ) .
Revised: However, it is not recommended toset p very large as updating the Cholesky decomposition requires O ( p 3 ) .

Original: The empirical impact ofp on computational time and performance is shown in Appendix E.
Revised: The empirical impact ofp on computational time and performance is shown in Appendix E.

Original: In our experiments, only mildvalues of p are necessary and the impact on computational is relatively small.
Revised: In our experiments, only mildvalues of p are necessary and the impact on computational is relatively small.

Original:  Experimental Results
Revised:  Experimental Results

Original: We evaluate ICK y on 3 different data sets: a synthetic data set, a remote sensing data set, and a dataset obtained from UCI Machine Learning Repository [13].
Revised: We evaluate ICK y on 3 different data sets: a synthetic data set, a remote sensing data set, and a dataset obtained from UCI Machine Learning Repository [13].

Original: Note that in all the 3 experiments, our
Revised: Note that in all the 3 experiments, our

Original: ICK y framework only consists of 2 kernels (i.e. M = 2 ), one NN-implied kernel and one chosenkernel function with trainable parameters.
Revised: ICK y framework only consists of 2 kernels (i.e. M = 2 ), one NN-implied kernel and one chosenkernel function with trainable parameters.

Original: To verify that ICK y can work with more than 2 kernels,we create another synthetic data set with 3 kernels and show the corresponding results in Appendix
Revised: To verify that ICK y can work with more than 2 kernels,we create another synthetic data set with 3 kernels and show the corresponding results in Appendix

Original: A.
Revised: A.

Original: In addition, the experimental results for ICK r is provided in Appendix B.
Revised: In addition, the experimental results for ICK r is provided in Appendix B.

Original: All experiments areconducted on a computer cluster equipped with a GeForce RTX 2080 Ti GPU.
Revised: All experiments areconducted on a computer cluster equipped with a GeForce RTX 2080 Ti GPU.

Original: The implementationdetails of all the experiments in this section are provided in Appendix G.
Revised: The implementationdetails of all the experiments in this section are provided in Appendix G.

Original: 5.1 Synthetic Data
Revised: 5.1 Synthetic Data

Original: To verify that ICK y can simulate a multiplicative kernel , we create a synthetic data set y ∼
Revised: To verify that ICK y can simulate a multiplicative kernel , we create a synthetic data set y ∼

Original: GP (0 , K 1 K 2 ) containing 3000 data points where x (1) ∈ [0 , 1] is the input for the linear kernel
Revised: GP (0 , K 1 K 2 ) containing 3000 data points where x (1) ∈ [0 , 1] is the input for the linear kernel

Original: K 1 and x (2) ∈ [0 , 2] is the input for the spectral mixture kernel [56] K 2 with 2 components.
Revised: K 1 and x (2) ∈ [0 , 2] is the input for the spectral mixture kernel [56] K 2 with 2 components.

Original: With
Revised: With

Original: ICK y , we process x (1) with a single-hidden-layer NN and x (2) with a spectral mixture kernel function.
Revised: ICK y , we process x (1) with a single-hidden-layer NN and x (2) with a spectral mixture kernel function.

Original: We evaluate ICK y on both a prediction task (where we ﬁrst randomly shufﬂe the data points and do a50:50 train-test split) and a forecasting task (where we use only the data points with x (2) < 0 .
Revised: We evaluate ICK y on both a prediction task (where we ﬁrst randomly shufﬂe the data points and do a50:50 train-test split) and a forecasting task (where we use only the data points with x (2) < 0 .

Original: 6 fortraining and the rest for testing).
Revised: 6 fortraining and the rest for testing).

Original: We then compare ICK y with two models: a plain multi-layer perceptron (MLP) applied to theconcatenated features and a novel multi-layer perceptron-random forest (MLP-RF) joint modelemployed by Zheng et al.
Revised: We then compare ICK y with two models: a plain multi-layer perceptron (MLP) applied to theconcatenated features and a novel multi-layer perceptron-random forest (MLP-RF) joint modelemployed by Zheng et al.

Original: [61] where MLP learns from x (1) and RF learns from x (2) .
Revised: [61] where MLP learns from x (1) and RF learns from x (2) .

Original: We believe
Revised: We believe

Original: MLP-RF serves as a good benchmark model as it is a joint model with similar architecture to our
Revised: MLP-RF serves as a good benchmark model as it is a joint model with similar architecture to our

Original: ICK y framework.
Revised: ICK y framework.

Original: To see how ICK y simulates the spectral mixture kernel, we plot only x (2) againstthe predicted value of y as shown in Figure 2.
Revised: To see how ICK y simulates the spectral mixture kernel, we plot only x (2) againstthe predicted value of y as shown in Figure 2.

Original: As can be seen from the ﬁgure, in the prediction task(top row), plain MLP only captures the linear trend.
Revised: As can be seen from the ﬁgure, in the prediction task(top row), plain MLP only captures the linear trend.

Original: MLP-RF only captures the mean of the spectralmixture components.
Revised: MLP-RF only captures the mean of the spectralmixture components.

Original: In contrast, our ICK y framework captures both the mean and the variance of thespectral mixture kernel.
Revised: In contrast, our ICK y framework captures both the mean and the variance of thespectral mixture kernel.

Original: In the forecasting task (bottom row), ICK y also outperforms plain MLP and
Revised: In the forecasting task (bottom row), ICK y also outperforms plain MLP and

Original: MLP-RF as it approximately captures the rising trend in the range of x (2) ∈ [0 . 6 , 1] .
Revised: MLP-RF as it approximately captures the rising trend in the range of x (2) ∈ [0 . 6 , 1] .

Original: When x (2) > 1 ,
Revised: When x (2) > 1 ,

Original: ICK y is unable to conﬁdently extrapolate, so it starts to " fail gracefully ," by which we mean that theprediction reverts to the mean of the prior (e.g., no observed information case.), just as would beexpected in a GP.
Revised: ICK y is unable to conﬁdently extrapolate, so it starts to " fail gracefully ," by which we mean that theprediction reverts to the mean of the prior (e.g., no observed information case.), just as would beexpected in a GP.

Original: However, we do not evaluate the posterior distribution to get a full sense of theposterior uncertainty.
Revised: However, we do not evaluate the posterior distribution to get a full sense of theposterior uncertainty.

Original: We also test plain MLP, MLP-RF, and ICK y on the prediction task using different number of trainingsamples.
Revised: We also test plain MLP, MLP-RF, and ICK y on the prediction task using different number of trainingsamples.

Original: As displayed in Figure 3, ICK y yields the smallest error among all the 3 frameworks evenwith very limited data.
Revised: As displayed in Figure 3, ICK y yields the smallest error among all the 3 frameworks evenwith very limited data.

Original: In addition, to test the robustness of ICK y , we conduct the same experimentson another synthetic data set in Appendix D to conﬁrm that ICK y can simulate an additive kernel .
Revised: In addition, to test the robustness of ICK y , we conduct the same experimentson another synthetic data set in Appendix D to conﬁrm that ICK y can simulate an additive kernel .

Original: 5.2 Remote Sensing Data
Revised: 5.2 Remote Sensing Data

Revised: [Figure]

Original: We believe ICK y will be particularly useful for remote sensing applications.
Revised: We believe ICK y will be particularly useful for re- mote sensing applications.

Original: In this experiment, wecollect remote sensing data from 51 air quality monitoring (AQM) stations located in the National Capital
Revised: In this experiment, wecollect remote sensing data from 51 air quality monitoring (AQM) stations located in the National Capital

Original: Territory (NCT) of Delhi and its satellite cities overthe period from January 1, 2018 to June 30, 2020 (see
Revised: Territory (NCT) of Delhi and its satellite cities overthe period from January 1, 2018 to June 30, 2020 (see

Original: Appendix F for notes on data availability).
Revised: Appendix F for notes on data availability).

Original: Each datapoint x = { x, t } contains 2 sources of information:
Revised: Each datapoint x = { x, t } contains 2 sources of information:

Original: a three-band natural color (red-blue-green) satelliteimage x as the high-dimensional information andthe corresponding timestamp as the low-dimensionalinformation.
Revised: a three-band natural color (red-blue-green) satelliteimage x as the high-dimensional information andthe corresponding timestamp as the low-dimensionalinformation.

Original: Note that we convert the timestampsinto numerical values t (where the day 2018-01-01corresponds to t = 0 ) before feeding them into themodels.
Revised: Note that we convert the timestampsinto numerical values t (where the day 2018-01-01corresponds to t = 0 ) before feeding them into themodels.

Original: Our goal is to predict the ground-level PM 2 .
Revised: Our goal is to predict the ground-level PM 2 .

Original: 5concentration ˆ y =
Revised: 5concentration ˆ y =

Original: f
Revised: f

Original: ( x, t ) using both sources of information.
Revised: ( x, t ) using both sources of information.

Original: We split the train and test data set based on t .
Revised: We split the train and test data set based on t .

Original: Speciﬁcally, we use all the data points with t ≥ 500 fortesting and the rest for training.
Revised: Speciﬁcally, we use all the data points with t ≥ 500 fortesting and the rest for training.

Original: As PM 2 . 5 varies with time on a yearly basis, we use an exponentialsine-squared kernel with a period of T = 365 (days) to process the low-dimensional informationt .
Revised: As PM 2 . 5 varies with time on a yearly basis, we use an exponentialsine-squared kernel with a period of T = 365 (days) to process the low-dimensional informationt .

Original: The satellite images are processed with a CNN.
Revised: The satellite images are processed with a CNN.

Original: Figure 4 shows the true versus the forecasted
Revised: Figure 4 shows the true versus the forecasted

Original: PM 2 .
Revised: PM 2 .

Original: 5 values by both ICK y and 2 benchmarks: a Convolutional Neural Network-Random Forest(CNN-RF) joint model [61, 62] (similar to the MLP-RF model in Section 5.1, where RF learns thetemporal variation of PM 2 .
Revised: 5 values by both ICK y and 2 benchmarks: a Convolutional Neural Network-Random Forest(CNN-RF) joint model [61, 62] (similar to the MLP-RF model in Section 5.1, where RF learns thetemporal variation of PM 2 .

Original: 5 and CNN captures the spatial variation of PM 2 .
Revised: 5 and CNN captures the spatial variation of PM 2 .

Original: 5 from satellite images)and a carefully designed CNN-RF model that maps t into two new features, sin(2 πt/ 365)
Revised: 5 from satellite images)and a carefully designed CNN-RF model that maps t into two new features, sin(2 πt/ 365)

Original: andcos(2 πt/ 365) , to explicitly model seasonality.
Revised: andcos(2 πt/ 365) , to explicitly model seasonality.

Original: As can be seen, ICK y outperforms both benchmarkswith the highest correlation coefﬁcients and the lowest errors on the forecasting task.
Revised: As can be seen, ICK y outperforms both benchmarkswith the highest correlation coefﬁcients and the lowest errors on the forecasting task.

Original: Speciﬁcally,regular CNN-RF joint model fails to forecast PM 2 . 5 as shown in Figure 4a.
Revised: Speciﬁcally,regular CNN-RF joint model fails to forecast PM 2 . 5 as shown in Figure 4a.

Original: After including seasonality,
Revised: After including seasonality,

Original: CNN-RF performs signiﬁcantly better as shown in Figure 4b, but the forecasted PM 2 .
Revised: CNN-RF performs signiﬁcantly better as shown in Figure 4b, but the forecasted PM 2 .

Original: 5 values are stillless smooth than those from ICK y
Revised: 5 values are stillless smooth than those from ICK y

Original: (Figure 4c) due to the discontinuous nature of the RF regressor[6, 19].
Revised: (Figure 4c) due to the discontinuous nature of the RF regressor[6, 19].

Original: We also visualize these results in the form of time series in Appendix F
Revised: We also visualize these results in the form of time series in Appendix F

Original: [Table]
Revised: [Table]

Original: We note that the inner product operation in ICK is similar in mathematical structure to attention-basedmechanisms [51] popular in many deep learning frameworks.
Revised: We note that the inner product operation in ICK is similar in mathematical structure to attention-basedmechanisms [51] popular in many deep learning frameworks.

Original: Therefore, we compare ICK y
Revised: Therefore, we compare ICK y

Original: with

Revised: with

Original: attention-based benchmarks that we constructured based off of a Vision Transformer (ViT)
Revised: attention-based benchmarks that we constructured based off of a Vision Transformer (ViT)

Original: [11]architecture, including ViT-RF, Seasonal ViT-RF, Seasonal DeepViT-RF [63], and Seasonal MAEViT-RF where ViT is pre-trained by a Masked Autoencoder [22], as displayed in Table 1.
Revised: [11]architecture, including ViT-RF, Seasonal ViT-RF, Seasonal DeepViT-RF [63], and Seasonal MAEViT-RF where ViT is pre-trained by a Masked Autoencoder [22], as displayed in Table 1.

Original: These usethe same RF and sinsusoidal mappings as described previously to input the temporal information intothe model.
Revised: These usethe same RF and sinsusoidal mappings as described previously to input the temporal information intothe model.

Original: We note that we are unaware of Vision Transformers being used in this manner, and thatall these models represent novel formulations.
Revised: We note that we are unaware of Vision Transformers being used in this manner, and thatall these models represent novel formulations.

Original: It can be observed that standard ViT-RF model fails toforecast PM 2 . 5 without seasonality incorporated, just as in CNN-RF.
Revised: It can be observed that standard ViT-RF model fails toforecast PM 2 . 5 without seasonality incorporated, just as in CNN-RF.

Original: After introducing seasonalityby mapping t into sinusoidal features, ViT-based joint models yield higher correlation and smallererror than CNN-RF but still underperform CNN-based ICK y .
Revised: After introducing seasonalityby mapping t into sinusoidal features, ViT-based joint models yield higher correlation and smallererror than CNN-RF but still underperform CNN-based ICK y .

Original: We also considered using a ViT-basedarchitecture for the CNN ICK y , and observed similar performance in these ICK y variants.
Revised: We also considered using a ViT-basedarchitecture for the CNN ICK y , and observed similar performance in these ICK y variants.

Original: 5.3 UCI Machine Learning Repository Data
Revised: 5.3 UCI Machine Learning Repository Data

Original: To see if our ICK y framework generalizes to other domains, we acquire another data set containingthe normalized productivity and corresponding features of garment workers from the UCI Machine
Revised: To see if our ICK y framework generalizes to other domains, we acquire another data set containingthe normalized productivity and corresponding features of garment workers from the UCI Machine

Original: Learning Repository.
Revised: Learning Repository.

Original: Imran et al.
Revised: Imran et al.

Original: [1] employed a dense MLP with 2 hidden layers to predict theworker productivity with collected features such as date, team number, targeted productivity, etc.
Revised: [1] employed a dense MLP with 2 hidden layers to predict theworker productivity with collected features such as date, team number, targeted productivity, etc.

Original: Totest our ICK y framework, we separate out the date and use it as the low-dimensional information.
Revised: Totest our ICK y framework, we separate out the date and use it as the low-dimensional information.

Original: The rest of the features (excluding the temporal information) are then concatenated together toserve as the high-dimensional information.
Revised: The rest of the features (excluding the temporal information) are then concatenated together toserve as the high-dimensional information.

Original: Observing that the daily averaged worker productivityhas an approximate monthly trend , we again use an exponential-sine-squared kernel.
Revised: Observing that the daily averaged worker productivityhas an approximate monthly trend , we again use an exponential-sine-squared kernel.

Original: The networkarchitecture of ICK y is the same as that of the two-hidden-layer MLP benchmark.
Revised: The networkarchitecture of ICK y is the same as that of the two-hidden-layer MLP benchmark.

Original: To demonstratethe strength of ICK y compared to other methods that equip NNs with GPs, we also add 2 additionalbenchmarks here: a Gaussian Neural Process (GNP) [7] and an Attentive Gaussian Neural Process(AGNP) [29].
Revised: To demonstratethe strength of ICK y compared to other methods that equip NNs with GPs, we also add 2 additionalbenchmarks here: a Gaussian Neural Process (GNP) [7] and an Attentive Gaussian Neural Process(AGNP) [29].

Original: Based on the results shown in Table 2, ICK y has the best performance when theperiod parameter of the kernel is set to be T = 30 (days) and it outperforms both MLP and NPbenchmarks by almost one order of magnitude .
Revised: Based on the results shown in Table 2, ICK y has the best performance when theperiod parameter of the kernel is set to be T = 30 (days) and it outperforms both MLP and NPbenchmarks by almost one order of magnitude .

Original: When we set T = 2 or T = 7 , this improvement isless signiﬁcant, which aligns with our initial observation that the daily averaged productivity has amonthly seasonal trend.
Revised: When we set T = 2 or T = 7 , this improvement isless signiﬁcant, which aligns with our initial observation that the daily averaged productivity has amonthly seasonal trend.

Original: It is also worth noting that the GNP benchmarks here yield larger errors than
Revised: It is also worth noting that the GNP benchmarks here yield larger errors than

Original: MLPs.
Revised: MLPs.

Original: A possible explanation is that GNP does not allow explicit assignment of a stationary kernel(as the kernel models a posterior covariance)
Revised: A possible explanation is that GNP does not allow explicit assignment of a stationary kernel(as the kernel models a posterior covariance)

Original: so it is hard for GNP to identify speciﬁc patterns in thedata such as seasonality without being given the pattern a priori .
Revised: so it is hard for GNP to identify speciﬁc patterns in thedata such as seasonality without being given the pattern a priori .

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  Discussion
Revised:  Discussion

Original: Efﬁciency, Flexibility, and Generalization Compared to exact composite GP models which scale
Revised: Efﬁciency, Flexibility, and Generalization Compared to exact composite GP models which scale

Original: O ( N 3 ) , the training process of our ICK framework is more efﬁcient as it leverages standard backpropagation to learn both the paramters of NN and the kernel function.
Revised: O ( N 3 ) , the training process of our ICK framework is more efﬁcient as it leverages standard backpropagation to learn both the paramters of NN and the kernel function.

Original: In addition, the networkarchitecture of ICK can be very simple, as can be seen in all 3 experiments of ours, which furtherreduces its time and space complexity.
Revised: In addition, the networkarchitecture of ICK can be very simple, as can be seen in all 3 experiments of ours [Figure] which furtherreduces its time and space complexity.

Original: Besides efﬁciency, our ICK framework is more ﬂexiblecompared to other joint models (i.e. BNNs and CNN-RF).
Revised: Besides efﬁciency, our ICK framework is more ﬂexiblecompared to other joint models (i [Figure] e. BNNs and CNN-RF).

Original: To be speciﬁc, the BNNs implemented by
Revised: To be speciﬁc, the BNNs implemented by

Original: Pearce et al.
Revised: Pearce et al.

Original: [43] cannot simulate complicated kernels such as the spectral mixture kernel we usein Section 5.1.
Revised: [43] cannot simulate complicated kernels such as the spectral mixture kernel we usein Section 5.1.

Original: The CNN-RF joint model implemented by Zheng et al.
Revised: The CNN-RF joint model implemented by Zheng et al.

Original: [61] requires us to carefullydesign the input pre-processing procedure.
Revised: [61] requires us to carefullydesign the input pre-processing procedure.

Original: Also, ICK generalizes well to unseen data even withvery limited training samples.
Revised: Also, ICK generalizes well to unseen data even withvery limited training samples.

Original: There is a potential concern that ICKy may run into computationalchallenges when a large number of inducing points are required.
Revised: There is a potential concern that ICKy may run into computationalchallenges when a large number of inducing points are required.

Original: This was not a problem in ourexperiments, but in large scale models this could be tackled by considering conjugate gradientmethods, which have been recently popular in GP inference [15].
Revised: This was not a problem in ourexperiments, but in large scale models this could be tackled by considering conjugate gradientmethods, which have been recently popular in GP inference [15].

Original: Limitations A major limitation of ICK lies in our method of combining latent representations as thenature of inner product (i.e. the effect of multiplying small numbers) may cause vanishing gradientproblems when we have a large number of sources of information (i.e. M is large).
Revised: Limitations A major limitation of ICK lies in our method of combining latent representations as thenature of inner product (i.e. the effect of multiplying small numbers) may cause vanishing gradientproblems when we have a large number of sources of information (i.e. M is large).

Original: Furthermore,this paper only discusses the theoretical relationship between ICK and composite GPR a priori .
Revised: Furthermore,this paper only discusses the theoretical relationship between ICK and composite GPR a priori .

Original: Thisrelationship will not exactly hold true a posteriori , although empirical results [34] and theoreticalresults [24] in slightly different contexts suggest that they may be close.
Revised: Thisrelationship will not exactly hold true a posteriori , although empirical results [34] and theoreticalresults [24] in slightly different contexts suggest that they may be close.

Original: Future work will evaluatethis gap.
Revised: Future work will evaluatethis gap by exploring Bayesian neural networks and a posteriori properties.

Original: Broader Impacts We believe our framework is extensively applicable to regression problems inmany ﬁelds of study involving high-dimensional data and multiple sources of information withperceptible trends, such as remote sensing, spatial statistics, or clinical diagnosis.
Revised: Broader Impacts We believe our framework is extensively applicable to regression problems inmany ﬁelds of study involving high-dimensional data and multiple sources of information withperceptible trends, such as remote sensing, spatial statistics, or clinical diagnosis.

Original:  Conclusion
Revised:  Conclusion

Original: This paper presents a novel yet surprisingly simple Implicit Composite Kernel (ICK) framework tolearn from hybrid data containing both high-dimensional information and low-dimensional information with prior knowledge.
Revised: This paper presents a novel yet surprisingly simple Implicit Composite Kernel (ICK) framework tolearn from hybrid data containing both high-dimensional information and low-dimensional information with prior knowledge.

Original: We ﬁrst analytically show the resemblance between ICK and composite
Revised: We ﬁrst analytically show the resemblance between ICK and composite

Original: GPR models and then conduct experiments using both synthetic and real-world data.
Revised: GPR models and then conduct experiments using both synthetic and real-world data.

Original: It appearsthat ICK outperforms various benchmark models in our experiments with lowest prediction errorsand highest correlations even with very limited data.
Revised: It appearsthat ICK outperforms various benchmark models in our experiments with lowest prediction errorsand highest correlations even with very limited data.

Original: Overall, we show that our ICK framework isexceptionally powerful when learning from hybrid data with our prior knowledge incorporated, andwe hope our work can inspire more future research on joint machine learning models, enhancing theirperformance, efﬁciency, ﬂexibility, and generalization capability.Checklist For all authors... [Table]
Revised: Overall, we show that our ICK framework isexceptionally powerful when learning from hybrid data with our prior knowledge incorporated, andwe hope our work can inspire more future research on joint machine learning models, enhancing theirperformance, efﬁciency, ﬂexibility, and generalization capability.

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes] See Section 1.

Original: [Table]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We areunaware of direct negative societal impacts.

Original: [Table]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: [Table]

Original: If you are including theoretical results...

Original: [Table]

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 4.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Section 4.

Original: [Table]

Original: If you ran experiments...

Original: [Table]

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: [Table]

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] The code, data,

Original: [Table]

Original: and instructions to reproduce the experimental results are provided via the anonymousrepository URL in the abstract.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5. 
Revised: 5. If you used crowdsourcing

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 5.1 and 5.3.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See section

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]

Original: [Table]

Original: (b) Did you mention the license of the assets?

Original: [Yes] See Appendix I

Original: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: We have included code, with the link in the abstract to an anonymous repository.

Original: This

Original: will be switched to a user-maintained repository if accepted.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: We do not release data.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content? [Yes] None of our datasets contain personally identiﬁable information or offensive content. If you used crowdsourcing or conducted research with human subjects...
Revised: or conducted research with human subjects.

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Figure]
Revised: [Figure]

Original: A ICK with More Than Two Kernels
Revised: A ICK with More Than Two Kernels

Original: Besides the visualization presented in Figure1, we also show our ICK framework for processing datax = (cid:8) x (1) , x (2) , ..., x ( M ) (cid:9) with M > 2 sources of information in Figure A2.
Revised: Besides the visualization presented in Figure we also show our ICK framework for processing datax = (cid:8) x (1) , x (2) , ..., x ( M ) (cid:9) with M > 2 sources of information in Figure A2.

Original: Here K (2) , ..., K ( M )represent different types of kernels with trainable parameters.
Revised: Here K (2) , ..., K ( M )represent different types of kernels with trainable parameters.

Original: The ﬁnal prediction is calculated by achained inner product of all extracted representations ˆ y = (cid:80) pk =1 (cid:81)
Revised: The ﬁnal prediction is calculated by achained inner product of all extracted representations ˆ y = (cid:80) pk =1 (cid:81)

Original: Mm =1 z ( m ) k .
Revised: Mm =1 z ( m ) k .

Original: To conﬁrm that ICK y can work with more than 2 kernels, we construct another synthetic data setcontaining 3000 data points in total.
Revised: To conﬁrm that ICK y can work with more than 2 kernels, we construct another synthetic data setcontaining 3000 data points in total.

Original: Each input x = (cid:8) x (1) , x (2) , x (3) (cid:9) has 3 sources of information.
Revised: Each input x = (cid:8) x (1) , x (2) , x (3) (cid:9) has 3 sources of information.

Original: The output y is generated by y = x
Revised: The output y is generated by y = x

Original: (3) tanh (cid:0) 2 x (1) cos 2 (cid:0) πx (2) / 50 (cid:1)(cid:1) + (cid:15) where (cid:15) is a Gaussian noiseterm.
Revised: (3) tanh (cid:0) 2 x (1) cos 2 (cid:0) πx (2) / 50 (cid:1)(cid:1) + (cid:15) where (cid:15) is a Gaussian noiseterm.

Original: We process x (1) with a small single-hidden-layer NN, x (2) with an exponential sine squaredkernel, and x (3) with a radial-basis function (RBF) kernel.
Revised: We process x (1) with a small single-hidden-layer NN, x (2) with an exponential sine squaredkernel, and x (3) with a radial-basis function (RBF) kernel.

Original: Figure A1 shows the prediction resultsas we progressively add more sources of information into our ICK y framework with correspondingkernel functions.
Revised: Figure A1 shows the prediction resultsas we progressively add more sources of information into our ICK y framework with correspondingkernel functions.

Original: It can be observed that ICK y yields both smallest error and highest correlation
Revised: It can be observed that ICK y yields both smallest error and highest correlation

Original: [Figure]
Revised: [Figure]

Original: with information from all 3 different sources.
Revised: with information from all 3 different sources.

Original: Hence, ICK y works well with the M = 3 case and theregression performance is improved as we add in more information related to the 
Revised: Hence, ICK y works well with the M = 3 case and theregression performance is improved as we add in more information related to the target.

Original: B Random Fourier Features
Revised: B Random Fourier FeaturesB.1 Methodology

Original: B.1 Methodology

Original: Random Fourier Features (RFF) is another popular approximation method used for kernel learning[45].
Revised: Random Fourier Features (RFF) is another popular approximation method used for kernel learning[45].

Original: Unlike the Nyström method which approximates the entire kernel matrix, RFF directly approximates the kernel function K using some randomized feature mapping φ : R D m → R 2 d m suchthat K (cid:16) x ( m ) i , x ( m [Equation] j [Equation]
Revised: Unlike the Nyström method which approximates the entire kernel matrix, RFF directly approximates the kernel function K using some randomized feature mapping φ : R D m → R 2 d m suchthat K (cid:16) x ( m ) i , x ( m ) j (cid:17) ≈ φ (cid:16) x ( m ) i (cid:17) T φ (cid:16) x ( m ) j (cid:17) .

Original: m ) j (cid:17) . To obtain the feature mapping φ , based on Bochner’stheorem, we ﬁrst compute the Fourier transform p of kernel K
Revised: To obtain the feature mapping φ , based on Bochner’stheorem, we ﬁrst compute the Fourier transform p of kernel K

Original: [Equation]
Revised: [Equation]

Original: where δ = x ( m )
Revised: where δ = x ( m )

Original: i − x ( m ) j .
Revised: i − x ( m ) j .

Original: Then we draw d m i.i.d.
Revised: Then we draw d m i.i.d.

Original: samples ω 1 , ω 2 , ..., ω
Revised: samples ω 1 , ω 2 , ..., ω

Original: d m from p ( ω ) and constructthe feature mapping φ as follows
Revised: d m from p ( ω ) and constructthe feature mapping φ as follows

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since φ (cid:0) x ( m ) (cid:1) ∈ R 2 d m , we need to set d m = p/ 2 when using RFF as a kernel-to-latent-spacetransformation.
Revised: Since φ (cid:0) x ( m ) (cid:1) ∈ R 2 d m , we need to set d m = p/ 2 when using RFF as a kernel-to-latent-spacetransformation.

Original: In addition, since RFF involves sampling from a distribution, the kernel parametersare thus not directly differentiable and we need to apply a reparameterization trick [35] to learn thoseparameters.
Revised: In addition, since RFF involves sampling from a distribution, the kernel parametersare thus not directly differentiable and we need to apply a reparameterization trick [35] to learn thoseparameters.

Original: B.2 Experimental Results
Revised: B.2 Experimental Results

Original: B.2.1 Synthetic Data
Revised: B.2.1 Synthetic Data

Original: We use the same toy data set where each data point x = (cid:8) x (1) , x (2) , x (3) (cid:9) contains 3 sources ofinformation as described in Appendix A.
Revised: We use the same toy data set where each data point x = (cid:8) x (1) , x (2) , x (3) (cid:9) contains 3 sources ofinformation as described in Appendix A.

Original: Also, we use the same types of kernels as those in ICK y asdiscussed in Appendix A.
Revised: Also, we use the same types of kernels as those in ICK y asdiscussed in Appendix A.

Original: The only difference here is that we use RFF instead of Nyström method totransform the kernel matrix into the latent space in ICK r framework.
Revised: The only difference here is that we use RFF instead of Nyström method totransform the kernel matrix into the latent space in ICK r framework.

Original: The results are displayed in Figure B1.
Revised: The results are displayed in Figure B1.

Original: It can be observed that when we add in only the sideinformation x (2) along with the exponential sine squared kernel, both the correlation and the predictiveperformance are improved (though not as good as the results from ICK y as shown in Figure A1).
Revised: It can be observed that when we add in only the sideinformation x (2) along with the exponential sine squared kernel, both the correlation and the predictiveperformance are improved (though not as good as the results from ICK y as shown in Figure A1).

Original: [Figure]
Revised: [Figure]

Original: However, after we further include x (3) with the RBF kernel, we realize that the parameters of ICK rbecome very hard to optimize and it fails to make valid predictions and starts to guess randomlyaround zero.
Revised: However, after we further include x (3) with the RBF kernel, we realize that the parameters of ICK rbecome very hard to optimize and it fails to make valid predictions and starts to guess randomlyaround zero.

Original: B.2.2 Remote Sensing Data
Revised: B.2.2 Remote Sensing Data

Original: We also try ICK r on the forecasting task using the remote sensing data (see Section 5.2) and comparethe results with those from ICK y .
Revised: We also try ICK r on the forecasting task using the remote sensing data (see Section 5.2) and comparethe results with those from ICK y .

Original: Each data point x = { x, t } contains a satellite image x as thehigh-dimensional information and its corresponding timestamp t as the low-dimensional information.
Revised: Each data point x = { x, t } contains a satellite image x as thehigh-dimensional information and its corresponding timestamp t as the low-dimensional information.

Original: The satellite images are processed with a two-layer CNN and the timestamps are processed with anexponential-sine-squared kernel with a period of T = 365 (days).
Revised: The satellite images are processed with a two-layer CNN and the timestamps are processed with anexponential-sine-squared kernel with a period of T = 365 (days).

Original: As can be observed from Figure
Revised: As can be observed from Figure

Original: B2, ICK r yields much higher error compared to ICK y .
Revised: B2, ICK r yields much higher error compared to ICK y .

Original: C Estimated Kernel Matrix and its Eigen-spectrum
Revised: C Estimated Kernel Matrix and its Eigen-spectrum

Original: [Figure]
Revised: [Figure]

Original: (c [Figure] We ﬁrst examine whether ICK y can retrieve the spectral mixture kernel in the prediction task.
Revised: We ﬁrst examine whether ICK y can retrieve the spectral mixture kernel in the prediction task.

Original: Afterﬁtting the parameters of the spectral mixture kernel in ICK y , we compute the kernel matrix K ICK yusing these learned parameters and compare it with the true kernel matrix K true by calculating theabsolute difference between them as displayed in Figure C1.
Revised: Afterﬁtting the parameters of the spectral mixture kernel in ICK y , we compute the kernel matrix K ICK yusing these learned parameters and compare it with the true kernel matrix K true by calculating theabsolute difference between them as displayed in Figure C1.

Original: As can be observed, K ICK y and K trueare similar and their absolute difference is relatively small, indicating that ICK y can approximatelyretrieve the spectral mixture kernel.
Revised: As can be observed, K ICK y and K trueare similar and their absolute difference is relatively small, indicating that ICK y can approximatelyretrieve the spectral mixture kernel.

Original: Yang et al.
Revised: Yang et al.

Original: [59] studied the fundamental differencebetween Nyström method and Random Fourier Features (RFF).
Revised: [59] studied the fundamental differencebetween Nyström method and Random Fourier Features (RFF).

Original: They conclude that Nyström-methodbased approaches can yield much better generalization error bound than RFF-based approaches if thereexists a large gap in the eigen-spectrum of the kernelmatrix.
Revised: They conclude that Nyström-methodbased approaches can yield much better generalization error bound than RFF-based approaches if thereexists a large gap in the eigen-spectrum of the kernelmatrix.

Original: This phenomenon is mainly caused by howthese two methods construct their basis functions.
Revised: This phenomenon is mainly caused by howthese two methods construct their basis functions.

Original: Inparticular, the basis functions used by RFF are sampled from a Gaussian distribution that is independentfrom the training examples, while the basis functionsused by the Nyström method are sampled from thetraining samples so they are data-dependent.
Revised: Inparticular, the basis functions used by RFF are sampled from a Gaussian distribution that is independentfrom the training examples, while the basis functionsused by the Nyström method are sampled from thetraining samples so they are data-dependent.

Original: In oursynthetic data experiments, we train our ICK framework using a batch size of 50.
Revised: In oursynthetic data experiments, we train our ICK framework using a batch size of 50.

Original: The eigenvalues of thekernel matrices computed from the ﬁrst 4 batches ofthe synthetic data set are displayed in Figure C2.
Revised: The eigenvalues of thekernel matrices computed from the ﬁrst 4 batches ofthe synthetic data set are displayed in Figure C2.

Original: Itcan be observed that the ﬁrst few eigenvalues of thekernel matrix are much larger than the remaining eigenvalues.
Revised: Itcan be observed that the ﬁrst few eigenvalues of thekernel matrix are much larger than the remaining eigenvalues.

Original: Namely, there exists a large gap in theeigen-spectrum of the kernel matrix, which helps explain why ICK y has a much better performancethan ICK r .
Revised: Namely, there exists a large gap in theeigen-spectrum of the kernel matrix, which helps explain why ICK y has a much better performancethan ICK r .

Original: D Simulation of an Additive Kernel
Revised: D Simulation of an Additive Kernel

Original: [Figure]
Revised: [Figure]

Original: While ICK is designed to capture multiplicative ker- nels, we evaluated how well it could capture additive kernels.
Revised: While ICK is designed to capture multiplicative kernels, we evaluated how well it could capture additive kernels.

Original: We conduct experiments using anothersynthetic data set generated by an additive kernely ∼ GP (0 , K 1 + K 2 ) with the same training settings.
Revised: We conduct experiments using anothersynthetic data set generated by an additive kernely ∼ GP (0 , K 1 + K 2 ) with the same training settings.

Original: As shown in Figure D1, ICK y again outperformsplain MLP and MLP-RF in both the prediction andthe forecasting tasks.
Revised: As shown in Figure D1, ICK y again outperformsplain MLP and MLP-RF in both the prediction andthe forecasting tasks.

Original: Moreover, we again test plain
Revised: Moreover, we again test plain

Original: MLP, MLP-RF, and ICK y on the prediction task usingdifferent number of training samples.
Revised: MLP, MLP-RF, and ICK y on the prediction task usingdifferent number of training samples.

Original: As displayedin Figure D2, ICK y yields the smallest error amongall the 3 frameworks.
Revised: As displayedin Figure D2, ICK y yields the smallest error amongall the 3 frameworks.

Original: Also, the performance gapbetween ICK y and the other 2 benchmark modelsshrink as we feed in more training data.
Revised: Also, the performance gapbetween ICK y and the other 2 benchmark modelsshrink as we feed in more training data.

Original: Therefore,we conclude ICK y is robust enough to simulate bothadditive and multiplicative kernels.
Revised: Therefore,we conclude ICK y is robust enough to simulate bothadditive and multiplicative kernels.

Original: E Number of Inducing Points
Revised: E Number of Inducing Points

Original: As discussed in Section 4.2.1, as we increase the number of inducing points p , we expect theapproximation error between the true kernel matrix K and the approximated kernel matrix ˆ K todecrease.
Revised: As discussed in Section 4.2.1, as we increase the number of inducing points p , we expect theapproximation error between the true kernel matrix K and the approximated kernel matrix ˆ K todecrease.

Original: Here, we empirically show how the value of p impacts our predictions.
Revised: Here, we empirically show how the value of p impacts our predictions.

Original: In Figure E1a, weplot the prediction error of ˆ y = f ICK y (cid:0) x (1) , x (2) , x (3) (cid:1) against the number of inducing points usingthe synthetic data generated in Appendix A. As can be observed, the prediction error drops sharply aswe raise p from a small value (e.g. p = 2 ).
Revised: In Figure E1a, weplot the prediction error of ˆ y = f ICK y (cid:0) x (1) , x (2) , x (3) (cid:1) against the number of inducing points usingthe synthetic data generated in Appendix A. As can be observed, the prediction error drops sharply aswe raise p from a small value (e.g. p = 2 ).

Original: When p is relatively large, increasing p yields smallerimprovement on the predictions.
Revised: When p is relatively large, increasing p yields smallerimprovement on the predictions.

Original: Additionally, in Figure E1b, we plot the total training time againstp .
Revised: Additionally, in Figure E1b, we plot the total training time againstp .

Original: The total training time is dependent on how long a single iteration takes and the total number ofepochs required.
Revised: The total training time is dependent on how long a single iteration takes and the total number ofepochs required.

Original: We note that once p > 80 the training time is relatively ﬂat, which is due to thefact that the total computation in the Cholesky is less than the computation in the neural network.
Revised: We note that once p > 80 the training time is relatively ﬂat, which is due to thefact that the total computation in the Cholesky is less than the computation in the neural network.

Original: Interestingly, it appears that when p is very small, ICK y takes longer to converge due to the need formany more epochs.
Revised: Interestingly, it appears that when p is very small, ICK y takes longer to converge due to the need formany more epochs.

Original: As we increase p , the training time goes down and then goes up again due to the
Revised: As we increase p , the training time goes down and then goes up again due to the

Original: [Figure]
Revised: [Figure]

Original: computational complexity, i.e. O ( p 3 ) , of the Cholesky decomposition.
Revised: computational complexity, i.e. O ( p 3 ) , of the Cholesky decomposition.

Original: Based on these observations,we are overly concerned about the computational complexity for reasonable values of p .
Revised: Based on these observations,we are overly concerned about the computational complexity for reasonable values of p .

Original: F Visualization of Remote Sensing Data as Time Series
Revised: F Visualization of Remote Sensing Data as Time Series

Original: To better illustrate the results in Section 5.2, we visualize those results in the form of time series.
Revised: To better illustrate the results in Section 5.2, we visualize those results in the form of time series.

Original: Asshown in Figure F1, the plain CNN-RF model does not work as it tends to forecast constant PM 2 . 5values.
Revised: Asshown in Figure F1, the plain CNN-RF model does not work as it tends to forecast constant PM 2 . 5values.

Original: In contrast, both the seasonal CNN-RF model and our ICK y framework captures the overalltrend of the true daily averaged PM 2 .
Revised: In contrast, both the seasonal CNN-RF model and our ICK y framework captures the overalltrend of the true daily averaged PM 2 .

Original: 5 values, but the forecasted values by ICK y are smoother andyield smaller error.
Revised: 5 values, but the forecasted values by ICK y are smoother andyield smaller error.

Original: [Figure]
Revised: [Figure]

Original: G Experimental Details
Revised: G Experimental Details

Original: G.1 Synthetic Data
Revised: G.1 Synthetic Data

Original: We use the GPytorch package [15] to generate the synthetic data.
Revised: We use the GPytorch package [15] to generate the synthetic data.

Original: Before feeding x (1) into MLP, weﬁrst map x (1) into higher dimension using an unsupervised algorithm called Totally Random Trees
Revised: Before feeding x (1) into MLP, weﬁrst map x (1) into higher dimension using an unsupervised algorithm called Totally Random Trees

Original: Embedding [48].
Revised: Embedding [48].

Original: All the MLP structures in this experiment (including those in MLP-RF and ICK y )contain one single fully connected (FC) layer of width 1000, which serves as a simple benchmarksince a one-hidden-layer MLP can only capture linear relationship between the input and output.
Revised: All the MLP structures in this experiment (including those in MLP-RF and ICK y )contain one single fully connected (FC) layer of width 1000, which serves as a simple benchmarksince a one-hidden-layer MLP can only capture linear relationship between the input and output.

Original: Formodel training, we optimize a Mean Squared Error (MSE) objective using Adam optimizer [30] witha weight decay of 0.1.
Revised: Formodel training, we optimize a Mean Squared Error (MSE) objective using Adam optimizer [30] with

Original: [Table]
Revised: [Table]

Original: G.2 Remote Sensing Data
Revised: G.2 Remote Sensing Data

Original: The model architecture and training details are listed in Table 3.
Revised: The model architecture and training details are listed in Table 3.

Original: Here p denotes the length of latentrepresentations z as discussed in Section 4.
Revised: Here p denotes the length of latentrepresentations z as discussed in Section 4.

Original: Note that we use stochastic gradient descent (SGD)optimizer with a momentum of 0.9 for ICK y as we realize that SGD helps ICK y ﬁnd a local minimumon the objective more efﬁciently.
Revised: Note that we use stochastic gradient descent (SGD)optimizer with a momentum of 0.9 for ICK y as we realize that SGD helps ICK y ﬁnd a local minimumon the objective more efﬁciently.

Original: We use MSE objective for ICK y and all benchmark models in thisexperiment.
Revised: We use MSE objective for ICK y and all benchmark models in thisexperiment.

Original: G.3 UCI Machine Learning Repository Data
Revised: G.3 UCI Machine Learning Repository Data

Original: The MLPs (including the MLP part in ICK y ) in this experiment share the same structure as the oneused in [1], which consist of 3 hidden layers of width 128, 32, and 32, respectively.
Revised: The MLPs (including the MLP part in ICK y ) in this experiment share the same structure as the oneused in [1], which consist of 3 hidden layers of width 128, 32, and 32, respectively.

Original: For plain MLP,cyclic MLP, and ICK y , we use the mean absolute error (MAE) objective to put less weight on theoutliers and thus enhance the model performance.
Revised: For plain MLP,cyclic MLP, and ICK y , we use the mean absolute error (MAE) objective to put less weight on theoutliers and thus enhance the model performance.

Original: For GNP and AGNP, we maximize a biased Monte
Revised: For GNP and AGNP, we maximize a biased Monte

Original: Carlo estimate of the log-likelihood objective as discussed in [37].
Revised: Carlo estimate of the log-likelihood objective as discussed in [37].

Original: All these objectives are optimizedby an Adam optimizer with β 1 = 0 .
Revised: All these objectives are optimizedby an Adam optimizer with β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: H Adapting ICK for Classiﬁcation
Revised: H Adapting ICK for Classiﬁcation

Original: While regression tasks are the primary motivation for this paper, there are many ways to adapt GPRfor classiﬁcation tasks.
Revised: While regression tasks are the primary motivation for this paper, there are many ways to adapt GPRfor classiﬁcation tasks.

Original: For example, a binary classiﬁcation model can be created by using a sigmoid[55] or probit link [8] on the output of the GP.
Revised: For example, a binary classiﬁcation model can be created by using a sigmoid[55] or probit link [8] on the output of the GP.

Original: Succinctly, given a function f ( x ) ∼ GP (0 , K ( x , x (cid:48) )) ,the binary outcome probability is be given as p ( y = 1 | f ( x ))
Revised: Succinctly, given a function f ( x ) ∼ GP (0 , K ( x , x (cid:48) )) ,the binary outcome probability is be given as p ( y = 1 | f ( x ))

Original: =
Revised: =

Original: σ ( f ( x )) .
Revised: σ ( f ( x )) .

Original: Likewise, a multipleclassiﬁcation model can be constructed by using a multi-output GP (or multiple GPs) and puttingthe outputs through a softmax function [55] or multinomial probit link [20].
Revised: Likewise, a multipleclassiﬁcation model can be constructed by using a multi-output GP (or multiple GPs) and puttingthe outputs through a softmax function [55] or multinomial probit link [20].

Original: This strategy can besummarized by calculating C different functions f c ( x ) ∼ GP (0 , K
Revised: This strategy can besummarized by calculating C different functions f c ( x ) ∼ GP (0 , K

Original: ( x , x (cid:48) )) for c = 1 , ..., C , where
Revised: ( x , x (cid:48) )) for c = 1 , ..., C , where

Original: C is the number of classes, and then calculating the class probabilities through a link function,p ( y | x ) = softmax ([ f 1 ( x ) , f 2 ( x ) , ..., f C ( x )]) .
Revised: C is the number of classes, and then calculating the class probabilities through a link function,p ( y | x ) = softmax ([ f 1 ( x ) , f 2 ( x ) , ..., f C ( x )]) .

Original: This same logic can be used to construct a multiple classiﬁcation model from ICK y .
Revised: This same logic can be used to construct a multiple classiﬁcation model from ICK y .

Original: Succinctly, letr c = f NN,c ( x (1) )
Revised: Succinctly, letr c = f NN,c ( x (1) )

Original: (cid:12) z (2) c , where f NN,c denotes a neural network speciﬁc to the c
Revised: (cid:12) z (2) c , where f NN,c denotes a neural network speciﬁc to the c

Original: th class and z (2) crepresents the Nyström approximation speciﬁc to the kernel for the c th class.
Revised: th class and z (2) crepresents the Nyström approximation speciﬁc to the kernel for the c th class.

Original: We note that often in a multi-output case the kernel parameters are shared, and so z (2) c would be an identical vector foreach class.
Revised: We note that often in a multi-output case the kernel parameters are shared, and so z (2) c would be an identical vector foreach class.

Original: Then, the output probabilities for a data sample as p ( y | x ) = softmax ([ r 1 , . .
Revised: Then, the output probabilities for a data sample as p ( y | x ) = softmax ([ r 1 , . .

Original: , r C ]) .
Revised: , r C ]) .

Original: Thisframework is learned with a cross-entropy loss.
Revised: Thisframework is learned with a cross-entropy loss.

Original: To provide proof-of-concept of this multiple classiﬁcation strategy, we implemented this model on aversion of Rotating MNIST.
Revised: To provide proof-of-concept of this multiple classiﬁcation strategy, we implemented this model on aversion of Rotating MNIST.

Original: In this task, a dataset was created by rotating each image in the datasetby a uniform random value φ ∈ [0 , 2 π ) , thus creating a dataset with 60,000 images each with anassociated rotation covariate φ .
Revised: In this task, a dataset was created by rotating each image in the datasetby a uniform random value φ ∈ [0 , 2 π ) , thus creating a dataset with 60,000 images each with anassociated rotation covariate φ .

Original: We implemented the above multiple classiﬁcation model with aperiodic kernel over the rotation angle.
Revised: We implemented the above multiple classiﬁcation model with aperiodic kernel over the rotation angle.

Original: This strategy yielded an accuracy of 92.3% on the validationdata.
Revised: This strategy yielded an accuracy of 92.3% on the validationdata.

Original: This is lower than methods such as spatial transformers [25] that report accuracy greater than99%.
Revised: This is lower than methods such as spatial transformers [25] that report accuracy greater than99%.

Original: However, those models explicitly use the fact that the information is simply rotated, whereas
Revised: However, those models explicitly use the fact that the information is simply rotated, whereas

Original: ICK is modeling a smooth transformation in the prediction function as a function of angle.
Revised: ICK is modeling a smooth transformation in the prediction function as a function of angle.

Original: This
Revised: This

Original: ICK classiﬁcation model is much closer in concept to the way Rotating MNIST is used to evaluateunsupervised domain adaptation.
Revised: ICK classiﬁcation model is much closer in concept to the way Rotating MNIST is used to evaluateunsupervised domain adaptation.

Original: While the evaluation strategy is different than our random validationset, the state-of-the-art accuracy on unsupervised domain adaption is 87.1% [52].
Revised: While the evaluation strategy is different than our random validationset, the state-of-the-art accuracy on unsupervised domain adaption is 87.1% [52].

Original: Due to the lack ofcomplete and fair comparisons, we are not claiming that ICK y is state-of-the-art for classiﬁcation, but
Revised: Due to the lack ofcomplete and fair comparisons, we are not claiming that ICK y is state-of-the-art for classiﬁcation, but

Original: ICK y ’s classiﬁcation model does seem reasonable and viable based upon this result.
Revised: ICK y ’s classiﬁcation model does seem reasonable and viable based upon this result.

Original: I Generation, Accessibility, and Restrictions of the Data
Revised: I Generation, Accessibility, and Restrictions of the Data

Original: The synthetic data y ∼ GP (0 , K 1 K 2 ) in Section 5.1 and y ∼ GP (0 , K 1 + K 2 ) in Appendix Dare generated using the GPyTorch package.
Revised: The synthetic data y ∼ GP (0 , K 1 K 2 ) in Section 5.1 and y ∼ GP (0 , K 1 + K 2 ) in Appendix Dare generated using the GPyTorch package.

Original: The remote sensing data in Section 5.2 is downloadedusing PlanetScope API whose content is protected by copyright and/or other intellectual propertylaws.
Revised: The remote sensing data in Section 5.2 is downloadedusing PlanetScope API whose content is protected by copyright and/or other intellectual propertylaws.

Original: To access the data on PlanetScope, the purchase of an end-user license is required.
Revised: To access the data on PlanetScope, the purchase of an end-user license is required.

Original: When thismanuscript is accepted, we will provide the codes we used to acquire the data.
Revised: When thismanuscript is accepted, we will provide the codes we used to acquire the data.

Original: The UCI machinelearning repository data we use in Section 5.3 has an open access license, meaning that the data isfreely available online.
Revised: The UCI machinelearning repository data we use in Section 5.3 has an open access license, meaning that the data isfreely available online.