Original: I NPUT D EPENDENT S PARSE G AUSSIAN P ROCESSES
Revised: I NPUT D EPENDENT S PARSE G AUSSIAN P ROCESSES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Gaussian Processes (GPs) are Bayesian models that provide uncertainty estimates associated to the predictions made.
Revised: Gaussian Processes (GPs) are Bayesian models that provide uncertainty estimates associated to the predictions made.

Original: They are also very ﬂexible due to their nonparametric nature.
Revised: They are also very ﬂexible due to their nonparametric nature.

Original: Nevertheless, GPs suffer from poor scalability as the number of training instances N increases. More precisely, they have a cubic cost with respect to N .
Revised: Nevertheless, GPs suffer from poor scalability as the number of training instances N increases since their cost is cubic in N .

Original: To overcome this problem, sparse GP approximations are often used, where a set of M (cid:28) N inducing points is introduced during training.
Revised: To overcome this problem, sparse GP approximations are often used, where a set of M (cid:28) N inducing points is introduced during training.

Original: The location of the inducing points is learned by considering them as parameters of an approximate posterior distribution q .
Revised: The location of the inducing points is learned by considering them as parameters of an approximate posterior distribution q .

Original: Sparse GPs, combined with variational inference for inferring q , reduce thetraining cost of GPs  to O ( M 3 ) .
Revised: Sparse GPs, combined with variational inference for inferring q , reduce the cost of GPs per iteration to O ( M 3 ) .

Original: Critically, the inducing points determine the ﬂexibility of the model and they are often located in regions of the input space where the latent function changes.
Revised: Critically, the inducing points determine the ﬂexibility of the model and they are often located in regions of the input space where the latent function changes.

Original: A limitation is, however, that for some learning tasks a large number of inducing points may be required to obtain a good prediction performance.
Revised: A limitation is, however, that for some learning tasks a large number of inducing points may be required to obtain a good prediction performance.

Original: To address this limitation, we propose here to amortize the computation of the inducing points locations, as well as the parameters of the variational posterior approximation q .
Revised: To address this limitation, we propose here to amortize the computation of the inducing points locations, as well as the parameters of the variational posterior approximation q .

Original: For this, we use a neural network that receives the observed data as an input and outputs the inducing points locations and the parameters of q .
Revised: For this, we use a neural network that receives the observed data as an input and outputs the inducing points locations and the parameters of q .

Original: We evaluate our method in several experiments, showing that it performs similar or better than other state-of-the-art sparse variational GP approaches.
Revised: We evaluate our method in several experiments, showing that it performs similar or better than other state-of-the-art sparse variational GP approaches.

Original: However, with our method the number of inducing points is reduced drastically due to their dependency on the input data.
Revised: However, with our method the number of inducing points is reduced drastically due to their dependency on the input data.

Original: This makes our method scale to larger datasets and have faster training and prediction times.
Revised: This makes our method scale to larger datasets and have faster training and prediction times.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gaussian Processes (GPs) are non-parametric models that can be used to address regression and classiﬁcation machine learning problems (Rasmussen & Williams, 2006).
Revised: Gaussian Processes (GPs) are non-parametric models that can be used to address regression and classiﬁcation machine learning problems (Rasmussen & Williams, 2006).

Original: GPs become more expressive as the number of training instances N grows and, since they are Bayesian models, they provide a predictive distribution that estimates the uncertainty associated to the predictions made.
Revised: GPs become more expressive as the number of training instances N grows and, since they are Bayesian models, they provide a predictive distribution that estimates the uncertainty associated to the predictions made.

Original: This uncertainty estimation or ability to know what is not known is critical in many practical applications (Gal, 2016).
Revised: This uncertainty estimation or ability to know what is not known is critical in many practical applications (Gal, 2016).

Original: Nevertheless, GPs suffer from poor scalability as their training cost is O ( N 3 )  due to the need of computing the inverse of acovariance matrix of size N × N .
Revised: Nevertheless, GPs suffer from poor scalability as their training cost is O ( N 3 ) per iteration due to the need of computing the inverse of a N × N covariance matrix.

Original: Another limitation is that approximate inference is required with non-Gaussian likelihoods (Rasmussen & Williams, 2006).
Revised: Another limitation is that approximate inference is needed with non-Gaussian likelihoods (Rasmussen & Williams, 2006).

Original: Sparse approximations can improve the cost of GPs (Rasmussen & Williams, 2006).
Revised: Sparse approximations can improve the cost of GPs (Rasmussen & Williams, 2006).

Original: The most popular ones introduce a set of M (cid:28) N inducing points (Snelson & Ghahramani, 2006; Titsias, 2009).
Revised: The most popular ones introduce a set of M (cid:28) N inducing points (Snelson & Ghahramani, 2006; Titsias, 2009).

Original: The inducing points and their associated posterior values completely specify the posterior process at test points.
Revised: The inducing points and their associated posterior values completely specify the posterior process at test points.

Original: In Snelson & Ghahramani (2006), the computational gain is obtained by assuming independence among the process values at the training points given the inducing points and their values.
Revised: In Snelson & Ghahramani (2006), the computational gain is obtained by assuming independence among the process values at the training points given the inducing points and their values.

Original: This can also be seen as using an approximate GP prior (Qui˜nonero-Candela &
Revised: This can also be seen as using an approximate GP prior (Qui˜nonero-Candela &

Original: Rasmussen, 2005).
Revised: Rasmussen, 2005).

Original: By contrast, in Titsias (2009) the computational gain is obtained by combining variational inference (VI) with a posterior approximation q that has a ﬁxed part and a tunable part.
Revised: By contrast, in Titsias (2009) the computational gain is obtained by combining variational inference (VI) with a posterior approximation q that has a ﬁxed part and a tunable part.

Original: In both methods thecomputational cost is reduced to O ( NM 2 )  and the inducing points, considered as model’s hyper-parameters, are learned by maximizing an estimate of the marginal likelihood.
Revised: In both methods the cost is reduced to O ( NM 2 ) per iteration and the inducing points, considered as model’s hyper-parameters, are learned by maximizing an estimate of the marginal likelihood.

Original: Importantly, the VI approach of Titsias (2009) maximizes a lower bound on the log-marginal likelihood as an indirect way of minimizing the KL-divergence between an approximate posterior distribution for the process values at the inducing points and the corresponding exact posterior.
Revised: Importantly, the VI approach of Titsias (2009) maximizes a lower bound on the log-marginal likelihood as an indirect way of minimizing the KL-divergence between an approximate posterior distribution for the process values at the inducing points and the corresponding exact posterior.

Original: The advantage is that the objective is expressed as a sum over the training instances, allowing for mini-batch training and stochastic optimization techniques to be applied on the objective (Hensman et al., 2015b).
Revised: The advantage is that the objective is expressed as a sum over the training instances, allowing for mini-batch training and stochastic optimization techniques to be applied on the objective (Hensman et al., 2015b).

Original: This reduces thetraining cost to O ( M 3 ) , making GPs scalable tovery large datasets.
Revised: This reduces the cost to O ( M 3 ) per iteration, making GPs scalable to large datasets.

Original: In sparse approximations one often observes in practice that after the optimization process the inducing points are located in regions of the input space in which the latent function changes (Snelson & Ghahramani, 2006; Titsias, 2009;
Revised: In sparse approximations one often observes in practice that after the optimization process the inducing points are located in regions of the input space in which the latent function changes (Snelson & Ghahramani, 2006; Titsias, 2009;

Original: Hensman et al., 2015a; Bauer et al., 2016).
Revised: Hensman et al., 2015a; Bauer et al., 2016).

Original: Therefore, the expressive power of the modelcritically depends on the number of inducing points M and their correct location on the input space.
Revised: Therefore, the expressive power of the model depends on the number of inducing points M and their correct location on the input space.

Original: Some problems may require a large number of inducing points, in the order of several hundreds, to get good prediction results (Hensman et al., 2015b; Shi et al., 2020; Tran et al., 2020).
Revised: Some problems may require a large number of inducing points, in the order of thousands, to get good prediction results (Hensman et al., 2015b; Shi et al., 2020; Tran et al., 2020).

Original: This makes training inducing point based sparse GPs difﬁcult in those problems.
Revised: This makes training inducing point based sparse GPs difﬁcult in those problems.

Original: There have been some attempts to improve the computational cost of sparse approximations, including using different sets of inducing points for the computation of the posterior mean and variance (Cheng & Boots, 2017).
Revised: There have been some attempts to improve the training cost of sparse approximations, including using different sets of inducing points for the computation of the posterior mean and variance (Cheng & Boots, 2017).

Original: Other approaches use an orthogonal decomposition of the GP that allows to introduce an extra set of inducing points with less cost (Shi et al., 2020).
Revised: Other approaches use an orthogonal decomposition of the GP that allows to introduce an extra set of inducing points with less cost (Shi et al., 2020).

Original: Finally, other methods consider a large set of inducing points, but restrict the computations for a particular data point to the nearest neighbors to that point from the set of inducing points (Tran et al., 2020).
Revised: Finally, other methods consider a large set of inducing points, but restrict the computations for a particular data point to the nearest neighbors to that point from the set of inducing points (Tran et al., 2020).

Original: In this work we are inspired by Tran et al.
Revised: In this work we are inspired by Tran et al.

Original: and propose a novel method to improve the computational cost of sparse GPs.
Revised: and propose a novel method to improve the training cost of sparse GPs.

Original: Our method also tries to produce a set of inducing points (and associated variational approximation q ) that are speciﬁc of each input data point.
Revised: Our method also tries to produce a set of inducing points (and associated variational approximation q ) that are speciﬁc of each input data point.

Original: For that, we note that some works in the literature have observed that one can learn the mappings from inputs to proposal distributions instead of directly optimizing their parameters (Kingma & Welling, 2014; Shu et al., 2018).
Revised: For that, we note that some works in the literature have observed that one can learn the mappings from inputs to proposal distributions instead of directly optimizing their parameters (Kingma & Welling, 2014; Shu et al., 2018).

Original: This approach, known as amortized variational inference, is a key contribution of variational auto-encoders (VAE) (Kingma & Welling, 2014), and has also been explored in the context of GP to solve other types of problems such as multi-class classiﬁcation with input noise (Villacampa-Calvo et al., 2021).
Revised: This approach, known as amortized variational inference, is a key contribution of variational auto-encoders (VAE) (Kingma & Welling, 2014), and has also been explored in the context of GP to solve other types of problems such as multi-class classiﬁcation with input noise (Villacampa-Calvo et al., 2021).

Original: Amortized inference has also been empirically shown to lead to useful regularization properties that improve the generalization performance (Shu et al., 2018).
Revised: Amortized inference has also been empirically shown to lead to useful regularization properties that improve the generalization performance (Shu et al., 2018).

Original: Speciﬁcally, here wepropose to combine sparse GPs with a neural network architecture to compute the  inducing pointslocations associated to each input point.
Revised: Speciﬁcally, here we combine sparse GPs with a neural network architecture that computes, for each potential data point, the associated inducing points to be used for prediction.

Original: Moreover, we also employ a neural network to carry out amortized VI to compute the parameters of theapproximate variational distribution q modeling the posterior distribution associated to the  values of the outputted inducing points.
Revised: We also employ a neural network to carry out amortized VI to compute the parameters of the variational distribution q approximating the posterior of the process values for the outputted inducing points.

Original: Critically, this approach allows  a  reduction in the  number of inducing pointsdrastically without losing expressive power, as now we have different sets of inducing points associated to each input location.
Revised: While the number of parameters that need optimization may increase with the use of a neural network, this approach allows for a big reduction in the total number of inducing points without losing expressive power. In particular, it enables different sets of inducing points associated to each input location.

Original: The inducing points are simply given by a mapping from the inputs provided by a neural network.
Revised: The inducing points are simply given by a mapping from the inputs provided by a neural network.

Original: We show on several experiments that the proposed method is able to perform similar or better than standard sparse GPs and competitive methods for improving the cost of sparse GPs (Tran et al., 2020;
Revised: We show on several experiments that the proposed method is able to perform similar or better than standard sparse GPs and competitive methods for improving the cost of sparse GPs (Tran et al., 2020;

Original: Shi et al., 2020).
Revised: Shi et al., 2020).

Original: However, the training and prediction times of our method are much better.
Revised: However, the training and prediction times of our method are much better.

Original:  G AUSSIAN PROCESSES
Revised:  G AUSSIAN PROCESSES

Original: A Gaussian Process (GP) is a stochastic process for which any ﬁnite set of variables has a Gaussian distribution (Rasmussen & Williams, 2006).
Revised: A Gaussian Process (GP) is a stochastic process for which any ﬁnite set of variables has a Gaussian distribution (Rasmussen & Williams, 2006).

Original: In a learning task, we use a GP as a prior over a latent function.
Revised: In a learning task, we use a GP as a prior over a latent function.

Original: Then, Bayes’ rule is used to get a posterior for that function given the observed data.
Revised: Then, Bayes’ rule is used to get a posterior for that function given the observed data.

Original: Consider a dataset D
Revised: Consider a dataset D

Original: = { ( x i , y i ) } Ni =1 , where each scalar y i is assumed to be obtained as y i =
Revised: = { ( x i , y i ) } Ni =1 , where each scalar y i is assumed to be obtained as y i =

Original: f ( x i ) + (cid:15) i , with f ( · ) a latent function and (cid:15)
Revised: f ( x i )+ (cid:15) i , with f ( · ) a latent function and (cid:15)

Original: i Gaussian noise with variance σ 2 . Namely, (cid:15) i ∼ N (0 , σ 2 ) .
Revised: i Gaussian noise with variance σ 2 , i.e. , (cid:15) i ∼ N (0 , σ 2 ) .

Original: We specify a prior distribution for f in the form of a GP, which is described by a mean function m ( x ) (often set to zero) and covariance function k ( x , x (cid:48) )
Revised: We specify a prior distribution for f in the form of a GP, which is described by a mean function m ( x ) (often set to zero) and covariance function k ( x , x (cid:48) )

Original: such thatk ( x , x (cid:48) ) = E [ f ( x ) f
Revised: such that

Original: ( x (cid:48) )] .
Revised: [Equation]

Original: Covariance functions typically have some parameters θ .
Revised: Covariance functions typically have some parameters θ .

Original: Given D , the predictive distribution for f at a new test point x (cid:63) is Gaussian with mean and variance given by
Revised: Given D , the predictive distribution for f at a new test point x (cid:63) is Gaussian with mean and variance given by

Original: [Equation]
Revised: [Equation]

Original: where µ ( x (cid:63) ) and σ 2 ( x (cid:63) ) are the prediction mean and variance, respectively.
Revised: where µ ( x (cid:63) ) and σ 2 ( x (cid:63) ) are the prediction mean and variance, respectively.

Original: k ( x (cid:63) ) is a vector with the covariances between f ( x (cid:63) ) and each f ( x i ) .
Revised: k ( x (cid:63) ) is a vector with the covariances between f ( x (cid:63) ) and each f ( x i ) .

Original: Similarly, K has the covariances between f ( x i ) and f ( x j ) for i, j = 1 , . . .
Revised: Similarly, K has the covariances between f ( x i ) and f ( x j ) for i, j = 1 , . . .

Original: , N .
Revised: , N .

Original: Finally, I stands for the identity matrix.
Revised: Finally, I stands for the identity matrix.

Original: A popular covariance function k ( · , · ) is the squared exponential, which assumes that f is smooth (Rasmussen & Williams, 2006).
Revised: Popular covariances functions k ( · , · ) are the squared exponential and the Mat´ern (Rasmussen & Williams, 2006).

Original: Its parameters, θ , and σ 2 cansimply be found by maximizing p ( y ) (Rasmussen & Williams, 2006).
Revised: Their parameters, θ , and σ 2 can be found by maximizing p ( y ) (Rasmussen & Williams, 2006).

Original: The computational complexity of this approach is O ( N 3 ) since it needs the inversion of K , a N × N matrix.
Revised: The computational complexity of this approach is O ( N 3 ) since it needs the inversion of K , a N × N matrix.

Original: This makes GPs unsuitable for large data sets.
Revised: This makes GPs unsuitable for large data sets.

Original: 2. S PARSE VARIATIONAL G AUSSIAN PROCESSES
Revised: 2. S PARSE VARIATIONAL G AUSSIAN PROCESSES

Original: Sparse approximations improve the cost of GPs.
Revised: Sparse approximations improve the cost of GPs.

Original: The most popular methods introduce, in the same input space as the original data, a new set of M (cid:28) N points , called the inducing points, denoted by Z
Revised: The most popular methods introduce, in the same input space as the original data, a new set of M (cid:28) N points , called the inducing points, denoted by Z

Original: = ( z 1 , . . .
Revised: = ( z 1 , . . .

Original: , z M ) T (Snelson & Ghahramani, 2006; Titsias, 2009).
Revised: , z M ) T (Snelson & Ghahramani, 2006; Titsias, 2009).

Original: Let the corresponding latent function values be u
Revised: Let the corresponding latent function values be u

Original: =
Revised: =

Original: ( f ( z 1 ) , . . .
Revised: ( f ( z 1 ) , . . .

Original: , f ( z M )) T .
Revised: , f ( z M )) T .

Original: The inducing points are not restricted to be part of the observed data and their location can be learned during training.
Revised: The inducing points are not restricted to be part of the observed data and their location can be learned during training.

Original: A GP prior is placed on u .
Revised: A GP prior is placed on u .

Original: Namely, p ( u ) ∼ N ( 0 , K Z ) , where K Z is a matrix with the covariances associated to each pair of points from
Revised: Namely, p ( u ) ∼ N ( 0 , K Z ) , where K Z is a matrix with the covariances associated to each pair of points from

Original: Z .
Revised: Z .

Original: The idea is that the posterior for f can be approximated in terms of the posterior for u .
Revised: The idea is that the posterior for f can be approximated in terms of the posterior for u .

Original: In this work we focus on a widely used variational inference (VI) approach to approximate the posterior for f (Titsias, 2009).
Revised: In this work we focus on a widely used variational inference (VI) approach to approximate the posterior for f (Titsias, 2009).

Original: Let f = ( f ( x 1 ) , . . .
Revised: Let f = ( f ( x 1 ) , . . .

Original: , f ( x N )) T .
Revised: , f ( x N )) T .

Original: In VI, the goal is to ﬁnd an approximate posterior for f and u , q ( f , u ) , that resembles as much as possible the true posterior p ( f , u | y ) .
Revised: In VI, the goal is to ﬁnd an approximate posterior for f and u , q ( f , u ) , that resembles as much as possible the true posterior p ( f , u | y ) .

Original: Critically, q is constrained to be q ( f , u ) =
Revised: Critically, q is constrained to be q ( f , u ) =

Original: p ( f | u ) q ( u ) , with p ( f | u ) ﬁxed and q ( u ) a tunable multivariate Gaussian.
Revised: p ( f | u ) q ( u ) , with p ( f | u ) ﬁxed and q ( u ) a tunable multivariate Gaussian.

Original: To ﬁnd q ( u ) a lower bound of the marginal likelihood is maximized.
Revised: To ﬁnd q ( u ) a lower bound of the marginal likelihood is maximized.

Original: This bound  is obtained through Jensen’s inequality, leading to the following expression, after some simpliﬁcations:
Revised: The evidence lower bound (or ELBO) is obtained via Jensen’s inequality, leading to (after some simpliﬁcations):

Original: [Equation]
Revised: [Equation]

Original: where p ( y i | f i ) is the model’s likelihood for the i -th point and KL [ ·|· ] is the Kullback-Leibler divergence between probability distributions.
Revised: where p ( y i | f i ) is the model’s likelihood for the i -th point and KL [ ·|· ] is the Kullback-Leibler divergence between probability distributions.

Original: In Titsias (2009), theydo optimize q ( u ) in closed-form.
Revised: In Titsias (2009), they optimize q ( u ) in closed-form.

Original: The resulting expression is then maximized to estimate Z , θ and σ 2 .
Revised: The resulting expression is then maximized to estimate Z , θ and σ 2 .

Original: This leads to a complexity of
Revised: This leads to a complexity of

Original: O ( NM 2 ) .
Revised: O ( NM 2 ) .

Original: However, if the variational posterior q ( u ) is optimized alongside with Z , θ and σ 2 , as proposed in Hensman et al.
Revised: However, if the variational posterior q ( u ) is optimized alongside with Z , θ and σ 2 , as proposed in Hensman et al.

Original: , the ELBO can be expressed as a sum over training instances, which allows for mini-batch training and stochastic optimization techniques.
Revised: , the ELBO can be expressed as a sum over training instances, which allows for mini-batch training and stochastic optimization techniques.

Original: Using stochastic variational inference (SVI) reduces the training cost to O ( M 3 )  (Hensman et al., 2013).
Revised: Using stochastic variational inference (SVI) reduces the training cost to O ( M 3 ) per iteration (Hensman et al., 2013).

Original: Importantly, the ﬁrst term in (2) is an expectation that has closed-form solution in the case of Gaussian likelihoods.
Revised: Importantly, the ﬁrst term in (2) is an expectation that has closed-form solution in the case of Gaussian likelihoods.

Original: It needs to be approximated for other cases, e.g. , binary classiﬁcation, either by quadrature or MCMC methods (Hensman et al., 2015b).
Revised: It needs to be approximated for other cases, e.g. , binary classiﬁcation, either by quadrature or MCMC methods (Hensman et al., 2015b).

Original: The second term is the KL-divergence between the variational posterior and the prior, which can be computed analytically since they are both Gaussians.
Revised: The second term is the KL-divergence between the variational posterior and the prior, which can be computed analytically since they are both Gaussians.

Original: After optimizing (2), one often observes that the  inducing points are located inthose regions of the input space in which the latent function changes (Titsias, 2009; 
Revised: The expressive power of the sparse GP critically depends on the number of inducing points M and on their correct placement in the input space via optimizing (2) (Titsias, 2009; Hensman et al., 2015a;

Original: Hensman et al., 2015a; Bauer et al., 2016).
Revised: Bauer et al., 2016).

Original: In consequence, the expressive power of the sparse GP critically depends on the number of inducing points M and on their correct placement.

Original: In some learning problems, however, a large number of inducing points, in the order of several hundred, is required to get goodprediction results (Hensman et al., 2015b; Shi et al., 2020; Tran et al., 2020).
Revised: In some problems several thousands of inducing points may be required to get good results (Hensman et al., 2015b; Shi et al., 2020; Tran et al., 2020).

Original: This makes difﬁcult and expensive using sparse GPs in those problems.
Revised: This makes difﬁcult and expensive using sparse GPs in those problems.

Original: In the next section we describe how to reduce the cost of this method.
Revised: In the next section we describe how to alleviate this.

Original:  I NPUT DEPENDENT SPARSE GP S
Revised:  I NPUT DEPENDENT SPARSE GP S

Original: We develop a new formulation of sparse GPs which for every given input computes the corresponding inducing points to be used for prediction, and also the associated parameters of the approximate distribution q .
Revised: We develop a new formulation of sparse GPs which for every given input computes the corresponding inducing points to be used for prediction, and also the associated parameters of the approximate distribution q .

Original: To achieve this, we consider a meta-point ˜ x that is used to determine the inducing points Z and the corresponding u .
Revised: To achieve this, we consider a meta-point ˜ x that is used to determine the inducing points Z and the corresponding u .

Original: Namely, now u depends on ˜ x , i.e. , u ∼ p ( u | ˜ x ) .
Revised: Namely, now u depends on ˜ x , i.e. , u ∼ p ( u | ˜ x ) .

Original: In particular, we set p ( u | ˜ x ) =
Revised: In particular, we set p ( u | ˜ x ) =

Original: N ( 0 , K Z (˜ x ) ) where the inducing points Z depend non-linearly, e.g. , via a deep neural network, on ˜ x .
Revised: N ( 0 , K Z (˜ x ) ) where the inducing points Z depend non-linearly, e.g. , via a deep neural network, on ˜ x .

Original: The joint distribution of u and ˜ x is then given by p ( u , ˜ x )
Revised: The joint distribution of u and ˜ x is then given by p ( u , ˜ x )

Original: =
Revised: =

Original: p ( u | ˜ x ) p (˜ x ) for some prior distribution p (˜ x ) .
Revised: p ( u | ˜ x ) p (˜ x ) for some prior distribution p (˜ x ) .

Original: Following (Tran et al., 2020), we can consider an implicit distribution p (˜ x ) .
Revised: Following (Tran et al., 2020), we can consider an implicit distribution p (˜ x ) .

Original: That is, its analytical form is unknown, but we can draw samples from it.
Revised: That is, its analytical form is unknown, but we can draw samples from it.

Original: Later on, we specify p (˜ x ) .
Revised: Later on, we specify p (˜ x ) .

Original: Note that the marginalized prior p ( u ) is no longer Gaussian.
Revised: Note that the marginalized prior p ( u ) is no longer Gaussian.

Original: However, we can show that this formulation does not impact on the prior over f .
Revised: However, we can show that this formulation does not impact on the prior over f .

Original: For an arbitrary selected meta-point ˜ x we have that
Revised: For an arbitrary selected meta-point ˜ x we have that

Original: [Equation]
Revised: [Equation]

Original: where K X , Z (˜ x ) are the cross-covariances between f and u .
Revised: where K X , Z (˜ x ) are the cross-covariances between f and u .

Original: Therefore, if u is marginalized out in (3), the prior for f is the standard GP prior and does not depend on ˜ x .
Revised: Therefore, if u is marginalized out in (3), the prior for f is the standard GP prior and does not depend on ˜ x .

Original: Hence, p ( f | ˜ x ) = p ( f ) .
Revised: Hence, p ( f | ˜ x ) = p ( f ) .

Original: Thus, p ( f , u ) = (cid:82) p ( f , u | ˜ x ) p (˜ x )
Revised: Thus, p ( f , u ) = (cid:82) p ( f , u | ˜ x ) p (˜ x )

Original: d ˜ x is a mixture of Gaussian densities, where the marginal over f is the same for every component of the mixture.
Revised: d ˜ x is a mixture of Gaussian densities, where the marginal over f is the same for every component of the mixture.

Original: In the standard sparse GP, the inducing points also have an impact on the variational approximation q via the ﬁxed conditional distribution p ( f | u ) (Titsias, 2009).
Revised: In the standard sparse GP, the inducing points also have an impact on the variational approximation q via the ﬁxed conditional distribution p ( f | u ) (Titsias, 2009).

Original: Therefore, we also incorporate the input dependence on ˜ x in q .
Revised: Therefore, we also incorporate the input dependence on ˜ x in q .

Original: This is done in the next section.
Revised: This is done in the next section.

Original: 3. L OWER BOUND ON THE LOG - MARGINAL LIKELIHOOD
Revised: 3. L OWER BOUND ON THE LOG - MARGINAL LIKELIHOOD

Original: We follow Tran et al.
Revised: We follow Tran et al.

Original: (2020) to derive a lower bound on the log-marginal likelihood of the extended model described above.
Revised: (2020) to derive a lower bound on the log-marginal likelihood of the extended model described above.

Original: Consider a posterior approximation of the form q ( f , u , ˜ x ) = p ( f | u ) q ( u | ˜ x ) p (˜ x ) , where only q ( u | ˜ x ) can be adjusted and the other factors are ﬁxed.
Revised: Consider a posterior approximation of the form q ( f , u , ˜ x ) = p ( f | u ) q ( u | ˜ x ) p (˜ x ) , where only q ( u | ˜ x ) can be adjusted and the other factors are ﬁxed.

Original: Using this posterior’s factorization and Jensen’s inequality we obtain the lower bound after some simpliﬁcations:
Revised: Using this posterior’s factorization and Jensen’s inequality we obtain the lower bound after some simpliﬁcations:

Original: [Equation]
Revised: [Equation]

Original: Now, assuming that p (˜ x ) is an implicit distribution, we can draw samples from it and approximate the expectation w.r.t p (˜ x ) .
Revised: Now, assuming that p (˜ x ) is an implicit distribution, we can draw samples from it and approximate the expectation w.r.t p (˜ x ) .

Original: Thus, for a meta-point sample ˜ x s from p (˜ x ) , (4) is approximated as
Revised: Thus, for a meta-point sample ˜ x s from p (˜ x ) , (4) is approximated as

Original: [Equation]
Revised: [Equation]

Original: We can evaluate (5) and its gradients to maximize the original objective in (4) using stochastic optimization techniques.
Revised: We can evaluate (5) and its gradients to maximize the original objective in (4) using stochastic optimization techniques.

Original: This is valid for any implicit distribution p (˜ x ) .
Revised: This is valid for any implicit distribution p (˜ x ) .

Original: Consider now that we use mini-batch-based training for optimization, and we set ˜ x s = x i .
Revised: Consider now that we use mini-batch-based training for optimization, and we set ˜ x s = x i .

Original: In this case, the value of ˜ x remains random, as it depends on the points ( x i , y i ) that are selected in the random mini-batch.
Revised: In this case, the value of ˜ x remains random, as it depends on the points ( x i , y i ) that are selected in the random mini-batch.

Original: This results in a method that computes different inducing points for each input location.
Revised: This results in a method that computes different inducing points for each input location.

Original: In practice, we use the same sample to approximate the expectation w.r.t.
Revised: In practice, we use the same sample to approximate the expectation w.r.t.

Original: p ( ˜ x ) and the sum across the data in (5).
Revised: p (˜ x ) and the sum across the data in (5).

Original: This could introduce a bias in the objective.
Revised: This could introduce a bias in the objective.

Original: However, such a reusing of the samples is done in Tran et al.
Revised: However, such a reusing of the samples is done in Tran et al.

Original: (2020) with good empirical results.
Revised: (2020) with good empirical results.

Original: Moreover, our experiments in Section 5 also validate this approximation.
Revised: Moreover, our experiments in Section 5 also validate this approximation.

Original: 3. A MORTIZED VARIATIONAL INFERENCE AND DEEP NEURAL NETWORKS
Revised: 3. A MORTIZED VARIATIONAL INFERENCE AND DEEP NEURAL NETWORKS

Original: Maximizing the lower bound ﬁnds the optimal approximate distribution q .
Revised: Maximizing the lower bound ﬁnds the optimal approximate distribution q .

Original: A problem, however, is that we have a potential large number of parameters to ﬁx, corresponding to each q ( u | x i ) .
Revised: A problem, however, is that we have a potential large number of parameters to ﬁx, corresponding to each q ( u | x i ) .

Original: In particular, if we set q ( u | x i ) to be Gaussian, we will have to infer different
Revised: In particular, if we set q ( u | x i ) to be Gaussian, we will have to infer different

Original: X nd
Revised: X nd

Original: X nd h (L)P h (1)P −h (1) h (1) h (2)P h (2)P −h (2) h (2) h (L)P h (L)P −h (L) h (L) means and covariance matrices for each different x i .
Revised: X nd h ( ) P h (1)P −h (1) h (1) h (2)P h (2)P −h (2) h (2) h (L)P h (L)P −h (L) h (L) means and covariance matrices for each different x i .

Original: This is expected to be memory inefﬁcient and to make optimization more difﬁcult.
Revised: This is expected to be memory inefﬁcient and to make optimization more difﬁcult.

Original: To reduce the number of parameters of our method we use amortized variational inference and specify a function that can generate these parameters for each x i (Shu et al., 2018).
Revised: To reduce the number of parameters of our method we use amortized variational inference and specify a function that can generate these parameters for each x i (Shu et al., 2018).

Original: More precisely, we set the mean and covariance matrix of q ( u | x i ) to be m ( x i ) and S ( x i ) , for some non-linear functions.
Revised: More precisely, we set the mean and covariance matrix of q ( u | x i ) to be m ( x i ) and S ( x i ) , for some non-linear functions.

Original: Deep neural networks (DNN) are ﬂexible models that can describe complicated functions.
Revised: Deep neural networks (DNN) are ﬂexible models that can describe complicated functions.

Original: In these models, the inputs go through several layers of non-linear transformations.
Revised: In these models, the inputs go through several layers of non-linear transformations.

Original: We use these models to compute the non-linearities that generate from x i the inducing points, Z ( x i ) , and the means and covariances of q ( u | x i ) , i.e. , m ( x i ) and S ( x i ) .
Revised: We use these models to compute the non-linearities that generate from x i the inducing points, Z ( x i ) , and the means and covariances of q ( u | x i ) , i.e. , m ( x i ) and S ( x i ) .

Original: The architecture employed is displayed in Figure 1.
Revised: The architecture employed is displayed in Figure 1.

Original: At the output of the DNN we obtain Z , a mean vector m and the Cholesky factor of the covariance matrix S = LL T .
Revised: At the output of the DNN we obtain Z , a mean vector m and the Cholesky factor of the covariance matrix S = LL T .

Original: The maximization of the lower bound in (4) when using DNNs for the non-linearities is shown in Algorithm 1.
Revised: The maximization of the lower bound in (4) when using DNNs for the non-linearities is shown in Algorithm 1.

Original: The required expectations are computed in closed-form, in regression.
Revised: The required expectations are computed in closed-form, in regression.

Original: In binary classiﬁcation, we use 1-dimensional quadrature, as in Hensman et al.
Revised: In binary classiﬁcation, we use 1-dimensional quadrature, as in Hensman et al.

Original: (2015a).
Revised: (2015a).

Original:  P REDICTIONS AND COMPUTATIONAL COST
Revised: 3. P REDICTIONS AND TRAINING COST

Original: At test the instances are not randomly chosen.
Revised: At test the instances are not randomly chosen.

Original: In that case, we simply set p (˜ x ) to be a deterministic distribution placed on the candidate point x (cid:63) .
Revised: In that case, we simply set p (˜ x ) to be a deterministic distribution placed on the candidate point x (cid:63) .

Original: The DNN is used to obtain the associated information.
Revised: The DNN is used to obtain the associated information.

Original: Namely, Z , and the parameters of q ( u | x (cid:63) ) , m and S .
Revised: Namely, Z , and the parameters of q ( u | x (cid:63) ) , m and S .

Original: The predictive distribution for f ( x (cid:63) ) is:
Revised: The predictive distribution for f ( x (cid:63) ) is:

Original: [Equation]
Revised: [Equation]

Original: Given this distribution for f ( x (cid:63) ) , the probability distribution for y (cid:63) can be computed in closed form in the case of regression problems and with 1-dimensional quadrature in the case of binary classiﬁcation.
Revised: Algorithm

Original: Input: D , M , neural network NNet with L hidden layers and P hidden units
Revised: Input: D , M , neural network NNet with L hidden layers and P hidden units

Original: Output: Optimal parameters of the model initialize neural network’s weights and kernel’s parameters θ while stopping criteria is False do
Revised: Output: Optimal parameters of the model initialize neural network’s weights and kernel’s parameters θ while stopping criteria is False do

Original: Log lk = 0 , KL div =gather mini-batch Mb of size n from D for ( x i , y i ) in Mb do ( Z x i , m x i , L x i )
Revised: Log lk = 0 , KL div =gather mini-batch Mb of size n from D for ( x i , y i ) in Mb do ( Z x i , m x i , L x i )

Original: =
Revised: =

Original: NNet ( x i ); Log lk += E
Revised: NNet ( x i ); Log lk += E

Original: q
Revised: q

Original: ( f i , u ) [log p ( y i | f i )] ; ; KL div += KL [ q ( u | x i ) | p ( u | x i )]
Revised: ( f i , u ) [log p ( y i | f i )] ;KL div += KL [ q ( u | x i ) | p ( u | x i )]

Original: ELBO ← Nn × Log lk − 1 n × KL div
Revised: ELBO ← Nn × Log lk − 1 n × KL div

Original: Update parameters of the model using the gradient of ELBO
Revised: Update parameters of the model using the gradient of ELBO

Revised: Given this distribution for f ( x (cid:63) ) , the probability distribution for y (cid:63) can be computed in closed form in the case of regression problems and with 1-dimensional quadrature in the case of binary classiﬁcation.

Revised: Note that (6) is only suitable for predictions at individual test points, as in Tran et al.

Revised: This can be a limitation in applications needing covariances.

Original:  The cost of our method is smaller than the cost of a standard sparse GP if a smaller number of inducing points M is used.
Revised: As a solution, one could consider for prediction the union of all input dependent inducing points for test points. This would be inconsistent with the proposed training method. Nonetheless, such approach can also be modiﬁed to consider the union of input dependent inducing points for the training points within a mini-batch, as in (Tran et al., 2020). The cost of our method is smaller than the cost of a standard sparse GP if a smaller number of inducing points M is used.

Original: The cost of a DNN with L layers, P hidden units, d i dimension of the input data, and output dimension d o is O ( nd i P + nP 2 L + nPd o + n
Revised: The cost of a DNN with L layers, P hidden units, d i dimension of the input data, and output dimension d o is O ( nd i P + nP 2 L + nPd o + n

Original: ( L + 1)) .
Revised: ( L + 1)) .

Original: The cost of the sparse GPs is O ( nM 3 ) , with n the mini-batch size.
Revised: The cost of the sparse GPs is O ( nM 3 ) , with n the mini-batch size.

Original: Therefore, the cost of our method  is
Revised: Therefore, the cost of our method per iteration is

Original: O ( nd i P + nP 2 L + nPd o + n
Revised: O ( nd i P + nP 2 L + nPd o + n

Original: ( L + 1) + nM 3 ) .
Revised: ( L + 1) + nM 3 ) .

Original: Since in our method the inducing points are input dependent, we expect to obtain good prediction results even for M values that are fairly small.
Revised: Since in our method the inducing points are input dependent, we expect to obtain good prediction results even for M values that are fairly small.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Early works on sparse GPs simply chose a subset of the training data for inference based on an information criterion (Csat ´ o & Opper, 2002; Lawrence et al., 2003; Seeger et al., 2003; Henao &
Revised: Early works on sparse GPs simply chose a subset of the training data for inference based on an information criterion (Csat ´ o & Opper, 2002; Lawrence et al., 2003; Seeger et al., 2003; Henao &

Original: Winther, 2012).
Revised: Winther, 2012).

Original: This approach is limited in practice and more advanced methods in which the inducing points need not be equal to the training points are believed to be superior.
Revised: This approach is limited in practice and more advanced methods in which the inducing points need not be equal to the training points are believed to be superior.

Original: In the literature there are several works analyzing and studying sparse GP approximations based on inducing points.
Revised: In the literature there are several works analyzing and studying sparse GP approximations based on inducing points.

Original: Some of these works include Qui ˜ nonero-Candela & Rasmussen (2005); Snelson & Ghahramani (2006); Naish-Guzman & Holden (2007); Titsias (2009); Bauer et al.
Revised: Some of these works include Qui˜nonero-Candela & Rasmussen (2005); Snelson & Ghahramani (2006); Naish-Guzman & Holden (2007); Titsias (2009); Bauer et al.

Original: Hern ´ andez-Lobato &
Revised: Hern ´ andez-Lobato &

Original: Hern ´ andez-Lobato (2016).
Revised: Hern ´ andez-Lobato (2016).

Original: We focus here on a variational approach (Titsias, 2009) which allows for stochastic optimization and mini-batch training (Hensman et al., 2013; 2015a).
Revised: We focus here on a variational approach (Titsias, 2009) which allows for stochastic optimization and mini-batch training (Hensman et al., 2013; 2015a).

Original: This enables learning in very large datasets with a cost of O ( M 3 ) , with M the number of inducing points.
Revised: This enables learning in very large datasets with a cost of O ( M 3 ) per iteration, with M the number of inducing points.

Original: In general, however, a large number of inducing variables is desirable to obtain a good approximation to the posterior distribution.

Original: For some problems, even several hundreds of inducing points may be needed to get good prediction results (Hensman et al., 2015b; Shi et al., 2020; Tran et al., 2020).
Revised: In some problems, however, several thousands of inducing points may be needed to get good prediction results (Hensman et al., 2015b; Shi et al., 2020; Tran et al., 2020).

Original: There is hence a need to improve thecomputational cost of sparse GP approximations, without losing expressive power.
Revised: There is hence a need to improve the cost of sparse GPs, without losing expressive power.

Original: One work addressing this task is that of Cheng & Boots (2017).
Revised: One work addressing this task is that of Cheng & Boots (2017).

Original: In that work it is proposed to decouple the process of inferring the posterior mean and variance, allowing to consider a different number of inducing points for each one.
Revised: In that work it is proposed to decouple the process of inferring the posterior mean and variance, allowing to consider a different number of inducing points for each one.

Original: Importantly, the computation of the mean achieves a linear complexity, which allows to have more expressive posterior means at a lower cost.
Revised: Importantly, the computation of the mean has a linear complexity, which allows to have more expressive posterior means at a lower cost.

Original: A disadvantage is that such an approach suffers from optimization difﬁculties.
Revised: A disadvantage is that such an approach suffers from optimization difﬁculties.

Original: An alternative decoupled parameterizationthat adopts an orthogonal basis in the meanis proposed in Salimbeni et al. 
Revised: An alternative decoupled parameterization adopts an orthogonal basis in the mean Salimbeni et al. (2018a).

Original: Such a method can be considered as a speciﬁc case of Shi et al.
Revised: Such a method can be considered as a speciﬁc case of Shi et al.

Original: There, it is introduced a new interpretation of sparse variational approximations for GP using inducing points.
Revised: There, the authors introduce a new interpretation of sparse variational approximations for GP using inducing points.

Original: For this, the GP is decomposed as a sum of two independent processes.
Revised: For this, the GP is decomposed as a sum of two independent processes.

Original: This leads to tighter lower bounds on the marginal likelihood and new inference algorithms that allow to consider two different sets of inducing points.
Revised: This leads to tighter lower bounds on the marginal likelihood and new inference algorithms considering two different sets of inducing points.

Original: This enables including more inducing points at a linear cost, instead of cubic.
Revised: This enables using more inducing points at a linear cost.

Original: Our work is closer to Tran et al.
Revised: Our work is closer to Tran et al.

Original: There, it is also describeda mechanism to consider input dependent inducing points in the context of sparse GP.
Revised: There, a mechanism is also described to consider input dependent inducing points in the context of sparse GP.

Original: However, the difference is signiﬁcant.
Revised: However, the difference is signiﬁcant.

Original: In particular, in Tran et al.
Revised: In particular, in Tran et al.

Original: (2020) a very large set of inducing points M is considered initially.
Revised: (2020) a very large set of inducing points M is considered initially.

Original: Then, for each input point, a subset of these inducing points is considered.
Revised: Then, for each input point, a subset of these inducing points is considered.

Original: This subset is obtained by ﬁnding the K (cid:28) M nearest inducing points to the current data instance x i .
Revised: This subset is obtained by ﬁnding the K (cid:28) M nearest inducing points to the current data instance x i .

Original: This approach signiﬁcantly reduces the cost of the standard sparse GP described in Titsias (2009).
Revised: This approach signiﬁcantly reduces the cost of the standard sparse GP described in Titsias (2009).

Original: However, it suffers from the difﬁculty of having to ﬁnd the K nearest neighbors for each point in a mini-batch, which is very expensive.
Revised: However, it suffers from the difﬁculty of having to ﬁnd the K nearest neighbors for each point in a mini-batch, which is very expensive.

Original: Therefore, the ﬁnal cost is higher than what would be thought initially.
Revised: Therefore, the ﬁnal cost is higher than what would be thought initially.

Original: Our method is expected to be better because of the extra ﬂexibility by the non-linear relation between x i and Z given by the DNN.
Revised: Our method is expected to be better because of the extra ﬂexibility by the non-linear relation between x i and Z given by the DNN.

Original: Furthermore, the DNN is expected to make a better use of GPU acceleration.
Revised: Furthermore, the DNN is expected to make a better use of GPU acceleration.

Original: Another method to improve the computational cost of GP is described in Wilson & Nickisch (2015); Evans & Nair (2018); Gardner et al.
Revised: Another method to improve the training cost of GP is described in Wilson & Nickisch (2015); Evans & Nair (2018); Gardner et al.

Original: It consists in placing the inducing points on a grid.
Revised: It consists in placing the inducing points on a grid.

Original: This allows to perform fast computation exploiting the inducing points structure.
Revised: This allows to perform fast computation exploiting the inducing points structure.

Original: One can easily consider values for M that are even larger than N .
Revised: One can easily consider values for M that are even larger than N .

Original: However, to get such beneﬁts the inducing points need to be ﬁxed due to the structure constraints.
Revised: However, to get such beneﬁts the inducing points need to be ﬁxed due to the structure constraints.

Original: This may be detrimental in high-dimensional problems.
Revised: This may be detrimental in high-dimensional problems.

Original: Instead of using inducing points, there are some works that scale GPs by approximating the posterior
Revised: Instead of using inducing points, there are some works that scale GPs by approximating the posterior

Original: GP process using an inference network (Shi et al., 2019; Sun et al., 2019).
Revised: GP process using an inference network (Shi et al., 2019; Sun et al., 2019).

Original: An inference network receives some random noise and outputs function values for each input.
Revised: An inference network receives some random noise and outputs function values for each input.

Original: Particular examples include among others Bayesian DNNs.
Revised: Particular examples include among others Bayesian DNNs.

Original: Inference networks are expected to lead to ﬂexible stochastic processes.
Revised: Inference networks are expected to lead to ﬂexible stochastic processes.

Original: However, it is difﬁcult to enforce that the approximate posterior process looks similar to the prior
Revised: However, it is difﬁcult to enforce that the approximate posterior process looks similar to the prior

Original: GP in regions where there is no data.
Revised: GP in regions where there is no data.

Original: For this, approximate inference is carried out on a ﬁnite subset of points chosen at random from the input space.
Revised: For this, approximate inference is carried out on a ﬁnite subset of points chosen at random from the input space.

Original: This is expected to lead to poor results in high-dimensional spaces.
Revised: This is expected to lead to poor results in high-dimensional spaces.

Original: Moreover, another problem of using an inference network is that tuning the prior GP hyper-parameters is challenging and has often to be done in a separate step.
Revised: Moreover, another problem of using an inference network is that tuning the prior GP hyper-parameters is challenging and has often to be done in a separate step.

Original: Amortized variational inference (Shu et al., 2018) has also been explored in the context of GPs in
Revised: Amortized variational inference (Shu et al., 2018) has also been explored in the context of GPs in

Original: Villacampa-Calvo et al.
Revised: Villacampa-Calvo et al.

Original: There, input noise is considered in a multi-class learning problem and the  variational parameters of the posterior approximation for the noiseless inputs are amortized using a DNN receiving both x i and y i .
Revised: There, input noise is considered in a multi-class learning problem and the performance of the ﬁnal model is improved by amortizing the variational parameters of the posterior approximation for the noiseless inputs, using a DNN that receives both x i and y i .

Original: Amortized VI improves the performance of the ﬁnal model.
Revised: Other sparse GPs in the literature do not fully rely on inducing points, e.g. , (Tresp, 2000; Snelson, 2007; Gramacy & Apley, 2015). These techniques, however, cannot use, in general, stochastic optimization and do not scale to very large problems.

Original: Finally,  our method also relates to transductive inference in the sense that it uses test point information, i.e , via the DNN, to ﬁnd the parameters of q and the inducing points, to compute the predictions.
Revised: Finally, sparse GPs, and our method, can beneﬁt from natural-gradients (Salimbeni et al., 2018b). They could result in an orthogonal improvement.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the performance of the proposed method, to which we refer to as Input Dependent Sparse
Revised: We evaluate the performance of the proposed method, to which we refer to as Input Dependent Sparse

Original: GP (IDSGP).
Revised: GP (IDSGP).

Original: We consider both regression and binary classiﬁcation with a probit likelihood.
Revised: We consider both regression and binary classiﬁcation with a probit likelihood.

Original: In this later case, we approximate the expectation in the lower bound using 1-dimensional quadrature, as in
Revised: In this later case, we approximate the expectation in the lower bound using 1-dimensional quadrature, as in

Original: Hensman et al.
Revised: Hensman et al.

Original: (2015a).
Revised: (2015a).

Original: An implementation of the proposed method in Tensorﬂow 2.0  is provided in the supplementary material (Abadi et al., 2015).
Revised: The code of the proposed method in Tensorﬂow 2.0 (Abadi et al., 2015) is provided in the supplementary material.

Original: In the experiments we compare results with the standard variational sparse GP (Titsias, 2009).
Revised: In the experiments we compare results with the standard variational sparse GP (Titsias, 2009).

Original: We refer to such a method as VSGP.
Revised: We refer to such a method as VSGP.

Original: We also compare results with two of the methods described in Section 4.
Revised: We also compare results with two of the methods described in Section 4.

Original: Namely, the sparse within sparse GP (SWSGP) described in Tran et al.
Revised: Namely, the sparse within sparse GP (SWSGP) described in Tran et al.

Original: (2020), and the sparse GP based on an orthogonal decomposition that allows to consider two different sets of inducing points (Shi et al., 2020).
Revised: (2020), and the sparse GP based on an orthogonal decomposition that allows to consider two different sets of inducing points (Shi et al., 2020).

Original: We refer to this last method as SOLVE.
Revised: We refer to this last method as SOLVE.

Original: All methods use a Mat ´ ern 3 / 2 covariance function (Rasmussen & Williams, 2006).
Revised: All methods use a Mat´ern 3 / 2 covariance function (Rasmussen & Williams, 2006).

Original: The DNN architecture employed in IDSGP is described in detail in Appendix B.
Revised: The DNN architecture employed in IDSGP is described in detail in Appendix B.

Original: 5. T OY PROBLEMS
Revised: 5. T OY PROBLEMS

Original: We show the posterior mean and standard deviation of each method on a 1-dimensional regression problem (Snelson & Ghahramani, 2006).
Revised: We show the posterior mean and standard deviation of each method on a 1-dimensional regression problem (Snelson & Ghahramani, 2006).

Original: We compare results with a full GP.
Revised: We compare results with a full GP.

Original: Figure 2 shows the results obtained, including the learned locations of the inducing points.
Revised: Figure 2 shows the results obtained, including the learned locations of the inducing points.

Original: In the case of IDSGP we show the locations of the inducing points for the point represented with a star at x = 3 . 9 .
Revised: In the case of IDSGP we show the locations of the inducing points for the point represented with a star at x = 3 . 9 .

Original: The number of inducing points, for each method, are indicated in the ﬁgure’s caption.
Revised: The number of inducing points, for each method, are indicated in the ﬁgure’s caption.

Original: We consider a small number of inducing points to study the beneﬁts of having input dependent inducing points.
Revised: We consider a small number of inducing points to study the beneﬁts of having input dependent inducing points.

Original: IDSGP uses smaller number of inducing points than the other methods.
Revised: IDSGP uses smaller number of inducing points than the other methods.

Original: The ﬁgure shows that, in regions with observed data, IDSGP’s predictions look closer to those of the full GP.
Revised: The ﬁgure shows that, in regions with observed data, IDSGP’s predictions look closer to those of the full GP.

Original: Appendix D.1 has results for an increasing number of inducing points M .
Revised: Appendix D.1 has results for an increasing number of inducing points M .

Original: They show that as M increases IDSGP becomes more similar to the full GP.
Revised: They show that as M increases IDSGP becomes more similar to the full GP.

Original: Figure 3 shows the decision boundary of each method on the banana classiﬁcation dataset (Hensman et al., 2013).
Revised: Figure 3 shows the decision boundary of each method on the banana classiﬁcation dataset (Hensman et al., 2013).

Original: IDSGP produces the most accurate boundaries.
Revised: IDSGP produces the most accurate boundaries.

Revised: [Figure]

Revised: [Figure]

Original: 5. E XPERIMENTS ON UCI DATASETS
Revised: E XPERIMENTS ON UCI DATASETS

Original: We consider several regression and binary classiﬁcation datasets extracted from the UCI repository (Dua & Graff, 2017).
Revised: We consider several regression and binary classiﬁcation datasets extracted from the UCI repository (Dua & Graff, 2017).

Original: The number of inducing points of IDSGP is set to M = 15 .
Revised: The number of inducing points of IDSGP is set to M = 15 .

Original: In SOLVE we use M 1 = 1024 and M 2 = 1024 inducing points.
Revised: In SOLVE we use M 1 = 1024 and M 2 = 1024 inducing points.

Original: In VSGP we set M = 1024 .
Revised: In VSGP we set M = 1024 .

Original: In SWSGP we set
Revised: In SWSGP we set

Original: [Figure]

Original: [Figure]

Original: M = 1024 and K = 50 neighbors.
Revised: M = 1024 and K = 50 neighbors.

Original: All the methods are trained using ADAM (Kingma & Ba, 2015) with a mini-batch size of 100 and a learning rate of 0 . 01 .
Revised: All the methods are trained using ADAM (Kingma & Ba, 2015) with a mini-batch size of 100 and a learning rate of 0 . 01 .

Original: In the classiﬁcation setting we use the same setup, but the number of inducing points of IDSGP is set to M = 3 .
Revised: In the classiﬁcation setting we use the same setup, but the number of inducing points of IDSGP is set to M = 3 .

Original: All methods are trained on a
Revised: All methods are trained on a

Original: Tesla P100 GPU with 16GB of memory.
Revised: Tesla P100 GPU with 16GB of memory.

Original: On each dataset we use 80% of the data for training and the rest for testing.
Revised: On each dataset we use 80% of the data for training and the rest for testing.

Original: We report results across 5 splits of the data since the datasets are already quite big.
Revised: We report results across 5 splits of the data since the datasets are already quite big.

Original: The average negative test log-likelihood of each method on each dataset is displayed in Table 1, for the regression datasets, and in Table 2, for the classiﬁcation datasets, respectively.
Revised: The average negative test log-likelihood of each method on each dataset is displayed in Table 1, for the regression datasets, and in Table 2, for the classiﬁcation datasets, respectively.

Original: The average rank of each method is also displayed at the last row of each table.
Revised: The average rank of each method is also displayed at the last row of each table.

Original: The RMSE and prediction accuracy results are similar to those displayed here.
Revised: The RMSE and prediction accuracy results are similar to those displayed here.

Original: They can be found in Appendix D.2 and D.3.
Revised: They can be found in Appendix D.3 and D.4.

Original: Each table also shows the number of instances N and dimensions d of each dataset.
Revised: Each table also shows the number of instances N and dimensions d of each dataset.

Original: We observe that in the regression datasets, the proposed method, IDSGP, obtains best results in 6 out of the 8 datasets.
Revised: We observe that in the regression datasets, the proposed method, IDSGP, obtains best results in 6 out of the 8 datasets.

Original: IDSGP also obtains the best average rank (closer to always performing best on each train / test data split).
Revised: IDSGP also obtains the best average rank (closer to always performing best on each train / test data split).

Original: This is remarkable given that IDSGP a much smaller number of inducing points ( e.g. , M =for IDSGP vs. M = 1024 for VSGP).
Revised: This is remarkable given that IDSGP a much smaller number of inducing points ( e.g. , M =for IDSGP vs. M = 1024 for VSGP).

Original: In classiﬁcation, however, all the methods seem to perform similar to each other and the differences between them are smaller.
Revised: In classiﬁcation, however, all the methods seem to perform similar to each other and the differences between them are smaller.

Original: Again IDSGP uses here a smaller number of M = 3 inducing points.
Revised: Again IDSGP uses here a smaller number of M = 3 inducing points.

Original: Increasing M in IDSGP does not improve the results.
Revised: Increasing M in IDSGP does not improve the results.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In these experiments we also measure the average training time per epoch [Table] for each method.
Revised: In these experiments we also measure the average training time per epoch, for each method.

Original: The results corresponding to the UCI regression datasets are displayed in Table 3.
Revised: The results corresponding to the UCI regression datasets are displayed in Table 3.

Original: The results for the UCI classiﬁcation datasets are found in Appendix D.3.
Revised: The results for the UCI classiﬁcation datasets are found in Appendix D.4.

Original: They look very similar to ones reported here.
Revised: They look very similar to ones reported here.

Original: We observe that the fastest method in terms of training time is the proposed approach.
Revised: We observe that the fastest method in terms of training time is the proposed approach.

Original: Namely, IDSGP.
Revised: Namely, IDSGP.

Original: Nevertheless, the speed-up obtained is impaired by the overhead of having to compute the output of the DNN and update its parameters.
Revised: Nevertheless, the speed-up obtained is impaired by the overhead of having to compute the output of the DNN and update its parameters.

Original: IDSGP also results in fastest prediction times than VSGP,
Revised: IDSGP also results in fastest prediction times than VSGP,

Original: SOLVE or SWSGP.
Revised: SOLVE or SWSGP.

Original: See AppendixD.2 and D.3  for further results showing average prediction times.
Revised: See Appendix D.3 and D.4 for further results showing average prediction times.

Original: [Table]
Revised: [Table]

Original: 5. L ARGE SCALE DATASETS
Revised: 5. L ARGE SCALE DATASETS

Original: A last set of experiments considers two very large datasets.
Revised: A last set of experiments considers two very large datasets.

Original: The ﬁrst dataset is the Airlines Delay binary classiﬁcation dataset, as described in Hern´andez-Lobato & Hern´andez-Lobato (2016) [Table] with
Revised: The ﬁrst dataset is the Airlines Delay binary classiﬁcation dataset, as described in Hern´andez-Lobato & Hern´andez-Lobato (2016), with

Original: N = 2 , 127 , 068 data instances and d = 8 attributes.
Revised: N = 2 , 127 , 068 data instances and d = 8 attributes.

Original: The second dataset is the Yellow taxi dataset, as described in Salimbeni & Deisenroth (2017), with N = 1 billion data-points and d = 9 attributes.
Revised: The second dataset is the Yellow taxi dataset, as described in Salimbeni & Deisenroth (2017), with N = 1 billion data-points and d = 9 attributes.

Original: In each dataset we use a test set of 10 , 000 instances chosen at random.
Revised: In each dataset we use a test set of 10 , 000 instances chosen at random.

Original: The number of inducing points is set to be equal to M = 50 in IDSGP.
Revised: The number of inducing points is set to be equal to M = 50 in IDSGP.

Original: In the other methods, we use the same number of inducing points as in the previous section.
Revised: In the other methods, we use the same number of inducing points as in the previous section.

Original: The mini-batch size is set to 100 .
Revised: The mini-batch size is set to 100 .

Original: Training is also performed on the same GPU as in the previous section.
Revised: Training is also performed on the same GPU as in the previous section.

Original: The ADAM learning rate is set to 0 .
Revised: The ADAM learning rate is set to 0 .

Original: The average negative test log-likelihood of each methodon the test set is displayed in Figure 4, for each dataset.
Revised: The average negative test log-likelihood of each method is displayed in Figure 4, for each dataset.

Original: We report performance in terms of the training time, in a log 10 scale.
Revised: We report performance in terms of the training time, in a log 10 scale.

Original: The results corresponding to the RMSE are very similar to the ones displayed here.
Revised: The results corresponding to the RMSE are very similar to the ones displayed here.

Original: They can be found in Appendix D.4.
Revised: They can be found in Appendix D.5.

Original: We observe that the proposed method IDSGP performs best on each dataset.
Revised: We observe that the proposed method IDSGP performs best on each dataset.

Original: In particular it is able to obtain a better performance in a smaller computational time.
Revised: In particular, it obtains a better performance in a smaller computational time.

Original: We believe this is a consequence of using a smaller number of inducing points, and also because of the extra ﬂexibility that the DNN provides for specifying the locations of the inducing points.
Revised: We believe this is a consequence of using a smaller number of inducing points, and also because of the extra ﬂexibility that the DNN provides for specifying the locations of the inducing points.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Gaussian processes (GPs) are ﬂexible models for regression and classiﬁcation.
Revised: Gaussian processes (GPs) are ﬂexible models for regression and classiﬁcation.

Original: However, they have a cost of O ( N 3 )  with N the number of training points.
Revised: However, they have a cost of O ( N 3 ) per iteration with N the number of training points.

Original: Sparse approximations based on M (cid:28) N inducing points reduce such a cost to O ( M 3 ) .
Revised: Sparse approximations based on M (cid:28) N inducing points reduce such a cost to O ( M 3 ) .

Original: A problem, however, is that in some situations a large number of inducing points have to be used in practice, since they determine the ﬂexibility of the resulting approximation.
Revised: A problem, however, is that in some situations a large number of inducing points have to be used in practice, since they determine the ﬂexibility of the resulting approximation.

Original: There is hence a need to reduce their computational cost.
Revised: There is hence a need to reduce their training cost.

Original: We have proposed here input dependent sparse GP (IDSGP), a method that can improve the training time and the ﬂexibility of sparse GP approximations.
Revised: We have proposed here input dependent sparse GP (IDSGP), a method that can improve the training time and the ﬂexibility of sparse GP approximations.

Original: IDSGP uses a deep neural network (DNN) to output speciﬁc inducing points for each point at which the predictive distribution of the GP needs to be computed.
Revised: IDSGP uses a deep neural network (DNN) to output speciﬁc inducing points for each point at which the predictive distribution of the GP needs to be computed.

Original: The DNN also outputs the parameters of the corresponding variational approximation on the inducing values associated to the inducing points.
Revised: The DNN also outputs the parameters of the corresponding variational approximation on the inducing values associated to the inducing points.

Original: IDSGP can be obtained under a formulation that considers an implicit distribution for the input instance to the DNN.
Revised: IDSGP can be obtained under a formulation that considers an implicit distribution for the input instance to the DNN.

Original: Importantly, such a formulation is shown to keep intact the GP prior on the latent function values associated to the training points.
Revised: Importantly, such a formulation is shown to keep intact the GP prior on the latent function values associated to the training points.

Original: The extra ﬂexibility provided by the DNN allows to signiﬁcantly reduce the number M of inducing points used in IDSGP.
Revised: The extra ﬂexibility provided by the DNN allows to signiﬁcantly reduce the number M of inducing points used in IDSGP.

Original: Such a model provides similar or better results than other sparse GP approximations from the literature at a smaller computational cost.
Revised: Such a model provides similar or better results than other sparse GP approximations from the literature at a smaller training cost.

Original: IDSGP has been evaluated on several regression and binary classiﬁcation problems from the UCI repository.
Revised: IDSGP has been evaluated on several regression and binary classiﬁcation problems from the UCI repository.

Original: The results obtained show that it improves the quality of the predictive distribution and reduces the computational cost of sparse GP approximations.
Revised: The results obtained show that it improves the quality of the predictive distribution and reduces the training cost of sparse GP approximations.

Original: Better results are most of the times obtained in regression problems.
Revised: Better results are most of the times obtained in regression problems.

Original: In classiﬁcation problems, however, the performances obtained are similar to those of the state-of-the-art, although the training and prediction times are always shorter.
Revised: In classiﬁcation problems, however, the performances obtained are similar to those of the state-of-the-art, although the training and prediction times are always shorter.

Original: The scalability of IDSGP is also illustrated on massive datasets for regression and binary classiﬁcation of up to 1 billion points.
Revised: The scalability of IDSGP is also illustrated on massive datasets for regression and binary classiﬁcation of up to 1 billion points.

Original: There, IDSGP also obtains better results than alternative sparse GP approximations at a smaller training cost.
Revised: There, IDSGP also obtains better results than alternative sparse GP approximations at a smaller training cost.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: The authors acknowledge to have read and commit to adhering to the ICLR Code of Ethics 1 .
Revised: The authors acknowledge to have read and commit to adhering to the ICLR Code of Ethics 1 .

Original: We do not see any direct potential negative societal impact, because this paper focuses on the development of a new methodology.
Revised: We do not see any direct potential negative societal impact, because this paper focuses on the development of a new methodology.

Original: We believe these would be indirect through the particular application in which the proposed method is used.
Revised: We believe these would be indirect through the particular application in which the proposed method is used.

Original: As one of the main advantages of GPs is that they provide uncertainty estimates associated with the predictions made, we think the potential harm of these models in society could arise in applications when this uncertainty estimation is critical.
Revised: As one of the main advantages of GPs is that they provide uncertainty estimates associated with the predictions made, we think the potential harm of these models in society could arise in applications when this uncertainty estimation is critical.

Original: For example, an AI system in which the decisions made can have an inﬂuence on people’s life, such as autonomous vehicles or automatic medical diagnosis tools.
Revised: For example, an AI system in which the decisions made can have an inﬂuence on people’s life, such as autonomous vehicles or automatic medical diagnosis tools.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Most of the details needed to reproduce this paper’s results are described in Section 5.
Revised: Most of the details needed to reproduce this paper’s results are described in Section 5.

Original: In Appendix B we give further details about the neural network architecture and initialization.
Revised: In Appendix B we give further details about the neural network architecture and initialization.

Original: We provide the code used to run the experiments in the supplementary material, with implementation of our method and the rest of methods of the experimental comparison.
Revised: We provide the code used to run the experiments in the supplementary material, with implementation of our method and the rest of methods of the experimental comparison.

Original: Regarding the pre-processing of the data, you can ﬁnd all the information needed to reproduce this step in Appendix A.
Revised: Regarding the pre-processing of the data, you can ﬁnd all the information needed to reproduce this step in Appendix A.

Original: A D ATASETS PRE - PROCESSING
Revised: A D ATASETS PRE - PROCESSING

Original: All the datasets are publicly available.
Revised: All the datasets are publicly available.

Original: The UCI repository datasets can be downloaded from the repository (Dua & Graff, 2017).
Revised: The UCI repository datasets can be downloaded from the repository (Dua & Graff, 2017).

Original: Yellow taxi dataset was preprocessed following Salimbeni & Deisenroth (2017) and downloaded from https://www1.nyc.gov/site/tlc/about/ tlc-trip-record-data.page , where we have used data records from year 2015 .
Revised: Yellow taxi dataset was preprocessed following Salimbeni & Deisenroth (2017) and downloaded from https://www1.nyc.gov/site/tlc/about/ tlc-trip-record-data.page , where we have used data records from year 2015 .

Original: Similarly, the Airlines Delay dataset was preprocessed following Hern´andez-Lobato & Hern´andez-Lobato (2016) and was downloaded from https://community.amstat.org/jointscsg-section/ dataexpo/dataexpo2009 , keeping only the records from January 2008 to April 2008 .
Revised: Similarly, the Airlines Delay dataset was preprocessed following Hern ´ andez-Lobato & Hern ´ andez-Lobato (2016) and was downloaded from https://community.amstat.org/jointscsg-section/ dataexpo/dataexpo2009 , keeping only the records from January 2008 to April 2008 .

Original: All datasets have been standardized using scikit-learn’s built-int StandardScaler class (Pedregosa et al., 2011), which removes the mean and scales to unit variance.
Revised: All datasets have been standardized using scikit-learn’s built-int StandardScaler class (Pedregosa et al., 2011), which removes the mean and scales to unit variance.

Original: B N EURAL N ETWORK A RCHITECTURE
Revised: B N EURAL N ETWORK A RCHITECTURE

Original: About the choice of architecture for the DNN we have tried to keep it small in order to take more advantage of the computational gain of the amortized scheme.
Revised: About the choice of architecture for the DNN we have tried to keep it small in order to take more advantage of the computational gain of the amortized scheme.

Original: In particular, we used a 2 hidden-layer with 50 hidden units network for the toy problems in Section 5.1, a 1 layer with 50 hidden units network for the UCI datasets in Section 5.2 and a 2 layer with 25 hidden units for the large scale datasets in Section 5.3.
Revised: In particular, we used a 2 hidden-layer with 50 hidden units network for the toy problems in Section 5.1, a 1 layer with 50 hidden units network for the UCI datasets in Section 5.2 and a 2 layer with 25 hidden units for the large scale datasets in Section 5.3.

Original: We used ReLu activation functions.
Revised: We used ReLu activation functions.

Original: Keeping the network small reduces the number of parameters to optimize making the optimization process easier.
Revised: Keeping the network small reduces the number of parameters to optimize making the optimization process easier.

Original: In all problems we are using fully-connected layers with batch normalization and no skip layers.
Revised: In all problems we are using fully-connected layers with batch normalization and no skip layers.

Original: Regarding the initialization of the weights, all were initialized using the Glorot initialization (Glorot & Bengio, 2010).
Revised: Regarding the initialization of the weights, all were initialized using the Glorot initialization (Glorot & Bengio, 2010).

Original: In our experiments we did not exhaustively explore the DNN architecture.
Revised: In our experiments we did not exhaustively explore the DNN architecture.

Original: This choice of architecture and initialization was based on some preliminary tests done before running the experiments.
Revised: This choice of architecture and initialization was based on some preliminary tests done before running the experiments.

Original: This does not mean that this is the best possible conﬁguration.
Revised: This does not mean that this is the best possible conﬁguration.

Original: We did not optimize the architecture of the neural network.
Revised: We did not optimize the architecture of the neural network.

Original: In practical applications, we suggest to run some preliminary tests in order to choose a conﬁguration that performs well.
Revised: In practical applications, we suggest to run some preliminary tests in order to choose a conﬁguration that performs well.

Original: The main suggestion, however, is to keep the network small as the input dependence will make the model expressive enough to still get very good results.
Revised: The main suggestion, however, is to keep the network small as the input dependence will make the model expressive enough to still get very good results.

Original: C C HOOSING THE N UMBER OF I NDUCING P OINTS
Revised: C C HOOSING THE N UMBER OF I NDUCING P OINTS

Original: We have observed that our proposed method IDSGP performs well in general with a fairly small number of inducing points, much smaller than the number of inducing points used in the other methods.
Revised: We have observed that our proposed method IDSGP performs well in general with a fairly small number of inducing points, much smaller than the number of inducing points used in the other methods.

Original: Namely, SOLVE, VSGP and SWSGP.
Revised: Namely, SOLVE, VSGP and SWSGP.

Original: This is probably related to the extra ﬂexibility of having input-dependent inducing points in IDSGP.
Revised: This is probably related to the extra ﬂexibility of having input-dependent inducing points in IDSGP.

Original: In very large datasets we recommend using around
Revised: In very large datasets we recommend using around

Original: M = 50 inducing points.
Revised: M = 50 inducing points.

Original: In medium-size regression datasets M = 15 inducing points seem enough.
Revised: In medium-size regression datasets M = 15 inducing points seem enough.

Original: In medium-size binary classiﬁcation datasets, however, a smaller number of inducing points is enough
Revised: In medium-size binary classiﬁcation datasets, however, a smaller number of inducing points is enough

Original: M = 3 .
Revised: M = 3 .

Original: We believe the reason is that binary classiﬁcation problems require less complicated latent functions.
Revised: We believe the reason is that binary classiﬁcation problems require less complicated latent functions.

Original: We did not optimize the number of inducing points.
Revised: We did not optimize the number of inducing points.

Original: In practical applications, we suggest to run some preliminary tests in order to choose a number of inducing points that performs well.
Revised: In practical applications, we suggest to run some preliminary tests in order to choose a number of inducing points that performs well.

Original: D E XTRA EXPERIMENTAL RESULTS
Revised: C. KL-D IVERGENCE M INIMIZATION

Revised: In this section we show that maximizing (4) effectively minimizes the KL-divergence between q (˜ x , f , u ) and p (˜ x , f , u | y ) . In particular, [Equation]

Revised: where we have used that the posterior is equal to the joint p (˜ x , f , u , y ) divided by a normalization constant, i.e. , the marginal likelihood.

Revised: Moreover, L is simply the lower bound deﬁned in (4).

Revised: Therefore, maximizing L effectively leads to the minimization of the KL-divergence between q (˜ x , f , u ) and p (˜ x , f , u | y ) .

Revised: D E XTRA EXPERIMENTAL RESULTS

Original: In this section, we include some extra results that do not ﬁt in the main manuscript.
Revised: In this section, we include some extra results that do not ﬁt in the main manuscript.

Original: Namely, the
Revised: Namely, the

Original: RMSE in the test set results and prediction times for the UCI regression datasets, and the accuracy in the test set, training and prediction times for the UCI classiﬁcation datasets.
Revised: RMSE in the test set results and prediction times for the UCI regression datasets, and the accuracy in the test set, training and prediction times for the UCI classiﬁcation datasets.

Original: In both cases, the setup is the same as described in Section 5 and the results are similar that the ones obtained in terms of the negative test log likelihood and training times in that section.
Revised: In both cases, the setup is the same as described in Section 5 and the results are similar that the ones obtained in terms of the negative test log likelihood and training times in that section.

Original: Finally, we include similar plots to those in Section 5.3 but in terms of the test RMSE for the Yellow Taxi dataset and in terms of the test classiﬁcation error for the Airline Delays dataset.
Revised: Finally, we include similar plots to those in Section 5.3 but in terms of the test RMSE for the Yellow Taxi dataset and in terms of the test classiﬁcation error for the Airline Delays dataset.

Original: D. T OY REGRESSION DATASETS
Revised: D. T OY REGRESSION DATASETS

Original: Our method looks more and more similar to the full GP as number of inducing points M increases.
Revised: Our method looks more and more similar to the full GP as number of inducing points M increases.

Original: However, with a small number of inducing points, it gives similar results to those of the full GP and similar to the results obtained when more inducing points are considered, which does not happen in the other methods.
Revised: However, with a small number of inducing points, it gives similar results to those of the full GP and similar to the results obtained when more inducing points are considered, which does not happen in the other methods.

Original: This is probably due to the extra ﬂexibility of the neural network.
Revised: This is probably due to the extra ﬂexibility of the neural network.

Original: The ﬁgures below (Figures 6 to 8) show the results of each method on the toy regression problem as we increase the number of inducing points M .
Revised: The ﬁgures below (Figures 6 to 8) show the results of each method on the toy regression problem as we increase the number of inducing points M .

Original: For M = 128 IDSGP gives almost the same results as VSGP.
Revised: For M = 128 IDSGP gives almost the same results as VSGP.

Original: [Table] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Figure][Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: Here we run the 1D toy regression experiment of Section 5.1 using the closed-form solution approach of Titsias (2009) for ﬁnding q .

Original: [Figure] [Table] [Figure] [Table]
Revised: More precisely, this method is exactly the same as the SVGP method we compare results with, but where the approximate distribution q is not optimized at all.

Original: [Figure] [Table] [Figure]
Revised: The reason for this is that it is possible to ﬁnd a closed-form solution for q .

Original: [Table]
Revised: However and importantly, the resulting method does not allow for mini-batch training. Since SVGP* does not allow for stochastic optimization, the batch size is set equal to the number of training points ( N = 200) . Figure 9 shows the ﬁt obtained for an increasing number of inducing points M .

Original: [Figure] [Figure] [Figure] [Figure]
Revised: The results are very similar to the ones of SVGP in Figure 7.

Original: D. UCI REGRESSION DATASETS
Revised: D. UCI REGRESSION DATASETS

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] D. UCI CLASSIFICATION DATASETS
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: DefaultOrCredit

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: D. L ARGE SCALE DATASETS
Revised: D. L ARGE SCALE DATASETS

Original: [Figure]
Revised: [Figure]

Revised: D. N EURAL NETWORK TRAINED VIA MAXIMUM LIKELIHOOD

Revised: In this subsection we show extra experiment results on the UCI datasets when using a neural network trained via maximum likelihood.

Revised: The architecture of the neural network is the same as the one of the network used in the proposed method IDSGP.

Revised: Training is done using ADAM.

Revised: The learning rate used is 0.001.

Revised: The mini-batch size is the same for the GP-based methods.

Revised: In regression, we use the neural network to predict the mean and variance of the Gaussian predictive distribution.

Revised: In classiﬁcation, we use a sigmoid activation function.

Revised: The average test negative log-likelihood obtained in each problem is shown in Table 9 and Table 10.

Revised: The results are high-lighted in bold-face when the NN performs worse than any other GP based method.

Revised: The tables show that, in the case of regression problems, most of the times the neural network performs worse than the GP based methods.

Revised: In the case of classiﬁcation problems, the performance of the neural network is worse in those problems in which the error is higher according to Table 6.

Revised: By contrast, in those problems in which the accuracy is almost equal to 100%, there are no differences or it performs slightly better.

Revised: [Table]

Revised: [Table]