Original: I NDUCING G AUSSIAN P ROCESS N ETWORKS
Revised: I NDUCING G AUSSIAN P ROCESS N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Gaussian processes (GPs) are powerful but computationally expensive machine learning models, requiring an estimate of the kernel covariance matrix for every prediction.
Revised: Gaussian processes (GPs) are powerful but computationally expensive machine learning models, requiring an estimate of the kernel covariance matrix for every prediction.

Original: In large and complex domains, such as graphs, sets, or images, the choice of suitable kernel can also be non-trivial to determine, providing an additional obstacle to the learning task.
Revised: In large and complex domains, such as graphs, sets, or images, the choice of suitable kernel can also be non-trivial to determine, providing an additional obstacle to the learning task.

Original: Over the last decade, these challenges have resulted in significant advances being made in terms of scalability and expressivity, exemplified by, e.g., the use of inducing points and neural network kernel approximations.
Revised: Over the last decade, these challenges have resulted in significant advances being made in terms of scalability and expressivity, exemplified by, e.g., the use of inducing points and neural network kernel approximations.

Original: In this paper, we propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points.
Revised: In this paper, we propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points.

Original: The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains while also facilitating scalable gradient-based learning methods.
Revised: The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains while also facilitating scalable gradient-based learning methods.

Original: We consider both regression and (binary) classification tasks and report on experimental results for real-world data sets showing that IGNs provide significant advances over state-ofthe-art methods.
Revised: We consider both regression and (binary) classification tasks and report on experimental results for real-world data sets showing that IGNs provide significant advances over state-ofthe-art methods.

Original: We also demonstrate how IGNs can be used to effectively model complex domains using neural network architectures.
Revised: We also demonstrate how IGNs can be used to effectively model complex domains using neural network architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gaussian processes are powerful and attractive machine learning models, in particular in situations where uncertainty estimation is critical for performance, such as for medical diagnosis (Dusenberry et al., 2020).Whereas the original Gaussian process formulation was limited in terms of scalability, there has been significant progress in scalable solutions with Quiñonero-Candela & Rasmussen (2005) providing an early unified framework based on inducing points as a representative proxy of the training data.
Revised: Gaussian processes are powerful and attractive machine learning models, in particular in situations where uncertainty estimation is critical for performance, such as for medical diagnosis (Dusenberry et al., 2020).Whereas the original Gaussian process formulation was limited in terms of scalability, there has been significant progress in scalable solutions with Quiñonero-Candela & Rasmussen (2005) providing an early unified framework based on inducing points as a representative proxy of the training data.

Original: The framework by Quiñonero-Candela & Rasmussen (2005) has also been extended to variational settings (Titsias, 2009; Wilson et al., 2016b), further enabling a probabilistic basis for reasoning about the number of inducing points (Uhrenholt et al., 2021).
Revised: The framework by Quiñonero-Candela & Rasmussen (2005) has also been extended to variational settings (Titsias, 2009; Wilson et al., 2016b; Bauer, 2016), further enabling a probabilistic basis for reasoning about the number of inducing points (Uhrenholt et al., 2021).

Original: In terms of computational scalability, methods for leveraging the available computational resources have recently been considered (Nguyen et al., 2019b; Wang et al., 2019), with Chen et al.
Revised: In terms of computational scalability, methods for leveraging the available computational resources have recently been considered (Nguyen et al., 2019b; Wang et al., 2019), with Chen et al.

Original: (2020) also providing insights into the theoretical underpinnings for gradient descent-based solutions in correlated settings (as in the case for GPs).
Revised: (2020) also providing insights into the theoretical underpinnings for gradient descent-based solutions in correlated settings (as in the case for GPs).

Original: Common for most of the inducing points-based approaches to scalability is that the inducing points live in the same space as the training points (see e.g. (Snelson & Ghahramani, 2006; Titsias, 2009;
Revised: Common for most of the inducing points-based approaches to scalability is that the inducing points live in the same space as the training points (see e.g. (Snelson & Ghahramani, 2006; Titsias, 2009;

Original: Hensman et al., 2013; Damianou & Lawrence, 2013)).
Revised: Hensman et al., 2013; Damianou & Lawrence, 2013)).

Original: However, learning inducing points in the input space can be challenging for complex domains (e.g. over graphs), domains with high dimensionality (e.g. images), or domains with varying cardinality (e.g. text or point clouds)
Revised: However, learning inducing points in the input space can be challenging for complex domains (e.g. over graphs), domains with high dimensionality (e.g. images), or domains with varying cardinality (e.g. text or point clouds)

Original: (Lee et al., 2019;
Revised: (Lee et al., 2019;

Original: Aitchison et al., 2021).
Revised: Aitchison et al., 2021).

Original: More recently, methods for reasoning about the inducing points directly in embedding space have also been considered, but these methods often constrain the positions of the inducing points (Wilson et al., 2016a) or the structure of the embedding space (Lázaro-Gredilla &
Revised: More recently, methods for reasoning about the inducing points directly in embedding space have also been considered, but these methods often constrain the positions of the inducing points (Wilson et al., 2016a) or the structure of the embedding space (Lázaro-Gredilla &

Original: Figueiras-Vidal, 2009; Bradshaw et al., 2017), or they rely on complex inference/learning procedures (Aitchison et al., 2021)
Revised: Figueiras-Vidal, 2009; Bradshaw et al., 2017), or they rely on complex inference/learning procedures (Aitchison et al., 2021)

Original: In this paper, we propose inducing Gaussian process networks (IGN) as a simple and scalable framework for jointly learning the inducing points and the (deep) kernel (Wilson et al., 2016a).
Revised: In this paper, we propose inducing Gaussian process networks (IGN) as a simple and scalable framework for jointly learning the inducing points and the (deep) kernel (Wilson et al., 2016a).

Original: Key to the framework is that the inducing points live in an unconstrained feature space rather than in the input space.
Revised: Key to the framework is that the inducing points live in an unconstrained feature space rather than in the input space.

Original: By defining the inducing points in the feature space together with an amortized pseudo-label function, we are able to represent the data distribution with a simple base kernel (such as
Revised: By defining the inducing points in the feature space together with an amortized pseudo-label function, we are able to represent the data distribution with a simple base kernel (such as

Original: For learning IGNs, we rely on a maximum likelihood-based learning objective that is optimized using mini-batch gradient descent (Chen et al., 2020).
Revised: For learning IGNs, we rely on a maximum likelihood-based learning objective that is optimized using mini-batch gradient descent (Chen et al., 2020).

Original: This setup allows the method to scale to large data sets as demonstrated in the experimental results.
Revised: This setup allows the method to scale to large data sets as demonstrated in the experimental results.

Original: Furthermore, by only having the inducing points defined in feature space, we can seamlessly employ standard gradient-based techniques for learning the inducing points (even when the inputs space is defined over complex discrete/hybrid objects) without the practical difficulties sometimes encountered when learning deep neural network structures.
Revised: Furthermore, by only having the inducing points defined in feature space, we can seamlessly employ standard gradient-based techniques for learning the inducing points (even when the inputs space is defined over complex discrete/hybrid objects) without the practical difficulties sometimes encountered when learning deep neural network structures.

Original: We evaluate the performance of the proposed framework on several well-known data sets and show significant improvements compared to state of the art methods.
Revised: We evaluate the performance of the proposed framework on several well-known data sets and show significant improvements compared to state of the art methods.

Original: We provide a qualitative analysis of the framework using a two-class version of the MNIST dataset.
Revised: We provide a qualitative analysis of the framework using a two-class version of the MNIST dataset.

Original: This is complemented by a more detailed quantitative analysis using the full MNIST and CIFAR10 data sets.
Revised: This is complemented by a more detailed quantitative analysis using the full MNIST and CIFAR10 data sets.

Original: Lastly, to demonstrate the versatility of the framework, we also provide sentiment analysis results for both a text-based and a graph-based dataset derived from the IMDB movie review dataset.
Revised: Lastly, to demonstrate the versatility of the framework, we also provide sentiment analysis results for both a text-based and a graph-based dataset derived from the IMDB movie review dataset.

Original:  T HE INDUCING G AUSSIAN PROCESS NETWORKS (IGN) FRAMEWORK
Revised:  T HE INDUCING G AUSSIAN PROCESS NETWORKS (IGN) FRAMEWORK

Original: We start by considering regression problems, defined over an input space X and a label space of observations R , modeled by a Gaussian process:
Revised: We start by considering regression problems, defined over an input space X and a label space of observations R , modeled by a Gaussian process:

Original: [Equation]
Revised: [Equation]

Original: where k ( · , · ) : X ×X → R denotes the kernel describing the prior covariance, and ϵ ∼ N (0 , σ 2 ϵ ) is the noise associated with the observations.
Revised: where k ( · , · ) : X ×X → R denotes the kernel describing the prior covariance, and ϵ ∼ N (0 , σ 2 ϵ ) is the noise associated with the observations.

Original: We assume access to a set of data points D
Revised: We assume access to a set of data points D

Original: = { ( x i , y i ) } n i = generated from the model in Equation 1, and we seek to learn the parameters that define k and σ ϵ in order to predict outputs for new points x ∗ in X .
Revised: = { ( x i , y i ) } n i = generated from the model in Equation 1, and we seek to learn the parameters that define k and σ ϵ in order to predict outputs for new points x ∗ in X .

Original: In what follows we shall use X and y to denote ( x 1 , . . .
Revised: In what follows we shall use X and y to denote ( x 1 , . . .

Original: , x n ) T and ( y 1 , . . .
Revised: , x n ) T and ( y 1 , . . .

Original: y n ) T , respectively.
Revised: y n ) T , respectively.

Original: Firstly, we propose to embed the input points using a neural network g θ g : X → R d parameterized by θ g .
Revised: Firstly, we propose to embed the input points using a neural network g θ g : X → R d parameterized by θ g .

Original: Secondly, for modeling the kernel function k , we introduce a set of m inducing points
Revised: Secondly, for modeling the kernel function k , we introduce a set of m inducing points

Original: Z = ( z 1 , . . .
Revised: Z = ( z 1 , . . .

Original: , z m ) T , z i ∈ R d , together with a (linear) pseudo-label function r θ r : R d → R parameterized by θ r .
Revised: , z m ) T , z i ∈ R d , together with a (linear) pseudo-label function r θ r : R d → R parameterized by θ r .

Original: We will use r =
Revised: We will use r =

Original: ( r θ r ( z 1 ) , . . .
Revised: ( r θ r ( z 1 ) , . . .

Original: , r θ r ( z m )) T to denote the evaluation of r on
Revised: , r θ r ( z m )) T to denote the evaluation of r on

Original: Z , where r will play a rôle similar to that of inducing variables (Quiñonero-Candela & Rasmussen, 2005).
Revised: Z , where r will play a rôle similar to that of inducing variables (Quiñonero-Candela & Rasmussen, 2005).

Original: In the remainder of this paper, we will sometimes drop the parameter subscripts θ g and θ r from g and r for ease of representation.
Revised: In the remainder of this paper, we will sometimes drop the parameter subscripts θ g and θ r from g and r for ease of representation.

Original: An illustration of the model and the relationship between the training data and the inducing points can be see in Figure 1.
Revised: An illustration of the model and the relationship between the training data and the inducing points can be see in Figure 1.

Original: We finally define k : R d × R d
Revised: We finally define k : R d × R d

Original: → R as the kernel between pairs of vectors in R d .
Revised: → R as the kernel between pairs of vectors in R d .

Original: In particular, we denote with
Revised: In particular, we denote with

Original: [Equation]
Revised: [Equation]

Original: the four matrices corresponding to the kernel evaluations for all pairs of observed inputs and inducing points.
Revised: the four matrices corresponding to the kernel evaluations for all pairs of observed inputs and inducing points.

Original: The joint distribution over the function values at the observed inputs and the inducing points is now given as
Revised: The joint distribution over the function values at the observed inputs and the inducing points is now given as

Original: [Equation]
Revised: [Equation]

Original: where we slightly overload notation by using r to denote the function values associated with the inducing points, even though they do not correspond to real observations.
Revised: where we slightly overload notation by using r to denote the function values associated with the inducing points, even though they do not correspond to real observations.

Original: For a given data set D , our goal is to jointly learn θ = { Z, θ g , θ r , σ ϵ } by considering the marginal likelihood:
Revised: For a given data set D , our goal is to jointly learn θ = { Z, θ g , θ r , σ ϵ } by considering the marginal likelihood:

Original: [Equation]
Revised: [Equation]

Original: where ˆ y is the predictive mean:
Revised: where ˆ y is the predictive mean:

Original: [Equation]
Revised: [Equation]

Original: and K X | Z is the posterior kernel given the inducing points, i.e.
Revised: and K X | Z is the posterior kernel given the inducing points, i.e.

Original: [Equation]
Revised: [Equation]

Original: Note that the kernel K X | Z is implicitly parameterized by θ g through the embedding function g θ g .
Revised: Note that the kernel K X | Z is implicitly parameterized by θ g through the embedding function g θ g .

Revised: Furthermore, by jointly optimizing θ r together with the other model parameters we establish the connection between f and r .

Original: As in (Chen et al., 2020), we define our objective function in terms of the log-likelihood of the posterior of Equation 3:
Revised: As in (Chen et al., 2020), we define our objective function in terms of the log-likelihood of the posterior of Equation 3:

Original: [Equation]
Revised: [Equation]

Original:  which we can minimize wrt.
Revised: (6) which we can minimize wrt.

Original: the parameters θ using mini-natch gradient descent with appropriate scaling factors for the gradients (Chen et al., 2020).
Revised: the parameters θ using mini-natch gradient descent with appropriate scaling factors for the gradients (Chen et al., 2020).

Original: The gradient of ℓ ( θ ; D ) wrt.
Revised: The gradient of ℓ ( θ ; D ) wrt.

Original: θ can be expressed as:
Revised: θ can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: where tr ( A ) represents the trace of a matrix A .
Revised: where tr ( A ) represents the trace of a matrix A .

Original: Note that in Equation 7 both ˆ y and K X | Z depend on θ and that the joint optimization of the model parameters, in particular θ r , establishes the connection between the Gaussian process and the pseudo-label function.
Revised: Note that in Equation 7 both ˆ y and K X | Z depend on θ and that the joint optimization of the model parameters, in particular θ r , establishes the connection between the Gaussian process and the pseudo-label function.

Original: Based on the chain rule of differentiation, we see that the gradient of Equation 7 with respect to the inducing points Z and the parameters θ r of the pseudo-label function do not depend on ∇ g θ g , that is the gradients of g
Revised: Based on the chain rule of differentiation, we see that the gradient of Equation 7 with respect to the inducing points Z and the parameters θ r of the pseudo-label function do not depend on ∇ g θ g , that is the gradients of g

Original: θ g have no impact on the updates of Z and θ r .
Revised: θ g have no impact on the updates of Z and θ r .

Original: Proposition 1.
Revised: Proposition 1.

Original: The gradients ∇ Z ℓ ( θ ; D ) and ∇ θ r ℓ ( θ ; D ) do not depend on ∇ g θ g .
Revised: The gradients ∇ Z ℓ ( θ ; D ) and ∇ θ r ℓ ( θ ; D ) do not depend on ∇ g θ g .

Original: This is a key advantage of IGNs as learning the inducing points is therefore not influenced by the gradient of a potentially complex embedding function g and the entailed optimization difficulties.
Revised: This is a key advantage of IGNs as learning the inducing points is therefore not influenced by the gradient of a potentially complex embedding function g and the entailed optimization difficulties.

Original: Additionally, as the inducing points (only) exists in R d , the underlying learning framework for the inducing points is indifferent to the structure of the input space X and whether it is discrete or continuous or defined over, e.g., graphs or images.
Revised: Additionally, as the inducing points (only) exists in R d , the underlying learning framework for the inducing points is indifferent to the structure of the input space X and whether it is discrete or continuous or defined over, e.g., graphs or images.

Original: Once the IGN parameters θ = { Z, θ g , θ r , σ ϵ } have been learned, we can find the predictive distribution for input points X ∗ and inducing points Z , by first considering the joint distribution over the associated function values
Revised: Once the IGN parameters θ = { Z, θ g , θ r , σ ϵ } have been learned, we can find the predictive distribution for input points X ∗ and inducing points Z , by first considering the joint distribution over the associated function values

Original: [Equation]
Revised: [Equation]

Original: which in turn gives the predictive distribution
Revised: which in turn gives the predictive distribution

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where
Revised: (8) where

Original: [Equation]
Revised: [Equation]

Original: In this section, we extend the Gaussian process networks to classification.
Revised: Finally, IGNs can straightforwardly be extended to solve classification tasks.

Original: For ease of exposition, we only consider binary classification problems, but the general method can straightforwardly be extended to a multi-valued setting.
Revised: For ease of exposition, we only consider binary classification problems, but the general method can straightforwardly be extended to a multi-valued setting.

Original: For the experiment results, (see Section 4.3) we therefore use a one-vs-all approach to solve multi-class classification tasks.

Original: We assume that the label space is given by Y = { 0 , 1 } and that we have access to a data set D
Revised: 1 That is, we assume that the label space is given by Y = { 0 , 1 } and that we have access to a data set D

Original: = { ( x i , y i ) } ni
Revised: = { ( x i , y i ) } ni

Original: =1 , where the underlying data generating process is defined by a latent function f ( x ) with a GP prior f ∼ GP (0 , k ( · , · )) and y | x ∼ Φ( f ( x )) , where Φ( · ) is the cumulative Gaussian function; see (Rasmussen & Williams, 2006).For inference, we calculate the posterior distribution p ( y | X ∗ , θ ) given by [Equation] Assuming that f ∗ ∼ N ( µ, σ 2 ) for some µ and σ we get (Rasmussen & Williams, 2006, Section 3.9): p ( y ∗ | x ∗ , θ ) =
Revised: =1 , where the underlying data generating process is defined by a latent function f ( x ) with a GP prior f ∼ GP (0 , k ( · , · )) and y | x ∼ Φ( f ( x )) , where Φ( · ) is the cumulative Gaussian function; see (Rasmussen & Williams, 2006).

Original: Φ

Original: [Equation]

Original: α ) with α

Original: [Equation]

Original: where µ and σ 2 can be found from the predictive distribution in Equation 8.

Original: For learning the IGN parameters θ we perform maximum likelihood estimation wrt.

Original: [Equation]

Original: In order to evaluate the integral, we rely on a Laplace approximation centered around [Equation] (found using Newton’s method) yielding the log-likelihood approximation [Equation]
Revised: In order to perform maximum likelihood estimation of the model parameters, we approximate the likelihood using a Laplace approximation.

Original: where c represent the accumulated constant terms and [Equation] f ) + log p ( f | X, θ )) | f
Revised: For the sake of compactness, we defer the full derivation to Appendix B.

Original: = ˆ f .

Original: The form of p ( ˆ f | X, θ ) is given in Equation 3 and ∇∇ (log p ( y | ˆ f ) can be found in, e.g., (Rasmussen & Williams, 2006).

Original: Optimizing this log-likelihood approximation can now straightforwardly be achieved by interleaving mini-batch gradient descent and Newton’s approximation.

Original: As for regression, we also have here that the gradient of (the Laplace approximation of) ℓ ( θ ; D ) wrt.

Original: the inducing points

Original: Z and θ r do not depend on ∇ θ g of the embedding function g .

Original: Hence, IGNs for classification enjoy the same properties when learning the inducing points as IGNs for regression.

Original: Additional details can be found in the appendix.

Original: 2. C OMPUTATIONAL COMPLEXITY
Revised: 2. C OMPUTATIONAL COMPLEXITY

Original: The main aspect related to the computational complexity of the IGN framework concerns the kernel computation K Z | X .
Revised: The main aspect related to the computational complexity of the IGN framework concerns the kernel computation K Z | X .

Original: If n z is the number of inducing points and b is the mini-batch size (recall that for training we use mini-batch gradient descent), the complexity of computing the kernel is cubic wrt n z , i.e. O ( n 3 z ) .
Revised: If n z is the number of inducing points and b is the mini-batch size (recall that for training we use mini-batch gradient descent), the complexity of computing the kernel is cubic wrt n z , i.e. O ( n 3 z ) .

Original: However, at inference time, we can omit the term connected to K − 1 ZZ , since it only needs to be efficiently computed once (Sharma et al., 2013).
Revised: However, at inference time, we can omit the term connected to K − 1 ZZ , since it only needs to be efficiently computed once (Sharma et al., 2013).

Original: If we at this phase further assume that n z > b , the complexity reduces to the computation of K ZX and K XX , i.e. O ( n z · b ) .
Revised: If we at this phase further assume that n z > b , the complexity reduces to the computation of K ZX and K XX , i.e. O ( n z · b ) .

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: The combination of kernels and neural networks has previously been explored, most notably in the context of deep kernel learning (Wilson et al., 2016a).
Revised: The combination of kernels and neural networks has previously been explored, most notably in the context of deep kernel learning (Wilson et al., 2016a).

Original: Using a a deep neural network architecture, (Wilson et al., 2016a) transform the input vectors into feature space based on which a base kernel is applied (here the RBF kernel and the spectral mixture base kernel (Wilson & Adams, 2013) are used).
Revised: Using a a deep neural network architecture, (Wilson et al., 2016a) transform the input vectors into feature space based on which a base kernel is applied (here the RBF kernel and the spectral mixture base kernel (Wilson & Adams, 2013) are used).

Original: The kernel parameters and the neural network weights are jointly learned by maximizing the marginal likelihood of the Gaussian process, but relying on a pre-training of the underlying deep neural network architecture.
Revised: The kernel parameters and the neural network weights are jointly learned by maximizing the marginal likelihood of the Gaussian process, but relying on a pre-training of the underlying deep neural network architecture.

Original: This work has been subsequently extended into a variational setting (Wilson et al., 2016b), also providing support for multi-class classification but again relying on a somewhat cumbersome pre-training.
Revised: This work has been subsequently extended into a variational setting (Wilson et al., 2016b), also providing support for multi-class classification but again relying on a somewhat cumbersome pre-training.

Original: A key difference between our approach and (Wilson et al., 2016a;b) is in our choice of inducing points.
Revised: A key difference between our approach and (Wilson et al., 2016a;b) is in our choice of inducing points.

Original: For instance, in (Wilson et al., 2016a;b), the inducing points are placed on a regular multidimensional lattice based on which the deep kernel is evaluated (van Amersfoort et al., 2022).
Revised: For instance, in (Wilson et al., 2016a;b), the inducing points are placed on a regular multidimensional lattice based on which the deep kernel is evaluated (van Amersfoort et al., 2022).

Original: In
Revised: In

Original: IGNs the inducing points are (only) defined in feature space, where they are treated as unconstrained parameters and learned jointly together with the neural network weights and (any) kernel parameters.
Revised: IGNs the inducing points are (only) defined in feature space, where they are treated as unconstrained parameters and learned jointly together with the neural network weights and (any) kernel parameters.

Original: The pseudo-label function of IGNs plays a role similar to that of inducing variables (Wilson et al., 2016b), but where each inducing variable is defined by its own set of variational parameters, the pseudo-label function offers an amortized approach by having the parameters of the pseudo-label function being shared among all inducing points.
Revised: The pseudo-label function of IGNs plays a role similar to that of inducing variables (Wilson et al., 2016b), but where each inducing variable is defined by its own set of variational parameters, the pseudo-label function offers an amortized approach by having the parameters of the pseudo-label function being shared among all inducing points.

Original: Furthermore, as shown in Section 4, our learning scheme is end-to-end and does not require any pre-training of networks as in (Wilson et al., 2016a).
Revised: Furthermore, as shown in Section 4, our learning scheme is end-to-end and does not require any pre-training of networks as in (Wilson et al., 2016a).

Original: Finally, IGNs rely on mini-batch maximum likelihood-based gradient descent (Chen et al., 2020), thus avoiding the GP-KISS kernel approximation of (Wilson et al., 2016a) and the grid layout of the inducing points together with the variational-based sampling setup of (Wilson et al., 2016b).
Revised: Finally, IGNs rely on mini-batch maximum likelihood-based gradient descent (Chen et al., 2020), thus avoiding the GP-KISS kernel approximation of (Wilson et al., 2016a) and the grid layout of the inducing points together with the variational-based sampling setup of (Wilson et al., 2016b).

Original: Several other related works exploit inducing points.
Revised: Several other related works exploit inducing points.

Original: For instance, (Titsias, 2009; Hensman et al., 2013; Damianou & Lawrence, 2013) propose to maximize a lower bound of the exact marginal likelihood to learn the inducing points in the input space, in contrast to our IGN framework where the inducing points are defined in a (transformed) feature space.
Revised: For instance, (Titsias, 2009; Hensman et al., 2013; Damianou & Lawrence, 2013) propose to maximize a lower bound of the exact marginal likelihood to learn the inducing points in the input space, in contrast to our IGN framework where the inducing points are defined in a (transformed) feature space.

Original: Using inducing point in a transformed is also explored in, e.g., (Lázaro-Gredilla & Figueiras-Vidal, 2009), but here the transformed space is fixed, e.g., in the form of the frequency domain of the input space.
Revised: Using inducing point in a transformed is also explored in, e.g., (Lázaro-Gredilla & Figueiras-Vidal, 2009), but here the transformed space is fixed, e.g., in the form of the frequency domain of the input space.

Original: In our IGN framework, we instead use a transformation that is jointly learned together with a (deep) kernel; moreover, our inducing points can (in principle) also live in a space different from the embedding space.
Revised: In our IGN framework, we instead use a transformation that is jointly learned together with a (deep) kernel; moreover, our inducing points can (in principle) also live in a space different from the embedding space.

Original: Closely related to IGNs is (Snelson & Ghahramani, 2006), where inducing points are jointly learned with kernel  parameters using gradient descent.
Revised: Closely related to IGNs is (Snelson & Ghahramani, 2006), where inducing points are jointly learned with kernel [Table] parameters using gradient descent.

Original: However, in contrast to IGNs, the continuous optimization of Z proposed by Snelson & Ghahramani (2006) is considerably more simplified as the inducing points are learned in the input space.
Revised: However, in contrast to IGNs, the continuous optimization of Z proposed by Snelson & Ghahramani (2006) is considerably more simplified as the inducing points are learned in the input space.

Original: Similarly to the proposed framework, Aitchison et al.
Revised: Similarly to the proposed framework, Aitchison et al.

Original: (2021) exploit inducing Gram matrix.
Revised: (2021) exploit inducing Gram matrix.

Original: The Gram matrix are used to sample inducing points in the feature space, and subsequent GP predictions are conditionally sampled on the inducing points in the features space.
Revised: The Gram matrix are used to sample inducing points in the feature space, and subsequent GP predictions are conditionally sampled on the inducing points in the features space.

Original: This process, in contrast to IGNs, relies on a complex doubly-stochastic variational inference process.
Revised: This process, in contrast to IGNs, relies on a complex doubly-stochastic variational inference process.

Original: The papers cited above are representatives of GP methods that are methodologically related to the proposed IGN framework.
Revised: The papers cited above are representatives of GP methods that are methodologically related to the proposed IGN framework.

Original: Not all of the cited methods are, however, in line with state of the art in terms of, e.g., accuracy results, hence the experimental section below also includes descriptions and comparisons of other GP methods, which forms the basis for the empirical evaluation and analysis.
Revised: Not all of the cited methods are, however, in line with state of the art in terms of, e.g., accuracy results, hence the experimental section below also includes descriptions and comparisons of other GP methods, which forms the basis for the empirical evaluation and analysis.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present and discuss experimental results showing the potential of the IGN framework on both regression and classification tasks.
Revised: In this section, we present and discuss experimental results showing the potential of the IGN framework on both regression and classification tasks.

Original: Both settings are aimed at showing the ability of IGNs to jointly learn the inducing points and the model parameters.
Revised: Both settings are aimed at showing the ability of IGNs to jointly learn the inducing points and the model parameters.

Original: In the experiments, we also compare IGNs against other state of the art methods that either learn the kernel parameters or the inducing points.
Revised: In the experiments, we also compare IGNs against other state of the art methods that either learn the kernel parameters or the inducing points.

Original: The variety of the experiments aim to provide insight into the properties of the framework and show the potential of IGNs for diverse real-world datasets, outperforming state-of-the-art methods in the majority of the cases.
Revised: The variety of the experiments aim to provide insight into the properties of the framework and show the potential of IGNs for diverse real-world datasets, outperforming state-of-the-art methods in the majority of the cases.

Original: 4. R EGRESSION TASKS
Revised: 4. R EGRESSION TASKS

Original: Following the setup described in (Chen et al., 2020), we compared our approach to deep kernel learning (DKL) (Wilson et al., 2016a), deep kernel processes (DIWP) (Aitchison et al., 2021), and sgGP (Chen et al., 2020) on several simulated and real regression benchmark datasets.
Revised: Following the setup described in (Chen et al., 2020), we compared our approach to deep kernel learning (DKL) (Wilson et al., 2016a), deep kernel processes (DIWP) (Aitchison et al., 2021), and sgGP (Chen et al., 2020) on several simulated and real regression benchmark datasets.

Original: Briefly, DKL exploits inducing points (in the input space) and an approximation of the kernel to scale to large[Table] datasets, DIWP defines deep kernel processes where positive definite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions.
Revised: Briefly, DKL exploits inducing points (in the embedding space) and an approximation of the kernel to scale to large datasets, DIWP defines deep kernel processes where positive definite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions.

Original: The sizes and feature dimensions for each dataset are reported in Appendix (see Table 3).
Revised: The sizes and feature dimensions for each dataset are reported in Appendix (see Table 3).

Original: The simulated datasets (Levy, Griewank and Borehole) are from the Virtual Library of Simulation
Revised: The simulated datasets (Levy, Griewank and Borehole) are from the Virtual Library of Simulation

Original: Experiments 1 , while the real datasets (Protein, PM2.5, Energy, Bike-hour, and Query) are from the
Revised: Experiments 2 , while the real datasets (Protein, PM2.5, Energy, Bike-hour, and Query) are from the

Original: UCI repository 2 .
Revised: UCI repository 3 .

Original: For each dataset, we repeated the experiments 10 times.
Revised: For each dataset, we repeated the experiments 10 times.

Original: In each experiment, the dataset was randomly split into a 60% training set and a 40% test set.
Revised: In each experiment, the dataset was randomly split into a 60% training set and a 40% test set.

Original: Furthermore, the training set was normalized to have 0 mean and 1 standard deviation, and the test set was scaled accordingly.
Revised: Furthermore, the training set was normalized to have 0 mean and 1 standard deviation, and the test set was scaled accordingly.

Original: For all the datasets, we considered a simple neural network with 3 stacked dense layers with 128 units and ReLU activations.
Revised: For all the datasets, we considered a simple neural network with 3 stacked dense layers with 128 units and ReLU activations.

Original: On top of those we added a final feature layer with 64 units.
Revised: On top of those we added a final feature layer with 64 units.

Original: We used 512 inducing points 3 having the same dimension as the feature layer (i.e., 64 dimensions somewhat arbitrarily chosen), an RBF kernel with γ = 1 . 0 for all dimensions, and a linear pseudo-label function.
Revised: We used 512 inducing points 4 having the same dimension as the feature layer (i.e., 64 dimensions somewhat arbitrarily chosen), an RBF kernel with γ = 1 . 0 for all dimensions, and a linear pseudo-label function.

Original: The model weights are updated with the Adamoptimizer for 500 epochs onmini-batches of size 128.
Revised: The model weights are updated with the Adam for 500 epochs on of size 128.

Original: Table 1 summarizes the results, where for each dataset we report the average and standard deviation of the root mean squared error and negative log-likelihood on the test set over the 10 experimental repetitions.
Revised: Table 1 summarizes the results, where for each dataset we report the average and standard deviation of the root mean squared error and negative log-likelihood on the test set over the 10 experimental repetitions.

Original: For Levy, Borehole, and Bike-hour datasets, our approach outperforms by a large margin the other methods.
Revised: For Levy, Borehole, and Bike-hour datasets, our approach outperforms by a large margin the other methods.

Original: While for PM2.5, Energy, and Query datasets our approach performed slightly worse than the others, we still obtained comparable results.
Revised: While for PM2.5, Energy, and Query datasets our approach performed slightly worse than the others, we still obtained comparable results.

Original: Overall, our method remained stable as the standard deviations over 10 experiments are small and comparable to sgGP.
Revised: Overall, our method remained stable as the standard deviations over 10 experiments are small and comparable to sgGP.

Original: Additional results using other straw-men methods can be found in Appendix B.
Revised: Additional results using other baseline methods can be found in Appendix C.

Original: Finally, we have also explored the effect of imposing a grid structure over the inducing points wrt.
Revised: Finally, for illustrating the effect of imposing a grid structure over the inducing points wrt.

Original: computation time and accuracy.
Revised: computation time and accuracy, we have made a comparison of IGN and DKL (Wilson et al., 2016a). Recall that DKL requires a n -dimensional grid whose cost grows exponentially with the amount of data. In this experiments, we use the exact same architecture in terms of layers and kernel as described in Section 4.1 for both IGN and DKL. Figure 2 shows the root mean square error for Levy and

Original: The results of these experiments can be found in Appendix C.
Revised: Griewank datasets. Our method, although requiring more inducing points, outperforms DKL in terms of computational time and root mean square errors (evaluated on the test sets). It was not possible to conduct experiments for DKL with more than 8 inducing points as we hit the memory limit.

Original: 4. T OY -MNIST
Revised: 4. T OY -MNIST

Original: Toy-MNIST is the same as MNIST, but limited to digits belonging to class 5 or 6.
Revised: Toy-MNIST is the same as MNIST, but limited to digits belonging to class 5 or 6.

Original: The aim of this experiments is to show the ability of IGNs to classify relatively ambiguous digits, and provide insights into the inducing points and the uncertainty associated with the predictions.
Revised: The aim of this experiments is to show the ability of IGNs to classify relatively ambiguous digits, and provide insights into the inducing points and the uncertainty associated with the predictions.

Original: In this case, the training and test sets consist of 11,339 and 1,850 images, respectively.
Revised: In this case, the training and test sets consist of 11,339 and 1,850 images, respectively.

Original: The image labels associated with classes 5 and 6 are replaced with 1 and -1, respectively, and we assume the exact same model specification as in Section 4.1, but now only using 64 inducing points.
Revised: The image labels associated with classes 5 and 6 are replaced with 1 and -1, respectively, and we assume the exact same model specification as in Section 4.1, but now only using 64 inducing points.

Original: By repeating the experiment 10 times we obtain a test set accuracy of 99 .
Revised: By repeating the experiment 10 times we obtain a test set accuracy of 99 .

Original: 25 ± 0 .
Revised: 25 ± 0 .

Original: For each of the 64 inducing points, we retrieve the closest image in the training set based on the chosen kernel in feature space, after which we refine each of the images (relative to the distance to the closest inducing point) using gradient descent with respect to the input image.
Revised: For each of the 64 inducing points, we retrieve the closest image in the training set based on the chosen kernel in feature space, after which we refine each of the images (relative to the distance to the closest inducing point) using gradient descent with respect to the input image.

Original: The 64 inducing points (in the feature space) and digit images corresponding to the inducing points are shown in Figure 2.
Revised: The 64 inducing points (in the feature space) and digit images corresponding to the inducing points are shown in Figure 3.

Original: For this experiment, the intuitive interpretation of the images associated with the inducing point also provides a degree of explainabilty of the model structure, even though the inducing points are learned in feature space.
Revised: For this experiment, the intuitive interpretation of the images associated with the inducing point also provides a degree of explainabilty, even though the inducing points are learned in feature space.

Original: As with standard Gaussian processes, IGNs retains the feature of uncertainty estimation.
Revised: As with standard Gaussian processes, IGNs retains the feature of uncertainty estimation.

Original: This is illustrated in Figure 3, which depicts a scatter plot of the images in the test set, given in terms of the expected value of the predictions (x-axis) and the variance of the predictions (y-axis).
Revised: This is illustrated in Figure 4, which depicts a scatter plot of the images in the test set, given in terms of the expected value of the predictions (x-axis) and the variance of the predictions (y-axis).

Original: The two distinct orange and blue clusters contain the digits corresponding to classes 6 and 5.
Revised: The two distinct orange and blue clusters contain the digits corresponding to classes 6 and 5.

Original: The figure also illustrates some of the extreme predictions: the two digits with means far from 0.5 and low variance correspond to clearly distinguishable digits, while the digit with a mean close to 0.5 and relatively high variance corresponds to a more ambiguous image.
Revised: The figure also illustrates some of the extreme predictions: the two digits with means far from 0.5 and low variance correspond to clearly distinguishable digits, while the digit with a mean close to 0.5 and relatively high variance corresponds to a more ambiguous image.

Original: 4. MNIST AND CIFAR 
Revised: 4. MNIST AND CIFAR

Original: In this section we evaluate IGN on computer vision classification tasks.
Revised: In this section we evaluate IGN on computer vision classification tasks.

Original: Specifically, here we focus on
Revised: Specifically, here we focus on

Original: MNIST and CIFAR10.
Revised: MNIST and CIFAR10.

Original: We compare our approach against deep kernel processes (DIWP) (Aitchison et al., 2021) as well as Deep Gaussian Processes (DGP) (Damianou & Lawrence, 2013), NNGP (Blundell et al., 2015), and stochastic variational deep kernel learning (SVI-DKL)
Revised: We compare our approach against deep kernel processes (DIWP) (Aitchison et al., 2021) as well as Deep Gaussian Processes (DGP) (Damianou & Lawrence, 2013), NNGP (Blundell et al., 2015), and stochastic variational deep kernel learning (SVI-DKL)

Original: (Wilson et al., 2016b).
Revised: (Wilson et al., 2016b).

Original: Briefly, DGP uses graphical models to nest layers of Gaussian processes, and NNGP adopts a backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network to estimate the uncertainty of the model.
Revised: Briefly, DGP uses graphical models to nest layers of Gaussian processes, and NNGP adopts a backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network to estimate the uncertainty of the model.

Original: For the sake of simplicity (see Section 2.1), we decompose with a one-vs-all approach each multiclass task into 10 binary tasks.
Revised: For the sake of simplicity , we decompose with a one-vs-all approach each multiclass task intobinary tasks.

Original: We again used the same model as in Section 4.1 (this model is comparable to the other competitor models in terms of number of parameters), but with 32 and 16 inducing points for MNIST and CIFAR10, respectively (the same number of inducing points was used by all methods).
Revised: We again used the same model as in Section 4.1 (this model is comparable to the other competitor models in terms of number of parameters), but with 32 and 16 inducing points for MNIST and CIFAR10, respectively (the same number of inducing points was used by all methods).

Original: For each of the datasets, we repeated the experiments 10 times and report the average accuracy and negative log-likelihood in Table 2.
Revised: For each of the datasets, we repeated the experiments 10 times and report the average accuracy and negative log-likelihood in Table 2.

Original: We observe that our method outperforms all the others in terms of accuracy, while keeping the standard deviation comparably small.
Revised: We observe that our method outperforms all the others in terms of accuracy, while keeping the standard deviation comparably small.

Original: The negative log-likelihood is better for CIFAR-10, but worse for MNIST, suggesting that training a full multi-classification model could be beneficial, see (Murphy, 2022).
Revised: The negative log-likelihood is better for CIFAR-10, but worse for MNIST, suggesting that training a full multi-classification model could be beneficial, see (Murphy, 2022).

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  S TRUCTURED DATASETS
Revised:  S TRUCTURED DATASETS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we investigate the potential of our model applied to datasets with more complex structure.
Revised: we investigate the potential of our model applied to datasets with more complex structure.

Original: Specifically
Revised: Specifically

Original: [Table]
Revised: [Table]

Original: we consider two classification tasks on text and graphs, respectively.
Revised: we consider two classification tasks on text and graphs, respectively.

Original: The datasets for both tasks are derived from IMDB:
Revised: The datasets for both tasks are derived from IMDB:

Original: • IMDB-T EXT (Maas et al., 2011) consists of 25,000 training reviews and 25,000 test reviews.
Revised: • IMDB-T EXT (Maas et al., 2011) consists of 25,000 training reviews and 25,000 test reviews.

Original: Positive and negative labels are balanced within the training and test sets.
Revised: Positive and negative labels are balanced within the training and test sets.

Original: We preprocess the
Revised: We preprocess the

Original: data
Revised: data

Original: by keeping the top 20,000 words and limiting the length of
Revised: by keeping the top 20,000 words and limiting the length of

Original: words
Revised: words

Original: in
Revised: in

Original: a review to
Revised: a review to

Original: • IMDB-G RAPH is a social network dataset contained in the collection described in (Yanardag & Vishwanathan, 2015).
Revised: • IMDB-G RAPH is a social network dataset contained in the collection described in (Yanardag & Vishwanathan, 2015).

Original: It consists of constructed genre-specific collaboration networks where nodes represent actresses/actors who are connected by an edge if they have appeared together in a movie of a given genre.
Revised: It consists of constructed genre-specific collaboration networks where nodes represent actresses/actors who are connected by an edge if they have appeared together in a movie of a given genre.

Original: Collaboration networks are generated for the genres Action and Romance for this dataset.
Revised: Collaboration networks are generated for the genres Action and Romance for this dataset.

Original: The data then consists of the ego-graphs for all actresses/actors in all genre networks, and the task is to identify the genre from which
Revised: The data then consists of the ego-graphs for all actresses/actors in all genre networks, and the task is to identify the genre from which

Original: an ego-graph has been extracted.
Revised: an ego-graph has been extracted.

Original: It contains 1, graphs with labels balanced among the two classes.
Revised: It contains 1, graphs with labels balanced among the two classes.

Original: For both experiments, we aim to show that our approach can jointly handle complex data and complex models for extracting features, hence the aim is not to compare against state-of-the-art methods.
Revised: For both experiments, we aim to show that our approach can jointly handle complex data and complex models for extracting features, hence the aim is not to compare against state-of-the-art methods.

Original: In particular, the two IGN instantiations for the two datsets differ only in the embedding functions/feature extractors being used.
Revised: In particular, the two IGN instantiations for the two datsets differ only in the embedding functions/feature extractors being used.

Original: For both datasets, we also provide baseline accuracies obtained by two classifiers sharing the architectures of the feature extractors in the IGNs.
Revised: For both datasets, we also provide baseline accuracies obtained by two classifiers sharing the architectures of the feature extractors in the IGNs.

Original: For IMDB-T EXT , we used 128 inducing points and trained a transformer-based model (Vaswani et al., 2017) to learn features from the reviews.
Revised: For IMDB-T EXT , we used 128 inducing points and trained a transformer-based model (Vaswani et al., 2017) to learn features from the reviews.

Original: We repeated the experiments 10 times for the baseline and our approach obtaining accuracies of 84 .
Revised: We repeated the experiments 10 times for the baseline and our approach obtaining accuracies of 84 .

Original: 35% ± 0 .
Revised: 35% ± 0 .

Original: 10 and 84 .
Revised: 10 and 84 .

Original: 43% ± 0 .
Revised: 43% ± 0 .

Original: 96 , respectively.
Revised: 96 , respectively.

Original: Similarly to Section 4.2, we can also inspect the learned inducing points.
Revised: Similarly to Section 4.2, we can also inspect the learned inducing points.

Original: For example, included below are the two training set examples (positive and negative) that are closets to two randomly chosen learned inducing points.
Revised: For example, included below are the two training set examples (positive and negative) that are closets to two randomly chosen learned inducing points.

Original: • I was literally preparing to hate this movie so believe me when I say this film is worth seeing ...
Revised: • I was literally preparing to hate this movie so believe me when I say this film is worth seeing ...

Original: my score 7 out of 10.
Revised: my score 7 out of 10.

Original: • I can’t believe that so many are comparing this movie to Argento’s ...
Revised: • I can’t believe that so many are comparing this movie to Argento’s ...

Original: If you’re looking for a good horror movie look elsewhere.
Revised: If you’re looking for a good horror movie look elsewhere.

Original: Similar to Section 4.2 we also here see that by associating the learned inducing points with training instances provides an additional element of explainability to(parts of) the learned model.
Revised: Similar to Section 4.2 we also here see that by associating the learned inducing points with training instances provides an additional element of explainability to the inducing points.

Original: For the IMDB-G RAPH , we follow the setup described by Tibo et al.
Revised: For the IMDB-G RAPH , we follow the setup described by Tibo et al.

Original: and represent each graph as a set of neighborhoods.
Revised: and represent each graph as a set of neighborhoods.

Original: We performed a 10 times 10 fold cross-validation for both the baseline and IGN.
Revised: We performed a 10 times 10 fold cross-validation for both the baseline and IGN.

Original: We ran 100 epochs of the Adam optimizer with learning rate 0.001 on mini-batches of size 32.
Revised: We ran 100 epochs of the Adam optimizer with learning rate 0.001 on mini-batches of size 32.

Original: We obtained accuracies equal to 72 .
Revised: We obtained accuracies equal to 72 .

Original: 49% ± 0 .
Revised: 49% ± 0 .

Original: 60 and 73 .
Revised: 60 and 73 .

Original: 40% ± 0 .
Revised: 40% ± 0 .

Original: 63 , for the baseline and IGN, respectively.
Revised: 63 , for the baseline and IGN, respectively.

Original: Without any particular effort, in terms of fine tuning model structure and parameters, our method outperforms most of the graph neural networks reported in (Nguyen 
Revised: Without any particular effort, in terms of fine tuning model structure and parameters, our method outperforms most of the graph neural networks reported in (Nguyen et al.,

Original:  A BLATION ON NUMBER OF INDUCING POINTS
Revised:  A BLATION ON NUMBER OF INDUCING POINTS

Original: Here we study the impact of the number of inducing points for the regression and classification tasks defined in the previous sections.
Revised: Here we study the impact of the number of inducing points for the regression and classification tasks defined in the previous sections.

Original: The variance of Gaussian processes estimations decrease with the number of training points (Rasmussen & Williams, 2006).
Revised: The variance of Gaussian processes estimations decrease with the number of training points (Rasmussen & Williams, 2006).

Original: In our case, the training points are replaced by inducing points but we retain similar result.
Revised: In our case, the training points are replaced by inducing points but we retain similar result.

Original: For the sake of completeness, we slightly reformulate the proposition and provide a proof in the supplementary material.
Revised: For the sake of completeness, we slightly reformulate the proposition and provide a proof in the supplementary material.

Original: Proposition 2.
Revised: Proposition 2.

Original: The variance of a test point ( x ∗ , f ∗ ) given a set of inducing points Z can never increase by the inclusion of an additional inducing point z ̸∈ Z .
Revised: The variance of a test point ( x ∗ , f ∗ ) given a set of inducing points Z can never increase by the inclusion of an additional inducing point z ̸∈ Z .

Original: When training IGNs with different number of inducing points, there are no guarantees that the same subset of inducing points will be learned during training.
Revised: When training IGNs with different number of inducing points, there are no guarantees that the same subset of inducing points will be learned during training.

Original: However, we still observe a consistent decrease in variance when the number of inducing points increases.
Revised: However, we still observe a consistent decrease in variance when the number of inducing points increases.

Original: For the regression tasks, we report here the results for all the real datasets described in Section 4.1.
Revised: For the regression tasks, we report here the results for all the real datasets described in Section 4.1.

Original: For the classification task we focus on IMDB-T EXT described in Section 4.4.
Revised: For the classification task we focus on IMDB-T EXT described in Section 4.4.

Original: In all the cases, we repeated the experiments 10 times varying the number of inducing points (4,8,16,32,64,128,256,512).
Revised: In all the cases, we repeated the experiments 10 times varying the number of inducing points (4,8,16,32,64,128,256,512).

Original: Figure 4 depicts the root mean square error (on the top), and the average variance with error bars (on the bottom) for the different datasets.
Revised: Figure 5 depicts the root mean square error (on the top), and the average variance with error bars (on the bottom) for the different datasets.

Original: From the figure we see that the average variance consistently decreases with the number of inducing points for all datasets.
Revised: From the figure we see that the average variance consistently decreases with the number of inducing points for all datasets.

Original: For some of the datasets with only a few inducing points, the error bars are lowe, but the root mean square error is always higher.
Revised: For some of the datasets with only a few inducing points, the error bars are lowe, but the root mean square error is always higher.

Original: Finally, it is worth noticing that even when the root mean square error stabilizes, the variance keeps decreasing with the number of inducing points.
Revised: Finally, it is worth noticing that even when the root mean square error stabilizes, the variance keeps decreasing with the number of inducing points.

Original:  C ONCLUSIONS AND FUTURE WORK
Revised:  C ONCLUSIONS AND FUTURE WORK

Original: While Gaussian processes are powerful machine learning methods that can estimate uncertainty, they remain intractable on large datasets.
Revised: While Gaussian processes are powerful machine learning methods that can estimate uncertainty, they remain intractable on large datasets.

Original: Several works have addressed this problem using inducing points but, to the best of our knowledge, these methods remain limited to comparatively simple datasets.
Revised: Several works have addressed this problem using inducing points but, to the best of our knowledge, these methods remain limited to comparatively simple datasets.

Original: In this paper, we introduce a framework for learning Gaussian processes for large and complex datasets by combining inducing points in feature space with neural network kernel approximations.
Revised: In this paper, we introduce a framework for learning Gaussian processes for large and complex datasets by combining inducing points in feature space with neural network kernel approximations.

Original: We empirically showed the ability of our method on standard machine learning benchmarks as well as on structured (graph and text) datasets, with the proposed method outperforming other state-of-the-art approaches.
Revised: We empirically showed the ability of our method on standard machine learning benchmarks as well as on structured (graph and text) datasets, with the proposed method outperforming other state-of-the-art approaches.

Original: The flexibility of our IGN framework enables the combination of Gaussian process training with complex deep learning architectures.
Revised: The flexibility of our IGN framework enables the combination of Gaussian process training with complex deep learning architectures.

Original: We believe that our method provides a basis for further research in calibration of uncertainty estimates in deep neural network models.
Revised: We believe that our method provides a basis for further research in calibration of uncertainty estimates in deep neural network models.

Original: Furthermore, as part of future work, we also plan to investigate how to position IGN within a fully variational setting (Titsias, 2009), exploring the hierarchical structure of the model, possibly in the context of deep kernel processes (Aitchison et al., 2021).
Revised: Furthermore, as part of future work, we also plan to investigate how to position IGN within a fully variational setting (Titsias, 2009), exploring the hierarchical structure of the model, possibly in the context of deep kernel processes (Aitchison et al., 2021).

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: The work reported in this paper is not related to human subjects, practices to data set releases, discrimination/bias/fairness concerns and research integrity issues.
Revised: The work reported in this paper is not related to human subjects, practices to data set releases, discrimination/bias/fairness concerns and research integrity issues.

Original: The work is also not related to any particular application domain.
Revised: The work is also not related to any particular application domain.

Original: The paper proposes a simple but general framework for learning inducing points in sparse Gaussian processes, also contributing with insights/explainability into the model (through the learned inducing points) and potentially improved accuracy and uncertainty estimates.
Revised: The paper proposes a simple but general framework for learning inducing points in sparse Gaussian processes, also contributing with insights/explainability into the model (through the learned inducing points) and potentially improved accuracy and uncertainty estimates.

Original: Depending on the application domain, the significance and interpretation of these insights and estimates may require careful scrutiny by domain experts, in particular for safety critical applications.
Revised: Depending on the application domain, the significance and interpretation of these insights and estimates may require careful scrutiny by domain experts, in particular for safety critical applications

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: An open-source implementation of our IGN framework has been uploaded as supplementary material together with the paper.
Revised: An open-source implementation of our IGN framework has been uploaded as supplementary material together with the paper.

Original: Upon acceptance of the paper, the code will be placed in a public GitHub repository.
Revised: Upon acceptance of the paper, the code will be placed in a public GitHub repository.

Original: The implementation also contains detailed information about the data sets, parameters, and random seeds used in the experiments.
Revised: The implementation also contains detailed information about the data sets, parameters, and random seeds used in the experiments.

Original: A A BOUT THE VARIANCE OF INDUCING POINTS
Revised: A A BOUT THE VARIANCE OF INDUCING POINTS

Original: Proposition 2.
Revised: Proposition 2.

Original: The variance of a test point ( x ∗ , f ∗ ) given a set of inducing points Z can never increase by the inclusion of an additional inducing point z ̸∈ Z .
Revised: The variance of a test point ( x ∗ , f ∗ ) given a set of inducing points Z can never increase by the inclusion of an additional inducing point z ̸∈ Z .

Original: Proof.
Revised: Proof.

Original: Let S and R be two sets containing inducing points.
Revised: Let S and R be two sets containing inducing points.

Original: Let K f ∗ , s | r be the variance associated with f ∗ conditioned only on the inducing point in R .
Revised: Let K f ∗ , s | r be the variance associated with f ∗ conditioned only on the inducing point in R .

Original: Let K f ∗ | s , r be the variance associated with f ∗ conditioned only on the inducing point in R ∪ S .
Revised: Let K f ∗ | s , r be the variance associated with f ∗ conditioned only on the inducing point in R ∪ S .

Original: If K ′ X ∗ X ∗ is the top-left scalar in K f ∗ , s | r representing the variance of f ∗ , then we prove that K f ∗ | s , r ≤ K ′ X ∗ X ∗ .
Revised: If K ′ X ∗ X ∗ is the top-left scalar in K f ∗ , s | r representing the variance of f ∗ , then we prove that K f ∗ | s , r ≤ K ′ X ∗ X ∗ .

Original: Let K be the covariance matrix of the Gaussian process having the following block structure:
Revised: Let K be the covariance matrix of the Gaussian process having the following block structure:

Original: [Equation]
Revised: [Equation]

Original: where the subscripts represents the two sets (among the observation { x ∗ } and inducing points in R and S ) used to evaluate the kernel matrix.
Revised: where the subscripts represents the two sets (among the observation { x ∗ } and inducing points in R and S ) used to evaluate the kernel matrix.

Original: We compute now K f ∗ , s | r as
Revised: We compute now K f ∗ , s | r as

Original: [Equation]
Revised: [Equation]

Original: Note that K ′ X ∗ X ∗ ≤ K X ∗ X ∗ .
Revised: Note that K ′ X ∗ X ∗ ≤ K X ∗ X ∗ .

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: which concludes the proof.

Original: A. IGN S FOR CLASSIFICATION
Revised: B IGN S FOR CLASSIFICATION

Original: Here we provide additional details about the full derivation (see Section 2.1 in the main paper) of
Revised: Here we provide additional details about the full derivation (see Section 2.1 in the main paper) of

Original: IGNs for classification.
Revised: IGNs for classification.

Original: For ease of exposition, we only consider binary classification problems, but the general method can straightforwardly be extended to a multi-valued setting.
Revised: For ease of exposition, we only consider binary classification problems, but the general method can straightforwardly be extended to a multi-valued setting.

Original: For the experiment results, (see Section 4.3 in the main paper) we therefore use a one-vs-all approach to solve multiclass classification tasks.
Revised: For the experiment results, (see Section 4.3 in the main paper) we therefore use a one-vs-all approach to solve multiclass classification tasks.

Original: We assume that the label space is given by Y = { 0 , 1 } and that we have access to a data set D
Revised: We assume that the label space is given by Y = { 0 , 1 } and that we have access to a data set D

Original: = { ( x i , y i ) } ni
Revised: = { ( x i , y i ) } ni

Original: =1 , where the underlying data generating process is defined by a latent function f ( x ) with a GP prior f ∼ GP (0 , k ( · , · )) and y | x ∼ Φ( f ( x )) , where Φ( · ) is the cumulative Gaussian function; see (Rasmussen & Williams, 2006).
Revised: =1 , where the underlying data generating process is defined by a latent function f ( x ) with a GP prior f ∼ GP (0 , k ( · , · )) and y | x ∼ Φ( f ( x )) , where Φ( · ) is the cumulative Gaussian function; see (Rasmussen & Williams, 2006).

Original: For inference, we calculate the posterior distribution p ( y | X ∗ , θ ) given by
Revised: For inference, we calculate the posterior distribution p ( y | X ∗ , θ ) given by

Original: [Equation]
Revised: [Equation]

Original: Assuming that f ∗ ∼ N ( µ, σ 2 ) for some µ and σ we get (Rasmussen & Williams, 2006, Section 3.9):
Revised: Assuming that f ∗ ∼ N ( µ, σ 2 ) for some µ and σ we get (Rasmussen & Williams, 2006, Section 3.9):

Original: [Equation]
Revised: [Equation]

Original: where µ and σ 2 can be found from the predictive distribution in Equation 8 (in the main paper).
Revised: where µ and σ 2 can be found from the predictive distribution in Equation 8 (in the main paper).

Original: For learning the IGN parameters θ we perform maximum likelihood estimation, but using the Laplace approximation to derive an approximation of the marginal likelihood.
Revised: For learning the IGN parameters θ we perform maximum likelihood estimation, but using the Laplace approximation to derive an approximation of the marginal likelihood.

Original: Firstly, the likelihood is defined as
Revised: Firstly, the likelihood is defined as

Original: [Equation]
Revised: [Equation]

Original: Denoting by
Revised: Denoting by

Original: [Equation]
Revised: [Equation]

Original: and employing the Laplace approximation in Equation 11, our goal is to maximize the following log-likelihood approximation
Revised: and employing the Laplace approximation in Equation 11, our goal is to maximize the following log-likelihood approximation

Original: [Equation]
Revised: [Equation]

Original: where c represent the accumulated constant terms and
Revised: where c represent the accumulated constant terms and

Original: [Equation]
Revised: [Equation]

Original: The form of p ( ˆ f | X, θ ) is given in Equation 3 (in the main paper) and using the shorthand notation
Revised: The form of p ( ˆ f | X, θ ) is given in Equation 3 (in the main paper) and using the shorthand notation

Original: [Equation]
Revised: [Equation]

Original: the log-likelihood in Equation 12 can be expressed as
Revised: the log-likelihood in Equation 12 can be expressed as

Original: [Equation]
Revised: [Equation]

Original: The term p ( y | f ) in Equation 13 factorizes, hence W = ∇∇ log p ( y | f ) is diagonal (Rasmussen &
Revised: The term p ( y | f ) in Equation 13 factorizes, hence W = ∇∇ log p ( y | f ) is diagonal (Rasmussen &

Original: Williams, 2006, Page 43) with
Revised: Williams, 2006, Page 43) with

Original: [Equation]
Revised: [Equation]

Original: From Equation 14, ∇∇ log p ( f | X, θ ) =
Revised: From Equation 14, ∇∇ log p ( f | X, θ ) =

Original: K − 1 , therefore A = − W − K − 1 .
Revised: K − 1 , therefore A = − W − K − 1 .

Original: We finally approximating the log-likelihood (omitting constant terms) as
Revised: We finally approximating the log-likelihood (omitting constant terms) as

Original: [Equation]
Revised: [Equation]

Original: Lastly, we find ˆ f using Newton’s approximation for T steps.
Revised: Lastly, we find ˆ f using Newton’s approximation for T steps.

Original: Starting with ˆ f 0 = 0 , for 1 ≤ t ≤ T , we have
Revised: Starting with ˆ f 0 = 0 , for 1 ≤ t ≤ T , we have

Original: [Equation]
Revised: [Equation]

Original: 1 ( W ˆ f t + ∇ log p ( y | ˆ f t )) , where ∇ log p ( y i | f i ) =
Revised: 1 ( W ˆ f t + ∇ log p ( y | ˆ f t )) , where ∇ log p ( y i | f i ) =

Original: y i N ( f i ) /
Revised: y i N ( f i ) /

Original: Φ( y i f i ) .
Revised: Φ( y i f i ) .

Original: B A DDITIONAL EXPERIMENTS ON REGRESSION TASKS
Revised: C A DDITIONAL EXPERIMENTS ON REGRESSION TASKS

Original: Following the setup described in (Chen et al., 2020), we compared our approach to PCG-based exact
Revised: Following the setup described in (Chen et al., 2020), we compared our approach to PCG-based exact

Original: GP (EGP) (Wang et al., 2019), sparse GP regression (SGPR) (Titsias, 2009), stochastic variational
Revised: GP (EGP) (Wang et al., 2019), sparse GP regression (SGPR) (Titsias, 2009), stochastic variational

Original: GP (SVGP) (Hensman et al., 2013), and sgGP (Chen et al., 2020) on several simulated and real regression benchmark datasets.
Revised: GP (SVGP) (Hensman et al., 2013), and sgGP (Chen et al., 2020) on several simulated and real regression benchmark datasets.

Original: Briefly, EGP leverages GPU parallelization and conjugate gradients to compute the exact covariance matrix on large datasets, SGPR uses a variational formulation for sparse approximations that jointly infers the inducing inputs and the kernel hyper-parameters by maximizing a lower bound of the true log marginal likelihood, SVGP variationally decomposes
Revised: Briefly, EGP leverages GPU parallelization and conjugate gradients to compute the exact covariance matrix on large datasets, SGPR uses a variational formulation for sparse approximations that jointly infers the inducing inputs and the kernel hyper-parameters by maximizing a lower bound of the true log marginal likelihood, SVGP variationally decomposes

Original: Gaussian processes to depend on a set of globally relevant inducing variables, and sgGP exploits stochastic gradient descent on the Gaussian process likelihood to learn the kernel hyper-parameters.
Revised: Gaussian processes to depend on a set of globally relevant inducing variables, and sgGP exploits stochastic gradient descent on the Gaussian process likelihood to learn the kernel hyper-parameters.

Original: C A DDITIONAL EXPERIMENTS ON COMPUTATION TIME AND ACCURACY

Original: For illustrating the effect of imposing a grid structure over the inducing points wrt.

Original: computation time and accuracy, we have made a comparison of IGN and DKL (Wilson et al., 2016a).

Original: Recall that DKL requires a n -dimensional grid whose cost grows exponentially with the amount of data.

Original: In this experiments, we use the exact same architecture in terms of layers and kernel as described in Section 4.1 for both IGN and DKL.

Original: Figure 5 shows the root mean square error for Levy and

Original: Griewank datasets.

Original: Our method, although requiring more inducing points, outperforms DKL in terms of computational time and root mean square errors (evaluated on the test sets).

Original: It was not possible to conduct experiments for DKL with more than 8 inducing points as we hit the memory limit.

Original: [Table]

Original: and Griewank datasets in function of the grid size (for DKL) and number of inducing points (for IGN).

Original: The colors encode the time (in seconds) that each epoch takes to be completed.

Original: DKL RMSE improves with the grid size but the computational time drastically increases.

Original: On the other hand, IGN requires more inducing points but the results are overall better and the computational time remains stable.

Original: All the models were run on

Original: CPU for having a fair comparison.

Original: D IMDB-T EXT I NDUCING P OINTS
Revised: D IMDB-T EXT I NDUCING P OINTS

Original: We report the training set reviews close to the inducing points for the IMBD-Text experiment presented in Section 4.4 for one of the ten runs.
Revised: We report the training set reviews close to the inducing points for the IMBD-Text experiment presented in Section 4.4 for one of the ten runs.

Original: Recall that we use 128 inducing points.
Revised: Recall that we use 128 inducing points.

Original: In this case, we obtained 60 unique inducing points corresponding to 32 positive and 28 negative training set reviews.
Revised: In this case, we obtained 60 unique inducing points corresponding to 32 positive and 28 negative training set reviews.

Original: The inducing points are all different to each other but there are collisions when choosing the closests points in the training set.
Revised: The inducing points are all different to each other but there are collisions when choosing the closests points in the training set.

Original: Note that the inducing points are balanced among the two classes.
Revised: Note that the inducing points are balanced among the two classes.

Original: For the sake of space, for each review we report the initial and last 25 words.
Revised: For the sake of space, for each review we report the initial and last 25 words.

Revised: [Table]

Original: positive - there have been several films about zorro
Revised: positive - there have been several films about zorro

Original: some even made in europe e g alain delon this role has also been played by outstanding actors ... good performance of hadley as zorro he was quick smart used well his whip and sword and his voice was the best for any zorro 2.
Revised: some even made in europe e g alain delon this role has also been played by outstanding actors ... good performance of hadley as zorro he was quick smart used well his whip and sword and his voice was the best for any zorro 2.

Original: negative - i love the frequently masters of horror series horror fans live in a constant lack of projects like this and the similar project with gave ... up has to have a payoff that exceeds build up not the other way around storytelling math 101 br br end of spoilers big oops 3.
Revised: negative - i love the frequently masters of horror series horror fans live in a constant lack of projects like this and the similar project with gave ... up has to have a payoff that exceeds build up not the other way around storytelling math 101 br br end of spoilers big oops 3.

Original: positive - i rented the film i don’t think it got a theatrical release here out expecting the worse the previews made the film look awful
Revised: positive - i rented the film i don’t think it got a theatrical release here out expecting the worse the previews made the film look awful

Original: i ... be proclaimed ’the worst film ever i recommend this film for anybody interested in the show a flawed but innovative and interesting piece of film 4.
Revised: i ... be proclaimed ’the worst film ever i recommend this film for anybody interested in the show a flawed but innovative and interesting piece of film 4.

Original: positive - one of the best of the fred astaire and rogers films great music by irving berlin solid support from randolph scott harriet nelson lucille ball ...
Revised: positive - one of the best of the fred astaire and rogers films great music by irving berlin solid support from randolph scott harriet nelson lucille ball ...

Original: jazzy and it’s a great song br br fun all the way although i got tired of we joined the navy after the third time 5.
Revised: jazzy and it’s a great song br br fun all the way although i got tired of we joined the navy after the third time 5.

Original: positive - forbidden planet rates as landmark in science fiction carefully staying within hard aspects of the genre science not fantasy nerds will love it while still ...
Revised: positive - forbidden planet rates as landmark in science fiction carefully staying within hard aspects of the genre science not fantasy nerds will love it while still ...

Original: the edge destroying its creator just as it did thousands of centuries earlier to the krell br br maybe the krell had teenage daughters too 6.
Revised: the edge destroying its creator just as it did thousands of centuries earlier to the krell br br maybe the krell had teenage daughters too 6.

Original: negative - the bad news is it’s still really dreadful i gave it a 2 because occasionally some of this slapstick parody actually seems funny br br ... this one and get this dvd back to the video store on time you’ll really hate yourself if you have to pay a late fee 7.
Revised: negative - the bad news is it’s still really dreadful i gave it a 2 because occasionally some of this slapstick parody actually seems funny br br ... this one and get this dvd back to the video store on time you’ll really hate yourself if you have to pay a late fee 7.

Original: positive - nice character development in a pretty cool milieu being a male i’m probably not qualified to totally understand it
Revised: positive - nice character development in a pretty cool milieu being a male i’m probably not qualified to totally understand it

Original: but they do a nice job ...
Revised: but they do a nice job ...

Original: but within this world it needed to happen good acting all around with something positive taking place in the lives of some pretty good people 8.
Revised: but within this world it needed to happen good acting all around with something positive taking place in the lives of some pretty good people 8.

Original: negative - mercifully there’s no video of this wannabe western that a stay afloat vehicle for big frank at a time when his career was floundering the ... you up late and having a bout of insomnia but if you can sit through it you’ve more than most of my movie buff friends 9.
Revised: negative - mercifully there’s no video of this wannabe western that a stay afloat vehicle for big frank at a time when his career was floundering the ... you up late and having a bout of insomnia but if you can sit through it you’ve more than most of my movie buff friends 9.

Original: negative - bela lugosi is an evil who sends brides poisoned on their wedding day steals the body in his fake ambulance hearse and takes it home ... in a discount store 2 for £1 which i think is a pretty accurate anyone paying more for this would be out of their mind 10.
Revised: negative - bela lugosi is an evil who sends brides poisoned on their wedding day steals the body in his fake ambulance hearse and takes it home ... in a discount store 2 for £1 which i think is a pretty accurate anyone paying more for this would be out of their mind 10.

Original: positive - its no surprise that busey later developed a in his this film is also a poor decision but one i enjoyed fully the first 5 ... wet myself some of best work by far rent or buy it today my vote is a perfect 10 on the poo meter that is 11.
Revised: positive - its no surprise that busey later developed a in his this film is also a poor decision but one i enjoyed fully the first 5 ... wet myself some of best work by far rent or buy it today my vote is a perfect 10 on the poo meter that is 11.

Original: positive - before sky i saw diane tender performance in this otherwise of a movie campers are invited to the camp of their youth and experience it ... comic acting turn by noted director sam raimi makes this a movie you can pull out again and again like looking up an old friend 12.
Revised: positive - before sky i saw diane tender performance in this otherwise of a movie campers are invited to the camp of their youth and experience it ... comic acting turn by noted director sam raimi makes this a movie you can pull out again and again like looking up an old friend 12.

Original: positive - the line of course is from the lord’s prayer thy will be done on earth as it is in heaven sweden especially its far north ...
Revised: positive - the line of course is from the lord’s prayer thy will be done on earth as it is in heaven sweden especially its far north ...

Original: the ending is what you make of it i guess
Revised: the ending is what you make of it i guess

Original: but it’s not spoiling it to say daniel achieves what he set out to do13.
Revised: but it’s not spoiling it to say daniel achieves what he set out to do

Original: positive - my certainly is a fair looking woman this film is a lost gem a dead on satire mockumentary of the early 90’s hip hop scene ...
Revised: positive - my certainly is a fair looking woman this film is a lost gem a dead on satire mockumentary of the early 90’s hip hop scene ...

Original: this regard i regard this movie like the 1000 islands of upstate new york it’s a wonderful little secret you want to keep to yourself 14.
Revised: this regard i regard this movie like the 1000 islands of upstate new york it’s a wonderful little secret you want to keep to yourself 14.

Original: positive - i rented this film from netflix for two reasons i was in the mood for what i thought would be a silly ’50s sci fi ... also generally very good and the acting is much better than one might expect i was particularly impressed with reeves jeff corey and walter reed 
Revised: positive - i rented this film from netflix for two reasons i was in the mood for what i thought would be a silly ’50s sci fi ... also generally very good and the acting is much better than one might expect i was particularly impressed with reeves jeff corey and walter reed 15.

Original: negative - brilliant book with wonderful characterizations and insights into human nature particularly the nature of addiction which still resonate strongly today br br as for the ... normally excellent but inappropriately cast actors all in all a weak adaptation your three hours would be better spent reading or re reading the book 16.
Revised: negative - brilliant book with wonderful characterizations and insights into human nature particularly the nature of addiction which still resonate strongly today br br as for the ... normally excellent but inappropriately cast actors all in all a weak adaptation your three hours would be better spent reading or re reading the book 16.

Original: negative - it’s boggles the mind how this movie was nominated for seven oscars and won one not because it’s abysmal or because given the collective credentials ... director hungry to be recognized it could’ve been morphed to something better but what’s left looks like a film nobody was really interested in making 17.
Revised: negative - it’s boggles the mind how this movie was nominated for seven oscars and won one not because it’s abysmal or because given the collective credentials ... director hungry to be recognized it could’ve been morphed to something better but what’s left looks like a film nobody was really interested in making 17.

Original: positive - a comedy of funny proportions from the guys that brought you south park and most of the guys from this movie has utterly disgusting and ... turn the sport sour and its up ta coop ta fix it and along the way you will laugh alot that’s all there is enjoy 18.
Revised: positive - a comedy of funny proportions from the guys that brought you south park and most of the guys from this movie has utterly disgusting and ... turn the sport sour and its up ta coop ta fix it and along the way you will laugh alot that’s all there is enjoy 18.

Original: negative - there was a bugs bunny cartoon titled baby buggy bunny
Revised: negative - there was a bugs bunny cartoon titled baby buggy bunny

Original: that was exactly this plot baby faced robbed a bank and the money in the ... the bugs bunny dvd it’s was much more original the first time 1954 plus you’ll get a lot more classic bugs bunny cartoons to boot 19.
Revised: that was exactly this plot baby faced robbed a bank and the money in the ... the bugs bunny dvd it’s was much more original the first time 1954 plus you’ll get a lot more classic bugs bunny cartoons to boot 19.

Original: negative - first off i really loved henry fool which puts me in a very small pool of movie goers parker posey is one of best actresses ... ride i’d be happy to spoil this movie for you
Revised: negative - first off i really loved henry fool which puts me in a very small pool of movie goers parker posey is one of best actresses ... ride i’d be happy to spoil this movie for you

Original: but it’s been done it’s rotten the fool franchise is dead long live henry fool 20.
Revised: but it’s been done it’s rotten the fool franchise is dead long live henry fool 20.

Original: positive - i imagine victorian literature slowly sinking into the of the increasingly distant past pulled down by the weight of its under skirts along comes television ... coarse have been made to modern tastes and without having felt preached to another bbc classic highly recommended this is howromantic literature should be 21.
Revised: positive - i imagine victorian literature slowly sinking into the of the increasingly distant past pulled down by the weight of its under skirts along comes television ... coarse have been made to modern tastes and without having felt preached to another bbc classic highly recommended this is how

Revised: romantic

Revised: literature should be

Original: negative - i knew this movie wasn’t going to be amazing
Revised: negative - i knew this movie wasn’t going to be amazing

Original: but i thought i would give it a chance i am a fan of luke wilson ...
Revised: but i thought i would give it a chance i am a fan of luke wilson ...

Original: the movie without people getting annoyed the movie had its moments but i’m glad i didn’t spend 9 50 to see it in the theater 22.
Revised: the movie without people getting annoyed the movie had its moments but i’m glad i didn’t spend 9 50 to see it in the theater 22.

Original: negative - i have read several good reviews that have defended and the various aspects of this film one thing i see over and over is annoyance ... of good and terrible acting i would recommend it for a cheap thrill but hardly a diamond in the rough that is micro budget horror 23.
Revised: negative - i have read several good reviews that have defended and the various aspects of this film one thing i see over and over is annoyance ... of good and terrible acting i would recommend it for a cheap thrill but hardly a diamond in the rough that is micro budget horror 23.

Original: negative - this movie is like the thousand cat and mouse movies that preceded it the following may look like a spoiler but it really just describes ... exiting the theater from a hollywood movie and if you have ever felt that way too heed my warning stay miles away from this movie 24.
Revised: negative - this movie is like the thousand cat and mouse movies that preceded it the following may look like a spoiler but it really just describes ... exiting the theater from a hollywood movie and if you have ever felt that way too heed my warning stay miles away from this movie 24.

Original: negative - i wonder who how and more importantly why the decision to call richard attenborough to direct the most singular sensation to hit broadway in many ... well michael douglas was in it true i forgot i’m absolutely wrong and you are absolutely right nothing like a richard attenborough michael douglas
Revised: negative - i wonder who how and more importantly why the decision to call richard attenborough to direct the most singular sensation to hit broadway in many ... well michael douglas was in it true i forgot i’m absolutely wrong and you are absolutely right nothing like a richard attenborough michael douglas

Original: musical 25.
Revised: musical 25.

Original: positive - this movie will go down down in history as one of the greats right along side of citizen kane casablanca and on the waterfront someone ... do yourself and your family a favor and buy it immediately i’m still holding out hope for a special edition dvd one of these days 26.
Revised: positive - this movie will go down down in history as one of the greats right along side of citizen kane casablanca and on the waterfront someone ... do yourself and your family a favor and buy it immediately i’m still holding out hope for a special edition dvd one of these days 26.

Original: negative - this is high grade cheese fare of b movie kung fu flicks bruce wannabe lee is played by bruce li
Revised: negative - this is high grade cheese fare of b movie kung fu flicks bruce wannabe lee is played by bruce li

Original: i think of course let’s ... flashback for a scene just shown 3 minutes ago they must’ve thought that only one with attention disorder could fully understand this film br br 27.
Revised: i think of course let’s ... flashback for a scene just shown 3 minutes ago they must’ve thought that only one with attention disorder could fully understand this film br br 27.

Original: negative - the only previous gordon film i had watched was the kiddie adventure the magic sword 1962 though i followed this soon after with empire of ... them then again this particular version is further sunk by the tacked on electronic score – which is wholly inappropriate and cheesy in the extreme28.
Revised: negative - the only previous gordon film i had watched was the kiddie adventure the magic sword 1962 though i followed this soon after with empire of ... them then again this particular version is further sunk by the tacked on electronic score – which is wholly inappropriate and cheesy in the extreme

Original: positive - hilarious evocative confusing brilliant film reminds me of or holy mountain lots of strange characters about and looking for what is it i laughed almost ... watch on screen or at his big slide show smart funny quirky and outrageously hot make more films write more books keep the nightmare alive 29.
Revised: positive - hilarious evocative confusing brilliant film reminds me of or holy mountain lots of strange characters about and looking for what is it i laughed almost ... watch on screen or at his big slide show smart funny quirky and outrageously hot make more films write more books keep the nightmare alive 29.

Original: positive - the villian in this movie is one mean sob and he seems to enjoy what he is doing that is what i guess makes him ... guess you can make up your own mind about the true ending
Revised: positive - the villian in this movie is one mean sob and he seems to enjoy what he is doing that is what i guess makes him ... guess you can make up your own mind about the true ending

Original: i’m left feeling that only one character should have survived at the end 
Revised: i’m left feeling that only one character should have survived at the end 30.

Original: negative - okay what the hell kind of trash have i been watching now the mountain has got to be one of the most incoherent and insane ...
Revised: negative - okay what the hell kind of trash have i been watching now the mountain has got to be one of the most incoherent and insane ...

Original: good heroine this is the type of european horror film that could have been legendary if only someone had bothered to write a structured screenplay 31.
Revised: good heroine this is the type of european horror film that could have been legendary if only someone had bothered to write a structured screenplay 31.

Original: positive - i found this movie to be very good in all areas the acting was brilliant from all characters especially ms stone and character just gets ... audience which was misled by some faulty terrible reviews about the movie before it even started you won’t regret it if you go see it 32.
Revised: positive - i found this movie to be very good in all areas the acting was brilliant from all characters especially ms stone and character just gets ... audience which was misled by some faulty terrible reviews about the movie before it even started you won’t regret it if you go see it 32.

Original: negative - a lot of horror fans seem to love scarecrows so i won’t be very popular in saying that i found it to be rather boring ... involving killer scarecrows to my knowledge apart from dark night of the scarecrow which is much better i would recommend that over scarecrows any day 33.
Revised: negative - a lot of horror fans seem to love scarecrows so i won’t be very popular in saying that i found it to be rather boring ... involving killer scarecrows to my knowledge apart from dark night of the scarecrow which is much better i would recommend that over scarecrows any day 33.

Original: negative - david mamet’s film debut has been hailed by many as a real thinking man’s movie a movie that makes you question everybody and everything i ... unfulfilled and if you like me predicted ahead of time that margaret was going to be conned you will find this revelation just as unsatisfying 34.
Revised: negative - david mamet’s film debut has been hailed by many as a real thinking man’s movie a movie that makes you question everybody and everything i ... unfulfilled and if you like me predicted ahead of time that margaret was going to be conned you will find this revelation just as unsatisfying 34.

Original: positive - while this was a better movie than 101 dalmations live action not animated version
Revised: positive - while this was a better movie than 101 dalmations live action not animated version

Original: i think it still fell a little short of what disney ...
Revised: i think it still fell a little short of what disney ...

Original: as so many disney films are here’s to hoping the third will be even better still because you know they probably want to make one 35.
Revised: as so many disney films are here’s to hoping the third will be even better still because you know they probably want to make one 35.

Original: positive - i think i read this someplace joe johnston director of the film and also one of the guys who founded industrial light and magic for ... first homer jr did not like the idea
Revised: positive - i think i read this someplace joe johnston director of the film and also one of the guys who founded industrial light and magic for ... first homer jr did not like the idea

Original: but he warmed up to it after the movie poster paperback novel came out and took off 36.
Revised: but he warmed up to it after the movie poster paperback novel came out and took off 36.

Original: positive - stephen king movies are a funny thing with me i either really love them or i loathe them some of the productions such as desperation ... very watchable and enjoyable adaption br br for uk readers this production has most recently been shown on sci fi and sky thriller horror channels 37.
Revised: positive - stephen king movies are a funny thing with me i either really love them or i loathe them some of the productions such as desperation ... very watchable and enjoyable adaption br br for uk readers this production has most recently been shown on sci fi and sky thriller horror channels 37.

Original: positive - if you’re researching ufo facts then this video is very important the of the video is the comments made by buzz he is without a ... in details should not detour your from viewing this video if nothing else it is interesting
Revised: positive - if you’re researching ufo facts then this video is very important the of the video is the comments made by buzz he is without a ... in details should not detour your from viewing this video if nothing else it is interesting

Original: and i recommend you watch with an open mind 38.
Revised: and i recommend you watch with an open mind 38.

Original: positive - rupert friend gives a performance as prince albert that lifts the young victoria to unexpected levels he is superb as we know queen victoria fell ... believe for a minute she was victoria no real sense of period it may no have been her fault but her prince deserved the crown 39.
Revised: positive - rupert friend gives a performance as prince albert that lifts the young victoria to unexpected levels he is superb as we know queen victoria fell ... believe for a minute she was victoria no real sense of period it may no have been her fault but her prince deserved the crown 39.

Original: positive - seeing moonstruck after so many years is a reminder of how sweet and funny this film was when it first appeared who knew that cher ... used to be at its best entertainment with no social significance whatsoever if they’d only lost that’s along the way it would have been perfect 40.
Revised: positive - seeing moonstruck after so many years is a reminder of how sweet and funny this film was when it first appeared who knew that cher ... used to be at its best entertainment with no social significance whatsoever if they’d only lost that’s along the way it would have been perfect 40.

Original: negative - i first learned of the wendigo many years ago in one of alvin scary stories books according to that story the wendigo after calling your ... to count br br anyway avoid it patricia clarkson and erik per sullivan dewey on malcolm in the middle have done far better than this 41.
Revised: negative - i first learned of the wendigo many years ago in one of alvin scary stories books according to that story the wendigo after calling your ... to count br br anyway avoid it patricia clarkson and erik per sullivan dewey on malcolm in the middle have done far better than this 41.

Original: positive - well maybe not immediately before the rodney king riots but even a few months before was timely enough my parents said that they saw it ...
Revised: positive - well maybe not immediately before the rodney king riots but even a few months before was timely enough my parents said that they saw it ...

Original: but either way grand canyon is a great movie it kevin kline as my favorite actor also starring mary mary louise parker and alfre woodard 42.
Revised: but either way grand canyon is a great movie it kevin kline as my favorite actor also starring mary mary louise parker and positive - finally

Original: positive - finally a movie where the audience is kept guessing until the end what will happen well we all kind of know that the lives of ... his drug and sex addictions and a father who finally discovers exactly what happened the day of the robbery this movie will get you thinking43.
Revised: a movie where the audience is kept guessing until the end what will happen well we all kind of know that the lives of ... his drug and sex addictions and a father who finally discovers exactly what happened the day of the robbery this movie will get you thinking

Original: negative - wow i don’t even really remember that much about this movie except that it stunk br br the plot’s basically a girl’s parents neglect her ...
Revised: negative - wow i don’t even really remember that much about this movie except that it stunk br br the plot’s basically a girl’s parents neglect her ...

Original: you do see it don’t expect much 1 out of 10 br br seriously if you want a pokemon movie rent pokemon the first movie 
Revised: you do see it don’t expect much 1 out of 10 br br seriously if you want a pokemon movie rent pokemon the first movie 44.

Original: positive - the group of people are travelling to in an awful bus led by a drunk conductor and his dumb son who likes to drive with ... bigger than him in the end the movie takes one turn and the trip becomes nothing but a swan s song of a dying country 45.
Revised: positive - the group of people are travelling to in an awful bus led by a drunk conductor and his dumb son who likes to drive with ... bigger than him in the end the movie takes one turn and the trip becomes nothing but a swan s song of a dying country 45.

Original: negative - yes this movie is a real thief it stole some shiny oscars from just because politicians wanted another war hero movie to boost the acceptance ... if we consider this title a reasonable piece of the u s wars are cool genre you surely have much better movies to choose from 46.
Revised: negative - yes this movie is a real thief it stole some shiny oscars from just because politicians wanted another war hero movie to boost the acceptance ... if we consider this title a reasonable piece of the u s wars are cool genre you surely have much better movies to choose from 46.

Original: positive - i was going through a list of oscar winners and was surprised to see that this film beat butch cassidy and the sundance kid for ... by hoffman to take this role otherwise he may have been typecast after the graduate anyway this considered an all time great for a reason 47.
Revised: positive - i was going through a list of oscar winners and was surprised to see that this film beat butch cassidy and the sundance kid for ... by hoffman to take this role otherwise he may have been typecast after the graduate anyway this considered an all time great for a reason 47.

Original: positive - inspirational tales about triumph of the human spirit are usually big turn offs for me the most surprising thing about men of honor is how ... doesn’t disappoint he creates a darkly funny portrait director george jr set out to make an old style flick and comes up with a winner 48.
Revised: positive - inspirational tales about triumph of the human spirit are usually big turn offs for me the most surprising thing about men of honor is how ... doesn’t disappoint he creates a darkly funny portrait director george jr set out to make an old style flick and comes up with a winner 48.

Original: negative - this movie was god awful from conception to execution the us needs to set up a star wars site in this remote country this is ...
Revised: negative - this movie was god awful from conception to execution the us needs to set up a star wars site in this remote country this is ...

Original: gymnast star in real life i would probably kick him in the face after a double with 2 1 2 twists in the layout position 49.
Revised: gymnast star in real life i would probably kick him in the face after a double with 2 1 2 twists in the layout position 49.

Original: positive - i wouldn’t call we’re back a story simply a kiddie version of jurassic park i found it more interesting than that like the former it ... kind i would actually say that john goodman doing voice here is sort of a precursor to his voice work in monsters inc worth seeing 50.
Revised: positive - i wouldn’t call we’re back a story simply a kiddie version of jurassic park i found it more interesting than that like the former it ... kind i would actually say that john goodman doing voice here is sort of a precursor to his voice work in monsters inc worth seeing 50.

Original: negative - chinese ghost story iii is a totally superfluous sequel to two excellent fantasy films the film delivers the spell casting special effects that one can ...
Revised: negative - chinese ghost story iii is a totally superfluous sequel to two excellent fantasy films the film delivers the spell casting special effects that one can ...

Original: a little extra money out of a successful formula they won’t be able to do it again the cash cow is now dead as a 51.
Revised: a little extra money out of a successful formula they won’t be able to do it again the cash cow is now dead as a 51.

Original: negative - the direction had clearly stated that this film’s idea and plot is totally original however as to those who have read comic we can clearly ... watching this thus making this movie getting what it shouldn’t have it has became one of the best budget films in china for this year 52.
Revised: negative - the direction had clearly stated that this film’s idea and plot is totally original however as to those who have read comic we can clearly ... watching this thus making this movie getting what it shouldn’t have it has became one of the best budget films in china for this year 52.

Original: negative - let me start by saying that i understand that invasion of the star creatures was meant to be a parody of the sci fi films ...
Revised: negative - let me start by saying that i understand that invasion of the star creatures was meant to be a parody of the sci fi films ...

Original: the double feature dvd with invasion of the bee girls that movie is academy award winning stuff in comparison with invasion of the star creatures 53.
Revised: the double feature dvd with invasion of the bee girls that movie is academy award winning stuff in comparison with invasion of the star creatures 53.

Original: positive - knowing when to end a movie is just as important as casting directing and acting and it’s nice to see when a director script get ... a mansion br br this is a great independent production and one that wastes little time getting going and it won’t waste your time either 54.
Revised: positive - knowing when to end a movie is just as important as casting directing and acting and it’s nice to see when a director script get ... a mansion br br this is a great independent production and one that wastes little time getting going and it won’t waste your time either 54.

Original: negative - italian born has inherited from her deceased lover karl an ultra modern and isolated house in the middle of the woods
Revised: negative - italian born has inherited from her deceased lover karl an ultra modern and isolated house in the middle of the woods

Original: it’s winter
Revised: it’s winter

Original: and she ... around for it
Revised: and she ... around for it

Original: i think i’ll give it the benefit of the doubt as it’s definitely not what i was expecting from this indie film 55.
Revised: i think i’ll give it the benefit of the doubt as it’s definitely not what i was expecting from this indie film 55.

Original: negative - is a horror comedy that doesn’t really have enough horror or comedy to qualify as one or the other it has one scene that is ... the movie that is weaker in general plot and spine because of production values that just shows you how uninteresting i found the look of 56.
Revised: negative - is a horror comedy that doesn’t really have enough horror or comedy to qualify as one or the other it has one scene that is ... the movie that is weaker in general plot and spine because of production values that just shows you how uninteresting i found the look of 56.

Original: positive - there were a lot of 50’s sci fi movies they were big draws for the drive in theaters a lot of them were crappy even ... worried about the invisible monster forbidden planet is a movie a sci fi fan can watch several times and find something new with each viewing 57.
Revised: positive - there were a lot of 50’s sci fi movies they were big draws for the drive in theaters a lot of them were crappy even ... worried about the invisible monster forbidden planet is a movie a sci fi fan can watch several times and find something new with each viewing 57.

Original: negative - curiously it is rene eyes and mouth not buddy the that emerge as the focal point of buddy a jim henson pictures production through francis ...
Revised: negative - curiously it is rene eyes and mouth not buddy the that emerge as the focal point of buddy a jim henson pictures production through francis ...

Original: thompson needs a good pick up shot she gives rene another extreme close up i wonder what the lipstick budget was on this picture from58.
Revised: thompson needs a good pick up shot she gives rene another extreme close up i wonder what the lipstick budget was on this picture from

Original: positive - now this is what i’d call a good horror with occult supernatural undertones this nice low budget french movie caught my attention from the very ... very confusing towards the end but redeems itself by the time it’s over br br
Revised: positive - now this is what i’d call a good horror with occult supernatural undertones this nice low budget french movie caught my attention from the very ... very confusing towards the end but redeems itself by the time it’s over br br

Original: i thought his was a very good movie59.
Revised: i thought his was a very good movie

Original: negative - and ethel buffs too will love her loud vocals as the wicked witch but this cartoon sequel to the wizard of oz is bereft of ... sure baby boomers will get a charge from it since it has been out of for so long as a curiosity item just fair from [Table] [Table] ants lacking suspense
Revised: negative - and ethel buffs too will love her loud vocals as the wicked witch but this cartoon sequel to the wizard of oz is bereft of ... sure baby boomers will get a charge from it since it has been out of for so long as a curiosity item just fair from 60.

Original: [Table] creepiness [Table]
Revised: negative - this is another one of those vs insects features a theme that was popular in the late 70’s only you can’t really call it horror ...

Original:  be [Table] ants [Table]
Revised: after having seen ants lacking suspense action thrills shocks and creepiness the only thing you’ll be left with after seeing ants is an annoying itch