Original: D EEP KERNEL PROCESSES
Revised: D EEP KERNEL PROCESSES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We deﬁne deep kernel processes in which positive deﬁnite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions.
Revised: We deﬁne deep kernel processes in which positive deﬁnite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions.

Original: Remarkably, we ﬁnd that deep Gaussian processes (DGPs), Bayesian neural networks (BNNs), inﬁnite BNNs, and inﬁnite BNNs with bottlenecks can all be written as deep kernel processes.
Revised: Remarkably, we ﬁnd that deep Gaussian processes (DGPs), Bayesian neural networks (BNNs), inﬁnite BNNs, and inﬁnite BNNs with bottlenecks can all be written as deep kernel processes.

Original: For DGPs the equivalence arises because the Gram matrix formed by the inner product of features is Wishart distributed, and as we show, standard isotropic kernels can be written entirely in terms of this Gram matrix — we do not need knowledge of the underlying features.
Revised: For DGPs the equivalence arises because the Gram matrix formed by the inner product of features is Wishart distributed, and as we show, standard isotropic kernels can be written entirely in terms of this Gram matrix — we do not need knowledge of the underlying features.

Original: We deﬁne a tractable deep kernel process, the deep inverse Wishart process, and give a doubly-stochastic inducing-point variational inference scheme that operates on the Gram matrices, not on the features, as in DGPs.
Revised: We deﬁne a tractable deep kernel process, the deep inverse Wishart process, and give a doubly-stochastic inducing-point variational inference scheme that operates on the Gram matrices, not on the features, as in DGPs.

Original: We show that the deep inverse Wishart process gives superior performance to DGPs and inﬁnite
Revised: We show that the deep inverse Wishart process gives superior performance to DGPs and inﬁnite

Original: BNNs on standard fully-connected baselines.
Revised: BNNs on standard fully-connected baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The deep learning revolution has shown us that effective performance on difﬁcult tasks such as image classiﬁcation (Krizhevsky et al., 2012) requires deep models with ﬂexible lower-layers that learn task-dependent representations.
Revised: The deep learning revolution has shown us that effective performance on difﬁcult tasks such as image classiﬁcation (Krizhevsky et al., 2012) requires deep models with ﬂexible lower-layers that learn task-dependent representations.

Original: Here, we consider whether these insights from the neural network literature can be applied to purely kernel-based methods.
Revised: Here, we consider whether these insights from the neural network literature can be applied to purely kernel-based methods.

Original: (Note that we do not consider deep
Revised: (Note that we do not consider deep

Original: Gaussian processes or DGPs to be “fully kernel-based” as they use a feature-based representation in intermediate layers).
Revised: Gaussian processes or DGPs to be “fully kernel-based” as they use a feature-based representation in intermediate layers).

Original: Importantly, deep kernel methods (e.g. Cho & Saul, 2009) already exist.
Revised: Importantly, deep kernel methods (e.g. Cho & Saul, 2009) already exist.

Original: In these methods, which are closely related to inﬁnite Bayesian neural networks (Lee et al., 2017; Matthews et al., 2018;
Revised: In these methods, which are closely related to inﬁnite Bayesian neural networks (Lee et al., 2017; Matthews et al., 2018;

Original: Garriga-Alonso et al., 2018; Novak et al., 2018), we take an initial kernel (usually the dot product of the input features) and perform a series of deterministic, parameter-free transformations to obtain an output kernel that we use in e.g. a support vector machine or Gaussian process.
Revised: Garriga-Alonso et al., 2018; Novak et al., 2018), we take an initial kernel (usually the dot product of the input features) and perform a series of deterministic, parameter-free transformations to obtain an output kernel that we use in e.g. a support vector machine or Gaussian process.

Original: However, the deterministic, parameter-free nature of the transformation from input to output kernel means that they lack the capability to learn an effective task-dependent representation (Aitchison, 2019).
Revised: However, the deterministic, parameter-free nature of the transformation from input to output kernel means that they lack the capability to learn a top-layer representation, which is believed to be crucial for the effectiveness of deep methods (Aitchison, 2019).

Original: To obtain the ﬂexibility necessary to learn a task-dependent representation, we propose deep kernel processes (DKPs), which combine nonlinear transformations of the kernel, as in Cho & Saul (2009) with a ﬂexible learned representation by exploiting a Wishart or inverse Wishart process (Dawid, 1981; Shah et al., 2014).
Revised: To obtain the ﬂexibility necessary to learn a task-dependent representation, we propose deep kernel processes (DKPs), which combine nonlinear transformations of the kernel, as in Cho & Saul (2009) with a ﬂexible learned representation by exploiting a Wishart or inverse Wishart process (Dawid, 1981; Shah et al., 2014).

Original: We ﬁnd that models ranging from DGPs (Damianou & Lawrence, 2013; Salimbeni & Deisenroth, 2017) to Bayesian neural networks (BNNs; Blundell et al., 2015,
Revised: We ﬁnd that models ranging from DGPs (Damianou & Lawrence, 2013; Salimbeni & Deisenroth, 2017) to Bayesian neural networks (BNNs; Blundell et al., 2015,

Original: App. C.1), inﬁnite BNNs (App. C.2) and inﬁnite BNNs with bottlenecks (App.
Revised: App. C.1), inﬁnite BNNs (App. C.2) and inﬁnite BNNs with bottlenecks (App.

Original: C.3) can be written as DKPs (i.e. only with kernel/Gram matrices, without needing features or weights).
Revised: C.3) can be written as DKPs (i.e. only with kernel/Gram matrices, without needing features or weights).

Original: Practically, we ﬁnd that the deep inverse Wishart process (DIWP), admits convenient forms for variational approximate posteriors, and we give a novel scheme for doubly-stochastic variational inference (DSVI) with inducing points purely in the kernel domain (as opposed to Salimbeni & Deisenroth, 2017, who described DSVI for standard feature-based DGPs), and demonstrate improved performance with carefully matched models on fully-connected benchmark datasets.
Revised: Practically, we ﬁnd that the deep inverse Wishart process (DIWP), admits convenient forms for variational approximate posteriors, and we give a novel scheme for doubly-stochastic variational inference (DSVI) with inducing points purely in the kernel domain (as opposed to Salimbeni & Deisenroth, 2017, who described DSVI for standard feature-based DGPs), and demonstrate improved performance with carefully matched models on fully-connected benchmark datasets.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We brieﬂy revise Wishart and inverse Wishart distributions.
Revised: We brieﬂy revise Wishart and inverse Wishart distributions.

Original: The Wishart distribution is a generalization of the gamma distribution that is deﬁned over positive semideﬁnite matrices.
Revised: The Wishart distribution is a generalization of the gamma distribution that is deﬁned over positive semideﬁnite matrices.

Original: Suppose that we have a collection of P -dimensional random variables x i with i ∈ { 1 , . . .

Original: , N } such that[Equation] then, [Equation]
Revised: Suppose that

Original: X
Revised: X

Original: K
Revised: K

Original: F
Revised: F

Original: G
Revised: G

Original: K
Revised: K

Original: F
Revised: F

Original: G
Revised: G

Original: K
Revised: K

Original: F
Revised: F

Original: Y
Revised: Y

Original: X
Revised: X

Original: G G
Revised: G G

Original: F
Revised: F

Original: Y
Revised: Y

Original: Layer Layer
Revised: Layer Layer

Revised: Output Layer we have a collection of P -dimensional random variables x i with i ∈ { 1 , . .

Revised: . , N }

Revised: such that

Original: Output Layer has Wishart distribution with scale matrix V and N degrees of freedom.
Revised: [Equation] then, [Equation] has Wishart distribution with scale matrix V and N degrees of freedom.

Original: When N > p − 1 , the density is,
Revised: When N > P − 1 , the density is,

Original: [Equation]
Revised: [Equation]

Original: where Γ p is the multivariate gamma function.
Revised: where Γ P is the multivariate gamma function.

Original: Further, the inverse, Σ − 1 has inverse Wishart distribution, W −(cid:0)
Revised: Further, the inverse, S − 1 has inverse Wishart distribution, W −(cid:0)

Original: [Equation] (cid:1) .
Revised: V − 1 , N (cid:1) .

Original: The inverse Wishart is deﬁned only for N > P − 1 and also has closed-form density.
Revised: The inverse Wishart is deﬁned only for N > P − 1 and also has closed-form density.

Original: Finally, we note that the Wishart distribution has mean N V while the inverse Wishart has mean V − 1 / ( N − P − 1) (for N > P + 1 ).
Revised: Finally, we note that the Wishart distribution has mean N V while the inverse Wishart has mean V − 1 / ( N − P − 1) (for N > P + 1 ).

Original:  D EEP KERNEL PROCESSES
Revised:  D EEP KERNEL PROCESSES

Original: We deﬁne a kernel process to be a set of distributions over positive deﬁnite matrices of different sizes, that are consistent under marginalisation (Dawid, 1981; Shah et al., 2014).
Revised: We deﬁne a kernel process to be a set of distributions over positive deﬁnite matrices of different sizes, that are consistent under marginalisation (Dawid, 1981; Shah et al., 2014).

Original: The two most common kernel processes are the Wishart process and inverse Wishart process, which we write in a slightly unusual form to ensure their expectation is K ,
Revised: The two most common kernel processes are the Wishart process and inverse Wishart process, which we write in a slightly unusual form to ensure their expectation is K .

Revised: We take G and G (cid:48) to be ﬁnite dimensional marginals of the underlying Wishart and inverse Wishart process,

Original: [Equation]
Revised: [Equation]

Original: and where we explicitly give the consistent marginal distributions over K ∗ , G ∗ and G (cid:48)∗ which are
Revised: and where we explicitly give the consistent marginal distributions over K ∗ , G ∗ and G (cid:48)∗ which are

Original: P ∗ × P ∗ principal submatrices of the P × P matrices K , G and G (cid:48) dropping the same rows and columns.
Revised: P ∗ × P ∗ principal submatrices of the P × P matrices K , G and G (cid:48) dropping the same rows and columns.

Original: In the inverse-Wishart distribution, δ is a positive parameter that can be understood as controlling the degree of variability, with larger values for δ implying smaller variability in G (cid:48) .
Revised: In the inverse-Wishart distribution, δ is a positive parameter that can be understood as controlling the degree of variability, with larger values for δ implying smaller variability in G (cid:48) .

Original: We deﬁne a deep kernel process by analogy with a DGP, as a composition of kernel processes, and show in App.
Revised: We deﬁne a deep kernel process by analogy with a DGP, as a composition of kernel processes, and show in App.

Original: A that under sensible assumptions any such composition is itself a kernel process.
Revised: A that under sensible assumptions any such composition is itself a kernel process.

Original: 3. DGP S WITH ISOTROPIC KERNELS ARE DEEP W ISHART PROCESSES
Revised: 3. DGP S WITH ISOTROPIC KERNELS ARE DEEP W ISHART PROCESSES

Original: We consider deep GPs of the form (Fig. 1 top) 
Revised: We consider deep GPs of the form (Fig. 1 top) with X ∈ R P × N K

Original: [Equation]
Revised: [Equation]

Original: where λ indexes hidden features so f (cid:96)λ is a single column of F (cid:96) .
Revised: for (cid:96) = 1 , K ( G [Equation] [Equation]

Revised: otherwise

Revised: [Equation]

Revised: (4a)

Revised: [Equation]

Revised: (4b)

Revised: [Equation]

Original:  Note, we have restricted ourselves to kernel functions, K , that can be written as functions of the Gram matrix, G (cid:96) , and do not require the full set of activations, F (cid:96) .
Revised: (4c) Here, F (cid:96) ∈ R P × N (cid:96) are the N (cid:96) hidden features in layer (cid:96) ; λ indexes hidden features so f (cid:96) λ is a single column of F (cid:96) , representing the value of the λ th feature for all training inputs. Note that K ( · ) is a function that takes a Gram matrix and returns a kernel matrix; whereas K (cid:96) is a (possibly random) variable representing a kernel matrix. Note, we have restricted ourselves to kernels, that can be written as functions of the Gram matrix, G (cid:96) , and do not require the full set of activations, F (cid:96) .

Original: As we describe later, this is not too restrictive, as it includes amongst others all isotropic kernels (i.e. those that can be written as a function of the distance between points Williams & Rasmussen, 2006).
Revised: As we describe later, this is not too restrictive, as it includes amongst others all isotropic kernels (i.e. those that can be written as a function of the distance between points Williams & Rasmussen, 2006).

Original: Note that we have a number of choices as to how to initialize the kernel in Eq.
Revised: Note that we have a number of choices as to how to initialize the kernel in Eq.

Original: (4a).
Revised: (4a).

Original: The current choice just uses a linear dot-product kernel, rather than immediately applying the kernel function K .
Revised: The current choice just uses a linear dot-product kernel, rather than immediately applying the kernel function K .

Original: This is both to ensure exact equivalence with inﬁnite NNs with bottlenecks (App. C.3) and also to highlight an interesting interpretation of this layer as Bayesian inference over generalised lengthscale hyperparameters in the squared-exponential kernel (App.
Revised: This is both to ensure exact equivalence with inﬁnite NNs with bottlenecks (App. C.3) and also to highlight an interesting interpretation of this layer as Bayesian inference over generalised lengthscale hyperparameters in the squared-exponential kernel (App.

Original: B e.g. Lalchand & Rasmussen, 2020).
Revised: B e.g. Lalchand & Rasmussen, 2020).

Original: For DGP regression, the outputs, Y , are most commonly given by a likelihood that can be written in terms of the output features, F L +1 .
Revised: For DGP regression, the outputs, Y , are most commonly given by a likelihood that can be written in terms of the output features, F L +1 .

Original: For instance, for regression, we might have,
Revised: For instance, for regression, the distribution of the λ th output feature column could be

Original: [Equation]
Revised: [Equation]

Original: but our methods can be used with many other forms for the likelihood, including e.g. classiﬁcation.
Revised: but our methods can be used with many other forms for the likelihood, including e.g. classiﬁcation.

Original: The generative process for the Gram matrices, G (cid:96) , consists of generating samples from a Gaussian distribution (Eq. 4b), and taking their product with themselves transposed (Eq. 4c).
Revised: The generative process for the Gram matrices, G (cid:96) , consists of generating samples from a Gaussian distribution (Eq. 4b), and taking their product with themselves transposed (Eq. 4c).

Original: This exactly matches the generative process for a Wishart distribution (Eq. 1), so we can write the Gram matrices,
Revised: This exactly matches the generative process for a Wishart distribution (Eq. 1), so we can write the Gram matrices,

Original: G (cid:96) , directly in terms of the kernel, without needing to sample features (Fig. 1 bottom),
Revised: G (cid:96) , directly in terms of the kernel, without needing to sample features (Fig. 1 bottom),

Original: [Equation]
Revised: [Equation]

Original: (cid:96) ∈

Original: [Equation]

Original: [Equation] L [Equation] Except at the output, the model is phrased entirely in terms of positive-deﬁnite kernels and Gram matrices, and is consistent under marginalisation (assuming a valid kernel) and is thus a DKP.
Revised: Except at the output, the model is phrased entirely in terms of positive-deﬁnite kernels and Gram matrices, and is consistent under marginalisation (assuming a valid kernel) and is thus a DKP.

Revised: At a high level, the model can be understood as alternatively sampling a Gram matrix (introducing ﬂexibility in the representation), and nonlinearly transforming the Gram matrix using a kernel (Fig. 2).

Original: This highlights a particularly simple interpretation of the DKP as an autoregressive process (Fig. 2).
Revised: This highlights a particularly simple interpretation of the DKP as an autoregressive process.

Original: In a standard autoregressive process, we might propagate the current vector, x t , through a deterministic function, f ( x t ) , and add zero-mean Gaussian noise, ξ ,
Revised: In a standard autoregressive process, we might propagate the current vector, x t , through a deterministic function, f ( x t ) , and add zero-mean Gaussian noise, ξ ,

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: By analogy, the next Gram matrix has expectation centered on a deterministic transformation of the previous Gram matrix,
Revised: By analogy, the next Gram matrix has expectation centered on a deterministic transformation of the previous Gram matrix,

Original: [Equation]
Revised: [Equation]

Original:  so G (cid:96)+1 can be written as this expectation plus a zero-mean random variable, Ξ  , that can be interpreted as noise,
Revised: (8) so G (cid:96) can be written as this expectation plus a zero-mean random variable, Ξ (cid:96) , that can be interpreted as noise,

Original: [Equation]
Revised: [Equation]

Original: Note that Ξ  is not in general positive deﬁnite, and may not have an analytically tractable distribution.
Revised: Note that Ξ (cid:96) is not in general positive deﬁnite, and may not have an analytically tractable distribution.

Original: This noise decreases as N (cid:96) increases,
Revised: This noise decreases as N (cid:96) increases,

Original: [Equation]
Revised: [Equation]

Original: Notably, as N (cid:96) tends to inﬁnity, the Wishart samples converge on their expectation, and the noise disappears, leaving us with a series of deterministic transformations of the Gram matrix.
Revised: Notably, as N (cid:96) tends to inﬁnity, the Wishart samples converge on their expectation, and the noise disappears, leaving us with a series of deterministic transformations of the Gram matrix.

Original: Therefore, we can understand a deep kernel process as alternatively adding “noise” to the kernel by sampling e.g. a Wishart or inverse Wishart distribution ( G 2 and G 3 in Fig. 2) and computing a nonlinear transformation of the kernel ( K ( G 2 ) and K ( G 3 ) in Fig. 2)
Revised: Therefore, we can understand a deep kernel process as alternatively adding “noise” to the kernel by sampling e.g. a Wishart or inverse Wishart distribution ( G 2 and G 3 in Fig. 2) and computing a nonlinear transformation of the kernel ( K ( G 2 ) and K ( G 3 ) in Fig. 2)

Original: Remember that we are restricted to kernels that can be written as a function of the Gram matrix,
Revised: Remember that we are restricted to kernels that can be written as a function of the Gram matrix,

Original: [Equation]
Revised: [Equation]

Revised: i n p u t i n d e x

Revised: K ( G 1 )

Revised: G

Revised: K ( G 2 )

Revised: G

Revised: K ( G 3 )   input index   i n p u t i n d e x   input index   input index   input index   input index

Revised: W i s h a r t

Revised: I n v e r s e W

Revised: i s h a r t where K features ( · ) takes a matrix of features, F (cid:96) , and returns the kernel matrix, K (cid:96) , and k is the usual kernel, which takes two feature vectors (rows of F (cid:96) ) and returns an element of the kernel matrix.

Original: This does not include all possible kernels because it is not possible to recover the features from the
Revised: This does not include all possible kernels because it is not possible to recover the features from the

Original: Gram matrix.
Revised: Gram matrix.

Original: In particular, the Gram matrix is invariant to unitary transformations of the features: the Gram matrix is the same for F (cid:96) and F (cid:48) (cid:96) =
Revised: In particular, the Gram matrix is invariant to unitary transformations of the features: the Gram matrix is the same for F (cid:96) and F (cid:48) (cid:96) =

Original: UF (cid:96) where U is a unitary matrix, such that UU T
Revised: UF (cid:96) where U is a unitary matrix, such that UU T

Original: =
Revised: =

Original: I ,
Revised: I ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Superﬁcially, this might seem very limiting — leaving us only with dot-product kernels (Williams & Rasmussen, 2006) such as,
Revised: Superﬁcially, this might seem very limiting — leaving us only with dot-product kernels (Williams & Rasmussen, 2006) such as,

Original: [Equation]
Revised: [Equation]

Original: i n p u t i n d e x

Original: K ( G 1 )

Original: G

Original: K ( G 2 )

Original: G

Original: K ( G 3 )   input index   i n p u t i n d e x   input index   input index   input index   input index

Original: W i s h a r t I n v e r s e W i s h a r t However, in reality, a far broader range of kernels ﬁt within this class.
Revised: However, in reality, a far broader range of kernels ﬁt within this class.

Original: Importantly, isotropic or radial basis function kernels including the squared exponential and Matern depend only on the squared distance between points, R , (Williams & Rasmussen, 2006)
Revised: Importantly, isotropic or radial basis function kernels including the squared exponential and Matern depend only on the squared distance between points, R , (Williams & Rasmussen, 2006)

Original: [Equation]
Revised: [Equation]

Original: These kernels can be written as a function of G , because the matrix of squared distances, R , can be computed from G ,
Revised: These kernels can be written as a function of G , because the matrix of squared distances, R , can be computed from G ,

Original: [Equation]
Revised: [Equation]

Original:  V ARIATIONAL INFERENCE IN DEEP KERNEL PROCESSES
Revised:  V ARIATIONAL INFERENCE IN DEEP KERNEL PROCESSES

Original: A key part of the motivation for developing deep kernel processes was that the posteriors over weights in a BNN or over features in a deep GP are extremely complex and multimodal, with a large number of symmetries that are not captured by standard approximate posteriors (MacKay, 1992; Moore, 2016; Pourzanjani et al., 2017).
Revised: A key part of the motivation for developing deep kernel processes was that the posteriors over weights in a BNN or over features in a deep GP are extremely complex and multimodal, with a large number of symmetries that are not captured by standard approximate posteriors (MacKay, 1992; Moore, 2016; Pourzanjani et al., 2017).

Original: For instance, in the Appendix we show that there are permutation symmetries in the prior and posteriors over weights in BNNs (App. D.1) and rotational symmetries in the prior and posterior over features in deep GPs with isotropic kernels (App. D.2).
Revised: For instance, in the Appendix we show that there are permutation symmetries in the prior and posteriors over weights in BNNs (App. D.1) and rotational symmetries in the prior and posterior over features in deep GPs with isotropic kernels (App. D.2).

Original: The inability to capture these symmetries in standard variational posteriors may introduce biases in the parameters inferred by variational inference, because the variational bound is not uniformly tight across the state-space (Turner & Sahani, 2011).
Revised: The inability to capture these symmetries in standard variational posteriors may introduce biases in the parameters inferred by variational inference, because the variational bound is not uniformly tight across the state-space (Turner & Sahani, 2011).

Original: Intuitively, these symmetries arise in DGPs with isotropic kernels because the features at the next layer depend only on the kernel matrix at the previous layer, and this kernel is invariant to unitary transformations of the features (Eq. 12).
Revised: Intuitively, these symmetries arise in DGPs with isotropic kernels because the features at the next layer depend only on the kernel matrix at the previous layer, and this kernel is invariant to unitary transformations of the features (Eq. 12).

Original: As such, we can sidestep these complex posterior symmetries by working directly with the Gram matrices as the random variables for variational inference.
Revised: As such, we can sidestep these complex posterior symmetries by working directly with the Gram matrices as the random variables for variational inference.

Original: We show that DGPs (Sec. 3.1) and inﬁnite NNs with bottlenecks (App. C.3) are deep Wishart processes, so a natural approach would be to deﬁne an approximate posterior over the Gram matrices in the deep Wishart process.
Revised: We show that DGPs (Sec. 3.1) and inﬁnite NNs with bottlenecks (App. C.3) are deep Wishart processes, so a natural approach would be to deﬁne an approximate posterior over the Gram matrices in the deep Wishart process.

Original: However, this turns out to be difﬁcult, predominantly because the approximate posterior we would like to use, the non-central Wishart (App. E), has a probability density function that is prohibitively costly and complex to evaluate in the inner loop of a deep learning model (Koev & Edelman, 2006).
Revised: However, this turns out to be difﬁcult, predominantly because the approximate posterior we would like to use, the non-central Wishart (App. E), has a probability density function that is prohibitively costly and complex to evaluate in the inner loop of a deep learning model (Koev & Edelman, 2006).

Original: Instead, we consider an inverse Wishart process prior, for which the inverse Wishart itself makes a good choice of approximate posterior.
Revised: Instead, we consider an inverse Wishart process prior, for which the inverse Wishart itself makes a good choice of approximate posterior.

Original: 4. T HE DEEP INVERSE W ISHART PROCESSES
Revised: 4. T HE DEEP INVERSE W ISHART PROCESSES

Original: By analogy with Eq.
Revised: By analogy with Eq.

Original: (6), our deep inverse Wishart processes (DIWPs) are given by
Revised: (6), our deep inverse Wishart processes (DIWPs) are given by

Original: [Equation]
Revised: [Equation]

Original: with 
Revised: with G 1 = [Equation] XΩX T

Original: [Equation]
Revised: [Equation]

Original:  (cid:96) 
Revised: 16a) [Equation] for (cid:96) ∈ { 2 , . . . L }

Original: [Equation]
Revised: [Equation]

Original: 16b
Revised: (16b)

Original: [Equation]
Revised: [Equation]

Revised: (16c) remember that X ∈ R P × N 0 , G (cid:96) ∈ R P × P and F (cid:96) ∈ R P × N L +1 .

Original: Note that at the input layer, K 0 = 1 N 0 X TX may be singular if there are more datapoints than features.
Revised: Note that at the input layer, K 0 = 1 N 0 XX T may be singular if there are more datapoints than features.

Original: Instead of attempting to use a singular Wishart distributions over G 1 , which would be complex and difﬁcult to work with (Bodnar & Okhrin, 2008; Bodnar et al., 2016), we instead use a full-rank an  N 0 × N 0 matrix, Ω  .
Revised: Instead of attempting to use a singular Wishart distributions over G 1 , which would be complex and difﬁcult to work with (Bodnar & Okhrin, 2008; Bodnar et al., 2016), we instead deﬁne an approximate posterior over the full-rank N 0 × N 0 matrix, Ω , and use G 1 = 1 N 0 XΩX T ∈ R P × P .

Original: Critically, the distributions in Eq.
Revised: Critically, the distributions in Eq.

Original: (16b) are consistent under marginalisation as long as δ (cid:96) is held constant (Dawid, 1981), with P taken to be the number of input points, or equivalently the size of K (cid:96) − 1 .
Revised: (16b) are consistent under marginalisation as long as δ (cid:96) is held constant (Dawid, 1981), with P taken to be the number of input points, or equivalently the size of K (cid:96) − 1 .

Original: Further, the deep inverse Wishart process retains the interpretation as a deterministic transformation of the kernel plus noise because the expectation is,
Revised: Further, the deep inverse Wishart process retains the interpretation as a deterministic transformation of the kernel plus noise because the expectation is,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The resulting inverse Wishart process does not have a direct interpretation as e.g. a deep GP, but does have more appealing properties for variational inference, as it is always full-rank and allows independent control over the approximate posterior mean and variance.
Revised: The resulting inverse Wishart process does not have a direct interpretation as e.g. a deep GP, but does have more appealing properties for variational inference, as it is always full-rank and allows independent control over the approximate posterior mean and variance.

Original: Finally, it is important to note that Wishart and inverse Wishart distributions do not differ as much as one might expect; the standard Wishart and standard inverse Wishart distributions have isotropic distributions over the eigenvectors so they only differ in terms of their distributions over eigenvalues, and these are often quite similar, especially if we consider a Wishart model with ResNet-like structure (App. H).
Revised: Finally, it is important to note that Wishart and inverse Wishart distributions do not differ as much as one might expect; the standard Wishart and standard inverse Wishart distributions have isotropic distributions over the eigenvectors so they only differ in terms of their distributions over eigenvalues, and these are often quite similar, especially if we consider a Wishart model with ResNet-like structure (App. H).

Original: 4. A N APPROXIMATE POSTERIOR FOR THE DEEP INVERSE W ISHART PROCESS
Revised: 4. A N APPROXIMATE POSTERIOR FOR THE DEEP INVERSE W ISHART PROCESS

Original: Choosing an appropriate and effective form for variational approximate posteriors is usually a difﬁcult research problem.
Revised: Choosing an appropriate and effective form for variational approximate posteriors is usually a difﬁcult research problem.

Original: Here, we take inspiration from Ober & Aitchison (2020) by exploiting the fact that the inverse-Wishart distribution is the conjugate prior for the covariance matrix of a multivariate Gaussian.
Revised: Here, we take inspiration from Ober & Aitchison (2020) by exploiting the fact that the inverse-Wishart distribution is the conjugate prior for the covariance matrix of a multivariate Gaussian.

Original: In particular, if we consider an inverse-Wishart prior over Σ  , which forms the covariance of Gaussian-distributed matrix, V  , consisting of columns v λ , 
Revised: In particular, if we consider an inverse-Wishart prior over Σ ∈ R P × P with mean δ Σ 0 , which forms the covariance of Gaussian-distributed matrix, V ∈ R P × P , consisting of columns v λ , [Equation] 18a [Equation]

Revised: 18c

Original: [Equation]
Revised: [Equation]

Original: Inspired by this exact posterior that is available in simple models, we choose the approximate posterior in our model to be,
Revised: Inspired by this exact posterior that is available in simple models, we choose the approximate posterior in our model to be,

Original: [Equation]
Revised: [Equation]

Original: (cid:96) − 1 )
Revised: (cid:1)

Original: + V (cid:96) V T(cid:96) , δ (cid:96) + γ (cid:96)
Revised: [Equation]

Original: + ( P + 1) (cid:1) , (19b)
Revised: (19b)

Original: Q ( FL +1 | G L ) =
Revised: Q [Equation] F

Original: (cid:81)
Revised: [Equation]

Original: N L +1 λ
Revised: G [Equation]

Original: =1 N (cid:0) f 
Revised: (cid:0) f [Equation] +

Original: L +1 λ ; Σ λ Λ λ v λ , Σ λ (cid:1) , where Σ λ = (cid:0)
Revised: [Equation] Σ λ Λ [Equation] Σ λ (cid:1) , where Σ λ = (cid:0)

Original: K − 1 ( G L ) + Λ λ (cid:1) − 1 , (19c) and where V 1 is a learned N 0 × N 0 matrix, { V (cid:96) } L(cid:96)
Revised: [Equation] (19c) and where V 1 is a learned N 0 × N 0 matrix, { V (cid:96) } L(cid:96)

Original: =2 area P × P  matrices and { γ (cid:96) } L(cid:96)
Revised: =2 are P × P learned matrices and { γ (cid:96) } L(cid:96)

Original: =1 are learned non-negative real numbers.
Revised: =1 are learned non-negative real numbers.

Original: For more details about the input layer, see App.
Revised: For more details about the input layer, see App.

Original: F. At the output layer, we take inspiration from the global inducing approximate posterior for DGPs from Ober & Aitchison (2020), with learned parameters  v λ  andthe positive deﬁnite matrices, Λ λ (see App. G).
Revised: F. At the output layer, we take inspiration from the global inducing approximate posterior for DGPs from Ober & Aitchison (2020), with learned parameters being vectors, v λ , and positive deﬁnite matrices, Λ λ (see App. G).

Original: In summary, the prior has parameters { δ (cid:96) } L(cid:96)
Revised: In summary, the prior has parameters { δ (cid:96) } L(cid:96)

Original: =1 (which also appears in the approximate posterior), and the posterior has parameters { V (cid:96) } L(cid:96)
Revised: =1 (which also appears in the approximate posterior), and the posterior has parameters { V (cid:96) } L(cid:96)

Original: =1 and { γ (cid:96) } L(cid:96)
Revised: =1 and { γ (cid:96) } L(cid:96)

Original: =
Revised: =

Original: 1 for the inverse-Wishart hidden layers, and { v λ } N L +1 λ
Revised: 1 for the inverse-Wishart hidden layers, and { v λ } N L +1 λ

Original: =1 and { Λ λ } N L +1 λ
Revised: =1 and { Λ λ } N L +1 λ

Original: =1 at the output.
Revised: =1 at the output.

Original: In all our experiments, we optimize all ﬁve parameters ( { δ (cid:96) , V (cid:96) , γ (cid:96) } L(cid:96)
Revised: In all our experiments, we optimize all ﬁve parameters ( { δ (cid:96) , V (cid:96) , γ (cid:96) } L(cid:96)

Original: =1 ) and ( { v λ , Λ λ } N L +1 λ =1 ) , and in addition, for inducing-point methods, we also optimize a single set of “global” inducing inputs, X i  , which are deﬁned only at the input layer.
Revised: =1 ) and ( { v λ , Λ λ } N L +1 λ =1 ) , and in addition, for inducing-point methods, we also optimize a single set of “global” inducing inputs, X i ∈ R P i × N 0 , which are deﬁned only at the input layer.

Original: 4. D OUBLY STOCHASTIC INDUCING - POINT VARIATIONAL INFERENCE IN DEEP INVERSE W ISHART PROCESSES
Revised: 4. D OUBLY STOCHASTIC INDUCING - POINT VARIATIONAL INFERENCE IN DEEP INVERSE W ISHART PROCESSES

Original: For efﬁcient inference in high-dimensional problems, we take inspiration from the DGP literature (Salimbeni & Deisenroth, 2017) by considering doubly-stochastic inducing-point deep inverse
Revised: For efﬁcient inference in high-dimensional problems, we take inspiration from the DGP literature (Salimbeni & Deisenroth, 2017) by considering doubly-stochastic inducing-point deep inverse

Original: Wishart processes.
Revised: Wishart processes.

Original: We begin by decomposing all variables into inducing and training (or test) points,
Revised: We begin by decomposing all variables into inducing and training (or test) points

Revised: X t ∈ R P t × N 0 ,

Revised: X = (cid:18)

Revised: X i X t (cid:19) ,

Revised: F L +1 =

Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: where e.g. G (cid:96) ii is P i × P
Revised: where e.g. G (cid:96) ii is P i × P

Original: i and G (cid:96) it is P
Revised: i and G (cid:96) it is P

Original: i × P t where P i is the number of inducing points, and P t is the number of testing/training points.
Revised: i × P t where P i is the number of inducing points, and P t is the number of testing/training points.

Original: Note that Ω does not decompose as it is N 0 × N 0 .
Revised: Note that Ω does not decompose as it is N 0 × N 0 .

Original: The full ELBO including latent variables for all the inducing and training points is,
Revised: The full ELBO including latent variables for all the inducing and training points is,

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over Q
Revised: where the expectation is taken over Q

Original: [Equation]
Revised: [Equation]

Original: The approximate posterior over inducing points,
Revised: The prior is given by combining all terms in Eq.

Original: Q
Revised: (16) for both inducing and test/train inputs,

Original: [Equation]
Revised: [Equation]

Original: L +1 i | X

Original: [Equation]
Revised: [Equation]

Original: (cid:1) , is given by Eq.
Revised: where the X -dependence enters on the right because G 1 = 1 N 0 XΩX T .

Original: (19), and again taking inspiration from
Revised: Taking inspiration from

Original: Salimbeni & Deisenroth (2017), the  approximate posterior over the train/test points is given by the conditional prior,
Revised: Salimbeni & Deisenroth (2017), the full approximate posterior is the product of an approximate posterior over inducing points and the conditional prior for train/test points,

Original: [Equation]
Revised: [Equation]

Original: As such, the ELBO simpliﬁes,
Revised: And the prior can be written in the same form, [Equation] We discuss the second terms (the conditional prior) in Eq. The ﬁrst terms (the prior and approximate posteriors over inducing points), are given by combining terms in Eq.

Revised: (16) and Eq.

Revised: P

Original: [Equation]
Revised: [Equation]

Revised: { G (cid:96)

Revised: [Equation]

Revised: L(cid:96)

Revised: =2 , F L +1 i

Revised: [Equation]

Revised: (cid:0)

Revised: G (cid:96) ii | G (cid:96) − 1 ii (cid:1) (cid:105)

Revised: P (cid:0)

Revised: F L +1 i | G L ii (cid:1) , (25)

Revised: Q (cid:0) Ω , { G (cid:96) ii } L (cid:96) =2 , F L +1 i

Revised: | X i (cid:1) =

Revised: Q ( Ω ) (cid:104)(cid:81)

Revised: L (cid:96) =

Revised: 2 Q (cid:0)

Revised: G (cid:96) ii | G (cid:96) − 1 ii (cid:1)(cid:105)

Revised: Q (cid:0)

Revised: F L +1 i | G L ii (cid:1) .

Revised: Substituting Eqs.

Revised: (23–26) into the ELBO (Eq. 21), the conditional prior cancels and we obtain,

Revised: L = E   log P (cid:0)

Revised: Y | F L +1 t (cid:1)

Revised: + log

Revised: Q ( Ω ) (cid:104)(cid:81)

Revised: L(cid:96)

Revised: =

Revised: 2 Q (cid:0)

Revised: G (cid:96) ii | G (cid:96) − 1 ii (cid:1)(cid:105)

Revised: Q (cid:0)

Revised: F L +1 i | G L ii (cid:1)

Revised: P ( Ω ) (cid:104)(cid:81)

Revised: L(cid:96)

Revised: =

Revised: 2 P (cid:0)

Revised: G (cid:96) ii | G (cid:96) − 1 ii (cid:1)(cid:105)

Original:  Importantly, the ﬁrst term is a summation across test datapoints, and the second term depends only on the inducing points, so as in Salimbeni & Deisenroth (2017) we can compute unbiased estimates of the expectation by taking only a minibatch of datapoints.
Revised: P (cid:0) F L +1 i | G L ii (cid:1)   . Importantly, the ﬁrst term is a summation across test/train datapoints, and the second term depends only on the inducing points, so as in Salimbeni & Deisenroth (2017) we can compute unbiased estimates of the expectation by taking only a minibatch of datapoints, and we never need to compute the density of the conditional prior in Eq.

Revised: (28), we only need to be able to sample it.

Original: Finally, to sample the test/training points, conditioned on the inducing points, we need to sample,
Revised: Finally, to sample the test/training points, conditioned on the inducing points, we need to sample,

Original: P
Revised: P

Original: [Equation]

Original: [Equation]

Original: { G

Original: [Equation]

Original: tt } L(cid:96)

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst distribution, P (cid:0)
Revised: The ﬁrst distribution, P (cid:0)

Original: [Equation] (cid:1) , is a multivariate Gaussian, and can be evaluated using methods from the GP literature (Williams & Rasmussen, 2006; Salimbeni & Deisenroth, 2017).
Revised: F L +1 t | F L +1 i , G L (cid:1) , is a multivariate Gaussian, and can be evaluated using methods from the GP literature (Williams & Rasmussen, 2006; Salimbeni & Deisenroth, 2017).

Original: The difﬁculties arise for the inverse Wishart terms, P (cid:0)
Revised: The difﬁculties arise for the inverse Wishart terms, P (cid:0)

Original: [Equation]
Revised: [Equation]

Original: Then, we note that samples from the jointprior over inducing and train/test locations can be written,
Revised: To sample this distribution, note that samples from the joint over inducing and train/test locations can be written,

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and where P i is the number of inducing inputs, and P t is the number of train/test inputs.
Revised: and where P i is the number of inducing inputs, and P t is the number of train/test inputs.

Original: Deﬁning the Schur complements,
Revised: Deﬁning the Schur complements,

Original: [Equation]
Revised: [Equation]

Original: We know that G (cid:96) tt · i and
Revised: We know that G (cid:96) tt · i and

Original: [Equation]
Revised: [Equation]

Revised: G

Original: distribution
Revised: [Equation] G [Equation] it have distribution, (Eaton, 1983)

Original: [Equation]
Revised: [Equation]

Original: where MN is the matrix normal.
Revised: where MN is the matrix normal.

Original: Now, G (cid:96) it and G (cid:96) tt , can be recovered by algebraic manipulation.
Revised: Now, G (cid:96) it and G (cid:96) tt , can be recovered by algebraic manipulation.

Original: Finally, because of the doubly stochastic form for the objective, we do not need to sample multiple of jointly consistent samples for test points; instead, (and as in DGPs Salimbeni & Deisenroth, 2017) we can independently sample each test point (App. I), which dramatically reduces computational complexity. 
Revised: Finally, because of the doubly stochastic form for the objective, we do not need to sample multiple of jointly consistent samples for test points; instead, (and as in DGPs Salimbeni & Deisenroth, 2017) we can independently sample each test point (App. I), which dramatically reduces computational complexity. We optimize using standard reparameterised variational inference (Kingma & Welling, 2013; Rezende et al., 2014) (Ober & Aitchison, 2020, for details on how to reparameterise samples from the Wishart, see).  C OMPUTATIONAL COMPLEXITY

Original: [Table]
Revised: As in non-deep GPs, the complexity is O ( P 3 ) for time and O ( P 2 ) for space for standard DKPs (the O ( P 3 ) time dependencies emerge e.g. because of inverses and determinants required for the inverse Wishart distributions). For DSVI, there is a P 3 i time and P 2 i space term for the inducing points, because the computations for inducing points are exactly the same as in the non-DSVI case.

Original: {+As we can treat each test/train point independently (App. I), the complexity for test/training points must scale linearly with P t , and this term has P 
+}
Revised: {+As we can treat each test/train point independently (App. I), the complexity for test/training points must scale linearly with P t , and this term has P 
+}

Revised: i time scaling, e.g. due to the matrix products in Eq.

Revised: Thus, the overall complexity for DSVI is O ( P 3 i + P 2 i P t ) for time and O

Revised: ( P 2 i + P i P t ) for space which is exactly the same as non-deep inducing GPs.

Revised: Thus, and exactly as in non-deep inducing-GPs, by using a small number of inducing points, we are able to convert a cubic dependence on the number of input points into a linear dependence, which gives considerably better scaling.

Revised: Surprisingly, this is substantially better than standard DGPs.

Revised: In standard DGPs, we allow the approximate posterior covariance for each feature to differ (Salimbeni & Deisenroth, 2017), in which case, we are in essence doing standard inducing-GP inference over N hidden features, which gives complexity of O ( NP 3 i + NP 2 i P t ) for time and O ( NP 2 i + NP i P t ) for space (Salimbeni & Deisenroth, 2017).

Revised: It is possible to improve this complexity by restricting the approximate posterior to have the same covariance for each point (but this restriction can be expected to harms performance).

Revised: [Table]

Original:  R ESULTS
Revised:  R ESULTS

Original: We began by comparing the performance of our deep inverse Wishart process (DIWP) against inﬁnite Bayesian neural networks (known as the neural network Gaussian process or NNGP) and DGPs.
Revised: We began by comparing the performance of our deep inverse Wishart process (DIWP) against inﬁnite Bayesian neural networks (known as the neural network Gaussian process or NNGP) and DGPs.

Original: To ensure sensible comparisons against the NNGP, we used a ReLU kernel in all models (Cho &
Revised: To ensure sensible comparisons against the NNGP, we used a ReLU kernel in all models (Cho &

Original: Saul, 2009).
Revised: Saul, 2009).

Original: For all models, we used three layers (two hidden layers and one output layer), with three applications of the kernel.
Revised: For all models, we used three layers (two hidden layers and one output layer), with three applications of the kernel.

Original: In each case, we used a learned bias and scale for each input feature, and trained for 8000 gradient steps with the Adam optimizer with 100 inducing points, a learning rate of 10 − 2 for the ﬁrst 4000 steps and 10 − 3 for the ﬁnal 4000 steps.
Revised: In each case, we used a learned bias and scale for each input feature, and trained for 8000 gradient steps with the Adam optimizer with 100 inducing points, a learning rate of 10 − 2 for the ﬁrst 4000 steps and 10 − 3 for the ﬁnal 4000 steps.

Original: For evaluation, we used 100 samples from the ﬁnal iteration of gradient descent, and for each training step we used 10 samples in the smaller datasets (boston, concrete, energy, wine, yacht), and 1 sample in the larger datasets.
Revised: For evaluation, we used 100 samples from the ﬁnal iteration of gradient descent, and for each training step we used 10 samples in the smaller datasets (boston, concrete, energy, wine, yacht), and 1 sample in the larger datasets.

Original: We found that DIWP usually gives better predictive performance and ELBOs.
Revised: We found that DIWP usually gives better predictive performance and ELBOs.

Original: We expected DIWP to be better than (or the same as) the NNGP as the NNGP was a special case of our DIWP (sending δ (cid:96) → ∞ sends the variance of the inverse Wishart to zero, so the model becomes equivalent to the
Revised: We expected DIWP to be better than (or the same as) the NNGP as the NNGP was a special case of our DIWP (sending δ (cid:96) → ∞ sends the variance of the inverse Wishart to zero, so the model becomes equivalent to the

Original: NNGP).
Revised: NNGP).

Original: We found that the DGP performs poorly in comparison to DIWP and NNGPs, and even to past baselines on all datasets except protein (which is by far the largest).
Revised: We found that the DGP performs poorly in comparison to DIWP and NNGPs, and even to past baselines on all datasets except protein (which is by far the largest).

Original: This is because we use a
Revised: This is because we use a

Original: ReLU, rather than a squared exponential kernel, as in (Salimbeni & Deisenroth, 2017), and because we used a plain feedforward architecture for all models.
Revised: ReLU, rather than a squared exponential kernel, as in (Salimbeni & Deisenroth, 2017), and because we used a plain feedforward architecture for all models.

Original: In contrast, Salimbeni & Deisenroth (2017) found that good performance with DGPs on even UCI datasets required a complex architecture involving skip connections.
Revised: In contrast, Salimbeni & Deisenroth (2017) found that good performance with DGPs on even UCI datasets required a complex architecture involving skip connections.

Original: Here, we used simple feedforward architectures, both to ensure a fair comparison to the other models, and to avoid the need for an architecture search.
Revised: Here, we used simple feedforward architectures, both to ensure a fair comparison to the other models, and to avoid the need for an architecture search.

Original: In addition, the inverse Wishart process is implicitly able to learn the network “width”, δ (cid:96) , whereas in the DGPs, the width is ﬁxed to be equal to the number of input features, following standard practice in the literature (e.g. Salimbeni & Deisenroth, 2017).
Revised: In addition, the inverse Wishart process is implicitly able to learn the network “width”, δ (cid:96) , whereas in the DGPs, the width is ﬁxed to be equal to the number of input features, following standard practice in the literature (e.g. Salimbeni & Deisenroth, 2017).

Original: Next, we considered fully-connected networks for small image classiﬁcation datasets (MNIST and
Revised: Next, we considered fully-connected networks for small image classiﬁcation datasets (MNIST and

Original: CIFAR-10).
Revised: CIFAR-10).

Original: We used the same models as in the previous section, with the omission of learned bias and scaling of the inputs.
Revised: We used the same models as in the previous section, with the omission of learned bias and scaling of the inputs.

Original: Note that we do not expect these methods to perform well relative to
Revised: Note that we do not expect these methods to perform well relative to

Original: [Table]
Revised: [Table]

Original: standard methods (e.g. CNNs) for these datasets, as we are using fully-connected networks with only 100 inducing points (whereas e.g. work in the NNGP literature uses the full 60 , 000 × 60 ,covariance matrix).
Revised: standard methods (e.g. CNNs) for these datasets, as we are using fully-connected networks with only 100 inducing points (whereas e.g. work in the NNGP literature uses the full 60 , 000 × 60 ,covariance matrix).

Original: Nonetheless, as the architectures are carefully matched, it provides another opportunity to compare the performance of DIWPs, NNGPs and DGPs.
Revised: Nonetheless, as the architectures are carefully matched, it provides another opportunity to compare the performance of DIWPs, NNGPs and DGPs.

Original: Again
Revised: Again

Original: [Table]
Revised: [Table]

Original: we found that DIWP usually gave statistically signiﬁcant but perhaps underwhelming gains in predictive performance (except for CIFAR-10 test-log-likelihood, where DIWP lagged by only 0 . 01 ).
Revised: we found that DIWP usually gave statistically signiﬁcant but perhaps underwhelming gains in predictive performance (except for CIFAR-10 test-log-likelihood, where DIWP lagged by only 0 . 01 ).

Original: Importantly, DIWP gives very large improvements in the ELBO, with gains of 0 . 09 against DGPs for MNIST and 0 .for
Revised: Importantly, DIWP gives very large improvements in the ELBO, with gains of 0 . 09 against DGPs for MNIST and 0 .for

Original: CIFAR-10 (App. K).
Revised: CIFAR-10 (App. K).

Original: For MNIST, remember that the ELBO must be negative (because both the log-likelihood for classiﬁcation and the KL-divergence term give negative contributions), so the improvement from − 0 .
Revised: For MNIST, remember that the ELBO must be negative (because both the log-likelihood for classiﬁcation and the KL-divergence term give negative contributions), so the improvement from − 0 .

Original: 301 to − 0 .
Revised: 301 to − 0 .

Original: 214 represents a dramatic change.
Revised: 214 represents a dramatic change.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Our ﬁrst contribution was the observation that DGPs with isotropic kernels can be written as deep
Revised: Our ﬁrst contribution was the observation that DGPs with isotropic kernels can be written as deep

Original: Wishart processes as the kernel depends only on the Gram matrix.
Revised: Wishart processes as the kernel depends only on the Gram matrix.

Original: We then gave similar observations for neural networks (App. C.1), inﬁnite neural networks (App. C.2) and inﬁnite network with bottlenecks (App.
Revised: We then gave similar observations for neural networks (App. C.1), inﬁnite neural networks (App. C.2) and inﬁnite network with bottlenecks (App.

Original: C.3, also see Aitchison, 2019).
Revised: C.3, also see Aitchison, 2019).

Original: These observations motivated us to consider the deep inverse Wishart process prior, which is a novel combination of two pre-existing elements: nonlinear transformations of the kernel (e.g. Cho & Saul, 2009) and inverse Wishart priors over kernels (e.g. Shah et al., 2014).
Revised: These observations motivated us to consider the deep inverse Wishart process prior, which is a novel combination of two pre-existing elements: nonlinear transformations of the kernel (e.g. Cho & Saul, 2009) and inverse Wishart priors over kernels (e.g. Shah et al., 2014).

Original: Deep nonlinear transformations of the kernel have been used in the inﬁnite neural network literature (Lee et al., 2017; Matthews et al., 2018) where they form deterministic, parameter-free kernels that do not have any ﬂexibility in the lower-layers (Aitchison, 2019).
Revised: Deep nonlinear transformations of the kernel have been used in the inﬁnite neural network literature (Lee et al., 2017; Matthews et al., 2018) where they form deterministic, parameter-free kernels that do not have any ﬂexibility in the lower-layers (Aitchison, 2019).

Original: Likewise, inverse-Wishart distributions have been suggested as priors over covariance matrices (Shah et al., 2014), but they considered a model without nonlinear transformations of the kernel.
Revised: Likewise, inverse-Wishart distributions have been suggested as priors over covariance matrices (Shah et al., 2014), but they considered a model without nonlinear transformations of the kernel.

Original: Surprisingly, without these nonlinear transformations, the inverse Wishart prior becomes equivalent to simply scaling the covariance with a scalar random variable (App.
Revised: Surprisingly, without these nonlinear transformations, the inverse Wishart prior becomes equivalent to simply scaling the covariance with a scalar random variable (App.

Original: L Shah et al., 2014).
Revised: L; Shah et al., 2014).

Original: Further linear (inverse)
Revised: Further linear (inverse)

Original: Wishart processes have been used in the ﬁnancial domain to model how the volatility of asset prices changes over time (Philipov & Glickman, 2006b;a; Asai & McAleer, 2009; Gourieroux & Sufana, 2010; Wilson & Ghahramani, 2010; Heaukulani & van der Wilk, 2019).
Revised: Wishart processes have been used in the ﬁnancial domain to model how the volatility of asset prices changes over time (Philipov & Glickman, 2006b;a; Asai & McAleer, 2009; Gourieroux & Sufana, 2010; Wilson & Ghahramani, 2010; Heaukulani & van der Wilk, 2019).

Original: Importantly, inference in these dynamical (inverse) Wishart processes is often performed by assuming ﬁxed, integer degrees of freedom, and working with underlying Gaussian distributed features.
Revised: Importantly, inference in these dynamical (inverse) Wishart processes is often performed by assuming ﬁxed, integer degrees of freedom, and working with underlying Gaussian distributed features.

Original: This approach allows us to leverage standard GP techniques (e.g. Kandemir & Hamprecht, 2015; Heaukulani & van der Wilk, 2019), but  the posterior over these features usually has rotational symmetries (App. D.2) that are not captured by standard variational posteriors.
Revised: This approach allows one to leverage standard GP techniques (e.g. Kandemir & Hamprecht, 2015; Heaukulani & van der Wilk, 2019), but it is not possible to optimize the degrees of freedom and the posterior over these features usually has rotational symmetries (App. D.2) that are not captured by standard variational posteriors.

Original: In contrast, we give a novel doubly-stochastic variational inducing point inference method that operates purely on Gram matrices and thus avoids needing to capture these symmetries.
Revised: In contrast, we give a novel doubly-stochastic variational inducing point inference method that operates purely on Gram matrices and thus avoids needing to capture these symmetries.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We proposed deep kernel processes which combine nonlinear transformations of the Gram matrix with sampling from matrix-variate distributions such as the inverse Wishart.
Revised: We proposed deep kernel processes which combine nonlinear transformations of the Gram matrix with sampling from matrix-variate distributions such as the inverse Wishart.

Original: We showed that DGPs,
Revised: We showed that DGPs,

Original: BNNs (App. C.1), inﬁnite BNNs (App. C.2) and inﬁnite BNNs with bottlenecks (App. C.3) are all instances of DKPs.
Revised: BNNs (App. C.1), inﬁnite BNNs (App. C.2) and inﬁnite BNNs with bottlenecks (App. C.3) are all instances of DKPs.

Original: We deﬁned a new family of deep inverse Wishart processes, and give a novel doubly-stochastic inducing point variational inference scheme that works purely in the space of
Revised: We deﬁned a new family of deep inverse Wishart processes, and give a novel doubly-stochastic inducing point variational inference scheme that works purely in the space of

Original: Gram matrices.
Revised: Gram matrices.

Original: DIWP performed better than NNGPs and DGPs on UCI, MNIST and CIFARbenchmarks.
Revised: DIWP performed better than NNGPs and DGPs on UCI, MNIST and CIFARbenchmarks.

Original: A DKP S ARE KERNEL PROCESSES
Revised: A DKP S ARE KERNEL PROCESSES

Original: We deﬁne a generic DKP to be K ( K )  .
Revised: We deﬁne a generic DKP to be K ( K ) , for a random matrix K ∈ R P × P .

Original: For instance, we could take,
Revised: For instance, we could take,

Original: [Equation]
Revised: [Equation]

Revised: where N is a positive integer and δ is a positive real number.

Original: A deep kernel process, D , is the composition of two (or more) underlying kernel processes, K 1 and K 2 ,
Revised: A deep kernel process, D , is the composition of two (or more) underlying kernel processes, K 1 and K 2 ,

Original: [Equation]
Revised: [Equation]

Original: We deﬁne K ∗ , G ∗ 1 and G ∗ 2 as principle submatrices of K , G 1 and G 2 respectively, dropping the same rows and columns.
Revised: We deﬁne K ∗ , G ∗ 1 and G ∗ 2 as principle submatrices of K , G 1 and G 2 respectively, dropping the same rows and columns.

Original: To establish that D is consistent under marginalisation, we use the consistency under marginalisation of K 1 and K
Revised: To establish that D is consistent under marginalisation, we use the consistency under marginalisation of K 1 and K

Original: [Equation]
Revised: [Equation]

Original: and the deﬁnition of the D as the composition of K 1 and K 2 (Eq. 29)
Revised: and the deﬁnition of the D as the composition of K 1 and K 2 (Eq. 33)

Original: [Equation]
Revised: [Equation]

Original: D is thus consistent under marginalisation, and hence is a kernel process.
Revised: D is thus consistent under marginalisation, and hence is a kernel process.

Original: Further, note that we can consider K to be a deterministic distribution that gives mass to only a single
Revised: Further, note that we can consider K to be a deterministic distribution that gives mass to only a single

Original: G .
Revised: G .

Original: In that case, K can be thought of as a deterministic function which must satisfy a corresponding consistency property,
Revised: In that case, K can be thought of as a deterministic function which must satisfy a corresponding consistency property,

Original: [Equation]
Revised: [Equation]

Original: and this is indeed satisﬁed by all deterministic transformations of kernels considered here.
Revised: and this is indeed satisﬁed by all deterministic transformations of kernels considered here.

Original: In practical terms, as long as G is always a valid kernel, it is sufﬁcient for the elements of G i (cid:54) =
Revised: In practical terms, as long as G is always a valid kernel, it is sufﬁcient for the elements of G i (cid:54) =

Original: j to depend only on K ij , K ii and K jj and for G ii to depend only on K jj , which is satisﬁed by e.g. the squared exponential kernel (Eq. 15) and by the ReLU kernel (Cho & Saul, 2009).
Revised: j to depend only on K ij , K ii and K jj and for G ii to depend only on K jj , which is satisﬁed by e.g. the squared exponential kernel (Eq. 15) and by the ReLU kernel (Cho & Saul, 2009).

Original: B T HE FIRST LAYER OF OUR DEEP GP AS B AYESIAN INFERENCE OVER A GENERALISED LENGTHSCALE
Revised: B T HE FIRST LAYER OF OUR DEEP GP AS B AYESIAN INFERENCE OVER A GENERALISED LENGTHSCALE

Original: In our deep GP architecture, we ﬁrst sample F 1  from a Gaussian with covariance K 0 =1 N 0 XX T (Eq. 4a).
Revised: In our deep GP architecture, we ﬁrst sample F 1 ∈ R P × N 1 from a Gaussian with covariance K 0 = N 0 XX T (Eq. 4a).

Original: This might seem odd, as the usual deep GP involves passing the input, X  , directly to the kernel function.
Revised: This might seem odd, as the usual deep GP involves passing the input, X ∈ R P × N 0 , directly to the kernel function.

Original: However, in the standard deep GP framework, the kernel (e.g. a squared exponential kernel) has lengthscale hyperparameters which can be inferred using Bayesian inference.
Revised: However, in the standard deep GP framework, the kernel (e.g. a squared exponential kernel) has lengthscale hyperparameters which can be inferred using Bayesian inference.

Original: In particular,
Revised: In particular,

Original: [Equation]
Revised: [Equation]

Revised: where k param is a new squared exponential kernel that explicitly includes hyperparmeters Ω ∈

Original:  where x i is the i th row of X and is thus a row-vector.
Revised: R N 0 × N 0 , and where x i is the i th row of X .

Original: Typically, in deep GPs, the parameter, Ω , is diagonal, and the diagonal elements correspond to the inverse square of the lengthscale, l i , (i.e. Ω ii = 1 /l 2 i ).
Revised: Typically, in deep GPs, the parameter, Ω , is diagonal, and the diagonal elements correspond to the inverse square of the lengthscale, l i , (i.e. Ω ii = 1 /l 2 i ).

Original: However, in many cases it may be useful to have a non-diagonal scaling.
Revised: However, in many cases it may be useful to have a non-diagonal scaling.

Original: For instance, we could use,
Revised: For instance, we could use,

Original: [Equation]
Revised: [Equation]

Original: ) which corresponds to,
Revised: ) which corresponds to,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Under our approach, we sample F = F 1 from Eq.
Revised: Under our approach, we sample F = F 1 from Eq.

Original: (4b), so F can be written as,
Revised: (4b), so F can be written as,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: X

Original: W
Revised: W

Original: F W
Revised: [Equation]

Original: F
Revised: F

Revised: [Equation]

Original: W
Revised: W

Revised: [Equation]

Original: F
Revised: F

Original: W
Revised: W

Original: F
Revised: F

Original: Y
Revised: Y

Original: X K
Revised: [Equation]

Original: F
Revised: F

Revised: [Equation]

Original: K
Revised: K

Revised: [Equation]

Original: F
Revised: F

Revised: [Equation]

Original: K
Revised: K

Revised: [Equation]

Original: F
Revised: F

Original: K
Revised: K

Original: F
Revised: F

Original: Y
Revised: Y

Original: X
Revised: X

Original: K
Revised: K

Revised: [Equation]

Original: K
Revised: K

Revised: [Equation]

Original: K K
Revised: K K

Original: F
Revised: F

Original: Y where f i is the i th row of F .
Revised: Y where f i is the i th row of F .

Original: Putting this into a squared exponential kernel without a lengthscale parameter,
Revised: Putting this into a squared exponential kernel without a lengthscale parameter,

Original: [Equation]
Revised: [Equation]

Original: We ﬁnd that a parameter-free squared exponential kernel applied to F is equivalent to a squaredexponential kernel with generalised lengthscale hyperparameters applied to the input.
Revised: We ﬁnd that a parameter-free squared exponential kernel applied to F is equivalent to a squaredexponential kernel with generalised lengthscale hyperparameters applied to the input.

Original: C BNN S AS DEEP KERNEL PROCESSES
Revised: C BNN S AS DEEP KERNEL PROCESSES

Original: Here we show that standard, ﬁnite BNNs, inﬁnite BNNs and inﬁnite BNNs with bottlenecks can be understood as deep kernel processes.
Revised: Here we show that standard, ﬁnite BNNs, inﬁnite BNNs and inﬁnite BNNs with bottlenecks can be understood as deep kernel processes.

Original: C. S TANARD FINITE BNN S ( AND GENERAL DGP S )
Revised: C. S TANARD FINITE BNN S ( AND GENERAL DGP S )

Original: Standard, ﬁnite BNNs are deep kernel processes, albeit ones which do not admit an analytic expression for the probability density.
Revised: Standard, ﬁnite BNNs are deep kernel processes, albeit ones which do not admit an analytic expression for the probability density.

Original: In particular, the prior for a standard Bayesian neural network (Fig. 3 top) is, 
Revised: In particular, the prior for a standard Bayesian neural network (Fig. 3 top) is, [Equation] (cid:96) [Equation]

Revised: otherwise ,

Original: [Equation]
Revised: [Equation]

Revised: 41b

Revised: [Equation]

Revised: where w (cid:96)λ is the λ th column of W (cid:96) .

Original: In the neural-network case, φ is a pointwise nonlinearity such as a ReLU.
Revised: In the neural-network case, φ is a pointwise nonlinearity such as a ReLU.

Original: Integrating out the weights, the features, F (cid:96) , become Gaussian distributed, as they depend linearly on the Gaussian distributed weights, W (cid:96) ,
Revised: Integrating out the weights, the features, F (cid:96) , become Gaussian distributed, as they depend linearly on the Gaussian distributed weights, W (cid:96) ,

Original: [Equation]
Revised: [Equation]

Original:  where
Revised: 42) where

Original: [Equation]
Revised: [Equation]

Original: Crucially, F (cid:96) depends on the previous layer activities, F (cid:96) − 1 only through the kernel, K (cid:96) .
Revised: Crucially, F (cid:96) depends on the previous layer activities, F (cid:96) − 1 only through the kernel, K (cid:96) .

Original: As such, we could write a generative model as (Fig.
Revised: As such, we could write a generative model as (Fig.

Original: 3 middle),
Revised: 3 middle),

Original: [Equation]
Revised: [Equation]

Original: where we have explicitly included the kernel, K (cid:96) , as a latent variable.
Revised: where we have explicitly included the kernel, K (cid:96) , as a latent variable.

Original: This form highlights that
Revised: This form highlights that

Original: BNNs are deep GPs, in the sense that F (cid:96)λ are Gaussian, with a kernel that depends on the activations from the previous layer.
Revised: BNNs are deep GPs, in the sense that F (cid:96)λ are Gaussian, with a kernel that depends on the activations from the previous layer.

Original: Indeed note that any deep GP (i.e. including those with kernels that cannot be written as a function of the Gram matrix) as a kernel, K (cid:96) , is by deﬁnition a matrix that can be written as the outer product of a potentially inﬁnite number of features, φ ( F (cid:96) ) where we allow φ to be a much richer class of functions than the usual pointwise nonlinearities (Hofmann et al., 2008).
Revised: Indeed note that any deep GP (i.e. including those with kernels that cannot be written as a function of the Gram matrix) as a kernel, K (cid:96) , is by deﬁnition a matrix that can be written as the outer product of a potentially inﬁnite number of features, φ ( F (cid:96) ) where we allow φ to be a much richer class of functions than the usual pointwise nonlinearities (Hofmann et al., 2008).

Original: We might now try to follow the approach we took above for deep GPs, and consider a Wishartdistributed Gram matrix, G (cid:96) = 1 N (cid:96) F (cid:96) F T(cid:96) .
Revised: We might now try to follow the approach we took above for deep GPs, and consider a Wishartdistributed Gram matrix, G (cid:96) = 1 N (cid:96) F (cid:96) F T(cid:96) .

Original: However, for BNNs we encounter an issue: we are not able to compute the kernel, K (cid:96) just using the Gram matrix, G (cid:96) : we need the full set of features, F (cid:96) .
Revised: However, for BNNs we encounter an issue: we are not able to compute the kernel, K (cid:96) just using the Gram matrix, G (cid:96) : we need the full set of features, F (cid:96) .

Original: Instead, we need an alternative approach to show that a neural network is a deep kernel process.
Revised: Instead, we need an alternative approach to show that a neural network is a deep kernel process.

Original: In particular, after integrating out the weights, the resulting distribution is chain-structured (Fig.middle), so in principle we can integrate out F (cid:96) to obtain a distribution over K (cid:96) conditioned on
Revised: In particular, after integrating out the weights, the resulting distribution is chain-structured (Fig.middle), so in principle we can integrate out F (cid:96) to obtain a distribution over K (cid:96) conditioned on

Original: K (cid:96) − 1 , giving the DKP model in Fig.
Revised: K (cid:96) − 1 , giving the DKP model in Fig.

Original: 3 (bottom),
Revised: 3 (bottom),

Original: [Equation]
Revised: [Equation]

Original: 41) where P ( F (cid:96) − 1 | K (cid:96) − 1 ) is given by Eq.
Revised: where P ( F (cid:96) − 1 | K (cid:96) − 1 ) is given by Eq.

Original: (40b) and δ  is the Dirac-delta function, representing the deterministic distribution, P ( K (cid:96) | F (cid:96) − 1 ) (Eq. 40a).
Revised: (44b) and δ D is the Dirac-delta function, representing the deterministic distribution, P ( K (cid:96) | F (cid:96) − 1 ) (Eq. 44a).

Original: Using this integral to write out the generative process only in terms of K (cid:96) gives the deep kernel process in Fig.
Revised: Using this integral to write out the generative process only in terms of K (cid:96) gives the deep kernel process in Fig.

Original: 3 (bottom).
Revised: 3 (bottom).

Original: While this distribution exists in principle, it cannot be evaluated analytically.
Revised: While this distribution exists in principle, it cannot be evaluated analytically.

Original: But we can explicitly evaluate the expected value of K (cid:96) given K (cid:96) − 1 using results from Cho & Saul (2009).
Revised: But we can explicitly evaluate the expected value of K (cid:96) given K (cid:96) − 1 using results from Cho & Saul (2009).

Original: In particular, we take Eq.
Revised: In particular, we take Eq.

Original: 40a, write out the matrix-multiplication explicitly as a series of vector outer products, and note that as f (cid:96)λ is IID across (cid:96) , the empirical average is equal to the expectation of a single term, which is computed by
Revised: 44a, write out the matrix-multiplication explicitly as a series of vector outer products, and note that as f (cid:96)λ is IID across (cid:96) , the empirical average is equal to the expectation of a single term, which is computed by

Original: Cho & Saul (2009),
Revised: Cho & Saul (2009),

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we deﬁne this expectation to be K ( K (cid:96) ) in the case of NNs.
Revised: we deﬁne this expectation to be K ( K (cid:96) ) in the case of NNs.

Original: C. I NFINITE NN S
Revised: C. I NFINITE NN S

Original: We have found that for standard ﬁnite neural networks, we were not able to compute the distribution over K (cid:96) conditioned on K (cid:96) − 1 (Eq.
Revised: We have found that for standard ﬁnite neural networks, we were not able to compute the distribution over K (cid:96) conditioned on K (cid:96) − 1 (Eq.

Original: To resolve this issue, one approach is to consider the limit of an inﬁnitely wide neural network.
Revised: To resolve this issue, one approach is to consider the limit of an inﬁnitely wide neural network.

Original: In this limit, the K (cid:96) becomes a deterministic function of K (cid:96) − 1 , as K (cid:96) can be written as the average of N (cid:96) IID outer products
Revised: In this limit, the K (cid:96) becomes a deterministic function of K (cid:96) − 1 , as K (cid:96) can be written as the average of N (cid:96) IID outer products

Original: [Equation]
Revised: [Equation]

Original: and as N (cid:96) grows to inﬁnity, the law of large numbers tells us that the average becomes equal to its expectation,
Revised: and as N (cid:96) grows to inﬁnity, the law of large numbers tells us that the average becomes equal to its expectation,

Original: [Equation]
Revised: [Equation]

Original: C. I NFINITE NN S WITH BOTTLENECKS
Revised: C. I NFINITE NN S WITH BOTTLENECKS

Original: In inﬁnite NNs

Original: [Equation]

Original:  the kernel is deterministic, meaning that there is no ﬂexibility/variability [Equation] and hence no capability for representation learning (Aitchison, 2019).
Revised: In inﬁnite NNs, the kernel is deterministic, meaning that there is no ﬂexibility/variability, and hence no capability for representation learning (Aitchison, 2019).

Original: Here, we consider inﬁnite networks with bottlenecks that combine the tractability of inﬁnite networks with the ﬂexibility of ﬁnite networks (Aitchison, 2019).
Revised: Here, we consider inﬁnite networks with bottlenecks that combine the tractability of inﬁnite networks with the ﬂexibility of ﬁnite networks (Aitchison, 2019).

Original: The trick is to separate ﬂexible, ﬁnite linear “bottlenecks” from inﬁnite-width nonlinearities.
Revised: The trick is to separate ﬂexible, ﬁnite linear “bottlenecks” from inﬁnite-width nonlinearities.

Original: We keep the nonlinearity inﬁnite in order to ensure that the output kernel is deterministic and can be computed using results from Cho & Saul (2009).
Revised: We keep the nonlinearity inﬁnite in order to ensure that the output kernel is deterministic and can be computed using results from Cho & Saul (2009).

Original: In particular, we use ﬁnite-width
Revised: In particular, we use ﬁnite-width

Original: X
Revised: X

Original: W
Revised: W

Original: F
Revised: F

Original: M
Revised: M

Original: F (cid:48)
Revised: F (cid:48)

Original: W
Revised: W

Original: F
Revised: F

Original: M
Revised: M

Original: F (cid:48)
Revised: F (cid:48)

Original: W
Revised: W

Original: F
Revised: F

Original: Y
Revised: Y

Original: X
Revised: X

Original: K
Revised: K

Original: F
Revised: F

Original: G
Revised: G

Original: F (cid:48)
Revised: F (cid:48)

Original: K
Revised: K

Original: F
Revised: F

Original: G
Revised: G

Original: F (cid:48) K
Revised: F (cid:48) K

Original: F
Revised: F

Original: Y
Revised: Y

Original: X
Revised: X

Original: K
Revised: K

Original: G
Revised: G

Original: K
Revised: K

Original: G
Revised: G

Original: K
Revised: K

Original: F
Revised: F

Original: Y
Revised: Y

Original: X
Revised: X

Original: G G
Revised: G G

Original: F
Revised: F

Original: Y
Revised: Y

Original: F (cid:96) ∈ R P × N (cid:96) and inﬁnite width F (cid:48) (cid:96) ∈ R P × M (cid:96) , (we send M (cid:96) to inﬁnity while leaving N (cid:96) ﬁnite),
Revised: F (cid:96) ∈ R P × N (cid:96) and inﬁnite width F (cid:48) (cid:96) ∈ R P × M (cid:96) , (we send M (cid:96) to inﬁnity while leaving N (cid:96) ﬁnite),

Original: P ( W (cid:96)
Revised: P ( W (cid:96)

Original: [Equation]
Revised: [Equation]

Original: (cid:81)
Revised: (cid:81)

Original: [Equation]
Revised: [Equation]

Original: (cid:0) w (cid:96)λ
Revised: (cid:0) w (cid:96)λ

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation]
Revised: [Equation]

Original: (cid:1)
Revised: (cid:1)

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: , (44a)
Revised: , (48a)

Original: F (cid:96) = (cid:26)
Revised: F (cid:96) = (cid:26)

Original: XW (cid:96) if (cid:96) = 1 , φ ( F (cid:48) (cid:96)
Revised: XW (cid:96) if (cid:96) = 1 , φ ( F (cid:48) (cid:96)

Original: [Equation]
Revised: [Equation]

Original: ) W (cid:96) otherwise , (44b)
Revised: ) W (cid:96) otherwise , (48b)

Original: P (
Revised: P (

Original: [Equation]
Revised: [Equation]

Original: (cid:81)
Revised: (cid:81)

Original: M (cid:96)
Revised: M (cid:96)

Original: [Equation]
Revised: [Equation]

Original: N (cid:0) m (cid:96)λ
Revised: N (cid:0) m (cid:96)λ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: /N (cid:96) (cid:1) , (44c)
Revised: /N (cid:96) (cid:1) , (48c)

Original: [Equation]
Revised: [Equation]

Original: 44d
Revised: 48d

Original: [Equation]
Revised: [Equation]

Original: This generative process is given graphically in Fig.
Revised: This generative process is given graphically in Fig.

Original: 4 (top).
Revised: 4 (top).

Original: Integrating over the expansion weights, M (cid:96)
Revised: Integrating over the expansion weights, M (cid:96)

Original: ∈
Revised: ∈

Original: R N (cid:96) × M (cid:96) , and the bottleneck weights, W (cid:96) ∈ R M (cid:96) − 1 × N (cid:96) , the generative model (Fig. 4 second row) can be rewritten,
Revised: R N (cid:96) × M (cid:96) , and the bottleneck weights, W (cid:96) ∈ R M (cid:96) − 1 × N (cid:96) , the generative model (Fig. 4 second row) can be rewritten,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Remembering that K (cid:96) +1 is the empirical mean of M (cid:96) IID terms, as M (cid:96) → ∞ it converges on its expectation 
Revised: Remembering that K (cid:96) +1 is the empirical mean of M (cid:96) IID terms, as M (cid:96) → ∞ it converges on its expectation [Equation]

Original: [Equation]
Revised: [Equation]

Original: 46) and we deﬁne the limit to be K ( G (cid:96) ) .
Revised: and we deﬁne the limit to be K ( G (cid:96) ) .

Original: Note if we use standard (e.g. ReLU) nonlinearities, we can use results from Cho & Saul (2009) to compute K ( G (cid:96) ) .
Revised: Note if we use standard (e.g. ReLU) nonlinearities, we can use results from Cho & Saul (2009) to compute K ( G (cid:96) ) .

Original: Thus, we get the following generative process,
Revised: Thus, we get the following generative process,

Original: [Equation]
Revised: [Equation]

Original: for (cid:96) = 1 ,
Revised: for (cid:96) = 1 ,

Original: [Equation]
Revised: [Equation]

Original: Finally, eliminating the deterministic kernels, K (cid:96) , from the model, we obtain exactly the deep GP generative model in Eq.
Revised: Finally, eliminating the deterministic kernels, K (cid:96) , from the model, we obtain exactly the deep GP generative model in Eq.

Original: 6 (Fig.
Revised: 6 (Fig.

Original: C.3 fourth row).
Revised: C.3 fourth row).

Original: D S TANDARD APPROXIMATE POSTERIORS OVER FEATURES AND WEIGHTS FAIL TO CAPTURE SYMMETRIES
Revised: D S TANDARD APPROXIMATE POSTERIORS OVER FEATURES AND WEIGHTS FAIL TO CAPTURE SYMMETRIES

Original: We have shown that it is possible to represent DGPs and a variety of NNs as deep kernel processes.
Revised: We have shown that it is possible to represent DGPs and a variety of NNs as deep kernel processes.

Original: Here, we argue that standard deep GP approximate posteriors are seriously ﬂawed, and that working with deep kernel processes may alleviate these ﬂaws.
Revised: Here, we argue that standard deep GP approximate posteriors are seriously ﬂawed, and that working with deep kernel processes may alleviate these ﬂaws.

Original: In particular, we show that the true DGP posterior has rotational symmetries and that the true BNN posterior has permutation symmetries that are not captured by standard variational posteriors.
Revised: In particular, we show that the true DGP posterior has rotational symmetries and that the true BNN posterior has permutation symmetries that are not captured by standard variational posteriors.

Original: D. P ERMUTATION SYMMETRIES IN DNN S POSTERIORS OVER WEIGHTS
Revised: D. P ERMUTATION SYMMETRIES IN DNN S POSTERIORS OVER WEIGHTS

Original: Permutation symmetries in neural network posteriors were known in classical work on Bayesian neural networks (e [Equation] g. MacKay, 1992).
Revised: Permutation symmetries in neural network posteriors were known in classical work on Bayesian neural networks (e.g. MacKay, 1992).

Original: Here, we spell out the argument in full.
Revised: Here, we spell out the argument in full.

Original: Taking P to be a permutation matrix (i [Equation] e. a unitary matrix with PP T
Revised: Taking P to be a permutation matrix (i.e. a unitary matrix with PP T

Original: = I with one 1 in every row and column), we have,[Equation] i
Revised: = I with one 1 in every row and column), we have,

Original: [Equation]
Revised: [Equation]

Original: e. permuting the input to a nonlinearity is equivalent to permuting its output.
Revised: i.e. permuting the input to a nonlinearity is equivalent to permuting its output.

Original: Expanding two steps of the recursion deﬁned by Eq.
Revised: Expanding two steps of the recursion deﬁned by Eq.

Original: (37b),
Revised: (41b),

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  multiplying by the identity,
Revised: 53) multiplying by the identity,

Original: [Equation]
Revised: [Equation]

Original:  applying Eq.
Revised: 54) where P ∈ R N (cid:96) − 1 × N (cid:96) − 1 , applying Eq.

Original: [Equation]
Revised: [Equation]

Original: ) deﬁning permuted weights,
Revised: 55) deﬁning permuted weights,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: the output is the same under the original or permuted weights,
Revised: the output is the same under the original or permuted weights,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Introducing a different perturbation between every pair of layers we get a more general symmetry,
Revised: Introducing a different perturbation between every pair of layers we get a more general symmetry,

Original: [Equation]
Revised: [Equation]

Original: for (cid:96) ∈ { 2 , . . . , L }
Revised: where P (cid:96) ∈ R N (cid:96) − 1 × N

Original: [Equation]
Revised: [Equation]

Original: 54b [Equation] 54c [Equation]
Revised: − 1 .

Original: As the output of the neural network is the same under any of these permutations the likelihoods for original and permuted weights are equal,
Revised: As the output of the neural network is the same under any of these permutations the likelihoods for original and permuted weights are equal,

Original: [Equation]
Revised: [Equation]

Original: [Equation] and as the prior over elements within a weight matrix is IID Gaussian (Eq. 37a), the prior probability density is equal under original and permuted weights,[Equation]
Revised: and as the prior over elements within a weight matrix is IID Gaussian (Eq. 41a), the prior probability density is equal under original and permuted weights,

Original: [Equation]
Revised: [Equation]

Original: Thus, the joint probability is invariant to permutations,
Revised: Thus, the joint probability is invariant to permutations,

Original: [Equation]
Revised: [Equation]

Original: ) and applying Bayes theorem, the posterior is invariant to permutations,
Revised: and applying Bayes theorem, the posterior is invariant to permutations,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Due in part to these permutation symmetries, the posterior distribution over weights is extremely complex and multimodal.
Revised: Due in part to these permutation symmetries, the posterior distribution over weights is extremely complex and multimodal.

Original: Importantly, it is not possible to capture these symmetries using standard variational posteriors over weights, such as factorised posteriors, but it is not necessary to capture these symmetries if we work with Gram matrices and kernels, which are invariant to permutations (and other unitary transformations; Eq. 12).
Revised: Importantly, it is not possible to capture these symmetries using standard variational posteriors over weights, such as factorised posteriors, but it is not necessary to capture these symmetries if we work with Gram matrices and kernels, which are invariant to permutations (and other unitary transformations; Eq. 12).

Original: D. R OTATIONAL SYMMETRIES IN DEEP GP POSTERIORS
Revised: D. R OTATIONAL SYMMETRIES IN DEEP GP POSTERIORS

Original: To show that deep GP posteriors are invariant to unitary transformations, U (cid:96)  , where U (cid:96) U T(cid:96)
Revised: To show that deep GP posteriors are invariant to unitary transformations, U (cid:96) ∈ R N (cid:96) × N (cid:96) , where U (cid:96) U T(cid:96)

Original: =
Revised: =

Original: I , we deﬁne transformed features, F (cid:48) (cid:96) ,F (cid:48)
Revised: I , we deﬁne transformed features, F (cid:48) (cid:96) ,

Original: [Equation]
Revised: [Equation]

Original: To evaluate P
Revised: To evaluate P

Original: [Equation]
Revised: [Equation]

Original: , we begin by substituting for F (cid:48)
Revised: , we begin by substituting for F (cid:48)

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: To evaluate P ( F (cid:48) (cid:96) | F (cid:96) − 1 ) , we substitute for F (cid:48) (cid:96) in the explicit form for the multivariate Gaussian probability density,
Revised: To evaluate P ( F (cid:48) (cid:96) | F (cid:96) − 1 ) , we substitute for F (cid:48) (cid:96) in the explicit form for the multivariate Gaussian probability density,

Original: [Equation]
Revised: [Equation]

Original: where K (cid:96) − 1 =
Revised: where K (cid:96) − 1 =

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original:  , and the constant depends only on F (cid:96) − 1 .
Revised: (cid:17) , and the constant depends only on F (cid:96) − 1 .

Original: Combining these derivations, each of these conditionals is invariant to rotations of F (cid:96) and F (cid:96) − 1 ,
Revised: Combining these derivations, each of these conditionals is invariant to rotations of F (cid:96) and F (cid:96) − 1 ,

Original: [Equation]
Revised: [Equation]

Original: The same argument can straightforwardly be extended to the inputs, P ( F 1 | X
Revised: The same argument can straightforwardly be extended to the inputs, P [Equation] F 1 | X

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and to the ﬁnal probability density
Revised: and to the ﬁnal probability density

Original: [Equation]
Revised: [Equation]

Original: for output activations
Revised: for output activations

Original: [Equation]
Revised: [Equation]

Original: F L +1 which is not invariant to permutations
Revised: F L +1 which is not invariant to permutations

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: applying Bayes theorem the posterior is invariant to rotations,
Revised: applying Bayes theorem the posterior is invariant to rotations,

Original: [Equation]
Revised: [Equation]

Original: Importantly, these posterior symmetries are not captured by standard variational posteriors with non-zero means (e.g. Salimbeni & Deisenroth, 2017).
Revised: Importantly, these posterior symmetries are not captured by standard variational posteriors with non-zero means (e.g. Salimbeni & Deisenroth, 2017).

Original: D. T HE TRUE POSTERIOR OVER FEATURES IN A DGP HAS ZERO MEAN
Revised: D. T HE TRUE POSTERIOR OVER FEATURES IN A DGP HAS ZERO MEAN

Original: We can use symmetry to show that the posterior of F (cid:96) has zero mean.
Revised: We can use symmetry to show that the posterior of F (cid:96) has zero mean.

Original: We begin by writing the expectation as an integral,
Revised: We begin by writing the expectation as an integral,

Original: [Equation]
Revised: [Equation]

Original: Changing variables in the integral to F (cid:48)
Revised: Changing variables in the integral to F (cid:48)

Original: = − F , and noting that the absolute value of the Jacobian is 1 , we have
Revised: = − F , and noting that the absolute value of the Jacobian is 1 , we have

Original: [Equation]
Revised: [Equation]

Original: using the symmetry of the posterior,
Revised: using the symmetry of the posterior,

Original: [Equation]
Revised: [Equation]

Original: the expectation is equal to minus itself, so it must be zero
Revised: the expectation is equal to minus itself, so it must be zero

Original: [Equation]
Revised: [Equation]

Original: E D IFFICULTIES WITH VI IN DEEP W ISHART PROCESSES
Revised: E D IFFICULTIES WITH VI IN DEEP W ISHART PROCESSES

Original: The deep Wishart generative process is well-deﬁned as long as we admit nonsingular Wishart distributions (Uhlig, 1994; Srivastava et al., 2003).
Revised: The deep Wishart generative process is well-deﬁned as long as we admit nonsingular Wishart distributions (Uhlig, 1994; Srivastava et al., 2003).

Original: The issue comes when we try to form a variational approximate posterior over low-rank positive deﬁnite matrices.
Revised: The issue comes when we try to form a variational approximate posterior over low-rank positive deﬁnite matrices.

Original: This is typically the case because the number of datapoints, P is usually far larger than the number of features.
Revised: This is typically the case because the number of datapoints, P is usually far larger than the number of features.

Original: In particular, the only convenient distribution over low-rank positive semideﬁnite matrices is the Wishart itself,
Revised: In particular, the only convenient distribution over low-rank positive semideﬁnite matrices is the Wishart itself,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: However, a key feature of most variational approximate posteriors is the ability to increase and decrease the variance, independent of other properties such as the mean, and in our case the rank of the matrix.
Revised: However, a key feature of most variational approximate posteriors is the ability to increase and decrease the variance, independent of other properties such as the mean, and in our case the rank of the matrix.

Original: For a Wishart, the mean and variance are given by,
Revised: For a Wishart, the mean and variance are given by,

Original: [Equation]
Revised: [Equation]

Original: Initially, this may look ﬁne: we can increase or decrease the variance by changing N (cid:96) .
Revised: Initially, this may look ﬁne: we can increase or decrease the variance by changing N (cid:96) .

Original: However, remember that N (cid:96) is the degrees of freedom, which controls the rank of the matrix, G (cid:96) .
Revised: However, remember that N (cid:96) is the degrees of freedom, which controls the rank of the matrix, G (cid:96) .

Original: As such,
Revised: As such,

Original: N (cid:96) is ﬁxed by the prior: the prior and approximate posterior must deﬁne distributions over matrices of the same rank.
Revised: N (cid:96) is ﬁxed by the prior: the prior and approximate posterior must deﬁne distributions over matrices of the same rank.

Original: And once N (cid:96) is ﬁxed, we no longer have independent control over the variance.
Revised: And once N (cid:96) is ﬁxed, we no longer have independent control over the variance.

Original: To go about resolving this issue, we need to ﬁnd a distribution over low-rank matrices with independent control of the mean and variance.
Revised: To go about resolving this issue, we need to ﬁnd a distribution over low-rank matrices with independent control of the mean and variance.

Original: The natural approach is to use a non-central Wishart, deﬁned as the outer product of Gaussian-distributed vectors with non-zero means.
Revised: The natural approach is to use a non-central Wishart, deﬁned as the outer product of Gaussian-distributed vectors with non-zero means.

Original: While this distribution is easy to sample from and does give independent control over the rank, mean and variance, its probability density is prohibitively costly and complex to evaluate (Koev & Edelman, 2006).
Revised: While this distribution is easy to sample from and does give independent control over the rank, mean and variance, its probability density is prohibitively costly and complex to evaluate (Koev & Edelman, 2006).

Original: F S INGULAR ( INVERSE ) W ISHART PROCESSES AT THE INPUT LAYER
Revised: F S INGULAR ( INVERSE ) W ISHART PROCESSES AT THE INPUT LAYER

Original: In almost all cases of interest, our the kernel functions K ( G ) return full-rank matrices, so we can use standard (inverse) Wishart distributions, which assume that the input matrix is full-rank.
Revised: In almost all cases of interest, our the kernel functions K ( G ) return full-rank matrices, so we can use standard (inverse) Wishart distributions, which assume that the input matrix is full-rank.

Original: However, this is not true at the input layer as K 

Revised: However, this is not true at the input layer as K

Original: = 1 N 0 XX T will often be low-rank.
Revised: = 1 N 0 XX T will often be low-rank.

Original: This requires us to use singular (inverse) Wishart distributions which in general are difﬁcult to work with (Uhlig, 1994;
Revised: This requires us to use singular (inverse) Wishart distributions which in general are difﬁcult to work with (Uhlig, 1994;

Original: Srivastava et al., 2003; Bodnar & Okhrin, 2008; Bodnar et al., 2016).
Revised: Srivastava et al., 2003; Bodnar & Okhrin, 2008; Bodnar et al., 2016).

Original: As such, instead we exploit knowledge of the input features to work with a smaller, full-rank matrix, Ω ∈ R N 0 × N 0 , where, remember, N 0 is the number of input features in X .
Revised: As such, instead we exploit knowledge of the input features to work with a smaller, full-rank matrix, Ω ∈ R N 0 × N 0 , where, remember, N 0 is the number of input features in X .

Original: For a deep Wishart process,
Revised: For a deep Wishart process,

Original: [Equation]
Revised: [Equation]

Original: where Ω ∼ W (cid:16) 1 N 1 I , N(cid:17) ,
Revised: where Ω ∼ W (cid:16) 1 N 1 I , N(cid:17) ,

Original: [Equation]
Revised: [Equation]

Original: 84) and for a deep inverse Wishart process,
Revised: 88) and for a deep inverse Wishart process,

Original: [Equation]
Revised: [Equation]

Original: , where 
Revised: , where Ω ∼

Original: [Equation]
Revised: [Equation]

Original: Now, we are able to use the full-rank matrix, Ω rather than the low-rank matrix, G 1 as the random variable for variational inference.
Revised: Now, we are able to use the full-rank matrix, Ω rather than the low-rank matrix, G 1 as the random variable for variational inference.

Original: For the approximate posterior over Ω , in a deep inverse Wishart process, we use
Revised: For the approximate posterior over Ω , in a deep inverse Wishart process, we use

Original: [Equation]
Revised: [Equation]

Original: Note in the usual case where there are fewer inducing points than input features, then the matrix Kwill be full-rank, and we can work with G 1 as the random variable as usual.
Revised: Note in the usual case where there are fewer inducing points than input features, then the matrix Kwill be full-rank, and we can work with G 1 as the random variable as usual.

Original: G A PPROXIMATE POSTERIORS OVER OUTPUT FEATURES
Revised: G A PPROXIMATE POSTERIORS OVER OUTPUT FEATURES

Original: To deﬁne approximate posteriors over inducing outputs, we are inspired by global inducing point methods (Ober & Aitchison, 2020).
Revised: To deﬁne approximate posteriors over inducing outputs, we are inspired by global inducing point methods (Ober & Aitchison, 2020).

Original: In particular, we take the approximate posterior to be the prior, multiplied by a “pseudo-likelihood”,
Revised: In particular, we take the approximate posterior to be the prior, multiplied by a “pseudo-likelihood”,

Original: [Equation]
Revised: [Equation]

Original: This is valid both for global inducing inputs and (for small datasets) training inputs, and the key thing to remember is that in either case, for any given input (e.g. an MNIST handwritten 2), there is a desired output (e.g. the class-label “2”), and the top-layer global inducing outputs, v λ , express these desired outcomes.
Revised: This is valid both for global inducing inputs and (for small datasets) training inputs, and the key thing to remember is that in either case, for any given input (e.g. an MNIST handwritten 2), there is a desired output (e.g. the class-label “2”), and the top-layer global inducing outputs, v λ , express these desired outcomes.

Original: Substituting for the prior,
Revised: Substituting for the prior,

Original: [Equation]
Revised: [Equation]

Original: and computing this value gives the approximate posterior in the main text (Eq. 19).
Revised: and computing this value gives the approximate posterior in the main text (Eq. 19).

Original: H U SING EIGENVALUES TO COMPARE DEEP W ISHART , DEEP RESIDUAL W ISHART AND INVERSE W ISHART PRIORS
Revised: H U SING EIGENVALUES TO COMPARE DEEP W ISHART , DEEP RESIDUAL W ISHART AND INVERSE W ISHART PRIORS

Original: One might be concerned that the deep inverse Wishart processes in which we can easily perform inference are different to the deep Wishart processes corresponding to BNNs (Sec.
Revised: One might be concerned that the deep inverse Wishart processes in which we can easily perform inference are different to the deep Wishart processes corresponding to BNNs (Sec.

Original: C.1)
Revised: C.1)

Original: and inﬁnite
Revised: and inﬁnite

Original: NNs with bottlenecks (App. C.3).
Revised: NNs with bottlenecks (App. C.3).

Original: To address these concerns, we begin by noting that the (inverse)
Revised: To address these concerns, we begin by noting that the (inverse)

Original: Wishart priors can be written in terms of samples from the standard (inverse) Wishart
Revised: Wishart priors can be written in terms of samples from the standard (inverse) Wishart

Original: [Equation]
Revised: [Equation]

Original: (89) where K = LL T such that,
Revised: (93) where K = LL T such that,

Original: [Equation]
Revised: [Equation]

Original: Note that as the standard Wishart and inverse Wishart have uniform distributions over the eigenvectors (Shah et al., 2014), they differ only in the distribution over eigenvalues of Ω and Ω (cid:48) .
Revised: Note that as the standard Wishart and inverse Wishart have uniform distributions over the eigenvectors (Shah et al., 2014), they differ only in the distribution over eigenvalues of Ω and Ω (cid:48) .

Original: We plotted the eigenvalue histogram for samples from a Wishart distribution with N = P = 2000 (Fig.top left).
Revised: We plotted the eigenvalue histogram for samples from a Wishart distribution with N = P = 2000 (Fig.top left).

Original: This corresponds to an IID Gaussian prior over weights, with 2000 features in the input and output layers.
Revised: This corresponds to an IID Gaussian prior over weights, with 2000 features in the input and output layers.

Original: Notably, there are many very small eigenvalues, which are undesirable as they eliminate information present in the input.
Revised: Notably, there are many very small eigenvalues, which are undesirable as they eliminate information present in the input.

Original: To eliminate these very small eigenvalues, a common approach is to use a ResNet-inspired architecture (which is done even in the deep GP literature, e.g. Salimbeni & Deisenroth, 2017).
Revised: To eliminate these very small eigenvalues, a common approach is to use a ResNet-inspired architecture (which is done even in the deep GP literature, e.g. Salimbeni & Deisenroth, 2017).

Original: To understand the eigenvalues in a residual layer, we deﬁne a
Revised: To understand the eigenvalues in a residual layer, we deﬁne a

Original: Res W distribution by taking the outer product of a weight matrix with itself,
Revised: Res W distribution by taking the outer product of a weight matrix with itself,

Original: [Equation]
Revised: [Equation]

Original: where the weight matrix is IID Gaussian, plus the identity matrix, with the identity matrix weighted as α ,
Revised: where the weight matrix is IID Gaussian, plus the identity matrix, with the identity matrix weighted as α ,

Original: [Equation]
Revised: [Equation]

Original: With α = 1 , there are still many very small eigenvalues, but these disappear as α increases.
Revised: With α = 1 , there are still many very small eigenvalues, but these disappear as α increases.

Original: We compared these distributions to inverse Wishart distributions (Fig. 5 bottom) with varying degrees of freedom.
Revised: We compared these distributions to inverse Wishart distributions (Fig. 5 bottom) with varying degrees of freedom.

Original: For all degrees of freedom, we found that inverse Wishart distributions do not produce very small eigenvalues, which would eliminate information.
Revised: For all degrees of freedom, we found that inverse Wishart distributions do not produce very small eigenvalues, which would eliminate information.

Original: As such, these eigenvalue distributions resemble those for Res W with α larger than 1 .
Revised: As such, these eigenvalue distributions resemble those for Res W with α larger than 1 .

Original: I D OUBLY STOCHASTIC VARIATIONAL INFERENCE IN DEEP INVERSE W ISHART PROCESSES
Revised: I D OUBLY STOCHASTIC VARIATIONAL INFERENCE IN DEEP INVERSE W ISHART PROCESSES

Original: Due to the doubly stochastic results in Sec. 4.3, we only need to compute the conditional distribution over a single test/train point (we do not need the joint distribution over a number of test points).
Revised: Due to the doubly stochastic results in Sec. 4.3, we only need to compute the conditional distribution over a single test/train point (we do not need the joint distribution over a number of test points).

Original: As such, we can decompose G  as,
Revised: As such, we can decompose G and Psi as,

Original: [Equation]
Revised: [Equation]

Original: [Figure] cy
Revised: f r equen cy

Original: W (cid:0) 1 N I , N (cid:1)
Revised: W (cid:0) 1 N I , N (cid:1)

Original: Res W ( N ,1)
Revised: Res W ( N ,1)

Original: Res W ( N ,2)
Revised: Res W ( N ,2)

Original: Res W ( N ,3)
Revised: Res W ( N ,3)

Original: [Figure]
Revised: [Figure]

Original:  r
Revised: f r

Revised: [Figure]

Revised: cy

Original: [Figure]
Revised: [Figure]

Revised: where G (cid:96) ii , Ψ ii ∈ R P i × P

Original: where g (cid:96) it  and ψ it  are column-vectors, and g (cid:96) tt and ψ tt are scalars.
Revised: i , g (cid:96) it ∈ R P i times 1 and ψ it ∈ R P i × 1 are column-vectors, and g (cid:96) tt and ψ tt are scalars.

Original: Taking the results in Eq.
Revised: Taking the results in Eq.

Original: (27) to the univariate case,
Revised: (31) to the univariate case,

Original: [Equation]
Revised: [Equation]

Original: As g (cid:96) tt · i is univariate, its distribution becomes Inverse Gamma,
Revised: As g (cid:96) tt · i is univariate, its distribution becomes Inverse Gamma,

Original: [Equation]
Revised: [Equation]

Revised: InverseGamma

Original: [Equation]
Revised: [Equation]

Original:  As g (cid:96)it is a vector rather than a matrix, its distribution becomes Gaussian,
Revised: [Equation] As g (cid:96)it is a vector rather than a matrix, its distribution becomes Gaussian,

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: J S AMPLES FROM THE 1D PRIOR AND APPROXIMATE POSTERIOR
Revised: J S AMPLES FROM THE 1D PRIOR AND APPROXIMATE POSTERIOR

Original: First, we drew samples from a one-layer (top) and two-layer (bottom) deep inverse Wishart process, with a squared-exponential kernel (Fig. 6).
Revised: First, we drew samples from a one-layer (top) and two-layer (bottom) deep inverse Wishart process, with a squared-exponential kernel (Fig. 6).

Original: We found considerable differences in the function family corresponding to different prior samples of the top-layer Gram matrix, G L (panels).
Revised: We found considerable differences in the function family corresponding to different prior samples of the top-layer Gram matrix, G L (panels).

Original: While differences across function classes in a one-layer IW process can be understood as equivalent to doing inference over a prior on the lengthscale, this is not true of the two-layer process, and to emphasise this, the panels for two-layer samples all have the same ﬁrst layer sample (equivalent to choosing a lengthscale), but different samples from the Gram matrix at the second layer.
Revised: While differences across function classes in a one-layer IW process can be understood as equivalent to doing inference over a prior on the lengthscale, this is not true of the two-layer process, and to emphasise this, the panels for two-layer samples all have the same ﬁrst layer sample (equivalent to choosing a lengthscale), but different samples from the Gram matrix at the second layer.

Original: The two-layer deep
Revised: The two-layer deep

Original: IW process panels use the same, ﬁxed input layer, so variability in the function class arises only from sampling G 2 .
Revised: IW process panels use the same, ﬁxed input layer, so variability in the function class arises only from sampling G 2 .

Original: Next, we exploited kernel ﬂexibilities in IW processes by training a one-layer deep IW model with a ﬁxed kernel bandwidth on data generated from various bandwidths.
Revised: Next, we exploited kernel ﬂexibilities in IW processes by training a one-layer deep IW model with a ﬁxed kernel bandwidth on data generated from various bandwidths.

Original: The ﬁrst row in Figureshows posterior samples from one-layer deep IW processes trained on different datasets.
Revised: The ﬁrst row in Figureshows posterior samples from one-layer deep IW processes trained on different datasets.

Original: For each panel, we ﬁrst sampled ﬁve full G 1 matrices using Eq.(27a) and (27b).
Revised: For each panel, we ﬁrst sampled ﬁve full G 1 matrices using Eq.(31a) and (31b).

Original: Then for each G 1 , we use Gaussian conditioning to get a posterior distribution on testing locations and drew one sample from the posterior plotted as a single line.
Revised: Then for each G 1 , we use Gaussian conditioning to get a posterior distribution on testing locations and drew one sample from the posterior plotted as a single line.

Original: Remarkably, these posterior samples exhibited wiggling behaviours that were consistent with training data even outside the training range, which highlighted the additional kernel ﬂexibility in IW processes.
Revised: Remarkably, these posterior samples exhibited wiggling behaviours that were consistent with training data even outside the training range, which highlighted the additional kernel ﬂexibility in IW processes.

Original: On the other hand, when model bandwidth was ﬁxed, samples from vanilla GPs with ﬁxed bandwidth in the second row displayed almost identical shapes outside the training range across different sets of training data.
Revised: On the other hand, when model bandwidth was ﬁxed, samples from vanilla GPs with ﬁxed bandwidth in the second row displayed almost identical shapes outside the training range across different sets of training data.

Original: [Figure]
Revised: [Figure]

Original: y 4 2 0 2x - -    y -4 -2 0 2x -4 -2 0 2x -4 -2 0 2x -4 -2 0 2x
Revised: y 4 2 0 2x - -    y -4 -2 0 2x -4 -2 0 2x -4 -2 0 2x -4 -2 0 2x

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: K W HY WE CARE ABOUT THE ELBO
Revised: K W HY WE CARE ABOUT THE ELBO

Original: While we have shown that DIWP offers some beneﬁts in predictive performance, it gives much more dramatic improvements in the ELBO.
Revised: While we have shown that DIWP offers some beneﬁts in predictive performance, it gives much more dramatic improvements in the ELBO.

Original: While we might think that predictive performance is the only goal, there are two reasons to believe that the ELBO itself is also an important metric.
Revised: While we might think that predictive performance is the only goal, there are two reasons to believe that the ELBO itself is also an important metric.

Original: First, the ELBO is very closely related to PAC-Bayesian generalisation bounds (e.g. Germain et al., 2016).
Revised: First, the ELBO is very closely related to PAC-Bayesian generalisation bounds (e.g. Germain et al., 2016).

Original: In particular, the bounds are generally written as the average training log-likelihood, plus the
Revised: In particular, the bounds are generally written as the average training log-likelihood, plus the

Original: KL-divergence between the approximate posterior over parameters and the prior.
Revised: KL-divergence between the approximate posterior over parameters and the prior.

Original: This mirrors the standard form for the ELBO,
Revised: This mirrors the standard form for the ELBO,

Original: [Equation]
Revised: [Equation]

Original: where x is all the data (here, the inputs, X and outputs, Y ), and z are all the latent variables.
Revised: where x is all the data (here, the inputs, X and outputs, Y ), and z are all the latent variables.

Original: Remarkably, Germain et al.
Revised: Remarkably, Germain et al.

Original: (e.g. 2016) present a bound on the test-log-likelihood that is exactly the ELBO per data point, up to additive constants.
Revised: (e.g. 2016) present a bound on the test-log-likelihood that is exactly the ELBO per data point, up to additive constants.

Original: As such, in certain circumstances, optimizing the
Revised: As such, in certain circumstances, optimizing the

Original: ELBO is equivalent to optimizing a PAC-Bayes bound on the test-log-likelihood.
Revised: ELBO is equivalent to optimizing a PAC-Bayes bound on the test-log-likelihood.

Original: Similar results are available in Rivasplata et al.
Revised: Similar results are available in Rivasplata et al.

Original: Second, we can write down an alternative form for the ELBO as the model evidence, minus the KL-divergence between the approximate and true posterior,
Revised: Second, we can write down an alternative form for the ELBO as the model evidence, minus the KL-divergence between the approximate and true posterior,

Original: [Equation]
Revised: [Equation]

Original: As such, for a ﬁxed generative model, and hence a ﬁxed value of the model evidence, log P ( x ) , the ELBO measures the closeness of the variational approximate posterior, Q ( z ) and the true posterior, P ( z | x ) .
Revised: As such, for a ﬁxed generative model, and hence a ﬁxed value of the model evidence, log P ( x ) , the ELBO measures the closeness of the variational approximate posterior, Q ( z ) and the true posterior, P ( z | x ) .

Original: As we are trying to perform Bayesian inference, our goal should be to make the approximate posterior as close as possible to the true posterior.
Revised: As we are trying to perform Bayesian inference, our goal should be to make the approximate posterior as close as possible to the true posterior.

Original: If, for instance, we can set Q ( z ) to givea better predictive performance, but be further from the true posterior, then that is ﬁne in certain settings, but not when the goal is inference.
Revised: If, for instance, we can set Q ( z ) to give better predictive performance, but be further from the true posterior, then that is ﬁne in certain settings, but not when the goal is inference.

Original: Obviously, it is desirable for the true and approximate posterior to be as close as possible, which corresponds to larger values of L (indeed, when the approximate posterior equals the true posterior, the KL-divergence is zero, and L = log P ( x ) ).
Revised: Obviously, it is desirable for the true and approximate posterior to be as close as possible, which corresponds to larger values of L (indeed, when the approximate posterior equals the true posterior, the KL-divergence is zero, and L = log P ( x ) ).

Original: L D IFFERENCES WITH S HAH ET AL . (2014)
Revised: L D IFFERENCES WITH S HAH ET AL . (2014)

Original: For a one-layer deep inverse Wishart process, using our deﬁnition in Eq.
Revised: For a one-layer deep inverse Wishart process, using our deﬁnition in Eq.

Original: [Equation]
Revised: [Equation]

Original:  Importantly, we do the nonlinear kernel transformation after sampling the inverse Wishart, so the inverse-Wishart sample acts as a generalised lengthscale hyperparameter (App. B), and hence dramatically changes the function family.
Revised: 104a [Equation] Importantly, we do the nonlinear kernel transformation after sampling the inverse Wishart, so the inverse-Wishart sample acts as a generalised lengthscale hyperparameter (App. B), and hence dramatically changes the function family.

Original: In contrast, for Shah et al.
Revised: In contrast, for Shah et al.

Original: (2014), the nonlinear kernel is computed before , the inverse Wishart is sampled, and the inverse Wishart sample is used directly as the covariance for the Gaussian,
Revised: (2014), the nonlinear kernel is computed before , the inverse Wishart is sampled, and the inverse Wishart sample is used directly as the covariance for the Gaussian,

Original: [Equation]
Revised: [Equation]

Original: This difference in ordering, and in particular, the lack of a nonlinear kernel transformation between the inverse-Wishart and the output is why Shah et al.
Revised: This difference in ordering, and in particular, the lack of a nonlinear kernel transformation between the inverse-Wishart and the output is why Shah et al.

Original: (2014) were able to ﬁnd trivial results in their model (that it is equivalent to multiplying the covariance by a random scale).
Revised: (2014) were able to ﬁnd trivial results in their model (that it is equivalent to multiplying the covariance by a random scale).