G AUSSIAN P ROCESS B EHAVIOUR IN W IDE D EEP N EURAL N ETWORKS
A BSTRACT
Whilst deep neural networks have shown great empirical success, there is still much work to be done to understand their theoretical properties.
In this paper, we study the relationship between Gaussian processes with a recursive kernel deﬁnition and random wide fully connected feedforward networks with more than one hidden layer.
We exhibit limiting procedures under which ﬁnite deep networks will converge in distribution to the corresponding Gaussian process.
To evaluate convergence rates empirically, we use maximum mean discrepancy.
We then exhibit situations where existing Bayesian deep networks are close to Gaussian processes in terms of the key quantities of interest.
Any Gaussian process has a ﬂat representation.
Since this behaviour may be undesirable in certain situations we discuss ways in which it might be prevented.
I NTRODUCTION
Deep feedforward neural networks have emerged as an essential component of modern machine learning.
As such there has been signiﬁcant research effort in trying to understand the theoretical properties of such models.
One important branch of such research is the study of random networks.
By assuming a probability distribution on the network parameters, a distribution is induced on the input to output function that such networks encode.
This has proved important in the study of initialisation and learning dynamics (Schoenholz et al., 2017) and expressivity (Poole et al., 2016).
It is, of course, essential in the study of Bayesian priors on networks (Neal, 1996).
The Bayesian approach makes little sense if prior assumptions are not understood, and distributional knowledge can be essential in ﬁnding good posterior approximations.
Since we typically want our networks to have high modelling capacity, it is natural to consider limit distributions of networks as they become large.
Whilst distributions on deep networks are generally challenging to work with exactly, the limiting behaviour can lead to more insight.
Further, as we shall see, networks used in the literature may be very close to this behaviour.
The seminal work in this area is that of Neal (1996), which showed that under certain conditions random neural networks with one hidden layer converge to a Gaussian process.
The question of the type of convergence is non-trivial and part of our discussion.
Historically this result was a signiﬁcant one because it provided a connection between ﬂexible Bayesian neural networks and
Gaussian processes (Williams, 1998; Rasmussen & Williams, 2006)
1. O UR CONTRIBUTIONS
We extend the theoretical understanding of random fully connected networks and their relationship to Gaussian processes.
In particular, we prove a rigorous result (Theorem 1) on the convergence of certain ﬁnite networks with more than one hidden layer to Gaussian processes.
Further, we empirically study the distance between ﬁnite networks and their Gaussian process analogues by using maximum mean discrepancy (Gretton et al., 2012) as a distance measure.
We ﬁnd that Bayesian deep networks from the literature can exhibit predictions that are close to Gaussian processes.
To demonstrate this, we systematically compare exact Gaussian process inference with ‘gold standard’ MCMC inference for Bayesian neural networks.
Our work is of relevance to the theoretical understanding of neural network initialisation and dynamics.
It is also important in the area of Bayesian deep networks because it demonstrates that Gaussian process behaviour can arise in more situations of practical interest than previously thought.
If this behaviour is desired then Gaussian process inference (exact and approximate) should also be considered.
In some scenarios, the behaviour may not be desired because it implies a lack of a hierarchical representation.
We therefore highlight promising ideas from the literature to prevent such behaviour.
1. R ELATED WORK
The case of random neural networks with one hidden layer was studied by Neal (1996).
Cho & Saul (2009) provided analytic expressions for single layer kernels including those corresponding to a rectiﬁed linear unit (ReLU).
They also studied recursive kernels designed to ‘mimic computation in large, multilayer neural nets’.
As discussed in Section 3 they arrived at the correct kernel recursion through an erroneous argument.
Such recursive kernels were later used with empirical success in the
Gaussian process literature (Krauth et al., 2017), with a similar justiﬁcation to that of Cho and Saul.
The ﬁrst case we are aware of using a Gaussian process construction with more than one hidden layer is the work of Hazan & Jaakkola (2015).
Their contribution is similar in content to Lemma 1 discussed here, and the work has had increasing interest from the kernel community (Mitrovic et al., 2017).
Recent work from Daniely et al.
(2016) uses the concept of ‘computational skeletons’ to give concentration bounds on the difference in the second order moments of large ﬁnite networks and their kernel analogue, with strong assumptions on the inputs.
The Gaussian process view given here, without strong input assumptions, is related but concerns not just the ﬁrst two moments of a random network but the full distribution.
As such the theorems we obtain are distinct.
A less obvious connection is to the recent series of papers studying deep networks using a mean ﬁeld approximation (Poole et al., 2016; Schoenholz et al., 2017).
In those papers a second order approximation gives equivalent behaviour to the kernel recursion.
By contrast, in this paper the claim is that the behaviour emerges as a consequence of increasing width and is therefore something that needs to be proved.
Another surprising connection is to the analysis of self-normalizing neural networks (Klambauer et al., 2017).
In their analysis the authors assume that the hidden layers are wide in order to invoke the central limit theorem.
The premise of the central limit theorem will only hold approximately in layers after the ﬁrst one and this theoretical barrier is something we discuss here.
An area that is less related than might be expected is that of ‘Deep Gaussian Processes’ (DGPs) (Damianou &
Lawrence, 2013).
As will be discussed in Section 6, narrow intermediate representations mean that the marginal behaviour is not close to that of a Gaussian process.
Duvenaud et al.
(2014) offer an analysis that largely applies to DGPs though they also study the Cho and Saul recursion with the motivating argument from the original paper.
T HE DEEP WIDE LIMIT
We consider a fully connected network as shown in Figure 1.
The inputs and outputs will be real valued vectors of dimension M and L respectively.
The network is fully connected.
The initial step and recursion are standard.
The initial step is:
[Equation]
[Figure]
We make the functional dependence on x explicit in our notation as it will help clarify what follows.
For a network with D hidden layers the recursion is, for each µ = 1 , . . .
, D ,
[Equation]
so that f ( D +1) ( x ) is the output of the network given input x .
φ denotes the non-linearity.
In all cases the equations hold for each value of i ; i ranges between 1 and H µ in Equation (2), and betweenand H µ +1 in Equation (3) except in the case of the ﬁnal activation where the top value is L .
The network could of course be modiﬁed to be probability simplex-valued by adding a softmax at the end.
A distribution on the parameters of the network will be assumed.
Conditional on the inputs, this induces a distribution on the activations and activities.
In particular we will assume independent normal distributions on the weights and biases
[Equation]
We will be interested in the behaviour of this network as the widths H µ becomes large.
The weight variances for µ ≥ 2 will be scaled according to the width of the network to avoid a divergence in the variance of the activities in this limit.
As will become apparent, the appropriate scaling is
[Equation]
The assumption is that ˆ C ( µ ) w will remain ﬁxed as we take the limit.
Neal (1996) analysed this problem for D = 1 , showing that as H 1 → ∞ , the values of f (2) i ( x ) , the output of the network in this case, converge to a certain multi-output Gaussian process if the activities have bounded variance.
Since our approach relies on the multivariate central limit theorem we will arrange the relevant terms into (column) vectors to make the linear algebra clearer.
Consider any two inputs x and x (cid:48) and all output functions ranging over the index i .
We deﬁne the vector f (2) ( x ) of length L whose elements are the numbers f (2) i ( x ) .
We deﬁne f (2) ( x (cid:48) ) similarly.
For the weight matrices deﬁned by w ( µ ) i,j for ﬁxed µ we use a ‘placeholder’ index • to return column and row vectors from the weight matrices.
In particular w (1) j, • denotes row j of the weight matrix at depth 1 .
Similarly, w (2) • ,j denotes column j at depth 2 .
The biases are given as column vectors b (1) and b (2) .
Finally we concatenate the two vectors f (2) ( x ) and f (2) ( x (cid:48) ) into a single column vector F (2) of size 2 L .
The vector in question takes the form
[Equation]
The beneﬁt of writing the relation in this form is that the applicability of the multivariate central limit theorem is immediately apparent.
Each of the vector terms on this right hand side is independent and identically distributed conditional on the inputs x and x (cid:48) .
By assumption, the activities have bounded variance.
The scaling we have chosen on the variances is precisely that required to ensure the applicability of the theorem.
Therefore as H becomes large F (2) converges in distribution to a multivariate normal distribution.
The limiting normal distribution is fully speciﬁed by its ﬁrst two moments.
Deﬁning γ ∼ N (0 , C (1) b ) , [Equation] N [Equation]
[Equation]
C
[Equation]
1) w I M ) , the moments in question are:
[Equation] Note that we could have taken a larger set of input points to give a larger vector F and again we would conclude that this vector converged in distribution to a multivariate normal distribution.
More formally, we can consider the set of possible inputs as an index set .
A set of consistent ﬁnite dimensional Gaussian distributions on an index set corresponds to a Gaussian process by the Kolmogorov extension theorem.
The Gaussian process in question is a distribution over functions deﬁned on the product σ -algebra, which has the relevant ﬁnite dimensional distributions as its marginals.
In the case of a multivariate normal distribution a set of variables having a covariance of zero implies that the variables are mutually independent.
Looking at Equation (9), we see that the limiting distribution has independence between different components i, j of the output.
Combining this with the recursion (2), we might intuitively suggest that the next layer also converges to a multivariate normal distribution in the limit of large H µ .
Indeed we state the following lemma, which we attribute to
Hazan & Jaakkola (2015):
Lemma 1 (Normal recursion) .
If the activations of a previous layer are normally distributed with moments:
[Equation]
Then under the recursion (2) and as H → ∞ the activations of the next layer converge in distribution to a normal distribution with moments
[Equation]
[Equation]
where K is a 2 × 2 matrix containing the input covariances.
Unfortunately the lemma is not sufﬁcient to show that the joint distribution of the activations of higher layers converge in distribution to a multivariate normals.
This is because for ﬁnite H the input activations do not have a multivariate normal distribution - this is only attained (weakly or in distribution) in the limit.
It could be the case that the rate at which the limit distribution is attained affects the distribution in subsequent layers.
We are able to offer the following theorem rigorously:
Theorem 1.
Consider a Bayesian deep neural network of the form in Equations (1) and (2) using
ReLU activation functions.
Then there exist strictly increasing width functions h µ : N (cid:55)→ N such that H
= h 1 ( n ) , . . .
, H D = h D ( n ) , and for any countable input set ( x [ i ]) ∞
i =1 , the distribution of the output of the network converges in distribution to a Gaussian process as n → ∞ .
A proof is included in the appendix.
We conjecture that a more general theorem will hold.
In particular we expect that the width functions h µ can be taken to be the identity and that the nonlinearity can be extended to monotone functions with well behaved tails.
Our conjecture is based on the intuition from Lemma 1 and from our experiments, in which we always take the width function to be the identity.
S PECIFIC KERNELS UNDER RECURSION
Cho & Saul (2009) suggest a family of kernels based on a recurrence designed to ‘mimic computation in large, multilayer neural nets’.
It is therefore of interest to see how this relates to deep wide Gaussian processes.
A kernel may be associated with a feature mapping Φ( x ) such that
K ( x, x (cid:48) )
=
Φ( x ) • Φ( x (cid:48) ) .
Cho and Saul deﬁne a recursive kernel through a new feature mapping by compositions such as Φ(Φ( x )) .
However this cannot be a legitimate way to create a kernel because such a composition represents a type error.
There is no reason to think the output dimension of the function Φ matches the input dimension and indeed the output dimension may well be inﬁnite.
Nevertheless, the paper provides an elegant solution to a different task: it derives closed form solution to the recursion from Lemma 1 (Hazan & Jaakkola, 2015) for the special case
[Equation]
where Θ is the Heaviside step function.
Speciﬁcally, the recursive approach of Cho & Saul (2009) can be adapted by using the fact that u (cid:62) z for z ∼ N (0 , LL (cid:62) ) is equivalent in distribution to ( L (cid:62) u ) (cid:62) ε with ε ∼ N (0 , I ) , and by optionally augmenting u to incorporate the bias.
Since r =corresponds to rectiﬁed linear units we apply this analytic kernel recursion in all of our experiments.
M EASURING CONVERGENCE USING MAXIMUM MEAN DISCREPANCY
In this section we use the kernel based two sample tests of Gretton et al.
(2012) to empirically measure the similarity of ﬁnite random neural networks to their Gaussian process analogues.
The maximum mean discrepancy (MMD) between two distributions P and Q is deﬁned as:
[Equation]
(15) where H denotes a reproducing kernel Hilbert space and || •
|| H denotes the corresponding norm.
It gives the biggest possible difference between expectations of a function under the two distributions under the constraint that the function has Hilbert space norm less than or equal to one.
We used the unbiased estimator of squared MMD given in Equation (3) of Gretton et al.
In this experiment and all those that follow we take weight variance parameters ˆ C ( µ ) w = 0 .
8 and bias variance C b = 0 .
We took 10 standard normal input points in 4 dimensions and pass them through 2000 independent random neural networks drawn from the distribution discussed in this paper.
This was then compared to 2000 samples drawn from the corresponding Gaussian process distribution.
The experiment was performed with different numbers of hidden layers and numbers of units per hidden layer.
We repeated each experiment 20 times which allows us to reduce variance in our results and give a simple estimate of measurement error.
The experiments use an RBF kernel for the MMD [Figure] estimate with lengthscale 1 / 2 .
In order to help give an intuitive sense of the distances involved we also include a comparison between two Gaussian processes with isotropic RBF kernels using the same MMD distance measure.
The kernel length scales for this pair of ‘calibration’ Gaussian processes are taken to be l and 2 l , where the characteristic length scale l = √ 8 is chosen to be sensible for the standard Normal input distribution on the four dimensional space.
The results of the experiment are shown in Figure 2.
We see that for each ﬁxed depth the network converges towards the corresponding Gaussian process as the width increases.
For the same number of hidden units per layer, the MMD distance between the networks and their Gaussian process analogue becomes higher as depth increases.
The rate of convergence to the Gaussian process is slower as the number of hidden layers is increased.
C OMPARING B AYESIAN D EEP N ETWORKS TO G AUSSIAN P ROCESSES
In this section we compare the behaviour of ﬁnite Bayesian deep networks of the form considered in this paper with their Gaussian process analogues.
If we make the networks wide enough the agreement will be very close.
It is also of interest, however, to consider the behaviour of networks actually used in the literature, so we use 3 hidden layers and 50 hidden units which is typical of the networks used by Hern´andez-Lobato & Adams (2015).
Fully connected Bayesian deep networks with ﬁnite variance priors on the weights have also been considered in other works (Graves, 2011;
Hern´andez-Lobato et al., 2016; Blundell et al., 2015), though the speciﬁc details vary.
We use rectiﬁed linear units and correct the variances to avoid a loss of prior variance as depth is increased.
Our general strategy was to compare exact Gaussian process inference against expensive ‘gold standard’ Markov Chain Monte Carlo (MCMC) methods.
We choose the latter because used correctly it works well enough to largely remove questions of posterior approximation quality from the calculus of comparison.
It does mean however that our empirical study does not extend to datasets which are large in terms of number of data points or dimensionality, where such inference is challenging.
We therefore sound a note of caution about extrapolating our empirical ﬁnite network conclusions too conﬁdently to this domain.
On the other hand, lower dimensional, prior dominated problems are generally regarded as an area of strength for Bayesian approaches and in this context our results are directly relevant.
We computed the posterior moments by the two different methods on some example datasets.
For the
MCMC we used Hamiltonian Monte Carlo (HMC) (Neal, 2010) updates interleaved with elliptical slice sampling (Murray et al., 2010).
We considered a simple one dimensional problem and a two dimensional real valued embedding of the four data point XOR problem.
We see in Figures 3 and(left) that the agreement in the posterior moments between the Gaussian process and the Bayesian deep network is very close.
A key quantity of interest in Bayesian machine learning is the marginal likelihood.
It is the normalising constant of the posterior distribution and gives a measure of the model ﬁt to the data.
For a
Bayesian neural network, it is generally very difﬁcult to compute, but with care and computational time it can be approximated using Hamiltonian annealed importance sampling (Sohl-Dickstein &
Culpepper, 2012).
The log-importance weights attained in this way constitute a stochastic lower bound on the marginal likelihood (Grosse et al., 2015).
Figure 4 (right) shows the result of such an experiment compared against the (extremely cheap) Gaussian process marginal likelihood computation on the XOR problem.
The value of the log-marginal likelihood computed in the two different ways agree to within a single nat which is negligible from a model selection perspective (Grosse et al., 2015).
Predictive log-likelihood is a measure of the quality of probabilistic predictions given by a Bayesian regression method on a test point.
To compare the two models we sampled 10 standard normal train and test points in 4 dimensions and passed them through a random network of the type under study to get regression targets.
We then discarded the true network parameters and compared the predictions of posterior inference between the two methods.
We also compared the marginal predictive distributions of a latent function value.
Figure 5 shows the results.
We see that the correspondence in predictive log-likelihood is close but not exact.
Similarly the marginal function values are close to those of a Gaussian process but are slightly more concentrated.
[Figure]
[Table]
[Figure]
[Table] [Figure] [Figure] [Figure]
[Figure]
N e u r a l n e t w o r k l o g d e n s i t y [Figure] [Figure] [Figure]
[Figure]
[Figure]
D e n s i t y  A VOIDING G AUSSIAN PROCESS BEHAVIOUR When using deep Bayesian neural networks as priors, the emergence of Gaussian priors raises important questions in the cases where it is applicable, even if one sets aside questions of computational tractability.
It has been argued in the literature that there are important cases where kernel machines with local kernels will perform badly (Bengio et al., 2005).
The analysis applies to the posterior mean of a Gaussian process.
The emergent kernels in our case are hyperparameter free.
Although they do not meet the strict deﬁnition of what could be considered ‘local’ the fact remains that any
Gaussian process with a ﬁxed kernel does not use a learnt hierarchical representation.
Such representations are widely regarded to be essential to the success of deep learning.
There is relevant literature here on learning the representation of a standard, usually structured, network composed with a Gaussian process (Wilson et al., 2016a;b; Al-Shedivat et al., 2017).
This differs from the assumed paradigm of this paper, where all model complexity is speciﬁed probabilistically and we do not assume convolutional, recurrent or other problem speciﬁc structure.
Within this paradigm, the question therefore arises as to what can be done to avoid marginal Gaussian process behaviour if it is not desired.
Speaking loosely, to stop the onset of the central limit theorem and the approximate analogues discussed in this paper one needs to make sure that one or more of its conditions is far from being met.
Since the chief conditions on the summands are independence, bounded variance and many terms, violating these assumptions will remove Gaussian process behaviour.
Deep Gaussian processes (Damianou & Lawrence, 2013) are not close to standard Gaussian processes marginally because they are typically used with narrow intermediate layers.
It can be challenging to choose the precise nature of these narrow layers a priori.
Neal (1996) suggests using networks with inﬁnite variance in the activities.
With a single hidden layer and correctly scaled, these networks become alpha stable processes in the wide limit.
Neal also discusses variants that destroy independence by coupling weights.
Our results about the emergence of Gaussian processes even with more than one hidden layer mean these ideas are of considerable interest going forward.
C ONCLUSIONS
Studying the limiting behaviour of distributions on feedforward networks has been a fruitful avenue for understanding these models historically.
In this paper we have extended the state of knowledge about the wide limit, including for networks with more than one hidden layer.
In particular, we have exhibited limit sequences of networks that converge in distribution to Gaussian processes with a certain recursively deﬁned kernel.
Our empirical study using MMD suggests that this behaviour is exhibited in a variety of models of size comparable to networks used in the literature.
This led us to juxtapose ﬁnite Bayesian neural networks with their Gaussian process analogues, ﬁnding that the agreement in terms of key predictors is close empirically.
If this Gaussian process behaviour is desired then exact and approximate inference using the analytic properties of Gaussian processes should be considered as an alternative to neural network inference.
Since Gaussian processes have an equivalent ﬂat representation then in the context of deep learning the behaviour may well not be desired and steps should be taken to avoid it.
We view these results as a new opportunity to further the understanding of neural networks in the work that follows.
Initialisation and learning dynamics are crucial topics of study in modern deep learning which require that we understand random networks.
Bayesian neural networks should offer a principled approach to generalisation but this relies on successfully approximating a clearly understood prior.
In illustrating the continued importance of Gaussian processes as limit distributions, we hope that our results will further research in these broader areas.
A CKNOWLEDGEMENTS
We wish to thank Neil Lawrence for helpful conversations.
We also thank the anonymous reviewers for their insights.
Alexander Matthews and Zoubin Ghahramani acknowledge the support of EPSRC
Grant EP/N014162/1 and EPSRC Grant EP/N510129/1 (The Alan Turing Institute).
Jiri Hron holds a Nokia CASE Studentship. Mark Rowland acknowledges support by EPSRC grant EP/L016516/ for the Cambridge Centre for Analysis. Richard E. Turner is supported by Google as well as EPSRC grants EP/M0269571 and EP/L000776/1. A P ROOF OF M AIN T HEOREM A. S TATEMENT OF THEOREM AND NOTATION
In this section, we provide a proof of the main theorem of the paper, which we begin by recalling.
Theorem 1.
Consider a Bayesian deep neural network of the form in Equations (1) and (2) using
ReLU activation functions.
Then there exist strictly increasing width functions h µ : N (cid:55)→ N such that H
= h 1 ( n ) , . . .
, H D = h D ( n ) , and for any countable input set ( x [ i ]) ∞
i =1 , the distribution of the output of the network converges in distribution to a Gaussian process as n → ∞ .
The theorem is proven via use of the propositions that follow below.
The broad structure of the proof is to use a particular variant of the Berry-Esseen inequality to upper bound how far each layer is from a multivariate normal distribution, and then to inductively propagate these inequalities through the network, leading to a bound on the distance between the output of the network for a collection of input points, and a multivariate Gaussian distribution.
These notions will be made precise below.
We begin in Section A.2 by stating the propositions that will be used in the proof of Theorem 1, but ﬁrst establish notation that will be used in the remainder of the appendix.
Given a ﬁnite set of inputs x [1] , . . .
, x [ n ] ∈ R M , we will write:
• f ( µ ) ( x ) for the random variables ( f ( µ ) ( x [ i ]))
ni =1 collectively taking values in R nH µ ; • f ( µ ) j ( x ) for the random variables ( f ( µ ) j ( x [ i ]))
ni =1 collectively taking values in R n ; • g ( µ ) ( x ) for the random variables ( g ( µ ) ( x [ i ]))
ni =1 collectively taking values in R nH µ ; • g ( µ ) j ( x ) for the random variables ( g ( µ ) j ( x [ i ]))
ni =1 collectively taking values in R n ;
Throughout, if U 1 , U 2 are random variables taking in values in some Euclidean space R d , we will deﬁne
[Equation]
Note that convergence of a sequence of random variables in this metric implies convergence in distribution.
We will also consider multivariate normal distributions ( Z ( µ )
j
( x [ i ]) | j = 1 , . .
, H µ , i = 1 , . . . , n ) with covariance matrices of block diagonal form, such that
[Equation]
To avoid writing this in full every time it is required, we will refer to this condition as blockwise independence with respect to the index j .
We will avoid speciﬁcation of all covariance values, deferring to the expression (13) given in the main paper.
Finally, to simplify notation, we will assume that the network output is one-dimensional.
Our proof trivially extends to arbitrary ﬁnite output dimension where the limiting distribution is a coordinate-wise independent multivariate GP.
A. S UPPORTING RESULTS
Proposition 1.
Let ε > 0 , and x [1] , . . .
, x [ n ] ∈ R M .
Let µ ∈ { 2 , . .
, D + 1 } , and let H k
= 2 H 2 k
+ for k = 1 , . . .
, D − 1 .
Then for H D sufﬁciently large, suppose the condition
[Equation]
holds, where Z ( µ − 1) ( x ) =
( Z ( µ − 1)
j
( x [ i ]) | j = 1 , . . .
H µ − 1 , i = 1 , . .
. , n ) is mean-zero multivariate normal, with blockwise independence with respect to the index j .
Then we have
[Equation]
where Z ( µ ) ( x ) =
( Z ( µ ) j
( x [ i ]) | j = 1 , . .
, H µ , i = 1 , . . . , n ) is mean-zero multivariate normal, with blockwise independence with respect to the index j .
Proposition 2.
Let ε > 0 , and x [1] , . . .
, x [ n ] ∈ R M .
If H k = 2 H 2 k +1 for k = 1 , . . .
, D − 1 , then for
H D sufﬁciently large, we have
[Equation]
where Z ( x ) is a mean-zero multivariate normal random variable.
In establishing the two propositions above, the following three lemmas will be useful.
Lemma 2.
Let ε > 0 , and let Z ( µ − 1) ( x ) =
( Z ( µ − 1) j
( x [ i ]) | j = 1 , . . . ,
[Equation]
[Equation]
=
[Equation]
, . . . , n ) be mean-zero multivariate normal, with blockwise independence with respect to the index j .
Let (cid:101) g ( µ − 1) ( x ) =
φ ( Z ( µ − 1) ( x )) , and let (cid:101) f ( µ ) ( x ) be given by
[Equation]
for i = 1 , . . . , n .
Then given ε > 0 , if H k = 2 H 2 k +1 for k = 1 , . .
, D − 1 , then for all sufﬁciently large H D we have:
[Equation]
where Z ( µ ) ( x ) =
( Z ( µ ) j
( x [ i ]) | j = 1 , . .
, H µ , i = 1 , . . . , n ) is mean-zero multivariate normal, with blockwise independence with respect to the index j .
Lemma 3.
Let Z ( µ − 1) ( x ) =
( Z ( µ − 1)
j
( x [ i ]) | j = 1 , . . .
, H µ − 1 , 1 , . .
. , n ) be mean-zero multivariate normal, with blockwise independence with respect to the index j , such that for some ε > 0 , d [Equation] ≤
[Equation]
Then, deﬁning (cid:101) f ( µ ) ( x ) by
[Equation]
in the particular case where φ is the elementwise ReLU function, we have
[Equation]
Lemma 4.
Let X 1 , . . .
, X H µ − 1 be iid random variables of the form X j = (cid:101) g ( µ − [Equation] j
[Equation] x [Equation] ⊗ (cid:101)
[Equation] [Equation] j , where ⊗ denotes the Kronecker product, (cid:101) g ( µ − 1) j ( x ) is deﬁned as in Lemma 2, and (cid:101) w ( µ ) • ,j is a multivariate normal variable taking values in R H µ with mean vector 0, and covariance ˆ C ( µ ) w I .
We denote the variance of X j by Σ ⊗ and its Schur decomposition as Σ ⊗ = Q ⊗ Λ ⊗ Q T ⊗ .
Then β = E (cid:104) (cid:107) Q ⊗ Λ − 1 / 2 ⊗
Q T ⊗ X j (cid:107) 3 (cid:105) ≤ C H µ ,n , where C H µ ,n ∈ R depends on H µ and n , but is independent of H µ − 1 .
Further, we have C H µ ,n =
O ( H 2 µ n 2 ) .
A. P ROOFS
Proof of Lemma 2.
We use a straightforward variant of a particular Berry-Esseen inequality described in Bentkus (2003).
We ﬁrst state this result from the literature, and then derive a straightforward variation that we will use in the sequel.
Theorem 2 (From Bentkus (2003)) .
Let X 1 , . . .
, X n be iid random variables taking values in R d , with mean vector 0, identity covariance matrix, and β = E (cid:2) (cid:107)
X i (cid:107) 3 (cid:3) < ∞ .
Let S n = 1 √ n (cid:80) ni
=
X i , and let Y be a standard d -dimensional multivariate normal random vector.
Then we have sup
[Equation]
We need a mildly modiﬁed version of this theorem to deal with iid random vectors X 1 , . . .
X n with non-identity covariance matrices.
To this end, suppose that Σ is the (full-rank) covariance matrix of each X i , with decomposition Σ = RR (cid:62) , for some invertible matrix R ; R can be obtained, for example, by using Cholesky or Schur decomposition.
The random variables R − 1 X 1 , . .
, R − 1 X n are then iid, mean zero and with identity covariance matrices, so we may apply Theorem 2 to obtain:
[Equation]
where β = E (cid:2) (cid:107) R − 1 X i (cid:107) 3 (cid:3) .
Now note that this is equivalent to
[Equation]
noting that RY ∼ N (0 , Σ) .
Since R is invertible, and recalling the deﬁnition of the distance d above, this is exactly equivalent to:
[Equation]
which is the variant of Bentkus’ result we will require in the sequel.
We apply this bound to the sum
[Equation]
Noting that the summands indexed by j are iid by assumption, with the expected third moment norm featuring in the Berry-Esseen inequality upper-bounded by β ≤ C H µ ,n , for some constant C H µ ,n depending on H µ and n , but independent of H µ − 1 (ﬁniteness of C H µ ,n follows from Lemma 4).
As a consequence, we have the following bound:
[Equation]
where Z (cid:48) ( x ) =
( Z (cid:48)
j ( x [ i ]) | j = 1 , . .
, H µ , i = 1 , . . . , n ) is mean-zero multivariate normal, with blockwise independence with respect to the index j .
We wish to demonstrate that this is less than or equal to 2 − ( D − ( µ − 1))
− n (cid:80)
Dk = µ H k ε when H D is sufﬁciently large.
This is equivalent to showing that
[Equation]
for all sufﬁciently large H D .
But note that with H k − 1 = 2 H 2 k for k = µ, . .
, D − 1 , the left-hand side converges to 0 as H D increases (using the bound obtained for C H µ ,n in Lemma 4), so for all H
D sufﬁciently large, we obtain
[Equation]
as required.
Adding the independent bias vector b ( µ ) immediately yields
[Equation]
where Z ( x ) is mean-zero multivariate normal, with the same block-diagonal covariance structure as described for Z (cid:48)
( x ) above, and 1 n ∈ R n is a vector of 1 ’s.
Proof of Lemma 3.
Let A ⊆ R nH µ be an arbitrary convex set.
First, observe that we have
[Equation]
Now, note that for ﬁxed w ( µ ) and b ( µ ) , the event
[Equation]
  is exactly that the vector φ ( (cid:101) f ( µ − 1) ( x )) lies in the preimage of the convex set A − 1 n ⊗ b ( µ ) under the linear map w ( µ ) , which is again a convex set.
Secondly, observe that for the speciﬁc ReLU nonlinearity φ , if C is an arbitrary convex set, then { ( f ( µ − 1) ( x ) | φ ( f ( µ − 1) ( x )) ∈ C } may be written as the disjoint union of at most 2 nH µ − 1 convex sets:
[Equation]
Applying the assumed bound in the statement of the lemma to each of these sets, we obtain
[Equation]
Substituting this bound into the conditional probability (17) yields
[Equation]
Since A was an arbitrary convex set, the proof is complete.
Proof of Lemma 4.
Note that by independence of (cid:101) g ( µ − 1)
j ( x ) from (cid:101) w ( µ ) • ,j we have that each X j has mean zero and covariance Σ ⊗ = Σ ⊗ ˆ C ( µ ) w I where Σ is the covariance matrix of (cid:101) g ( µ − 1)
j ( x ) .
By standard properties of the Kronecker product, the Schur decomposition of Σ ⊗ is ( Q Λ Q T ) ⊗ ( ˆ C ( µ ) w I ) where Q Λ Q T is the Schur decomposition of Σ .
Simple algebraic manipulation yields:
[Equation]
Notice that the random variable ( ˆ C ( µ ) w ) − 1 / 2 (cid:101) w ( µ ) • ,j follows the R H µ -dimensional standard normal distribution
[Equation]
and thus its squared norm follows the chi-squared distribution with H µ degrees of freedom, which is also known as the Gamma ( H µ / 2 , 1 / 2) distribution.
Exponentiating to the power of 3 /and taking the expectation [Equation] we obtain:
[Equation]
Finally,
[Equation]
Q Λ
[Equation]
( x )
[Equation]
2 min where λ min is the smallest value on the diagonal of Λ .
If the activation φ does not increase the norm of the input vector (as is the case for rectiﬁed linear), we have (cid:107) (cid:101) [Equation]
µ
[Equation]
[Equation]
( x ) (cid:107)
[Equation]
almost surely, where Z ( µ − 1) j ( x ) follows the known n-dimensional normal distribution with mean zero and covariance matrix whose Schur decomposition will be denoted as U Ψ U T .
Using standard Gaussian identities, we can write
[Equation]
where ε ∼ N (0 , I n ) and ψ max is the highest entry on the diagonal of Ψ .
Putting it all together, we arrive at the desired upper bound C H µ ,n
[Equation]
Because ψ max and λ min are derived from the distribution of the limiting variable (cid:101)
[Equation]
φ ( Z [Equation] [Equation]
( x )) , which only depends on µ , the bound only depends on H µ and n as desired.
Further, noting that Γ(( x +3) / 2) / Γ( x/ 2)
=
O ( x 2 ) , we have that C H µ ,n =
O ( H 2 µ n 2 ) , as required.
Proof of Proposition 1.
We ﬁrst apply Lemma 3 to the assumed inequality
[Equation]
to obtain
[Equation]
We apply Lemma 2 so that for H D sufﬁciently large, we have
[Equation]
Applying the triangle inequality then yields
[Equation]
as required.
Proof of Proposition 2.
The idea of the proof is to chain Proposition 1 together across the layers of the network.
We ﬁx ε > 0 , and apply Proposition 1 to each layer of the network, yielding the following set of implications for H D sufﬁciently large:
[Equation]
for µ ∈ { 2 , . . .
, D + 1 } .
Finally, note that from the deﬁnition of the network, the distribution of f (1) ( x ) is exactly multivariate normal with the required covariance structure, so that d ( f (1) ( x ) , Z (1) ( x )) = 0 , completing the proof.
Proof of Theorem 1.
To prove that ( f ( D +1) ( x [ i ]))
∞ i =1 converges weakly to a Gaussian process with respect to the metric ρ on R N given by:
[Equation]
it is sufﬁcient (Billingsley, 1999, p. 19) to prove weak convergence of the ﬁnite-dimensional marginals of the process to multivariate Gaussian random variables, with covariance matrix matching that speciﬁed by the kernel of the proposed Gaussian process.
To this end, let I be a ﬁnite subset of N , and consider the inputs ( x [ i ]) i ∈ I .
We may now apply
Proposition 2 to obtain weak convergence of the joint distribution of the output variables of the network, ( f ( D +1) ( x [ i ]))
i ∈ I to a multivariate Gaussian with the correct covariance matrix.
As the ﬁnite subset of inputs was arbitrary, we are done.