D EEP B AYESIAN C ONVOLUTIONAL N ETWORKS WITH M ANY C HANNELS ARE G AUSSIAN P ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
There is a previously identiﬁed equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).
This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, inﬁnitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP.
In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels.
We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible.
Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical.
As a consequence, translation equivariance in ﬁnite channel CNNs trained with stochastic gradient descent (SGD) has no corresponding property in the Bayesian treatment of the inﬁnite channel limit – a qualitative difference between the two regimes that is not present in the FCN case.
We conﬁrm experimentally, that while in some scenarios the performance of SGD-trained ﬁnite CNNs approaches that of the corresponding
GPs as the channel count increases, with careful tuning SGD-trained CNNs can signiﬁcantly outperform their corresponding GPs.
I NTRODUCTION
Neural networks (NNs) demonstrate remarkable performance (He et al., 2016;
Oord et al., 2016;
Silver et al., 2017; Vaswani et al., 2017), but are still only poorly understood from a theoretical perspective
(Goodfellow et al., 2015; Choromanska et al., 2015; Pascanu et al., 2014; Zhang et al., 2017).
NN performance is often motivated in terms of model architectures, initializations, and training procedures together specifying biases, constraints, or implicit priors over the class of functions learned by a network.
This induced structure in learned functions is believed to be well matched to structure inherent in many practical machine learning tasks, and in many real-world datasets.
For instance, properties of NNs which are believed to make them well suited to modeling the world include: hierarchy and compositionality (Lin et al., 2017; Poggio et al., 2017), Markovian dynamics (Ti ˇ no et al., 2004; 2007), and equivariances in time and space for RNNs (Werbos, 1988) and CNNs (Fukushima & Miyake, 1982; Rumelhart et al., 1985) respectively.
The recent discovery of an equivalence between deep neural networks and GPs (Lee et al., 2018;
Matthews et al., 2018b) allow us to express an analytic form for the prior over functions encoded by deep NN architectures and initializations.
This transforms an implicit prior over functions into an explicit prior , which can be analytically interrogated and reasoned about 1 .
Previous work studying these Neural Network-equivalent Gaussian Processes (NN-GPs) has established the correspondence only for fully connected networks (FCNs).
Additionally, previous work has not used analysis of NN-GPs to gain speciﬁc insights into the equivalent NNs.
In the present work, we extend the equivalence between NNs and NN-GPs to deep Convolutional
Neural Networks (CNNs), both with and without pooling.
CNNs are a particularly interesting architecture for study, since they are frequently held forth as a success of motivating NN design based on invariances and equivariances of the physical world (Cohen & Welling, 2016) – speciﬁcally, designing a NN to respect translation equivariance (Fukushima & Miyake, 1982; Rumelhart et al., 1985).
As we will see in this work, absent pooling, this quality can vanish in the Bayesian treatment of the inﬁnite channel (number of convolutional ﬁlters) limit.
The speciﬁc novel contributions of the present work are:
We show analytically that CNNs with many channels, trained in a fully Bayesian fashion, correspond to an NN-GP ( § 2, § 3).
We show this for CNNs both with and without pooling, with arbitrary convolutional striding, and with both same and valid padding.
We prove convergence as the number of channels in hidden layers approach inﬁnity simultaneously (i.e. min (cid:8) n 1 , . . .
, n L (cid:9) → ∞ , see § A.5.4 for details), strengthening and extending the result of Matthews et al.
(2018a)
under mild conditions on the nonlinearity derivative.
We show that in the absence of pooling, the NN-GP for a CNN and a Locally Connected Network (LCN) are identical ( § 5.1).
An LCN has the same local connectivity pattern as a CNN, but without weight sharing or translation equivariance.
We experimentally compare trained CNNs and LCNs and ﬁnd that under certain conditions both perform similarly to the respective NN-GP (Figure 3, b, c).
Moreover, both architectures tend to perform better with increased channel count, suggesting that similarly to FCNs (Neyshabur et al., 2015;
Novak et al., 2018)
CNNs beneﬁt from overparameterization (Figure 3, a, b), corroborating a similar trend observed in Canziani et al.
(2016, Figure 2).
However, we also show that careful tuning of hyperparameters allows ﬁnite CNNs trained with SGD to outperform their corresponding NN-GP by a signiﬁcant margin.
We experimentally disentangle and quantify the contributions stemming from local connectivity, equivariance, and invariance in a convolutional model in one such setting (Table 1).
We introduce a Monte Carlo method to compute NN-GP kernels for situations (such as CNNs with pooling) where evaluating the NN-GP is otherwise computationally infeasible ( § 4).
We stress that we do not evaluate ﬁnite width Bayesian networks nor do we make any claims about their performance relative to the inﬁnite width GP studied here or ﬁnite width SGD-trained networks.
While this is an interesting subject to pursue (see Matthews et al.
(2018a); Neal (1994))
, it is outside of the scope of this paper.
1. R ELATED WORK
In early work on neural network priors, Neal (1994) demonstrated that, in a fully connected network with a single hidden layer, certain natural priors over network parameters give rise to a Gaussian process prior over functions when the number of hidden units is taken to be inﬁnite.
Follow-up work extended the conditions under which this correspondence applied (Williams, 1997; Le Roux & Bengio, 2007; Hazan & Jaakkola, 2015).
An exactly analogous correspondence for inﬁnite width, ﬁnite depth deep fully connected networks was developed recently in Lee et al.
; Matthews et al.
(2018b), with Matthews et al.
(2018a)
extending the convergence guarantees to a more general class of nonlinearities and width growth rates.
The line of work examining signal propagation in random deep networks (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; Hanin & Rolnick, 2018; Chen et al., 2018) is related to the construction of the GPs we consider.
They apply a mean ﬁeld approximation in which the pre-activation signal is replaced with a Gaussian, and the derivation of the covariance function with depth is the same as for the kernel function of a corresponding GP.
Recently, Xiao et al.
(2017b; 2018) extended this to convolutional architectures without pooling.
Xiao et al.
(2018) also analyzed properties of the convolutional kernel at large depths to construct a phase diagram which will be relevant to NN-GP performance, as discussed in § A.2.
Compositional kernels coming from wide convolutional and fully connected layers also appeared outside of the GP context.
Cho & Saul (2009) derived closed-form compositional kernels for rectiﬁed polynomial activations (including ReLU ).
Daniely et al.
(2016) proved approximation guarantees between a network and its corresponding kernel, and show that empirical kernels will converge as the number of channels increases.
There is a line of work considering stacking of GPs, such as deep GP s (Lawrence & Moore, 2007;
Damianou & Lawrence, 2013).
These no longer correspond to GPs, though they can describe a rich class of probabilistic models beyond GPs.
Alternatively, deep kernel learning (Wilson et al., 2016b;a; Bradshaw et al., 2017) utilizes GPs with base kernels which take in features produced by a deep neural network (often a CNN), and train the resulting model end-to-end.
Finally, van der Wilk et al.
(2017) incorporates convolutional structure into GP kernels, with follow-up work stacking multiple such GPs (Kumar et al., 2018; Blomqvist et al., 2018; Anonymous, 2019) to produce a deep convolutional GP (which is no longer a GP).
Our work differs from all of these in that our
GP corresponds exactly to a fully Bayesian CNN in the inﬁnite channel limit, when all layers are taken to be of inﬁnite size.
We remark that while alternative models, such as deep GPs, do include inﬁnite-sized layers in their construction, they do not treat all layers in this way – for instance, through insertion of bottleneck layers which are kept ﬁnite.
While it remains to be seen exactly which limit is applicable for understanding realistic CNN architectures in practice, the limit we consider is natural for a large class of CNNs, namely those for which all layers sizes are large and rather comparable in size.
Borovykh (2018) analyzes the convergence of CNN outputs at different spatial locations (or different timepoints for a temporal CNN) to a GP for a single input example.
Thus, while they also consider a
GP limit (and perform an experiment comparing posterior GP predictions to an SGD-trained CNN), they do not address the dependence of network outputs on multiple input examples, and thus their model is unable to generate predictions on a test set consisting of new input examples.
In concurrent work, Garriga-Alonso et al.
(2018) derive an NN-GP kernel equivalent to one of the kernels considered in our work.
In addition to explicitly specifying kernels corresponding to pooling and vectorizing, we also compare the NN-GP performance to ﬁnite width SGD-trained CNNs and analyze the differences between the two models.
M ANY - CHANNEL B AYESIAN CNN S ARE G AUSSIAN PROCESSES
P RELIMINARIES
General setup.
For simplicity of presentation we consider 1D convolutional networks with circularly-padded activations.
Our analysis is straighforward to extend to higher dimensions, using zero (“same”) or no (“valid”) padding, and strided convolutions ( § A.3).
We consider a series of
L + 1 convolutional hidden layers, l = 0 , ..., L .
Random weights and biases.
The parameters of the network are the convolutional ﬁlters and biases, ω lij,β and b li , respectively, with outgoing (incoming) channel index i ( j ) and ﬁlter relative spatial location β ∈ [ ± k ] ≡ {− k, . . .
. , k } .
2 Assume a Gaussian prior on both the ﬁlter weights and biases,
[Equation]
The weight and bias variances are σ 2 ω , σ 2 b , respectively.
n l is the number of channels (ﬁlters) in layer l , 2 k + 1 is the ﬁlter size, and v β is the fraction of the receptive ﬁeld variance at location β (with (cid:80) β v β = 1 ).
In experiments we utilize uniform v β = 1 / (2 k + 1) , but nonuniform v β (cid:54)
= 1 / (2 k + 1) should enable kernel properties that are better suited for ultra-deep networks, as in Xiao et al.
Inputs, pre-activations, and activations.
Let X denote a set of input images (training set or validation set or both).
The network has activations y l ( x ) and pre-activations z l ( x ) for each input image x ∈ X ⊂ R n 0 d , with input channel count n 0 ∈ N , number of pixels d ∈ N , where y li,α ( x ) ≡ (cid:26) x i,α l
[Equation]
We emphasize the dependence of y li,α ( x ) and z li,α ( x ) on the input x .
φ : R → R is a nonlinearity (with elementwise application to higher-dimensional inputs).
Similarly to Xiao et al.
, y l is assumed to be circularly-padded and the spatial size d hence remains constant throughout the network.
See Figure 4 for a visual depiction of our notation.
Activation covariance.
A recurring quantity in this work will be the empirical uncentered covariance matrix K l of the activations y l , deﬁned as
[Equation]
K l is therefore a random variable indexed by two inputs x, x (cid:48) and two spatial locations α, α (cid:48)
(the dependence on layer widths n 1 , . .
. , n l , as well as weights and biases, is implied and by default not stated explicitly).
K 0 , the empirical uncentered covariance of inputs, is deterministic.
Shapes and indexing.
Whenever an index is omitted, the variable is assumed to contain all possible entries along the respective dimension.
E.g. y 0 is a vector of size |X| n 0 d , (cid:2) K l (cid:3) α,α (cid:48) is a matrix of shape |X| × |X| , z lj is a vector of size |X| d , etc.
Our work concerns proving that the top-layer pre-activations z L converge in distribution to an |X| n L +1 d -variate normal random vector with a particular covariance matrix of shape (cid:0) |X| n L +1 d (cid:1) × (cid:0) |X| n L +1 d (cid:1) as min (cid:8) n 1 , . . . , n L (cid:9) → ∞ .
We emphasize that only the channels in hidden layers are taken to inﬁnity, and n L +1 , the number of channels in the top-layer pre-activations z L , remains ﬁxed.
Therefore for convergence proofs, we always consider z l , y l , as well as any of their indexed subsets like z lj , y li,α to be 1D vector random variables, while K l , as well as any of their indexed subsets (when applicable, e.g. (cid:2) K l (cid:3) α,α (cid:48) , (cid:2) K l (cid:3) ( x, x (cid:48) ) ) to be 2D matrix random variables.
2. C ORRESPONDENCE BETWEEN G AUSSIAN PROCESSES AND B AYESIAN DEEP CNN S WITH INFINITELY MANY CHANNELS
We next consider the prior over outputs z L computed by a CNN in the limit of inﬁnitely many channels in the hidden (excluding input and output) layers, min (cid:8) n 1 , . . .
, n L (cid:9) → ∞ , and derive its equivalence to a GP with a compositional kernel.
This section provides a brief argument showing that z L is normally distributed conditioned on previous layer activation covariance K L , which itself becomes deterministic in the inﬁnite limit.
As we elaborate in § A.5.4, this allows to conclude convergence in distribution of the outputs to a Gaussian with the respective deterministic covariance limit.
2.2. A SINGLE CONVOLUTIONAL LAYER IS A GP CONDITIONED ON THE UNCENTERED COVARIANCE MATRIX OF THE PREVIOUS LAYER ’ S ACTIVATIONS
As can be seen in Equation 2, the pre-activations z l are an afﬁne transformation of the multivariate
Gaussian (cid:8) ω l , b l (cid:9) , speciﬁed by the previous layer’s activations y l .
An afﬁne transformation of a multivariate Gaussian is itself a Gaussian with a covariance matrix that can be derived straightforwardly.
Speciﬁcally,
[Equation]
(4) where I n l +1 is an n l +1 × n l +1 identity matrix, and A (cid:0) K l (cid:1) is the covariance of the pre-activations z li and is derived in Xiao et al.
Precisely, A : R |X| d ×|X| d → R |X| d ×|X| d is an afﬁne transformation (a cross-correlation operator followed by a shifting operator) deﬁned as follows:
[Equation]
Notice that the covariance matrix in Equation 4 factorizes due to the fact that separate channels (cid:8) z li (cid:9) n l +1 i =1 are i.i.d.
conditioned on y l , due to i.i.d.
weights and biases (cid:8) ω li , b li (cid:9) n l +1 i =1 .
We further remark that per Equation 4 the normal distribution of (cid:0) z l | y l (cid:1) only depends on K l , hence the random variable (cid:0) z l | K l (cid:1) has the same distribution by the law of total expectation:
[Equation]
2.2. A CTIVATION COVARIANCE MATRIX BECOMES DETERMINISTIC WITH INCREASING CHANNEL COUNT
It follows from Equation 6 that the summands in Equation 3 are i.i.d.
conditioned on K l − 1 .
Subject to weak restrictions on the nonlinearity φ , we can apply the weak law of large numbers and conclude that the covariance matrix K l becomes deterministic in the inﬁnite channel limit in layer l (note that pre-activations z l remain stochastic).
Precisely,
[Equation]
where
[Equation]
The decoupling of the kernel “propagation” into C and A is highly convenient since A is a simple afﬁne transformation of the kernel (see Equation 5), and C is a well-studied map in literature (see § A.7.4), and for nonlinearities such as ReLU (Nair & Hinton, 2010) and the error function ( erf ) C can be computed in closed form as derived in Cho & Saul (2009) and Williams (1997) respectively.
We refer the reader to Xiao et al.
(2018, Lemma A.1) for complete derivation of the limiting value in Equation 7.
A less obvious result is that, under slightly stronger assumptions on φ , the top-layer activation covariance K L becomes unconditionally (dependence on observed deterministic inputs y 0 is implied) deterministic as channels in all hidden layers grow to inﬁnity simultaneously:
[Equation]
, (9) i.e. K L ∞ is ( C ◦ A ) applied L times to K 0 , the deterministic input covariance.
We prove this in § A.5.4 (Theorem A.4).
2.2. A CONDITIONALLY NORMAL RANDOM VARIABLE BECOMES NORMAL IF ITS COVARIANCE BECOMES DETERMINISTIC
§ 2.2.1 established that (cid:0) z L | K L (cid:1) is Gaussian, and § 2.2.2 established that its covariance matrix
A (cid:0) K L (cid:1) ⊗ I n L +1 converges in probability to a deterministic A (cid:0) K L ∞ (cid:1) ⊗ I n l +1 in the inﬁnite channel limit (since K L P → K L ∞ , and A ( · ) ⊗ I n L +1 : R |X| d ×|X| d → R n L +1 |X| d × n L +1 |X| d is continuous).
As we establish in § A.5.
(Theorem A.5), this is sufﬁcient to conclude with the following result.
Result.
If φ : R → R has an exponentially-bounded derivative almost everywhere, i.e. ∃ a, b ∈ R : ∀ x ∈ R | φ (cid:48)
( x ) | ≤ a exp ( bx ) a.e., then the following convergence in distribution holds:
[Equation]
T RANSFORMING A GP OVER SPATIAL LOCATIONS INTO A GP OVER CLASSES
In § 2.2 we have shown that in the inﬁnite channel limit a deep CNN is a GP indexed by input samples, output spatial locations, and output channel indices.
Further, its covariance matrix K L ∞ can be computed in closed form.
Here we show that transformations to obtain class predictions that are common in CNN classiﬁers can be represented as either vectorization or projection (as long as we treat classiﬁcation as regression, similarly to Lee et al.
Both of these operations preserve the GP equivalence and allow the computation of the covariance matrix of the respective GP (now indexed by input samples and target classes) as a simple transformation of K L ∞ .
In the following we will denote C as the number of target classes, ¯ z L +1 ( x ) ∈ R C as the outputs of the CNN-based classiﬁer, and ¯ ω L +1 , ¯ b L +1 as the last (number L + 1 ) layer weights and biases.
For each class i we will denote the empirical uncentered covariance of the outputs as
[Equation]
a random variable of shape |X| × |X| .
We will show that in the scenarios considered below in § 3. and § 3.2, the outputs (cid:0) ¯
z L +1 i | y 0 (cid:1) will converge in distribution to a multivariate Gaussian (i.i.d.
for every class i ) as min { n 1 , . . .
, n L +1 } → ∞ with covariance denoted as K ∞ ∈ R |X|×|X| , or, jointly, (cid:0) ¯
z L +1 | y 0 (cid:1) D
→ N (0 , K ∞ ⊗ I C ) .
However, as in § 2.2, we postpone the formal derivation until § A.5.
(Theorem A.5).
3. V ECTORIZATION
One common readout strategy is to vectorize (ﬂatten) the output of the last convolutional layer z L ( x ) ∈ R n L +1 d into a vector 5 vec (cid:2)
z L ( x ) (cid:3) ∈ R n L +1 d and stack a fully connected layer on top:
[Equation]
where the weights ¯ ω L +1 ∈ R C × n L +1 d
and biases ¯ b L +1 ∈ R C are i.i.d.
Gaussian, ¯ ω
L +1 ij ∼
N (cid:0)
[Equation]
It is easy to verify that (cid:0)
¯
z L +1 i
| K L +
[Equation]
is a multivariate Gaussian with the covariance:
[Equation]
The argument of § 2.2 can then be extended to conclude that in the limit of inﬁnite width (cid:0)
¯ z L +
i
[Equation]
converges in distribution to a multivariate Gaussian (i.i.d.
for each class i ) with covariance
[Equation]
A sample network using this readout strategy is depicted in Figure 4.
Note that as observed in Xiao et al.
(2018), to compute any diagonal terms of (cid:2) K l +1 ∞ (cid:3) ( x, x (cid:48) ) , one needs only the corresponding diagonal terms of (cid:2) K l ∞ (cid:3) ( x, x (cid:48) ) .
Consequently, we only need to compute (cid:110)(cid:2) K l ∞ (cid:3) α,α ( x, x (cid:48) )
: x, x (cid:48) ∈ X , α ∈ { 1 . . .
d } (cid:111) l =0 ,...,L and the memory cost is O (cid:0) |X| 2 d (cid:1) (or
O ( d ) per covariance entry in an iterative or distributed setting).
Note that this approach ignores pixel-pixel covariances and produces a GP corresponding to a locally-connected network (see § 5.1).
3. P ROJECTION
Another approach is a projection collapsing the spatial dimensions.
Let h ∈ R
d be a deterministic vector,
¯ ω
L +1 ij ∼ N (cid:0) 0 , σ 2 ω /n
L +1 (cid:1) , and ¯ b L +1 be the same as in § 3.1.
Deﬁne the output to be
[Equation]
, leading to (analogously to § 3.1) (15)
[Equation]
Examples of this approach include
[Figure]
[Figure]
MC-CNN-GP with pooling
CNN-GP
CNN-GP (center pixel)
CNN-GP (without padding)
FCN-GP
[Figure]
cc u r a c y
Global average pooling: take h = 1 d 1 d .
Then
[Equation]
This approach corresponds to applying global average pooling right after the last convolutional layer.
This approach takes all pixel-pixel covariances into consideration and makes the kernel translation invariant.
However, it requires O (cid:0) |X| 2 d 2 (cid:1) memory to compute the sample-sample covariance of the GP (or O (cid:0) d 2 (cid:1)
per covariance entry in an iterative or distributed setting).
It is impractical to use this method to analytically evaluate the GP, and we propose to use a Monte Carlo approach (see § 4).
Subsampling one particular pixel: take h =
e α ,
[Equation]
This approach makes use of only one pixel-pixel covariance, and requires the same amount of memory as vectorization ( § 3.1) to compute.
We compare the performance of presented strategies in Figure 1.
Note that all described strategies admit stacking additional FC layers on top while retaining the GP equivalence, using a derivation analogous to § 2.2 (Lee et al., 2018; Matthews et al., 2018b).
M ONTE C ARLO EVALUATION OF INTRACTABLE GP KERNELS
We introduce a Monte Carlo estimation method for NN-GP kernels which are computationally impractical to compute analytically, or for which we do not know the analytic form.
Similar in spirit to traditional random feature methods (Rahimi & Recht, 2007), the core idea is to instantiate many
M CC
NN - G
P    0.
A cc u r a c y log2(#Samples) l og ( # C h a nn e l s )    − 6.
l og ( D i s t
a n ce ) log2(#Samples) l og ( # C h a nn e l s ) random ﬁnite width networks and use the empirical uncentered covariances of activations to estimate the Monte Carlo-GP (MC-GP) kernel,
[Equation]
(19) where θ consists of M draws of the weights and biases from their prior distribution, θ m ∼ p ( θ ) , and n is the width or number of channels in hidden layers.
The MC-GP kernel converges to the analytic kernel with increasing width, lim n →∞ K ln,M = K l ∞ in probability.
For ﬁnite width networks, the uncertainty in K ln,M is Var (cid:2)
K ln,M (cid:3) =
Var θ (cid:2) K ln ( θ ) (cid:3) /M .
From
Daniely et al.
, we know that Var θ (cid:2)
K ln ( θ ) (cid:3) ∝ 1 n , which leads to Var θ [ K ln,M ] ∝ 1 Mn .
For ﬁnite n , K ln,M is also a biased estimate of K l ∞ , where the bias depends solely on network width.
We do not currently have an analytic form for this bias, but we can see in Figures 2 and 7 that for the hyperparameters we probe it is small relative to the variance.
In particular, (cid:13)(cid:13) K ln,M ( θ ) − K L ∞ (cid:13)(cid:13) 2 F is nearly constant for constant Mn .
We thus treat Mn as the effective sample size for the Monte
Carlo kernel estimate.
Increasing M and reducing n can reduce memory cost, though potentially at the expense of increased compute time and bias.
In a non-distributed setting, the MC-GP reduces the memory requirements to compute GP pool from
O (cid:16) |X| 2 d 2 (cid:17) to O (cid:16) |X| 2 + n 2 + nd (cid:17) , making the evaluation of CNN-GPs with pooling practical.
D ISCUSSION 5. B AYESIAN CNN S WITH MANY CHANNELS ARE IDENTICAL TO LOCALLY CONNECTED NETWORKS , IN THE ABSENCE OF POOLING
Locally Connected Networks (LCNs) (Fukushima, 1975; Lecun, 1989) are CNNs without weight sharing between spatial locations.
LCNs preserve the connectivity pattern, and thus topology, of a
CNN.
However, they do not possess the equivariance property of a CNN – if an input is translated, the latent representation in an LCN will be completely different, rather than also being translated.
The CNN-GP predictions without spatial pooling in § 3.1 and § 3.2.2 depend only on sample-sample covariances, and do not depend on pixel-pixel covariances.
LCNs destroy pixel-pixel covariances: (cid:2)
K L ∞ (cid:3)
LCN α,α (cid:48) ( x, x (cid:48) )
= 0 , for α (cid:54) =
α (cid:48) and all x, x (cid:48) ∈ X and L > 0 .
However, LCNs preserve the covariances between input examples at every pixel: (cid:2) K L ∞ (cid:3)
LCN α,α ( x, x (cid:48) )
= (cid:2) K L ∞ (cid:3) CNN α,α ( x, x (cid:48) ) .
As a result, in the absence of pooling, LCN-GPs and CNN-GPs are identical.
Moreover, LCN-GPs with pooling are identical to CNN-GPs with vectorization of the top layer (under suitable scaling of y L +1 ).
We conﬁrm these ﬁndings experimentally in trained networks in the limit of large width in
Figure 3 (b), as well as by demonstrating convergence of MC-GPs of the respective architectures to the same CNN-GP (modulo scaling of y L +1 ) in Figures 2 and 7.
5. P OOLING LEVERAGES EQUIVARIANCE TO PROVIDE INVARIANCE
The only kernel leveraging pixel-pixel covariances is that of the CNN-GP with pooling.
This enables the predictions of this GP and the corresponding CNN to be invariant to translations (modulo edge effects) – a beneﬁcial quality for an image classiﬁer.
We observe strong experimental evidence supporting the beneﬁts of invariance throughout this work (Figures 1, 2, 3 (b); Tables 1, 2), in both
CNNs and CNN-GPs.
5. F INITE CHANNEL SGD- TRAINED CNN S CAN OUTPERFORM INFINITE CHANNEL B AYESIAN CNN S , IN THE ABSENCE OF POOLING
In the absence of pooling, the beneﬁts of equivariance and weight sharing are more challenging to explain in terms of Bayesian priors on class predictions (since without pooling equivariance is not a property of the outputs, but only of intermediary representations).
Indeed, in this work we ﬁnd that the performance of ﬁnite width SGD-trained CNNs often approaches that of their CNN-GP counterpart (Figure 3, b, c) 7 , suggesting that in those cases equivariance does not play a beneﬁcial role in SGD-trained networks.
However, as can be seen in Tables 1, 2 and Figure 3 (c), the best CNN overall outperforms the best CNN-GP by a signiﬁcant margin – an observation speciﬁc to CNNs and not FCNs or LCNs.
We observe this gap in performance especially in the case of ReLU networks trained with a large learning rate.
In Table 1 we demonstrate this large gap in performance by evaluating different models with equivalent architecure and hyperparameter settings, chosen for good SGD-trained CNN performance.
We conjecture that equivariance, a property lacking in LCNs and the Bayesian treatment of the inﬁnite channel CNN limit, contributes to the performance of SGD-trained ﬁnite channel CNNs with the correct settings of hyperparameters.
Nonetheless, more work is needed to disentangle and quantify the separate contributions of stochastic optimization and ﬁnite width effects 8 to differences in performance between CNNs with weight sharing and their corresponding CNN-GPs.
C ONCLUSION
In this work we have derived a Gaussian process that corresponds to a deep fully Bayesian CNN with inﬁnitely many channels.
The covariance of this GP can be efﬁciently computed either in closed form or by using Monte Carlo sampling, depending on the architecture.
The CNN-GP achieves state of the art results for GPs without trainable kernels on CIFAR10.
It can perform competitively with CNNs (that ﬁt the training set) of equivalent architecture and weight priors, which makes it an appealing choice for small datasets, as it eliminates all training-related hyperparameters.
However, we found that the best overall performance is achieved by ﬁnite SGDtrained CNNs and not by their inﬁnite Bayesian counterparts.
We hope our work stimulates future research into disentangling the contributions of the two qualities (Bayesian treatment and inﬁnite width) to the performance gap observed.
[Figure] [Figure]
No Pooling
Global Average Pooling
[Figure]
SGD-trained CNNs can outperform their corresponding CNN-GPs.
Each point corresponds to the test accuracy of: ( y -axis) a speciﬁc CNN-GP; ( x -axis) the best (on validation) CNN with the same architectural hyper-parameters selected among the 100%-accurate models on the full training
CIFAR10 dataset with different learning rates, weight decay and number of channels.
While CNN-GP appears competitive against 100% -accurate CNNs (above the diagonal), the best CNNs overall outperform CNN-GPs by a signiﬁcant margin (below the diagonal, right).
For further analysis of factors leading to similar or diverging behavior between SGD-trained ﬁnite CNNs and inﬁnite Bayesian CNNs see Tables 1 and 2.
Experimental details: all networks have reached 100% training accuracy on CIFAR10.
Values in (b) are reported on an 0.5K/4K train/validation subset downsampled to 8 × 8 for computational reasons.
See § A.7.5 and § A.7.1 for full experimental details of (a, c) and (b) plots respectively.
[Table]
[Table]
A A PPENDIX A. A DDITIONAL F IGURES
[Equation]
z 1 ( x ) y 2 ( x ) ¯ z 2 ( x ) ϕ n 0 =n 1 =n 1 =n 2 =n 2 =
C =k
= ( 3 , 3 ) k
= ( 3 , 3 ) d 0 = ( 8 ,) d 1 = ( 6 , 6 ) d 2 = ( 4 , 4 ) ϕ
[Equation]
is assumed in the text, yet our formalism easily extends to the model displayed ( § A.3).
Further, while displayed (pre-)activations have 3D shapes, in the text we treat them as 1D vectors ( § 2.1).
[Figure]
[Figure]
(vertical axis) of best (in terms of validation loss) neural networks as the number of channels increases (horizontal axis).
While perfect 0 loss is not achieved (but 100% accuracy is), we observe no consistent improvement when increasing the capacity of the network (left to right).
This eliminates underﬁtting as a possible explanation for why small models perform worse in Figure 3 (b).
See Figure 5 for validation loss and § A.7.1 for experimental details.
M CC
NN - G
P w it
h poo li ng    0.
A cc u r a c y log2(#Samples) l og ( # C h a nn e l s )    − 6.
l og ( D i s t
a n ce ) log2(#Samples) l og ( # C h a nn e l s )
M CL C
N - G
P    0.
A c c u r a c y log2(#Samples)
l
og ( # C h a nn e l s )    − 6.
l og ( D i s t
a n ce ) log2(#Samples) l og ( # C h a nn e l s )
M CL C
N - G
P w it h
P oo li ng    0.
A cc u r a c y log2(#Samples) l og ( # C h a nn e l s )    − 6.
l og ( D i s t
a n ce ) log2(#Samples) l og ( # C h a nn e l s )
M CF C
N - G
P    0.
A cc u r a c y log2(#Samples) l og ( # C h a nn e l s )    − 6.
l og ( D i s t a n ce ) log2(#Samples) l og
[Figure]
e l
[Figure]
Depth:
Phase boundary
C
NN - G
P 0.
A cc u r a c y
Weight variance
B i a s v a r i a n ce
F C
N - G
P
A. R ELATIONSHIP TO D EEP S IGNAL P ROPAGATION
The recurrence relation linking the GP kernel at layer l + 1 to that of layer l following from Equation 10 (i.e. K l +1 ∞ =
( C ◦ A ) (cid:0) K l ∞ (cid:1) ) is precisely the covariance map examined in a series of related papers on signal propagation (Xiao et al., 2018;
Poole et al., 2016; Schoenholz et al., 2017; Lee et al., 2018) (modulo notational differences; denoted as F , C or e.g. A (cid:63) C in Xiao et al.
In those works, the action of this map on hidden-state covariance matrices was interpreted as deﬁning a dynamical system whose large-depth behavior informs aspects of trainability.
In particular, as l → ∞ , K l +1 ∞ =
[Equation]
i.e. the covariance approaches a ﬁxed point
K ∗∞ .
The convergence to a ﬁxed point is problematic for learning because the hidden states no longer contain information that can distinguish different pairs of inputs.
It is similarly problematic for GPs, as the kernel becomes pathological as it approaches a ﬁxed point.
Precisely, in the chaotic regime outputs of the GP become asymptotically decorrelated and therefore independent, while in the ordered regime they approach perfect correlation of 1 .
Either of these scenarios captures no information about the training data in the kernel and makes learning infeasible.
This problem can be ameliorated by judicious hyperparameter selection, which can reduce the rate of exponential convergence to the ﬁxed point.
For hyperpameters chosen on a critical line separating two untrainable phases, the convergence rates slow to polynomial, and very deep networks can be trained, and inference with deep NN-GP kernels can be performed – see Table 3.
A. S TRIDED CONVOLUTIONS AND AVERAGE POOLING IN INTERMEDIATE LAYERS
Our analysis in the main text can easily be extended to cover average pooling and strided convolutions (applied before the pointwise nonlinearity).
Recall that conditioned on K l the pre-activation z lj
( x ) ∈ R d 1 is a zero-mean multivariate Gaussian.
Let B ∈ R
d 2 × d 1 denote a linear operator.
Then
Bz lj ( x ) ∈ R d 2 is a zero-mean Gaussian, and the covariance is
[Equation]
One can easily see that (cid:8) Bz lj (cid:12)(cid:12) K l (cid:9) j are i.i.d.
multivariate Gaussian as well.
Strided convolution .
Strided convolution is equivalent to a non-strided convolution composed with subsampling.
Let s ∈ N denote size of the stride.
Then the strided convolution is equivalent to choosing B as follows: B ij = δ ( is − j ) for i ∈ { 0 , 1 , . . .
( d 2 − 1) } .
Average pooling .
Average pooling with stride s and window size ws is equivalent to choosing
B ij = 1 /ws for i = 0 , 1 , . . .
( d 2 − 1) and j = is, . . .
, ( is + ws − 1) .
ND convolutions.
Note that our analysis in the main text (1D) easily extends to higher-dimensional convolutions by replacing integer pixel indices and sizes d, α, β with tuples (see also Figure 4).
In Equation 2 β values would have to span the hypercube [ ± k ] N = {− k, . . .
, k } N in the preactivation deﬁnition.
Similarly, in § 3 the normalizing factor d ( d 2 ) should be the product (squared) of its entries, and summations over α, β should span the [ d 0 ] × · · · × [ d N ] hypercube as well.
The deﬁnition of the kernel propagation operator A in Equation 5 will remain exactly the same, so long as β is summed over the hypercube, and the variance weights remain respectively normalized (cid:80) β v β = 1 .
A. R EVIEW OF EXACT B AYESIAN REGRESSION WITH GP S
Our discussion in the paper has focused on model priors .
A crucial beneﬁt we derive by mapping to a GP is that Bayesian inference is straightforward to implement and can be done exactly for regression (Rasmussen & Williams, 2006, chapter 2), requiring only simple linear algebra.
Let
X denote training inputs x 1 , ..., x |X| , t T =
( t 1 , ..., t |X| ) training targets, and collectively D for the training set.
The integral over the posterior can be evaluated analytically to give a posterior predictive distribution on a test point y ∗ which is Normal, ( z ∗ |D , y ∗ ) ∼ N (cid:0) µ ∗ , σ 2 ∗ (cid:1) , with
[Equation]
[Equation]
We use the shorthand K ( X , X ) to denote the |X| × |X| matrix formed by evaluating the GP covariance on the training inputs, and likewise K ( x ∗ , X ) is a |X| -length vector formed from the covariance between the test input and training inputs.
Computationally, the costly step in GP posterior predictions comes from the matrix inversion, which in all experiments were carried out exactly, and typically scales as O ( |X| 3 ) (though algorithms scaling as O ( |X| 2 . 4 ) exist for sufﬁciently large matrices).
Nonetheless, there is a broad literature on approximate Bayesian inference with GPs which can be utilized for efﬁcient implementation (Rasmussen & Williams, 2006, chapter 8); (Qui ˜ noneroCandela & Rasmussen, 2005; Titsias, 2009).
A. E QUIVALENCE BETWEEN RANDOMLY INITIALIZED NN S AND GP S
In this section, we present two different approaches, the sequential limit ( § A.5.3) and simultaneous limit ( § A.5.4), to illustrate the relationship between many-channels Bayesian CNNs and GPs.
Sequential limit ( § A.5.3) involves taking the inﬁnite channel limit in hidden layers in a sequence, starting from bottom (closest to inputs) layers and going upwards (to the outputs), i.e. n 1 → ∞ , . .
. , n L → ∞ .
Note that this approach in fact constructs a GP using a NN architecture to deﬁne its covariance, and does not provide guarantees on actual convergence of large but ﬁnite Bayesian CNNs to GPs, which is of most practical interest.
However, it has the following beneﬁts:
Weak assumptions on the NN activation function φ and on the distribution of the NN parameters.
The arguments can be easily extended to more complicated network architectures, e
g. architectures with max pooling, dropout, etc. 3.
This approach provides a straightforward and intuitive way to compute the covariance of the Gaussian process without diving into mathematical details.
Simultaneous limit ( § A.5.4) considers growing the number of channels in hidden layers uniformly, i.e. min (cid:8) n 1 , . .
. , n L (cid:9) → ∞ .
This approach establishes convergence of ﬁnite channel Bayesian
CNNs to GPs and is thus a more practically relevant result.
However, it makes stronger assumptions, and the proof is more involved.
We hightlight that the GPs obtained by the two approaches are identical.
In both sections, we only provide the arguments for CNNs.
It is straightforward (and in fact simpler) to extend them to LCNs and FCNs.
Indeed, an FCN is a particular case of a CNN where the inputs and ﬁlters have singular spatial dimensions ( d = 1 , k = 0 ).
For LCNs, the proof goes through in an identical fashion if we replace A with A LCN deﬁned as (cid:2)
A LCN ( K ) (cid:3) α,α (cid:48)
( x, x (cid:48) )
≡ δ
A.5. S ETUP
Probability space.
Let P be a collection of countably many mutually independent random variables (R.V.s ) deﬁned on some probability space (Ω , F , P ) , where F is a σ -algebra deﬁned on Ω and P is a probability measure.
Here P ≡ W ∪ B ∪ H is the collection of parameters used to deﬁne neural networks:
(i) Weights.
W = (cid:83) l ∈ N W l and W l = (cid:110) ω
lij,β : i, j ∈ N , β ∈ [ ± k ] (cid:111) , where [ ± k ] ≡ [ − k, k ] ∩ Z .
We assume ω lij,β are i.i.d.
R.V.s with mean zero and ﬁnite variance 0 < σ 2 ω < ∞ .
When l = 0 , we further assume they are Gaussian distributed.
(ii) Biases .
B =
(cid:83) l ∈ N B l and B l = (cid:8) b lj : j ∈ N (cid:9) .
We assume b lj are i
i
d. Gaussian with mean zero and variance 0 ≤ σ 2 b < ∞ .
(iii) Place-holder.
H is a place-holder to store extra (if needed) R
V.s , e.g. parameters coming from the ﬁnal dense layer.
Inputs.
We will consider a ﬁxed X ⊆ R n 0 ×
d to denote the inputs, with input channel count n 0 , number of pixels d .
Assume x (cid:54) =
for all x ∈ X and |X| , the cardinality of the inputs, is ﬁnite.
However, the simultaneous limit ( § A.5.4) result can be extended to a countably-inﬁnite input indexing spaces X for certain topologies via an argument presented in Matthews et al.
(2018b, section 2.2), allowing to infer weak convergence on X from convergence on any ﬁnite subset (which is the case we consider in this text; see also Billingsley (1999, page 19) for details).
Notation, shapes, and indexing.
We adopt the notation, shape, and indexing convention similar to § 2.1, which the reader is encouraged to review.
We emphasize that whenever an index is omitted, the variable is assumed to contain all possible entries along the respective dimension (e.g. whole X if x is omitted, or all n l channels if the channel i is omitted).
In addition, to simplify the notation, we set v β = 1 2 k +1 .
A.5. P RELIMINARY
We will use the following well-known theorem.
Theorem A.1.
Let X, { X n } n ∈ N be R.V.s in R m .
The following are equivalent:
(i) X n D → X (converges in distribution / converges weakly), (ii) (Portmanteau Theorem)
For all bounded continuous function f : R m → R
[Equation]
(iii) (L ´
evy’s Continuity Theorem)
The characteristic functions of X n , i.e. E
[Equation]
X n
[Equation]
converge to that of X pointwisely, i.e. for all t ∈ R m ,
[Equation]
where i denotes the imaginary unit.
Using the equivalence between (i) and (iii), it is straightforward to show that
[Equation]
In particular,
[Equation]
A.5. S EQUENTIAL LIMIT
In this section, we construct a Gaussian process using an inﬁnite channel CNN with the limits taking sequentially.
We will show that pre-activation functions of different channels are i.i.d.
Gaussian with mean zero and covariance that can be computed recursively.
Let Ψ 1 denote the following space of functions:
Uniformly square-integrable: for every R ≥ 1 ,
[Equation]
Let N ∗ =
N \{ 0 } .
We construct Gaussian processes recursively using an inﬁnite width neural network architecture with the following procedure:
(i)
If l > 0 , assume { z l − 1 , ∞ i } i ∈ N ∗ are i.i.d.
Gaussian with mean zero and variance K l − 1 ∞ and independent from future events (i.e. independent from the σ -alegbra generated by all R.V.s with layer index greater than ( l − 1) ).
For n ∈ N ∗ , let
[Equation]
where [ n ] ≡
n
} and
[Equation]
±
[Equation]
] ≡
[Equation]
(ii) Prove that for any m ≥ 1 , (cid:104)
z l,ni (cid:105) i ∈ [ m ] ⊆
R |X| d converges in distribution to a multivariate normal with mean zero and covariance A (cid:0) K l ∞ (cid:1) ⊗ I m , where K l ∞ is deﬁned identically to Equation 10.
(iii) Deﬁne random variable (cid:104)
z l, ∞
i (cid:105) i ∈ [ m ] in (Ω , F , P ) with the same distribution as the limit in (ii) so that they are also independent from any future events, i.e. independent from the σ -algebra generated by all R.V.s with layer index greater than l .
In (iii), we implicitly assume the probability space (Ω , F , P ) has enough capacity to ﬁt in all R.V.s generated by the above procedure, if not we will extend the sample space Ω using product measures.
In what follows, we use the central limit theorem to prove (ii).
Theorem A.2.
If φ ∈ Ψ 1 , then for every l ≥ 0 and every m ≥ 1 , (cid:104) z l,ni (cid:105) i ∈ [ m ] ⊆
R |X| d converges in distribution to a multivariate normal with mean zero and covariance A (cid:0) K l ∞ (cid:1) ⊗ I m .
Proof.
We proceed by induction.
This is obvious in the base case l = 0 , since the weights and biases are assumed to be independent Gaussian.
Now we assume the theorem holds for l − 1 .
This implies { y li } i ∈ N ∗ =
(cid:110) φ ( z l − 1 , ∞ i ) (cid:111)
i ∈ N ∗ are i.i.d.
random variables.
Choose a vector a ≡ [ a i ( x )]
Ti ∈ [ m ] ,x ∈X ∈ R m |X| .
Then (cid:88)
i ∈ [ m ] x ∈X
[Equation]
It is not difﬁcult to see that u j are i.i.d.
and v is Gaussian.
Then we can use the central limit theorem to conclude that the above converges in distribution to a Gaussian, once we verify the second moment of u j is ﬁnite.
Using the fact that (cid:110) ω lij,β (cid:111)
i,β is a collection of independent R.V.s and integrating over these R.V.s ﬁrst, we get
[Equation]
where by ¯ A we denote the linear transformation part of A from Equation 5, i.e. A without the translation term σ 2 b :
[Equation]
To prove ﬁniteness of the second moment in Equation 36, it is sufﬁcient to show that all the diagonal terms of K l ∞ are ﬁnite.
This easily follows from the assumption φ ∈ Ψ 1 and the deﬁnition of
K l
[Equation]
Together with the distribution of v (whose covariance is straightforward to compute), the joint distribution of (cid:104) z l,ni (cid:105) i ∈ [ m ] converges weakly to a mean zero Gaussian with covaraince matrix A (cid:0) K l ∞ (cid:1) ⊗ I m by Theorem A.1 (Equation 26).
Remark A.1.
The results of Theorem A.2 can be strengthened / extended in many directions.
(1) Same analysis carries over (and the covariance matrix can be computed without much extra effort) if we stack a channel-wise deterministic afﬁne transform after the convolution operator.
Note that average pooling (not max pooling, which is not afﬁne), global average pooling and the convolutional striding are particular examples of such afﬁne tranforms.
Moreover, valid padding (i
e. no padding) convolution can be regarded as subsampling (a linear projection) operator composed with the regular circular padding.
(2) Same analysis applies to max pooling, but computing the covariance may require non-trivial effort.
Let m denote the max pooling operator and assume it is applied right after the activation function.
The assumption (cid:8) y li (cid:9)
i
[Equation] [Equation]
[Equation]
i ∈ [ n ] are i.i.d.
implies (cid:8) m (cid:0) y li (cid:1)(cid:9) i ∈
[ n ] are also i
i
d. Then we can proceed exactly as above except for verifying the ﬁniteness of second moment of m ( y l i ) with the following trivial estimate:
[Equation]
where s is the window size of the max pooling.
In general, one can stack a channel-wise deterministic operator op on { y li } so long as the second moment of { op ( y li ) } is ﬁnite.
One can also stack a stochastic operator (e.g. dropout), so long as the outputs are still channel-wisely i.i.d.
and have ﬁnite second moments.
A.5. S IMULTANEOUS LIMIT
In this section, we present a sufﬁcient condition on the activation function φ so that the neural networks converge to a Gaussian process as all the widths approach to inﬁnity simultaneously.
Precisely, let t ∈ N and for each l ≥ 0 , let n l : N → N be the width function at layer l (by convention n 0 ( t )
= n 0 ).
We are interested in the simultaneous limit n l = n l ( t ) → ∞ as t → ∞ , i.e., for any ﬁxed L ≥
[Equation]
Deﬁne a sequence of ﬁnite channel CNNs as follows:
[Equation]
This network induces a sequence of covariances matrices K lt (which are R.V.s ): for l ≥ 0 and t ≥ 0 , for x, x (cid:48) ∈
X
[Equation]
We make an extra assumption on the parameters.
Assumption : all R.V.s in W are Gaussian distributed.
Notation.
Let PSD m denote the set of m × m positive semi-deﬁnite matrices and for R ≥ 1 , deﬁne
[Equation]
1 ≤ α ≤ m } .
Further let T ∞ :
PSD 2 → R be a function given by
[Equation]
and C k ( φ, R ) (may equal to ∞ ) denotes the uniform upper bound for the k -th moment
[Equation]
Let Ψ denotes the space of measurable functions with the following properties:
Uniformly square-integrable: C 2 ( φ, R ) < ∞ .
Lipschitz continuity: for every R ≥ 1 , there exists β = β ( φ, R ) > 0 such that for all Σ , Σ (cid:48) ∈
[Equation]
Uniform convergence in probability: for every R ≥ 1 and every ε > 0 there exists a positive sequence ρ n ( φ, ε, R ) with ρ n ( φ, ε, R ) → 0 as n → ∞ such that for every Σ ∈ PSD 2 ( R ) and any { ( x i , y i
[Equation]
We will also use Ψ 1 , Ψ 2 and Ψ 3 to denote the spaces of functions satisfying property 1, propertyand property 3, respectively.
It is not difﬁcult to see that for every i , Ψ
i is a vector space, and so is Ψ =
∩ i Ψ i .
Finally, we say that a function f : R → R is exponentially bounded if there exist a, b > 0 such that
[Equation]
We now prove our main result presented in § 2.2.3 through the following three theorems.
Theorem A.3.
If φ (cid:48) is exponentially bounded then φ ∈ Ψ .
Theorem A.4.
If φ ∈ Ψ , then for l ≥ 0 , K lt P → K l ∞ .
Theorem A.5.
If for l ≥ 0 , K lt P → K l ∞ and m ≥ 1 , the joint distribution of (cid:104)
z l,tj (cid:105) j ∈ [ m ] converges in distribution to a multivariate normal distribution with mean zero and covariance A (cid:0) K l ∞ (cid:1) ⊗ I m .
The proofs of Theorems A.3, A.4, and A.5 can be found in § A.5.7, § A.5.6, and § A.5.5 respectively.
A.5. P ROOF OF T HEOREM A.
It sufﬁces to prove that for any vector [
[Equation]
Indeed
[Equation]
the characteristic function
[Equation]
Note that conditioned on (cid:110)
y l,tj (cid:111)
j ∈ [ n l ( t )] , the exponent in the above expression is just a linear combination of independent Gaussian R.V.s (cid:110) ω
lij,β , b li (cid:111) , which is also a Gaussian.
Integrating out these
R.V.s using the formula of the characteristic function of Gaussian distribution yields
[Equation]
where have used K lt P −→ K l ∞ in the last step.
Therefore, Equation 50 is true by Theorem A.1 (iii).
Remark A.2.
We brieﬂy comment how to handle the cases when stacking an average pooling, a subsampling or a dense layer after ﬂattening the activations in the last layer.
(i)
Global Average pooling / subsampling.
Let B ∈ R 1 × d be any deterministic linear functional deﬁned on R d .
The fact that
[Equation]
implies that the empirical covariance of (cid:110)
By l,tj (cid:111)
[Equation] [Equation]
where B ⊗|X| ∈ R 1 × d |X| , |X| copies of B .
Invoking the same “characteristic function” arguments as above, it is not difﬁcult to show that stacking a dense layer (assuming the weights and biases are drawn from i.i.d.
Gaussian with mean zero and variances σ 2 ω and σ 2 b , and are properly normalized) on top of (cid:110)
By l,tj (cid:111) the outputs are i.i.d.
Gaussian with mean zero and covariance σ 2 ω B ⊗|X| K l ∞ (cid:0) B ⊗|X| (cid:1) T + σ 2 b .
Taking B = (cid:0) 1 d , . .
, 1 d (cid:1) ∈
R 1 × d (cid:0) or B =
e α ∈ R 1 × d (cid:1) implies the result of global average pooling ( § 3.2.1, Equation 17), or subsampling ( § 3.2.2, Equation 18).
(ii) Vectorization and a dense layer.
Let (cid:8) ω lij,α (cid:9) i ∈ [ m ] ,j ∈ [ n l ( t )] ,
α ∈
[ d ] be the weights of the dense layer, ω lij,α represents the weight connecting the α -th pixel of the j -channel to the i -th output.
Note that the range of α is [ d ] not [ ± k ] because there is no weight sharing.
Deﬁne the outputs to be
[Equation]
Now let [ a i ( x )] i ∈ [ m ] ,x ∈X ∈ R m |X| and compute the characteristic function of
[Equation]
Using the fact E ω ij,α ω i (cid:48) j (cid:48) ,α (cid:48)
=
unless ( ij, α ) =
( i (cid:48) j (cid:48) , α (cid:48) )
and integrating out the R.V.s of the dense layer, the characteristic function is equal to
E
[Equation]
) where ¯ tr denotes the mean trace operator acting on the pixel by pixel matrix, i.e. the functional computing the mean of the diagonal terms of the pixel by pixel matrix.
Therefore [ f i ] i ∈ [ m ] converges weakly to a mean zero Gaussian with covariance (cid:2) σ 2 ω ¯ tr (cid:0) K l ∞ ( x, x
[Equation]
) (cid:1)
+ σ
[Equation]
(cid:3) x
[Equation]
x
[Equation]
∈X ⊗
I m in the case of vectorization ( § 3.1).
A.5. P ROOF OF T HEOREM A.
Proof.
We recall K Lt and K L ∞ to be random matrices in R d |X|× d |X| , and we will prove convergence K Lt
P −−−→ t →∞
K L ∞ with respect to (cid:107) · (cid:107) ∞ , the pointwise (cid:96) ∞ -norm ( i.e. (cid:107) K (cid:107) ∞ =
max x,x (cid:48) ,α,α (cid:48) | K α,α (cid:48)
( x, x (cid:48) )
| ) .
Note that due to ﬁnite dimensionality of K L , convergence w.r.t.
all other norms follows.
We ﬁrst note that the afﬁne transform A is σ 2 ω -Lipschitz and property 2 of Ψ implies that the C operator is β -Lipschitz.
Indeed, if we consider
[Equation]
63) then [ C ( K )]
α,α (cid:48) ( x, x (cid:48) )
=
T ∞ (Σ) .
Thus C ◦ A is σ 2 ω β -Lipschitz.
We now prove the theorem by induction.
Assume K lt P → K l ∞ as t → ∞ (obvious for l = 0 ).
Let ε > 0 be sufﬁciently small so that the ε 2 β -neighborhood of A ( K l ∞ ) is contained in PSD |X|
d ( R ) , where we take R to be large enough for K l ∞ to be an interior point of PSD |X| d ( R ) .
Since
[Equation]
to prove K l +1 t
P → K l +1 ∞ , it sufﬁces to show that for every δ > 0 , there is a t ∗ such that for all t > t ∗ ,
P (cid:16)(cid:13)(cid:13)
[Equation]
A
[Equation]
K l
[Equation]
lt (cid:1)(cid:13)(cid:13) ∞ >
[Equation]
(cid:17)
[Equation]
P (cid:16)(cid:13)(cid:13)
[Equation]
By our induction assumption, there is a t l such that for all t > t l
[Equation]
Since C ◦ A is σ 2 ω β -Lipschitz, then
[Equation]
To bound the second term in Equation
[Equation]
let U ( t ) denotes the event
[Equation]
and U ( t ) c its complement.
For all t > t l it’s probability is
[Equation]
[ assumption on small ε ]
[Equation] [Equation]
[Equation] [Equation]
Finally, denote
[Equation]
The fact
[Equation]

 implies
[Equation]
Where the maximum is taken over all ( x, x (cid:48) , α, α (cid:48) )
∈
X 2 ×
[ d ] .
Consider a ﬁxed κ ∈ PSD |X| d , and deﬁne Σ
[Equation]
75) a deterministic matrix in PSD 2 .
Then
[Equation]
) and, conditioned on K lt ,
[Equation]
77) where (cid:110)(cid:16) z l,ti,α ( x ) , z l,ti,α (cid:48)
( x (cid:48) ) (cid:17)
| K lt = κ (cid:111) i ∈
[ n l +1 ( t )] are i.i.d.
∼ N (0 , Σ ( κ, α, α (cid:48) , x, x (cid:48) )) .
Then if Σ ( κ, α, α (cid:48) , x, x (cid:48) ) ∈ PSD 2 ( R ) we can apply property 3 of Ψ to conclude that:
[Equation]
[Equation]
However, if Σ ( κ, α, α (cid:48) , x, x (cid:48) )
(cid:54)∈ PSD 2 ( R ) , then necessarily A ( κ ) (cid:54)∈ PSD |X| d ( R ) (since Σ ( κ, α, α (cid:48) , x, x (cid:48) )
∈ PSD 2 ), ensuring that P (cid:0) U ( t ) | K lt = κ (cid:1) = 0 .
Therefore Equation 78 holds for any κ ∈ PSD |X| d , and for any ( x, x (cid:48) , α, α (cid:48) )
∈
X 2 ×
[ d ] 2 .
We further remark that ρ n l +1 ( t ) (cid:0) φ, R, ε 2 (cid:1) is deterministic and does not depend on ( κ, x, x (cid:48) , α, α (cid:48) ) .
Marginalizing out K lt and maximizing over ( x, x (cid:48) , α, α (cid:48) ) in Equation 78 we conclude that
[Equation]
x (cid:48)
[Equation]
Since ρ n (cid:0) φ, R, ε 2 (cid:1) → 0 as n → ∞ , there exists n such that for any n l +1 ( t ) ≥ n ,
[Equation]
and, substituting this bound in Equation 74,
[Equation]
Therefore we just need to choose t l +1 > t l so that n l +1 ( t ) ≥ n for all t > t l +1 .
Remark A.3.
We list some directions to strengthen / extent the results of Theorem A.4 (and thus
Theorem A.5) using the above framework.
Consider stacking a deterministic
channel-wise linear operator right after the convolutional layer.
Again, strided convolution, convolution with no (“valid”) padding and (non-global) average pooling are particular examples of this category.
Let B ∈ R d (cid:48)
×
d denote a linear operator.
Then the recurrent formula between two consecutive layers is
[Equation]
where B is the linear operator on the covariance matrix induced by B .
Conditioned on
K lt , since the outputs after applying the linear operator B are still i.i.d.
Gaussian (and the property 3 is applicable), the analysis in the above proof can carry over with A replaced by
B ◦ A .
More generally, one may consider inserting an operator op (e.g. max-pooling, dropout and more interestingly, normalization) in some hidden layer.
It is possible to drop the Gaussianity assumption on the weight parameters by incorporating the proof of the central limit theorem using characteristic function into the above arguments.
A.5. P ROOF OF T HEOREM A.
Note that exponentially bounded functions contain all polynomials, and are closed under multiplication and integration in the sense that for any constant C the function
[Equation]
is also exponentially bounded.
Theorem A.3 is a consequence of the following lemma.
Lemma A.6.
The following is true:
1. for k ≥ 1 , C k ( φ, R ) < ∞ if φ is exponentially bounded.
φ ∈ Ψ 2 if φ (cid:48) exists a.e. and is exponentially bounded.
φ ∈ Ψ 3 if C 4 ( φ, R ) < ∞ .
Indeed, if φ (cid:48) is exponentially bounded, then φ is also exponentially bounded.
By the above lemma, φ ∈ Ψ .
Proof of Lemma A.6.
We prove the ﬁrst statement.
Assume | φ ( x ) | ≤ ae b | x | .
[Equation]
Thus
[Equation]
To prove the second statement, let Σ , Σ (cid:48) ∈
PSD 2 ( R ) and deﬁne A (similarly for A (cid:48) ):
[Equation]
Then AA T = Σ (and A (cid:48)
A (cid:48) T = Σ (cid:48) ).
Let
A ( t ) ≡ (1 − t )
A + tA (cid:48) , t ∈ [0 , 1] (87) and f ( w ) ≡ φ ( x ) φ ( y ) where w ≡ ( x, y ) T .
Since φ (cid:48) is exponentially bounded, φ is also exponentially bounded.
In addition, p ( (cid:107) w (cid:107) 2 ) (cid:107)∇ f ( w ) (cid:107) 2 is exponentially bounded for any polynomial p ( (cid:107) w (cid:107) 2 ) .
Applying the Mean Value Theorem (we use the notation (cid:46) to hide the dependence on R and other absolute constants)
[Equation]
Note that the operator norm is bounded by the inﬁnity norm (up to a multiplicity constant) and (cid:107) w
[Equation] [Equation]
f (
[Equation]
[Table]
[Equation]
2 is exponentially bounded.
There is a constant a (hidden in (cid:46) ) and b such that the above is bounded by
[Equation]
Here we have applied the facts
[Equation]
and
[Equation]
R
[Equation]
3. Chebyshev’s inequality implies
[Equation]
Remark A.4.
In practice the 1 /n decay bound obtained by Chebyshev’s inequality in equationis often too weak to be useful.
However, if φ is linearly bounded, then one can obtain an exponential decay bound via the following concentration inequality:
Lemma A.7.
If | φ ( x ) | ≤ a + b | x | a.e., then there is an absolute constant c > 0 and a constant κ = κ ( a, b, R ) > 0 such that property 3 (Equation 48) holds with
[Equation]
Proof.
We postpone the proof of the following claim.
Claim A.8.
Assume | φ ( x ) | ≤ a + b | x | .
Then there is a κ = κ ( a, b, R ) such that for all Σ ∈
PSD 2 ( R ) and all p ≥ 1 ,
[Equation]
Claim A.8 and the triangle inequality imply
[Equation]
We can apply Bernstein-type inequality (Vershynin, 2010, Lemma 5.16) to conclude that there is a c > 0 such that for every Σ ∈ PSD 2 ( R ) and any { ( x i , y i ) } ni =1 i.i.d.
∼ N (0 , Σ)
[Equation]
n
[Equation]
It remains to prove Claim
A
[Equation]
For p ≥ 1 ,
[Equation]
We applied Cauchy-Schwarz’ inequality in the ﬁrst inequality, the triangle inequality in the second one, the fact Σ 11 , Σ 22 ≤ R in the third one, absolute moments estimate of standard Gaussian in the fourth one, where c (cid:48) is a constant such that
[Equation]
A. G LOSSARY
We use the following shorthands in this work:
NN - neural network; 2.
CNN - convolutional neural network; 3.
LCN - locally-connected network, a.k.a.
convolutional network without weight sharing; 4.
FCN - fully connected network, a
k.a. multilayer perceptron (MLP); 5.
GP - Gaussian process; 6.
X-GP - a GP equivalent to a Bayesian inﬁnitely wide neural network of architecture X ( § 2).
MC-(X-)-GP - a Monte Carlo estimate ( § 4) of the X-GP.
Width, (number of) ﬁlters, (number of) channels represent the same property for CNNs and LCNs.
Pooling - referring to architectures as “with” or “without pooling” means having a single global average pooling layer (collapsing the spatial dimensions of the activations y L +1 )
before the ﬁnal linear FC layer giving the regression outputs z L +1 .
Invariance and equivariance are always discussed w.r.t.
translations in the spatial dimensions of the inputs.
A. E XPERIMENTAL S ETUP
Throughout this work we only consider 3 × 3 (possibly unshared) convolutional ﬁlters with strideand no dilation.
All inputs are normalized to have zero mean and unit variance, i.e. lie on the d − dimensional sphere of radius √ d , where d is the total dimensionality of the input.
All labels are treated as regression targets with zero mean.
I.e. for a single-class classiﬁcation problem with C classes targets are C − dimensional vectors with − 1 /C and ( C − 1) /C
entries in incorrect and correct class indices respectively.
If a subset of a full dataset is considered for computational reasons, it is randomly selected in a balanced fashion.
No data augmentation is used.
All experiments were implemented in Tensorﬂow (Abadi et al., 2016) and executed with the help of
Vizier (Golovin et al., 2017).
All neural networks are trained using Adam (Kingma & Ba, 2015) minimizing the mean squared error loss.
A.7. M ANY - CHANNEL CNN S AND LCN S
Relevant Figures: 3 (b), 5, 6.
We use a training and validation subsets of CIFAR10 of sizes 500 and 4000 respectively.
All images are bilinearly downsampled to 8 × 8 pixels.
All models have 3 hidden layers with an erf nonlinearity.
No (“valid”) padding is used.
Weight and bias variances are set to σ 2 ω ≈ 1 .
7562 and σ 2 b ≈ 0 .
1841 , corresponding to the preactivation variance ﬁxed point q ∗ =
(Poole et al., 2016) for the erf nonlinearity.
NN training proceeds for 2 19 gradient updates, but aborts if no progress on training loss is observed for the last 100 epochs.
If the training loss does not reduce by at least 10 − 4 for 20 epochs, the learning rate is divided by 10 .
All computations are done with 32 -bit precision.
The following NN parameters are considered 9 :
Architecture: CNN or LCN. 2.
Pooling: no pooling or a single global average pooling (averaging over spatial dimensions) before the ﬁnal FC layer.
Number of channels: 2 k for k from 0 to 12 .
Initial learning rate: 10 − k for k from 0 to 15 .
Weight decay: 0 and 10 − k for k from 0 to 8 . 6.
Batch size: 10 , 25 , 50 , 100 , 200 .
For NNs, all models are ﬁltered to only 100% -accurate ones on the training set and then for each conﬁguration of { architecture, pooling, number of channels } the model with the lowest validation loss is selected among the conﬁgurations of { learning rate, weight decay, batch size } .
For GPs, the same CNN-GP is plotted against CNN and LCN networks without pooling.
For
LCN with pooling, inference was done with an appropriately rescaled CNN-GP kernel, i.e. (cid:0)
K vec ∞ − σ 2 b (cid:1) /d
+ σ 2 b , where d is the spatial size of the penultimate layer.
For CNNs with pooling, a Monte Carlo estimate was computed (see § 4) with n = 2 12 ﬁlters and M = 2 6 samples.
For GP inference, the initial diagonal regularization term applied to the training convariance matrix is 10 − 10 ; if the cholesky decompisition fails, the regularization term is increased by a factor ofuntil it either succeeeds or reaches the value of 10 5 , at which point the trial is considered to have failed.
A.7. M ONTE C ARLO E VALUATION OF I NTRACTABLE GP K ERNELS
Relevant Figures: 2, 7.
We use the same setup as in § A.7.1, but training and validation sets of sizes 2000 and 4000 respectively.
For MC-GPs we consider the number of channels n (width in FCN setting) and number of NN instantiations M to accept values of 2 k for k from 0 to 10 .
Kernel distance is computed as:
[Equation]
where K ∞ is substituted with K 2 10 , 2 10 for the CNN-GP pooling case (due to impracticality of computing the exact K pool ∞ ).
GPs are regularized in the same fashion as in § A.7.1, but the regularization factor starts at 10 − 4 and ends at 10 10 and is multiplied by the mean of the training covariance diagonal.
A.7. T RANSFORMING A GP OVER SPATIAL LOCATIONS INTO A GP OVER CLASSES
Relevant Figure: 1.
We use the same setup as in § A.7.2, but rescale the input images to size of 31 × 31 , so that at depth 15 the spatial dimension collapses to a 1 × 1 patch if no padding is used (hence the curve of the
CNN-GP without padding halting at that depth).
For MC-CNN-GP with pooling, we use samples of networks with n = 16 ﬁlters.
Due to computational complexity we only consider depths up to 31 for this architecture.
The number of samples M was selected independently for each depth among (cid:8) 2 k (cid:9) for k from 0 to 15 to maximize the validation accuracy on a separate 500 -points validation set.
This allowed us to avoid the poor conditioning of the kernel.
GPs are regularized in the same fashion as in § A.7.1, but for MLP-GP the multiplicative factor starts at 10 − 4 and ends at 10 10 .
A.7. R ELATIONSHIP TO D EEP S IGNAL P ROPAGATION
Relevant Table: 3.
We use a training and validation subsets of CIFAR10 of sizes 500 and 1000 respectively.
We use the erf nonlinearity.
For CNN-GP, images are zero-padded (“same” padding) to maintain the spatial shape of the activations as they are propagated through the network.
Weight and bias variances (horizontal axis σ 2 ω and vertical axis σ 2 b respectively) are sampled from a uniform grid of size 50 × 50 on the range [0 . 1 , 5] ×
[0 , 2] including the endpoints.
All computations are done with 64 -bit precision.
GPs are regularized in the same fashion as in § A.7.1, but the regularization factor is multiplied by the mean of the training covariance diagonal.
If the experiment fails due to numerical reasons, 0 .
1 (random chance) validation accuracy is reported.
A.7. CNN-GP ON FULL DATASETS
Relevant Table: 2, Figure 3 (a, c).
We use full training, validation, and test sets of sizes 50000 , 10000 , and 10000 respectively for
MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017a), 45000 , 5000 , and 10000 for
CIFAR10 (Krizhevsky, 2009).
We use validation accuracy to select the best conﬁguration for each model (we do not retrain on valdiation sets).
GPs are computed with 64 -bit precision, and NNs are trained with 32 -bit precision.
GPs are regularized in the same fashion as in § A.7.4.
Zero-padding (“same”) is used.
The following parameters are considered:
Architecture: CNN or FCN.
Nonlinearity: erf or ReLU .
Depth: 2 k for k from 0 to 4 (and up to 2 5 for MNIST and Fashion-MNIST datasets).
Weight and bias variances.
For erf : q ∗ from { 0 . 1 , 1 , 2 , . . .
For ReLU : a ﬁxed weight variance σ 2 ω
=
+ 4 e − 16 and bias variance σ 2 b from { 0 . 1 , 1 , 2 , . .
On CIFAR10, we additionally train NNs for 2 18 gradient updates with a batch size of 128 with corresponding parameters in addition to
Pooling: no pooling or a single global average pooling (averaging over spatial dimensions) before the ﬁnal FC layer (only for CNNs).
Number of channels or width: 2 k for k from 1 to 9 (and up to 2 10 for CNNs with pooling in Figure 3, a).
Learning rate: 10 − k × 2
( width × q ∗ ) for k from 5 to 9 , where width is substituted with the number of channels for CNNs and q ∗ is substituted with σ 2 b for ReLU networks.
“Small learning rate” in Table 2 refers to k ∈ { 8 , 9 } .
Weight decay: 0 and 10 − k for k from 0 to 5 .
For NNs, all models are ﬁltered to only 100% -accurate ones on the training set (expect for values in parentheses in Table 2).
The reported values are then reported for models that achieve the best validation accuracy.
A.7. M ODEL COMPARISON ON CIFAR
Relevant Table: 1.
We use the complete CIFAR10 dataset as described in § A.7.5 and consider 8-layer ReLU models with weight and bias variances of σ 2 ω = 2 and σ 2 b = 0 . 01 .
The number of channels / width is set to 2 5 , 2 10 and 2 12 for LCN, CNN, and FCN respectively.
GPs are computed with 64 -bit precision, and NNs are trained with 32 -bit precision.
No padding (“valid”) is used.
NN training proceeds for 2 18 gradient updates with batch size 64 , but aborts if no progress on training loss is observed for the last 10 epochs.
If the training loss does not reduce by at least 10 −for 2 epochs, the learning rate is divided by 10 .
Values for NNs are reported for the best validation accuracy over different learning rates ( 10 − k for k from 2 to 12 ) and weight decay values ( 0 and 10 − k for k from 2 to 7 ).
For GPs, validation accuracy is maximized over initial diagonal regularization terms applied to the training convariance matrix: 10 − k × [mean of the diagonal] for k among 2 , 4 and 9 (if the cholesky decompisition fails, the regularization term is increased by a factor of 10 until it succeeds or k reaches the value of 10 ).