D EEP C ONVOLUTIONAL N ETWORKS AS SHALLOW G AUSSIAN P ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
We show that the output of a (residual) convolutional neural network (CNN) with an appropriate prior over the weights and biases is a Gaussian process
GP
) in the limit of inﬁnitely many convolutional ﬁlters, extending similar results for dense networks.
For a CNN, the equivalent kernel can be computed exactly and, unlike
“deep kernels”, has very few parameters: only the hyperparameters of the original
CNN.
Further, we show that this kernel has two properties that allow it to be computed efﬁciently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one ﬁlter per layer.
The kernel equivalent to a layer
ResNet obtains
% classiﬁcation error on
MNIST, a new record for GPs with a comparable number of parameters.
I NTRODUCTION
Convolutional Neural Networks (CNNs) have powerful pattern-recognition capabilities that have recently given dramatic improvements in important tasks such as image classiﬁcation (Krizhevsky et al., 2012).
However, as CNNs are increasingly being applied in real-world, safety-critical domains, their vulnerability to adversarial examples (Szegedy et al., 2013; Kurakin et al., 2016), and their poor uncertainty estimates are becoming increasingly problematic.
Bayesian inference is a theoretically principled and demonstrably successful (Snoek et al., 2012; Deisenroth & Rasmussen, 2011) framework for learning in the face of uncertainty, which may also help to address the problems of adversarial examples (Gal & Smith, 2018).
Unfortunately, Bayesian inference in CNNs is extremely difﬁcult due to the very large number of parameters, requiring highly approximate factorised variational approximations (Blundell et al., 2015; Gal & Ghahramani, 2015), or requiring the storage (Lakshminarayanan et al., 2017) of large numbers of posterior samples (Welling & Teh, 2011; Mandt et al., 2017).
Other methods such as those based on Gaussian Processes (GPs) are more amenable to Bayesian inference, allowing us to compute the posterior uncertainty exactly (Rasmussen & Williams, 2006).
This raises the question of whether it might be possible to combine the pattern-recognition capabilities of CNNs with exact probabilistic computations in GPs.
Two such approaches exist in the literature.
First, deep convolutional kernels (Wilson et al., 2016) parameterise a GP kernel using the weights and biases of a CNN, which is used to embed the input images into some latent space before computing their similarity.
The CNN parameters of the resulting kernel then have to be optimised by gradient descent.
However, the large number of kernel parameters in the CNN reintroduces the risk of overconﬁdence and overﬁtting.
To avoid this risk, we need to infer a posterior over the CNN kernel parameters, which is as difﬁcult as directly inferring a posterior over the parameters of the original CNN.
Second, it is possible to deﬁne a convolutional GP (van der
Wilk et al., 2017) or a deep convolutional GP (Kumar et al., 2018)
by deﬁning a GP that takes an image patch as input, and using that GP as a component in a larger CNN-like system.
However, inference in such systems is very computationally expensive, at least without the use of potentially severe variational approximations (van der Wilk et al., 2017).
An alternative approach is suggested by the underlying connection between Bayesian neural networks (NNs) and GPs.
In particular, Neal (1996) showed that the function deﬁned by a single-layer fully-connected NN with inﬁnitely many hidden units, and random independent zero-mean weights and biases is equivalent to a GP, implying that we can do exact Bayesian inference in such a NN by working with the equivalent GP.
Recently, this result was extended to arbitrarily deep fullyconnected NNs with inﬁnitely many hidden units at each layer (Lee et al., 2017; Matthews et al., 2018a).
However, these fully-connected networks are rarely used in practice, as they are unable to exploit important properties of images such as translational invariance, raising the question of whether state-of-the-art architectures such as CNNs (LeCun et al., 1990) and ResNets
(He et al., 2016a) have equivalent GP representations.
Here, we answer in the afﬁrmative, giving the GP kernel corresponding to arbitrarily deep CNNs and to (convolutional) residual neural networks (He et al., 2016a).
In this case, if each hidden layer has an inﬁnite number of convolutional ﬁlters , the network prior is equivalent to a GP.
Furthermore, we show that two properties of the GP kernel induced by a CNN allow it to be computed very efﬁciently.
First, in previous work it was necessary to compute the covariance matrix for the output of a single convolutional ﬁlter applied at all possible locations within a single image (
van der
Wilk et al., 2017), which was prohibitively computationally expensive.
In contrast, under our prior, the downstream weights are independent with zero-mean, which decorrelates the contribution from each location, and implies that it is necessary only to track the patch variances, and not their covariances.
Second, while it is still necessary to compute the variance of the output of a convolutional ﬁlter applied at all locations within the image, the speciﬁc structure of the kernel induced by the CNN means that the variance at every location can be computed simultaneously and efﬁciently as a convolution.
Finally, we empirically demonstrate the performance increase coming from adding translationinvariant structure to the GP prior.
Without computing any gradients, and without augmenting the training set (e.g. using translations), we obtain 0.84% error rate on the MNIST classiﬁcation benchmark, setting a new record for nonparametric GP-based methods.
GP BEHAVIOUR IN A CNN
For clarity of exposition, we will treat the case of a 2D convolutional NN.
The result applies straightforwardly to n D convolutions, dilated convolutions and upconvolutions (“deconvolutions”), since they can be represented as linear transformations with tied coefﬁcients (see Fig. 1).
2. A 2D CONVOLUTIONAL NETWORK PRIOR
The network takes an arbitrary input image X of height H (0) and width D (0) , as a C (0) ×
( H (0) D (0) ) real matrix.
Each row, which we denote x 1 , x 2 , . .
, x C (0) , corresponds to a channel of the image (e.g. C (0)
= 3 for RGB), ﬂattened to form a vector.
The ﬁrst activations A (1) ( X ) are a linear transformation of the inputs.
For i ∈ { 1 , . . .
, C (1) } :
[Equation]
We consider a network with L hidden layers.
The other activations of the network, from A (2) ( X ) up to A ( L +1) ( X ) , are deﬁned recursively:
[Equation] [Equation]
The activations A ( (cid:96) )
( X ) are C ( (cid:96) )
× ( H ( (cid:96) )
D ( (cid:96) ) ) matrices.
Each row a ( (cid:96) +1)
i represents the ﬂattened j th channel of the image that results from applying a convolutional ﬁlter to φ ( A ( (cid:96) )
( X )) .
The structure of the pseudo-weight matrices W ( (cid:96) +1)
i,j and biases b ( (cid:96) +1) i , for i ∈ { 1 , . .
, C ( (cid:96) +1) } and j ∈ { 1 , . . .
, C ( (cid:96) ) } , depends on the architecture.
For a convolutional layer, each row of W ( (cid:96) +1)
i,j represents a position of the ﬁlter, such that the dot product of all the rows with the image vector x j represents applying the convolutional ﬁlter U ( (cid:96) +1)
i,j to the j th channel.
Thus, the elements of
Filter U
[Figure]
Input image’s j th channel: =
Resulting convolution
[Figure]
µ t h c onv .
p a t c h each row of W ( (cid:96) +1)
i,j are: 0 where the ﬁlter does not apply and the corresponding element of U ( (cid:96) +1)
i,j where it does, as illustrated in Fig.
The outputs of the network are the last activations, A ( L +1)
( X ) .
In the classiﬁcation or regression setting, the outputs are not spatially extended, so we have H ( L +1)
=
D ( L +1) = 1 , which is equivalent to a fully-connected output layer.
In this case, the pseudo-weights W ( L +1)
i,j only have one row, and the activations a ( L +1) i are single-element vectors.
Finally, we deﬁne the prior distribution over functions by making the ﬁlters U ( (cid:96) )
i,j and biases b ( (cid:96) )
i be independent Gaussian random variables (RVs).
For each layer (cid:96) , channels i, j and locations within the ﬁlter x, y :
[Equation]
[Equation]
Note that, to keep the activation variance constant, the weight variance is divided by the number of input channels.
The weight variance can also be divided by the number of elements of the ﬁlter, which makes it equivalent to the NN weight initialisation scheme introduced by He et al.
(2016a).
2. A RGUMENT FOR GP BEHAVIOUR
We follow the proofs by Lee et al.
(2017) and Matthews et al.
(2018a) to show that the output of the
CNN described in the previous section, A ( L +1) , deﬁnes a GP indexed by the inputs, X .
Their proof (Lee et al., 2017) proceeds by applying the multivariate Central Limit Theorem (CLT) to each layer in sequence, i.e. taking the limit as N (1) → ∞ , then N (2) → ∞ etc, where N ( (cid:96) ) is the number of hidden units in layer (cid:96) .
By analogy, we sequentially apply the multivariate CLT by taking the limit as the number of channels goes to inﬁnity, i.e. C (1) → ∞ , then C (2) → ∞ etc.
While this is the simplest approach to taking the limits, other potentially more realistic approaches also exist (Matthews et al., 2018a).
The fundamental quantity we consider is a vector formed by concatenating the feature maps (or equivalently channels), a ( (cid:96) )
j ( X ) and a ( (cid:96) )
j ( X (cid:48) ) from data points X and X (cid:48) ,
[Equation]
This quantity (and the following arguments) can all be extended to the case of countably ﬁnite numbers of input points.
Induction base case.
For any pair of data points, X and X (cid:48) the feature-maps corresponding to the j th channel, a (1) j
( X , X (cid:48) ) have a multivariate Gaussian joint distribution.
This is because each element is a linear combination of shared Gaussian random variables: the biases, b (0) j and the ﬁlters,
U (0) j, : .
Following Eq.
[Equation]
where 1 is a vector of all-ones.
While the elements within a feature map display strong correlations, different feature maps are independent and identically distributed (iid) conditioned on the data (i.e. a (1) i ( X , X (cid:48) ) and a (1) i (cid:48)
( X , X (cid:48) ) are iid for i (cid:54)
=
i (cid:48) ), because the parameters for different feature-maps (i.e. the biases, b (1) i and the ﬁlters, W (1) i, : ) are themselves iid.
Induction step.
Consider the feature maps at the (cid:96) th layer, a ( (cid:96) )
j ( X , X (cid:48) ) , to be iid multivariate
Gaussian RVs (i.e. for j (cid:54) =
j (cid:48) , a ( (cid:96) )
j ( X , X (cid:48) ) and a ( (cid:96) )
j (cid:48)
( X , X (cid:48) ) are iid).
Our goal is to show that, taking the number of channels at layer (cid:96) to inﬁnity (i.e. C ( (cid:96) ) → ∞ ), the same properties hold at the next layer (i.e. all feature maps, a ( (cid:96) +1)
i ( X , X (cid:48) ) , are iid multivariate Gaussian RVs).
Writing eq.
(2) for two training examples, X and X (cid:48) , we obtain,
[Equation]
We begin by showing that a ( (cid:96) +1)
i ( X , X (cid:48) ) is a multivariate Gaussian RV.
The ﬁrst term is multivariate Gaussian, as it is a linear function of b ( (cid:96) +1)
i , which is itself iid Gaussian.
We can apply the multivariate CLT to show that the second term is also Gaussian, because, in the limit as C ( (cid:96) )
→ ∞ , it is the sum of inﬁnitely many iid terms: a ( (cid:96) )
j ( X , X (cid:48) ) are iid by assumption, and W ( (cid:96) +1)
i,j are iid by deﬁnition.
Note that the same argument applies to all feature maps jointly, so all elements of
A ( (cid:96) +1)
( X , X (cid:48) ) (deﬁned by analogy with eq. 4) are jointly multivariate Gaussian.
Following Lee et al.
(2017), to complete the argument, we need to show that the output feature maps are iid, i.e. a ( (cid:96) +1)
i
[Equation]
) and a ( (cid:96) +1) i (cid:48)
( X , X (cid:48) ) are iid for i (cid:54)
=
i (cid:48) .
They are identically distributed, as b ( (cid:96) +1) i and W ( (cid:96) +1)
i,j are iid and φ ( a ( (cid:96) )
j ( X , X (cid:48) )) is shared.
To show that they are independent, remember that a ( (cid:96) +1) i ( X , X (cid:48) ) and a ( (cid:96) +1) i (cid:48)
( X , X (cid:48) ) are jointly Gaussian, so it is sufﬁcient to show that they are uncorrelated, and we can show that they are uncorrelated because the weights, W ( (cid:96) +1)
i,j are independent with zero-mean, eliminating any correlations that might arise through the shared
RV, φ ( a ( (cid:96) )
j ( X , X (cid:48) )) .
In the appendix, we consider the more complex case where we take limits simultaneously.
T HE C ONV N ET AND R ES N ET KERNELS
Here we derive a computationally efﬁcient kernel corresponding to the CNN described in the previous section.
It is surprising that we can compute the kernel efﬁciently because the feature maps, a ( (cid:96) )
i ( X ) , display rich covariance structure due to the shared convolutional ﬁlter.
Computing and representing these covariances would be prohibitively computationally expensive.
However, in many cases we only need the variance of the output, e.g. in the case of classiﬁcation or regression with a ﬁnal dense layer.
It turns out that this propagates backwards through the convolutional network, implying that for every layer, we only need the “diagonal covariance” of the activations: the covariance between the corresponding elements of a ( (cid:96) )
j ( X ) and a ( (cid:96)
[Table] [Table]
3. GP MEAN AND COVARIANCE
A GP is completely speciﬁed by its mean and covariance (kernel) functions.
These give the parameters of the joint Gaussian distribution of the RVs indexed by any two inputs, X and X (cid:48) .
For the purposes of computing the mean and covariance, it is easiest to consider the network as being written entirely in index notation,
[Equation]
where (cid:96) and (cid:96) + 1 denote the input and output layers respectively, j and i ∈ { 1 , . .
, C ( (cid:96) +1) } denote the input and output channels, and ν and µ ∈ { 1 , . . .
, H ( (cid:96) +1) D (
[Equation]
+1) } denote the location within the input and output channel or feature-maps.
The mean function is thus easy to compute
[Equation]
as b ( (cid:96) +1)
i and W ( (cid:96) +1)
i,j,µ,ν have zero mean, and W ( (cid:96) +1)
i,j,ν,µ are independent of the activations at the previous layer, φ ( A ( (cid:96) )
j,ν ( X )) .
Now we show that it is possible to efﬁciently compute the covariance function.
This is surprising because for many networks, we need to compute the covariance of activations between all pairs of locations in the feature map
(i.e. C (cid:104) A ( (cid:96) +1) i,µ
( X ) , A ( (cid:96) +1) i,µ (cid:48)
( X (cid:48) )
(cid:105) for µ, µ (cid:48) ∈ { 1 , . . .
, H ( (cid:96) +1) D ( (cid:96) +1) } ) and this object is extremely high-dimensional, N
( H ( (cid:96) +1) D ( (cid:96) +1) ) 2 .
However, it turns out that we only need to consider the “diagonal” covariance, (i.e. we only need C (cid:104) A
( (cid:96) +1) i,µ ( X ) , A ( (cid:96) +1) i,µ
( X (cid:48) )
(cid:105) for µ ∈ { 1 , . . .
, H ( (cid:96) +1) D ( (cid:96) +1) } ), which is a more manageable quantity of size N
( H ( (cid:96) +1) D ( (cid:96) +1) ) .
This is true at the output layer ( L + 1) : in order to achieve an output suitable for classiﬁcation or regression, we use only a single output location H ( L +1)
=
D ( L +1) = 1 , with a number of “channels” equal to the number of of outputs/classes, so it is only possible to compute the covariance at that single location.
We now show that, if we only need the covariance at corresponding locations in the outputs, we only need the covariance at corresponding locations in the inputs, and this requirement propagates backwards through the network.
Formally, as the activations are composed of a sum of terms, their covariance is the sum of the covariances of all those underlying terms,
[Equation]
As the terms in the covariance have mean zero
[Equation]
and as the weights and activations from the previous layer are independent,
[Equation]
The weights are independent for different channels: W ( (cid:96) +1)
i,j and W ( (cid:96) +1)
i,j (cid:48) are iid for j (cid:54) =
j (cid:48) , so
E (cid:104) W ( (cid:96) +1)
i,j,µ,ν W ( (cid:96) +1)
i,j (cid:48) ,µ,ν (cid:48) (cid:105) = 0 for j (cid:54) =
j (cid:48)
Further, each row µ of the weight matrices W (
[Equation]
+1)
i,j only contains independent variables or zeros (Fig. 1), so E (cid:104) W ( (cid:96) +1)
i,j,µ,ν W ( (cid:96) +1)
i,j (cid:48) ,µ,ν (cid:48) (cid:105) = 0 for ν (cid:54) =
ν (cid:48) .
Thus, we can eliminate the sums over j (cid:48) and ν (cid:48) :
C
[Equation]
i
[Equation]
µ
The µ th row of W ( (cid:96) +1)
i,j is zero for indices ν that do not belong to its convolutional patch, so we can restrict the sum over ν to that region.
We also deﬁne v (1) g
( X , X (cid:48) ) , to emphasise that the covariances are independent of the output channel, j .
The variance of the ﬁrst layer is
[Equation]
And we do the same for the other layers,
[Equation]
where
[Equation]
is the covariance of the activations, which is again independent of the channel.
3. C OVARIANCE OF THE ACTIVITIES
The elementwise covariance in the right-hand side of Eq.
(11) can be computed in closed form for many choices of φ if the activations are Gaussian.
For each element of the activations, one needs to keep track of the 3 distinct entries of the bivariate covariance matrix between the inputs,
K ( (cid:96) +1) µ ( X , X ) , K ( (cid:96) +1) µ ( X , X (cid:48) ) and K ( (cid:96) +1) µ ( X (cid:48) , X (cid:48) ) .
For example, for the ReLU nonlinearity ( φ ( x ) = max(0 , x ) ), one can adapt Cho & Saul (2009) in the same way as Matthews et al.
(2018a, section 3) to obtain
[Equation]
where θ ( (cid:96) ) ν =
[Equation]
K ( (cid:96) )
ν
[Equation]
X , X (cid:48)
[Equation]
/ (cid:113) K
[Equation]
(cid:96) )
ν
( X
[Equation]
X
[Equation]
K ( (cid:96) )
ν
( X (cid:48)
[Equation]
X (cid:48)
[Equation]
(cid:19) .
3. E FFICIENCY OF THE C ONV N ET KERNEL
We now have all the pieces for computing the kernel, as written in Algorithm 1.
Putting together Eq.
(11) and Eq.
(13) gives us the surprising result that the diagonal covariances of the activations at layer (cid:96) + 1 only depend on the diagonal covariances of the activations at layer (cid:96) .
This is very important, because it makes the computational cost of the kernel be within a constant factor of the cost of a forward pass for the equivalent CNN with 1 ﬁlter per layer.
Thus, the algorithm is more efﬁcient that one would naively think.
A priori, one needs to compute the covariance between all the elements of a ( (cid:96) )
j ( X ) and a ( (cid:96) )
j ( X (cid:48) ) combined, yielding a 2 H ( (cid:96) )
D ( (cid:96) )
×
2 H ( (cid:96) ) D ( (cid:96) ) covariance matrix for every pair of points.
Instead, we only need to keep track of a
H ( (cid:96) ) D ( (cid:96) )
-dimensional vector per layer and pair of points.
Furthermore, the particular form for the kernel (eq. 1 and eq. 2) implies that the required variances and covariances at all required locations can be computed efﬁciently as a convolution.
3. K ERNEL FOR A RESIDUAL CNN
The induction step in the argument for GP behaviour from Sec.
2.2 depends only on the previous activations being iid Gaussian.
Since all the activations are iid Gaussian, we can add skip connections between the activations of different layers while preserving GP behaviour, e.g. A ( (cid:96) +1) and A ( (cid:96) − s ) where s is the number of layers that the skip connection spans.
If we change the NN recursion (Eq. 2) to
[Equation]
then the kernel recursion (Eq. 11) becomes
[Equation]
This way of adding skip connections is equivalent to the “pre-activation” shortcuts described by He et al.
(2016b).
Remarkably, the natural way of adding residual connections to NNs is the one that performed best in their empirical evaluations.
E XPERIMENTS
We evaluate our kernel on the MNIST handwritten digit classiﬁcation task.
Classiﬁcation likelihoods are not conjugate for GPs, so we must make an approximation, and we follow Lee et al.
(2017), in re-framing classiﬁcation as multi-output regression.
The training set is split into N = 50000 training and 10000 validation examples.
The regression targets Y ∈ {− 1 , 1 } N × 10 are a one-hot encoding of the example’s class: y n,c
= 1 if the n th example belongs to class c , and − 1 otherwise.
Training is exact conjugate likelihood GP regression with noiseless targets Y
(Rasmussen &
Williams, 2006).
First we compute the N × N kernel matrix K xx , which contains the kernel between every pair of examples.
Then we compute K − 1 xx Y using a linear system solver.
The test set has N T = 10000 examples.
We compute the N T × N matrix K x ∗ x , the kernel between each test example and all the training examples.
The predictions are given by the row-wise maximum of K x ∗ x K − 1 xx Y .
For the “ConvNet GP” and “Residual CNN GP”, (Table 1) we optimise the kernel hyperparameters by random search.
We draw M random hyperparameter samples, compute the resulting kernel’s performance in the validation set, and pick the highest performing run.
The kernel hyperparameters are: σ 2 b , σ 2 w ; the number of layers; the convolution stride, ﬁlter sizes and edge behaviour; the nonlinearity (we consider the error function and ReLU); and the frequency of residual skip connections (for Residual CNN GPs).
We do not retrain the model on the validation set after choosing hyperparameters.
The “ResNet GP” (Table 1) is the kernel equivalent to a 32-layer version of the basic residual architecture by He et al.
(2016a).
The differences are: an initial 3 × 3 convolutional layer and a ﬁnal dense layer instead of average pooling.
We chose to remove the pooling because computing its output variance requires the off-diagonal elements of the ﬁlter covariance, in which case we could not exploit the efﬁciency gains described in Sec. 3.3.
We found that the despite it not being optimised, the 32-layer ResNet GP outperformed all other comparable architectures (Table 1), including the NNGP in Lee et al.
(2017), which is state-ofthe-art for non-convolutional networks, and convolutional GPs (van der
Wilk et al., 2017; Kumar et al., 2018).
That said, our results have not reached state-of-the-art for methods that incorporate a
[Table]
parametric neural network, such as a standard ResNet (Chen et al., 2018) and a Gaussian process with a deep neural network kernel Bradshaw et al.
To check whether the GP limit is applicable to relatively small networks used practically (with of the order of 100 channels in the ﬁrst layers), we randomly sampled 10 , 000 32-layer ResNets, with 3, 10, 30 and 100 channels in the ﬁrst layers, and, following the usual practice for ResNets we increase the number the number of hidden units when we downsample the feature maps.
The probability density plots show a good match around 100 channels (Fig. 2A), which matches a more sensitive graphical procedure based on quantile-quantile plots (Fig. 2B).
Notably, even for only 30 channels, the moments match closely (Fig.
2C).
For comparison, typical ResNets use from 64 (He et al., 2016a) to 192 (Zagoruyko & Komodakis, 2016) channels in their ﬁrst layers.
We believe that this is because the moment propagation equations only require the Gaussianity assumption for propagation through the relu, and presumably this is robust to non-Gaussian input activations.
Computational efﬁciency.
Asymptotically, computing the kernel matrix takes O ( N 2 LD ) time, where L is the number of layers in the network and D is the dimensionality of the input, and inverting the kernel matrix takes O ( N 3 ) .
As such, we expect that for very large datasets, inverting the kernel matrix will dominate the computation time.
However, on MNIST, N 3 is only around a factor of 10 larger than N 2 LD .
In practice, we found that it was more expensive to compute the kernel matrix than to invert it.
For the ResNet kernel, the most expensive, computing K xx , and K xx ∗ for validation and test took 3 h 40 min on two Tesla P100 GPUs.
In contrast, inverting K xx and computing validation and test performance took 43 .
25 ± 8 . 8 seconds on a single Tesla P100 GPU.
R ELATED WORK
Van der Wilk et al.
(van der Wilk et al., 2017) also adapted GPs to image classiﬁcation.
They deﬁned a prior on functions f that takes an image and outputs a scalar.
First, draw a function g ∼ GP (0 , k p ( X , X (cid:48) )) .
Then, f is the sum of the output of g applied to each of the convolutional patches.
Their approach is also inspired by convolutional NNs, but their kernel k p is applied to all pairs of patches of X and X (cid:48) .
This makes their convolutional kernel expensive to evaluate, requiring inter-domain inducing point approximations to remain tractable.
The kernels in this work, directly motivated by the inﬁnite-ﬁlter limit of a CNN, only apply something like k p to the corresponding pairs of patches within X and X (cid:48)
(Eq. 10).
As such, the CNN kernels are cheaper to compute and exhibit superior performance (Table 1), despite the use of an approximate likelihood function.
Kumar et al.
(2018) deﬁne a prior over functions by stacking several GPs with van der Wilk’s convolutional kernel, forming a “Deep GP” (Damianou & Lawrence, 2013).
In contrast, the kernel in this paper conﬁnes all hierarchy to the deﬁnition of the kernel, and the resulting GPs is shallow.
−  output 0.
0.5 pd f
C= −  output 0.
C= −  output 0. C= −  output 0.
C= −  limiting q.
−  s a m p l ed q .
−  limiting q.
−  −  limiting q. −  −  limiting q.
−     limiting cov.
s a m p l ed c o v .
limiting cov. limiting cov.
limiting cov.
A
B
C
Wilson et al. (2016) introduced and Bradshaw et al. (2017) improved deep kernel learning. The inputs to a classic GP kernel k (e.g. RBF) are preprocessed by applying a feature extractor g (
a deep
NN)
prior to computing the kernel: k deep ( X , X (cid:48) ) :
=
k ( g ( X ; θ ) , g ( X (cid:48) , θ )) .
The NN parameters are optimised by gradient ascent using the likelihood as the objective, as in standard GP kernel learning (Rasmussen & Williams, 2006, Chapter 5).
Since deep kernel learning incorporates a state-of-the-art
NN with over 10 6 parameters, we expect it to perform similarly to a NN applied directly to the task of image classiﬁcation.
At present both CNNs and deep kernel learning display superior performance to the GP kernels in this work.
However, the kernels deﬁned here have far fewer parameters (around 10 , compared to their 10 6 ).
Borovykh (2018) also suggests that a CNN exhibits GP behaviour.
However, they take the inﬁnite limit with respect to the ﬁlter size , not the number of ﬁlters.
Thus, their inﬁnite network is inapplicable to real data which is always of ﬁnite dimension.
Finally, there is a series of papers analysing the mean-ﬁeld behaviour of deep NNs and CNNs which aims to ﬁnd good random initializations, i.e. those that do not exhibit vanishing or exploding gradients or activations (Schoenholz et al., 2016; Yang & Schoenholz, 2017).
Apart from their very different focus, the key difference to our work is that they compute the variance for a single trainingexample, whereas to obtain the GPs kernel, we additionally need to compute the output covariances for different training/test examples (Xiao et al., 2018).
C ONCLUSIONS AND FUTURE WORK
We have shown that deep Bayesian CNNs with inﬁnitely many ﬁlters are equivalent to a GP with a recursive kernel.
We also derived the kernel for the GP equivalent to a CNN, and showed that, in handwritten digit classiﬁcation, it outperforms all previous GP approaches that do not incorpo- rate a parametric NN into the kernel.
Given that most state-of-the-art neural networks incorporate structure (convolutional or otherwise) into their architecture, the equivalence between CNNs and
GPs is potentially of considerable practical relevance.
In particular, we hope to apply GP CNNs in domains as widespread as adversarial examples, lifelong learning and k-shot learning, and we hope to improve them by developing efﬁcient multi-layered inducing point approximation schemes
A PPENDIX 7. T ECHNICAL NOTES ON LIMITS
The key technical issues in the proof (and the key differences between Lee et al. 2017 and Matthews et al. 2018b) arise from exactly how and where we take limits.
In particular, consider the activations as being functions of the activities at the previous layer,
[Equation]
[Equation]
Now, there are two approaches to taking limits.
First, both our argument in the main text, and the argument in Lee et al.
(2017) is valid if we are able to take limits “inside” the network,
[Equation]
However, Matthews et al.
(2018a;b) argue that is preferable to take limits “outside” the network.
In particular, Matthews et al.
(2018b) take the limit with all layers simultaneously,
[Equation]
(18) where C ( (cid:96) )
=
C ( (cid:96) ) ( n ) goes to inﬁnity as n → ∞ .
That said, similar technical issues arise if we take limits in sequence, but outside the network.
7. E XTENDING THE DERIVATIONS OF M ATTHEWS ET AL . (2018 B ) TO THE CONVOLUTIONAL CASE
In the main text, we follow Lee et al.
(2017) in sequentially taking the limit of each layer to inﬁnity (i.e. C (1) → ∞ , then C (2) → ∞ etc.).
This dramatically simpliﬁed the argument, because taking the number of units in the previous layer to inﬁnity means that the inputs from that layer are exactly
Gaussian distributed.
However, Matthews et al.
(2018b) argue that the more practically relevant limit is where we take all layers to inﬁnity simultaneously.
This raises considerable additional difﬁculties, because we must reason about convergence in the case where the previous layer is ﬁnite.
Note that this section is not intended to stand independently: it is intended to be read alongside Matthews et al.
(2018b),
and we use several of their results without proof.
Mirroring Deﬁnition 3 in Matthews et al.
(2018b), we begin by choosing a set of “width” functions,
C ( (cid:96) )
( n ) , for (cid:96) ∈ { 1 , . . .
, L } which all approach inﬁnity as n → ∞ .
In Matthews et al.
(2018b), these functions described the number of hidden units in each layer, whereas here they describe the number of channels.
Our goal is then to extend the proofs in Matthews et al.
(2018b)
(in particular, of theorem 4), to show that the output of our convolutional networks converge in distribution to a
Gaussian process as n → ∞ , with mean zero and covariance given by the recursion in Eqs.
The proof in Matthews et al.
(2018b) has three main steps.
First, they use the Cramér-Wold device, to reduce the full problem to that of proving convergence of scalar random variables to a Gaussian with speciﬁed variance.
Second, if the previous layers have ﬁnite numbers of channels, then the channels a ( (cid:96) )
j ( X ) and a ( (cid:96) )
j ( X (cid:48) ) are uncorrelated but no longer independent, so we cannot apply the
CLT directly, as we did in the main text.
Instead, they write the activations as a sum of exchangeable random variables, and derive an adapted CLT for exchangeable (rather than independent) random variables (Blum et al., 1958).
Third, they show that moment conditions required by their exchangeable CLT are satisﬁed.
To extend their proofs to the convolutional case, we begin by deﬁning our networks in a form that is easier to manipulate and as close as possible to Eq.
(21-23) in Matthews et al.
(2018b),
[Equation]
where,
[Equation]
The ﬁrst step is to use the Cramér-Wold device (Lemma 6 in Matthews et al., 2018b), which indicates that convergence in distribution of a sequence of ﬁnite-dimensional vectors is equivalent to convergence on all possible linear projections to the corresponding real-valued random variable.
Mirroring
Eq. 25 in Matthews et al.
(2018b), we consider convergence of random vectors, f ( (cid:96) ) i,µ ( x )[ n ] − b
( (cid:96) )
i , projected onto α ( x,i,µ ) ,
T
[Equation]
where L ⊂ X × N × { 1 , . .
, H ( (cid:96) )
D ( (cid:96) ) } is a ﬁnite set of tuples of data points and channel indicies, i , and indicies of elements within channels/feature maps, µ .
The sufﬁx [ n ] indicates width functions that are instantiated with input, n .
Now, we must prove that these projections converge in distribution a Gaussian.
We begin by deﬁning summands, as in Eq. 26 in Matthews et al.
(2018b),
[Equation]
such that the projections can be written as a sum of the summands, exactly as in Eq.
27 in Matthews et al.
(2018b),
[Equation] [Equation]
Now we can apply the exchangeable CLT to prove that T ( (cid:96) )
( L , α ) [ n ] converges to the limiting
Gaussian implied by the recursions in the main text.
To apply the exchangeable CLT, the ﬁrst step is to mirror Lemma 8 in Matthews et al.
(2018b), in showing that for each ﬁxed n and (cid:96) ∈ { 2 , . . .
, L + 1 } , the summands, γ ( (cid:96) )
j ( L , α ) [ n ] are exchangeable with respect to the index j .
In particular, we apply de Finetti’s theorem, which states that a sequence of random variables is exchangeable if and only if they are i.i.d.
conditional on some set of random variables, so it is sufﬁcient to exhibit such a set of random variables.
Mirroring Eq.
29 in Matthews et al.
(2018b), we apply the recursion, γ ( (cid:96)
[Equation]
L ,
[Equation]
[ n ]
[Equation]
=
[Equation] [Equation]
As such, the summands are iid conditional on the ﬁnite set of random variables (cid:110) g
( (cid:96) − 2) k,ξ ( x )[ n ] : k ∈ { 1 , . . .
, C ( (cid:96) − 2) } , ξ ∈ { 1 , . . .
, H ( (cid:96) − 2) D ( (cid:96) − 2) } , x ∈ L X (cid:111) , where L X is the set of input points in L .
The exchangeable CLT in Lemma 10 in Matthews et al.
(2018b) indicates that T ( (cid:96) )
( L , α ) [ n ] converges in distribution to N (cid:0) 0 , σ 2 ∗ (cid:1) if the summands are exchangeable (which we showed above), and if three conditions hold,
a) E n (cid:104) γ (
[Table]
j γ
[Table]
=
[Table]
b) lim n →∞ E n (cid:20)(cid:16) γ
[Table]
j
[Table]
=
[Table] [Table]
c) E n (cid:104) | γ ( (cid:96) )
j | 3 (cid:105) =
o (cid:16)(cid:112) C (
[Table]
) ( n ) (cid:17)
Condition a) follows immediately as the summands are uncorrelated and zero-mean.
Conditions b) and c) are more involved as convergence in distribution in the previous layers does not imply convergence in moments for our activation functions.
We begin by considering the extension of Lemma 20 in Matthews et al.
(2018b), which allow us to show conditions b) and c) above, even in the case of unbounded but linearly enveloped nonlinearities (Deﬁnition 1 in Matthews et al., 2018b).
Lemma 20 states that the eighth moments of f ( t ) i,µ ( x )[ n ] are bounded by a ﬁnite constant independent of n ∈ N .
We prove this by induction.
The base case is trivial, as f (1) j,µ ( x )[ n ] is Gaussian.
Following Matthews et al.
(2018b), assume the condition holds up to (cid:96) − 1 , and show that the condition holds for layer (cid:96) .
Using Eq.
(21), we can bound the activations at layer (cid:96) ,
[Equation]

[Equation]
Following Eq.
48 in Matthews et al.
(2018b), which uses Lemma 19 in Matthews et al.
(2018b), we have,
E
[Equation] [Equation]
where
[Equation]
∈ µ th patch ( x )[ n ] is the set of post-nonlinearities corresponding to j ∈ { 1 , . .
, C ( (cid:96) − 1) ( n ) } and ν ∈ µ th patch.
Following Matthews et al.
(2018b), observe that,
[Equation]
by the linear envelope property, | φ ( u ) | ≤ c + m | u | .
Following Matthews et al.
(2018b), we substitute this bound back into Eq.
(28) and suppress a multiplicative constant independent of x and n ,
E 
[Equation]
This can be multiplied out, yielding a weighted sum of expectations of the form,
[Equation]
[Figure]
with p i ∈ { 0 , 1 , 2 } for i = 1 , 2 , 3 , 4 , and k, l, r, q ∈ { 1 , . . .
, C ( (cid:96) − 1) ( n ) } , and ν, ξ, π, ρ ∈ µ th patch where the weights of these terms are independent of n .
Using Lemma 18 in Matthews et al.
(2018b), each of these terms is bounded if the eighth moments of f ( (cid:96) − 1) k,µ ( x )[ n ] are bounded, which is our inductive hypothesis.
The number of terms in the expanded sum is upper bounded by (cid:0) 2 C ( (cid:96) − 1) ( n ) | µ th patch | (cid:1) 4 , where | µ th patch | is the number of elements in a convolutional patch.
Thus, we can use the same constant for any n due to the 1 / (cid:0) C
( (cid:96) − 1) ( n ) (cid:1) 4 scaling.
As in Matthews et al.
(2018b),
noting that f ( (cid:96) − 1)
j,µ ( x )[ n ] are exchangeable over j for any x and n concludes the proof.
Using this result, we can obtain a straightforward adaptation of Lemmas 15, 16 and 21 in Matthews et al.
(2018b).
Lemma 15 gives condition b), Lemma 16 gives condition c); Lemma 15 requires uniform integrability, which is established by Lemma 21.
7. C ALIBRATION OF G AUSSIAN PROCESS UNCERTAINTY
It is important to check that the estimates of uncertainty produced by our Gaussian process are reasonable.
However, to make this assessment, we needed to use a proper likelihood, and not the squared-error loss in the main text.
We therefore used our kernel to perform the full, multi-class classiﬁcation problem in GPﬂow Matthews et al.
(2017), with a RobustMax likelihood (Hernándezlobato et al., 2011).
The more difﬁcult non-conjugate inference problem forced us to useinducing points, randomly chosen from the training inputs.
Both our kernel and an RBF kernel have similar calibration curves, that closely track the diagonal, indicating accurate uncertainty estimation.
However, even in the inducing point setting, our convolutional kernel gave considerably better performance than the RBF kernel (2.4% error vs 3.4% error).