A J OHNSON –L INDENSTRAUSS F RAMEWORK FOR R ANDOMLY I NITIALIZED CNN S
A BSTRACT
How does the geometric representation of a dataset change after the application of each randomly initialized layer of a neural network?
The celebrated Johnson–
Lindenstrauss lemma answers this question for linear fully-connected neural networks (FNNs), stating that the geometry is essentially preserved.
For FNNs with the ReLU activation, the angle between two inputs contracts according to a known mapping.
The question for non-linear convolutional neural networks (CNNs) becomes much more intricate.
To answer this question, we introduce a geometric framework.
For linear CNNs, we show that the Johnson–Lindenstrauss lemma continues to hold, namely, that the angle between two inputs is preserved.
For
CNNs with ReLU activation, on the other hand, the behavior is richer: The angle between the outputs contracts, where the level of contraction depends on the nature of the inputs.
In particular, after one layer, the geometry of natural images is essentially preserved, whereas for Gaussian correlated inputs, CNNs exhibit the same contracting behavior as FNNs with ReLU activation.
I NTRODUCTION
Neural networks have become a standard tool in multiple scientiﬁc ﬁelds, due to their success in classiﬁcation (and estimation) tasks.
Conceptually, this success is achieved since better representation is allowed by each subsequent layer until linear separability is achieved at the last (linear) layer.
Indeed, in many disciplines involving real-world tasks, such as computer vision and natural language processing, the training process is biased toward these favorable representations.
This bias is a product of several factors, with the neural-network initialization playing a pivotal role (Sutskever et al., 2013).
Therefore, we concentrate in this work on studying the initialization of neural networks, with the following question guiding this work.
How does the geometric representation of a dataset change after the application of each randomly initialized layer of a neural network?
To answer this, we study how the following two geometrical quantities change after each layer.
[Equation]
The scalar inner product between vectors x and y
[Equation]
The cosine similarity (or simply similarity) between vectors x and y .
The similarity ρ ∈ [ − 1 , 1] between x and y equals ρ = cos( θ ) , where θ is the angle between them.
Consider ﬁrst one layer of a fully connected neural network (FNN) with an identity activation (linear
FNN) that is initialized by
independent identically distributed (i.i.d.)
Gaussian weights with mean zero and variance 1 /N , where N is the number of neurons in that layer.
This random linear FNN induces an isometric embedding of the dataset, namely, the similarity ρ in between any two inputs, x in and y in , is preserved together with their norm:
[Equation]
where ρ out is the similarity between the resulting random (due to the multiplication by the random weights) outputs, X out and Y out (respectively), and ¯ ρ out is the mean output similarity deﬁned by ρ out : = h X out , Y out i k X out k k Y out
[Equation]
, and
[Equation]
The proof of this isometric relation between the input and output similarities follows from the celebrated Johnson–Lindenstrauss lemma (Johnson and Lindenstrauss, 1984; Dasgupta and Gupta, 1999).
This lemma states that a random linear map of dimension N preserves the distance between any two points up to an ǫ > 0 contraction/expansion with probability at least 1 − δ for all
N > c log(1 /δ )
/ǫ 2 for an absolute constant c .
In the context of randomly initialized linear FNNs, this result means that, for a number of neurons N that satisﬁes N > c
log(
/δ )
/ǫ 2 ,
[Equation]
So, conceptually, the Johnson–Lindenstrauss lemma studies how inner products (or geometry) change, in expectation, after applying a random transformation and how well an average of these random transformations is concentrated around the expectation.
This is the exact setting of randomly initialized neural networks.
The random transformations consist of a random projection (multiplying the dataset by a random matrix) which is followed by a non-linearity.
Naturally, adding a non-linearity complicates the picture.
Let us focus on the case where the activation function is a rectiﬁed linear unit (ReLU).
That is, consider a random fully-connected layer with
ReLU activation initialized with i.i.d.
zero-mean Gaussian weights and two different inputs.
For this case, Cho and Saul (2009), Giryes et al.
(2016), and Daniely et al.
(2016) proved that
[Equation]
Following Daniely et al.
, we refer to the resulting function in (2) of ¯ ρ out in ρ in as the dual activation of the ReLU; this function is represented in ﬁgure 1 by the yellow curve.
One may easily verify that the dual activation (2) of the ReLU satisﬁes ρ out > ρ in for ρ in 6 = 1 , meaning that it is a contraction .
Consequently, for deep FNNs, which comprise multiple layers, random initialization results in a collapse of all inputs with the same norm (a sphere) to a single point at the output of the FNN (equivalently, the entire dataset collapses to a single straight line).
Intuitively, this collapse is an unfavorable starting point for optimization.
To see why, consider the gradient ∇ w ( i ) j of the weight w ( i ) j of some neuron j in a deep layer
i in a randomly initialized ReLU
FNN.
By the chain rule (backpropagation), this gradient is proportional to the output of the previous layer a ( i − 1) for the corresponding input, i.e., it holds that ∇ w ( i )
j ∝ a ( i − 1) .
If the collapse is present already at layer i , this output is essentially proportional to a ﬁxed vector a ∗ .
But this implies that, in the gradient update, the weights of the deep layer will move roughly along a straight line which would impede, in turn, the process of achieving linear separability.
Indeed, it is considered that for a FNN to train well, its input–output Jacobian needs to exhibit dynamical isometry upon initialization (Saxe et al., 2014).
Namely, the singular values of the Jacobian ∂x out /∂x in must be concentrated around 1 , where x out and x in denote the input and output of the
FNN, respectively.
If the dataset collapses to a line, x out is essentially invariant to x in (up to a change in its norm), suggesting that the singular values of ∂x out /∂x in are close to zero.
Therefore, randomly initialized FNNs exhibit the opposite behavior from dynamical isometry and hence do not train well. [Figure]
1. O UR C ONTRIBUTION
Our main interest lies in the following question.
Does the contraction observed in randomly initialized ReLU FNNs carry over to convolutional neural networks (CNNs)?
As we will show, qualitatively, the answer is yes.
However, quantitatively, the answer is more subtle as is illustrated in ﬁgure 1.
In this ﬁgure, the similarity between pairs of inputs sampled at random from standard natural image datasets—Fashion MNIST (F-MNIST), CIFAR-10, and ImageNet—are displayed against the corresponding output of a randomly initialized CNN layer.
For these datasets, clearly ρ out ≈ ρ in meaning that the relation of ReLU FNNs (2)—represented in ﬁgure 1 by the yellow curve—breaks down.
That said, for inputs consisting of i.i.d.
zero-mean Gaussians (and ﬁlters comprising i.i.d.
zero-mean
Gaussian weights as before) with a Pearson correlation coefﬁcient ρ between corresponding entries (and independent otherwise), the relation in (2) between ¯ ρ out and ρ in of ReLU FNNs does hold for
ReLU CNNs as well, as illustrated in ﬁgure 1a.
This dataset-dependent behavior, observed in ﬁgure 1, suggests that, in contrast to randomlyinitialized FNNs which behave according to (2), randomly-initialized CNNs exhibit a richer behavior: ¯ ρ out does not depend only on ρ in but on the inputs x in and y in themselves.
Therefore, in this work, we characterize the behavior of ¯ ρ out after applying one layer in randomly initialized CNNs.
We start by considering randomly initialized CNNs with general activation functions.
We show in theorem 1 that the expected (over the ﬁlters) inner product E
[ h X out , Y out i ] and the mean similarity ¯
ρ out depend on x in and y in (and not just
h x in , y in i ) by extending the dual-activation notion of Daniely et al.
In theorem 2, we further prove that, by taking multiple independent ﬁlters, h X out , Y out i and ρ out of (1) concentrate around E [ h X out , Y out i ] and ¯ ρ
We then specialize these results to linear CNNs (with identity activation) and derive a convolutionbased variant of the Johnson–Lindenstrauss lemma that shows that h X out , Y out i
≈ h x in ,
y in i and ρ out ≈ ρ in for linear CNNs, both in expectation ( ¯ ρ out = ρ in for the latter) and with high probability.
For randomly initialized ReLU CNNs, we derive the following tight upper and lower bounds for ¯ ρ out in terms of ρ in in theorem 3:
[Equation]
These bounds imply, in turn, that for ρ in 6 = 1 each ReLU CNN layer is contracting.
In theorem 4 we prove that ¯ ρ out for random Gaussian data satisﬁes the relation (2), in accordance with ﬁgure 1a.
To explain the (almost) isometric behavior of CNNs for natural images (ﬁgure 1), we note that many natural images consist of large, relative to the ﬁlter size, approximately monochromatic patches.
This observation leads to a simple model of black and white (binary) images with “large patches”.
To describe this model mathematically, we deﬁne a notion of a shared boundary between two images in deﬁnition 2, and model large patches by bodies whose area is large compared to the shared boundary.
We prove that ¯ ρ out
≈ ρ in for this model, meaning that the lower bound in (3) is in fact tight.
1. R ELATED W ORK
In this paper, we study how various inputs are embedded by randomly initialized convolutional neural networks.
Neural tangent kernel (NTK)
(Jacot et al., 2018) is a related line of work.
This setting studies the inﬁnite width limit (among other assumptions) in which one can consider neural network training as regression over a ﬁxed kernel; this kernel is the NTK.
There are two factors that affect the calculation of the NTK:
The embedding of the input space at initialization and the gradients at initialization.
In this paper we study the ﬁrst one.
Arora et al.
and Bietti and Mairal (2019) give expressions for the NTK and the convolutional NTK (CNTK).
Theorem 1 may be implicitly deduced from those expressions.
Arora et al.
(2019) provide concentration bounds for the NTK of fully connected networks with ﬁnite width.
Bietti and Mairal (2019) derive smoothness properties for NTKs, e.g., upper bounds on the deformation induced by the NTK in terms of the initial Euclidean distance between the inputs.
A related approach to NTK is taken in (Bietti, 2021) where convolutional kernel networks (CKN) are used.
Standard initialization techniques use the Glorot initializtion (Glorot and Bengio, 2010) and He initialization (He et al., 2015).
Both were introduced to prevent the gradients from exploding/vanishing.
On a similar note, Hanin and Rolnick (2018) discuss how to prevent exploding/vanishing mean activation length—which corresponds to the gradients to some degree—in FNN with and without skip connections.
For a comprehensive review on other techniques see (Narkhede et al., 2021-06-28).
Schoenholz et al.
; Poole et al.
; Yang and Schoenholz (2017) study the initialization of FNNs using mean-ﬁeld theory, in a setting where the width of the network is inﬁnite.
They demonstrate that for some activation functions there exists an initialization variance such that the network does not suffer from vanishing/exploding gradients.
In this case, the network is said to be initialized at the edge of chaos.
As mentioned earlier, Saxe et al.
(2014) introduced a stronger requirement than that of moderate gradients at initialization, in the form of dynamical isometry .
for linear FNNs, they showed that orthogonal initialization achieves dynamical isometry whereas Gaussian i.i.d.
initialization does not.
For non-linear FNNs, Pennington et al.
(2017) show that the hyperbolic tangent (tanh) activation can achieve dynamical isometry while ReLU FNNs with Gaussian i.i.d.
initialization or orthogonal initialization cannot.
In contrast, Burkholz and Dubatovka (2019) show that ReLU FNNs achieve dynamical isometry by moving away from i.i.d.
initialization.
And lastly, Tarnowski et al.
(2019) show that residual networks achieve dynamical isometry over a broad spectrum of activations (including ReLU) and initializations.
Xiao et al.
trained tanh (and not ReLU) CNNs with 10000 layers that achieve dynamical isometry using the delta-orthogonal initialization.
Similarly, Zhang et al.
(2019) trained residual
CNNs with 10000 layers without batch normalization using ﬁxup-initialization that prevents exploding/vanishing gradients.
S ETTING AND D EFINITIONS
The output z ∈ R n × n of a single ﬁlter of a CNN is given by
[Equation]
where x ∈ R n × n × d is the input; F ∈ R r ×
r × d is the ﬁlter (or kernel) of this layer; and σ : R → R is the activation function of this layer and is understood to apply entrywise when applied to tensors.
The dimensions satisfy n, d, r ∈ N , r ≤ n .
The ﬁrst two dimensions of x and z are the image dimensions (width and height), while the third dimension is the number of channels, and is most commonly equal to either 1 for grayscale images and 3 for RGB color images; r is the dimension of the ﬁlter; ‘ ∗ ’ denotes the cyclic convolution operation:
[Equation]
where the subtraction operation in the arguments is the subtraction operation modulo n .
In this work, we will concentrate on the ReLU activation function which is given by σ ( x ) = max { 0 , x } .
Typically, a single layer of a CNN consists of multiple ﬁlters F 1 , . .
. , F N .
The following deﬁnition is an adaptation of the dual activation of Daniely et al.
(2016) to CNNs.
Deﬁnition 1 (Dual activation) .
The dual activation function b
[Equation]
[Equation]
n × R n × R → R of an activation function σ is deﬁned as
[Equation]
for input vectors x, y ∈ R n , and a parameter ν > 0 , where
[Equation]
For tensors x, y ∈ R n × n × d , the dual activation b σ is deﬁned as in (5) with the inner products and norms replaced by the standard inner product between tensors:
[Equation]
In the special case of an activation that is homogeneous, i.e., satisfying σ ( cx ) = cσ ( x ) for c ≥ 0 , we use an additional notation for − 1 ≤ ρ ≤ 1 , with some abuse of notation:
[Equation]
This is the dual activation function of Daniely et al.
Note that, by this deﬁnition, b σ (1) = 1 .
We will be mostly interested in the ReLU activation, denoted by R , for which the following holds (see, e.g., (Daniely et al., 2016, Table 1 and Section C of supplement))
[Equation]
M AIN R ESULTS
In this section, we present the main results of this work.
The proofs of all the results in this section may be found in the appendix.
Theorem 1.
Let x, y ∈ R n × n × d be inputs to a convolution ﬁlter F ∈ R r × r × d with r ≤ n and activation function σ such that the entries of F are i.i.d.
Gaussian with variance ν 2 .
Then,
[Equation]
In particular, for a homogeneous activation function we have
[Equation]
where ρ ij = [Equation] and [Equation]
has been deﬁned in (6) .
In particular, due to (9), for homogeneous activations we can choose ν = 1 / ( ν σ r ) and get E (cid:2) σ [Equation] (cid:3) =
k x k 2 .
That is, the norms are preserved in expectation for every input x .
Note that theorem 1 may be deduced as a special case from existing more general formulas; see, e.g.,
Arora et al.
; Bietti and Mairal (2019).
Nevertheless, it is an important starting point for us.
While theorem 1 holds in the mean sense, it does not hold for a speciﬁc realization of a single ﬁlter F , in general.
The next theorem, whose proofs relies on the notions of sub-Gaussian and subexponential RVs and their concentration of measure, states that theorem 1 holds approximately for a large enough number of applied ﬁlters.
We apply basic, general concentration bounds on the activation.
There exist sharper bounds in special cases, e.g., in the context of cosine similarity and ReLU, see (Buchanan et al., 2021, app.
E).
Theorem 2.
Let x, y ∈ R n × n × d be inputs to N ﬁlters F 1 , . .
. , F N ∈ R r × r × d with r ≤ n such that
[Equation]
all the entries of all the ﬁlters are i.i.d.
Gaussian with zero mean and variance ν 2 , and a Lipschitz continuous activation function σ with a Lipschitz constant L and satisfying σ (0) = 0 .
Then, for δ > 0 , however small,
[Equation]
for N > max
[Equation]
log 2 n 2 δ , where K =
Dν 2 L 2 R 2 n 2 /ǫ
and D > 0 is an absolute constant.
Remark 1.
Clearly, k [ x ] rij k and k [ y ] rij k in (10) may be bounded by k x k and k y k , respectively.
However, since r is typically much smaller than n , we prefer to state the result in terms of the norms of [ x ] rij and [ y ] rij .
Theorem 2 states a concentration result in terms of the inner product.
We now give a parallel result for homogeneous activations in terms of the cosine similarity ρ out .
To that end, consider a homogeneous activation σ and N random ﬁlters F 1 , . .
. , F N ; for these ﬁlters, the quantities of (1) for two inputs x and y are equal to
[Equation]
where ρ ij is deﬁned in theorem 1.
Applying theorem 2 to these quantities, yields the following.
Corollary 1.
Let x, y be inputs to N ﬁlters F 1 , . .
, F N such that
[Equation]
all the entries of the ﬁlters are i.i.d.
Gaussian with zero mean and variance ν 2 , and a Lipschitz homogeneous activation σ with Lipschitz constant L .
Then, for 0 < ǫ ≤ 1 / 10 and δ > 0 ,
P
[Equation]
|
[Equation]
x, y )
[Equation]
x, y ) | ≥ ǫ (cid:1) ≤ δ for N > max( K, K 2 ) log 6 n 2 δ , where K = DL 2 R 2 n 2 ǫν 2 σ rand D > 0 is an absolute constant.
Remark 2.
To make sense out of the constants in corollary 1, we point out that applying it to inputs x, y ∈ {± 1 } n × n × d (so that k x k 2 =
k y k 2 = dn 2 and k [ x ] rij k 2 =
k [ y ] rij k 2 = dr 2 ) results in
K = DL 2 ǫν 2 σ , which does not depend on n or ν 2 .
There remains a log n 2 factor that increases with n .
In the remainder of this section, we derive an analogous result to the Johnson–Lindenstrauss lemma (Johnson and Lindenstrauss, 1984; Dasgupta and Gupta, 1999) in section 3.1 for random linear
CNNs, namely, CNNs with identity activation functions.
We then move to treating random CNNs with ReLU activation functions: We derive upper and lower bounds on the inner product between the outputs in section 3.2, and prove that they are achievable for Gaussian inputs in section 3.3, and for large convex-body inputs in section 3.4, respectively.
3. L INEAR CNN
Theorem 1 yields a variant of the Johnson–Lindenstrauss lemma where the random projection is replaced with random ﬁltering, that is, by applying a (properly normalized) random ﬁlter, the inner product is preserved (equivalently, the angle).
Lemma 1. Consider the setting of theorem 1 with ν = 1 /r .
Then, E [ h F ∗ x, F ∗ y i ]
= h x, y i .
Lemma 2.
Consider the setting of theorem 2 with ν = 1 /r (and σ ( x ) ≡ x ).
Then,
[Equation]
for N > max (cid:0) K
[Equation]
(cid:1) log 2 n 2 δ , where K = DR
[Equation]
and D > 0 is an absolute constant.
Related result appears in Krahmer et al.
(2014) (theorem 1.1) which shows norm preservation of sparse vectors for convolutional operators with a ﬁlter dimension that equals the input dimension.
3. CNN WITH R E LU A CTIVATION
The following theorem implies that every layer of a neural network is contracting in expectation (over F ).
That is, the expected inner product between any two data points will get larger with each random CNN layer.
We also develop an upper bound on the new correlation.
Theorem 3.
Consider the setting of theorem 1 with a ReLU activation function and variance ν = 2 /r 2 , for inputs x and y with similarity ρ .
Then,
[Equation]
Remark 3.
All the bounds in theorem 3 are tight.
For simplicity, we present examples where the unit vectors x, y are ﬂat, i.e., they are not tensors and r = 1 .
Similar examples can be drawn for tensor inputs and r > 1 .
The upper bound is realized for x, y ∈ R 2 of the form x = (cid:18)q 1+ ρ 2 , q 1 − ρ(cid:19) ,
y =
(cid:18)q 1+ ρ 2 , − q 1 − ρ(cid:19) .
Similarly, x, y ∈ R 3 with x = ( √ 1 − ρ, √ ρ, 0) , y = (0 , −√ ρ, √ 1 − ρ ) satisfy h x, y i = − ρ and E [ σ ( F ∗ x ) σ ( F ∗ y )]
= 0 for r = 1 and 0 ≤ ρ ≤ 1 .
Finally, we can take x = ( √− ρ, √ ρ, 0) and y = (0 , √ ρ, √ 1 − ρ ) to obtain h x, y i
=
E [ σ ( F ∗ x ) σ ( F ∗ y )]
= ρ .
These examples are illustrated in ﬁgure 3.
Remark 4.
Although the lower bound is tight, in most typical scenarios it will be strict, so, in expectation, the convolutional layers will contract the dataset after application of enough layers.
To see why, observe (23c); equality is achieved iff ρ ij = 1 ∨ [ x ] rij = 0 ∨ [ y ] rij = 0 .
For real data sets this will never hold over all ( i, j ) .
This can be observed in ﬁgures 1b, 1c, and 1d where the linear ﬁt has a slope of 0.97.
3. G AUSSIAN I NPUTS
The following result shows that the behavior reported by (Daniely et al., 2016)
[and illustrated in ﬁgure 1a by the orange curve] for FNNs holds for CNNs with Gaussian correlated inputs; this is illustrated in ﬁgure 1a.
For simplicity, we state the result for d = 1 ( n × n inputs instead of n × n × d ); the extension to d > 1 is straightforward.
Theorem 4.
Let X, Y ∈ R n × n be zero-mean jointly Gaussian with the following auto- and crosscorrelation functions:
[Equation]
i.e., X and Y have pairwise i.i.d.
entries which are correlated with a Pearson correlation coefﬁcient ρ .
Assume further that the ﬁlter F ∈ R r × r comprises zero-mean i.i.d.
Gaussian entries with variance 2 /r 2 and is independent of ( X, Y ) .
Then, for a ReLU activation function σ ,
[Equation]
3. S IMPLE B LACK AND W HITE I MAGE M ODEL
This section provides insight why we get roughly an isometry when embedding natural images via a random convolution followed by ReLU.
As a conceptual model, we will work with binary pictures { 0 , 1 } n × n or equivalently as subsets of Z n × Z n .
Also, when considering real-life pictures, an apparent characteristic is that they consist of relatively large monochromatic patches, so for the next theorem, one should keep in mind pictures that consist of large convex bodies.
To state our results, we deﬁne a notion of a shared r -boundary.
Deﬁnition 2 (Boundary) .
Let A, B ⊂ Z n × Z n and let r ∈ [ n ] .
Then, the r -boundary of the intersection between A and B , denoted by ∂ r ( A, B ) , is deﬁned as the set of all pixels ( i,
j ) ∈
Z n × Z n such that
[Equation]
15a
[Equation]
15b
[Equation]
where the addition in (15a) and (15b) is over Z n .
In words, a pixel ( i, j ) belongs to the r -boundary ∂ r
( A, B ) if, considering the square with edge size 2 r + 1 centered at the pixel: (1) the square intersects both A and B , (2) the square is not contained in A or is not contained in B .
Example 1.
Let A and B be axis-aligned rectangles of sizes a × b and c × d , respectively, and an intersection of size e × f where e, f ≥ 2 r .
Then, | ∂ r ( A, B ) | = 4 r ( e + f ) .
This is illustrated in ﬁgure 2.
The next theorem bounds the inner product between A and B after applying a ReLU convolution layer in terms of the r -boundary of the intersection of A and B .
Theorem 5.
Let A, B ⊂ [ n ] × [ n ] and a convolution ﬁlter F ∈ R (2 r +1) ×
(2 r +1)
such that the entries in F are i.i.d.
Gaussians with variance 2 / (2 r + 1) 2 .
Then,
[Equation]
16a
[Equation]
Theorem 5 and corollary 1 imply
[Equation]
with high probability for a large enough number of ﬁlters N .
This shows that a set of images consisting of “large patches”, meaning that ∂
r ( A,B ) k
A kk B k is small (as in example 1 for small r ), is embedded almost-isometrically by a random ReLU convolutional layer.
Moreover, any set of images may be embedded almost-isometrically this way.
To see how, ﬁx r while increasing artiﬁcially (digitally) the resolution of the images.
The latter would yield ∂ r ( A,B )
k
A kk B k → 0 as the resolution tends to inﬁnity.
[Figure]
D ISCUSSION 4. G RAYSCALE VERSUS RGB I MAGES
The curious reader might wonder how the analysis over black-and-white (B/W) images can be extended to grayscale and RGB images.
While we concentrated on B/W images in this work, the deﬁnition of a shared boundary readily extends to grayscale images as does the result of theorem 5.
This, in turn, explains the behavior of the grayscale-image dataset F-MNIST in ﬁgure 1b.
For RGB images, the framework developed in this work does not hold in general.
To see why, consider two extremely simple RGB images: Red —consisting of all-zero green and blue channels and a constant unit-norm red channel, and Green —consisting of all-zero red and blue channels and a constant unit-norm green channel.
Clearly, h Red, Green i = 0 .
However, a simple computation, in the spirit of this work, shows that E [ h R( F ∗ Red ) , R( F ∗ Green ) i ]
= 1 /π
=
b
R (0) which is far from 0 .
That is, for such monochromatic pairs, the network behaves as a fully connected network (2) and not according to the almost isometric behavior of theorem 5 and ﬁgure 1d.
So why does an almost isometric relation appear in ﬁgures 1c and 1d?
The answer is somewhat surprising and of an independent interest.
It appears that for the RGB datasets CIFAR-10 and ImageNet, the RGB channels have high cosine similarity (which might not be surprising on its own), and additionally all the three channels have roughly the same norm.
Quantitatively—averaged over 10 4 images that were picked uniformly at random from the ImageNet dataset—the angle between two channels is ~ 12 . 5 ◦ and the relative difference between the channel norms is ~ 9 . 2% .
This phenomenon implies that our analysis for black and white images (as well as its extension to grayscale images) holds for RGB natural-images datasets.
That said, there are images that are roughly monochromatic (namely, predominantly single-color images) but their measure is small and therefore these images can be treated as outliers.
4. F UTURE W ORK
Figure 1 demonstrates that a randomly initialized convolutional layer induces an almost-isometric yet contracting embedding of natural images (mind the linear ﬁt with slope ~ 0.97 in the ﬁgure), and therefore the contraction intensiﬁes (“worsens”) with each additional layer.
In practice, the datasets are preprocessed by operations such as mean removal and normalization.
These operations induce a sharper contraction.
Since a sharp collapse in the deeper layers implies failure to achieve dynamical isometry, a natural goal for a practitioner would be to prevent such a collapse.
Saxe et al.
(2014) propose to replace the i.i.d.
Gaussian initialization with orthogonal initialization that is chosen uniformly over the orthogonal group for linear FNNs.
However, the latter would fail to prevent the collapse phenomenon as well (see (Pennington et al., 2017)).
This can be shown by a similar calculation to that of the dual activation of ReLU with i.i.d.
Gaussian initialization.
In fact, the dual activations of both initializations will coincide in the limit of large input dimensions.
That said, the collapse can be prevented by moving further away from i.i.d.
initialization.
For example, by drawing the weights of every ﬁlter from the Gaussian distribution under the condition that the ﬁlter’s sum of weights is zero.
Curiously, also including batch normalization between the layers prevents such a collapse.
In both cases, the opposite phenomenon to a collapse happens, an expansion.
In the deeper layers, the dataset pre-activation vectors become orthogonal to one another.
Understanding this expansion and why it holds for seemingly unrelated modiﬁcations is a compelling line of research.
A CKNOWLEDGMENTS
The work of I. Nachum and M. Gastpar in this paper was supported in part by the Swiss National
Science Foundation under Grant 200364.
The work of A. Khina was supported by the I SRAEL S CIENCE F OUNDATION (grants No. 2077/20) and the WIN Consortium through the Israel Ministry of Economy and Industry.
A N OTATION R , N , Z
The sets of real , natural, and
integer numbers
, respectively.
∨ , ∧
The logical ‘or’ and ‘and’ operators, respectively.
Z n ,
[Table]
The
sets { 0 , 1 , . .
n − 1 } and { 0 , 1 , . .
n
−
[Table]
The (
i,
j entry of the matrix
A and
the (
i,
j, k
) entry
of the tensor T , respectively.
[ T ] rij
The sub tensor of T formed from rows i to i + r − 1 and columns j to j + r − 1 (and full third “layer” dimension).
[Table]
The identity matrix of dimensions k × k .
vec(
[Table]
The vectorization operation of a matrix or a tensor in some systemic order.
h x, y
i
The standard inner product between vectors x and y .
For matrix or tensor x and y (of the same dimensions) the inner product is deﬁned as h x, y i
=
h vec( x ) , vec( y ) i .
k x k
The standard Euclidean norm, induced by the standard inner product: k x k = p h x, x i .
ρ
The (cosine) similarity.
The similarity between x and y is deﬁned as ρ =
h x,y i k x kk y k .
¯ ρ
The mean similarity.
¯ ρ between random X and Y is deﬁned as ¯ ρ =
E [ h X,Y i ] q E [ k X k 2 ] E [ k Y k 2 ] .
σ , ˆ σ
An activation function and its dual, respectively, as deﬁned in deﬁnition 1.
R( x )
The ReLU applied to x : R( x ) :
= max { 0 , x } ; for a tensor x the R is applied entrywise.
b
R ( ρ )
The dual activation of the ReLU at ρ ∈ [ − 1 , 1] :
[Equation]
N ( µ, C )
A Gaussian vector with mean vector µ and covariance matrix C .
B P ROOF OF THEOREM
To prove theorem 1, we ﬁrst prove the following simple result.
Lemma 3.
Let u, v ∈ R k be two (deterministic) vectors, and let G,
H ∈ R k be zero-mean jointly
Gaussian vectors with the following auto- and cross-correlation matrices:
[Equation]
Then,
[Equation]
is a zero-mean Gaussian vector with a covariance matrix
[Equation]
Proof of lemma 3.
Z is a Gaussian vector as a linear combination of two jointly Gaussian vectors.
Furthermore,
[Equation]
We are now ready to prove theorem
[Equation]
Proof of theorem 1.
The equation (7) is proved as follows.
[Equation]
where (17a) follows from the deﬁnition of the cyclic convolution (4) and the linearity of expectation, and (17b) follows from deﬁnition 1 using lemma 3 with F taking the role of G and H (with ρ = 1) , and [ x ] rij and [ y ] rij taking the roles of u and v , respectively.
Now
[Equation]
we set out to prove (8) under the assumption that σ is homogeneous and the following notation.
[Equation]
where (18a) follows from deﬁnition 1, (18b) follows from the homogeneity of σ , (18c) follows from standard random variable calculus, and (18d) follows from (6).
Finally, to obtain (9), we apply (8) for x = y and use that ρ ij = 1 and b σ (1) = 1 :
[Equation]
C P ROOF OF THEOREM
To prove the concentration of measure results of this work, we will make use of the following deﬁnition.
Deﬁnition 3.
The Orlicz norm of a random variable (RV) X with respect to (w.r.t.)
a convex function ψ : [0 , ∞ )
→ [0 , ∞ )
such that ψ (0) =
and lim x →∞ ψ ( x ) = ∞ is deﬁned by
[Equation]
In particular, X is said to be sub-Gaussian if k X k ψ 2 < ∞ , and sub-exponential if k X k ψ 1 < ∞ , where ψ p ( x ) :=
exp { x p } − 1 for p ≥ 1 .
The following two results, whose proofs are available in (Vershynin, 2018, Ch. 2 and 5.2), will be useful for the proof of theorem 2.
Lemma 4.
Let X and Y be sub-Gaussian random variables (not necessarily independent).
Then,
Sum of independent sub-Gaussians.
If X and Y are also independent, then their sum, X + Y , is sub-Gaussian.
Moreover, k X + Y k 2 ψ 2 ≤ C
[Equation]
for an absolute constant C .
The same holds (with the same constant C ) also for sums of multiple independent sub-Gaussian RVs.
Centering.
X − E
[ X ] is sub-Gaussian.
Moreover, k X − E
[ X ] k ψ 2 ≤ C k X k ψ 2 for an absolute constant C . 3.
Lipschitz functions of Gaussians.
If X is a centered Gaussian and f is a function with Lipschitz constant L , then k f ( X ) − E [ f ( X )]
k ψ 2 ≤ C k X k ψ 2 for an absolute constant C . 4.
Product of sub-Gaussians.
XY is sub-exponential.
Moreover, k XY k ψ 1 ≤ k X k ψ 2 k
Y k ψ 2 .
Theorem 6 (Bernstein’s inequality for sub-exponentials) .
Let X 1 , . . .
X N be independent zeromean sub-exponential RVs.
Then,
[Equation]
where K = max i
k X i k ψ 1 and c > 0 is an absolute constant.
We are now ready to prove the desired concentration-of-measure result.
Proof of theorem 2.
h F ℓ , [ x ] r ij i and h F ℓ , [ y ] r ij i are (jointly)
Gaussian RVs—as linear combinations of i.i.d.
Gaussian RVs—with mean zero and variances ν(cid:13)(cid:13)
[ x ] rij (cid:13)(cid:13) 2 and ν(cid:13)(cid:13)
[ y ] rij (cid:13)(cid:13) 2 , respectively, for all ℓ ∈ [ N ] and i, j ∈ Z n .
Hence, h F ℓ , [ x ] r ij i and h F ℓ , [ y ] r ij i are also sub-Gaussian with (cid:13)(cid:13)
[Equation]
Z n for some universal constant C 0 > 0 .
Deﬁne now
[Equation]
Since σ is Lipschitz continuous with a Lipschitz constant L , and since the inner products h F ℓ , [ x ] rij i and h F ℓ , [ y ] rij i are centered Gaussians, by property 3 in lemma 4 it follows that X ijℓ
and Y ijℓ are sub-Gaussian with
[Equation]
Due to the assumption σ (0) =
and the fact that σ has Lipschitz constant L , inequality | σ ( x ) | ≤
L | x | holds for every x ∈ R .
Accordingly, we can bound the expectation of E [ X ijℓ ] with
[Equation]
[Equation]
where we applied the Cauchy–Schwarz inequality.
Similarly, it holds that
[Equation]
Since by homogeneity of norm it also holds for any constant D ∈ R that k D k ψ 2 ≤ C 0 | D | , by triangle inequality it then follows from (19), (20) and (21) that
[Equation]
Therefore, subsequent application of properties 4 and 2 of lemma 4 to X ijℓ Y ijℓ , yields
[Equation]
for all ℓ ∈ [ N ] and all i, j ∈ Z n , where C > 0 is is the absolute constant from lemma 4.
Now it follows
[Equation]
where (23a) follows from theorem 1 and (19), (23b) follows from the triangle inequality, (23d) follows from the union bound, and (23e) follows from (22) and theorem 6.
Finally, noting that (11) holds for N > max (cid:16)
[-C 40 C 2 ν 4 L 4 R 4 n 4 16 cǫ, C 
-][Equation]
concludes the proof.
D P ROOF OF COROLLARY
Let us start by inspecting (12) to observe that, by homogeneity of σ , neither the value of ¯ ρ out nor the distribution of ρ out depend on ν or the norms of x and y .
Therefore, let us assume without loss of generality that x and y are unit vectors, as well as that ν = 1 ν σ r .
Furthermore, note that homogeneity of σ implies σ (0) = 0 , so theorem 2 can be applied with activation function σ .
Let A := 1 N
[Equation]
and A ′ :
= 1 r
P i,j ∈ Z n k [ x ] r ij k · k [ y ] r ij k b σ ( ρ ij ) .
Comparing against (12), we see that ρ out =
A √ BC and that ¯ ρ out = A ′ .
We now apply theorem 2 three times, for pairs of vectors ( x, y ) , ( x, x ) and ( y, y ) , respectively, with parameters of ǫ/ 5 and δ/ 3 .
Using equations (7)–(9) from theorem 1, we check that indeed each of the three following relations holds for N > max( K, K 2 ) log 6 n 2 δ , except with probability δ/ 3 :
[Equation]
By union bound, all three inequalities in (24) hold simultaneously except with probability δ .
Finally, by elementary inequalities we establish that if (24) holds for some A, A ′ , B, C ∈ R such that | A | ≤ 1 and 0 < ǫ ≤ 1 / 10 , then it also holds that
[Equation]
Since this occurs except with probability δ , the proof is concluded.
E P ROOF OF LEMMATA 1 AND
Proof of lemma 1.
Lemma 1 may be viewed as a special case of theorem 1 with an identity activation function σ .
Therefore,
[Equation]
where ( a ) follows from theorem 1, ( b ) follows from deﬁnition 1, and ( c ) holds since ν 2 = 1 /r 2 .
Proof of lemma 2.
Again, lemma 2 follows from theorem 2 for σ ( x ) ≡ x (and hence L = 1 ) and from lemma 1.
F P ROOF OF THEOREM
We start with the lower bound.
Deﬁne ρ ij :=
h [ x ] rij , [ y ] rij i
k
[ x ] rij [Equation] [ y ] rij k for all i, j ∈ Z n .
Then, for all i, j ∈ Z n , h [ x ] r ij , [ y ] r ij [Equation]
(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13) [ y ] r ij [Equation] ij (25a) ≤ (cid:13)(cid:13)
[Equation]
x
[Equation]
25b
[Equation]
25c
[Equation]
where (25a) holds by the deﬁnition of ρ ij , (25b) holds since
[Equation]
∀ [Equation] and (25c) follows from (Daniely et al., 2016, Table 1 and Section C of supplement).
Thus, h x, y i = 1 /r
[Equation]
where the inequality follows from (25), and the last step is due to (17).
Now, since the ReLU activation is non-negative, 0 ≤ E [ σ ( F ∗ x ) σ ( F ∗ y )] , which completes the proof of the left inequality in (13).
For the upper bound, we use the following convexity argument.
First, by homogeneity of ReLU, we can assume without loss of generality that x and y have unit norm.
Therefore, it remains to show
E [ σ ( F ∗ x ) σ ( F ∗ y )] ≤ 1+ ρ 2 for x, y such that k x k = k y k
= 1 and h x, y i = ρ .
Recall that ρ ij = h
[ x ] rij , [ y ] rij i
k
[ x ] rij kk [ y ] rij k .
We expand the deﬁnitions in the same way as in (26) and (25c)
[Equation]
[Equation]
It is easily checked that
[Equation]
[Equation]
{+) = 
+}
and that b
R ( x ) is convex for − 1 ≤ x ≤ 1 .
Accordingly, using the decomposition ρ = 1+ ρ 2 · 1+ 1 − ρ 2 · ( − 1) , we have by Jensen’s inequality for every i, j ∈ Z n [Equation] [Figure]
[Equation]
Substituting into (27),
[Equation]
where in (28) we applied the Cauchy–Schwarz inequality and used the initial assumption k x k =
k y k
= 1 .
G P ROOF OF THEOREM
The proof follows from the following steps.
[Equation] 29a) [Equation] 29b
[Equation]
29d) where (29a) follows from the law of total expectation; (29b) follows from (14), from the independence of F in ( X, Y ) , and from lemma 3 with F taking the roles of u and v , and [ X ] rij and [ Y ] rij taking the roles of G and H , resulting in X F and Y F which are zero-mean jointly Gaussian with variances k F k 2 /n 2 and a Pearson correlation coefﬁcient ρ ; (29c) follows from the homogeneity of the ReLU activation function and from (Cho and Saul, 2009, eq.
(6)), (Giryes et al., 2016, Theorem 4), (Daniely et al., 2016, Section 8); (29d) holds by the deﬁnition of F .
H P ROOF OF THEOREM
In this section, we will use the following additional notations.
[ A ] ¯ rij will denote the submatrix of A formed from rows i − r to i + r and columns j − r to j + r , and 1 k × k will denote a k × k matrix with all entries equal 1 .
The main effort in the proof is establishing (16a).
We apply theorem 1, so we have E [ h σ ( F ∗ A ) , σ ( F ∗ B ) i ] =
P i,j b σ ([ A ] ¯ rij , [ B ] ¯ rij , 2 / (2 r + 1)) .
And speciﬁcally, for the ReLU activation, it holds that
[Equation]
where ρ ij = h
[ A ] ¯ rij , [ B ] ¯ rij
i
[Equation]
A natural way to show that the convolutional layer induces isometry in expectation would be to show
[Equation]
for all pairs ( i, j ) .
Unfortunately, this does not hold for all pairs.
However, we will show that the only pairs for which it does not hold belong to the r -boundary ∂ r
( A, B ) .
There are two cases to consider A ij · B ij = 0 and A ij · B ij = 1 since A and B are binary.
Let
[Equation]
2 r + 1) (cid:1) .
Let E
= { ( i,
j ) ∈
Z n : A ij · B ij =
∧ b σ ij 6 = 0 } and
E
= { ( i,
j ) ∈
Z n : A ij · B ij = 1 ∧ b σ ij 6 = 1 } .
Since clearly 0 ≤ b σ ij ≤ 1 and h A, B i = |{ ( i, j ) :
A ij · B ij = 1 }| , we have
[Equation]
In particular, if we show E 0 , E 1 ⊆
∂ r ( A, B ) , then (16a) will be established.
This is what we show in the rest of the proof.
For readability, we repeat here the two conditions for a pair ( i, j ) to be included in the boundary, see deﬁnition 2.
Namely, ( i, j ) belongs to ∂ r ( A, B ) if and only if both conditions below hold:
∃ a 1 , b 1 , c 1 , d 1 ∈ {− r, . . .
, r } : ( i + a 1 , j + b 1 ) ∈
A ∧ ( i + c 1 , j + d 1 ) ∈ B . 2.
∃ a 2 , b 2 , c 2 , d 2 ∈ {− r, . . .
, r } : ( i + a 2 , j + b 2 )
/ ∈
A ∨ ( i + c 2 , j + d 2 ) / ∈ B .
Let ( i, j ) be such that A ij · B ij = 0 .
By equation (30), b σ ij = 0 if and only if [ A ] ¯ rij = 0 or [ B ] ¯ rij = 0 .
So, when A ij · B ij = 0 , for b σ ij 6 = 0 to hold it is necessary that there are a 1 , b 1 , c 1 , d 1 ∈ {− r, . . .
, r } such that A
i + a 1 ,j + b 1 = 1 and A i + c 1 ,j + d 1 = 1 which corresponds to the ﬁrst item in deﬁnition 2.
The second item holds with a 2 = b 2 =
c
=
d 2 = 0 since A ij · B ij =
so either A ij = 0 or
B ij = 0 .
This shows that pairs of indices for which A ij · B ij = 0 and b σ ij 6 = 0 are contained in ∂ r ( A, B ) , in other words that E
0 ⊆
∂ r ( A, B ) .
Now take ( i, j ) such that A ij · B ij = 1 .
By equation (30), b σ ij = 1 if and only if [ A ] ¯ rij = 1 (2 r +1) × (2 r +1) and [ B ] ¯ rij = 1 (2 r +1) × (2 r +1) .
So, when A ij · B ij = 1 and b σ ij 6 = 1 , then there are a 2 , b 2 , c 2 , d 2 ∈ {− r, . . .
, r } such that A
i + a 2 ,j + b 2 = 0 or B i + c 2 ,j + d 2 = 0 which corresponds to the second item in deﬁnition 2.
The ﬁrst item holds with a 1 = b 1 = c 1 = d 1 = 0 since A ij · B ij = 1 so both A ij = 1 and B ij = 1 .
This shows that pairs of indices for which A ij · B ij = 1 and b σ ij 6 = 1 are contained in ∂ r ( A, B ) , or equivalently that E 1 ⊆
∂ r ( A, B ) .
This concludes the proof of (16a).
As for (16b), it follows easily by substituting in the formula k
A − B k 2 = k
A k 2 + k B k 2 − 2 h A, B i .
We then use the fact E (cid:2)(cid:13)(cid:13)
R( F ∗ A )(cid:13)(cid:13)(cid:3) =
k
A k 2 , which follows from (9), recalling that ν 2 = 2 / (