Feature Learning in L 2 -regularized DNNs: Attraction/Repulsion and Sparsity
Abstract
We study the loss surface of DNNs with L 2 regularization.
We show that the loss in terms of the parameters can be reformulated into a loss in terms of the layerwise activations Z ℓ of the training set.
This reformulation reveals the dynamics behind feature learning: each hidden representations Z ℓ are optimal w.r.t.
to an attraction/repulsion problem and interpolate between the input and output representations, keeping as little information from the input as necessary to construct the activation of the next layer.
For positively homogeneous nonlinearities, the loss can be further reformulated in terms of the covariances of the hidden representations, which takes the form of a partially convex optimization over a convex cone.
This second reformulation allows us to prove a sparsity result for homogeneous
DNNs: any local minimum of the L 2 -regularized loss can be achieved with at most N ( N + 1) neurons in each hidden layer (where N is the size of the training set).
We show that this bound is tight by giving an example of a local minimum that requires N 2 / 4 hidden neurons.
But we also observe numerically that in more traditional settings much less than N 2 neurons are required to reach the minima.
Introduction
It is generally believed that the success of deep learning hinges on the ability of deep neural networks (DNNs) to learn features that are well suited to the task they are trained on.
There is however little understanding of what these features are and how they are selected by the network.
On the other hand recent results [12] have shown that it is possible to train DNNs without feature learning.
This suggests the existence of two regimes of DNNs, a kernel regime (also called lazy or
NTK regime) without feature learning and an active regime where features are learned.
The presence or absence of feature learning can depend on multiple factors, such as the initialization/parametrization of DNNs [3, 25, 17, 13], very large depths [10] or large learning rate [16, 5].
In this paper, we focus on the impact of L 2 regularization on feature learning in DNNs.
This analysis is further motivated by recent results [8, 9, 4] which show that the implicit bias of gradient descent on losses such as the cross entropy (which decay exponentially towards infinity) is essentially the same as the bias induced by L 2 regularization in DNNs.
Generally, the bias induced by the addition of L 2 -regularization on the parameters θ of a model f θ can be described by the representation cost R ( f ) =
min θ : f θ = f
∥ θ ∥ 2 , since min θ C ( f θ ) + λ ∥ θ ∥ 2 = min f C ( f ) + λR ( f ) .
In deep linear networks, the addition of L 2 regularization on the parameters corresponds to the addition of an L p -Schatten norm regularization to the represented matrix, with p = 2 / L where L is the depth of the network [9, 6].
This implies a sparsity effect that increases with depth L .
In non-linear networks the sparsity effect of L 2 -regularization has been described for shallow networks ( L = 2 ) in [1, 20, 23] or for shallow non-linear networks with added linear layers [19].
Though it seems natural that this effect should become stronger for deeper networks, to our knowledge little theoretical work has been done in this area.
1.1 Contributions
In this paper, we study the minima of the loss of L 2 regularized fully-connected DNNs of depth L .
We propose two reformulations of the loss:
The first reformulation expresses the loss in terms of the representations Z 1 , . . .
, Z L (the layer pre-activations of every input in the training set) of every layer of the network.
This reformulation has the advantage of being local - the optimal choice of a layer Z ℓ only depends on its neighboring layers Z ℓ − 1 and Z ℓ +1 .
The optimal choice of representation Z ℓ is at the balance between an attractive force (determined by the previous layer) and a repulsive force (coming from the next layer).
It illustrates how the representations Z 1 , . . .
, Z L −interpolates between the input layer Z 0 and output layer Z L . 2.
The second reformulation expresses the loss in terms of the covariances of the representation before applying the non-linearity K ℓ = Z Tℓ Z ℓ and after the non-linearity K σℓ =
( Z σℓ ) T Z σℓ .
For positively homogeneous non-linearities and when the number of neurons n ℓ in every hidden layer ℓ is larger or equal to N ( N + 1) for N the number of datapoints, this reformulation is an optimization of a (partially convex) loss over the covariances ( K 1 , K σ 1 ) , . . .
, ( K L − 1 , K σL − 1 ) and the outputs Z L , restricted to a (translated) convex cone.
This reformulation does not depend on the number of neurons n ℓ in the hidden layers (as long as n ℓ ≥ N ( N + 1) ).
The second reformulation implies that for positively homogeneous non-linearities such as the ReLU, as the number of neurons in the hidden layers n ℓ increase, the global minimum of the L 2 -regularized loss goes down until reaching a plateau (i.e. adding neurons does not lead to an improvement in the loss).
This illustrates the sparsity effect of L 2 -regularization, where the optimum reached on a very large network is equivalent to a much smaller network.
The start of the plateau hence gives a measure of sparsity of the global minimum.
We show that the minimal number of neurons n ℓ to reach this plateau is determined by a notion of rank Rank σ ( K ℓ , K σ ℓ ) of the covariance pairs.
We show that Rank σ ( K ℓ , K σℓ ) ≤ N ( N + 1) , i.e. the plateau must start before N ( N + 1) and that the scaling of this upper bound is tight by giving an example dataset such that at the optimum Rank σ ( K ℓ , K σℓ ) ≥ N 2 / 4 .
We also present other datasets where the start of the plateau is either constant or grows linearly with the number of datapoints.
We also observe empirically that the plateau can start at much smaller widths for real data such as MNIST and the teacher/student setting.
Setup
We consider fully-connected deep neural networks with L + 1 layers, numbered from 0 (the input layer) to L (the output layer), with nonlinear activation function σ : R → R (e.g. the ReLU σ ( x ) = max { 0 , x } )).
Each layer ℓ contains n ℓ neurons and we denote n = ( n 1 , . . .
, n L ) the widths of the network.
Given an input dataset { x 1 , . . .
x N } ⊂ R n 0 of size N , we consider the data matrix
X = ( x 0 , . . .
, x n ) ∈ R n 0 × N , and encode the activations and preactivations of the whole data set by considering the pre-activations Z ℓ ( X ; W ) ∈ R n ℓ × N and activations Z σℓ ( X ; W ) ∈ R ( n ℓ +1) × N given by:
[Equation]
where W = ( W ℓ ) ℓ =1 ,...,L is the collection of n ℓ × ( n ℓ − 1 + 1)
-dim weight matrices W ℓ , σ ( Z ℓ ) is obtained by applying elementwise the nonlinearity σ to the matrix Z ℓ , and the scalar β ∈ R represents the amount of bias (i.e. when β =
there is no bias, when β = 1 this definition is equivalent to the traditional definition of bias).
The output of the network is the pre-activation of the L -th layer Z L .
We often drop the dependence on the weights W and on the dataset X and simply write Z ℓ and Z σℓ .
We denote f W : R n 0 → R n L the network function , which maps an input x to the pre-activation at the last layer.
2.1 L 2 -Regularized Loss and Representation Cost
Given a general cost functional C : R n L × N → R , the L 2 -regularized loss of DNNs of widths n is
[Equation]
where ∥ W ∥ is the L 2 -norm of W understood as a vector.
Note that ∥ W ∥ 2 = (cid:80) Lℓ =
∥ W ℓ ∥ 2 F where ∥·∥ F denotes the Frobenius norm.
From now on, we often omit to specify the widths n and simply write L λ .
The additional regularization cost should bias the network toward low norm solutions.
This bias on the parameters leads to a bias in function space, which is described by the so-called representation cost R n ( f ) defined on functions f : R n 0 → R n L :
[Equation]
where the minimum is taken over all choices of parameters W of a width n network, with fixed β bias amount, such that the network function f W equals f .
By convention, if no such parameters exist then R n ( f ) = + ∞ .
Similarly, given an input-output pair X ∈ R n 0 × N , Y ∈ R n L × N , the representation cost R n ( X, Y ) is:
[Equation]
with again the convention that if there exists no weight W such that Z L ( X, W )
=
Y , then
R n ( X, Y ) =
+ ∞ .
The representation cost R n naturally describes the bias induced by the L 2regularized loss of DNNs since:
[Equation]
Two Reformulations of the Regularized Loss: Hidden Representation and Covariance Optimization
We now provide two reformulations of the L 2 -regularized loss L λ ( W ) and representation cost
R n ( X, Y ) , which both put emphasis on the hidden representations Z ℓ and how they are progressively modified throughout the neural network.
The first reformulation holds for general non-linearities while the second only applies to networks with homogeneous nonlinearities.
3.1 Feature optimization : attraction/repulsion
The key observation is that the weights W ℓ can be decomposed as follows:
[Equation]
[Figure]
where the residual matrix ˜ W ℓ is orthogonal to Z σℓ − 1 , i.e. ˜ W ℓ Z σℓ − 1 = 0 , and ( · ) + is the Moore-Penrose pseudo-inverse.
This stems from the fact that W ℓ = W ℓ P Im Z σℓ − 1 + ˜ W ℓ where ˜ W ℓ := W ℓ P
(Im Z σℓ − 1 ) ⊥ , and P
Im Z σℓ − 1 , resp.
P (Im Z σℓ − 1 ) ⊥ , is the orthogonal projection on Im Z σℓ − 1 , resp.
on the orthogonal complement of Im Z σℓ − 1 ; one concludes using the facts that P Im Z σℓ − 1 =
Z σℓ − 1 (cid:0)
Z σℓ − 1 (cid:1) + and Z ℓ =
W ℓ Z σℓ − 1 .
Note that the matrix ˜ W ℓ does not affect either the hidden representations Z ℓ nor the output Z L .
Besides, the Frobenius norm of W ℓ can be rewritten as || W ℓ || 2 F = || Z ℓ (cid:0) Z σℓ
[Equation]
1 (cid:1) + || 2 F + || ˜ W ℓ
|| 2 F .
When minimizing the L 2 -regularized cost, it is therefore always optimal to consider null residual matrices ˜
W ℓ = 0 , resulting in a reformulation of the cost which only depends on the pre-activations Z ℓ :
Proposition 1.
The infimum of L λ ( W ) =
C ( Z L ( X ; W ))
+ λ ∥ W ∥ 2 , over the parameters W ∈ R P is equal to the infimum of
[Equation]
over the set Z of hidden representations Z = ( Z ℓ ) ℓ =1 ,...,L such that Z ℓ ∈ R n ℓ × N , Im Z Tℓ +1 ⊂
Im ( Z σℓ ) T , with the notations Z σ 0 = (cid:18)
Xβ 1 TN (cid:19) and Z σℓ = (cid:18) σ ( Z ℓ ) β 1 TN (cid:19) .
Furthermore, if W is a local minimizer of L λ then ( Z 1 ( X ; W ) , . . .
, Z L ( X ; W )) is a local minimizer of L rλ .
Conversely, keeping the same notations, if ( Z ℓ ) ℓ =1 ,...,L is a local minimizer of L rλ , then
W =
( Z ℓ ( Z σℓ − 1 ) + ) ℓ =1 ,...,L is a local minimizer of L λ .
Note that one can also reformulate the representation cost:
[Equation]
The representation in terms of the output and hidden representations have several interesting properties, especially when it comes to minimization:
The optimization becomes local in the sense that all terms and constraints depend either only on the output cost C ( Z L ) or on two neighboring terms (e.g. (cid:13)(cid:13)(cid:13) Z ℓ (cid:0) Z σℓ − 1 (cid:1) + (cid:13)(cid:13)(cid:13) 2 F ).
As a result, the (projected) gradient of the loss L rλ ( Z 1 , ..., Z L ) w.r.t.
to Z ℓ only depends on Z ℓ − 1 , Z ℓ and Z ℓ +1 .
This is in contrast to the optimization of L λ ( W ) , where the gradient of C ( Z L ) with respect to W ℓ depends on all parameters W 1 , . . .
, W L .
The value (cid:13)(cid:13)(cid:13) Z ℓ (cid:0) Z σℓ − 1 (cid:1) + (cid:13)(cid:13)(cid:13) 2 F represents a ’multiplicative distance’ between Z ℓ and Z σℓ −(in contrast to the ’additive distance’ (cid:13)(cid:13)
Z ℓ − Z σℓ − 1 (cid:13)(cid:13) 2 F ); the representation Z ℓ therefore interpolates multiplicatively between Z ℓ − 1 and Z ℓ +1 .
This is most obvious for linear networks (i.e. σ = id and β = 0 ): in this case, one can check that at any global minimizer, the covariances of the hidden layers equal Z Tℓ Z ℓ = X T
( X − T Z TL Z L X − 1 ) ℓL X , interpolating between the input covariance X T X and output covariance Z TL Z L .
A lot of work has been done to propose biologically plausible training methods for DNNs [2, 11], in contrast to backpropagation which is not local.
A line of work [21, 18, 22], propose a biologically plausible optimization technique which minimizes a cost which closely resembles our first reformulation, with the multiplicative distances (cid:13)(cid:13)(cid:13) Z ℓ (cid:0) Z σℓ − 1 (cid:1) + (cid:13)(cid:13)(cid:13) 2 F replaced by additive ones (cid:13)(cid:13)
Z ℓ − Z σℓ − 1 (cid:13)(cid:13) 2 F .
Due to this change, there is no direct correspondence between the networks trained with this biologically plausible technique and those trained with backpropagation.
If one could extend this training technique to work with multiplicative distances one could guarantee such a direct correspondence.
The optimization leads to an attraction-repulsion algorithm.
If we optimize only on the term Z ℓ and fix all other representations, the only two terms that depend on Z ℓ are (cid:13)(cid:13)(cid:13)
Z ℓ (cid:0) Z σℓ − 1 (cid:1) + (cid:13)(cid:13)(cid:13) 2 F and (cid:13)(cid:13)(cid:13) Z ℓ +1 ( Z σℓ ) + (cid:13)(cid:13)(cid:13) 2 F .
The former term is attractive as it pushes the representations Z ℓ towards the origin (and hence pushes the representations at depth ℓ of every input towards each other), especially along directions where Z σℓ − 1 is small.
The latter term is repulsive as it pushes the representations Z σℓ away from the origin, especially along directions where Z ℓ +1 is large.
This attraction-repulsion process is similar to the Information Bottleneck theory [24]: the repulsive term ensure that Z ℓ keeps enough information about the inputs to reconstruct Z ℓ +1 , while the attractive term pushes Z ℓ to keep as little information as possible.
The attraction and repulsion forces of of the ℓ -th layer are the derivative ∂ Z ℓ (cid:13)(cid:13)(cid:13) Z ℓ (cid:0)
Z σℓ − 1 (cid:1) + (cid:13)(cid:13)(cid:13) 2 F and ∂ Z ℓ (cid:13)(cid:13)(cid:13) Z ℓ +1 ( Z σℓ ) + (cid:13)(cid:13)(cid:13) 2 F which are both n ℓ × N matrices.
One can visualize these forces either column by column (each column corresponding to a datapoint i = 1 , . . .
, N ) or line by line (each line corresponding to a neuron k = 1 , . .
. , n ℓ ).
These two visualizations of the forces are presented in
Figure 1 for the two hidden layers of a depth L = 3 network, projected to the 2 largest principal components of the columns resp.
lines of Z ℓ .
Figures 1a and 1c illustrate how the inputs corresponding to different classes are pushed away from each other, leading to a clustering effect.
Figures 1b and 1d show that the neurons naturally align along rays starting from the origin.
This happens for homogeneous non-linearities, such as the ReLU in this example, because if two neurons k, k ′ have proportional activations, i.e. Z ℓ,k = αZ ℓ,k for some α ∈ R , then their attractive and repulsive forces will also be proportional with the same scaling α .
As a result, the neuron k is stable, i.e. the attraction and repulsion cancel each other, if and only if the neuron k ′ is stable.
This phenomenon can be interpreted as a form of sparsity: a group of aligned neurons can be replaced by a single neuron without changing the resulting function Z L .
Can we guarantee a degree of sparsity in the hidden representations?
Can we bound the number of aligned groups in a neuron?
In the next section, we introduce a further reformulation of the loss which allows us to partially answer these questions.
3.2 Covariance learning : partial convex optimization for positively homogeneous nonlinearities
The loss of the first reformulation L rλ depends on the hidden representations Z ℓ and Z σℓ only through the covariances K ℓ =
Z Tℓ Z ℓ and K σℓ =
( Z σℓ ) T Z σℓ , since (cid:13)(cid:13)(cid:13) Z ℓ (cid:0) Z σℓ − 1 (cid:1) + (cid:13)(cid:13)(cid:13) 2 F =
Tr (cid:104) K ℓ (cid:0) K σℓ − 1 (cid:1) + (cid:105) .
Hence, we provide a second reformulation expressed in terms of the tuple of covariance pairs K
= (( K 1 , K σ 1 ) , . . .
, ( K L − 1 , K σL − 1 )) and the outputs Z L .
Using the notations K σ 0 =
X T X + β 2 1 N × N and K L =
Z TL Z L , we define:
[Equation]
It remains to identify the set K n ( X ) of covariances K and outputs Z L which can be represented by a width n network with inputs X .
For positively homogeneous nonlinearities of degree 1 such as the ReLU (i.e. when σ ( λx ) =
λσ ( x ) for any positive λ ), the set K n ( X ) can be expressed using the notion of conical hulls.
Definition 2.
The conical hull of Ω ⊂
R d is the set cone (Ω) :=
[Equation]
and
its
[Equation]
-conical hull for m ≥ 1 is the set
[Equation]
Note that by Caratheodory’s theorem for conical hulls, cone m (Ω) =
cone (Ω) for any m ≥ d .
We now proceed to the description of the set K n ( X ) and obtain the second formulation of the Lregularized loss and of the representation loss R n in terms of covariances:
Proposition 3.
For positively homogeneous non-linearities σ , the infimum of L λ ( W ) =
C ( Z L ( X ; W ))
+ λ ∥ W ∥ 2 , over the parameters W ∈ R P is equal to the infimum over K n ( X ) of
[Equation]
The set K n ( X ) is the set of covariances K
= (( K 1 , K σ 1 ) , . . .
, ( K L − 1 , K σL − 1 )) and outputs Z L such that for all hidden layer ℓ = 1 , . . .
, L − 1 :
• the pair ( K ℓ , K σℓ ) belongs to the (translated) n ℓ -conical hull
[Equation]
[Equation]
• Im K ℓ ⊂ Im K σℓ − 1 , with the notation K σ
=
X T X + β 2 1 N × N , and Im Z L ⊂
Im K σL − 1 .
Note that one can also reformulate the representation cost:
[Equation]
Remark 4 .
In contrast to the first loss L rλ whose local minima were in correspondence with the local minima of the original loss L λ , the second loss L k λ can in some cases have strictly less critical points and local minima.
Indeed, since the map W (cid:55)→ ( K , Z L ) is continuous, if ( K (W) , Z L ( W )) is a local minimum, then so is W .
However, the converse is not true: we provide a counterexample in the
Appendix, i.e. a set of weights W of a depth L = 2 network which is a local minimum of L λ and such that the corresponding ( K , Z L ) is not a local minimum of L k λ .
Since the dimension of the space of pairs of symmetric N × N matrices is N ( N +1) , if n ℓ ≥ N ( N +1) then, by the Caratheodory’s theorem for conical hulls,
[Equation]
Hence, as soon as n ℓ ≥ N ( N + 1) for all hidden layers, the set K n ( X ) does not depend on the list of widths n .
We denote
[Equation]
this width-independent set.
The following proposition shows that for sufficiently wide networks with a positively homogeneous nonlinearity, training a deep DNN with L 2 regularization is equivalent to a partially convex optimization over a translated convex cone.
Proposition 5.
The set K ( X ) is a translated convex cone: after the suitable translation, it is equal to its conical hull.
The cost L k λ ( K , Z L ) is partially convex w.r.t.
to the outputs Z L and the pairs ( K ℓ , K σℓ ) , i.e. it is convex if one fixes the other parameters and let only ( K ℓ , K σℓ ) , or Z L , vary.
3.3 Direct optimization of the reformulations
It is natural at this point to wonder whether one could optimize directly over the representations
Z (using the first reformulation) or over the covariances K and output Z L (using the second reformulation), and whether this would have an advantage over the traditional optimization of the weights.
For the first reformulation, one can simply use the projected gradient descent with updates given by
[Equation]
for any projection P Z to the constraint space Z .
For example, a projection is obtained by mapping
Z ℓ to Z ℓ P Im Z σℓ − 1 sequentially from ℓ = 1 to ℓ = L .
Note however that the loss explodes as the constraints become unsatisfied so that for gradient flow, there is no need for the projections.
This suggests that these projections might also be unnecessary as long as the learning rate is small enough.
For more details, see Appendix B.1.
For the second reformulation, there is no obvious way to compute a projection to the constraint space
K : the cone S is spanned by an infinite amount of points and we do not have an explicit formula for the dual cone S ∗ .
Frank-Wolfe optimization can be used to overcome the need for computing the projections.
However, these direct optimizations of the reformulations lead to issues of computational complexity and stability.
First, the computation of the gradients ∇L rλ
( Z ) and ∇L k λ
( K , Z L ) requires solving a linear equation of dimension N , which is very costly, in contrast to the traditional optimization of the weights W for which the gradient can be computed very efficiently.
Second, if Z σℓ is not full-rank, the computation of its pseudo-inverse ( Z σℓ ) + and the projection P Im Z σℓ are very unstable.
Therefore, if we only have finite-precision knowledge of Z ℓ , we cannot reliably compute ( Z σℓ ) + nor P Im Z σℓ .
Although it could be possible to solve these problems (e.g. using the Tikhonov regularization for the unstability problem) and to develop efficient algorithms to optimize both reformulations efficiently, we decided in this paper to focus on the theoretical implications of these reformulations.
Sparsity of the Regularized Optimum for Homogeneous DNNs
In this section, we assume that the non-linearity is positively homogenous.
Under this assumption, the second reformulation of the loss (and of the representation cost) holds and implies the existence of a sparsity phenomenon.
First observe that as the widths n increase, both the global minimizer of the loss min W L λ ( W ) and the representation cost R n ( X, Y ) diminish.
We denote by L λ, n the L 2 -regularized loss of DNNs with widths n .
Recall that the depth L is fixed.
Proposition 6.
If n ≤ n ′ (in the sense that n ℓ ≤ n ′ ℓ for all ℓ and n 0 = n ′ 0 and n L = n ′ L ), then
[Equation]
and for any X ∈ R n 0 × N and Y ∈ R n L × N , R n ( X, Y ) ≥ R n ′ ( X, Y ) .
Proof.
Let us assume that the parameters W ∗ are optimal for a width n network, the parameters can be mapped to parameters of a wider network by adding ‘dead’ neurons (i.e. neurons with zero incoming and outcoming weights) without changing the network function f W ∗ nor the norm of the parameters ∥ W ∥ .
For DNNs with positively homogeneous nonlinearities, a direct consequence of our reformulation through the hidden covariances is that both the global minimum of the loss L λ and the representation cost R n ( X, Y ) plateau for any widths n such that n ℓ ≥ N ( N + 1) .
Proposition 7.
For any positively homogeneous nonlinearity σ , any widths n and n ′ such that n 0 = n ′ 0 , n L = n ′ L and for all ℓ = 1 , . . .
, L − 1 , n ℓ , n ′ ℓ ≥ N ( N + 1) , for all λ > 0 , we have:
[Equation]
Under the same conditions R n ( X, Y ) =
R n ′ ( X, Y
Proof.
This follows directly from the second reformulation and the fact that by Caratheodory’s theorem for conical hulls, S n = S if n ≥ N ( N + 1) (see discussion after Remark 4).
We can therefore define a width-independent representation cost R ( X, Y ) (which still depends on the fixed depth L ) equal to the representation cost R n ( X, Y ) of any sufficiently wide network.
4.1 Rank of the Hidden Representations
Now that we have revealed the plateau phenomenon, a natural question that we investigate in this section is when does this plateau begin.
In order to do so, we introduce the notion of rank
Rank σ ( K, K σ ) of a pair of Gram matrices ( K, K σ ) ∈ S which is the minimal number k
such that
[Equation]
(4.1) for some z 1 , . . .
, z k ∈ R N .
This notion of rank describes exactly the minimal number of neurons required to recover a set of covariances K :
Proposition 8.
Let ( K , Z L ) ∈ K
( X ) , then there are parameters W of a width n network with covariances and outputs K if and only if n ℓ ≥ Rank σ ( K ℓ , K σℓ ) for all ℓ = 1 , . . .
, L − 1 .
We can now describe the plateau R = { n : min W ∈ R P n L λ, n ( W ) =
min m min W ∈ R P
m L λ, m ( W ) } , i.e. the set of widths n such that the minimum min W ∈ R P n L λ, n ( W ) is optimal over all possible widths:
Corollary 9.
Let K min be the set of covariances sequences ( K , Z L ) which are global minima of the second reformulation.
We have that n ∈ R if and only if there is a ( K , Z L ) ∈ K min such that n ℓ ≥ Rank σ ( K ℓ , K σℓ ) .
Hence, the investigation of Rank σ ( · , · ) is crucial to understand where the plateau begins; unfortunately, it can be difficult to compute.
However, from its definition and the Caratheodory’s theorem for conical hulls (see our discussion after Remark 4), we have the following natural bounds:
Lemma 10.
For any pair ( K, K σ ) ∈ S , we have Rank ( K ) ≤ Rank σ ( K, K σ ) ≤ N ( N + 1) .
We show in the next section that the order of magnitude of the upper bound is tight.
More specifically, we construct a dataset for which any global optimum satisfies Rank σ ( K 1 , K σ 1 ) ≥ N 2 / 4 .
This implies that, in this example, the plateau transition occurs when the number of hidden neurons is of order
O ( N 2 ) .
Note however that, in our numerical experiments (see Figure 2), the rank of the global optimum can be much smaller for more traditional dataset such as MNIST.
Remark 11 .
The start of the plateau measures a notion of sparsity of the learned network, since the networks learned in the plateau are equivalent to a network at the start of the plateau, i.e. large networks are equivalent in terms of their covariances and outputs ( K , Z L ) to a (potentially much) smaller network.
Even though the set of pairs ( K ℓ , K σℓ ) in the cone S that are not full rank has measure zero, the optimal representations ( K ℓ , K σℓ ) always lie on the border of the cone S (since the derivative of the cost L kλ w.r.t.
( K ℓ , K σℓ ) never vanishes) where the rank is lower.
More precisely, the rank is determined by the dimension of the smallest face that contains the optimum (e.g. the pairs ( K ℓ , K σℓ ) on the edges of S have rank at most 2 for example, while those on the vertices are rank 1).
We can identify different degrees of sparsity depending on how the rank of the hidden representations scales with the number of datapoints N : if the rank is o ( N ) the covariances K ℓ , K σℓ are low-rank (in the traditional linear sense) and for shallow networks the effective number of parameters (i.e. the number of parameters at the start of the plateau) is o ( N 2 ) , if the rank is o ( √ N ) then for deep networks the effective number of parameters o ( N ) .
This could explain why very large networks with ‘too many parameters’ are able to generalize, since their effective number of parameters is of the order of the number datapoints.
Very large networks can therefore be trained safely knowing that thanks to L 2 -regularization, the network is able to recognize what is the ’right’ width of the network.
4.2 Tightness of the Upper-Bound
In this section, we construct a pair of input and output datasets X and Y , both in R N × N , such that for any optimal parameters W of a ReLU network of depth L = 2 with no bias ( β = 0 ), the rank of the hidden representation Rank σ ( K 1 , K σ 1 ) is greater than N 2 / 4 .
Note that one can write the decomposition (4.1) as K = C T C and K σ = B T B where C
=
( z 1 , . . .
, z k ) and B = ReLU( C ) is obtained by applying elementwise the ReLU to C .
Key to our construction is the fact that B is then a matrix with non-negative entries: the matrix K σ is completely positive and Rank σ ( K, K σ ) can be studied using the CP-rank of K :
Definition 12.
A N × N matrix A is completely positive if A = B T B for a k × N matrix B with non-negative entries.
The CP-rank Rank cp ( A ) of a completely positive matrix
A is the minimal integer k such A = B T B for a k × N matrix B with non-negative entries.
When σ is the ReLU, the kernel K σℓ is completely positive for all hidden layers ℓ , and thus
[Equation]
In order to obtain the tightness of the upper bound, we proceed in two steps: first, we construct a completely positive matrix A with high CP-rank, and then construct inputs X and outputs Y such that the optimal hidden covariance K 1 = K σ 1 for a depth L = 2 network equals the matrix A .
As shown in [7], bi-partite graphs can be used to construct matrices with high CP-rank.
We refine this by showing that graphs on N vertices without cliques of 3 or more vertices lead to N × N matrices with CP-rank equal to the number of edges, and as a corollary, we construct a completely positive matrix with CP-rank equal to N 2 / 4 .
Proposition 13.
Given a graph G with N vertices and k edges, consider the k × N matrix E with entries E ev = 1 if the vertex v is an endpoint of the edge e and E ev = 0 otherwise.
The matrix
A =
E T E is completely positive and if the graph G contains no cliques of 3 or more vertices then
Rank cp ( A ) =
k .
Hence, to obtain a completely positive matrix of high CP-rank, it remains to find a graph with no cliques and as many edges as possible.
For even N , we consider the complete bipartite graph, i.e. the graph with two groups of size N/ 2 and with edges between any two vertices iff they belong to different groups.
For this graph, the matrix B N = E T E takes the form of a block matrix:
[Equation]
where 1 N 2 is the N/ 2 × N/ 2 matrix with all ones entries.
Since this bipartite graph has no cliques and N 2 / 4 edges, from the previous proposition, we obtain Rank cp ( B N )
=
N 2 4 .
The following proposition shows how for any completely positive matrix (with CP-rank k ) there is a dataset such that a shallow ReLU network will have a hidden representation pair ( K 1 , K σ 1 ) of rank k :
Proposition 14.
Consider a width- n shallow network ( L = 2 ) with ReLU activation, no bias β = 0 , n 0 = N , n 1 ≥ N ( N + 1) , input dataset X N = I N , and any output dataset Y N such that (cid:0) Y TN Y N (cid:1)is a completely positive matrix with CP-rank k .
At any global minimum of R n ( X N , Y N ) , we have Rank σ ( K 1 , K σ 1 ) =
k .
Furthermore for λ small enough, at any global minimum of L MSE λ, n ( W ) =
1 N ∥ Y ( X N ; W ) − Y N ∥ 2 F + λ ∥
W ∥ 2 , we have
Rank σ ( K 1 , K σ 1 ) ≥ k .
By Proposition 14, with the outputs Y N = B N , the rank of the hidden representations (and the start the plateau) is larger or equal to N 2 4 .
This shows that the order N 2 of the bound of Lemma 10 is tight when it comes to data-agnostic bounds.
However under certain assumptions on the data one can guarantee a much earlier plateau.
For example, if we instead apply Proposition 14 to a task closer to classification, where the columns of the outputs Y N ∈ R n L × N are one-hot vectors, then ( Y TN Y N ) 12 is (up to permutations of the columns/lines) a block diagonal matrix with n L constant positive blocks, which is completely positive with CP-rank equal to the number of classes n L .
This is in line with our empirical experiments in
Figure 2 where we observe in MNIST a plateau starting roughly at a width of 10, which is the number of classes.
Another example where the structure of the data leads to an earlier plateau is when the input and output dimensions are both 1, in which case we can guarantee that the start of the plateau grows at most linearly with the number of datapoints N :
Proposition 15.
Consider shallow networks ( L = 2 ) with scalar inputs and outputs ( n 0 = n 2 = 1 ), a
ReLU nonlinearity, and a dataset X, Y ∈ R 1 × N .
Both the representation cost R n ( X, Y ) and global minimum min W L λ, n ( W ) for any λ > 0 are independent of the width n 1 as long as n 1 ≥ 4 N .
More generally, we propose to view the start of the plateau as an indicator of how well a certain task is adapted to a DNN architecture.
An early plateau suggests that the network is able to solve the task optimally with very few neurons, in contrast to a late plateau.
The fact that the optimal network requires few neurons (and hence few parameters) can be used to guarantee good generalization.
4.3 Conclusion
We have given two reformulations of the loss of L 2 -regularized DNNs.
The first works for a general non-linearity and shows how the hidden representations of the inputs Z 1 , . . .
, Z L − 1 are learned to interpolate between the input and output representations, as a balance between attraction and repulsion forces for every layer.
The second reformulation for homogeneous non-linearities allows us to analyze a sparsity effect of L 2 -regularized DNNs, where the learned networks are equivalent to another network with much fewer neurons.
This effect can be visualized by the appearance of a plateau in the minimal loss as the number of neurons grows, the earlier the plateau, the sparser the solution, since an early plateau means that very few neurons were required to obtain the same loss as a network with an infinite number of neurons.
We show that this plateau cannot start later than N ( N + 1) , and then show that the order of this bound is tight by constructing a toy dataset for which the plateau starts at N 2 / 4 , however, we observe that on more traditional datasets, the start of the plateau can be much earlier.
Acknowledgements
C. Hongler acknowledges support from the Blavatnik Family Foundation, the Latsis Foundation, and the NCCR Swissmap.
[-Ψ( Z i ) , however this sequence might not be convergent since Ψ is not continuous, however we know the sequence is bounded, since ∥ W i ∥ 
-]
[-Tℓ +
-]
[-Tℓ +
-]
[-Such weights a 1 , a 2 , b 1 , b 2 must satisfy a 1 = b 1 and a 
-]
[-Im K σ 
-]
[-This follows from the fact that if K 1 ̸ = K σ 1 , then the pair ( K σ 1 , K σ 1 ) has a strictly lower cost than the pair ( K 1 , K σ 1 ) : for any Z 1 such that K 
-]
[-There are more than 4 neurons whose cusp lies between two inputs x i and x i +
-]