P ARALLEL D EEP N EURAL N ETWORKS WITH MODIFIED REGULARIZATIONS H AVE Z ERO D UALITY G AP
A BSTRACT
Training deep neural networks is a challenging non-convex optimization problem.
Recent work has proven that the strong duality holds (which means zero duality gap) for regularized ﬁnite-width two-layer ReLU networks and consequently provided an equivalent convex training problem.
However, extending this result to deeper networks remains to be an open problem.
In this paper, we prove that the duality gap for deeper linear networks with vector outputs is non-zero.
In contrast, we show that the zero duality gap can be obtained by stacking standard deep networks in parallel, which we call a parallel architecture, and modifying the regularization.
Therefore, we prove the strong duality and existence of equivalent convex problems that enable globally optimal training of deep networks.
As a by-product of our analysis, we demonstrate that the weight decay regularization on the network parameters explicitly encourages low-rank solutions via closed-form expressions.
In addition, we show that strong duality holds for three-layer standard ReLU networks given rank-1 data matrices.
I NTRODUCTION
Deep neural networks demonstrate outstanding representation and generalization abilities in popular learning problems ranging from computer vision, natural language processing to recommendation system.
Although the training problem of deep neural networks is a highly non-convex optimization problem, simple ﬁrst order gradient based algorithms, such as stochastic gradient descent, can ﬁnd a solution with good generalization properties.
However, due to the non-convex and non-linear nature of the training problem, underlying theoretical reasons for this remains an open problem.
The Lagrangian dual problem (Boyd et al., 2004) plays an important role in the theory of convex and non-convex optimization.
For convex optimization problems, the convex duality is an important tool to determine their optimal values and to characterize the optimal solutions.
Even for a non-convex primal problem, the dual problem is a convex optimization problem the can be solved efﬁciently.
As a result of weak duality, the optimal value of the dual problem serves as a non-trivial lower bound for the optimal primal objective value.
Although the duality gap is non-zero for non-convex problems, the dual problem provides a convex relaxation of the non-convex primal problem.
For example, the semi-deﬁnite programming relaxation of the two-way partitioning problem can be derived from its dual problem (Boyd et al., 2004).
The convex duality also has important applications in machine learning.
In Paternain et al.
(2019), the design problem of an all-encompassing reward can be formulated as a constrained reinforcement learning problem, which is shown to have zero duality.
This property gives a theoretical convergence guarantee of the primal-dual algorithm for solving this problem.
Meanwhile, the minimax generative adversarial net (GAN) training problem can be tackled using duality (Farnia & Tse, 2018).
In lines of recent works, the convex duality can also be applied for analyzing the optimal layer weights of two-layer neural networks with linear or ReLU activations (Ergen & Pilanci, 2019; Pilanci & Ergen, 2020; Ergen & Pilanci, 2020a;b).
Based on the convex duality framework, the training problem of two-layer neural networks with ReLU activation can be represented in terms of a single convex program in Pilanci & Ergen (2020).
Such convex optimization formulations are extended to two-layer and three-layer convolutional neural network training problems in Ergen & Pilanci (2021b).
Strong duality also holds for deep linear neural networks with scalar output (Ergen & Pilanci, 2021a).
The convex optimization formulation essentially gives a detailed characterization of the global optimum of the training problem.
This enables us to examine in numerical experiments whether popular optimizers for neural networks, such as gradient descent or stochastic gradient descent, converge to the global optimum of the training loss.
Admittedly, a zero duality gap is hard to achieve for deep neural networks, especially for those with vector outputs.
This imposes more difﬁculty to understand deep neural networks from the convex optimization lens.
Fortunately, neural networks with parallel structures (also known as multi-branch architecture) appear to be easier to train.
Practically, the usage of parallel neural networks dates back to AlexNet (Krizhevsky et al., 2012).
Modern neural network architecture including Inception (Szegedy et al., 2017), Xception (Chollet, 2017) and SqueezeNet (Iandola et al., 2016) utilize the parallel structure.
As the “parallel” version of ResNet (He et al., 2016a;b), ResNeXt (Xie et al., 2017) and Wide ResNet (Zagoruyko & Komodakis, 2016) exhibit improved performance on many applications.
Recently, it was shown that neural networks with parallel architectures have smaller duality gaps (Zhang et al., 2019) compared to standard neural networks.
Furthermore, Ergen &
Pilanci (2021c;e) proved that there is no duality gap for parallel architectures with three-layers.
On the other hand, it is known that overparameterized parallel neural networks have benign training landscapes (Haeffele & Vidal, 2017; Ergen & Pilanci, 2019).
The parallel models with the over-parameterization are essentially neural networks in the mean-ﬁeld regime (Nitanda & Suzuki, 2017; Mei et al., 2018; Chizat & Bach, 2018; Mei et al., 2019;
Rotskoff et al., 2019; Sirignano &
Spiliopoulos, 2020; Akiyama & Suzuki, 2021; Chizat, 2021; Nitanda et al., 2020).
The deep linear model is also of great interests in the machine learning community.
For training (cid:96) 2 loss with deep linear networks using Schatten norm regularization, Zhang et al.
(2019) show that there is no duality gap.
The implicit regularization in training deep linear networks has been studied in Ji & Telgarsky (2018); Arora et al.
Moroshko et al.
From another perspective, the standard two-layer network is equivalent to the parallel two-layer network.
This may also explain why there is no duality gap for two-layer neural networks.
1. C ONTRIBUTIONS
Following the convex duality framework introduced in Ergen & Pilanci (2021a; 2020a), which showed the duality gap is zero for two-layer networks, we go beyond two-layer and study the convex duality for vector-output deep neural networks with linear activation and ReLU activation.
Surprisingly, we prove that three-layer networks may have duality gaps depending on their architecture, unlike two-layer neural networks which always have zero duality gap .
We summarize our contributions as follows
• For training standard vector-output deep linear networks using (cid:96) 2 regularization, we precisely calculate the optimal value of the primal and dual problems and show that the duality gap is non-zero , i.e., Lagrangian relaxation is inexact.
We also demonstrate that the (cid:96) 2regularization on the parameter explicitly forces a tendency toward a low-rank solution, which is boosted with the depth.
However, we show that the optimal solution is available in closed-form .
• For parallel deep linear networks, with certain convex regularization, we show that the duality gap is zero , i.e, Lagrangian relaxation is exact.
• For parallel deep ReLU networks of arbitrary depth, with certain convex regularization and sufﬁcient number of branches, we prove strong duality, i.e., show that the duality gap is zero .
Remarkably, this guarantees that there is a convex program equivalent to the original deep ReLU neural network problem.
We summarize the duality gaps for parallel/standard neural network in Table 1.
1. N OTATIONS
We use bold capital letters to represent matrices and bold lowercase letters to represent vectors.
Denote [ n ] = { 1 , . . . , n } .
For a matrix W l ∈ R m l − 1 × m l , for i ∈ [ m l − 1 ] and j ∈ [ m l ] , we denote w col l,i as its i -th column and w row l,j as its j -th row.
Throughout the paper, X ∈ R N × d is the data matrix consisting linear activation
ReLU activation
L = L =
L >
L = L =
L >standard networks previous work (cid:51) ( 0 ) (cid:55) (cid:55) (cid:51) ( 0 ) (cid:55) (cid:55) this paper (cid:51) ( 0 ) (cid:51) ( (cid:54) = 0 ) (cid:51) ( (cid:54) = 0 ) (cid:51) ( 0 ) (cid:55) (cid:55) parallel networks previous work (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:55) this paper (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) (cid:51) ( 0 ) we compare our duality gap characterization with previous literature.
check of d dimensional N samples and
Y ∈
R N × K is the label matrix for a regression/classiﬁcation task with K outputs.
We use the letter P ( D ) for the optimal value of the primal (dual) problem.
1. M OTIVATIONS AND B ACKGROUND
Recently a series of papers (Pilanci & Ergen, 2020; Ergen & Pilanci, 2021a; 2020a) studied two-layer neural networks via convex duality and proved that strong duality holds for these architectures.
Particularly, these prior works consider the following weight decay regularized training framework for classiﬁcation/regression tasks.
Given a data matrix X ∈ R N × d consisting of d dimensional N samples and the corresponding label matrix y ∈ R N , the weight-decay regularized training problem for a scalar-output neural network with m hidden neurons can be written as follows
[Equation]
where W 1 ∈ R d × m and w 2 ∈ R m are the layer weights, β > 0 is a regularization parameter, and φ is the activation function, which can be linear φ ( z )
=
z or ReLU φ ( z ) =
max { z, 0 } .
Then, one can take the dual of (1) with respect to W 1 and w 2 obtain the following dual optimization problem
[Equation]
We ﬁrst note that since the training problem (1) is non-convex, strong duality may not hold, i.e.,
P ≥ D .
Surprisingly, as shown in Pilanci & Ergen (2020); Ergen & Pilanci (2021a; 2020a), strong duality in fact holds, i.e., P = D , for two-layer networks and therefore one can derive exact convex representations for the non-convex training problem in (1).
However, extensions of this approach to deeper and state-of-the-art architectures are not available in the literature.
Based on this observation, the central question we address in this paper is:
Does strong duality hold for deep neural networks?
Depending on the answer to the question above, an immediate next questions we address is
Can we characterize the duality gap (P-D)?
Is there an architecture for which strong duality holds regardless of the depth?
Consequently, throughout the paper, we provide a full characterization of convex duality for deeper neural networks.
We observe that the dual of the convex dual problem of the nonconvex minimum norm problem of deep networks correspond to a minimum norm problem of deep networks with parallel branches.
Based on this characterization, we propose a modiﬁed architecture for which strong duality holds regardless of depth.
1. O RGANIZATION
This paper is organized as follows.
In Section 2, we review standard neural networks and introduce parallel architectures.
For deep linear networks, we derive primal and dual problems for both standard and parallel architectures and provide calculations of optimal values of these problems in Section 3.
We derive primal and dual problems for three-layer ReLU networks with standard architecture and precisely calculate the optimal values for whitened data in Section 4.
We also show that deep ReLU networks with parallel structures have no duality gap.
S TANDARD NEURAL NETWORKS VS PARALLEL ARCHITECTURES
We brieﬂy review the convex duality theory for two-layer neural networks in Appendix A. To extend the theory to deep neural networks, we ﬁst consider the L -layer neural network with the standard architecture:
[Equation]
where φ is the activation function, W l ∈ R m l − 1 × m l is the weight matrix in the l -th layer and θ =
( W 1 , . . .
, W L ) represents the parameter of the neural network.
We then introduce the neural network with parallel architectures:
[Equation]
Here for l ∈
[ L − 1] , the l -th layer has m weight matrices W l,j ∈ R m l − 1 × m l where j ∈ [ m ] .
Speciﬁcally, we let m L − 1 = 1 to make each parallel branch as a scalar-output neural network.
In short, we can view the output A L − 1 from a parallel neural network as a concatenation of m scalar-output standard neural work.
In Figures 1 and 2, we provide examples of neural networks with standard and parallel architectures.
We shall emphasize that for L = 2 , the standard neural network is identical to the parallel neural network.
We next present a summary of our main result.
Theorem 1 (main result)
For L ≥ 3 , there exists an activation function φ and a L -layer standard neural network deﬁned in (3) such that the strong duality does not hold, i.e., P > D .
In contrast, for any L -layer parallel neural network deﬁned in (4) with linear or ReLU activations and sufﬁciently large number of branches, strong duality holds, i.e., P = D .
We elaborate on the primal problem with optimal value P and the dual problem with optimal value
D in Section 3 and 4.
D EEP LINEAR NETWORKS 3. S TANDARD DEEP LINEAR NETWORKS
[Figure]
[Figure]
We ﬁrst consider the neural network with standard architecture, i.e., f θ ( X ) =
XW 1 . . .
W L .
Consider the following minimum norm optimization problem:
[Equation]
where the variables are W 1 , .
, W L .
As shown in the
Proposition 3.1 in (Ergen & Pilanci, 2021a), by introducing a scale parameter t , the problem (5) can be reformulated as
[Equation]
where the subproblem P lin ( t ) is deﬁned as
P
[Equation]
To be speciﬁc, these two formulations have the same optimal value and the optimal solutions of one problem can be rescaled into the optimal solution of another solution.
Based on the rescaling of parameters in P lin ( t ) , we characterize the dual problem of P lin ( t ) and its bi-dual, i.e., dual of the dual problem.
Proposition
The dual problem of P lin ( t ) is a convex optimization problem given by
[Equation]
There exists a threshold of the number of branches m ∗ ≤ KN + 1 such that D lin ( t ) =
BD lin ( t ) , where BD lin ( t ) is the optimal value of the bi-dual problem
[Equation]
Detailed derivation of the dual and the bi-dual problems are provided in Appendix C.1.
As Λ = 0 is a strict feasible point for the dual problem, the optimal dual solutions exist due to classical results in strong duality for convex problems.
The reason why we do not directly take the dual of P lin is that the objective function in P lin involves the weights of ﬁrst L − 1 layer, which prevents obtaining a non-trivial dual problem.
An interesting observation is that the bi-dual problem is related to the minimum norm problem of a parallel neural network with balanced weights.
Namely, the Frobenius norm of the weight matrices { W l,j } L − 2 l =1 in each branch j ∈ [ m ] has the same upper bound t .
To calculate the value P lin ( t ) for ﬁxed t ∈ R , we introduce the deﬁnition of Schatten- p norm.
Deﬁnition 1 For a matrix A ∈ R m × n and p > 0 , the Schatten- p quasi-norm of A is deﬁned as
[Equation]
where σ i ( A ) is the i -th largest singular value of A .
The following proposition provides a closed-form solution for the sub-problem P lin ( t ) and determines its optimal value.
Proposition 2 Suppose that W ∈ R d × K with rank r is given.
Assume that m l ≥ r for l = 1 , . . .
, L − 1 .
Consider the following optimization problem:
[Equation]
Then, the optimal value of the problem (7) is given by L 2 (cid:107) W (cid:107) 2 /LS 2 /L .
Suppose that W = UΣV T .
The optimal value is achieved when
[Equation]
Here U
=
U , U L = V and for l = 1 , . . .
, L − 1 , U l ∈ R m l × r satisﬁes that U Tl U
l = I r .
To the best of our knowledge, this result was not known previously.
Proposition 2 implies that P lin can be equivalently written as
[Equation]
Denote X † as the pseudo inverse of X .
Although the objective is non-convex for L ≥ 3 , this problem has a closed-form solution as we show next.
Theorem 2 Suppose that X † Y =
UΣV T is the singular value decomposition and let r := rank ( X † Y ) .
Assume that m l ≥ r for l = 1 , . . .
, L − 1 .
The optimal solution to P lin is given in closed-form as follows:
[Equation]
where U
=
U , U L = V .
For l = 1 , . . .
, L − 1 , U l ∈ R m l × r satisﬁes U Tl U
l = I r .
Based on Theorem 2, the optimal value of P lin ( t ) and D lin ( t ) can be precisely calculated as follows.
Theorem 3 Assume that m l ≥ rank ( X † Y ) for l = 1 , . . .
, L − 1 .
For ﬁxed t > 0 , the optimal value of P lin ( t ) and D lin ( t ) are given by
[Equation]
and
[Equation]
Here (cid:107) · (cid:107) ∗ represents the nuclear norm.
P lin ( t ) =
D lin ( t ) if and only if the singular values of X † Y are equal.
As a result, if the singular values of X † Y are not equal to the same value, the duality gap exists, i.e.,
P > D , for standard deep linear networks with L ≥ 3 .
We note that the optimal scale parameter t for the primal problem P lin is given by t ∗
=
(cid:107) W ∗ (cid:107) 1 /LS 2 /L .
This proves the ﬁrst part of Theorem 1.
We conclude that, the deep linear network training problem has a duality gap whenever the depth is three or more.
In contrast, there exists no duality gap for depth two.
Nevertheless, the optimal solution can be obtained in closed form as we have shown.
In the following section, we introduce a parallel multi-branch architecture that always has zero duality gap regardless of the depth.
3. P ARALLEL DEEP LINEAR NEURAL NETWORKS
Now we consider the parallel multi-branch network structure as deﬁned in Section
[Equation]
and consider the corresponding minimum norm optimization problem:
[Equation]
Due to a rescaling to achieve the lower bound of the inequality of arithmetic and geometric means
[Equation]
we can formulate the problem (12) in the following way.
In other words, two formulations (12) and (13) have the same optimal value and the optimal solutions of one problem can be mapped to the optimal solutions of another problem.
Proposition
The problem (12) can be formulated as
[Equation]
We note that z 2 /L is a non-convex function of z and we cannot hope to obtain a non-trivial dual.
To solve this issue, we consider the (cid:107) · (cid:107) LF regularized objective given by
[Equation]
Utilizing the arithmetic and geometric mean (AM-GM) inequality, we can rescale the parameters and formulate (14).
To be speciﬁc, the two formulations (14) and (15) have the same optimal value and the optimal solutions of one problem can be rescaled to the optimal solutions of another problem and vice versa.
Proposition 4 The problem (14) can be formulated as
[Equation]
The dual problem of P prllin is a convex problem
[Equation]
In contrary to the standard linear network model, the strong duality holds for the parallel linear network training problem (14).
Theorem 4 There exists a critical width m ∗ ≤ KN + 1 such that as long as the number of branches m ≥ m ∗ , the strong duality holds for the problem (14) .
Namely, P prllin =
D p rl lin .
The optimal values are both L 2 (cid:107) X † Y (cid:107) ∗ .
This implies that there exist equivalent convex problems which achieve the global optimum of the deep parallel linear network.
Comparatively, optimizing deep parallel linear neural networks can be much easier than optimizing deep standard linear networks.
N EURAL NETWORKS WITH R E LU ACTIVATION 4. S TANDARD THREE - LAYER R E LU NETWORKS
We ﬁrst focus on the three-layer ReLU network with standard architecture.
Speciﬁcally, we set φ ( z ) =
max { z, 0 } .
Consider the minimum norm problem
[Equation]
Here we denote ( z ) + = max { z, 0 } .
Similarly, by introducing a scale parameter t , this problem can be formulated as P ReLU =
min t> 0 12 t 2 + P ReLU ( t ) , where P ReLU ( t ) is deﬁned as
[Equation]
The proof is analagous to the proof of Proposition 3.1 in (Ergen & Pilanci, 2021a).
To be speciﬁc, these two formulations have the same optimal value and their optimal solutions can be mutually transformed into each other.
For W 1 ∈ R d × m , we deﬁne the set
[Equation]
We derive the convex dual problem of P ReLU ( t ) in the following proposition.
Proposition
The dual problem of P ReLU ( t ) deﬁned in (18) is a convex problem deﬁned as
[Equation]
There exists a threshold of the number of branches m ∗ ≤ KN +1 such that D ReLU ( t ) =
BD ReLU ( t ) where BD ReLU ( t ) is the optimal value of the bi-dual problem
[Equation]
We note that the bi-dual problem deﬁned in (21) indeed optimizes with a parallel neural network satisfying (cid:107) W 1 ,j (cid:107) F ≤ t, (cid:107) w col2 ,j (cid:107) 2 ≤ 1 , j ∈ [ m ∗ ] .
For the case where the data matrix is with rankand the neural network is with scalar output, we show that there is no duality gap.
We extend the result in (Ergen & Pilanci, 2021d) from two-layer ReLU networks to three-layer ReLU networks.
Theorem 5 For a three-layer scalar-output ReLU network, let X
= ca T 0 be a rank-one data matrix.
Then, strong duality holds, i.e., P ReLU ( t ) =
D ReLU ( t ) .
Suppose that λ ∗ is the optimal solution to the dual problem D ReLU ( t ) , then the optimal weights for each layer can be formulated as
[Equation]
Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and
[Equation]
For general standard three-layer neural networks, although we have BD ReLU ( t )
=
D ReLU ( t ) , it may not hold that P ReLU ( t )
=
D ReLU ( t ) as the bi-dual problem corresponds to optimizing a parallel neural network instead of a standard neural network to ﬁt the labels.
To theoretically justify that the duality gap can be zero, we consider a parallel multi-branch architecture for ReLU networks in the next section.
4. P ARALLEL DEEP R E LU NETWORKS
For the corresponding parallel architecture, we show that there is no duality gap for arbitrary depth
ReLU networks, as long as the number of branches is large enough.
Consider the following minimum norm problem:
[Equation]
As the ReLU activation is homogeneous, we can rescale the parameter to reformulate (22) and derive the dual problem.
We note that two formulations (22) and (23) have the same optimal value and the optimal solutions of one problem can be rescaled to the optimal solutions of another problem and vice versa.
Proposition 6 The problem (22) can be reformulated as
[Equation]
The dual problem of (23) is a convex problem deﬁned as
[Equation]
For deep parallel ReLU networks, we show that with sufﬁcient number of parallel branches, the strong duality holds, i.e., P = D .
Theorem 6 Let m ∗ be the threshold of the number of branches, which is upper bounded by KN + 1 .
Then, as long as the number of branches m ≥ m ∗ , the strong duality holds for (23) in the sense that
P
[Equation]
Similar to case of parallel deep linear networks, the parallel deep ReLU network also achieves zero-duality gap.
Therefore, to ﬁnd the global optimum for parallel deep ReLU network is equivalent to solve a convex program.
This proves the second part of Theorem 1.
Based on the strong duality results, assuming that we can obtain an optimal solution to the convex dual problem (24), then we can construct an optimal solution to the primal problem (23) as follows.
Theorem 7 Let Λ ∗ be the optimal solution to (24) .
Denote the set of maximizers
[Equation]
as { v 1 , . . .
, v m ∗ } , where v i =
(( XW 1 ,i )
+ . . .
W L − 2 ,i ) + w L − 1 ,i ) + with (cid:107) W l,i (cid:107) F ≤ 1 , l ∈
[ L − 2] and (cid:107) w L − 1 ,i (cid:107) 2 ≤ 1 and m ∗ ≤ KN + 1 is the critical threshold of the number of branches.
Let w row L, 1 , . . .
, w row L,m ∗ be an optimal solution to the convex problem
[Equation]
Then, ( W 1 , . . .
, W L ) is an optimal solution to (23) .
We note that ﬁnding the set of maximizers in (25) can be challenging in practice due to the highdimensionality of the constraint set.
C ONCLUSION
We present the convex duality framework for standard neural networks, considering both multi-layer linear networks and three-layer ReLU networks with rank-1.
In stark contrast to the two-layer case, the duality gap can be non-zero for neural networks with depth three or more.
Meanwhile, for neural networks with parallel architecture, with the regularization of L -th power of Frobenius norm in the parameters, we show that strong duality holds and the duality gap reduces to zero.
A limitation of our work is that we primarily focus on minimum norm interpolation problems.
We believe that our results can be easily generalized to a regularized training problems with general loss function, including squared loss, logistic loss, hinge loss, etc..
Another interesting research direction is investigating the complexity of solving our convex dual problems.
Although the number of variables can be high for deep networks, the convex duality framework offers a rigorous theoretical perspective to the structure of optimal solutions.
These problems can also shed light into the optimization landscape of their equivalent non-convex formulations.
We note that it is not yet clear whether convex formulations of deep networks present practical gains in training.
However, in Mishkin et al.
; Pilanci & Ergen (2020)
it was shown that convex formulations provide signiﬁcant computational speed-ups in training two-layer neural networks.
Furthermore, similar convex analyses was also applied various architectures including batch normalization Ergen et al. (2022b), vector output networks Sahiner et al. (2021), threshold/binary networks Ergen et al. (2023), GANs Sahiner et al.
(2022a), autoregressive models Gupta et al.
(2021), and Transformer
Ergen et al.
(2022a)
; Sahiner et al. (2022b). A CKNOWLEDGEMENTS This work was partially supported by the National Science Foundation (NSF) under grants ECCS2037304, DMS-2134248, NSF CAREER award CCF-2236829, the U.S. Army Research Ofﬁce Early
Career Award W911NF-21-1-0242, and the ACCESS – AI Chip Center for Emerging Smart Systems, sponsored by InnoHK funding, Hong Kong SAR.
A C ONVEX DUALITY FOR TWO - LAYER NEURAL NETWORKS
We brieﬂy review the convex duality theory for two-layer neural networks introduced in Ergen &
Pilanci (2021a; 2020a). Consider the following weight-decay regularized training problem for a vector-output neural network architecture with m hidden neurons
[Equation]
where W 1 ∈ R d × m and W 2 ∈ R m × K are the variables, and β > 0 is a regularization parameter.
Here φ is the activation function, which can be linear φ ( z ) =
z or ReLU φ ( z ) =
max { z, 0 } .
As long as the network is sufﬁciently overparameterized, there exists a feasible point for such that φ ( XW 1 ) W 2 = Y .
Then, a minimum norm variant 1 of the training problem in (27) is given by
[Equation] [Equation]
As shown in Pilanci & Ergen (2020), after a suitable rescaling, this problem can be reformulated as
[Equation]
(29) where [ m ] = { 1 , . . .
, m } .
Here w row2 ,j represents the j -th row of W 2 and w col1 ,j denotes the j -th column of W 1 .
The rescaling does not change the solution to (28).
By taking the dual with respect to W 1 and W 2 , the dual problem of (29) with respect to variables is a convex optimization problem given by
[Equation]
where Λ ∈ R N × K is the dual variable.
Provided that m ≥ m ∗ , where m ∗ is a critical threshold of width upper bounded by m ∗ ≤ N + 1 , the strong duality holds, i.e., the optimal value of the primal problem (29) equals to the optimal value of the dual problem (30).
B D EEP LINEAR NETWORKS WITH GENERAL LOSS FUNCTIONS
We consider deep linear networks with general loss functions, i.e.,
[Equation]
where (cid:96) ( Z , Y ) is a general loss function and β > 0 is a regularization parameter.
According to
Proposition 2, the above problem is equivalent to
[Equation]
[Equation]
The (cid:96) 2 regularization term becomes the Schatten- 2 /L quasi-norm on W to the power 2 /L .
Suppose that there exists W such that l ( XW , Y ) = 0 .
With β → 0 , asymptotically, the optimal solution to the problem (31) converges to the optimal solution of
[Equation]
In other words, the (cid:96) 2 regularization explicitly regularizes the training problem to ﬁnd a low-rank solution W .
C P ROOFS OF MAIN RESULTS FOR LINEAR NETWORKS C. P ROOF OF P ROPOSITION
Consider the Lagrangian function
[Equation]
Here Λ ∈ R N × K is the dual variable.
We note that
[Equation]
Here I ( A ) is 0 if the statement A is true.
Otherwise it is + ∞ .
For ﬁxed W 1 , . . .
[Equation]
W L − 1 , the constraint on W L is linear so we can exchange the order of max Λ and min W L in the second line of (34).
By exchanging the order of min and max , we obtain the dual problem
[Equation]
Now we derive the bi-dual problem.
The dual problem can be reformulated as
[Equation]
Here the set Θ is deﬁned as
[Equation]
By writing θ =
( W 1 , . . .
, W L − 2 , w L − 1 ) , the dual of the problem (36) is given by
[Equation]
[Equation]
Here µ : Σ → R K is a signed vector measure and Σ is a σ -ﬁeld of subsets of Θ .
The norm (cid:107) µ (cid:107) TV is the total variation of µ , which can be calculated by
[Equation]
where we write µ =  
[Equation]
µ
[Equation]
 .
The formulation in (38) has inﬁnite width in each layer.
According to
Theorem 10 in Appendix G, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.
Therefore, we can rewrite the problem (38) as
[Equation]
Here the variables are W i,j for i ∈
[ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .
As the strong duality holds for the problem (40) and (36), we can reformulate the problem of D lin ( t ) as the bi-dual problem (40).
C. P ROOF OF P ROPOSITION
We restate Proposition 2 with details.
Proposition 7 Suppose that W ∈ R d × K with rank r is given.
Consider the following optimization problem:
[Equation]
in variables W i ∈
R m i − 1 × m i .
Here m
= d , m L = K and m i ≥ r for i = 1 , . . .
, L − 1 .
Then, the optimal value of the problem (41) is given by
[Equation]
[Equation]
Suppose that W = UΣV T .
The optimal value can be achieved when
[Equation]
Here U i ∈
R r × m i satisﬁes that U
Ti U
i
=
I .
We start with two lemmas.
Lemma 1 Suppose that A ∈ S n × n is a positive semi-deﬁnite matrix.
Then, for any 0 < p < 1 , we have
[Equation]
Here λ i is the i -th largest eigenvalue of A .
Lemma 2 Suppose that P ∈ R d × d is a projection matrix, i.e., P 2 = P .
Then, for arbitrary
W ∈ R d × K , we have
[Equation]
where σ i ( W ) represents the i -th largest singular value of W .
Now, we present the proof for Proposition 2.
For L = 1 , the statement apparently holds.
Suppose that for L = l this statement holds.
For L = l + 1 , by writing A = W 2 . . .
W l +1 , we have
[Equation]
Suppose that t is ﬁxed.
It is sufﬁcient to consider the following problem:
[Equation]
Suppose that there exists W 1 and A such that W = W 1 A .
Then, we have WA † A
= W 1 AA † A =
W .
As WA † = W 1 AA † , according to Lemma 2, (cid:107) WA † (cid:107) F ≤ (cid:107) W 1 (cid:107) F ≤ t .
Therefore, ( WA † , A ) is also feasible for the problem (46).
Hence, the problem (46) is equivalent to
[Equation]
Assume that W is with rank r .
Suppose that A = UΣV T , where Σ ∈ R r 0 × r 0 .
Here r 0 ≥ r .
Then, we have A † =
VΣ − 1 U T .
We note that
[Equation]
Denote G ( V ) =
V T W T WV .
This implies that r
[Equation]
i
[Equation]
G
[Equation]
Therefore, we have (cid:32)
[Equation]
i =
σ i
[Equation]
A )
[Equation]
G
[Equation]
V
[Equation]
ii
[Equation]
V
[Equation]
As WV T V = W , the non-zero eigenvalues of G ( V ) are exactly the non-zero eigenvalues of
WVV T W T = WW T , i.e., the square of non-zero singular values of W .
From Lemma 1, we have r 0 (cid:88)
[Equation]
Therefore, we have
[Equation]
This also implies that
[Equation]
Suppose that W = (cid:80) ri =1 u i σ i v Ti is the SVD of W .
We can let
A =
[Equation]
t
[Equation]
Here (cid:107) ρ i (cid:107) 2 = 1 and ρ
Ti ρ j =
for i (cid:54) =
j .
Then, W 1 A = W and (cid:107) W 1 (cid:107) F ≤ t .
We also have (cid:107)
[Equation]
=
[Equation]
[Equation]
l
[Equation]
=
[Equation]
1) i =
[Equation]
l
[Equation]
i =
[Equation]
[Equation]
[Equation]
l .
In summary, we have
[Equation]
This completes the proof.
C. P ROOF OF T HEOREM
From Proposition 2, the minimum norm problem (5) is equivalent to
[Equation]
in variable W ∈ R d × K .
According to Lemma 2, for any feasible W satisfying XW = Y , because
X † XW = X † Y and X † X is a projection matrix, we have
[Equation]
We also note that XX † Y =
XX † XW = XW = Y .
Therefore, X † Y is also feasible for the problem (54).
This indicates that P lin =
L 2 (cid:107) X † Y (cid:107)
/LS 2 /L .
C. P ROOF OF T HEOREM
For a feasible point ( W 1 , . . .
, W L ) for
P lin ( t ) , we note that
[Equation]
is feasible for
P lin (1) .
This implies that t L − 2 P lin ( t ) =
P lin (1) , or equivalently, P lin ( t ) =
t − ( L − 2)
P lin (1) .
Recall that
[Equation]
From Theorem 2, we have P lin =
L 2 (cid:107) X † Y (cid:107)
/LS 2 /L .
This implies that P lin (1) =
(cid:107) X † Y (cid:107) S 2 /L and
P lin ( t ) =
t − ( L − 2) (cid:107) X † Y (cid:107) S 2 /L .
For the dual problem D lin ( t ) deﬁned in (35), we note that
[Equation]
The equality can be achieved when W l = t u l u Tl +1 for l ∈
[ L − 2] , where (cid:107) u l (cid:107) 2 = 1 for l = 1 , . . .
, L − 1 .
Speciﬁcally, we set u L − 1 = w L − 1 and let u 0 as right singular vector corresponds to the largest singular value of Λ T X .
Therefore, the constraints on Λ is equivalent to
[Equation]
[Equation]
Thus, according to the Von Neumann’s trace inequality, it follows
[Equation]
Suppose that X † Y
=
UΣV T is the singular value decomposition.
Let Σ = diag ( σ 1 , . . .
, σ r ) where σ 1 ≥ σ 2 ≥ · · · ≥ σ r > 0 and r = rank ( X † Y ) .
We note that
[Equation]
The equality holds if and only if σ 1 = · · · = σ r .
This is because for given x ∈ (0 , 1) and a ≥ 1 , ( a + x p ) 1 /p is strictly decreasing w.r.t.
p ∈ (0 , 1] .
As a result, we have D lin ( t ) =
t − ( L − 2) (cid:107) X † Y (cid:107) ∗ ≤ t − ( L − 2) (cid:107) X † Y (cid:107) S
/L
=
P lin ( t ) .
The equality is achieved if and only if the singular values of X † Y are the same.
In other words, the inequality is strict when X † Y has different singular values.
Then, the duality gap exists for the standard neural network.
C. P ROOF OF P ROPOSITION
For simplicity, we write W L − 1 ,j = w col L − 1 ,j and W L,j = w row L,j for j ∈ [ m ] .
For the j -th branch of the parallel network, let ˆ W l,j = α l,j W l,j for l ∈
[ L ] .
Here α l,j > 0 for l ∈
[ L ] and they satisﬁes that (cid:81)
Ll =
α l,j = 1 for j ∈ [ m ] .
Therefore, we have
[Equation]
This implies that { ˆ W l,j } l ∈ [ L ] ,j ∈ [ m ] is also feasible for the problem (12).
According to the the inequality of arithmetic and geometric means, the objective function in (12) is lower bounded by
[Equation]
The equality is achieved when α l,j =
[Equation]
for l ∈ [ L ] and j ∈ [ m ] .
As the scaling operation does not change (cid:81) Ll =1 (cid:107) W l,j (cid:107) 2 /LF , we can simply let (cid:107) W l,j (cid:107) F = 1 and the lower bound becomes L 2 (cid:80) mi =1 (cid:107) W L,j (cid:107) 2 /LF
=
L 2 (cid:80) mi =1 (cid:107) w row L,j (cid:107) 2 /L.
This completes the proof.
C. P ROOF OF P ROPOSITION
We ﬁrst show that the problem (14) is equivalent to (15).
The proof is analogous to the proof of
Proposition 3.
For simplicity, we write W L − 1 ,j = w col L − 1 ,j and W L,j = w row L,j for j ∈ [ m ] .
Let α l,j > 0 for l ∈
[ L ] and they satisﬁes that (cid:81)
Ll =
α l,j = 1 for j ∈ [ m ] .
Consider another parallel network { ˆ W l,j
} l ∈ [ L ] ,j ∈ [ m ] whose j -th branch is deﬁned by ˆ W l,j = α l,j W l,j for l ∈
[ L ] .
As (cid:81)
Ll =1 α l,j = 1 , we have
[Equation]
This implies that { ˆ W l,j } l ∈ [ L ] ,j ∈ [ m ] is also feasible for the problem (14).
According to the the inequality of arithmetic and geometric means, the objective function in (12) is lower bounded by
[Equation]
The equality is achieved when α l,j =
[Equation]
for l ∈ [ L ] and j ∈ [ m ] .
As the scaling operation does not change (cid:81) Ll =1 (cid:107) W l,j (cid:107) F , we can simply let (cid:107) W l,j (cid:107) F = 1 and the lower bound becomes
[Equation]
w row L,j (cid:107) 2 .
Hence, the problem (14) is equivalent to (15).
For the problem (15)
[Equation]
we consider the Lagrangian function
[Equation]
The primal problem is equivalent to
[Equation]
The dual problem follows
[Equation]
C. P ROOF OF T HEOREM
We can rewrite the dual problem as
[Equation]
where the set Θ is deﬁned as
[Equation]
By writing θ =
( W 1 , . . .
, W L − 2 , w L − 1 ) , the bi-dual problem, i.e., the dual problem of (69), is given by
[Equation]
Here µ : Σ → R K is a signed vector measure, where Σ is a σ -ﬁeld of subsets of Θ and (cid:107) µ (cid:107) TV is its total variation.
The formulation in (71) has inﬁnite width in each layer.
According to Theorem 10 in
Appendix G, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.
Therefore, there exists a critical threshold of the number of branchs m ∗ < KN + 1 such that we can rewrite the problem (71) as
[Equation]
Here the variables are W l,j for l ∈
[ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .
This is equivalent to (15).
As the strong duality holds for the problem (69) and (71), the primal problem (15) is equivalent to the dual problem (69) as long as m ≥ m ∗ .
Now, we compute the optimal value of D prllin .
Similar to the proof of Theorem 3, we can show that the constraints in the dual problem (69) is equivalent to
[Equation]
Therefore, we have
[Equation]
This implies that
[Equation]
D S TAIRS OF DUALITY GAP FOR STANDARD DEEP LINEAR NETWORKS
We consider partially dualizing the non-convex optimization problem by exchanging a subset of the minimization problems with respect to the hidden layers.
Consider the Lagrangian for the primal problem of standard deep linear network
[Equation]
By changing the order of L − 2 min s and the max in (75), for l = 0 , 1 , . . .
, L − 2 , we can deﬁne the l -th partial “dual” problem
D
[Equation]
For l = 0 , D ( l ) lin ( t ) corresponds the primal problem P lin ( t ) , while for l = L − 2 , D ( l ) lin ( t ) is the dual problem D lin ( t ) .
From the following proposition, we illustrate that the dual problem of D ( l ) lin ( t ) corresponds to a minimum norm problem of a neural network with parallel structure.
Proposition 8 There exists a threshold of the number of branches m ∗ ≤ KN + 1 such that the problem D ( l ) lin ( t ) is equivalent to the “bi-dual” problem
[Equation]
where the variables are
W i ∈
R m i − 1 ×
m i for i ∈
[ l ] , W i,j ∈ R m i − 1 × m i for i =
l + 1 , . . .
, L − 2 , j ∈ [ m ∗ ] , W L − 1 ∈ R m L − 2 × m ∗ and W L ∈ R m ∗ × m L .
We can interpret the problem (77) as the minimum norm problem of a linear network with parallel structures in ( l + 1) -th to ( L − 2) -th layers.
This indicates that for l = 0 , 1 , . . .
, L − 2 , the bi-dual formulation of D ( l ) lin ( t ) can be viewed as an interpolation from a network with standard structure to a network with parallel structure.
Now, we calculate the exact value of D ( l ) lin ( t ) .
Proposition 9 The optimal value D ( l ) lin ( t ) follows
[Equation]
Suppose that the eigenvalues X † Y are not identical to each other.
Then, we have
[Equation]
In Figure
[Equation]
we plot D ( l ) lin ( t ) for l = 0 , . . .
, 5 for an example.
D. P ROOF OF P ROPOSITION
We note that
[Equation]
Therefore
[Equation]
we can rewrite D ( l ) lin ( t ) as
[Equation]
From the equation (10), we note that
[Equation]
This completes the proof
[Equation]
E P ROOFS OF MAIN RESULTS FOR R E LU NETWORKS E. P ROOF OF P ROPOSITION
For the problem of P ( t ) , introduce the Lagrangian function
[Equation]
According to the convex duality of two-layer ReLU network
[Equation]
we have
[Equation]
By changing the min and max
[Equation]
we obtain the dual problem.
[Equation]
The dual of the dual problem writes
[Equation]
Here µ is a signed vector measure and (cid:107) µ (cid:107) TV is its total variation.
Similar to the proof of Proposition 1, we can ﬁnd a ﬁnite representation for the optimal measure and transform this problem to
[Equation] [Equation]
Here m ∗ ≤ KN + 1 .
This completes the proof.
P ROOF OF T HEOREM
For rank-1 data matrix that X
= ca T 0 , suppose that A 1 = ( XW 1 ) + .
It is easy to observe that
[Equation]
1 , − ,
Here we let a 1 , + = ( W T 1 a 0 )
+ and a 1 , − = ( − W T 1 a 0 )
+ .
For a three-layer network, suppose that λ ∗ is the optimal solution to the dual problem D ReLU ( t ) .
We consider the extreme points deﬁned by
[Equation]
For ﬁxed W 1 , because a T 1 , + a 1 , − = 0 , suppose that
[Equation]
where r T a 1 , + = r T a 1 , − =
and (cid:107) r (cid:107) 2 = 1 .
The maximization problem on w 2 reduces to
[Equation]
If ( λ ∗ ) T ( c ) + and ( λ ∗ ) T ( − c ) + have different signs, then the optimal value is
[Equation]
[Equation]
And the corresponding optimal w 2 is w 2 = a 1 , + / (cid:107) a 1 , + (cid:107) 2 or w 2 = a 1 , − / (cid:107) a 1 , − (cid:107) 2 .
Then, the problem becomes
[Equation]
We note that
[Equation]
Thus the optimal W 1 is given by
[Equation]
Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and ρ 1 ∈ R m l + satisﬁes (cid:107) ρ 1 (cid:107) = 1 .
This implies that the optimal w 2 is given by w 2 = ρ 1 .
On the other hand
[Equation]
if ( λ ∗ ) T ( c ) + and ( λ ∗ ) T ( − c ) + have same signs, then, the optimal w 2 follows
[Equation]
The maximization problem of W 1 is equivalent to
[Equation]
By noting that
[Equation]
the optimal W 1 is given by
[Equation]
Here ρ 0 = a 0 / (cid:107) a 0 (cid:107) 2 and ρ 1 ∈ R m 1 + satisﬁes (cid:107) ρ 1 (cid:107) = 1 .
E. P ROOF OF P ROPOSITION
Analogous to the proof of Proposition 4, we can reformulate (22) into (23).
The rest of the proof is analogous to the proof of Proposition 4.
For the problem (23), we consider the Lagrangian function
[Equation]
The primal problem is equivalent to
[Equation]
By exchanging the order of min and max
[Equation]
the dual problem follows
[Equation]
E. P ROOF OF T HEOREM
The proof is analogous to the proof of Theorem 4.
We can rewrite the dual problem as
[Equation]
where the set Θ is deﬁned as
[Equation]
By writing θ =
( W 1 , . . .
, W L − 2 , w L − 1 ) , the bi-dual problem,
i
[Equation]
e., the dual problem of (92), is given by
[Equation]
Here µ : Σ → R K is a signed vector measure, where Σ is a σ -ﬁeld of subsets of Θ and (cid:107) µ (cid:107) TV is its total variation.
The formulation in (94) has inﬁnite width in each layer.
According to Theorem 10 in
Appendix G, the measure µ in the integral can be represented by ﬁnitely many Dirac delta functions.
Therefore, there exists m ∗ ≤ KN + 1 such that we can rewrite the problem (94) as min
[Equation]
Here the variables are W l,j for l ∈
[ L − 2] and j ∈ [ m ∗ ] , W L − 1 and W L .
This is equivalent to (23).
As the strong duality holds for the problem (92) and (94), the primal problem (23) is equivalent to the dual problem (92) as long as m ≥ m ∗ .
E. P ROOF OF T HEOREM
Consider the following dual problem
[Equation]
Apparently we have D prl , sub ReLU ≤ D prlReLU .
As Λ ∗ is the optimal solution to D prlReLU and Λ ∗ is feasible to D prl , sub ReLU , we have D prl , sub ReLU ≥ D prlReLU .
This implies that D prl , sub ReLU =
D prlReLU .
We note that (26) is the dual problem of (96).
Therefore, as a corollary of Theorem 6, we have
[Equation]
Therefore, ( W 1 , . . .
, W L ) is the optimal solution to (23).
F P ROOFS OF AUXILIARY RESULTS F. P ROOF OF L EMMA
Denote a ∈ R n such that a i =
A ii and denote b ∈ R n such that b i =
λ
i ( A ) .
We can show that a is majorized by b , i.e., for k ∈ [ n − 1] , we have k (cid:88) i = a ( i ) ≤ k (cid:88)
i =
b ( i )
[Equation]
(97) and (cid:80) ni
=1 a i = (cid:80) ni
=1 b i .
Here a ( i ) is the i -th largest entry in a .
We ﬁrst note that n (cid:88) i =
A ii = tr( A ) = n (cid:88) i =
λ
i ( A ) .
On the other hand, for k ∈ [ n − 1] , we have
[Equation]
(cid:88)
[Equation]
Therefore, a is majorized by b .
As f ( x ) = − x p is a convex function, according to the Karamata’s inequality, we have
[Equation]
This completes the proof.
F. P ROOF OF L EMMA
According to the min-max principle for singular value, we have
[Equation]
As P is a projection matrix, for arbitrary x ∈ R d , we have (cid:107) PWx (cid:107) 2 ≤ (cid:107)
Wx (cid:107) 2 .
Therefore, we have
[Equation]
This completes the proof.
G C ARATHEODORY ’ S THEOREM AND FINITE REPRESENTATION
We ﬁrst review a generalized version of Caratheodory’s theorem introduced in (Rosset et al., 2007).
Theorem 8 Let µ be a positive measure supported on a bounded subset D ⊆ R N .
Then, there exists a measure ν whose support is a ﬁnite subset of D , { z 1 , . .
z k } , with k ≤ N + 1 such that (cid:90)
D zdµ
[Equation]
(cid:88)
[Equation]
=
[Equation] [Equation]
and (cid:107) µ (cid:107) TV =
(cid:107) ν (cid:107) TV .
We can generalize this theorem to signed vector measures.
Theorem 9 Let µ : Σ → R K be a signed vector measure supported on a bounded subset D ⊆ R N .
Here Σ is a σ -ﬁeld of subsets of D .
Then, there exists a measure ν whose support is a ﬁnite subset of
D , { z 1 , . . .
, z k } , with k ≤ KN + 1 such that
[Equation]
and (cid:107) ν (cid:107) TV = (cid:107) µ (cid:107) TV .
P ROOF Let µ be a signed vector measure supported on a bounded subset D ⊆ R N .
Consider the extended set ˜ D
= { zu T | z ∈ D, u ∈ R K , (cid:107) u (cid:107) 2 = 1 } .
Then, µ corresponds to a scalar-valued measure ˜ µ on the set ˜ D and (cid:107) µ (cid:107) TV =
(cid:107) ˜
µ
(cid:107) TV .
We note that ˜ D is also bounded.
Therefore, by applying Theorem 8 to the set ˜ D and the measure ˜ µ , there exists a measure ˜ ν whose support is a ﬁnite subset of ˜ D , { z 1 u T 1 , . .
z k u Tk } , with k ≤ KN + 1 such that (cid:90)
˜ D Zd ˜ µ ( Z ) =
k (cid:88)
i =
z i u Ti d ˜ ν
( z i u Ti ) , (101) and (cid:107) ˜ µ (cid:107) TV = (cid:107) ˜
ν (cid:107) TV .
We can deﬁne ν as the signed vector measure whose support is a ﬁnite subset { z 1 , . . .
, z k } and dν ( z i )
=
u i d ˜
( z i u i ) .
Then, (cid:107) ν (cid:107) TV = (cid:107) ˜
ν (cid:107) TV = (cid:107) ˜ µ (cid:107) TV
= (cid:107) µ (cid:107) TV .
This completes the proof.
Now we are ready to present the theorem about the ﬁnite representation of a signed-vector measure.
Theorem 10 Suppose that θ is the parameter with a bounded domain Θ ⊆ R p and φ ( X , θ ) : R N × d × Θ
→
R
N is an embedding of the parameter into the feature space.
Consider the following optimization problem
[Equation]
Assume that an optimal solution to (102) exists.
Then, there exists an optimal solution ˆ µ supported on at most KN + 1 features in Θ .
P ROOF Let ˆ µ be an optimal solution to (102).
We can deﬁne a measure ˆ P on R N as the push-forward of ˆ µ by ˆ P ( B ) =
ˆ µ ( { θ | φ ( X, θ ) ∈ B } ) .
Denote D = { φ ( X, θ ) | θ ∈ Θ } .
We note that ˆ P is supported on D and D is bounded.
By applying Theorem 9 to the set D and the measure ˆ P , we can ﬁnd a measure Q whose support is a ﬁnite subset of D , { z 1 , . . .
, z k } with k ≤ KN + 1 .
For each z i ∈ D , we can ﬁnd
θ i such that φ ( X, θ i ) =
z i .
Then, ˜ µ = (cid:80) ki
=1 δ ( θ − θ i ) dQ
( z i ) is an optimal solution to (102) with at most KN + 1 features and (cid:107)
˜ µ (cid:107) TV = (cid:107) µ (cid:107) TV .
Here δ ( · ) is the Dirac delta measure.