G LOBALLY O PTIMAL T RAINING OF N EURAL N ETWORKS WITH T HRESHOLD A CTIVATION F UNCTIONS
A BSTRACT
Threshold activation functions are highly preferable in neural networks due to their efﬁciency in hardware implementations.
Moreover, their mode of operation is more interpretable and resembles that of biological neurons.
However, traditional gradient based algorithms such as Gradient Descent cannot be used to train the parameters of neural networks with threshold activations since the activation function has zero gradient except at a single non-differentiable point.
To this end, we study weight decay regularized training problems of deep neural networks with threshold activations.
We ﬁrst show that regularized deep threshold network training problems can be equivalently formulated as a standard convex optimization problem, which parallels the LASSO method, provided that the last hidden layer width exceeds a certain threshold.
We also derive an alternative simpliﬁed convex optimization formulation when the dataset can be shattered at a certain layer of the network.
We also prove that the shattering condition holds when a certain layer width exceeds O ( √ n/L ) , where n is the number of samples and L is the network depth.
We corroborate our theoretical results with various numerical experiments.
I NTRODUCTION
In the past decade, deep neural networks have proven remarkably useful in solving challenging problems and become popular in many applications.
The choice of activation plays a crucial role in their performance and practical implementation.
In particular, even though neural networks with popular activation functions such as ReLU are successfully employed, they require advanced computational resources in training and evaluation, e.g., Graphical Processing Units (GPUs) (Coates et al., 2013).
Consequently, training such deep networks is an extremely challenging especially on low power devices.
On the other hand, the threshold activation offers a multitude of advantages: (1) computational efﬁciency, (2) compression/quantization to binary latent dimension, (3) interpretability.
Unfortunately, gradient based optimization methods fail in optimizing threshold activation networks due to the fact that the gradient is zero almost everywhere.
To close this gap, we analyze the training problem of deep neural networks with the threshold activation function deﬁned as
[Equation]
where s ∈ R is a trainable amplitude parameter for the neuron.
Our main result is that globally optimal deep threshold networks can be trained by solving a convex optimization problem.
1. W HY SHOULD WE CARE ABOUT THRESHOLD NETWORKS ?
Neural networks with threshold activations are highly desirable due to the following reasons:
• Since the threshold activation (1) is restricted to take values in { 0 , s } , threshold neural network models are far more suitable for hardware implementations (Bartlett & Downs, 1992; Corwin et al., 1994).
Speciﬁcally, these networks have signiﬁcantly lower memory footprint, less computational complexity, and consume less energy (Helwegen et al., 2019). [Figure]
• Modern neural networks have extremely large number of full precision trainable parameters so that several computational barriers emerge during hardware implementations.
One approach to mitigate these issues is reducing the network size by grouping the parameters via a hash function (Hubara et al., 2017; Chen et al., 2015).
However, this still requires full precision training before the application of the hash function and thus fails to remedy the computational issues.
On the other hand, neural networks with threshold activations need a minimal amount of bits.
• Another approach to reduce the complexity is to quantize the weights and activations of the network (Hubara et al., 2017) and the threshold activation is inherently in a two level quantized form.
• The threshold activation is a valid model to simulate the behaviour of a biological neuron as detailed in Jain et al.
Therefore, progress in this research ﬁeld could shed light into the connection between biological and artiﬁcial neural networks.
1. R ELATED W ORK
Although threshold networks are essential for several practical applications as detailed in the previous section, training their parameters is a difﬁcult non-differentiable optimization problem due to the discrete nature in (1).
For training of deep neural networks with popular activations, the common practice is to use ﬁrst order gradient based algorithms such as Gradient Descent (GD) since the well known backpropagation algorithm efﬁciently calculates the gradient with respect to parameters.
However, the threshold activation in (1) has zero gradient except at a single non-differentiable point zero, and therefore, one cannot directly use gradient based algorithms to train the parameters of the network.
In order to remedy this issue numerous heuristic algorithms have been proposed in the literature as detailed below but they still fail to globally optimize the training objective (see Figure 1).
The Straight-Through Estimator (STE) is a widely used heuristics to train threshold networks (Bengio et al., 2013; Hinton, 2012).
Since the gradient is zero almost everywhere, Bengio et al.
Hinton (2012) proposed replacing the threshold activation with the identity function during only the backward pass.
Later on, this approach is extended to employ various forms of the ReLU activation function,
e.g., clipped ReLU, vanilla ReLU, Leaky ReLU (Yin et al., 2019b;
Cai et al., 2017; Xiao et al.)
, during the backward pass.
Additionally, clipped versions of the identity function were also used as an alternative to STE (Hubara et al., 2017;
Courbariaux et al., 2016; Rastegari et al., 2016).
1. C ONTRIBUTIONS
• We introduce polynomial-time trainable convex formulations of regularized deep threshold network training problems provided that a layer width exceeds a threshold detailed in Table 1.
In Theorem 2.2, we prove that the original non-convex training problem for two-layer networks is equivalent to standard convex optimization problems.
• We show that deep threshold network training problems are equivalent to standard convex optimization problems in Theorem 3.2.
In stark contrast to two-layer networks, deep threshold networks can have a richer set of hyperplane arrangements due to multiple nonlinear layers (see Lemma 3.5). [Table]
• In Section 3. we characterize the evolution of the set of hyperplane arrangements and consequently hidden layer representation space as a recursive process (see Figure 3) as the network gets deeper.
• We prove that when a certain layer width exceeds O ( √ n/L ) , the regularized L -layer threshold network training further simpliﬁes to a problem that can be solved in O ( n ) time.
Notation and preliminaries: We use lowercase and uppercase bold letters to denote vectors and matrices, respectively.
We also use [ n ] to denote the set { 1 , 2 , . . . , n } .
We denote the training data matrix consisting of d dimensional n samples as X ∈ R n × d , label vector as y ∈ R n , and the l th layer weights of a neural network as W ( l ) ∈
R m l − 1 × m l , where m 0 = d, m L = 1 .
T WO - LAYER THRESHOLD NETWORKS
We ﬁrst consider the following two-layer threshold network
[Equation]
where the set of the trainable parameters are W (1)
∈ R d × m , s ∈ R m , w (
2) ∈
R m and θ is a compact representation for the parameters, i.e., θ :=
{ W (1) , s , w (2) } .
Note that we include bias terms by concatenating a vector of ones to X .
Next, consider the weight decay regularized training objective
[Equation]
Now, we apply a scaling between variables s j and w (2) j to reach an equivalent optimization problem.
Lemma 2.1 (Optimal scaling) .
The training problem in (3) can be equivalently stated as
[Equation]
where Θ s :
= { θ : | s j | = 1 , ∀ j ∈ [ m ] } .
We next deﬁne the set of hyperplane arrangement patterns of the data matrix X as
[Equation]
We denote the distinct elements of the set H ( X ) by d 1 , . . .
, d P ∈ { 0 , 1 } n , where P := |H
( X ) | is the number of hyperplane arrangements.
Using this ﬁxed set of hyperplane arrangements { d i } Pi =1 , we next prove that (4) is equivalent to the standard Lasso method (Tibshirani, 1996).
Theorem 2.2.
Let m ≥ m ∗ , then the non-convex regularized training problem (4) is equivalent to
[Equation]
[Equation]
where D = [ dd. .
d P ] is a ﬁxed n × P matrix.
Here, m ∗ is the cardinality of the optimal solution, which satisﬁes m ∗ ≤ n + 1 .
Also, it holds that P noncvx2 =
P cvx2 .
Theorem 2.2 proves that the original non-convex training problem in (3) can be equivalently solved as a standard convex Lasso problem using the hyperplane arrangement patterns as features.
Surprisingly, the non-zero support of the convex optimizer in (6) matches to that of the optimal weight-decay regularized threshold activation neurons in the non-convex problem (3).
This brings us two major advantages over the standard non-convex training:
• Since (6) is a standard convex problem, it can be globally optimized without resorting to non-convex optimization heuristics, e g., initialization schemes, learning rate schedules etc.
• Since (6) is a convex Lasso problem
, there exists many efﬁcient solvers (Efron et al., 2004).
2. S IMPLIFIED CONVEX FORMULATION FOR COMPLETE ARRANGEMENTS
We now show that if the set of hyperplane arrangements of X is complete, i.e., H = { 0 , 1 } n contains all boolean sequences of length n , then the non-convex optimization problem in (3) can be simpliﬁed.
We call these instances complete arrangements .
In the case of two-layer threshold networks, complete arrangements emerge when the width of the network exceeds a threshold, speciﬁcally m ≥ n + 2 .
We note that the m ≥ n regime, also known as memorization, has been extensively studied in the recent literature (Bubeck et al., 2020; de Dios & Bruna, 2020; Pilanci & Ergen, 2020; Rosset et al., 2007).
Particularly, these studies showed that as long as the width exceeds the number of samples, there exists a neural network model that can exactly ﬁt an arbitrary dataset.
Vershynin (2020); Bartlett et al.
(2019) further improved the condition on the width by utilizing the expressive power of deeper networks and developed more sophisticated weight construction algorithms to ﬁt the data.
Theorem 2.3.
We assume that the set of hyperplane arrangements of X is complete, i.e., equal to the set of all length- n Boolean sequences” H = { 0 , 1 } n .
Suppose m ≥ n + 2 , then (4) is equivalent to
[Equation]
(7) and it holds that P noncvx
= P cvx v 2 .
Also, one can construct an optimal network with n + 2 neurons in time O ( n ) based on the optimal solution to the convex problem (7) .
Based on Theorem 2.3, when the data matrix can be shattered, i.e., all 2 n possible y ∈ { 0 , 1 } n labelings of the data points can be separated via a linear classiﬁer, it follows that the set of hyperplane arrangements is complete.
Consequently, the non-convex problem in (3) further simpliﬁes to (7).
2. T RAINING COMPLEXITY
We ﬁrst brieﬂy summarize our complexity results for solving (6) and (7), and then provide the details of the derivations below.
Our analysis reveals two interesting regimes:
• (incomplete arrangements) When n +1 ≥ m ≥ m ∗ , we can solve (6) in O ( n 3 r ) complexity,
where r := rank ( X ) .
Notice this is polynomial-time whenever the rank r is ﬁxed.
• (complete arrangements) When m ≥ n +2 , we can solve (7) in closed-form, and the reconstruction of the non-convex parameters W (1) and w (2) only O ( n ) time independent of d .
Computational complexity of (6) :
To solve the optimization problem in (6), we ﬁrst enumerate all possible hyperplane arrangements { d i } Pi =1 .
It is well known that given a rank- r data matrix the number of hyperplane arrangements P is upper bounded by (Stanley et al., 2004; Cover, 1965)
[Equation]
where r = rank ( X ) ≤ min( n, d ) .
Furthermore, these can be enumerated in O ( n r ) (Edelsbrunner et al., 1986).
Then, the complexity for solving (6) is O
( P 3 )
≈ O ( n 3 r )
(Efron et al., 2004).
Computational complexity of (7) :
The problem in (7) is the proximal operator of a polyhedral norm.
Since the problem is separable over the positive and negative parts of the parameter vector δ , the optimal solution can be obtained by applying two proximal steps (Parikh & Boyd, 2014).
As noted in
Theorem 2.3, the reconstruction of the non-convex parameters W (1) and w (2) requires O ( n ) time.
2. A GEOMETRIC INTERPRETATION
To provide a geometric interpretation, we consider the weakly regularized case where β → 0 .
In this case, (6) reduces to the following minimum norm interpolation problem
[Equation]
Proposition 2.4.
The minimum (cid:96) 1 norm interpolation problem in (9) can be equivalently stated as
[Equation]
where Conv( A ) denotes the convex hull of a set A .
This corresponds to the gauge function (see Rockafellar (2015)) of the hyperplane arrangement patterns and their negatives.
We provide visualizations of the convex set Conv {± d j , ∀ j ∈
[ P ] } in Figures 3 and 4 (see Appendix) for Example 3.1.
Proposition 2.4 implies that the non-convex threshold network training problem in (3) implicitly represents the label vector y as the convex combination of the hyperplane arrangements determined by the data matrix X .
Therefore, we explicitly characterize the representation space of threshold networks.
We also remark that this interpretation extends to arbitrary depth as shown in Theorem 3.2.
2. H OW TO OPTIMIZE THE HIDDEN LAYER WEIGHTS ?
After training via the proposed convex programs in (6) and (7), we need to reconstruct the layer weights of the non-convex model in (2).
We ﬁrst construct the optimal hyperplane arrangements d i for (7) as detailed in Appendix A.4.
Then, we have the following prediction model (2).
Notice changing w (1) j to any other vector w (cid:48)
j with same norm and such that 1 { Xw (1)
j
[Equation]
does not change the optimal training objective.
Therefore, there are multiple global optima and the one we choose might impact the generalization performance as discussed in Section 5.
D EEP THRESHOLD NETWORKS
We now analyze L -layer parallel deep threshold networks model with m L − 1 subnetworks deﬁned as
[Equation]
(10) where θ := {{ W ( l ) k , s ( l ) k } Ll =1 } m L − 1 k =1 , θ ∈ Θ:= { θ : W ( l ) k ∈ R m l − 1 × m l , s ( l ) k ∈ R m l , ∀ l, k } ,
X (
[Equation]
and the subscript k is the index for the subnetworks (see Ergen & Pilanci (2021a;b); Wang et al.
(2023) for more details about parallel networks).
We next show that the standard weight decay regularized training problem can be cast as an (cid:96) 1 -norm minimization problem as in Lemma 2.1.
Lemma 3.1.
The following L -layer regularized threshold network training problem
[Equation]
can be reformulated as
[Equation]
where Θ s := { θ ∈ Θ : | s ( L − 1) k | = 1 } .
C HARACTERIZING THE SET OF HYPERPLANE ARRANGEMENTS FOR DEEP NETWORKS
We ﬁrst deﬁne hyperplane arrangements for L -layer networks with a single subnetwork ( m L − 1 = so we drop the index k ). We denote the set of hyperplane arrangements as
[Equation]
We also denote the elements of the set H L ( X ) by d 1 , . . .
, d P L − 1 ∈ { 0 , 1 } n and P L − 1 = |H
L ( X ) | is the number of hyperplane arrangements in the layer L − 1 .
To construct the hyperplane arrangement matrix D ( l ) , we deﬁne a matrix valued operator as follows
[Equation]
Here, the operator A ( · ) outputs a matrix whose columns contain all possible hyperplane arrangements corresponding to its input matrix as in (5).
In particular, D (1) denotes the arrangements for the ﬁrst layer given the input matrix X .
The notation D ( l ) S ∈ { 0 , 1 } n × m l denotes the submatrix of
D ( l ) ∈ { 0 , 1 } n × P l indexed by the subset S of its columns, where the index S runs over all subsets of size m l .
Finally, (cid:116) is an operator that takes a union of these column vectors and outputs a matrix of size n × P l +1 containing these as columns.
Note that we may omit repeated columns and denote the total number of unique columns as P l +1 , since this does not change the value of our convex program.
We next provide an analytical example describing the construction of the matrix D ( l ) .
Example 3.1.
We illustrate an example with the training data X =
[ − 1 1; 0 1; 1 1] ∈ R 3 × 2 .
Inspecting the data samples (rows of X ), we observe that all possible arrangement patterns are
[Equation] [Equation]
0(cid:35)
[Equation]
= 6 . For the second layer, we ﬁrst specify the number of neurons in the ﬁrst layer as m 1 = 2 .
Thus, we need to consider all possible column pairs in (14) .
We have
D (1) { 1 , 2 } = (cid:34) 000(cid:35)
=
⇒ A (cid:32)(cid:34) 000(cid:35)(cid:33)
= (cid:34) 0 0 10 0 10 1 1(cid:35)
D (1) { 1 , 3 } = (cid:34) 000(cid:35)
=
⇒ A (cid:32)(cid:34) {+
 [Equation]+}
= [-(cid:34) 0 
-][Equation]
[Equation] [Equation]
We then construct the hyperplane arrangement matrix as
[Equation]
which shows that P 2 = 8 .
Consequently, we obtain the maximum possible arrangement patterns, i.e., { 0 , 1 } 3 , in the second layer even though we are not able to obtain some of these patterns in the ﬁrst layer in (14) .
We also provide a three dimensional visualization of this example in Figure 3.
3. P OLYNOMIAL - TIME TRAINABLE CONVEX FORMULATION
Based on the procedure described in Section 3.1 to compute the arrangement matrix D ( l ) , we now derive an exact formulation for the non-convex training problem in (12).
Theorem 3.2.
Suppose that m L − 1 ≥ m ∗ , then the non-convex training problem (12) is equivalent to
[Equation]
(15) where D ( L − 1) ∈ { 0 , 1 } n × P L − 1 is a ﬁxed matrix constructed via (13) and m ∗ denotes the cardinality of the optimal solution, which satisﬁes where m ∗ ≤ n + 1 .
Also, it holds that P noncvx L = P cvx L .
Theorem 3.2 shows that two-layer and deep networks simplify to very similar convex Lasso problems (i.e. (6) and (15)).
However, the set of hyperplane arrangements is larger for deep networks as analyzed in Section 3.1.
Thus, the structure of the diagonal matrix D and the problem dimensionality are signiﬁcantly different for these problems.
3. S IMPLIFIED CONVEX FORMULATION
Here, we show that the data X can be shattered at a certain layer, i.e., H l ( X ) = { 0 , 1 } n for a certain l ∈ [ L ] , if the number of hidden neurons in a certain layer m l satisﬁes m l ≥ C √ n/L .
Then we can alternatively formulate (11) as a simpler convex problem.
Therefore, compared to the two-layer networks in Section 2.1, we substantially improve the condition on layer width by beneﬁting from the depth L , which also conﬁrms the benign impact of the depth on the optimization.
Lemma 3.3.
If ∃ l, C such that m l ≥ C √ n/L , then the set of hyperplane arrangements is complete, i.e., H L ( X ) = { 0 , 1 } n .
We next use Lemma 3.3 to derive a simpler form of (11).
Corollary 3.4.
As a direct consequence of Theorem 2.3 and Lemma 3.3, the non-convex deep threshold network training problem in (12) can be cast as the following convex program
[Equation]
Surprisingly, both two-layer and deep networks share the same convex formulation in this case.
However, notice that two-layer networks require a condition on the data matrix in Theorem 2.
whereas the result in Corollary 3.4 requires a milder condition on the layer widths.
3. T RAINING COMPLEXITY
Here, we ﬁrst brieﬂy summarize our complexity results for the convex training of deep networks.
Based on the convex problems in (15) and Corollary 3.4, we have two regimes:
• When n +1 ≥ m L − 1 ≥ m ∗ , we solve (15) with O ( n 3 r (cid:81)
L − 2 k
=1 m k ) complexity, where r := rank ( X ) .
Note that this is polynomial-time when r and the number of neurons in each layer { m l } L − 2 l =1 are constants.
• When ∃ l, C : m l ≥ C √ n/L , we solve (7) in closed-form, and the reconstruction of the non-convex parameters requires O ( n ) time as proven in Appendix A.9.
Computational complexity for (15) :
We ﬁrst need to obtain an upperbound on the problem dimensionality P
L − 1 , which is stated in the next result.
Lemma 3.5.
The cardinality of the hyperplane arrangement set for an L -layer network H L ( X ) can be bounded as |H
L ( X ) | =
P L − 1 (cid:46) O
( n r (cid:81)
L − 2 k
=1 m k ) , where r = rank( X ) and m l denotes the number of hidden neurons in the l th layer.
Lemma 3.5 shows that the set of hyperplane arrangements gets signiﬁcantly larger as the depth of the network increases.
However, the cardinality of this set is still a polynomial term since that r < min { n, d } and { m l } L − 2 l =1 are ﬁxed constants.
To solve (15), we ﬁrst enumerate all possible arrangements { d i } P L − 1 i =1 to construct the matrix D ( L − 1) .
Then, we solve a standard convex Lasso problem, which requires O ( P 3 L − 1 ) complexity (Efron et al., 2004).
Thus, based on Lemma 3.5, the overall complexity is O ( P 3 L − 2 )
≈ O ( n 3 r (cid:81)
L − 2 k
=1 m k ) .
Computational complexity for (7) :
Since Corollary 3.4 yields (7), the complexity is O ( n ) time.
E XTENSIONS TO ARBITRARY LOSS FUNCTIONS
In the previous sections, we considered squared error as the loss function to give a clear description of our approach.
However, all the derivations extend to arbitrary convex loss.
Now, we consider the regularized training problem with a convex loss function L ( · , y ) , e.g., hinge loss, cross entropy,
[Equation]
Then, we have the following generic loss results.
Corollary 4.1.
Theorem 3.2 implies that when m L − 1 ≥ m ∗ , (16) can be equivalently stated as
[Equation]
Alternatively when H L ( X ) = { 0 , 1 } n , the equivalent convex problem is
[Equation]
based on Corollary 3.4.
[Figure] Corollary 4.1 shows that (17) and (18) are equivalent to the non-convex training problem in (16).
More importantly, they can be globally optimized via efﬁcient convex optimization solvers.
E XPERIMENTS
In this section 1 , we present numerical experiments verifying our theoretical results in the previous sections.
As discussed in Section 2.4, after solving the proposed convex problems in (15) and (7), there exist multiple set of weight matrices yielding the same optimal objective value.
Therefore, to have a good generalization performance on test data, we use some heuristic methods for the construction of the non-convex parameters { W ( l ) } Ll =1 .
Below, we provide details regarding the weight construction and review some baseline non-convex training methods.
Convex-Lasso: To solve the problem (15), we ﬁrst approximate arrangement patterns of the data matrix X by generating i.i.d.
Gaussian weights G ∈ R d × ˜ P and subsample the arrangement patterns via 1 [ XG ≥ 0] .
Then, we use G as the hidden layer weights to construct the network.
We repeat this process for every layer.
Notice that here we sample a ﬁxed subset of arrangements instead of enumerating all possible P arrangements.
Thus, this approximately solves (15) by subsampling its decision variables, however, it still performs signiﬁcantly better than standard non-convex training.
Convex-PI:
After solving (7), to recover the hidden layer weights of (2), we solve 1 { Xw (1) j ≥ 0 } =
d j as w (1) j
= X † d j , where X † denotes the pseudo-inverse of X .
The resulting weights w (1) j enforce the preactivations
Xw (1) j to be zero or one.
Thus, if an entry is slightly higher or less than zero due to [Table] precision issues during the pseudo-inverse, it might give wrong output after the threshold activation.
To avoid such cases, we use 0 .
5 threshold in the test phase, i.e., 1 { X test w (1) j ≥ 0 . 5 } .
Convex-SVM: Another approach to solve 1 { Xw (1) j ≥ 0 } =
d j is to use Support Vector Machines (SVMs), which ﬁnd the maximum margin vector.
Particularly, we set the zero entries of d i as −and then directly run the SVM to get the maximum margin hidden neurons corresponding to this arrangement.
Since the labels are in the form { +1 , − 1 } in this case, we do not need additional thresholding as in the previous approach.
Nonconvex-STE (Bengio et al., 2013):
This is the standard non-convex training algorithm, where the threshold activations is replaced with the identity function during the backward pass.
STE Variants: We also benchmark against variants of STE.
Speciﬁcally, we replace the threshold activation with ReLU ( Nonconvex-ReLU (Yin et al., 2019a)), Leaky ReLU ( Nonconvex-LReLU (Xiao et al.)), and clipped ReLU ( Nonconvex-CReLU (Cai et al., 2017)) during the backward pass.
Synthetic Datasets: We compare the performances of Convex-PI and Convex-SVM trained via (7) with the non-convex heuristic methods mentioned above.
We ﬁrst run each non-convex heuristic for ﬁve different initializations and then plot the best performing one in Figure 1.
This experiment clearly shows that the non-convex heuristics fail to achieve the globally optimal training performance provided by our convex approaches.
For the same setup, we also compare the training and test accuracies for three different regimes, i.e., n > d , n = d , and n < d .
As seen in Figure 2, our convex approaches not only globally optimize the training objective but also generalize well on the test data.
Real Datasets: In Table 2, we compare the test accuracies of two-layer threshold network trained via our convex formulation in (15), i.e., Convex-Lasso and the non-convex heuristics mentioned above.
For this experiment, we use CIFAR-10 (Krizhevsky et al., 2014), MNIST (LeCun), and the datasets in the UCI repository (Dua & Graff, 2017) which are preprocessed as in Fern ´ andez-Delgado et al.
Here, our convex training approach achieves the highest test accuracy for most of the datasets while the non-convex heuristics perform well only for a few datasets.
Therefore, we also validates the good generalization capabilities of the proposed convex training methods on real datasets.
C ONCLUSION
We proved that the training problem of regularized deep threshold networks can be equivalently formulated as a standard convex optimization problem with a ﬁxed data matrix consisting of hyperplane arrangements determined by the data matrix and layer weights.
Since the proposed formulation parallels the well studied Lasso model, we gain two major advantages over the standard non-convex training methods: 1) We globally optimize the network without resorting to any optimization heuristic or extensive hyperparameter search (e.g., learning rate schedule and initialization scheme); 2) We efﬁciently solve the training problem using specialized solvers for Lasso.
We also provided a computational complexity analysis and showed that the proposed convex program can be solved in polynomial-time.
Moreover, when a layer width exceeds a certain threshold, a simpler alternative convex formulation can be solved in O ( n ) independent of the feature dimension.
Lastly, as a by product of our analysis, we characterize the recursive process behind the set of hyperplane arrangements for deep networks.
Even though this set rapidly grows as the network gets deeper, globally optimizing the resulting Lasso problem still requires polynomial-time complexity for ﬁxed data rank.  A CKNOWLEDGEMENTS
This work was partially supported by the National Science Foundation under grants CCF-2236829,
ECCS- 2037304 and DMS-2134248, the U.S. Army Research Ofﬁce Early Career Award W911NF21-1-0242, Stanford Precourt Institute and the ACCESS – AI Chip Center for Emerging Smart
Systems sponsored by InnoHK funding, Hong Kong SAR.
Appendix Table of Contents A Proofs of the results in the main paper
A.1 Lemma 2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Proposition 2.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Theorem 2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Theorem 2.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.5 Lemma 3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.6 Theorem 3.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.7 Lemma 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.8 Lemma 3.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.9 Corollary 3.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.10 Corollary 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Additional experiments and details
B.1 Experiment in Figure 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Experiment in Table 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Two-layer experiments in Figure 6, 7, 8, and 9 . . . . . . . . . . . . . . . . . .
B.4 Three-layer experiments with a representation matrix in Figure 10, 11, 12, and
B.5 Standard three-layer experiments in Figure 1 and 2 . . . . . . . . . . . . . . . .
[Figure]
Standard ten-layer experiments in Figure 14 . . . . . . . . . . . . . . . . . . .
[Figure]
Here, we visualize the convex constraint in Proposition 2.4, which is the implicit representation revealed by convex analysis.
We observe that the hidden representation space becomes richer with an additional nonlinear layer (from 2-layer to 3-layer), and the resulting symmetry in the convex hull enables the simpler convex formulation (7).
A P ROOFS OF THE RESULTS IN THE MAIN PAPER A. L EMMA 2.
Proof of Lemma 2.1 .
We ﬁrst restate the original training problem below.
[Equation]
[Equation]
As already noted in the main paper the loss function is invariant to the norm of w (1)
j
and we may have w (1) j
→ 0 to reduce the regularization cost.
Therefore, we can omit the regularization penalty
[Figure]
on w (1) j
to simplify (19) as
[Equation]
We now note that one can rescale the parameters as
[Equation]
without the changing the network’s output, i.e.,
[Equation]
where α j > 0 .
We next note that
[Equation]
where the equality holds when α j = (cid:114) | w
[Equation]
j [Equation] | s j | .
Thus, we obtain the following reformulation of the objective function in (19) where the regularization term takes a multiplicative form
[Equation]
Next
[Equation]
we apply a variable change for the new formulation in (20) as follows
[Equation]
With the variable change above (20) can be equivalently written as
[Equation]
This concludes the proof and yields the following equivalent formulation of (21)
[Equation]
A. P ROPOSITION 2.
Proof of Proposition 2.4 .
Using the reparameterization for Lasso problems, we rewrite (9) as
[Equation]
We next introduce a slack variable t ≥ 0 such that (22) can be rewritten as
[Equation]
We now rescale w + j , w − j with 1 /t to obtain the following equivalent of (23)
[Equation]
The problem in (24) implies that ∃ w + j , w − j ≥ 0 s .
t .
[Equation]
where Conv denotes the convex hull operation.
Therefore, (9) can be equivalenly formulated as
[Equation]
A. T HEOREM 2.
Proof of Theorem 2.2 .
We ﬁrst remark that the activations can only be either zero or one, i.e., σ s
[Equation]
{ 0 , 1 } n , since | s j | = 1 , ∀ j ∈ [ m ] .
Therefore, based on Lemma 2.1, we reformulated (4) as
[Equation]
where we denote the elements of the set H ( X ) by d 1 , . .
, d P ∈ { 0 , 1 } n and P is the number of hyperplane arrangements.
The above problem is similar to Lasso (Tibshirani, 1996), although it is non-convex due to the discrete variables
d i 1 , ..., d i m .
These m hyperplane arrangement patterns are discrete optimization variables along with the coefﬁcient vector w .
In fact, the above problem is equivalent to a cardinality-constrained Lasso problem.
We have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
where inequality (i) holds by weak duality, inequality (ii) follows from augmenting the set of constraints | z T d i | ≤ β to hold for any sign pattern d ∈ H ( X ) , and inequality (iii) follows from enumerating all possible sign patterns, i.e., H ( X )
= { d 1 , d 2 , . . .
, d P } .
We now prove that strong duality in fact holds,
i
[Equation]
e., P noncvx2 = D cvx2 .
We ﬁrst form the Lagrangian of the dual problem D cvx
[Equation]
(26) where, in equality (i) follows from the fact that D cvx2 is a convex optimization problem satisfying
Slater’s conditions so that strong duality holds (Boyd & Vandenberghe, 2004), i.e., D cvx2 =
P cvx2 , and the matrix D ∈ { 0 , 1 } n × P is deﬁned as D := [ d [Equation]
d
[Equation]
P ] . Now, based on the strong duality results in Pilanci & Ergen (2020), there exist a threshold for the number of neurons, i.e., denoted as m ∗ , such that if m ≥ m ∗ then strong duality holds for the original non-convex training problem (25), i.e., P noncvx
=
D cvx2 =
P cvx2 .
Thus, (26) is exactly equivalent to the original non-convex training problem (3).
A. T HEOREM 2.
Proof of Theorem 2.3 .
Following the proof of Theorem 2.2, we ﬁrst note that strong duality holds for the problem in (25), i.e., P noncvx2 = D cvx2 .
Under the assumption that H ( X ) = { 0 , 1 } n , the dual constraint max d ∈H ( X )
|
z T d | ≤ β is equivalent to { max d ∈ [0 , 1] n z T d ≤ β } ∪ { max d [Equation] ∈ [0 , 1] [Equation]
− z
[Equation]
d
[Equation]
≤ β } .
Forming the Lagrangian based on this reformulation of the dual constraint, we have
[Equation]
where, in equality (i), we used the change of variables
d ≡ t d and d (cid:48) ≡ t (cid:48)
d (cid:48) , and, in equality (ii), we used the fact that the objective function 12 (cid:107) y (cid:107) 22 − 12 (cid:107) y − z (cid:107)
+ z T ( d (cid:48) − d ) + β ( t (cid:48)
+ t ) is strongly concave in z and convex in ( d , d (cid:48) , t, t (cid:48) ) and the constraints are linear, so that strong duality holds and we can switch the order of minimization and maximization.
Given a feasible point ( d , d (cid:48) , t, t (cid:48) ) , i.e., d ∈ [0 , t ] n and d (cid:48) ∈ [0 , t (cid:48) ] n , we set δ = d (cid:48) − d .
Note that (cid:107) ( δ ) + (cid:107) ∞ ≤ (cid:107) ( d (cid:48) )
+ (cid:107) ∞
= (cid:107) d (cid:48) (cid:107) ∞ ≤ t (cid:48) .
Similarly, (cid:107) ( − δ ) + (cid:107) ∞ ≤ t .
This implies that
[Equation]
Conversely, given δ ∈ R n , we set d = ( δ ) + , t = (cid:107) d
[Equation]
, d
[Equation]
= ( −
[Equation]
and t
[Equation]
=
[Equation]
d
[Equation]
(cid:107) ∞ .
It holds that ( d , d (cid:48) , t, t (cid:48) ) is feasible with same objective value, and consequently, the above inequality is an equality, i.e., D cvx2 =
P cvx v 2 .
Optimal threshold network construction: We now show how to construct an optimal threshold network given an optimal solution to the convex problem (7).
Let δ ∈ R n be an optimal solution.
Set d = ( δ ) + and d (cid:48)
= ( − δ ) + .
We have d ∈ [0 , (cid:107) d (cid:107) ∞ ] n and d (cid:48) ∈
[0 , (cid:107) d (cid:48) (cid:107) ∞ ] n .
It is easy to show that we can transform δ such that, for each index i ∈ [ n ] , either the i -th coordinate of d is active or the i -th coordinate of d (cid:48) is active.
Therefore, by Caratheodory’s theorem, there exist n + , n − ≥ 1 such that n − + n + ≤ n , and d 1 , . . .
, d n + +1 ∈ { 0 , 1 } n and γ 1 , . . .
, γ n + +1 ≥ 0 such that (cid:80) n + +1 i =1 γ i = 1 and d = (cid:107) d (cid:107) ∞ (cid:80) n + +1 i =1 γ i d i , and, d (cid:48) 1 , . . .
, d (cid:48) n − +1 ∈ { 0 , 1 } n and γ (cid:48) 1 , . .
, γ (cid:48) n − +1 ≥ 0 such that (cid:80) n − +1 i = γ (cid:48)
i =and d (cid:48)
= (cid:107) d (cid:48) (cid:107) ∞ (cid:80)
n − +1 i = γ (cid:48)
i d (cid:48)
i , with n − + n + ≤ n .
Then, we can pick w (1)1 , . . .
, w (1) n + +1 , w (2)1 , . . . , w (2) n + +1 , s 1 , . . .
, s n + +1 such that 1 { Xw (1) i ≥ 0 } =
d i , w (1) i
= (cid:107) d (cid:107) ∞
γ i and s i = − 1 , and, w (1) n + +2 , . . .
, w (1) n + + n − +2 , w (2) n + +2 , . . .
, w (2) n + + n − +2 , s n + +2 , . . .
, s n + + n − +2 such that 1 { Xw (1) i ≥ 0 } =
d (cid:48)
i − ( n + +1) , α i = (cid:107) d (cid:48) (cid:107) ∞
γ
i − ( n + +1) and s i = 1 .
Note that, given δ ∈ R n , ﬁnding the corresponding d 1 , . . .
, d n + +1 , γ 1 , . . .
, γ n + +1 and d (cid:48) 1 , . .
, d (cid:48) n + +1 , γ (cid:48) 1 , . .
, γ (cid:48) n + +1 takes time
O ( n + + n + ) =
O ( n ) .
A. L EMMA 3.
Proof of Lemma 3.1 .
We ﬁrst restate the standard weight decay regularized training problem below
[Equation]
Then, we remark that the loss function in (27), i.e., 12 (cid:107) f θ,L ( X ) − y (cid:107) 22 is invariant to the norms of hidden layer weights {{ W ( l ) k } L − 1 l =1 } m L − 1 k =1 and amplitude parameters {{ s ( l ) k } L − 2 l =1 } m L − 1 k =1 .
Therefore, (27) can be rewritten as
[Equation]
Then, we can directly apply the scaling in the proof of Lemma 2.1 to obtain the claimed (cid:96) 1 -norm regularized optimization problem.
A. T HEOREM 3.
Proof of Theorem 3.2 .
We ﬁrst remark that the last hidden layer activations can only be either zero or one, i.e., X
( L − 1) k ∈ { 0 , 1 } n × m L − 1 , since | s ( L − 1) k
| = 1 , ∀ k ∈ [ m L − 1 ] .
Therefore, based on Lemma 3.1, we reformulated (12) as
[Equation]
As in (25), the above problem is similar to Lasso, although it is non-convex due to the discrete variables
d i 1 , ..., d i mL − 1 .
These m L − 1 hyperplane arrangement patterns are discrete optimization variables along with the coefﬁcient vector w .
We now note that (29) has following dual problem
[Equation]
Since the above dual form is the same with the dual problem of two-layer networks, i.e., D cvx L = D cvx2 , we directly follow the proof of Theorem 2.2 to obtain the following bidual formulation.
[Equation]
where D ( L − 1) =
[ d 1 , d , . . .
, d P L − 1 ] .
Hence, based on the strong duality results in Pilanci & Ergen (2020), there exist a threshold for the number of neurons, i.e., denoted as m ∗ , such that if m L − 1 ≥ m ∗ then strong duality holds for the original non-convex training problem (25), i.e., P noncvx L
[Equation]
A. L EMMA 3.
Proof of Lemma 3.3 .
Based on the construction in (Bartlett et al., 2019), a deep network with L layers and W parameters, where W = (cid:80) Li =
1 m i − 1 m i in our notation, can shatter a dataset of the size C 1 WL log( W ) , where C 1 is a constant.
Based on this complexity results, if we choose m l = m = C √ n/L, ∀ l ∈ [ L ] , the number of samples that can be shattered by an L -layer threshold network is # of data samples that can be shattered [Equation] 1 WL
[Equation]
W
[Equation]
provided that n (cid:29) L , which is usually holds in practice and we choose the constant C such that
C = 1 /C 21 .
Therefore, as we make the architecture deeper, we can in fact improve the assumption m ≥ n + 2 assumption in the two-layer networks case by beneﬁting from the depth.
A. L EMMA 3.
Proof of Proposition 3.5 .
Let us start with the L = 2 case for which hyperplane arrangement set can described as detailed in (5)
[Equation]
Now, in order to construct H 3 ( X ) {+(with m L − 1 = 
 so we drop the index k ),+} we need to choose marrangements from H 2 ( X ) and then consider all hyperplane arrangements for each of these choices.
Particularly, since |H 2 ( X ) | =
P 1 , where
[Equation]
from (8), we have (cid:0) P 1 m 1 (cid:1) choices and each of them yields a different activation matrix denoted as
X (1) i ∈
R n × m 1 .
Based on the upperbound (8), each X (1) i can generate O ( n m 1 ) patterns.
Therefore, overall, the set of possible arrangement in the second layer is as follows
[Equation]
which implies that
[Equation]
This analysis can be recursively extended to the l th layer where the hyperplane arrangement set and the corresponding cardinality values can be computed as follows
[Equation]
A. C OROLLARY 3.
Proof of Corollary 3.4 .
We ﬁrst remark that the last hidden layer activations can only be either zero or one, i.e., X
( L − 1) k ∈ { 0 , 1 } n × m L − 1 , since | s ( L − 1) k | = 1 , ∀ k ∈ [ m L − 1 ] .
Therefore, based on
Lemma 3.1, we reformulated (12) as
[Equation]
As in (25), the above problem is similar to Lasso, although it is non-convex due to the discrete variables
d i 1 , ..., d i mL − 1 .
These m L − 1 hyperplane arrangement patterns are discrete optimization variables along with the coefﬁcient vector w .
We now note that (31) has following dual problem
[Equation]
Since H L ( X ) = { 0 , 1 } n by Lemma 3.3, all the steps in the proof of Theorem 2.3 directly follow.
Optimal deep threshold network construction: For the last two layers’ weights, we exactly follow the weight construction procedure in the Proof of Theorem 2.3 as detailed below.
Let δ ∈ R n be an optimal solution.
Set d = ( δ ) + and d (cid:48)
= ( − δ ) + .
We have d ∈ [0 , (cid:107) d (cid:107) ∞ ] n and d (cid:48) ∈
[0 , (cid:107) d (cid:48) (cid:107) ∞ ] n .
It is easy to show that we can transform δ such that, for each index i ∈ [ n ] , either the i -th coordinate of d is active or the i -th coordinate of d (cid:48) is active.
Therefore, by Caratheodory’s theorem, there exist n + , n − ≥ 1 such that n − + n + ≤ n , and d 1 , . . .
, d n + +1 ∈ { 0 , 1 } n and γ 1 , . . .
, γ n + +1 ≥ 0 such that (cid:80) n + +1 i =1 γ i = 1 and d = (cid:107) d (cid:107) ∞ (cid:80) n + +1 i =1 γ i d i , and, d (cid:48) 1 , . . .
, d (cid:48) n − +1 ∈ { 0 , 1 } n and γ (cid:48) 1 , . .
, γ (cid:48) n − +1 ≥ 0 such that (cid:80) n − +1 i = γ (cid:48)
i = 1 and d (cid:48)
= (cid:107) d (cid:48) (cid:107) ∞ (cid:80)
n − +1 i = γ (cid:48)
i d (cid:48)
i , with n − + n + ≤ n .
Then, we can pick w ( L − 1), . . .
, w ( L − 1) n + +1 , w ( L ) 1 , . . . , w ( L ) n + +1 , s ( L − 1), . . .
, s ( L − 1) n + +
such that 1 { X ( L − 2) w ( L − 1)
i ≥ 0 } =
d i , w ( L − 1) i
= (cid:107) d (cid:107) ∞
γ i and s i = − 1 , and, w ( L − 1) n + +2 , . . .
, w (1) n + + n − +2 , w ( L ) n + +2 , . . . , w ( L ) n + + n − +2 , s ( L − 1) n + +2 , . . .
, s ( L − 1) n + + n − +2 such that 1 { X ( L − 2) w ( L − 1)
i ≥ 0 } =
d (cid:48)
i − ( n + +1) , α i = (cid:107) d (cid:48) (cid:107) ∞
γ
i − ( n + +1) and s i = 1 .
Note that, given δ ∈ R n , ﬁnding the corresponding d 1 , . . .
, d n + +1 , γ 1 , . . .
, γ n + +1 and d (cid:48) 1 , . .
, d (cid:48) n + +1 , γ (cid:48) 1 , . .
, γ (cid:48) n + +1 takes time O ( n + + n + )
=
O ( n ) .
Then, the rest of the layer weights can be reconstructed using the construction procedure detailed in (Bartlett et al., 2019).
A. C OROLLARY 4.
Proof of Corollary 4.1 .
We ﬁrst apply the scaling in Lemma 3.1 and then follow the same steps in the proof of Theorem 2.3 to get the following dual problem
[Equation]
where L ∗ is the Fenchel conjugate of L and deﬁned as
[Equation]
Therefore, we obtain a generic version of the dual problem in (30), where we can arbitrarily choose the network’s depth and the convex loss function.
Then the rest of the proof directly follows from
Theorem 3.2 and Corollary 3.4.
B A DDITIONAL EXPERIMENTS AND DETAILS
In this section, we present additional numerical experiments and further experimental details that are not presented in the main paper due to the page limit.
We ﬁrst note that all of the experiments in the paper are run on a single laptop with Intel(R) Core(TM) i7-7700HQ CPU and 16GB of RAM. [Figure]
E XPERIMENT IN F IGURE
For the non-convex heuristic, we repeat the training process using 5 independent initializations, however, all trials fail to converge to the global minimum obtained by our convex optimal method, and lack stability.
We provide experimental details in Appendix B.
For the experiment in Figure 5, we consider a simple one dimensional experiment, where the data matrix is X =
[ − 2 , − 1 , 0 , 1 , 2] T .
Using this data matrix, we generate the corresponding labels y by simply forward propagating the data through randomly initialized two-layer threshold networks with m 1 = 2 neurons as described in (2).
We then run our convex training method in (6) and the non-convex training heuristic STE (Bengio et al., 2013) on the objective with the regularization coefﬁcient β = 1 e − 2 .
For a fair comparison, we used P 1 = 24 for the convex method and m 1 = 24 for STE.
We also tune the learning rate of STE by performing a grid search on the set { 5 e − 1 , 1 e − 1 , 5 e − 2 , 1 e − 2 , 5 e − 3 , 1 e − 3 } .
As illustrated in Figure 5, the non-convex training heuristic STE fails to achieve the global minimum obtained by our convex training algorithm fordifferent initialization trials.
B. E XPERIMENT IN T ABLE
Here, we provide a test performance comparison on on CIFAR-10 (Krizhevsky et al., 2014), MNIST (LeCun), and the datasets taken from UCI Machine Learning Repository (Dua & Graff, 2017), where we follow the preprocesing in Fern´andez-Delgado et al.
(2014) such that 750 ≤ n ≤ 5000 (see
Table 3 for the exact dataset sizes).
We particularly consider a conventional binary classiﬁcation framework with two-layer networks and performed simulations over multiple seeds and compare the mean/std of the test accuracies of non-convex heuristics trained with SGDnamely NonconvexSTE (Bengio et al., 2013),
Nonconvex-ReLU (Yin et al., 2019a), Nonconvex-LReLU (Xiao et al.) and Nonconvex-CReLU (Cai et al., 2017), with our convex program in (6), i.e., Convex-Lasso .
For the non-convex training, we use the SGD optimizer.
We also use the 80% − 20% splitting
[Table]
ratio for the training and test sets of the UCI datasets.
We tune the regularization coefﬁcient β and the learning rate µ for the non-convex program by performing a grid search on the sets β list = { 1 e − 6 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 0 . 5 , 1 , 5 } and µ list =
{ 1 e − 3 , 5 e − 3 , 1 e − 2 , 1 e − 1 } , respectively.
In all experiments, we also decayed the selected learning rate systematically using
PyTorch’s (Paszke et al., 2019) scheduler ReduceLROnPlateau .
Moreover, we choose the number of neurons, number of epochs (ne), batch size (bs) as m = 1000 , bs = 5000 , bs = n , respectively.
Our convex approach achieves highest test accuracy for precisely 9 of 13 datasets whereas the best non-convex heuristic achieves the highest test accuracy only for 4 datasets.
This experiment veriﬁes that our convex training approach not only globally optimize the training objective but also usually generalizes well on the test data.
In addition, our convex training approach is shown to be signiﬁcantly more time efﬁcient than standard non-convex training.
[Figure]
with four standard non-convex training heuristics (STE and its variants in Section 5).
To generate the dataset, we randomly initialize a two-layer network and then sample a random i.i.d.
Gaussian data matrix X ∈ R n × d .
Then, we set the labels as y =
sgn(tanh( XW (1) )
w (2) ) where sgn and tanh are the sign and hyperbolic tangent functions.
We speciﬁcally choose the regularization coefﬁcient β = 1 e − 3 and run the training algorithms with m = 1000 neurons for different ( n, d ) combinations.
For each non-convex method, we repeat the training with 5 different initialization and then plot the best performing non-convex method for each initialization trial.
In each case, our convex training algorithm achieves signiﬁcantly lower objective than all the non-convex heuristics.
We also indicate the time taken to solve the convex program with a marker.
[Figure] [Figure]
B. T WO - LAYER EXPERIMENTS IN F IGURE 6, 7, 8, AND
Here, we compare two-layer threshold network training performance of our convex program (6), which we call Convex-Lasso , with standard non-convex training heuristics, namely Nonconvex-STE (Bengio et al., 2013), Nonconvex-ReLU (Yin et al., 2019a), Nonconvex-LReLU (Xiao et al.) and
Nonconvex-CReLU (Cai et al., 2017).
For the non-convex heuristics, we train a standard twolayer threshold network with the SGD optimizer.
In all experiments, learning rates are initialized to be 0 .
and they are decayed systematically using PyTorch’s (Paszke et al., 2019) scheduler
ReduceLROnPlateau .
To generate a dataset, we ﬁrst sample an i.i.d.
Gaussian data matrix X
and then obtain the corresponding labels as y =
sgn(tanh( XW (1) )
w (2) ) where sgn and tanh are the sign and hyperbolic tangent functions, respectively.
Here, we denote the the ground truth parameters as W (1) ∈ R d × m ∗ and w (2) ∈
R m ∗ , where m ∗ = 20 is the number of neurons in the ground truth model.
Notice that we use tanh and sign in the ground truth model to have balanced label distribution y ∈
+1 , − 1 } n .
We also note that for all the experiments, we choose the regularization coefﬁcient as β = 1 e − 3 .
We now emphasize that to have a fair comparison with the non-convex heuristics, we ﬁrst randomly sample a small subset of hyperplane arrangements and then solve (6) with this ﬁxed small subset.
Speciﬁcally, instead of enumerating all possible arrangements { d i } Pi =1 , we randomly sample a subset { d i j } m j =1 to have a fair comparison with the non-convex neural network training with m hidden neurons.
So, Convex-Lasso is an approximate way to solve the convex program (6) yet it still performs extremely well in our experiments.
We also note that the other convex approaches
Convex-PI and Convex-SVM exactly solve the proposed convex programs.
In Figure 6, we compare training performances.
We particularly solve the convex optimization problem (6) once.
However, for the non-convex training heuristics, we try ﬁve different initializations.
[Figure]
We then select the trial with the lowest objective value to plot.
In the ﬁgure, we plot the objective value deﬁned in (3).
As it can be seen from the ﬁgures, Convex-Lasso achieves much lower objective value than the non-convex heuristics in three different regimes, i.e., overparameterized ( n < d ), underparameterized ( n > d ) and moderately parameterized (( n = d )).
This is mainly because of the non-convex and heuristic nature of the standard optimizers.
Moreover, we illustrate training and test accuracies of these training algorithms in Figure 7, 8, and 9 for different ( n, d ) combinations.
To generate the test data with 3000 , samples we use the same ground truth model deﬁned above.
Here we observe that in some cases even though Convex-Lasso provides a better training performance, it might yield worse test accuracies than the non-convex heuristic especially in the low data regime ( n ≤ d ) .
Therefore, understanding the generalization performance and optimal weight reconstruction (discussed in Section 2.4) remains to be an open problem.
[Equation]
three-layer network and then sample i.i.d.
Gaussian data matrix.
We then set the labels as y = sgn(tanh(tanh( XW (1) ) W (2) ) w (3) ) .
To have our convex formulation, we require H ( X ) to be complete.
Thus, we ﬁrst use a random representation matrix H ∈ R d × M , where M = 1000 , and then apply the transformation ˜ X
= σ ( XH ) .
We also apply this transformation to the non-convex methods as if we train a two-layer networks on the modiﬁed data matrix ˜ X .
We also indicate the time taken to solve the convex programs with markers.
We again observe that our convex training approach achieves lower objective value than all the non-convex heuristic training methods in all initialization trials.
B. T HREE - LAYER EXPERIMENTS WITH A REPRESENTATION MATRIX IN F IGURE 10, 11, 12, AND
We also compare our alternative convex formulation in (7) with non-convex approaches for three-layer network training.
To do so we ﬁrst generate a dataset via the following ground truth model y =
[Figure] [Figure]
( n ≤ d ).
sgn(tanh(tanh( XW (1) ) W (2) )
w (3) ) , where W (1) ∈ R
d
× m ∗ 1 , W (2) ∈
R m ∗ 1 × m ∗ 2 and w (
3) ∈
R m ∗and we choose m ∗ [-
-]1 = m ∗ 2 = 20 for all the experiments.
As it is described in Theorem 2.3, we require
H ( X ) to be complete in this case.
To ensure that, we transform the data matrix X using a random representation matrix.
In particular, we ﬁrst generate a random representation matrix H ∈ R d × M and then multiply it with the data matrix followed by a threshold function.
Therefore, effectively, we obtain a new data matrix ˜ X
= σ ( XH ) .
By choosing M large enough, which is M = 1000 in our experiments, we are able to enforce ˜ X to be full rank and thus the set of hyperplane arrangements
H ( ˜ X ) is complete as assumed in Theorem 2.3.
We also apply the same steps for the non-convex training, i.e., we train the networks as if we perform a two-layer network training on the data matrix ˜
X .
Notice that we also provide standard three-layer network training comparison, where all of three layers are trainable, in Section B.5.
More importantly, after solving the convex problem (7), we need to construct a neural network that gives the same objective value.
As discussed in Section 2.4, there are numerous ways to reconstruct the non-convex network weights and we particularly use Convex-PI and Convex-SVM as detailed in Section 5, which seem to have good generalization performance.
We also note that these approaches are exact in the sense that they globally optimize the objective in B.3.
Additionally, we have n neurons in our reconstructed neural network to have fair comparison with the non-convex training with n hidden neurons.
In Figure 10, we compare objective values and observe that Convex-PI and Convex-SVM , which solve the convex problem 7, obtain the globally optimal objective value whereas all trials of the non-convex training heuristics are stuck at a local minimum.
Figure 11, 12, and 13 show that our convex training approaches also yield better test performance in all cases unlike the two-layer training in Section B.3.
Again, for the testing phase we generate 3000 samples via the ground truth model deﬁned above.
[Figure]
B. S TANDARD THREE - LAYER EXPERIMENTS IN F IGURE 1 AND
Here, we compare our convex program in (7) with 3-layer networks trained with non-convex heuristics.
Similar to the case in Section B.4 we generate a dataset via the ground truth model y =
sgn(tanh(tanh( XW (1) )
W (2) )
w (3) ) , where X ∈ R n × d , W (1) ∈ R
d × m ∗ 1 , W (
) ∈
R m ∗ 1 × m ∗and w (
3) ∈
R m ∗
and we choose m ∗
= m ∗ 2 = 20 for all the experiments.
In contrast to Section B.4, we use a representation matrix H ∈ R d × M , where M = 1000 , and then apply the transformation ˜ X
= σ ( XH ) for the convex methods.
We perform standard training procedure for three-layer non-convex networks, i.e., we train a fully three-layer network without any sort of representation matrix.
To have fair comparison, we choose m 1 =
M and m 2 = n for both non-convex and convex settings.
In Figure 1, we compare the objective values and observe that our convex training approaches achieve a global optimum in all cases unlike the non-convex training heuristics.
We also provide the test and training accuracies for three-layer networks trained with different ( n, d ) pairs in Figure 2.
In all cases our convex approaches outperform the non-convex heuristics in terms of test accuracy.
[Figure]
we in Figure 1.
As in the previous experiment, our convex training approaches yields outperforms the non-convex heuristics in both training and test metrics.
B. S TANDARD TEN - LAYER EXPERIMENTS IN F IGURE
In order to verify the effectiveness of the proposed convex formulations in training deeper networks, we consider a threshold network training problem for a 10 -layer network, i.e., (10) with L = 10 .
We directly follow the same setup with Section B.5 for ( n, d, β ) =
(100 , 20 , 1 e − 3) .
Since in this case the network is signiﬁcantly deeper than the previous cases, all of the non-convex training heuristics failed to ﬁt the training data, i.e., they couldn’t achieve 100% training accuracy.
Therefore, for the non-convex training, we also include Batch Normalization in between hidden layers to stabilize and improve the training performance.
In Figure 14, we present the results for this experiment.
Here, we observe that our convex training approaches provide a globally optimal training performance and
yield higher test accuracies than all of the non-convex heuristics that are further supported by Batch Normalization