T HE H IDDEN C ONVEX O PTIMIZATION L ANDSCAPE OF R EGULARIZED T WO -L AYER R E LU N ETWORKS : AN E XACT C HARACTERIZATION OF O PTIMAL S OLUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We prove that ﬁnding all globally optimal two-layer ReLU neural networks can be performed by solving a convex optimization program with cone constraints.
Our analysis is novel, characterizes all optimal solutions and does not leverage duality-based analysis which was recently used to lift neural network training into convex spaces.
Given the set of solutions of our convex optimization program, we show how to construct exactly the entire set of optimal neural networks.
We provide a detailed characterization of this optimal set and its invariant transformations.
As additional consequences of our convex perspective, (i) we establish that Clarke stationary points found by stochastic gradient descent correspond to the global optimum of a subsampled convex problem (ii) we provide a polynomial-time algorithm for checking if a neural network is a global minimum of the training loss (iii) we provide an explicit construction of a continuous path between any neural network and the global minimum of its sublevel set and (iv) characterize the minimal size of the hidden layer so that the neural network optimization landscape has no spurious valleys.
Overall, we provide a rich framework for studying the landscape of neural network training loss through convexity.
I NTRODUCTION
Let X ∈ R n × d and y ∈ R n be the data matrix and the label vector.
Given a number of neurons m (cid:62) 1 and a regularization parameter β > 0 , we consider the regularized optimization problem
[Equation]
where Θ m = R d × m × R m , θ = ( U, α ) , u i is the i -th column of U ∈ R d × m
and α i is the i -th coefﬁcient of α ∈ R m .
Here we focus on the ReLU activation, i.e., σ ( z ) = max { z, 0 } and absorb the label y ∈ R n in the loss function (cid:96) :
R n → R , which is assumed to be convex (e.g., logistic, hinge, squared loss).
The model (cid:80) mi =1 σ ( Xu i ) α i in (1) can be easily extended to the one with bias term by adding a column of 1 ’s into the data X .
We refer to an element θ ∈ Θ m as a neural network and to each pair ( u i , α i ) as a neuron.
We denote the set of optimal neural network as
[Equation]
We denote the best training loss achievable by a neural as P ∗ =
inf m (cid:62) 1 P ∗ m .
The ReLU activation induces a natural partition of the parameter space.
We denote D 1 , . . .
, D p as all possible values of diag ( 1 ( Xu (cid:62) 0)) .
We introduce the corresponding convex cones C i = { u ∈ R d | (2 D i − I ) Xu (cid:62) 0 } for i ∈
[ p ] where we denote [ p ] = { 1 , . .
. , p } .
From this partition we have the local linearization σ ( Xu ) =
D i
Xu , for u ∈ C i .
We let D i + p = − D
i for i ∈ [ p ] and C i + p =
C i for i ∈ [ p ] . Such a partition of the parameter space has regained attention in the recent literature.
In fact, Pilanci & Ergen (2020) recently showed that an optimal neural network θ ∗ ∈ Θ m for any m (cid:62) 2 p can be
[Equation]
where we introduced the convex feasible set W : = { W = ( w 1 , . .
. , w 2 p ) | w i ∈ C i } .
In a nutshell, this equivalence can be intuitively explained as follows.
From the constraint w i ∈ C i , we obtain the local linearization σ ( Xw i ) =
D i
Xw
i for i ∈ [ p ] and σ ( Xw i ) =
− D i
Xw
i for i ∈
[ p + 1 , 2 p ] .
By choosing neurons ( u i , α i ) such that w i = | α i | u i , α i (cid:62) 0 for i ∈
[ p ] and α i (cid:54) 0 for i (cid:62) p + 1 , we further obtain by positive homogeneity of the ReLU that
[Equation]
From the fact that the cones C 1 , . . .
, C p cover the entire space, Pilanci & Ergen (2020) establish the equality P ∗ c = P ∗ , and show that an optimal neural network can be constructed from an optimal solution w ∗ 1 , . . .
, w ∗ p .
In this work, we explore in more depth the mapping from the optimal set of solutions W ∗ of the convex program (4) to the set of optimal neural networks Θ ∗ m .
In fact, our main contribution is to show how to construct the set Θ ∗ m given W ∗ through simple mappings and transformations.
Throughout our analysis, we unveil some novel necessary conditions for a neural network to be optimal and we illustrate the relevance of these conditions by relating them to usual necessary conditions for optimality (e.g., Clarke stationarity).
1. P RIOR AND RELATED WORK
Literature on the theory of deep learning is now extremely vast, and we mention here only a few classical results and recent works which focus on the properties of the neural network training loss, from an optimization perspective (e.g., the convergence of ﬁrst-order methods, absence of spurious minima in the loss landscape, and, convex re-formulations of neural network training), and in the case of single-layer with ReLU-type activations.
Several recent works considered over-parameterized neural networks in the inﬁnite-width limit.
In particular, it is known that in this regime, gradient descent converges to an optimal solution, see (Jacot et al., 2018; Du et al., 2018; Allen-Zhu et al., 2018; Nguyen, 2021).
Further analysis in (Chizat & Bach, 2018) showed that almost no hidden neurons move from their initial values to actively learn useful features, so that this regime resembles that of kernel training and the inﬁnite-width limit infuses convexity.
Wang & Lin (2021) showed that with an explicit regularizer based on the scaled variation norm, overparametrization is generally harmless to two-layer ReLU networks.
However, experiments in (Arora et al., 2016) suggest that this kernel approximation is unable to fully explain the success of non-convex neural network models.
Convexity arguments in neural networks were proposed in the recent literature (Bengio et al., 2006;
Bach, 2017).
However, existing works, except (Pilanci & Ergen, 2020), are restricted to inﬁnitely wide networks.
In turn, Bengio et al.
(2006) and Bach (2017) consider greedy neuron-wise optimization strategies for the inﬁnite-dimensional optimization problem, which requires solving non-convex problems at every step to train a shallow neural network.
In contrast, in our work, we reveal the hidden convex optimization landscape for any ﬁnite number of hidden neurons.
Besides the convexity properties of inﬁnitely wide networks, many works derived lower bounds on the hidden layer size to guarantee the absence of spurious minima.
Venturi et al.
(2019) showed that the un-regularized (i.e. β = 0 ) objective L β has no spurious local minima provided that the number of neurons satisﬁes m (cid:62) n , and a similar result was shown in (Livni et al., 2014).
Similar results were derived for deep networks.
For instance, Soudry & Carmon (2016) showed that under a dropout-like noise assumption, there exist no differentiable spurious minima if the product of the dimensions of the layer weights exceeds n and this result matches the classical lower bound (Baum, 1988) on the minimal width of a neural network to implement any dichotomy for inputs in general position.
In a similar vein, Nguyen & Hein (2017) showed that no spurious minima occur provided that one of the layer’s inner width exceeds n and under additional non-degeneracy conditions.
For activations other than the ReLU (e.g., linear, quadratic, polynomial), similar lower bounds were derived in (Venturi et al., 2019; Du & Lee, 2018; Soltanolkotabi et al., 2018).
These analyses are typically based on the idea that when m (cid:38) n then it is very likely that the features σ ( Xu 1 ) , . . .
, σ ( Xu m ) form a basis of R n so that the training problem reduces to ﬁnding a linear model with weights
α 1 , . .
α m which perfectly ﬁts the labels.
For the hinge loss and linear separable data, (Wang et al., 2019) show that the modiﬁed stochastic gradient descent method can achieve global optimality despite the presence of spurious local minima and saddle points.
The training landscape of neural networks is of great interest for theoretical analysis in the optimization of neural networks.
An important perspective is to analyze the landscape via paths through the parameter space, see (Vidal et al., 2017).
Indeed, in (Haeffele & Vidal, 2015; 2017; Sharifnassab et al., 2019), it is shown that there exists a non-increasing path in objective value from every point to the global minimum with mild assumption on the layer width.
The existence of such paths also indicates that the level sets of the training loss are connected (Freeman & Bruna, 2016; Venturi et al., 2019; Nguyen, 2019; Nguyen et al., 2021) and there is no bad local valley (Nguyen & Hein, 2017).
However, with regularization, the training problem is more challenging.
Intuitively, it reduces the set of optimal solutions to those with small norms.
Without regularization (i.e., β = 0 ), it should be noted that the set of optimal solutions always contains inﬁnitely many points.
For instance, with ReLU activations, it holds that if θ ∗ =
{ ( u ∗ i , α ∗ i ) } m i =1 is an optimal neural network, then any re-scaling of θ ∗ in the form { ( u ∗ i γ
i , γ i α ∗ i ) } mi =1 (with γ 1 , . . .
, γ m > 0 ) has the same objective value and is thus optimal.
With regularization, this manifold is reduced to a single point.
It is then natural to expect that this minimal size of the hidden layer must increase.
Further, the aforementioned analyses do not extend since regularization also penalizes the norms of the u
i ’s, and one cannot simply generate such a basis of R n based on the features σ ( Xu 1 ) , . . .
, σ ( Xu m ) by random sampling and then overﬁtting the labels.
Recently, Pilanci & Ergen (2020); Ergen & Pilanci (2020) show that two-layer ReLU neural networks can be optimized exactly via ﬁnite-dimensional convex programs with complexity polynomial in the number of samples and hidden neurons.
As indicated in Pilanci & Ergen (2020), the worst-case complexity is exponential in the dimension of the training samples unless P = NP .
1. S UMMARY OF OUR CONTRIBUTIONS
In Section 2, we introduce the notions of minimal neural networks and nearly minimal neural networks.
These two notions are closely related to the plateau and the edge of the plateau of the loss landscape.
In Section 3, we show that any minimal neural network θ can be represented, via an explicit map, in the convex feasible space W as a point W ( θ ) such that L β ( θ ) (cid:62) L cβ ( W ( θ )) , and vice-versa.
This structural result provides a mathematically rich perspective to characterize optimal neural networks through the lens of convexity.
We then provide an exact characterization of the set of all global optima of the nonconvex problem, which include all nearly minimal neural networks generated via the optimal solutions of the convex program.
In Section 4, we show that any Clarke stationary point θ with respect to L β is a nearly minimal neural network.
This provides a preliminary structure on the solutions found by stochastic gradient descent (SGD), as it has been recently shown (see, for instance, Corollary 5.11 in Davis et al.
that the limit points of SGD applied to neural network optimization are Clarke stationary.
More importantly, we show that Clarke stationary point θ with respect to L β also corresponds to a global minimum of a subsampled convex problem.
We also provide a polynomial-time algorithm (in the sample size n and the hidden-layer size m ) in order to test whether a neural network is globally optimal.
In Section 5, we show that any neural network is path-connected to a succinct representation (with at most n + 1 non-zero neurons) and this path is with constant objective value.
Then, from the convex perspective of two-layer ReLU neural networks, we provide an explicit path of non-increasing loss between θ and θ (cid:48) , where θ (cid:48) is the global optimum of the non-convex training problem.
This establishes that the training loss L β has no spurious local minima, provided that the number of neurons is sufﬁciently large.
1. N OTATIONS
We ﬁrst present an alternative interpretation of the cones C i and the diagonal matrices
D i for i ∈
[ p ] .
The ReLU activation function partitions the space of neurons u ∈ R d into linearly separated regions, that is, given a binary vector s ∈ { 0 , 1 } n , the set of neurons u ∈ R d such that 1 ( Xu (cid:62) 0)
= s is a convex cone in R d , if not empty.
We enumerate the closures of all these cones as C 1 , . . .
, C p
and we set C i
+ p
=
C i for i ∈
[ p ] .
For i ∈ [ p ] , we introduce the corresponding diagonal matrices
D i
=
diag
( 1 ( Xu (cid:62) 0)) for an arbitrary u ∈ C i , and D i + p = − D i .
Here the number p is the number of dichotomies that the data matrix X can realize.
It is upper bounded by p (cid:54) 2 r (cid:16) e ( n − 1) r (cid:17) r where r = rank ( X ) , see (Cover, 1965).
Beyond the dichotomies of the space of neurons u ∈ R d , we further introduce the partitions (trichotomies) { I + , I 0 , I − } of [ n ] such that there exists a solution vector u ∈ R d verifying ( Xu ) k >if k ∈ I + , ( Xu ) k
=
if k ∈
I 0 and ( Xu ) k < 0 if k ∈ I − .
Clearly, there exists a ﬁnite number q of such trichotomies and q is trivially upper bounded by 3 n .
For the j -th trichotomy { I + , I 0 , I − } , we deﬁne the n × n diagonal matrix T j with k -th diagonal element ( T j ) kk = 1 if k ∈
I + , ( T j ) kk =
if k ∈ I 0 and ( T j ) kk = 0 if k ∈ I − .
Such trichotomies are also discussed in Phuong & Lampert (2020).
For each j = 1 , . . .
, q , we deﬁne Q j as the closed convex cone of solution vectors for the j -th trichotomy { I + , I 0 , I − } .
We consider a partition { B 1 , . . .
, B 2 q } of the neurons’ parameter space where B i : =
Q i × R > 0 for j = 1 , . . .
, q and B j :
=
Q j − q × R < 0 for j = q +1 , . . .
, 2 q .
We augment the set of diagonal matrices { T j } qj =1 by setting T j = − T j − q for j = q + 1 , . .
, 2 q .
For a neuron pair ( u, α ) ∈ R d × R , we denote B ( u, α ) as the unique B i such that ( u, α ) ∈ B i .
The notion of path-connected sublevel set is introduced as follows.
Deﬁnition 1.
We write θ (cid:73) θ (cid:48) if the neural network θ (cid:48) ∈
Θ m belongs to the path-connected sublevel set (or valley) of θ ∈ Θ m .
Namely, there exists a continuous path γ : [0 , 1] → Θ m such
that γ (0) = θ , γ (1) =
θ (cid:48) and
t (cid:55)→ L β ( γ ( t )) is non-increasing.
We denote the valley of θ as Ω( θ ) :
= { θ (cid:
) ∈
Θ m | θ (cid:73) θ (cid:48) } .
We say that θ ∈ Θ m is non-spurious if θ (cid:73) θ ∗ for some θ ∗ ∈ argmin θ (cid:
) ∈
Θ m L β ( θ (cid:48) ) .
Otherwise, we say that θ and its valley Ω( θ ) are spurious.
M INIMAL NEURAL NETWORKS AND NEARLY MINIMAL NEURAL NETWORKS
We start with the notion of minimal neural networks and nearly minimal neural networks.
Minimal neural networks enjoy a well-structured representation which is useful to understand the optimality properties of two-layer neural networks.
Deﬁnition 2 (Minimal neural networks) .
We say that a neural network θ is minimal if (i) it is scaled, i.e., (cid:107)
u
i (cid:107) 2 = | α i | for i ∈ [ m ] and (ii) the cones B ( u, α ) of each of its non-zero neurons ( u, α ) are pairwise distinct.
That is, a minimal neural network has at most a single non-zero neuron per cone
B i .
We denote by Θ min m the set of minimal neural networks with m neurons.
Note that any minimal neural network has at most 2 q non-zero neurons since there are 2 q cones
B i and at most one neuron per cone.
Next, we introduce a slightly less structured class of neural networks that one can interpret as ’split’ versions of minimal neural networks, and can have an arbitrary number of non-zero neurons.
Deﬁnition 3 (Nearly minimal neural networks) .
We say that a neural network θ is nearly minimal if (i) it is scaled and (ii) for any two non-zero neurons ( u, α ) , ( v, β ) of θ , if B ( u, α )
=
B ( v, β ) then u and v are positively colinear, i.e., there exists λ (cid:62)
such that u
= λv .
We denote by (cid:101)
Θ min m the set of nearly minimal neural networks with m neurons.
It trivially holds that Θ min m ⊂ (cid:101) Θ min m .
For a nearly minimal neural network, by merging the neurons corresponding to the same trichotomies, we can reformulate it into a minimal neural network without changing the objective value.
2. F ROM NEARLY MINIMAL TO MINIMAL NEURAL NETWORKS
Nearly minimal neural networks have the property that any two neurons which share at least one active cone must be positively colinear.
As we establish next, these colinear neurons can be continuously merged together along a path of constant objective value, resulting in a minimal neural network.
Formally, we let θ ∈ (cid:101)
Θ min m be a nearly minimal neural network with m neurons and we ﬁx ( w, γ ) ∈ θ a non-zero neuron.
Let ( w 2 , γ 2 ) , . . .
, ( w k , γ k ) ∈
θ be the other non-zero neurons such that for each j = 2 , . . .
, k , we have sign ( γ ) =
sign ( γ j ) , and, w and w j are positively colinear.
Write ( w 1 , γ 1 ) :
= ( w, γ ) , and deﬁne the merged neuron ( w m , γ m ) as w m : = (cid:80) kj =1 | γ j | w j (cid:113) (cid:107) (cid:80) kj =1 | γ j | w j (cid:107) 2 and γ m : = sign ( γ ) (cid:107) w m (cid:107) 2 .
Let M ( θ ) be a copy of θ where each such set of k positively colinear neurons { ( w 1 , γ 1 ) , . . .
, ( w k , γ k ) } is replaced by the k neurons { ( w m , γ m ) , (0 , 0) , . . .
We refer to M ( θ ) as the merged version of θ .
The next result states relevant properties of M ( θ ) .
Proposition 1.
Let θ ∈ (cid:101)
Θ min m .
Then, the following results hold.
The merged neural network M ( θ ) is a minimal neural network.
We have θ (cid:73) M ( θ ) , and the continuous path from θ to M ( θ ) has constant objective value.
If M ( θ ) is a local minimum of L β , then θ is also a local minimum of L β .
Intuitively, merging the colinear neurons preserves the active cones and leaves a single neuron per cone, so that M ( θ ) is indeed minimal.
The third property essentially follows from the fact that
M ( θ ) has more degrees of freedom than θ since it has more neurons equal to 0 .
In addition, the continuous path of constant objective value from θ to M ( θ ) can be explicitly constructed (see the proof in Appendix B.1).
M APPING MINIMAL NEURAL NETWORKS TO A CONVEX OPTIMIZATION LANDSCAPE
We provide here an explicit map from the set of minimal neural networks to the feasible set W of the convex program (4), and vice-versa.
For W = ( w 1 , . . .
, w 2 p ) ∈ W , we let (cid:107) W (cid:107) 0 be number of the non-zero vectors in w 1 , . . .
, w 2 p .
Deﬁne
[Equation]
First, we introduce the map θ (cid:55)→ W ( θ ) from Θ min m to W m where for each i = 1 , . . .
, 2 p , we set
[Equation]
and such that each non-zero neuron ( u j , α j ) contributes only to a single w i .
To understand the latter, note that each cone B ( u j , α j ) might be a subset of several (adjacent) cones B i and hence, one might need to choose which w i a neuron ( u j , α j ) contributes to.
These ties can be resolved arbitrarily without affecting any of our results.
Conversely, we construct a map W (cid:55)→ θ ( W ) from W m to Θ min m by setting θ ( W ) = { ( u i , α i ) } mi = where the ( u i , α i ) are deﬁned as follows.
Denote i 1 < · · · < i m the indices such that if i (cid:54)∈ { i 1 , . .
, i m } then w i = 0 .
Take the index J (if any) such that i J (cid:54) p
and i J +1 (cid:62) p + 1 .
Let { K 1 , . . .
, K (cid:96) } be a partition of { i 1 , . .
, i J } in terms of the repartition of w i 1 , . . .
, w i J into the cones { Q 1 , . . .
Q q } .
Similarly, let { K (cid:96) +1 , . . .
, K (cid:96) + (cid:96) (cid:48) } be a partition of { i J +1 , . .
, i m } in terms of the repartition of w i J +1 , . . .
, w i m into the cones { Q 1 , . . .
Q q } .
Then, for 1 (cid:54) j (cid:54) (cid:96) + (cid:96) (cid:48) , we set
[Equation]
where γ j = 1 if j (cid:54) (cid:96) and γ j = − 1 if j > (cid:96) .
Finally, for (cid:96) + (cid:96) (cid:48) + 1 (cid:54)
j
(cid:54) m , we set ( u j , α j ) =
As stated in the next result, these mappings can only improve the training loss.
Proposition 2.
It holds that for any W ∈ W m we have L β ( θ ( W ))
(cid:54)
L cβ ( W ) , and, for any (cid:101) θ ∈ Θ min m , we have L cβ ( W ( (cid:101) θ ))
(cid:54) L β ( (cid:101) θ ) .
Furthermore, it holds that θ ( W ( (cid:101) θ ))
∈ Ω( (cid:101) θ ) .
These mappings between minimal neural networks and the convex feasible set provide a rich structure to address the optimality properties of neural networks.
In Figure 1, we provide an illustration of the Then, we have L β ( u, α ) = (1 − max { u, 0 } α ) 2 + 12 ( | u | 2 + | α | 2 ) . The convex objective is then
L cβ ( v, w ) =
(1 − v + w ) 2 + ( | v | + | w | ) subject to v, w (cid:62) 0 .
The set of minimal neural networks corresponds to | u | = | α | , which includes the optima. Further, the optimal values of the two functions match and are equal to 0 . 75 , and attained at ( u, α
= (
/ √
√ v
= v this
3. T HE GLOBAL OPTIMAL SET OF NEURAL NETWORKS
Let m ∗ =
min W ∈W ∗ (cid:107) W (cid:107) 0 .
As a consequence of Caratheodory’s theorem, we have the following upper bound on the minimal cardinality m ∗ of an optimal solution.
Lemma 1.
It holds that m ∗ (cid:54) n + 1 .
Further, for any m (cid:62) m ∗ , we have that P ∗ m = inf k (cid:62) 1 P ∗ k .
From the deﬁnition of m ∗ , it clearly holds that W ∗ m (cid:54) = ∅ for m (cid:62) m ∗ .
Then, we present the mapping from the optimal solution to the convex problem (4) to a globally optimal neural network for the non-convex problem (1).
Lemma 2.
Let W = ( w 1 , . . .
, w 2 p ) ∈
W ∗ , and denote by I
= { i 1 , . . .
, i (cid:107) W (cid:107) 0 } ⊂ [2 p ] the set of indices such that w ∗
i (cid:54) = 0 for i ∈
I .
We set
[Equation]
for i j ∈
I .
Here γ i
= 1 if i (cid:54) p and γ i = −
if i > p .
Then, it holds that θ = { ( u i , α i ) } (cid:107) W (cid:107)
i = is an optimal neural network, i.e., L β ( θ ) =
P ∗ .
We denote the above mapping (9) by ψ , and we set Θ cvx m = ψ ( W ∗ m ) .
According to Lemma 2, it holds that Θ cvx m ⊆
Θ ∗ m .
Given a neuron ( u, α ) , we say that a collection of neurons { ( u j , α j ) } kj =1 is a splitting of ( u, α ) if ( u j , α j ) =
( √ γ j u, √ γ j α ) for some γ j (cid:62) 0 and (cid:80) kj =1 γ j = 1 .
Given a neural network θ = { ( u i , α i ) } mi , a splitting of θ is any neural network θ (cid:48)
∈
Θ m such that the non-zero neurons of θ (cid:48) can be partitioned into splittings of the neurons of θ .
Similarly, split neurons can be merged back to their original form.
We denote by ˜Θ cvx m the set of splittings generated from Θ cvx m .
We provide an exact characterization of the optimal set in the following theorem.
Theorem 1.
Suppose that m (cid:62) m ∗ .
It holds that Θ ∗ m = ˜Θ cvx m .
Namely, all optimal solutions of the nonconvex loss can be found via the optimal solutions of the convex program (4) up to permutation and splitting/merging of the neurons as deﬁned above.
We compare our result with the result in (Pilanci & Ergen, 2020) as follows.
Essentially, Pilanci &
Ergen (2020) show how to construct one globally optimal solution of the nonconvex loss by solving the convex program, while Theorem 1 shows how to construct the entire set of global optimum of the nonconvex loss.
The relations among W ∗ m , Θ cvx m , ˜Θ cvx m and Θ ∗ m is illustrated in Figure 2.
Example 1.
We consider a toy example, where X = (cid:34) [Equation] , Y = [Equation]
(cid:35) and β = 0 .
In this case, p = 6 and we can enumerate the diagonal matrices D i as
[Equation]
[Figure]
merge
Proposition convexify split
Theorem ReLU [Figure] The optimal solution to the convex problem (4) is given by W ∗ =
( w ∗ 1 , . . .
, w ∗ 2 p ) , where W ∗ only consists of one non-zero block w ∗ 5 =
[0 . 86 , − 0 .
79] T .
Therefore, the set of the global minimizers of the nonconvex loss L β consists of all nearly minimal neural network θ = { ( u i , α i ) } mi =1 satisfying
[Equation]
where (cid:80) mi =1 γ i = 1 and γ i (cid:62) 0 are arbitrary.
These correspond to the split versions of the single neuron w ∗ 5 .
We investigate numerically our result: for m = 5 , we run gradient descent (GD) on the nonconvex loss L β until we ﬁnd a nearly stationary neural network { ( u i , α i ) }
i
=1 .
We plot the points α
i u i as well as w ∗ 5 in Figure 3 in the Appendix.
C HARACTERIZATION OF ALL LOCAL MINIMA
Minimal neural networks form a subset considerably smaller than the entire space of neural networks, and they do contain all the global optima.
In this section, we show that ﬁrst-order methods can ﬁnd networks that can be merged to a minimal representation.
Moreover, we exhibit the existence of a path of strictly decreasing objective value from any neural network to a minimal representation.
This may suggest that minimal neural networks are the right notion to study the complexity of the loss landscape.
4. SGD FINDS A NEARLY MINIMAL NEURAL NETWORK
The limit points of SGD are almost surely Clarke stationary with respect to L β (see, e.g., (Davis et al., 2020; Bolte & Pauwels, 2019)).
We show next that any Clarke stationary point w.r.t.
the loss L ( θ ) is in fact a nearly minimal neural network.
This shows that SGD ﬁnds a neural network which can be merged to a minimal representation.
Theorem 2.
Fix m (cid:62) 1 .
Any Clarke stationary point θ of the non-convex loss function L β over Θ m is a nearly minimal neural network.
Consequently, any local minimum of L β is nearly minimal.
As an additional motivation for studying nearly minimal neural networks, we establish the following.
Proposition 3.
Let θ ∈ Θ m be any neural network.
There exists a continuous path in Θ m from θ to a nearly minimal neural network along which the loss function is (strictly) decreasing.
The proof of Theorem 2 is deferred to Appendix B.4 and that of Proposition 3 to Appendix B.5.
Both proofs are based on the same transformations of a neural network θ which decreases the training loss: scaling the neural network and then aligning the non-zero neurons which belong to the same cones
B i so that they become positively colinear.
These transformations leave the predictions unchanged due to the piecewise linear structure of the activation function but decrease the value of the regularization term.
Thus, our notions of minimal representations are intimately related to (i) the piecewise linear structure of the activation function and (ii) the regularization effect.
We emphasize again that these two features of neural network training are commonly used in practice (e.g., ReLU and weight decay).
Combining Proposition 3 and Proposition 1, we immediately obtain the following result.
Corollary 1.
The valley Ω( θ ) of any neural network θ contains a minimal one.
Further, if the valley Ω( θ ) is non-spurious, then it contains an optimal neural network which is minimal.
Interestingly, we are able to provide an explicit construction of the map from a neural network θ to a nearly minimal representation, and this map is based on the aforementioned transformations (scaling and aligning; see the proof of Proposition 3 for details).
Hence, the study of the optimality properties of a neural network can be narrowed down to the structured class (cid:101)
Θ min m which contains the limit points of SGD.
Next, we establish that we can go further by considering the class of minimal neural networks Θ min m .
4. C LARKE ’ S STATIONARY POINT AND SUBSAMPLED CONVEX PROGRAM
Consider the convex program with trichotomies:
[Equation]
The convex program with trichotomies also provides a convex optimization formulation of the regularized neural network training problem (1).
Proposition 4.
The convex program (12) with trichotomies has the optimal value P ∗ .
Given a subset I ⊆
[ q ] , we can also consider a subsampled convex program with trichotomies: [Equation]
[Equation]
We show the connection of the Clarke’s stationary point of the nonconvex loss function L β and the optimal solution of the subsampled convex program (13) as follows.
Theorem 3.
Suppose that θ is a Clarke’s stationary point of the nonconvex loss function L β .
Let
I = { i ∈ [ q ] | there exists k ∈ [ m ] such that T j = diag(sign( Xu k )) } .
Then, θ corresponds to a global optimum of the subsampled convex program (13) .
In other words, any local minimum of the nonconvex loss (1) can be characterized as a global minimum of a subsampled convex program (13); further, the optimality gap is equal to the gap between the subsampled problem (13) and the full convex program (12).
4. S UBSAMPLED CONVEX PROGRAM AND VERIFYING GLOBAL OPTIMALITY
We established that a stationary point of the non-convex training loss is a global optimum of a subsampled convex program.
Here, we build on this observation to design a procedure to check whether a neural network is in fact a global minimizer.
Our key theoretical contribution is to provide such an algorithm that runs in polynomial time of sample size n .
We ﬁrst note that the set { D i } pi =1 can be constructed in polynomial time of n via standard results from geometry and hyperplane arrangements in (Cover, 1965;
Winder, 1966; Ojha, 2000).
Consider a feasible point W = ( w 1 , . .
w 2 p ) ∈ W of the convex program (4).
Note that each constraint w i ∈
C i is a linear inequality constraint.
Indeed, as described in Section 2, each C i is the convex cone of solution vectors for a dichotomy { I + , I − } of { 1 , . . . , n } .
Writing X ( i )+ (resp.
X ( i ) − ) the subset of rows of X indexed by I + (resp.
I − ), we have u ∈ C i if and only X ( i ) + u (cid:23) 0 and X ( i ) − u (cid:22) 0 .
Using these notations, the convex program (4) can be reformulated as
[Equation]
[Equation]
[Equation]
p , where (cid:98) y c ( W ) =
(cid:80) 2 pi =1 D
i
Xw
i .
Hence, given a feasible point W ∗ = ( w 1 , . . . , w 2 p ) ∈ W to the convex program (4), it holds that W ∗ is a global minimizer if and only if W ∗ satisﬁes the KarushKuhn-Tucker (KKT) conditions (see (Boyd et al., 2004)) of (4).
Here, W ∗ ∈ W satisﬁes the
KKT conditions if, for each i
= 1 , . .
. , 2 p , there exist ζ ( i )+ , ζ ( i ) − (cid:23)
such that (cid:104) ζ ( i )+ , X ( i )+ w ∗ i (cid:105) =
(cid:104) ζ ( i ) − , X
( i ) − w ∗
i (cid:105) =
and
[Equation]
otherwise . [Equation] This amounts to solving a system with 2 np variables of 2 np linear inequalities, n 0 convex quadratic inequalities and (2 p − n 0 )( d + 2) linear equalities, where n 0 is the number of variables w ∗ i equal to 0 , and this can be done efﬁciently using standard convex solvers, in time polynomial in the sample size n .
The next result establishes the link between checking the KKT conditions of the above program and checking whether a neural network is a global optimum.
Its proof is deferred to Appendix B.8.
Proposition 5.
Let (cid:101) θ ∈ Θ min m be a minimal neural network.
Suppose that W ( (cid:101) θ ) satisﬁes the KKT conditions as described above.
Then, θ ( W ( (cid:101) θ )) is a global optimum of the loss L β .
In the above result, the minimal neural network assumption is not restrictive, since any local minima of the loss must be a nearly minimal neural network (Theorem 2), and then, any nearly minimal neural network can be reduced to a minimal one along a continuous path of constant value (Theorem 1).
N ON - SPURIOUS VALLEYS AND CONVEX LANDSCAPE
The subsampled convex program relates to the optima of SGD.
It is then of interest to understand the landscape when m is much smaller than the number of cones.
Here we show that a critical threshold is n + 1 + m ∗ for having a path of non-increasing value.
While this may be an open problem, it is reasonable to expect SGD to behave better in that case, and thus to ﬁnd a global minimum.
For an arbitrary neural network θ , we can ﬁnd a point θ (cid:48) with at most n + 1 non-zero neurons such that they are connected with a path with constant objective values.
Proposition 6.
Given a scaled neural network θ ∈ Θ m with m (cid:62) n + 1 , there exists a neural network θ (cid:48) with at most n + 1 non-zero neurons and there exists a path with constant objective value between θ and θ (cid:48) .
Namely, θ (cid:73) θ (cid:48) and θ (cid:48) (cid:73) θ .
A direct corollary of Proposition 6 is that for any global optimum θ ∗ of L β , we can ﬁnd a succinct representation ˜ θ ∗ with at most n + 1 non-zero neurons and there exists a path between ˜ θ ∗ and θ ∗ such that the objective value is constant.
Based on Proposition 6, we can also show that there is no spurious valley.
The following result states the absence of spurious valleys for the training loss as soon as m (cid:38) n .
Proposition 7.
Let m (cid:62) n + 1 + m ∗ .
Then, it holds that for any neural network θ ∈ Θ m , we have θ (cid:73) θ ∗ for some θ
∗ ∈
Θ ∗ m .
In other words, provided that m (cid:62) n + 1 + m ∗ , all strict local minima are global .
Compared to the standard lower bound m (cid:62) n for the unregularized case in (Venturi et al., 2019; Livni et al., 2014), we have an additional term m ∗ (cid:54) n + 1 induced by weight decay.
As known in the literature (Freeman & Bruna, 2016; Venturi et al., 2019; Vidal et al., 2017), the loss landscape of an over-parameterized shallow neural network is almost convex.
Essentially, for a sufﬁciently wide neural network, for any θ 0 , θ 1 and λ ∈ [0 , 1] , we can ﬁnd θ λ such that f ( x ; θ λ )
=
λf ( x ; θ 0 )
+ (1 − λ )
f ( x ; θ 1 ) .
From a perspective of convex formulation of two layer neural network, we give a sufﬁcient upper bound on the width of neural network to ensure the convex landscape in terms of realizations.
Essentially, as long as m (cid:62) 2( n + 1) , for any two neural network realizations f ( x ; θ 0 ) and f ( x ; θ 0 ) , we can ﬁnd succinct representations ˜ θ 1 , ˜ θ 2 such that θ i (cid:73) ˜
θ i and ˜
θ i (cid:73)
θ i for i = 1 , 2 .
Then, for any λ ∈ [0 , 1] , we can construct θ λ such that
[Equation]
The construction of θ λ is straightforward.
From Proposition 6, we can take ˜ θ i as a neural network with at most n + 1 non-zero neurons for i = 1 , 2 .
Given m (cid:62) 2( n + 1) , following the proof of
Proposition 7, we can construct θ λ satisfying (16).
C ONCLUSION
We present a methodology to ﬁnd all global solutions to the training problem of two-layer ReLU networks via solving a convex optimization problem.
We provide an exact characterization of globally optimal solutions, and local minimizers in terms of convex problems.
We provide a procedure to generate the entire global optimum.
A polynomial-time algorithm is presented to examine whether a neural network is globally optimal to the nonconvex training loss.
A current limitation is that the results apply to two-layer networks.
However, we conjecture that our results can be extended to arbitrary depth with appropriate convex formulations.
A F IGURE IN E XAMPLE
We plot the points α i u i as well as w ∗ 5 in Figure 3.
B P ROOFS OF MAIN RESULTS
Thoughout the appendix, we will use the following notations.
For W = ( w 1 , . . .
, w 2 p ) , we deﬁne (cid:98) y c
( W ) =
(cid:80) 2 pi =1 D
i
Xw
i .
For θ =
( U, α ) , where U ∈ R d × m and α ∈ R m , we deﬁne (cid:98) y
( θ ) = (cid:80) mj
=1 σ ( Xu j ) α j .
B. P ROOF OF P ROPOSITION
According to the construction of M ( θ ) , there is at most one non-zero neuron per cone, and, each neuron ( w, γ ) satisﬁes (cid:107) w (cid:107) =
| γ | , i.e., M ( θ ) is minimal.
Fix a cone B .
Let ( w 1 , γ 1 ) , . . .
, ( w k , γ k ) be the neurons of θ such that B ( u i , w i )
=
B for i = 1 , . . .
, k , and let w m = (cid:80) kj =1 | γ j | w j (cid:113)(cid:80)
kj =1 | γ j | w j and γ m = sign ( γ 1 ) · (cid:112) (cid:107) w m (cid:107) 2 be the merged neuron.
Since θ is nearly minimal, we know that w 1 , . .
, w k are positively colinear.
For t ∈ [0 , 1] , deﬁne θ ( t ) such that it has k neurons associated with the cone B given by
[Equation]
where j (cid:62) 2 .
Note that for all j (cid:62) 1 , all vectors w j ( t ) , w j , w m
are positively colinear, that (cid:107) w j ( t ) (cid:107) 2 =
|
γ j ( t ) | and that sign ( γ j ( t ))
=
sign ( γ 1 ) .
Further, note that ( w 1 (0) , γ 1 (0))
= ( w 1 , γ 1 ) , ( w 1 (1) , γ 1 (1)) =
( w m , γ m ) and for j (cid:62) 2 , ( w j (0) , γ j (0))
= ( w j , γ j ) , ( w j (1) , γ j (1)) =
Consider the neural networks θ ( t ) with neurons ( w j ( t ) , γ j ( t )) respectively deﬁned for each cone
B .
It holds that θ (0)
= θ and θ (1) =
M ( θ ) .
Then, the contribution of the neurons in B to the predictions (cid:98)
y
( θ ( t )) are given by
[Equation]
γ )
[Equation]
σ
[Equation]
γ
[Equation] [Equation]
Xw
[Equation]
γ
[Equation]
Thus, the predictions (cid:98) y
( θ ( t )) are constant as a function of t .
Similarly, we claim that the regularization term R ( θ ( t )) is constant as a function of t .
Since the neurons are scaled, the contribution of the cone
B to the regularization term is given by (up to the constant β )
[Equation]
where the third equality follows from the triangular equality when all vectors are positively colinear.
Thus, we have explicited a continuous path from θ to M ( θ ) such that L β is constant along that path.
Now, we show that if M ( θ ) is a local minimum then θ is also a local minimum.
We proceed with the converse.
Assume that θ is not a local minimum of L β .
For a cone B , let ( w 1 , γ 1 ) , . . .
, ( w k , γ k ) be the neurons of θ such that B ( w i , γ i ) =
B .
Let ( w m , γ m ) be the merged neuron.
If θ is not a local minimum, then there exists a small perturbation θ ε : = { ( u εi , α εi ) } mi =1 of the neurons of θ such that (i)
L β ( θ ε ) <
L β ( θ ) , (ii) sign ( γ εi ) =
sign ( γ i ) and (iii) for each i = 1 , . . .
, m , we have I + ( σ ( Xu i )) ⊆
I + ( σ ( Xu εi ))
and I − ( σ ( Xu i )) ⊆
I − ( σ ( Xu εi )) , where we use the notation
I + ( z ) : = { i ∈ { 1 , . . . , n } | z i > 0 } and I − ( z ) : =
{ i ∈ { 1 , . . . , n } |
z
i < 0 } for a vector z ∈ R n .
Then, we deﬁne for t ∈ [0 , 1] ,
[Equation]
where j (cid:62) 2 .
Let θ ( t ) the neural network with neurons ( w j ( t ) , γ j ( t )) deﬁned as above for each cone
B .
Then, the contribution of a cone B to the predictions (cid:98) y
( θ ( t )) is given by k
[Equation]
Due to the above property (iii), we have that
[Equation]
Thus
[Equation]
the contribution of the cone B to the predictions is
[Equation]
Summing over all the cones, we ﬁnd that
[Equation]
Similarly
[Equation]
the contribution of the cone B to the regularization term R ( θ ( t )) is k
[Equation]
where the last inequality is due to the triangular inequality.
Thus, we obtain that
[Equation]
Hence, we get that L β ( θ ( t ))
(cid:54) (1 − t )
L β ( θ ε )+ t L β ( M ( θ )) .
Since L β ( θ ε ) <
L β ( θ ) =
L β ( M ( θ )) , we have that L β ( θ ( t )) <
L β ( M ( θ )) for any t < 1 .
This concludes the proof.
B. P ROOF OF P ROPOSITION
Let θ ∈ Θ m , and consider the point W ( θ ) , as deﬁned in (7), whose expression is given by
[Equation]
and such that each non-zero neuron ( u j , α j ) contributes only to a single w i ( θ ) .
We prove that the mapping θ (cid:55)→ W ( θ ) is well-deﬁned.
Each set P i is a cone and w i ( θ ) is a positive linear combination of elements of P i .
It follows that w i ( θ ) ∈ P i , and W ( θ ) ∈ W .
Further, θ has m neurons and each neuron ( u j , α j ) contributes only to a single w i ( θ ) .
It follows that at most m variables among { w 1 ( θ ) , . . .
, w 2 p ( θ ) } are non-zero, and W ( θ ) ∈ W m .
Hence, the mapping θ (cid:55)→ W ( θ ) is well-deﬁned from Θ m to W m .
We show that L cβ ( W ( θ )) (cid:54) L β ( θ ) .
We note that
[Equation]
Note that for a neuron ( u j , α j ) such that B ( u j , α j ) ⊆
C i , we have that
D
i X | α j | u j = σ ( Xu j ) α j .
It implies that
[Equation]
and consequently, (cid:96) ( (cid:98) y c
( W ( θ )))
= (cid:96) ( (cid:98) y
( θ )) .
On the other hand, we have
[Equation]
where the last inequality follows from triangular inequality.
Since the neurons ( u j , α j ) are scaled, we get that (cid:80) m j =1 | α j |(cid:107) u j (cid:107) 2 = 1
(cid:80) m j =1 |
α j | 2 + (cid:107) u j (cid:107) 22 , and we ﬁnally obtain that L cβ ( W ( θ ))
(cid:54)
L β ( θ ) .
We show that the mapping W (cid:55)→ θ ( W ) is well-deﬁned.
Based on the construction (8), it holds that the non-zero neurons ( u j , α j ) belong to pairwise distinct cones in { B 1 , . . .
, B 2 q } .
Further, the neurons are scaled.
Hence, θ ( W ) is a minimal neural network with m neurons.
We prove that L β ( θ ( W )) (cid:54)
L cβ ( W ) .
Denote by D ( j ) the diagonal matrix associated with B ( u j , α j ) for 1 (cid:54) j (cid:54) l + l (cid:48) .
We have that
[Equation]
=
[Equation]
It holds by construction that for each i ∈ K j , D ( j )
Xw i
= D i
Xw
i .
Therefore, we ﬁnd that (cid:98) y
( θ ( W ))
=
(cid:98)
y c
( W ) and (cid:96) ( (cid:98) y
( θ ( W )))
= (cid:96) ( (cid:98) y c
( W )) .
On the other hand, we have that m
[Equation]
j (cid:107)
[Equation]
= 2 p (cid:88) i
= (cid:107) v i (cid:107) 2 , where inequality (i) follows from triangular inequality.
Hence, we obtain that L β ( θ ( v ))
(cid:54)
L c β ( v ) .
B. P ROOF OF T HEOREM
First, we show that ˜Θ cvx m ⊆
Θ ∗ m .
Let θ ∈ Θ cvx m , and consider θ (cid:48) a split version of θ .
Let ( u, α ) be a neuron of θ , and { ( u j , α j ) } kj =
the neurons of θ (cid:48) which correspond to the split of ( u, α ) .
We have (cid:80) mj
=1 σ ( Xu j ) α j = (cid:80) mj
=
γ j σ ( Xu )
α = σ ( Xu ) α because (cid:80) kj =1 γ j = 1 .
Furthermore, 12 (cid:80) kj =1 (cid:107) u j (cid:107)
+ | α j | 2 = 12 (cid:80) kj =
1 γ j ( (cid:107) u (cid:107)
+ | α | 2 )
=
12 ( (cid:107) u (cid:107)
+
| α | 2 ) , whence L ( θ ) =
L ( θ (cid:48) ) .
Consequently, ˜Θ cvx m ⊆
Θ ∗ m .
It remains to show that Θ ∗ m ⊆ ˜Θ cvx m .
Let θ ∈ Θ ∗ m .
Due to the strong convexity of the regularization term and the re-scaling invariance of the term σ ( Xu j ) α j , we must have that (cid:107) u j (cid:107) 2 = | α j | for each neuron ( u j , α j ) of θ .
We partition the neurons of θ such that the neurons in each partition { ( u i , α i ) } ki =1 belong to the same cone C (in the sense that u i α i ∈ C ), and the cones are pairwise distinct across partitions.
Due to the regularization term, it is straightforward to show that the neurons must be positively colinear, and that this corresponds to a split.
Thus, Θ m ⊆ ˜Θ cvx m and this concludes the proof.
B. P ROOF OF T HEOREM 2: C LARKE STATIONARY POINTS ARE NEARLY MINIMAL NEURAL NETWORKS
We review the deﬁnition of the Clarke subdifferential Clarke (1975) of f .
At x ∈ R d , this is deﬁned as
[Equation]
where D : = { x ∈ R d | f differentiable at x } .
In particular, it holds that R d \ D has measure equal to zero Borwein & Lewis (2010) under mild assumptions on f .
Then, we say that x ∈ R d is Clarke stationary with respect to f if 0 ∈ ∂
C f ( x ) .
Let θ ∈ Θ m be Clarke stationary, i.e., there exist λ (1) , . . .
, λ ( N ) > 0 and sequences { θ (1) k } k , . .
. , { θ ( N ) k } k such that (cid:80) Nj =1 λ ( j ) = 1 , lim k →∞ θ ( j ) k
= θ for each j = 1 , . . .
, N , the loss function L β is differentiable at each θ ( j ) k and
[Equation]
P ART 1: T HE NEURAL NETWORK θ MUST BE SCALED .
By contradiction, we assume ﬁrst that the neural network θ is unscaled, i.e., there exists a neuron ( u, α ) such that (cid:107) u (cid:107) 2 (cid:54) =
| α | .
Write ( u ( j ) k , α ( j ) k ) the corresponding neuron of each θ ( j ) k .
Since lim k →∞ ( u ( j ) k , α ( j ) k ) = ( u, α ) , up to extracting subsequences, we can assume that the neurons ( u ( j ) k , α ( j ) k ) are also unscaled, i.e., (cid:107) u ( j ) k (cid:107) 2 (cid:54) =
| α ( j ) k | for all k (cid:62) 1 and j = 1 , . . .
, N .
C ASE 1: α (cid:54) =
Since lim k →∞ α ( j ) k
= α , up to extracting subsequences, we can assume that α ( j ) k (cid:54) =
for all k (cid:62)and j = 1 , . . .
, N .
Then, for each j = 1 , . .
. , N and k (cid:62) 1 and for t ∈ [0 , 1] , we deﬁne the neural network θ ( j ) k ( t ) as a copy of θ ( j ) k except for the neuron ( u ( j ) k , α ( j ) k ) that we replace by
[Equation]
where γ ( j ) k ∗ = (cid:114) (cid:107) u
( j ) k (cid:107)| α ( j ) k
| , γ ( j ) k ( t ) = 1 + t ( γ ( j ) k ∗ − 1)
and we use the improper notation u ( j ) k γ ( j ) k ( t )
=if u ( j ) k
= 0 .
Note that θ ( j )
k
( t ) deﬁnes a continuous path from θ ( j ) k
= θ ( j ) k (0) to the scaled neural network θ ( j ) k (1) .
Further, since σ is positively homogeneous, it holds that for any t ∈ [0 , 1] ,
[Equation]
so that that the function L β ( θ ( j ) k ( t )) is constant as a function of t ∈ [0 , 1] .
On the other hand, the regularization term satisﬁes
[Equation]
Note that the function g ( j ) k ( t ) :
= L β ( θ ( j ) k ( t )) is differentiable, and simple algebra yields
[Equation]
Hence, lim
[Equation]
(cid:107) 22 ) .
Since | α | (cid:54) =
(cid:107) u (cid:107) 2 , it follows that lim k ∞
d g ( j ) k d t (0) < 0 .
On the other hand, we have that
[Equation]
Simple algebra yields that lim k ∞ d u ( j ) k
( t =0)
d t
[Equation]
and lim
[Equation]
(cid:16)(cid:113) (cid:107) u (cid:107) 2 | α | − 1 (cid:17) · α .
Thus, the limit d θ : = lim k ∞
d θ ( j ) k d t (0) is constant (independent of the index j ) and
N (cid:88)
[Equation]
This is contradiction with the fact that (cid:80) N j =1 λ ( j ) lim k →∞ d g ( j ) k d t (0) < 0 .
Therefore, in the case u (cid:54) =
and α (cid:54) = 0 , we must have that (cid:107) u (cid:107)
=
| α | .
C
[Equation] [Equation]
u
(cid:54)
[Equation]
The proof proceeds exactly in the same way, except that we deﬁne
[Equation]
where γ ( j ) k ∗ = (cid:114)
| α ( j ) k
| (cid:107) u
( j ) k (cid:107) 2 , γ ( j ) k ( t )
= 1 + t ( γ ( j ) k ∗ − 1)
and we use the convention α ( j ) k γ ( j ) k ( t )
=
if α ( j ) k = 0 .
P ART 2: N ON - ZERO NEURONS WHICH SHARE THE SAME ACTIVATION CONE ARE POSITIVELY COLINEAR
According to the ﬁrst part of the proof, we can assume that the neural network θ is scaled.
(S PECIAL CASE ) T HE NEURAL NETWORK θ IS A DIFFERENTIABLE POINT OF L β .
In order to provide some intuition about the proof, let us assume ﬁrst that L β is differentiable at θ .
By contradiction, we suppose that there exist two non-zero neurons ( u, α ) and ( v, β ) such that
B ( u, α ) =
B ( v, β ) , and, u and v are not positively colinear.
Further, let us assume that α, β >(the case α, β < 0 follows the same lines).
Deﬁne w : = αu + βv .
Note that w has the same sign pattern as u and v .
For t ∈ [0 , 1] , we set
[Equation]
Note that B ( u ( t ) , α ( t ))
=
B ( u, α ) =
B ( v, β ) =
B ( v ( t ) , β ( t )) .
Further, we deﬁne θ ( t ) as a copy of θ where we replace the two neurons ( u, α ) and ( v, β ) by ( u ( t ) , α ( t )) and ( v ( t ) , β ( t )) .
Note that θ ( t ) deﬁnes a continuous path in Θ m starting at θ .
Then, we introduce the two functions
[Equation]
so that L β ( θ ( t )) =
g ( t ) + β · h ( t ) .
First, we claim that g ( t ) is constant over [0 , 1] .
Indeed, this follows from the fact
[Equation]
where the ﬁrst equality comes from the fact that B ( u ( t ) , α ( t ))
=
B ( v ( t ) , β ( t )) .
Hence, we have (cid:98) y
( θ ( t )) =
(cid:98) y
( θ ) and g ( t ) is constant.
On the other hand, the function h ( t ) is clearly differentiable, and simple algebra yields that
[Equation]
Since u and v are not colinear, it holds by Cauchy-Schwarz inequality that ( αu ) (cid:62) ( βv ) < (cid:107) αu (cid:107) 2 (cid:107) βv
(cid:107) 2 , and thus,
[Equation]
βv (cid:107) (cid:18) (cid:107) αu (cid:107)
+ (cid:107) βv
(cid:107)(cid:19) =
, that is, d h (0)d t < 0 .
Thus, we ﬁnally obtain that d L β ( θ (0))
d t < 0 , which contradicts the stationarity of θ .
(G ENERAL CASE ) T HE NEURAL NETWORK θ IS NOT NECESSARILY A DIFFERENTIABLE POINT OF L β .
Now, let us generalize the above proof to the case where L β is not necessarily differentiable at θ .
For a vector z ∈ R n , we use the notations I + ( z ) :
= { i ∈ { 1 , . . .
, n } | z i > 0 } , I 0 ( z ) :
= { i ∈ { 1 , . . .
, n } | z i = 0 } and I − ( z ) : = { i ∈ { 1 , . . . , n } |
z i < 0 } .
Since θ is a Clarke stationary point of L β , we know that there exist λ (1) , . . .
, λ ( N ) > 0 and sequences { θ (1) k } k , . .
. , { θ ( N ) k } k such that (cid:80) Nj =1 λ ( j ) = 1 , lim k →∞ θ ( j ) k
= θ for each j = 1 , . . .
, N , the loss function L β is differentiable at each θ ( j ) k and 0 =
N (cid:88)
[Equation]
=
[Equation] [Equation]
[Equation] [Equation]
For each k (cid:62)
and j = 1 , . .
, N , up to extracting subsequences, we can assume that u ( j ) k , α ( j ) k , v ( j ) k , β ( j ) k (cid:54) = 0 , and, α ( j ) k and β ( j ) k have the same sign (let us say positive).
Further, up to extracting subsequences again, we can assume that the sign patterns I + ( Xu ( j ) k ) and I − ( Xu ( j ) k ))
(resp.
I + ( Xv ( j ) k ) and I − ( Xv ( j ) k )) ) remain constant (independent of k ).
Since the sign patterns of
Xu and Xv are equal by assumption, and, since lim k ∞ u ( j ) k
= u and lim k ∞
v ( j ) k
= v , it follows that
[Equation]
18) and
[Equation]
We denote T ( j ) ( u ) and T ( j ) ( v ) the diagonal matrices (as introduced in Section 2) which correspond to the sign patterns of u ( j ) k and v ( j ) k , and which are independent of k by assumption.
Then, using (18) and (19), it follows that
[Equation]
The above equalities will be crucial later on in our analysis.
Then, for each neural network θ ( j ) k , we can construct a similar path θ ( j )
k ( t ) as in the differentiable case, that is, we set w ( j ) k : = α ( j ) k u ( j ) k + β ( j ) k v ( j ) k , and (cid:101) u ( j )
[Equation]
Similarly to the differentiable case
[Equation]
we also deﬁne the functions
[Equation]
First, we claim that lim k →∞ d g ( j ) k (0) d t = 0 .
Indeed, we have
[Equation]
Taking the limit k → ∞
[Equation]
we obtain that
[Equation]
, ∇
[Equation]
Using (20), we get that ( T ( j ) ( u ) − T ( j ) ( v ))
X ( v − u ) = 0 , and consequently, the claimed equality lim k →∞ d g
[Equation]
j
[Equation]
k
[Equation] [Equation]
d t = 0 .
On the other hand, the function h ( j ) k ( t ) is clearly differentiable, and simple algebra yields that
[Equation]
Since u and v are not colinear, it holds by Cauchy-Schwarz inequality that ( αu ) (cid:62) ( βv ) < (cid:107) αu
[Equation]
2 (cid:107) βv
(cid:107) 2 , and thus, lim
[Equation]
that is, lim k →∞ d h ( j ) k
[Equation] [Equation]
< 0 .
Thus, we ﬁnally obtain that
[Equation]
and further, that
[Equation]
<
[Equation]
However, it holds that
[Equation]
It is immediate to see that
[Equation]
does not depend on the index j , so that
[Equation]
That is, we obtained both that
[Equation]
< and
[Equation]
= 0 , which is a contradiction.
This concludes the proof that θ must be a nearly minimal neural network.
B. P ROOF OF P ROPOSITION 3: R EDUCTION TO N EARLY M INIMAL N EURAL N ETWORKS ALONG A P ATH OF D ECREASING O BJECTIVE V ALUE
We consider reductions similar to those in the proof of Theorem 2, in order to construct a path θ ( t ) ∈
Θ m for t ∈ [0 , 1] such that θ (0)
= θ , θ (1) ∈ (cid:101)
Θ min m and L β ( θ ( t )) is strictly decreasing.
Naturally, we assume that θ is not nearly minimal, otherwise, there is nothing to show.
P ART 1: T HE NEURAL NETWORK θ IS UNSCALED .
We claim that there exists a path θ ( t ) ∈
Θ m for t ∈ [0 , 1] such that θ (0)
= θ , θ (1) is scaled and
L β ( θ ( t )) is strictly decreasing.
Suppose that the neural network is unscaled (if not, go directly to Part 2).
Then, for each neuron ( u, α ) of θ such that (cid:107) u (cid:107) 2 (cid:54) = | α | , deﬁne u (
[Equation]
= (cid:40)(cid:112)
[Equation]
if u, α (cid:54) = 0 , 0 otherwise , α
[Equation]
(cid:40)
[Equation]
if α (cid:54) = 0 , 0 otherwise , where γ ( t ) = (cid:112) | α | + t ( (cid:112) (cid:107) u (cid:107) 2 − (cid:112) | α | ) .
Simple algebra yields that ( u (0) , α (0)) =
( u, α ) and (cid:107) u (1) (cid:107) 2 =
(cid:112) | α |(cid:107) u (cid:107) 2 = | α (1) | ,
so that θ (0) = θ and θ (1) is scaled.
By positive homogeneity of σ , we have that σ ( Xu ( t ))
α ( t ) =
σ ( Xu ) α , which further implies that (cid:98)
y
( θ ( t )) =
(cid:98) y
( θ ) and L ( (cid:98) y
( θ ( t ))) is constant as a function of t .
We claim that the regularization term R ( θ ( t )) is strictly decreasing as a function of t .
Indeed, it holds that
[Equation]
The minimizer of the function γ ∈ R (cid:55)−→ | α |
[Equation] [Equation]
u
[Equation]
22 + γ 2 |
α | | α | 2 is given by γ ∗ = (cid:112) (cid:107) u (cid:107) 2 , which is also equal to γ (1) , and the minimal value of the latter function is given by 2 (cid:107) u (cid:107) 2 |
α | , which is strictly smaller than (cid:107) u (cid:107) 22 + | α | 2 since (cid:107) u (cid:107) 2 (cid:54) =
| α | .
Thus, the function t (cid:55)→ R ( θ ( t )) is minimized at t = 1 , and R ( θ (1)) <
R ( θ ) .
Lastly, observe that t (cid:55)→ R ( θ ( t )) is a convex function, which implies that it must be strictly decreasing over [0 , 1] .
This concludes the ﬁrst part of the proof.
P ART 2: T HE NEURAL NETWORK θ IS SCALED BUT NOT NEARLY MINIMAL .
If the neural network θ is scaled but not nearly minimal, we claim that there exists a continuous path θ ( t ) for t ∈ [0 , 1] such that θ (0)
= θ , θ (1) is nearly minimal, and L β ( θ ( t )) is strictly decreasing.
For each cone B ∈ { B 1 , . . .
, B 2 q } , we consider the non-zero neurons U B : = { ( u, α ) } of θ such that B ( u, α )
=
B .
By assumption, there exists at least one cone B such that U B has more than two elements which are not positively colinear.
Then, for each cone B , we set w = (cid:80) ( u,α ) ∈U B | α | u , and, for each ( u, α ) ∈
U B and for t ∈ [0 , 1] ,
[Equation]
where |U B | is the cardinality of the set U B .
Note that B ( u ( t ) , α ( t ))
=
B ( u, α ) =
B , and each neuron ( u ( t ) , α ( t )) is scaled.
Further, we deﬁne θ ( t ) the neural network with neurons ( u ( t ) , α ( t )) .
It holds that θ ( t ) deﬁnes a continuous path in Θ m starting at θ , and ending at a nearly minimal neural network.
Then, we introduce the two function g ( t ) =
L ( (cid:98) y
( θ ( t ))) and h ( t ) :
=
R ( θ ( t )) , so that
L β ( θ ( t )) =
g ( t ) + β · h ( t ) .
First, we claim that g ( t ) is constant over [0 , 1] .
Indeed, this comes from the fact that for each cone B ,
[Equation]
The ﬁrst (resp. third) equality holds from the fact that the neurons ( u ( t ) , α ( t )) (resp.
( u, α ) ) have the same active cone B .
Thus, (cid:98) y
( θ ( t )) =
(cid:98) y
( θ ) and g ( t ) is indeed constant.
On the other hand, we claim that the function h ( t ) is strictly decreasing.
Indeed, observe ﬁrst that
[Equation]
where the second equality holds since the neurons ( u ( t ) , α ( t )) are scaled.
Thus, it is immediate to verify that the function h is differentiable, and
[Equation]
Clearly, h (cid:48) ( t ) is strictly increasing (since there exists, by assumption, at least one cone B and a neuron ( u, α
) ∈
U B such that w |U B | (cid:54) =
| α | u ).
Therefore, it sufﬁces to verify that h (cid:48) (1) (cid:54) 0 .
Simple algebra yields actually that h (cid:48) (1) = 0 , which concludes the proof.
B. P ROOF OF P ROPOSITION
The proof with trichotomies is almost identical to the proof of dichotomies in Pilanci & Ergen (2020);
Sahiner et al.
We start with the dual representation of P ∗ :
[Equation] [Equation]
Here (cid:96) ∗ ( λ ) :
= max v { λ T v − (cid:96) ( v ) } is the Fenchel conjugate function of (cid:96) .
We note that the singlesided dual constraint has an equivalent formulation using trichotomies:
[Equation]
Similarly, the other side of the dual constraint can be formulated as
[Equation]
Therefore, we can rewrite P ∗ as
[Equation]
For simplicity [Equation] we denote T j + q = T j for j ∈ [ q ] .
We now formulate the Lagrangian
[Equation]
By Sion’s minimax theorem [Equation] we can switch the max and min [Equation]
and then minimize over λ .
Following this, we obtain
[Equation]
By rescaling the variable w j = ν
j w j
[Equation]
we can reformulate P ∗ as
[Equation]
Minimizing with respect to ν yields
[Equation]
This completes the proof.
B. P ROOF OF T HEOREM
According to Proposition 1 and
[Equation]
we can assume that θ is a minimal neural network.
Denote ˜ λ = ∇ (cid:96) (cid:16)(cid:80) m j =1 ( Xu j ) + α j (cid:17) .
From the deﬁnition of Clarke’s stationary point, for j ∈ [ m ] with u j (cid:54) = 0 , we have
[Equation]
The ﬁrst line in (21) is equivalent to that there exists δ j ∈ [0 , 1] N such that [Equation]
[Equation]
Here ˜ D
j = diag( I ( Xu j (cid:62) 0)) and
˜ S j = diag( I ( Xu j = 0)) .
As u j (cid:54) = 0 and α j (cid:54) = 0 , this implies that
[Equation]
For the second line in (21), we can also rewrite it as
[Equation]
Therefore, we have (cid:107) u j (cid:107)
=
| α j | and
[Equation] [Equation]
For the subsampled convex program (13), the KKT conditions are given by: for i ∈
I , there exists ζ ( i ) (cid:23) 0 and ξ ( i ) such that
[Equation]
Here S i is a diagonal matrix satisfying that ( S i )
jj = 1 if j ∈ I 0 and ( S i )
jj =
if j ∈ I + ∪ I − , where { I + , I 0 , I − } is the i -th trichotomy.
The vector λ ∈ R N is deﬁned as λ = ∇ (cid:96) (cid:0)(cid:80) i ∈I
( T i )
+ X
( w i − w i + q ) (cid:1) .
As θ is a minimal neural network, there exists a bijective mapping between non-zero neurons ( u j , α j ) and i ∈ I .
For i ∈ I , suppose that T i =
diag(sign( Xu j )) .
If α j > 0 , we let w i = α j u j , w i + q = 0 , otherwise, we let
[Equation]
As the mapping between non-zero neurons ( u j , α j ) and i ∈ I is bijective, we note that (cid:80) mj
=
( Xu j ) + α j = (cid:80) i ∈I
X T ¯ D i ( w i − w i + q ) .
This implies that ˜ λ
= λ .
On the other hand, by taking ζ ( i ) = 0 , ξ ( i ) =
diag( δ j )˜ λ, ζ
( i + q ) = 0 , ξ ( i + q ) =
− diag( δ j )˜ λ , as ˜ D j = ( T i ) + and ˜ S
j = S i , the KKT conditions (26) are satisﬁed.
Therefore, W = { w i , w i + q | i ∈ I} is a global optimum of the subsampled convex program (13).
B. P ROOF OF P ROPOSITION
Let (cid:101) θ ∈ Θ m be a minimal neural network, and suppose that W ( (cid:101) θ ) satisﬁes the KKT conditions of the optimization problem (4).
Since the latter is a convex optimization problem, it follows that W ( (cid:101) θ ) is a global minimum.
From Proposition 2, we have that P ∗ (cid:54)
L β ( θ ( W ( (cid:101) θ )))
(cid:54) L cβ ( W ( (cid:101) θ ))
=
P ∗ c =
P ∗ , it follows that L β ( θ ( W ( (cid:101) θ ))
=
P ∗ and θ ( W ( (cid:101) θ )) is a global minimizer of L β , which yields the claimed result.
B. P ROOF OF P ROPOSITION
Without loss of generality, we can assume that θ is scaled.
Otherwise, we know from the proof of Proposition 3 that θ can be reduced to a scaled neural network along a continuous path of non-increasing training loss.
We follow the same steps as in the proof of Lemma 1.
Denote the neurons of θ by ( u 1 , α 1 ) , . . .
, ( u m , α m ) .
We have that (cid:98) y
( θ ) = (cid:80) mi =1 (cid:101)
λ i z i , where z
i = sign
[Equation]
|
[Equation]
X
[Equation]
(cid:17) and (cid:101) λ i =
[Equation]
i |
[Equation]
=
[Equation]
u j (cid:107)| α j | .
Thus, (cid:98) y ( θ ) ∈
Conv { z 1 , . .
z m } .
From Lemma 3, we know that there exist i 1 , . .
, i n +1 and λ 1 , . . .
, λ n +1 (cid:62)
such that (cid:80) n +1 j =1 λ j =and (cid:98) y
( θ ) = (cid:80) n +1 j =1 λ j z
i j .
Plugging-in the expressions of the z i j , it follows that
[Equation]
where
[Equation]
Further, we have that
[Equation]
where the last equality follows from the fact that (cid:80) n +1 j =1 λ j = 1 .
Setting (cid:101) θ the neural network with neurons ( (cid:101) u i j , (cid:101) α i j ) for j = 1 , . . . , n + 1 and ( (cid:101) u i , (cid:101) α i ) =
(0 , 0) for i ∈ { 1 , . . .
, m } \ { i 1 , . . .
, i n +1 } , we obtain that (cid:101) θ ∈ Θ m and L β ( (cid:101) θ ) (cid:54)
L β ( θ ) .
Now, we deﬁne a continuous path between θ and (cid:101) θ , as follows.
For t ∈ [0 , 1] , j = 1 , . . .
, n + 1 and i ∈ { 1 , . . .
, m } \ { i 1 , . . .
, i n +1 } , we set u i j
[Equation]
t ) =
[Equation]
u
[Equation]
, and θ ( t ) the neural network with neurons { ( u i ( t ) , α i ( t )) } mi =1 .
Note that θ ( t ) is scaled, and m (cid:88)
i =
(cid:107) u
i ( t ) (cid:107)| α i ( t ) | = n +1 (cid:88)
j = (cid:107) (1 − t ) u
i j | α i j |
[Equation]
t (cid:101) u
i j | (cid:101) α i j |(cid:107) + (cid:88) i =1 ,...,n +1 i (cid:54)
=
i 1 ,...,i n + (1 − t ) (cid:107) u
i (cid:107)| α i | =
( i ) (1 − t ) n +1 (cid:88)
j = | α i j |(cid:107) u i j (cid:107)
+ (1 − t ) (cid:88)
i =1 ,...,n +1 i (cid:54)
=
i 1 ,...,i n + (cid:107) u
i (cid:107)| α i | + t n +1 (cid:88)
j = | (cid:101) α i j |(cid:107) u
i j (cid:107) =
( ii ) (1 − t ) R
( θ ) + t R ( (cid:101) θ ) = ( iii ) R ( θ ) , where equality (i) follows from the triangular inequality and the fact that (cid:101) u i j and u i j are positively colinear; equality (ii) follows from the fact that (cid:101) θ and θ are scaled; equality (iii) holds since R ( θ ) =
R ( (cid:101) θ ) .
Thus, the function t (cid:55)→ R ( θ ( t )) is constant over [0 , 1] .
On the other hand, we have
[Equation]
, where equality (i) holds since the u i j and (cid:101) u
i j are positively colinear and the α i j and (cid:101) α
i j have same signs; equality (ii) holds since (cid:98) y
( (cid:101) θ )
= (cid:98) y
( θ ) .
Consequently, the function t (cid:55)→ L β ( θ ( t )) is constant over [0 , 1] , and this concludes the proof of the fact that θ (cid:73) (cid:101) θ .
B. P ROOF OF P ROPOSITION
First, according to Proposition 6, given θ ∈ Θ m with m (cid:62) n + 1 + m ∗ , there exists a neural network (cid:101) θ with at most n + 1 non-zero neurons such that L β ( (cid:101) θ ) (cid:54)
L β ( θ ) .
According to Lemma 1, there exists θ ∗ =
{ ( u ∗ i , α ∗ i ) } mi =
an optimal neural network with at most m ∗ non-zero neurons.
Up to a permutation of the zero neurons of (cid:101) θ and those of θ ∗ , since m (cid:62) n +
+ m ∗ , we can assume without loss of generality that ( u ∗ i , α ∗ i ) =
(0 , 0) for i = m ∗ + 1 , . .
, m and ( (cid:101) u
i , (cid:101) α i ) =
(0 , 0) for i = 1 , . .
, m ∗ .
Now, we deﬁne a continuous path between (cid:101) θ
and θ ∗ .
For i = 1 , . . .
, m ∗ and j = m ∗ + 1 , . .
, m , we set the neural network θ ( t ) ∈
Θ m with neurons
[Equation]
Clearly, we have θ (0) =
(cid:101) θ and θ (1)
= θ ∗ .
Further, θ ( t ) is scaled and it is easily veriﬁed that
[Equation]
θ ) .
This immediately implies that the function t (cid:55)→ L β ( θ ( t )) is convex over [0 , 1] .
Since it achieves a minimum at t = 1 , it follows that it is non-increasing, and this concludes the proof of Proposition 7.
C P ROOFS OF INTERMEDIATE RESULTS C. P ROOF OF L EMMA
Proof.
It holds that (cid:80) i ∈I σ ( Xu i )
α i = (cid:80) i ∈I γ i σ ( Xw i ) =
(cid:80) i ∈I D i
Xw i , whence (cid:96) ( (cid:80) i ∈I σ ( Xu i )
α i ) =
(cid:96) ( (cid:80) i ∈I
D
i
Xw i ) .
On the other hand, we have (cid:107) w i j (cid:107)
= (cid:107) u j (cid:107)
|
α j | .
Note that (cid:107) u j (cid:107) 2 = | α j | and thus, (cid:107) u j (cid:107)
|
α j | = 12 ( (cid:107) u j (cid:107) 22 + | α j | 2 ) .
Consequently, L β ( θ ) =
L cβ c ( w ∗ ) .
From Pilanci & Ergen (2020), we know that P ∗ =
P ∗ c .
Hence, L β ( θ ) =
P ∗ .
C. P ROOF OF L EMMA
We aim to show that m ∗ (cid:54) n + 1 and P ∗ m for any m (cid:62) m ∗ .
We leverage the following result which is known as Caratheodory’s theorem.
Lemma 3.
Let z 1 , . .
z m ∈ R n .
Suppose that y ∈ Conv { z 1 , . .
z m } .
Then, there exist indices i 1 , . .
, i n +1 ∈ { 1 , . . .
, m } such that y ∈ Conv { z
i 1 , . . .
, z i n +1 } .
Suppose that θ is an optimal neural network with m (cid:62) n + 1 neurons, and denote its neurons by ( u 1 , α 1 ) , . . .
, ( u m , α m ) .
We have that (cid:98) y
( θ ) = (cid:80) mi =1 (cid:101)
λ i z i , where z
i = sign ( α i ) ( (cid:80) mj
=1 (cid:107)
u j
(cid:107)| α j | ) σ (cid:16)
X u i (cid:107) u
i (cid:107) (cid:17) and (cid:101) λ
i
=
(cid:107) u
i (cid:107)| α i | (cid:80) mj
=1 (cid:107)
u j
(cid:107)| α j | .
Thus, (cid:98) y ( θ ) ∈
Conv { z 1 , . .
z m } .
From Lemma 3, we know that there exist i 1 , . .
, i n +1 and λ 1 , . . .
, λ n +1 (cid:62)
such that (cid:80) n +1 j =1 λ j =and (cid:98) y
( θ ) = (cid:80) n +1 j =1 λ j z
i j .
Plugging-in the expressions of the z i j , it follows that (cid:98) y
( θ ) =
n +1 (cid:88)
j = λ j sign ( α i j
[Equation]
where 
[Equation]
Further, we have that
[Equation]
where the last equality follows from the fact that (cid:80) n +1 j =1 λ j .
We deﬁne the neural network (cid:101) θ with neurons ( (cid:101) u i j , (cid:101) α i j ) .
We have that (cid:101) θ ∈ Θ n +1 and P ∗ n +1 (cid:54)
L β ( (cid:101) θ ) =
L β ( θ ) =
P ∗ m .
Since P ∗ n +1 (cid:62) P ∗ m for any m (cid:62) n + 1 , it follows from the previous set of inequalities that L β ( (cid:101) θ )
=
P ∗ n +
=
P ∗ m , and this holds for any m (cid:62) n + 1 .
Therefore, P ∗ n +1 =
P ∗ and L β ( (cid:101) θ )
=
P ∗ .
We set (cid:102) W = W ( (cid:101) θ ) .
We know from Proposition 2 that W ( (cid:101) θ ) ∈ W n +1 and L c β ( (cid:102) W ) (cid:54) L β ( (cid:101) θ ) .
Hence,
P ∗ c (cid:54) P ∗ .
We also know that L β ( θ ( (cid:102) W ))
(cid:54) L cβ ( (cid:102) W ) .
This implies that P ∗ c =
P ∗ and (cid:102) W is an optimal solution to (4).
Consequently, m ∗ (cid:54) n + 1 .
It remains to show that for any m (cid:62) m ∗ , we have P ∗ m
= P ∗ .
This follows again from Proposition 2.
Indeed, let W ∗ be an optimal solution to (4)
such that W ∗ ∈
W m ∗ .
Set θ ∗ =
θ ( v ∗ ) .
We know that θ ∗ ∈
Θ m ∗ , and L β ( θ ∗ ) (cid:54)
L cβ ( W ∗ ) =
P ∗ c =
P ∗ .
Hence, θ ∗ achieves P ∗ and this implies that for any m (cid:62) m ∗ , we have P ∗ m =
P ∗ .
D V ERIFICATION OF THE OPTIMAL SET
We review a standard method to determine whether a convex optimization problem has unique solution.
Consider a convex optimization problem
[Equation]
27) in the variable x ∈ R d .
Here f and f i for i ∈ [ m ] are convex functions.
Suppose that we calculate one optimal solution x ∗ and the corresponding optimal value f ∗ .
We can determine whether x ∗ is the unique optimal solution of (27) as follows.
For j ∈ [ d ] , consider the following convex optimization problems
[Equation]
These problems give the upper bound and the lower bound of the value of the i -th index in the optimal set of (27).
Suppose that p ub j − p lb j (cid:54) (cid:15) for certain small (cid:15) > 0 , for instance, (cid:15) = 10 − 8 .
Then, the radius of the optimal set with respect to the (cid:96) ∞ norm is upper-bounded by (cid:15) .
Therefore, we can be conﬁdent that x ∗ is the unique optimal solution up to numerical tolerance.
We have veriﬁed numerically that the convex optimization problem in Example 1 in section 3.1 has a unique optimal solution.