A Combinatorial Perspective on the Optimization of Shallow ReLU Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
The NP-hard problem of optimizing a shallow ReLU network can be characterizedas a combinatorial search over the activation pattern for each training examplefollowed by a constrained convex problem given a fixed set of activation patterns.
We explore the implications of this combinatorial aspect of ReLU optimizationin this work.
We show that it can be naturally modeled via a geometric andcombinatoric object known as a zonotope with its vertex set isomorphic to the setof feasible activation patterns.
This assists in analysis and provides a foundationfor further research.
We provide an example of its usefulness when we explorethe sensitivity of the optimal loss to perturbations of the training data.
Later wediscuss methods of zonotope vertex selection and its relevance to optimization.
Overparameterization assists in training by making a randomly chosen vertex morelikely to contain a good solution.
We then introduce a novel polynomial-time vertexselection procedure that provably picks a vertex containing the global optimumusing only double the minimum number of parameters required to fit the data.
We further introduce a local greedy search heuristic over zonotope vertices anddemonstrate that it outperforms gradient descent on underparameterized problems.
Introduction
Neural networks have become commonplace in a variety of applications.
They are typically trained tominimize a loss on a given dataset of labeled examples using a variant of stochastic gradient descent.
However, our theoretical knowledge of neural networks and their training lags behind their practicaldevelopments.
Single-layer ReLU networks are an appealing subject for theoretical study.
The universal approximation theorem guarantees their expressive power while their relative simplicity makes analysistractable (Hornik, 1991).
We restrict ourselves in this paper to studying empirical risk minimization(ERM) as was done in previous works (Du et al., 2018; Oymak & Soltanolkotabi, 2020) which isjustified since the train set performance tends to upper bound the test set performance.
Furthermore,modern neural networks achieve zero training loss but nevertheless generalize well (Kaplan et al.
,2020; Nakkiran et al., 2021).
Minimizing the training loss of a shallow ReLU network is a nonconvexoptimization problem.
Finding its global minima is difficult and can in fact be shown to be NP-hardin general (Goel et al., 2020).
Arora et al.
(2016) provide an explicit algorithm for finding the globalminima by solving a set of convex optimization problems; however, the size of this set is exponentialin both the input dimension d
and the number of hidden units m .
In this paper, we explore the combinatorial structure implicit in the global optimization algorithm of
Arora et al. (2016).
We start by using tools from polyhedral geometry to characterize the set of convexoptimization problems and describe the relationships between the subproblems.
Notably, we areable to create a special type of polytope called a zonotope (McMullen, 1971) whose vertices have a one-to-one correspondence with the convex subproblems and whose faces represent information abouttheir relationships.
We then explore the combinatorial optimization problem implicit in shallow ReLUnetwork empirical risk minimization using the zonotope formalism to help interpret our findings andassist in some proofs.
Since the computational complexity of optimization problems shapes our approach to solving them,we examine the reductions of NP-hard problems introduced in Goel et al.
The datasetsproduced have examples that are not in general position (i.e. they have nontrivial affine dependencies),which differs from most real-world datasets.
We prove that the global optimum of the loss of a shallow
ReLU network over such a dataset can have a discontinuous jump for arbitrarily small perturbationsof the data, which has a very natural interpretation in our zonotope formalism.
This means that aproof of the NP-hardness of ReLU optimization over training examples in general position does notfollow from the results Goel et al.
(2020) via a simple continuity argument.
We therefore present amodification of their proof that produces a dataset in general position.
In contrast to the NP-hardness of general ReLU optimization, sufficient overparameterization allowsgradient descent to provably converge to a global optimum in polynomial time, as demonstrated by(Du et al., 2018; Oymak & Soltanolkotabi, 2020).
We interpret the proof methods generally used inthese works as asserting that sufficient overparameterization allows gradient descent to bypass muchof the combinatorial search over zonotope vertices by having a randomly chosen vertex be close toone with a good solution with high probability.
We then introduce a novel algorithm that finds a goodzonotope vertex in polynomial time requiring only about twice the minimum number of hidden unitsrequired to fit the dataset.
Finally, we explore how gradient descent interacts with this combinatorial structure.
We provideempirical evidence that it can perform some aspects of combinatorial search but present an informalargument that it is suboptimal.
We reinforce this claim by showing that a greedy local search heuristicover the vertices of the zonotope outperforms gradient descent on some toy synthetic problems andsimplifications of real-world tasks.
In contrast to the NP-hard worst case, these results suggest thatthe combinatorial searches encountered in practice might be relatively tractable.
We summarize our contributions as follows.
• We are the first to provide an in-depth exposition of the combinatorial structure arising fromthe set of feasible activation patterns that is implicit in shallow ReLU network optimization.
Inparticular, we show that this structure can be characterized exactly as a Cartesian power of thezonotope generated by the set of training examples.
• We use this formalism to prove necessary conditions for the global optimum of a shallow ReLU
network to be discontinuous with respect to the training dataset.
We show that this impliesthat previous NP-hardness proofs of ReLU optimization do not automatically apply to datasetssatisfying realistic assumptions, which we rectify by presenting a modification that does producea dataset in general position.
• We explore the role that combinatorial considerations play in the relationship between overparameterization and optimization difficulty.
In particular, we introduce a novel polynomial-timealgorithm fitting a generic dataset using twice the minimum number of parameters
needed.• We introduce a novel heuristic algorithm that performs a greedy search along edges of a zonotope
and show that it outperforms gradient descent on some toy datasets.
We hope that the tools we introduce are generally useful in furthering our understanding of ReLUnetworks.
Notably, they have deep connections to several well-established areas of mathematics(McMullen, 1971; Richter-Gebert & Ziegler, 2017; Ziegler, 2012), which might allow researchers toquickly make new insights by drawing upon existing results in those fields.
Empirical Risk Minimization for ReLU Networks
A single ReLU layer consists of an affine transformation followed by a coordinate-wise applicationof the ReLU nonlinearity ϕ
( x ) = max { x, 0 } .
We can represent an affine transformation from 85 R d → R m by an m × ( d + 1) matrix by representing its inputs in homogeneous coordinates, i.e.by appending a ( d + 1) -th coordinate to network inputs that is always equal to 1.
Hence a single
ReLU layer with parameters W can be written as f W ( x ) =
ϕ
( W ¯ x ) , where ¯ x denotes x expressedin homogeneous coordinates.
A single hidden layer ReLU network consists of a single ReLU layer followed by an affine transformation.
We focus on the case of a network with scalar output, so thesecond layer can be represented by a vector v ∈ R m +1 .
Although the second layer parameters aretrained jointly with the first layer in practice, we assume that they are fixed for our analysis.
Thisparallels simplifying assumptions made in previous work (Du et al., 2018).
Let ℓ : R × R → R be a convex loss function such as MSE or cross-entropy.
Since the second layerparameters are fixed, we can incorporate them into a modified loss function ˜ ℓ : R m × R → R givenby ˜ ℓ ( z , y ) =
ℓ ( v T ¯ z , y ) that operates directly on the first layer’s activations z .
This modified lossfunction is convex since it is the composition of a convex function with an affine function.
Suppose we are given D
= { ( x i , y i ) } Ni =1 as the training dataset with x i ∈ R d and y i ∈ R for alli = 1 , . . .
, N .
Throughout this paper, we assume that N > d + 1 .
Sometimes we will represent adataset by a matrix X ∈ R ( d +1)
× N with each column corresponding to an example and its labels asthe vector y ∈ R N .
We say that D is in general position if there exist no nontrivial affine dependenciesbetween the columns of X .
The empirical loss L ( W ) , also known as the empirical risk, is defined asthe mean per-example loss
[Equation]
The goal of ERM is to find a set of parameters W that minimizes this loss.
Arora et al.
(2016)were the first to introduce an algorithm for exact ERM.
We adapt their algorithm for the case offixed second layer weights in algorithm 1, which has a running time of O ( N md poly( N, m, d )) .
Thealgorithm works by iterating over all feasible activation patterns
[Equation]
The subset of parameters corresponding to an activation patterns, which we call an activation region,can be expressed via a set of linear inequalities.
Within a single activation region, the map fromparameter values to ReLU layer activations over the train dataset is linear.
Hence we can solve aconstrained convex optimization problem to get the optimal parameters in each activation region.
Namely for a given A ∈ A , we solve for W ∈ R m × ( d +1) in the followingminimize
[Equation]
where ⊙ denotes the Hadamard product and a i denotes the i -th column of A .
The global optimumthen becomes the best optimum found over the entire set of activation regions.
Thus single-layer
ReLU network ERM can be expressed as a combinatorial search over activation patterns with aconvex optimization step per pattern.
Zonotope Formalism
While Arora et al.
(2016) mention that A arises from a set of hyperplanes induced by the training examples, they only use this connection to bound its cardinality |A| =
O ( N md ) .
However, hyperplanearrangements are well-studied geometric and combinatoric objects (Richter-Gebert & Ziegler, 2017;
Stanley et al., 2004).
As such, we will see that we can use this connection to better characterize thecombinatorial aspects of ReLU optimization.
Our mathematical tools for describing the combinatorial structure of shallow ReLU network optimization include oriented hyperplane arrangements, polyhedral sets, polyhedral complexes, andzonotopes.
Appendix A provides an approachable overview of these topics for unfamiliar readers.
Single Hidden Unit We start by considering a single ReLU unit f w ( x ) =
ϕ ( w T ¯ x ) parameterizedby the vector w ∈ R d +1 .
Looking at its behavior as w ranges over R d +1 on a single trainingexample x
i , we see that there are two linear regimes depending on the sign of w T ¯ x .
They areseparated by the hyperplane in parameter space satisfying w T ¯ x = 0 .
We can describe such behaviormathematically as an oriented hyperplane with the sign of w T ¯ x providing its orientation.
Thecollection of oriented hyperplanes associated to each training example { ¯ x i } Ni =1 is known as anoriented hyperplane arrangement (Richter-Gebert & Ziegler, 2017).
Algorithm 1 Exact ERM (Arora et al., 2016)
Input: data D = { x i , y i } Ni =1 , 2nd layer v ∈ R m +
A ⊆ { 0 , 1 } m × N {feasible activation patterns (2)}
W ∗ ∈ R m × ( d +1) {random initialization} for A ∈
A do
W ← solution of (3) if L ( ˜ W ) <
L ( W ∗ ) then
W ∗ ← W end if end for return W ∗
The structure imposed on parameter space
R d +1 by this oriented hyperplane arrangement can be described as a polyhedral complex (Ziegler, 2012), which is a collectionof polyhedral sets and their faces that fittogether in a “nice” way.
The polyhedralcomplex R induced by the training set willcontain codimension 0 sets called chambers.
These correspond exactly to activation regions.
Activation patterns havea one-to-one correspondence with the tuple of hyperplane orientations associatedto each chamber.
The center panel of fig.
1provides an illustration of R for an example dataset.
The dual zonotope of a polyhedral complex is a single polytope providing an alternate representationof its combinatorial structure (Ziegler, 2012).
Each dimension k member of the polyhedral complexhas a corresponding codimension k face in the dual zonotope.
Incidence relations between themembers of the complex are preserved in the dual zonotope.
Generally, a zonotope can be describedas the image of an N -dimensional hypercube under a linear map whose columns are known as itsgenerators (McMullen, 1971).
Each vertex of the zonotope thus is a weighted sum of its generatorswith coefficients belonging to { 0 , 1 } .
The dual zonotope Z of our polyhedral complex R has the training examples { ¯ x i } Ni =1 as its generators.
The vertices of Z have a one-to-one correspondence to activation regions of our network.
When avertex is expressed as a weighted sum over the generators, the coefficient { 0 , 1 } of each generatorequals its corresponding example’s value in the region’s activation pattern.
The right panel of fig.
1shows an example zonotope Z and its duality with the polyhedral complex R .
These correspondences allow us to assign additional structure to the set of activation patterns A ratherthan just treating it as an unstructured set.
For example, the 1-skeleton of the zonotope Z , which is thegraph formed by its vertices and edges, provides a means of traversing the set of activation patterns.
Furthermore, we can directly make connections between the training dataset and the activation patternstructure by making use of the fact that Z is generated by the training examples.
Multiple Hidden Units In the multiple hidden unit setting, i.e. m > 1 , note that parameter spacebecomes an m -fold Cartesian product of single-unit parameter spaces.
Furthermore, we are free toset the activation pattern for each unit independently of the others.
As the combinatorial structure foreach hidden unit can be described using the zonotope Z , the combinatorial structure for a multiplehidden unit network is described by the m -fold Cartesian product Z m =
(cid:81) mi
=1 Z .
As noted inappendix A.3.1, Z m is also a zonotope.
Each vertex of Z m corresponds to a product of m vertices of
Z .
As in the single unit case, there is a one-to-one correspondence between the vertices of Z m andthe set of activation patterns A .
ReLU Optimization
4.1 NP-Hardness
Given the additional structure we have imposed on the set of activation patterns in algorithm 1,it is natural to ask whether we can use it to develop a global optimization algorithm that is moreefficient than a brute-force search over activation patterns.
Unfortunately, several works (Goel et al.
,2020; Froese et al., 2021) have demonstrated that global optimization of a shallow ReLU network is
NP-hard.
Nevertheless, this does not preclude the existence of an efficient combinatorial optimizergiven certain conditions on the input dataset.
Since the zonotope
Z m encapsulates the combinatorialstructure of the optimization problem, we look to see if properties of Z m can be related to thedifficulty of combinatorial optimization.
Nontrivial affine dependencies between training examples influence the combinatorial structure ofthe Z m .
Since the reductions of NP-hard problems to ReLU optimization done in Goel et al.
Froese et al.
(2021) create datasets with such nontrivial dependencies, it is natural to ask whether it is
NP-hard to optimize a shallow ReLU network over a training dataset in general position.
4.1.1 Discontinuity of the Global Optimum
If the global minimum of the loss is always continuous with respect to the input dataset, then the
NP-hardness of optimization over arbitrary datasets in general position would follow from continuitysince every set of points is arbitrarily close to a set in general position.
However, we can prove thatsuch continuity holds unconditionally for ReLU optimization only in the case where the trainingdataset is in general position.
We give a sketch of the proof here along with some analysis of thefailure cases that can happen when the data are not in general positions.
We provide a full proof inappendix C.
Theorem 4.1.
Suppose we are given a dataset D
= { ( x i , y i ) }
Ni =1 in general position and somem ∈ N .
Let L ∗ ( D ) denote the global minimum of the loss (1) over the dataset D for a shallow ReLUnetwork with m units.
Given any ϵ > 0 , some δ > 0 exists such | L ∗ ( D ) − L ∗ ( D ϵ ) | < δ for anydataset D ϵ
= { ( x ′ i , y i ) }
Ni =1 satisfying ∥ x
i − x ′ i ∥ 2 ≤ ϵ .
Proof sketch.
For a small enough perturbation, we can prove that the datasets’ zonotopes are combinatorially equivalent.
Hence their sets of feasible activation patterns will be exactly the same.
Usingthe fact that any subset of a set in general position is also in general position, we can then show thatthe constrained convex optimization problem associated with each vertex is continuous with respectto the input dataset.
Since the global minimum of the loss is just the minimum of the optimal loss foreach vertex, its continuity follows from the fact that the composition of two continuous functions iscontinuous.
When the dataset D is not in general position, there are two possible ways in which breaking ofnontrivial affine dependencies between examples can cause the global minimum of the loss to becomediscontinuous.
The first is that the globally optimal vertex in the perturbed zonotope exists in theoriginal zonotope, but its associated constrained convex optimization problem is discontinuous withrespect to the dataset.
This can happen when there are nontrivial affine dependencies that get brokenamongst the active examples in the vertex.
The second way is that the globally optimal vertex of theperturbed zonotope does not exist in the original zonotope.
Geometrically, we can think of such avertex as resulting from the breakdown of a non-parallelepiped higher dimension face (Gover, 2014).
4.1.2 Analysis of Reductions
We can use this characterization of the instabilities of the global optimum to perturbations in thetraining data to analyze the reductions of NP-hard problems used in Goel et al.
We focuson the reduction of the NP-hard set cover problem to optimization of a single bias-free ReLU.
Inappendix D.2, we provide an explicit example of an arbitrarily small perturbation that results in theglobal minimum of the loss being independent of the solution to the set cover problem.
To the best of our knowledge, existing reductions of NP-hard problems to ReLU optimization allcreate datasets that are not in general position (Goel et al., 2020; Froese et al., 2021).
Therefore,we present a modification of the set cover reduction that produces a dataset in general position.
Seeappendix D.3 for details of this modification along with a proof that it is indeed a reduction of the setcover problem.
We thus have the following statement.
Theorem 4.2.
Optimizing a single ReLU over a dataset in general position is NP-hard.
We also note that the affine dependencies between training examples does not represent all of thecombinatorial structure captured by the zonotope.
Any dataset in general position containing Npoints belonging to R d corresponds to the same matroid (Oxley, 2006).
Such a matroid is called auniform matroid.
However, the orientation information present in hyperplane arrangements addsadditional structure to produce an oriented matroid.
Notably, two datasets in general position canhave non-equivalent oriented matroids (Richter-Gebert & Ziegler, 2017).
Thus all we have provenis that a common assumption on the matroid structure of the training examples can still lead to NPhard optimization problems.
Regardless, the zonotope formalism contains additional combinatorialstructure that may prove insightful to the complexity of optimization.
4.2 Polynomial Time Optimization via Overparameterization
Even though ReLU network optimization is NP-hard in general, it can be shown that overparameterization allows for gradient descent to converge to the global minimum in polynomial time (Duet al., 2018; Zou & Gu, 2019; Oymak & Soltanolkotabi, 2020; Allen-Zhu et al., 2019).
This is not acontradiction since optimization of overparameterized ReLU networks is a strict subset of the set ofall ReLU optimization problems.
The general proof method of these works usually involves demonstrating that overparameterizationresults in activation patterns not changing much throughout training.
This allows gradient descentto effectively bypass the combinatorial search of the outer loop in algorithm 1.
The remainingoptimization problem can then be shown to be similar to the constrained convex optimization problem(3) by assuming that the second layer is frozen.
Using the zonotope formalism, we can interpretthese results as saying that a sufficiently large number of hidden units m guarantees with highprobability that a randomly chosen vertex corresponds to a region of parameter space containing aglobal minimum of the loss.
The random vertex selection corresponds to the weight initializationscheme.
This can be justified theoretically through a connection to random feature models.
Here we assumethat the first layer is frozen, and the second layer forms a linear model over the random first layerfeatures.
As the number of units m increases past the number of training examples N , the set of firstlayer activations can become linearly independent.
The probability of this approaches 1 as m → ∞ .
Whether all of the parameters within an activation region produce linearly dependent activations canbe shown to depend solely on its activation pattern when the dataset is in general position.
To test this, we ran experiments comparing batch gradient descent to solving (3) for a randomly chosenvertex on some toy datasets.
We created synthetic datasets by first choosing the input dimension dand a positive integer m gen .
To get the training examples, we sampled N =
( d + 1) m gen points in
R d i.i.d.
from the standard Gaussian distribution.
We then sampled the weights of a shallow ReLUnetwork with m gen units from independent standard Gaussian distributions.
We used this networkto create the labels for our synthetic dataset.
See appendix G.1.1 for details on the data generationprocess.
We also created toy binary classification datasets from MNIST (LeCun et al., 2010) and Fashion
MNIST (Xiao et al., 2017) by choosing two classes, 5/9 and coat/pullover, respectively, to differentiate.
We used the first d ∈ { 8 , 16 } components of the PCA whitened data and selected N ∈ { 350 , 700 }examples for our train sets.
See appendix G.2 for details.
We present some of our results in fig.
See appendix G for details of the training procedures and forresults for more d, m gen and d, N pairs.
On synthetic data, we see that the random vertex methodfinds a good solution for overparameterization factors of 4 and up.
However, gradient descent tendsto arrive at reasonably good solutions for lower levels of overparameterization while the randomvertex method fails.
This was a general trend that we observed across different d, m gen pairs on thesynthetic datasets and d, N pairs on the binary classification datasets.
Note that the Fashion MNIST networks represented in fig.
2 were relatively underparameterized with the maximal size of 64 unitsbeing overparameterized by only a factor of about 1.5.
This demonstrates that gradient descent can perform some aspects of the combinatorial search overzonotope vertices.
We hypothesize that the gradient tends to be smaller within activation regionswith a good optimum and thus gradient descent is more likely to stay within a good activation region.
Conversely, the larger gradients within activation regions with poor optima makes it more likelythat a gradient descent step will move the parameters out of those regions.
We can thus think ofgradient descent as performing a pseudo-annealing process over the vertices of the zonotope sincethe likelihood of moving from one vertex to another decreases as the parameters settle into betteractivation regions.
4.2.1 Tighter Bounds
We now introduce a novel vertex selection scheme that runs in polynomial time and requires minimaloverparameterization.
Suppose D
=
( x i , y i ) } Ni =1 is a dataset in general position.
Assume that theexamples are ordered by the value of their last coordinate, which we suppose is unique WLOG (i.e.e Td x i <
e Td x j for i < j ).
If not provided in this format, this can be accomplished in O ( N log N )time.
We now split the dataset into ⌈ Nd +1 ⌉ chunks containing at most d + 1 examples.
We write each chunkas D k
= { ( x i , y i ) } min( N,k
( d +1))
i =( k − 1)( d +1)+1 .
Since each D k contains a contiguous chunk of examples sortedalong an axis in coordinate space, we see that we can always find a hyperplane separating D k and
D k ′ for k ̸ = k ′ .
For each k = 1 , . . .
, ⌈ Nd +1 ⌉ , we add two units to our ReLU network and assign them the activationpattern of 0 for examples belong to a D k ′ with k ′ <
k and 1 for the remaining examples.
One of theunits will be multiplied by +1 in the second layer while the other will be multiplied by -1.
Hence thenetwork contains a total of 2 ⌈ N d +1 ⌉ hidden units.
We prove in appendix E that a set of weights with that activation pattern exists such that the output ofthe network on training examples exactly matches their labels.
The key idea in the proof is that wecan sequentially fit the examples in the k -th chunk without undoing our progress in fitting the chunksbefore it.
Finding the optimal value of the weights is a constrained convex optimization problem thatcan be solved in polynomial time.
This proves the following statement.
Theorem 4.3.
Given a dataset containing N examples belonging to R d in general position, a shallow
ReLU network containing 2 ⌈ Nd +
⌉ hidden units can be found in polynomial time that exactly fits thedataset.
To the best of our knowledge, this is the tightest known bound on the amount of overparameterizationneeded to find the global optimum of a ReLU network in polynomial time.
A simple argumentcomparing the number of unknowns and the number of equations demonstrates that we need at least
Nd +1 hidden units to exactly fit an arbitrary dataset with a shallow ReLU network.
Hence our methoduses only about twice as many hidden units as is necessary to fit the data.
However, we emphasizethat this ReLU optimization scheme is primarily of theoretical interest since we find in practice thatthe resulting ReLU network tends to be very ill-conditioned.
Heuristic
Input: data D = { x i , y i } Ni =1 , 2nd layer v ∈ R m +1 , max steps T ∈ N
A 0 ∈ vert( Z m ) for t ∈ { 0 , . . .
, T } do
A t +1 ← A t for A ′ ∈ neighbors( A t ) do if L ∗ ( A ′ ; D ) <
L ∗ ( A t +1 ; D ) then
A t +1 ← A ′ end if end for if A t +1 =
A t then return A t end if end for return A T
4.3 Relevance to Optimization in Practice
Practically all optimization of ReLU networks in practice uses some variant of gradient descent withan overparameterized network.
As the degree of overparameterization goes down, gradient descentbegins to arrive at increasingly suboptimal solutions (Nakkiran et al., 2021).
In section 4.2, we hypothesized how gradient descent can find activation regions containing goodoptima.
However, the gradient of the loss is inherently a local property in parameter space whilethe space’s decomposition into activation regions is inherently global.
Boundaries between regionscorrespond to discontinuities in the gradient of the loss.
We hypothesize that these properties leadto little direct information about the optimization problem being used to inform gradient descent’straversal over zonotope vertices.
Hence we suspect that algorithms that explicitly traverse zonotopevertices using some loss-based criteria can outperform gradient descent in the underparameterized- tomildly-overparameterized regimes.
4.3.1 Difficulty of Combinatorial Search
Unless P = NP , we are unlikely to find an efficient algorithm to perform the combinatorial searchin algorithm 1 for arbitrary datasets (Goel et al., 2020).
However, this does not preclude the existenceof heuristics that tend to work well on problems encountered in practice.
We investigated this by using a greedy local search (GLS) over the graph formed by the zonotope’s1-skeleton.
We start by selecting a vertex at random and find its corresponding optimal loss by solvinga convex program.
We iterate over its neighboring vertices and compute their optimal losses as well.
We then move to the neighboring vertex with the lowest loss and repeat the process until we arriveat a vertex with lower loss than its neighbors.
We then take that vertex’s optimal parameters as ourapproximation to the global minimization problem.
This algorithm is defined in detail in algorithm 2.
We also experimented with some additional heuristics that help the GLS converge faster by reducingthe number of convex problems solved at each step.
For example, we can greedily move to the firstneighboring vertex encountered with a lower loss, which significantly decreases the time per stepin the early stages of training.
We can further improve this by using geometric information about asolution’s relative location in its activation region to try certain vertices first.
We call the algorithmwith these heuristics modified greedy local search (mGLS) and define it in detail in appendix F.
Note that these heuristics are not guaranteed to return a global minimizer of the loss.
Furthermore,there are an exponential number of vertices in the zonotope, so there are no immediate guarantees of them taking less than exponential time to run.
However, each step takes polynomial time sinceeach vertex has O ( mN ) neighbors, so each step solves a polynomial number of convex optimizationproblems.
We ran experiments comparing these heuristics to gradient descent on toy datasets generated in thesame way as in section 4.2.
We used GLS on the synthetic data and mGLS on the MNIST and
Fashion MNIST derived data.
We present some of our results in fig.
See appendix G for details ofthe training procedures and for results for more values of d, m gen and d, N pairs.
On synthetic data,the GLS heuristics significantly outperformed gradient descent.
On binary classification tasks, mGLSoutperformed gradient descent for networks with moderate levels of underparameterization andperformed similarly otherwise.
We observed a similar trend across the rest of the d, m gen and d, Npairs.
These results support our hypothesis that gradient descent is suboptimal in the combinatorialsearch over activation patterns.
Furthermore, they suggest that this combinatorial optimization mighttend to be tractable in practice.
Related Work
Some of the concepts in this work also arise in Zhang et al.
A key difference is that theyanalyze the activation regions in input space given a ReLU network with fixed parameters.
We can,in fact, use their tropical geometric approach to derive our zonotope formalism for a single ReLUunit.
To do so, the roles of the weights and data must be swapped; we instead use a fixed data matrixand varying vector of weights while they use a fixed weight matrix and varying vector of data.
Ouruse of a zonotope generated by the training examples, however, is novel.
Misiakos et al.
(2021) show that the approximation error between two shallow ReLU networksdepends on the the Hausdorff distance between the zonotopes generated by the each network’s units.
Bach (2017) also use a Hausdorff distance between zonotopes in the context of neural networkoptimization.
Goel et al.
(2020) provides proofs of the NP-hardness of optimization of shallow ReLU networks andthe hardness of even finding an approximate solution.
Froese et al.
(2021) extend these results andshow that the brute force search in Arora et al.
(2016) cannot be avoided in the worst case.
Du et al.
(2018) was one the first works to prove that overparameterization in shallow ReLU networksallows gradient descent to converge to a global optimum in polynomial time.
Their bound of Ω( N 6 )on the number of hidden units needed for convergence was improved upon by subsequent work (Ji &
Telgarsky, 2019; Daniely, 2019).
For example, Oymak & Soltanolkotabi (2020) proved a bound ofΩ( N 2 /d ) .
Dey et al.
(2020) provide an example of a heuristic algorithm that searches over activation patterns fora single ReLU unit.
Their algorithm operates on the principle that examples with large positive labelsare more likely to belong to the active set in good solutions.
It does not use any of the properties ofthe combinatorial structure that we explore in this work.
Other works have also explored the search over activation patterns.
For example, Pilanci & Ergen(2020) and Wang et al. (2021) represent global optimization of shallow ReLU networks with ℓ 2regularization using a convex optimization problem that operates simultaneously over all activationpatterns for a single unit.
Multiple units are handled by summing over the activations with different activation patterns.
This leads to exponential complexity in the data dimension d but avoidsexponential complexity in the number of units m .
Conclusion
We introduced a novel characterization of the combinatorial structure of activation patterns implicitin the optimization of shallow ReLU networks.
We showed that it can be described as a Cartesianproduct of zonotopes generated by the training examples.
We used this zonotope formalism to explore aspects of the optimization of shallow ReLU networks.
It provides a natural way to describe instabilities of the global minimum to perturbations of thedataset.
We then related this to work on the NP-hardness of global ReLU optimization.
In particular, we demonstrated that this optimization problem is still NP-hard even when restricted to datasets ingeneral position, which is commonly assumed of data in practice.
We then explored how combinatorial considerations play into the relationship between overparameterization and polynomial-time optimization of shallow ReLU networks.
Namely we interpret knownresults for gradient descent as stating that a randomly chosen zonotope vertex will be close to onewhose activation region contains a good local optimum.
We then provide empirical evidence thatsufficient overparameterization makes it highly likely that a randomly chosen vertex has a good localoptimum.
We also provide a polynomial-time algorithm that can find a vertex containing the globaloptimum using approximately twice the minimum number of hidden units needed to fit the datasetexactly.
Finally, we provide a GLS heuristic over zonotope vertices that outperforms gradient descenton some toy problems.
In future work we plan to theoretically and empirically explore heuristics and algorithms that performwell on real-world datasets.
We are interested in seeing if weak conditions on the training data, suchas being in general position, exist allowing for a polynomial-time global optimization.
We also hopeto extend our theory to analyze how vertex choice impacts generalization.
Further insights might bederived by exploring the connections of hyperplane arrangements to tropical geometry and orientedmatroids (Stanley et al., 2004; Oxley, 2006; Maclagan & Sturmfels, 2015).
One caveat of our theory is that it applies to only a shallow ReLU network.
However, the conceptsof activation patterns are still meaningful for deep ReLU networks.
Notably, one can show thatactivation regions in parameter space are semialgebraic sets for deep ReLU networks.
Real algebraicgeometry should thus provide tools to extend our analysis to deep ReLU networks (Basu, 2014;
Bochnak et al., 2013).
We hope that further research along these avenues will deepen our understanding of neural networktraining and enable improvements to training in practice.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our contributions are primarily theoretical.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] We provide proofs inthe appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] We have included this in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)?
[Yes] We include standard deviations for all results in the
tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the datasets
and software libraries used.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[No] The datasets used (MNIST, Fashion MNIST) are standard in
machine learning.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] They are widely used datasets with pictures of
handwritten digits and items of clothing.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]