Principled Deep Neural Network Training through Linear Programming
Anonymous authors
Paper under double-blind review
Abstract
Deep Learning has received signiﬁcant attention due to its impressive performance in many state-of-the-art learning tasks.
Unfortunately, while very powerful, Deep Learning is not well understood theoretically and in particular only recently results for the complexity of training deep neural networks have been obtained.
In this work we show that large classes of deep neural networks with various architectures (e.g., DNNs, CNNs, Binary Neural
Networks, and ResNets), activation functions (e.g., ReLUs and leaky ReLUs), and loss functions (e.g., Hinge loss, Euclidean loss, etc) can be trained to near optimality with desired target accuracy using linear programming in time that is exponential in the input data and parameter space dimension and polynomial in the size of the data set; improvements of the dependence in the input dimension are known to be unlikely assuming P (cid:44) NP , and improving the dependence on the parameter space dimension remains open.
In particular, we obtain polynomial time algorithms for training for a given ﬁxed network architecture.
Our work applies more broadly to empirical risk minimization problems which allows us to generalize various previous results and obtain new complexity results for previously unstudied architectures in the proper learning setting.
Introduction
Deep Learning is a powerful tool for modeling complex learning tasks.
Its versatility allows for nuanced architectures that capture various setups of interest and has demonstrated a nearly unrivaled performance on state-of-the-art learning tasks across many domains.
At the same time, the fundamental behavior of Deep
Learning methods is not well understood.
One particular aspect that recently gained signiﬁcant interest is the computational complexity of training such networks.
The basic training problem is usually formulated as an empirical risk minimization problem (ERM) that can be phrased as
[Equation]
where (cid:96) is some loss function , ( ˆ x i , ˆ y i )
Di = 1 is an i.i.d.
sample from some data distribution D , and f is a neural network architecture parameterized by φ ∈ Φ with Φ being the parameter space of the considered architecture (e.g., network weights).
The empirical risk minimization problem is solved in lieu of the general risk minimization problem (GRM) min φ ∈ Φ (cid:69) ( x , y )∈D [ (cid:96) ( f ( x , φ ) , y )] which is usually impossible to solve due to the inaccessibility of D .
Several works have studied the training problem for speciﬁc architectures, both in the proper and improper learning setup.
In the former, the resulting “predictor” obtained from (1) is always of the form f (· , ˆ φ ) for some ˆ φ ∈ Φ , whereas in the latter, the predictor is allowed to be outside the class of functions { f (· , φ ) : φ ∈ Φ } as long as it satisﬁes certain approximation guarantee to the solution of (1) 1 .
In both cases, all results basically establish trainability in time that is exponential in the network parameters but polynomial in the amount of data.
In this work we complement and signiﬁcantly extend previous work by providing a principled method to convert the empirical risk minimization problem in (1) associated with the learning problem for various architectures into a linear programming problem (LP) in the proper learning setting.
The obtained linear programming formulations are of size roughly exponential in the input dimension and in the parameter space dimension and linear in the size of the data-set.
This result provides new bounds on the computational complexity of the training problem.
For an overview on Complexity Theory we refer the reader to Arora and Barak (2009).
Related Work
Our work is most closely related to Goel et al.
(2017), Zhang et al.
(2016), and Arora et al.
In Zhang et al.
the authors show that (cid:96) 1 -regularized networks can be learned improperly in polynomial time in the size of the data (with a possibly exponential architecture dependent constant) for networks with ReLU-like activations (but not actual ReLUs) and an arbitrary number of layers k .
These results were then generalized in
Goel et al.
(2017) to actual ReLU activations.
In both cases the improper learning setup is considered, i.e., the learned predictor is not a neural network itself and the learning problem is solved approximately for a given target accuracy.
In contrast to these works, Arora et al.
(2018) considered proper and exact learning however only for k = 2 (i.e., one hidden layer).
In relation to these works, we consider the proper learning setup for an arbitrary number of layers k and a wide range of activations, loss functions, and architectures.
As previous works, except for Arora et al.
(2018), we consider the approximate learning setup as we are solving the empirical risk minimization problem and we also establish generalization of our so-trained models.
Our approach makes use of Bienstock and Muñoz (2018) that allows for reformulating non-convex optimization problems with small treewidth and discrete as well as continuous decision variables as an approximate linear programming formulations.
To the best of our knowledge, there are no previous papers that propose LP-based approaches for training neural networks.
There are, however, proposed uses of Mixed-Integer and Linear Programming technology in other aspects of Deep Learning.
Some examples of this include feature visualization (Fischetti and Jo, 2018), generating adversarial examples (Cheng et al., 2017; Anonymous, 2019a; Fischetti and Jo, 2018), counting linear regions of a Deep Neural Network (Serra et al., 2017), performing inference (Amos et al., 2017) and providing strong convex relaxations for trained neural networks (Anderson et al., 2018).
Contribution
We ﬁrst establish a general framework that allows us to reformulate (regularized)
ERM problems arising in
Deep Learning (among others!) into approximate linear programs with explicit bounds on their complexity.
The resulting methodology allows for providing complexity upper bounds for speciﬁc setups simply by plugging-in complexity measures for the constituting elements such as layer architecture, activation functions, and loss functions.
In particular our approach overcomes limitations of previous approaches in terms of handling the accuracy of approximations of non-linearities used in the approximation functions to achieve the overall target accuracy.
Principled Training through LPs.
If (cid:15) > 0 is arbitrary, then for any sample size D there exists a dataindependent linear program, i.e., the LP can be written down before seeing the data, with the following properties:
1 In the case of Neural Networks, for example, improper learning could lead to a predictor that does not correspond to a Neural Network, but that might behave closely to one.
Solving the ERM problem to (cid:15) -optimality.
The linear program describes a polyhedron P such that for every realized data set ( ˆ X , ˆ Y ) =
( ˆ x i , ˆ y i ) Di
= 1 there is a face F ˆ X ,
ˆ Y ⊆ P such that optimizing certain linear function over
F ˆ X , ˆ Y solves (1) to (cid:15) -optimality returning a feasible parametrization ˜ φ ∈ Φ which is part of our hypothesis class (i.e., we consider proper learning ).
The face F ˆ
X , ˆ Y ⊆ P is simply obtained from P by ﬁxing certain variables of the linear program using the values of the actual sample; equivalently, by Farkas’ lemma, this can be achieved by modifying the objective function to ensure optimization over the face belonging to the data.
As such, the linear program has a build-once-solve-many feature.
We will also show that a possible data-dependent LP formulation is meaningless (see Appendix B).
Size of the linear program.
The size , measured as bit complexity, of the linear program is roughly
O (( 2 L/ (cid:15) )
N + n + m D ) where L is a constant depending on (cid:96) , f , and Φ that we will introduce later, n , m are the dimensions of the data points, i.e., ˆ x i ∈ (cid:82) n and ˆ y i ∈ (cid:82) m for all
i ∈
[ D ] , and N is the dimension of the parameter space Φ .
The overall learning algorithm is obtained then by formulating and solving the linear program, e.g., with the ellipsoid method whose running time is polynomial in the size of the input
Grötschel et al.
Even sharper size bounds can be obtained for speciﬁc architectures assuming network structure (see Appendix F) and our approach immediately extends to regularized ERMs (see Appendix E).
It is important to mention that the constant L measures a certain Lipschitzness of the ERM training problem.
While not exactly requiring Lipschitz continuity in the same way, Lipschitz constants have been used before for measuring complexity in the improper learning framework (see Goel et al.
and more recently have been shown to be linked to generalization in Gouk et al.
Generalization.
Additionally, we establish that the solutions obtained for the ERM problem via our linear programming approach generalize, utilizing techniques from stochastic optimization.
We also show that using our approach one can obtain a signiﬁcant improvement on the results of Goel et al.
(2017) when approximating the general risk minimization problem .
Due to space limitations, however, we relegate this discussion to
Appendix I.
Throughout this work we assume both data and parameters to be well-scaled, which is a common assumption and mainly serves to simplify the representation of our results; the main assumption is the reasonable boundedness, which can be assumed without signiﬁcant loss of generality as actual computations assume boundedness in any case (see also Liao et al.
(2018) for arguments advocating the use of normalized coeﬃcients in neural networks).
More speciﬁcally, we assume Φ ⊆
[− 1 , 1 ] N as well as ( x , y ) ∼ D satisﬁes ( x , y ) ∈ [− 1 , 1 ] n × [− 1 , 1 ] m .
We point out three important features of our results.
First, we provide a solution method that has provable optimality guarantees for the ERM problem, ensures generalization, and linear dependency on the data (in terms of the complexity of the LP) without assuming convexity of the optimization problem.
To the best of our knowledge, the only result presenting optimality guarantees in a proper learning , non-convex setting is that of Arora et al.
Second, the linear program that we construct for a given sample size D is data-independent in the sense that it can be written down before seeing the actual data realization and as such it encodes reasonable approximations of all possible data sets that can be given as an input to the ERM problem.
This in particular shows that our linear programs are not simply discretizing space: if one considers a discretization of data contained in [− 1 , 1 ] n × [− 1 , 1 ] m , the total number of possible data sets of size D is exponential in D , which makes the linear dependence on D of the size of our LPs a remarkable feature.
Finally, our approach can be directly extended to handle commonly used regularizers as we show in
Appendix E; for ease of presentation though we omit regularizers throughout our main discussions.
Complexity results for various network architectures.
We apply our methodology to various well-known neural network architectures and either generalize previous results or provide completely new results.
We provide an overview of our results in Table 1, where k is the number of layers, w is width of the network, n / m are the input/output dimensions and N is the total number of parameters.
We use G to denote the directed graph deﬁning the neural network and ∆ the maximum vertex in-degree in G .
In all results the node
[Table]
computations are linear with bias term and normalized coeﬃcients, and activation functions with Lipschitz constant at most 1 and with 0 as a ﬁxed point; these include ReLU, Leaky ReLU, eLU, Tanh , among others.
We would like to point out that certain improvements in the results in Table 1 can be obtained by further specifying if the ERM problem corresponds to regression or classiﬁcation .
For example, the choice of loss functions and the nature of the output data y (discrete or continuous) typically rely on this.
We can exploit such features in the construction of the LPs (see the proof of Theorem 3.1) and provide a sharper bound on the LP size.
Nonetheless, these improvements are not especially signiﬁcant and in the interest of clarity and brevity we prefer to provide a uniﬁed discussion on ERM.
Missing proofs have been relegated to the appendix due to space limitations.
The complexity of the training problem for the Fully Connected DNN case is arguably the most studied and, to the best of our knowledge, all training algorithms with approximation or optimality guarantees have a polynomial dependency on D only after ﬁxing the architecture (depth, width, input dimension, etc.).
In our setting, once the architecture is ﬁxed, we obtain a polynomial dependence in both D and 1 / (cid:15) .
Moreover, our results show that in the bounded case one can obtain a training algorithm with polynomial dependence on D across all architectures, assuming very little on the speciﬁc details of the network (loss, activation, etc).
This answers an open question left by Arora et al.
(2018) regarding the possibility of a training algorithm with polynomial dependence on D .
In addition, we show that a uniform LP can be obtained without compromising that dependence on D .
The reader might wonder if the exponential dependence on the other parameters of our LP sizes can be improved, namely the input dimension n + m and the parameter space dimension
N (we are ignoring for the moment the exponent involving the depth k , as it will be typically dominated by N ).
The dependence on the input dimension is unlikely to be improved due to the NP-hardness result in Blum and Rivest (1992), whereas obtaining a polynomial dependence on the parameter space dimension remains open (see Arora et al.
A recent paper Anonymous (2019b) provides an NP-hard DNN training problem that becomes polynomially solvable when the input dimension is ﬁxed.
However, this result considers a ﬁxed architecture, thus the parameter space dimension is a constant and the running time is measured with respect to D .
Preliminaries
In the following let [ n ] (cid:17) { 1 , . . ., n } and [ n ] 0 (cid:17) { 0 , . . ., n } .
Given a graph H , we will use V ( H ) and E ( H ) to denote the vertex-set and edge-set of
H , respectively, and δ H ( u ) will be the set of edges incident to vertex u .
We will need:
Deﬁnition 2.1.
For a function g : K ⊆ (cid:82) n → (cid:82) , we denote its Lipschitz constant with respect to the p -norm over K as L p ( g ) , satisfying | g ( x ) − g
( y )| ≤ L p ( g )(cid:107) x − y (cid:107) p for all x , y ∈ K (whenever it exists).
Moreover, in the following let (cid:69) ω ∈ Ω [·] and (cid:86) ω ∈ Ω [·] denote the expectation and variance with respect to the random variable ω ∈ Ω , respectively.
2.1 Empirical Risk Minimization
The basic ERM problem is typically of the form (1), where (cid:96) is some loss function , ( ˆ x i , ˆ y i )
Di = 1 is an i.i.d.
sample from some data distribution D that we have reasonable sampling access to, and f is a model that is parametrized by φ ∈ Φ .
We consider the proper learning setting here, where the computed solution to the ERM problem has to belong to the hypothesis class induced by Φ ; for a detailed discussion see Appendix A.2.
We next deﬁne the Lipschitz constant of an ERM problem with respect to the inﬁnity norm.
Deﬁnition 2.2.
Consider the ERM problem (1) with parameters D , Φ , (cid:96), f .
We deﬁne the Architecture
Lipschitz Constant L( D , Φ , (cid:96), f ) as
[Equation]
over the domain K = [− 1 , 1 ] n × Φ × [− 1 , 1 ] m .
We emphasize that in (2) we are considering the data-dependent entries as variables as well, and not only the parameters Φ as it is usually done in the literature.
This is because we will construct data-independent LPs, a subtlety that will become clear later.
2.2 Neural Networks
A neural network can be understood as a function f deﬁned over a directed graph that maps inputs x ∈ (cid:82) n to f ( x ) ∈ (cid:82) m .
The directed graph G =
( V , E ) , which represents the network architecture, often naturally decomposes into layers V = (cid:207)
i ∈
[ k ] 0 V i with V i ⊆ V , where V 0 is referred to as the input layer and V k as the output layer .
To all other layers we refer to as hidden layers .
These graphs do neither have to be acyclic (as in the case of recurrent neural networks ) nor does the layer decomposition imply that arcs are only allowed between adjacent layers (as in the case of ResNets ).
In feed-forward networks, however, the graph is assumed to be acyclic.
For the unfamiliar reader we provide a more formal deﬁnition in Appendix A.3.
2.3 Binary Optimization problems with small treewidth
We will introduce the key concepts that we need to formulate and solve binary optimization problems with small treewidth, which will be the main workhorse behind our results.
The treewidth of a graph is a parameter used to measure how tree-like a given graph is.
Among all its equivalent deﬁnitions, the one we will use in this work is the following:
Deﬁnition 2.3.
Let G be an undirected graph.
A tree-decomposition (Robertson and Seymour (1986)) of G is a pair ( T , Q ) where T is a tree and Q = { Q t : t ∈ V ( T )} is a family of subsets of V ( G ) such that
(i)
For all v ∈ V ( G ) , the set { t ∈ V ( T ) : v ∈ Q t } forms a sub-tree T v of T , and (ii) For each { u , v } ∈ E ( G ) there is a t ∈ V ( T )
such that { u , v } ⊆
Q t , i.e., t ∈ T u ∩ T v .
The width of the decomposition is deﬁned as max {| Q t | : t ∈ V ( T )} − 1.
The treewidth of G is the minimum width over all tree-decompositions of
G .
We refer to the Q t as bags as customary.
An example of a tree-decomposition is given in Figure 2 in
Appendix A.1.
In addition to width , another important feature of a tree-decomposition ( T , Q ) we use is the size of the tree-decomposition given by | V ( T )| .
Consider a problem of the form ( BO ) :
min c T x + d T y s.t.
f i ( x ) ≥ 0 , i ∈ [ m ] g
[Equation]
( x ) =
y j , j ∈ [ p ] x
[Equation]
[Equation]
y ∈ (cid:82) p , where the f i and g j are arbitrary functions that we access via a function value oracle, i.e., an oracle that returns the function upon presentation with an input.
We will further use the concept of intersection graph .
Deﬁnition 2.4.
The intersection graph Γ [I] for an instance I of BO is the undirected graph which has a vertex for each x variable and an edge for each pair of x variables that appear in any common constraint.
Note that in the above deﬁnition we have ignored the y variables which will be of great importance later.
The sparsity of a problem is now given by the treewidth of its intersection graph and we obtain:
Theorem 2.5.
Consider an instance I of problem BO .
If Γ [I] has a tree-decomposition ( T , Q ) of width ω , there is an exact linear programming reformulation of I with O ( 2 ω (| V ( T )| + p )) variables and constraints.
Theorem 2.5 is an immediate generalization of a theorem in Bienstock and Muñoz (2018) distinguishing the variables y , which do not need to be binary in nature, but are fully determined by the binary variables x .
The proof is omitted as it is almost identical to the proof in Bienstock and Muñoz (2018).
For the sake of completeness, we include a proof sketch in Appendix C.1.
Approximation to ERM via data-independent LPs
We will now show how we can obtain an approximate LP formulation for the ERM problem.
A notable feature is that our LP formulation is data-independent in the sense that we can write down the LP, for a given sample size D , before having seen the actual data; the LP is later specialized to a given data set by ﬁxing some of its variables.
This subtlety is extremely important as it prevents trivial solutions, where some non-deterministic guess provides a solution to the ERM problem for a given data set and then simply writes down a small LP that outputs the network conﬁguration; such an LP would be of small size (the typical notion of complexity used for LPs) however not eﬃciently computable.
By making the construction independent of the data we circumvent this issue; we provide a short discussion in Appendix B and refer the interested reader to Braun et al.
(2016; 2015); Braun and Pokutta (2018+) for an in-depth discussion of this subtlety.
Slightly simplifying, we might say for now that in the same way we do not want algorithms to be designed for a ﬁxed data set , we do not want to construct LPs for a speciﬁc data set but for a wide range of data sets.
As mentioned before, we assume Φ ⊆
[− 1 , 1 ] N as well as ( x , y ) ∼ D satisﬁes ( x , y ) ∈ [− 1 , 1 ] n × [− 1 , 1 ] m as normalization to simplify the exposition.
Since the BO problem only considers linear objective functions, we begin by reformulating the ERM problem (1) in the following equivalent form:
[Equation]
3.1 Approximation of the feasible region via an (cid:15) -grid
Motivated by this reformulation, we study an approximation to the following set:
[Equation]
The variables ( x i , y i ) Di
= 1 denote the data variables, that will be assigned values upon a speciﬁcation of a data set of sample size D .
Let r ∈ (cid:82) with − 1 ≤ r ≤ 1.
Given γ ∈ ( 0 , 1 ) we can approximate r as a sum of inverse powers of 2, within additive error proportional to γ .
For N γ (cid:17) (cid:100) log 2 γ − 1 (cid:101) there exist values z h ∈ { 0 , 1 } with h ∈ [ N γ ] , so that
[Equation]
Our strategy is now to approximately represent the x , y , φ variables via these binary approximations, i.e., as − 1 + 2 ·
[Equation]
L γ h = 1 2 − h z h where each z h is a (new) binary variable.
Deﬁne (cid:15) = 2 γ L , where L =
L
( D , Φ , (cid:96), f ) is the architecture Lipschitz constant deﬁned in (2), and consider the following approximation of S ( D , Φ , (cid:96), f ) :
[Equation]
We can readily describe the error of the approximation of S ( D , Φ , (cid:96), f ) by S (cid:15)
( D , Φ , (cid:96), f ) in the ERM problem (1) induced by the discretization:
Lemma 3.1.
For any ( x 1 , . . .
, x D , y 1 , . . .
, y D , φ, L ) ∈ S ( D , Φ , (cid:96), f ) there is ( ˆ x 1 , . . .
, ˆ x D , ˆ y 1 , . . .
, ˆ y D , ˆ φ, ˆ L ) ∈
S (cid:15) ( D , Φ , (cid:96), f ) such that (cid:12)(cid:12)
[Equation]
(cid:205)
[Equation]
By substituting out the x , y , φ by means of the equations of S (cid:15) ( D , Φ , (cid:96), f ) , we obtain a feasible region as BO .
3.2 Linear reformulation of the binary approximation
So far, we have phrased the ERM problem (1) in terms of a binary optimization problem using a discretization of the continuous variables.
This in and of itself is neither insightful nor useful.
In this section we will perform the key step, reformulating the convex hull of S (cid:15)
( D , Φ , (cid:96), f ) as a moderate-sized linear program by means of Theorem 2.5 exploiting small treewidth of the ERM problem.
After replacing the ( x , y , φ ) variables in S (cid:15) ( D , Φ , (cid:96), f ) using the z variables, we can see that the intersection graph of
S (cid:15)
[Equation] [Equation]
[Equation]
, f ) is given by Figure 1a, where we use ( x , y , φ ) as stand-ins for corresponding the binary variables z x , z y , z φ .
Recall that the intersection graph does not include the L variables.
It is not hard to see that a valid tree-decomposition for this graph is given by Figure 1b.
This tree-decomposition has size D and width
N γ ( n + m + N ) − 1 (much less than the N γ ( N + Dn + Dm ) binary variables).
This yields our main theorem:
Main Theorem 3.1.
Let D ∈ (cid:78) be a given sample size.
Then conv ( S (cid:15) ( D , Φ , (cid:96), f )) admits a linear programming formulation with the following properties:
(a) The linear program has no more than 4 D ( 2 L/ (cid:15) )
n + m + N variables and 2 D ( 2 ( 2 L/ (cid:15) ) n + m + N + 1 ) constraints.
We refer to the resulting polytope as P S (cid:15)
(b) The linear program can be constructed in time O
(( 2 L/ (cid:15) )
n + m + N D
plus the time required for O (( 2 L/ (cid:15) )
n + m + N ) evaluations of (cid:96) and f .
φφφ 3 φφ N x 1 , yx 2 , yx 3 , y 3 x D , y D
[Figure]
(cid:15) (
[Figure]
(cid:96)
[Figure]
f ) φ x 1 , yφ x 2 , y· · · φ x D , y D
[Figure]
(c) Fix any data set ( ˆ X , ˆ Y )
=
( ˆ x i , ˆ y i ) Di
= 1 of size D , with ˆ x i ∈ [− 1 , 1 ] n and ˆ y i ∈ [− 1 , 1 ] m .
There exists a face
F ˆ X , ˆ Y of P S (cid:15) such that
[Equation]
[Figure]
satisﬁes (cid:12)(cid:12) 1 D (cid:205)
Di = 1 (cid:2) (cid:96) ( f ( ˆ x i , φ ∗ ) , ˆ y i ) − (cid:96) ( f ( ˆ x i , ˜ φ ) , ˆ y i ) (cid:3)(cid:12)(cid:12) ≤ 2 (cid:15), where φ ∗ ∈ [− 1 , 1 ] N is an optimal solution to the ERM problem (1) with input data ( ˆ X , ˆ Y ) .
This means that solving an LP using an appropriate face of
P S (cid:15) solves the ERM problem (1) within an additive error 2 (cid:15) .
(d) The face F ˆ X , ˆ Y arises by substituting in actual data for the data-variables x , y , which determine the approximations z x , z y and is used to ﬁxed additional variables of the LP.
Proof.
The proof of part (a) follows directly from Theorem 2.5 using N γ = (cid:100) log ( 2 L/ (cid:15) )(cid:101) along with the tree-decomposition of Figure 1b, which implies | V ( T (cid:48) )
| + p = 2 D in this case.
Parts (b), (c) and (d) rely on the explicit construction for Theorem 2.5 and they are given in Appendix D.
(cid:3)
Observe that equivalently, by Farkas’ lemma, optimizing over the face can be also achieved by modifying the objective function in a straightforward way.
Also note that the number of evaluations of (cid:96) and f is independent of D .
We would like to further point out that we can provide an interesting reﬁnement of the theorem from above: if Φ has an inherent network structure (as in the case of Neural Networks) one can exploit treewidth-based sparsity of the network itself in order to obtain a smaller linear program with the same approximation guarantees as before.
This allows us to reduce the exponent in the exponential term of the LP size to an expression that depends on the sparsity of the network , instead of its size.
For brevity of exposition, we relegate this discussion to Appendix F.
Another signiﬁcant improvement can be obtained by using more information about the input data.
Assuming extra structure on the input, one could potentially improve the exponential input-dimension dependency of the LP size.
We relegate the discussion of this feature to Remark
D.1 in the Appendix, as it requires the explicit construction of the LP, which we also provide in the Appendix.
Complexity for Specific Architectures 4.1 Fully-connected layers with ReLU activations and normalized coefficients
We consider a Deep Neural Network f : (cid:82) n → (cid:82) m with k layers given by f = T k ◦ σ ◦ · · · ◦ T 2 ◦ σ ◦ T 1 , where σ is the ReLU activation function σ ( x ) (cid:17) max { 0 , x } applied component-wise and each T i : (cid:82) w i − 1 → (cid:82) w i is an aﬃne linear function.
Here w 0 = n is the dimension of the input data and w k = m is the dimension of the output of the network.
We write T i ( z )
=
A i z + b i for i ∈
[ k ] and assume (cid:107) A i (cid:107) ∞ ≤ 1, (cid:107)
b i (cid:107) ∞ ≤ 1 via normalization.
Thus, if v is a node in layer i , the node computation performed in v is of the form ˆ a T z + ˆ b , where ˆ a is a row of A i and ˆ b is a component of b i .
Note that in this case the dimension of the parameter space Φ is exactly the number of edges of the network.
Hence, we use N to represent the number of edges as well.
We begin with a short technical Lemma, with which we can immediately establish the following corollary.
Lemma 4.1.
For every i ∈ [ k − 1 ] 0 deﬁne U i = (cid:205) ij = 0 w j .
If (cid:107) z (cid:107) ∞ ≤ U i then (cid:107) T
i + 1 ( z )(cid:107) ∞
≤ U i + 1 .
Corollary 4.2.
If Φ is the class of Neural Networks with k layers, N edges, ReLU activations, and normalized coeﬃcients, then conv ( S (cid:15)
( D , Φ , (cid:96), f )) can be formulated via a linear program of size
O (( 2 L ∞ ( (cid:96) )
w O ( k 2 ) / (cid:15) ) n + m + N D ) , where w = max i ∈
[ k − 1 ] 0 w i and L ∞ ( (cid:96) ) is the Lipschitz constant of (cid:96) (· , ·) over [− U k , U k ] m × [− 1 , 1 ] m .
The linear program can be constructed in time O
(( 2 L ∞ ( (cid:96) )
w O ( k 2 ) / (cid:15) ) n + m + N D ) plus the time required for O (( 2 L ∞ ( (cid:96) )
w O ( k 2 ) / (cid:15) ) n + m + N ) evaluations of (cid:96) and f .
Proof.
Proving that the architecture Lipschitz constant is L ∞ ( (cid:96) )
w O ( k 2 ) suﬃces.
Note that all node computations take the form h ( z , a , b ) =
z T a + b for a ∈ [− 1 , 1 ] w and b ∈ [− 1 , 1 ] .
The only diﬀerence is made in the domain of z , which varies from layer to layer.
The 1-norm of the gradient of h is at most (cid:107) z (cid:107)
+ (cid:107) a (cid:107) 1 + 1 ≤ (cid:107) z (cid:107) 1 + w
+
which, in virtue of Lemma 4.1, implies that a node computation on layer
i (where the weights are considered variables as well) has Lipschitz constant at most (cid:205) ij =
w j + w + 1 ≤ (cid:205) ij = 0 2 w j = 2 w i + 1 − 1 w −≤ 2 w i
+ 1 .
On the other hand, for (cid:107)( a , b ) − ( a (cid:48) , b (cid:48) )(cid:107) ∞ ≤ γ and z ∈ [− U
i , U i ] , it holds that | h ( z , a , b ) − h
( z (cid:48) , a (cid:48) , b (cid:48) )
| ≤w i + 1 (cid:107)( z − z (cid:48) , a − a (cid:48) , b − b (cid:48) )
(cid:107) ∞ ≤ 2 w i + 1 max {(cid:107)
z − z (cid:48) (cid:107) ∞ , γ } , which shows that the Lipschitz constants can be multiplied layer-by-layer to obtain the overall architecture Lipschitz constant.
Since ReLUs have Lipschitz constant equal to 1, and (cid:206) ki
= 1 2 w i + 1 = w
O
( k 2 ) , whenever w ≥ 2, we conclude the architecture Lipschitz constant is L ∞ ( (cid:96) )
w O ( k 2 ) .
(cid:3)
The reader might have noticed that a sharper bound for the Lipschitz constant above could have been used, however we chose simpler bounds for the sake of presentation.
It is worthwhile to compare the previous lemma to the following closely related result.
Theorem 4.3.
(Arora et al., 2018, Theorem 4.1) Let Φ be the class of Neural Networks with 1 hidden layer ( k = 2 ), convex loss function (cid:96) , ReLU activations and output dimension m = 1 .
There exists an algorithm to ﬁnd a global optimum of the ERM problem in time O
( 2 w D nw poly ( D , n , w )) .
Remark 4.4 .
We point out a few key diﬀerences of this result with the algorithm we can obtain from solving the LP in Corollary 4.2: (a) One advantage of our result is the benign dependency on D .
An algorithm that solves the training problem using our proposed LP has polynomial dependency on the data-size regardless of the architecture.
(b) As we have mentioned before, our approach is able to construct an LP before seeing the data.
(c) The dependency on w of our algorithm is also polynomial.
To be fair, we are including an extra parameter N —the number of edges of the Neural Network—on which the size of our LP depends exponentially.
(d) We are able to handle any output dimension m and any number of layers k .
(e) We do not assume convexity of the loss function (cid:96) , which causes the resulting LP size to depend on how well behaved (cid:96) is in terms of its Lipschitzness.
(f)
The result of Arora et al.
(2018) has two advantages over our result: there is no boundedness assumption on the coeﬃcients, and they are able to provide a globally optimal solution instead of an (cid:15) -approximation.
4.2 ResNets, CNNs, and alternative activations
Corollary 4.2 can be directly generalized to handle other architectures as well, as the key features we used before are the acyclic structure of a standard Neural Network and the well behaved Lipschitz constant of the
ReLU function.
These features are present in many other architectures, and yield the following result.
Lemma 4.5.
Let Φ denote the class of feed-forward Neural Networks with k layers, as deﬁned in Section 2.2, N edges, aﬃne node computations, activation functions a i : (cid:82) → (cid:82) with Lipschitz constant at most 1 and such that a ( 0 )
= 0 , and normalized coeﬃcients.
Then conv ( S (cid:15) ( D , Φ , (cid:96), f )) can be formulated via a linear program of size
O (( 2 L ∞ ( (cid:96) )
∆ O ( k 2 ) / (cid:15) ) n + m + N D ) , where ∆ denotes the maximum vertex in-degree of the network and L ∞ ( (cid:96) ) is the Lipschitz constant of (cid:96) (· , ·) over [− U k , U k ] m × [− 1 , 1 ] m .
The linear program can be constructed in time O
(( 2 L ∞ ( (cid:96) )
∆ O ( k 2 ) / (cid:15) ) n + m + N D ) plus the time required for O (( 2 L ∞ ( (cid:96) )
∆ O ( k 2 ) / (cid:15) ) n + m + N ) evaluations of (cid:96) and f .
Corollary 4.6.
The ERM problem (1) over Deep Residual Networks (ResNets) with 1-Lipschitz activations can be solved to (cid:15) -optimality in time poly ( ∆ , 1 / (cid:15), D ) whenever the network size and number of layers are ﬁxed.
Another interesting point can be made with respect to Convolutional Neural Networks (CNN).
In these, convolutional layers are included which help to signiﬁcantly reduce the number of parameters involved in the neural network.
From a theoretical perspective, a CNN can be obtained by simply enforcing certain parameters of a fully-connected DNN to be equal.
This implies that Lemma 4.5 can also be applied to CNNs, with the key diﬀerence residing in parameter N , which is the dimension of the parameter space and does not correspond to the number of edges in a CNN.
In Table 1 we provide explicit LP sizes for common architectures.
These results can be directly obtained from Lemma 4.5, using the speciﬁc Lipschitz constants of the loss functions.
We provide explicit computations in Appendix G.
Conclusion and Final remarks
We have presented a novel framework which shows that training a wide variety of neural networks can be done in time which depends polynomially on the data set size, while satisfying a predetermined arbitrary optimality tolerance.
Our approach is realized by approaching training through the lens of linear programming.
Moreover, we show that training using a particular data set is closely related to the face structure of a data-independent polytope.
Our contributions not only improve the best known algorithmic results for neural network training with optimality/approximation guarantees, but also shed new light on (theoretical) neural network training by bringing together concepts of graph theory, polyhedral geometry, and non-convex optimization as a tool for
Deep Learning.
While the LPs we are constructing are large, and likely to be diﬃcult to solve by straightforward use of our formulation, we strongly believe the theoretical foundations we lay here can also have practical implications in the Machine Learning community.
First of all, we emphasize that all our architecture dependent terms are worst-case bounds, which can be improved by assuming more structure in the corresponding problems.
Additionally, the history of Linear Programming has provided many interesting cases of extremely large LPs that can be solved to near-optimality without necessarily generating the complete LP description.
In these cases the theoretical understanding of the LP structure is crucial to drive the development of incremental solution strategies.
Finally, we would like to point out an interesting connection between the way our approach works and the current most practically eﬀective training algorithm: stochastic gradient descent.
Our LP approach implicitly "decomposes" the problem for each data point, and the LP merges them back together without losing any information nor optimality guarantee, even in the non-convex setting.
This is the core reason why our LP has a linear dependence on D , and bears close resemblance to SGD where single data points (or batches of those) are used in a given step.
As such, our results might provide a new perspective, through low treewidth, on why the current practical algorithms work so well, and perhaps hints at a synergy between the two approaches.
We believe this can be an interesting path to bring our ideas to practice.
A Further definitions A.1 Complementary discussion on Treewidth
The concept of treewidth is an important concept in the context of solving optimization problems with ‘sparse’ structure.
An alternative deﬁnition to Deﬁnition 2.3 of treewidth that the reader might ﬁnd useful is the following; recall that a chordal graph is a graph where every induced cycle has length exactly 3.
Deﬁnition A.1.
An undirected graph G =
( V , E ) has treewidth ≤ ω if there exists a chordal graph H
=
( V , E (cid:48) ) with
E ⊆ E (cid:48) and clique number ≤ ω + 1.
H in the deﬁnition above is sometimes referred to as a chordal completion of G .
In Figure 2 we present an example of a graph and a valid tree-decomposition.
The reader can easily verify that the conditions of
Deﬁnition 2.3 are met in this example.
Moreover, using Deﬁnition A.1 one can verify that the treewidth of the graph in Figure 2 is exactly 2.
Two important folklore results we use in Section C.1 and Section F are the following.
Lemma A.2.
Let G be a graph with a valid tree-decomposition ( T , Q ) of width ω .
Then there exists a valid tree-decomposition ( T (cid:48) , Q (cid:48) ) of width at most ω such that | V ( T (cid:48) )
| ∈
O (| V ( G )|) .
Lemma A.3.
Let G be a graph with a valid tree-decomposition ( T , Q ) and
K ⊆
V ( G ) a clique of G .
Then there exists t ∈ T such that K ⊆ Q t .
A.2 Proper vs. improper learning
An important distinction is the type of solution to the ERM that we allow.
In proper learning we require the solution to satisfy φ ∈ Φ , i.e., the model has to be from the considered model class induced by Φ and takes the form f (· , φ ∗ ) for some φ ∗ ∈ Ω , with
[Equation]
and this can be relaxed to (cid:15) -approximate
(proper) learning by allowing for an additive error (cid:15) > 0 in the above.
In contrast, in improper learning we allow for a model g (·) , that cannot be obtained as f (· , φ ) with φ ∈ Φ , satisfying
[Equation]
with a similar approximate version.
As we mentioned in the main body, this article considers the proper learning setup.
A.3 Neural Networks
In a Neural Network, the graph G deﬁning the network can be partitioned in layers.
This means that
V ( G ) =
(cid:207) ki
=
V i for some sets V i —the layers of the network.
Each vertex v ∈ V i with i ∈
[ k ] 0 has an associated set of in-nodes denoted by δ
+ ( v ) ⊆
V , so that ( w , v ) ∈ E for all w ∈ δ + ( v ) and an associated set of out-nodes δ − ( v ) ⊆
V deﬁned analogously.
If i = 0, then δ + ( v ) are the inputs (from data) and if i =
k , then δ − ( v ) are the outputs of the network.
Moreover, each node v ∈ V performs a node computation g i
( δ + ( v )) , where g i : (cid:82) | δ + ( v )| → (cid:82) with i ∈ [ k ] is typically a smooth function (often these are linear or aﬃne linear functions) and then the node activation is computed as a i ( g i ( δ + ( v ))) , where a i : (cid:82) → (cid:82) with i ∈ [ k ] is a (not necessarily smooth) function (e.g., ReLU activations of the form a i ( x ) = max { 0 , x } ) and the value on all out-nodes w ∈ δ − ( v ) is set to a i ( g i ( δ + ( v ))) for nodes in layer
i ∈
[ k ] .
In feed-forward networks, we can further assume that if v ∈ V
i , then δ + ( v ) ⊆
∪ i − 1 j =
V j , i.e., all arcs move forward in the layers.
B Further discussion Data-dependent vs. independent LPs
As mentioned before, the assumption that the construction of the LP is independent of the speciﬁc data is important and reasonable as it basically prevents us from constructing an LP for a speciﬁc data set , which would be akin to designing an algorithm for a speciﬁc data set in ERM problem (1).
To further illustrate the point, suppose we would do the latter, then a correct algorithm would be a simple print statement of the optimal conﬁguration ¯ φ .
Clearly this is nonsensical and we want the algorithm to work for all types of data sets as inputs.
We have a similar requirement for the construction of the LP, with the only diﬀerence that number of data points D has to be known at time of construction.
As such LPs more closely resemble a circuit model of computation (similar to the complexity class P/poly); see Braun et al.
(2016; 2015); Braun and Pokutta (2018+) for details.
The curious reader might still wonder how our main result changes if we allow the LPs in Theorem 3.1 to be data-dependent , i.e., if we construct a speciﬁc linear program after we have seen the data set:
Remark B.1 .
To obtain a data-dependent linear program we can follow the same approach as in Section 3 and certainly produce an LP that will provide the same approximation guarantees for a ﬁxed data set.
Moreover, since the construction of Theorem 2.5 explained in Appendix C.1 involves an enumeration over a discretization of Φ ⊆ [− 1 , 1 ] N , one can compute the (approximately) optimal solution in advance and solve this data-dependent ERM problem as a trivial LP with N constraints, each constraint simply ﬁxing a variable to a value; the analog to the print statement from above.
The time needed to generate such an LP is
O (( 2 L/ (cid:15) ) N ) (the number of possible discretized conﬁgurations) via at most O (( 2 L/ (cid:15) )
N D ) evaluations of (cid:96) and f (one per each enumerated conﬁguration and data-point).
This result is not particularly insightful, as it is based on a straight-forward enumeration which takes a signiﬁcant amount of time, considering that it only serves one data set.
On the other hand, our result shows that by including the input data as a variable , we do not induce an exponential term in the size of the data set
D
and we can keep the number function evaluations to be roughly the same.
Similarities with SGD-based training
Our approach shares some similarities with stochastic gradient descent (SGD) based training: data points are considered separately (or in small batches) and the method (in case of SGD) or the LP (in our case) ensure that the information gained from a single data point is integrated into the overall ERM solution.
In the case of
SGD this happens through sequential updates of the form x t + 1 ← x t − η ∇ f i ( x t ) , where i is a random function corresponding to a training data point ( ˆ X i , ˆ Y i ) from the ERM problem.
In our case, it is the LP that ‘glues together’ solutions obtained from single training data points by means of leveraging the low treewidth.
This is reﬂected in the linear dependence in D in the problem formulation size.
C Missing proofs
Proof of Lemma 3.1.
Choose binary values ˜
z so as to attain the approximation for variables x , y , φ as in (6) and deﬁne ˆ x , ˆ y , ˆ φ, ˆ L from ˜ z
according to the deﬁnition of S (cid:15) ( D , Φ , (cid:96), f ) .
Since
[Equation]
by Lipschitzness we obtain | L d − ˆ L d | ≤ (cid:15) .
The result then follows.
[Equation]
Proof of Lemma 4.1.
The result can be veriﬁed directly, since for a ∈ [− 1 , 1 ] w and b ∈ [− 1 , 1 ] it holds | z T a + b | ≤ w (cid:107) z (cid:107) ∞ + 1.
[Equation]
Proof of Lemma 4.5.
The proof follows almost directly from the proof of Corollary 4.2.
The two main diﬀerences are (1) the input dimension of a node computation, which can be at most ∆ instead of w and (2) the fact that an activation function a with Lipchitz constant 1 and that a ( 0 ) = 0 satisﬁes | a ( z )| ≤ | z | , thus the domain of each node computation computed in Lemma 4.1 applies.
The layer-by-layer argument can be applied as the network is feed-forward.
[Equation]
C.1 Proof of Theorem 2.
Let us recall the deﬁnition of BO: ( BO ) : min c T x + d T y subject to : f i ( x ) ≥i ∈ [ m ] g j ( x ) =
y j j ∈ [ p ] x ∈ { 0 , 1 } n ,
We sketch the proof of
Theorem 2.5.
Consider an instance I of problem BO .
If Γ [I] has a tree-decomposition ( T , Q ) of width ω , there is an exact linear programming reformulation of I with O ( 2 ω (| V ( T )| + p )) variables and constraints.
Proof.
Since the support of each f i induces a clique in the intersection graph, there must exist a bag Q such that supp ( f i ) ⊆
Q (Lemma A.3).
The same holds for each g j .
We modify the tree-decomposition to include the y j variables the following way:
• For each j ∈ [ p ] , choose a bag Q containing supp ( g j ) and add a new bag Q (cid:48)
( j ) consisting of Q ∪ { y j } and connected to Q .
• We do this for every j ∈ [ p ] , with a diﬀerent Q (cid:48)
( j ) for each diﬀerent j .
This creates a new tree-decomposition ( T (cid:48) , Q (cid:48) ) of width at most ω + 1, which has each variable y j contained in a single bag Q (cid:48)
( j ) which is a leaf .
• The size of the tree-decomposition is | T (cid:48)
| = | T | + p .
From here, we proceed as follows:
• For each t ∈ T (cid:48) , if Q (cid:48) t (cid:51) y j for some j ∈ [ p ] , then we construct
[Equation]
for
[Equation]
otherwise we simply construct
[Equation]
Note that these sets have size at most 2 | Q (cid:48) t | .
• We deﬁne variables X
[ Y , N ] where Y , N form a partition of Q (cid:48)
t 1 ∩ Q (cid:48)
t 2 .
These are at most 2 ω | V ( T (cid:48) )| .
• For each t ∈ T (cid:48) and v ∈ F t , we create a variable λ v .
These are at most 2 ω | V ( T (cid:48) )| .
We formulate the following linear optimization problem ( LBO ) : min c T x + d T y subject to :
[Equation]
Note that the notation in the last constraint is justiﬁed since by construction supp ( g j ) ⊆ Q (cid:48)
( j ) .
The proof of the fact that LBO is equivalent to BO follows from the arguments in Bienstock and Muñoz (2018).
The key diﬀerence justifying the addition of the y variables relies in the fact that they only appear in leaves of the tree decomposition ( T (cid:48) , Q (cid:48) ) , and thus in no intersection of two bags.
The gluing argument using variables X
[ Y , N ] then follows directly, as it is then only needed for the x variables to be binary.
We can substitute out the x and y variables and obtain an LP whose variables are only λ v and X [ Y , N ] .
This produces an LP with at most 2 · 2 ω | V ( T (cid:48) )| variables and ( 2 · 2 ω + 1 )
| V ( T (cid:48) )
| constraints.
This proves the size of the LP is O ( 2 ω (| V ( T )| + p )) as required.
(cid:3)
D Data-dependent Faces and Construction Time D.1 Construction Time in Theorem 3.
In this Section we show how to construct the polytope in Theorem 3.1.
We ﬁrst recall the following deﬁnition:
S ( D , Φ , (cid:96), f ) = {( x 1 , . . .
, x D , y 1 , . . .
, y D , φ, L ) ∈ [− 1 , 1 ] ( n + m )× D × Φ × (cid:82) D : L d = (cid:96) ( f ( x d , φ ) , y d )} and recall that S (cid:15)
( D , Φ , (cid:96), f ) is a discretized version of the set mentioned above.
From the tree-decomposition detailed in Section 3.2, we see that data-dependent variables x , y , L are partitioned in diﬀerent bags for each data d ∈ [ D ] .
Let us index the bags using d .
Since all data variables have the same domain, the sets F d we construct in the proof of Theorem 2.5 will be the same for all d ∈ [ D ] .
Using this observation, we can construct the LP as follows:
Fix, say, d = 1 and enumerate all binary vectors corresponding to the discretization of x 1 , y 1 , φ . 2.
Compute (cid:96) ( f ( x 1 , φ ) , y 1 ) .
This will take O (( 2 L/ (cid:15) ) n + m + N ) function evaluations of f and (cid:96) .
This deﬁnes the set F 1 . 3.
Duplicate this set D times, and associate each copy with a bag indexed by d ∈ [ D ] .
For each d ∈ [ D ] , and each v ∈ F d create a variable λ v . 5.
For each d ∈ [ D − 1 ] , create variables X
[ Y , N ] corresponding to the intersection of bags d and d + 1.
This will create O (( 2 L/ (cid:15) ) N ) variables, since the only variables in the intersections are the discretized φ variables.
Formulate LBO .
The only evaluations of (cid:96) and f are performed in the construction of F 1 .
As for the additional computations, the bottleneck lies in creating all λ variables, which takes time O (( 2 L/ (cid:15) ) n + m + N D ) .
Remark D.1 .
Note that in step 1 of the LP construction we are enumerating all possible discretized values of x 1 , y 1 , i.e., we are implicitly assuming all points in [− 1 , 1 ] n + m are possible inputs.
This is reﬂected in the ( 2 L/ (cid:15) ) n + m term in the LP size estimation, and is the sole source for the exponential dependence on the input dimension.
If one were to use another discretization method (or a diﬀerent “point generation” technique) using more information about the input data, this term could be improved and the exponential dependency on the input dimension of the LP size could be alleviated signiﬁcantly.
We leave the full development of this improvement for future work.
D.2 Data-dependent faces of the data-independent polytope
Consider a ﬁxed data set ( ˆ X , ˆ Y )
=
( ˆ x i , ˆ y i ) Di
= 1 and let φ ∗ be an optimal solution to the ERM problem with input data ( ˆ X , ˆ Y ) .
Consider now binary variables
z ˆ x , z ˆ y to attain the approximation (6) and deﬁne ˜ x , ˜ y
from z ˆ x , z ˆ y , i.e., ˜ x di = − 1 + 2 (cid:205) N γ h = 1 2 − h
z
ˆ x d i , h and similarly for ˜ y .
Deﬁne the set
[Equation]
and similarly as before deﬁne S (cid:15) ( ˜ X , ˜ Y , Φ , (cid:96), f ) to be the discretized version (on variables φ ).
The following Lemma shows the quality of approximation to the ERM problem obtained using S ( ˜ X , ˜ Y , Φ , (cid:96), f ) and subsequently S (cid:15) ( ˜ X , ˜ Y , Φ , (cid:96), f ) .
Lemma D.2.
For any ( φ, L ) ∈ S ( ˜ X , ˜ Y , Φ , (cid:96), f ) there exists ( φ (cid:48) , L (cid:48) )
∈ S (cid:15) ( ˜ X , ˜ Y , Φ , (cid:96), f ) such that
[Equation]
Additionally, for every φ ∈ Φ , there exists ( φ (cid:48) , L (cid:48) )
∈ S (cid:15) ( ˜ X , ˜ Y , Φ , (cid:96), f ) such that
[Equation]
Proof.
The ﬁrst inequality follows from the same proof as in Lemma 3.1.
For the second inequality, let φ (cid:48) be the binary approximation to φ , and L (cid:48) deﬁned by L (cid:48)
d = (cid:96) ( f ( ˜ x d , φ (cid:48) ) , ˜ y d ) .
Since ˜ x , ˜ y , φ (cid:48) are approximations to
ˆ x , ˆ y , φ , by Lipschitzness we know that
[Equation]
Corollary D.3.
Let ( ˆ φ, ˆ L ) deﬁned as
[Equation]
L ) ∈ argmin
[Equation]
Then
[Equation]
Proof.
Since ˆ φ ∈ Φ , and φ ∗ is a “true” optimal solution to the ERM problem, we immediately have
[Equation]
On the other hand, by the previous Lemma we know there exists ( φ (cid:48) , L (cid:48) )
∈ S (cid:15) ( ˜ X , ˜ Y , Φ , (cid:96), f ) such that
[Equation]
Note that since the objective is linear, the optimization problem in the previous Corollary is equivalent if we replace S (cid:15) ( ˜ X , ˜ Y , Φ , (cid:96), f ) by its convex hull.
Therefore the only missing link to the face property of the data-independent polytope is the following:
Lemma D.4.
conv ( S (cid:15) ( ˜ X , ˜ Y , Φ , (cid:96), f )) is a face of conv ( S (cid:15)
( D , Φ , (cid:96), f )) .
Proof.
The proof follows from simply ﬁxing variables in the corresponding LBO that describes conv ( S (cid:15)
( D , Φ , (cid:96), f )) .
For every d ∈ [ D ] and v ∈ F d , we simply need to make λ v = 0 whenever the ( x , y ) components of v do not correspond to ˜ X , ˜ Y .
We know this is well deﬁned, since ˜ X , ˜ Y are already discretized, thus there must be some v ∈ F d corresponding to them.
The structure of the resulting LP is the same as LBO , so the fact that it is exactly conv ( S (cid:15) ( ˜ X , ˜ Y , Φ , (cid:96), f )) follows.
The fact that it is a face of conv ( S (cid:15) ( D , Φ , (cid:96), f )) follows from the fact that the procedure simply ﬁxed some inequalities to be tight.
(cid:3)
E Regularized ERM
A common practice to avoid over-ﬁtting is the inclusion of regularizer terms in (1).
This leads to problems of the form
[Equation]
where R (·) is a function, typically a norm, and λ > 0 is a parameter to control the strength of the regularization.
Regularization is generally used to promote generalization and discourage over-ﬁtting of the obtained ERM solution.
The reader might notice that our arguments in Section 3 regarding the epigraph reformulation of the
ERM problem and the tree-decomposition of its intersection graph can be applied as well, since the regularizer term does not add any extra interaction between the data-dependent variables.
The previous analysis extends immediately to the case with regularizers after appropriate modiﬁcation of the architecture Lipschitz constant L to include R (·) .
Deﬁnition E.1.
Consider a regularized ERM problem (9) with parameters D , Φ , (cid:96), f , R , and λ .
We deﬁne its
Architecture Lipschitz Constant L( D , Φ , (cid:96), f , R , λ ) as
[Equation]
over the domain K = [− 1 , 1 ] n × Φ × [− 1 , 1 ] m .
F ERM under Network Structure
So far we have considered general ERM problems exploiting only the structure of the ERM induced by the ﬁnite sum formulations.
We will now study ERM under Network Structure, i.e., speciﬁcally ERM problems as they arise in the context of Neural Network training.
We will see that in the case of Neural Networks, we can exploit the sparsity of the network itself to obtain better LP formulations of conv ( S (cid:15)
( D , Φ , (cid:96), f )) .
Suppose the network is deﬁned by a graph G , and recall that in this case, Φ ⊆
[− 1 , 1 ] E (G) .
By using additional auxiliary variables s representing the node computations and activations, we can describe S ( D , Φ , (cid:96), f ) in the following way:
[Equation]
The only diﬀerence with our original description of S ( D , Φ , (cid:96), f ) in (5) is that we explicitly “store” node computations in variables s .
These new variables will allow us to better use the structure of G .
Assumption F.1 .
To apply our approach in this context we need to further assume Φ to be the class of Neural
Networks with normalized coeﬃcients and bounded node computations .
This means that we restrict to the case when s ∈ [− 1 , 1 ] | V (G)| D .
Under Assumption F.1 we can easily derive an analog description of S (cid:15)
( D , Φ , (cid:96), f ) using this node-based representation of S (cid:15)
( D , Φ , (cid:96), f ) .
In such description we also include a binary representation of the auxiliary variables s .
Let Γ be the intersection graph of such a formulation of S (cid:15)
( D , Φ , (cid:96), f ) and Γ φ be the sub-graph of Γ induced by variables φ .
Using a tree-decomposition ( T , Q ) of Γ φ we can construct a tree-decomposition of Γ the following way:
We duplicate the decomposition D times ( T i , Q i ) Di = 1 , where each
We connect the trees T i in a way that the resulting graph is a tree (e.g., they can be simply concatenated one after the other).
To each bag Q it with t ∈ T i and i ∈
[ D ] , we add all the data-dependent variables L d and the binary variables associated with the discretization of x d , s · , d , and y d .
This adds N γ (| V (G)| + n + m ) additional variables to each bag, as there is only one variable s per data point per vertex of G .
It is not hard to see that this is a valid tree-decomposition of Γ , of size | T | · D —since the bags were duplicated
D times— and width N γ ( t w ( Γ φ ) + | V (G)| + n + m ) .
We now turn to providing a bound to t w ( Γ φ ) .
To this end we observe the following:
The architecture variables φ are associated to edges of G .
Moreover, two variables φ e , φ f , with e , f ∈ E appear in a common constraint if and only if there is a vertex v such that e , f ∈ δ + ( v ) .
This implies that Γ φ is a sub-graph of the line graph of G .
Recall that the line graph of a graph G is obtained by creating a node for each edge of G and connecting two nodes whenever the respective edges share a common endpoint.
The treewidth of a line graph is related to the treewidth of the base graph (see Bienstock (1990); Calinescu et al.
; Atserias (2008); Harvey and Wood (2018)).
More speciﬁcally, t w ( Γ φ ) ∈
O ( t w (G) ∆ (G)) where ∆ denotes the maximum vertex degree.
Addititionally, using Lemma A.2 we may assume | T | ≤ | E (G)| , since Γ φ has at most | E (G)| nodes.
Putting everything together we obtain:
Lemma F.2.
If there is an underlying network structure G in the ERM problem and the node computations are bounded, then conv ( S (cid:15) ( D , Φ , (cid:96), f )) admits a linear programming formulation with no more than 2 D (| E (G)| + 1 ) (cid:18) 2 L (cid:15) (cid:19)
O ( tw (G) ∆ (G) + | V (G)| + n + m ) and
D (| E (G)| + 1 ) (cid:32) 2 (cid:18) 2 L (cid:15) (cid:19)
O
( tw (G) ∆ (G) + | V (G)| + n + m ) + 1 (cid:33) variables and constraints, respectively.
Moreover, given a tree-decomposition of the network G , the linear program can be constructed in time O (cid:16) D | E (G)| ( 2 L/ (cid:15) )
O ( tw (G) ∆ (G) + | V (G)| + n + m ) (cid:17) plus the time required for
O (cid:16) | E (G)| ( 2 L/ (cid:15) )
O ( tw (G) ∆ (G) + | V (G)| + n + m ) (cid:17) evaluations of (cid:96) and f .
G Explicit Lipschitz constants of common Loss Functions
In Section 4 we speciﬁed our results —the size of the data-independent LPs— for feed-forward networks with 1-Lipschitz activation functions.
However, we kept as a parameter L ∞ ( (cid:96) ) ; the Lipschitz constant of (cid:96) (· , ·) over [− U k , U k ] m × [− 1 , 1 ] m , with U k
=
(cid:205)
kj =
w j a valid bound on the output of the node computations, as proved in Lemma 4.1.
Note that U k ≤ w k + 1 .
In this Section we compute this Lipschitz constant for various common loss functions.
It is important to mention that we are interested in the Lipschitznes of (cid:96) with respect to both the output layer and the data-dependent variables as well —not a usual consideration in the literature
Note that a bound on the Lipschitz constant L ∞ ( (cid:96) ) is given by sup z , y (cid:107)∇ (cid:96) ( z , y )(cid:107) 1 .
• Quadratic Loss (cid:96) ( z , y )
= (cid:107)
z − y (cid:107) 22 .
In this case it is easy to see that (cid:107)∇ (cid:96) ( z , y )(cid:107) 1 = 4 (cid:107)
z − y (cid:107) 1 ≤ 4 m ( U k + 1 ) ≤ 4 m ( w k + 1 + 1 )
• Absolute Loss (cid:96) ( z , y ) =
(cid:107)
z − y (cid:107) 1 .
In this case we can directly verify that the Lipschitz constant with respect to the inﬁnity norm is at most 2 m .
• Cross Entropy Loss with Soft-max Layer.
In this case we include the Soft-max computation in the deﬁnition of (cid:96) , therefore
[Equation]
where S ( z ) is the Soft-max function deﬁned as
[Equation]
A folklore result is
[Equation]
Additionally,
[Equation]
which in principle cannot be bounded.
Nonetheless, since we are interested in the domain [− U k
[Equation]
U k ] of z
[Equation]
we obtain
[Equation]
which implies that L ∞ ( (cid:96) ) ≤ 2 m ( log ( m )
+ 2 U k ) ≤ 2 m ( log ( m ) +
2 w k + 1 ) .
• Hinge Loss (cid:96) ( z , y )
=
max { 1 − z T x , 0 } .
Using a similar argument as for the Quadratic Loss, one can easily see that the Lipschitz constant with respect to the inﬁnity norm is at most m
( U k + 1 ) ≤ m ( w k + 1 + 1 ) .
H
Binarized Neural Networks
A Binarized activation unit (BiU) is parametrized by p + 1 values b , a 1 , . .
., a p .
Upon a binary input vector z 1 , z 2 , . . .
, z p the output is binary value y deﬁned by:
[Equation]
Now suppose we form a network using BiUs, possibly using diﬀerent values for the parameter p .
In terms of the training problem we have a family of (binary) vectors x 1 , . . .
, x D in (cid:82) n and binary labels and corresponding binary label vectors y 1 , . . .
, y D in (cid:82) m , and as before we want to solve the ERM problem (1).
Here, the parametrization φ refers to a choice for the pair ( a , b ) at each unit.
In the speciﬁc case of a network withnodes in the ﬁrst layer and 1 node in the second layer, and m = 1, Blum and Rivest (1992) showed that it is
NP-hard to train the network so as to obtain zero loss, when n = D .
Moreover, the authors argued that even if the parameters ( a , b ) are restricted to be in {− 1 , 1 } , the problem remains NP-Hard.
See Courbariaux et al.
(2016) for an empirically eﬃcient training algorithm for BiUs.
In this section we apply our techniques to the ERM problem (1) to obtain an exact polynomial-size dataindependent formulation for each ﬁxed network (but arbitrary D ) when the parameters ( a , b ) are restricted to be in {− 1 , 1 } .
We begin by noticing that we can reformulate (1) using an epigraph formulation as in (4).
Moreover, since the data points in a BiU are binary, if we keep the data points as variables, the resulting linear-objective optimization problem is a binary optimization problem as BO .
This allows us to claim the following:
Theorem H.1.
Consider a graph G , p ∈ (cid:78) and D ∈ (cid:78) .
There exists a linear program of size
[Equation]
such that any BiU ERM problem of the form (1) is equivalent to optimizing a linear function over a face of P , where P is the polytope deﬁned by the linear program.
Constructing the linear program takes time
O ( 2 p | V (G)| D ) plus the time required for O ( 2 p | V (G)| ) evaluations of f and (cid:96) .
Proof.
The result follows from applying Theorem 2.5 directly to the epigraph formulation of BiU keeping x and y as variables.
In this case an approximation is not necessary.
The construction time and the data-independence follow along the same arguments used in the approximate setting before.
[Equation]
The following corollary is immediate.
Corollary H.2.
The ERM problem (1) over BiUs can be solved in polynomial time for any D , whenever p and the network structure G are ﬁxed.
I Linear Programming-based Training Generalizes
In this section we will show that the ERM solutions obtained via Linear Programming generalize to the
General Risk Minimization problem.
Here we show generalization as customary in stochastic optimization, exploiting the Lipschitzness of the model to be trained; we refer the interested reader to Ahmed (2017);
Shapiro et al.
(2009) for an in-depth discussion.
In a ﬁrst step, we further precise notation as required for our analysis.
To this end, recall that the General Risk Minimization (GRM) is deﬁned as min φ ∈ Φ GRM ( φ ) (cid:17)
min φ ∈ Φ (cid:69) ( x , y )∈D [ (cid:96) ( f ( x , φ ) , y )] , where (cid:96) is some loss function , f is a neural network architecture with parameter space Φ , and ( x , y ) ∈ (cid:82) n + m drawn from the distribution D .
We solve the ﬁnite sum problem, i.e., the empirical risk minimization problem min φ ∈ Φ ERM X , Y ( φ ) (cid:17) min φ ∈ Φ 1 D (cid:205)
Di = 1 (cid:96) ( f ( x i , φ ) , y i ) , instead, where ( X , Y ) =
( x i , y i )
Di = 1 is an i.i.d.
sample from data distribution D of size D .
We will show in this section, for any 1 > α > 0, (cid:15) > 0, we can choose a (reasonably small!)
sample size D , so that with probability 1 − α it holds:
[Equation]
where ¯ φ ≤ max φ ∈ Φ ERM X , Y ( φ ) + (cid:15) is an (cid:15) -approximate solution to ERM X , Y for i.i.d.-sampled data ( X , Y ) ∼ D .
As the size of the linear program that we use for training only linearly depends on the number of data points, this also implies that we will have a linear program of reasonable size as a function of α and (cid:15) .
The following proposition summaries the generalization argument used in stochastic programming as presented in Ahmed (2017) (see also Shapiro et al.
Proposition I.1.
Consider the optimization problem
[Equation]
where γ ( ω ) is a random parameter with ω ∈ Ω a set of parameters, X ⊆ (cid:82) n a ﬁnite set, and F :
X × Ω → (cid:82) is a function.
Given i.i.d.
samples γ 1 , . .
., γ D of γ ( ω ) , consider the ﬁnite sum problem
[Equation]
If ¯ x ∈ X is an (cid:15) -approximate solution to (11) , i.e., 1 D (cid:205)
[Equation]
≤
[Equation]
∈[ D ] F ( x , γ i ) + (cid:15) and
[Equation]
log
[Equation]
where α > 0 and σ 2 = max x ∈ X (cid:86) ω ∈ Ω
[ F ( x , γ ( ω ))] , then with probability 1 − α it holds:
[Equation]
Let σ 2 = max φ ∈ Φ (cid:86) ( x , y )∈D [ (cid:96) ( f ( x , φ ) , y )] .
We will now establish generalization by means of Proposition I. and a straightforward discretization argument.
By assumption from above Φ ⊆ [− 1 , 1 ] N for some N ∈ (cid:78) .
Let Φ ν ⊆
Φ ⊆
[− 1 , 1 ] N be a ν -net of Φ , i.e., for all φ ∈ Φ there exists ¯ φ ∈
Φ ν with (cid:107) φ − ¯ φ (cid:107) ∞ ≤ ν .
Furthermore let
L the be architecture Lipschitz constant, as deﬁned in (2) (or (10)).
Theorem I.2.
[Generalization] Let ¯ φ ∈
Φ be an (cid:15) -approximate solution to min φ ∈ Φ ERM X , Y ( φ ) with (cid:15) > 0 , i.e., ERM X , Y ( ¯ φ ) ≤ min φ ∈
Φ ERM X , Y ( φ ) + (cid:15).
If D ≥
[Equation]
σ
[Equation]
2 log (
[Equation] [Equation]
α , with L and σ 2 as above, then with probability 1 − α it holds GRM ( ¯ φ ) ≤ min φ ∈
Φ GRM ( φ ) + 6 (cid:15), i.e., ¯ φ is a 6 (cid:15) -approximate solution to min φ ∈ Φ GRM ( φ ) .
Proof.
Let ¯ φ be as above.
With the choice ν (cid:17) (cid:15) /L , there exists ˜ φ ∈ Φ ν , so that (cid:107) ˜
φ − ¯ φ (cid:107) ∞ ≤ ν and hence by
Lipschitzness,
[Equation]
so that ERM X , Y ( ˜ φ ) ≤ min φ ∈ Φ ν ERM X , Y ( φ ) + 2 (cid:15) .
As D ≥ 4 σ 2 (cid:15) 2 log (( 2 L)/ (cid:15) )
N α , with probability 1 − α
we have
GRM ( ˜ φ ) ≤ min φ ∈ Φ ν GRM ( φ ) + 4 (cid:15) by Proposition I.1.
If now ¯ φ G = argmin φ ∈ Φ GRM ( φ ) and ˜ φ G ∈
Φ ν with (cid:107) ¯ φ G − ˜ φ G (cid:107) ∞
≤ ν , by Lipschitzness we have | GRM ( ¯ φ G ) − GRM ( ˜ φ G )| ≤ (cid:15) .
Now
[Equation]
(by optimality)
[Equation]
(by Lipschitzness) .
Together with | GRM ( ¯ φ ) − GRM ( ˜ φ )| ≤ (cid:15) as (cid:107) ˜ φ − ¯ φ (cid:107) ∞ ≤ ν it follows
[Equation]
which completes the proof.
[Equation]
We are ready to formulate the following corollary combining Theorem I.2 and Main Theorem 3.1.
Corollary I.3 (LP-based Training for General Risk Minimization) .
Let D be a data distribution as above.
Further, let 1 > α > 0 and (cid:15) > 0 , then there exists a linear program with the following properties:
(a) The linear program has size O (cid:18)(cid:16)
2 L (cid:15) (cid:17) n + m + N (cid:16)
[Equation]
and can be constructed in time
O
[Equation]
plus the time required for O (cid:16) ( 2 L/ (cid:15) ) n + m + N (cid:17) evaluations of (cid:96) and f , where L and σ 2 as above.
(b) With probability ( 1 − α ) it holds GRM ( ¯ φ ) ≤ min φ ∈ Φ GRM
[Equation]
φ )
[Equation]
6 (cid:15) , where ¯ φ is an optimal solution to the linear program obtained for the respective sample of D of size 4 σ
[Equation] [Equation]
α
Similar corollaries hold, combining Theorem I.2 with the respective alternative statements from Section 4.
Of particular interest for what follows is the LP size in the case of a neural network with k layers with width w , which becomes
[Equation] [Equation]
A closely related result regarding an approximation to the GRM problem for neural networks is provided by
Goel et al.
(2017) in the improper learning setting.
The following corollary to (Goel et al., 2017, Corollary 4.5) can be directly obtained, rephrased to match our notation:
Theorem I.4 (Goel et al.
There exists an algorithm that outputs ˜ φ such that with probability 1 − α , for any distribution D and loss function (cid:96) which is convex, L -Lipschitz in the ﬁrst argument and b bounded on [− 2 √ w , √ w ] , GRM ( ˜ φ )
≤ min φ ∈ Φ GRM ( φ ) + (cid:15) , where Φ is the class of neural networks with k hidden layers, width w , output dimension m = 1 , ReLU activations and normalized weights.
The algorithm runs in time at most
[Equation]
Remark I.5 .
In contrast to the above result of Goel et al. (2017), note that in our paper we consider the proper learning setting, where we actually obtain a neural network. In addition we point out several key diﬀerences between Theorem I.4 and the algorithmic version of our result when solving the LP in Corollary I.3 of size as (12): (a) In (13), the dependency on the input dimension is better than in (12).
(b) The dependency on the Lipschitz constant is signiﬁcantly better in (12), although we have to point out that we are relying on the Lipschitz constant with respect to all inputs of the loss function and in a potentially larger domain.
(c) The dependency on (cid:15) is also better in (12).
(d) We are not assuming convexity of (cid:96) and we consider general m .
(e) The dependency on k in (12) is much more benign than the one in (13), which is doubly exponential.
Remark I.6 .
Since the ﬁrst submission of this article, a manuscript by Manurangsi and Reichman (2018) was published which extended the results by Goel et al.
This work provides an algorithm with similar characteristics to the one by Goel et al.
but in the proper learning setting, for depth-2 ReLU networks with convex loss functions.
The running time of the algorithm (rephrased to match our notation) is ( n / α ) O ( 1 ) 2 ( w / (cid:15) )
O ( 1 ) .
Analogous to the comparison in Remark I.5, we obtain a much better dependence with respect to (cid:15) and we do not rely on convexity of the loss function or on constant depth of the neural network.