Improved Bounds on Neural Complexity for Representing Piecewise Linear Functions
Author(s
Abstract
A deep neural network using rectified linear units represents a continuous piecewiselinear (CPWL) function and vice versa.
Recent results in the literature estimatedthat the number of neurons needed to exactly represent any CPWL function growsexponentially with the number of pieces or exponentially in terms of the factorialof the number of distinct linear components.
Moreover, such growth is amplifiedlinearly with the input dimension.
These existing results seem to indicate thatthe cost of representing a CPWL function is expensive.
In this paper, we proposemuch tighter bounds and establish a polynomial time algorithm to find a networksatisfying these bounds for any given CPWL function.
We prove that the numberof hidden neurons required to exactly represent any CPWL function is at most aquadratic function of the number of pieces.
In contrast to all previous results, thisupper bound is invariant to the input dimension.
Besides the number of pieces, wealso study the number of distinct linear components in CPWL functions.
Whensuch a number is also given, we prove that the quadratic complexity turns intobilinear, which implies a lower neural complexity because the number of distinctlinear components is always not greater than the minimum number of pieces in a
CPWL function.
When the number of pieces is unknown, we prove that, in termsof the number of distinct linear components, the neural complexity of any CPWLfunction is at most polynomial growth for low-dimensional inputs and a factorialgrowth for the worst-case scenario, which are significantly better than existingresults in the literature.
Introduction
The rectified linear unit (ReLU)
[Fukushima, 1980, Nair and Hinton, 2010] activation has beenby far the most widely used nonlinearity and successful building block in deep neural networks(DNNs).
Numerous architectures based on ReLU DNNs have achieved remarkable performanceor state-of-the-art accuracy in speech processing [Zeiler et al., 2013, Maas et al., 2013], computervision [Krizhevsky et al., 2012, Simonyan and Zisserman, 2015, He et al., 2016], medical imagesegmentation [Ronneberger et al., 2015], game playing [Mnih et al., 2015, Silver et al., 2016], andnatural language processing [Vaswani et al., 2017], just to name a few.
Besides such unprecedentedempirical success, ReLU DNNs are also probably the most understandable nonlinear deep learningmodels due to their ability to be “un-rectified”
[Hwang and Heinecke, 2019].
The ability to demystify ReLU DNNs via “un-rectifying ReLUs” dates back to a seminal workby Pascanu et al.
in 2014.
Because each of ReLUs in a hidden layer divides the space of thepreceding layer’s output into two half spaces whose ReLU response is affine in one half space andexactly zero in the other, the layer of ReLUs can be replaced by an input-dependent diagonal matrixwhose diagonal elements are ones for firing ReLUs and zeros for non-firing ReLUs.
Based on this n =n = 2 n =invariant to n
Number of pieces q
N u m b e r
o f h
i dd e n n e u r on s h
Number of distinct linear components k
N u m b e r
o f h
i dd e n n e u r on s h
Theorem 1 and[Hertrich et al., 2021]
[He et al., 2020] rationale, Pascanu et al.
[2014] proved that a neural network using ReLUs divides the input spaceinto many linear regions such that the network itself is an affine function within every region.
Twoexcellent visualizations are shown in Figure 2 in [Hanin and Rolnick, 2019a] and [Hanin and Rolnick,2019b].
At this point, it is quite evident that any ReLU network exactly represents a CPWL function.
Pascanu et al.
also proved that the maximum number of linear regions for any ReLU network with asingle hidden layer is equivalent to the number of connected components induced by arrangementsof hyperplanes in general position where each hyperplane corresponds to a ReLU in the hiddenlayer.
Such a number can be computed in a closed form by Zaslavsky’s Theorem [Zaslavsky, 1975].
Furthermore, they showed that the maximum number of linear regions can be bounded from belowby exponential growth in terms of the number of hidden layers, leading to a conclusion that ReLU
DNNs can generate more linear regions than their shallow counterparts.
In the same year, Montúfaret al.
improved such a lower bound and gave the first upper bound for the maximum number of linearregions.
These bounds and their assumptions were later improved by [Montúfar, 2017, Raghu et al.
,2017, Arora et al., 2018, Serra et al., 2018, Hinz and van de Geer, 2019], just to name a few.
We referreaders to Hinz’s doctoral thesis for a thorough discussion on the upper bound of the number of linearregions.
Because a CPWL function with more pieces can better approximate any given continuousfunction and a ReLU DNN exactly represents a CPWL [Arora et al., 2018], a ReLU DNN with morelinear regions in general exhibits stronger expressivity.
In summary, this “un-rectifying” perspectiveprovides us a new angle to understand ReLU DNNs, and the results in some ways align with theadvance in approximation theory demonstrating the expressivity.
Despite these advancements in linear regions, the complexity of a ReLU DNN that exactly representsa given CPWL function remains largely unexplored.
One can find that this question is the oppositedirection of the above-mentioned line of research.
Although Arora et al.
[2018] proved that any
CPWL function can be exactly represented by a ReLU DNN with a bounded depth, any estimatesregarding the width or number of neurons of such a network were not given.
The resources requiredfor a ReLU neural network to exactly represent a CPWL function remained unknown until He et al.[2020] first gave an estimate to the complexity of a ReLU network that realizes any given CPWLfunction.
They proved that the number of neurons is bounded from above by exponential growthin terms of the product between number of pieces and the number of distinct linear components ofa given CPWL function.
Such an exponential bound also grows linearly with the input dimension.
Because the number of pieces is an upper bound of the number of distinct linear components for any
CPWL function [Tarela and Martinez, 1999, He et al., 2020], the bound grows exponentially with the quadratic number of pieces, which seems to imply that the cost for representing a CPWL function bya ReLU DNN is exceedingly high.
The most recent upper bound can be inferred from a recent work by Hertrich et al.
[2021] althoughthe number of hidden neurons was not directly given.
Hertrich et al.
[2021] proved the first widthbound in terms of the number of pieces under the same depth used by Arora et al.
[2018] and He et al.[2020].
In particular, they proved that the width of a ReLU network that represents any given CPWLfunction can be polynomially bounded from above in terms of the number of pieces.
However, theorder of such a polynomial is a quadratic function of the input dimension, which can be immenselylarge for a small number of pieces if the input dimension is large.
This bound grows larger withthe input dimension even though the underlying CPWL function is just a one-hidden-layer ReLUnetwork using only one ReLU.
In this paper, we provide improved bounds showing that any CPWL function can be representedby a ReLU DNN whose neural complexity is bounded from above by functions with much slowergrowth (see Figure 1).
Our results imply that one can exactly realize any given CPWL function bya ReLU network at a much lower cost.
Moreover, in addition to guarantee the existence, we alsogive a polynomial time algorithm to exactly find a network satisfying our bounds.
To the best of ourknowledge, our results regarding the computational resource for a ReLU network, i.e., the number ofhidden neurons, are the lowest upper bounds in the existing literature and the algorithm is the firsttailored procedure to find a network representation from any given CPWL function.
Key results andmain contributions of this paper are highlighted below.
1.1 Key results and contributions
Quadratic bounds.
We prove that any CPWL function with q pieces can be represented by a ReLUnetwork whose total number of hidden neurons is bounded from above by a quadratic function of q .
We also give the corresponding upper bounds for the maximum width, i.e., the maximum numberof neurons per hidden layer, and the number of layers for such a network.
The maximum width isbounded from above by O ( q 2 ) and the number of layers is bounded from above by a logarithmicfunction of q , i.e., O (log 2 q ) .
These bounds are invariant to the input dimension.
For any affinefunction, the upper bounds for the maximum width and the number of hidden neurons are zero.
Further improvements on neural complexity.
When the number of distinct linear componentsk of any CPWL function is given along with the number of pieces q , the quadratic bounds O ( q 2 )for the number of hidden neurons and the maximum width turn into bilinear bounds of k and q , i.e.,
O ( kq ) .
Such a change reduces the neural complexity because k ≤ q and q is usually much largerthan k with a high dimensional input.
Still, these bounds are independent of the input dimension.
Finding a network satisfying bilinear bounds.
We establish a polynomial time algorithm (see
Algorithm 1) that finds a ReLU network representing any given CPWL function.
The networkfound by the algorithm satisfies the bilinear bounds on the number of hidden neurons and themaximum width, and the logarithmic bound on the number of layers.
Note that such an algorithmalso guarantees that one can always reverse-engineer at least one ReLU network from the function itcomputes.
Compared to the general-purpose reverse-engineering algorithm proposed by Rolnick and
Kording [2020], our algorithm specializes in the situation when the pieces of CPWL are given.
Improved bounds from a perspective of linear components.
When the number of pieces of a
CPWL function is unknown and only the number of linear components k is available, we prove thatthe number of hidden neurons and maximum width are bounded from above by factorial growth.
More precisely, O ( k · k !) .
The number of layers is bounded from above by linearithmic growth,or O ( k log 2 k ) .
However, if the input dimension n grows sufficiently slower than k , e.g., O (cid:16) √ k (cid:17) ,then bounds for the number of hidden neurons and maximum width reduce to polynomial growthfunctions of order 2 n + 1 ; and the linearithmic growth reduces to O ( n log 2 k ) for the depth.
A new approach to choosing depth.
Instead of scaling the depth of a ReLU network with theinput dimension [Arora et al., 2018, He et al., 2020, Hertrich et al., 2021], we reveal that constructinga ReLU network whose depth is scaled with the number of pieces of the given CPWL function ismore advantageous.
Such a scaling turns out to be the key to derive better upper bounds.
This insight is provided by the max-min representation of CPWL functions [Tarela et al., 1990].
The importanceof this scaling on the depth in ReLU networks has not been well recognized by existing bounds in theliterature.
We discuss implications of different representations in Section 4.
Preliminaries
Notation and definitions used in this paper are set up and clarified in this section.
The set{ 1 , 2 , · · · , m } is denoted by [ m ] .
I [ condition ] is an indicator function that gives 1 if the conditionis true, and 0 otherwise.
The CPWL function is defined by Definition 1 below.
Definition 1.
A function p : R n → R is said to be CPWL if there exists a finite number of closedsubsets of R n , say {X i } i ∈ [ m ] , such that (a) R n = (cid:83) i ∈
[ m ] X i ; (b) p is affine on X i , ∀ i ∈ [ m ] .
A family of closed convex subsets of R n satisfying Definition 1 are also referred to as convex regions,affine pieces or simply pieces for a CPWL function in this paper.
Definition 1 follows the definitionof CPWL functions by Ovchinnikov [2002].
Notice that there are different definitions in the literature.
For example, Chua and Deng [1988] and Arora et al.
[2018] defined a CPWL function on a finitenumber of polyhedral regions.
However, their definitions are essentially the same as Definition 1because any family of closed subsets satisfying Definition 1 can be decomposed into polyhedralregions.
It is possible that some of the closed subsets satisfying Definition 1 are non-convex eventhough the number of them reaches the minimum (see Figure 2 in [Wang and Sun, 2005]).
Thecontinuity is implied by Definition 1 due to the subsets being closed.
Because the goal of this paper is to bound the complexity of a ReLU DNN that exactly represents anygiven CPWL function, it is necessary to be able to measure the complexity of a CPWL function.
Thecomplexity of a CPWL function can be described using two quite different perspectives.
One is thenumber of pieces q , which is the number of closed convex subsets satisfying Definition 1.
Becausethis number has a minimum and any finite number above the minimum can be a valid m in Definition1, the bounds become obviously loose when the number of pieces is not the minimum.
Without lossof generality, we are interested in the number q when it is the minimum.
The other is the number ofdistinct linear components k .
A linear component of a CPWL function is defined in Definition 2.
Definition 2.
An affine function f is said to be a linear component of a CPWL function p if thereexists a nonempty subset M ⊆
[ m ] such that f ( x ) = p ( x ) , ∀ x ∈ (cid:83) i
∈M X i where {X i } i ∈ [ m ] is afamily of minimum number of closed subsets satisfying Definition 1.
A greater q or k gives a CPWL function more degrees of freedom because a CPWL function allowedto use q + 1 pieces or k + 1 arbitrary linear components can represent any CPWL function with qpieces or k distinct linear components and still have a flexibility to modify existing affine maps orincrease the number of distinct affine maps of the CPWL function.
Although increasing them bothleads to a CPWL function with a higher flexibility, the speed of upgrading degrees of freedom isdifferent from each other.
Note that a CPWL function with q pieces can never have more than qdistinct linear components and a CPWL function with k distinct linear components can easily havemore than k minimum number of pieces.
Such a difference in a 1 -dimensional case can be clearlyobserved from Figure 1 in [Tarela and Martinez, 1999].
Note that it is possible for two disjoint subsetsfrom a minimum number of closed subsets satisfying Definition 1 to share the same linear component.
In other words, a linear component can be reused by multiple pieces.
Hence, increasing k gives muchfaster growth than increasing q for the complexity and expressivity of CPWL functions.
We define the ReLU activation function in Definition 3.
The ReLU network defined in Definition 4is a simple architecture which is usually referred to as a ReLU multi-layer perceptron .
Definition 5defines the corresponding number of hidden neurons, depth, and the maximum width.
Definition 3.
The rectified linear unit (ReLU) activation function σ is defined as σ ( x ) = max(0 , x ) .
The ReLU layer or vector-valued rectified linear activation function σ k is defined as σ k ( x ) =(cid:2) σ ( x 1 ) σ ( x 2 ) · · · σ ( x k ) (cid:3) T where x = (cid:2) xx· · · x k (cid:3) T .
Definition 4.
Let l be any positive integer.
A function g : R k
→ R k l is said to be an l -layer
ReLU network if there exist weights
W i ∈
R k
i × k
i − 1 and b i ∈
R k i for i ∈
[ l ] such that theinput-output relationship of the network satisfies g ( x ) =
h l ( x ) where h 1 ( x ) =
W 1 x + b 1 andh
i ( x ) =
W i σ k
i − 1 (cid:0) h
i − 1 ( x ) (cid:1)
+ b i for every i ∈ [ l ] \ [1] .
Definition 5.
The sum (cid:80) L − 1 l =1 k l and the maximum max l ∈ [ L − 1] k l for L > 1 are referred to as thenumber of hidden neurons and the maximum width of an L -layer ReLU network, respectively.
Any1 -layer ReLU network is said to have 0 hidden neurons and a maximum width of 0 .
An l -layer ReLUnetwork is said to have depth l and l − 1 hidden layers.
Upper bounds on neural complexity for representing CPWL functions
The correspondence between CPWL functions and ReLU networks was first clearly confirmed by
Theorem 2.1 in [Arora et al., 2018] although a weaker version of the correspondence can be inferredfrom Proposition 4.1 in [Goodfellow et al., 2013].
Arora et al.
[2018] proved that every ReLUnetwork R n → R exactly represents a CPWL function, and the converse is also true, i.e., every
CPWL function can be exactly represented by a ReLU network.
One of the key steps used by Aroraet al.
[2018] to construct a ReLU network from any given CPWL function relies on an importantrepresentation result by Wang and Sun [2005], stating that any CPWL function can be represented bya sum of a finite number of max- η -affine functions [Magnani and Boyd, 2009] whose sign may beflipped and η is bounded from above by n + 1 .
The implication of using this representation is laterdiscussed in Section 4.1.
The bound η ≤ n + 1 in the representation allowed Arora et al.
[2018] tofurther prove that there exists a ReLU DNN with at most
[Equation]
1) hidden layers to exactly realize any given CPWL function.
However, the computational resourcerequired for a ReLU network to exactly represent any CPWL function had not been available in theliterature until the work by He et al.
3.1 Upper bounds in prior work
He et al.
[2020] proved that a CPWL function R n → R with q pieces and k linear components canbe represented by a ReLU network whose number of neurons is given by
[Equation]
for k ≥ n + 1 and O (cid:0)
n 2 kq (cid:1) for k < n + 1 .
The number of hidden layers in such a ReLU DNN isalso bounded from above by (cid:6) log 2 ( n + 1) (cid:7) , the same as the bound derived by Arora et al.
One of their significant contributions in our view is that they utilize the number of pieces and linearcomponents of a CPWL function to bound the complexity of the equivalent ReLU network.
Ingeneral, the number of pieces q is much larger than the number of linear components k for a CPWLfunction.
He et al.
[2020] also proved the relationship
[Equation]
for any CPWL function.
Note that the bounds in (3) on the number of pieces q and linear componentsk were first mentioned by Tarela and Martinez [1999] who developed the lattice representation of
CPWL function.
Asymptotically, the bounds proved by He et al.
[2020] are amplified linearly withthe input dimension n for any fixed k .
Due to (3), they can be further bounded from above by
O (cid:16) n 2 q 2 +( n +1)( q − n − 1) (cid:17) and O (cid:16) n 2 q 2 (cid:17) in terms of q and n .
On the other hand, in terms of k and n ,they can be further bounded from above by O (cid:16) n 2 k · k !+
( n +1)
( k − n − 1) (cid:17) and O (cid:0) n 2 k · k ! (cid:1) .
Because thesebounds grow much faster than exponential growth, they seem to suggest that the cost of computing a
CPWL function via a ReLU network is exceptionally high.
Hertrich et al.
[2021] proved that any CPWL function R n → R with q pieces can be represented by a
ReLU network whose maximum width is O (cid:16) q 2 n 2 +3 n +1 (cid:17) under the same number of hidden layers(cid:6) log 2 ( n + 1) (cid:7) .
Hence, the number of hidden neurons must be bounded from above by
[Equation]
Note that we infer this bound by taking the product of the depth and the maximum width.
Thebound given by (4) can grow slower than O (cid:16) n 2 q 2 (cid:17) , but it grows faster than O (cid:16) n 2 q 2 (cid:17) if the inputdimension n grows sufficiently faster than the number of pieces q .
Bounds expressed in terms of thenumber of distinct linear components were not given in [Hertrich et al., 2021].
3.2 Improved upper bounds
We show that any CPWL function can be represented by a ReLU network whose number of hiddenneurons is bounded by much slower growth functions.
We state our main results in Theorem 1, 2 and3, and focus on their impact in this subsection.
Each one of them is tailored to a specific complexitymeasure of the CPWL function.
Their proof sketches are deferred to Section 4.2.
We first focus onthe case when the number of linear components is unknown and the complexity of the CPWL is onlymeasured by the number of pieces, a scenario used in [Hertrich et al., 2021].
Theorem 1.
Any CPWL function p : R n → R with q pieces can be represented by a ReLU networkwhose number of layers l , maximum width w , and number of hidden neurons h satisfy
[Equation] [Equation]
(6) and
[Equation]
Furthermore, Algorithm 1 finds such a network in poly ( n, q ) time.
Algorithm 1 Find a ReLU network that computes a given continuous piecewise linear function f f ← run Algorithm linear components do 3:
∅ ▷ See Table 1 in Appendix C in the supplementary material for complexity analysis 4: do 5:
7: end
8: end for 9: v
i ← run Algorithm 2 with { f m } m ∈A i using the minimum type 10: end for 11: v ← run Algorithm 3 with v 1 , v 2 , · · · , v q ▷ Combine q ReLU networks in parallel 12: u ← run Algorithm 2 with (cid:110)(cid:2) ss· · · s q (cid:3) T
(cid:55)→ s m (cid:111) m
∈ [ q ] using the maximum type
: g ← run Algorithm 4 with v and u
▷ Find a ReLU network for the composition u ◦ v
The proof of Theorem 1 is deferred to Appendix B.4 in the supplementary material.
Algorithm 6, 2,3, and 4 used by Algorithm 1 are deferred to Appendix C in the supplementary material and will bediscussed later (before presenting Theorem 2).
Because 2 ⌈ log 2 q ⌉ < 2 q , the upper bound in (7) can befurther bounded from above by 6 q 2 + 2 ⌈ log 2 q ⌉ q + 3 q − 2 ⌈ log 2 q ⌉ − 3 , leading to the asymptoticbounds l = O (log 2 q ) , w = O ( q 2 ) , and h = O ( q 2 ) .
Since the bound given by Theorem 5.2 in Heet al.
[2020] can be lower bounded by O (cid:16) n 2 q 2 (cid:17) , it grows exponentially faster than our bound of hgiven in Theorem 1.
On the other hand, the upper bound given by (4) is at least polynomially largerthan our bound of h and the order of this polynomial grows quadratically with the input dimension n .
Note that such a polynomial becomes an exponential when the growth in n is not slower than q .
Suchdifferences are illustrated by the figure on the left-hand side of Figure 1.
The bounds in Theorem 1are independent of the input dimension n .
Hence, one can realize any given CPWL function using arelatively small ReLU network even though the input dimension n is huge.
In terms of the maximum width, the upper bound given by (6) is at least polynomially smaller than theone given by Hertrich et al.
In contrast to the bound for the number of layers in [Arora et al.
He et al., 2020, Hertrich et al., 2021] that grows logarithmically with the input dimension n , ourbound in Theorem 1 grows logarithmically with the number of pieces q .
Therefore, the ReLU networkfound by Algorithm 1 in general becomes deeper when the CPWL becomes more complex for a fixed
1 2 3 4 5 6 7 8 9 10 10101010k
U pp e r bound o f h 1 2 3 4 5 6 7 8 9 10 10101010k U pp e r bound o f h 1 2 3 4 5 6 7 8 9 10 10101010k
U pp e r bound o f h n = 1 n =n = 5 n = input dimension.
On the other hand, the network remains the same depth even for arbitrarily larger nas long as q is fixed.
Taking an affine function for example, a 1 -layer ReLU network with 0 hiddenneurons is the solution given by Theorem 1.
However, the bound given by [Arora et al., 2018, Heet al., 2020, Hertrich et al., 2021] keeps increasing the depth for a larger n .
Algorithm 1 is a polynomial time algorithm, following from the proof of Theorem 2.
Algorithm 2finds a ReLU network that computes a max-affine or min-affine function [Magnani and Boyd, 2009].
Algorithm 3 concatenates two given ReLU networks in parallel and returns another ReLU networkcomputing the concatenation of two outputs.
Algorithm 4 finds a ReLU network that representsa composition of two given ReLU networks.
These algorithms are basic manipulations of ReLUnetworks required in Algorithm 1.
Notice that Algorithm 1 does not need to be given any linearcomponents or completely know the CPWL function because every distinct linear component can befound by Algorithm 6, which only needs to be given a closed ϵ -ball in the interior of every piece of a
CPWL function p and observe the output of p when feeding an input.
Algorithm 6 solves a system oflinear equations for every piece of p to find the corresponding linear component.
Every system oflinear equations here has a unique solution because the interior of each of pieces is nonempty.
Thenonemptyness is guaranteed by Lemma 12(a) in Appendix A in the supplementary material.
Theorem 2 discusses the case when the number of linear components and pieces are both known.
Theorem 2.
Any CPWL function p : R n → R with k linear components and q pieces can berepresented by a ReLU network whose number of layers l , maximum width w , and number of hiddenneurons h satisfy l ≤ ⌈ log 2 q ⌉ + ⌈ log 2 k
+ 1 , w ≤ I [ k > 1] (cid:108) 3 k 2 (cid:109) q, and
[Equation]
[Equation]
Furthermore, Algorithm 1 finds such a network in poly ( n, k, q ) time.
The proof of Theorem 2 is deferred to Appendix B.3 in the supplementary material.
The bounds in
Theorem 2 are in general tighter and always no worse than those in Theorem 1 because q is never lessthan k but can be much larger than k .
Asymptotically, l = O (log 2 q ) ,
w = O ( kq ) , and h =
O ( kq ) .
The bound given by Theorem 5.2 in He et al.
[2020] increases exponentially faster than the bound ofh in Theorem 2.
When the number of linear components is the only complexity measure of the CPWL function, weresort to Theorem 3 below.
Theorem 3.
Any CPWL function p : R n → R with k linear components can be represented by a
ReLU network whose number of layers l , maximum width w , and number of hidden neurons h satisfy
[Equation]
> 1] (cid:108) 3 k 2 (cid:109) ϕ
( n, k ) , and
[Equation]
(9) where
[Equation]
The proof of Theorem 3 is deferred to Appendix B.5 in the supplementary material.
Becauseϕ ( n, k ) ≤ k ! , the worst-case asymptotic bounds for l , w and h are l =
O
( k log 2 k ) , w = O ( k · k !) ,and
h =
O ( k · k !) , respectively.
However, it holds that (cid:80) ni =0 (cid:0) k 2 − k
i (cid:1) ≤ k 2 n , so the asymptoticbounds are l = O ( n log 2 k ) , w = O (cid:0) k 2 n +1 (cid:1) , and
h =
O (cid:0) k 2 n +1 (cid:1) when n grows sufficiently slowerthan k .
For example, n = O (cid:16) √ k (cid:17) .
In this case, w and h are bounded from above by a polynomialof order 2 √ k + 1 , which grows slower than factorial growth.
Such an advantage for small n isillustrated by the figure on the left-hand side of Figure 2.
Since the bound given by Theorem 5.2 in [He et al., 2020] can be bounded from below by O (cid:0) n 2 k · k ! (cid:1) ,it at the minimum grows exponentially larger than the upper bound of h in Theorem 3.
For sufficientlysmall n , the relative order of growth is even larger.
The figure on the right-hand side of Figure 2illustrates such a large difference.
For k = 5 , n 2 k · k ! ≈ 7 . 92 × 10 28 if n = 1 , while our bound of h isless than 3 .
7 × 10 3 for any n .
The difference is extremely large even though k is small under n = 1 .
The middle plot in Figure 2 shows that (4) increases much faster when n becomes larger.
The upperbound of h in Theorem 3 is better than (4) for any n , which can be clearly seen from Figure 2.
Lemma 1.
Let P n,k be the set of all CPWL functions with exactly k distinct linear components suchthat p : R n → R , ∀ p ∈ P n,k .
Let C n,k ( p ) be the collection of all families of closed convex subsetssatisfying Definition 1 for any p ∈ P n,k .
Then, k ≤ min Q∈C n,k ( p ) |Q| ≤ ϕ ( n, k ) .
The proof of Lemma 1 is deferred to Appendix B.1 in the supplementary material.
Clearly, ϕ ( n, k ) isa better upper bound of q compared to the bound q ≤ k !
given by He et al.
When n growssufficiently slower than k , the bound ϕ ( n, k ) can be exponentially smaller than k ! .
Although the upper bounds in our main results are much better, it is still possible to find a ReLUnetwork whose hidden neurons are fewer than the bounds in Theorem 2 to exactly represent a given
CPWL function.
Tight bounds for the case when n = 1 was first given by Theorem 2.2 in [Aroraet al., 2018].
However, it seems more difficult to bound the size of a network from below for ahigh dimensional input.
To the best of our knowledge, we are not aware of any tight bounds in theliterature for the exact ReLU network representation of general CPWL functions using an arbitrary n .
Representations of CPWL functions have different implications on depth
We reveal implications of using different representations of CPWL functions and their impact onconstructing ReLU networks.
We first discuss the popular representation used by prior work and theimplicit constraint imposed by such a representation.
4.1 Constrained depth
Arora et al.
He et al.
[2020], and Hertrich et al.
[2021] proved the same bound for the numberof layers and the work relies on the following representation of CPWL function:
[Equation]
(11) where σ j ∈ { +1 , − 1 } and η ( j ) is a subset of [ J ] such that (cid:12)(cid:12)
η
( j ) (cid:12)(cid:12) ≤ n + 1 for all j ∈
[ J ] .
That is, asum of a finite number of max- η -affine functions whose sign may be flipped.
(11) was established by
Theorem 1 in Wang and Sun [2005] which is essentially the same as Theorem 1 in Wang [2004] thatemphasizes on the difference of two convex piecewise linear functions.
This result was also used by
Goodfellow et al.
[2013] for the maxout network (Proposition 4.1).
The depth given by (1) does not scale with the complexity of the CPWL function.
This feature directlycomes from using a ReLU network to realize each of max- η -affine functions in (11) and concatenatingall of them together.
Because the size of η ( j ) is bounded from above by n + 1 , the depth can bemade to depend solely on n .
Such a treatment seems to be the only way if one considers a CPWLfunction represented by (11).
As a result, the ReLU network is forced to use a depth constrainedby the input dimension to represent the given CPWL function, which in turn requires more hiddenneurons.
Because we do not use (11), our networks are not limited by such an implication.
4.2 Proof sketch for unconstrained depth
We give a proof sketch in this subsection for our main results.
By using a different representation, thedepth of a ReLU network is able to be scaled with the complexity measure, i.e., number of pieces, ofany given CPWL function to accommodate the high expressivity.
By Theorem 4.2 in [Tarela and Martinez, 1999], any CPWL function p can be represented as
[Equation]
(12) for all x ∈ R n where A ( X ) =
(cid:8) i ∈
[ k ] | f i ( x ) ≥ p ( x ) , ∀ x ∈ X (cid:9) is the set of indices of linearcomponents that have values greater than or equal to p ( x ) for all x ∈ X , and Q is any family ofclosed convex subsets of R n satisfying Definition 1.
We have used f 1 , f 2 , · · · , f k to denote the kdistinct linear components of p .
Notice that Theorem 4.2 in [Tarela and Martinez, 1999] was firststated by Theorem 7 in [Tarela et al., 1990].
Both are essentially the same, but Theorem 4.2 in [Tarelaand Martinez, 1999] emphasizes the convexity of each of regions in the domain.
Both theoremsare also fundamentally equivalent to Theorem 2.1 in [Ovchinnikov, 2002].
Notice that one of theconcluding remarks in [Ovchinnikov, 2002] pointed out that the convexity of the input space is anessential assumption.
The entire space R n satisfies such an assumption.
In addition, Ovchinnikovpointed out that the max-min representation also holds for vector-valued CPWL functions.
Hence, itis possible to generalize the bounds in our results to vector-valued CPWL functions.
Using the representation in (12) and Lemma 2 below, we are able to prove Theorem 2 by boundingthe size of Q and A ( X ) .
Theorem 1 and 3 can be proved by applying Lemma 1 to Theorem 2.
Notethat the size |Q| in (12) is the key for the depth of a ReLU network to be able to scale with q .
Lemma 2.
Let m be any positive integer.
Define l ( m ) =
⌈ log 2 m ⌉ + 1 , w ( m ) =
I [ m > 1] (cid:6) 3 m 2 (cid:7) ,and the following sequence for any positive integer k ,
[Equation]
Then, there exists an l ( m )
-layer
ReLU network g : R n → R with r ( m ) hidden neurons and amaximum width of w ( m ) such that g computes the extremum of f 1 ( x ) , f 2 ( x ) , · · · , f m ( x ) , i.e.,g ( x ) = max i ∈ [ m ] f i ( x ) or g ( x ) =
min i ∈ [ m ] f i ( x ) for all x ∈ R n under any m scalar-valued affinefunctions f 1 , f 2 , · · · , f m .
Furthermore, Algorithm 2 finds such a network in poly ( m, n ) time.
The proof of Lemma 2 is deferred to Appendix B.2 in the supplementary material.
One can also viewl ( m ) , w ( m ) , and r ( m ) as upper bounds for the number of layers, maximum width, and the numberof hidden neurons.
Because r ( m ) < 6 m − 3 by Lemma 6, the bound for the number of hiddenneurons r ( m ) is tighter than the bound 8 m − 4 given by Lemma D.3 in [Arora et al., 2018] or Lemma5.4 in [He et al., 2020] (these two lemmas are essentially the same).
The bound for the numberof layers remains the same as the one given by Lemma D.3 in [Arora et al., 2018].
By combining
Lemma 2 with Lemma 3, Lemma 4, and Lemma 8, we can easily perform the same job on computingthe extremum of multiple scalar-valued ReLU networks as Lemma D.3 does in [Arora et al., 2018].
Lemma 6, 3, 4, and 8 are given in Appendix A in the supplementary material.
Broader impact
Our results guarantee that any CPWL function can be exactly computed by a ReLU neural networkat a more manageable cost.
This assurance is crucial because CPWL functions are important toolsin many applications.
Such an assurance also relates DNNs closer to CPWL functions and allowsresearchers and engineers to understand the expressivity of DNNs from a different perspective.
Wefocus on simple ReLU networks (ReLU multi-layer perceptrons) in this paper, but it may be possibleto derive bounds for other activation functions and advanced neural network architectures such asmaxout networks [Goodfellow et al., 2013], residual networks [He et al., 2016], densely connectednetworks [Huang et al., 2017], and other nonlinear networks [Chen et al., 2021], by making some(possibly mild) assumptions.
Our contributions advance the fundamental understanding of the linkbetween ReLU networks and CPWL functions.
Checklist
For all authors...
(a) Do the main claims made in the
abstract and introduction accurately
reflect
the paper’scontributions
and scope?
[Yes] The main claims are summarized in Section
(b) Did you describe the limitations of your work?
[Yes] See the last paragraph of Section
(c) Did you discuss any potential negative societal impacts of your work?
[No] This is a
theoretical paper
and we are not aware of any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All definitions
and assumptions are stated or referenced before the results.
(b) Did you include complete proofs of all theoretical results?
[Yes] Appendix B in the
supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/
A
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]