D EPTH -W IDTH T RADE - OFFS FOR R E LU N ETWORKS VIA S HARKOVSKY ’ S T HEOREM
A BSTRACT
Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory.
In a seminal paper, Telgarsky highlighted the beneﬁts of depth by presenting a family of functions (based on simple triangular waves) for which DNNs achieve zero classiﬁcation error, whereas shallow networks with fewer than exponentially many nodes incur constant error.
Even though Telgarsky’s work reveals the limitations of shallow neural networks, it doesn’t inform us on why these functions are difﬁcult to represent and in fact he states it as a tantalizing open question to characterize those functions that cannot be well-approximated by smaller depths.
In this work, we point to a new connection between DNNs expressivity and
Sharkovsky’s Theorem from dynamical systems, that enables us to characterize the depth-width trade-offs of ReLU networks for representing functions based on the presence of a generalized notion of ﬁxed points, called periodic points (a ﬁxed point is a point of period 1).
Motivated by our observation that the triangle waves used in Telgarsky’s work contain points of period 3 – a period that is special in that it implies chaotic behaviour based on the celebrated result by Li-Yorke – we proceed to give general lower bounds for the width needed to represent periodic functions as a function of the depth.
Technically, the crux of our approach is based on an eigenvalue analysis of the dynamical systems associated with such functions.
I NTRODUCTION
In approximation theory, one typically tries to understand how to best approximate a complicated family of functions using simpler functions as building blocks.
For instance, Weierstrass (1885) proved a general result stating that every continuous function can be uniformly approximated as closely as desired by a polynomial.
It wasn’t until later that Vitushkin (1959) gave quantitative bounds between the approximation error and the polynomial’s degree.
Drifting away from polynomials and given the recent breakthroughs of deep learning in a variety of difﬁcult tasks like image classiﬁcation, natural language processing, game playing and self-driving cars, researchers have tried to understand the approximation theory that governs neural networks.
This question of neural network expressivity , i.e. how architectural properties like the depth, width or the activation units affect the functions it can compute, has been a fundamental ongoing challenge with a rich history.
A classical result by Cybenko (1989), Hornik et al.
, Fukushima (1980) demonstrates the expressive power of neural networks: it states that even two layered neural networks (using well known activation functions) can approximate any continuous function on a bounded domain.
The caveat is that the size of such networks may be exponential in the dimension of the input, which makes them highly susceptible to overﬁtting as well as impractical, since one can always add extra layers in their model aiming at increasing the representational power of the neural network.
More recently, in a seminal paper by Telgarsky (2016), it was shown that there exist functions that can be represented by DNNs, i.e, by some particular choice of weights on their edges (and for a wide variety of standard activation units in their layers), yet cannot be approximated by shallow networks unless they are exponentially large.
More concretely, he showed that for any positive integer k , there exist neural networks with Θ( k 3 ) layers, Θ(1) nodes per layer, and Θ(1) distinct parameters which cannot be approximated by networks with O ( k ) layers, unless they have Ω(2 k ) nodes.
At a high level, he uses the number of oscillations present in certain functions as a notion of “complexity” that distinguishes between deep and shallow networks’ representation capabilities via the following three facts: a) functions with few oscillations poorly approximate functions with many oscillations, b) functions computed by networks with few layers must have few oscillations and c) functions computed by networks with many layers can have many oscillations.
Our main contribution is a novel connection between the theory of dynamical systems and the representational power of DNNs via the well-studied notion of periodic points , a notion that captures the important notion of ﬁxed points of a continuous function.
Deﬁnition 1 (Period) .
We say that a (continuous) Lipschitz function f : [0 , 1] → [0 , 1] contains a point of period n ≥ 1 if there exists a point x 0 ∈ [0 , 1] such that 1 : f n ( x 0 ) =
xand (point of period n ) f k ( x 0 ) (cid:54) = x 0 , ∀ 1 ≤ k ≤ n − 1 .
In particular, all numbers in C = { x 0 , f ( x 0 ) , f ( f ( x 0 )) , . . .
, f n − 1 ( x 0 ) } are distinct, each of which is a point of period n and the set C is called a cycle (or orbit) of period n .
Observe that since f : [0 , 1] → [0 , 1] is continuous, it certainly has at least one point of period 1, which is called a ﬁxed point.
For the rest of this paper, we focus on (continuous) Lipschitz functions f : [0 , 1] → [0 , 1] , unless otherwise stated.
Note that the choice of interval [0 , 1] is for simplicity of our presentation and that our results will hold for any closed interval [ a, b ] .
As we observe, points of period 3 are contained in both Telgarsky (2016) and Schmitt (2000) constructions and this could as well have been a coincidence, however we show that the existence of periodic points of certain periods are actually one of the reasons explaining why depth is needed to represent functions that contain them (otherwise exponential width is required) .
Towards this direction, we will make use of a deep result in the literature of iterated dynamical systems called
Sharkovsky’s Theorem (Sharkovsky (1964; 1965)).
1. S HARKOVSKY ’ S T HEOREM
Consider the set of positive natural numbers N ∗ = { 1 , 2 , . . . } and deﬁne the following (decreasing) ordering (cid:46) called Sharkovsky’s ordering as follows:
3 (cid:46) 5 (cid:46) 7 (cid:46) · · · (cid:46) (odd numbers bigger than one) (cid:46) 2 · 3 (cid:46) 2 · 5 (cid:46) 2 · 7 (cid:46) · · · (cid:46) (odd multiples of two but not two) (cid:46) 2 2 · 3 (cid:46) 2 2 · 5 (cid:46) 2 2 · 7 (cid:46) · · · (cid:46) (odd multiples of four but not four) ...
(cid:46) · · · (cid:46) 2 4 (cid:46) 2
(cid:46) 2 2 (cid:46) 2 (cid:46) 1 (powers of two in decreasing order) .
This is a total ordering; we write l (cid:46) r or r (cid:47) l whenever l is to the left of r .
Sharkovsky showed that this ordering describes which numbers can be periods for a continuous map on an interval; allowed periods need to be a sufﬁx of the Sharkovsky ordering:
Theorem 1.1 (Sharkovsky’s “Forcing” Theorem, Sharkovsky (1964; 1965)) .
Let I be a closed interval and f : I → I be a continuous map.
If n is a period for f and n (cid:46) n (cid:48) , then n (cid:48) is also a period for f .
Remark 1.1 .
Note that the number 3 is the maximum period according to Sharkovsky’s ordering, so an important corollary is that a function having a point of period 3, must also have points of any period.
This special corollary is a weaker version of Sharkovsky’s theorem and was proved some years later 2 in a celebrated result by Li & Yorke (1975), who coined the term “chaos” as used in
Mathematics.
We conclude the subsection with the deﬁnition of a prime period of a function f .
Deﬁnition 2 (Prime period) .
A function f has prime period n as long as it has a cycle of period n , but has no cycles with period greater than n according to the Sharkovsky ordering.
For example, in the interval [0,1], the function f ( x ) = 1 − x has prime period 2, since f ( f ( x ))
= 1 − (1 − x ) =
x so all points are periodic with period 2, except the ﬁxed point at 1/2.
Before formally stating our main theorems, we present an illustrative example inspired from Telgarsky’s triangle wave construction and we connect it to DNNs’ sensitivity to weight perturbations and their representational power.
1. S ENSITIVITY A NALYSIS - A M OTIVATING E XAMPLE
An important ingredient in Telgarsky’s proof, was the “triangular wave” function (sometimes referred to as the tent map or sawtooth) depicted in Figure 1b and given by:
[Equation]
He shows that the composition of t ( x ; 2) with itself k times (denoted by t k ( x ; 2) ), will create exponentially (in k ) many oscillations and as a result he is able to show a separation for the classiﬁcation error when using a shallow vs a deep neural network as a predictor.
Our starting point is the observation that the triangular wave function t ( x ; 2) contains points of period 3, e.g. ( 29 → 49 → 89 → 29 ) .
It follows in particular, that t ( x ; 2) exhibits Li-Yorke Chaos (Li & Yorke (1975)) in the sense that it contains all periods.
The compositions of such functions will look highly complex (see Figure 2) and in fact Telgarsky heavily relied on the highly oscillatory behavior of t ( x ; 2) to prove his depth separation result.
However, his result doesn’t inform us on what would happen if one used a slightly modiﬁed version of the triangle wave t ( x ; 2) .
Observe that since a simple neural network with one hidden layer can represent the function t ( x ; 2) , the question is basically equivalent to asking how modifying the weights on the edges of the neural network can affect its representational power (see Figure 1), hence the title of the current subsection.
The main question is can we have a general theory that informs us on when will the function composition be hard to represent and when not?
Our paper’s main point is to provide an answer by checking if the function at hand has a simple property, relating to the presence of chaotic behavior.
To illustrate our point, consider the generalized triangle wave function t ( x ; µ ) parameterized by µ :
[Equation]
This function parameterized by µ ranges from [0 , µ/ 2] and is closely related to the logistic map f ( x ) :
= rx (1 − x ) used in Schmitt (2000) and exhibits a variety of limiting behaviors: for instance, it converges to a stable ﬁxed point when µ ≤ 1 , it exhibits chaos when µ = 2 etc.
3 Instead of µ = 2 , if we set µ = 1 , we get the network depicted in Figure 1c, 1d.
Note that compositions of t ( x ; 1) (created by the same neural network architecture but with slightly different weights), behave completely differently since in the µ = 1 case, we will not get a highly − 0 .   − x t ( x
[Figure] [Figure] [Figure]
0.7 0.8 0.9 1 x0.1 0.
0.9t ( x ; 2 )
[Figure]
− 0 .
− x t ( x ; 1)
[Figure]
[Figure]
0.7 0.8 0.9 1 x0.1 0.
0.9t ( x
[Figure]
[Figure]
oscillatory behavior.
This can be seen in Figure 3.
One difference between the two cases is the relative position of the map with the line y
=
x
and this seems to be pointing that ﬁxed points and their generalizations i.e. periodic orbits play an important role when dealing with function compositions.
Indeed, despite the wide range of possibilities one can expect by composing such functions, as we show, their behavior can be characterized using tools from dynamical systems; the exponential growth in complexity (or lack thereof) of these compositions can be explained by invoking a fundamental property of these continuous functions on bounded intervals which is the existence (or not) of periodic points of certain periods.
Similarly, we can argue about changing the parameters of the logistic map which is given by f ( x ; r ) :
= rx (1 − x ) used in Schmitt (2000) for sigmoidal networks (where f ( x ; 4) was used).
The properties of the logistic map are well known and was ﬁrst studied by Robert May and Mitchell
Feigenbaum (May (1976) and Feigenbaum (1976)).
It is known that as one varies the parameter r , the logistic map gives rise to a plethora of different behaviors, hence the same is true for when one slightly perturbs the weights of a neural net used to represent the map.
Please refer to Appendix C for some ﬁgures that illuminate these differences in the logistic map.
1. I NFORMAL S TATEMENTS OF M AIN T HEOREMS
We demonstrate that a simple property of f governs the depth-width trade-offs in order to represent it and we give quantitative bounds for them.
This simple property has to do with the periods that the function f contains.
Informally, our ﬁrst main theorem states that if a function f contains periodic points with certain periods, then composing f with itself many times, will result in exponentially many oscillations, giving rise to complicated behaviors and chaos:
Theorem 1.2.
Let f : [0 , 1] → [0 , 1] be a continuous function.
Assume that there exists a cycle of period n , where n = m · p with p being an odd number greater than one and with m being a power of two (it might be m = 1 ). Then, there exist x, y ∈ [0 , 1] such that the function f mt (taking mt compositions of f with itself) “oscillates” (also look Deﬁnition 4) at least ρ t times between [Figure] [Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
1 x0.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] x and y for all t ∈ N ∗ , where ρ is the positive root greater than one of the polynomial equation λ p − 1 − λ p − 2 − 1 = 0 .
Our second main theorem then draws the connection between the number of oscillations a function has and the depth-width trade-offs needed:
Theorem 1.3.
Let k be a positive integer and f be a function as above.
We set ρ to be the positive root greater than one of the polynomial equation λ p − 1 − λ p − 2 − 1 = 0 .
We can construct a sequence of points ( x i , y i ) 2 ni =1 with n := (cid:98) ρ k (cid:99)such that the classiﬁcation error of the function f mk is zero, whereas the classiﬁcation error of any neural network with l layers and u nodes per layer, where u ≤ ρ kl 8 , necessarily has classiﬁcation error ≥ 14 .
Formal statements for the two theorems can be found in Section 3 and Section 4.
Using these theorems, we draw connections with previous results Telgarsky (2016), Schmitt (2000) in a uniﬁed way, thus identifying chaotic behavior as the main underlying thread for depth-width trade-offs.
Technically, our approach is based on an eigenvalue analysis of certain matrices associated with such periodic functions.
1. O THER R ELATED W ORK
Understanding the beneﬁts of depths on the expressive power a speciﬁc computational model can have, is an important area of research spanning different computational models and results come in the ﬂavor of depth separation arguments.
Roughly speaking, many of the results in this area rely on a suitably deﬁned notion of “complexity” of a function we would like to represent, and then proceed by proving that under this notion, deep models have signiﬁcantly more power than shallower models.
For example, if the computational model of interest is the family of boolean or threshold circuits, depth lower bounds are given in Hastad (1986);
Rossman et al.
(2015); H ˚ astad (1987);
Parberry et al.
; Kane & Williams (2016).
Furthermore, people have analyzed sumproduct networks (summation and product nodes) and studied trade-offs for depth (Delalleau &
Bengio (2011); Martens & Medabalimi (2014)).
Coming closer to neural networks computation where the activation units can be general real-valued functions, important previous results include Eldan & Shamir (2016);
Telgarsky (2015; 2016);
Schmitt (2000); Montufar et al.
(2014); Malach & Shalev-Shwartz (2019)
; Poole et al. (2016);
Raghu et al.
; Arora et al. (2016); Liang & Srikant (2016); Kileel et al.
Regarding the aforementioned notions of “complexity” used in depth separation arguments, examples include the notion of global curvature (Poole et al.
trajectory length (Raghu et al.
, number of oscillations (Telgarsky (2015; 2016) and Schmitt (2000)), number of linear regions (Montufar et al.
(2014)), fractals (Malach & Shalev-Shwartz (2019)) and more.
Our work is more closely related to Telgarsky (2015; 2016), and Schmitt (2000) since it is easy to see that their maps are chaotic, but we conjecture that many of the notions of complexity introduced in this line of research to showcase beneﬁts of depth actually arise due to chaotic behavior.
In this sense, we conjecture that chaotic behavior is the main culprit for the failure of neural networks to represent certain functions, unless they are sufﬁciently deep (or have exponential width).
Moreover, other works that have exploited the powerful result by Li-Yorke (in online learning frameworks) are Palaiopanos et al.
(2017); Chotibut et al.
F URTHER B ACKGROUND : T HE C OVERING L EMMA
The crux of the proof of Sharkovsky’s theorem provided by Burns & Hasselblatt (2011) contains a covering lemma that will be our starting point to prove our main results.
Before we proceed with the statement of the Covering Lemma, we provide one more important deﬁnition.
Deﬁnition 3 (Covering relation) .
Let f be a function and I 1 , I 2 be two closed intervals.
We say that
I 1 covers I 2 under f , denoted by I 1 f −→ I 2 as long as I 2 ⊆
f ( I 1 ) .
[Figure] For example, the triangle wave t ( x ; 2) that has the period 3 point 29 (recall 29 → 49 → 89 → 29 ) naturally deﬁnes two intervals I 1 = [ 29 , 49 ] and I 2 = [ 49 , 89 ] with the covering relations: I 1 f −→ I 2 ,
I 2 f −→ I
and I 2 f −→ I 1 .
Lemma 1 (Covering Lemma for odd periods) .
Let f : [0 , 1] → [0 , 1] be a continuous function and assume f has a cycle C of period n , where n > 1 is an odd number.
Denote β 0 , ..., β n − 1 ∈ C the elements of the cycle in increasing order and deﬁne the sequence of closed intervals I 0 , ..., I n −where
I i
=
[ β i , β i +1 ] (they have pairwise disjoint interiors).
Then, there exists a sub-collection of the aforementioned intervals (not necessarily in the same ordering) J 0 , ...
J r with 1 ≤ r ≤ n −such that the following covering relation holds:
J i f −→ J
i +1 , for 1 ≤ i ≤ r − 1 , 2.
J r f −→ J 0 and J 0 f −→ J 0 ∪ J 1 .
For a pictorial illustration of the Covering Lemma, see Figure 4.
In particular, observe that for n =we get r = 1 so the covering relation is as in Figure 4.
We conclude this section with the formal deﬁnition of crossings (or oscillations) and we refer the reader to Figure 5 for some examples.
Deﬁnition 4 (Crossings) .
We say that a continuous function f : [0 , 1] → [0 , 1] crosses the interval [ x, y ] with x, y ∈ [0 , 1] if there exist a, b such that f ( a ) =
x and f ( b ) =
y .
Moreover we denote
C x,y ( f ) the number of times f crosses [ x, y ] .
That is C x,y ( f ) =
t if there exist numbers a 1 , b 1 < a 2 , b 2 < · · · < a t , b t in [0 , 1] so that f ( a i ) = x and f ( b i )
=
y for all 1 ≤ i ≤ t .
Observe that if
I f,x,y is used to denote 4 the number of intervals the function ˜ f x,y ( z ) :
=
[ f ( z ) ≥ x + y 2 ] is piecewise constant and partitions [0 , 1] , then C x,y ( f ) ≤ I f,x,y .
P ERIODS D ETERMINE THE N UMBER OF C ROSSINGS 3. P ERIOD THAT IS NOT A POWER OF TWO IMPLIES EXPONENTIAL CROSSINGS
In this section, we prove our main theorem, the statement of which is given below.
Technically, we make use of the Lemma 1 (Covering Lemma) to show the exponential growth of the number of crossings.
Theorem 3.1.
Let f : [0 , 1] → [0 , 1] be a continuous function.
Assume that there exists a cycle of period n where n = m · p , p is an odd number greater than one and m being a power of two (it might be m = 1 ).
It holds that there exist x, y ∈
[0 , 1] so that C x,y ( f mt ) is c t for all t ∈ N ∗ , where c is the positive root greater than one of the polynomial equation λ p − 1 − λ p − 2 − 1 = 0 .
Counting the number of oscillations.
For a given continuous function f : [0 , 1] → [0 , 1] , let
J 0 , . .
, J r , where 1 ≤ r ≤ n − 2 , be the intervals as promised from Lemma 1.
We deﬁne a sequence of vectors δ t ∈ N r +1 such that δ ti is deﬁned as the number of times the function f t crosses the interval J i for all 0 ≤ i ≤ r .
In particular we deﬁne f 0 to be the identity function and hence δ
=
, 1) (all ones vector).
For what follows, we will try to express recursively δ t in terms of δ t − 1 and in the end we will show that δ k 0 is Ω( c k ) where c is some constant that depends on r .
To build some intuition, we ﬁrst analyze the case of period three and then we prove the general case.
3.1. W ARM UP : T HE CASE OF PERIOD 3 AND THE F IBONACCI SEQUENCE
Assume that f has a cycle of period 3, that is the numbers { x 0 , f ( x 0 ) , f 2 ( x 0 ) } are distinct and f 3 ( x 0 ) =
x 0 for some x 0 ∈ [0 , 1] .
Let β 0 < β 1 < β 2 be the numbers x 0 , f ( x 0 ) , f 2 ( x 0 ) in increasing order.
We deﬁne I
=
[ β 0 , β 1 ] and I 1 =
[ β 1 , β 2 ] .
From Lemma 1, when n = 3 , we can see that r = 1 and thus we have the following possibilities for the covering relations:
• Either I
f −→ I 0 ∪ I 1 , • or I 1 f −→ I 0 ∪ I 1 .
We deﬁne J 0 to be the interval among I 0 , I 1 that involves the self-loop covering and J 1 to be the remaining interval.
Deﬁne δ t ∈ N 2 as above, and so we get that:
[Equation]
[Equation]
where δ
= 1 and δ
= 1 .
The matrix A := (cid:18) 11(cid:19) can be interpreted as the adjacency matrix that corresponds to the covering relations between J 0 , J 1 (which consists of a directed cycle with a self-loop at vertex J 0 ).
The reason we have an inequality instead of an equality is because the
Covering Lemma only guarantees that the number of times J 0 “covers” J 0 and J 1 is at least one and not necessarily exactly one.
We set α
= δ
and we deﬁne α t +1 =
Aα t .
It is clear that δ t ≥ α t (entry-wise) for all t ∈ N .
Moreover, α t 0 is the well-known Fibonacci sequence F t +1 (with F 0 = F 1 = 1 ), therefore α t 0 =
[Equation]
We conclude that δ t 0 ≥ (cid:16) 1+ √ 5 2 (cid:17) t .
See also Figure 5 for a pictorial illustration about the proof for t = 1 , 2 , 3 , 4 .
3.1. E VERY PERIOD GREATER THAN 3 BUT NOT POWER OF TWO
Assume that f has a cycle of period n > 3 with n odd, that is the numbers { x 0 , f ( x 0 ) , f 2 ( x 0 ) , ..., f n − 1 ( x 0 ) } are distinct and f n ( x 0 ) =
x 0 for some x 0 ∈ [0 , 1] .
Let β 0 < β 1 < β 2 < ... <
β n − 1 be the numbers x 0 , f ( x 0 ) , f 2 ( x 0 ) , ..., f n − 1 ( x 0 ) in increasing order.
We deﬁne I i =
[ β i , β i +1 ] for 0 ≤ i ≤ n − 2 .
From Lemma 1 it follows that there is a subcollection of the intervals
I 0 , ..., I n − 2 (with not necessarily the same ordering) J 0 , ..., J r ( 1 ≤ r ≤ n − 2 ) such that
J i f −→ J
i +1 , for 1 ≤ i ≤ r − 1 , 2.
J r f −→ J 0 and J 0 f −→ J 0 ∪ J 1 .
The interval J 0 is the one that involves the self-loop covering.
As in the case for n = 3 , we deﬁne δ t which is in N r +1 , with δ ti capturing the number of times f t crosses the interval J i .
We get that:
[Equation]
where δ
=
, 1) (all ones vector) and A ∈ R ( r +1) ×
( r +1) is deﬁned to be:
[Equation]
In words, A is the adjacency matrix of a graph with r +1 nodes that is a directed cycle that involves a self-loop at vertex J 0 .
We deﬁne α t in a similar way as in the case for period three, i.e., α t +1 =
Aα t and α
=
δ
so that δ t ≥ α t (entry-wise) for all t ∈ N .
We can easily observe that the following holds: α t +1 =
A t +1 α 0 .
Our next plan is to compute a lower bound on the spectral radius of the matrix A (cid:62) (denoted by sp ( A (cid:62) ) ) with the following claim (proof in Appendix A).
Claim 3.1.
The characteristic polynomial of A (cid:62) is:
[Equation]
Let us call ρ r the largest root in absolute value of the polynomial π ( λ ) in A.1.
Since A is a nonnegative matrix, the largest root in absolute value is actually a positive real number (by the PerronFrobenius theorem).
It is easy to see that the polynomial in A.1 has always a root greater than one and less than two (by Bolzano’s theorem, see π (1) =
− 1 < 0 and π (2) = 2 r +1 − 2 r − 1 = 2 r − 1 > 0 ).
Hence we have sp ( A ) =
ρ r > 1 .
Furthermore, it is easy to see that since A is a non-negative matrix (and powers of A are also non-negative), it holds that
[Equation]
for all t ≥ 1 , that is the row with the largest sum of its entries is the ﬁrst row (row for i = 0 ).
Using the fact that
[Equation]
that is the spectral radius of a matrix is always at most any matrix norm, we conclude that (cid:80)
[-r j =
 A t 
 j ≥ ρ t r .-][Equation]
The case of odd period greater than three follows by noting that (cid:80) rj =
A t
j = α t 0 , thus δ t 0 ≥ α t 0 ≥ ρ tr .
Observe that for period three, we have that r = 1 and also ρ 1 = 1+ √ 5(the largest root of λ 2 − λ − 1 = 0 ).
We would like to make the following two remarks:
Remark 3.1 .
The spectral radius ρ r is strictly decreasing in r : this is easy to see since ρ r > 1 and is satisfying the equation x r +1 − x r = 1 (note that x r +1 − x r is increasing in r for x > 1 ).
This implies that smaller odd periods can potentially have a number of crossings that grows at faster rates than larger odd periods, hence giving rise to more complex behaviors.
See also Remark 4.1.
Remark 3.2 (The case of even period but not power of two) .
Our result above is applied for cycles of period n = m · n (cid:48) where m is a power of two and n (cid:48) is an odd number greater than one.
The trick is to observe that if a function has cycle of period n , then f m has a cycle of period n (cid:48) (which is an odd number greater than one).
Therefore, the number of oscillations C x,y ( f mt ) with x, y being the endpoints of J 0 , is at least ρ tn (cid:48) − 2 for t ∈ N .
Proof of Theorem 3.1.
The proof now follows from the case analysis carried out in Sections 3.1.1, 3.1.2 and Remark 3.2.
3. P ERIOD THAT IS A POWER OF TWO MAY HAVE POLYNOMIAL CROSSINGS
Lemma 2 (Period power of two - proof in Appendix A) .
There exist continuous functions f with prime period n that is a power of two so that the number of crossings C x,y ( f t ) scales at most polynomially with t for any x, y ∈ [0 , 1] .
P ERIOD -D EPENDENT L OWER B OUNDS FOR DNN S
Building on Telgarsky (2015; 2016), the representation power of different networks will be measured via the classiﬁcation error.
For a given collection of n points ( x i , y i ) ni =1 with y i ∈ { 0 , 1 } , one can deﬁne the classiﬁcation error of a function g to be ( ˜ g is just the thresholded value of g ):
[Equation]
In this section, we argue that functions with cycles of period not a power of two, will have compositions for which any shallow neural network will have classiﬁcation error a positive constant.
Assume we are given a continuous function f : [0 , 1] → [0 , 1] so that f has a cycle of period m × p where p is an odd number greater than one and m is a power of two.
From Theorem 3.1, there exist x, y ∈ [0 , 1] so that C x,y ( f tm ) is at least ρ tp − 2 2 , where ρ r is deﬁned to be the root that is greater than one of the polynomial equation
λ r +1 − λ r − 1 = 0 .
We set ρ :=
ρ p − 2 , h :=
f k · m and assume that g : [0 , 1] → [0 , 1] is a neural network with l layers and u nodes (ReLU activations) per layer.
In
Lemma 2.1 of Telgarsky (2015), it is proved that a neural network with u ReLU units per layer and with l layers is piecewise afﬁne with at most (2 m ) l pieces.
We deﬁne as ˜ h
( z ) =
[ h ( z ) ≥ x + y 2 ] and ˜ g ( z ) =
1 [ g ( z ) ≥ x + y 2 ]
(note that we changed the threshold to be x + y 2 instead of 12 that was used in Telgarsky (2015)).
Since C x,y ( h ) is at least ρ k , it holds that there exist points ( x i , y i ) 2 ni
=1 with n := (cid:98) ρ k (cid:99)such that h ( x j ) = x , y j = 0 for j odd and h ( x j ) =
y , y j = 1 for j even.
It is clear that for this collection of points the classiﬁcation error of the function h is zero, whereas the classiﬁcation error for function g is bounded from below by
[Equation]
The above inequality is an application of Lemma 2.2 of Telgarsky (2015) (with careful counting it has been slightly improved).
By choosing u to be at most ρ kl 8 it holds that the classiﬁcation error
R ( g ) ≥ 14 for any neural network g with u ReLUs and l layers.
The above discussion implies the following theorem:
Theorem 4.1 (Classiﬁcation Error Theorem) .
Let k be a positive integer and f be a function of period m × p with p an odd number greater than one and m being a power of two (it might hold m = 1 ).
We set ρ to be the positive root greater than one of the polynomial equation λ p − 1 − λ p − 2 − 1 = 0 .
We can construct a sequence of points ( x i , y i ) 2 ni =1 with n := (cid:98) ρ k (cid:99) 2 so that the classiﬁcation error of function f mk is zero, whereas the classiﬁcation error of any neural network of l layers and u nodes per layer with u ≤ ρ kl 8 satisﬁes R ( g ) ≥ 14 .
Remark 4.1 .
Observe that if the number of units u per layer is constant and the number of layers l is o ( k ) , then the classiﬁcation error is always a positive constant for any neural network (whereas for f mk is zero).
Moreover, observe that since ρ is decreasing in p (recall p is the odd factor of the period), it holds that the classiﬁcation error decreases as p increases (with ﬁxed number of layers and nodes per layer).
This indicates that the composition of functions with large odd period is simpler than of functions with small odd period (period greater than one) following the intuition we have from the Sharkovsky’s ordering. A CKNOWLEDGMENTS
Vaggos Chatziafratis is partially supported by an Onassis Foundation Scholarship.
Sai Ganesh Nagarajan would like to acknowledge SUTD President’s Graduate Fellowship (SUTD-PGF). Ioannis Panageas and Xiao Wang would like to acknowledge NRF-NRFFAI1-2019-0003, SRG ISTD136 and ANR NRF 0095 ALIAS. Part of this project happened while the authors were visiting the
Simons Foundation program “Foundations of Deep Learning” at Berkeley and would like to thank the organizers for their hospitality.
A A PPENDIX
Claim A.1.
The characteristic polynomial of A (cid:62) is:
[Equation]
(A.1)
Proof.
Let I denote the identity matrix of size ( r + 1) ×
( r + 1) .
We consider the matrix:
A (cid:62) − λI =   1 − λ    . . .
− λ   . . .
− λ  . . . − λ     . . .
− λ   .
Observe that λ = 0 , 1 are not eigenvalues of the matrix A (cid:62) .
, hence we can multiply the ﬁrst row byλ − 1 , the second row by 1 λ ( λ − 1) , the third row by 1 λ 2 ( λ − 1) ,..., the i -th row by 1 λ i − 1 ( λ − 1) (and so on) and add them to the last row.
Let B be the resulting matrix:
B =    1 − λ    . . .
− λ   . . . − λ  . . .
− λ     . . .
− λ + 1 λ r − 1 ( λ − 1) 
  .
It is clear that det ( B ) =
as an equation has the same roots as det ( A (cid:62) − λI ) = 0 .
Since B is an upper triangular matrix, it follows that
[Equation]
We conclude that the eigenvalues of A (cid:62) (and hence of A ) must be roots of ( λ r − λ r − 1 ) λ − 1 and the claim follows.
Lemma 3 (Period power of two) .
There exist continuous functions f with prime period n that is a power of two so that the number of crossings C x,y ( f t ) scales at most polynomially with t for any x, y ∈ [0 , 1] .
Proof.
The easiest example one can construct is the function f : [0 , 1] → [0 , 1] that is deﬁned f ( x ) = 1 − x. Observe that for any a ∈ [0 , 1] one has f ( f ( a ))
= a and moreover if a (cid:54) =
12 then f ( a ) (cid:54) = a .
Hence f is a function of prime period two.
It is also clear that f t ( x ) =
x if t is even and f t ( x ) = 1 − x if t is odd, so the number of crossings is always one for all t ∈ N ∗ .
One other less trivial example is the following function (see also Figure 6):
[Equation]
It is not hard to see that this function has prime period four ( f (1)
= 4 , f (4) = 2 , f (2) = 3 , f (3) = 1 ).
Let J 0 =
[1 , 2] , J 1 = [2 , 3] , J 2 = [3 , 4] .
It is clear that
• f
( J 0 )
=
J 2 , f ( J 1 ) =
J 0 ∪ J 1 and f ( J 2 ) =
J 0 .
By letting δ ti be the number of crossings of the function f for the interval J i ( i ∈ { 0 , 1 , 2 } ), one has recursively   δ t + δ t + δ t + 
 = (cid:32) 0 10 11
[Equation]
where δ
=
(1 , 1 , 1) (all ones vector).
It is easy to observe that the matrix A =
(cid:32)[-
-]
[Equation] [Equation] 1 0(cid:33) has spectral radius one (as opposed to the case of odd period greater than one) and moreover it holds
5 x0.
[Table]
(a) This ﬁgure captures one composition of function f .
Observe that f crosses the interval [2 , 3] two times (once for x ∈ [1 , 2] and once for x ∈ [2 , 3] ) and it crosses the interval [1 , 2] once.
In particular, δ 1 =
5 x0.
[Table]
This ﬁgure captures
two compositions of func- tion f . Observe that f crosses the interval [2 ,
three times (two times for x ∈ [2 , 3] and once for x ∈ [1 , 2] ) and it crosses the interval [1 , 2] two times (once for x ∈ [1 , 2] and once for x ∈ [2 , 3] ).
In particular, δ 2 = (3 , 2) .
[Table]
c)
This ﬁgure captures three compositions of func- tion
f .
Observe that f
crosses
the
interval [
3] ﬁve times (three times for x ∈ [2 , 3] and twice for x ∈ [1 , 2] ) and it crosses the interval [1 , 2] three times (once for x ∈ [1 , 2] and twice for x ∈ [2 , 3] ).
In particular, δ 3 = (5 , 3) . [Table]
[Table]
d) This ﬁgure captures four compositions of func- tion f .
Observe that f crosses the interval [2 , 3] eight times (ﬁve times for x ∈ [2 , 3] and three times for x ∈ [1 , 2] ) and it crosses the interval [1 , 2] ﬁve times (twice for x ∈ [1 , 2] and three times for x ∈ [2 , 3] ).
In particular, δ 4 = (8 , 5) .
[Figure]
6 x      f
( x ) (a) δ 1 = (1 , 1 , 2 , 2) .
6 x      f 2 ( x ) (b) δ 2 = (2 , 2 , 3 , 2) .
6 x      f 3 ( x ) (c) δ 3 = (2 , 3 , 5 , 4) .
6 x      f 4 ( x ) (d) δ 4 = (4 , 5 , 7 , 5) .
that (cid:80)
i
=
(cid:80)
j =
A tij = t + 3 for all t ∈ N ∗ .
We conclude that α t 0 + α t 1 + α t 2 = t + 3 , therefore the number of crossings for J 0 , J 1 , J 2 of the function f t grows linearly with t (and not exponentially).
Since the function we deﬁned is of prime period four and is piecewise monotone (and so is any composition with itself) in each interval J 0 , J 1 , J 2 , we conclude that the number of crossings of f t for any possible pairs of values is at most linear in t .
B A N E XAMPLE WHICH IS PERIOD 5 BUT NOT PERIOD 3 (L I & Y ORKE (1975))
Here, we show an example function, that has a point of period 5, but not period 3, thereby respecting the Sharkovsky ordering.
Our proof approach for general odd periods is similar to the case of period 3, by using the induced covering graph and counting the crossings over each interval.
This is illustrated in Figure 7.
C T HE H ETEROGENEITY OF THE L OGISTIC M AP
In this section, we illustrate how the compositions of the logistic map f ( x ; r ) :
= rx (1 − x ) behaves as r varies slightly.
We give certain examples in the form of Figure 8.
It is known that the map when r = 3 . 9 , has a point of period 3.
In contrast when r is reduced to 3 .
5 the map has a point of period 4 and further bringing r down to 3 .
2 will ensure that the map has a point of period 2.
The ﬁgures below illustrate how the oscillations grow under these scenarios.
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
1 x0.
f 6 ( x ) (f) Here
f 6 ( x ; 3 . 2) is shown.
[Figure] [Figure]
[Figure] [Figure]
D F URTHER DISCUSSIONS
In this section, we provide some additional theoretical and experimental remarks on our characterization.
D. I NCORPORATING B IAS TERMS
If we add a bias term in the ReLU activation unit, e.g., use max( v, (cid:15) ) instead of max( v, 0) for the activation gates, where (cid:15) is a small number (positive or negative), then our results do not change; in particular our trade-off in Theorem 4.1 still holds (since the Lemma 2.2 from Telgarsky (2015) is for general sawtooth functions).
But, if one adds the bias term to the function f itself, then things get more interesting indeed: Suppose f has some period p where p is not a power of two; due to bifurcation phenomena (i.e., phenomena arising because we are at critical regimes of parameters such as the parameter µ in our generalized triangle wave function), then the compositions of the function ( f +bias term) with itself may give rise to qualitatively different behaviors compared to f .
In particular, the function ( f +bias term) might not have period p anymore.
Intuitively, one can think that the small bias term is ampliﬁed after many compositions and is not negligible anymore.
One such example is the triangle function f ( x ) = φx for 0 ≤ x ≤ 0 .
5 and φ (1 − x ) for 1 / 2 ≤ x ≤ 1 , where φ = (1 + √ 5) / 2 is the golden ratio.
This function has period 3, see Figure 9a.
However, if we consider the function g ( x ) =
( φ − (cid:15) )
x for 0 ≤ x ≤ 0 . 5 and ( φ − (cid:15) )
(1 − x ) for 0 .
5 ≤ x ≤ 1 with (cid:15) > 0 (arbitrarily small positive) then g does not have period 3, see Figure 9b.
In this sense, period as a property can be brittle to numerical changes if we are at the critical point.
D. S OME E XPERIMENTAL E VIDENCE
In this section, we provide experimental evidence for our depth separation results by training a neural network of constant width, but with increasing depth on a classiﬁcation task that closely resembles the n -alternating points problem that appeared in Telgarsky (2015) and is the foundation of our separation results as well.
As mentioned before, this is a speciﬁc instance of a function that has a point of period 3.
For simplicity, we do not consider this original problem exactly but rather a “smoothed” variant of it, in order to make it more amenable to the training procedure.
Our goal is to create a diagram showing how the classiﬁcation error drops as a function of the depth of the network for a ﬁxed value of the width.
We create 8000 equally spaced points from [0,1] (in increasing order), where the ﬁrst 1000 points are of label 0, the second 1000 are label 1 and this label alternates every 1000 points.
This is what we call a “smoothed” alternating point problem.
Although, the theory would have used the classical 8-alternating points to argue about the lower bounds, in practice, performing training of deep ( and above layers) and narrow networks (hidden layers with less than 4 neurons) with very few data points is a major challenge, see for instance
Lu et al.
Apart from the separation results that we show in theory, we show empirically that deep networks generally do improve the accuracy in this task compared to the shallow network and in fact a deep network with 5 layers can reach an accuracy of 99.04%.
Any additional uncertainties in the error is generally attributed to the training procedure.
To perform the experiments, we vary the depth of the neural network (excluding the input and the output layer) as d = 1 , 2 , 3 , 4 , 5 .
In addition, we ﬁx the neurons for each layer to be 6.
All activations are ReLU’s, while the last layer is the classiﬁer that uses a sigmoid to output probabilities.
Each model adds one extra hidden layer and we make use of the same hyper-parameters to train all networks.
Moreover, we require the training error or the classiﬁcation error to tend to 0 during the training procedure, i.e, we will try and overﬁt the data (as we try to demonstrate a representation result, rather than a statistical/generalization result).
Thus, for the actual training we use the same parameters to train all the different models using the “ADAM” optimizer Kingma & Ba (2014) and make the epochs to be 200 in order to enable overﬁtting.
To record the training error, we verify that the training saturates by seeing the performance over the epochs and report by default the error in the last epoch.
The results are shown in Figure 10.
D. P ERIOD AS A N ATURAL C HARACTERIZATION
In a nutshell, our paper provides a “natural” property of a function (periodic points of certain periods) and then derive depth-width trade-offs based on it.
This addresses some questions raised not only in Telgarsky (2016; 2015)’s works, but also in the paper Poole et al.
(2016) that seeks to provide a natural, general measure of functional complexity helping us understand the beneﬁts of depth.
On the contrary, many of the previous depth separation results take a worst case approach for the representation question (showing that there exist functions implemented by deep networks that are hard to approximate with a shallow net).
However, it is not clear whether such analysis applies to the typical instances arising in practice of neural-networks.
We believe that our work together with
Telgarsky (2016; 2015) and the paper Eldan & Shamir (2016) show a depth separation argument for very natural functions, such as the triangle waves or the indicator function of the unit ball.
Given a speciﬁc prediction task in practice, how could one assess the period?
We believe that this would be extremely useful yet a very difﬁcult question that seems to be outside the reach of current techniques in the literature.
Previous works and our work so far are able to present depth separation for representing certain functions.
We point out that, intuitively, our characterization result consists of a certiﬁcate informing us qualitatively and quantitatively about which functions have complicated compositions and which not.
Similar to computational problems in class NP, if one is given the certiﬁcate (the points ( x 1 , . .
, x p ) , then one can easily verify (if we have oracle access to evaluate the function f ), if the given function has a p -periodic cycle with points ( x 1 , . . .
, x p ) .
Nevertheless, we believe that ﬁnding the certiﬁcate for arbitrary continuous functions is not a straightforward problem, except maybe for particular restricted classes of functions.
Having said that, we want to emphasize that in many prediction problems that are inspired by physics, one may a priori expect to have complicated dynamics behavior and hence require deeper networks for better performance.
Such examples include efforts to solve the notorious 3-body problem or turbulent ﬂows showing empirical evidence that complex physical processes require deep networks (see for instance, Ling et al.
(2016) and
Breen et al.
(2019) that uses a 10 layered neural network).