R ETHINK
D EPTH
S EPARATION
WITH
I NTRA - LAYER
L INKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The depth separation theory is nowadays widely accepted as an effective explanation for the power of depth, which consists of two parts: i) there exists a function representable by a deep network; ii) such a function cannot be represented by a shallow network whose width is lower than a threshold.
Here, we report that adding intra-layer links can greatly improve a network’s representation capability through the bound estimation, explicit construction, and functional space analysis.
Then, we modify the depth separation theory by showing that a shallow network with intra-layer links does not need to go as wide as before to express some hard functions constructed by a deep network.
Such functions include the renowned ”sawtooth” functions.
Our results supplement the existing depth separation theory by examining its limit in a broader domain.
Also, our results suggest that once configured with an appropriate structure, a shallow and wide network may have expressive power on a par with a deep network.
I NTRODUCTION
Due to the widespread applications of deep networks in many important fields (LeCun et al., 2015), mathematically understanding the power of deep networks has been a central problem in deep learning theory (Poggio et al., 2020).
The key issue is figuring out how expressive a deep network is or how increasing depth promotes the expressivity of a neural network better than increasing width.
In this regard, there have been a plethora of studies on the expressivity of deep networks, which are collectively referred to as the depth separation theory.
A popular idea to demonstrate the expressivity of depth is the complexity characterization that introduces appropriate complexity measures for functions represented by neural networks (Pascanu et al., 2013; Montufar et al., 2014; Telgarsky, 2015; Mont ´ ufar, 2017; Serra et al., 2018; Hu & Zhang, 2018;
Xiong et al., 2020; Bianchini & Scarselli, 2014; Raghu et al., 2017), and then reports that increasing depth can greatly boost such a complexity measure.
In contrast, a more concrete way to show the power of depth is to construct functions that can be expressed by a small network of a given depth, but cannot be approximated by shallower networks, unless its width is sufficiently large (Telgarsky, 2015; 2016; Arora et al., 2016; Eldan & Shamir, 2016; Safran & Shamir, 2017; Venturi et al., 2021).
For example, Eldan & Shamir (2016) constructed a radial function and used Fourier spectrum analysis to show that a two-hidden-layer network can represent it with a polynomial number of neurons, but a one-hidden-layer network needs an exponential number of neurons to achieve the same level of error.
Telgarsky (2015) employed a ReLU network to build a one-dimensional ”sawtooth” function whose number of pieces scales exponentially over the depth.
As such, a deep network can construct a sawtooth function with many pieces, while a shallow network cannot unless it is very wide.
Arora et al.
(2016) derived the upper bound of the maximal number of pieces for a univariate ReLU network, and used this bound to elaborate the separation between a deep and a shallow network.
In a broad sense, we summarize the elements of establishing a depth separation theorem as the following: i) there exists a function representable by a deep network; ii) such a function cannot be represented by a shallow network whose width is lower than a threshold.
The depth separation theory is nowadays widely accepted as an effective explanation for the power of depth.
However, we argue that depth separation does not hold when we slightly adjust the structure of the shallow networks.
Our investigation is on ReLU networks. As shown in Figure 1(c), we find that inserting intra-layer links, which forces a neuron to take the outputs of its neighboring neuron, can greatly increase the maximum number of pieces represented by a shallow net- work.
As such, we can modify the statement of depth separation: without the need of going as wide as before, a shallow network can express as a complicated function as a deep network could.
[Figure]
[Figure] … O O
Our result is valuable in two aspects.
On the one hand, it non-trivially supplements the depth separation theory.
In reality, a neural network often is not feedforward but uses shortcuts to link distant layers to facilitate feature reuse and easy training.
Exploring the depth separation in the shortcut paradigm reveals the limit of the existing theory and motivates us to rethink the genuine power of depth.
On the other hand, the superiority of depth over width nowadays seems to become a doctrine for most deep learning practitioners.
However, studies such as width-depth equivalence (Fan et al., 2020) and representation ability of wide networks (Lu et al., 2017; Levine et al., 2020) show the essential role of width.
In the same vein, our work suggests the potential of wide networks in expressivity. Once configured with an appropriate structure, a shallow and wide network may have expressive power on a par with a deep network.
Note that adding intra-layer links is not equivalent to increasing depth.
The common understanding of increasing depth is to increase the number of layers, while intra-layer links are just to connect neurons in the same layer, which is a slight change to the original network.
Specifically, our roadmap to the modification of depth separation theorems includes two milestones.
1) Through bound analysis (Theorems 4, 6, and 8), explicit construction (Propositions 1, 2, and 3), and functional space analysis (Theorem 10), we substantiate that a network with intra-layer links can produce much more pieces than a feedforward network, and the gain is at most exponential, i.e. , ( 32 ) k , where k is the number of hidden layers.
2) Since intra-layer links can yield more pieces, they can be used to modify depth separation theorems by empowering a shallow network to represent a function constructed by a deep network, even if the width of this shallow network is lower than the prescribed threshold.
The modification is done in the cases of k 2 vs 3 (Theorem 12) and k 2 vs k (Theorem 13, the famous sawtooth function (Telgarsky, 2015)).
Also, we point out that the depth separation cannot be fully accomplished based on the bound analysis, unless the bound is proved to be tight. Thus, Arora et al.
(2016)’s depth separation theorem might need to be re-examined.
To summarize, our contributions are threefold.
We point out the limitation of the depth separation and propose to consider inserting intra-layer links in shallow networks.
2) We show via bound estimation, explicit construction, and functional space analysis that intra-layer links can make a
ReLU network produce more pieces. We modify the depth separation result including the famous Telgarsky (2015)’s theorem by demonstrating that a shallow network with intra-layer links does not need to go as wide as before to represent a function constructed by a deep network.  RELATED WORK
Recently, a plethora of depth separation studies have shown the superiority of deep networks over shallow ones from the points of view of complexity analysis and constructive analysis.
The complexity analysis is to characterize the complexity of the function represented by a neural network, thereby demonstrating that increasing depth can greatly maximize such a complexity measure.
Currently, one of the most popular complexity measures is the number of linear regions because it conforms to the functional structure of the widely-used ReLU networks.
For example, Pascanu et al.
(2013); Montufar et al.
(2014); Mont´ufar (2017); Serra et al.
; Hu & Zhang (2018); Hanin & Rolnick (2019) estimated the bound of the number of linear regions generated by a fully-connected ReLU network by applying Zaslavsky’s Theorem (Zaslavsky, 1997).
Xiong et al.
(2020) offered the first upper and lower bounds of the number of linear regions for convolutional networks.
Other complexity measures include classification capabilities (Malach & Shalev-Shwartz, 2019), Betti numbers (Bianchini & Scarselli, 2014), trajectory lengths (Raghu et al., 2017), global curvature (Poole et al., 2016), and topological entropy (Bu et al., 2020).
Please note that using complexity measures to justify the power of depth demands a tight bound estimation.
Otherwise, it is insufficient to say that shallow networks cannot be as powerful as deep networks, since deep networks cannot reach the upper bound.
The construction analysis is to find a family of functions that are hard to approximate by a shallow network, but can be efficiently approximated by a deep network.
Eldan & Shamir (2016) built a special radial function that is expressible by a 3-layer neural network with a polynomial number of neurons, but a 2-layer network can do the same level approximation only with an exponential number of neurons.
Later, Safran & Shamir (2017) extended this result to a ball function, which is a more natural separation result.
Venturi et al.
(2021) generalized the construction of this type to a non-radial function.
Telgarsky (2015; 2016) used an O ( k 2 ) -layer network to construct a sawtooth function.
Given that such a function has an exponential number of pieces, it cannot be expressed by an O ( k ) -layer network, unless the width is O (exp( k )) .
Arora et al.
(2016) estimated the maximal number of pieces a network can produce, and established the size-piece relation to advance the depth separation results from ( k 2 , k ) to ( k , k ′ ), where k ′ <
k .
Other smart constructions include polynomials (Rolnick & Tegmark, 2017), functions of a compositional structure (Poggio et al., 2017), Gaussian mixture models (Jalali et al., 2019), and so on.
Our work also highlights the construction, and we use an intra-linked network to more efficiently build a sawtooth function.
N OTATION AND D EFINITION
Notation 1 (Feedforward networks).
For an R w 0 → R ReLU DNN with widths w 1 , . . .
, w k of k hidden layers, we use f 0 = (cid:104) f (1)0 , . . . , f ( w 0 )(cid:105) = x ∈ R w 0 to denote the input of the network. Let f i = (cid:104) f
(1) i , . . . ,
f ( w i ) i (cid:105) ∈ R w i , i = 1 , · · · , k, be the vector composed of outputs of all neurons in the i -th layer, for i = 1 , . . .
, k , j = 1 , . . .
, w i , we use g ( j ) i = (cid:68) a ( j ) i , f i − 1 (cid:69) + b ( j ) i to denote the pre-activation of the j -th neuron of the i -th layer, where a ( j ) i ∈
R w i − 1 , b ( j ) i ∈ R are parameters. Then f ( j [-+
-]) i =
σ (cid:16) g
( j ) i (cid:17) , where σ ( · ) is the ReLU activation.
The output of this network is g k +1 = ⟨ a k , f k ⟩ + b k , where a k ∈ R w k , b k ∈ R are parameters.
Notation 2 (Intra-linked networks).
For an R w 0 → R ReLU DNN with every 2 paired neurons linked in each hidden layer and widths w 1 , . . .
, w k of k hidden layers, we use ˜ f {+
 = (cid:104) ˜ f ˜ f+}
( w 0 )(cid:105) = x ∈ R w 0 to denote the input of the network. Let ˜ f i = (cid:104) ˜
f (1) i , . . . , ˜ f ( w i ) i (cid:105) ∈
R w i be the vector composed of the neurons in the i -th layer, then for i = 1 , . . . , k , j = 1 , . . .
, w i , we use ˜ g ( j ) i
= (cid:68) ˜ a ( j ) i , ˜ f i − 1 (cid:69) +˜ b ( j ) i to denote the pre-activation of the j -th neuron in the i -th layer, where ˜ a ( j ) i ∈ R w i − 1 , ˜ b ( j ) i ∈
R are some parameters. In an intra-linked network, the j -th and ( j +1) -th neurons are linked, and the ( j + 2) -th and ( j + 3) -th neurons are linked, we prescribe ˜ f
( j ) i
= σ (cid:16) g ( j ) i (cid:17) , ˜ f
( j +1) i =
σ (cid:16) ˜ g
( j +1)
i − ˜ f
( j ) i (cid:17) , ˜ f ( j +2)
i = σ (cid:16) ˜ g
( j +2) i (cid:17) , ˜ f
( j +3) i = σ (cid:16) ˜ g ( j +3)
i − ˜ f
( j +2) i (cid:17) .
Similar with the classical network, the output of the network is ˜ g
k +
=
(cid:68) ˜ a k , ˜ f
k (cid:69) + ˜ b k , where ˜ a k ∈ R w k , ˜ b k ∈ R are parameters.
Notation 3 (Sawtooth function). A piecewise linear (PWL) function g : [ a, b ] → R is of ” Nsawtooth” shape, if g = ( − 1) n − 1 (cid:0) x − ( n − 1) · b − aN (cid:1) , x ∈ (cid:2) ( n − 1) · b − aN , n · b − aN (cid:3) , for n ∈ [ N ] .
Definition 1 (Width and depth of a feedforward network (Arora et al., 2016)) .
For any number of hidden layers k ∈ N , input and output dimensions w 0 , w k +1 ∈ N , an R w 0 → R w k +1 feedforward network is given by specifying a sequence of k natural numbers w 1 , w 2 , . . .
, w k representing widths of the hidden layers.
The depth of the network is defined as k + 1 .
The width of the network is max { w 1 , . . .
, w k } .
Definition 2 (Width and depth of a shortcut network (Fan et al., 2020)) .
Given a shortcut network Π , we delete the minimum number of links to make the resultant network Π ′ a feedforward network without isolated neurons.
Then, we define the width and depth of Π as the width and depth of Π ′ .
Admittedly, defining the width and depth of a network embedded with shortcuts is tricky.
A reasonably good definition should conform to the customary understanding of depth and width.
For example, the ResNet (He et al., 2016) shall not be taken as a wide network in the light of the proposed definition; otherwise, it conflicts with practitioners’ common sense.
Our definition for width and depth can fit the common sense for the network that is formed by slightly modifying a feed- forward network, e.g. , ResNet, DenseNet (Huang et al., 2017), and S3Net (Fan et al., 2018). Per our definition, the width and depth of intra-linked networks are also max { w 1 , . . . , w k } and k + 1 , respectively, the same as the width and depth of the corresponding feedforward network.
R ETHINK THE D EPTH S EPARATION WITH I NTRA - LAYER L INKS
Since our focus is the network using ReLU activation and related estimation of the number of pieces, the seminal depth separation theorems closest to us are the following:
Theorem 1 (Depth separation k 2 vs k (Telgarsky, 2015; 2016)) .
For a natural number k ≥ 1 , there exists a sawtooth function representable by an R → R (2 k 2 + 1) -layer feedforward ReLU DNN of width 2 such that if it is also representable by a ( k + 1) -layer feedforward ReLU DNN, this ( k + 1) -layer feedforward ReLU DNN should at least have the width of 2 k − 1 .
Theorem 2 (Depth separation k vs k ′ (Arora et al., 2016)) .
For every pair of natural numbers k ≥ 1 , w ≥ 2 , there exists a function representable by an R → R ( k + 1) -layer feedforward ReLU
DNN of width w such that if it is also representable by a ( k ′ + 1) -layer feedforward ReLU DNN for any k ′ ≤ k , this ( k ′ + 1) -layer feedforward ReLU DNN has width at least 12 w kk ′ .
Despite being one-dimensional, the above results convincingly reveal that increasing depth can make a ReLU network express a much more complicated function, which is the heart of depth separation.
Here, we shed new light on the depth separation problem with intra-layer links.
Our primary argument is that if intra-layer links shown in Figure 1(c) are inserted, there exist shallow networks that previously cannot express some hard functions constructed by deep networks now can do the job.
Our investigation consists of two parts.
First, we substantiate that adding intra-layer links can greatly increase the number of pieces via bound estimation, explicit construction, and functional space analysis.
Then, adding intra-layer links can empower the shallow networks to represent complicated functions such as sawtooth functions, without the need of going as wide as before.
4. I NTRA -L AYER L INKS C AN I NCREASE THE N UMBER OF P IECES 4.1. U PPER B OUND E STIMATION
Lemma 3.
Let g : R → R be a PWL function with w + 1 pieces, then the breakpoints of f := σ ( g ) consists of two parts: some old breakpoints of g and at most w + 1 newly produced breakpoints.
Furthermore, f has w + 1 new breakpoints if and only if g has w + 1 distinct zero points.
Proof.
A direct calculus.
Theorem 4 (Upper bound of feedforward networks) .
Let f : R → R be a PWL function represented by an R → R ReLU DNN with depth k + 1 and widths w 1 , . . .
, w k of k hidden layers.
Then f has at most (cid:81) k
i =1 ( w i + 1) number of pieces.
This bound is the univariate case of the bound: (cid:81) ki
=1 (cid:80) nj =0 (cid:0) w i j (cid:1) , derived in Mont´ufar (2017) for n -dimensional inputs.
In Appendix B, we offer constructions to show that this bound is achievable in a depth-bounded but width-unbounded network (depth=3)
(Proposition 4) and a width-bounded (width=3) but depth-unbounded network (Proposition 5) in one-dimensional space. Previously many bounds Pascanu et al. (2013); Montufar et al. (2014); Mont´ufar (2017); Xiong et al.
(2020) on linear regions were derived, however, it is unknown that these bounds are vacuous or tight, particularly for networks with more than one hidden layer.
What makes Propositions 4 and 5 special is that they for the first time substantiate that Mont´ufar (2017)’s bound is tight over three-layer and deeper networks, although these results are for the one-dimensional case.
Remark 1 (Sharpening the bound in (Arora et al., 2016)).
Previously, Arora et al.
(2016) computed the number of pieces produced by a network of depth k + 1 and widths w 1 , . . .
, w k as 2 k +1 · ( w 1 + 1) w 2 · · · w k .
The reason why their bound has an exponential term is that when considering how ReLU activation increases the number of pieces, they repetitively computed the old breakpoints generated in the previous layer.
Our Lemma 3 implies that the ReLU activation in fact cannot generate as many as double pieces.
Since Arora et al.
(2016)’s bound is loose, their depth separation theorem needs to be re-examined.
Lemma 5.
Let g 1 , g 2 : R → R be two PWL functions with totally w breakpoints.
Set f 1 :=
σ ( g 1 ) and f 2 := σ ( g 2 − f 1 ) .
Then the breakpoints of f 2 consist of three parts: some breakpoints of g 2 , some breakpoints of f 1 , and at most 2 w + 2 newly produced breakpoints.
Furthermore, f 2 has 2 w + 2 newly produced breakpoints if and only if g 2 − f 1 has 2 w + 2 distinct zero points.
Proof.
A direct corollary of Lemma 3.
Let us illustrate why the intra-linked architecture can produce more pieces.
Given two PWL functions g 1 and g 2 which has totally w breakpoints, in the feedforward architecture, σ ( g 1 ) and σ ( g 2 ) have totally at most 3 w + 2 breakpoints, which contains at most w old breakpoints of g 1 , g 2 and at most 2 w +2 newly produced breakpoints.
However, in the intra-linked architecture, σ ( g 2 − σ ( g 1 )) can produce more breakpoints because σ ( g 1 ) has two states: activated or deactivated. Then, σ ( g 1 ) and σ ( g 2 − σ ( g 1 )) consist of at most w old breakpoints of g 1 , g 2 and ( w +1)+(2 w +2)
= 3 w + newly produced breakpoints.
Theorem 6 (Upper bound of intra-linked networks) .
Let f : R → R be a PWL function represented by a ReLU DNN with depth k + 1 , widths w 1 , . . .
, w k , and every two neurons linked in each hidden layer as Figure 1(c).
Assuming that w 1 , . .
. , w k are even, f has at most (cid:81) k
i =1 (cid:0) {+
 2+} w i + 1 (cid:1) pieces.
Proof.
We prove by induction on k .
For the base case k = 1 , we assume for every odd j , the neurons ˜ f
( j ) 1 and ˜ f
( j +1)are linked.
The number of breakpoints of ˜ f
( j ) 1 , j = 1 , . . . , w 1 , is at most 2 + ( − 1) j .
Hence, the first layer yields at most 32 w 1 + 1 pieces.
For the induction step, we assume that for some k ≥ 1 , any R → R ReLU DNN with every two neurons linked in each hidden layer, depth k + 1 and widths w 1 , . . .
, w k of k hidden layers produces at most (cid:81) ki
=1 (cid:0) 32 w i + 1 (cid:1) pieces.
Now we consider any R → R ReLU DNN with every two neurons linked in each hidden layer, depth k + 2 and widths w 1 , . . .
, w k +1 of k + 1 hidden layers.
By the induction hypothesis, each ˜ g ( j ) k + has at most (cid:81) ki =1 (cid:0) 32 w i + 1 (cid:1) − 1 breakpoints.
Then the breakpoints of σ (˜ g ( j ) k +1 ) consist of some breakpoints of ˜ g
( j
) k +1 and at most (cid:81) ki
=1 (cid:0) 32 w i + 1 (cid:1) newly generated breakpoints.
Then ˜ g
( j +1)
k
+1 − ˜ f
( j ) k
+ has at most 2 · (cid:81) ki =1 (cid:0) 32 w i + 1 (cid:1) − 1 breakpoints, based on Lemma 5.
The breakpoints of ˜ f ( j +1) k + = σ (˜ g
( j +1)
k +1 − ˜ f ( j ) k +1 ) consist of some breakpoints of ˜ g
( j +1) k
+1 − ˜ f
( j ) k +1 and at most 2 · (cid:81) ki
=1 (cid:0) 32 w i + 1 (cid:1) newly generated breakpoints.
Note that ˜ g (1)
k +1 , . . .
, ˜ g ( w k +1 ) k + have totally at most (cid:81) ki
=1 (cid:0) 32 w i + 1 (cid:1) −breakpoints.
In all, the number of pieces we can therefore get is at most 1 + w k +  · (cid:32) k
(cid:89) i
=
(cid:18)2 w i +(cid:19) + 2 · k (cid:89) i
= (cid:18)2 w i +(cid:19)(cid:33)
+ k (cid:89) i = (cid:18)2 w i +(cid:19) − 1 = k +1 (cid:89)
i = (cid:18)2 w i +(cid:19) .
In the following theorems, we offer the bound estimation for high-dimensional cases. The detailed proof for Theorem 8 is put into Appendix A.
Theorem 7 (Upper Bound of Feedforward Networks (Mont´ufar, 2017)) .
Let f : R n → R be a PWL function represented by an R n → R ReLU DNN with depth k +1 and widths w 1 , . . . , w k of k hidden layers. Then f has at most (cid:81) ki =1 (cid:80) nj =0 (cid:0) w i
j (cid:1) linear regions.
Theorem 8 (Upper Bound of Intra-linked Networks) .
Let f : R n → R be a PWL function represented by an R n → R ReLU DNN with every two neurons linked in each hidden layer, depth k + 1 and widths w 1 , . . . , w k of k hidden layers.
We assume each w i is even.
Then f has at most (cid:81)
[Equation]
(cid:1) linear regions.
Remark 2.
Although both adding a new layer (going deep) and adding intra-layer links involve composition, their mechanisms of producing pieces are fundamentally different. While the mechanism of going deep is the repetition effect (multiplication), i.e. , the function value of the function being composed is oscillating, and each oscillation can generate corresponding pieces. The mechanism of intra-layer links is the gating effect (addition). The neuron being embedded have two activation states, and each state is leveraged by the neuron being linked to produce a breakpoint.
Such a mechanism essentially conforms to the parallelism, which is of width paradigm.
4.1. E XPLICIT C ONSTRUCTION .
Despite that the bound estimation offers some light, to convincingly illustrate that intra-layer links can increase the number of pieces, we need to supply the explicit construction for the intra-linked networks.
The number of pieces in the construction should be bigger than either the upper bound of feedforward networks or the maximal number a feedforward network can achieve.
Specifically, the constructions for intra-linked networks in Propositions 1 and 2 have a number of pieces larger than the upper bounds of feedforward networks.
In Proposition 3, by enumerating all possible cases, we present a construction for an intra-linked network of width 2 and arbitrary depth whose number of pieces is larger than what a feedforward network of width 2 and arbitrary depth possibly achieves.
Proposition 1 [-> 3 , we let f ( j )= σ ( − 2 x − 2( j − 3)) and g (1)2 = − (cid:16) f (1) + f (2) − f (3) + (cid:80) w 1 j =4 ( − 1) j − 1 f ( j ) 1 − 3 − 1 w 2 + (cid:17) g ( i ) 
-](The bound (cid:81) ki
=1 (cid:0) 3 w i 2 + 1 (cid:1) is tight for a two-hidden-layer intra-linked network) .
Given an R → R two-hidden-layer ReLU network, with every two neurons linked in each hidden layer, for any even w 1 ≥ 6 , w 2 ≥ 4 , there exists a PWL function represented by such a network, whose number of pieces is (cid:0) 3 w 1 2 + 1 (cid:1) (cid:0) 3 w 2 2 + 1 (cid:1) .
Proof.
To guarantee the bound (cid:81) ki =1 (cid:0) 3 w i 2 + 1 (cid:1) is tight, the following two conditions should be satisfied: (i) ˜
g
( j ) i and ˜ g
( j +1)
i − ˜ f ( j ) i have as many zero points as possible so that σ (˜ g ( j ) i ) and σ (˜ g ( j +1) i − ˜ f ( j ) i ) can produce the maximal number of breakpoints; (ii) all old breakpoints of (cid:110)
˜ g (1) i , . . . , ˜ g ( w i ) i (cid:111) are reserved by ˜ g
( j ) i +1 , an affine transform of (cid:110) ˜ f (1) i , [Equation] We first consider the first hidden layer.
Let
[Equation]
When w 1 = 6 , we set ˜ g
= − 29 ˜ f
[Equation]
When w> 6 , for each odd j > 6 , let ˜ f ( j ) [Equation] −
[Equation]
σ (cid:16) − [Equation] x − a j ) − ˜ f ( j ) 1 (cid:17) , where a j = − 192 − 9 (cid:0) j − 12 − 3 (cid:1) ,
then the output of the first layer is expressed as the following:
[Equation]
which has 32 w 1 + 1 pieces and whose adjacent pieces have slopes of opposite signs.
Note that any line y = b , where b ∈ ( − 13 / 2 , − 6) , can cross all pieces of ˜ g + b .
Thus, g fulfills the conditions of Lemma 5.
We divide the breakpoints of ˜ g into two parts: B upper = { x : x is a breakpoint of ˜ g and ˜ g ( x ) > b } and B lower = { x : x is a breakpoint of ˜ g and ˜ g ( x ) ≤ b } . We refer to their counts as # B upper and # B lower , respectively.
[Figure]
Next, we construct the second hidden layer.
˜ f
= σ (˜ g + b 1 ) , where b 1 ∈ ( − 13 / 2 , − 6) , has 32 w 1 + 1 new breakpoints. Then by choosing some scaling parameter a ∈ (0 , 1) bias b 2 to fulfill
Lemma 5, we can also make a ˜ g + b 2 − ˜ f (1)has 3 w 1 + 2 distinct zero-points, which implies ˜ f (2):= σ (cid:16) a ˜ g + b 2 − ˜ f
(1)2 (cid:17) has 3 w 1 + 2 newly produced breakpoints.
Therefore, the affine combination of ˜ f (1) and ˜ f (2) contains all breakpoints of B upper , and has # B upper + (cid:0) 32 w 1 + 1 (cid:1) + (3 w 1 + 2) breakpoints. To reserve all the breakpoints of ˜ g , we do the similar thing for − ˜ g to gain ˜ f (3) and ˜ f (4)2 , whose affine combination has # B lower + (cid:0)
32 w 1 + 1 (cid:1) +
(3 w 1 + 2) breakpoints, which contains all breakpoints in B lower , and shares no breakpoints with the affine combination of (cid:110) ˜ f (1)2 , ˜ f (2)2 (cid:111) . Hence, the affine combination of (cid:110) ˜ f
(1)2 , ˜ f (2)2 , ˜ f (3)2 , ˜ f (4)2 (cid:111) has # B upper + # B lower + 2 · (cid:0) 3 w 1 2 + 1 (cid:1) + 2 · (3 w 1 + 2) = (cid:0) 3 w 1 2 (cid:1) + 6 · (cid:0) 3 w 1 2 + 1 (cid:1) breaking points, which contains all the breakpoints of ˜ g .
(cid:110) ˜ f (1)2 , ˜ f
(2)2 , ˜ f (3)2 , ˜ f (4)2 (cid:111) are visualized in Figure 2. Repeating this procedure by selecting different b 1 , a, b 2 , we can construct the remaining { ˜ f ( i ) 2 } w 2 i =5 such that the affine transformation of { ˜ f
( i ) 2 } w 2 i = has pieces of
[Equation]
Proposition 2 (Use intra-linked networks to achieve a sawtooth function with (cid:81) ki =1 (cid:0) 3 w i 2 (cid:1) pieces) .
There exists a [0 , 1] → R function represented by an intra-linked ReLU DNN with depth k + 1 and width w 1 , . . .
, w k of k hidden layers, whose number of pieces is at least 3 w 1 2 · . . .
· 3 w k 2 .
Proof.
Let ϕ ( x ) = x defined over [0 , ∆] .
The core of the proof is to use a one-hidden-layer network of w ≥ 2 neurons to create 3 w 2 pieces from ϕ ( x ) .
[Figure]
Let δ = 2∆3 w .
Set ˜ g
(1) = 3 ϕ − 3 δ , ˜ f =
σ (cid:0) ˜ g (1) (cid:1) , ˜ g (2) =
ϕ , ˜ f =
σ (cid:16) ˜ g (2) − ˜ f (1) + δ (cid:17) , and ˜ g
(2 j +1)
= 4 ϕ − 4(3 j + 1) δ , f (2 j +1) = σ (cid:0)
˜ g
[Equation]
j +1) (cid:1) , ˜ g
(2 j +2)
= 2 ϕ − 6 jδ , ˜ f
(2 j +2)
=
σ (cid:16) ˜ g (2 j +2) − ˜ f (2 j +1) (cid:17) , for all j = 1 , . .
, w/ 2 − 1 .
The output of this one-hidden-layer network is
[Equation]
which has 3 w 2 pieces on [0 , ∆] .
ξ ∆ ,w ( x ) is of slope ( − 1) j on [ jδ, ( j + 1) δ ] , j = 0 , . .
, 3 w/ 2 − 1 , and ranges from 0 to δ on each piece.
Figure 3 shows how the affine transformation of { ˜ f (1) , ˜ f (2) , ˜ f (3) , ˜ f (4) } constructs a sawtooth function of 6 pieces. Please note that flipping ϕ ( x ) or translating ϕ ( x ) will not prevent ξ ∆ ,w ( ϕ ( x )) from generating 3 w 2 pieces. The targeted intra-linked ReLU network with depth k + 1 and width w 1 , . . .
, w k of k hidden layers is designed as [Equation] where ∆ i = 1 / (cid:16) (cid:81) i − 1 j =1 3 w i 2 (cid:17) .
Proposition 3 (Intra-layer links can greatly increase the number of pieces in an R → R ReLU network with width 2 and arbitrary depth) .
Let f : R → R be a PWL function represented by an R → R ( k + 1) -layer ReLU DNN with widths 2 of all k hidden layers.
Then number of pieces of f is at most (cid:40)
[Equation]
if k is even,
[Equation]
if k is odd.
There exists an R → R ( k + 1) -layer 2 -wide ReLU DNN, with neurons linked in each hidden layer, which can produce at least 7 · 3 k − 2 + 2 pieces.
Proof.
The proof is put in Appendix C.
4.1. F UNCTIONAL S PACE A NALYSIS
The above constructive analyses demonstrate that in the maximal sense, intra-layer links can empower a feedforward network to represent a function with more pieces.
Now, we move one step forward by showing that intra-layer links can surprisingly expand the functional space of a feedforward network.
The reason why this result is surprising is that one tends to think an intra-linked network produces an exclusively different function from a feedforward network.
However, here we report that an intra-linked one-hidden-layer network of two neurons can express a feedforward one-hidden-layer network of two neurons (Lemma 9), and the opposite doesn’t hold true. Furthermore, given an arbitrary feedforward ReLU network, adding intra-layer links in the first layer can definitely expand its functional space (Theorem 10).
Lemma 9.
Let f (1)
=
σ (cid:0) a (1) x
+ b (1) (cid:1) , f (2) =
σ (cid:0) a (2) x
+ b (2) (cid:1) , f
= c (1) f (1) + c (2) f
(2) + d , where a (1) a (2) > 0 , b (1) , b (2) , c (1) , c (2) , and d ∈ R , there exists some ˜ a (1) , ˜ a (2) , ˜ b (1) , ˜ b (2) , ˜ c (1) , ˜ c (2) , and ˜ d ∈ R such that f = ˜ c
(1) ˜ f
(1) + ˜ c
(2) ˜ f
(2) + ˜ d , where ˜ f (1) =
σ (cid:16) ˜ a (1) x + ˜ b (1) (cid:17) , ˜ f (2)
=
σ (cid:16) ˜ a (2) x
+ ˜ b (2) − ˜ f
(1) (cid:17) .
Proof.
Without loss of generality, we assume a (1) , a (2) > 0 and − b (2) a (2) < − b (1) a (1) .
Then f is of slope 0 , c (2) a (2) , and c (1) a (1) + c (2) a (2) on (cid:16) −∞ , − b (2) a (2) (cid:105) , (cid:104) − b (2) a (2) , − b (1) a (1) (cid:105) and (cid:104) − b (1) a (1) , ∞ (cid:17) , respectively.
Now we choose ˜ a (1) , ˜ a (2) satisfying 0 <
˜ a (1) <
˜ a (2) , and set ˜ b ( i ) =
b ( i ) a ( i ) .
˜ a ( i ) , i = 1 , 2 .
Then ˜ f
(1) is of slope 0 and ˜ a (1) on (cid:16) −∞ , − b (1) a (1) (cid:105) and (cid:104) − b (1) a (1) , ∞ (cid:17) , respectively, while ˜ f (2) is of slope 0, ˜ a (2) , and ˜ a (2) − ˜ a (1) on (cid:16) −∞ , − b (2) a (2) (cid:17) , (cid:104) − b (2) a (2) , − b (1) a (1) (cid:105) , and (cid:16) − b (1) a (1) , ∞ (cid:17) , respectively.
Hence, let ˜ c (2) =
c (2) a (2) /
˜ a (2) , ˜ c (1) =
(cid:0) c (1) a (1) + c (2) a (2) − ˜ c (2) (cid:0) ˜ a (2) − ˜ a (1) (cid:1)(cid:1) /
˜ a (1) , and ˜ d
= d , we have f = ˜ c (1) ˜ f
(1) + ˜ c
(2) ˜ f
(2) + ˜ d .
Theorem 10.
Let f be any R → R PWL representable by a classical ( k + 1) -layer ReLU DNN with widths w 1 > 2 , . . .
, w k of k hidden layers.
Then, f can also be represented by a ( k + 1) -layer
ReLU DNN with widths w 1 , . . .
, w k of k hidden layers, with neurons in the first layer linked.
Proof.
Let the output of the j -th neuron of the first layer in the feedforward network be f ( j ) 1 ( x ) =
σ (cid:16) a ( j ) 1 x + b ( j ) 1 (cid:17) , j = 1 , . . .
, w 1 .
Since the feedforward network is invariant to permutating neurons, we can link the arbitrary j -th and j ′ -th neuron if a ( j ) 1 a ( j ′ )> 0 , which directly concludes the proof according to Lemma 9.
4. M ODIFY THE D EPTH S EPARATION T HEOREM WITH I NTRA - LAYER L INKS
In a broad sense, the depth separation theorem consists of two elements: i) there exists a function representable by a deep network; ii) such a function cannot be represented by a shallow network whose width is lower than a threshold.
Since adding intra-layer links can generally improve the capability of a network, if one adds intra-layer links to a shallow network, the function constructed by a deep network can be represented by a shallow network, even if the width of this shallow network is still lower than the threshold.
Theorems 12 and 13 modify the depth separation k 2 vs 3 and k 2 vs k , respectively, by presenting that a shallow network with intra-layer links only needs to go 23 times as wide as before to express the same function.
Lemma 11 (A network with width=2 can approximate any univariate PWL function (Fan et al., 2018)) .
Given a univariate PWL function with n pieces p ( x ) , there exists a ( n + 1) -layer network D ( x ) with two neurons in each layer such that f ( x ) = D ( x ) .
Theorem 12 (Modify the depth separation k 2 vs 3) .
For every k ≥ 2 , there exists a function p ( x ) that can be represented by a ( k 2 + 1) -layer ReLU DNN with 2 nodes in each layer, such that it cannot be represented by a classical 3 -layer ReLU DNN W 2 ( x ) with width less than k − 1 , but can be represented by a 2 -layer, 2( k − 1)-wide intra-linked ReLU DNN ˜
W 2 ( x ) .
Proof.
Combining Theorem 4, Proposition 1, and Lemma 11 straightly concludes the proof.
Theorem 13 (Modify the depth separation k 2 vs k ) .
For every k ≥ 1 , there is a [0 , 1] → R PWL function p ( x ) represented by a feedforward (2 k 2 +1) -layer ReLU DNN with at most 6 nodes in each layer, such that it cannot be represented by a classical ( k + 1) -layer ReLU DNN W k ( x ) with width less than 6 k , but can be represented by a ( k + 1) -layer intra-linked ReLU DNN ˜
W k ( x ) with width no more than 4 · 6 k − 1 .
Proof.
Per (Telgarsky, 2016)’s construction, a feedforward (2 k 2 + 1) -layer ReLU DNN with at most 2 nodes in each layer can produce a sawtooth function of 2 k 2 pieces. Similarly, a feedforward (2 k 2 +1) -layer ReLU DNN with at most 6 nodes in each layer can have 6 k 2 pieces. Thus, it follows
Theorem 4 that any classical ( k + 1) -layer ReLU DNN W k
( x ) with width less than 6 k − 1 cannot generate 6 k 2 pieces. However, according to the construction in Proposition 2, let w 1 = w 2 = · · · = w k = 4 · 6 k − 1 , an intra-linked network can exactly express a sawtooth function with 6 k 2 pieces.
Remark 3.
Theorems 12 and 13 implicate that intra-layer links can reduce the bar of the width by 1/3.
Although it is not an exponential reduction, our highlight is the existence of such shallow networks that can be transformed by intra-layer links to have the representation power on a par with a deep network. Such shallow networks go against the predictions of depth separation theory.
Furthermore, suppose every n neurons are intra-linked in each layer, the upper bound of the number of pieces by a network of k hidden layers with widths w 1 , . . .
, w k is (cid:81) ki
=1 (cid:16) (2 n − 1) w i n + 1 (cid:17) .
Therefore, by intra-linking more neurons, the bar of the width can be substantially reduced, and more shallow networks will become counterexamples of the depth separation theory.
DISCUSSION AND CONCLUSION
Please note that intra-layer links are fundamentally different from residual connections or other kinds of shortcuts. The former is to link the neurons inside a layer, while the latter is to connect all outputs of the previous layer to the next layer or later layers.
We take the intra-layer links as inner links, while the shortcuts of ResNet (He et al., 2016) and so on are outer links.
It is widely recognized that outer links can facilitate networks in the dimension of depth, e.g. , the residual connection solves the training issues of deep networks and allows the network to go very deep.
In contrast, our theoretical results show that inner links can promote the representation capability of networks in the dimension of width, e.g. , the network with inner links does not need to go as wide as before to represent the same function.
By leveraging the capabilities of a network in both width and depth domains, we believe that the synergy of inner and outer links in a network will further contribute to more powerful networks.
More favorably, both inner and outer links do not incorporate new parameters; therefore, their synergy is likely to enhance model efficiency.
In this draft, via bound estimation, dedicated construction, and functional space analysis, we have shown that a network with intra-layer links is much more expressive than a feedforward one.
Then, we have modified the depth separation results to that a shallow network that previously cannot express some functions constructed by deep networks now can do the job with intra-layer links.
Our results supplement the existing depth separation theory, and suggest that the potential of wide networks can be released by an appropriate structure.
Future endeavors can be put into training wide networks using intra-layer links to achieve comparable performance with deep networks.
A P ROOF OF T HEOREM
Lemma 14 (Zaslavsky’s Theorem Zaslavsky (1975); Stanley (2004)) .
Let A =
{ H i ⊂ V : 1 ≤ i ≤ m } be an arrangement in R n .
Then, the number of regions for the arrangement
A satisfies
[Equation]
[-Then f has at most (cid:81) ki =1 (cid:80) nj =0 (cid:0) 
-]
Proof.
We prove by induction on k .
For the base case k = 1 , ˜ f
(2 i −
[Equation]
[Equation]
(cid:17) produces one hyperplane in the input space R n .
Furthermore, ˜ f (2 i [Equation] [Equation] σ
[Equation]
− σ (cid:16)
[Equation]
(cid:17)(cid:17) produces at most two hyperplanes in the input space R n .
Therefore, in total, the w 1 neurons in the first layer produces (1 + 2) · w 1 2 = 3 w 1 2 hyperplanes in the input space R n .
Then by Zaslavsky’s Theorem, it will produce at most (cid:80) nj =0 (cid:0) w 1 +1 j (cid:1) linear regions in the input space
R n .
For the induction step, we assume that for some k ≥ 1 , any R n → R ReLU DNN with every two neurons linked in each hidden layer, depth k +1 and widths w 1 , . . .
, w k of k hidden layers produces at most (cid:81) ki
=1 (cid:80) nj =0 (cid:0) [-
 wi-]3 w i 2 +1 j (cid:1) linear regions.
Now we consider any R n → R ReLU DNN with every two neurons linked in each hidden layer, depth k +2 and widths w 1 , . . .
, w k +1 of k +1 hidden layers.
Then for each linear region S produced by the first k + 1 layers, again, ˜ f (2 i − 1) k
+ = σ (cid:16) ˜
g
(2 i − 1) k +1 (cid:17) produces one hyperplane in S .
Furthermore, ˜ f (2 i ) k
+1 =
σ (cid:16) ˜
g (2 i )
k +1 − ˜ f
(2 i − 1) k
+ (cid:17) =
σ (cid:16) ˜ g
(2 i )
k +1 − σ (cid:16) ˜ g
(2 i − 1) k +1 (cid:17)(cid:17) produces at most two hyperplanes in the S .
Therefore, in total, the w k +1 neurons in the k + 1 layer produces (1 + 2) · w k +1= 3 w k +1hyperplanes in S .
Then by Zaslavsky’s Theorem, it will produce at most (cid:80) nj =0 (cid:0)
w k +1 +
j (cid:1) linear regions in S .
Thus f has at most (cid:81) ki =1 (cid:80) nj =0 (cid:0)
wi 2 +1 j (cid:1) linear regions.
B S UPPLEMENTARY R ESULTS FOR THE T IGHTNESS OF T HEOREM
Proposition 4 (The bound (cid:81) ki
=1 ( w i + 1) is tight for a depth-bounded but width-unbounded network) . Given an R → R two-hidden-layer ReLU network, for any width w 1 ≥ 3 , w 2 ≥ 2 in the first and second hidden layers, there exists a PWL function represented by such a network, whose number of pieces is ( w 1 + 1) ( w 2 + 1) .
Breakpoints =-
Breakpoints =-      -0. Breakpoints = Proof.
To guarantee the bound (cid:81) ki =1 ( w i + 1) is tight, the following two requirements should be met: (i) each g ( j ) i , i = 0 , 1 , 2 , j = 1 , . . .
, w i , has distinct zero points that are as much as its number of pieces, so that the activation step can produce the most new breakpoints; (ii) the breakpoints of each g ( j ) ( i +1) , i = 0 , 1 , 2 , j = 1 , . .
, w i +1 , as the affine combination of (cid:110) f
(1) i , . . . , f ( w i ) i (cid:111) , contains all the breakpoints of (cid:110) g (1) i , . . , g ( w i ) i (cid:111) , so that all the old breakpoints are reserved.
Now we give the proof in detail.
Let f (1)1 ( x ) =
σ (3 x ) ,
f (2)1 ( x ) = σ ( − x + 3) , f (3)1 ( x ) = σ (cid:0) 3 2 x − 3 2 (cid:1) .
When w 1 = 3 , we set
[Equation]
When w 1 > 3 , we let f ( j )= σ ( − 2 x − 2( j − 3)) and
[Equation]
Then g ( j )has w 1 + 1 distinct zero points.
Hence for j = 1 , . . .
, w 2 , the breakpoints of f ( j )= σ (cid:16) g
( j ) 2 (cid:17) keeps all breakpoints of g ( j ) 2 and yields w 1 +1 new breakpoints. Note that f ( j ) 2 and f ( j ) 2 do not share new breakpoints, and f (1)2 and f (2)2 covers all the breakpoints of (cid:110) g ( j ) 2 (cid:111)
w 2 j =1 . Therefore, the total number of pieces via an affine combination of f (1)2 , . . . , f ( w 2 )is ( w 1 + 1) ( w 2 + 1) pieces.
Proposition 5 (The bound (cid:81) ki
=1 ( w i + 1) is tight for a width-bounded but depth-unbounded network) .
Given an R → R ReLU network with width w for the first layer and 3 for the other layers, for any depth k ≥ 2 , there exists a PWL function represented by such a network, whose number of pieces is ( w + 1) · 4 k − 1 .
Proof.
Let f (1)1 , . . .
f ( w )be the same as in Proposition 4.
Let ˜ g
[Equation]
We set
[Equation]
Now we continue our proof by induction.
Assume we have constructed f (1) i , f (2)
i and f (3) i , i ≥ 2 .
Then we set ˜ g i
[Equation]
and
[Equation]
Through a direct calculus, we know ˜
g
i +1 has ( w + 1) · 4 i − 1 pieces with opposite slope in every two adjoint pieces and ranges from 0 to 3 / 6 i in each piece except the leftmost and rightmost piece, which implied we can totally obtain ( w + 1) · 4 k − 1 pieces. C P ROOF OF P ROPOSITION Proof.
For the first assertion, we claim that each pre-activation g ( j ) i , 2 ≤ i ≤ k , j = 1 , 2 , cannot have its every two adjacent pieces of slope with different sign, which implies the activation cannot produce the most breakpoints as in Lemma 3.
In fact, g ( j ) 2 , j = 1 , 2 , has at most 3 pieces.
If some g ( j ) 2 has 3 pieces, then by exhaustion , we know either it has a 0 -slope, or it has two adjacent pieces with slopes of the same sign (see Figure 5).
Hence, f ( j ) 2 , j = 1 , 2 , has at most 2 newly produced breakpoints.
Then the output of the 2 -nd layer has at most 2 + 2 × 2 = 6 breakpoints, i.e., 7 pieces.
Applying the similar method to each piece, we can get our result via a simple induction step.
[Figure]
[Figure]
) (c) (e) (f) (g) (h) (i)
Now we come to the second assertion.
For convenience, we say an R → R PWL function f is of ”triangle-trapezoid-triangle” shape on [ a, b ] ⊂
R , if there exists a partition of [ a, b ] : a < x 1 < x 2 < · · · < x 6 < b and a positive constant c , such that
[Equation]
linear connection , otherwise.
Given a PWL function f : R → R of ”triangle-trapezoid-triangle” shape on [ a, b ] , with corresponding partition a < x 1 < x 2 < · · · < x 6 < b and f ( a ) =
c > 0 , if we set g (
[Equation]
then g = − 14 f (1) + f (2) + c 8 is of ”triangle-trapezoid-triangle” shape on [ a, x 2 ] , [ x 2 , x 5 ] and [ x 5 , b ] respectively.
Using this fact, we can construct a PWL function represented by a ( k + 1) -layer 2 -wide intra-linked
ReLU DNN, which has 7 · 3 k − 2 + 2 pieces.
Actually, set
[Equation]
+
[Equation]
˜
[Equation]
+ 32 , then through a direct calculus, 14 ˜ f
(1)2 + ˜ f
(2)2 − 38 is of ”triangle-trapezoid-triangle” shape on [ − 1 , 1] .
Using the fact above repeatedly, we can construct a PWL function represented by a R → R ( k + 1) - layer, 2 -wide, intra-linked ReLU DNN, which is constant on ( −∞ , − 1] ∪ [1 , ∞ ) and of ”triangletrapezoid-triangle” shape on (cid:104) − 1 + 2 n 3 k − 2 , − 1 + 2( n +1) 3 k − 2 (cid:105) , n = 0 , . .
, 3 k − 2 − 1 .