Neural Network Architecture Beyond Width and Depth
Abstract
This paper proposes a new neural network architecture by introducing an additional dimension called height beyond width and depth.
Neural network architectures with height, width, and depth as hyper-parameters are called three-dimensional architectures.
It is shown that neural networks with three-dimensional architectures are significantly more expressive than the ones with two-dimensional architectures (those with only width and depth as hyper-parameters), e.g., standard fully connected networks.
The new network architecture is constructed recursively via a nested structure, and hence we call a network with the new architecture nested network (NestNet).
A NestNet of height s is built with each hidden neuron activated by a NestNet of height ≤ s − 1 .
When s = 1 , a NestNet degenerates to a standard network with a two-dimensional architecture.
It is proved by construction that height- s ReLU NestNets with
O( n ) parameters can approximate 1 -Lipschitz continuous functions on [ 0 ,] d with an error O( n −( s + 1 )/ d ) , while the optimal approximation error of standard ReLU networks with O( n ) parameters is
O( n − 2 / d ) .
Furthermore, such a result is extended to generic continuous functions on [ 0 ,] d with the approximation error characterized by the modulus of continuity.
Finally, we use numerical experimentation to show the advantages of the super-approximation power of ReLU NestNets.
Introduction
In this paper we design a new neural network architecture by introducing one more dimension, called height, in addition to width and depth in the characterization of dimensions of neural networks.
We call neural network architectures with height, width and depth as hyper-parameters three-dimensional architectures.
It is proved by construction that neural networks with three-dimensional architectures improve the approximation power significantly, compared to standard networks with two-dimensional architectures (those with only width and depth as hyper-parameters).
The approximation power of standard neural networks has been widely studied in recent years.
The optimality of the approximation of standard fully-connected rectified linear unit (ReLU) networks (e.g., see [35, 40, 49, 52]) implies limited room for further improvements.
This motivates us to design a new neural network architecture by introducing an additional dimension of height beyond width and depth.
We will focus on the ReLU ( max { 0 ,x } ) activation function and use it to demonstrate our ideas.
Our new network architecture is constructed recursively via a nested structure, and hence we call a neural network with the new architecture nested network ( NestNet ).
A NestNet of height s is built with each hidden neuron activated by a NestNet of height ≤ s − 1 .
In the case of s = 1 , a NestNet degenerates to a standard network with a two-dimensional architecture.
Let us use a simple example to explain the height of a NestNet.
We say a network is activated by ϱ 1 , ⋯ ,ϱ r if each hidden neuron of this network is activated by one of ϱ 1 , ⋯ ,ϱ r .
Here, ϱ 1 , ⋯ ,ϱ r are trainable functions mapping R to R .
Then, a network of height s ≥ 2 can be regarded as a ( ϱ 1 , ⋯ ,ϱ r ) -activated network, where ϱ 1 , ⋯ ,ϱ r are (realized by) networks of height ≤ s − 1 .
See an example of a height- 2 network in Figure 1.
The network therein can be regarded as a ( ϱ 1 ,ϱ) -activated network, where ϱ 1 and ϱ 2 are (realized by) networks of height 1 (i.e., standard networks).
The number of parameters in the network of Figureis the sum of the numbers of parameters in L 0 , L 1 , L 2 and ϱ 1 ,ϱ 2 . [Figure] L
[Figure]
L L [Figure] We remark that a NestNet can be regarded as a sufficiently large standard network by expanding all of its sub-network activation functions.
We propose the nested network architecture since it shares the parameters via repetitions of sub-network activation functions.
In other words, a NestNet can provide a special parameter-sharing scheme.
This is the key reason why the NestNet has much better approximation power than the standard network.
If we regard the network in Figure 1 as a NestNet of height 2 , then the number of parameters is the sum of the numbers of parameters in L 0 , L 1 , L 2 and ϱ 1 ,ϱ 2 .
However, if we expand the network in Figure 1 to a large standard network, then the number of parameters in ϱ 1 and ϱ 2 will be added many times for computing the total number of parameters.
Next, let us discuss our new network architecture from the perspective of hyper-parameters.
We call the network architecture with only width as a hyper-parameter one-dimensional architecture.
Its depth and height are both equal to one.
Neural networks with this type of architecture are generally called shallow networks.
See an example in Figure 2(a).
We call the network architecture with only width and depth as hyper-parameters two-dimensional architecture.
Its height is equal to one.
Neural networks with this type of architecture are generally called deep networks.
See an example in Figure 2(b).
We call the network architecture with height, width, and depth as hyper-parameters three-dimensional architecture, which is proposed in this paper.
Neural networks with this type of architecture are called NestNets.
See an example in Figure 2(c).
One may refer to Table 1 for the approximation power of networks with these three types of architectures discussed above.
[Table]
nested composition this paper
Our main contributions are summarized as follows.
We first propose a three-dimensional neural network architecture by introducing one more dimension called height beyond width and depth.
We show that neural networks with three-dimensional architectures are significantly more expressive than standard networks.
In particular, we prove that height- s ReLU NestNets with
O( n ) parameters can approximate 1 -Lipschitz continuous functions on [ 0 ,] d with an error O( n −( s + 1 )/ d ) , which is much better than the optimal error O( n − 2 / d ) of standard ReLU networks with O( n ) parameters.
In [Figure] the case of s + ≥ d , the approximation error is bounded by O( n −( s + 1 )/ d ) ≤ O( n −) , which means we overcome the curse of dimensionality.
Furthermore, we extend our result to generic continuous functions with the approximation error characterized by the modulus of continuity.
See Theorem 2. and Corollary 2.2 for more details.
Finally, we conduct simple experiments to show the numerical advantages of the super-approximation power of ReLU NestNets.
The rest of this paper is organized as follows.
In Section 2, we present the main results, provide the ideas of proving them, and discuss related work.
The detailed proofs of the main results are placed in the appendix.
Next, we conduct experiments to show the advantages of the super-approximation power of ReLU NestNets in Section 3.
Finally, Section 4 concludes this paper with a short discussion.
Main results and related work
In this section, we first present our main results and discuss the proof ideas.
The detailed proofs of the main results are placed in the appendix.
Next, we discuss related work from multiple perspectives.
2.1 Main results
We use NN s { n } for n,s ∈ N to denote the set of functions realized by height- s ReLU NestNets with as most n parameters.
We will give the mathematical definition of NN s { n } .
We first discuss some notations regarding affine linear maps.
We use L to denote the set of all affine linear maps, i.e.,
L ∶= {
L ∶ L ( x ) = W x + b , W ∈ R d 2 × d 1 , b ∈
R d 2 , d 1 ,d∈
N + } .
Let # L denote the number of parameters in L ∈ L , i.e., # L = ( d+ ) dif L ( x ) =
W x + b for W ∈ R d 2 × d 1 and b ∈ R d 2 .
We use ⃗ g =
( ϱ 1 , ⋯ ,ϱ k ) to denote an activation function vector, where ϱ i ∶ R → R is an activation function for each i ∈ { 1 , ⋯ ,k } .
When ⃗ g = ( ϱ 1 , ⋯ ,ϱ k ) is applied to a vector input x = ( x 1 , ⋯ ,x k ) , ⃗ g ( x ) =
( ϱ( x) , ⋯ , ϱ k ( x k )) for any x = ( x 1 , ⋯ ,x k ) ∈ R k .
Let set (⃗ g ) denote the function set containing all entries (functions) in ⃗ g .
For example, if ⃗ g = ( ϱ 1 ,ϱ 2 ,ϱ 3 ,ϱ 2 ,ϱ) , then set (⃗ g ) = { ϱ 1 ,ϱ 2 ,ϱ} .
To define NN s { n } for n,s ∈
N recursively, we first consider the degenerate case. Define NN
[Equation]
where id R ∶ R → R is the identity map.
That is, we regard the identity map and ReLU as heightReLU NestNets with n parameters or as height- s ReLU NestNets with 0 parameters.
Next, let us present the recursive step.
For n,s ∈ N [Equation] , a (vector-valued) function ϕ ∈ NN s { n } has the following form:
[Equation]
where L 0 , ⋯ , L m ∈ L are affine linear maps.
Moreover, Equation (1) satisfies the following two conditions:
• Condition on activation functions: m ⋃
[Equation]
= 1 set (⃗ g
i
[Equation]
{ ϱ
⋯ [Equation]
ϱ
r and
ϱ j
∈ s −
[Equation]
[Equation] [Equation] [Equation] Here, ⃗ g i is an activation function vector for each i ∈ { 1 , ⋯ ,m } .
All entries in ⃗ g 1 , ⋯ , ⃗ g m form an activation function set { ϱ 1 , ⋯ ,ϱ r } . For each j ∈ { 1 , ⋯ ,r } , ϱ j can be realized by a height- i NestNet with ≤ n j parameters for some i
= i j [Equation] s − 1 .
This condition means each hidden neuron is activated by a NestNet of lower height.
• Condition on the number of parameters:
[Equation]
This condition means the total number of parameters is no more than n .
The total number of parameters is calculated by adding two parts.
The first one is the number of parameters in affine linear maps L 0 , ⋯ , L m .
The other part is the number of parameters in the activation set { ϱ 1 , ⋯ ,ϱ r } formed by the entries in activation function vectors ⃗
[Equation]
Then, with two conditions in Equations (2) and (3), we can define NN s { n } for n,s ∈ N + as follows: NN
[Equation]
for j = 1 , ⋯ [Equation]
r
[Equation]
m ∑
[Equation]
We remark that, in the definition above, m can be equal to 0 . In this case, the function ϕ degenerates to an affine linear map. In the NestNet example in Figure 1, the function ϕ therein is in ⋃ n ∈ N
NN { n } and the activation function vectors ⃗ g 1 and ⃗ g 2 can be represented as ⃗ g= ( ϱ 1 , ϱ 2 , ϱ 1 , ϱ) and ⃗ g
[Equation]
Moreover, the activation function set containing all entries in ⃗ g 1 and ⃗ g 2 is a subset of ⋃ n ∈ N NN { n } , i.e., { ϱ 1 ,ϱ} ⊆
⋃ n ∈ N NN { n } .
Let C ([ 0 ,] d ) denote the set of continuous functions on [ 0 ,] d .
By convention, the modulus of continuity of a continuous function f ∈ C ([ 0 ,] d ) is defined as ω f ( r ) ∶= sup {∣ f ( x ) − f ( y )
∣ ∶ ∥ x − y ∥ ≤ r, x , y ∈ [ 0 ,] d } for any r ≥ 0 .
Under these settings, we can find a function in NN s {O( n )} to approximate f ∈ C ([ 0 ,] d ) with an approximation error O( ω f ( n −( s + 1 )/ d )) , as shown in the main theorem below.
Theorem 2.1.
Given a continuous function f ∈ C ([ 0 ,] d ) , for any n,s ∈ N + and p ∈ [ 1 , ∞] , there exists ϕ ∈ NN s { C s,d ( n + )} such that ∥ ϕ ( x ) − f ( x )∥
L p ([ 0 , 1 ] d ) ≤ √ dω f ( n −( s + 1 )/ d ) , where C s,d = 10 3 d( s + ) 2 if p ∈ [ 1 , ∞) and C s,d = 10 d + 3 d( s + ) 2 if p = ∞ .
We remark that the constant C s,d in Theorem 2.1 is valid for all n ∈ N + .
As we shall see later, C s,d can be greatly reduced if one only cares about large n ∈ N + .
Generally, it is challenging to simplify the approximation error in Theorem 2.1 to make it explicitly depend on n due to the complexity of ω f (⋅) .
However, the approximation error can be simplified to an explicit one depending on n in the case of special target function spaces like Hölder continuous function space.
To be exact, if f is a Hölder continuous function on [ 0 ,] d of order α ∈ ( 0 ,] with a Hölder constant λ > 0 , then ∣ f ( x ) − f ( y )∣ ≤ λ ∥ x − y ∥ αfor any x , y ∈ [ 0 ,] d , implying ω f ( r ) ≤ λr α for any r ≥ 0 .
This means we can get an exponentially small approximation error 7 λ √ dn −( s + 1 ) α / d as shown in Corollary 2.2 below.
Corollary 2.2.
Suppose f is a Hölder continuous function on [ 0 ,] d of order α ∈ ( 0 ,] with a
Hölder constant λ > 0 .
For any n,s ∈ N + and p ∈ [ 1 , ∞] , there exists ϕ ∈ NN s { C s,d ( n + )} such that
[Equation]
where C s,d = 10 3 d( s + ) 2 if p ∈ [ 1 , ∞) and C s,d = 10 d + 3 d ( s + ) 2 if p = ∞ .
In Corollary 2.2, if α = 1 , i.e., f is a Lipschitz continuous function with a Lipschitz constant λ > 0 , then the approximation error can be further simplified to 7 λ √ dn −( s + 1 )/ d .
See Table 1 for the comparison of the approximation error of 1 -Lipschitz continuous functions on [ 0 ,] d approximated by ReLU NestNets and standard ReLU networks. 2.2 Sketch of proving Theorem 2.  We will discuss how to prove Theorem 2.1.
Given a target function f ∈
C ([ 0 ,] d ) , the key point is to construct an almost piecewise constant function realized by a ReLU NestNet to approximate f well except for a small region.
Then we can get the desired result by dealing with the approximation in this small region.
We divide the sketch of proving Theorem 2.1 into three main steps. First, we divide [ 0 ,] d into a union of cubes { Q β } β ∈{ 0 , 1 , ⋯ ,K − 1 } d and a small region Ω with K = O( n ( s + 1 )/ d ) .
Each Q β is associated with a representative x β ∈
Q β for each vector index β .
See Figure 3 for an illustration for K = 4 and d = 2 .
Next, we design a vector-valued function Φ( x ) to map the whole cube Q β to its index β for each β .
Here, Φ 1 can be defined/constructed via Φ
[Equation]
where each one-dimensional function ϕ 1 is a step function outside a small region.
We can efficiently construct ReLU NestNets with the desired size to approximate such an almost step function ϕ 1 with sufficiently many “steps” by using the composition architecture of ReLU
NestNets.
See the appendix for the detailed construction.
Finally, we need to construct a function ϕ 2 realized by a ReLU NestNet to map β approximately to f ( x β ) for each β ∈ { 0 , 1 , ⋯ ,K − } d .
Then we have ϕ○
Φ( x ) =
ϕ( β ) ≈ f ( x β ) ≈ f
( x ) for any x ∈
Q β and each β , implying ϕ ∶= ϕ
○ Φ≈ f on [ 0 ,] d / Ω .
Then, we can get a good approximation on [ 0 ,] d by using Lemma 3 .
4 of our previous paper [24] to deal with the approximation inside Ω .
We remark that, in the construction of ϕ∶
R d →
R , we only need to care about the values of ϕ 2 at a set of K d points { 0 , 1 , ⋯ ,K − } d .
As we shall see later, this is the key point to ease the design of a ReLU NestNet with the desired size to realize ϕ 2 .
See Figure 3 for an illustration of the above steps.
Observe that in Figure 3, we have ϕ ( x ) = ϕ
[Equation]
○ Φ
[Equation]
( x ) =
[Equation]
( β )
[Equation]
f (
[Equation]
[Equation]
≈ f ( x ) for any x ∈
Q β and each β ∈ { 0 , 1 , ⋯ ,K − } d .
That means ϕ − f is bounded by E+
E 2 on [ 0 ,] d / Ω .
For any x ∈
Q β and each β , we have ∥ x β − x ∥ ≤ √ d /
K (cid:212)⇒ ∣ f ( x β ) − f ( x )∣ ≤ ω f (√ d /
K ) (cid:212)⇒
E≤ ω f (√ d /
K ) .
Q 0 ,
Q 1 ,
Q 2 ,
Q 3 ,
Q 0 ,
Q 1 ,
Q 2 ,
Q 3 ,
Q 0 ,
Q 1 ,
Q 2 ,
Q 3 ,
Q 0 ,
Q 1 ,
Q 2 ,
Q 3 ,x β Ω
Q β Φ( x ) = β for x ∈
Q β
A set of d -dimensional indices: β ∈ { 0 , 1 , ⋯ ,K − } d ϕ( β ) ≈ f ( x β ) A set of function values at representatives: { f ( x β ) ∶ β ∈ { 0 , 1 , ⋯ ,K − } d
) =
ϕ
○ Φ( x ) =
ϕ( β ) ≈ f ( x β ) ≈
f ( x )
∈
{ 0 , 1 , ⋯ ,K − } d .
The upper bound of E 1 is determined by the construction of ϕ∶
R d →
R .
As stated previously, we only need to care about the values of ϕ 2 at a set of K d points { 0 , 1 , ⋯ ,K − } d ⊆
R d , which gives us much freedom to control E 1 .
As we shall see later, E 1 can be bounded by
O( ω f (√ d /
K )) .
Therefore, ϕ − f is controlled by
O( ω f (√ d /
K )) outside Ω , from which we deduce the desired approximation error on [ 0 ,] d / Ω since K = O( n −( s + 1 )/ d ) .
Finally, by using Lemma 3 . 4 of our previous paper [24] to deal with the approximation inside Ω , we can get the desired approximation error on [ 0 ,] d .
2.3 Related work
We first compare our results with existing ones from an approximation perspective.
Next, we discuss the parameter-sharing schemes of neural networks. Finally, we connect our NestNet architecture to existing trainable activation functions.
Discussion from an approximation perspective
The study of the approximation power of deep neural networks has become an active topic in recent years.
This topic has been extensively studied from many perspectives, e.g., in terms of combinatorics [27], topology [7], information theory [29], fat-shattering dimension [1, 21], Vapnik-Chervonenkis (VC) dimension [6, 14, 32], classical approximation theory [3, 4, 8, 9, 10, 11, 12, 13, 18, 22, 24, 25, 28, 34, 35, 38, 39, 42, 48, 49, 52, 53], etc.
To the best of our knowledge, the study of neural network approximation has two main stages: shallow (one-hidden-layer) networks and deep networks.
In the early works of neural network approximation, the approximation power of shallow networks is investigated.
In particular, the universal approximation theorem [11, 17, 18], without approximation error estimate, showed that a sufficiently large neural network can approximate a target function in a certain function space arbitrarily well.
For one-hidden-layer neural networks of width n and sufficiently smooth functions, an asymptotic approximation error O( n − 1 /) in the L 2 -norm is proved in [4, 5], leveraging an idea that is similar to Monte Carlo sampling for high-dimensional integrals.
Recently, a large number of works focus on the study of deep neural networks.
It is shown in [35, 49, 52] that the optimal approximation error is [Equation] n − 2 / d ) by using ReLU networks with n parameters to approximate 1 -Lipschitz continuous functions on [ 0 ,] d .
This optimal approximation error follows a natural question: How can we get a better approximation error?
Generally, there are two ideas to get better errors.
The first one is to consider smaller function spaces, e.g., smooth functions [24, 50] and band-limited functions [26].
The other one is to introduce new networks, e.g., Floor-ReLU networks [36], Floor-Exponential-Step (FLES) networks [37], and (Sin, ReLU, 2 x )-activated networks [20].
This paper proposes a three-dimensional neural network architecture by introducing one more dimension called height beyond width and depth.
As shown in Theorem 2.1 and Corollary 2.2, neural networks with three-dimensional architectures are significantly more expressive than the ones with two-dimensional architectures.
We will conduct experiments to explore the numerical properties of NestNets in Section 3. Discussion from a parameter-sharing perspective As discussed previously, our NestNet architecture can be regarded as a sufficiently large standard network architecture with a specific parameter-sharing scheme.
Parameter-sharing schemes are used in neural networks to control the overall number of parameters for reducing memory and communication costs. There are two common parameter-sharing schemes for a neural network. The first scheme is to share parameters in the same layer. A typical network example with this scheme is the convolutional neural network (CNN).
In CNN architectures, filters in a CNN layer are shared for all channels, which means the parameters in the filters are shared. The second scheme is to share parameters across different layers of networks, e.g., recurrent neural networks. In the NestNet architecture, we share parameters via repetitions of sub-network activation functions. Both of parameter-sharing schemes discussed just above are used in the NestNet architecture.
The nested architecture of NestNets gives us much freedom to determine how many parameters to share. Beyond parameter-sharing schemes for a neural network, there are also parameter-sharing schemes among different neural networks or models, especially for multi-task Learning. One may refer to [30, 33, 44, 45, 46, 51] for more discussion on parameter sharing in neural networks.
Connection to trainable activation functions
The key idea of trainable activation functions is to add a small number of trainable parameters to existing activation functions.
Let us present several existing trainable activation functions as follows.
A ReLU-like function is introduced in [15] by modifying the negative part of ReLU using a trainable parameter α , i.e., the parametric ReLU (PReLU) is defined as PReLU ( x ) ∶= { x if
[Equation]
αx if x < 0 . A variant of ELU unit is introduced in [43] by adding two trainable parameters β,γ > 0 , i.e., the parametric
ELU (PELU) is given by PELU [Equation] [Equation] β ( exp
[Equation]
x /
[Equation] [Equation] 1 ) x if x < 0 . Authors in [31] propose a type of Flexible ReLU (FReLU), which is defined via FReLU
[Equation] =
ReLU ( x + α ) + β, where α and β are two trainable parameters.
One may refer to [2] for a survey of modern trainable activation functions.
To the best of our knowledge, most existing trainable activation functions can be regarded as parametric variants of the original activation functions.
That is, they are attained via parameterizing the original activation functions with a small number of (typically 1 or 2 ) trainable parameters.
By contrast, activation functions in our NestNets are much more flexible.
They can be (realized by) either complicated or simple sub-NestNets.
That is, we can freely determine the number of parameters in the activation functions of NestNets.
In other words, in NestNets, we can randomly distribute the parameters in the affine linear maps and activation functions.
In short, compared to the networks with existing trainable activation functions, our NestNets are more flexible and have much more freedom in the choice of activation functions.
Experimentation
In this section, we will conduct experiments as a proof of concept to explore the numerical properties of ReLU NestNets. It is challenging to tune the hyper-parameters of large NestNets due to their nested architectures. Thus, our experimentation focuses on relatively small NestNets of height 2 and we introduce a simple sub-network activation function ϱ , which is realized by a trainable one-hidden-layer ReLU network of width 3 . To be exact, ϱ is given by
[Equation]
where w 0 , w 1 , b∈
R 3 and b∈ R are trainable parameters. There are 10 parameters in ϱ . The initial settings for ϱ in our experiments are w= ( 1 , 1 ,) , w= ( 1 , 1 , − )
[Equation]
b
[Equation]
= (− 0 . [Equation] − 0 . , and b= 0 .
We believe that NestNets can achieve good results in some real-world applications if proper optimization algorithms are developed for NestNets.
In this paper, we only consider two classification problems: a synthetic classification problem based on the Archimedean spiral in Section 3.1 and an image classification problem corresponding to a standard benchmark dataset Fashion-MNIST [47] in Section 3.2. We remark that a classification function can be continuously extended to R d if each class of samples are located in a bounded closed subset of R d and these subsets are pairwise disjoint. That means we can apply our theory to classification problems. 3.1 Archimedean spiral
We will design a binary classification experiment by constructing two disjoint sets based on the
Archimedean spiral, which can be described by the equation r = a + bθ in polar coordinates ( r,θ ) for given a,b ∈
R .
Let us first define two curves (Archimedean spirals) as follows: ̃C i ∶= {( x,y )
∶ x = r i cos θ, y = r i sin θ, r i =
a
i + b i θ, θ ∈ [ 0 ,sπ ]} , for i = 0 , 1 , where a= 0 , a= 1 , b= b= / π , and s = 30 .
To simplify the discussion below, we normalize ̃C i as
C i ⊆ [ 0 ,] 2 , where C i is defined by C i ∶= {( x,y )
∶ x = ̃ x 2 ( s + 2 ) + 12 , y = ̃ y 2 ( s + 2 ) + 12 , (̃ x, ̃ y ) ∈ ̃C
i } , for i = 0 , 1 .
Then, we can define the two desired sets as follows:
S i ∶= {( u,v ) ∶ √( u − x ) + ( v − y ) ≤ ε, ( x
,y ) ∈ C i } , for i = 0 , 1 , where ε = 0 .
005 in our experiments.
See an illustration for S 0 and S 1 in Figure 4.
[Figure]
[Figure]
To explore the numerical performance of NestNets, we design NestNets and standard networks to classify samples in
S ⋃S 1 .
We adopt four-hidden-layer fully connected network architecture of width 20 , 35 , or 50 .
To make the optimization more stable, we add the layers of batch normalization [19].
See Figure 5 for an illustration of the full network architecture.
In Figure 5, FC and ActFun are short of fully connected layer and activation function, respectively.
ActFun is ReLU for standard networks, while for NestNets, ActFun is the learnable sub-network activation function ϱ given in
Equation (4).
Before presenting the experiment results, let us present the hyper-parameters for training the networks mentioned above.
For each i ∈ { 0 ,} , we randomly choose× 10 5 training samples and× 10 4 test samples in
S i with label i .
Then, we use these× 10 5 training samples to train the networks and use these× 10 4 test samples to compute the test accuracy.
We use the cross-entropy loss function to evaluate the loss between the networks and the target classification function.
The number of epochs and the batch size are set to 500 and 512 , respectively.
We adopt RAdam [23] as the optimization method.
In epochs( i − ) + 1 to 5 i for i = 1 , 2 , ⋯ , 100 , the learning rate is 0 {+.× 
+}
.×
9 i − 1 for the parameters in ϱ and 0 .× 0 . 9 i − 1 for all other parameters.
We remark that all training (test) samples are standardized before training, i.e., we rescale the samples to have a mean of 0 and a standard deviation of 1 .
Finally, let us present the experiment results to compare the numerical performances of NestNets and standard networks.
We adopt the average of test accuracies in the last 100 epochs as the target test accuracy.
As we can see from Table 2 and Figure 6, by adding 10 more parameters (stored in ϱ ), NestNets achieve much better test accuracies than standard networks though slightly more training time is required.
In an “unfair” comparison, the test accuracy attained by the NestNet with 1 .×
10parameters is still better than that of the standard network with 7 .× 10 3 parameters.
This numerically verifies that the NestNet has much better approximation power than the standard network. 3.2 Fashion-MNIST
We will design convolutional neural network (CNN) architectures activated by ReLU or the subnetwork activation function ϱ given in Equation (4) to classify image samples in Fashion-MNIST [47].
[Table]
accuracy
[Figure]
ReLU ≈ 2532 s 0.
[Figure]
+ sub-network ( ϱ ) ≈ 4016 s 0.
[Figure]
ReLU ≈ 2595 s 0. [Figure] + sub-network ( ϱ ) ≈ 4104 s 0. [Figure]
ReLU ≈ 2642 s 0. [Figure] ( ϱ ) [Figure]
[Figure]
This dataset consists of a training set of× 10 4 samples and a test set of 10 4 samples. Each sample is a× 28 grayscale image, associated with a label from 10 classes.
To compare the numerical performances of NestNets and standard networks, we design a standard CNN architecture and a
NestNet architecture that is constructed by replacing a few activation functions of a standard CNN network by the sub-network activation function ϱ . For simplicity, we denote the standard CNN and the NestNet as CNN1 and CNN2. To make the optimization more stable, we add the layers of dropout [16, 41] and batch normalization [19].
See illustrations of CNN1 and CNN2 in Figure 7. We present more details of them in Table 3. [Figure] [Figure]
Before presenting the numerical results [Figure] let us present the hyper-parameters for training two CNN architectures above.
We use the cross-entropy loss function to evaluate the loss between the CNNs and the target classification function. The number of epochs and the batch size are set to 500 and 128 , respectively. We adopt RAdam [23] as the optimization method and the weight decay of the optimizer is 0 . 0001 . In epochs( i − )
{++ 1 to 5 i for i = 1 , 2 , ⋯ , 100 , the learning rate is 0 .× 
 .× 9 i − for the parameters in ϱ and 0 .× 0 . 9 i − 1 for all other parameters.+}
All training (test) samples in the Fashion-MNIST dataset are standardized in our experiment, i.e., we rescale all training (test) samples to have a mean of 0 and a standard deviation of 1 . In the settings above, we repeat the experiment 18 times and discard 3 top-performing and 3 bottom-performing trials by using the average of test accuracy in the last 100 epochs as the performance criterion. For each epoch, we adopt the average of test accuracies in the rest 12 trials as the target test accuracy.
Next, let us present the experiment results to compare the numerical performances of CNN1 and
CNN2.
The test accuracy comparison of CNN1 and CNN2 is summarized in Table 4.
[Table]
For each of CNN1 and CNN2, we present the training time, the largest test accuracy, the average of the largest 100 test accuracies, and the average of test accuracies in the last 100 epochs.
For an intuitive comparison, we also provide illustrations of the test accuracy over epochs for CNN1 and CNN2 in Figure 8. As we can see from Table 4 and Figure 8, CNN2 performs better than CNN though slightly more training time and 10 more parameters are required.
This numerically shows that the NestNet is significantly more expressive than the standard network. [Figure] over epochs.
Conclusion
This paper proposes a three-dimensional neural network architecture by introducing one more dimension called height beyond width and depth.
We show by construction that neural networks with three-dimensional architectures are significantly more expressive than the ones with two-dimensional architectures.
We use simple numerical examples to show the advantages of the super-approximation power of ReLU NestNets, which is regarded as a proof of possibility.
It would be of great interest to further explore the numerical performance of NestNets to bridge our theoretical results to applications.
We believe that NestNets can be further developed and applied to real-world applications.
We remark that our analysis is limited to the ReLU activation function and the (Hölder) continuous function space.
It would be interesting to generalize our results to other activation functions (e.g., tanh and sigmoid functions) and other function spaces (e.g, Lebesgue and Sobolev spaces).
Acknowledgments
Z. Shen was supported by Distinguished Professorship of National University of Singapore.
H. Yang was partially supported by the US National Science Foundation under award DMS-2244988, DMS2206333, and the Office of Naval Research Award N00014-23-1-2007.