E XPRESSIVENESS OF N EURAL N ETWORKS H AVING W
IDTH E QUAL OR B ELOW
THE I NPUT D IMENSION
Anonymous authors
Paper under double-blind review
A BSTRACT
The understanding about the minimum width of deep neural networks needed to ensure universal approximation for different activation functions has progressively been extended (Park et al., 2021).
In particular, with respect to approximation on general compact sets in the input space, a network width less than or equal to the input dimension excludes universal approximation.
In this work, we focus on network functions of width less than or equal to the latter critical bound.
We prove a maximum principle from which we conclude that for all continuous and monotonic activation functions, universal approximation of arbitrary continuous functions is impossible on sets that coincide with the boundary of an open set plus an inner point.
Conversely, we prove that in this regime, the exact ﬁt of partially constant functions on disjoint compact sets is still possible for ReLU network functions under some conditions on the mutual location of these components.
I NTRODUCTION
In the course of the increasing popularity of deep neural networks for applications in technical, ecological and many other ﬁelds, there has been huge progress in the research on understanding the mathematical properties of the mathematical mapping implemented by a deep neural network.
The approximation properties, or expressiveness, of neural network functions have attracted intense interest.
The central result in this ﬁeld is the classic universal approximation theorem , which states that any continuous function can be approximated with arbitrary accuracy (in terms of uniform approximation or L p norms) by neural network functions that have only one hidden layer for nearly every activation function (Cybenko, 1989; Hornik, 1991).
In recent years, these kind of results have become more precise, for instance, in terms of estimates on the order of magnitude of parameters needed to achieve an approximation of a certain accuracy (Petersen & Voigtlaender, 2018;
Yarotsky, 2017; 2018), or by investigating the number off piece-wise linear regions (Montufar et al., 2014;
Serra et al., 2018; Hanin & Rolnick, 2019); we refer to (DeVore et al., 2020) for an overview on recent developments.
Since empirical observations showed that depth has a signiﬁcant impact on the performance of neural networks, a lot of research has been dedicated to the effect of depth on the expressive power of neural networks (Telgarsky, 2016; Mhaskar & Poggio, 2016; Montufar et al., 2014;
Raghu et al., 2017; Rolnick & Tegmark, 2018; Lin et al., 2017; Cohen et al., 2016).
Besides this, the role of width in expressiveness of network functions has been investigated (Lu et al., 2017; Hanin & Sellke, 2017; Hanin, 2019; Johnson, 2018; Kidger & Lyons, 2020; Park et al., 2021).
An important result states that a width larger than the input dimension is needed to allow universal approximation on arbitrary compact sets.
It has also been shown that the capability to learn disconnected or bounded decision regions depends on whether the width is larger than the input dimension (Nguyen et al., 2018; Beise et al., 2021).
In a further line of research, the capabilities of neural networks to memorize ﬁnite sample sets is investigated (Yun et al., 2019; Vershynin, 2020;
Bubeck et al., 2020; Park et al., 2020), which provides mathematical foundations to the surprising ﬁndings that neural networks can interpolate the training data (Zhang et al., 2017) in practise.
In this work, we investigate what kind of subsets M ⊂ R n admit universal approximation by neural network functions with a maximum layer width less than or equal to the input dimension, i.e. network functions that violate the necessary condition for universal approximation given in Hanin (2019) and Park et al. (2021) for arbitrary compact sets.
Let us introduce the following notation: For a set D ⊂ R n we denote by D ◦ the set of interior points, by D its closure and by ∂D
=
D \ D ◦ the boundary of D .
By (cid:107)·(cid:107) we mean the Euclidean norm.
For a linear subspace U ⊂ R n , we denote by dim( U ) its dimension and by P U : R n → U the orthogonal linear projection on U .
We denote by e j the unit vector of the j -th coordinate axis in R n .
For some v ∈ R n and j ∈ { 1 , ..., n } , we write v ( j ) for the j − th component of v .
For some depth L ∈ N , we consider neural network functions of some , F : R n 0 → R n L where n 0 is called the input dimension and n L the output dimension.
Our network functions have the following form
[Equation]
where A j ( x ) = σ ( W j x + b j ) with W j ∈ R n j × n j − 1 (weights), b j ∈ R n j (bias), where j = 1 , ..., L , and σ : R → R the activation function.
The application of σ and the inverse image σ − 1 are understood to be applied element wise to vectors or subsets of R n .
The widely used activation function rectiﬁed linear unit , shortly ReLU, is deﬁned by t (cid:55)→ max { t, 0 } .
As a shorthand notation of the function implemented by the ﬁrst k layers of a network function, we set
[Equation]
for k ∈ { 1 , ..., L − 1 } .
We call n j the width of layer j = 1 , ..., L .
The width of the network is deﬁned as ω ( F ) = max { n j : j = 1 , ..., L } and L is called the depth of the network.
For m, L ∈ N and an activation function σ , NN kσ ( m, L ) is the set of network functions F : R n 0 → R k of the form (1) with activation σ and of maximum width m , i.e. ω ( F ) ≤ m , and depth L , i.e. L layers including the ﬁnal linear layer.
In case that R k
= R we omit the latter and write NN σ ( m, L ) instead, and in case the the depth is not speciﬁed, meaning that arbitrary depth is allowed, we write NN kσ ( m ) ,
NN σ ( m ) , respectively.
We say that a compact subset M ⊂ R n 0 admits universal uniform approximation of functions in some function class of mappings from M to R by network function of NN σ ( m, L ) or NN σ ( m ) , if for every f : M → R in that class and every ε > 0 , there exists some F ∈ NN σ ( m, L ) , respectively
F ∈ NN σ ( m ) , such that
[Equation]
R ELATED WORK
Approximation properties of width bounded neural networks have been studied in several papers.
A common goal in these lines of research is to provide upper and lower bounds on the minimum width of the network, i.e. the minimum number of neurons per layers, here denoted by ω min , needed for universal approximation in C ( M, R n L ) , i.e. the space of continuous functions from a compact set M to R n L endowed with the norm of uniform convergence, and L p ( M, R n L ) with some set M ⊂ R n 0 , i.e. the space of functions f = ( f 1 , ..., f n L ) from M to R n L such that every | f j | p is integrable over M , endowed with the usual L p norm.
Since in this work we are interested in universal approximation in C ( M, R ) , we refer to Lu et al.
(2017), Kidger & Lyons (2020) and Park et al.
(2021) for the case of L p approximation.
For the case of C ( M, R n L ) , it was proven in Hanin & Sellke (2017) that n
[Equation]
which has been tightened in Park et al.
to
[Equation]
where STEP refers to the threshold activation that maps to 1 on { x ≥ 0 } and 0 otherwise.
It has been shown at the same time in Johnson (2018) and Beise et al.
that
[Equation]
The activation functions allowed in Johnson (2018) are slightly more general, as it is only required that they admit arbitrary accurate uniform approximation by injective continuous functions on arbitrary compact subset of R .
In Kidger & Lyons (2020) the above upper bounds are extended to further classes of activation functions.
For these results and a general overview on recent developments in this ﬁeld, we refer to Park et al.
As we also formulate results for ﬁnite sets, our work partially exhibits relations to the research on ﬁnite sample memorization (Yun et al., 2019; Vershynin, 2020;
Bubeck et al., 2020; Park et al., 2020).
In this work, we only consider width less than or equal to n 0 and investigate uniform approximation on certain compacts subsets of the input space.
According to the above result (6), we cannot expect universal approximation for all compact M ⊂ R n 0 .
However, a common assumption in applications is that approximation is only needed on a certain subset.
We derive some topological conditions on
M that allow or exclude a kind of universal approximation under these circumstances.
Although narrow neural networks, as they are considered in this work, are not common in practical applications, many networks exhibit a decaying layer width at the later layers.
From this perspective, our results give theoretical insights on the kind feature extraction earlier layers need to implement in order to allow the later layers to solve a given machine learning task.
Our main contributions are as follows:
A maximum principle is given for F ∈ NN σ ( n 0 ) with σ continuous and monotonic.
This allows to conclude that the lower bound ω min ≥ n 0 + 1 in (6) is sharp for a wide range of subsets M , e.g. when M = ∂D ∪ { c } for some D with non-empty interior and c ∈ D ◦ .
We show that for the case of two disjoint compact sets, the existence of a cone-like sector that contains one of these sets and does not intersect with the other one, is sufﬁcient to allow exact ﬁt of functions that take constant values on each of these sets by network functions from NN ReLU ( n 0 , 4) .
A weaker result is concluded for the case of multiple pairwise compact components.
M AXIMUM PRINCIPLE
Let us recall that NN σ ( n 0 ) designates the set of network functions with activation function σ , having maximum width n 0 and arbitrary depth.
In this section, we will prove a maximum principle for network functions of NN σ ( n 0 ) for a wide class of activation functions.
This principle can be viewed as a root cause why universal approximation with functions from NN σ ( n 0 ) on arbitrary compact sets is impossible as shown in Johnson (2018) and Beise et al.
(2021), see (6).
This also immediately leads to a topological condition on subsets of R n 0 that do not admit universal approximation by functions in NN σ ( n 0 ) .
Theorem 1 (Maximum Principle) .
Let M be some compact subset of R n 0 and σ a continuous, monotonic activation function.
Then every F ∈ NN σ ( n 0 ) takes its maximum value at the boundary ∂M .
Note that, considering − F instead of F , the latter result also implies that the minimum value is taken on ∂M .
This implies that universal uniform approximation of continuous functions is impossible on compact sets with non-empty interior.
Even more, it can be concluded that:
Corollary 1.
Let M be some compact subset of R n 0 such that ∂D ⊂ M and M ∩ D ◦ (cid:54) = ∅ for some non-empty open set D ⊂ R n .
Then M does not admit universal uniform approximation of continuous function by network functions from NN σ ( n 0 ) .
The latter results naturally lead to the following question : Is it possible to uniformly approximate arbitrary continuous functions on the unit sphere (in R n 0 ) with arbitrary accuracy by network functions as they are considered in Theorem 1?
Some positive approximation results are given in the next section, but a gap to an answer to the question remains.
It is not clear to us whether the conditions of Theorem 1 can be weakened in a way that it also applies to network function having width larger than the input dimensions under certain circumstances.
The following might be a natural question in this context: Let F : R n 0 → R be a neural network function of arbitrary width but with weight matrices having rank less than or equal to n 0 .
Does a maximum principle similar to Theorem 1 apply in this case?
The following example shows that the maximum principle in Theorem 1 does not hold in this case. Example 1. Let M = [ − 1 , 1] and let W 1 ∈ R 2 × 1 and b 1 ∈ R 2 implement the following linear afﬁne mapping
[Equation]
Then M 1 = ReLU( W 1 M + b 1 ) is given by M 1 = M 1 , 1 ∪ M 1 , 2 where [Equation] and
[Equation]
< 0 }
[Equation]
[Equation]
and
[Equation]
[Equation]
Now, let v = (1 , 1 / 2) T ∈ R 2 , W 2 ∈ R 2 × 2 and b 2 ∈ R 2 such that x (cid:55)→ W 2 x + b 2 implements the orthogonal projection onto the hypersurface V :=
{ x ∈ R 2 : x T v = (1 , 0) v } .
That is, the hypersurface orthogonal to v which intersects (1 , 0) T .
Then both, M 1 , 1 and M 1 , 2 are mapped to the line M 2 :=
{ y ∈ R 2 : y = (1 , 0)+ λ ( − 1 / 2 , 1) , 0 ≤ λ ≤ 1 / 2 } and since those vectors are contained in the ﬁrst quadrant we have
[Equation]
It can further easily be veriﬁed that, under the latter two layer network function, 0 , which is an inner point of M , is the only vector that is mapped to the extreme point (1 , 0) of M 2 .
Now it is easy to ﬁnd a weight W 3 ∈ R 2 such that the linear map x
(cid:55)→ W 3 x as a mapping from M 2 to R takes its minimum or maximum value at (1 , 0) only.
By concatenation of additional components to the input dimension such that M = [ − 1 , 1] × [0 , 1] d , d ∈ N , and adaptation of the weights W 1 , W 2 and bias b 1 , b 2 in a way that these additional components are mapped identically under the action of the two layer network function x (cid:55)→
ReLU( W 2 (ReLU( W 1 x + b 1 ) + b 1 ) , the above example can be extended to higher dimensions.
The following proposition provides the main observation for the proof of Theorem 1.
Proposition 1.
Let M be a compact subset of R n 0 , σ a continuous, monotonic activation function and F ∈ NN n 0 σ ( n 0 ) such that all weight matrices W j , j = 1 , ..., L , are square and have full rank.
If for some x ∈ M ◦ the image F ( x ) is a boundary point of F ( M ) , i.e. F ( x ) ∈ ∂F ( M ) , then there is an ˜ x ∈ ∂M such that F (˜ x ) =
F ( x ) .
Proof.
In case that σ is injective, the whole network function F is injective since the weight matrices are assumed to be square and of full rank.
Hence, the invariance of the domain theorem gives that no x ∈ M ◦ can be mapped to ∂F ( M ) .
The remaining case is that σ is partially constant.
Then, given an x ∈ M ◦ with F ( x ) ∈ ∂F ( M ) , let ˜ k be the smallest k ∈ { 1 , ..., L − 1 } such that σ (
F ˜ k
( x )) ∈ ∂σ ( F ˜ k ( M )) .
Notice that such a ˜ k must exist since, according to the invariance of the domain theorem, the last injective, linear afﬁne mapping x
(cid:55)→ W L x cannot map inner points to boundary points.
Then, necessarily for at least one component of F ˜ k ( x ) , a partially constant part of σ is active.
More explicitly, say ( F ˜ k
( x ))
( j ) ∈ [ a j , b j ] ⊂ R with σ ( t ) =
c j , where c j is the constant value taken for all t ∈ [ a j , b j ] , for some indices j ∈ I ⊂ { 1 , ..., n 0 } .
In case that one of those intervals can be enlarged arbitrarily to either [ a j , ∞ ) or ( −∞ , b j ] , which would be the case for ReLU activation, we can ﬁnd a t ∈ R such that x ˜ k = F ˜ k
( x ) + te j ∈ ∂F ˜
k
( M ) and with σ ( x ˜ k )
=
σ ( F ˜ k
( x )) .
In a next step we show that such an x ˜ k can also be found in cases where the length of the intervals [ a j , b j ] are upper bounded.
In those cases we may assume that the intervals [ a j , b j ] are maximum in the sense that at their left and right end σ is not constant any more, and I is maximum in the sense that for every other index, the activation function σ is injective in a neighbourhood.
Indeed, if such components would not exist, the component wise application of σ would be injective in a small neighbourhood of F ˜ k
( x ) and the invariance of the domain theorem would exclude that σ (
F ˜ k ( x )) ∈ ∂σ ( F ˜ k ( M )) .
By the selection of ˜ k , F ˜ k
( x ) is still an inner point of F ˜ k ( M ) .
The fact that the inner point F ˜ k ( x ) ∈ ( F ˜ k ( M )) ◦ is mapped to a boundary point by the element wise application of σ implies that there is some non-empty ˜ I
⊂ I with corresponding t j ∈ R such that
[Equation]
Indeed, otherwise,
[Equation]
where for some small ε > 0 and all j ∈ I , t j may be chosen arbitrarily in ( − (
F ˜ k ( x ))
( j ) + a j − ε, − (
F ˜ k ( x ))
( j ) + b j + ε ) and ε > | t j | > 0 for j / ∈ I .
By the deﬁnition of I and the corresponding intervals [ a j , b j ] , this would imply that σ (cid:0) F ˜ k
( x ) (cid:1) is an inner point of σ (cid:0) F ˜ k ( M ) (cid:1) .
Now, (8) implies that there is a ˜ x ∈ M such that
[Equation]
with t j as in (7).
Considering that W ˜ k is square and has full rank, the inverse image of F ˜ k (˜ x ) under x (cid:55)→ W ˜ k x + b ˜ k is a boundary point of σ ( F ˜ k − 1 ( M )) in case that ˜ k > 1 and a boundary point of M in case when ˜ k
= 1 .
In the last case we are done.
In the ﬁrst case, the same reasoning can be applied to ˜ x instead of x , and the iterative application ﬁnally leads to a boundary point of M that is mapped to σ (cid:0)
F ˜ k
(˜ x ) (cid:1)
=
σ (cid:0) F ˜ k
( x ) (cid:1) and hence follows the same trajectory when passed through the last layers from ˜ k to L .
Proof.
(Theorem 1)
We may assume that M ◦ is non-empty and that every weight matrix W j , j = 1 , ..., L − 1 , is square and of full rank.
Otherwise the result is trivial, because F ( M ) =
∂F ( M ) .
Hence the assumption of Proposition 1 holds for ˜ F =
A L − 1 ◦ ...
◦
A 1 .
Since the linear mapping x (cid:55)→
W L x from R n L − 1 to R takes its maximum and minimum values on ∂ ˜ F ( M ) , the latter proposition also implies that F takes its maximum value on ∂M .
A PPROXIMATION PROPERTIES ON SUBSETS
In this section we give some approximation results that show that, despite the limitations induced by Theorem 1, the network functions considered in this work still allow some weaker kind of universal approximation on subsets.
The ﬁrst result in this section follows from existing results on approximation by network functions.
The ﬁrst point can be deduced from Hanin & Sellke (2017,
Theorem 1).
The second point can be derived from Hanin & Sellke (2017, Proposition 2) or from
Hardt & Ma (2017, proof of Theorem 3.2.).
Proposition 2.
Let M be some compact subset of R n 0 .
If there exists a subspace U ⊂ R n 0 with dim( U ) < n 0 such that P U is injective as a mapping from M to U , then M admits universal uniform approximation of continuous function by network functions in NN ReLU (dim( U ) + 1) .
Let M = { x 1 , ..., x m } ⊂ R n 0 be a ﬁnite set and f : M → R some mapping.
Then there exists a network function F ∈ NN ReLU (2) such that F ( x ) =
f
( x ) for all x ∈ M .
The proof is almost straightforward, given the results from the aforementioned works.
A proof can be found in the appendix.
We next extend the second statement of the latter proposition to the case of two disjoint n 0 -dimensional compact sets.
Theorem 1 shows that arbitrary uniform approximation is not admitted on such domains.
However, exact ﬁt is possible for functions that are constant on each of these sets under certain geometrical properties on their mutual location.
The approximation of such functions arises naturally in classiﬁcation tasks where each point in a particular disjoint compact set belongs to the same class, i.e. these sets constitute different disjoint clusters of the data.
Theorem 2.
Let M = K 1 ∪ K 2 , with disjoint compact sets K 1 , K 2 ⊂ R n 0 such that for some c ∈ R n 0 and some linearly independent v 1 , ..., v n 0 ∈ R n 0 the set K 1 is contained in the sector
[Equation]
and K 2 ⊂ R n 0 \ S .
Then for every function f : M → R that takes constant values on K 1 and K 2 , respectively, there exists a network function F ∈ NN ReLU ( n 0 , 4) such that f ( x )
= F ( x ) for every x ∈ M .
The conditions of Theorem 2 are veriﬁed and inspected by numerical experiments in Section 6.
However, to us it is unclear whether the assumptions on the mutual location of the compact components in
Theorem 2 can be signiﬁcantly weakened.
Increasing the depth will certainly allow more complex conﬁgurations.
However, from Theorem 1 is clear that K 2 cannot completely enclose K 1 when the maximum width is upper bounded by n 0 , no matter how many layers the network contains.
We think that closing the described gap could entail important insights for the theory of neural networks.
For the proof of the latter result we need the following.
Proposition 3.
Let K , M be two compact sets in R n 0 that are strictly separable by a linear hyper surface.
Then there exists an F ∈ NN n 0 ReLU ( n 0 , 2) such that F ( K ) is a single vector in R n 0 \ M and F ( x ) =
x for all x ∈ M .
Moreover, F can be arranged in a way that given ε >min x ∈ K (cid:107) F ( K ) − x (cid:107) < ε.
As a consequence of Proposition 3 and Proposition 2, we obtain the following for the case of more than two distinct sets under considerably more restrictive condition as in Theorem 2.
Corollary 2.
Let M =
(cid:83)
dj
=1 K j with pairwise disjoint compact sets K j ⊂ R n 0 , j = 1 , ..., d such that for every K j there exist v j ∈ R n 0 and q j ∈ R such that v Tj x > q j for all x ∈ K j , and v Tj x < q j for all x ∈ M \ K j , i.e. there are linear hypersurfaces separating each K j from the remaining K l , l (cid:54) =
j .
Then for every function f : M → R that takes constant values on every K j , j = 1 , ..., d , there exists a network function F ∈ NN ReLU ( n 0 ) with f ( x ) =
F ( x ) for every x ∈ M .
Proof. (Corollary 2) By means of Proposition 3, we ﬁnd an F 1 ∈ NN n 0 ReLU ( n 0 ) such that F 1 ( x ) = x for x ∈ (cid:83) dj =2 K j and a 1 :=
F 1 ( K 1 ) is a single vector in R n 0 \ (cid:83)
dj
=2 K j .
Proposition 3 also allows us to choose F 1 in a way that a 1 is sufﬁciently close to K 1 that the conditions of Corollary 2 still apply to (cid:83) dj
=2 K j ∪ { a 1 } .
We can iteratively apply Proposition 3 to ﬁnd F 1 , ..., F d ∈ NN n 0 ReLU ( n 0 ) such that
[Equation]
for pairwise distinct a 1 , ..., a d ∈ R n 0 .
Now, for a given f : M → R with f ( x ) =
y j for all x ∈ K j , j = 1 , ..., d , Proposition 2 yields an ˜ F ∈ NN ReLU (2) with ˜ F ( a j ) =
y j for all j = 1 , ..., d .
The desired network function is then obtained by F :=
˜ F ◦ F d ◦ ... ◦ F 1 .
Proof.
(Proposition 3) Let us remind that e 1 , ..., e n 0 are the standard basis vectors in R n 0 .
By assumption, there is a v ∈ R n 0 , with (cid:107) v (cid:107) = 1 and a q ∈ R such that v T x > q for x ∈ K and v T x < q for x ∈ M .
For a given ε > 0 we may assume that the linear hyper surface deﬁned by v, q ,
H :=
{ x ∈ R n 0 : v T x = q } is so near to K that for some a ∈ K that realizes the minimum distance of K to H we have [Equation]
[Equation]
Indeed, otherwise we can shift the hyper surface accordingly by increasing q .
Then, by the fact that (cid:107) v (cid:107) = 1 , (9) means that a has a distance equal to ε/ 2 to H and ˜ a :
= a − ε/ 2 v ∈ H is the unique vector in H that realizes this distance.
Let V 1 ∈ R n 0 × n 0 such that x (cid:55)→ V 1 x implements a length preserving rotation that maps v to − e 1 .
Then x (cid:55)→ V 1 x − V 1 ˜ a maps H to span( e 2 , ..., e n 0 ) and
K 1 :=
V 1 K − V 1 ˜ a is contained in the half space of vectors with negative ﬁrst component and with a 1 :=
V 1 a − V 1 ˜ a = − ε/ 2 e 1 , and M 1 :=
V 1 M − V 1 ˜ a is contained in the half space of vectors with positive ﬁrst component.
By compactness of K 1 and M 1 , we can ﬁnd u 1 , ..., u n 0 ∈ span( e 2 , ..., e n 0 ) , such that for a δ > 0 , − u 1 − δe 1 , ..., − u n 0 − δe 1 are linearly independent and such that with δ sufﬁciently small
[Equation]
Let V 2 ∈ R n 0 × n 0 such that x (cid:55)→ V 2 x maps − u j − δe 1 to − e j for j = 1 , ..., n 0 .
Then, by linearity, every u j + δe 1 is mapped to e j for j = 1 , ..., n 0 .
Thus S − is mapped to the cone of vectors having negative components only, and S + is mapped to the cone of vectors having positive components only.
Hence, the application of ReLU maps all of V 2 K 1 to 0 and maps V 2 M 1 identically.
We can now apply the inverse of the linear afﬁne mappings which will map V 2 M 1 back to M , i.e. for
[Equation]
With this F we have F ( M ) =
M and F ( K ) =
F ( a ) =
˜ a .
Note that by (9) and the deﬁnition of ˜ a
[Equation]
which concludes the proof.
Proof.
(Theorem 2) Let V be the matrix that results from the concatenation of the columns v 1 , ..., v nand set W 1 := − V − 1 and b 1 :=
W 1 ( − c ) .
Then the mapping x (cid:55)→ W 1 x maps the v j to − e j , j = 1 , ..., n 0 and W 1 K 1 + b 1 is a subset of
[Equation]
and W 1 K 2 + b 1 ⊂ R n 0 \ S − .
Thus, the one layer network function F 1 ( x ) :
=
ReLU( W 1 x + b 1 ) maps all x ∈ K 1 to 0 and x ∈ K 2 are mapped to
[Equation]
j , λ j ≥ 0 } \ { 0 } . Hence, with a sufﬁciently small q > 0 , F 1 ( K 1 ) and F 1 ( K 2 ) are strictly separated by the linear hyper surface { x ∈ R n 0 : (1 , 1 , ...., 1) x = q } .
The twofold application of Proposition 3 yields two network functions that can be concatenated to a three layer network function F 2 ∈ NN n 0 ReLU ( n 0 , 3) with F 2 ( x ) =
W 4 ReLU( W 3 ReLU( W 2 x + b 2 ) + b 3 ) + b 4 and such that F 2 ( F 1 ( K 1 )) and F 2 ( F 1 ( K 2 )) are two distinct vectors u 1 , u 2 in R n 0 , respectively.
For a given f : M → R that takes constant values on K 1 and K 2 , say a 1 , a 2 ∈ R , respectively, we attach a linear afﬁne layer x
(cid:55)→ w T x + b 5 , where w ∈ R n 0 , b 5 ∈ R such that w T u 1 + b 5 = a 1 and w T u 2 + b 5 = a 2 .
This mapping is integrated in the ﬁnal layer of F 2 ◦ F 1 and ﬁnally gives
[Equation]
We conclude this section with a comment on implications of our results on generalisation.
In Zhang et al.
(2017) the researchers made the observation that deep neural networks can interpolate randomly annotated data, which shows the enormous capabilities of large networks to memorize data.
In
Proposition 2, Theorem 2 and Corollary 3 we show that network functions of width less than or equal than to input dimension in many conﬁgurations also implement an exact ﬁt of the training data.
At the same time these network functions are restricted by the maximum principle given in Theorem 1, which may contradict the underlying distribution of the data at hand.
U NIQUENESS
A maximum principle similar to Theorem 1 also applies for harmonic and holomorphic functions, that is, the null space of the Laplace operator and the null space of ∂ z := 1 / 2( ∂ x + i∂ y )
( x real part, y imaginary part), respectively (Rudin, 2006).
As a consequence, a uniqueness theorem applies which states that two harmonic, respectively holomorphic, functions coincide on the interior of a given set when they coincide on its boundary, respectively.
In view of Theorem 1, it is hence natural to ask whether a similar result applies in the case of network functions in NN σ ( n 0 ) .
However, the proof for the case of harmonic and holomorphic functions relies on the fact that these functions form a vector space, which is not the case for the set NN σ ( n 0 ) .
The following examples show that a uniqueness theorem does not hold, in general, for the network functions with (strictly) monotonic activation functions considered in this work.
Example 2.
Let σ α ( x ) be the leaky ReLU function deﬁned by σ α ( x ) = x for x ≥ 0 and σ α ( x ) =
αx for x < 0 with α > 0 .
Then
[Equation]
both map − 1 to 0 and 1 to 1 , but the ﬁrst one maps 0 to α/ (1 + α ) , while the second one maps 0 to 1 / 2 and hence do not coincide unless α = 1 .
Example 3.
A similar example can be obtained for analytic activation functions.
Indeed, let σ be the sigmoid function.
For x (cid:55)→ σ ( ax + b ) , one can arrange a > 0 , and b ∈ R , such that in one case only a concave and in a second case a convex excerpt of sigmoid is active for x in [0,1], respectively, and such that both functions coincide at 0 and 1 .
As the ﬁrst one is concave and the second one is convex on [0,1] and both are non-linear, they do not coincide on the interior of [0,1].
A simple uniqueness result follows from the following observation.
Proposition 4.
Let M be a compact subset of R n 0 , F ∈ NN n 0 ReLU ( n 0 ) and B = F − 1 (( F ( M )) ◦ ) the inverse image of the inner points of F ( M ) .
Then F is linear afﬁne and bijective as a mapping from B to ( F ( M )) ◦ .
A proof of Proposition 4 is given in the appendix.
As a consequence we have.
Corollary 3.
Let F, G ∈ NN n 0 ReLU ( n 0 ) and M ⊂ R n 0 be some compact set.
If F and G coincide on a set of vectors v 1 , ..., v n 0 +1 in
[Equation]
G ( M )) ◦ ) ,
such that v j − v n 0 +1 , j = 1 , ..., n 0 are linearly independent, then F ( x ) =
G ( x ) for all x ∈ K .
It should be noted that in the latter corollary, the fact that F, G coincide on K does not imply that F and G are the same network functions as functions on R n 0 .
N UMERICAL EXPERIMENT
We provide numerical results to illustrate that neural networks learn transformations similar to the construction of our theoretical derivations in the proof of Theorem 2.
We formulate a toy example consisting of several (6 or 8) pair-wise disjoint balls of the same class encasing a center ball of a different class.
The border balls use a radius of 0 .
125 and a subset of the centres (0 . 25 , 0 . 25) , (0 . 5 , 0 . 25) , (0 . 75 , 0 . 25) , (0 . 25 , 0 . 5) , (0 . 75 , 0 . 5) , (0 . 25 , 0 . 75) , (0 . 5 , 0 . 75) and (0 .
75) while the center ball uses a radius of 0 .
01 and the center (0 . 5 , 0 . 5) .
We generated randomly and uniformlydata points for each border ball, however, as the classes would hence be unbalanced, we generated as many data points for the center ball as there are in total for the border balls.
We used a multilayer perceptron (MLP) consisting of 4 layers, all of width 2, and the ReLU activation function (as stated in the theorem).
For illustration purposes, we used an input dimension of 2 such that the neural network functions are of the form F ∈ NN ReLU (2) .
We used a batch size of 16, the Adam optimizer with a learning rate of 0.001 and we trained each model for 500 epochs using the mean-squared error (MSE) as our cost function.
The different datasets and results are reported in Fig.
We rate an experiment as successful when the universal uniform approximation condition (3) tends to zero.
It can be observed in Fig.
1 (a) that, when the conditions of Theorem 2 are fulﬁlled, the neural network can approximate the function correctly and the different layers are learning transformations similar to the constructions in the proof.
When the conditions are violated, as is shown in Fig.
1 (b), the universal uniform approximation condition does not tend to zero and the intermediate layer transformations need to violate the constructions in the proof.
Additional results are reported in Fig.
2 in the appendix.
C ONCLUSION
We identiﬁed a maximum principle that holds in the case of width less than or equal to the input dimension for all common activation functions, which can be interpreted as a root cause why universal approximation is not possible under the aforementioned conditions.
On the positive side we proved [Figure] that, although being not sufﬁcient for universal approximation in the general sense, ReLU network functions of width less than or equal to the input dimension are sufﬁcient to implement functions that give zero or arbitrary small training error in some machine learning tasks.
In particular, we have shown that for the case of two disjoint compact sets, exact ﬁt in a two class classiﬁcation task is feasible with 4 layers, provided the sets can be separated by a cone like sector.
However, there remains a gap between our positive results and limitation implied by the maximum principle.
Our numerical experiments show that neural networks learn transformations similar to our theoretical derivations.
R EPRODUCIBILITY S TATEMENT
This paper is a contribution to the theory of neural networks, which includes some numerical examples to illustrate our ﬁndings.
For this purpose, we work only with artiﬁcial two-dimensional data and simple network architectures, each of which is described in detail and is therefore easily reproducible.
Further, we provide a code implementation in the supplementary material to reproduce the results of our paper.
A proof is given for all the theorems, propositions and lemma either in the main paper or in the appendix.
A A PPENDIX P ROOF P ROPOSITION
Proof.
We start with the proof of (1).
It is clear that P U followed by a linear isomorphism E : U → R m with m = dim( U ) < n 0 can be implemented by x (cid:55)→ W 1 x + b 1 with some W 1 ∈ R m × n 0 , b 1 ∈ R m .
As M is compact and hence bounded, one can further arrange b 1 in a way that W 1 x + b 1 ≥holds (component wise) for every x ∈ M , which implies that ReLU( W 1 x + b 1 )
=
W 1 x + b 1 for all x ∈ M .
Thus g ( x ) =
ReLU( W 1 x + b 1 ) maps M bijectively to M 1 :=
ReLU( W 1 M + b 1 ) ⊂ R m .
Now, uniform approximation of a given continuous function f from M to R amounts to uniform approximation of continuous function from M 1 to R .
By Tietze’s Extension Theorem (Rudin, 2006) we can apply Theorem 1 from Hanin & Sellke (2017) which yields arbitrary accurate uniform approximation of continuous function on rectangular domains in R m by network functions in
NN ReLU ( m + 1) .
To show (2), let W 1 ∈ R 1 × n 0 and b 1 ∈ R such that y j = W 1 x j + b 1 , j = 1 , ..., m , are distinct points
M 1 = { y 1 , ..., y m } ⊂ [0 , ∞ ) .
It is then clear that also F 0 ( x ) :
=
ReLU( W 1 x + b 1 ) maps M to M 1 .
Now, for some function f : M → R (which is automatically continuous as M is ﬁnite) we have to ﬁnd a network function F 1 ∈ NN ReLU (2) such that F 1 ( y j ) =
f
( x j ) for j = 1 , ..., m .
To this end, we observe that the mapping y j (cid:55)→ f ( x j ) j = 1 , ..., m can be interpolated by a max-min string, see
Hanin & Sellke (2017, Deﬁnition 1) for a deﬁnition, so that Hanin & Sellke (2017, Proposition 2) yields the desired F 1 .
Alternatively, one could follow the proof of Hardt & Ma (2017, Theorem 3.2.), observing that on the non-negative y j , j = 1 , ..., m , the residual ReLU network function constructed in the latter can be written as a ReLU network function in the sense of this work with width two in our case.
Finally, we have that F = F 1 ◦ F 0 ∈ NN ReLU (2) and F ( x j ) =
f
( x j ) for all j = 1 , ..., m .
P ROOF P ROPOSITION
Proof.
It is clear that the existence of inner points in F ( M ) requires that every weight matrix of F is square and of full rank.
Indeed, otherwise the range of M is reduced to a manifold of dimension less than or equal to n 0 − 1 after a layer that violates this condition.
From then on the range remains without inner points.
If L = 1 , then we are done, since in this case F reduces to an isomorphism on R n 0 .
Otherwise, let M j :=
A j ( M j − 1 ) for j = 1 , ..., L and M 0 := M .
Hence F ( M ) =
M L .
Since W L is non-singular, u L ∈ M L is an inner point of M L if and only if u L = W L u L − 1 for some unique point u L − 1 ∈ M ◦ L − 1 .
This implies that component wise u L − 1 > 0 and further, by the fact that W L − 1 is non-singular, u L − 1 = W L − 1 u L − 2 + b L − 1 for some unique point u
L − 1 ∈ M ◦ L − 2 .
The exact same reasoning can now iteratively be applied down to the ﬁrst layer.
This gives a sequence of u j ∈ M ◦ j , j = 1 , ..., L so that component wise u j > 0 holds for every j .
Hence, u j = W j u j − 1 + b j for j = 1 , ..., L .
Thus, for the inverse image B = F − 1 ( F [Equation] M ) [Equation]
) ⊂ [Equation] ◦ , the mapping F reduces to an bijective linear afﬁne mapping. N UMERICAL E XPERIMENT
[Figure]
which is applied to the of to form the of the