U NIVERSAL A PPROXIMATION WITH D EEP N ARROW N ETWORKS
Anonymous authors Paper under double-blind review
A BSTRACT
The classical Universal Approximation Theorem certiﬁes that the universal approximation property holds for the class of neural networks of arbitrary width.
Here we consider the natural ‘dual’ theorem for width-bounded networks of arbitrary depth.
Precisely, let n be the number of inputs neurons, m be the number of output neurons, and let ρ be any nonafﬁne continuous function, with a continuous nonzero derivative at some point.
Then we show that the class of neural networks of arbitrary depth, width n + m +2 , and activation function ρ , exhibits the universal approximation property with respect to the uniform norm on compact subsets of R n .
This covers every activation function possible to use in practice; in particular this includes polynomial activation functions, making this genuinely different to the classical case.
We go on to consider extensions of this result.
First we show an analogous result for a certain class of nowhere differentiable activation functions.
Second we establish an analogous result for noncompact domains, by showing that deep narrow networks with the ReLU activation function exhibit the universal approximation property with respect to the p -norm on R n .
Finally we show that width of only n + m + 1 sufﬁces for ‘most’ activation functions.
I NTRODUCTION
The Universal Approximation Theorem (Cybenko, 1989; Hornik, 1991; Pinkus, 1999) states that universal approximation holds for the class of neural networks with a single hidden layer of arbitrary width, with any continuous nonpolynomial activation function:
Theorem 1.1.
Let ρ : R → R be any continuous function.
Let N ρn represent the class of neural networks with activation function ρ , with n neurons in the input layer, one neuron in the output layer, and one hidden layer with an arbitrary number of neurons.
Let K ⊆ R n be compact.
Then
N ρn is dense in C ( K ) if and only if ρ is nonpolynomial.
What if arbitrary width is replaced with arbitrary depth?
Put more precisely, can networks of bounded width and arbitrary depth provide universal approximation?
In some sense this poses a question ‘dual’ to the problem answered by the classical Universal Approximation Theorem.
We refer to networks of this type as deep, narrow networks.
Furthermore we might ask how narrow the network may be, and what activation functions may be admitted.
We provide a near-complete answer to these various questions.
Universal approximation may be established with respect to more than one topology.
Continuous activation functions beget networks representing continuous functions.
Thus when working with respect to the uniform norm, it is natural to seek density in C ( K ; R m ) for K ⊆ R n .
When working with respect to the p -norm, it is natural to seek density in L p ( R n ; R m ) for p ∈ [1 , ∞ ) .
In this latter case we may hope to generalise to noncompact domains, as functions in L p ( R n ; R m ) must exhibit some sort of decay.
The primary motivation for this work stems from the work of Lu et al.
(2017), who study this question in the special case of the popular ReLU activation function, and who establish density in L 1 ( R n ) .
The other notable result we are aware of is the work of Hanin & Sellke (2017), who show another special case: they also consider the ReLU activation function, and establish density in
C ( K ; R m ) for K ⊆ R n compact.
This article demonstrates generalisations of these results, in particular to general activation functions, without relying on the strong algebraic and analytic properties of the ReLU activation function.
This also improves certain results speciﬁc to the ReLU.
The rest of the paper is laid out as follows.
Section 2 discusses existing work.
Section 3 provides a brief summary of our results; these are then presented in detail in Section 4.
Section 5 is the conclusion.
Several proofs are deferred to the appendices, due to length and technical content.
C ONTEXT
Some positive results have been established showing that particular classes of networks are dense in certain spaces.
Some negative results have also been established, showing that insufﬁciently wide networks will fail to be dense.
Hanin & Sellke (2017) have shown that deep narrow networks with the ReLU activation function exhibit the universal approximation property in C ( K ; R m ) for K ⊆ R n compact.
Lu et al.
(2017) have shown that deep narrow networks with the ReLU activation function exhibit the universal approximation property in L 1 ( R n ) , whilst Lin & Jegelka (2018) have shown that a particular description of residual networks, with the ReLU activation function, also exhibit the universal approximation property in this space.
We are not aware of any results for the general case of
L p ( R n ; R m ) for p ∈ [1 , ∞ ) .
We do not know of any positive results applying to activation functions other than the ReLU.
Regarding widths insufﬁcient for a class of deep narrow networks to exhibit the universal approximation property, consider the case of a network with n input neurons and a single output neuron.
For certain activation functions, Johnson (2019) shows that width n is insufﬁcient to give density in
C ( K ) .
For the ReLU activation function, Lu et al.
(2017) show that width n is insufﬁcient to give density in L 1 ( R n ) , and that width n − 1 is insufﬁcient in L 1 ([ − 1 , 1] n ) .
For the ReLU activation function, Hanin & Sellke (2017) shows that width n is insufﬁcient to give density in C ( K ) , and that in fact that this is the greatest possible width not achieving universal approximation in this context.
The precise minimum width for activation functions other than ReLU, or for multiple output neurons, remains unknown.
Everything discussed so far is in the most general case of approximating functions on Euclidean space: in the language of machine learning, they are regression tasks.
There has been some related work in the special case of classiﬁcation tasks, for example Beise et al.
; Szymanski & McCane (2012); Rojas (2003);
Nguyen et al.
There has also been some related work in the special case of certain ﬁnite domains; Le Roux & Bengio (2010) show that networks with sigmoid activation function and width n can approximate any distribution on { 0 , 1 } n .
See also
Sutskever & Hinton (2008).
Mont ´ ufar (2014) considers the analogous scenario for distributions on { 0 , 1 , . . .
, q − 1 } n .
S UMMARY OF R ESULTS
Deﬁnition 3.1.
Let ρ : R → R and n, m, k ∈ N .
Then let NN ρn,m,k represent the class of functions R n → R m described by neural networks with n neurons in the input layer, m neurons in the output layer, k neurons in each hidden layer, and an arbitrary number of hidden layers, such that every neuron in every hidden layer has activation function ρ , and every neuron in the output layer has the identity activation function.
Our central result is the following theorem.
Theorem 3.2.
Let ρ : R → R be any continuous function which is continuously differentiable at at least one point, with nonzero derivative at that point.
Let K ⊆ R n be compact.
Then NN ρn,m,n + m + is dense in C ( K ; R m ) .
The technical condition is very weak; in particular it is satisﬁed by every piecewise- C 1 function not identically zero.
Thus any activation function that one might practically imagine using on a computer must satisfy this property.
Theorem 3.2 is proved by handling particular classes of activation functions as special cases.
First we have the result for nonpolynomial activation functions, for which the width can be made slightly smaller.
Theorem 4.4.
Let ρ : R → R be any continuous nonpolynomial function which is continuously differentiable at at least one point, with nonzero derivative at that point.
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +1 is dense in C ( K ; R m ) .
We observe a corollary for noncompact domains, which generalises Lu et al.
(2017, Theorem 1) to multiple output neurons, a narrower width, and L p for p (cid:62) 1 instead of just p = 1 .
Corollary 4.6.
Let ρ be the ReLU activation function.
Let p ∈
[1 , ∞ ) .
Then NN ρn,m,n + m +1 is dense in L p ( R n ; R m ) .
Moving on to polynomial activation functions, the smaller width of n + m + 1 also sufﬁces for a large class of polynomials.
Theorem 4.8.
Let ρ : R → R be any polynomial for which there exists a point α ∈ R such that ρ (cid:48)
( α ) =
and ρ (cid:48)(cid:48)
( α ) (cid:54) = 0 .
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +1 is dense in C ( K ; R m ) .
The simplest example of such a ρ is x (cid:55)→ x 2 .
Note that in the classical arbitrary-width case it is both necessary and sufﬁcient that the activation function be nonpolynomial.
Here, however, the same restriction does not hold.
Polynomial activation functions are a reasonable choice in this context.
The technical restrictions on the polynomial may be lifted by allowing the full n + m + 2 neurons per hidden layer.
Theorem 4.10.
Let ρ : R → R be any nonafﬁne polynomial.
Let K ⊆ R n be compact.
Then
NN ρn,m,n + m +2 is dense in C ( K ; R m ) .
It is clear that Theorems 4.4 and 4.10 together imply Theorem 3.2.
Finally we observe that even pathological cases not satisfying the technical condition of Theorem 3.2 may exhibit the universal approximation property.
Proposition 4.13.
Let w : R → R be any bounded continuous nowhere differentiable function.
Let ρ ( x ) = sin( x ) + w ( x )e − x , which will also be nowhere differentiable.
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +1 is dense in C ( K ; R m ) .
Whilst not of direct practical application, this result exempliﬁes that little necessarily needs to be assumed about an activation function to understand the corresponding class of neural networks.
Remark 3.3.
Every proof in this article is constructive, and can in principle be traced so as to determine how depth changes with approximation error.
We have instead chosen to focus on quantifying the width necessary for universal approximation.
In fact there are places in our arguments where we have used a deeper network over a shallower one, when the deeper network is more easily explained.
Remark 3.4.
An understanding of universal approximation in deep narrow networks is applicable to an understanding of bottlenecks, when information must be discarded due to space constraints, for example in autoencoders (Bengio et al., 2006).
This article demonstrates that certain narrow networks will not constitute a bottleneck; a converse example is Johnson (2019), who demonstrates that networks of insufﬁcient width are forced to maintain certain topological invariants.
U NIVERSAL APPROXIMATION 4. P RELIMINARIES
Remark 4.1.
A neuron is usually deﬁned as an activation function composed with an afﬁne function.
For ease, we shall extend the deﬁnition of a neuron to allow it to represent a function of the form ψ ◦ ρ ◦ φ , where ψ and φ are afﬁne functions, and ρ is the activation function.
This does not increase the representational power of the network, as the new afﬁne functions may be absorbed into the afﬁne parts of the next layer, but it will make the neural representation of many functions easier to present.
We refer to these as enhanced neurons .
It is similarly allowable to take afﬁne combinations of multiple enhanced neurons; we will use this fact as well.
One of the key ideas behind our constructions is that most reasonable activation functions can be taken to approximate the identity function.
Indeed, this is essentially the notion that differentiability captures: that a function is locally afﬁne.
This makes it possible to treat neurons as ‘registers’, in which information may be stored and preserved through the layers.
This allows for preserving the input values between layers, which is crucial to performing computations in a memory-bounded regime.
Thus our constructions have strong overtones of space-limited algorithm design in traditional computer science settings.
Lemma 4.2.
Let ρ : R → R be any continuous function which is continuously differentiable at at least one point, with nonzero derivative at that point.
Let L ⊆
R be compact.
Then a single enhanced neuron with activation function ρ may uniformly approximate the identity function ι : R → R on L , with arbitrarily small error.
Proof.
By assumption, as ρ is continuously differentiable, there exists [ a, b ] ⊆ R with a (cid:54)
=
b , on some neighbourhood of which ρ is differentiable, and α ∈ ( a, b ) at which ρ (cid:48) is continuous, and for which ρ (cid:48) ( α ) is nonzero.
For h ∈ R \ { 0 } , let φ h ( x ) = hx
+ α , and let
[Equation]
Then
[Equation]
is of the form that an enhanced neuron can represent.
Then for all u ∈ [ a, b ] , by the Mean Value
Theorem there exists ξ u between u and α such that
[Equation]
and hence
[Equation]
for h sufﬁciently small that φ h ( L ) ⊆
[ a, b ] .
Now let ρ (cid:48) have modulus of continuity ω on [ a, b ] .
Let ι : R → R represent the identity function.
Then for all x ∈ L ,
[Equation]
and so ι h → ι uniformly over L .
Notation.
Throughout the rest of this paper ι h will be used to denote such an approximation to the identity function, where ι h → ι uniformly as h → 0 .
An enhnaced neuron may be described as performing (for example) the computation x
(cid:55)→ ι h (4 x +3) .
This is possible as the afﬁne transformation x (cid:55)→ 4 x +
and the afﬁne transformation φ h (from the description of ι h ) may be combined together into a single afﬁne transformation.
4. N ONPOLYNOMIAL ACTIVATION FUNCTIONS
We consider the ‘Register Model’, which represents a simpliﬁcation of a neural network.
Proposition 4.3 (Register Model) .
Let ρ : R → R be any continuous nonpolynomial function.
Let
I ρn,m,n + m +1 represent the class of neural networks with n neurons in the input layer, m neurons in the output layer, n + m + 1 neurons in each hidden layer, an arbitrary number of hidden layers, and for which n + m of the neurons in each hidden layer have the identity activation function, and one neuron in each hidden layer has activation function ρ .
Let K ⊆ R n be compact.
Then I ρn,m,n + m + is dense in C ( K ; R m ) .
(cid:80) Mj =
1 σ j xx· · · x n σ M =
σ M ( x 1 , . . .
, x n ) (cid:80) M − 1 j =1 σ j · · · xx· · · x n σ 3 =
σ 3 ( x 1 , . . .
, x n ) σ 1 + σxx· · · x n σ 2 = σ 2 ( x 1 , . . . , x n ) σxx· · · x n σ 1 = σ 1 ( x 1 , . . .
, x n ) xx· · · x n
See Appendix A for the proof.
A simpliﬁed depiction of the proof of the Register Model is shown in Figure 1, for the special case of m = 1 .
It uses n neurons in each layer as registers to preserve the input values.
A single neuron in each layer performs a computation based off of the input values, which were preserved in the previous layer.
The remaining neuron in each layer also acts a register, gradually summing up the results of the computation neurons.
The computation neurons may be shown to exist by the classical
Universal Approximation Theorem.
The Register Model is similar to Hanin & Sellke (2017), who have a related construction speciﬁc to the ReLU.
The idea of the Register Model may also be thought of as thematically similar to residual networks, as in Lin & Jegelka (2018): in both cases the network is almost applying the identity transformation at each layer, with only a small amount of nonlinearity.
Theorem 4.4.
Let ρ : R → R be any continuous nonpolynomial function which is continuously differentiable at at least one point, with nonzero derivative at that point.
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +1 is dense in C ( K ; R m ) .
Proof.
Let f ∈ C
( K ; R m ) and ε > 0 .
Set up a neural network as in the Register Model (Proposition 4.3), approximating f to within ε/ 2 .
Every neuron requiring an identity activation function in the Register Model will instead approximate the identity, in the manner of Lemma 4.2.
Uniform continuity preserves uniform convergence, compactness is preserved by continuous functions, and a composition of two uniformly convergent sequences of functions with uniformly continuous limits is again uniformly convergent.
So as a neural network is a layer-by-layer composition of functions then the new model can be taken within ε/ 2 of the Register Model, with respect to (cid:107) · (cid:107) ∞ in K , by taking h sufﬁciently small.
Remark 4.5.
This of course implies approximation in L p ( K, R m ) for p ∈ [1 , ∞ ) .
However, when ρ is the ReLU activation function, then the next corollary shows that in fact the result may be generalised to unbounded domains.
Corollary 4.6.
Let ρ be the ReLU activation function.
Let p ∈
[1 , ∞ ) .
Then NN ρn,m,n + m +1 is dense in L p ( R n ; R m ) .
See Appendix B for the proof.
Given some f ∈ L p ( R n ; R m ) , the essential idea of the proof is to choose a compact set K ⊆ R n on which f places most of its mass, and ﬁnd a neural approximation to f on K in the manner of
Theorem 4.4.
Once this is done, a cut-off function is applied outside the set, so that the network takes the value zero in R n \ K .
The interesting bit is ﬁnding a neural representation of such cut-off behaviour.
In particular the ‘obvious’ thing to do – multiply by a cut-off function – does not appear to have a suitable neural representation, as merely approximating the multiplication operation is not necessarily enough on an unbounded domain.
Instead the strategy is to take a maximum and a minimum with suitable cut-off functions.
4. P OLYNOMIAL ACTIVATION FUNCTIONS
For the classical Universal Approximation Theorem, it was necessary that the activation function be nonpolynomial.
However that turns out to be unnecessary here; deep narrow networks are different to shallow wide networks, and polynomial activations functions are reasonable choices.
We begin with the simplest possible nonafﬁne polynomial, namely ρ ( x ) = x 2 .
Proposition 4.7 (Square Model) .
Let ρ ( x ) = x 2 .
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +1 is dense in C ( K ; R m ) .
See Appendix C for the proof.
As might be expected, density is established with the help of the
Stone–Weierstrass theorem, reducing the problem to the approximation of arbitrary polynomials.
We remark that it is actually straightforward to ﬁnd a construction showing that NN ρn,m,n + m +2 is dense in C ( K ; R m ) when ρ ( x ) = x 2 , note the increased width.
This is because the square activation function can be used to perform multiplication, via xy =
(( x + y ) 2 − ( x − y ) 2 ) / 4 , and this makes it easy to construct arbitrary polynomials.
In fact this is what is done in the proof of Proposition 4.7 for ﬁnding m − 1 of the m outputs, when there is still a ‘spare’ neuron in each layer.
It is computing the ﬁnal output that actually requires the bulk of the work.
The key to this argument is a width-efﬁcient approximation to division.
It is a consequence of Proposition 4.7 that any (polynomial) activation function which can approximate the square activation function, in a suitable manner, is also capable of universal approximation.
Theorem 4.8.
Let ρ : R → R be any polynomial for which there exists a point α ∈ R such that ρ (cid:48)
( α ) =
and ρ (cid:48)(cid:48)
( α ) (cid:54) = 0 .
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +1 is dense in C ( K ; R m ) .
Proof.
Let h ∈ R \ { 0 } .
Deﬁne ρ h : R → R by
[Equation]
Then, taking a Taylor explansion around α ,
[Equation]
Let s ( x ) = x 2 .
Then ρ h → s uniformly over any compact set as h → 0 .
Now set up a network as in the Square Model (Proposition 4.7), with every neuron using the square activation function.
Call this network N .
Create a network N h by copying N and giving every neuron in the network the activation function ρ h instead.
Uniform continuity preserves uniform convergence, compactness is preserved by continuous functions, and a composition of two uniformly convergent sequences of functions with uniformly continuous limits is again uniformly convergent.
So as a neural network is a layer-by-layer composition of functions, then the difference between N and N h , with respect to (cid:107) · (cid:107) ∞ on K , may be taken arbitrarily small by taking h arbitrarily small. Furthermore note that ρ h is just
ρ pre- and post-composed with afﬁne functions. (Note that there is only one term in the deﬁnition of ρ h ( x ) which depends on x .) This means that any network which may be represented with activation function ρ h may be precisely represented with activation function ρ , by combining the afﬁne transformations involved.
Remark 4.9.
That ρ is polynomial is never really used in the proof of Theorem 4.8.
Only a certain amount of differentiability is required, and all such nonpolynomial functions are already covered by Theorem 4.4, as a nonzero second derivative at α implies a nonzero ﬁrst derivative somewhere close to α .
Nonetheless in principle this provides another possible construction by which certain networks may be shown to exhibit universal approximation.
Note that the converse strategy (applying nonpolynomial techniques to the polynomial case) fails.
This is because the Register Model requires nonpolynomial activation functions due to its dependence on the classical Universal Approximation Theorem.
φ 1 ( η )φ 2 ( η )· · · φ n + m +1 ( η )=
⇒ Expand α 1 ,n + m +1 =
ι ( α 1 ,n + m ) α 2 ,n + m +1 =
ι ( α 2 ,n + m ) · · · α n + m +1 ,n + m +
=
( α n + m +1 ,n + m )· · · α 1 , 2 =
ι
( α 1 , 1 ) α 2 , 2 =
( α 2 , 1 )· · · α n + m +1 , 2 =
ι ( α n + m +1 , 1 )
α 1 , 1 = φ 1 ( η )α 2 , 1 = φ 2 ( η ) · · · α n + m +1 , 1 = φ n + m +
[Figure]
( η )
Theorem 4.10.
Let ρ : R → R be any nonafﬁne polynomial.
Let K ⊆ R n be compact.
Then
NN ρn,m,n + m +2 is dense in C ( K ; R m ) .
Proof.
Fix α ∈ R such that ρ (cid:48)(cid:48)
( α ) (cid:54) = 0 , which exists as ρ is nonafﬁne.
Now let h ∈ (0 , ∞ ) .
Deﬁne σ h : R → R by σ
[Equation]
=
[Equation] [Equation]
Then Taylor expanding ρ ( α + hx ) and ρ ( α − hx ) around α ,
[Equation]
ρ (
[Equation]
h
[Equation]
) h
[Equation]
) = x 2 + O ( hx 3 ) .
Observe that σ h needs precisely two operations of ρ on (afﬁne transformations of) x , and so may be computed by two enhanced neurons with activation function ρ .
Thus the operation of a single enhanced neuron with square activation function may be approximated by two enhanced neurons with activation function ρ .
Let N be a network as in the Square Model (Proposition 4.7) with every neuron using the square activation function.
Let (cid:96) be any hidden layer of N ; it contains n + m +1 neurons.
Let η be a vector of the values of the neurons of the previous layer.
Let φ i be the afﬁne part of the i th neuron of (cid:96) , so that (cid:96) computes φ 1 ( η ) 2 , . .
φ n + m +1 ( η ) 2 .
Then this may equivalently be calculated with n + m + layers of n + m + 1 neurons each, with n + m of the neurons in each of these new layers using the identity function, and one neuron using the square activation function.
The ﬁrst of these new layers applies the φ i , and the i th layer squares the value of the i th neuron.
See Figure 2.
Apply this procedure to every layer of N ; call the resulting network (cid:101) N .
It will compute exactly the same function as N , and will have n + m + 1 times as many layers, but will use only a single squaring operation in each layer.
Create a copy of (cid:101) N , call it (cid:101) N h .
Replace its identity activation functions with approximations in the manner of Lemma 4.2, using activation function ρ .
Replace its square activation functions (one in each layer) by approximations in the manner described above with σ h ; this requires an extra neuron in each hidden layer, so that the network is now of width n + m + 2 .
Thus (cid:101) N h uses the activation function ρ throughout.
Uniform continuity preserves uniform convergence, compactness is preserved by continuous functions, and a composition of two uniformly convergent sequences of functions with uniformly continuous limits is again uniformly convergent.
So as a neural network is a layer-by-layer composition of functions, then the difference between (cid:101) N h and (cid:101) N , with respect to (cid:107) · (cid:107) ∞ on K , may be taken arbitrarily small by taking h arbitrarily small.
Remark 4.11.
It is possible to construct shallower networks analogous to (cid:101) N .
The proof of Proposition 4.7 in Appendix C, uses most of the network’s neurons to approximate the identity anyway; only a few in each layer are used to square a valued that is desired to be squared.
These are the only neurons that actually require the procedure used in Figure 2 and the proof of Theorem 4.10.
4. N ONDIFFERENTIABLE ACTIVATION FUNCTIONS
Although not of direct practical application, results for nondifferentiable activation functions demonstrate how certain pathological cases are still capable of being handled.
Lemma 4.12.
Let w : R → R be any bounded continuous nowhere differentiable function.
Let ρ ( x ) = sin( x ) + w ( x )e − x .
Let L ⊆
R be compact.
Then a single enhanced neuron with activation function ρ may uniformly approximate the identity function ι : R → R on L , with arbitrarily small error.
Proof.
For h ∈ R \ { 0 } and A ∈ 2 π N , let φ h,A ( x ) =
hx + A , and let ψ ( x ) = x/h .
Let ι h,A = ψ h ◦ ρ ◦ φ h,A , which is of the form that an enhanced neuron can represent.
Then jointly taking h small enough and
A large enough it is clear that ι h,A may be taken uniformly close to ι on L .
Proposition 4.13.
Let w : R → R be any bounded continuous nowhere differentiable function.
Let ρ ( x ) = sin( x ) + w ( x )e − x , which will also be nowhere differentiable.
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +1 is dense in C ( K ; R m ) .
Proof.
As the proof of Theorem 4.4, except substituting Lemma 4.12 for Lemma 4.2.
This manner of proof may be extended to other nondifferentiable activation functions as well.
C ONCLUSION
There is a large literature on theoretical properties of neural networks, but much of it deals only with the ReLU.
However how to select an activation function remains a poorly understood topic, and many other options have been proposed: leaky ReLU, PReLU, RRelu, ELU, SELU and other more exotic activation functions as well.
Our central contribution is to provide results for universal approximation using general activation functions (Theorems 3.2, 4.4, 4.8 and 4.10).
In contrast to previous work, these results do not rely on the nice properties of the ReLU, and in particular do not rely on its explicit description.
The techniques we use are straightforward, and robust enough to handle even the pathological case of nondifferentiable activation functions (Proposition 4.13).
We also consider approximation in L p norm (Remark 4.5), and generalise previous work to smaller widths, multiple output neurons, and p (cid:62) 1 in place of p = 1 (Corollary 4.6).
In contrast to much previous work, every result we show also handles the general case of multiple output neurons.
A CKNOWLEDGEMENTS
(Redacted from anonymised submission)
A P ROOF OF THE R EGISTER M ODEL (P ROPOSITION 4.3)
First, we recall the classical Universal Approximation Theorem (Pinkus, 1999):
Theorem 1.1.
Let ρ : R → R be any continuous function.
Let N ρn represent the class of neural networks with activation function ρ , with n neurons in the input layer, one neuron in the output layer, and one hidden layer with an arbitrary number of neurons.
Let K ⊆ R n be compact.
Then
N ρn is dense in C ( K ) if and only if ρ is nonpolynomial.
The Register Model is created by suitably reorganising the neurons from a collection of such shallow networks.
Proposition 4.3 (Register Model) .
Let ρ : R → R be any continuous nonpolynomial function.
Let
I ρn,m,n + m +1 represent the class of neural networks with n neurons in the input layer, m neurons in the output layer, n + m + 1 neurons in each hidden layer, an arbitrary number of hidden layers, and for which n + m of the neurons in each hidden layer have the identity activation function, and one neuron in each hidden layer has activation function ρ .
Let K ⊆ R n be compact.
Then I ρn,m,n + m + is dense in C ( K ; R m ) .
Proof.
Fix f ∈ C ( K ; R m ) .
Let f = ( f 1 , . . .
, f m ) .
Fix ε > 0 .
By Theorem 1.1, there exist singlehidden-layer neural networks g 1 , . .
g m ∈ N ρn with activation function ρ approximating f 1 , . . .
, f m respectively.
Each approximation is to within error ε with respect to (cid:107) · (cid:107) ∞ on K .
Let each g
i have β i hidden neurons.
Let σ i,j represent the operation of its j th hidden neuron, for j ∈ { 1 , . .
, β i } .
In keeping with the idea of enhanced neurons, let each σ i,j include the afﬁne function that comes after it in the output layer of g i , so that g
i = (cid:80) β i j =1 σ i,j .
Let M = (cid:80) mi
=1 β i .
We seek to construct a neural network N ∈
I ρn,m,n + m +1 .
Given input ( x 1 , . . .
, x n ) ∈ R n , it will output ( G 1 , . . . ,
G m ) ∈
R m , such that G i
=
g i ( x 1 , . . . , x n ) for each i .
That is, it will compute all of the shallow networks g 1 , . . .
, g m .
Thus it will approximate f to within error ε with respect to (cid:107) · (cid:107) ∞ on K .
The construction of N is mostly easily expressed pictorially; see Figure 3.
In each cell, representing a neuron, we deﬁne its value as a function of the values of the neurons in the previous layer.
In every layer, all but one of the neurons uses the identity activation function ι : R → R , whilst one neuron in each layer performs a computation of the form σ i,j .
The construction can be summed up as follows.
Each layer has n + m + 1 neurons, arranged into a group of n neurons, a group of a single neuron, and a group of m neurons.
The ﬁrst n neurons in each layer simply record the input ( x 1 , . .
, x n ) , by applying an identity activation function.
We refer to these as the ‘in-register neurons’.
Next we consider g 1 , . .
. , g m , which are all shallow networks.
The neurons in the hidden layers of g 1 , . .
. , g m are arranged ‘vertically’ in our deep network, one in each layer.
This is the neuron in each layer that uses the activation function ρ .
We refer to these as the ‘computation neurons’.
Each computation neuron performs its computation based off of the inputs preserved in the in-register neurons.
The ﬁnal group of m neurons also use the identity activation function; their afﬁne parts gradually sum up the results of the computation neurons.
We refer to these as the ‘out-register neurons’.
The i th out-register neuron in each layer will sum up the results of the computation neurons computing σ i,j for all j ∈ { 1 , . .
, β i } .
Finally, the neurons in the output layer of the network are connected to the out-register neurons of the ﬁnal hidden layer.
As each of the neurons in the output layer has, as usual, the identity activation function, they will now have computed the desired results.
[Table]
[Table] [Table]
=
[Table]
=
[Table]
[Table] [Table]
[Table] [Table]
=
[Table]
+
[Table]
=
[Table]
=
[Table]
=
[Table]
= σ
[Table]
[Table] [Table]
[Table] [Table]
=
[Table] [Table]
n
[Table]
=
[Table]
n , 1 )
[Table]
= σ 1 , 2 ( γ 1 , 1 , ..., γ n , 1 ) ζ 1 ,= ι ( τ) ζ 2 ,=· · · ζ m ,=γ 1 ,= ι ( x 1 ) γ 2 ,= ι ( x 2 ) · · · γ n ,= ι ( x n ) τ 1 = σ 1 , 1 ( x 1 ,..
., x n ) ζ 1 ,=ζ 2 ,=· · · ζ m ,=xx· · · x n .
F i gu r e 3 :
T h e t h
i c k li n e s d e li
m it g r oup
s o f l a y e r s ; t h e i t
h
g r oup
c
o m pu t
e s σ i , 1 , . . .
, σ i , β i .
T h
e i npu t s t o t h e n e t w o r k
a r e x 1 , .
x n , d e p i c t e d a t t h e bo tt o m .
T h e ou t pu t s fr o m t
h e n e t w o r k a r e
G 1 , .
G m , d e p i
c t e d a t t h e t op .
T h
e i d e n tit y ac ti v a ti on f un c ti on
R →
R i s d e no t e d ι .
B P ROOF OF C OROLLARY 4.
Lemma B.1.
Let a, b, c, d ∈ R be such that a < b < c < d .
Let U a,b,c,d : R → R be the unique continuous piecewise afﬁne function which is one on [ b, c ] and zero on ( −∞ , a ] ∪ [ d, ∞ ) .
Then two layers of two enhanced neurons each, with ReLU activation function, may exactly represent the function U a,b,c,d .
Proof.
Let x ∈ R be the input.
Let m 1 = 1 / ( b − a ) .
Let m 2 = 1 / ( d − c ) .
Let η 1 , η 2 represent the ﬁrst neuron in each layer, and ζ 1 , ζ 2 represent the second neuron in each layer.
We assign them values as follows.
[Equation]
Then U a,b,c,d ( x ) =
ζ 2 − η 2 .
(This ﬁnal afﬁne transformation is allowed, in keeping with the notion of enhanced neurons.)
Lemma B.2.
One layer of two enhanced neurons, with ReLU activation function, may exactly represent the function ( x, y )
(cid:55)→ min { x, y } on [0 , ∞ ) 2 .
Proof.
Let the ﬁrst neuron compute η = max { 0 , x − y } .
Let the second neuron compute ζ = max { 0 , x } .
Then min { x, y } = ζ − η .
Corollary 4.6.
Let ρ be the ReLU activation function.
Let p ∈
[1 , ∞ ) .
Then NN ρn,m,n + m +1 is dense in L p ( R n ; R m ) .
Proof.
Let f ∈ L p ( R n ; R m ) and ε > 0 .
For simplicity assume that R m is endowed with the (cid:107) · (cid:107) ∞ norm; other norms are of course equivalent.
Let (cid:98) f
= ( (cid:98) f 1 , . . .
, (cid:98) f m ) ∈ C c
( R n ; R m ) be such that
[Equation]
(B.1)
Let
[Equation]
B.
[Equation]
and
[Equation]
Pick a 1 , b 1 , . .
. , a n , b n ∈ R such that J deﬁned by
J =
[ a 1 , b 1 ] × · · · × [ a n , b n ] is such that supp (cid:98) f ⊆ J .
Furthermore, for δ > 0 that we shall ﬁx in a moment, let
[Equation]
δ,
[Equation]
and let K be deﬁned by
K =
[ A 1 , B 1 ] × · · · × [ A n , B n ] .
Fix δ small enough that
[Equation]
Let g
=
( g 1 , . .
g m ) ∈
NN ρn,m,n + m +1 be such that
[Equation]
which exists by Theorem 4.4.
Note that g is deﬁned on all of R n ; it simply happens to be close to (cid:98) f on K .
In particular it will takes values close to zero on K \ J , and may take arbitrary values in R n \ K .
By equations (B.2), (B.3), (B.5), it is the case that
[Equation]
Now consider the network describing g ; it will be modiﬁed slightly.
The goal is to create a network which takes value g on J , zero in R n \ K , and moves between these values in the interface region
K \ J .
Such a network will provide a suitable approximation to (cid:98) f .
This is done by ﬁrst constructing a function which is approximately the indicator function for J , with support in K ; call such a function
U .
The idea then is to construct a neural representation of G i deﬁned by
[Equation]
Provided | K \ J | is small enough then G
=
( G 1 , . . .
, G m ) will be the desired approximation; this is proved this below.
We move on to presenting the neural representation of this construction.
First we observe that because the activation function is the ReLU, then the identity approximations used in the proof of Theorem 4.4 may in fact exactly represent the identity function on some compact set: x (cid:55)→ max { 0 , x + N } − N is exactly the identity function, for suitably large N , and is of the form that an enhanced neuron may represent.
This observation isn’t strictly necessary for the proof, but it does simplify the presentation somewhat, as the values preserved in the in-register neurons of g are now exactly the inputs x = ( x 1 , . . .
, x n ) for x ∈ K .
For sufﬁciently negative x i , outside of
K , they will take the value − N instead, but by insisting that is N sufﬁciently large that
[Equation]
for all i , then this will not be an issue for the proof.
So take the network representing g , and remove the output layer.
(If the output layer is performing any afﬁne transformations then treat them as being part of the ﬁnal hidden layer, in the manner of enhanced neurons.
Thus the output layer that is being removed is just applying the identity function to the out-register neurons.)
Some more hidden layers will be placed on top, and then a new output layer will be placed on top.
In the following description, all neurons not otherwise speciﬁed will be performing the identity function, so as to preserve the values of the corresponding neurons in the preceding layer.
As all functions involved are continuous and K is compact, and compactness is preserved by continuous functions, and continuous functions are bounded on compact sets, then this is possible for all x ∈ K by taking N large enough.
The ﬁrst task is to modify the value stored in the in-register neurons corresponding to x 1 .
At present it stores the value x 1 ; by using this in-register neuron and the computation neuron in two extra layers, its value may be replaced with U A 1 ,a 1 ,b 1 ,B 1 ( x 1 ) , via Lemma B.1.
Place another two layers on top, and use them to replace the value of x 2 in the second in-register neuron with U A 2 ,a 2 ,b 2 ,B 2 ( x 2 ) , and so on.
The in-register neurons now store the values ( U A 1 ,a 1 ,b 1 ,B 1 ( x 1 ) , . .
U A n ,a n ,b n ,B n ( x n )) .
Once this is complete, place another layer on top and use the ﬁrst two in-register neurons to compute the minimum of their values, in the manner of Lemma B.2, thus computing min { U A 1 ,a 1 ,b 1 ,B 1 ( x 1 ) , U A 2 ,a 2 ,b 2 ,B 2 ( x 2 ) } .
Place another layer on top and use another two in- register neurons to compute the minimum of this value and the value presently stored in the third in-register neuron, that is U A 3 ,a 3 ,b 3 ,B 3 ( x 3 ) , so that min { U A 1 ,a 1 ,b 1 ,B 1 ( x 1 ) , U A 2 ,a 2 ,b 2 ,B 2 ( x 2 ) , U A 3 ,a 3 ,b 3 ,B 3 ( x 3 ) } has now been computed.
Continue to repeat this process until the in-register neurons have computed.
[Equation]
Observe how U represents an approximation to the indicator function for J , with support in K , evaluated at ( x 1 , . . .
, x n ) .
This is a highly destructive set of operations: the network no longer remembers the values of its inputs.
Thankfully, it no longer needs them.
Note how the small foible regarding how an in-register neuron would only record − N instead of x i , for x i < − N , is not an issue.
This is because of equation (B.7), which implies that U A i ,a i ,b i ,B i ( x i )
= 
 =
U
A i ,a i ,b i ,B i ( − N ) , thus leaving the value of U unaffected.
The out-register neurons presently store the values g 1 , . .
. , g m , where g i =
g i ( x 1 , . . . , x n ) .
Now add another layer.
Let the value of its out-register neurons be θ 1 , . .
, θ m , where
[Equation]
Add one more hidden layer.
Let the value of its out-register neurons be λ 1 , . . .
, λ m , where
[Equation]
Finally place the output layer on top.
Let the value of its neurons be G 1 , . . .
, G m , where
[Equation]
Then in fact
G
[Equation]
as desired.
All that remains to show is that G
=
( G 1 , . . .
, G m ) of this form is indeed a suitable approximation.
First, as G and g coincide in J , and by equation (B.5),
[Equation]
=
[Equation]
< ε 3 .
(B.9)
Secondly, by equations (B.2), (B.3), (B.6), (B.8) and then equation (B.4), (cid:32)(cid:90)
K \ J
[Equation]
(cid:98)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:33)
[Equation]
/p (cid:54)
[Equation]
\ J |
[Equation]
/p sup x ∈ K \ J (cid:12)(cid:12)(cid:12) (cid:98) f
( x )
[Equation]
G ( x ) (cid:12)(cid:12)(cid:12) (cid:54) | K \ J | 1 /p ·
[Equation]
max {| C | , | c |} < ε 3 .
(B.10)
Thirdly, (cid:32)(cid:90)
R n \ K (cid:12)(cid:12)(cid:12) (cid:98) f
( x ) − G ( x ) (cid:12)(cid:12)(cid:12) p d x (cid:33) 1 /p = 0 , (B.11) as both (cid:98) f and G have support in K .
So by equations (B.1), (B.9), (B.10) and (B.11),
[Equation]
ε
[Equation]
C P ROOF OF THE S QUARE M ODEL (P ROPOSITION 4.7)
Lemma C.1.
One layer of two enhanced neurons, with square activation function, may exactly represent the multiplication function ( x, y )
(cid:55)→ xy on R 2 .
Proof.
Let the ﬁrst neuron compute η = ( x + y ) 2 / 4 .
Let the second neuron compute ζ = ( x − y ) 2 / 4 .
Then xy =
η − ζ .
Lemma C.2.
Fix L ⊆ R 2 compact.
Three layers of two enhanced neurons each, with square activation function, may uniformly approximate ( x, y )
(cid:55)→ ( x 2 , y ( x + 1)) arbitrarily well on L .
Proof.
Let h, s ∈ R \ { 0 } .
Let η 1 , η 2 , η 3 represent the ﬁrst neuron in each layer; let ζ 1 , ζ 2 , ζrepresent the second neuron in each layer.
Let ι h represent an approximation to the identity in the manner of Lemma 4.2.
Using ‘ ≈ ’ as an informal notation to represent ‘equal to up to the use of ι h in place ι ’, just to help keep track of why we are performing these operations, assign values to η 1 , η 2 , η 3 and ζ 1 , ζ 2 , ζ 3 as follows:
[Equation]
And so η 3 may be taken arbitrarily close to x 2 and ζ 3 may be taken arbitrarily close to y ( x +1) , with respect to (cid:107) · (cid:107) ∞ on L ,
Proposition C.3.
Fix L ⊆ (0 , 2) compact.
Then multiple layers of two enhanced neurons each, with square activation function, may uniformly approximate x (cid:55)→ 1 /x arbitrarily well on L .
(Unlike Lemma C.2, the number of layers necessary will depend on the quality of approximation.)
Proof.
First note that
[Equation]
as n → ∞ , uniformly over compact subsets of ( − 1 , 1) .
Thus,
[Equation]
uniformly over L .
This has the following neural approximation: let η
=
(1 − x ) 2 and ζ
= ι h (2 − x ) be the neurons in the ﬁrst layer, where ι h is some approximation of the identity as in Lemma 4.2.
Let κ h represent an approximation to ( x, y )
(cid:55)→ ( x 2 , y ( x + 1)) in the manner of Lemma C.2, with error made arbitrarily small as h → 0 .
Now for i ∈ { 1 , 4 , 7 , 10 , . . .
, 3 n − 2 } , recursively deﬁne ( η i +3 , ζ i +3 )
=
κ h ( η i , ζ i ) , where we increase the index by three to represent the fact that three layers are used to perform this operation.
So up to approximation, η i +3 ≈ ( η i ) 2 , and ζ i +3 ≈
ζ i ( η i + 1) .
So ζ 3 n +1 → (2 − x ) (cid:81) ni
=1 (1 + (1 − x ) 2 i ) uniformly over L as h → 0 .
Thus the result is obtained by taking ﬁrst n large enough and then h small enough.
Remark C.4.
Our approach to Proposition C.3 is to ﬁnd a suitable polynomial approximation of the reciprocal function, and then represent that with a network of multiple layers of two neurons each.
It is fortunate, then, that this polynomial happens to be of a form that may be represented by such a network, as it is not clear that this should necessarily be the case for all polynomials.
Even if Proposition 4.7 were already known, it requires a network of width three to represent arbitrary polynomials R
→ R , whereas Proposition C.3 uses a network of only width two.
It remains unclear whether an arbitrary-depth network of width two, with square activation function, is capable of universal approximation in C ( K ) .
Proposition 4.7 (Square Model) .
Let ρ ( x ) = x 2 .
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +1 is dense in C ( K ; R m ) .
Proof.
Fix f ∈ C ( K ; R m ) .
Let f = ( f 1 , . . .
, f m ) .
Fix ε > 0 .
By precomposing with an afﬁne function – which may be absorbed into the ﬁrst layer of the network – assume without loss of generality that
[Equation]
By the Stone–Weierstrass theorem there exist polynomials g 1 , . .
g m in x 1 , . . .
x n approximating f 1 , . . .
, f m to within ε/ 3 with respect to (cid:107) · (cid:107) ∞ .
We will construct a network that evaluates arbitrarily good approximations to g 1 , . .
. , g n .
There are a total of n + m + 1 neurons in each layer; group them as in the proof of the Register Model (Proposition 4.3), see Appendix A, so that in each layer there is a group of n neurons that we refer to as ‘in-register neurons’, a single neuron that we refer to as the ‘computation neuron’, and a group of m neurons that we refer to as the ‘out-register neurons’.
As before, every in-register neuron will simply apply an approximate identity function to the corresponding in-register neuron in the previous layer, so that they preserve the inputs to the network, up to an arbitrarily good approximation of the identity.
(For now, at least – later, when constructing the approximation to g 1 , which will be the ﬁnal approximation that is handled, then these neurons will be repurposed to perform that computation.)
For the sake of sanity of notation, we shall suppress this detail in our notation, and refer to our neurons in later layers as having e.g. ‘ x 1 ’ as an input to them to them; in practice this means some arbitrarily good approximation to x 1 .
The out-register neurons will eventually store the desired outputs of the network; thus there is an out-register neuron in each layer ‘corresponding’ to each of the g i .
Now suppose m > 1 ; if m = 1 then this paragraph and the next three paragraphs may simply be skipped.
It is easy to build a network approximating g 2 , . . .
g m , as there is at least one ‘extra’ neuron per layer that is available to use: the out-register neuron corresponding to g 1 .
The strategy is as follows.
Let g
= (cid:80) Nj =1 δ j , where each δ j is a monomial.
Using the computation neuron and the ‘extra’ out-register neuron in each layer, perform successive multiplications in the manner of Lemma
C.1 to compute the value of δ 1 .
For example, if δ 1 = x 21 x 2 x 3 , then this chain of multiplications is x 1 ( x 1 ( x 2 x 3 )) .
The computation neuron and the ‘extra’ out-register neuron in the ﬁrst layer compute the multiplication α = x 2 x 3 , these neurons in the second layer compute the multiplication β
= x 1 α , and these neurons in the third layer compute x 1 β .
This value is then stored in the out-register neuron corresponding to g 2 – by using the afﬁne part of the operation of this neuron – and kept through the layers via approximate identity functions, as per Lemma 4.2.
This process is then repeated for δ 2 .
The result is then added on – via the afﬁne part of the operation of the out-register neuron corresponding to g 2 – to the value stored in the out-register neuron corresponding to g 2 .
Repeat for all j until all of the δ j have been computed and the out-register neuron stores an approximation to g 2 .
This is only an approximation in that it requires the use of approximate identity functions; other than that it is exact.
As such, by taking sufﬁciently good approximations of the identity function, this will be a uniform approximation to g 2 over K .
Now repeat this whole process for g 3 , . . .
g m .
For the rest of the layers of the network, the out-register neurons corresponding to g 2 , . .
. , g m will now simply apply approximate identity functions to maintain their values: these will eventually form the outputs of the network.
Let these computed values be denoted (cid:98) g 2 , . . .
, (cid:98) g m .
(With the ‘hat’ notation becuse of the fact that these are not the values g 2 , . . .
, g m , due to the approximate identity functions in between.)
The difﬁcult bit is computing an approximation to g 1 , as it must be done without the ‘extra’ neuron in each layer.
In total, then, in each layer, there are n +2 neurons available: the n in-register neurons (which have so far been storing the inputs x 1 , . . .
x n ), the computation neuron, and the out-register neuron corresponding to g 1 .
Written in terms of monomials, let g 1 = (cid:80)
Mj =1 γ j .
Then g 1 may be written as
[Equation]
Note that this description is deﬁned over K , as K is bounded away from the origin.
In particular, let γ j = (cid:81)
nk
=1 x θ j,k k , for θ j,k ∈ N 0 .
Substituting this in, g
[Equation]
=
[Equation]
Now let sup K be deﬁned by
[Equation]
so that 1 < sup K < 2 .
Let r be an approximation to x (cid:55)→ 1 /x in the manner of Proposition C.3, with the L of that proposition given by
[Equation]
where α > 0 is taken small enough that the inclusion holds.
Let r a denote r composed a times.
By taking r to be a suitably good approximation, we may ensure that (cid:101) g 1 deﬁned by
[Equation]
[Equation]
· (cid:33)(cid:33)(cid:33) is an approximation to g 1 in K , to within ε/ 3 , with respect to (cid:107) · (cid:107) ∞ .
This is possible by equations (C.1) and (C.2); in particular the approximation should be sufﬁciently precise that
[Equation]
which is why α > 0 is needed: note how r 2 , and thus r 4 , r 6 , ..., r 2 M − 2 , are approximately the identity function on L .
This description of (cid:101) g 1 is now amenable to representation with a neural network.
The key fact about this description of (cid:101) g 1 is that, working from the most nested set of brackets outwards, the value of (cid:101) gmay be computed by performing a single chain of multiplications and additions, and occasionally taking the reciprocal of all of the input values.
So let the computation neuron and the out-register neuron corresponding to g 1 perform the multiplications, layer-by-layer, to compute (cid:81)
nk =1 x θ M,k k , in the manner of Lemma C.1.
Store this value in the out-register neuron.
Now use the computation neurons and the in-register neurons corresponding to x 1 (across multiple layers) to compute r ( x 1 ) , in the manner of Proposition C.3: eventually the in-register neuron is now storing r ( x 1 ) ≈ 1 /x 1 .
Repeat for the other in-register neurons, so that they are collectively storing r ( x 1 ) , . . .
, r ( x n ) .
Now the computation neuron and the out-register neuron may start multipying r ( x 1 ) , . . .
, r ( x n ) on to (cid:81) nk =1 x θ M,k k (which is the value presently stored in the out-register neuron) the appropriate number to times to compute (cid:2)(cid:81)
nk =
1 r ( x k )
θ M − 1 ,k (cid:3) (cid:104)(cid:81)
nk =1 x θ M,k k (cid:105) , by Lemma C.1.
Store this value in the out-register neuron.
Then add one (using the afﬁne part of a layer).
The out-register neuron has now computed the expression in the innermost bracket in the description of (cid:101) g 1 .
The general pattern is now clear: apply r to all of the in-register neurons again to compute r 2 ( x i ) , multiply them on to the value in the out-register neuron, and so on.
Eventually the out-register neuron corresponding to g 1 will have computed the value (cid:101) g 1 .
Actually, it will have computed an approximation (cid:98) g 1 to this value, because of the identity approximations involved.
By taking all of the (many) identity approximations throughout the network to be suitably precise, the values of (cid:101) g 1 and (cid:98) g 1 may be taken within ε/ 3 of each other, and the values of (cid:98) g 2 , . . .
, (cid:98) g m
and g 2 , . .
. , g m may be taken within 2 ε/ 3 of each other, in each case with respect to (cid:107) · (cid:107) ∞ on K .
Thus ( (cid:98) g 1 , . .
, (cid:98) g m ) approximates f with total error no more than ε , and the proof is complete.