Size and depth of monotone neural networks: interpolation and approximation
Abstract
Monotone functions and data sets arise in a variety of applications.
We study the interpolation problem for monotone data sets: The input is a monotone data set with n points, and the goal is to find a size and depth efficient monotone neural network with non negative parameters and threshold units that interpolates the data set.
We show that there are monotone data sets that cannot be interpolated by a monotone network of depth 2 .
On the other hand, we prove that for every monotone data set with n points in R d , there exists an interpolating monotone network of depth 4 and size O ( nd ) .
Our interpolation result implies that every monotone function over [0 , 1] d can be approximated arbitrarily well by a depth4 monotone network, improving the previous best-known construction of depth d + 1 .
Finally, building on results from Boolean circuit complexity, we show that the inductive bias of having positive parameters can lead to a super-polynomial blow-up in the number of neurons when approximating monotone functions.
Introduction
The recent successes of neural networks are owed, at least in part, to their great approximation and interpolation power.
However, some prediction tasks require their predictors to possess specific properties.
This work focuses on monotonicity and studies the effect on overall expressive power when restricting attention to monotone neural networks .
Given x, y ∈ R d we consider the partial ordering, x ≥ y ⇐⇒ for every i = 1 , . . .
d, [ x ] i ≥ [ y ] i .
Here, and throughout the paper we use [ x ] i for the i th coordinate of x .
A function f : [0 , 1] d → R is monotone 1 if for every two vectors x, y ∈ [0 , 1] d , x ≥ y =
⇒ f
( x ) ≥ f ( y ) .
Monotone functions arise in several fields such as economics, operations research, statistics, healthcare and engineering.
For example, larger houses typically result in larger prices, and certain features are monotonically related to option pricing [12] and bond rating [9].
As monotonicity constraints abound, there are specialized statistical methods aimed at fitting and modeling monotonic functions such as Isotonic Regression [1, 22, 24] as well as many other works related to monotone approximation [7,17,42].
Neural networks are no exception: Several works are devoted to the study of approximating monotone functions using neural networks [9,36,39].
When using a network to approximate a monotone function, one might try to“force” the network to be monotone.
A natural way to achieve this is to consider only networks where every parameter
(other than the biases) is non-negative 2 .
Towards this aim, we introduce the following class of monotone networks .
Recall that the building blocks of a neural network are the single one-dimensional neurons σ w,b ( x ) = σ ( ⟨ w, x ⟩ + b ) where x ∈ R k is an input of the neuron,
w ∈ R
k is a weight parametrizing the neuron, σ : R → R is the activation function, and b ∈ R is a bias term.
Two popular choices for activation functions, that we shall also consider, are the ReLU activation function σ ( z ) = max(0 , z ) and the threshold activation function σ ( z )
= 1 ( z ≥ 0) , which equals 1 if z ≥ 0 and zero otherwise.
We slightly abuse the term and say that a network is monotone if every single neuron is a monotone function.
Since both ReLU and threshold are monotone, this requirement, of having every neuron a monotone function, translates to w having all positive entries.
Such a restriction on the weights can be seen as an inductive bias reflecting prior knowledge that the functions we wish to approximate are monotone.
One advantage of having such a “positivity bias” is that it guarantees the monotonicity of the network.
Ensuring that a machine learning model approximating a monotone function is indeed monotone is often desirable [15, 25, 26].
Current learning methods such as stochastic gradient descent and back-propagation for training a network are not guaranteed (when applied to a monotone training set) to return a network computing a monotone function.
Furthermore, while there are methods for certifying that a given neural network implements a monotone function [25,39], the task of certifying monotonicity remains a non-trivial task.
Restricting the weights raises several questions on the behavior of monotone neural networks compared to their more general, unconstrained, counterparts.
For example, should we still expect monotone networks to be able to approximate arbitrary monotone functions within an arbitrarily small error?
Continuing a line of work on monotone networks, [9,36], we further elucidate the above comparison and uncover some similarities (a universal approximation theorem) and some surprising differences (the interplay of depth and monotonicity) between monotone and general networks.
We will mainly be interested in expressiveness, the ability to approximate monotone functions and to interpolate monotone data sets using monotone neural networks of constant depth.
1.1 Our contributions
First, it is not a priori clear that monotone networks can approximate every possible monotone function or even fit any monotone data set .
Here a monotone data set is a set of n labeled points ( x i , y i ) i ∈
[ n ] ∈ ( R d × R ) n with the properties
[Equation]
In the monotone interpolation problem we seek to find a monotone network N such that for every i ∈ [ n ] , N ( x i ) =
y i .
On expressive power and interpolation: While it is well-known that general networks with
ReLU activation are universal approximators (can approximate any continuous function on a bounded domain), perhaps surprisingly, the same is not true for monotone networks and monotone functions.
Namely, there are monotone functions that cannot be approximated within an arbitrary small additive error by a monotone network with ReLU gates regardless of the size and depth of the network.
This fact was mentioned in [25]: We provide a proof for completeness.
Lemma 1.
There exists a monotone function f : [0 , 1] → R and a constant c > 0 , such that for any monotone network N with ReLU gates, there exists x ∈ [0 , 1] , such that
[Equation]
Proof.
It is known that a sum of convex functions f
i , (cid:80) α
i f i is convex provided that for every i, α i ≥ 0 .
It is also known that the maximum of convex functions g i , max i { g i } is a convex function.
It follows from the definition of the ReLU gate (in particular, ReLU is a convex function) that a neural network with positive weights at all neurons is convex.
As there are monotone functions that are not convex, the result follows.
For a concrete example one may take the function f ( x ) = √ x for which the result holds with c = 18 .
In light of the above, we shall henceforth only consider monotone networks with a threshold activation function and discuss, for now, the problem of interpolation.
For general networks (no restriction on the weights) with threshold activation, it has been established, in the work of Baum [3], that even with 2 layers, for any labeled data set in R d , there exists an interpolating network.
Our next result is another negative result, which shows that there is an inherent loss of expressive power, when transitioning to 2 -layered monotone threshold networks, provided that the dimension is at least two.
When the input is real-valued there always exists an interpolating monotone network.
This is a simple fact whose proof is omitted: It can be proved using similar ideas to those in [9].
Lemma 2.
Let d ≥ 2 .
There exists a monotone data set ( x i , y i ) i ∈ [ n ] ∈ ( R d × R ) n , such that any depth- 2 monotone network N , with threshold activation must satisfy,
[Equation]
for some i ∈ [ n ] .
Given the limitation of depth-2 monotone networks, it may seem that, like the case of ReLUs, the class of monotone networks with threshold activation is too limited, in the sense that it cannot approximate any monotone function, with constant depth (allowing the depth to scale with the input dimension was considered in [9], see below).
One reason for such a belief is that, for non-monotone networks, depth 2 suffices to ensure universality: any continuous function over a bounded domain can be approximated by a depth- 2 network [2,8,19] and this universality result holds for networks with threshold or ReLUs as activation functions.
Our first main result supports the contrary to this belief.
We establish a depth separation result for monotone threshold networks, and show that, by slightly increasing the number of layers, monotone networks can interpolate arbitrary monotone data sets.
Thereafter a simple argument shows that monotone networks of bounded depth become universal approximators of monotone functions.
As noted, this is in sharp contrast to general neural networks, where adding extra layers can affect the efficiency of the representation [13], but does not change the expressive power.
Theorem 1.
Let ( x i , y i )
i ∈ [ n ] ∈ ( R d × R )
n be a monotone data set.
There exists a monotone threshold network N , with 4 layers and O ( nd ) neurons such that,
[Equation]
for every i ∈ [ n ] .
Moreover, if the set ( x i ) i ∈ [ n ] is totally-ordered, in the sense that, for every i, j ∈ [ n ] , either x i ≤ x j or x j ≥ x
i , then one may take N to have 3 layers and O ( n ) neurons.
We also complement Theorem 1 with a lower bound that shows that the number of neurons we use is essentially tight, up to the dependence on the dimension.
Lemma 3.
There exists a monotone data set ( x i , y i ) i ∈
[ n ] ⊂ ( R d × R ) n such that, if N is an interpolating monotone threshold network, the first layer of N must contain n units.
Moreover, this lower bound holds when the set ( x i ) i ∈ [ n ] is totally-ordered.
The lower bound of Lemma 3 demonstrates another important distinction between monotone and general neural networks.
According to [3], higher-dimensions allow general networks, with 2 layers, to be more compact.
Since the number of parameters in the networks increase with the dimension, one can interpolate labeled data sets in general position with only O (cid:0) nd (cid:1) neurons.
Moreover, for deeper networks, a recent line of work, initiated in [41], shows that O ( √ n ) neurons suffice.
Lemma 3 shows that monotone networks cannot enjoy the same speed-up, either dimensional or from depth, in efficiency.
Since we are dealing with monotone functions, our interpolation results immediately imply a universal approximation theorem for monotone networks of depth 4 .
Theorem 2.
Let f : [0 , 1] d
→ R be a continuous monotone function and let ε > 0 .
Then, there exists a monotone threshold network N , with 4 layers, such that, for every x ∈ [0 , 1] d , | N ( x ) − f ( x ) | ≤ ε.
If the function f is L -Lipschitz, for some L > 0 , one can take N to have O (cid:18) d (cid:16) L √ d ε (cid:17) d (cid:19) neurons.
While it was previously proven, in [9], that monotone networks with threshold activation can approximate any monotone function, the depth in the approximating network given by [9] scales linearly with the input dimension.
Our result is thus a significant improvement which only requires constant depth.
When looking at the size of the network, the linear depth construction, in [9], iteratively evaluates Riemann sums, and builds a network which is piecewise-constant on a grid.
Hence, for L -Lipschitz functions , it would require a similar amount of neurons.
Again, we see that comparable results can be obtained while maintaining constant depth.
Whether one can achieve similar results to Theorems 1 and 2 with only 3 layers is an interesting question which we leave for future research.
Efficiency when compared to general networks: We have shown that, with 4 layers, monotone networks can serve as universal approximates.
However, even if a monotone network can approximate a monotone function arbitrarily well it might be that it requires a much larger size, when compared to unconstrained networks.
In this case, the cost of having a much larger network might outweigh the benefit of having a network that is guaranteed to be monotone.
In our second main result we verify that this can sometimes be the case.
We show that using monotone networks to approximate, in the ℓ ∞ -norm, a monotone function h : [0 , 1] d → R can lead to a super-polynomial blow-up in the number of neurons.
Namely, we demonstrate a smooth monotone function h : [0 , 1] d → R with a poly ( d ) Lipschitz constant such that h can be approximated within an additive error of ε > 0 by a general neural network with poly ( d ) neurons.
Yet any monotone network approximating h within error smaller than 12 requires super-polynomial size in d .
Theorem 3.
There exists a monotone function h : [0 , 1] d → R , such that:
• Any monotone threshold network N which satisfies,
| N ( x ) − h ( x ) | < 1 2 , for every x ∈ [0 , 1] d , must have e Ω(log 2 d ) neurons.
• For every ε > 0 , there exists a general threshold network N , which has poly( d ) neurons and such that, | N ( x ) − h ( x ) | < ε, for every x ∈ [0 , 1] d .
Related work
We are not aware of previous work studying the interpolation problem for monotone data sets using monotone networks.
For general data sets and interpolation using networks with no positivity requirement there is extensive research regarding the size and depth needed to achieve interpolation [5, 10, 41, 45] starting with the seminal work of Baum studying this problem for networks with threshold units [3].
Known constructions of neural networks achieving interpolations are nonmonotone: they may result in negative parameters even for monotone data sets.
Several works have studied approximating monotone (real) functions over a bounded domain using a monotone network.
Sill [36] provides a construction of a monotone network (all parameters are nonnegative) with depth 3 where the first layer consists of linear units divided into groups, the second layer consists of max gates where each group of linear units of the first layer is fed to a different gate and a final gate computing the minimum of all outputs from the second layer.
It is proven in [36] that this class of networks can approximate every monotone function over [0 , 1] d .
We remark that this is very different than the setting considered in this present work.
First, using both min and max gates in the same architecture with positive parameters does not fall into the modern paradigm of an activation function.
Moreover, we are not aware of works prior to Theorem 2, that show how to implement or approximate min and max gates, with arbitrary fan-ins, using constant depth monotone networks .
3 Finally, the results from [36] focus on approximating arbitrary monotone functions and do not consider the monotone interpolation problem studied here.
real numbers. Later, the problem of approximating arbitrary monotone functions with networks having nonnegative parameters using more standard activation functions such as thresholds or sigmoids has been studied in [9].
In particular [9] gives a recursive construction showing how to approximate in the ℓ ∞ norm an arbitrary monotone function using a network of depth d + 1 ( d -hidden layers) with threshold units and non-negative parameters.
In addition [9] provides a construction of a monotone function g : [0 , 1] 2 → R that cannot be approximated in the ℓ ∞ norm with an error smaller than 1 / 8 by a network of depth 2 with sigmoid activation and non-negative parameters, regardless of the number of neurons in the network.
Our Lemma 2 concerns networks with threshold gates and applies to arbitrary dimension larger than 1 .
It can also be extended to provide monotone functions that cannot be approximated by monotone networks with thresholds of depth 2 .
Lower bounds for monotone models of computation have been proven for a variety of models [11], including monotone De Morgan 4 circuits [14,18,31,32], monotone arithmetic circuits and computations [6,21,44], which correspond to polynomials with non-negative coefficients, and circuits with monotone real gates [20, 29] whose inputs and outputs are Boolean .
One difference between our separation result regarding monotone and non-monotone networks and these works is that we consider real differentiable functions as opposed to Boolean functions.
Furthermore we need functions that can be computed by an unconstrained neural network of polynomial size .
In contrast, known lower bounds for circuits with real monotone gates apply to Boolean functions that are believed to be intractable (require a super polynomial number of gates) to compute even with non-monotone circuits (e.g., deciding if a graph contains a clique of a given size).
Finally our model of computation of neural networks with threshold gates differs from arithmetic circuits [35] which use gates that compute polynomial functions.
To achieve our separation result we begin with a Boolean function m , which requires a superpolynomial size to compute by any Boolean circuit with monotone threshold gates, but can be computed efficiently with arbitrary threshold circuits: the existence of m follows from [32].
Thereafter we show how to smoothly extend m to have domain [0 , 1] d while preserving monotonicity.
Proving lower bounds for neural networks with a continuous domain by extending a Boolean function f for which lower bounds are known to a function f ′ whose domain is [0 , 1] d has been done before [40,43].
However the extension method in these works do not yield a function that is monotone.
Therefore, we use a different method based on the multi-linear extension.
Preliminaries and notation
We work on R d , with the Euclidean inner product ⟨· , ·⟩ .
For k ∈ N , we denote [ k ] = { 1 , 2 , . . .
, k } and use { e i }
i ∈
[ d ] for standard unit vectors in R d .
That is, for i ∈
[ d ] , e i = (0 , . . .
, 0 (cid:124) (cid:123)(cid:122) (cid:125)
i − 1 times , 1 , 0 , . . .
,(cid:124) (cid:123)(cid:122) (cid:125)
d − i times ) .
For x ∈ R d and i ∈ [ d ] , we write [ x ] i := ⟨ x, e i ⟩ , the i th coordinate of x .
With a slight abuse of notation, when the dimension of the input changes to, say, R k , we will also use { e i } i ∈ [ k ] to stand for standard unit vectors in R k .
To avoid confusion, we will always make sure to make the dimension explicit.
A neural network of depth L is a function N : R d → R , which can be written as a composition, N
[Equation]
) , where for ℓ ∈ [ L ] , N ℓ : R d ℓ → R d ℓ +1 is a layer.
We set d 1 = d , d L +1 = 1 and term d ℓ +1 as the width of layer ℓ .
Each layer is composed of single neurons in the following way: for i ∈
[ d ℓ +1 ] , [ N ℓ ( x )]
i =
σ ( ⟨ w ℓi , x ⟩ + b ℓi ) where σ : R → R is the activation function, w ℓi ∈ R d ℓ is the weight vector, and b ℓi ∈ R is the bias.
The only exception is the last layer which is an affine functional of the previous layers, N L ( x ) = ⟨ w L , x ⟩ + b L .
For a weight vector w L ∈ R d L and bias b L ∈ R .
Suppose that the activation function is monotone.
We say that a network N is monotone, if, for every ℓ ∈ [ L ] and i ∈
[ d ℓ +1 ] , the weights vector w ℓi has all positive coordinates.
In other words, [ w ℓi ] j ≥ 0 , for every j ∈ [ d ℓ ] .
A counter-example to expressibility
We outline the ideas behind Lemma 2.
We find it instructive to consider the case d = 2 .
Lemmafollows via a simple generalization of this case: a proof 5 of the Lemma can be found in Section A.1.
Recall that σ ( x ) = 1 ( x ≥ 0) is the threshold function and consider the monotone set, [Equation]
Assume towards a contradiction that there exists a network N ( x ) :=
[Equation]
which interpolates the set.
Set
[Equation]
The set I is the set of all neurons which are active on x 3 , and, since N ( x 3 ) = 1 , I is non-empty.
We also define,
[Equation]
It is clear that I 1 ∪ I 2 = I. Observe that for m ∈ I 1 , by monotonicity, we have ⟨ x 1 , w m ⟩ = 2[ w m ] 1 ≥ [ w m ] 1 + [ w m ] 2 = ⟨ x 3 , w m ⟩ .
Since the same also holds for m ∈ I 2 and x 2 , we have, [Equation]
≥
[Equation]
[Equation] Hence, either N ( x 1 ) ≥ 12 or N ( x 2 ) ≥ 12 .
Four layers suffice with threshold activation
Let ( x i , y i )
ni
=1 ∈ ( R d × R )
n be a monotone data set, and assume, with no loss of generality,
[Equation]
If, for some i, i ′ ∈ [ n ] with i ̸ = i ′ we have y i
= y i ′ , and x i ≤ x i ′ , we will assume i < i ′ .
Other ties are resolved arbitrarily.
Note that the assumption that the y i ’s are positive holds without loss of generality, as one can always add a constant to the output of the network to handle negative labels.
This section is dedicated to the proof of Theorem 1, and we will show that one can interpolate the above set using a monotone network with 3 hidden layers.
The first hidden layer is of width dn and the second and third of width n .
Throughout we shall use σ ( t ) = 1 ( t ≥ 0) , for the threshold function.
For ℓ ∈ { 1 , 2 , 3 } we will also write ( w ℓi , b ℓi ) for the weights of the i th neuron in level ℓ .
We shall also use the shorthand, σ ℓi ( x ) = σ ( ⟨ x, w ℓi ⟩ − b ℓi ) .
We first describe the first two layers.
The second layer serves as a monotone embedding into R n .
We emphasize this fact by denoting the second layer as E : R d → R n , with i th coordinate is given by, [ E ( x )] i = σ 2 i ( N 1 ( x )) , where [ N 1 ( x )]
j = σ 1 j ( x ) , for j = 1 , . .
. , nd , are the outputs of the first layer.
First hidden layer: The first hidden layer has dn units.
For j = 1 , . . .
, dn .
We let e i be the i th standard basis vector in R d and define,
[Equation]
In other words, w 1 j = e ( j mod d )+1 and b 1 j = ⟨ x ⌈ jd ⌉ , e ( j mod d )+1 ⟩ (the addition of 1 offsets the fact that mod operations can result in 0 ).
To get a feeling of what the layer does, suppose that j ≡ r mod d , then unit j is activated on input x iff the ( r + 1) th entry of x is at least the ( r + 1) th entry of x ⌈ jd ⌉ .
Second hidden layer: The second layer has n units.
For j = 1 , . . . , nd , with a slight abuse of notation we now use e j for the j th standard basis vector in R nd and define unit i = 1 , . . . , n , σ 2 i : R nd → R , by,
[Equation]
Explicitly, w 2 i = (cid:80) d r =1 e d ( i − 1)+ r and b 2 i = d .
With this construction in hand, the following is the main property of the first two layers.
Lemma 4.
Let i = 1 , . . .
, n .
Then, [ E ( x )] i =
if and only if x ≥ x i .
Otherwise, [ E ( x )] i = 0 .
Proof.
By construction, we have [ E ( x )] i =
if
and only if (cid:80) dr =1 σ 1 d ( i − 1)+ r ( x ) ≥ d .
For each r ∈ [ d ] , σ 1 d ( i − 1)+ r ( x ) ∈ { 0 , 1 } .
Thus, [ E ( x )] i =
if and only if, for every r ∈
[ d ] , σ 1 d ( i − 1)+ r ( x ) = 1 .
But σ 1 d ( i − 1)+ r ( x ) = 1 is equivalent to [ x ] r ≥ [ x i ] r .
Since this must hold for every r ∈ [ d ] , we conclude x ≥ x i .
The following corollary is now immediate.
Corollary 4.
Fix j ∈ [ n ] and let i ∈ [ n ] .
If j < i , then [ E ( x j )]
i = 0 .
If j ≥ i , then there exists i ′ ≥ i such that [ E ( x j )] i ′ =
Proof.
For the first item, if j < i , by the ordering of the labels (2), we know that x j ≱ x i .
By
Lemma 4, [ E ( x j )] i = 0 .
For the second item, by construction, [ E ( x j )] j = 1 .
Since j ≥ i , the claim concludes.
The third hidden layer: The third layer contains n units with weights given by w 3 i = (cid:80) nr = i e r and b 3 i = 1 .
Thus,
[Equation]
Lemma 5.
Fix j ∈ [ n ] and let i ∈ [ n ] .
σ 3 i ( E ( x j ))
=
if j ≥ i and σ 3 i ( E ( x j )) = 0 otherwise.
Proof.
By Corollary 4, [ E ( x j )] r = 0 , for every r > j .
In particular, if i > j , by (3), we get, σ 3 i ( E ( x j ))
=
σ ( − 1) = 0 .
On the other hand, if j ≥ i , then by Corollary 4, there exists i ′ ≥
i such that [ E ( x j )] i ′ =
1 , and, σ 3 i ( E ( x j ))
=
σ ([ E ( x j )]
i ′ − 1)
= 1 .
The final layer: The fourth and final layer is a linear functional of the output of the third layer.
Formally, for x ∈ R d , the output of the network is, N ( x ) = n (cid:80)
i =
[ w 4 ] i σ 3 i ( E ( x )) , for some weights vector w 4 ∈ R .
To complete the construction we now define the entries of w 4 , as [ w 4 ] i =
y
i − y i −with y
= 0 .
We are now ready to prove Theorem 1.
Proof of Theorem 1.
Consider the function N ( x ) = n (cid:80) [-i =
-][Equation] =1 [
[Equation] 4 ] i σ 3 i ( E ( x )) described above.
Clearly, it is a network with 3 hidden layers, by construction.
To see that it is monotone, observe that for ℓ ∈ { 1 , 2 , 3 } each w ℓi is a sum of standard basis vectors, and thus has non-negative entries.
The weight vector w 4 also has non-negative entries, since, by assumption, y i ≥
y i − 1 .
We now show that N interpolates the data set ( x j , y j ) nj =1 .
Indeed, fix j ∈ [ n ] .
By Lemma 5, we have, N[-( x j ) = n (cid:80) i =
 [ w 4 ] i σ 3 i ( E ( x j )) = j (cid:80) i =
-]
[-( y i − y i − 1 ) = y j − y 
 = y j .-][Equation]
The proof is complete, for the general case.
To handle the case of totally-ordered ( x i ) i ∈ [ n ] , we slightly alter the construction of the first two layers, and compress them into a single layer satisfying Lemma 4, and hence Lemma 5.
The total-order of ( x i ) i ∈ [ n ] implies the following:
For every i ∈ [ n ] , there exists r ( i ) ∈
[ d ] , such that for any j ∈ [ n ] , [ x i ] r ( i ) <
[ x j ] r ( i ) if and only if i < j .
In words, for every point in the set, there exists a coordinate which separates it from all the smaller points.
We thus define w 1 i =
e r ( i ) and b i = 1 .
From the above it is clear that σ
[Equation]
As in the general case, we define E ( x ) : R d → R n by [ E ( x )] i
= σ 1 i ( x ) , and note that Lemma 4 holds for this construction of E as well.
The next two layers are constructed exactly like in the general case and the same proof holds.
We conclude this section with two comments.
First, as noted, our interpolation scheme can be used to prove that any continuous monotone function can be approximated by a monotone threshold network which is Theorem 2.
A proof can be found in Section A.3.
Second, interpolating monotone networks are wide: It turns out, Ω( n ) neurons may be needed for monotone interpolation.
A proof of this (Lemma 3) can be found in Section A.2.
For general threshold networks (no restriction on the weights) one can interpolates n points using O ( √ n + f ( δ )) neurons where f depends on the minimal distance between any two of the data points [30, 41].
Hence Lemma 3 shows that there exist monotone data sets such that interpolating them with a monotone network entails a quadratic blowup in the size of the network even if the data set is well separated.
A super polynomial separation between the size of monotone and arbitrary threshold networks
By the universal approximation result for monotone threshold networks, Theorem 2, we can approximate monotone functions by monotone networks.
Are there functions such that monotone networks approximating them provably require a much larger size than networks which are allowed to have negative parameters as well?
We show that the answer is positive when seeking an ℓ ∞ - approximation smaller than ε for any ε ∈ [0 , 1 / 2) .
Our proof of this fact builds on findings from monotone complexity theory of Boolean functions.
Given an undirected graph G =
( V, E ) with 2 n vertices a matching M is a set of pairwise disjoint edges.
A perfect matching is a matching of size n (which is largest possible).
There are efficient algorithms for deciding if a bipartite graph has a perfect matching [23].
Furthermore, by standard results that convert Turing machines that decide an algorithmic problem with inputs of size n in time t ( n ) to threshold circuits with O ( t ( n ) 2 ) gates, [33, 37], it follows that there is a network of size polynomial in n that decides, given the incidence matrix of a graph, whether it has a perfect matching.
A seminal result by Rzaborov [32] shows that the monotone complexity of the matching function is not polynomial.
Theorem 5.
Let g be the Boolean function that receives the adjacency matrix of a 2 n -vertex graph
G and decides if G has a perfect matching.
Then, any Boolean circuit with AND and OR gates that computes g has size n Ω(log n ) .
Furthermore, the same lower bound applies if the graph is restricted to be a bipartite graph G ( A, B, E ) where | A | = | B | = n is the bi-partition of the graph.
Definition 6 (Matching probabilities in non-homogeneous random bipartite graphs) .
Let p = ( p ij ) ni,j =1 ∈ [0 , 1] n × n and define G ( p ) to be a random bipartite graph on vertex set [ n ] × [ n ] , such that each edge ( i, j ) appears independently with probability p ij , for every 1 ≤ i, j ≤ n .
Define m : [0 , 1] n × n → [0 , 1] as 6 , m ( p ) =
P ( G ( p ) contains a perfect matching ) .
When p ∈ { 0 , 1 } n × n , m ( p ) reduces to the indicator function of a perfect matching in a given bipartite graph.
Thus m ( p ) should be thought of as the harmonic (or multi-linear) extension of the indicator function to the solid cube [0 , 1] n × n .
Theorem 3 is an immediate consequence of the following more specific theorem, which is our main result for this section.
Theorem 7.
The function m defined above is a smooth monotone function with Lipschitz constant ≤ n , which satisfies the following: If N is a monotone threshold network of size
e o ((log n ) 2 ) , there exists p ∈ [0 , 1] n × n , such that, | N ( p ) − m ( p ) | ≥ 12 .
Furthermore, for every fixed ε > 0 there exist a general threshold network N of polynomial size in n , such that for all p ∈ [0 , 1] n × n , | N ( p ) − m ( p ) | ≤ ε.
Our proof of Theorem 7 is divided into three parts.
We first establish the relevant properties of m .
We then show that m cannot be approximated by a monotone network of polynomial size.
Finally, we show that m can be approximated, arbitrarily well, by a general network with polynomial size.
We begin by collecting several simple facts about the function m .
All missing proofs can be found in Section A.4.
A standard coupling argument (omitted) shows that m is monotone.
We have that m is Lipschitz continuous: Let p , p ′ ∈ [0 , 1] n × n .
Then, | m ( p ) − m ( p ′ )
| ≤ n ∥ p − p ′ ∥ .
This is a simple consequence of the fact that m is the harmonic extension of a bounded function.
The fact that m is smooth can again be seen from the multi-linear extension.
Proofs that m has these properties can be found in Section A.4.1.
The proof (in Section A.4.2) that the function m cannot be approximated by a monotone network of polynomial size uses the fact [4] that a threshold gate with s inputs, non negative parameters and Boolean inputs can be computed by a monotone De Morgan circuit of size poly ( s ) .
Therefore, if there was a monotone network of size e o ((log n ) 2 ) approximating m ( x ) we could replace each gate with a monotone De Morgan circuit entailing a polynomial blowup to the size of the network.
This in turn would imply the existence of a monotone
De Morgan circuit of size e o ((log n ) 2 ) computing m over Boolean inputs which would contradict Theorem 5.
Summarizing:
Lemma 6.
If N is a monotone threshold network of size e o ((log n ) 2 ) , there exists p ∈ [0 , 1] n × n , such that, | N ( p ) − m ( p ) | ≥ 12 .
Finally, we show how to approximate m with a network (without weight restrictions) of polynomial size.
To estimate the probability a graph G ( p ) drawn according to a probability vector p has a perfect matching we can realize independently (polynomially) many copies of graphs distributed as
G ( p ) and estimate the number of times a perfect matching is detected.
In order to implement this idea two issues need to be addressed: the use of randomness by the algorithm (our neural networks do not use randomness) and the algorithm dealing with probability vectors in [0 , 1] n × n that may need infinitely many bits to be represented in binary expansion.
We first present a randomized polynomial-time algorithm, denoted A , for approximating m ( p ) .
We then show how to implement it with a (deterministic) threshold network.
Algorithm A works as follows.
Let q () , r () be polynomials to be defined later.
First, the algorithm (with input p ) only considers the q ( n ) most significant bits in the binary representation of every coordinate in p .
Next it realizes r ( n ) independent copies of G ( p ) .
It checks 8 for each one of these copies whether it contains a perfect matching of size n .
Let t be the number of times a perfect matching is detected ( t depends on p : We omit this dependency to lighten up notation).
The algorithm outputs A ( p ) :
= t r ( n ) .
Clearly the running time of this algorithm is polynomial.
Let (cid:101) p be the vector obtained from p when considering the q ( n ) most significant bits in each coordinate, and observe A ( p ) law =
A ( (cid:101) p ) .
We next show how to implement this algorithm by a neural network of polynomial size that does not use randomness.
For a proof please see Section A.4.3.
Lemma 7.
Let δ ∈ (0 , 1) be a fixed constant.
Then, if ˜ p ∈ [0 , 1] n × n is such that every coordinate ˜p ij can be represented by at most q ( n ) bits there exists a neural network of polynomial size N such that for every p ∈ [0 , 1] n × n , | m ( p ) − N ( p )) | ≤ δ + [Equation]
We can now prove the following Lemma concluding the proof of Theorem 7.
Lemma 8.
For every fixed ε > 0 there exist a general threshold network N of polynomial size in n , such that for all p ∈ [0 , 1] n × n , | N ( p ) − m ( p ) | ≤ ε.
Proof.
Set δ =
ε 2 and let N be the network constructed in Lemma 7 with accuracy parameter δ .
Choose q ( n ) which satisfies q ( n ) > log( 4 n 4 ε 2 ) .
Thus, Lemma 7 implies, for every p ∈ [0 , 1] n × n :
[Equation]
Conclusion
We studied neural networks with nonnegative weights and examined their power and limitation in approximating monotone functions.
Our results reveal that for the ReLU activation, restricting the weights to be nonnegative severally limits the ability of the model to express monotone functions. For threshold activation we have shown that the restriction to positive parameters is less severe and that universality can be achieved at constant depth. In addition, we have shown that monotone neural networks can be much more resource consuming, in terms of the number of neurons needed to approximate monotone functions. We focused on the threshold activation function.
It is an interesting direction to extend our results for other activation functions such as sigmoids. For the universality result of depth 4 monotone function it seems plausible one could approximate thresholds by sigmoids to prove that monotone networks of depth 4 with sigmoids are universal approximators of monotone functions. For our lower bounds based on the matching function m it appears that new ideas are needed to show a super polynomial separation between the size needed for monotone as opposed to arbitrary networks with sigmoids to approximate m . The field of Boolean circuit complexity has been used before to address theoretical questions related to neural networks (e.g., [6, 16, 28, 34]) and we believe that additional insights could be found by studying the intersection of circuit complexity and deep learning.
With regard to monotone neural networks it is likely that stronger lower bounds can be proved when the depth of the network is bounded.
Proving separation between monotone and non monotone networks with respect to the square loss is another avenue for further research.
One aspect we did not consider here is learning neural networks with positive parameters using gradient descent.
It would be interesting to examine the efficacy of gradient methods both empirically and theoretically.
Such study could lead to further insights regarding methods that ensure that a neural network approximating a monotone function is indeed monotone.
Finally, we did not deal with generalization properties of monotone networks: Devising tight generalization bounds for monotone networks is left for future study.
A Omitted proofs A.1 Counter-example to expressibility when d ≥
Proof of Lemma 2.
Consider the following monotone set of d + 1 data points in R d with d ≥ 2 . For i ∈ [ d ] , x i = d · e i , is a vector whose i th coordinate is d and all other coordinates are 0 , and set y i = 0 . We further set x d +1 = , 1) (the all 1 ’s vector) with y d +1 = 1 .
Suppose towards a contradiction there is a monotone depth- 2 threshold network
N ( x ) =
r (cid:88) m = a m σ ( ⟨ x, w m ⟩ − b m ) , with N ( x d +1 ) = 1 and for every i ∈ [ d ] , N ( x i ) = 0 .
We prove the result while assuming that the bias of the output layer is 0 . Since the bias just adds a constant to every output it is straightforward to take it into account.
Denote,
[Equation]
Since N ( x d +1 ) = 1 , we have that I is non-empty. For j ∈ [ d ] , let I j := { m ∈ I | [ w m ] j = max { [ w m ] 1 , . . .
, [ w m ] d }} .
Clearly I = d (cid:83) j =1 I j and we can assume, with no loss of generality, that this is a disjoint union.
Now, by following the exact same logic as in (1), [Equation] Therefore there exists j ∈ [ d ] with N ( x j ) ≥ 1 d > 0 , which is a contradiction. A.2 Lower bounds
Proof of Lemma 3.
Let ( x i , y i )
i ∈ [ n ] ∈ (cid:0) R d × R (cid:1)
{+n be a monotone data set, such that for every 1 ≤ i ≤ n − 1 , x i ≤ x i +
+}
and y i ̸
= y i +1 . Suppose that the first layer of N has k units and denote it by N 1 : R d → R k .
Now, let 1 ≤ i < j ≤ n .
Since, every unit is monotone, and since x i ≤ x j , we have for ℓ ∈ [ k ] the implication [Equation] Denote I i = { ℓ ∈ [ k ] | [ N 1 ( x i )] ℓ ̸ = 0 } .
The above shows the following chain condition,
[Equation]
Suppose that k < n , then as { I i } ni =1 is an ascending chain of subsets in [ k ] , necessarily there exists an i < n , such that I
i
= I i +1 , which implies N 1 ( x i ) = N 1 ( x i +1 ) .
We conclude that
N ( x i ) =
N ( x i +1 ) , which cannot happen as y i ̸
= y i +1 . Hence, necessarily k > n . A.3 Universal approximation
Proof of Theorem 2. Since f is continuous and [0 , 1] d is compact, f is uniformly continuous. Hence, there exists δ > 0 , such that [Equation]
Set δ d = δ √ d and consider the grid,
[Equation]
I.e. G δ is a uniform grid of points with spaces of width δ d . Define now a monotone data set ( x j , f ( x j )) x j ∈ G δ . By Theorem 1 there exists a network N , such that N ( x j ) =
f ( x j ) for every x j ∈ G δ . We claim that this is the approximating network. Let x ∈ [0 , 1] d and let x − (resp.
x + ) be the closest point to x in G δ such that x ≥ x − (resp. x ≤ x + ). Observe that, x − and x + are vertices of a sub-cube in [0 , 1] d of side length δ d . Thus, ∥ x − − x + ∥ ≤ √ dδ d = δ.
Now, since both N and f are monotone,
[Equation]
Assume now that f is L -Lipschitz and Let us now estimate the size of the network.
According to
Theorem 1, the network has O ( d | G δ | ) neurons.
Since f is L -Lipschitz, one can control the uniform continuity parameter and take δ = εL .
Hence,
[Equation]
A.4.1 Properties of m
Towards the proof of Theorem 7, we start by collecting several facts about the function m .
The first one follows from standard coupling arguments.
We omit the proof.
Claim 8.
m is monotone.
We now show that m is smooth and Lipschitz continuous.
Those are immediate consequences of the fact that m is the harmonic extension of a bounded function.
For completeness, below, we give more self-contained arguments.
Lemma 9.
Let p , p ′ ∈ [0 , 1] n × n .
Then,
[Equation]
In other words, m is n -Lipschitz.
Proof.
Let p , p ′′ ∈ [0 , 1] n × n be such that p − p ′′ = ρe ij , where, for i, j ∈ [ n ] , e ij is a standard basis vector in R n × n and ρ > 0 .
Let U := { U ij } ni,j =1 be i.i.d. random variables, uniformly distributed on [0 , 1] . We couple the random graphs, G ( p ) , G ( p ′′ ) in the following way,
[Equation]
We slightly abuse notation, and write m ( G ( p )) for the indicator that G ( p ) contains a perfect matching (and similarly for G ( p ′′ ) ). It is clear that,
[Equation]
Moreover if G ij is the sigma algebra generated by U \ { U ij } , we have.
[Equation]
Since G ( p ) and G ( p ′′ ) agree on every edge other than ( i, j ) , it is readily seen that, almost surely, E [ m ( G ( p )) − m ( G ( p ′′ ))) | G ij ] ≤ ρ . Hence, [Equation] p
[Equation]
m
[Equation]
p
[Equation]
| ≤ ρ. For general p , p ′ ∈ [0 , 1] n × n with Cauchy-Schwartz’s inequality, we conclude,
[Equation]
Lemma 10. The function m is a C ∞ function. Proof. We sketch the proof.
Enumerate all n ! matching of the complete bipartite graph with A = B = n and denote this set by M . For M ∈ M let A M be the event that M occurs in a graph
G ( p ) sampled according to the input vector p . Then the probability G ( p ) has a perfect matching is P = Pr ( (cid:83) M ∈M A M ) .
Since each edge occurs independently we have using inclusion-exclusion that P is a sum of polynomials in the coordinates of p . The claim follows. A.4.2 The function m cannot be approximated by a polynomial monotone network Proof of Lemma 6.
Suppose towards a contradiction that there is a monotone network N of size s = e o (log( n ) 2 ) that approximates m in [0 , 1] n × n within error less than 12 . Then, restricting N to Boolean inputs, and applying a threshold gate to the output, based on whether the output is larger than 12 would yield a monotone threshold circuit C N that computes m (exactly) on Boolean inputs and has size s + 1 .
It is known that the monotone circuit complexity of a circuit with AND and OR gates (De Morgan circuit) computing a threshold function with positive coefficients is polynomial [4]. Therefore, we claim that, the existence of C N entails the existence of a monotone circuit C ′ N with AND and OR gates of size 9 O ( s t ) , for some constant t ≥ 1 , that decides if a bipartite graph has a perfect matching. Indeed, simply replace every positive threshold gate, g , in C N by a monotone De Morgan circuit C g computing g . As the size of C g is polynomial in the number of inputs to g (and hence upper bounded by s k for an appropriate constant k ) these replacements result in a polynomial blowup with respect to the size of C N : the size of C ′ N is at most ( s + 1) s k
= O ( s k +1 ) .
Therefore setting t to be k +we have that the size of C ′ n is at most O ( s t ) = e t · o (log 2 n ) = e o (log 2 n ) .
Since e log( n ) 2 = n log( n ) , this contradicts Theorem 5.
A.4.3 Approximating m with a general network
This subsection is devoted to the proof of Lemma 7. Recall that A is a polynomial time random algorithm, that approximates m . Moreover, on input p , A first truncates every coordinate of p to the first q ( n ) most significant bits. Therefore, if (cid:101) p is the vector obtained from p by truncating bits, we have an equality in law,
[Equation]
Keeping this in mind, we shall first require the following technical result.
Lemma 11.
Let δ ∈ (0 , 1) be an accuracy parameter.
Then, if ˜p ∈ [0 , 1] n × n is such that every coordinate ˜p ij can be represented by at most q ( n ) bits,
[Equation]
As a consequence, for every p ∈ [0 , 1] n × n ,
[Equation]
Proof.
We first establish the first part.
Indeed, since A ( ˜p ) can be realized a linear combination of i.i.d.
Bernoulli’s and since E [ A ( ˜p )]
= m ( ˜p ) , (5) is a consequence of Chernoff’s inequality and follows directly from the discussion in [27, Page 73].
Now, ˜p was obtained from p by keeping the q ( n ) most significant bits.
Thus, ∥ p − ˜p ∥ ≤ (cid:113) n 2 2 q ( n ) , and by Lemma 9,
[Equation]
So, because of (4), and (5),
[Equation]
We may now prove Lemma 7.
Proof of Lemma 7.
By standard results regarding universality of Boolean circuits [33,37] algorithm
A can be implemented by a neural network N ′ of polynomial size that get as additional input n 2 q ( n ) r ( n ) random bits.
In light of this, our task is to show how to get rid of these random bits.
The number of points in [0 , 1] n × n represented by at most q ( n ) bits in each coordinate is at most 2 n 2 q ( n ) .
On the other hand by the first part of Lemma 11 the probability for a given input ˜p (with each coordinate in ˜p represented by at most q ( n ) bits) that | N ′
( ˜p ) − m ( ˜p ) | ≥ δ is 2 − Ω( r ( n )) .
As long as r ( n ) is a polynomial with a large enough degree, with respect to q ( n ) , we have that 2 n 2 q ( n ) − Ω( r ( n )) < 1 .
Hence, there must exist a fixed choice of for the n 2 q ( n ) r ( n ) (random) bits used by the algorithm such that for every input p the additive error of the neural network N ′ on p is no larger than δ .
Fixing these bits and hard-wiring them to N ′ results in the desired neural network N.
The final result follows from the proof of the second part of Lemma 11.