O N THE O PTIMAL M EMORIZATION P OWER OF R E LU N EURAL N ETWORKS
A BSTRACT
We study the memorization power of feedforward ReLU neural networks.
We show that such networks can memorize any N points that satisfy a mild separability assumption using ˜ O (cid:16) √ N (cid:17) parameters.
Known VC-dimension upper bounds imply that memorizing N samples requires Ω( √ N ) parameters, and hence our construction is optimal up to logarithmic factors.
We also give a generalized construction for networks with depth bounded by 1 ≤ L ≤ √ N , for memorizing N samples using ˜ O
( N/L ) parameters.
This bound is also optimal up to logarithmic factors.
Our construction uses weights with large bit complexity.
We prove that having such a large bit complexity is both necessary and sufﬁcient for memorization with a sub-linear number of parameters.
I NTRODUCTION
The expressive power of neural networks has been widely studied in many previous works.
These works study different aspects of expressiveness, such as the universal approximation property (Cybenko, 1989; Leshno et al., 1993) and the beneﬁts of depth in neural networks (Telgarsky, 2016;
Eldan and Shamir, 2016; Safran and Shamir, 2017; Daniely, 2017; Chatziafratis et al., 2019).
Another central and well studied question is about their memorization power.
The problem of memorization in neural networks can be viewed in the following way: For every dataset of N labeled samples ( x 1 , y 1 ) , . . .
, ( x N , y N ) ∈
X × Y , construct a network N : X → Y such that N ( x i ) =
y i for every i = 1 , . . .
, N .
Many works have shown results regarding the memorization power of neural networks, using different assumptions on the activation function and data samples (see e.g. Huang and Babri (1998);
Huang (2003); Baum (1988); Vershynin (2020);
Daniely (2019; 2020); Bubeck et al.
; Park et al.
(2020); Hardt and Ma (2016)
; Yun et al.
; Zhang et al.
(2021); Nguyen and Hein (2018)
; Rajput et al.
(2021); Sontag (1997)).
The question of memorization also have practical implications on phenomenons such as ”double descent” (Belkin et al.
; Nakkiran et al.
(2019)) which connects the memorization power of neural networks with their generalization capabilities.
A trivial lower bound on the required size of the network for memorizing N labeled points is implied by the VC dimension of the network (cf.
Shalev-Shwartz and Ben-David (2014) ).
That is, if a network with a certain size cannot shatter any speciﬁc set of N points, then it certainly cannot memorize all sets of N points.
Known VC dimension bounds for networks with W parameters is on the order of O ( W 2 )
(Goldberg and Jerrum (1995); Bartlett et al.
Hence, it follows that memorizing N samples would require at least Ω (cid:0) N 1 / 2 (cid:1) parameters.
The best known upper bound is given in Park et al.
(2020), where it is shown that memorizing N data samples can be done using a neural network with O (cid:0) N 2 / 3 (cid:1) parameters.
Thus, there is a clear gap between the lower and upper bounds, although we note that the upper bound is for memorization of any set of data samples, while the lower bound is for shattering a single set of data samples.
In this paper we ask the following questions:
What is the minimal number of parameters that are required to memorize N labeled data samples?
Is the task of memorizing any set of N data samples more difﬁcult than shattering a single set of N samples?
We answer these questions by providing a construction of a ReLU feedforward neural network which achieves the lower bound up to logarithmic factors.
In this construction we use a very deep neural network, but with a constant width of 12 .
In more details, our main result is the following:
Theorem 1.1 (informal statement) .
Let ( x 1 , y 1 ) , . . .
, ( x N , y N ) ∈ R
d × { 1 , . . .
, C } be a set of N labeled samples of a constant dimension d , with (cid:107) x i (cid:107) ≤ r for every i and (cid:107) x
i − x j (cid:107) ≥ δ for every i (cid:54) =
j .
Then, there exists a ReLU neural network F : R d → R with width 12 , depth ˜ O (cid:16) √ N (cid:17) , and ˜ O (cid:16) √ N (cid:17) parameters, such that F ( x i ) =
y i for every i ∈ [ N ] , where the notation ˜ O ( · ) hides logarithmic factors in N, C, r, δ − 1 .
Comparing this result to the known VC bounds, we show that, up to logarithmic factors, our construction is optimal.
This also shows, quite surprisingly, that up to logarithmic factors, the task of memorizing any set of N points is not more difﬁcult than shattering a single set of N points, under the mild separability assumption of the data samples.
We note that this result can also be extended to regression tasks (see Remark 3.3).
In our construction, the depth of the network is ˜Θ (cid:16) √ N (cid:17) .
We also give a generalized construction where the depth of the network is limited to some 1 ≤ L ≤ √ N .
In this case, the number of parameters in our construction is ˜ O (cid:0) NL (cid:1) (see Theorem 5.1).
We compare this result to the VC-dimension bound from Bartlett et al.
(2019), and show that our construction is optimal up to logarithmic factors.
Our construction uses a bit extraction technique, inspired by Telgarsky’s triangle function (Telgarsky (2016)), and by Safran and Shamir (2017).
Using this technique, we are able to use weights with bit complexity ˜Θ( √ N ) , and deep neural networks to “extract” the bits of information from the specially crafted weights of the network.
We also generalize our results to the case of having a bounded bit complexity restriction on the weights.
We show both lower (Theorem 6.1) and upper (Theorem 6.2) bounds, proving that memorizing N points using a network with N 1 − (cid:15) parameters, for some (cid:15) ∈ [0 , 0 . 5] can be done if the bit complexity of each weight is ˜Θ ( N (cid:15) ) .
Hence, our construction is also optimal, up to logarithmic factors, w.r.t the bit complexity of the network.
We emphasize that also in previous works showing non-trivial VC bounds (e.g. Bartlett et al.
(1998; 2019)) weights with large bit complexity are used.
We note that increasing the bit complexity beyond
N 1 / 2 cannot be used to further reduce the number of parameters (see the discussion in section 4).
R ELATED WORK M EMORIZATION – UPPER BOUNDS
The problem of memorizing arbitrary data points with neural networks has a rich history.
Baum (1988) studied memorization in single-hidden-layer neural networks with the threshold activation, and showed that (cid:100) Nd (cid:101) neurons sufﬁce to memorize N arbitrary points in general position in R d with binary labels.
Bubeck et al.
(2020) extended the construction of Baum (1988) and showed that single-hidden-layer ReLU networks with 4 · (cid:100) Nd (cid:101) hidden neurons can memorize N points in general position with arbitrary real labels.
In Huang et al.
and Sartori and Antsaklis (1991) it is shown that single-hidden-layer networks with the threshold activation can memorize any arbitrary set of N points, even if they are not in general position, using N − 1 neurons.
Huang and Babri (1998) proved a similar result for any bounded non-linear activation function σ where either lim z →∞ σ ( z ) or lim z →−∞ σ ( z ) exists.
Zhang et al.
(2021) proved that single-hidden-layer ReLU networks can memorize arbitrary N points in R d with arbitrary real labels using N neurons and 2 N + d parameters.
Huang (2003) showed that two-hidden-layers networks with the sigmoid activation can memorize
N points with O ( √ N ) neurons, but the number of parameters is still linear in N .
Yun et al.
(2019) proved a similar result for ReLU (and hard-tanh) networks.
Vershynin (2020) showed that threshold and ReLU networks can memorize N binary-labeled unit vectors in R d separated by a distance of δ > 0 , using ˜ O (cid:16) e 1 /δ 2 + √ N (cid:17) neurons and ˜ O (cid:16) e 1 /δ 2 ( d + √ N ) + N (cid:17) parameters.
Rajput et al.
(2021) improved the dependence on δ by giving a construction with ˜ O (cid:16) 1 δ + √ N (cid:17) neurons and ˜ O (cid:0) dδ + N (cid:1) parameters.
This result holds only for threshold networks, but does not assume that the inputs are on the unit sphere.
The memorization power of more speciﬁc architectures was also studied.
Hardt and Ma (2016) proved that residual ReLU networks with O ( N ) neurons can memorize N points on the unit sphere separated by a constant distance.
Nguyen and Hein (2018) considered convolutional neural networks and showed, under certain assumptions, memorization using O ( N ) neurons.
Note that in all the results mentioned above the number of parameters is at least linear in N .
Our work is inspired by Park et al.
(2020), that established a ﬁrst memorization result with a sub-linear number of parameters.
They showed that neural networks with sigmoidal or ReLU activations can memorize N points in R d separated by a normalized distance of δ , using O (cid:0) N 2 / 3 + log(1 /δ )
(cid:1) parameters (where the dimension d is constant).
Thus, in this work we improve the dependence on
N from N 2 / 3 to √ N (up to logarithmic factors), which is optimal.
We also note that the ﬁrst stage in our construction is similar to the ﬁrst stage in theirs.
In Park et al.
there is also a discussion about the bit complexity of the network, and the necessity that the bit complexity depends on N to achieve memorization with a sub-linear number of parameters.
Finally, optimization aspects of memorization were studied in Bubeck et al.
(2020); Daniely (2019; 2020).
M EMORIZATION – LOWER BOUNDS
Moreover, lower bounds on the number of parameters required for memorization are implied by bounds on the VC dimension of neural networks.
Indeed, if W parameters are not sufﬁcient for shattering even a single set of size N , then they are clearly not sufﬁcient for memorizing all sets of size N .
The VC dimension of neural networks has been extensively studied in recent decades (cf.
Anthony and Bartlett (2009); Bartlett et al.
The most relevant results for our work are by Goldberg and Jerrum (1995) and Bartlett et al.
We discuss these results and their implications in Sections 4 and 5.
Trade-offs between the number of parameters of the network and the Lipschitz parameter of the prediction function in memorizing a given dataset are studies in Bubeck et al.
; Bubeck and
Sellke (2021).
T HE BENEFITS OF DEPTH
In this work we show that deep networks have signiﬁcantly more memorization power.
Quite a few theoretical works in recent years have explored the beneﬁcial effect of depth on increasing the expressiveness of neural networks (e.g., Martens et al.
; Eldan and Shamir (2016); Telgarsky (2016); Liang and Srikant (2016); Daniely (2017); Safran and Shamir (2017); Yarotsky (2017);
Safran et al.
; Chatziafratis et al.
(2019); Vardi and Shamir (2020);
Bresler and Nagaraj (2020);
Venturi et al.
(2021); Vardi et al.
The beneﬁts of depth in the context of the VC dimension is implied by, e.g., Bartlett et al.
Finally, Park et al.
(2020) already demonstrated that deep networks have more memorization power than shallow ones, albeit with a weaker bound than ours.
P RELIMINARIES N OTATIONS
For n ∈ N and i ≤ j we denote by BIN i : j ( n ) the string of bits in places i until j inclusive, in the binary representation of n and treat it as an integer (in binary basis).
For example, BIN 1:3 (32) = 4 .
We denote by LEN ( n ) the number of bits in its binary representation.
We denote BIN i ( n ) :=
BIN i : i ( n ) .
For a function f
and i ∈
N we denote by f ( i ) the composition of f with itself i times.
We denote vectors in bold face.
We use the ˜ O ( N ) notation to hide logarithmic factors, and use O ( N ) to hide constant factors.
For n ∈ N we denote [ n ] := { 1 , . . . , n } .
We say that a hypothesis class H shatters the points x 1 , . . .
, x N ∈ X if for every y 1 , . .
, y N ∈ {± 1 } there is h ∈ H
s.t.
h ( x i ) =
y i for every i = 1 , . . .
, N .
N EURAL N ETWORKS
We denote by σ ( z ) :
= max { 0 , z } the ReLU function.
In this paper we only consider neural networks with the ReLU activation.
Let d ∈ N be the data input dimension.
We deﬁne a neural network of depth L as N : R d → R where N ( x ) is computed recursively by
• h (1) =
σ (cid:0) W (1) x
+ b (1) (cid:1) for W (1) ∈
R m 1 × d , b (1) ∈
R m• h
( i ) =
σ (cid:0) W ( i ) h
( i − 1)
+ b ( i ) (cid:1) for W ( i ) ∈
R m i × m i − 1 , b ( i ) ∈
R m i for i = 2 , . . .
, L −• N ( x ) =
h ( L ) =
W ( L ) h
( L − 1)
+ b ( L ) for W ( L ) ∈ R 1 × m L − 1 , b ( L ) ∈
R
The width of the network is max { m 1 , . . .
, m L − 1 } .
We deﬁne the number of parameters of the network as the number of weights of N which are non-zero.
It corresponds to the number of edges when we view the neural network as a directed acyclic graph.
We note that this deﬁnition is standard in the literature on VC dimension bounds for neural networks (cf.
Bartlett et al.
B IT C OMPLEXITY
We refer to the bit complexity of a weight as the number of bits required to represent the weight.
Throughout the paper we use only weights which have a ﬁnite bit complexity.
Speciﬁcally, for n ∈ N , its bit complexity is (cid:100) log n (cid:101) .
The bit complexity of the network is the maximal bit complexity of its weights.
I NPUT D ATA S AMPLES
In this work we assume that we are given N labeled data samples ( x 1 , y 1 ) , . . .
, ( x N , y N ) ∈ R
d × [ C ] , and our goal is to construct a network F : R d → R such that F ( x i ) =
y i for every i ∈ [ N ] .
We will assume that there is a separation parameter δ > 0 such that for every i (cid:54) =
j we have (cid:107) x
i − x j (cid:107) ≥ δ .
We will also assume that the data samples have a bounded norm, i.e. there is r > 0 such that (cid:107) x
i (cid:107) ≤ r for every i ∈ [ N ] .
Note that any neural network that does not ignore input neurons must have at least d parameters.
Existing memorization results (e.g., Park et al.
(2020)) assume that d is constant.
As we discuss in Remark 3.2, in our work we assume that d is at most O ( √ N ) .
We also note that since our bounds depend logarithmically on δ , then even if δ depends polynomially on N it will not change our asymptotic bounds.
M EMORIZATION U SING ˜ O (cid:16) √ N (cid:17) PARAMETERS
In this section we prove that given a ﬁnite set of N labeled data points, there is a network with ˜ O (cid:16) √ N (cid:17) parameters which memorizes them.
Formally, we have the following:
Theorem 3.1.
Let N, d, C ∈ N , and r, δ > 0 , and let ( x 1 , y 1 ) , . . .
, ( x N , y N ) ∈ R
d × [ C ] be a set of N labeled samples with (cid:107) x i (cid:107) ≤ r for every i and (cid:107) x
i − x j (cid:107) ≥ δ for every i (cid:54) =
j .
Denote
R :=
10 rN 2 δ − 1 √ πd .
Then, there exists a neural network F : R d → R with width 12 and depth
[Equation]
such that F ( x i ) =
y i for every i ∈ [ N ] .
From the above theorem, we get that the total number of parameters is ˜ O (cid:16) d + √ N (cid:17) .
For d =
O ( √ N ) (see Remark 3.2 below) we get the lower bound presented in Theorem 1.1.
Note that except for the number of data points N and the dimension d , the number of parameters of the network depends only logarithmically on all the other parameters of the problem (namely, on C, r, δ − 1 ).
We also note that the construction can be improved by some constant factors, but we preferred simplicity over minor improvements.
Speciﬁcally, we hypothesize that it is possible to give a construction using width 3 (instead of 12 ) similar to the result in Park et al.
To simplify the terms in the theorem, we assume that r ≥ 1 , otherwise (i.e., if all data points have a norm smaller than 1 )
we
just ﬁx r := 1 and get the same result.
In the same manner, we assume that δ ≤ 1 , otherwise we just ﬁx δ := 1 .
Remark 3.2 (Dependence on d ) .
Note that any neural network that does not ignore input neurons must have at least d parameters.
In our construction the ﬁrst layer consists of a single neuron (i.e width 1), which means that the number of parameters in the ﬁrst layer is d + 1 .
Hence, this dependence on d is unavoidable .
Previous works (e.g., Park et al.
(2020)) assumed that d is constant.
In our work, to achieve the bound of ˜ O (cid:16) √ N (cid:17) parameters we can assume that either d is constant or it may depend on N with d = O (cid:16) √ N (cid:17) .
Remark 3.3 (From classiﬁcation to regression) .
Although the theorem considers multi-class classiﬁcation, it is possible to use the method suggested in Park et al.
(2020) to extend the result to regression.
Namely, if the output is in some bounded interval, then we can partition it into smaller intervals of length (cid:15) each.
We deﬁne a set of output classes C with O (cid:0) 1 (cid:15) (cid:1) classes, such that each class corresponds to an interval.
Now, we can use Theorem 3.1 to get an (cid:15) approximation of the output, while the number of parameters is linear in log (cid:0) 1 (cid:15) (cid:1) .
3. P ROOF I NTUITION
Below we describe the main ideas of the proof.
The full proof can be found in Appendix A.
The proof is divided into three stages, where at each stage we construct a network F i for i ∈ { 1 , 2 , 3 } , and the ﬁnal network is F = F 3 ◦ F 2 ◦ F 1 .
Each subnetwork F i has width O (1) , but the depth varies for each such subnetwork.
Stage I: We project the data points from R d to R .
This stage is similar to the one used in the proof of the main result from Park et al.
We use a network with 2 layers for the projection.
With the correct scaling, the output of this network on x 1 , . . .
, x N ∈ R
d are points x 1 , . . .
, x N ∈ R with | x i − x j | ≥ 2 for i (cid:54) =
j , and | x i | ≤ R for every i where
R =
O (cid:0) N 2 drδ − 1 (cid:1) .
Stage II: Following the previous stage, our goal is now to memorize ( x 1 , y 1 ) , . . .
, ( x N , y N ) ∈ R ×
[ C ] , where the x i ’s are separated by a distance of at least 2 .
We can also reorder the indices to assume w.l.o.g that x 1 < x 2 < · · · < x N .
We split the data points into (cid:112) N log( N ) intervals, each containing (cid:113) N log( N ) data points.
We also construct crafted integers w 1 . . .
, w √ N log( N ) and u 1 , . . .
, u √ N log( N ) in the following way: Note that by the previous stage, if we round x i to (cid:98) x i (cid:99) , it can be represented using log( R ) bits.
Also, each y i can be represented by at most log( C ) bits.
Suppose that x i is the k -th data point in the j -th interval (where j ∈ (cid:104)(cid:112) N log( N ) (cid:105) and k ∈ (cid:110) 0 , . .
, (cid:113) N log( N ) − 1 (cid:111) )
, then we deﬁne u j , w j such that
[Table]
[Equation]
That is, for each interval j , the number u j has log( R ) bits which represent the integral value of the k -th data point in this interval.
In the same manner, w j has log( C ) bits which represent the label of the k -th data point in this interval.
This is true for all the k data points in the j -th interval, hence w j and u j are represented with log( R ) · (cid:113) N log ( N ) and log( C ) · (cid:113) N log ( N ) bits respectively.
We construct a network F 2 : R → R 3 such that F 2 ( x i ) =
(cid:32) x i w
[Equation]
u [Equation] (cid:33) for each x i , where the i -th data point is in the j i -th interval.
The integers w j and u j are used as parameters of the network F 2 .
Stage III:
In this stage we construct a network which uses a bit extraction technique adapting Telgarsky’s triangle function (Telgarsky, 2016) to extract the relevant information out of the crafted integers from the previous stage.
In more details, given the input (cid:32) xwu (cid:33) , we sequentially extract from u the bits in places k · log( R ) + 1 until ( k + 1) · log( R ) for k ∈ (cid:110) 0 , 1 , . . .
, (cid:113) N log( N ) − 1 (cid:111) , and check whether x is at distance at most 1 from the integer represented by those bits.
If it is, then we extract the bits in places k · log( C ) + 1 until ( k + 1) · log( C ) from w , and output the integer represented by those bits.
By the previous stage, for each x i we know that u includes the encoding of (cid:98)
x i (cid:99) , and since | x i − x j | ≥ 2 for every j (cid:54) =
i (by the ﬁrst stage), there is exactly one such x i .
Hence, the output of this network is the correct label for each x i .
The construction is inspired by the works of Bartlett et al.
; Park et al.
Speciﬁcally, the ﬁrst stage uses similar results from Park et al.
(2020) on projection onto a 1 -dimensional space.
The main difference from previous constructions is in stages II and III where in our construction we encode the input data points in the weights of the network.
This is the main technique which allows us to reduce the required number of parameters from the ˜ O (cid:0) N 2 / 3 (cid:1) bound in Park et al. (2020) to ˜ O (cid:0) N 1 / 2 (cid:1) . We also note, that although we use the convention of counting the number of parameters without zero-weights, in this section we do count zero weights, hence this result is comparable to Park et al.
(2020) where they do count zero weights.
O N THE O PTIMAL N UMBER OF P ARAMETERS
In section 3 we showed that a network with width O (1) and depth ˜ O (cid:16) √ N (cid:17) can perfectly memorize
N data points, hence only ˜ O (cid:16) √ N (cid:17) parameters are required for memorization.
In this section, we compare our results with the known lower bounds.
First, note that our problem contains additional parameters besides N .
Namely, d, C, r and δ .
As we discussed in Remark 3.2, we assume that d is either constant or depends on N with d = O ( √ N ) .
In this comparison we also assume that r and δ − 1 are either constants or depend polynomially on
N .
We note that for a constant d , either r or δ − 1 must depend on N .
For example, assume d =and r = 1 / 2 , then to have N point in R d with norm at most r and distance at least δ from one another we must have that δ − 1 ≥ N − 1 .
Hence, we can bound R ≤ poly ( N ) (where R is deﬁned in Theorem 3.1), which implies log( R ) =
O (log( N )) .
Moreover, we assume that C ≤ poly ( N ) , because it is reasonable to expect that the number of output classes is not larger than the number of data points.
Hence, also log( C ) =
O (log( N )) .
In regression tasks, as discussed in Remark 3.3, we can choose C to consist of (cid:6) 1 (cid:15) (cid:7) classes where (cid:15) is either a constant or depends at most polynomially on the other parameters.
Using that R, C ≤ poly ( N ) , and tracing back the bounds given in Theorem 3.1, we get that the number of parameters of the network is O (cid:16) (cid:112) N log( N ) (cid:17) .
Note that here the O ( · ) notation only hides constant factors, which can also be exactly calculated using Theorem 3.1.
By Goldberg and Jerrum (1995), the VC dimension of the class of ReLU networks with W parameters is O ( W 2 ) .
Thus, the maximal number of points that can be shattered is O ( W 2 ) .
Hence, if we can shatter N points then W = Ω( √ N ) .
In particular, it gives a Ω( √ N ) lower bound for the number of parameters required to memorize N inputs.
Thus, the gap between our upper bound and the above lower bound is only O (cid:16)(cid:112) log( N ) (cid:17) , which is sub-logarithmic.
Moreover, it implies that the number of parameters required to shatter one size- N set is roughly equal (up to a sub-logarithmic factor) to the number of parameters required to memorize all size- N sets (that satisfy some mild assumptions).
Thus, perhaps surprisingly, the task of memorizing all size- N sets is not signiﬁcantly harder than shattering a single size- N set.
If we further assume that C depends on N , i.e., the number of classes depends on the number of samples (or in regression tasks, as we discussed in Remark 3.3, the accuracy depends on the number of samples), then we can show that our bound is tight up to constant terms.
Thus, in this case the (cid:112) log( N ) factor is unavoidable.
Formally, we have the following lemma:
Lemma 4.1.
Let C, N ∈ N , and assume that C = N (cid:15) for some constant (cid:15) > 0 .
If we can express all the functions of the form g : [ N ] → [ C ] using neural networks with W parameters, then
W = Ω (cid:16)(cid:112) N log( N ) (cid:17) .
Proof.
Let F be the class of all the functions f : [ N ] × [ (cid:15) log( N )]
→ { 0 , 1 } .
The VC-dimension bound from Goldberg and Jerrum (1995) implies that expressing all the functions from F with neural networks requires W = Ω (cid:16)(cid:112) N log( N ) (cid:17) parameters, since it is equivalent to shattering a set of size (cid:15)N log( N ) .
Assume that we can express all the functions of the form g : [ N ] →
[ C ] using networks with W (cid:48) parameters.
Given some function f ∈ F we can express it with a neural network as follows: deﬁne a function g : [ N ] → [ C ] such that for every n ∈ [ N ] and i ∈ [ (cid:15) log( N )] , the i -th bit of g ( n ) is f ( n, i ) .
We construct a neural network for f , such that for an input ( n, i ) it computes g ( n ) (using W (cid:48) parameters) and outputs its i -th bit.
The extraction of the i -th bit can be implemented using the bit extraction technique from the proof of Theorem 3.1.
Overall, this network requires O ( W (cid:48)
+ log( N )) parameters, and in this manner we can express all the functions in F .
This shows that
W =
O ( W (cid:48)
+ log( N )) ,
but since we also have W = Ω (cid:16)(cid:112) N log( N ) (cid:17) then
W (cid:48)
=
Ω (cid:16)(cid:112) N log( N ) (cid:17) .
L IMITING THE D EPTH
In this section we generalize Theorem 3.1 to the case of having a bounded depth.
We will then compare our upper bound on memorization to the VC-dimension bound from Bartlett et al.
We have the following:
Theorem 5.1.
Assume the same setting as in Theorem 3.1, and denote R :
=
dNCrδ − 1 .
Let L ∈ [ √ N ] .
Then, there exists a network F : R d → R with width O (cid:0) NL 2 (cid:1) , depth O
[Equation]
and a total of O (cid:18)
[Equation]
that
[Equation]
i
[Equation]
[ N ] [Equation]
Note that for L = √ N we get a similar bound to Theorem 3.1.
In the proof we partition the dataset into NL 2 subsets, each containing L 2 data points.
We then use Theorem 3.1 on each such subset to construct a subnetwork of depth O ( L · log( R )) and width O (1) to memorize the data points in the subset.
We construct these subnetworks such that their output on each data point outside of their corresponding subset is zero.
Finally we stack these subnetworks to construct a wide network, whose output is the sum of the outputs of the subnetworks.
By Bartlett et al.
(2019), the VC dimension of the class of ReLU networks with W parameters and depth L is O ( WL log( W )) .
It implies that if we can shatter N points with networks of depth L , then W = Ω (cid:16)
N L log( N ) (cid:17) .
As in section 4, assume that d is either constant or at most O ( √ N ) , and that r, δ − 1 and C are bounded by some poly ( N ) .
Theorem 5.1 implies that we can memorize any
N points (under the mild separability assumption) using networks of depth L and W =
˜
O (cid:0) NL (cid:1) parameters.
Therefore, the gap between our upper bound and the above lower bound is logarithmic.
It also implies that, up to logarithmic factors, the task of memorizing any set of N points using depth- L neural networks is not more difﬁcult than shattering a single set of N points with depth- L networks.
Proof of Theorem 5.1.
We ﬁrst construct a network H : R d → R in the same manner as the construction of F 1 is the ﬁrst stage of Theorem 3.1 (this construction is detailed in Lemma A.2 in the appendix).
This is a 2-layer network with width 1 .
We denote the output of H on the samples x 1 , . . .
, x N as x 1 , . .
, x N .
Note that by the construction | x i | ≤ O ( R ) for every i ∈ [ N ] and | x i − x j | ≥ 2 for every i (cid:54) =
j .
We split the inputs to NL 2 subsets of size L 2 each, we denote these subsets as I 1 , . . .
, I NL 2 (assume w.l.o.g. that NL 2 is an integer, otherwise replace it with (cid:6) NL 2 (cid:7) ).
For each subset I k we use stages II and III from Theorem 3.1 to construct a network F k such that F k ( x i ) =
y i if x i ∈
I
k .
Note that we only need to use the last two stages since we have already done the projection stage, hence are data samples are already one-dimensional.
We construct a network ˜ F : R → R N/L 2 such that:
[Equation]
We also deﬁne a network G : R N/L 2 → R which adds up all its inputs.
Finally, we construct the network F : R d → R as F :=
G ◦ ˜ F ◦ H .
By the construction of each F k from Theorem 3.1, for every x ∈ R , if | x − x i | ≥ 2 for every i ∈
I
k , then F k ( x ) = 0 .
For every i ∈ [ N ] , there is exactly one k ∈
[ N/L 2 ] such that i ∈ I k .
Using the projection H , for this k we get that F k ( x i ) =
y i , and for every (cid:96) (cid:54)
=
k we get that F (cid:96) ( x i )
= 0 .
This means that F ( x i ) =
y i for every i ∈ [ N ] .
The depth of F is the sum of the depths of its subnetwork.
Both H and G have depth O (1) , and by
Theorem 3.1, the depth of F k is at most O
[Equation]
, since each such network memorizes
L 2 samples.
Hence, the total depth of F is O (cid:18)
[Equation]
Finally, the width of F is the maximal width of its subnetworks.
The width of G and H is 1 .
The width of each F k is also O (1) by Theorem 3.1, hence the width of ˜ F is O (cid:0) NL 2 (cid:1) , which is also the width of F .
Recall that the number of parameters of a network is the number of non-zero weights.
In our case, the network consists of NL 2 subnetworks, where the weights between these subnetworks are zero.
Each such subnetwork has depth
O (cid:18)
L √ log( L ) · log( R ) (cid:19) and width O (1) .
Also, the projection phase requires O ( d ) parameters.
In total, the number of parameters in the network is
O (cid:18)
N L √ log( L ) · log( R ) + d (cid:19) .
B IT C OMPLEXITY - L OWER AND U PPER B OUNDS
In the proof of Theorem 3.1 the bit complexity of the network is roughly (cid:112) N log( N ) (See Theorem A.1 in the appendix).
On one hand, having such large bit complexity allows us to ”store” and ”extract” information from the weights of the network using bit extraction techniques.
This enable us to memorize N data points using signiﬁcantly less than N parameters.
On the other hand, having large bit complexity makes the network difﬁcult to implement on ﬁnite precision machines.
In this section we argue that having large bit complexity is necessary and sufﬁcient if we want to construct a network that memorizes N data points with less than N parameters.
We show both upper and lower bounds on the required bit complexity, and prove that, up to logarithmic factors, our construction is optimal w.r.t.
the bit complexity.
First, we show a simple upper bound on the VC dimension of neural networks with bounded bit complexity.
For this upper bound, we view the neural network as a directed graph with U neurons, and where each weight is represented by B bits.
Note that representing each parameter requires log( B )+2 log( U ) bits, that is log( B ) bits for the weights magnitude, and 2 log( U ) bits for the input and output neuron of each edge.
Theorem 6.1.
Let H be the hypothesis class of ReLU neural networks with W parameters, where each parameter is represented by B bits.
Then, the VC dimension of H is O ( WB + W log( W )) .
Proof.
We claim that H is a ﬁnite hypothesis class with at most 2 O ( WB + W log( W )) different functions.
Let f be a neural network in H , and suppose that it has at most W parameters and U ≤ W neurons.
Then, each weight of f can be represented by at most B + 2 log( U ) bits, namely, 2 log( U ) bits for the indices of the input and output neurons of the edge, and B bits for the weight magnitude.
Hence, f is represented by at most O ( W ( B + log( U )))
bits.
Since U ≤ W we get that every function in H can be represented by at most O ( W ( B +log( W ))) bits, which gives the bound on the size of H .
An upper bound on the VC dimension of ﬁnite classes is the log of their size (cf.
ShalevShwartz and Ben-David (2014)), and hence the VC dimension of H is O
( WB + W log( W )) .
This lemma can be interpreted in the following way: Assume that we want to shatter a single set of N points, and we use ReLU neural networks with N 1 − (cid:15) parameters for some (cid:15) ∈ (cid:2) 0 , 12 (cid:3) .
Then, the bit complexity of each weight in the network must be at least Ω( N (cid:15) ) .
Thus, to shatter N points with less than N parameters, we must have a neural network with bit complexity that depends on N .
Also, for (cid:15) = 12 , our construction in Theorem 3.1 (which memorizes any set of N points) is optimal, up to logarithmic terms, since having bit complexity of Ω( √ N ) is unavoidable.
We emphasize that existing works that show non-trivial VC bounds also use large bit complexity (e.g. Bartlett et al.
We now show an upper bound on the number of parameters of a network for memorizing N data points, assuming that the bit complexity of the network is bounded:
Theorem 6.2.
Assume the same setting as in Theorem 3.1, and denote R :
=
dNCrδ − 1 .
Let B ∈
[ √ N ] .
Then, there exists a network F : R d → R with bit complexity O (cid:18)
B √ log( B ) · log( R ) (cid:19) , depth
O (cid:18) N √ log( B )
B · log( R ) (cid:19) and width O (1) such that F ( x i ) =
y i for every i ∈ [ N ] .
The full proof can be found in Appendix B.
The proof idea is to partition the dataset into NB 2 subsets, each containing B 2 data points.
For each such subset we construct a subnetwork using Theorem 3.
to memorize the points in this subset.
We concatenate all these subnetwork to create one deep network, such that the output of each subnetwork is added to the output of the previous one.
Using a speciﬁc projection from Lemma A.2 in the appendix, this enables each subnetwork to outputfor each data point which is not in the corresponding subset.
We get that the concatenated network successfully memorizes the N given data points.
Assume, as in the previous sections, that r, δ − 1 and C are bounded by some poly ( N ) and d is bounded by O ( √ N ) .
Theorem 6.2 implies that we can memorize any N points (under mild assumptions), using networks with bit complexity B and ˜ O (cid:0) NB (cid:1) parameters.
More speciﬁcally, we can memorize N points using networks with bit complexity ˜
O
( N (cid:15) ) and ˜ O (cid:0)
N 1 − (cid:15) (cid:1) parameters, for (cid:15) ∈ (cid:2) 0 ,
12 (cid:3) .
Up to logarithmic factors in N , this matches the bound implied by Theorem 6.1.
C ONCLUSIONS
In this work we showed that memorization of N separated points can be done using a feedforward
ReLU neural network with ˜ O (cid:16) √ N (cid:17) parameters.
We also showed that this construction is optimal up to logarithmic terms.
This result is generalized for the cases of having a bounded depth network, and a network with bounded bit complexity.
In both cases, our constructions are optimal up to logarithmic terms.
An interesting future direction is to understand the connection between our results and the optimization process of neural networks.
In more details, it would be interesting to study whether training neural networks with standard algorithms (e.g. GD or SGD) can converge to a solution which memorizes N data samples while the network have signiﬁcantly less than N parameters.
Another future direction is to study the connection between the bounds from this paper and the generalization capacity of neural networks.
The double descent phenomenon (Belkin et al.
(2019)) suggests that after a network crosses the ”interpolation threshold”, it is able to generalize well.
Our results suggest that this threshold may be much smaller than N for a dataset with N samples, and it would be interesting to study if this is true also in practice.
Finally, it would be interesting to understand whether the logarithmic terms on our upper bounds are indeed necessary for the construction, or these are artifacts of our proof techniques.
If these logarithmic terms are not necessary, then it will show that for neural network, the tasks of shattering a single set of N points, and memorizing any set of N points are exactly as difﬁcult (maybe up to constant factors).
A P ROOF FROM SECTION
We ﬁrst give the proofs for each of the three stages of the construction, then we combine all the stages to prove Theorem 3.1.
For convenience, we rewrite the theorem, and also state the bound on the bit complexity of the network:
Theorem A.1.
Let N, d, C ∈ N , r, δ > 0 , and let ( x 1 , y 1 ) , . . .
, ( x N , y N ) ∈ R
d × [ C ] be a set of N labeled samples with (cid:107) x i (cid:107) ≤ r for every i and (cid:107) x
i − x j (cid:107) ≥ δ for every i (cid:54) =
j .
Denote R := 10 rN 2 δ − 1 √ πd .
Then, there exists a neural network F : R d → R with width 12, depth O (cid:16)(cid:112) N log( N ) + (cid:113) N log( N ) · max { log( R ) , log( C ) } (cid:17) and bit complexity bounded by
O (cid:16) log( d ) + (cid:113) N log( N ) · max { log( R ) , log( C ) } (cid:17) such that F ( x i ) =
y i for every i ∈ [ N ] .
A. S TAGE I: P ROJECTING ONTO A ONE - DIMENSIONAL SUBSPACE
Lemma A.2.
Let x 1 , . . .
, x N ∈ R d with (cid:107) x i (cid:107) ≤ r for every i and (cid:107) x
i − x j (cid:107) ≥ δ for every i (cid:54) =
j .
Then, there exists a neural network F : R d → R with width 1 , depth 2 , and bit complexity log (cid:0) 3 drN 2 √ πδ − 1 (cid:1) , such that 0 ≤ F ( x i ) ≤ 10 rN 2 δ − 1 √ πd for every i ∈ [ N ] and | F ( x i ) −
F ( x j ) | ≥ 2 for every i (cid:54) =
j .
To show this lemma we use a similar argument to the projection phase from Park et al.
(2020), where the main difference is that we use weights with bounded bit complexity.
Speciﬁcally, we use the following:
Lemma A.3 (Lemma 8 from Park et al.
Let N, d ∈ N , then for any distinct x 1 , . . .
, x N ∈ R
d there exists a unit vector u ∈ R d such that for all i (cid:54) =
j :
[Equation]
Proof of Lemma A.2.
We ﬁrst use Lemma A.3 to ﬁnd u ∈ R d that satisﬁes Eq.
Note that (cid:107) u (cid:107) = 1 , hence every coordinate of u is smaller than 1 .
We deﬁne ˜
u ∈ R d such that each of its coordinates is equal to the ﬁrst (cid:100) log (cid:0) dN 2 √ π (cid:1) (cid:101) bits of the corresponding coordinate of u .
Note that
[Equation]
1 N
[Equation]
For every i (cid:54) =
j we have that:
[Equation] [Equation]
We also have for every i ∈ [ N ] :
[Equation]
Let b := − min { 0 , min i {(cid:98)
˜ u (cid:62) x i (cid:99)}} + 1 , and note that by Eq.
(3) we have b ≤ 2 r + 1 .
We deﬁne the network F : R d → R in the following way:
[Equation]
We show the correctness of the construction.
Let i (cid:54) =
j , then we have that:
[Equation]
where the second equality is because by the deﬁnition of b we have that ˜ u
(cid:62) x
i + b ≥ 0 for every i , and the last inequality is by Eq.
(2) and the assumption that (cid:107) x
i − x j (cid:107) ≥ δ for every i (cid:54) =
j .
Now, let i ∈
[ N ] , then we have:
[Equation]
where the second to last inequality is since b ≤ 2 r + 1 and Eq.
The network F has depth 2 and width 1 .
The bit complexity of the network can be bounded by log (cid:0) 3 drN 2 √ πδ − 1 (cid:1) .
This is because each coordinate of ˜ u can be represented using (cid:100) log( dN 2 √ π ) (cid:101) bits, the bias b can be represented using at most (cid:100) log(2 r +1) (cid:101) bits and the weight in the second layer can be represented using (cid:100) log(2 N 2 √ πdδ − 1 ) (cid:101) bits.
A. S TAGE II: F INDING THE RIGHT SUBSET
Lemma A.4.
Let x 1 < · · · < x N < R with R > 0 , x i ∈ R for every i ∈ [ N ] and | x i − x j | ≥for every i (cid:54) =
j .
Let m ∈ N with m < N and let w 1 , . .
w m ∈ N where LEN ( w j ) ≤ b for every j ∈ [ m ] .
Let k :=
(cid:6) Nm (cid:7) .
Then, there exists a neural network F : R → R with width 4 , depth 3 m +and bit complexity b + (cid:100) log( R ) (cid:101) such that for every i ∈ [ N ] we have that F ( x i ) =
w (cid:100) ik (cid:101) .
Proof.
Let j ∈ [ m ] .
We deﬁne network blocks ˜ F j : R → R and F j : R 2 → R 2 in the following way: First, we use Lemma A.6 to construct ˜ F j such that ˜ F j ( x ) = 1 for every x ∈ [ (cid:98) x j · k − k +1 (cid:99) , (cid:98) x j · k + 1 (cid:99) ] , and ˜ F j ( x ) = 0 for x < (cid:98)
x j · k − k +1 (cid:99) − 12 or x > (cid:98) x j · k + 1 (cid:99) + 12 .
In particular, ˜ F j ( x i ) =
if i ∈ [ j · k − k + 1 , j · k ] , and ˜ F j ( x i ) = 0 otherwise.
Note that since k is deﬁned with ceil, it is possible that j · k ≥ N , if this is the case we replace j · k with N .
Next, we deﬁne:
[Equation]
Finally we deﬁne the network F ( x ) =
[Equation] [Equation]
(we can use one extra layer to augment the input with an extra coordinate of 0 ).
We show that this construction is correct.
Note that for every i ∈ [ N ] , if we denote j = (cid:100)
ik (cid:101) , then ˜ F j ( x i )
= 1 and for every j (cid:48)
(cid:54) =
j we have ˜ F j (cid:48)
( x i )
= 0 .
By the construction of F we get that
F ( x i ) =
w (cid:100) ik (cid:101) .
The width of F at every layer j is at most the width of ˜ F j + 2 , since we also keep copies of both x and y , hence the width is at most 4 .
The depth of ˜ F j is 2, and F is a composition of the ˜ F j ’s with an extra layer for the input to get x (cid:55)→ (cid:18)
x
(cid:19) , and an extra layer in the output to extract the last coordinate.
Hence, its depth is 3 m + 2 .
The bit complexity is bounded by the sum of the bit complexity of ˜ F j and the weights w j , hence it is bounded by b + (cid:100) log( R ) (cid:101) .
A. S TAGE III: B IT E XTRACTION FROM THE C RAFTED W EIGHTS
Lemma A.5.
Let ρ, n, c ∈ N .
Let u ∈ N with LEN ( u ) =
ρ · n and let w ∈ N with
LEN ( w ) =
c · n .
Assume that for any (cid:96), k ∈ { 0 , 1 , . . .
, n − 1 } with (cid:96) (cid:54) =
k we have that (cid:12) (cid:12) BIN ρ · (cid:96) +1: ρ · ( (cid:96) +1) ( u ) − BIN ρ · k +1: ρ · ( k +1) ( u ) (cid:12) (cid:12) ≥ 2 .
Then, there exists a network F : R 3 → R with width 12 , depth 3 n · max { ρ, c } + 2 n + 2 and bit complexity n max { ρ, c } + 2 , such that for every x > 0 , if there exist j ∈ { 0 , 1 , . . .
, n − 1 } where (cid:98) x
(cid:99)
=
BIN ρ · j +1: ρ · ( j +1) ( u ) , then:
F [Equation] =
[Equation] +1: c · ( j +1) ( w ) .
Proof.
Let i ∈ { 0 , 1 , . . .
, n − 1 } , and denote by ϕ
( z ) :
= σ ( σ (2 z ) − σ (4 z − 2))
the triangle function due to Telgarsky (2016).
We construct the following neural network F i :
[Equation]
where we deﬁne y i :
= BIN i · c +1:(
i +1) ·
c ( w ) if x ∈ [ BIN i · ρ +1:(
i +1) · ρ ( u ) , BIN i · ρ +1:(
i +1) ·
ρ ( u )+1] , and y i =
if x > BIN i · ρ +1:( i +1) · ρ ( u ) + 32 or x < BIN i · ρ +1:( i +1) · ρ ( u ) − 12 .
The construction uses two basic building blocks: The ﬁrst is using Lemma A.7 twice, for u and w .
This way we construct two smaller networks F wi , F ui , such that:
[Equation]
The second is to construct y i .
We use Lemma A.8 with inputs x and BIN i · ρ +1:( i +1) · ρ ( u ) , and denote the output of this network by ˜ y i .
We use the following 1 -layer network:
[Equation]
Note that if ˜ y i =
1 then the output of the above network is BIN i · c +1:(
i +1) ·
c ( w ) , and if ˜ y i =
then the output of the network is 0 since BIN i · c +1:(
i +1) ·
c ( w ) ≤ 2 c .
The last layer of F i is just adding the output y i to y , and using the identity on the other coordinates which are being kept as the output, while the other coordinates (namely, the last output coordinates of F wi and F ui ) do not appear in the output of F i .
Also note that all the inputs and outputs of the networks and basic building blocks are positive, hence σ acts as the identity on them.
This means that if F ui is deeper than F wi (or the other way around), then we can just add identity layers to F wi which do not change the output, but make the depths of both networks equal.
Finally we deﬁne:
[Equation]
where: (1) G : R 5 → R is a 1-layer network that outputs the last coordinate of the input, and; (2)
H : R 3 → R 6 is a 1-layer network such that:
[Equation]
  where we assume that x, w, u > 0 .
We show the correctness of the construction.
We have that F (cid:32)(cid:32) xw u (cid:33)(cid:33) =
(cid:80) n − 1 i =
y i .
Assume there exists j ∈ { 0 , 1 . . .
, n − 1 } such that (cid:98) x (cid:99) =
BIN ρ · j +1: ρ · ( j +1) ( u ) .
Then, by the construction of y i we have that y j = BIN c · j +1: c · ( j +1) ( w ) , and for every other (cid:96) (cid:54) =
j , since (cid:12)(cid:12) BIN ρ · (cid:96) +1:
ρ · ( (cid:96) +1) ( u ) − BIN ρ · j +1: ρ · ( j +1) ( u ) (cid:12)(cid:12) ≥ 2 , we must have that x > BIN ρ · (cid:96) +1: ρ · ( (cid:96) +1)
( u ) + 32 or x < BIN ρ · (cid:96) +1:
ρ · ( (cid:96) +1) ( u ) − 12 , which means that y (cid:96) = 0 .
In total we get the the output of F is equal to (cid:80) n − 1 i =
y i =
y j = BIN c · j +1: c · ( j +1) ( w ) as required.
We will now calculate the width, depth and bit complexity of the network F .
The width of each
F wi and F ui is equal to 5 by Lemma A.7, and we need two more neurons for x and y .
In total, the width is bounded by 12 .
Note that all other parts of the network (i.e. H, G and the network from
Lemma A.8) require less width than this bound.
For the depth, by Lemma A.7, the depth of each
F ui and F wi is at most 3 · max { ρ, c } , adding the construction from Lemma A.8, the depth of each F i is at most 3 · max { ρ, c } + 2 .
Summing over all i , and adding the depth of G and H we get that the depth of F is bounded by 3 n · max { ρ, c } + 2 n + 2 .
Finally, the bit complexity of F ui , F wi and H is bounded by n max { ρ, c } + 2 , and all other parts of the network require less bit complexity.
Hence, the bit complexity of F is bounded by n · max { ρ, c } + 2 .
A. P ROOF OF T HEOREM 3.
The construction is done in three phases.
We ﬁrst project the points onto a 1-dimensional subspace, where the projection preserves distances up to some error.
The second step is to split the points into (cid:112)
N log( N ) subsets, and to extract two weights containing “hints” about the points and their labels.
The third step is to parse the hints using an efﬁcient bit extraction method, and to output the correct label for each point.
Throughout the proof we assume w.l.o.g. that the following terms are integers since we use them as indices: (cid:112) N log( N ) , (cid:113)
N log( N ) , log( R ) , log( C ) .
If any of them is not an integer we can just replace it with its ceil (i.e. (cid:100) log( R ) (cid:101) instead of log( R ) ).
This replacement changes these numbers by at most 1 , which in turn can only increase the number of parameters or bit complexity by at most a constant factor.
Since we give the result using the O ( · ) notation, this does not change it.
Also, note that the width of the network is independent of such terms.
For the ﬁrst stage, we use Lemma A.2 to construct a network F 1 : R d → R such that F 1 ( x i ) ≤ 10 rN 2 δ − 1 √ πd for every i ∈ [ N ] and | F 1 ( x i ) − F 1 ( x j ) | ≥ 2 for every i (cid:54) =
j .
We denote R := 10 rN 2 δ − 1 √ πd , and we denote the output of F 1 on the samples x 1 , . . .
, x N ∈ R d as x 1 , . .
, x N ∈ R for simplicity.
We also assume that the x i ’s are in increasing order, otherwise we reorder the indices.
This concludes the ﬁrst stage.
For the second stage, we deﬁne two sets of integers w 1 , . . .
, w √ N log( N ) each represented by (cid:113) N log( N ) · log( C ) bits, and u 1 , . . .
, u √ N log( N ) each represented by (cid:113) N log( N ) · log( R ) bits, in the following way:
For every i ∈ [ N ] let j := (cid:24) i · (cid:113) log( N ) N (cid:25) and k :
= i (cid:16) mod (cid:113) N log( N ) (cid:17) .
We set:
BIN k · log( C )+1:(
k +1)
· log( C ) ( w j ) =
y
i
BIN k · log( R )+1:(
k +1)
· log( R ) ( u j ) =
(cid:98) x
i (cid:99) .
We now use Lemma A.4 twice to construct two networks F w 2 : R → R and F u 2 : R → R such that
F w 2 ( x i ) =
w j i and F u 2 ( x i ) =
u j i for j i =
(cid:24) i · (cid:113) log
( N ) N (cid:25) .
We construct the network F 2 : R → R to be a concatenation of the two networks, with an additional coordinate which outputs σ ( x ) (since the inputs x i are positive, this coordinate just output the exact input).
Namely, we construct a network such that for every i ∈ [ N ] we have
[Equation]
where j i =
[Equation]
For the third stage, we use Lemma A.5 to construct a network F 3 : R 3 → R such that for every x > 0 , if there exist j ∈ (cid:26) 0 , 1 , . . .
, (cid:24)(cid:113) log( N ) N (cid:25) −(cid:27) such that (cid:98) x
(cid:99)
=
BIN log( R ) · j +1:log( R ) · ( j +1) ( u ) , then:
[Equation]
Finally, we construct the network F : R d → R as F ( x ) =
F 3 ◦ F 2 ◦ F 1 ( x ) .
We show the correctness of the construction.
Let i ∈ [ N ] , and let y :
= F ( x i ) .
By the construction of F 2 and the numbers w 1 , . . . , w √ N log( N ) , u 1 , . . . , u √ N log( N ) , if we denote j := (cid:24) i · (cid:113) log( N ) N (cid:25) and k := i (cid:16) mod (cid:113) N log( N ) (cid:17) , then we have that: (1) F 2 ◦ F 1 ( x i ) =
(cid:32) F 1 ( x i ) w j u j (cid:33) ; (2)
BIN log( R ) · k +1:log( R ) · ( k +1) ( u j ) = (cid:98) F 1 ( x i ) (cid:99) ; and (3) BIN log( C ) · k +1:log( C ) · ( k +1) ( w j ) =
y i .
Finally, by the construction of F 3 we get that y = F 3 ◦ F 2 ◦ F 1 ( x i ) =
BIN log( C ) ·
k +1:log( C ) · ( k +1) ( w j ) =
y i as required.
The width of the network F , namely the maximal width of its subnetworks, is the width of F 3 , which is 12 .
The depth of F is the sum of the depths of each of its subnetworks.
The depth of F 1 is 2, the depth of F 2 is O (cid:16)(cid:112)
N log( N ) (cid:17) and the depth of Fis O (cid:16)(cid:113)
N log( N ) · max { log( R ) , log( C ) } (cid:17) .
Hence, the total depth of F can be bounded by
O (cid:16)(cid:112) N log( N ) + (cid:113) N log( N ) · max { log( R ) , log( C ) } (cid:17) .
The bit complexity of F is the maximal bit complexity of its subnetworks.
The bit complexity of F 1 is log (cid:0) 3 drN 2 √ πδ − 1 (cid:1)
=
log( d ) + log( R/ 3) , the bit complexity of F 2 is O (cid:16)(cid:113) N log( N ) · max { log( R ) , log( C ) } (cid:17) and the bit complexity of F 3 is also
O (cid:16)(cid:113)
N log( N ) · max { log( R ) , log( C ) } (cid:17) .
In total, the bit complexity of F can be bounded by
O (cid:16) log( d ) + (cid:113) N log( N ) · max { log( R ) , log( C ) } (cid:17) .
A. A UXILIARY L EMMAS
Lemma A.6.
Let a, b ∈ N with a < b .
Then, there exists a neural network F with depth 2 , width 2 and bit complexity LEN ( b ) such that F ( x ) = 1 for x ∈ [ a, b ] and F ( x ) = 0 for x > b + 12 or x < a − 12 .
Proof.
Consider the following neural network:
[Equation]
It is easy to see that this networks satisﬁes the requirements.
Also, its bit complexity is at most
LEN ( b ) , since a < b , hence a can be represented by at most LEN ( b ) bits.
Lemma A.7.
Let n ∈ N and let i, j ∈ N with i < j ≤ n .
Denote Telgarsky’s triangle function by ϕ ( z ) :
= σ ( σ (2 z ) − σ (4 z − 2)) .
Then, there exists a neural network F : R 2 → R 3 with width 5 , depth 3( j − i + 1) , and bit complexity n + 2 , such that for any x ∈ N with LEN ( x ) ≤ n , if the input of F is (cid:18) ϕ
( i − 1) (cid:0) x 2 n + [-
 [Equation]-]1 2 n +1 (cid:1) ϕ
( i
[Equation]
1) (cid:0) x 2 n + 1 2 n +2 (cid:1) (cid:19) , then it outputs:
[Equation]
Proof.
We use Telgarsky’s function to extract bits.
Let x ∈ N with LEN ( x ) = n , and let i ∈ N with i ≤ n .
Then, we have that:
[Equation]
The intuition behind Eq.
(4) is the following: The function ϕ ( i ) is a piecewise linear function with 2 i − 1 ”bumps”.
Each such ”bump” consists of two linear parts with a slope of [-
-]2 i , the ﬁrst linear part goes from 0 to 1, and the second goes from 1 to 0.
Let x ∈ N with at most n bits in its binary representation.
It can be seen that the i -th bit of x is 1 if ϕ ( i ) (cid:0) x 2 n + 1 2 n +1 (cid:1) is on the second linear part (i.e. descending from 1 to 0) and its i -th bit is 0 otherwise.
This shows that ϕ ( i ) (cid:0) x 2 n + 1 2 n +2 (cid:1) − ϕ
( i ) (cid:0) x 2 n + 1 2 n +1 (cid:1) is equal to 2 i − n − {+
+} if the i
The correctness of Eq.
(4) follows.
Let j, i ∈ N with i <
j and denote c :
= j − i .
Using the construction above as a building block, we construct a network which outputs BIN i : j ( x ) in the following way: For (cid:96) ∈ { 0 , 1 , . .
. , c } deﬁne
F (cid:96) : R 3 → R 3 to be the neural network, such that for an input
[Equation]
y
[Equation]
[Equation]
outputs
[Equation]

 .
The network F (cid:96) is obtained by composing the ﬁrst two coordinates with
Telgarsky’s function ϕ , and then the last coordinate is calculated using Eq.
Finally, we deﬁne:
F :=
F c ◦ · · · ◦ F 0 , and we augment the input of F 0 , such that its last coordinate is zero.
The output of F is as required since
[Equation]
Finally, we compute the width, depth and bit complexity of F .
Its width is bounded by twice the width of ϕ (which is 2 ), plus an extra neuron for computing the output y , hence its width is bounded by 5 .
Its depth is equal to 3( j − i + 1) , since we need two layers to compute ϕ and an extra layer to compute the output y .
Finally, the bit complexity is bounded by n + 2 , since the largest weight in the network is at most 2 n +2 .
Lemma A.8.
There exists a network F : R 2 → R with width 2 depth 2 and bit complexity 2 such that F (cid:18)(cid:18) xy (cid:19)(cid:19)
= 1 if x ∈ [ y, y + 1] and F (cid:18)(cid:18)
xy (cid:19)(cid:19) =
if x > y + 32 or x < y − 12 .
Proof.
We consider the following neural network:
[Equation]
It is easy to see that this network satisﬁes the requirements.
It has width 2 , depth 2 and bit complexity 2 .
B P ROOF FROM SECTION
Proof of Theorem 6.2.
We ﬁrst use Lemma A.2 to construct a network H : R d → R in the same manner of the construction of F 1 is the ﬁrst stage of Theorem 3.1.
This is a 2-layer network with width 1 and bit complexity of O (log( R )) .
We denote the output of H on the samples x 1 , . . .
, x N as x 1 , . .
, x N .
Note that by the construction | x i | ≤ O ( R ) for every i ∈ [ N ] and | x i − x j | ≥ 2 for every i (cid:54) =
j .
We now split the inputs to N B 2 subsets of size B 2 each, we denote these subsets as I 1 , . .
, I N B 2 (We assume w.l.o.g. that NB 2 is an integer, otherwise we can replace it with (cid:6) NB 2 (cid:7) ).
For each subset I k we use Theorem 3.1 to construct a network F k which memorizes the points in I k , with the following changes : (1) There is an
additional coordinate which memorizes the input and output
as
is ; and (2) The output of the network is added to the output of the network for the previous subset k − 1 .
For k = 1 , we set this coordinate to be zero.
That is, if x i ∈
I k then F k (cid:18)(cid:18) x i y (cid:19)(cid:19) =
(cid:18) x i y
+ y i (cid:19) , otherwise F k (cid:18)(cid:18)
x i y (cid:19)(cid:19) =
(cid:18) x i y (cid:19) .
Finally, we construct the network F : R d → R as:
[Equation]
where G (cid:18)(cid:18) xy (cid:19)(cid:19) =
y .
By the construction of each F k from Theorem 3.1, for every x ∈ R , if | x − x i | ≥ 2 for every i ∈
I
k , then F k ( x ) = 0 .
For every i ∈ [ N ] , there is exactly one k ∈
[ N/B 2 ]
such that i ∈ I k .
Using the projection H , for this k we get that F k (cid:18)(cid:18) x
i y (cid:19)(cid:19) =
(cid:18) x i y
+ y i (cid:19) , and for every (cid:96) (cid:54)
=
k we get that
F (cid:96) (cid:18)(cid:18) x i y (cid:19)(cid:19) =
(cid:18) x i y (cid:19) .
This means that F ( x i ) =
y i for every i ∈ [ N ] .
By Lemma A.4 and Lemma A.5, since each F k is used to memorize B 2 samples, then its bit complexity is bounded by O (cid:16) B (cid:112) log( B ) · log( R ) (cid:17) bits, hence this is also the bit complexity of F .
The depth of each component F k is O (cid:16) B (cid:112) log( B ) · log( R ) (cid:17) , and there are NB 2 such components.
Hence, the depth of the network F is O (cid:18) N √ log( B )
B log( R ) (cid:19) .
The width of F is bounded by the width of each component, which is O (1) .