T HE C OMPARATIVE P OWER OF R E LU N ETWORKS AND P OLYNOMIAL K ERNELS IN THE P RESENCE OF S PARSE L ATENT S TRUCTURE
Anonymous authors
Paper under double-blind review
A BSTRACT
There has been a large amount of interest, both in the past and particularly recently, into the relative advantage of different families of universal function approximators, for instance neural networks, polynomials, rational functions, etc.
However, current research has focused almost exclusively on understanding this problem in a worst-case setting : e.g. characterizing the best L 1 or L 1 approximation in a box (or sometimes, even under an adversarially constructed data distribution.)
In this setting many classical tools from approximation theory can be effectively used.
However, in typical applications we expect data to be high dimensional, but structured – so, it would only be important to approximate the desired function well on the relevant part of its domain, e.g. a small manifold on which real input data actually lies.
Moreover, even within this domain the desired quality of approximation may not be uniform; for instance in classiﬁcation problems, the approximation needs to be more accurate near the decision boundary.
These issues, to the best of our knowledge, have remain unexplored until now.
With this in mind, we analyze the performance of neural networks and polynomial kernels in a natural regression setting where the data enjoys sparse latent structure, and the labels depend in a simple way on the latent variables.
We give an almost-tight theoretical analysis of the performance of both neural networks and polynomials for this problem, as well as verify our theory with simulations.
Our results both involve new (complex-analytic) techniques, which may be of independent interest, and show substantial qualitative differences with what is known in the worst-case setting.
I NTRODUCTION
The concept of representational power has been always of great interest in machine learning.
In part the reason for this is that classes of “universal approximators” abound – e.g. polynomials, radial bases, rational functions, etc.
Some of these were known to mathematicians as early as Bernstein and Lebesgue 1 – yet it is apparent that not all such classes perform well empirically.
In recent years, the class of choice is neural networks in tasks as simple as supervised classiﬁcation, and as complicated as reinforcement learning – inspiring an immense amount of theoretical study.
Research has focus on several angles of this question, e.g. comparative power to other classes of functions (Yarotsky, 2017; Safran and Shamir, 2017; Telgarsky, 2017), the role of depth and the importance of architecture (Telgarsky, 2016; Safran and Shamir, 2017; Eldan and Shamir, 2016), and many other topics such as their generalization properties and choice of optimization procedure (Hardt et al., 2016; Zhang et al., 2017; Bartlett et al., 2017).
Our results fall in the ﬁrst category: comparing the relative power of polynomial kernels and ReLU networks – with a signiﬁcant twist, that makes our results more relevant to real-life settings.
The ﬂavor of existing results in this subject is roughly the following: every function in a class Ccan be approximately represented as a function in a different class C 2 , with some blowup in the size/complexity of the function (e.g. degree, number of nodes, depth).
The unsatisfying aspect of such results is the “worst-case” way in which the approximation is measured: typically, one picks a domain coarsely relevant for the approximation (e.g. an interval or a box), and considers
the L 1 ; L 2 ; L 1 ; : : : norm of the difference between the two functions on this domain.
In some of the constructions (e.g. (Eldan and Shamir, 2016; Safran and Shamir, 2017)), the evaluation is even more adversarial: it’s the mean-square error over a specially-designed measure.
Instead, in practically relevant settings, it’s reasonable to expect that approximating a predictor function well only on some “relevant domain” would sufﬁce, e.g. near the prediction boundary or near a lower-dimensional manifold on which the data lives, as would be the case in settings like images, videos, ﬁnancial data, etc.
A good image classiﬁer need not care about “typical” data points from the ℓ 1 -ball, which mostly look like white noise.
The difﬁculty with the above question is that it’s not immediate how to formalize what the “relevant domain” is or how to model the data distribution.
We tackle here a particularly simple (but natural) incarnation of this question: namely, when the data distribution has sparse latent structure , and all we ask is to predict a linear function of the latent variables based upon (noisy) observations.
The assumption of sparsity is very natural in the context of realistic, high-dimensional data: sparsity under the correct choice of basis is essentially the reason that methods such as lossy image compression work well, and it is also the engine behind the entire ﬁeld of compressed sensing (Donoho, 2006).
O VERVIEW OF RESULTS
We will be considering a regression task where the data has a sparse latent structure.
More precisely, we wish to ﬁt pairs of (observables, labels)
( X; Y ) generated by a ( latent-variable ) process: (cid:15)
Sample a latent vector Z 2 R m from H , where H is a distribution over sparse vectors.
(cid:15)
To produce X 2 R n , set X = AZ + (cid:24) , where the noise (cid:24) (cid:24) subG ( (cid:27) 2 ) is a subgaussian random vector with variance proxy (cid:27) 2 (e.g. N (0 ; (cid:27) 2 I ) ).
(cid:15) To produce Y 2 R , we set Y = ⟨ w; Z ⟩ .
We hope the reader is reminded of classical setups like sparse linear regression, compressive sensing and sparse coding: indeed, this distribution on the data distribution X is standard in these setups.
In our setting, we additionally attach a regression task to this data distribution, wherein the labels Y are linearly generated 2 by a predictor w from the latent vector Z .
Note our interest is slightly different than usual: in the traditional setup, we are interested in the statistical/algorithmic problem of inferring Z , given X as input (the former studying the optimal rates of “reconstruction” for Z , the latter efﬁcient algorithms for doing so).
In particular, we do not typically care about the particular form of the predictor as long as it is efﬁciently computable.
By contrast, we want to understand how well different subsets of universal approximator families can ﬁt the data points ( X; Y ) .
Namely, regardless of the speciﬁcs of the training procedure, the end will be an element of some function class like a linear function of a kernel embedding of X , or a neural network.
Therefore, we ask if these classes are rich enough to reconstruct Y given X accurately (i.e. compared to the Bayes-optimal estimator E [ Y j X ] ): if the answer is negative, then we know our predictor will perform poorly, no matter the training method.
We measure the performance of these estimators in the natural 3 distributional sense: expected reconstruction error, E [( ^ Y (cid:0) Y ) 2 ] .
Informally, what we will show is the following.
Theorem (Informal) .
For the problem of predicting Y given X in the generative model for data described above, it holds that:
(1) Small two-layer ReLU networks achieve close to the statistically optimal rate.
(2) Polynomial predictors of degree lower than log m achieve a statistical rate which is substantially worse.
(In fact, in a certain sense, close to “trivial”.)
Conversely, polynomial predictors of degree
O ((log n ) 2 ) achieve close to the statistically optimal rate.
The lower bound in (2) is relevant since ﬁtting a polynomial to data points of the form ( x i ; y i ) requires 4 searching through the space of multivariate polynomials of degree Ω(log m ) which has dimension m Ω(log( m )) , and thus even writing down all of the variables in this optimization problem takes super-polynomial time.
Practical aspects of using polynomial kernels even with much lower degree than this have been an important concern and topic of empirical research; see for example (Chang et al., 2010) and references within.
On the other hand, the upper bound in (2) shows that our analysis is essentially tight: greater than polylog ( m ) degree is not required to achieve good statistical performance, which is qualitatively different from the situation in worst-case analyses (see
Section 4.2.2 for more details).
Our mathematical analysis closely matches the observed behavior in experiments: see Section 6.
For formal statements of the theorems, see Section 4.
P RIOR W ORK
There has been a large body of work studying the ability of neural networks to approximate polynomials and various classes of well-behaved functions, such as recent work ( Yarotsky, 2017; Safran and Shamir, 2017; Telgarsky, 2017; Poggio et al., 2017).
These results exclusively focus on the worst-case setting where the goal is to ﬁnd a network close to some function in some norm (e.g. Lor L 1 -norm, often under an adversarially chosen measure).
In contrast there is little work on the problem of approximating ReLU networks by polynomials, mostly because it is well-known by classical results of approximation theory (Newman et al., 1964;
DeVore and Lorentz, 1993) that polynomials of degree Ω(1 =ϵ ) are required to approximate even a single ReLU function within error ϵ in L 1 -norm on [ (cid:0) 1 ; 1] .
On the other hand, we will show that if we do not seek to achieve ϵ -error everywhere for the ReLU (in particular not near the nonsmooth point at 0 ) we can build good approximations to ReLU using polynomials of degree only
O (log 2 (1 =ϵ ))
(see discussion in Section 4.2.2 and Theorem 5.2).
Because of the trivial Ω(1 =ϵ ) lower bound for worst-case approximation of ReLU networks by polynomials, ( Telgarsky, 2017) studied the related problem of approximating a neural network by rational functions.
(A classical result of approximation theory (Newman et al., 1964) shows that rational functions of degree O (log 2 (1 =ϵ )) can get within ϵ -error of the absolute value function.)
In particular, (Telgarsky, 2017) shows that rational functions of degree polylog (1 =ϵ ) can get within ϵ distance in L 1 -norm of bounded depth ReLU neural networks.
Somewhat related is also the work of (Livni et al., 2014) who considered neural networks with quadratic activations and related their expressivity to that of sigmoidal networks in the depthcase building on results of ( Shalev-Shwartz et al., 2011) for approximating sigmoids.
The result in (Shalev-Shwartz et al., 2011) is also proved using complex-analytic tools, though the details are substantially different.
The work of (Zhang et al., 2016) studied the power of kernel regression methods to simulate a certain class of neural networks. More precisely, they bounded the ℓ 2 norm of kernel regression models approximating neural networks with bounded depth, “nice” activation functions (not including ReLU), and small input and edge weights. By standard generalization theory, this gives a corresponding sample complexity result for improper learning via kernels. In our setting, their result does not apply: ﬁrst, the network of interest has ReLU activations; even ignoring this issue, their bounds would be roughly exponential in n because the ℓ 2 norm of the network’s input vector is large, of order (cid:2)( p n ) .
dataset.
However, since training data sets are often very large, this is not necessarily helpful.
More usefully, in the classiﬁcation context, the optimization may become feasible due to the existence of a smaller number of support vectors .
In this case, the ability of algorithms to efﬁciently ﬁnd a good function depends on more complex interactions between the particular choice of kernel and the margin properties of the data; we leave analyzing these kinds of quantities for future work.
See Proposition 1 of Zhang et al.
There is a vast literature on high dimensional regression and compressed sensing which we do not attempt to survey, since the main goal of our paper is not to develop new techniques for sparse regression but rather to analyze the representation power of kernel methods and neural networks.
Some relevant references for sparse recovery can be found in (Vershynin, 2018;
Rigollet, 2017).
We only emphasize that the upper bound via soft thresholding we show (Theorem 4.1) is implicit in the literature on high-dimensional statistics; we include the proofs here solely for completeness.
M AIN R ESULTS
In this section we will give formal statements of the results and give some insight into the techniques used.
First, we state the assumptions on the parameters of our generative model:
(cid:15) Z is sparse: more precisely, j supp ( Z ) j (cid:20) k and ∥ Z ∥ 1 (cid:20) M with high probability.(cid:15) A is a (cid:22) -incoherent n (cid:2) m matrix, which means that ∥ A ⊤ A (cid:0) I ∥ 1 (cid:20) (cid:22) for some (cid:22) (cid:21) 0 .
(cid:15) ∥ w ∥ 1 = 1 (w.l.o.g., since changing the magnitude of w rescales Y )
The assumption on A is standard in the literature on sparse recovery (see reference texts (Rigollet, 2017; Moitra, 2018)).
In general one needs an assumption like this (or a stronger one, such as the
RIP property) in order to guarantee that standard algorithms such as LASSO actually work for sparse recovery.
For the reader not familiar with this literature, this property is a proxy for the matrix being “random-like” – e.g. a matrix with i.i.d.
entries of the form (cid:6) 1 = p n has (cid:22) =
O (1 = p n ) , even when m >> n .
We also note that for notational convenience, we will denote ∥ A ∥ 1 = max i;j j A i;j j .
Before proceeding to the results, we note that the ﬁrst-time reader may freely assume that (cid:22) =and n = m ; the results are still interesting in this setting and no important technical idea is needed for the more general case.
For the upper bounds, we have included results for the more general setting (with (cid:22) (cid:21) 0 ) to show that our results are relevant even to very high-dimensional settings where m >> n .
We have only proven the lower bound in the case (cid:22) = {+
+} : this is the easiest setting for algorithms, so this makes the lower bounds the strongest.
4. R EGRESSION U SING R E LU N ETWORKS
We prove the following theorem, which shows that small 2-layer ReLU networks can achieve an almost optimal statistical rate.
Let us denote the soft threshold function with threshold (cid:28) as (cid:26) (cid:28) ( x ) :
= sgn ( x ) min(0 ; j x j (cid:0) (cid:28) ) =
ReLU ( x (cid:0) (cid:28) ) (cid:0) ReLU ( (cid:0) x + (cid:28) ) .
Let’s introduce the notation (cid:26) (cid:10) m (cid:28) to denote the map given by applying (cid:26) (cid:28) coordinate-wise to a vector in R m .
Consider the following estimator (for y ), corresponding to a 2-layer neural network:
[Equation]
We can prove the following result for the estimator (see Appendix A of the supplement):
Theorem 4.1 (2-layer ReLU) .
With high probability, the estimator ^ Y NN satisﬁes
[Equation]
Notice that the size of the ReLU net is comparable to the input: one of the layers has the same dimension as A , the other the same dimension as w .
Furthermore, to interpret this result, recall that we think of (cid:22) as quite small – in particular (cid:22) ≪ 1 .
Thus the error of the estimator is essentially
O ( (cid:27) 2 k 2 log( m )) , i.e. essentially j (cid:27) j error “per-nonzero-coordinate”.
It can be shown that this upper bound is nearly information-theoretically optimal (see Remark B.1), except that there is an additional factor of k .
This additional factor is artiﬁcial and can be removed with added technical effort; we show how to do this in the (cid:22) = 0 case in Theorem A.1.
We emphasize that the analysis of this kind of soft thresholding estimator is implicit in much of the literature on sparse linear regression.
For completeness, we include a complete and self-contained proof of Theorem 4.1 in Section A.
4. R EGRESSION U SING P OLYNOMIALS 4.2. L OWER B OUNDS FOR L OW -D EGREE P OLYNOMIALS
We ﬁrst show that polynomials of degree smaller than O (log m ) essentially cannot achieve a “nontrivial” statistical rate.
This holds even in the easiest case for the dictionary A : when it’s the identity matrix.
More precisely, we consider the situation in which A is an orthogonal matrix (i.e. (cid:22) = 0 ; m = n ), w 2 f(cid:6) 1 g m , the noise distribution is Gaussian N (0 ; (cid:27) 2 I ) , and the entries of Z are independentlywith probability 1 (cid:0)
k =m and N (0 ; (cid:13) 2 ) with probability k =m .
Then we show
Theorem 4.2.
Suppose k < m= 2 and f is a multivariate degree d polynomial.
Then
[Equation]
To parse the result, observe that the numerator is of order (cid:13) 2 k which is the error of the trivial estimator 7 and the denominator is close to 1 unless d is sufﬁciently large with respect to m .
More precisely, assuming the signal-to-noise ratio (cid:13)=(cid:27) does not grow too quickly with respect to m , we see that the denominator is close to 1 unless d d = Ω( p m ) , i.e. unless d is of size Ω((log m ) = log log m ) .
On a technical note we observe that this statement is given with respect to expectation but a similar one can be made with high probability, see Remark B.2.
4.2. N EARLY M ATCHING U PPER B OUNDS
The lower bound of the previous section leaves open the possibility that polynomials of degree
O ( polylog ( m )) still do not sufﬁce to perform sparse regression and solve our inference problem;
Indeed, it is a well-known fact (see e.g. (Telgarsky, 2017))
that to approximate a single ReLU to ϵ -closeness in inﬁnity norm in [ (cid:0) 1 ; 1] requires polynomials of degree poly (1 =ϵ ) ; this follows from standard facts in approximation theory ( DeVore and Lorentz, 1993) since ReLU is not a smooth function.
Proceeding with this “worst-case” way of thinking: our upper bound follows by designing a polynomial approximation to ReLU into our neural network construction; since estimates for Y typically accumulate error from estimating each of the m coordinates of Z , to guarantee accurate reconstruction we would need mϵ to be small.
Plugging in the the best approximation to ReLU in inﬁnity norm, we would need a Ω( p m ) -degree polynomial for this to yield a multivariate polynomial with similar statistical performance to the 2-layer ReLU network which computes ^ Y NN .
Thus, naively, we might suspect that the degree of the kernel needs to be as high as p m to get a reasonable approximation.
Surprisingly, we show this intuition is incorrect!
In fact, we show how using only a polylog ( m ) degree polynomial, our converted ReLU network has similar statistical performance.
Formally this is summarized by the following theorem, where ^ Y d;M is the corresponding version of ^ Y NN formed by replacing each ReLU by our polynomial approximation.
Theorem 4.3.
Suppose (cid:28) = (cid:2)( (cid:27) √ [Equation] + [Equation]
and d
[Equation]
[Equation]
With high probability, the estimator ^ Y d;M satisﬁes
[Equation]
The idea behind our construction is described in Section 5.3.
Our methods are novel and may be of independent interest; we are not aware of a way to get this result using only generic techniques such as FT-Molliﬁcation (Diakonikolas et al., 2010).
O VERVIEW OF PROOFS
In this section, we will sketch the ideas behind the proofs of our results.
The full proofs are relegated to the appropriate appendices.
We proceed with each of our results in turn.
5. U PPER BOUND FOR R E LU N ETWORKS
As previously mentioned, this kind of result is well-known in the literature on sparse regression and we include a proof primarily for completeness.
The intuition is simple: the estimator ^ Z NN can make use of the non-linearity in the soft threshold to zero out the coordinates in the estimate A ⊤ X which are small and thus “reliably” not in the support of the true z .
Thus, the estimator only makes mistakes on the non-zero coordinates.
The full proofs are in Section A.
5. L OWER BOUND : PROOF SKETCH OF T HEOREM 4.
The proof of Theorem 4.2 has two main ideas, which we detail below:
(1) A structural lemma, which shows that the optimal predictor has a “decoupled” structure along the coordinates of the latent variable.
An analysis of this decoupled estimator using a bias-variance calculation in an appropriately chosen basis.
The full proofs of this Section are in Appendix B.
5.2. S TRUCTURE OF THE O PTIMAL E STIMATOR
As explained above, our structural lemma shows that the optimal low-degree polynomial estimator decouples along the coordinates of the latent variable.
In order to understand why this should be true, ﬁrst observe that the optimal estimator for Y = ⟨ w; Z ⟩ given X has a particularly simple structure.
Concretely, the optimal estimator is the conditional expectation E [ ⟨ w; Z ⟩j X ]
=
∑ i w i E
[ Z i j X ] , so the optimal estimator for Y simply reconstructs Z as well as possible coordinate-wise, then takes an inner product with w .
With this in mind, note the coordinates of Z are independent in our setting, so optimal estimation of Z i should not depend in any way on reconstructing Z j for j ̸ = i . This allows us to show that the optimal polynomial of degree d to estimate Y has no “mixed monomials” in an appropriate basis.
This is the content of the next lemma, whose proof is in Appendix B.
Lemma 5.1.
Suppose X = AZ + (cid:24) where A is an orthogonal m (cid:2) m matrix, Z has independent entries and (cid:24) (cid:24) N (0 ; (cid:27) 2 Id ) .
Then there exists a unique minimizer f (cid:3) d over all degree d polynomials f d of the square-loss, E [ [Equation] f
[Equation]
A [Equation] ) 2 ] and furthermore f (cid:3) d has no mixed monomials.
In other words, we can write f (cid:3) d ( A ⊤ X )
=
∑ i f (cid:3) d;i (( A ⊤ X ) i ) where each of the f (cid:3) d;i are univariate degree d polynomials.
5.2. F OURIER A NALYSIS AND B IAS -V ARIANCE T RADE -O FF
Once we have reduced to considering estimators with decoupled structure, it becomes feasible to analyze the performance of all possible low degree polynomials using a bias-variance calculation in a carefully chosen basis. This is the second (and more involved) step in the proof.
In order to perform the calculation, we need to apply Fourier analytic methods, so we need to switch to an orthonormal basis.
Since the noise we chose for the lower bound instance is Gaussian 9 , a natural choice is the Hermite polynomials.
We review the deﬁnition of the Hermite polynomials in Appendix B, but for the purposes of this proof overview, the Hermite polynomials are polynomials H n ( x ) indexed by multi-indices n 2 N mwith the important property that they are orthogonal with respect to the standard m-variate Gaussian distribution, namely
[Equation]
1 ; otherwise
From this, we can derive Plancherel’s Theorem in this basis:
Theorem 5.1 (Plancherel’s Theorem in Hermite Basis) .
Let f ( x ) = ∑ n b f ( n ) H n ( x=(cid:27) ) , then
[Equation]
[Equation] We use this theorem, along with the structural Lemma 5.1 to perform a bias-variance tradeoff analysis of any predictor: namely, we show (1) If the Fourier coefﬁcients j b f ( n ) j are large, then the estimator will be very sensitive to noise (i.e. has too high of a variance ).
(2) On the other hand, if j b f ( n ) j is small and f is low-degree, then the estimator cannot match the correct mean
well regardless of noise (i.e. has too high of a bias ).
Efﬁcient application of Plancherel’s theorem is key to proving both results: in the ﬁrst case, we apply it over the randomness in the noise (cid:24) , and in the second case, we apply it over the randomness in the latent vector Z , which has Gaussian entries conditioned on its support.
Note that when f is sufﬁciently high-degree, it can effectively take advantage of the difference in scales between the noise and the signal to achieve both low bias and low variance simultaneously: see the following upper bound section for details.
5. U PPER B OUND : P ROOF S KETCH OF T HEOREM 4.
As previously mentioned, it’s a result from classical approximation theory that no low-degree polynomial is close to the ReLU function on all of [ (cid:0) 1 ; 1] .
The crux of these results is that it’s hard to approximate ReLU well at 0, its point of non-smoothness.
However, in our setting precisely approximating ReLU everywhere is not important for getting a good regression rate: instead, the approximation needs to be very close to 0 when the input is negative, and only very coarsely accurate otherwise.
The reason for this is the intuition we described for 2-layer ReLU networks: the property of ReLU that is useful in this setting is it’s “denoising” ability – the fact that it zeroes out negative inputs.
Consequently, we design a polynomial approximation to ReLU of degree O (log 2 n ) which sacriﬁces accuracy near the point of non-smoothness in favor of closeness to 0 in the negative region.
More precisely, we prove the following theorem, in which the parameter (cid:28) in our theorem controls the trade-off between the polynomial p d being close to 0 for x < 0 and being close to x for x > 0 .
Theorem 5.2.
Suppose R > 0 , 0 < (cid:28) < 1 = 2 and d (cid:21) 7 .
Then there exists a polynomial p d = p d;(cid:28);R of degree d such that for x 2 [ (cid:0) R; 0]
[Equation]
and for x 2 [0 ; R ] ,
[Equation]
The proof of this theorem proceeds in two steps: (1) First, one takes a “soft-max” molliﬁcation of ReLU of the form g (cid:12) ( x ) :
= 1 (cid:12) log(1 + e (cid:12)x ) with an appropriately tuned (cid:12) , so that g (cid:12) is sufﬁciently close to ReLU.
(2) Second, if (cid:12) is not too large, we prove that the poles (in the complex plane) of the function g (cid:12) are not too close to the origin.
This, it turns out, governs the polynomial approximability of g (cid:12) due to a powerful theorem of Bernstein in complex analysis.
(See Theorem C.1, and it’s quantitative analogue we prove as Theorem C.2.)
Once we have this approximation to ReLU, we directly plug it into our 2-Layer ReLU network estimator from Section 4.1 to prove Theorem 4.3.
The full proofs are in Appendix C.
[Table]
S IMULATIONS
Finally, we provide synthetic experiments to verify the predictions from Theorem 4.2 and Theorem 4.3.
The setup is as follows: we generate a large synthetic data set (with n = m and (cid:22) = 0 ) in the following fashion: (cid:15)
A is a random orthogonal matrix and w is sampled from a n -dimensional standard Gaussian.
(cid:15)
Z 2 R n is sampled by including each coordinate with probability k=n , and sampling a standard Gaussian for each included coordinate.
(cid:15) X and Y are sampled according to the generative model in Section 2, using Gaussian noise with standard deviation (cid:27) .
For each ﬁxed degree, we ﬁt a polynomial using least-squares regression, and evaluate the performance on a corresponding test set 10 generated in the same fashion (reusing the same A and w ).
Solving the regression problem for large degrees is intractable using standard training methods; to overcome this issue, we used structural observation in Lemma 5.1 to reduce the regression problem for estimating Y from X to that of estimating Z i given X i , which is a much lower dimensional problem.
The results of the experiment are in Figure 1, graphed on a log scale.
All experiments were run with k = 5 and (cid:27) = 0 : 06 .
We see that for low degrees, i.e. before our prediction error is close to the information-theoretic limit, the log-error decays roughly linearly with respect to polynomial degree.
This matches the prediction of the lower bound in Theorem 4.2 after taking a log of the right-hand-side.
For completeness, we also evaluate the baseline 2-Layer ReLU network described in Section 4.1 in the same experimental setup.
Table 1 shows the test error of the baseline 2-Layer ReLU network and, for comparison, the best polynomial of degree 17 in the same experiment.
Despite the high degree, the ReLU network is still slightly better.
C ONCLUSIONS
In this paper, we considered the problem of providing representation lower and upper bounds for different classes of universal approximators in a natural statistical setup that exhibits sparse latent structure.
We hope this will inspire researchers to move beyond the worst-case setup when considering the representational power of different function classes.
The techniques we develop are interesting in their own right: unlike standard approximation theory setups, we need to design polynomials which may only need to be accurate in certain regions.
Conceivably, in classiﬁcation setups, similar wisdom may be helpful: the approximator needs to only be accurate near the decision boundary.
Finally, we conclude with a tantalizing open problem: In general it is possible to obtain non-trivial sparse recovery guarantees for LASSO even when the sparsity k is nearly of the same order as n un- [Figure] der assumptions such as RIP.
Since LASSO can be computed quickly using iterated soft thresholding (ISTA and FISTA, see Beck and Teboulle (2009)), we see that sufﬁciently deep neural networks can compute a near-optimal solution in this setting as well.
It would be interesting to determine whether shallower networks and polynomials of degree polylog ( n ) can achieve a similar guarantees
A U PPER BOUND FOR 2-L AYER R E L U N ETWORKS
We will ﬁrst prove a bound on the error of the soft-thresholding estimator ^ Z NN (Lemma A.2), which corresponds to the hidden layer of the neural network: this is essentially a standard fact in highdimensional statistics (see reference text (Rigollet, 2017)).
The idea is that the soft thresholding will correctly zero-out most of the coordinates in the support while adding only a small additional error to the coordinates outside the support.
From the recovery guarantee for ^ Z NN , we will then deduce Theorem 4.1.
Towards proving the above result, we ﬁrst need an estimate on the bias of A ⊤ x , i.e. the error without noise:
Lemma A.1.
Suppose A is (cid:22) -incoherent i.e. ∥ A ⊤ A (cid:0) Id ∥ 1 (cid:20) (cid:22) .
Then for any z , ∥ A ⊤ Az (cid:0) z ∥ 1 (cid:20) (cid:22) ∥ z ∥ 1 .
Proof.
We have
[Equation]
so applying the incoherence assumption we have j ( A ⊤ Az ) i (cid:0)
z i j (cid:20) (cid:22) ∥
z ∥ 1 .
Using this we can analyze the error in thresholding.
Lemma A.2.
Suppose A is (cid:22) -incoherent i.e. ∥ A ⊤ A (cid:0) I ∥ 1 (cid:20) (cid:22) .
Let z be an arbitrary ﬁxed vector such that ∥ z ∥ 1 (cid:20) M and j supp ( z ) j (cid:20) k .
Suppose x = Az + (cid:24) where (cid:24) (cid:24) N (0 ; (cid:27) 2 I n (cid:2) n ) . Then for some (cid:28) = (cid:2)( (cid:27) √ (1 + (cid:22) )
log m + (cid:22)M ) and ^ z = (cid:26) (cid:10) n (cid:28) ( A ⊤ x ) , with high probability we have ∥ ^ z (cid:0) z ∥ 1 (cid:20) 2 (cid:28) and supp (^ z ) (cid:26) supp ( z ) .
Proof.
Observe that
[Equation]
Note that entry i of A ⊤ (cid:24) is ⟨ A i ; (cid:24) ⟩ where {+∥ A i ∥ 
+}
(cid:20) (1 + (cid:22) ) so ( A t (cid:24) ) i is subgaussian with variance proxy at most [-σ 
-](cid:27) 2 (1 + (cid:22) ) .
By concentration and union bound, with high probability all coordinates not in the true support are thresholded to 0.
Similarly we see that for each of the coordinates in the support, an error of at most 2 (cid:28) is made.
From the above lemma, we can easily prove the main theorem of this section:
Proof of Theorem 4.1.
When the high probability above event happens, we have the following upper bound by Holder’s inequality:
[Equation]
For the lower bounds we will be interested mostly in the case when (cid:22) = 0 , i.e. A is orthogonal and so m = n , the coordinates of Z are independent and each is nonzero with probability at most k=n , and the noise is Gaussian.
Then the error estimate we had in the previous theorem specializes to O ( (cid:27) 2 k 2 log( n )) , but under these assumptions we know that the information-theoretic optimal is actually (cid:27) 2 k log( n ) .
While not very important to the ﬂow of the paper, for completeness we can improve the analysis to eliminate the extra factor of k , without changing the algorithm:
Theorem A.1.
Suppose A is orthogonal (hence m = n ), the coordinates of Z are independent, and (cid:24) (cid:24) N (0 ; (cid:27) 2 I ) .
Then
[Equation]
Proof.
In this case [Equation] we have A ⊤ X
= Z + (cid:24) ′ where (cid:24) ′ (cid:24) N (0 ; (cid:27) 2 I ) .
Therefore the coordinates of ^ Z are independent of each other [Equation] and so we see
[Equation]
Let E i denote the event that j (cid:24) ′ j i > (cid:28) .
Then
[Equation]
where the ﬁrst inequality follows as in Lemma A.2, the second inequality uses that j (cid:26) (cid:28) ( x ) (cid:0) x j (cid:20) (cid:28) , the third uses that ( a + b ) 2 = a 2 +2 ab + b 2 (cid:20) 2 a 2 +2 b 2 by Young’s inequality, and the last inequality follows from standard tail bounds on Gaussians.
We see the last expression is O ( k(cid:27) 2 log( m ))
so we have proved the result.
B L OWER B OUNDS FOR P OLYNOMIAL K ERNELS
In this section, prove the lower bounds for polynomial kernels.
We recall the lower bound instance: the noise distribution is N (0 ; (cid:27) 2 Id ) and the distribution for Z is s.t.
every coordinate is ﬁrst chosen to be non-zero with probability k=n , and if it is non-zero, it’s set as an independent sample from N (0 ; (cid:13) 2 ) .
This construction makes Z approximately k -sparse with high probability while making its coordinates independent.
We choose A as an arbitrary orthogonal matrix, so m = n .
We choose w to be an arbitrary (cid:6) 1 sign vector, so w 2 i = 1 for every i .
As a warmup, we ﬁrst show that linear predictors, and subsequently ﬁxed low degree polynomials cannot achieve the information-theoretic rate 12 of O ( (cid:27) 2 k log n ) – in fact, we will show that they achieve a “trivial” rate.
Furthermore, we will show that even if the degree of our polynomials is growing with n , if d = o (log n= log log n ) the state of affairs is similar.
B. W ARMUP : L INEAR P REDICTORS
As a warmup, and to illustrate the main ideas of the proof techniques, we ﬁrst consider the case of linear predictors.
(i.e. kernels of degree 1.)
The main idea is to use a bias-variance trade-off: namely, we show that the linear predictor we use, say f ( x ) = ⟨ ~ w; x ⟩ either has to have too high of a variance (when ∥ ~ w ∥ is large), or otherwise has too high of a bias.
(Recall, the bias captures how well the predictor captures the expectation.)
We prove:
Theorem B.1.
For any ~ w 2 R n ,
[Equation]
Before giving the proof, let us see how the theorem should be interpreted.
The trivial estimator which always returns 0 makes error of order (cid:13) 2 K and a good estimator (such as thresholding) should instead make error of order (cid:27) 2 K log n when (cid:13) >> (cid:27) p log n .
The next theorem shows that as long as the signal to noise ratio is not too high , more speciﬁcally as long as {+(cid:13) 
+} ( k=n ) =
o ( (cid:27) 2 ) , any linear estimator must make square loss of Ω( (cid:13) 2 k ) , i.e. not signiﬁcantly better than the trivial 0 estimate.
Note that the most interesting (and difﬁcult) regime is when the signal is not too much larger than the noise, e.g. (cid:13) 2 = (cid:27) 2 polylog ( n ) in which case it is deﬁnitely true that (cid:13) 2 ( k=n ) << (cid:27) 2 .
Proof.
Note that ⟨ ~
[Equation]
; x
[Equation]
which gives the following bias-variance decomposition for the square loss:
[Equation]
where in the second-to-last step we used that the covariance matrix of Z is (cid:13) 2 ( k [Equation] I , and in the last step we used that A is orthogonal.
Now observe that if we ﬁx R = ∥ ~ w ∥ 2 , then by the Pythagorean theorem the minimizer of the square loss is given by the projection of Aw onto the R -dilated unit sphere, so
~ w = √ R 2 =m ( Aw ) since ∥
Aw ∥ 2 = ∥ w ∥ 2 = p m .
In this case the square loss is then of the form
[Equation]
and the risk is minimized when
[Equation]
i.e. when
[Equation]
so the minimum square loss is
[Equation]
since m = n .
B. S TRUCTURE OF THE O PTIMAL E STIMATOR : P ROOF OF L EMMA 5.
Proof of Lemma 5.1.
Let X ′ =
A ⊤ X , so by orthogonality X ′ =
Z + (cid:24) ′ where (cid:24) ′ (cid:24) N (0 ; (cid:27) 2 Id ) .
Observe that if we look at the optimum over all functions f , we see that
[Equation]
where where in the ﬁrst step we used that the conditional expectation minimizes the squared loss, in the second step we used linearity of conditional expectation, and in the last step we used that Z i is independent of X ′̸
= i .
By the Pythagorean theorem, the optimal degree d polynomial f (cid:3) d is just the projection of ∑ i w i E
[ Z i j X ′ i ] onto the space of degree d polynomials.
On the other hand observe that
[Equation]
so the optimal projection f (cid:3) d is just ∑ i w i f (cid:3) i;d
( X ′ i ) where f (cid:3) i;d is just the projection of each of the
E [ Z i j X ′ i ] .
Therefore f (cid:3) d has no mixed monomials.
Remark B.1.
The previous calculation shows additionally that the problem of minimizing the squared loss for predicting Y is equivalent to that of minimizing the squared loss for the sparse regression problem of recovering Z .
It is a well-known fact that the information theoretic rate for sparse regression (with our normalization convention) is (cid:2)( (cid:27) 2 k ) (see for example (Rigollet, 2017)), and so the information-theoretic rate for predicting Y is the same, and is matched by Theorem A.1.
B. B IAS -V ARIANCE , F OURIER A NALYSIS AND P ROOF OF T HEOREM 4.
We recall that the lower bound for polynomials combines the observation of Lemma 5.1 with a biasvariance tradeoff calculation using Fourier analysis on orthogonal polynomials.
Concretely, since the noise we chose for the lower bound instance is Gaussian, the most convenient basis will be the Hermite polynomals.
We recall the probabilist’s Hermite polynomial He n ( x ) , deﬁned by the recurrence relation
[Equation]
(1) where He 0 ( x ) = 1 ; He 1 ( x ) = x .
In terms of this, the normalized Hermite polynomial H n ( x ) is
[Equation]
Let H n ( x ) for a vector of indices n 2 N m 0 denote the multivariate polynomial (cid:5) mi =1 H n i ( x i ) .
It’s easy to see the polynomials H n ( x ) form an orthogonal basis with respect to the standard m-variate
Gaussian distribution.
As a consequence, we get
[Equation]
which gives us Plancherel’s theorem:
Theorem B.2 (Plancherel in Hermite basis) .
Let f ( x ) = ∑ n b f ( n ) H n ( x=(cid:27) ) , then
[Equation]
We can use Plancherel’s theorem to get lower bounds on the noise sensitivity of degree d polynomials.
This will be an analogue of the variance.
Lemma B.1.
[Variance analogue in Hermite basis] Let f ( x ) =
∑ n b f ( n ) H n ( x=(cid:27) ) and let f ̸ =0 := f (cid:0) b f (0) .
Then
[Equation]
Proof.
First we suppose Z (and thus Y ) is ﬁxed and consider the randomness of the noise.
Let S denote the support of Z .
Recall that A ⊤ x = Z + (cid:24) ′ where (cid:24) ′ (cid:24) N (0 ; (cid:27) 2 I n (cid:2) n ) .
Deﬁne f Z (cid:24) ) :
= f ( Z + (cid:24) ) (cid:0) Y , then by Plancherel
[Equation]
Furthermore
[Equation]
because ( (cid:24) ′
+ Z ) j S c
= (cid:24) ′
j S c so by expanding out f Z in terms of the fourier expansion of f , we see c f Z ( n ) =
b f ( n ) for n such that supp ( n ) ̸(cid:26) S .
Finally the probability n (cid:26) S for n ̸ = 0 is upper bounded by the probability a single element of its support is in S , which is k=n . Therefore
[Equation] which proves the result. Next we give a lower bound for the bias, showing that if ∥ b f ̸ =0 ∥ 22 is small for a low-degree polynomial, it cannot accurately predict y .
Here we will assume f is of the form given by Lemma 5.1.
Lemma B.2 (Low variance implies high bias) .
Suppose f is a multivariate polynomial of degree d with no mixed monomials, i.e. f ( x )
=
∑ i f i ( x i ) where f i is a univariate polynomial of degree d .
Expand f in terms of Hermite polynomials as f ( x ) =
∑ n b f ( n ) H n ( x=(cid:27) ) .
Then
[Equation]
vuut
[Equation]
Before proving the lemma, let us see how it proves the main theorem:
Proof of Theorem 4.2.
By Lemma 5.1, Lemma B.1, and Lemma B.2 we have that for the f which minimizes the square loss among degree d polynomials, we have a variance-type lower bound
[Equation]
[Equation]
Let ∥ b f i ∥ 2 :=
[Equation]
ni =1 j ^ f ( ke i )
[-(cid:113)(cid:80) ni =
 | ˆ f ( ke i ) |-]j 2 .
Then averaging these lower bounds and simplifying using k < n=gives
[Equation]
Returning to the proof of Lemma B.2, we have:
Proof of Lemma B.2.
Since f has no mixed monomials, we get for the Hermite expansion that b f ( n ) =
unless j supp ( n ) j (cid:20) 1 .
Let X ′ :
=
A ⊤ X = Z + (cid:24) ′ where (cid:24) ′ (cid:24) N (0 ; (cid:27) 2 I ) .
Next observe by independence that
[Equation]
where the last inequality follows since there is a k=n chance that Z i (cid:24) N (0 ; (cid:27) 2 I ) , equivalently that
Z i ̸ = 0 .
By the conditional Jensen’s inequality we have
[Equation]
Observe that f i ( X ′ i ) = {+∑ dk =
 b f+}
[Equation]
ke [Equation]
H k
[Equation]
Z
[Equation] = [Equation] + (cid:24)=(cid:27) ) and let g i ( Z i ) :
=
E [ f i ( X ′ i ) j Z i ] (cid:0) Z i , so then g i is a polynomial of degree d in Z i .
Write the Hermite polynomial expansion of g i in terms of
H k ( Z i =(cid:13) ) as
[Equation]
then by Plancherel’s formula
[Equation]
and it remains to lower bound j b g
i (1) j .
By orthogonality and direct computation,
[Equation]
Now we upper bound the last term
E
[Equation]
[Equation]
[Equation]
[Equation]
where the second equality is by the law of total expectation and the last inequality is CauchySchwarz.
Using the recurrence relation (1), we can bound the sum of the absolute value of the coefﬁcients of H k ( x ) by k k = p k ! (cid:20) k k .
We can also bound the moments of the absolute value of a Gaussian by E (cid:24) (cid:24) N (0 ; 1) [ j (cid:24) j k ] (cid:20) k k .
Therefore by Holder’s inequality
[Equation]
Therefore by reverse triangle inequality
[Equation]
Remark B.2.
Remarks on results:
We make a few remarks regarding the results in this section.
Recall that (cid:13) 2 k is the square loss of the trivial zero-estimator.
Suppose as before that (cid:13) = (cid:2)( (cid:27) 2 polylog ( n )) , then we see that if d = o (log n= log log n ) then the denominator of the lower bound tends to 1 , hence any such polynomial estimator has a rate no better than that of the trivial zero-estimate.
It is possible to derive a similar statement to Theorem 4.2 that holds with high probability instead of in expectation for polynomials of degree o (log n= log log n ) .
All that is needed is to bound the contribution to the expectation from very rare tail events when the realization of the noise (cid:24) is atypically large.
Since the polynomials we consider are very low degree o (log n= log log n ) , they can only grow at a rate of x d = x o (log( n ) = loglog n ) ; thus standard growth rate estimates (e.g. the Remez inequality) combined with the Gaussian tails of the noise can be used to show that a polynomial which behaves reasonably in the high-probability region (e.g. which has small w.h.p.
error) cannot contribute a large amount to the expectation in the tail region.
C U PPER B OUNDS FOR P OLYNOMIAL K ERNELS
In this section, we construct polynomials achieving close to the information-theoretic optimal rate of degree only O (log 2 m ) .
Recall this is nearly optimal due to our previous lower bound of Ω(log n ) .
As previously mentioned, the key technical result here will be Theorem 5.2, giving the construction of a new polynomial approximation to ReLU.
Before proceeding to the proof of that theorem, we show how it implies the ﬁnal result, Theorem 4.3.
Towards that, we substitute our polynomial construction for (cid:26) (cid:28) into our ReLU neural network and derive the analogous version of Lemma A.2.
First, deﬁne M (cid:28) = M + 2 (cid:28) and let
[Equation]
) where p is the polynomial constructed in Theorem 5.2.
We then have:
Lemma C.1.
Suppose ϵ; (cid:28) > 0 and M (cid:21) 1 .
Then for all d (cid:21) d 0 = Ω( M (cid:28) (cid:28) log 2 ( M (cid:28) ϵ(cid:28) )) , for j x j 2 ( (cid:28); M (cid:28) ) we have j ~ [Equation] d;(cid:28);
[Equation]
x ) [Equation] + ϵ and for j x j (cid:20) (cid:28) we have
[Equation]
Proof.
By the guarantee of Theorem 5.2, we see that for for j x j (cid:20) (cid:28) that
[Equation]
Thus we see that taking d = Ω( M (cid:28) [Equation] log 2 ( M (cid:28) ϵ(cid:28) )) sufﬁces to make the latter expression at most ϵ .
Similarly for j x j > (cid:28) we know that
[Equation]
and taking d = Ω( M (cid:28) (cid:28) log [-
-]2 ( M (cid:28) ϵ(cid:28) )) with sufﬁciently large constant guarantees the middle term is at most (cid:28) and the last term is at most ϵ .
Using this, we can show that if we use a polynomial of degree Ω(( M=(cid:27) p log n ) log 2 m ) we can achieve similar statistical performance to the ReLu network:
Lemma C.2.
Suppose A is (cid:22) -incoherent i.e. ∥ A ⊤ A (cid:0) Id ∥ 1 (cid:20) (cid:22) .
Let z be an arbitrary ﬁxed vector such that ∥ z ∥ 1 (cid:20) M and j supp ( z ) j (cid:20) k .
Suppose x = Az + (cid:24) where (cid:24) (cid:24) N (0 ; (cid:27) 2 Id n (cid:2) n ) .
Then for some (cid:28) = (cid:2)( (cid:27) √ (1 + (cid:22) ) log m + (cid:22)M ) , for any d (cid:21) d [-
-]0 =
Ω( M (cid:28) (cid:28) log 2 ( M (cid:28) m=(cid:28) 2 )) , if we take ^ z :
= ~ (cid:26) (cid:10) n d;(cid:28);M ( A ⊤ x ) , then with high probability we have ∥ ^ z (cid:0) z ∥ 1 (cid:20) 6 k(cid:28) .
Proof.
Apply Lemma C.1 with ϵ =
(cid:28)=m .
Then we see for j x j 2 ( (cid:28); M (cid:28) ) we havn
[Equation]
and for j x j (cid:20) (cid:28) we have
[Equation]
m
Observe that
[Equation]
Note that entry i of A ⊤ (cid:24) is ⟨ A i ; (cid:24) ⟩ where {+∥ A i ∥ 
+}
(cid:20) (1 + (cid:22) ) so ( A t (cid:24) ) i is Gaussian with variance at most (cid:27)
(1 + (cid:22) ) .
By choosing (cid:28) with sufﬁciently large constant, then applying the sub-Gaussian tail bound and union bound, with high probability all coordinates not in the true support are thresholded to at most (cid:28)=m .
Similarly we see that for each of the coordinates in the support, an error of at most 5 (cid:28) is made.
Therefore ∥ ^ z (cid:0) z ∥ 1 (cid:20) 5 k(cid:28) + m ( (cid:28)=m ) (cid:20) 6 k(cid:28) .
Now we have all the ingredients to prove Theorem 4.3:
Proof of Theorem 4.3.
Deﬁne an estimate for Y by taking ^ Z d;M := ~ (cid:26) (cid:10) n d;(cid:28);M ( A ⊤ X ) where (cid:28) is deﬁned as in the Lemma, and then taking ^ Y d;M := ⟨ w; ^ Z d;M ⟩ .
Applying the previous Lemma, we get analogous versions of Theorem 4.1 by the same argument as in that theorem.
Finally, we return to the proof of the key Theorem 5.2:
Proof of Theorem 5.2.
We start with the case where R = 1 = 2 .
We build the approximation in two steps.
First we approximate ReLu by the following “annealed” version of ReLu, for parameters (cid:12) > (cid:25); (cid:28) > 0 to be optimized later:
[Equation]
Observe that when we look at negative inputs, g (cid:12) ( (cid:0) x ) = 1 (cid:12) log(1 + e (cid:0) (cid:12)x ) (cid:20) 1 (cid:12) e (cid:0) (cid:12)x .
Therefore when x < 0 , f (cid:12) ( x ) (cid:20) 1 (cid:12) e (cid:0) (cid:12)(cid:28) .
For the second step,, we need to show f (cid:12) can be well-approximated by low-degree polynomials.
In fact, because f (cid:12) is analytic in a neighborhood of the origin, it turns out that its optimal rate of approximation is determined exactly by its complex-analytic properties.
More precisely, deﬁne D (cid:26) to be the region bounded by the ellipse in C
= R 2 centered at the origin with equation
[Equation]
with semi-axes a = 12 ( (cid:26) + (cid:26) (cid:0) 1 ) and b = 12 j (cid:26) (cid:0) (cid:26) (cid:0) 1 j ; the focii of the ellipse are (cid:6) 1 .
For an arbitrary function f : [ (cid:0) 1 ; 1] ! R , let E d ( f ) denote the error of the best polynomial approximation of degree d in inﬁnity norm on the interval [ (cid:0) 1 ; 1] of f .
Then the following theorem of Bernstein exactly characterizes the growth rate of E d ( f ) :
Theorem C.1 (Theorem 7.8.1, (DeVore and Lorentz, 1993)) .
Let f be a function deﬁned on [ (cid:0) 1 ; 1] .
Let (cid:26) 0 be the supremum of all (cid:26) such that f has an analytic extension on D (cid:26) .
Then
[Equation]
For our application we need only the upper bound
and we need a quantitative estimate for ﬁnite n .
Following the proof of the upper bound in (DeVore and Lorentz, 1993), we get the following result:
Theorem C.2.
Suppose f is analytic on the interior of D (cid:26) 1 and j f ( z ) j (cid:20) M on the closure of D (cid:26) 1 .
Then
[Equation]
The proof is fairly simple: by writing f in terms of cos( x ) one gets an expansion into Chebyshev polynomials and it sufﬁces to bound the coefﬁcients of the corresponding Fourier series: to do this, we write them as integrals over the unit circle, and use the analyticity assumption on D (cid:26) 1 to contour shift the integral to a different circle, which immediately gives us the desired exponential decay.
For details see ( DeVore and Lorentz, 1993).
We will now apply this theorem to g (cid:12) .
First, we claim that g (cid:12) is analytic on D (cid:26) 1 where (cid:26) 1 is the solution to this equation for the semi-axis of the ellipse:
[Equation]
which is
[Equation]
To see this, ﬁrst extend log to the complex plane by taking a branch cut at ( (cid:0)1 ; 0] .
To prove g (cid:12) is analytic on D (cid:26) 1 , we just need to prove that 1 + e (cid:12)z avoids ( (cid:0)1 ; 0] for z 2 D (cid:26) 1 .
This follows because by the deﬁnition of (cid:26) 1 , for every z 2 D (cid:26) 1 , ℑ ( z ) < (cid:25) 2 (cid:12) hence ℜ (1 + e (cid:12)z ) (cid:21) 1 .
We also see that for z 2 D (cid:26) 1 ,
[Equation]
Therefore by Theorem C.2 we have
[Equation]
) (cid:20) 12 (cid:12) (cid:25) ( [Equation] (cid:25)= [Equation]
(cid:12) [Equation] (cid:0) n (cid:20) 12 (cid:12) (cid:25) [Equation]
(cid:0) [Equation] (cid:12) where in the last step we used that 1 + x (cid:21) exp( x= 2) for x < 1 = 2 and that (cid:12) > (cid:25) . Let ~ g (cid:12);d denote the best polynomial approximation to g (cid:12) of degree d and let ~ f (cid:12);(cid:28);d = ~
g (cid:12);d ( x (cid:0) (cid:28) )
Thus for x 2 [ (cid:0) 1 + (cid:28); 0] , j
[Equation]
Take (cid:12) = √ (cid:25)d= 4 (cid:28) and require d > 7 so that (cid:12) > 1 , then for x 2 [ (cid:0) 1 + (cid:28); 0] , [Equation]
[Equation]
For x 2 (0 ; 1 (cid:0) (cid:28) ] we have by the 1-Lipschitz property of g (cid:12) and calculus that
[Equation]
so
[Equation]
Plugging in our value of (cid:12) and using log 2 (cid:20) 1 gives
[Equation]
Now the result for general R follows by taking p
d ( x ) = 2 R ~ f (cid:12);(cid:28);d ( x= 2 R ) , since 2 R (cid:1) ReLu ( x= 2 R ) = ReLu ( x ) and [ (cid:0) 1 = 2 ; 1 = 2] (cid:26) [ (cid:0) 1 + (cid:28); 1 (cid:0) (cid:28) ] .