Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?
Anonymous Author(s)
AffiliationAddress email
Abstract
We study the theory of neural network (NN) from the lens of classical nonparametric regression problems with a focus on NN’s ability to adaptively estimatefunctions with heterogeneous smoothness — a property of functions in Besov or
Bounded Variation (BV) classes.
Existing work on this problem requires tuningthe NN architecture based on the function spaces and sample sizes.
We consider a“Parallel NN” variant of deep ReLU networks and show that the standard weightdecay is equivalent to promoting the ℓ p -sparsity ( 0 < p < 1 ) of the coefficientvector of an end-to-end learned function bases, i e., a dictionary.
Using this equivalence, we further establish that by tuning only the weight decay, such Parallel
NN achieves an estimation error arbitrarily close to the minimax rates for both the
Besov and BV classes.
Notably, it gets exponentially closer to minimax optimalas the NN gets deeper.
Our research sheds new lights on why depth matters andhow NNs are more powerful than kernel methods.
Introduction
Why do deep neural networks (DNNs) work better?
They are universal function approximators [6],but so are splines and kernels.
They learn data-driven representations, but so are the shallower andlinear counterparts such as matrix factorization.
There is surprisingly little theoretical understandingon why DNNs are superior to these classical alternatives.
In this paper, we study DNNs in nonparametric regression problems — a classical branch of statistical theory and methods with more than half a century of associated literature [25, 7, 46, 10, 23, 37,33].
Nonparametric regression addresses the following fundamental problem:
• Let y i
=
f
( x i ) + Noise for i = 1 , ..., n .
How can we estimate a function f using data points( x 1 , y 1 ) , ..., ( x n , y n ) in conjunction with the knowledge that f belongs to a
function class F ?
Function class F typically imposes only weak regularity assumptions such as smoothness, whichmakes nonparametric regression widely applicable to real-life applications under weak assumptions.
Local adaptivity.
A subset of nonparametric regression techniques were shown to have the propertyof local adaptivity [24] in both theory and practice.
These include wavelet smoothing [10], locallyadaptive regression splines [24], trend filtering [40, 47] and adaptive local polynomials [2, 3].
Wesay a nonparametric regression technique is locally adaptive if it can cater to local differences insmoothness, hence allowing more accurate estimation of functions with varying smoothness andabrupt changes.
In light of such a distinction, it is natural to consider the following question.
Are NNs locally adaptive ,
i
e., optimal in learning functions with heterogeneous
smoothness?
This is a timely question to ask, partly because the bulk of recent theory of NN leverages its asymptotic Reproducing Kernel Hilbert Space (RKHS) in the overparameterized regime [21, 5, 1].
RKHSbased approaches, e.g., kernel ridge regression with any fixed kernels are suboptimal in estimatingfunctions with heterogeneous smoothness [9].
Therefore, existing deep learning theory based on
RKHS does not satisfactorily explain the advantages of neural networks over kernel methods.
We build upon the recent work of Suzuki [39] and Parhi and Nowak [29] who provided encouragingfirst answers to the question above about the local adaptivity of NNs.
Specifically, Parhi and Nowak[29, Theorem 8] showed that a two-layer truncated power function activated neural network witha non-standard regularization is equivalent to the locally adaptive regression splines (LARS)
This connection implies that such non-standard NNs achieve the minimax rate for the (higher order)bounded variation (BV) classes.
We provide a detailed discussion about this work in Section B.
Suzuki [39] showed that multilayer ReLU DNNs can achieve minimax rate for the Besov class,but requires the width, depth and an artificially imposed sparsity-level of the DNN weights to becarefully calibrated according to parameters of the Besov class, thus is quite different from how
DNNs are typically trained in practice.
In this paper, we aim at addressing the same locally adaptivity question for a more commonly usedneural network with standard weight decayed training.
Parallel neural networks.
We restrict our attention on a special network architecture called parallelneural network [18, 15] which learns an ensemble of subnetworks — each being a multilayer ReLU
DNNs.
Parallel NNs have been shown to be more well-behaved both theoretically [18, 51, 16, 15, 14]and empirically [50, 44].
Moreover, the idea of parallel NNs was used in many successful NNarchitectures such as SqueezeNet, ResNext and Inception (see [15] and the references therein).
Weight decay.
Weight decay is a common method in deep learning to reduce overfitting.
Empirically, the regularizer is not necessarily explicit.
Many tricks in deep learning, including earlystopping [48], quantization [20], and dropout [45] have similar effect as weight decay.
In this paper,we make no assumption on the training method thus there is no (implicit) regularizers apart fromweight decay.
Summary of results.
Our main contributions are:
We prove that the (standard) weight decay in training an L -layer parallel ReLU-activatedneural network is equivalent to a sparse ℓ p penalty term (where p = 2 /L ) on the linearcoefficients of a learned representation.2.
We show that neural networks can approximate B-spline basis functions of any order without the need of choosing the order parameter manually.
In other words, neural networks
can adapt to functions of different order of smoothness, and even functions with different
smoothness in different regions in their domain.
We show that the estimation error of weight decayed parallel ReLU neural network decreases polynomially with the number of samples up to a constant error for estimatingfunctions with heterogeneous smoothness in the both BV and Besov classes, and the exponential term in the error rate is close to the minimax rate.
Notably, the method requirestuning only the weight decay parameter.4.
We find that deeper models achieve closer to the optimal error rate.
This result helps explainwhy deep neural networks can achieve better performance than shallow ones empirically.
The above results separate NNs with any linear methods such as kernel ridge regression.
To thebest of our knowledge, we are the first to demonstrate that standard techniques (“weight decay”and ReLU activation) suffice for DNNs in achieving the optimal rates for estimating BV and Besovfunctions.
Preliminary2.1 Notation and Problem Setup.
We denote regular font letters as scalars, bold lower case letters as vectors and bold upper case lettersas matrices.
a ≲ b means a ≤ Cb for some constant C that does not depend on a or b , and a ≂ b
[Table]
Let f 0 be the target function to be estimated.
The training dataset is D n := { ( x i , y i ) , y i =
f 0 ( x i )
+ϵ i , i ∈ [ n ] } , where x i are fixed and
ϵ i are zero-mean, independent Gaussian noises with variance σ 2 .
In the following discussion, we assume x i ∈
[0 , 1] d , f 0 ( x i ) ∈ [ − 1 , 1] , ∀ i .
We will be comparing estimators under the mean square error (MSE), defined as
[Equation]
The optimal worst-case MSE is described by R ( F ) :
= min ˆ f max f 0 ∈F MSE( ˆ f ) , we say thatˆ f is optimal if MSE( ˆ f ) ≂ R ( F ) .
The empirical (square error) loss is defined as ˆ L ( ˆ f ) :=1 n (cid:80) ni
=
( ˆ f ( x i ) − y i ) 2 .
The corresponding population loss is L ( ˆ f ) :
=
E [ 1 n (cid:80) ni
=
( ˆ f ( x i ) − y ′
i )
2 | ˆ f ]where y ′
i are new data points.
It is clear that E [ L ( ˆ f )]
=
MSE[ ˆ f ] + σ 2 .
2.2 Besov Spaces and Bound Variation Space
Besov space , denoted as B αp,q , is a flexible function class parameterized by α, p, q whose definitionis deferred to Section C.1.
Here α ≥ 0 determines the smoothness of functions, 1 ≤ p ≤ ∞determines the averaging (quasi-)norm over locations, 1 ≤ q ≤ ∞ determines the averaging (quasi-)norm over scale which plays a relatively minor role.
Smaller p is more forgiving to inhomogeneityand loosely speaking, when the function domain is bounded, smaller p induces a larger functionspace.
On the other hand, it is easy to see from definition that B αp,q ⊂ B αp,q ′ , if q < q ′ .
Without lossof generalizability, in the following discussion we will only focus on B αp, ∞ .
When p = 1 , the Besov space allows higher inhomogeneity, and it is more general than the Sobolevor Hölder space.
Bounded variation (BV) space is a more interpretable class of functions with spatially heterogeneous smoothness [10].
It is defined through the total variation (TV) of a function.
For( m + 1) th differentiable function f : [0 , 1] → R , the m th order total variation is defined as
TV ( m ) ( f ) :
=
TV ( f ( m +1) ) =
(cid:82) [0 , 1] | f
( m +1)
( x ) | dx, and the corresponding m th order Bounded
Variation class BV ( m ) :
= { f : TV ( f ( m ) ) < ∞} .
The more general definition is given in Section C.2.
Bounded variation class is tightly connected to Besov classes.
Specifically [8]:
[Equation]
This allows the results derived for the Besov space to be easily applied to BV space.
Minimax MSE It is well known that minimax rate for Besov and 1D BV classes are O ( n −
α 2 α + d )and O ( n − (2 m +2) / (2 m +3) ) respectively .
The minimax rate for linear estimators in 1D BV classes isknown to be O ( n − (2 m +1) / (2 m +2) )
x W (1)
W (2) . . .
W ( L − 1)
W ( L )
W (2)
W (2) M . . .
W ( L − 1)
W ( L − 1) M . . .
W ( L )
W ( L ) M y min
f j L ( (cid:80) j f j ) + λ (cid:80)
Lℓ =1 (cid:80) Mj
=
∥ W ( ℓ ) j ∥ 2 F .
W ( ℓ ) :
W ( ℓ ) W ( ℓ )
W ( ℓ ) M
[Figure]
diagonal weights
[Figure]
x ¯ W (1) ¯ W (2) . . .
¯ W ( L − 1) ¯ W ( L ) ¯ W (2)
¯ W (2) M . . .
¯ W ( L − 1)¯ W
( L − 1) M . . .
¯ W ( L )¯ W ( L ) M . . .
aa M y min { a j , ¯ f j } L ( (cid:80)
j a j ¯ f j ) s.t.
(cid:80) Mj
[Figure]
| a j | 2 /L ≤ P ′ .
[Figure]
Main Results: Parallel ReLU DNNs
Consider a parallel neural network containing M multi layer perceptrons (MLP) with ReLU activation functions called subnetworks .
Each subnetwork has width w and depth L .
The input is fed toall the subnetworks, and the output of the parallel NN is the summation of the output of each subnetwork.
The architecture of a parallel neural network is shown in Figure 1a.
This parallel neuralnetwork is equivalent to a vanilla neural network with block diagonal weights in all but the first andthe last layers (Figure 1(c)).
Let W ( ℓ ) j and b ( ℓ ) j denote the weight and bias in the ℓ -th layer in thej -th subnetwork respectively.
Training this model with weight decay returns:
[Equation]
where f ( x ) = (cid:80)
Mj
=
f k ( x ) denotes the parallel neural network, f j ( · ) denotes the j -th subnetwork,and λ > 0 is a fixed scaling factor.
Theorem 1.
For any fixed α − d/p > 1 , q ≥ 1 , L ≥ 3 , for any f 0 ∈ B αp,q , given an L -layer parallelneural network satisfying
• The width of each subnetwork is fixed and large enough: w ≳ d .
See Theorem 9 for the
detail.
• The number of subnetworks is large enough: M ≳ m
d n
[Equation]
pL )
where m
= ⌈ α − 1 ⌉ .
With proper choice of the parameter of weight decay λ , the solution ˆ f parameterized by (2) satisfies
[Equation]
(3) where ˜ O shows the scale up to a logarithmic factor, and the trailing constant term decreases exponentially with L .
We explain the proof idea in the next section,but defer the extended form of the theorem and the fullproof to Section F. Before that, we comment on a few interesting aspects of the result.
Near optimal rates and the effect of depth.
The first term in the MSE bound is the estimation errorand the second term is (part of) the approximation error of this NN.
Recall that the minimax rate ofa Besov class is O ( n − 2 α 2 α + d ) thus as the depth parameter L increases it can get arbitrarily
Corollary 2.
Under the conditions of Theorem 1, for any f 0 ∈ B αp,q , there is a numerical constant
C such that when we choose C log n ≤ L ≤ 100 C log ,
[Equation]
where ˜ O hides only logarithmic factors and the o (1) factor in the exponent is O (1 / log( n )) .
This result says that deeper parallel neural networks achieves lower error and gets closer to thestatistical limit.
Overparameterization and sparsity.
We also note that the result does not depend on M as longas M is large enough.
This means that the neural network can be arbitrarily overparameterizedwhile not overfitting.
The underlying reason is sparsity .
As it will become clearer in the proofsketch, weight decayed training of a parallel L-layer ReLU NNs is equivalent to a sparse regressionproblem with an ℓ p penalty assigned to the coefficient vector of a learned dictionary.
Here p = 2 /Lwhich promotes even sparser solutions than an ℓ 1 penalty.
No architecture tuning.
For any fixed L , the required architecture of the model does not dependon the dataset or the target function ( n, α ) expect the number of subnetworks M , for which the onlyrequirement is being large enough.
As a result, one can design a model using a large guess on M ,and achieve the claimed near-optimal error rate by only tuning the weight decay parameter.
Bounded variation classes.
Thanks to the Besov space embedding of the BV class (1), our theoremalso implies the result for the BV class in 1 D .
Corollary 3.
If the target function is in bounded variation class f 0 ∈ BV ( m ) , For any fixed L ≥ 3 ,for a neural network satisfying the requirements in Theorem 1 with d = 1 and with proper choice ofthe parameter of weight decay λ , the NN ˆ f parameterized by (5) satisfies
[Equation]
Const.
where ˜ O shows the scale up to a logarithmic factor, and the trailing constant term decreases exponentially with L .
It is known that any linear estimators such as kernel smoothing and smoothing splines cannot havean error lower than O ( n − (2 m +1) / (2 m +2) ) for BV ( m ) [10].
This partly explains the advantage of
DNNs over kernels.
Representation learning and adaptivity.
The results also shed a light on the role of representationlearning in DNN’s ability to adapt.
Specifically, different from the two-layer NN in [29], whichachieves the minimax rate of BV ( m ) by choosing appropriate activation functions using each m ,each subnetwork of a parallel NN can learn to approximate the spline basis of an arbitrary order,which means that if we choose L to be sufficiently large, such Parallel NN with optimally tuned λ issimultaneously near optimal for m = 1 , 2 , 3 , . . . .
In fact, even if different regions of the space hasdifferent orders of smoothness, the NN will still be able to learn appropriate basis functions in eachlocal region.
To the best of our knowledge, this is a property that none of the classical nonparametricregression methods possess.
Synthesis v.s. analysis methods.
Our result could also inspire new ideas in estimator design.
There are two families of methods in non-parametric estimation.
One called synthesis frameworkwhich focuses on constructing appropriate basis functions to encode the contemplated structuresand regress the data to such basis, e.g., wavelets [10].
The other is called analysis framework whichuses analysis regularization on the data directly (see, e.g., RKHS methods [37] or trend filtering[40]).
It appears to us that parallel NN is doing both simultaneously.
It has a parametric familycapable to synthesizing an O ( n ) subset of an exponentially large family of basis, then implicitlyuse sparsity-inducing analysis regularization to select the relevant basis functions.
In this way theestimator does not actually have to explicitly represent that exponentially large set of basis functions,thus computationally more efficient.
Proof Overview
We start by first proving that a parallel neural network trained with weight decay is equivalent to anℓ p -sparse regression problem with representation learning (Section 4.1); which helps decompose its
MSE into an estimation error and approxmation error.
Then we bound the two terms in Section 4.2and Section 4.3 respectively.
4.1 Equivalence to ℓ p Sparse Regression with a Learned Feature Representation
It is widely known that ReLU function is 1-homogeneous: σ ( ax ) = aσ ( x ) , ∀ a ≥ 0 , x ∈ R .
In anyconsecutive two layers in a neural network (or a subnetwork), one can multiply the weight and biasin one layer with a positive constant, and divide the weight in another layer with the same constant.
The neural network after such transformation is equivalent to the original one:
[Equation]
This property allows us to reformulate (2) to an ℓ p sparsity constraint problem:
Proposition 4.
Fix the input dataset D n and a constant c 1 > 0 .
For any λ > 0 , there exists P ′ > 0such that (2) is equivalent to the following problem:
[Equation]
where ¯ f j ( · ) is a subnetwork with parameters ¯W ( ℓ ) j , ¯ b ( ℓ ) j .
This equivalent model is demonstated in Figure 1b.
The proof can be found in Section D.1.
Theconstraint ∥
¯W (1) j ∥ F ≲ √ d, ∥ ¯W ( ℓ ) j ∥ F ≲ √ w, ∀ ℓ > 1 is typical in deep learning for better numericalstability.
The equivalent model in Proposition 4 is also a parallel neural network, but it appends onelayer with parameters { a k } at the end of the neural network and the constraint on the Frobeniusnorm is converted to the 2 /L norm on the factors { a k } .
Since L ≫ 2 in a typical application,2 /L
≪ 1 and this constraint can enforce a sparser model than that in Section B.
There are two useful implications of Proposition 4.
First, it gives an intuitive explanation on howa weight decayed Parallel NN works.
Specifically, it can be viewed as a sparse linear regressionwith representation learning.
Second, the conversion into the constrained form allows us to adaptgeneric statistical learning machinery (a self-bounding argument) from Suzuki [39, Proposition 4]for studying this constrained ERM problem.
The adaptation is nontrivial because (1) our regression problem has a fixed design (so data points arenot iid); (2) there is an unconstrained subspace with no bounded metric entropy.
Specifically, our
Proposition 15 shows that the MSE of the regression problem can be bounded by
[Equation]
in which F decomposes into F ∥ ×F ⊥ , where F ⊥ is an unconstrained subspace with finite dimension,and F ∥ is a compact set in the orthogonal complement with a δ -covering number of N ( F ∥ , δ, ∥·∥ ∞ )in ∥·∥ ∞ -norm.
This decomposes MSE into an approximation errorand an estimation error.
The novelanalysis of these two represents the major technical contribution of this paper.
4.2 Estimation Error Analysis
The decomposition above reveals that to bound the estimation error, it suffices to compute the covering number of the constraint set in the sup-norm of the function it represents.
Previous results that bound the covering number of neural networks [49, 39] depends on the widthof the neural networks explicitly, which cannot be applied when analysing a potentially infinitelywide neural network.
In this section, we leverage the ℓ p -norm bounded coefficients to avoid thedependence in M in the covering number bound.
Theorem 5.
The covering number of the model defined in (5) apart from the bias in the last layersatisfies
[Equation]
The proof can be found in Section D.2.
It requires the following lemma:
Lemma 6.
log N ( G , δ ) ≲
k log(1 /δ ) for some finite c 3 , and for any g ∈ G , | a | ≤ 1 , we haveag ∈ G .
The covering number of F = (cid:110)(cid:80) Mi
=1 a
i g i (cid:12)(cid:12)(cid:12) g i ∈ G , ∥ a ∥ pp
[Equation]
P
[Equation]
< p <
[Equation]
for any P > 0satisfies
[Equation]
up to a double logarithmic factor.
See Section D.3 for the proof of Lemma 6.
The covering number in Theorem 5 does not dependon the number of subnetworks M .
In other words, it provides a bound of estimation error for anarbitrarily wide parallel neural network as long as the total Frobenius norm is bounded.
4.3 Approximation Error Analysis
The approximation error analysis involves two steps.
In Section 4.3.1, we analyse how a subnetworkcan approximate a B-spline basis.
Then in Section 4.3.2 we show that a sparse linear combinationof B-spline bases approximates Besov functions.
Both add up to the total error in approximating
Besov functions with a parallel neural network (Theorem 9).
4.3.1 Approximation Error of B-spline Basis Function
As is shown in Section C.1, functions in Besov space can be alternatively represented in a sequencespace via the coefficients of a cardinal B-spline basis.
In this section we study the approximationability of ReLU neural networks to B-spline basis function.
Proposition 7.
Let M m,k, s be the B-spline of order m with scale 2 − k in each dimension and positions ∈ R
d : M m,k, s ( x ) :=
M m (2 k ( x − s )) , M m is defined in (11).
There exists a parallel neuralnetwork that has the structure and satisfy the constraint in Proposition 4 for d -dimensional inputand one output, containing M
=
O ( m d ) subnetworks, each of which has width w =
O ( d ) anddepth L =
O (log( c ( m, d ) /ϵ )) for some constant w, c that depends only on m and d , denoted as˜ M m ( x ) , x ∈ R d , such that
• | ˜ M m,k, s ( x ) − M m,k, s ( x ) | ≤ ϵ , if 0 ≤ 2 k
( x i − s i ) ≤ m + 1 , ∀ i ∈
[ d ] ,• ˜ M m,k, s ( x ) = 0 ,
otherwise.• The weights in the last layer satisfy ∥ a ∥ 2 /L 2 /L ≲ 2 k m d e 2 md/L .
The proof can be found in Section E.1.
Note that the product of the coefficients among all the layersare proportional to 2 k , instead of 2 km when approximating truncated power basis functions.
This isbecause the transformation from M m to M m,k, s only scales the domain of the function by 2 k , whilethe codomain of the function is not changed.
To apply the transformation to the neural network, oneonly need to scale weights in the first layer by 2 k , which is equivalent to scaling the weights in eachlayer bt 2 k/L and adjusting the bias according.
4.3.2 Approximation Error in Besov Space
With the results given in Section 4.3.1, we can estimate the approximation error of parallel ReLUneural networks to functions in Besov space.
Proposition 8.
Let α − d/p > 1 , r > 0 .
For any function in Besov space f 0 ∈ B αp,q and any positive integer ¯ M , there is an ¯ M -sparse approximation using B-spline basis of order m satisfying 0 < α < min( m, m − 1 + 1 /p ) :
ˇ f ¯ M = (cid:80) ¯ Mi =1 a k
i , s i M m,k i , s i for any positive integer
¯ M such that the approximation error is bounded as ∥
ˇ f ¯ M − f
∥
r ≲ ¯ M − α/d ∥ f 0 ∥ B αp,q , and the coefficients satisfy
[Equation]
The proof can be found in Section E.2.
Remark 1.
The requirement in Proposition 8: α − d/p > 1 is stronger than the condition typicallyfound in approximation theorem α − d/p ≥
[11], so-called “Boundary of continuity”, or thecondition in Suzuki [39] α > d (1 /p − 1 /r )
+ .
This is because although the functions in B αp,q when0 ≤ α − d/p < 1 can be approximated by B-spline basis, the sum of weighted coefficients may notconverge.
One simple example is the step function f step ( x ) = 1 ( x ≥ 0 . 5) , f step ∈ B 11 , ∞ .
Althoughit can be decomposed using first order B-spline basis as in (10), the summation of the coefficients isinfinite.
Actually one only needs a ReLU neural network with one hidden layer and two neurons toapproximate this function to arbitrary precision, but the weight need to go to infinity.
Theorem 9.
Under the same condition as Proposition 8, for any positive integer ¯ M , any functionin Besov space f 0 ∈ B αp,q can be approximated by a parallel neural network with no less than
O ( m d ¯ M ) number of subnetworks satisfying:
Each subnetwork has width w =
O
( d ) and depth L .2.
The weights in each layer satisfy ∥ ¯W ( ℓ ) k
∥ F ≤ O ( √ w ) except the first layer ∥ ¯W (1) k
∥ F ≤O ( √ d ) ,
The scaling factors have bounded 2 /L
-norm: ∥{ a j }∥ 2 /L 2 /L ≲ m d e 2 md/
L ¯ M 1 − 2 / ( pL ) .4.
The approximation error is bounded by
[Equation]
where
c 4 , c 5 , c 6 are constants that depend only on m, d and p .
Here ¯ M is the number of “active” subnetworks, which is not to be confused with the number ofsubnetworks at initialization.
The proof can be found in Section E.3.
Using the estimation error in Theorem 5 and approximation error in Theorem 9, by choosing ¯ M tominimax the total error, we can conclude the sample complexity of parallel neural networks usingweight decay, which is the main result (Theorem 1) of this paper.
See Section F for the detail.
Experiment
We empirically compare a parallel neural network (PNN) and a vanilla ReLU neural network (NN)with smoothing spline, trend filtering (TF) [40], and wavelet denoising.
Trend filtering can beviewed as a more efficient discrete spline version of locally adaptive regression spline and enjoys thesame optimal rates for the BV classes.
Wavelet denoising is also known to be minimax-optimal forthe BV classes.
The results are shown in Figure 2.
We use two target functions: a Doppler functionwhose frequency is decreasing(Figure 2(a)-(c)), and a combination of piecewise linear function andpiecewise cubic function, or “vary” function (Figure 2(d)-(f)).
We repeat each experiment 10 timesand take the average.
The shallow area in Figure 2(b)(e) shows 95% confidence interval by invertingthe Wald’s test.
The degree of freedom (DoF) is computed based on Tibshirani [41].
As can be shown in the figure, both TF and wavelet denoising can adapt to the different levels ofsmoothness in the target function, while smoothing splines tend to be oversmoothed where the targetfunction is less smooth (the left side in (a)(d), enlarged in (g)).
The prediction of PNN is similar to
TF and wavelet denoising and shows local adaptivity.
Besides, the MSE of PNN almost follows thesame trend as TF and wavelet denoising which is consistent with our theoretical understanding thatthe error rate of neural network is closer to locally adaptive methods.
Notably PNN, TF and waveletdenoising achieve lower error at a much smaller degree-of-freedom than smoothing splines.
In a vanilla NN, weight decay is equivalent to ℓ 1 regularizer in any two successive layers, but to thebest of our knowledge it does not lead to sparse representation learning unless some specific sparsestructure is enforced.
While our theory does not apply to vanilla neural networks, the results seemto suggest the NN behaves similar to smoothing spline and is not locally adaptive.
There are some mild drops in the best MSE one can achieve with NN vs TF in both examples.
Weare surprised that the drop is small because NN needs to learn the basis functions that TF essentiallyhard-coded.
The additional price to pay for using a more adaptive and more flexible representationlearning method seems not high at all.
[Figure] [Figure]
[Figure] [Figure]
In Figure 2(c)(f), we give the output all the “active” subnetwork, i.e. the subnetworks whose output isnot a constant.
Notice that the number of active subnetworks is much smaller than the initialization.
This is because weight decay induces ℓ p sparsity and the weight in most of the subnetworks reducestowards 0 after training.
More details are shown in Section G.
In Figure 2(h)(i), we plot the MSE versus the number of training samples for “Doppler” and “Vary”respectively.
It is clear that parallel NN works the best overall.
In (i), we further compare the scalingof the MSE against the minimax rate ( n − 4 / 5 ) and the minimax linear rate ( n − 3 / 4 ), i.e., the best ratekernel methods could achieve.
As is predicted by our theory, when n is large, the MSE of parallelneural networks and trend filtering decreases at almost the same rate as the minimax rate, whilesmoothing splines, as expected, is converging at the (suboptimal) minimax linear rate.
Interestingly,vanilla NN seems to converge at the optimal rate too on this example.
It remains an open questionwhether vanila NN is merely “lucky” on this example, or it also achieves the minimax rate for allfunctions in BV(m).
Conclusion and Discussion
In this paper, we show that a deep parallel neural network can be locally adaptive by tuning onlythe weight decay parameter.
This confirms that neural networks can be nearly optimal in learning functions with heterogeneous smoothness which separates them from kernel methods.
We provethat training an L layer parallel neural network with weight decay is equivalent to an ℓ 2 /L -penalizedregression model with representation learning.
Since in typical application L ≫ 2 , this shows thatweight decay promotes a sparse linear combination of the learned bases.
Using this method, weproved that a parallel neural network can achieve close to the minimax rate in the Besov space andbounded variation (BV) space.
Our result reveals that one do not need to specify the smoothness parameter α (or m ).
Neural networks can adapt to different degree of smoothness, or choose differentparameters for different regions of the domain of the target function.
This is a new type of adaptivity not possessed by traditional adaptive nonparametric regression methods like locally adaptiveregression spline or trend filtering.
On the other hand, as the depth of neural network L increases, 2 /L tends to 0 and the error ratemoves closer to the minimax rate of Besov and BV space.
This indicates that when the sample sizeis large enough, deeper models have smaller error than shallower models, and helps explain whyempirically deep neural networks has better performance than shallow neural networks.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This
work does not have potential negative societal impacts to the best of our knowledge.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] All the proofs are in
the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] Although we
did not publish the code, the experiments are very simple and can be easily reproduced
using the information given in Section G.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section G.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We provided the confidence interval in Figure 2.
(d) Did you include the total amount of compute and the type of resources used (e
g.,
type of GPUs, internal cluster, or cloud provider)?
[No] The experiments are light-
weighted and can be finished in an ordinary PC in a reasonable time.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] We only used
synthetic dataset.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Other related works
Besides Parhi and Nowak [29] which we discussed earlier, Parhi and Nowak [30, 31] also leveragedthe connections between NNs and splines.
Parhi and Nowak [30] focused on characterizing thevariational form of multi-layer NN.
Parhi and Nowak [31] showed that two-layer ReLU activated
NN achieves minimax rate for a BV class of order 1 but did not cover multilayer NNs nor BV classwith order > 1 , which is our focus.
The connection between weight-decay regularization with sparsity-inducing penalties in two-layer
NNs is folklore and used by Neyshabur et al.
[27], Savarese et al.
[35], Ongie et al.
[28], Ergen and
Pilanci [13, 16], Parhi and Nowak [29, 31], Pilanci and Ergen [32].
The key underlying technique —an application of the AM-GM inequality (which we used in this paper as well) — can be traced backto Srebro et al.
[38] (see a recent exposition by Tibshirani [42]).
Tibshirani [42] also generalized theresult to multi-layered NNs, but with a simple (element-wise) connections.
Ergen and Pilanci [14]generalized the results ti a three-layer parallel neural network, and proved its equivalence to an ℓ 1sparse model, but this requires a non-standard regularizer.
Besides, similar effect have been foundin convolution neural networks [12].
The approximation-theoretic and estimation-theoretic research for neural network has a long historytoo [6, 4, 49, 36, 39].
Most existing work considered the Holder, Sobolev spaces and their extensions, which contain only homogeneously smooth functions and cannot demonstrate the advantageof NNs over kernels.
The only exception is Suzuki [39] which, as we discussed earlier, requiresmodifications to NN architecture for each class.
In contrast, we require tuning only the standardweight decay parameter.
B Two-layer Neural Network with Truncated Power Activation Functions
We start by recapping the result of Parhi and Nowak [29] and formalizing its implication in estimating BV functions.
Parhi and Nowak [29] considered a two layer neural network with truncatedpower activation function.
Let the neural network be
[Equation]
where w j , v j denote the weight in the first and second layer respectively, b j denote the bias in thefirst layer, c ( x ) is a polynomial of order up to m , σ m ( x ) :
= max( x, 0) m .
Parhi and Nowak [29,
Theorem 8] showed that when M is large enough, The optimization problem
[Equation]
is equivalent to the locally adaptive regression spline:
[Equation]
which optimizes over arbitrary functions that is m -times weakly differentiable.
The latter wasstudied in Mammen and van de Geer [24], which leads to the following MSE:
Theorem 10.
Let M ≥ n − m , and ˆ f be the function (7) parameterized by the minimizer of (8) ,then
[Equation]
We show a simpler proof in the univariate case due to Tibshirani [43]:
Proof.
As is shown in Parhi and Nowak [29, Theorem 8], the minimizer of (8) satisfy
[Equation]
so the TV of the neural network f NN is
[Equation]
which shown that (8) is equivalent to the locally adaptive regression spline (9) as long as the numberof knots in (9) is no more than M .
Furthermore, it is easy to check that any spline with knots nomore than M can be expressed as a two layer neural network (8).
It suffices to prove that the solutionin (9) has no more than n − m number of knots.
Mammen and van de Geer [24, Proposition 1] showed that there is a solution to (9)
ˆ f
( x ) such thatˆ f ( x ) is a m th order spline with a finite number of knots but did not give a bound.
Let the number ofknots be M , we can represent ˆ f using the truncated power basis
[Equation]
where t j are the knots, c ( x ) is a polynomial of order up to m , and define σ ( m )
j ( x ) =
( x − t j ) m + .
Mammen and van de Geer [24] however did not give a bound on M .
Parhi and Nowak [29]’s
Theorem 1 implies that M ≤ n − m .
Its proof is quite technical and applies more generally to ahigher dimensional generalization of the BV class.
Tibshirani [43] communicated to us the following elegant argument to prove the same using elementary convex analysis and linear algebra, which we present below.
Define Π m ( f ) as the L 2 ( P n ) projection of f onto polynomials of degree up to m , Π ⊥ m ( f ) :
=
f − Π m ( f ) .
It is easy to see that
[Equation]
Denote f ( x 1: n ) :
= { f ( x 1 ) , . . .
, f ( x n ) } ∈ R n as a vector of all the predictions at the sample points.
[Equation]
where conv denotes the convex hull of a set.
The convex hull conv {± σ ( m )
j
[Equation]
x 1: n ) } · (cid:80) Mj =1 | a j | is ann -dimensional space, and polynomials of order up to m is an m + 1 dimensional space, so the setdefined above has dimension n − m − 1 .
By Carathéodory’s theorem, there is a subset of points inthis space
[Equation]
such that
[Equation]
In other word, there exist a subset of knots { ˜ t j , j ∈ [ n − m ] } that perfectly recovers Π ⊥ m ˆ f ( x ) at allthe sample points, and the TV of this function is no larger than ˆ f .
This shows that
[Equation]
for all x i in n onbservation points.
The MSE of locally adaptivity regressive spline (9) was studied in Mammen and van de Geer [24,
Section 3], which equals the error rate given in Theorem 10.
This indicates that the neural network (7) is minimax optimal for BV ( m ) .
Let us explain a few the key observations behind this equivalence.
(a) The truncated power functions (together with an m th order polynomial) spans the space of an m th order spline.
(b) The neural network in (7) is equivalent to a free-knot spline with M knots (up to reparameterization).
(c) A solution to (9) is a spline with at most n − m knots [29, Theorem 8].
(d) Finally, by the AM-GM inequality
[Equation]
where c j = v j | w j | m is the coefficient of the corresponding j th truncated power basis.
The m thorder total variation of a spline is equal to (cid:80) j | c j | .
It is not hard to check that the loss functiondepends only on c j , thus the optimal solution will always take “ = ” in the AM-GM inequality.
C Introduction To Common Function Classes
In the following definition define Ω be the domain of the function classes, which will be omitted inthe definition.
C.1 Besov Class
Definition 1.
Modulus of smoothness: For a function f ∈ L p (Ω) for some 1 ≤ p ≤ ∞ , the r -thmodulus of smoothness is defined by
[Equation]
otherwise .
Definition 2.
Besov space:
For 1 ≤ p, q ≤ ∞ , α > 0 , r :=
⌈ α ⌉ + 1 , define
[Equation]
and define the norm of Besov space as:
[Equation]
A function f is in the Besov space B αp,q if ∥ f ∥ B αp,q is finite.
Note that the Besov space for 0 < p, q < 1 is also defined, but in this case it is a quasi-Banach spaceinstead of a Banach space and will not be covered in this paper.
Functions in Besov space can be decomposed using B-spline basis functions.
Any function f in
Besov space B αp,q , α > d/p can be decomposed using B-spline of order m, m > α : let x ∈ R d ,
[Equation]
10) where J ( k ) :
= { 2 − k s : s ∈ [ − m, 2 k + m ] d ⊂ Z d } , M m,k, s ( x ) :=
M m (2 k ( x − s )) , and M k ( x ) =(cid:81) di =1 M k ( x i ) is the cardinal B-spline basis function which can be expressed as a polynomial:
[Equation]
Furthermore, the norm of Besov space is equivalent to the sequence norm:
[Equation]
See e.g. D˜ung
[11, Theorem 2.2] for the proof.
The Besov space is closely connected to other function spaces including the Hölder space ( C α ) andthe Sobolev space ( W αp ).
Specifically, if the domain of the functions is d -dimensional [39, 34],
• ∀ α ∈
N
, B αp, 1 ⊂ W αp ⊂ B αp,
∞
, and
B α 2 , 2 = W α 2 .
• For 0 < α < ∞ and α ∈ N , C α = B α ∞ , ∞ .
• If α > d/p , B αp,q ⊂ C 0 .
C.2 Other Function Spaces
Definition 3.
Hölder space: let m ∈ N , the m -th order Holder class is defined as
[Equation]
where D a denotes the weak derivative.
Note that fraction order of Hölder space can also be defined.
For simplicity, we will not cover thatcase in this paper.
Definition 4.
Sobolev space: let m ∈ N , 1 ≤ p ≤ ∞ , the Sobolev norm is defined as
[Equation]
the Sobolev space is the set of functions with finite Sobolev norm:
[Equation]
Definition 5.
Total Variation (TV): The total variation (TV) of a function f on an interval [ a, b ] isdefined as
[Equation]
where the P is taken among all the partitions of the interval [ a, b ] .
In many applications, functions with stronger smoothness conditions are needed, which can be measured by high order total variation.
Definition 6.
High order total variation: the m -th order total variation is the total variation of the( m − 1) -th order derivative
[Equation]
Definition 7.
Bounded variation (BV): The m -th order bounded variation class is the set of functionswhose total variation (TV) is bounded.
[Equation]
D Proof of Estimation Error
D.1 Equivalence Between Parallel Neural Networks and p -norm Penalized Problems
Proposition 4.
Fix the input dataset D n and a constant c 1 > 0 .
For any λ > 0 , there exists P ′ > 0such that (2) is equivalent to the following problem:
[Equation]
where ¯ f j ( · ) is a subnetwork with parameters ¯W ( ℓ ) j , ¯ b ( ℓ ) j .
Proof.
Using Langrange’s method, one can easily find (2) is equivalent to a constrained optimizationproblem:
[Equation]
for some constant P that depends on λ and the dataset D .
We make use of the property from (4) to minimize the constraint term in (12) while keeping thisneural network equivalent to the original one.
Specifically, let W (1) , b (1) , . . .
W ( L ) , b ( L ) be theparameters of an L -layer neural network.
[Equation]
which is equivalent to
[Equation]
as long as α ℓ > 0 , (cid:81)
L ℓ =1 α L = (cid:81)
Lℓ =1 ∥ W ( ℓ ) ∥ F , where ˜ W ( ℓ ) :=
W
[Equation]
ℓ )
∥ W ( ℓ ) ∥ F .
By the AM-GM inequality, the ℓ 2 regularizer of the latter neural network is
[Equation]
and equality is reached when α 1 = α 2 = · · · = α L .
In other word, in the problem (2), it suffices toconsider the network that satisfies
[Equation]
Using (4) again, one can find that the neural network is also equivalent to
[Equation]
where∥
[Equation]
where the last two equality comes from the assumption (13).
Choosing β ( ℓ ) =
c 1 √ w expect ℓ = 1where β (1) =
c 1 √ d , and scaling ¯ b ( ℓ ) accordingly and taking the constraint in (12) into (14) finishesthe proof.
D.2 Covering Number of Parallel Neural Networks
Theorem 5.
The covering number of the model defined in (5) apart from the bias in the last layersatisfies
[Equation]
The proof relies on the covering number of each subnetwork in a parallel neural network (Lemma 11), observing that | f ( x ) | ≤ 2 L − 1 w L − 1 √ d under the condition in Lemma 11, and then apply Lemma 6.
We argue that our choice of condition on ∥ b ( ℓ ) ∥ 2 in Lemma 11 is sufficient to analyzing the model apart from the bias in the last layer, because it guarantees that √
[Equation]
∥ b
[Equation]
∥ 2 .
This leads to
[Equation]
If this condition is not met, W ( ℓ ) A ℓ − 1 ( x ) + b ( ℓ ) is either always positive or always negativefor all feasible x along at least one dimension.
If ( W ( ℓ ) A ℓ − 1 ( x ) + b ( ℓ ) )
i is always negative,one can replace b ( ℓ ) )
i with − max x ∥ W ( ℓ ) A ℓ − 1 ( x ) ∥ ∞ without changing the output of this modelfor any feasible x .
If ( W ( ℓ ) A ℓ − 1 ( x ) + b ( ℓ ) )
i is always positive, one can replace b ( ℓ ) )
i withmax x ∥ W ( ℓ ) A ℓ − 1 ( x ) ∥ ∞ , and adjust the bias in the next layer such that the output of this modelis not changed for any feasible x .
In either cases, one can replace the bias b ( ℓ ) with another one withsmaller norm while keeping the model equivalent except the bias in the last layer.
Lemma 11.
Let F ⊆ { f : R d → R } denote the set of L -layer neural network (or a subnetwork ina parallel neural network) with width w in each hidden layer.
It has the form
[Equation]
and σ ( · ) is the ReLU activation function, the input satisfy ∥ x
∥ 2 ≤ 1 , then the supremum normδ -covering number of F obeys
[Equation]
where c 7 is a constant depending only on d , and c 8 is a constant that depend on d, w and L .
Proof.
First study two neural networks which differ by only one layer.
Let g ℓ , g ′ ℓ be two neural networks satisfying (15) with parameters W 1 , b 1 , . .
, W L , b L and W ′ 1 , b ′ 1 , . . .
, W ′ L , b ′ L respectively.
Furthermore, the parameters in these two models are the same except the ℓ -th layer, which satisfy
[Equation]
Denote the model as
[Equation]
where A ℓ ( x ) = σ ( W ℓ − 1 σ ( . . .
σ ( W 1 x + b 1 ) . . . )
+ b ℓ − 1 ) denotes the first ℓ − 1 layers in the neuralnetwork, and A ℓ ( x ) =
W L σ ( . . .
σ ( W ℓ +1 σ ( x )+ b ℓ +1 ) . . . )
+ b L ) denotes the last L − ℓ − 1 layers,with definition
A 1 ( x ) = x , B L ( x ) = x .
Now focus on bounding ∥A ( x ) ∥ .
Let W ∈ R m × m ′ , ∥ W ∥ F ≤ √ m ′ , x
∈ R m ′ , b ∈ R m , ∥ b
∥ 2 ≤ 640 √ m
[Equation]
where we make use of ∥ · ∥ 2 ≤ ∥ · ∥ F .
Because of that,
[Equation] [Equation]
Then focus on B ( x ) .
Let W ∈ R m × m ′ , ∥ W ∥ F ≤ √ m ′ , x , x ′ ∈
R m ′ , b ∈ R m ,
∥ b
∥ 2 ≤ √ m .
Furthermore, ∥ x − x ′ ∥ 2 ≤ ϵ , then
[Equation]
which indicates that ∥B ( x ) − B ( x ) ′ ∥ 2 ≤ ( √ w ) L − ℓ
∥ x − x ′ ∥
Finally, for any W , W ′ ∈ R m × m ′ , x ∈ R m ′ , b , b ′ ∈ R m , one have
[Equation]
In summary,
[Equation]
Let f ( x ) , f ′ ( x ) be two neural networks satisfying (15) with parameters W 1 , b 1 , . .
, W L , b L and
W ′ 1 , b ′ 1 , . . .
, W ′ L , b ′ L respectively, and ∥ W ℓ − W ′ ℓ
∥ F ≤ ϵ ℓ , ∥ b ℓ − b ′ ℓ ∥ F ≤ ˜ ϵ ℓ .
Further define f ℓ be theneural network with parameters W 1 , b 1 , . .
. , W ℓ , b ℓ , W ′ ℓ +1 , b ′ ℓ +1 , . .
, W ′ L , b ′ L , then
[Equation]
w
[Equation]
−
[Equation]
For any δ > 0 , one can choose
[Equation]
such that | f ( x ) − f ′ ( x ) | ≤ δ .
On the other hand, the ϵ -covering number of { W ∈ R m × m ′ :
∥ W ∥ F ≤ √ m ′ } on Frobenius normis no larger than (2 √ m ′ /ϵ
+ 1) m × m ′ , and the ¯ ϵ -covering number of { b ∈ R
m : ∥ b ∥ 2 ≤ 1 } oninfinity norm is no larger than (2 / ¯ ϵ + 1) m .
The entropy of this neural network can be bounded by
[Equation]
D.3 Covering Number of p -Norm Constrained Linear Combination
Lemma 6.
log N ( G , δ ) ≲
k log(1 /δ ) for some finite c 3 , and for any g ∈ G , | a | ≤ 1 , we haveag ∈ G .
The covering number of F = (cid:110)(cid:80) Mi
=1 a
i g i (cid:12)(cid:12)(cid:12) g i ∈ G , ∥ a ∥ pp ≤ P, 0 < p < 1 (cid:111) for any P > 0satisfies
[Equation]
up to a double logarithmic factor.
Proof.
Let ϵ be a positive constant.
Without the loss of generality, we can sort the coefficients indescending order in terms of their absolute values.
There exists a positive integer M (as a functionof ϵ ), such that | a i | ≥ ϵ for i ≤ M , and | a i | <
ϵ for i > M .
By definition, M ϵ p ≤ (cid:80) M i =1 | a i | p ≤ P so M ≤ P/ϵ p , and | a i | p ≤ P, | a i | ≤ P 1 /p for all i .
Furthermore,(cid:88)
[Equation]
>
[Equation]
−
[Equation]
Let ˜ g
i = arg min g ∈ ˜ G
∥ g − a i P 1 /p g i ∥ ∞ where ˜ G is the δ ′ -convering set of G .
By definition of thecovering set,
[Equation]
∞
[Equation]
− p .
Choosing
[Equation]
18) we have M ≤ P 11 − p ( δ/ 2 c 3 ) − p 1 − p , M P 1 /p δ ′ ≤ δ/ 2 , c 3 Pϵ 1 − p ≤ δ/ 2 , so (17) ≤ δ .
One cancompute the covering number of F bylog N
[Equation]
F ,
[Equation]
Taking (18) into (19) finishes the proof.
E Proof of Approximation Error
E.1 Approximation of Neural Networks to B-spline Basis Functions
Proposition 7.
Let M m,k, s be the B-spline of order m with scale 2 − k in each dimension andposition s ∈ R d : M m,k, s ( x ) :
= M m (2 k ( x − s )) , M m is defined in (11).
There exists a parallelneural network that has the structure and satisfy the constraint in Proposition 4 for d -dimensionalinput and one output, containing M
=
O ( m d ) subnetworks, each of which has width w =
O ( d )and depth L
= O (log( c ( m, d ) /ϵ )) for some constant w, c that depends only on m and d , denotedas ˜ M m ( x ) , x ∈ R d , such that
• | ˜ M m,k, s ( x ) − M m,k, s ( x ) | ≤ ϵ , if 0 ≤ 2 k
( x i − s i ) ≤ m + 1 , ∀ i ∈
[ d ] ,• ˜ M m,k, s ( x ) = 0 , otherwise.
• The weights in the last layer satisfy ∥ a ∥ 2 /L 2 /L ≲ 2 k m d e 2 md/L .
We follow the method developed in Yarotsky [49], Suzuki [39], while putting our attention on bounding the Frobenius norm of the weights.
Lemma 12 (Yarotsky [49, Proposition 3]) .
: There exists a neural network with two-dimensionalinput and one output f × ( x, y ) , with constant width and depth O (log(1 /δ )) , and the weight in eachlayer is bounded by a global constant c 1 , such that
• | f × ( x, y ) − xy | ≤ δ, ∀ 0 ≤ x, y ≤ 1 ,• f × ( x, y ) =
0 , ∀ x = 0 or y = 0 .
We first prove a special case of Proposition 7 on the unscaled, unshifted B-spline basis function byfixing k = 0 , s = 0 :
Proposition 13.
There exists a parallel neural network that has the structure and satisfy the constraint in Proposition 4 for d -dimensional input and one output, containing M = ⌈ ( m + 1) / 2 ⌉ d =
O ( m d ) subnetworks, each of which has width w =
O ( d ) and depth L
= O (log( c ( m, d ) /ϵ ))
forsome constant w, c that depends only on m and d , denoted as ˜ M m ( x ) , x ∈ R d , such that
• | ˜
M m ( x ) − M m ( x ) | ≤ ϵ , if 0 ≤ x i ≤ m + 1 , ∀ i ∈
[ d ] , while
M
m ( · )
denote m -th order
B-spline basis function
and
c
only depends on
m
and d .
• ˜ M m ( x ) = 0 , if x i ≤ 0 or x i ≥ m + 1 for any i ∈
[ d ] .
• The weights in the last layer satisfy ∥ a ∥ 2 /L 2 /L ≲ m d e 2 md/L .
Proof.
We first show that one can use a neural network with constant width w 0 , depth L ≂log( m/ϵ 1 ) and bounded norm
∥ W (1)
∥
F ≤ O ( √ d ) , ∥ W ( ℓ ) ∥
F ≤ O ( √ w ) , ∀ ℓ = 2 , . . .
, L toapproximate truncated power basis function up to accuracy ϵ 1 in the range [0 , 1] .
Let m =(cid:80) ⌈ log 2 m ⌉ i
= m
i 2 i , m i ∈ { 0 , 1 } be the binary digits of m , and define ¯ m j = (cid:80) ij =
m i , γ = ⌈ log 2 m ⌉ , then for any x
[Equation] [Equation]
Notice that each line of equation only depends on the line immediately below.
Replacing themultiply operator × with the neural network approximation shown in Lemma 12 demonstrates thearchitecture of such neural network approximation.
For any x, y ∈ [0 , 1] , let | f × ( x, y )
− xy | ≤δ, | x − ˜ x | ≤ δ 1 , | y − δy | ≤
δ 2 , then | f × (˜ x, ˜ y ) − xy | ≤ δ 1 + δ 2 + δ .
Taking this into (20) shows thatϵ 1 ≂ 2 γ δ ≂ mδ , where ϵ 1 is the upper bound on the approximate error to truncated power basis oforder m and δ is the approximation error to a single multiply operator as in Lemma 12.
A univariate B-spline basis can be expressed using truncated power basis, and observing that it issymmetric around ( m + 1) / 2 :
[Equation]
A multivariate ( d -dimensional) B-spline basis function can be expressed as the product of truncatedpower basis functions and thus can be decomposed as
[Equation]
Using Lemma 12, one can construct a parallel neural network containing M = ⌈ ( m + 1) / 2 ⌉ d =
O ( m d ) subnetworks, and each subnetwork corresponds to one polynomial term in (21).
Using theresults above, the approximation of this constructed neural network can be bounded by
[Equation]
where we applied Stirling’s approximation and δ and ϵ 1 has the same definition as above.
Choosingδ =
ϵ d ( e 2 m √ m +1) , and recall ϵ 1 ≂ mδ proves the approximation error.
To bound the norm of the factors ∥ a ∥ 2 /L 2 /L , first observe that
[Equation]
where the first inequality is from (cid:0) m
+1 j i (cid:1) ≤ 2 m +1 , the last equality is from Stirling’s appropximation.
Finally,
[Equation]
which finishes the proof.
The proof of the Proposition 7 for general k, s follows by appending one more layer in the front, aswe show below.
Proof of Proposition 7.
Using the neural network proposed in Proposition 13, one can construct aneural network for appropximating M m,k, s by adding one layer before the first layer:
[Equation]
The unused neurons in the first hidden layer is zero padded.
The Frobenius norm of the weight is2 k
∥
I d ∥ F = 2 k √ d .
Following the proof of Proposition 4, rescaling the weight in this layer by 2 − k ,and the weight matrix in the last layer by 2 k , and scaling the bias properly, one can verify that thisneural network satisfy the statement.
E.2 Sparse approximation of Besov functions using B-spline wavelets
Proposition 8.
Let α − d/p > 1 , r > 0 .
For any function in Besov space f 0 ∈ B αp,q and any positive integer ¯ M , there is an ¯ M -sparse approximation using B-spline basis of order m satisfying 0 < α < min( m, m − 1 + 1 /p ) :
ˇ f ¯ M = (cid:80) ¯ Mi =1 a k
i , s i M m,k i , s i for any positive integer
¯ M such that the approximation error is bounded as ∥
ˇ f ¯ M − f
∥
r ≲ ¯ M − α/d ∥ f 0 ∥ B αp,q , and the coefficients satisfy
[Equation]
The proof is divided into three steps:
Bound the 0-norm and the 1-norm of the coefficients of B-spline basis in order to approxi-
mate an arbitrary function in Besov space up to any ϵ > 0 .
Bound p -norm of the coefficients of B-spline basis functions where 0 < p < 1 using theresults above .3.
Add the approximation to neural network to B-spline basis computed in Section 4.3.1 intoStep 2.
Proof.
D ˜ ung
[11, Theorem 3.1] Suzuki [39, Lemma 2] proposed an adaptive sampling recoverymethod that approximates a function in Besov space.
The method is divided into two cases: whenp ≥ r , and when p < r .
When p ≥ r , there exists a sequence of scalars λ j , j ∈ P d ( µ ) , P d ( µ ) :=
{ j ∈ Z d : | j i | ≤ µ, ∀ i ∈d } for some positive µ , for arbitrary positive integer ¯ k , the linear operator
[Equation]
has bounded approximation error
[Equation]
where ¯ f is the extrapolation of f , J (¯ k, m, d ) :
= { s : 2 ¯ k s ∈ Z d , − m/ 2 ≤ 2 ¯ k s i ≤ 2 ¯ k + m/ 2 , ∀ i ∈
[ d ] } .
See D˜ung [11, 2.6-2.7] for the detail of the extrapolation as well as references for options ofsequence λ j .
Furthermore, Q ¯ k ( f ) ∈ B αp,q so it can be decomposed in the form (10) with M = (cid:80) ¯ kk =
(2 k + m −1)
d ≲ 2 ¯ kd components and ∥{ ˜ c k, s } k, s ∥ ≲
∥ Q ¯ k ( f ) ∥ B αp,q ≲ ∥ f ∥ B αp,q where ˜ c k, s is the coefficients ofthe decomposition of Q ¯ k ( f ) .
Choosing ¯ k ≂ log 2 M/d leads to the desired approximation error.
On the other hand, when p < r , there exists a greedy algorithm that constructs
[Equation]
where ¯ k ≂ log 2 ( M ) , k ∗ =
[ ϵ − 1 log( λM )]
+ ¯ k + 1 , n k
=
[ λM 2 − ϵ
k − ¯ k ) ] for some 0 <
ϵ <α/δ − 1 , δ =
d (1 /p − 1 /r ) , λ > 0 , such that
[Equation]
and
[Equation]
See D˜ung [11, Theorem 3.1] for the detail.
Finally, since α − d/p > 1 ,
[Equation]
22) where the first line is because for arbitrary vectors a i , i ∈ [ n ] , ∥ (cid:80) ni
=1 a i ∥ p ≤ (cid:80) ni =1 ∥ a i ∥ p , thethird line is because the sequence norm of B-spline decomposition is equivalent to the norm in
Besov space (see Section C.1) .
Note that when α − d/p = 1 , the sequence norm (22) is bounded (up to a factor of constant) byk ∗ ∥ f ∥ B αp,q , which can be proven by following (22) except the last line.
This adds a logarithmic termwith respect to ¯ M compared with the result in Proposition 8.
This will add a logarithmic factor tothe MSE.
We will not focus on this case in this paper of simplicity.
E.3 Sparse approximation of Besov functions using Parallel Neural Networks
Theorem 9.
Under the same condition as Proposition 8, for any positive integer ¯ M , any functionin Besov space f 0 ∈ B αp,q can be approximated by a parallel neural network with no less than
O ( m d ¯ M ) number of subnetworks satisfying:
Each subnetwork has width w =
O
( d ) and depth L .
The weights in each layer satisfy ∥ ¯W ( ℓ ) k
∥ F ≤ O ( √ w ) except the first layer ∥ ¯W (1) k
∥ F ≤O ( √ d ) ,
The scaling factors have bounded 2 /L
-norm: ∥{ a j }∥ 2 /L 2 /L ≲ m d e 2 md/
L ¯ M 1 − 2 / ( pL ) .4.
The approximation error is bounded by
[Equation]
where c 4 , c 5 , c 6 are constants that depend only on m, d and p .
We first prove the following lemma.
Lemma 14.
For any a ∈ R ¯ M , 0 < p ′ < p , it holds that:
[Equation]
Proof.
[Equation]
The first inequality uses a Holder’s inequality with conjugate pair pp ′ and 1 / (1 − p ′ p ) .
Proof of Theorem 9.
Using Proposition 8, one can construct ¯ M number of PNN each O ( m d ) subnetworks according to Proposition 7, and in each PNN, such that each PNN represents one B-splinebasis function.
The weights in the last layer of each PNN is scaled to match the coefficients in Proposition 8.
Taking p ′ in Lemma 14 as 2 /L and combining with Proposition 7 finishes the proof.
F Proof of the Main Theorem
Theorem 1 extended form.
For any fixed α − d/p > 1 , q ≥ 1 , L ≥ 3 , for any f 0 ∈ B αp,q , given an
L -layer parallel neural network satisfying
• The width of each subnetwork is fixed and large enough: w ≳ d .
See Theorem 9 for the
detail.
• The number of subnetworks is large enough: M ≳ m
d n
[Equation]
d
[Equation]
1 − 2 / ( pL ) where m = ⌈ α − 1 ⌉ .
With proper choice of the parameter of weight decay λ , the solution ˆ f parameterized by (2) satisfies
[Equation]
where ˜ O shows the scale up to a logarithmic factor, and c 6 is the constant defined in Theorem 9.
Proof.
First recall the relationship between covering number (entropy) and estimation error:
Proposition 15.
Let F ⊆ { R d → [ − F, F ] } be a set of functions.
Assume that F can be decomposedinto two orthogonal spaces F = F ∥ × F ⊥ where F ⊥ is an affine space with dimension of N. Letf 0 ∈ { R d → [ − F, F ] } be the target function and ˆ f be the least squares estimator in F :
[Equation]
then it holds that
[Equation]
The proof of Proposition 15 is defered to the section below.
We choose F as the set of functionsthat can be represented by a parallel neural network as stated, the (null) space F ⊥ =
{ f : f ( x ) =constant } be the set of functions with constant output, which has dimension 1.
This space capturesthe bias in the last layer, while the other parameters contributes to the projection in F ∥ .
See Section D.2 for how we handle the bias in the other layers.
One can find that F ∥ is the set of functionsthat can be represented by a parallel neural network as stated, and further satisfy (cid:80) ni
=1 f ( x i )
= 0 .
Because F ∥ ⊆
F , N ( F ∥ , δ ) ≤ N ( F , δ ) for all δ > 0 , and the latter is studied in Theorem 5.
In Theorem 1, the width of each subnetwork is no less than what is required in Theorem 9, while thedepth and norm constraint are the same, so the approximation error is no more
that that in Theorem 9.
Choosing r = 2 , p = 2 /L , and taking Theorem 5 and Theorem 9 into this Proposition 15, one gets
[Equation]
where ∥ f ∥ B αp,q , m and d taken as constants
The stated MSE is obtained by choosing
[Equation]
Note that there exists a weight decay parameter λ ′ such that the (2 /L ) -norm of the coefficientsof the parallel neural network satisfy that ∥{ a j }∥ 2 /L 2 /L
= m d e 2 md/L ∥{ ˜ a j, ¯ M }∥ 2 /L 2 /L where { ˜ a j, ¯ M }is the coefficient of the particular ¯ M -sparse approximation, although { a j } is not necessarily ¯ Msparse.
Empirically, one only need to guarantee that during initialization, the number of subnetworks
M ≥ ¯ M such that the ¯ M -sparse approximation is feasible, thus the approximation error boundfrom Theorem 9 can be applied.
Theorem 9 also says that ∥{ a j }∥ 2 /L 2 /L =
m d e 2 md/L ∥{ ˜ a j, ¯ M }∥ 2 /L 2 /L ≲¯ M 1 − 2 /pL , thus we can apply the covering number bound from Theorem 5 with P ′ =
¯ M 1 − 2 /pL .
Finally, if λ is optimally chosen, then it achieves a smaller MSE than this particular λ ′ , which hasbeen proven to be no more than O ( ¯ M − α/d ) and completes the proof.
Proof of Proposition 15.
For any function f ∈ F , define f ⊥ = arg min
h ∈F ⊥ (cid:80) ni
=
( f ( x i ) −h ( x i ))
2 be the projection of f to F ⊥ , and define f ∥ = f − f ⊥ be the projection to the orthogonal complement.
Note that f ∥ is not necessarily in F ∥ .
However, if f ∈ F , then f ∥ ∈ F ∥ .
y i ⊥ andy i ∥ are defined by creating a function f
y such that f y ( x i ) =
y i , ∀ i , e.g. via interpolation.
Because
F ∥ and F ⊥ are orthononal, the empirical loss and population loss can be decomposed in the sameway:
[Equation]
such that L ( f ) =
L ∥ ( f ) + L ⊥ ( f ) , ˆ L ( f ) =
ˆ L ∥ ( f ) + ˆ L ⊥ ( f ) .
This can be verified by decomposing ˆ f, f 0 and y into two orthogonal components as shown above, and observing that 822 (cid:80)
[Equation]
First prove the following claim
Claim 16.
Assume that ˆ f = arg min f ∈F ˆ L ( f ) is the empirical risk minimizer.
Then ˆ f ⊥ =arg min f ∈F ⊥ ˆ L ⊥ ( f ) , ˆ f ∥ = arg min f ∈F ∥ ˆ L ∥ ( f ) , where ˆ f ⊥ is the projections of ˆ f in F ⊥ , andˆ f ∥
= ˆ f − ˆ f ⊥ respectively.
Proof.
Since ˆ f ∈ F , by definition ˆ f ∥ ∈ F ∥ .
Assume that there exist ˆ f ′⊥ , ˆ f ′∥ , and either ˆ L ⊥ ( ˆ f ′⊥ ) <
ˆ L ⊥ ( ˆ f ⊥ ) , or ˆ L ∥ ( ˆ f ′∥ ) <
ˆ L ∥ ( ˆ f ∥ ) .
Then
[Equation]
which shows that ˆ f is not the minimizer of ˆ L ( f ) and violates the assumption.
Then we bound MSE ⊥ ( f ) .
We convert this part into a finite dimension least square problem:
[Equation]
The forth line comes from our assumption that F ⊥ is orthogonal to F ∥ , so ∀ f ∈ F ⊥ , f + f 0 ⊥
+ ϵ ⊥is orthogonal to ϵ ∥ .
Let the basis function of F ⊥ be h 1 , h 2 , . .
, h N , the above problem can be reparameterized as
[Equation]
where X ∈ R n × N : X i =
h j ( x i ) , y = y
⊥ + ϵ , y 0 ⊥ =
[ f 0 ⊥ ( x 1 ) , . . .
, f 0 ⊥ ( x n )] , ϵ =
[ ϵ 1 , . .
ϵ n ] .
This problem has a closed-form solution
[Equation]
Observe that f 0 ⊥ ∈ F ⊥ , let y
⊥ =
X θ ∗ ,The MSE of this problem can be computed by
[Equation]
X T ϵ ∥
Observing that Π :=
X ( X T X ) − 1 X T is an idempotent and independent projection whose rank is
N , and that E [ ϵϵ T ] =
σ 2 I , we get
[Equation]
which concludes that
[Equation]
See also [19, Proposition 1].
Next we study MSE ∥
( ˆ f ) .
Denote ˜ σ 2 ∥ = 1 n (cid:80) ni
=
ϵ 2 i ∥ , E = max i
| ϵ i | .
Using Jensen’s inequality andunion bound, we have
[Equation]
Taking expectation over both sides, we get
[Equation]
maximizing the right hand side over t yields
[Equation]
Let ˜ F ∥ be the covering set of F ∥ = { f ∥ : f ∈ F} .
For any ˜ f ∥ ∈ ˜ F ∥ ,
[Equation]
The first term can be bounded using Bernstein’s inequality: let h i
=
ϵ i ( f j ∥ ( x i ) − f 0 ∥ ( x i )) , bydefinition | h i | ≤ 2 EF ,
[Equation]
using Bernstein’s inequality, for any ˜ f ∥ ∈ ˜ F ∥ , with probably at least 1 − δ p ,
[Equation]
the last inequality holds true for all ϵ > 0 .
The union bound shows that with probably at least 1 − δ ,for all ˜ f ∥ ∈ ˜ F ∥ ,
[Equation]
By rearanging the terms and using the definition of L ( ˜ f ∥ ) , we get
[Equation]
Taking the expectation (over D ) on both sides, and notice that E [˜ σ 2 ∥ ]
= n − Nn σ 2 .
Furthermore, forany random variable X, E
[ X ] = (cid:82) ∞
−∞ xdP ( X ≤ x ) , we get
[Equation]
where the integration can be computed by replacing δ with e x .
Though it is not integrable under
Riemann integral, it is integrable under Lebesgue integration.
Similarly, let ˇ f ∥ = arg min f ∈F ∥
L ∥ ( f ) ,
[Equation]
with probably at least 1 − δ q , for any ϵ > 0 ,−n
[Equation]
Taking the expectation on both sides,
[Equation]
Finally, let ˆ f ∗ :=
arg min
[Equation]
∈ ˜
[Equation]
∥ (cid:80) n i =1 ( ˆ f ∥ ( x i ) − f ( x i )) 2 be the projection of ˆ f ∥ in its δ -coveringspace,
[Equation]
=
[Equation]
and similarly
[Equation] [Equation]
We can conclude that
[Equation]
where the first line comes from (24), and second comes from (26), the thid line is becauseˆ f ∥ = arg min f ∈F ∥ ˆ L ∥ ( f ) , and the last line comes from (25).
We also use that fact that ˆ L ∥ ( ˆ f ) ≤ˆ L ∥ ( f ) , ∀ f .
Noticing that MSE( ˆ f )
=
MSE ∥
( ˆ f ) + MSE ⊥
( ˆ f ) , combining this with (23) finishes theproof.
G Detailed experimental setup
G.1 Target Functions
The doppler function used in Figure 2(d)-(f) isf ( x ) = sin(4 / ( x + 0 . 01))
+ 1 . 5 .
The “vary” function used in Figure 2(g)-(i) isf ( x ) =
M 1 ( x/ 0 . 01)
+ M 1 (( x − 0 . + M 1 (( x − 0 .
+ M 1 (( x − 0 .
+ M 3 (( x − 0 . + M 3 (( x − 0 .
+ M 3 (( x − 0 . + M 3 (( x − 0 .
68) / 0 . 08) , where M 1 , M 3 are first and third order Cardinal B-spline bases functions respectively.
We uniformly take 256 samples from 0 to 1 in the piecewise cubic function experiment, and uniformly1000 samples from 0 to 1 in the doppler function and “vary” function experiment.
We add zeromean independent (white) Gaussian noise to the observations.
The standard derivation of noise is0.4 in the doppler function experiment and 0.1 in the “vary” function experiment.
G.2 Training/Fitting Method
In the piecewise polynomial function (“vary”) experiment, the depth of the PNN L = 10 , the widthof each subnetwork w = 10 , and the model contains M = 500 subnetworks.
The depth of NN is also10, and the width is 200 such that the NN and PNN have almost the same number of parameters.
Inthe doppler function experiment, the depth of the PNN L = 12 , the width of each subnetwork w =10 , and the model contains M = 2000 subnetworks, because this problem requires a more complexmodel to fit.
The depth of NN is 12, and the width is 400.
We used Adam optimizer with learning rateof 10 − 3 .
We first train the neural network layer by layer without weight decay.
Specifically, we startwith a two-layer neural network with the same number of subnetworks and the same width in eachsubnetwork, then train a three layer neural network by initializing the first layer using the trainedtwo layer one, until the desired depth is reached.
After that, we turn the weight decay parameter andtrain it until convergence.
In both trend filtering and smoothing spline experiment, the order is 3,and in wavelet denoising experiment, we use sym4 wavelet with soft thresholding.
We implementthe trend filtering problem according to Tibshirani [40] using CVXPY, and use MOSEK to solvethe convex optimization problem.
We directly call R function smooth.spline to solve smoothingspline.
G.3 Post Processing
The degree of freedom of smoothing spline is returned by the solver in R, which is rounded to thenearest integer when plotting.
To estimate the degree of freedom of trend filtering, for each choiceof λ , we repeated the experiment for 10 times and compute the average number of nonzero knots asestimated degree of freedom.
For neural networks, we use the definition [41]:
[Equation]
(27) where df denotes the degree of freedom, σ 2 is the variance of the noise, y are the labels, ˆ y arethe predictions and y ′ are independent copy of y .
We find that estimating (27) directly by samplingleads to large error when the degree of freedom is small.
Instead, we compute
[Equation]
(28) where ˆ df is the estimated degree of freedom, E denotes the empirical average (sample mean), y
isthe target function and ¯ y 0 is the mean of the target function in its domain.
Proposition 17.
The expectation of (28) over the dataset D equals (27).
Proof.
[Equation]
where D denotes the dataset.
In the third line, we make use of the fact that E [ ϵ i ] = 0 , E [ ϵ 2 i ] =
σ 2 ,and in the last line, we make use of E
[ ϵ ′ i ]
= 0 , E
[ ϵ ′ i 2 ]
= σ 2 , and ϵ ′
i are independent of y i and y 0 ,
i
One can easily check that a “zero predictor” (a predictor that always predict ¯ y 0 , and it always predicts0 if the target function has zero mean) always has an estimated degree of freedom of 0.
In Figure 2(h)(i), we take the minimum MSE over different choices of λ , and plot the average over10 runs.
Due to optimization issue, sometimes the neural networks are stuck at bad local minimaand the empirical loss is larger than the global minimum by orders of magnitude.
To deal with thisproblem, in Figure 2(h)(i), we manually detect these results by removing the experiments wherethe MSE is larger than 1.5 times the average MSE under the same setting, and remove them beforecomputing the average.
G.4 More experimental resultsG.4.1 Regularization weight vs degree-of-freedom
As we explained in the previous section, the degree of freedom is the exact information-theoreticmeasure of the generalization gap.
A Larger degree-of-freedom implies more overfitting.
[Figure]
In figure Figure 3, we show the relationship between the estimated degree of freedom and the scalingfactor of the regularizer λ in a parallel neural network and in trend filtering.
As is shown in the
[Figure] [Figure]
c)
[Figure] [Figure]
Trend filtering ObservationTarget function prediction (d) 0.
Smoothing
[Figure]
ObservationTarget function
Prediction (e) 0.
[Figure] [Figure]
[Figure] [Figure]
(f) figure, generally speaking as λ decreases towards 0 , the degree of freedom should increase too.
However, for parallel neural networks, if λ is very close to 0 , the estimated degree of freedom willnot increase although the degree of freedom is much smaller than the number of parameters —actually even smaller than the number of subnetworks.
Instead, it actually decreases a little.
Thiseffect has not been observed in other nonparametic regression methods, e.g. trend filtering, whichoverfits every noisy datapoint perfectly when λ → 0 .
But for the neural networks, even if we donot regularize at all, the among of overfitting is still relatively mild 30 / 256 vs 80 / 1000 .
In ourexperiments using neural networks, when λ is small, we denoise the estimated degree of freedomusing isotonic regression.
We do not know the exact reason of this curious observation.
Our hypothesis is that it might berelated to issues with optimization, i.e., the optimizer ends up at a local minimum that generalizesbetter than a global minimum; or it could be connected to the “double descent” behavior of DNN[26] under over-parameterization.
G.4.2 Detailed numerical results
In order to allow the readers to view our result in detail, we plot the numerical experiment results of
Figure 5.
G.4.3 Practical equivalence between the weight-decayed two-layer NN and L1-TrendFiltering
In this section we investigate the equivalence of two-layer NN and the locally adaptive regressionsplines from Section B.
In the special case when m = 1 the special regularization reduces to weightdecay and the non-standard truncated power activation becomes ReLU.
We compare L1 trend filtering [22] (shown to be equivalent to locally adaptive regression splines by Tibshirani [40]) andan overparameterized version of the neural network for all regularization parameter λ > 0 , i.e.,a regularization path.
The results are shown in Figure 6.
It is clear that as the weight decay increases, it induces sparsity in the number of knots it selects similarly to L1-Trend Filtering, and theregularization path matches up nearly perfectly even though NNs are also learning knots locations.
[Figure]