Asymptotic Properties for Bayesian Neural Network in Besov Space
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Neural networks have shown great predictive power when dealing with variousunstructured data such as images and natural languages.
The Bayesian neuralnetwork captures the uncertainty of prediction by putting a prior distribution for theparameter of the model and computing the posterior distribution.
In this paper, weshow that the Bayesian neural network using spike-and-slab prior has consistencywith nearly minimax convergence rate when the true regression function is in the
Besov space.
Even when the smoothness of the regression function is unknown thesame posterior convergence rate holds and thus the spike and slab prior is adaptiveto the smoothness of the regression function.
We also consider the shrinkage priorand show that it has the same convergence rate.
In other words, we propose apractical Bayesian neural network with guaranteed asymptotic properties.
Introduction
The neural network is a machine learning technique which can handle complex structures in databy combining a linear transformation and non-linear transformations [Goodfellow et al., 2016].
Theneural network can show great predictive power when dealing with unstructured data ranging fromimage data to natural language processing.
Moreover, the end-to-end learning is possible by graspingthe inherent structure of data only with inputs and outputs without explicit intermediate steps.
Suchproperties have played major roles in the rapid development of numerous artiﬁcial intelligenceapplications.
The Bayesian neural network (BNN) provides the uncertainty of the prediction from the probabilitydistribution of the model parameters.
Before observing the data, A prior distribution that reﬂects priorbeliefs about the model is put over the parameter space of the neural network model.
As observationsare added, a posterior distribution is computed by updating the prior distribution via Bayes’ rule.
Neal [1996], Williams [1996] shows that a shallow BNN which has only one inﬁnite-width hiddenlayer converges to a Gaussian process regression model.
This means that the shallow BNN is anapproximation of the nonparametric Bayesian regression model.
Lee et al.
[2017], Matthews et al.[2018] extended this result to the deep BNN model.
However, it is difﬁcult to compute the exact posterior distribution of the BNN since the normalizingconstant of the posterior distribution is intractable.
Therefore, various methods have been proposedto infer BNN through approximate Bayesian computation algorithms.
The methods based on the
Markov chain Monte Carlo (MCMC) algorithm such as Gibbs sampler [Geman and Geman, 1984,
Casella and George, 1992] and Hamiltonian Monte Carlo (HMC) [Neal et al., 2011] is effective toapproximate true posterior.
By constructing a Markov chain that has the desired distribution as itsstationary distribution, one can obtain a sample of the desired distribution.
Recently, scalable MCMCalgorithms such as stochastic gradient Langevin dynamics [Welling and Teh, 2011] and stochasticgradient HMC [Chen et al., 2014] are proposed.
The variational inference (VI) is an approximation method that ﬁnds the closest variational distribution from the posterior distribution.
VI methodchanges the Bayesian inference problem to an optimization problem.
Graves [2011] proposed to use a Gaussian distribution as variational distribution for BNN and Kingma and Welling [2013] suggestedthe reparameterization trick for Gaussian variational distribution.
Practically, VI methods for BNNare faster and more stable than MCMC methods.
Blundell et al.
[2015] suggested Bayes by Backpropalgorithm for BNN via variational inference.
There are also studies that approximate BNN with thedropout
[Gal and Ghahramani, 2016] or the Gaussian process [Lee et al., 2017].
1.1 Related works
The performance of the neural network is justiﬁed by the universal approximation capability of theneural network.
It is known that even a shallow neural network can approximate arbitrary continuousfunctions sufﬁciently accurately in L p -sense
[Cybenko, 1989, Hornik, 1991], and as the networkdeepens, the expressive power of the neural network gets stronger [Delalleau and Bengio, 2011,
Bianchini and Scarselli, 2014, Telgarsky, 2015, 2016].
Lee III [1999] showed that the BNN has theuniversal approximation property.
As a measure of the expressive power of a model, consider the range of functional spaces which amodel can express.
The choice of the activation function can affect the expressive power of a model,and in this paper, we consider the Rectiﬁed Linear Unit (ReLU)
[Nair and Hinton, 2010], whichis the most popular activation function in practice.
Yarotsky [2017] evaluated the approximationerror of the neural network using the ReLU activation function (ReLU network) for functions in Hölder space.
Schmidt-Hieber [2020] showed that the regularized least squared estimator performedby ReLU network in nonparametric regression problem converges to the true regression functionwith nearly minimax rate up to logarithmic factors on Hölder space.
Suzuki [2018] extended theresults in Schmidt-Hieber [2020] from the Hölder space to the Besov space which contains Hölderspace and Sobolev space.
Polson and Ro ˇ cková [2018] proved a Bayesian version of the results in
Schmidt-Hieber [2020].
They proved the Bayesian ReLU network has posterior consistency when thepriors on the parameters of the ReLU network are spike-and-slab priors.
Chérief-Abdellatif [2020]shows that the results in Polson and Roˇcková [2018] are still valid when variational methods areapplied.
All of the above results assume a sparse structure in the neural networks.
Sparsity is often assumed inhigh-dimensional models for asymptotic results.
Castillo et al.
[2015] showed theoretical propertiesincluding posterior consistency and rates of consistency of the high-dimensional linear regressionmodel under the spike-and-slab prior.
Song and Liang [2017] extended the results in Castillo et al.[2015] to general shrinkage priors which are more feasible than spike-and-slab prior.
1.2 Main contribution
We show that the posterior of the Bayesian ReLU network with a spike-and-slab prior with appropriatewidth, depth, and sparsity level is consistent with a nearly minimax rate.
We extend the result of
Suzuki [2018] to the Bayesian framework and the results of Polson and Roˇcková [2018] to the Besovspace.
As in Polson and Roˇcková [2018], we show that the same convergence rate is achieved evenwhen the smoothness parameter of the true regression function is not known.
That is, the ReLUnetwork has rich expressive power, and this ability is not constrained even without the smoothnessinformation of the true function.
We get similar results for the shrinkage prior.
We try to narrow the gap between the study of the theoretical properties and the practical applicationof the Bayesian ReLU networks.
For instance, Gaussian distribution is frequently considered asa prior distribution in practical Bayesian ReLU networks.
In this case, the posterior distributionmay not be consistent since the posterior becomes heavy tail distribution over the parameter space[Vladimirova et al., 2019].
Approaches to interpreting Bayesian neural networks as GP [Lee et al. ,2017] avoid this problem by setting the variance of the prior distribution to be proportional to thewidth of the network.
We suggest similar sufﬁcient conditions for the posterior consistency of the Bayesian ReLU networks.
Compared to precedent studys, our setting is closer to practical use.
Theresults in Suzuki [2018] and Polson and Roˇcková [2018] require to solve optimization problem underthe l 0 constraint or compute posterior from spike-and-slab prior respectively.
We suggest BNN model,which is easy to infer and consistent with (nearly) optimal convergence rate.
The rest of the paper is organized as follows.
We set up the problem and introduce the necessaryconcepts in Section 2.
We present the obtained theoretical results in Section 3 and numerical examplesin Section 4.
A summary of the paper and a discussion are in Section 5.
Preliminaries2.1 Notation
The ﬂoor function (cid:98) x (cid:99) denotes the largest integer less than or equal to x ∈ R , and (cid:107) a (cid:107) p is anl p -norm of a real vector a .
The u -th derivative of a d -dimensional function f is denoted by D u fwith u ∈ { 0 , 1 , · · ·} d .
The covering number N ( (cid:15), A, d ) is the minimal number of d -balls of radius(cid:15) > 0 necessary to cover a set A .
We denote the dirac-delta function at a ∈ R by δ a .
We use A (cid:46) Bas shorthand for the inequality A ≤ CB for some constant C > 0 and A ∨ B as shorthand for themax { A, B } .
2.2 Model
Suppose that n input-output observations D n =
( X i , y i ) ni
=1 ⊂ [0 , 1] d × R are i.i.d.
random samplefrom a regression model
[Equation]
where ( ξ i ) ni =1 is an i.i.d.
sequence of Gaussian noises N (0 , σ 2 ) with known variance σ 2 > 0 and f 0is the true regression function belonging to the space F .
In frequentist statistics, the maximum likelihood estimation is used to estimate the function f 0 .
Fromthe machine learning viewpoint, it is equivalent to ﬁnding ˆ f that minimizes the square loss.
Suzuki[2018] showed that f 0 can be estimated with a nearly optimal rate in this setting.
In practice, a sparse regularization term r ( f ) such as dropout and L 1 -regularizer is also consideredtogether with a loss function.
This is equivalent to estimating the MAP (maximum a posteriori) in
Bayesian modeling.
The posterior distribution gives not only predictive values but also the uncertaintyof the prediction.
Thus, in Bayesian statistics, it is important to ﬁnd sufﬁcient conditions for a priordistribution or a regularizer which guarantees the consistency or optimal convergence rate of theposterior.
2.3 Functional spaces
In this section, we introduce functional spaces that statisticians have studied for smoothness offunctions.
Let Ω =
[0 , 1] d ⊂ R d be the domain of functions we consider.
The L p -norm of a function is deﬁned as follows:
[Equation]
where f is a real-valued function deﬁned on Ω .
The L p space is the functional space consisting offunctions with bounded L p -norm and is denoted by L p (Ω)
= { f : (cid:107) f (cid:107) L p < ∞} .
Let s > 0 be the smoothness parameter and m = (cid:98)
s (cid:99) .
The s -Hölder norm of function f is deﬁned by
[Equation]
The s -Hölder space C s (Ω) is deﬁned as a set of functions with bounded s -Hölder norm.
We call sthe smoothness parameter of the s -Hölder space.
Let k ∈ N and 1 ≤ p ≤ ∞ .
The Sobolev norm of afunction f is deﬁned by
[Equation]
The Sobolev space W k,p (Ω) is deﬁned by the functional space consisting of functions with bounded
Sobolev norm.
Note the notion of the smoothness of a function is related to the differentiability of the function.
The
Besov space extends the concept of the smoothness.
Before deﬁning the Besov space, deﬁne the r -thmodulus of smoothness of the function f as follows [Gine and Nickl, 2016]:
[Equation]
[Figure]
[Equation]
For 0 < p, q ≤ ∞ , s > 0 , r
= (cid:98) s (cid:99) + 1 , deﬁne the Besov norm of a function f by
[Equation]
The Besov space B sp,q (Ω) is deﬁned as the set of functions with ﬁnite Besov norms [Gine and Nickl,2016], i.e.,
[Equation]
Note that Besov spaces can be deﬁned without the differentiability and continuity of functions, and
[Figure]
f 1 ( x ) on [0 , 1]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] (b) f 2 ( x ) on [0 , 1] 133 are more general than the Hölder space and the Sobolev space, which are subspaces of Besov spaces[Suzuki, 2018, Sawano, 2018, Triebel, 1983].
A Besov space may contain complicated functions thatcannot be expressed by traditional statistical models.
For instance, the Cantor function f 1 , Figure 1(a), does not belong to any Sobolev space since it does not have a weak derivative but belongs to a
Besov space [Sawano, 2018].
In fact, f 1 ∈ C log2 / log3 ([0 , 1])
⊂ B log2 / log3 ∞ , ∞ ([0 , 1]) .
Also,
[Equation]
(6) is in B 3 / 2 1 , 1 but not in any Hölder space.
Figure 1 have plots of f 1 and f 2 .
Note the minimax convergence rate of a Besov function f 0 in (1) has a lower bound n − s 2 s + d [Gineand Nickl, 2016, Donoho and Johnstone, 1998].
2.4 Sparse neural network
Deﬁne (deep) neural network space generated by a parameter space Θ as follows:
[Equation]
In this paper, we consider the following B -bounded and S -sparse parameter space [Suzuki, 2018]
[Equation]
and denote S -sparse neural network space with B -bounded parameters Φ(Θ( L, W, S, B )) asΦ( L, W, S, B ) .
Main results
Consider F is a Besov space B sp,q ([0 , 1] d ) .
For each n ∈ N , and θ ∈ Θ , Let P ( n ) θ be the probabilitymeasure of the data D n =
( X i , y i ) ni
=1 with a density function p ( n ) θ with respect to the measure µ ( n )and Π be a prior distribution of θ .
The posterior distribution is given by
[Equation]
[Equation]
Let
[Equation]
be the norm on empirical measure P Xn
= 1 n (cid:80) ni
=
δ X i .
Suppose that 0 < p, q ≤ ∞ , δ :=
d/p < s < ∞ and set ν = ( s − δ ) / (2 δ ) .
Assume that m ∈ Nsatisﬁes 0 < s < min { m, m − 1 + 1 /p } .
Let N n = (cid:100) n d/ (2 s + d ) (cid:101) , W {+
+} = 6 dm ( m + 2) + 2 d and L n = 3 + 2 (cid:100) log(cid:18)
[Equation]
m τ n [Equation] (cid:19) [Equation]
[Equation] (cid:101) , [Equation] =
[Equation]
B n = O ( N ξn ) , (10) where c ( d,m ) =
(cid:16) 1 + 2 de (2 e ) m √ m (cid:17) , τ
n = N − s/d n (log N n ) − 1 and ξ = min { 1 , ν − 1 + d − 1 } .
Note the following theorems are the similar as the result in Suzuki [2018], a result for a frequentiststatistics, and shows that the BNNs also converge in nearly minimax rate.
Suzuki [2018] showed thatfor the least squared estimator ˆ f ∈ Φ( L n , W n , S n , B n ) ,
[Equation]
(11) where E D n [ A ] is a expectation of A with respect to the training data D n .
Spike-and-slab prior
This result is an extension of Theorem 6.1 in Polson and Roˇcková [2018].
They showed that the
Bayesian ReLU networks is consistent with nearly minimax rates on Hölder space.
We extend thefunctional space from the Hölder space to the Besov space which contains Hölder spaces and Sobolevspace.
Assume the following spike-and-slab prior π ( θ )
[Equation]
where ˜ π
= U ([ − B n , B n ]) and T n = | Θ( L n , W n , S n , B n ) | .
Theorem 1.
Under the model (1) and prior distribution (12), the posterior distribution concentrates at the rate (cid:15) n = n − s/ (2 s + d ) (log n ) 3 / 2 .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.3.
Adaptive estimation
This result is an extension of Theorem 6.2 in Polson and Roˇcková [2018] to the Besov space.
Theyshowed that the Bayesian ReLU networks adapt to smoothness on the Hölder space under thespike-and-slab prior.
Theorem 2.
Assume the model (1), prior distribution (12),
[Equation]
for λ, ρ, β > 0 , and
[Equation]
where W 1 = 6 dm ( m + 2)
+ 2 d for some m ∈ N .
Suppose that 0 < p, q ≤ ∞ and d/p < s < min { m, m − 1+1 /p } .
The posterior distribution concentrates at the rate (cid:15) n = n − s/ (2 s + d ) (log n ) 3 / 2 .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.4.
In Theorem 1, the prior distribution depends on the smoothness parameters p , q , and s .
Theorem 2shows that, as in Polson and Ro ˇ cková [2018], a similar result can be obtained by considering theappropriate prior distribution even if the values of these parameters are unknown.
Note it can beeasily shown that BNNs do not overﬁt as in Polson and Roˇcková [2018].
Shrinkage prior
Asymptotic properties of the spike-and-slab prior can be obtained using Theorem 1, but its practicaluse for large neural network models is prohibited due to computation time.
The following priordistributions are more practical than a spike-and-slab prior in terms of computation.
The result to be introduced in this section shows that all the aforementioned results still hold evenunder shrinkage prior.
Assume the following prior
[Equation]
where T n = | Θ( L n , W n , S n , B n ) | and g ( t ) is acontinuous, symmetric density function and decreasing on t > 0 .
Theorem 3.
Let (cid:15) n = n − s/ (2 s + d ) (log n ) 3 / 2 .
Under the model (1) and prior distribution (15), assumethat
[Equation]
satisﬁes
[Equation] [Equation]
and
[Equation]
where η n = exp( − Kn(cid:15) 2 n /S n ) for some K, K 0 > 4 .
Then the posterior distribution concentrates at the rate (cid:15) n .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.5.
Note that condition (17) is a continuous approximation for the spike part of the (12).
The nextcondition (18) means that the prior should have enough thick tail to sample true function.
The lastcondition (19) restricts the thickness of the tail to prevent function divergent.
Unlike the other priors, the shrinkage prior is relatively straightforward to implement.
For example, toimplement the model in Theorem 1, one can consider the MCMC algorithm
[Sun et al., 2022] whichcombines Gibbs sampler, the Metropolis-Hastings algorithm, and the SGHMC algorithm.
Shrinkageprior avoids the posterior computation with varying dimensions, and enables feasible computation.
Example 1 (Gaussian mixture prior) .
Let (cid:15) n = n − s/ (2 s + d ) (log n ) 3 / 2 , ψ and Ψ be density and inverse survival functions of the standard Gaussian distribution respectively.
When 1 / 2 < ξ ≤ 1 and B n (cid:16) N ξn ,
[Equation]
with
[Equation]
and a n / Ψ (cid:18) π 2 n π 1 n [Equation] (cid:19)(cid:21)(cid:19)
[Equation]
(cid:19)(cid:21)(cid:19) satisﬁes condition of Theorem 3.
Numerical examples
We consider a problem of estimating the Besov functions f 1 and f 2 .
We sampled n points fromx ∼ U (0 , 1) and y from ∼ N ( f 1 ( x ) , σ 20 ) and N ( f 2 ( x ) , σ 20 ) respectively, where σ 20 = 0 .
Weassume the Gaussian mixture prior distribution for each parameter as Example 1. Speciﬁcally, we set
B n = 10 N ξn , K 0 = 5 , η n = exp( − K 0 n(cid:15) 2 n /S n ) a
[Equation]
[Equation] L n σ [Equation] In Table 1 and Table 2, the parameters to estimate each function were presented.
As the number ofsamples increases, the complexity of the neural network model increases.
Its speed is affected by thecomplexity of the true regression function.
[Table] [Table]
We ﬁt the model using the Bayes by Backprop algorithm with the python PyTorch package [Paszkeet al., 2019]. Experiments were run on a GPU server with one Nvidia RTX 3090. The code andinstruction related to the experiment were provided in the supplementary material.
Figure 2 and Figure 3 show the results.
For both problems, as the number of the data increase, mean functions getcloser to the true regression function and empirical errors decrease.
Conclusions
In this paper, we justify the theoretical properties of neural networks that are currently widely used.
Speciﬁcally, we focused on the asymptotic properties of BNNs and extend results in Polson and
Roˇcková [2018] to the Besov space from the Hölder space.
Theorem 1 shows that the BNN witha shrinkage prior has consistency posterior with a nearly minimax rate at Besov functions withknown smoothness.
Theorem 3 extends the result to the shrinkage prior respectively to closing thegap.
According to these theorems, BNNs have posterior consistency and (nearly) optimal rates byassuming proper prior such as Gaussian mixture prior.
We found there are three sufﬁcient conditionsto achieve posterior consistency, (1) enough mass around zero, (2) thick tail to sample true function(3) while not too much.
Theorem 2 shows that even if the smoothness of the function is not known,the same rate can be obtained by considering a prior on the architecture of the neural networks.
The results in this paper are an extension of the results of Suzuki [2018] to BNN also.
We suggesta practical method that can achieve the optimal rate while providing the uncertainty of the prediction.
Note they showed the theoretical properties of the neural networks in the Besov space vianonparametric regression based on B-spline.
In other words, similar results can be obtained using the
Bayesian LARK B-spline model [Park et al., 2021].
Experiments on performance comparison withother models will also be of great help in establishing the properties of BNN.
We obtained results under the limited assumption, setting the networks fully connected and usingthe ReLU activation function.
The results of this study are at a basic level compared to the needsfor BNNs in the industrial ﬁeld, such as visual data analysis and natural language processing.
It isimportant to research the theoretical properties of BNNs used in the practical area through ongoingstudies.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplementary
material.
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A
crowdsourcing or research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Posterior consistency
Lemma 1 (Lemma 2 in Ghosal and Van Der Vaart [2007]) .
Suppose that observation X ( n ) =
( X 1 , · · · , X n ) of independent observations
X i .
Assume that the distribution P θ,i of the i th component
X i has a density p θ,i relative to a σ -ﬁnite measure µ i .
Let P ( n ) θ =
[Equation]
ni =1 P θ,i be the product measures and
[Equation]
Then there exists test φ n such that P ( n ) θ 0 φ n ≤ e − 12 nd 2 n ( θ 0 ,θ 1 ) and P ( n ) θ (1 − φ n ) ≤ e − 12 nd 2 n ( θ 0 ,θ 1 ) forall θ ∈ Θ such that d n ( θ, θ 1 ) ≤ 118 d n ( θ 0 , θ 1 ) .
Proof.
See Ghosal and Van Der Vaart [2007].
Let deﬁne divergences
[Equation]
[Equation] and set
B ∗ n ( θ 0 , (cid:15) ; k ) =
(cid:40)
[Equation]
Here, the C k is the constant satisfying
[Equation]
for k ≥ 2 .
The existence of C k is guaranteed by Marcinkiewiz-Zygmund inequality.
Then thefollowings hold
Lemma 2 (Theorem 4 of Ghosal and Van Der Vaart [2007]) .
Let P ( n ) θ be product measures andd n = 1 n (cid:80)
ni
=1 (cid:82) ( √ p θ,i −√ p θ (cid:48) ,i ) 2 dµ i .
Suppose that for a sequence (cid:15) n → 0 such that n(cid:15) 2 n is boundedaway from zero, some k > 1 , all sufﬁciently large j and sets Θ n ⊂ Θ , the following conditions hold: [Equation] [Equation]
[Equation]
Then P ( n ) θ
(Π n ( θ : d n ( θ, θ 0 ) ≥ M n (cid:15) n | D n ) → 0 for any sequence M n → ∞ .
Proof.
See Ghosal and Van Der Vaart [2007].
The following is an adapted version of the Lemma 2, which introduced in Polson and Roˇcková [2018](white noise model and k = 2 ).
Assume the model (1).
Let
[Equation]
If there exist d > 2 and F n ⊂ F such that
[Equation]
for any
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
B Consistency of neural network
The upper bound of the covering number of the neural network space is given by as follows.
Lemma 3 (Lemma 3 in Suzuki [2018]) .
∀ δ > 0 ,
[Equation]
Proof.
See appendix C.
For L, W, S ∈ N and B, a > 0 , deﬁne a functional space
[Equation]
(30) where Θ( L,
[Equation]
, S,
[Equation]
[Equation]
(cid:111) .
Lemma 4.
∀ δ ≥ 2 aL ( B ∨ 1) L − 1 ( W + 1) L ,log N ( δ, Φ( L, W, S, B, a ) ,
[Equation]
Proof.
See appendix C.
The following lemma shows that for any function f 0 in the Besov space, there exists neural networkswhich is closed enough to f 0 .
Lemma 5 (Proposition 1 in Suzuki [2018] with r = ∞ ) .
Suppose that 0 < p, q ≤ ∞ , δ :=
d/p <s < ∞ and ν =
( s − δ ) / (2 δ ) .
Assume that N ∈ N is sufﬁciently large and m ∈ N satisﬁes0 < s < min { m, m − 1 + 1 /p } .
Let W
= 6 dm ( m + 2) + 2 d .
Then,
[Equation]
32) for
[Equation]
33) where c ( d,m ) = (cid:16) 1 + 2 de (2 e ) m √ m (cid:17) and τ n = N − s/d n (log N n ) − 1 .
Proof.
See Suzuki [2018].
C Proof of theorems
C.1 Proof of lemma 3
Proof.
Let denote f ∈ Φ( L, W, S, B ) as
[Equation]
[Equation]
A
[Equation]
(34) for k = 2 , · · · , L , and A − L +1 ( f )( x ) =
A +1 ( f )( x ) = x .
Then, f ( x
[Equation]
[Equation] ( x ) . Since f ∈ Φ( L, W, S, B ) ,(cid:13)(cid:13) A + k ( f ) (cid:13)(cid:13)
L ∞ ≤ max j (cid:13)(cid:13)(cid:13) W ( k − 1) j, : (cid:13)(cid:13)(cid:13) 1 (cid:13)(cid:13)
A + k − 1 ( f
[Equation]
L ∞ +
[Equation]
b
[Equation]
k
[Equation] [Equation]
L ∞ ≤ WB (cid:13)(cid:13) [Equation]
+
[Equation]
( f
[Equation]
+ B ≤ ( W [Equation] [Equation] (cid:13)(cid:13)
A + k
[Equation]
( f [Equation] (cid:13)(cid:13) L ∞ ≤ ( W + 1) k − 1 ( B ∨ 1) k −
[Equation]
(35) and similarly
[Equation]
[Equation]
Fix (cid:15) > 0 and θ ∈ Θ( L, W, S, B ) . For any θ ∗ ∈ Θ( L, W, S, B ) which satisﬁes (cid:107) θ − θ ∗ (cid:107) ∞ < (cid:15) ,
[Equation]
W + 1) L .
Let s be the number of nonzero component of θ , s ≤ S .
Consider a subspace Θ θ ( L, W, S, B ) of the parameter space
Θ( L, W, S, B ) consists of parameters which have s nonzero component.
Choose θ 1 , · · · , θ M form each grid divided with length (cid:15) ∗ =
δ L ( B ∨ 1)
L − 1 ( W +1)
L over Θ θ ( L, W, S, B ) .
Then
[Equation]
Note
[Equation]
and the number of cases of choose s nonzero components is
[Equation]
Thus,
[Equation]
and we get desired results taking the logarithm of both sides.
C.2 Proof of lemma
Proof.
Let ˜ θ ( a ) =
( θ i I ( | θ i | > a )) .
Then (cid:13)(cid:13)(cid:13) θ − ˜ θ ( a ) (cid:13)(cid:13)(cid:13) ∞ ≤ a and ˜Θ( L, W, S, B, a ) =
(cid:110)
˜
θ ( a ) : θ ∈ Θ( L, W, S, B, a ) (cid:111) =
Θ( L, W, S, B ) .
As in the proof of Lemma 3,
[Equation]
(39) for any f θ , f ˜ θ ( a ) ∈
Φ( L, W, S, B, a, F ) .
Let s ≤ S be the number of nonzero component of ˜ θ ( a ) .
Consider a subspace ˜Θ θ ( L, W, S, B, a ) of ˜Θ( L, W, S, B, a ) consists of parameters which have s nonzero component.
Choose θ 1 , · · · , θ M form each grid divided with length (cid:15) ∗ =
[Equation]
+1) L over ˜Θ θ ( L, W, S, B, a ) .
Then
[Equation]
from triangular inequality.
We get desired results in the similar way as in the proof of the Lemma3.
C.3 Proof of theorem 1
Proof.
Let
A [Equation]
It is enough to show that there exists a constant C > 2 which satisﬁes
[Equation]
for sufﬁciently large n .
Let F n = F = Φ( L n , W n , S n , B n ) .
First, (c) is trivial since
F − F n = ∅ .
Since
[Equation]
and Lemma 3,
[Equation] (a) holds.
Here, the last inequality holds from
[Equation]
Next, from Lemma 5, there is a constant C 1 > 0 and ˆ f n = f ˆ θ ∈ F such that
[Equation]
Let ˆ γ and ˆ θ ˆ γ be index and value of nonzero components of ˆ θ respectively.
Let Θ(ˆ γ ; L n , W n , S n , B n )be a subset of parameter space Θ( L n , W n , S n , B n ) consists of parameters which have nonzerocomponents at ˆ γ only and F (ˆ γ ) =
Φ(ˆ γ ; L n , W n , S n , B n ) be the neural network space generated byΘ(ˆ γ ; L n , W n , S n , B n ) .
Note
[Equation]
for sufﬁciently large n .
As in the proof of Lemma 3,
[Equation]
=
[Equation]
Thus,
[Equation]
for sufﬁciently large n .
C.4 Proof of theorem 2
Proof.
Let F = Φ .
It is enough to show that there exists a constant C (cid:48) > 2 which satisﬁes
[Equation]
[Equation]
for sufﬁciently large n .
For sufﬁciently large constants C N , C B > 0 , let ˜ N n = C N N n , ˜ L n =
L ( ˜ N n ) , ˜ B n = C B n(cid:15) 2 n and
[Equation]
First, show that (a) holds.
From Lemma 3,
N
[Equation]
n
[Equation]
and
[Equation]
[Equation]
Next
[Equation]
show that (b) holds. For N n [Equation]
L n in Theorem [Equation] N n < ˜ N n , L n < ˜ L n .
Moreover, since B n ≤ N n (cid:46) n(cid:15) 2 n , there exists a constant D > 0 such that
[Equation] [Equation]
Then
[Equation]
[Equation]
holds and (b) holds for C (cid:48) =
C + 3 D .
From Π
[Equation]
and Chernoff bound , for any positive number t > 0 ,
[Equation]
Letting t = log ˜ N n , log ˜ L n respectively, we get
[Equation]
Also, since
[Equation]
for sufﬁciently large C N , C B > 0 and π ( B > ˜ B n ) = exp (cid:16) − β ˜ B n (cid:17) , (c) holds.
C.5 Proof of theorem
Proof.
It is enough to show that there exists a constant C > 2 which satisﬁes
[Equation]
for sufﬁciently large n .
Let F n = Φ( L n , W n , S n , B n , a n ) and F = Φ( L n , W n , T n ) .
It is easy toshow that (a) holds from Lemma 4 as in the proof of the Theorem 1.
By the assumption,Π( F − F n ) ≤ π (cid:0)
[Equation]
| θ i | > B n (cid:1) + π (cid:32) T n (cid:88) i =
I ( | θ i | > a n ) > S n (cid:33) =
(cid:0) 1 − (
[Equation]
v n ) T n (cid:1)
[Equation]
P ( S
[Equation]
S n | S
[Equation]
( T n , 1 − u n ))
[Equation]
T
[Equation]
n
[Equation]
o ( |
[Equation]
n | )
[Equation]
(cid:18)
[Equation]
T n (cid:26)(cid:18)
[Equation]
n
[Equation]
n (cid:19) log
[Equation]
n
[Equation]
n u n
[Equation]
n
T n log
S n /T n
[Equation]
u n (cid:27)(cid:19) =
o (cid:16)
[Equation]
(cid:17)
[Equation]
(cid:18)
[Equation]
(cid:18)
[Equation]
(cid:19)
[Equation]
u n
[Equation]
log
S
[Equation]
/T n
[Equation]
n (cid:19)
[Equation]
o (cid:16)
[Equation]
(cid:17)
[Equation]
(cid:18)
[Equation]
(cid:18)
[Equation]
n T n (cid:19)
[Equation]
1 − η n
[Equation]
n
[Equation]
T n
[Equation]
n
[Equation]
n log
S n /T n 1 − u n (cid:19)
[Equation]
(cid:16)
[Equation]
(cid:17)
[Equation]
(cid:18)
[Equation]
(cid:18)
[Equation]
n (cid:19) (cid:18) (
[Equation]
η n )
[Equation]
n
[Equation]
T n
[Equation]
T n
[Equation]
o (cid:18) (1 − η n )
[Equation]
n /T n 1 − S n /T n (cid:19)(cid:19) − Kn(cid:15) 2 n (cid:19)
[Equation]
[Equation]
for some 4 < K 1 < K .
Letting C
= min { K 0 , K 1 } − 2 , (c) holds.
We use tail bound for binomialdistribution in Arratia and Gordon [1989] for the second inequality.
Next, from Lemma 5, there is aconstant C 1 > 0 and ˆ f n = f ˆ θ ∈ F n such that
[Equation]
Let ˆ γ and ˆ θ ˆ γ be index and value of nonzero components of ˆ θ respectively.
Let
[Equation]
and ˜Θ(ˆ γ ; L n , W n , S n , B n , a n ) be a subset of parameter space ˜Θ
( L n , W n , S n , B n , a n ) consists ofparameters which have nonzero components at ˆ γ only and F n (ˆ γ ) =
˜Φ(ˆ γ ; L n , W n , S n , B n , a n ) bethe neural network space generated by ˜Θ(ˆ γ ; L n , W n , S n , B n , a n ) .
Note
[Equation]
56) for sufﬁciently large n .
As in the proof of Lemma 3,
[Equation]
where t n = [Equation]
W
[Equation]
+1)
Ln L n ( B n ∨
[Equation]
Ln − 1 .
Letting
[Equation]
[Equation]
[Equation]