Asymptotic Properties for Bayesian Neural Network in Besov Space
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Neural networks have shown great predictive power when dealing with variousunstructured data such as images and natural languages.
The Bayesian neural network captures the uncertainty of prediction by putting a prior distribution for theparameter of the model and computing the posterior distribution.
In this paper, weshow that the Bayesian neural network using spike-and-slab prior has consistencywith nearly minimax convergence rate when the true regression function is in the
Besov space.
Even when the smoothness of the regression function is unknownthe same posterior convergence rate holds and thus the spike and slab prior is adaptive to the smoothness of the regression function.
We also consider the shrinkageprior, which is more feasible than other priors, and show that it has the same convergence rate.
In other words, we propose a practical Bayesian neural networkwith guaranteed asymptotic properties
Introduction
The neural network is a machine learning technique which can handle complex structures in databy combining a linear transformation and non-linear transformations [Goodfellow et al., 2016].
Theneural network can show great predictive power when dealing with unstructured data ranging fromimage data to natural language processing.
Moreover, the end-to-end learning is possible by grasping the inherent structure of data only with inputs and outputs without explicit intermediate steps.
Such properties have played major roles in the rapid development of numerous artiﬁcial intelligenceapplications.
The Bayesian neural network (BNN) provides the uncertainty of the prediction from the probabilitydistribution of the model parameters.
Before observing the data, a prior distribution that reﬂects priorbeliefs about the model is put over the parameter space of the neural network model.
As observationsare added, a posterior distribution is computed by updating the prior distribution via Bayes’ rule.
Neal [1996], Williams [1996] shows that a shallow BNN which has only one inﬁnite-width hiddenlayer converges to a Gaussian process regression model.
This means that the shallow BNN is anapproximation of the nonparametric Bayesian regression model.
Lee et al.
[2017], Matthews et al.[2018] extended this result to the deep BNN model.
However, it is difﬁcult to compute the exact posterior distribution of the BNN since the normalizingconstant of the posterior distribution is intractable.
Therefore, various methods have been proposedto infer BNN through approximate Bayesian computation algorithms.
The methods based on the
Markov chain Monte Carlo (MCMC) algorithm such as Gibbs sampler [Geman and Geman, 1984,
Casella and George, 1992] and Hamiltonian Monte Carlo (HMC) [Neal et al., 2011] is effectiveto approximate true posterior.
By constructing a Markov chain that has the desired distribution asits stationary distribution, one can obtain a sample of the desired distribution.
Recently, scalable MCMC algorithms such as stochastic gradient Langevin dynamics [Welling and Teh, 2011] and stochastic gradient HMC [Chen et al., 2014] were proposed.
The variational inference (VI) is anapproximation method that ﬁnds the closest variational distribution from the posterior distribution.
VI method changes the Bayesian inference problem to an optimization problem.
Graves [2011] proposed to use a Gaussian distribution as variational distribution for BNN and Kingma and Welling[2013] suggested the reparameterization trick for Gaussian variational distribution.
Practically, VImethods for BNN are faster and more stable than MCMC methods.
Blundell et al.
[2015] suggested Bayes by Backprop algorithm for BNN via variational inference.
There are also studies that approximate BNN with the dropout [Gal and Ghahramani, 2016] or the Gaussian process [Lee et al.,2017].
1.1 Related works
The performance of the neural network is justiﬁed by the universal approximation capability of theneural network.
It is known that even a shallow neural network can approximate arbitrary continuousfunctions sufﬁciently accurately in L p -sense
[Cybenko, 1989, Hornik, 1991], and as the networkdeepens, the expressive power of the neural network gets stronger [Delalleau and Bengio, 2011,
Bianchini and Scarselli, 2014, Telgarsky, 2015, 2016].
Lee III [1999] showed that the BNN has theuniversal approximation property.
As a measure of the expressive power of a model, consider the range of functional spaces which amodel can express.
The choice of the activation function can affect the expressive power of a model,and in this paper, we consider the Rectiﬁed Linear Unit (ReLU) ReLU ( x )
= max { 0 , x } [Nair and Hinton, 2010], which is the most popular activation function in practice.
Yarotsky [2017] evaluatedthe approximation error of the neural network using the ReLU activation function (ReLU network)for functions in Hölder space.
Schmidt-Hieber [2020] showed that the regularized least squaredestimator performed by ReLU network in nonparametric regression problem converges to the trueregression function with nearly minimax rate up to logarithmic factors on Hölder space.
Suzuki[2018] extended the results in Schmidt-Hieber [2020] from the Hölder space to the Besov spacewhich contains Hölder space and Sobolev space.
Polson and Roˇcková [2018] proved a Bayesianversion of the results in Schmidt-Hieber [2020].
They proved the Bayesian ReLU network hasposterior consistency, the posterior probability is concentrated to the true parameter as collectingdata, when the priors on the parameters of the ReLU network are spike-and-slab priors.
ChériefAbdellatif [2020] shows that the results in Polson and Roˇcková [2018] are still valid when variationalmethods are applied.
All of the above results assume a sparse structure in the neural networks.
Sparsity is often assumed inhigh-dimensional models for asymptotic results.
Castillo et al.
[2015] showed theoretical propertiesincluding posterior consistency and rates of consistency of the high-dimensional linear regressionmodel under the spike-and-slab prior.
Song and Liang [2017] extended the results in Castillo et al.[2015] to general shrinkage priors which are more feasible than the spike-and-slab prior.
1.2 Main contribution
We show that the posterior of the Bayesian ReLU network with a spike-and-slab prior with appropriate width, depth, and sparsity level is consistent with a nearly minimax rate.
We extend the resultof Suzuki [2018] to the Bayesian framework and the results of Polson and Roˇcková [2018] to the Besov space.
As in Polson and Roˇcková [2018], we show that the same convergence rate is achievedeven when the smoothness parameter of the true regression function is not known.
That is, the ReLUnetwork has rich expressive power, and this ability is not constrained even without the smoothnessinformation of the true function.
We get similar results for the shrinkage prior.
We try to narrow the gap between the study of the theoretical properties and the practical application of the Bayesian ReLU networks.
For instance, Gaussian distribution is frequently consideredas a prior distribution in practical Bayesian ReLU networks.
In this case, the posterior distribution may not be consistent since the posterior becomes heavy tail distribution over the parameterspace [Vladimirova et al., 2019].
Approaches to interpreting Bayesian neural networks as Gaussian
Process [Lee et al., 2017] avoid this problem by setting the variance of the prior distribution to beproportional to the width of the network.
We suggest similar sufﬁcient conditions for the posteriorconsistency of the Bayesian ReLU networks.
Compared to precedent studys, our setting is closerto practical use.
The results in Suzuki [2018] and Polson and Roˇcková [2018] require to solve the optimization problem under the l 0 constraint or compute posterior from spike-and-slab prior respectively.
We suggest BNN model, which is easy to infer and consistent with the (nearly) optimalconvergence rate.
The rest of the paper is organized as follows.
We set up the problem and introduce the necessary concepts in Section 2.
We present the obtained theoretical results in Section 3 and numerical examplesin Section 4.
A summary of the paper and a discussion are in Section 5.
Preliminaries
2.1 Notation
The ﬂoor function ⌊ x ⌋ denotes the largest integer less than or equal to x ∈ R , and ∥ a ∥ p is an l pnorm of a real vector a .
The u -th derivative of a d -dimensional function f is denoted by D u f withu ∈ { 0 , 1 , · · ·} d .
The covering number N ( ϵ, A, d ) is the minimal number of d -balls of radius ϵ > 0necessary to cover a set A .
We denote the dirac-delta function at a ∈ R by δ a .
We use A ≲ B asshorthand for the inequality A ≤ CB for some constant C > 0 and A ∨ B as shorthand for themax { A, B } .
2.2 Model
Suppose that n input-output observations D n =
( X i , y i ) ni
=1 ⊂ [0 , 1] d × R are i.i.d.
random samplefrom a regression model
[Equation]
(1) where ( ξ i ) ni =1 is an i.i.d.
sequence of Gaussian noises N (cid:0) 0 , σ 2 (cid:1) with known variance σ 2 > 0 andf 0 is the true regression function belonging to the space F .
In frequentist statistics, the maximum likelihood estimation is used to estimate the function f 0 .
Fromthe machine learning viewpoint, it is equivalent to ﬁnding ˆ f that minimizes the square loss.
Suzuki[2018] showed that f 0 can be estimated with a nearly optimal rate in this setting.
In practice, a sparse regularization term r ( f ) such as dropout and l 1 -regularizer is also consideredtogether with a loss function.
This is equivalent to estimating the MAP (maximum a posteriori) in
Bayesian modeling.
The posterior distribution gives not only predictive values but also the uncertainty of the prediction.
Thus, in Bayesian statistics, it is important to ﬁnd sufﬁcient conditions fora prior distribution or a regularizer which guarantees the consistency or optimal convergence rate ofthe posterior.
2.3 Functional spaces
In this section, we introduce functional spaces that statisticians have studied for smoothness of functions.
Let Ω =
[0 , 1] d ⊂ R d be the domain of functions we consider.
The L p -norm of a function is deﬁned as follows:
[Equation]
where f is a real-valued function deﬁned on Ω .
The L p space is the functional space consisting offunctions with bounded L p -norm and is denoted by L p (Ω)
= { f : ∥ f ∥ L p < ∞} .
Let s > 0 be the smoothness parameter and m =
⌊ s ⌋ .
The s -Hölder norm of function f is deﬁnedby
[Equation]
The s -Hölder space C s (Ω) is deﬁned as a set of functions with bounded s -Hölder norm.
We call sthe smoothness parameter of the s -Hölder space.
Let k ∈ N and 1 ≤ p ≤ ∞ .
The Sobolev norm ofa function f is deﬁned by
[Equation]
The Sobolev space W k,p (Ω) is deﬁned by the functional space consisting of functions with bounded
Sobolev norm.
Note the notion of the smoothness of a function is related to the differentiability of the function.
The
Besov space extends the concept of the smoothness.
Before deﬁning the Besov space, deﬁne ther -th modulus of smoothness of the function f as follows [Gine and Nickl, 2016]:
[Equation]
[Equation]
For 0 < p, q ≤ ∞ , s > 0 , r =
⌊ s ⌋ + 1 , deﬁne the Besov norm of a function f by
[Equation]
The Besov space B sp,q (Ω) is deﬁned as the set of functions with ﬁnite Besov norms [Gine and Nickl,2016], i.e., [Equation]
[Equation]
Note that Besov spaces can be deﬁned without the differentiability and continuity of functions, and
[Figure]
a)
[Figure]
x )
[Figure] [Figure]
[Figure]
138 are more general than the Hölder space and the Sobolev space, which are subspaces of Besov spaces[Suzuki, 2018, Sawano, 2018, Triebel, 1983].
A Besov space may contain complicated functions thatcannot be expressed by traditional statistical models.
For instance, the Cantor function f 1 , Figure 1(a), does not belong to any Sobolev space since it does not have a weak derivative but belongs to a
Besov space [Sawano, 2018].
In fact, f 1 ∈ C log2 / log3 ([0 , 1])
⊂ B log2 / log3 ∞ , ∞ ([0 , 1]) .
Also,
[Equation]
is in B 3 / 2 1 , 1 but not in any Hölder space.
Figure 1 have plots of f 1 and f 2 .
Note the minimax convergence rate of a Besov function f 0 in (1) has a lower bound n − s 2 s + d [Gineand Nickl, 2016, Donoho and Johnstone, 1998].
Deﬁne (deep) neural network space generated by a parameter space Θ as follows:
[Equation]
where ζ is an activation function.
In this paper, we consider ζ ( x ) = ReLU ( x ) and the following B -bounded and S -sparse parameter space [Suzuki, 2018]
[Equation]
and denote S -sparse neural network space with B -bounded parameters Φ(Θ( L, W, S, B )) asΦ( L, W, S, B ) .
Main results
Consider F is a Besov space B sp,q ([0 , 1] d ) .
For each n ∈ N , and θ ∈ Θ , let P ( n ) θ be the probabilitymeasure of the data D n =
( X i , y i ) ni
=1 with a density function p ( n ) θ with respect to the measure µ ( n )and Π be a prior distribution of θ .
The posterior distribution is given by
[Equation]
for A ⊂ Θ .
Let ∥ f
[Equation]
be the norm on empirical measure P
[Equation]
[-=
-]=
[Equation] P [Equation] =1 δ X i .
Suppose that 0 < p, q ≤ ∞ , ω : =
d/p < s < ∞ and set ν = ( s − ω ) / (2 ω ) .
Assume that m ∈ Nsatisﬁes 0 < s < min { m, m − 1 + 1 /p } .
Let N n = ⌈ n d/ (2 s + d ) ⌉ , W [-
-]0 = 6 dm ( m + 2) + 2 d and
[Equation]
where c ( d,m ) = (cid:16) 1 + 2 de
[Equation]
) m √ m (cid:17) −, τ
n = N − s/d n (log N n ) − 1 and ξ = min { 1 , ν − 1 + d − 1 } .
Note the following theorems are the similar as the result in Suzuki [2018], a result for a frequentiststatistics, and shows that the BNNs also converge in nearly minimax rate.
Suzuki [2018] showedthat for the least squared estimator [Equation]
[Equation]
where E D n [ A ] is an expectation of A with respect to the training data D n .
Spike-and-slab prior
This result is an extension of Theorem 6.1 in Polson and Roˇcková [2018].
They showed that the
Bayesian ReLU networks is consistent with nearly minimax rates on Hölder space.
We extendthe functional space from the Hölder space to the Besov space which contains Hölder spaces and Sobolev space.
Assume the following spike-and-slab prior π ( θ )
[Equation]
where ˜ π
= U ([ − B n , B n ]) and T n = | Θ( L n , W n , S n , B n ) | .
Theorem 1.
Under the model (1) and prior distribution (12), the posterior distribution concentrates at the rate ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.3.
Adaptive estimation
This result is an extension of Theorem 6.2 in Polson and Roˇcková [2018] to the Besov space.
Theyshowed that the Bayesian ReLU networks adapt to smoothness on the Hölder space under the spikeand-slab prior.
Theorem 2.
Assume the model (1), prior distribution (12),
[Equation]
(13) for λ, ρ, β > 0 , andΦ = ∞ [ N = ∞ [ L = ∞ [ B = Φ( L,
[Equation]
where W 1 = 6 dm ( m + 2)
+ 2 d for some m ∈ N .
Suppose that 0 < p, q ≤ ∞ and d/p < s < min { m, m − 1+1 /p } .
The posterior distribution concentrates at the rate ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.4.
In Theorem 1, the prior distribution depends on the smoothness parameters p , q , and s .
Theorem2 shows that, as in Polson and Roˇcková [2018], a similar result can be obtained by considering theappropriate prior distribution even if the values of these parameters are unknown.
Note it can beeasily shown that BNNs do not overﬁt as in Polson and Roˇcková [2018].
Shrinkage prior
Asymptotic properties of the spike-and-slab prior can be obtained using Theorem 1, but its practicaluse for large neural network models is prohibited due to computation time.
The following priordistributions are more practical than a spike-and-slab prior in terms of computation.
The result to be introduced in this section shows that all the aforementioned results still hold evenunder shrinkage prior.
Assume the following prior
[Equation]
where T n = | Θ( L n , W n , S n , B n ) | and g ( t ) is a continuous, symmetric density function and decreasing on t > 0 .
Theorem 3.
Let ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 .
Under the model (1) and prior distribution (15),assume that
[Equation]
satisﬁes
[Equation] [Equation] [Equation]
and
[Equation]
[Equation] where η n = exp( − Knϵ 2 n /S n ) for some K, K 0 > 4 .
Then the posterior distribution concentrates at the rate ϵ n .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.5.
Note that condition (17) is a continuous approximation for the spike part of the (12).
The nextcondition (18) means that the prior should have enough thick tail to sample true function.
The lastcondition (19) restricts the thickness of the tail to prevent function divergent.
Unlike the other priors, the shrinkage prior is relatively straightforward to implement.
For example,to implement the model in Theorem 1, one can consider the MCMC algorithm
[Sun et al., 2022]which combines Gibbs sampler, the Metropolis-Hastings algorithm, and the SGHMC algorithm.
Shrinkage prior avoids the posterior computation with varying dimensions, and enables feasiblecomputation.
Example 1 (Gaussian mixture prior) .
Let ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 , ψ and Ψ be density and inverse survival functions of the standard Gaussian distribution N (0 , 1) respectively.
When 1 / 2 < ξ ≤ 1 and B n ≍ N ξn ,
[Equation]
with
[Equation]
and
[Equation]
satisﬁes condition of Theorem 3.
Numerical examples
However, these results are still not enough practical to be used.
As the dimensionality d of the dataincreases, the complexity of the neural network to obtain the theoretical optimality increases rapidly.
For instance, consider a problem of estimating the Besov functions f 1 and f 2 .
Assume the Gaussianmixture prior distribution for each parameter as Example 1,
[Equation]
(cid:19)(cid:21)(cid:19) .
In Table 1 and Table 2, the hyperparameters (model parameters) to estimate each function werepresented.
These values are calculated numerically and may vary depending on the computer environment.
As the number of samples increases, the complexity of the neural network model increases.
Its speed is affected by the complexity of the true regression function.
As can be seen from bothtables, the smaller of the variances is extremely small.
Considering the numerical precision, it isindistinguishable from spike-and-slab prior [Ishwaran and Rao, 2005].
[Table] [Table]
Instead, although the conditions of the theorem are not satisﬁed, we checked that the two functionscan be estimated even with a smaller model.
The results are presented in the Appendix D.
Conclusions
In this paper, we justify the theoretical properties of neural networks that are currently widely used.
Speciﬁcally, we focused on the asymptotic properties of BNNs and extend results in Polson and
Roˇcková [2018] to the Besov space from the Hölder space.
Theorem 1 shows that the BNN with aspike-and-slab prior has consistency posterior with a nearly minimax rate at Besov functions withknown smoothness.
Theorem 3 extends the result to the shrinkage prior.
According to these theorems, BNNs have posterior consistency and (nearly) optimal rates by assuming proper prior such as Gaussian mixture prior.
We found there are three sufﬁcient conditions to achieve posterior consistency, (1) enough mass around zero, (2) thick tail to sample true function (3) while not too much.
Theorem 2 shows that even if the smoothness of the function is not known, the same rate can beobtained by considering a prior on the architecture of the neural networks.
The results in this paper are an extension of the results of Suzuki [2018] to BNN also.
We suggesta practical method that can achieve the optimal rate while providing the uncertainty of the prediction.
Note they showed the theoretical properties of the neural networks in the Besov space vianonparametric regression based on B-spline.
In other words, similar results can be obtained usingthe Bayesian LARK B-spline model [Park et al., 2021].
In future works, performance comparison experiments with other models will also be of great help inestablishing the properties of BNN.
Empirically, we know that there are ﬁelds in which decision-treebased model such random forest or neural network models perform better than traditional statisticalmodels that ﬁnd smooth functional relationships such as Kernel regression.
For these data, it isnatural that there would be a non-smooth relationship between the explanatory variable and theresponse variable.
For this reason, it is essential to compare the relationship between Bayesianneural network models and staitical model for follow-up studies.
We obtained results under the limited assumption, setting the networks fully connected and usingthe ReLU activation function.
Also, as mentioned in the numerical experiment section, the resultsare not enough practical to be used.
In addition, we does not suggest result for the adaptive versionof the Theorem 3 yet.
It is a necessary future study to propose a general shrinkage prior distributionthat satisﬁes all conditions even when the parameters of the network are varying.
The results of thisstudy are at a basic level compared to the needs for BNNs in the industrial ﬁeld, such as visual dataanalysis and natural language processing.
It is important to research the theoretical properties of BNNs used in the practical area through ongoing studies.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplementary
material.
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data [Table]
you’re using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁ- [Table]
able information or offensive content?
[N/A]
[Table] If you used crowdsourcing or conducted research with human subjects... [Table]
(a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]
A Posterior consistency
Lemma 1 (Lemma 2 in Ghosal and Van Der Vaart [2007]) .
Suppose that observation X ( n ) =
( X 1 , · · · , X n ) of independent observations
X i .
Assume that the distribution P θ,i of the i th
component X i has a density p θ,i relative to a σ -ﬁnite measure µ i .
Let P ( n ) θ = N ni
=1 P θ,i be the product measures and
[Equation]
Then there exists test ϕ n such that P ( n ) θ 0 ϕ n ≤ e − 12 nd 2 n ( θ 0 ,θ 1 ) and P ( n ) θ (1 − ϕ n ) ≤ e − 12 nd 2 n ( θ 0 ,θ 1 ) forall θ ∈ Θ such that d n ( θ, θ 1 ) ≤ 118 d n ( θ 0 , θ 1 ) .
Proof.
See Ghosal and Van Der Vaart [2007].
Let deﬁne divergences
[Equation] [Equation]
(22) and set
B
[Equation]
Here, the C k is the constant satisfying
[Equation]
for k ≥ 2 .
The existence of C k is guaranteed by Marcinkiewiz-Zygmund inequality.
Then thefollowings hold
Lemma 2 (Theorem 4 of Ghosal and Van Der Vaart [2007]) .
Let P ( n ) θ be product measures andd n = 1 n
P ni =1 R ( √ p θ,i − √ p θ ′ ,i ) 2 dµ i .
Suppose that for a sequence ϵ n → 0 such that nϵ 2 n isbounded away from zero, some k > 1 , all sufﬁciently large j and sets Θ n ⊂ Θ , the followingconditions hold:
[Equation]
Then P ( n ) θ
(Π n ( θ : d n ( θ, θ 0 ) ≥ M n ϵ n | D n ) → 0 for any sequence M n → ∞ .
Proof.
See Ghosal and Van Der Vaart [2007].
The following is an adapted version of the Lemma 2, which introduced in Polson and Roˇcková[2018] (white noise model and k = 2 ).
Assume the model (1).
Let
[Equation]
If there exist d > 2 and F n ⊂ F such that
[Equation]
for any ϵ n → 0 , nϵ 2 n → ∞ ,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
B Consistency of neural network
The upper bound of the covering number of the neural network space is given by as follows.
Lemma 3 (Lemma 3 in Suzuki [2018]) .
[Equation]
[Equation]
for W, L ≥ 3 .
Proof.
See appendix C.
For L, W, S ∈ N and B, a > 0 , deﬁne a functional space
[Equation]
[Equation] where Θ( L,
[Equation]
= n
[Equation]
o .
Lemma 4.
∀ ϵ ≥ 2 aL ( B ∨ 1) L − 1 ( W + 1) L ,
[Equation]
(31) for W, L ≥ 3 .
Proof.
See appendix [-C.
-]C.
The following lemma shows that for any function f 0 in the Besov space, there exists neural networkswhich is closed enough to f 0 .
Lemma 5 (Proposition 1 in Suzuki [2018] with r = ∞ ) .
Suppose that 0 < p, q ≤ ∞ , ω : =
d/p <s < ∞ and ν =
( s − ω ) / (2 ω ) .
Assume that N ∈ N is sufﬁciently large and m ∈ N satisﬁes0 < s < min { m, m − 1 + 1 /p } .
Let W
= 6 dm ( m + 2) + 2 d .
Then,
[Equation]
for
[Equation]
33) where c ( d,m ) = (cid:16) 1 + 2 de (2 e ) m √ m (cid:17) −and τ n = N − s/d n (log N n ) − 1 .
Proof.
See Suzuki [2018].
C Proof of theorems
C.1 Proof of lemma 3
Proof.
Let denote f ∈ Φ( L, W, S, B ) as
[Equation]
34) for k = 2 , · · · , L
[Equation]
and A − L +1 ( f )( x ) =
A +1 ( f )( x ) = x .
Then,
[Equation]
Since f ∈ Φ( L, W, S, B ) ,
[Equation]
[Equation]
for all x , where A j, : is the j -th row of the matrix A .
Similarly,
[Equation]
[Equation]
Fix ϵ > 0 and θ ∈ Θ( L, W, S, B ) .
For any θ ∗ ∈ Θ( L, W, S, B ) which satisﬁes ∥ θ − θ ∗ ∥ ∞ <
ϵ ,
[Equation]
≤
[Equation]
≤
[Equation]
≤ ϵL [Equation] B ∨ 1) L − 1 ( W + 1) L [Equation]
Let s be the number of nonzero component of θ , s ≤ S .
Consider a subspace Θ θ ( L, W, S, B ) of the parameter space
Θ( L, W, S, B ) consists of parameters which have s nonzero component.
Choose θ 1 , · · · , θ M form each grid divided with length ϵ ∗ =
ϵ L ( B ∨ 1)
L − 1 ( W +1)
L over Θ θ ( L, W, S, B ) .
Then
[Equation]
Note
[Equation]
for W, L ≥ 3 and the number of cases of choose s nonzero components
[Equation]
Thus,
[Equation]
38) and we get desired results taking the logarithm of both sides.
C.2 Proof of lemma
Proof.
Let ˜ θ ( a ) =
( θ i I ( | θ i | > a )) .
Then (cid:13)(cid:13)(cid:13) θ − ˜ θ ( a ) (cid:13)(cid:13)(cid:13) ∞ ≤ a and ˜Θ( L
[Equation]
n
[Equation] a
[Equation]
As in the proof of Lemma 3,
[Equation]
(39) for any f θ , f ˜ θ ( a ) ∈
Φ( L, W, S, B, a, F ) .
Let s ≤ S be the number of nonzero component of ˜ θ ( a ) .
Consider a subspace ˜Θ θ ( L, W, S, B, a ) of ˜Θ( L, W, S, B, a ) consists of parameters which have s nonzero component.
Choose θ 1 , · · · , θ M form each grid divided with length ϵ ∗ = ϵ/ 2 L ( B ∨ 1)
L − 1 ( W +1)
L over ˜Θ θ ( L, W, S, B, a ) .
Then
[Equation]
from triangular inequality.
We get desired results in the similar way as in the proof of the Lemma3.
C.3 Proof of theorem 1
Proof.
Let
[Equation]
It is enough to show that there exists a constant C > 2 which satisﬁes
[Equation]
for sufﬁciently large n .
Let F n = F = Φ( L n , W n , S n , B n ) .
First, (c) is trivial since
F − F n = ∅ .
Since { f ∈ F n : ∥ f ∥ L ∞ ≤ ϵ } ⊂ { f ∈ F n : ∥ f ∥ n ≤ ϵ } and Lemma 3,
[Equation]
for sufﬁciently large n .
Thus, (a) holds.
Here, the last inequality holds from
[Equation]
Next, from Lemma 5, there is a constant C 1 > 0 and ˆ f n = f ˆ θ ∈ F such that
[Equation]
for sufﬁciently large n .
Let ˆ γ and ˆ θ ˆ γ be index and value of nonzero components of ˆ θ respectively.
LetΘ(ˆ γ ; L n , W n , S n , B n ) be a subset of parameter space Θ( L n , W n , S n , B n ) consists of parameterswhich have nonzero components at ˆ γ only and F (ˆ γ ) =
Φ(ˆ γ ; L n , W n , S n , B n ) be the neural networkspace generated by Θ(ˆ γ ; L n , W n , S n , B n ) .
Note
[Equation]
for sufﬁciently large n .
As in the proof of Lemma 3,Π
[Equation]
≥ [Equation] = exp [Equation]
(cid:18)
[Equation]
W n [Equation] [Equation] B n
[Equation]
[Equation]
ϵ n [Equation] Thus, [Equation]
C.4 Proof of theorem 2
Proof.
Let F = Φ .
It is enough to show that there exists a constant C ′ > 2 which satisﬁes
[Equation] [Equation]
for sufﬁciently large n .
For sufﬁciently large constants C N , C B > 0 , let ˜ N n = C N N n , ˜ L n =
[Equation] ˜ [Equation] = C B nϵ 2 n and
[Equation]
First, show that (a) holds.
From Lemma 3,
N
[Equation]
ϵ
[Equation]
[Equation]
and
[Equation]
N
[Equation]
for sufﬁciently large n .
Next, show that (b) holds.
For N n , L n in Theorem 1, N n < ˜ N n , L n < ˜ L n .
Moreover, since B n ≤ N n ≤ nϵ 2 n for sufﬁciently large n , there exists a constant D > 0 such thatπ
[Equation]
N n
[Equation] [Equation]
for sufﬁciently large n .
Then
[Equation]
holds and (b) holds for C ′ =
C + 3 D .
From
[Equation]
and Chernoff bound , for any positive number t > 0 ,
[Equation]
Letting t = log ˜ N n , log ˜ L n respectively, we get
[Equation]
Also, since
[Equation]
for sufﬁciently large C N , C B > 0 and π 3 ( B > ˜ B n ) = exp [Equation] , (c) holds.
C.5 Proof of theorem
Proof.
It is enough to show that there exists a constant C > 2 which satisﬁes(a) sup
ϵ>ϵ
n log N ( ϵ/ 36 , A ϵ, 1 ∩ F n , ∥·∥ n ) ≤ nϵ 2 n(b ) − log Π
( A ϵ
n , 1 )
[Equation]
[Equation]
for sufﬁciently large n .
Let F n = Φ( L n , W n , S n , B n , a n ) and F = Φ( L n , W n , T n ) .
It is easy toshow that (a) holds from Lemma 4 as in the proof of the Theorem 1.
By the assumption,
[Equation]
o
[Equation]
[Equation]
S
[Equation]
[Equation]
[Equation]
o
[Equation]
for some 4 < K 1 < K .
Letting C
= min { K 0 , K 1 } − 2 , (c) holds.
We use tail bound for binomialdistribution in Arratia and Gordon [1989] for the second inequality.
Next, from Lemma 5, there is aconstant C 1 > 0 and ˆ f n = f ˆ θ ∈ F n such that
[Equation]
Let ˆ γ and ˆ θ ˆ γ be index and value of nonzero components of ˆ θ respectively.
Let
[Equation]
and ˜Θ(ˆ γ ; L n , W n , S n , B n , a n ) be a subset of parameter space ˜Θ
( L n , W n , S n , B n , a n ) consists ofparameters which have nonzero components at ˆ γ only and F n (ˆ γ ) =
˜Φ(ˆ γ ; L n , W n , S n , B n , a n ) bethe neural network space generated by ˜Θ(ˆ γ ; L n , W n , S n , B n , a n ) .
Note
[Equation]
for sufﬁciently large n .
As in the proof of Lemma 3,
[Equation]
57) where t n = ϵ n 2( W n +1) Ln L n ( B n ∨ 1) Ln − 1 .
Letting y n =
Z B n B n − t n g ( t ) dt ≥ t n g ( B n ) , −
[Equation]
D Numerical Examples
Consider a problem of estimating the Besov functions f 1 and f 2 . Assume the Gaussian mixtureprior distribution for each parameter as Example 1. We sampled n points from x ∼ U (0 , 1) and yfrom N (cid:0)
f 1 ( x ) , σ 20 (cid:1) and N (cid:0) f 2 ( x ) , σ 20 (cid:1) respectively, where σ 20 = 0 . 01 .
As mentioned in Section
[Table]
we used a smaller model that was weaker than the conditions of thetheorem.
We ﬁx the depth L n = 5 and the width W n = 200 of the model and use same prioras in the theorem (Gaussian mixture BNN).
Also, the smaller of the variances is set to σ 1 n ←max { 0 .
001 , σ 1 n } to avoid numerical precision issues.
We summarize the hyperparameters of themodel used in the experiment in Table 3 and Table 4.
[Table]
[Table]
We ﬁt the model using the NUTS algorithm [Hoffman et al., 2014] with the python Pyro [Binghamet al., 2019] and PyTorch [Paszke et al., 2019] packages.
Experiments were run on a GPU serverwith one Nvidia RTX 3090.
The code and instruction related to the experiment are provided in thesupplementary material.
Figure 2 and Figure 3 show the results.
For both problems, as the numberof the data increase, mean functions get closer to the true regression function and empirical error 495 ∥
f − f i ∥ n decrease.
D.2 Gaussian Prior
We performed the exact same experiment as before using Gaussian prior (Gaussian BNN).
The standard deviation of the Gaussian prior is the same as σ 2 n .
Figure 4 and Figure 5 show the results.
Forboth problems, as the number of the data increase, mean functions get closer to the true regressionfunction and empirical error ∥ f − f i ∥ n decrease.
D.3 Comparison
We compare two model in terms of empirical errors (training error)
∥
f − f i ∥ n and L 2 norms (testerror) e f = ∥
f − f i ∥ 2 between true function f i and sampled function f .
Speciﬁcally, approximatee f as ˆ e f using Riemann sum,
[Equation]
with (cid:8) x ∗ k : x ∗ k = 1 n ∗ , 1 ≤ k ≤ n ∗ (cid:9) .
As in Table 5 and Table 6, both model give similar results, butthe Gaussian mixture BNN give slightly better results.
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Table]
[Table]