Asymptotic Properties for Bayesian Neural Network in Besov Space
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural networks have shown great predictive power when dealing with variousunstructured data such as images and natural languages.
The Bayesian neuralnetwork captures the uncertainty of prediction by putting a prior distribution for theparameter of the model and computing the posterior distribution.
In this paper, weshow that the Bayesian neural network using spike-and-slab prior has consistencywith nearly minimax convergence rate when the true regression function is in the
Besov space.
Even when the smoothness of the regression function is unknown thesame posterior convergence rate holds and thus the spike and slab prior is adaptiveto the smoothness of the regression function.
We also consider the shrinkageprior, which is more feasible than other priors, and show that it has the sameconvergence rate.
In other words, we propose a practical Bayesian neural networkwith guaranteed asymptotic properties
Introduction
The neural network is a machine learning technique which can handle complex structures in databy combining a linear transformation and non-linear transformations [Goodfellow et al., 2016].
Theneural network can show great predictive power when dealing with unstructured data ranging fromimage data to natural language processing.
Moreover, the end-to-end learning is possible by graspingthe inherent structure of data only with inputs and outputs without explicit intermediate steps.
Suchproperties have played major roles in the rapid development of numerous artificial intelligenceapplications.
The Bayesian neural network (BNN) provides the uncertainty of the prediction from the probabilitydistribution of the model parameters.
Before observing the data, a prior distribution that reflects priorbeliefs about the model is put over the parameter space of the neural network model.
As observationsare added, a posterior distribution is computed by updating the prior distribution via Bayes’ rule.
Neal [1996], Williams [1996] shows that a shallow BNN which has only one infinite-width hiddenlayer converges to a Gaussian process regression model.
This means that the shallow BNN is anapproximation of the nonparametric Bayesian regression model.
Lee et al.
[2017], Matthews et al.[2018] extended this result to the deep BNN model.
However, it is difficult to compute the exact posterior distribution of the BNN since the normalizingconstant of the posterior distribution is intractable.
Therefore, various methods have been proposedto infer BNN through approximate Bayesian computation algorithms.
The methods based on the
Markov chain Monte Carlo (MCMC) algorithm such as Gibbs sampler [Geman and Geman, 1984,
Casella and George, 1992] and Hamiltonian Monte Carlo (HMC) [Neal et al., 2011] is effectiveto approximate true posterior.
By constructing a Markov chain that has the desired distribution asits stationary distribution, one can obtain a sample of the desired distribution.
Recently, scalable
MCMC algorithms such as stochastic gradient Langevin dynamics [Welling and Teh, 2011] and stochastic gradient HMC [Chen et al., 2014] were proposed.
The variational inference (VI) is anapproximation method that finds the closest variational distribution from the posterior distribution.
VImethod changes the Bayesian inference problem to an optimization problem.
Graves [2011] proposedto use a Gaussian distribution as variational distribution for BNN and Kingma and Welling [2013]suggested the reparameterization trick for Gaussian variational distribution.
Practically, VI methodsfor BNN are faster and more stable than MCMC methods.
Blundell et al.
[2015] suggested Bayes by
Backprop algorithm for BNN via variational inference.
There are also studies that approximate BNNwith the dropout [Gal and Ghahramani, 2016] or the Gaussian process [Lee et al., 2017].
1.1 Related works
The performance of the neural network is justified by the universal approximation capability of theneural network.
It is known that even a shallow neural network can approximate arbitrary continuousfunctions sufficiently accurately in L p -sense
[Cybenko, 1989, Hornik, 1991], and as the networkdeepens, the expressive power of the neural network gets stronger [Delalleau and Bengio, 2011,
Bianchini and Scarselli, 2014, Telgarsky, 2015, 2016].
Lee III [1999] showed that the BNN has theuniversal approximation property.
As a measure of the expressive power of a model, consider the range of functional spaces which amodel can express.
The choice of the activation function can affect the expressive power of a model,and in this paper, we consider the Rectified Linear Unit (ReLU) ReLU ( x )
= max { 0 , x } [Nair and
Hinton, 2010], which is the most popular activation function in practice.
Yarotsky [2017] evaluatedthe approximation error of the neural network using the ReLU activation function (ReLU network) forfunctions in Hölder space.
Schmidt-Hieber [2020] showed that the regularized least squared estimatorperformed by ReLU network in nonparametric regression problem converges to the true regressionfunction with nearly minimax rate up to logarithmic factors on Hölder space.
Suzuki [2018] extendedthe results in Schmidt-Hieber [2020] from the Hölder space to the Besov space which contains Hölderspace and Sobolev space.
Polson and Ro ˇ cková [2018] proved a Bayesian version of the results in Schmidt-Hieber [2020].
They proved the Bayesian ReLU network has posterior consistency, theposterior probability is concentrated to the true parameter as collecting data, when the priors on theparameters of the ReLU network are spike-and-slab priors.
Chérief-Abdellatif [2020] shows that theresults in Polson and Roˇcková [2018] are still valid when variational methods are applied.
All of the above results assume a sparse structure in the neural networks.
Sparsity is often assumed inhigh-dimensional models for asymptotic results.
Castillo et al.
[2015] showed theoretical propertiesincluding posterior consistency and rates of consistency of the high-dimensional linear regressionmodel under the spike-and-slab prior.
Song and Liang [2017] extended the results in Castillo et al.[2015] to general shrinkage priors which are more feasible than the spike-and-slab prior.
1.2 Main contribution
We show that the posterior of the Bayesian ReLU network with a spike-and-slab prior with appropriatewidth, depth, and sparsity level is consistent with a nearly minimax rate.
We extend the result of Suzuki [2018] to the Bayesian framework and the results of Polson and Roˇcková [2018] to the Besovspace.
As in Polson and Roˇcková [2018], we show that the same convergence rate is achieved evenwhen the smoothness parameter of the true regression function is not known.
That is, the ReLUnetwork has rich expressive power, and this ability is not constrained even without the smoothnessinformation of the true function.
We get similar results for the shrinkage prior.
We try to narrow the gap between the study of the theoretical properties and the practical applicationof the Bayesian ReLU networks.
For instance, Gaussian distribution is frequently considered asa prior distribution in practical Bayesian ReLU networks.
In this case, the posterior distributionmay not be consistent since the posterior becomes heavy tail distribution over the parameter space[Vladimirova et al., 2019].
Approaches to interpreting Bayesian neural networks as Gaussian Process[Lee et al., 2017] avoid this problem by setting the variance of the prior distribution to be proportionalto the width of the network.
We suggest similar sufficient conditions for the posterior consistencyof the Bayesian ReLU networks.
Compared to precedent studys, our setting is closer to practicaluse.
The results in Suzuki [2018] and Polson and Roˇcková [2018] require to solve the optimizationproblem under the l 0 constraint or compute posterior from spike-and-slab prior respectively.
Wesuggest BNN model, which is easy to infer and consistent with the (nearly) optimal convergence rate.
The rest of the paper is organized as follows.
We set up the problem and introduce the necessaryconcepts in Section 2.
We present the obtained theoretical results in Section 3 and numerical examplesin Section 4.
A summary of the paper and a discussion are in Section 5.
Preliminaries2.1 Notation
The floor function ⌊ x ⌋ denotes the largest integer less than or equal to x ∈ R , and ∥ a ∥ p is anl p -norm of a real vector a .
The u -th derivative of a d -dimensional function f is denoted by D u fwith u ∈ { 0 , 1 , · · ·} d .
The covering number N ( ϵ, A, d ) is the minimal number of d -balls of radiusϵ > 0 necessary to cover a set A .
We denote the dirac-delta function at a ∈ R by δ a .
We use A ≲ Bas shorthand for the inequality A ≤ CB for some constant C > 0 and A ∨ B as shorthand for themax { A, B } .
2.2 Model
Suppose that n input-output observations D n =
( X i , y i ) ni
=1 ⊂ [0 , 1] d × R are i.i.d.
random samplefrom a regression model
[Equation]
where ( ξ i ) ni =1 is an i.i.d.
sequence of Gaussian noises N (cid:0) 0 , σ 2 (cid:1) with known variance σ 2 > 0 andf 0 is the true regression function belonging to the space F .
In frequentist statistics, the maximum likelihood estimation is used to estimate the function f 0 .
Fromthe machine learning viewpoint, it is equivalent to finding ˆ f that minimizes the square loss.
Suzuki[2018] showed that f 0 can be estimated with a nearly optimal rate in this setting.
In practice, a sparse regularization term r ( f ) such as dropout and l 1 -regularizer is also consideredtogether with a loss function.
This is equivalent to estimating the MAP (maximum a posteriori) in
Bayesian modeling.
The posterior distribution gives not only predictive values but also the uncertaintyof the prediction.
Thus, in Bayesian statistics, it is important to find sufficient conditions for a priordistribution or a regularizer which guarantees the consistency or optimal convergence rate of theposterior.
2.3 Functional spaces
In this section, we introduce functional spaces that statisticians have studied for smoothness offunctions.
Let Ω =
[0 , 1] d ⊂ R d be the domain of functions we consider.
The L p -norm of a function is defined as follows:
[Equation]
where f is a real-valued function defined on Ω .
The L p space is the functional space consisting offunctions with bounded L p -norm and is denoted by L p (Ω)
= { f : ∥ f
∥ L p < ∞} .
Let s > 0 be the smoothness parameter and m =
⌊ s ⌋ .
The s -Hölder norm of function f is defined by
[Equation]
The s -Hölder space C s (Ω) is defined as a set of functions with bounded s -Hölder norm.
We call sthe smoothness parameter of the s -Hölder space.
Let k ∈ N and 1 ≤ p ≤ ∞ .
The Sobolev norm of afunction f is defined by
[Equation]
The Sobolev space W k,p (Ω) is defined by the functional space consisting of functions with bounded
Sobolev norm.
Note the notion of the smoothness of a function is related to the differentiability of the function.
The
Besov space extends the concept of the smoothness.
Before defining the Besov space, define the r -thmodulus of smoothness of the function f as follows [Gine and Nickl, 2016]:
[Equation]
[Figure]
[Equation]
For 0 < p, q ≤ ∞ , s > 0 , r =
⌊ s ⌋ + 1 , define the Besov norm of a function f by
[Equation]
The Besov space B sp,q (Ω) is defined as the set of functions with finite Besov norms [Gine and Nickl,2016], i.e.,
[Equation]
Note that Besov spaces can be defined without the differentiability and continuity of functions, and
[Figure]
134 are more general than the Hölder space and the Sobolev space, which are subspaces of Besov spaces[Suzuki, 2018, Sawano, 2018, Triebel, 1983].
A Besov space may contain complicated functions thatcannot be expressed by traditional statistical models.
For instance, the Cantor function f 1 , Figure 1(a), does not belong to any Sobolev space since it does not have a weak derivative but belongs to a
Besov space [Sawano, 2018].
In fact, f 1 ∈ C log2 / log3 ([0 , 1])
⊂ B log2 / log3 ∞ , ∞ ([0 , 1]) .
Also,
[Equation]
is in B 3 / 2 1 , 1 but not in any Hölder space.
Figure 1 have plots of f 1 and f 2 .
Note the minimax convergence rate of a Besov function f 0 in (1) has a lower bound n − s 2 s + d [Gineand Nickl, 2016, Donoho and Johnstone, 1998].
2.4 Sparse neural network
Define (deep) neural network space generated by a parameter space Θ as follows:
[Equation]
where ζ is an activation function.
In this paper, we consider ζ ( x ) =
ReLU ( x ) and the following
B -bounded and S -sparse parameter space [Suzuki, 2018]
[Equation]
and denote S -sparse neural network space with B -bounded parameters Φ(Θ( L, W, S, B )) asΦ( L, W, S, B ) .
Main results
Consider F is a Besov space B sp,q ([0 , 1] d ) .
For each n ∈ N , and θ ∈ Θ , let P ( n ) θ be the probabilitymeasure of the data D n =
( X i , y i ) ni
=1 with a density function p ( n ) θ with respect to the measure µ ( n )and Π be a prior distribution of θ .
The posterior distribution is given by
[Equation]
for A ⊂ Θ .
Let ∥ f
[Equation]
n =
[Equation]
be the norm on empirical measure P Xn
= 1 n (cid:80) ni
{+=
+} δ X i .
Suppose that 0 < p, q ≤ ∞ , ω :
=
d/p < s < ∞ and set ν = ( s − ω ) / (2 ω ) .
Assume that m ∈ Nsatisfies 0 < s < min { m, m − 1 + 1 /p } .
Let N n = ⌈ n d/ (2 s + d ) ⌉ , W 0 = 6 dm ( m + 2) + 2 d and
[Equation]
where c ( d,m ) = (cid:16) 1 + 2 de (2 e )
m √ m (cid:17) − 1 , τ
n = N − s/d n (log N n ) − 1 and ξ = min { 1 , ν − 1 + d − 1 } .
Note the following theorems are the similar as the result in Suzuki [2018], a result for a frequentiststatistics, and shows that the BNNs also converge in nearly minimax rate.
Suzuki [2018] showed thatfor the least squared estimator ˆ f ∈ Φ( L n , W n , S n , B n ) ,
[Equation]
where E D n [ A ] is an expectation of A with respect to the training data D n .
Spike-and-slab prior
This result is an extension of Theorem 6.1 in Polson and Roˇcková [2018].
They showed that the
Bayesian ReLU networks is consistent with nearly minimax rates on Hölder space.
We extend thefunctional space from the Hölder space to the Besov space which contains Hölder spaces and Sobolevspace.
Assume the following spike-and-slab prior π ( θ )
[Equation]
where ˜ π
= U ([ − B n , B n ]) and T n = | Θ( L n , W n , S n , B n ) | .
Theorem 1.
Under the model (1) and prior distribution (12), the posterior distribution concentrates at the rate ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 .
That is, Π(
[Equation]
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.3.
Adaptive estimation
This result is an extension of Theorem 6.2 in Polson and Roˇcková [2018] to the Besov space.
Theyshowed that the Bayesian ReLU networks adapt to smoothness on the Hölder space under thespike-and-slab prior.
Theorem 2.
Assume the model (1), prior distribution (12),
[Equation]
) for λ, ρ, β > 0 , and
[Equation]
where W 1 = 6 dm ( m + 2)
+ 2 d for some m ∈ N .
Suppose that 0 < p, q ≤ ∞ and d/p < s < min { m, m − 1+1 /p } .
The posterior distribution concentrates at the rate ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.4.
In Theorem 1, the prior distribution depends on the smoothness parameters p , q , and s .
Theorem 2shows that, as in Polson and Ro ˇ cková [2018], a similar result can be obtained by considering theappropriate prior distribution even if the values of these parameters are unknown.
Note it can beeasily shown that BNNs do not overfit as in Polson and Roˇcková [2018].
Shrinkage prior
Asymptotic properties of the spike-and-slab prior can be obtained using Theorem 1, but its practicaluse for large neural network models is prohibited due to computation time.
The following priordistributions are more practical than a spike-and-slab prior in terms of computation.
The result to be introduced in this section shows that all the aforementioned results still hold evenunder shrinkage prior.
Assume the following prior
[Equation]
where T n = | Θ( L n , W n , S n , B n ) | and g ( t ) is a continuous, symmetric density function and decreasing on t > 0 .
Theorem 3.
Let ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 .
Under the model (1) and prior distribution (15), assumethat
[Equation]
satisfies
[Equation]
[Equation]
and
[Equation]
where η n = exp( − Knϵ 2 n /S n ) for some K, K 0 > 4 .
Then the posterior distribution concentrates at the rate ϵ n .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.5.
Note that condition (17) is a continuous approximation for the spike part of the (12).
The nextcondition (18) means that the prior should have enough thick tail to sample true function.
The lastcondition (19) restricts the thickness of the tail to prevent function divergent.
Unlike the other priors, the shrinkage prior is relatively straightforward to implement.
For example, toimplement the model in Theorem 1, one can consider the MCMC algorithm
[Sun et al., 2022] whichcombines Gibbs sampler, the Metropolis-Hastings algorithm, and the SGHMC algorithm.
Shrinkageprior avoids the posterior computation with varying dimensions, and enables feasible computation.
Example 1 (Gaussian mixture prior) .
Let ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 , ψ and Ψ be density and inverse survival functions of the standard Gaussian distribution N (0 , 1) respectively.
When 1 / 2 < ξ ≤and B n ≍ N ξn ,
[Equation]
with
[Equation]
and
[Equation]
satisfies condition of Theorem 3.
Numerical examples
However, these results are still not enough practical to be used.
As the dimensionality d of the dataincreases, the complexity of the neural network to obtain the theoretical optimality increases rapidly.
For instance, consider a problem of estimating the Besov functions f 1 and f 2 .
Assume the Gaussianmixture prior distribution for each parameter as Example 1,
[Equation]
In Table 1 and Table 2, the hyperparameters (model parameters) to estimate each function werepresented.
These values are calculated numerically and may vary depending on the computerenvironment.
As the number of samples increases, the complexity of the neural network modelincreases.
Its speed is affected by the complexity of the true regression function.
As can be seen fromboth tables, the smaller of the variances is extremely small.
Considering the numerical precision, it isindistinguishable from spike-and-slab prior [Ishwaran and Rao, 2005].
[Table] [Table]
Instead, although the conditions of the theorem are not satisfied, we checked that the two functionscan be estimated even with a smaller model.
The results are presented in the Appendix D.
Conclusions
In this paper, we justify the theoretical properties of neural networks that are currently widely used.
Specifically, we focused on the asymptotic properties of BNNs and extend results in Polson and
Ro ˇ cková [2018] to the Besov space from the Hölder space.
Theorem 1 shows that the BNN witha spike-and-slab prior has consistency posterior with a nearly minimax rate at Besov functionswith known smoothness.
Theorem 3 extends the result to the shrinkage prior.
According to thesetheorems, BNNs have posterior consistency and (nearly) optimal rates by assuming proper priorsuch as Gaussian mixture prior.
We found there are three sufficient conditions to achieve posteriorconsistency, (1) enough mass around zero, (2) thick tail to sample true function (3) while not toomuch.
Theorem 2 shows that even if the smoothness of the function is not known, the same rate canbe obtained by considering a prior on the architecture of the neural networks.
The results in this paper are an extension of the results of Suzuki [2018] to BNN also.
We suggest apractical method that can achieve the optimal rate while providing the uncertainty of the prediction.
Note they showed the theoretical properties of the neural networks in the Besov space via nonparametric regression based on B-spline.
In other words, similar results can be obtained using the Bayesian LARK B-spline model
[Park et al., 2021].
In future works, performance comparison experiments with other models will also be of great help inestablishing the properties of BNN.
Empirically, we know that there are fields in which decision-treebased model such random forest or neural network models perform better than traditional statisticalmodels that find smooth functional relationships such as Kernel regression.
For these data, it is natural that there would be a non-smooth relationship between the explanatory variable and the responsevariable.
For this reason, it is essential to compare the relationship between Bayesian neural networkmodels and staitical model for follow-up studies.
We obtained results under the limited assumption, setting the networks fully connected and using the ReLU activation function.
Also, as mentioned in the numerical experiment section, the results are notenough practical to be used.
In addition, we does not suggest result for the adaptive version of the Theorem 3 yet.
It is a necessary future study to propose a general shrinkage prior distribution thatsatisfies all conditions even when the parameters of the network are varying.
The results of this studyare at a basic level compared to the needs for BNNs in the industrial field, such as visual data analysisand natural language processing.
It is important to research the theoretical properties of BNNs usedin the practical area through ongoing studies.
Checklist
For all authors..
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplementary
material.
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
you describe any potential participant risks, with links to Institutional Review
[Table]
approvals, if applicable
[N/A
[Table]
Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
A Posterior consistency
Lemma 1 (Lemma 2 in Ghosal and Van Der Vaart [2007]) .
Suppose that observation X ( n ) =
( X 1 , · · · , X n ) of independent observations
X i .
Assume that the distribution P θ,i of the i th component
X i has a density p θ,i relative to a σ -finite measure µ i .
Let P ( n ) θ = [Equation] ni
[Equation] ,i be the product measures and
[Equation]
Then there exists test ϕ n such that P ( n ) θ 0 ϕ n ≤ e − 12 nd 2 n ( θ 0 ,θ 1 ) and P ( n ) θ (1 − ϕ n ) ≤ e − 12 nd 2 n ( θ 0 ,θ 1 ) forall θ ∈ Θ such that d n ( θ, θ 1 ) ≤ 118 d n ( θ 0 , θ 1 ) .
Proof.
See Ghosal and Van Der Vaart [2007].
Let define divergences
[Equation]
(22) and set
B ∗ n ( θ 0 , ϵ
[Equation]
k
[Equation] = [Equation] n
[Equation]
Here, the C k is the constant satisfying
[Equation]
for k ≥ 2 .
The existence of C k is guaranteed by Marcinkiewiz-Zygmund inequality.
Then thefollowings hold
Lemma 2 (Theorem 4 of Ghosal and Van Der Vaart [2007]) .
Let P ( n ) θ be product measures andd n = 1 n (cid:80) ni =
[Equation] ( √ p θ,i −√ p θ ′ ,i ) 2 dµ i .
Suppose that for a sequence ϵ n → 0 such that nϵ 2 n is boundedaway from zero, some k > 1 , all sufficiently large j and sets Θ n ⊂ Θ , the following conditions hold:
[Equation]
Then P ( n ) θ
(Π n ( θ : d n ( θ, θ 0 ) ≥ M n ϵ n | D n ) → 0 for any sequence M n → ∞ .
Proof.
See Ghosal and Van Der Vaart [2007].
The following is an adapted version of the Lemma 2, which introduced in Polson and Roˇcková [2018](white noise model and k = 2 ).
Assume the model (1).
Let
[Equation]
If there exist d > 2 and F n ⊂ F such that
[Equation]
for any ϵ n → 0 , nϵ 2 n → ∞ ,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
B Consistency of neural network
The upper bound of the covering number of the neural network space is given by as follows.
Lemma 3 (Lemma 3 in Suzuki [2018]) .
[Equation]
[Equation]
[Equation]
for W, L ≥ 3 .
Proof.
See appendix C.
For L, W, S ∈ N and B, a > 0 , define a functional space
[Equation]
30) where Θ( L, W, S, B
[Equation]
a ) = [Equation] [Equation] |
[Equation]
a ) [Equation] n
[Equation]
S [Equation] B ) (cid:111) .
Lemma 4.
∀ ϵ ≥ 2 aL ( B ∨ 1)
L − 1 ( W + 1) L ,
[Equation]
[Equation]
(31) for W, L ≥ 3 .
Proof.
See appendix {+C.
+}
The following lemma shows that for any function f 0 in the Besov space, there exists neural networkswhich is closed enough to f 0 .
Lemma 5 (Proposition 1 in Suzuki [2018] with r = ∞ ) .
Suppose that 0 < p, q ≤ ∞ , ω :
=
d/p <s < ∞ and ν =
( s − ω ) / (2 ω ) .
Assume that N ∈ N is sufficiently large and m ∈ N satisfies0 < s < min { m, m − 1 + 1 /p } .
Let W
= 6 dm ( m + 2) + 2 d .
Then,
[Equation]
32) for
[Equation]
33) where c ( d,m ) = (cid:16) 1 + 2 de (2 e ) m √ m (cid:17) − 1 and τ n = N − s/d n (log N n ) − 1 .
Proof.
See Suzuki [2018].
C Proof of theorems
C.1 Proof of lemma 3
Proof.
Let denote f ∈ Φ( L, W, S, B ) as
[Equation]
34) for k = 2 , · · · , L , and A − L +1 ( f )( x ) =
A +1 ( f )( x ) = x .
Then,
[Equation]
Since f ∈ Φ( L, W, S, B ) ,
[Equation]
[Equation]
≤
[Equation]
∞ [Equation] [Equation] 35) for all x , where A j, : is the j -th row of the matrix A .
Similarly,
[Equation] [Equation]
Fix ϵ > 0 and θ ∈ Θ( L, W, S, B ) .
For any θ ∗ ∈ Θ( L, W, S, B ) which satisfies ∥ θ − θ ∗ ∥ ∞ <
ϵ ,
[Equation]
≤
[Equation]
B
[Equation]
[Equation]
( W
[Equation]
Let s be the number of nonzero component of θ , s ≤ S .
Consider a subspace Θ θ ( L, W, S, B ) of the parameter space
Θ( L, W, S, B ) consists of parameters which have s nonzero component.
Choose θ 1 , · · · , θ M form each grid divided with length ϵ ∗ =
ϵ L ( B ∨ 1)
L − 1 ( W +1)
L over Θ θ ( L, W, S, B ) .
Then f θ ∈
M (cid:91) m = { f : ∥ f − f θ m ∥ L ∞ < ϵ } .
Note
T = | Θ
[Equation]
[Equation]
[Equation]
[Equation]
|
[Equation]
l =
[Equation]
( W + 1)
[Equation] ≤ ( L + 1) [Equation] + 1) 2 ≤ ( W + 1) L for W, L ≥ 3 and the number of cases of choose s nonzero components is
[Equation]
Thus,
[Equation]
≤ [Equation] and we get desired results taking the logarithm of both sides.
C.2 Proof of lemma
Proof.
Let ˜ θ ( a ) =
( θ i I ( | θ i | > a )) .
Then (cid:13)(cid:13)(cid:13) θ − ˜ θ ( a ) (cid:13)(cid:13)(cid:13) ∞ ≤ a and ˜Θ( L, W, S, B, a ) =
(cid:110)
˜
θ ( a ) : θ ∈ Θ( L, W, S, B, a ) (cid:111) =
Θ( L, W, S, B ) .
As in the proof of Lemma 3,
[Equation]
(39) for any f θ , f ˜ θ ( a ) ∈
Φ( L, W, S, B, a, F ) .
Let s ≤ S be the number of nonzero component of ˜ θ ( a ) .
Consider a subspace ˜Θ θ ( L, W, S, B, a ) of ˜Θ( L, W, S, B, a ) consists of parameters which have s nonzero component.
Choose θ 1 , · · · , θ M form each grid divided with length ϵ ∗ =
[Equation]
over ˜Θ θ ( L, W, S, B, a ) .
Then
[Equation]
from triangular inequality.
We get desired results in the similar way as in the proof of the Lemma3.
C.3 Proof of theorem 1
Proof.
Let
A ϵ,M := { f ∈ F : ∥ f − f 0 ∥ n ≤ Mϵ } .
It is enough to show that there exists a constant C > 2 which satisfies
[Equation]
for sufficiently large n .
Let F n = F = Φ( L n , W n , S n , B n ) .
First, (c) is trivial since
F − F n = ∅ .
Since { f ∈ F n : ∥ f ∥ L ∞ ≤ ϵ } ⊂ { f ∈ F n : ∥ f ∥ n ≤ ϵ } and Lemma 3,
[Equation]
for sufficiently large n .
Thus, (a) holds.
Here, the last inequality holds from
[Equation]
Next, from Lemma 5, there is a constant C 1 > 0 and ˆ f n = f ˆ θ ∈ F such that
[Equation]
for sufficiently large n .
Let ˆ γ and ˆ θ ˆ γ be index and value of nonzero components of ˆ θ respectively.
LetΘ(ˆ γ ; L n , W n , S n , B n ) be a subset of parameter space Θ( L n , W n , S n , B n ) consists of parameterswhich have nonzero components at ˆ γ only and F (ˆ γ ) =
Φ(ˆ γ ; L n , W n , S n , B n ) be the neural networkspace generated by Θ(ˆ γ ; L n , W n , S n , B n ) .
Note
[Equation]
for sufficiently large n .
As in the proof of Lemma 3,
[Equation]
Thus,
[Equation]
C.4 Proof of theorem
Proof.
Let F = Φ .
It is enough to show that there exists a constant C ′ > 2 which satisfies
[Equation]
for sufficiently large n .
For sufficiently large constants C N
[Equation]
C B > [Equation] let ˜ N n = C N N n , ˜ L n = L
[Equation]
, ˜
[Equation]
n =
[Equation]
B nϵ 2 n and
[Equation]
First, show that (a) holds.
From Lemma 3,
[Equation]
and
[Equation]
for sufficiently large n .
Next, show that (b) holds.
For N n , L n in Theorem 1, N n < ˜ N n , L n < ˜ L n .
Moreover, since B n ≤ N n ≤ nϵ 2 n for sufficiently large n , there exists a constant D > 0 such that
[Equation]
[Equation]
for sufficiently large n .
Then
[Equation]
holds and (b) holds for C ′ =
C + 3 D .
From Π
[Equation]
and Chernoff bound , for any positive number t > 0 ,
[Equation]
Letting t = log ˜ N n , log ˜ L n respectively, we get
[Equation]
Also, since
[Equation]
for sufficiently large C N , C B > 0 and π 3 ( B > ˜ B n ) = exp (cid:16) − β ˜ B n (cid:17) , (c) holds.
C.5 Proof of theorem
Proof.
It is enough to show that there exists a constant C > 2 which satisfies(a) sup ϵ>ϵ n log N ( ϵ/ 36 , A ϵ, 1 ∩ F n , ∥·∥ n ) ≤ nϵ 2 n(b)
− log Π
( A ϵ n , 1 ) ≤ Cnϵ 2 n(c
[Equation]
for sufficiently large n .
Let F n = Φ( L n , W n , S n , B n , a n ) and F = Φ( L n , W n , T n ) .
It is easy toshow that (a) holds from Lemma 4 as in the proof of the Theorem 1.
By the assumption,Π( F − F n
[Equation]
for some 4 < K 1 < K .
Letting C
= min { K 0 , K 1 } − 2 , (c) holds.
We use tail bound for binomialdistribution in Arratia and Gordon [1989] for the second inequality.
Next, from Lemma 5, there is aconstant C 1 > 0 and ˆ f n = f ˆ θ ∈ F n such that
[Equation]
Let ˆ γ and ˆ θ ˆ γ be index and value of nonzero components of ˆ θ respectively.
Let
[Equation]
and ˜Θ(ˆ γ ; L n , W n , S n , B n , a n ) be a subset of parameter space ˜Θ
( L n , W n , S n , B n , a n ) consists ofparameters which have nonzero components at ˆ γ only and F n (ˆ γ ) =
˜Φ(ˆ γ ; L n , W n , S n , B n , a n ) bethe neural network space generated by ˜Θ(ˆ γ ; L n , W n , S n , B n , a n ) .
Note
[Equation]
[Equation]
56) for sufficiently large n .
As in the proof of Lemma 3,
[Equation]
(57) where t n = ϵ n 2( W n +1)
Ln L n ( B n ∨ 1)
Ln − 1 .
Letting y n = (cid:90) B n
B n − t n g ( t ) dt ≥ t n g ( B n ) ,
[Equation]
D Numerical Examples
D.1 Gaussian Mixture Prior
Consider a problem of estimating the Besov functions f 1 and f 2 .
Assume the Gaussian mixture priordistribution for each parameter as Example 1.
We sampled n points from x ∼ U (0 , 1) and y from
N (cid:0) f 1 ( x ) , σ 20 (cid:1) and N (cid:0) f 2 ( x ) , σ 20 (cid:1) respectively, where σ 20 = 0 . 01 .
As mentioned in Section
[Table]
we used a smaller model that was weaker than the conditions of thetheorem.
We fix the depth L n = 5 and the width W n = 200 of the model and use same prioras in the theorem (Gaussian mixture BNN).
Also, the smaller of the variances is set to σ 1 n ←max { 0 .
001 , σ 1 n } to avoid numerical precision issues.
We summarize the hyperparameters of themodel used in the experiment in Table 3 and Table
[Table]
[Table]
[Table]
We fit the model using the NUTS algorithm [Hoffman et al., 2014] with the python Pyro [Binghamet al., 2019] and PyTorch [Paszke et al., 2019] packages.
Experiments were run on a GPU serverwith one Nvidia RTX 3090.
The code and instruction related to the experiment are provided in thesupplementary material.
Figure 2 and Figure 3 show the results.
For both problems, as the number increase
∥
D.2 Gaussian Prior
We performed the exact same experiment as before using Gaussian prior (Gaussian BNN).
Thestandard deviation of the Gaussian prior is the same as σ 2 n .
Figure 4 and Figure 5 show the results.
For both problems, as the number of the data increase, mean functions get closer to the true regressionfunction and empirical error ∥ f − f i ∥ n decrease.
D.3 Comparison
We compare two model in terms of empirical errors (training error)
∥
f − f i ∥ n and L 2 norms (testerror) e f = ∥
f − f i ∥ 2 between true function f i and sampled function f .
Specifically, approximatee f as ˆ e f using Riemann sum,
[Equation]
with (cid:8) x ∗ k : x ∗ k = 1 n ∗ , 1 ≤ k ≤ n ∗ (cid:9) .
As in Table 5 and Table 6, both model give similar results, butthe Gaussian mixture BNN give slightly better results.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table] [Table]