Asymptotic Properties for Bayesian Neural Network in Besov Space
Abstract
Neural networks have shown great predictive power when dealing with various unstructured data such as images and natural languages.
The Bayesian neural network captures the uncertainty of prediction by putting a prior distribution for the parameter of the model and computing the posterior distribution.
In this paper, we show that the Bayesian neural network using spike-and-slab prior has consistency with nearly minimax convergence rate when the true regression function is in the Besov space.
Even when the smoothness of the regression function is unknown the same posterior convergence rate holds and thus the spike-and-slab prior is adaptive to the smoothness of the regression function.
We also consider the shrinkage prior, which is more feasible than other priors, and show that it has the same convergence rate.
In other words, we propose a practical Bayesian neural network with guaranteed asymptotic properties.
Introduction
The neural network is a machine learning technique that can handle complex structures in data by combining linear transformations and non-linear transformations [Goodfellow et al., 2016].
The neural network can show great predictive power when dealing with unstructured data ranging from image data to natural language processing.
Moreover, end-to-end learning is possible by grasping the inherent structure of data only with inputs and outputs without explicit intermediate steps.
Such properties have played major roles in the rapid development of numerous artiﬁcial intelligence applications.
The Bayesian neural network (BNN) provides the uncertainty of the prediction from the probability distribution of the model parameters.
Before observing the data, a prior distribution that reﬂects prior beliefs about the model is put over the parameter space of the neural network model.
As observations are added, a posterior distribution is computed by updating the prior distribution via
Bayes’ rule.
Neal [1996], Williams [1996] showed that a shallow BNN which has only one inﬁnitewidth hidden layer converges to a Gaussian process regression model.
This means that the shallow
BNN can be regarded as a ﬁnite approximation of the nonparametric Bayesian regression model.
Lee et al.
[2017], Matthews et al.
[2018] extended this result to the deep BNN model.
However, it is difﬁcult to compute the exact posterior distribution of the BNN since the normalizing constant of the posterior distribution is intractable.
Therefore, various methods have been proposed to infer BNN through approximate Bayesian computation algorithms.
The methods based on the
Markov chain Monte Carlo (MCMC) algorithm such as the Gibbs sampler [Geman and Geman, 1984, Casella and George, 1992] and Hamiltonian Monte Carlo (HMC) [Neal et al., 2011] is effective to approximate true posterior.
By constructing a Markov chain that has the desired distribution as its stationary distribution, one can obtain a sample of the desired distribution.
Recently, scal36th Conference on Neural Information Processing Systems (NeurIPS 2022).
able MCMC algorithms such as stochastic gradient Langevin dynamics [Welling and Teh, 2011] and stochastic gradient HMC (SGHMC, Chen et al.
[2014]) were proposed.
The variational inference (VI) is an approximation method that ﬁnds the closest variational distribution from the posterior distribution.
VI method changes the Bayesian inference problem to an optimization problem.
Graves [2011] proposed to use of a Gaussian distribution as variational distribution for BNN and
Kingma and Welling [2013] suggested the reparameterization trick for Gaussian variational distribution.
Practically, VI methods for BNN are faster and more stable than MCMC methods.
Blundell et al.
[2015] suggested Bayes by Backprop algorithm for BNN via variational inference.
There are also studies that approximate BNN with the dropout [Gal and Ghahramani, 2016] or the Gaussian process [Lee et al., 2017].
1.1 Related works
The performance of the neural network is justiﬁed by the universal approximation capability of the neural network.
It is known that even a shallow neural network can approximate arbitrary continuous functions sufﬁciently precisely in L p -sense
[Cybenko, 1989, Hornik, 1991], and as the network deepens, the expressive power of the neural network gets stronger [Delalleau and Bengio, 2011,
Bianchini and Scarselli, 2014, Telgarsky, 2015, 2016].
Lee III [1999] showed that the BNN has the universal approximation property.
As a measure of the expressive power of a model, consider the range of function spaces that a model can express.
The choice of the activation function can affect the expressive power of a model, and in this paper, we consider the Rectiﬁed Linear Unit (ReLU) ReLU ( x )
= max { 0 , x } [Nair and Hinton, 2010], which is the most popular activation function in practice.
Yarotsky [2017] evaluated the approximation error of the neural network using the ReLU activation function (ReLU network) for functions in the Sobolev space.
Schmidt-Hieber [2020] showed that the regularized least squared estimator performed by the ReLU network in a nonparametric regression problem converges to the true regression function with nearly minimax rate up to logarithmic factors on the Hölder space.
Suzuki [2018] extended the results in Schmidt-Hieber [2020] from the Hölder space to the Besov space which contains the Hölder and Sobolev spaces.
Polson and Roˇcková [2018] proved a Bayesian version of the results in Schmidt-Hieber [2020].
They proved the Bayesian ReLU network has posterior consistency, the posterior probability is concentrated on the true parameter as collecting data, when the priors on the parameters of the ReLU network are spike-and-slab priors.
Chérief-Abdellatif [2020] showed that the results in Polson and Roˇcková [2018] are still valid when variational methods are applied.
All of the above results assume a sparse structure in the neural networks.
Sparsity is often assumed in high-dimensional models for asymptotic results.
Castillo et al.
[2015] showed theoretical properties including posterior consistency and rates of consistency of the high-dimensional linear regression model under the spike-and-slab prior.
Song and Liang [2017] extended the results in Castillo et al.
[2015] to the shrinkage priors which are more feasible than the spike-and-slab prior.
1.2 Main contribution
We show that the posterior of the Bayesian ReLU network with a spike-and-slab prior with appropriate width, depth, and sparsity level is consistent with a nearly minimax rate.
We extend the result of Suzuki [2018] to the Bayesian framework and the results of Polson and Roˇcková [2018] to the
Besov space.
As in Polson and Roˇcková [2018], we show that the same convergence rate is achieved even when the smoothness parameter of the true regression function is not known.
That is, the ReLU network has rich expressive power, and this ability is not constrained even without the smoothness information of the true function.
We get similar results for the shrinkage prior.
We try to narrow the gap between the study of the theoretical properties and the practical application of the Bayesian ReLU networks.
For instance, Gaussian distribution is frequently considered as a prior distribution in practical Bayesian ReLU networks.
In this case, the posterior distribution may not be consistent since the posterior becomes heavy tail distribution over the parameter space [Vladimirova et al., 2019].
Approaches to interpreting the Bayesian neural networks as Gaussian
Process [Lee et al., 2017] avoid this problem by setting the variance of the prior distribution to be proportional to the width of the network.
We suggest similar sufﬁcient conditions for the posterior consistency of the Bayesian ReLU networks.
Compared to precedent studies, our setting is closer to practical use.
The results in Suzuki [2018] and Polson and Roˇcková [2018] require to solve the optimization problem under the l 0 constraint or compute posterior from spike-and-slab prior respectively.
We suggest BNN model, which is easy to infer and consistent with the (nearly) optimal convergence rate.
The rest of the paper is organized as follows.
We set up the problem and introduce the necessary concepts in Section 2.
We present the obtained theoretical results in Section 3 and numerical examples in Section 4.
A summary of the paper and a discussion are in Section 5.
Preliminaries 2.1 Notation
The ﬂoor function ⌊ x ⌋ denotes the largest integer less than or equal to x ∈ R , the ceiling function ⌈ x ⌉ denotes the smallest integer greater than or equal to x ∈ R , and ∥ a ∥ p is an l p -norm of a real vector a .
The u -th derivative of a d -dimensional function f is denoted by D u f with u ∈ { 0 , 1 , · · ·} d .
The covering number N ( ϵ, A, d ) is the minimal number of d -balls of radius ϵ > 0 necessary to cover a set A .
We denote the dirac-delta function at a ∈ R by δ a .
We use A ≲ B as shorthand for the inequality A ≤ CB for some constant C > 0 and A ∨ B as shorthand for the max { A, B } .
2.2 Model
Suppose that n input-output observations D n =
( X i , y i ) ni
=1 ⊂ [0 , 1] d × R are i.i.d.
random sample from a regression model
[Equation]
where ( ξ i ) ni =1 is an i.i.d.
sequence of Gaussian noises N (cid:0) 0 , σ 2 (cid:1) with known variance σ 2 > 0 and f 0 is the true regression function belonging to the space F .
In frequentist statistics, the maximum likelihood estimation is used to estimate the function f 0 .
From the machine learning viewpoint, it is equivalent to ﬁnding ˆ f that minimizes the square loss.
Suzuki [2018] showed that f 0 can be estimated with a nearly optimal rate in this setting.
In practice, a sparse regularization term r ( f ) such as dropout and l 1 -regularizer is also considered together with a loss function.
This is equivalent to estimating the MAP (maximum a posteriori) in the Bayesian modeling.
The posterior distribution gives not only predictive values but also the uncertainty of the prediction.
Thus, in the Bayesian statistics, it is important to ﬁnd sufﬁcient conditions for a prior distribution or a regularizer which guarantees the consistency or optimal convergence rate of the posterior.
2.3 Function spaces
In this section, we introduce function spaces that statisticians have studied for the smoothness of functions.
Let Ω =
[0 , 1] d ⊂ R d be the domain of functions we consider.
The L p -norm of a function is deﬁned as follows:
[Equation]
where f is a real-valued function deﬁned on Ω .
The L p space is the function space consisting of functions with bounded L p -norm and is denoted by L p (Ω)
=
{ f : ∥ f
∥
L p < ∞} .
Let s > 0 be the smoothness parameter and m =
⌊ s ⌋ .
The s -Hölder norm of function f is deﬁned by
[Equation]
The s -Hölder space C s (Ω) is deﬁned as a set of functions with bounded s -Hölder norm.
We call s the smoothness parameter of the s -Hölder space.
Let k ∈ N and 1 ≤ p ≤ ∞ .
The Sobolev norm of a function f is deﬁned by
[Equation]
The Sobolev space W k,p (Ω) is deﬁned by the function space consisting of functions with bounded
Sobolev norm.
Note the notion of the smoothness of a function is related to the differentiability of the function.
The
Besov space extends the concept of smoothness.
Before deﬁning the Besov space, deﬁne the r -th modulus of smoothness of the function f as follows [Gine and Nickl, 2016]:
[Equation]
For 0 < p, q ≤ ∞ , s > 0 , r =
⌊ s ⌋ + 1 , deﬁne the Besov norm of a function f by
[Equation]
The Besov space B sp,q (Ω) is deﬁned as the set of functions with ﬁnite Besov norms [Gine and Nickl, 2016], i.e.,
[Equation]
Note that the Besov spaces can be deﬁned without the differentiability and continuity of functions, and are more general than the Hölder and Sobolev spaces, which are subspaces of the Besov spaces.
The Besov space may contain complicated functions including discontinuous function if d/p ≥ s .
For instance, the Cantor function f 1 does not belong to any Sobolev space since it does not have a weak derivative but belongs to a Besov space [Sawano, 2018].
In fact, f 1 ∈ C log2 / log3 ([0 , 1])
⊂
B log2 / log3 ∞ , ∞ ([0 , 1]) .
Also,
[Equation]
is not in any Hölder space.
Note when p < 2 , the smoothness of the Besov space is inhomogeneous [Suzuki, 2018].
Donoho and Johnstone [1994] suggested spatially variable functions including Blocks
[Equation]
where sgn ( x ) is a function that extracts the sign of a real number x , and HeaviSine
[Equation]
Since f 2 , f 3 and f 4 are functions of bounded variation, in B 11 , ∞ ([0 , 1])
[Peetre, 1976, Suzuki, 2018].
We plot the functions f 1 , f 2 , f 3 and f 4 in Figure 1.
2.4 Sparse neural network
Deﬁne (deep) neural network space generated by a parameter space Θ as follows:
[Equation]
[Figure]
x
[Figure] [Figure]
[Figure]
where ζ is an activation function.
In this paper, we consider ζ ( x ) =
ReLU ( x ) and the following
B -bounded and S -sparse parameter space [Suzuki, 2018]
[Equation]
X l
[Equation]
(cid:16)(cid:13)(cid:13)(cid:13) W ( l ) (cid:13)(cid:13)(cid:13)
+ (cid:13)(cid:13)(cid:13) b ( l ) (cid:13)(cid:13)(cid:13) (cid:17) ≤ S, ∥ θ ∥ ∞ =
max l n(cid:13)(cid:13)(cid:13) W ( l ) (cid:13)(cid:13)(cid:13)
[Equation]
10) and denote S -sparse neural network space with B -bounded parameters Φ(Θ( L, W, S, B ))
as Φ( L, W, S, B ) .
We can deﬁne the entire neural network model space as follows
[Equation]
Main results
Let UB ( F ) = (cid:8) f ∈ L ∞ ([0 , 1] d ) : | f ( x ) | ≤ F, ∀ x ∈ [0 , 1] d (cid:9) be an uniformly bounded family of functions with bound F > 0 .
Consider F = B sp,q ([0 , 1] d ) ∩ UB ( F ) .
For each n ∈ N , and f ∈ F , let P ( n ) f be the probability measure of the data D n =
( X i , y i ) ni
=1 with a density function p ( n ) f with respect to the measure µ ( n ) and Π be a prior distribution of f .
The posterior distribution is given by
[Equation]
for A ⊂ Θ .
Let
[Equation]
be the norm on empirical measure P Xn = 1 n P ni =
δ X i and
[Equation]
where P X is a marginal distribution of X .
Suppose that P X has a bounded density function p X ( x ) on [0 , 1] d .
Without loss of generality, we assume that 0 ≤ p X ( x ) ≤ R ≤ 2 .
Suppose that 0 < F < ∞ , 0 < p, q ≤ ∞ , ω :=
d (1 /p − 1 / 2) + < s < ∞ and set ν = ( s − ω ) / (2 ω ) .
Assume that m ∈ N satisﬁes 0 < s < min { m, m − 1 + 1 /p } .
Let N n = ⌈ n d/ (2 s + d ) ⌉ , W 0 = 6 dm ( m + 2) + 2 d and
L n = L
[Equation]
(13) where c ( d,m ) = (cid:16) 1 + 2 de (2 e ) m √ m (cid:17) −, L ( N n ) = 3 + 2 ⌈ log(cid:16)
[Equation]
d ∨ m
[Equation]
n
[Equation]
(cid:17) + 5 ⌉⌈ log
[Equation]
( d
[Equation]
m ) ⌉ , τ n =
N − s/d − ( ν − 1 + d − 1 )( d/p − s )
[Equation]
n (log N n ) − 1 and Ξ = ( ν − 1 + d − 1 )(1 ∨ ( d/p − s ) + ) .
Note the theorems in this section are similar to the results in Suzuki [2018], which studied the asymptotic properties of the least squared estimator.
They showed that for the least squared estimator
[Equation]
where ¯ f is the clipping of f deﬁned by ¯ f = min { max { f, − F } , F } for f 0 ∈ UB ( F ) and E D n [ A ] is an expectation of A with respect to the training data D n .
Note the minimax convergence rate of a
Besov function f 0 in (1) has a lower bound n − s 2 s + d [Gine and Nickl, 2016, Donoho and Johnstone, 1998].
Donoho and Johnstone [1998] also showed that the minimax rate of a linear estimator has a lower bound, n − s − (1 /p − 1 / 2)+
s +1 − 2(1 /p − 1 / 2)+ for d = 1 , s > 1 /p and 1 ≤ p, q ≤ ∞ or s = p = q = 1 .
Thus, any linear estimator cannot achieve the minimax optimal rate on the Besov space for p < 2 and is dominated by the neural network model.
Spike-and-slab prior
This result is an extension of Theorem 6.1 in Polson and Roˇcková [2018].
They showed that the
Bayesian ReLU networks are consistent with nearly minimax rates in the Hölder space.
We extend the function space from the Hölder space to the Besov space which contains the Hölder and Sobolev spaces.
Assume the following spike-and-slab prior π ( θ )
[Equation]
where ˜ π ( θ j | L, W, S, B ) =
U ( θ j ; [ − B, B ]) and T = | Θ( L, W, S, B ) | .
Theorem 1.
Under model (1) and prior distribution (15) and (16).
Suppose that 0 < F < ∞ , 0 < p, q ≤ ∞ and d (1 /p − 1 / 2) + < s .
Then the posterior distribution concentrates at the true function with a rate ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.4.
As mentioned in Suzuki [2018], the condition d (1 /p − 1 / 2) + <
s indicates that f 0 ∈ B sp,q ([0 , 1] d ) is in L 2 ([0 , 1] d ) and can be discontinuous.
When p = q = ∞ , B sp,q ([0 , 1] d ) =
C s ([0 , 1] d ) and the result by Polson and Roˇcková [2018] is a special case of Theorem 1.
Adaptive estimation
This result is an extension of Theorem 6.2 in Polson and Roˇcková [2018] to the Besov space.
They showed that the Bayesian ReLU networks adapt to smoothness on the Hölder space under the spikeand-slab prior.
Theorem 2.
Assume model (1).
Let
[Equation] [Equation]
Consider the following prior
[Equation]
Z = 1 ,
[Equation] (18) and prior distribution (15) of θ given ˜ L n ( H n ) , ˜ W n ( H n , N ) , ˜ S n ( H n , N ) , ˜ B n ( H n , N ) on the function space Φ
[Equation]
for any H n → ∞ and λ N > 0 .
Suppose that 0 < F < ∞ , 0 < p, q ≤ ∞ and d (1 /p − 1 / 2) + < s < min { m, m − 1 + 1 /p } .
The posterior distribution concentrates at the true function with a rate ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.5.
In Theorem 1, the prior distribution depends on the smoothness parameters p , q , and s .
Theorem 2 shows that, as in Polson and Roˇcková [2018], a similar result can be obtained by considering the appropriate prior distribution even if the values of these parameters are unknown.
Note it can be easily shown that BNNs do not overﬁt as in Polson and Roˇcková [2018].
Shrinkage prior
Asymptotic properties of the spike-and-slab prior can be obtained using Theorem 1, but its practical use for large neural network models is unsuitable due to the computational cost.
The following prior distributions are more practical than a spike-and-slab prior in terms of computation.
The result to be introduced in this section shows that all the aforementioned results still hold even under the shrinkage prior.
Assume the following prior
[Equation]
where T = | Θ( L, W, S, B ) | , g ( t ) :
= g ( t | L, W, S, B ) is a continuous, symmetric density function and decreasing on t > 0 .
Theorem 3.
Assume model (1), prior distribution (16) and (20).
Suppose that 0 < F < ∞ , 0 < p, q ≤ ∞ and d (1 /p − 1 / 2) + < s < min { m, m − 1 + 1 /p } .
Let ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 and g ( t ) be a function such that
[Equation]
satisﬁes
[Equation]
and
[Equation]
where η n = exp( − Knϵ 2 n /S n ) for some K, K 0 >
The posterior distribution concentrates at the true function with a rate ϵ n .
That is,
[Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.6.
Note that condition (22) is a continuous approximation for the spike part of (15).
The next condition (23) means that the prior should have enough thick tail to sample true function.
The last condition (24) restricts the thickness of the tail to prevent a function from being divergent.
Unlike the other priors, the shrinkage prior is relatively straightforward to implement.
For example, to implement the model in Theorem 1, one can consider the MCMC algorithm
[Sun et al., 2022] which combines Gibbs sampler, the Metropolis-Hastings algorithm, and the SGHMC algorithm.
The shrinkage prior avoids the posterior computation with varying dimensions, and enables feasible computation.
Note that the conditions are sufﬁcient conditions for the neural network model necessary to estimate the function in the worst case.
In other words, we may infer true function f 0 sufﬁciently even for a prior distribution that satisﬁes only some conditions.
Example 1 (Gaussian prior) .
Let ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 and ψ be a density functions of the standard Gaussian distribution N (0 , 1) . [Equation] satisﬁes (24).
Example 2 (Gaussian mixture prior) .
Let ϵ n = n − s/ (2 s + d ) (log n ) 3 / 2 , ψ and Ψ be density and inverse survival functions of the standard Gaussian distribution N (0 , 1) respectively.
When B n ≍ N min { 1 , Ξ } n ,
[Equation]
with
[Equation]
and
[Equation]
satisﬁes
Numerical examples
However, these results are still not enough practical to be used.
As the scale of the data n increases, the complexity of the neural network to obtain the theoretical optimality increases rapidly.
For instance, consider a problem of estimating the Besov functions f 1 , f 2 , f 3 and f 4 .
Assume the
Gaussian mixture prior distribution for each parameter as in Example 2,
[Equation]
In Table 1 and Table
[Table]
the hyperparameters (model parameters) to estimate each function were presented.
These values are calculated numerically and may vary depending on the computer environment.
Its speed is affected by the complexity of the true regression function.
As can be seen from the tables, the smaller variance is almost zero.
Considering the numerical precision, it is indistinguishable from spike-and-slab prior [Ishwaran and Rao, 2005].
[Table] [Table]
In practice
[Table]
even the smaller models can approximate the true function with a sufﬁcient precision.
We present the numerical experiments with the four functions in Appendix D.
Conclusions
In this paper
[Table]
we justify the theoretical properties of neural networks that are currently widely used.
Speciﬁcally, we focused on the asymptotic properties of the Bayesian neural networks and extend results in Polson and Roˇcková [2018] to the Besov space from the Hölder space.
Theorem 1 shows that the Bayesian neural network with a spike-and-slab prior has consistency posterior with a nearly minimax rate at Besov functions with known smoothness.
Theorem 3 extends the result to the shrinkage prior.
According to these theorems, the Bayesian neural networks have posterior consistency and (nearly) optimal rates by assuming proper prior.
We found there are three sufﬁcient conditions to achieve posterior consistency, (1) enough mass around zero, (2) thick tail to sample true function (3) while not too much.
Theorem 2 shows that even if the smoothness of the function is not known, the same rate can be obtained by considering a prior on the architecture of the neural networks.
The results in this paper are an extension of the results of Suzuki [2018] on the Bayesian neural networks also.
We suggest a practical method that can achieve the optimal rate while providing the uncertainty of the prediction.
Note they showed the theoretical properties of the neural networks in the Besov space via nonparametric regression based on B-spline.
In other words, similar results can be obtained using the Bayesian LARK B-spline model [Park et al., 2021].
In future works, performance comparison experiments with other models will also be of great help in establishing the properties of the Bayesian neural network.
Empirically, we know that there are ﬁelds in which decision-tree-based models such as random forest or neural network models perform better than traditional statistical models that ﬁnd smooth functional relationships such as kernel regression.
For these data, it is natural that there would be a non-smooth relationship between the explanatory variable and the response variable.
For this reason, it is essential to compare Bayesian neural network models and statistical models for future studies.
We obtained results under the limited assumption, setting the networks fully connected and using the ReLU activation function.
Also, as mentioned in Section 4, the results are not enough practical to be used.
In addition, we do not suggest a result for the adaptive version of Theorem 3 yet.
It is a necessary future study to propose a general shrinkage prior distribution that satisﬁes all conditions even when the parameters of the network are varying.
The results of this study are at a basic level compared to the needs for the Bayesian neural networks in the industrial ﬁeld, such as visual data analysis and natural language processing.
It is important to research the theoretical properties of the
Bayesian neural networks used in the practical area through ongoing studies.
Acknowledgments
Kyeongwon Lee and Jaeyong Lee were supported by the National Research Foundation of
Korea (NRF) grant funded by the Korea government(MSIT)
(No.
2018R1A2A3074973 and 2020R1A4A1018207).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3 and Appendix C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See supplementary material.
(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Posterior consistency
Lemma 1 (Lemma 2 in Ghosal and Van Der Vaart [2007]) .
Suppose that observation X ( n ) =
( X 1 , · · · , X n ) of independent observations
X i .
Assume that the distribution P θ,i of the i th
component X i has a density p θ,i relative to a σ -ﬁnite measure µ i .
Let P ( n ) θ = N ni
=1 P θ,i be the product measures and average Hellinger distance
[Equation]
Then there exists test ϕ n such that P ( n ) θ 0 ϕ n ≤ e − 12 nd 2 n ( θ 0 ,θ 1 ) and P ( n ) θ (1 − ϕ n ) ≤ e − 12 nd 2 n ( θ 0 ,θ 1 ) for all θ ∈ Θ such that d n ( θ, θ 1 ) ≤ 118 d n ( θ 0 , θ 1 ) .
Proof.
See Ghosal and Van Der Vaart [2007].
Let deﬁne divergences
[Equation]
[Equation] [Equation]
27) and set
[Equation]
Here, the C k is the constant satisfying
[Equation]
for k ≥ 2 .
The existence of C k is guaranteed by Marcinkiewiz-Zygmund inequality.
The following lemma gives a sufﬁcient condition for obtaining posterior consistency.
Lemma 2 (Theorem 4 of Ghosal and Van Der Vaart [2007]) .
Let P ( n ) θ be product measures and d n ( θ 0 ,
[Equation]
) =
[Equation]
P
[Equation]
=
R ( √ p θ 0 ,i − √ p θ,i ) 2 dµ i .
Suppose that for a sequence ϵ n → 0 such that nϵ 2 n is bounded away from zero, some k > 1 , all sufﬁciently large j and sets Θ n ⊂ Θ which satisﬁes following conditions:
[Equation]
Then P ( n ) θ
(Π( θ : d n ( θ, θ 0 ) ≥ M n ϵ n | D n ) → 0 for any sequence M n → ∞ .
Proof.
See Ghosal and Van Der Vaart [2007].
In Ghosal and Van Der Vaart [2007], they mentioned that it can be replaced by any other distance d n for which the conclusion of Lemma 1 holds.
Moreover, for k = 2 , Lemma 2 work with the smaller neighborhood
[Equation]
instead of B ∗ n ( θ 0 , ϵ n ; k ) .
The following is an adapted version of Lemma 2, which was introduced in Polson and Roˇcková [2018].
Lemma 3.
Assume model (1).
Suppose that F is uniformly bounded.
Let
[Equation]
If there exist C > 2 /σ 2 and F n ⊂ F such that
[Equation]
for any ϵ n → 0 , nϵ 2 n
[Equation] [Equation]
in P ( n ) f 0 -probability as n → ∞ for any M n → ∞ .
Proof.
See appendix C.1.
B Consistency of neural network
The upper bound of the covering number of the neural network space is given by as follows.
Lemma 4 (Lemma 3 in Suzuki [2018]) .
∀ ϵ > 0 ,
[Equation] [Equation]
for W, L ≥ 3 .
Proof.
See appendix C.2.
For L, W, S ∈ N and B, a > 0 , deﬁne a function space
[Equation]
, (36) where
[Equation]
o
[Equation]
Lemma 5.
∀ ϵ ≥ 2 aL ( B ∨ 1)
L −
[Equation]
for W, L ≥ 3 .
Proof.
See appendix C.3.
The following lemma shows that for any function f 0 in the Besov space, there exist neural networks which is closed enough to f 0 .
Lemma 6 (Proposition 1 in Suzuki [2018]) .
Suppose that 0 < p, q, r ≤ ∞ , ω :=
d (1 /p − 1 /r ) + < s < ∞ and ν =
( s − ω ) / (2 ω ) .
Assume that N ∈ N is sufﬁciently large and m ∈ N satisﬁes 0 < s < min { m, m − 1 + 1 /p } .
Let W
= 6 dm ( m + 2) + 2 d .
Then, sup
[Equation]
38) for
L =
[Equation]
(39) where U ( H ) is the unit ball of a quasi-Banach space H ,
c ( d,m )
= (cid:16) 1 + 2 de (2 e ) m √ m (cid:17)
−and τ ( N ) =
N − s/d − ( ν − 1 + d − 1 )( d/p − s )
+ (log N ) − 1 .
Proof.
See Suzuki [2018].
C Proof of theorems C.1 Proof of Lemma
Proof.
Let d n ( f 0 , f ) =
∥
f 0 − f ∥ n .
As in Ghosal and Van Der Vaart [2007], page 214, we may use the norm d n instead of the average Hellinger distance d n and
[Equation]
for all i = 1 , 2 , · · · , n .
Then,
[Equation] [Equation]
and it is enough to show that
[Equation]
43) for all sufﬁciently large j .
[Equation] [Equation]
and
[Equation]
) for all sufﬁciently large j .
C.2 Proof of Lemma
Proof.
Let denote f ∈ Φ( L, W, S, B ) as
[Equation]
ζ ◦
[Equation]
W
[Equation] [Equation]
b
[Equation] [Equation]
A
[Equation]
(46) for k = 2 , · · · , L , and A − L +1 ( f )( x ) =
A +1 ( f )( x ) = x .
Then, f
[Equation] [Equation]
( W ( k ) ·
[Equation]
( k )
[Equation]
A
[Equation]
( f )( x ) .
Since f ∈ Φ( L, W, S, B ) ,
[Equation]
[Equation]
for all x , where A j, : is the j -th row of the matrix A .
Similarly,
[Equation]
Fix ϵ > 0 and θ ∈ Θ( L, W, S, B ) .
For any θ ∗ ∈ Θ( L, W, S, B ) which satisﬁes ∥ θ − θ ∗ ∥ ∞ <
ϵ ,
[Equation]
B ∨ 1)
[Equation]
W +
[Equation]
Let s be the number of nonzero components of θ , s ≤ S .
Consider a subspace Θ θ ( L, W, S, B ) of the parameter space
Θ( L, W, S, B ) consists of parameters which have s nonzero component.
Choose θ 1 , · · · , θ M from each grid divided with length ϵ ∗ =
ϵ L ( B ∨ 1)
L − 1 ( W +1)
L over Θ θ ( L, W, S, B ) .
Then f θ ∈
M [ m = {
[Equation]
Note
T =
[Equation] [Equation]
[Equation]
1) L for W, L ≥ 3 , and the number of cases of choose s nonzero components are (cid:18)
[Equation]
Thus,
N ( ϵ,
[Equation]
( L,
[Equation] [Equation]
, B ) , ∥·∥
[Equation]
and we get desired results by taking the logarithm of both sides.
C.3 Proof of Lemma
Proof.
Let ˜ θ ( a ) =
( θ i I ( | θ i | > a )) .
Then (cid:13)(cid:13)(cid:13) θ − ˜ θ ( a ) (cid:13)(cid:13)(cid:13) ∞ ≤ a and ˜Θ( L, W, S, B, a ) =
n ˜ θ ( a ) : θ ∈ Θ( L, W, S, B, a ) o =
Θ( L, W, S, B ) .
As in the proof of Lemma 4, (cid:13)(cid:13)(cid:13)
f θ − f ˜ θ ( a ) (cid:13)(cid:13)(cid:13) L ∞ ≤ aL ( B ∨ 1) L − 1 ( W + 1) L ≤ ϵ/(51) for any f θ , f ˜ θ ( a ) ∈ Φ( L, W, S, B, a, F ) .
Let s ≤ S be the number of nonzero components of ˜ θ ( a ) .
Consider a subspace ˜Θ θ ( L, W, S, B, a ) of ˜Θ( L, W, S, B, a ) consisting of parameters which have s nonzero component.
Choose θ 1 , · · · , θ M from each grid divided with length ϵ ∗ =
ϵ/
L ( B ∨ 1) L − 1 ( W +1)
L over ˜Θ θ ( L, W, S, B, a ) .
Then
[Equation]
from triangular inequality.
We get desired results in a similar way as in the proof of Lemma 4.
C.4 Proof of Theorem
Proof.
Let F = Φ ∩ UB ( F ) and
[Equation]
From Lemma
[Equation]
it is enough to show that there exist a constant C > 2 /σ 2 and F n ⊂ F which satisfy
[Equation]
for sufﬁciently large n .
Let F n = Φ( L n , W n , S n , B n ) ∩UB ( F ) .
First, (c) is trivial from (16).
From { f ∈ F n : ∥ f ∥ L ∞ ≤ ϵ } ⊂ { f ∈ F n : ∥ f ∥ n ≤ ϵ } and Lemma 4,
[Equation]
for sufﬁciently large n .
Thus, (a) holds.
Here
[Equation]
the last inequality holds from
[Equation]
Next, from Lemma 6, there is a constant C 1 > 0 and ˆ f n = f ˆ θ ∈ F such that
[Equation]
for sufﬁciently large n .
Moreover, by the assumptions and the strong law of large numbers,
[Equation]
[Equation]
for sufﬁciently large n P ( n ) f 0 -almost surely.
Let ˆ γ and ˆ θ ˆ γ be index and value of nonzero components of ˆ θ respectively.
Let Θ(ˆ γ ; L n , W n , S n , B n ) be a subset of parameter space Θ( L n , W n , S n , B n ) consists of parameters which have nonzero components at ˆ γ only and F n (ˆ γ ) =
Φ(ˆ γ ; L n , W n , S n , B n ) ∩ UB ( F ) be an uniformly bounded neural network space generated by Θ(ˆ γ ; L n , W n , S n , B n ) .
Note
[Equation]
n ,
[Equation]
for sufﬁciently large n .
As in the proof of Lemma 4,
[Equation]
Thus,
[Equation]
for sufﬁciently large n .
C.5 Proof of Theorem
Proof.
Let F = Φ ∩ UB ( F ) .
From Lemma 3, it is enough to show that there exist a constant
C ′ > 2 /σ 2 and F n ⊂ F which satisfy
[Equation]
for sufﬁciently large n .
From (53), we can choose
[Equation]
[Equation]
Let ˜ N n = C N N n ,
[Equation]
A for sufﬁciently large C N > 0 and π N ( N ) be a density function of N .
First, show that (a) holds.
From Lemma 4,
[Equation]
and
[Equation]
≲ nϵ
[Equation]
n .
[Equation]
for sufﬁciently large n .
Next, show that (b) holds.
Note N n (log n ) 3 ≲ nϵ 2 n and L n
[Equation]
for N n , L n , W n , S n , B n in Theorem 1 and sufﬁciently large n .
Thus, there exists a constant
D > 0 such that
[Equation]
and
[Equation] [Equation]
holds for sufﬁciently large n .
(b) holds for C ′ =
max (cid:8) C + D, 1 + 2 /σ 2 (cid:9) .
From Π
[Equation]
and Chernoff bound , for any positive number t, Z 0 > 0 ,
[Equation]
[Equation]
Letting t = log Z 0 , we get
[Equation]
Thus,
[Equation]
67) for sufﬁciently large C N > 0 .
(c) holds.
C.6 Proof of Theorem
Proof.
Let F = Φ ∩ UB ( F ) .
From Lemma 3, it is enough to show that there exist a constant
C ′′ > 2 /σ 2 and F n ⊂ F which satisfy
[Equation]
for sufﬁciently large n .
Let F n = Φ( L n , W n , S n , B n , a n ) ∩UB
( F ) .
It is easy to show that (a) holds from Lemma 5 as in the proof of Theorem 1.
By the assumption,
[Equation]
o (cid:16)
[Equation]
[Equation]
o (cid:16)
[Equation]
nϵ
[Equation]
(cid:17) + exp (cid:18)
T n (cid:18)
[Equation]
for some 4 < K 1 < K and 4 < K 2 < K 0 .
Letting C ′′
=
(min { K 1 , K 2 }− 2) /σ 2 , (c) holds.
We use
Bernoulli’s inequality and a tail bound for binomial distribution in Arratia and Gordon [1989] for the second inequality.
Next, as in the proof of Theorem
[Equation]
there is a constant C 1 > 0 and ˆ f n = f ˆ θ ∈ F n such that
[Equation]
[Equation]
for sufﬁciently large n almost surely.
Let ˆ γ and ˆ θ ˆ γ be index and value of nonzero components of ˆ θ respectively.
Let
[Equation]
( L, W, S, B ) and ˜Θ(ˆ γ ; L n , W n , S n , B n , a n ) be a subset of parameter space ˜Θ ( L n , W n , S n , B n , a n ) consists of parameters which have nonzero components at ˆ γ only and F n (ˆ γ ) =
˜Φ(ˆ γ ; L n , W n , S n , B n , a n ) ∩
UB ( F ) be an uniformly bounded neural network space generated by ˜Θ(ˆ γ ; L n , W n , S n , B n , a n ) .
Note
[Equation]
for sufﬁciently large n .
As in the proof of Lemma 4,
[Equation]
where t n = ϵ n 4( W n +1)
Ln L n ( B n ∨ 1) Ln
[Equation]
Letting
[Equation]
D Numerical Examples D.1 Gaussian Mixture Prior
Consider a problem of estimating the Besov functions f 1 , f 2 , f 3 and f 4 .
Assume the Gaussian mixture prior distribution for each parameter as in Example 2.
We sampled n points from x ∼
U (0 , 1) and y from N (cid:0) f i ( x ) , σ 2 i (cid:1) for i = 1 , 2 , 3 and 4 .
We set σ 21 =
σ 22 = 0 .
01 2 and σ 23 = σ 24 = 0 .
As mentioned in Section 4, we used a smaller model that was weaker than the conditions of the theorem.
We ﬁx the depth L n = 5 , the width W n = 200 and set N n ← W n /W 0 of the model.
We consider the prior in Example 2 (Gaussian mixture BNN) and set the smaller variance σ 1 n ← max { 0 . 001 , σ 1 n } to avoid numerical precision issues.
We ﬁt the model using the NUTS algorithm [Hoffman et al., 2014] with the python Pyro [Bingham et al., 2019] and PyTorch [Paszke et al., 2019] packages.
Experiments were run on a GPU server with Nvidia GeForce GTX TITAN X and RTX 3090.
The code and instructions for the experiment are provided in the supplementary material.
Figure 2 shows the results.
Overall, as the number of the data n increase, the mean functions get closer to the true regression function.
D.2 Gaussian Prior
We performed the same experiment as before using Gaussian prior (Gaussian BNN).
The standard deviation of the Gaussian prior is the same as σ 2 n in the Gaussian mixture BNN
(see Example 1).
Figure 3 shows the results.
Overall, as the number of the data n increase, the mean functions get closer to the true regression function.
D.3 Comparison
We compare two model in terms of empirical errors (training error)
1 n P ni =1 |
y i − f ( x i ) | 2 and Lnorms (test error)
e f = ∥
f − f
∥
L 2 between true function f 0 and sampled function f .
We approxi-
[Figure] [Figure]
[Figure] [Figure]
[Figure]
y
[Figure]
[Figure] [Figure]
y
[Figure]
[Figure] [Figure]
1.0 x      y
[Table]
1.0 x    y
[Table]
1.0 x
[Table]
y
[Table]
1.0 x      y
[Table]
[Table] [Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
y
[Figure]
y
[Figure]
[Figure] [Figure]
1.0 x     y
[Table]
1.0 x        y
[Table]
1.0 x        y
[Table]
1.0 x      y
[Table]
y
[Table] [Table]
mate e f as ˆ e f using Riemann sum,
[Equation]
As in Table 3 and Table
[Table]
both models give similar results.
For the functions f 1 and f 2 , Gaussian mixture BNN shows better results for both errors.
For the functions f 3 and f 4 , Gaussian BNN gives slightly better results for the test error.
However, as shown in Figure 2 and Figure 3, the differences are negligible
[Table]
[Table] [Table]