metadata: authors: 0: author: Ley,Eduardo
1: author: Steel, Mark F.J.
count: World
docna: 0: docna: Mixtures of g-priors for Bayesian Model
            Averaging  with economic application
docty: Policy Research Working Paper
subtopic: Consumption,Fiscal & Monetary Policy,Economic Conditions and Volatility,Economic Stabilization,Energy and Mining,Inequality
teratopic: Culture and Development,Information and Communication Technologies,Science and Technology Development,Education
abstracts: cdata!: This paper examines the issue of
            variable selection in linear regression modeling, where
            there is a potentially large amount of possible covariates
            and economic theory offers insufficient guidance on how to
            select the appropriate subset. In this context, Bayesian
            Model Averaging presents a formal Bayesian solution to
            dealing with model uncertainty. The main interest here is
            the effect of the prior on the results, such as posterior
            inclusion probabilities of regressors and predictive
            performance. The authors combine a Binomial-Beta prior on
            model size with a g-prior on the coefficients of each model.
            In addition, they assign a hyperprior to g, as the choice of
            g has been found to have a large impact on the results. For
            the prior on g, they examine the Zellner-Siow prior and a
            class of Beta shrinkage priors, which covers most choices in
            the recent literature. The authors propose a benchmark Beta
            prior, inspired by earlier findings with fixed g, and show
            it leads to consistent model selection. Inference is
            conducted through a Markov chain Monte Carlo sampler over
            model space and g. The authors examine the performance of
            the various priors in the context of simulated and real
            data. For the latter, they consider two important
            applications in economics, namely cross-country growth
            regression and returns to schooling. Recommendations for
            applied users are provided.
repnme: repnme: Mixtures of g-priors for Bayesian Model
            Averaging  with economic application
display_title: Mixtures of g-priors for Bayesian
            Model Averaging with economic application
listing_relative_url: /research/2011/07/14643155/mixtures-g-priors-bayesian-model-averaging-economic-application
topicv3: Macroeconomic Vulnerability and Debt
            m1326057 1104,Poverty m1326687 695,Macroeconomic Stability and Growth
            m1326923 1088,Macroeconomic and Structural Policies
            m1327380 439,Iron m1503514 2931
publishtoextweb_dt: 2011-07-01T00:00:00Z
historic_topic: Culture and Development,Information and Communication Technologies,Science and Technology Development,Education
no_of_pages: 35
author_keys: 0: author_key: 913182
model_extractions: <mark>WPS5732</mark> Policy Research Working Paper 5732 Mixtures of g-priors for Bayesian Model Averaging with Economic Application _Eduardo Ley Mark F.J. Steel_ The World Bank Poverty Reduction and Economic Management Network Economic Policy & Debt Department July 2011
Policy Research Working Paper 5732

# **Abstract**

This paper examines the issue of variable selection in linear regression modeling, where there is a potentially large amount of possible covariates and economic theory offers insufficient guidance on how to select the appropriate subset. In this context, Bayesian Model Averaging presents a formal Bayesian solution to dealing with model uncertainty. The main interest here is the effect of the prior on the results, such as posterior inclusion probabilities of regressors and predictive performance. The authors combine a Binomial-Beta prior on model size with a g-prior on the coefficients of each model. In addition, they assign a hyperprior to g, as the choice of g has been found to have a large impact on the results. For the prior on g, they examine the Zellner-Siow prior and a class of Beta shrinkage priors, which covers most choices in the recent literature. The authors propose a benchmark Beta prior, inspired by earlier findings with fixed g, and show it leads to consistent model selection. Inference is conducted through a Markov chain Monte Carlo sampler over model space and g. The authors examine the performance of the various priors in the context of simulated and real data. For the latter, they consider two important applications in economics, namely cross-country growth regression and returns to schooling. Recommendations for applied users are provided.

This paper is a product of the Economic Policy & Debt Department, Poverty Reduction and Economic Management Network. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http:// econ.worldbank.org. The authors may be contacted at M.F.Steel@stats.warwick.ac.uk and eley@worldbank.org .

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Mixtures of** **_g_ -priors for Bayesian Model Averaging with Economic Application**

Eduardo Ley Mark F.J. Steel The World Bank University of Warwick

## **JEL classification:** C11, O47

**Keywords:** Consistency; Model uncertainty; Posterior odds; Prediction; Robustness
# **1. Introduction**

We focus on problems of variable selection where we have a potentially large amount of covariates in a linear regression context, and the relevant theory does not offer enough guidance on how to select the “appropriate” subset, based on a sample of small or moderate size _n_ . This problem naturally occurs in various applications in economics, such as cross-country growth regression (Brock _et al._ , 2003) or estimating the returns to schooling (Tobias and Li, 2004). Examples of both of these areas of application will be given. As they are quite different in terms of the ratio of observations to potential regressors, _n/k_ , many other econometric scenarios will be covered as intermediate cases.

The use of Bayesian Model Averaging (BMA) provides a natural solution to model uncertainty, based on formal probabilistic reasoning, and it has been shown to lead to better predictions than simply selecting and using one model. See Raftery _et al._ (1997), Hoeting _et al._ (1999) and Fern ́andez _et al._ (2001a) for discussions of BMA in linear regression.

We can view the problem of variable selection in regression models as one of inducing sparsity or parsimony and there are two main approaches in the Bayesian literature. One is through the use of shrinkage priors, which goes back to Strawderman (1971) in the context of estimating a multivariate normal mean. These are absolutely continuous priors on all regression coefficients, but are such that some of the regression coefficients will be “close” to zero in the posterior, inducing sparsity (although strictly speaking zero has prior Lebesgue measure zero). The second approach is the one adopted here, where we assign prior point mass at zero for each of the regression coefficients, thus allowing for formal exclusion of covariates and we have to deal with many models that need comparing, which we will typically average over. This was coined the “spike-and-slab” approach in Mitchell and Beauchamp (1988). Even though we use the second approach, some ideas of “shrinkage” will be borrowed from the first literature, appropriately adapted to suit our needs.

In previous studies of variable selection in linear regression using _g_ -priors, it has been noted that the choice of _g_ is crucial for the behaviour of BMA procedures. In addition, the prior on the model space is an important element of the model, particularly in the way it penalizes larger models. If a priori each covariate is included independently with probability _θ_ in the model, the interaction between _θ_ and _g_ was explored in some detail in Ley and Steel (2009). They recommend the use of a hierarchical prior on _θ_ as a way to make the analysis more robust with respect to prior assumptions on the model space.

In this paper we go one step further and the hierarchical Bayesian model explored here has a hyperprior on _θ_ (which leads to an integral to compute prior model probabilities, which can fortunately be solved analytically) and a hyperprior on _g_ , which leads to an integral for the marginal likelihood that is solved by adding _g_ into the MCMC procedure by an extra Metropolis-within-Gibbs step.

There have been a number of recent proposals for prior distributions to use on _g_ ; the paper reviews these approaches and compares them in a common framework of priors that

> We are grateful for constructive comments from two referees, Martin Feldkircher and participants of the first European Seminar on Bayesian Econometrics held at the Erasmus University, Rotterdam (November 4–5, 2010).

1
induce a Beta distribution on the shrinkage factor (corresponding to _g_ or to _g/n_ ). The one prior that does not fit in this setting is the prior proposed by Zellner and Siow (1980). Based on earlier recommendations for fixed values of _g_ , we propose a benchmark Beta class of priors, and investigate its properties. An added advantage of using random _g_ is that the information paradox of Liang _et al._ (2008) can be avoided. In addition, we want the priors to lead to consistency in the sense of Fern ́andez _et al._ (2001a). We also discuss and propose an estimation method for the marginal information that the sample provides on _g_ . This leads quite naturally to estimating the Bayes factors between models with different priors on _g_ or different fixed values of _g_ . The effect of the proposed hyperpriors on both _θ_ and _g_ on posterior model probabilities is examined in some detail through the implied penalties for model complexity and lack of fit. We investigate the behaviour of the various priors in BMA with simulated data and various different sets of real data relating to economic applications; two sets of macroeconomic growth data and one data set regarding returns to eduction. We focus mostly on posterior probability on the model that generated the data and the rate of consistency for the simulated data, while we assess prediction performance and compute Bayes factors between priors for the real data. On the basis of both theoretical properties and empirical performance we provide recommendations for the applied user.

Section 2 introduces the Bayesian model, whereas Section 3 discusses the hyperpriors on _g_ . The information in the sample regarding _g_ and Bayes factors between model with different priors on _g_ are examined in Section 4. Section 5 examines the induced penalties for complexity and lack of fit, while Section 6 briefly mentions computational issues. Applications to simulated and real data follow in Sections 7 and 8, respectively. Finally, conclusions and recommendations are given in Section 9.

# **2. The Bayesian Model**

We adopt a Normal linear regression model for _n_ observations, grouped in a vector _y_ , using an intercept, _α_ , and explanatory variables from a set of _k_ possible regressors in _Z_ . We allow for any subset of the (standardized) variables in _Z_ to appear in the model. This results in a model space of 2<sup>_k_</sup> possible models, which will thus be characterized by the selection of regressors. We call model _Mj_ the model with the 0 _≤ kj ≤ k_ regressors grouped in _Zj_ , leading to where _ιn_ is a vector of _n_ ones, _βj ∈R_<sup>_kj_</sup> groups the relevant regression coefficients and _σ ∈R_ + is a scale parameter.

For the parameters in a given model _Mj_ , we follow Fern ́andez _et al._ (2001a) and adopt a combination of a “non-informative” improper prior on the common intercept and scale and a so-called _g_ -prior (Zellner, 1986)<sup>1</sup> on the regression coefficients with prior density

> 1 There is one difference with respect to the notation in Fern ́andez _et al._ (2001a); in line with most of the literature, in this paper _g_ denotes a variance factor rather than a precision factor.

2
where _fN_<sup>_q_(</sup><sup>_w | m, V_)denotesthedensityfunctionofa</sup><sup>_q_-dimensionalNormaldistribution</sup> on _w_ with mean _m_ and covariance matrix _V_ . The regression coefficients not appearing in _Mj_ are exactly zero, represented by a prior point mass at zero (this is the “spike-and-slab” idea). Of course, we need a proper prior on _βj_ in (2), as an improper prior would not lead to meaningful Bayes factors. The so-called “benchmark” prior structure in (2), sometimes with small variations, is shared by most of the recent literature on covariate selection in linear models—see, _e.g._ , Clyde and George (2004) for a survey. As mentioned in Berger and Pericchi (1996), there is a potential danger in assigning the same improper prior on common parameters, but the prior structure used here can be motivated through invariance arguments (see Berger, Pericchi and Varshavsky, 1998) and is generally accepted to be a reasonable choice.

One advantage of this prior is that we only need to choose a single scalar parameter _g_ . A large fraction of the literature in this area has dealt with this choice of _g_ , and it is clear that posterior and predictive inference critically depends on _g_ ; see Ley and Steel (2009) and Eicher _et al._ (2011) for examples of this in the context of growth regressions. Popular values for _g_ in the literature are to take _g_ = _n_ , which corresponds to assigning the same amount of information to the conditional prior of _β_ as is contained in one observation—the so-called “unit information prior” of Kass and Wasserman (1995), also favoured in Eicher _et al._ (2011)—or to take _g_ = _k_<sup>2</sup> as suggested by the Risk Inflation Citerion (RIC) of Foster and George (1994). Fern ́andez _et al._ (2001a) recommend the “benchmark” choice of _g_ = max _{n, k_<sup>2</sup> _}_ . As a natural Bayesian response to the uncertainty regarding the choice of _g_ , we will put a hyperprior on _g_ , allowing for the data to influence the inference on the now random _g_ . This makes the analysis more robust with respect to the assumptions on _g_ and has also been used, among others, in Liang _et al._ (2008), Bottolo and Richardson (2008), Cui and George (2008), and Feldkircher and Zeugner (2009).

Thus, denoting by _p_ ( _g | Mj_ ) the prior for _g >_ 0 (which could depend on _Mj_ ), we have the following prior for all parameters in _Mj_

In other words, we can interpret the prior on _g_ as a way of extending the normal prior in (2) to a scale mixture of normals, which has substantially more flexible tails—see Andrews and Mallows (1974) and Fern ́andez and Steel (2000).

Now we can write the marginal likelihood as the following (with a proportionality constant that is the same for all models, including the null model): where _Rj_<sup>2istheusualcoefficientofdeterminationformodel</sup><sup>_Mj_;</sup><sup>_R_</sup> _j_<sup>2=1</sup><sup>_−y′QX_</sup> _j_<sup>_y/_(</sup><sup>_y −_</sup> ̄ ̄ _yιn_ )<sup>_′_</sup> ( _y − yιn_ ), where _QA_ = [ _I − A_ ( _A_<sup>_′_</sup> _A_ )<sup>_−_1</sup> _A_<sup>_′_</sup> ], and _Xj_ = ( _ιn Zj_ ) is the design matrix of model _Mj_ , which we assume to be of full column rank. This marginalised likelihood (4) is simply the sampling density integrated out with the prior, and is of critical importance as the ratio of marginal likelihoods of any two models is the Bayes factor between these models.

3
Note that the prior on _g_ needs to be proper, as the null model does not involve _g_ and an improper prior on _g_ would thus lead to arbitrary Bayes factors versus the null model. An important quantity in evaluating the properties of priors on _g_ will be the shrinkage factor, which is _δ_ = _g/_ (1 + _g_ ). The posterior mean of _βj_ for each given model will be the OLS solution times this shrinkage factor.

For constructing the prior over model space, we assume that each potential regressor is independently included in the model with probability _θ_ . As in Brown _et al._ (1998), Nott and Kohn (2005) and Ley and Steel (2009), we also consider putting a hyperprior on _θ_ , which is then assigned a Beta distribution. Ley and Steel (2009) illustrate that such a random _θ_ approach renders the analysis much more robust with respect to prior assumptions. They advocate the parameterization where _m_ is then the prior mean model size. The resulting hierarchical prior over model space is less restrictive than the one with fixed _θ_ and the choice of _m_ is shown in Ley and Steel (2009) not to matter too much in practice. The prior on model size induced by (5) is a Binomial-Beta distribution—see, _e.g._ , Bernardo and Smith (1994, p. 117).

# **3. Hyperpriors for** _g_

# _3.1. Zellner-Siow Prior_

Inspired by Jeffreys’ (1961) arguments for using Cauchy priors in model comparison problems related to a univariate normal mean, Zellner and Siow (1980) proposed the use of multivariate Cauchy priors in regression problems. As is well known, and mentioned in Liang _et al._ (2008), Student- _t_ form priors (like the Cauchy) can easily be expressed as scale mixtures of normals and thus, the Zellner-Siow prior corresponds to a particular choice of _p_ ( _g_ ) in (3). In particular, Zellner and Siow implicitly propose an inverted Gamma distribution with parameters<sup><u>1</u></sup> 2<sup>and</sup><sup>_<u>n</u>_</sup> 2<sup>,leadingto</sup> This implies the following distribution for the shrinkage factor _δ_ = _g/_ (1 + _g_ ): 4
# _3.2. Beta Shrinkage Priors_

A relatively large number of priors in the literature, in fact, imply a Beta prior distribution for the shrinkage factor _δ_ . A Beta( _b, c_ ) prior on the shrinkage factor induces the following prior on _g_ : which is called an inverted Beta distribution in Zellner (1971, p. 375) and is also known as a Gamma-Gamma distribution (Bernardo and Smith, 1994, p. 120) in the statistics literature. This has the following properties: and has a mode equal to ( _b −_ 1) _/_ ( _c_ + 1) provided _b >_ 1. This inverted Beta prior on _g_ leads to the following prior on the regression coefficients, marginalised over _g_ : where Ψ denotes the confluent hypergeometric function (Gradshteyn and Ryzhik, 1994, p. 1085). Note that these Beta shrinkage priors have a density in the right tail that behaves like _g_<sup>_−_(1+</sup><sup>_c_)</sup> , thus leading to very fat tails for small values of _c_ .

The so-called hyper- _g_ prior, proposed by Liang _et al._ (2008), corresponds to _b_ = 1 and _<u>a</u> c_ = 2<sup>_−_1;with</sup><sup>_a>_2toensureaproperprior.</sup> This class includes priors used by Strawderman (1971) for the normal means problem. Cui and George (2008) propose to use _a_ = 4 in the context of model selection with known _σ_ . Bottolo and Richardson (2008) adopt a hyper- _g_ prior with _a_ = 2, but make it proper by truncating the right tail at max _{n, k_<sup>2</sup> _}_ , which is the benchmark value for _g_ proposed by Fern ́andez _et al._ (2001a). Feldkircher and Zeugner (2009) propose a hyper- _g_ prior with a value of _a_ that leads to the same mean shrinkage factor as the unit information prior or the RIC prior. These hyper- _g_ priors have a finite nonzero limit as _g →_ 0.

Another prior corresponding to a Beta shrinkage prior is the horseshoe prior of Carvalho _et al._ (2010), where both _b_ and _c_ are taken to be<sup><u>1</u></sup> 2<sup>.TheshrinkagefactorthushasaU-</sup> shaped prior where the spike around zero corresponds to very strong shrinkage and induces zero coefficients in their shrinkage prior framework, and the spike around one describes the signal.

Whereas the horseshoe prior was explicitly developed for a different setting (shrinkage priors rather than our spike-and-slab framework), and the hyper- _g_ has roots in the shrinkage literature, the latter prior has been proposed explicitly for problems where we do have prior point masses to deal with formal exclusion of regressors—in Liang _et al._ (2008), Cui and George (2008), and Bottolo and Richardson (2008). However, in our spike-and-slab case, we do not need to rely on shrinkage to exclude regressors: this is formally allowed for 5
by the prior on the model space. Thus, we would expect that we don’t really need such a large prior mass around small values of the shrinkage factor _δ_ . Nevertheless, the horseshoe prior has an accumulation of mass towards zero for both _δ_ and _g_ . In terms of _g_ , both the horseshoe prior and the hyper- _g_ prior always decrease in _g_ , and the horseshoe prior even has an asymptote at zero. A graphical illustration of the various priors is provided in Figure 2, discussed later.

One way to adapt these priors (partly) motivated by the shrinkage literature to our current setting, is to realize that typically the shrinkage priors are not used in a _g_ -prior (where the conditional covariance of the regression coefficients is proportional to the inverse information matrix) but in a ridge-type prior structure (where this conditional covariance is proportional to the identity matrix). In the _g_ -prior framework used here, we need to account for the fact that information accrues with sample size and the inverse information matrix is of order 1 _/n_ . Thus, a fair comparison with a shrinkage ridge-type prior setup would perhaps be to apply the shrinkage prior to _g/n_ rather than _g_ . Equivalently, we then use a Beta( _b, c_ ) prior on _g/_ ( _n_ + _g_ ). Starting from the hyper- _g_ prior, this leads directly to the hyper- _g/n_ prior of Liang _et al._ (2008). Similarly, we shall denote the prior thus derived from the horseshoe prior by horseshoe _/n_ prior. Of course, this does not change the fact that these priors are always decreasing in _g_ , but it does move some mass towards larger values of _g_ , and makes the right tail of the horseshoe/ _n_ prior on _δ_ much thicker than the left (see Figure 2). In addition, it solves an inconsistency problem. As shown in Liang _et al._ (2008), the Zellner-Siow prior and the hyper- _g/n_ prior are consistent in the sense described in Subsection 3.4. This is also the case for the horseshoe/ _n_ prior, but not for any prior on _g_ that does not depend on _n_ . Subsection 3.4 presents more results in this respect.

Maruyama and George (2010) propose to choose _b_ + _c_ = ( _n − kj −_ 1) _/_ 2 and _c <_ 1 _/_ 2 in (6). This choice is motivated by the fact that the integral in (4) then has a simple analytic solution and thus Bayes factors can be computed as easily as in the case with fixed _g_ . In particular, we obtain (up to a common proportionality constant) whenever _c <_ ( _n − kj −_ 1) _/_ 2. As a default value, Maruyama and George (2010) propose to take _c_ = 1 _/_ 4. However, note that this choice implies that the prior on _g_ depends on the model we are considering (through the model size). Formally, this is allowed, but it may make it slightly harder to interpret the role of _g_ . Maruyama and George (2010) show that consistency holds with this prior.

Extending the prior for robust estimation of Berger (1985), Forte _et al._ (2010) effectively propose the use of a truncated version of (7) with _b_ = 1 and _c_ = 1 _/_ 2, while truncating _g_ to be greater than ( _n_ + 1) _/_ ( _kj_ + 3) _−_ 1. They prove consistency of the resulting model choice procedure and provide a closed-form expression for the Bayes factor. Through the truncation point, the prior on _g_ , again, becomes model-specific.

From fixed _g_ analyses with growth data (Eicher _et al._ , 2011; Ley and Steel, 2009) it seems that rather large values of _g_ could be preferable. In particular, values like _g_ = _n_ or even 6
_g_ = _k_<sup>2</sup> are the most used in this literature. With the hyper- _g_ we can only assign large prior mass to regions with high _g_ by taking _a_ very close to 2, which gives us a fat tail for _g_ , but does not really change the shape of the prior. In order to propose an alternative class of priors in the next subsection, we initially focus on the shrinkage factor, _δ_ .

# _3.3. A Benchmark Beta Prior_

If we start from a Beta( _b, c_ ) prior on the shrinkage factor _δ_ = _g/_ (1 + _g_ ), we can base our proposal on ensuring that prior moments are reasonable (they always exist, as we are dealing with a finite support). In particular, let us set the mean shrinkage factor equal to the one that corresponds to the Fern ́andez _et al._ (2001a) recommendation _g_ = max _{n, k_<sup>2</sup> _}_ . This fixes one parameter as a function of the other, as then _b/c_ = max _{n, k_<sup>2</sup> _}_ . The second parameter can then be chosen by considering the spread around this mean: where is the chosen prior mean of the shrinkage factor, and thus larger _c_ corresponds to a tighter prior around this mean.

For example, for the cross-country growth data used in Fern ́andez _et al._ (2001b) with _k_ = 41 and _n_ = 72, the value _c_ = 1 would correspond to a prior standard deviation of the shrinkage factor of 0.0006 (approximately equal to 1 _− d_ ) and the corresponding _b_ = 1681. For _c_ = 1 _/_ 1000 we obtain a prior standard deviation of 0.015 (approximately 25 times 1 _− d_ ) and this corresponds to _b_ = 1 _._ 681. For likely choices of _c_ (say, _c_ = 1 _,_ 0 _._ 1 _,_ 0 _._ 01) and max _{n, k_<sup>2</sup> _} >_ 100 (as for typical datasets in econometrics) we have no prior moments for _g_ , but we do have an interior mode equal to which is approximately max _{n, k_<sup>2</sup> _}/_ 2 for _c_ = 1 and is a lot smaller for the smaller values of _c_ .

Figure 1 shows three prior densities (expressed in logs to make matters easier to visualize) of _g_ and _δ_ from this class of benchmark priors. The figure illustrates the effect of _c_ : For _c_ = 1 the density is most concentrated while for the smallest _c_ the prior assigns a lot of the mass to the far right tail of _g_ (for example, _P_ ( _g >_ 5 _,_ 000) = 0 _._ 95 for _c_ = 0 _._ 01). Notably, all priors for _g_ have an interior mode. As _c →∞_ the benchmark Beta prior tends in the limit to the case with fixed _g_ = max _{n, k_<sup>2</sup> _}_ .

As a special case of the benchmark Beta class, we can get a single hyper- _g_ prior by choosing _c_ = 1 _/_ max _{n, k_<sup>2</sup> _}_ . This would lead to a hyper- _g_ prior with 7
Benchmark priors for _c_ = 1 (solid), _c_ = 0 _._ 1 (dashed), and _c_ = 0 _._ 01 (dotted) <!-- Start of picture text --> �8<br>c � 1 20<br>�9<br>�10 c � 0.1 15<br>�11<br>�12 c � 0.01 10<br>c � 1<br>�13 c � 0.1<br>5<br>c � 0.01<br>�14<br>�15 0<br>0 500 1000 1500 2000 2500 0.9990 0.9992 0.9994 0.9996 0.9998 1.0000<br>g ∆� g ��1� g �<br>�� gp ∆�� p<br>log log<br><!-- End of picture text -->

**Fig. 1.** Beta Benchmark Priors for _g_ and _δ_ ( _n_ = 72, _k_ = 41) Vertical lines correspond to _g_ = max _{n, k_<sup>2</sup> _}_

In fact, this would effectively correspond to taking the prior setting with the smallest _a_ of the two proposals in Feldkircher and Zeugner (2009). Note that even though the mean shrinkage factor is the same, this is a very different prior, where we achieve the mean shrinkage by a very small _c_ (with fixed _b_ = 1) rather than the benchmark choice of a very large _b_ (with fixed _c_ , chosen to give a reasonable prior spread of _δ_ ).

Figure 2 helps us understand the differences between the various priors. It displays the log densities of _g_ and _δ_ for all the priors discussed here (using _n_ = 72 and _k_ = 41 as in the first growth dataset in Section 7). Interior modes for _g_ occur for the Zellner-Siow, MaruyamaGeorge, Forte _et al._<sup>2</sup> and benchmark priors. All other priors tend to a positive constant or infinity<sup>3</sup> as _g →_ 0. In order to get the same mean shrinkage factor as the benchmark Beta prior, the Feldkircher-Zeugner priors thus need to decrease very rapidly with _g_ in order to compensate for the mass close to _g_ = 0 and then have a very fat far right tail (almost of the order 1 _/g_ ). As a consequence, the prior probability assigned by the Feldkircher-Zeugner priors to _δ > d_ (with _d_ defined as in the beginning of this subsection) is large: over 0.9 when _a_ = 2 + 2 _/n_ and 0.9986 for _a_ = 2 + 2 _/k_<sup>2</sup> . For the benchmark prior this probability decreases from 0.96 to 0.63 as _c_ ranges from 0.01 to 1. Particularly small prior probabilities for large values of _δ_ are associated with the Bottolo-Richardson prior ( _P_ ( _δ > d_ ) = 0 due to the truncation) and the hyper- _g_ ( _a_ = 3 _,_ 4) and horseshoe priors, which have relatively thin right tails for _g_ . The Zellner-Siow prior for _g_ has the same right tail behaviour as the hyper- _g_ with _a_ = 3 and the horseshoe prior and also leads to relatively small mass on _δ > d_ (0.16). The Maruyama-George and Forte _et al._ priors are intermediate cases in this respect. Remember also that the latter two priors depend on the model _Mj_ through its model size, _kj_ . Thus, denoting model size by _W_ , we compute the marginal prior for _g_ as

> 2 Note also that the Forte _et al._ prior for _g_ is truncated away from zero, in that _g >_ (( _n_ + 1) _/_ ( _k_ + 3)) _−_ 1. In addition, this prior, computed as in (8), is multimodal, with the truncation leading to a saw-tooth effect for small and moderate values of _g_ (the last discontinuity is at _g_ = ( _n −_ 2) _/_ 3).

> 3 For the horseshoe and horseshoe/ _n_ priors.

8
follows: where _P_ ( _W_ = _w_ ) is the probability mass function of the Binomial( _k, θ_ ) (for fixed _θ_ ) or Binomial-Beta distribution (for random _θ_ , and as given in Ley and Steel, 2009). Figure 2 presents the marginal Maruyama-George and Forte _et al._ priors for the random _θ_ case with _m_ = 7.

With the exception of the hyper- _g_ with _a_ = 4 and the horseshoe priors, all priors concentrate their mass on very small amounts of shrinkage. This is appropriate in our setting, since we do need to shrink regression coefficients to zero in order to indicate that certain regressors are not important. Instead, we use point masses at zero to formally exclude regressors.

Table 1 summarizes the definition and some of the main properties of the different prior distributions. The names of the priors are as in the text, except for “F-Z,” which denotes the Feldkircher-Zeugner prior. The numbers refer to prior numbers as used later in the empirical sections. The column “truncated” indicates any truncation of the support for _g_ and the column “model spec.” indicates whether the prior is specific to any given model. In the column “mean” we report whether the prior mean exists (and if so, under which conditions). The availability of analytical expressions for the Bayes factors is indicated in “BF”, while the last column “cons.” refers to consistency as discussed in the next subsection. Finally, note that if the prior mean of _g_ does not exist, neither will the posterior mean. The reason for that is simply that the posterior equals the prior for the null model (since that model does not involve _g_ ). Thus, even if the null model has only a very small posterior probability, the latter is not zero and the overall posterior mean of _g_ will be

# _3.4. Consistency and Information Paradox_

As stressed in Liang _et al._ (2008), the limiting behaviour of the Bayes factors can be an interesting guideline for the choice of priors on _g_ . They mention the “information paradox,” which occurs if a model _Mj_ gets overwhelming data support (so that _Rj_<sup>2</sup><sup>_→_1)andifthen</sup> the Bayes factor of _Mj_ with respect to the null model does not go to infinity. It is clear from (4) that this Bayes factor would tend to � (1+ _g_ )<sup>(</sup><sup>_n−_1</sup><sup>_−kj_)</sup><sup>_/_2</sup> _p_ ( _g_ )d _g_ . Liang _et al._ (2008) show that the Zellner-Siow prior and the hyper- _g_ prior with _a ≤ n − kj_ + 1 avoid the paradox, whereas no fixed _g_ would do so. If we adopt the general Beta shrinkage prior, we obtain the result that the paradox is removed if and only if _c <_ ( _n −_ 1 _− kj_ ) _/_ 2, which exactly corresponds to the findings of Liang _et al._ (2008). Typical values of _c_ chosen in the benchmark Beta prior would certainly comply with this condition.

Consistency implies that all posterior mass tends to be allocated to the true model ( _i.e._ , the model that generated the data) if the latter is in the model space as the number of observations goes to infinity. This was introduced in Fern ́andez _et al._ (2001a) and is called “model selection consistency” in Liang _et al._ (2008). The latter paper remarks that the 9
<!-- Start of picture text --> H ; |<br>— t ] ,<br>I ;<br>-|i<br>1 L<br>-_—<br>VYSs ,ae . {| o | L<br>— wee | I<br>fo ——-— === LL || H<br>_ / ween —_is==] h|1 e<br>-IJ77<br>| | ee f l<br>- I q = eee<br>] Toe en ce cucececeenceeseseeeeeeeeeeee? |<br>6= (( +)<br>fz<br>L- f<br>bos, |<br>_ .<br>, |<br>oe, |<br>S _— — 4<br>~<br>= 4\ “Sees... L |<br>NC<br>SAL !<br>_<br>~— 2.1<br>LL ——— <<br>LL — —_-<br>7 eee ee eee |<br>b= (( +)<br>t BO pt oe "7 Tt I Ll a<br>;|1<br>| 1<br>| 1<br>_ t ps ; ;<br>E[ Sa. ~ ~~ 44<br>~ TS 4 _<br>~ =. 4 ~—<br>; 2<br>—_<br>r y ?<br>— t s ; !<br>L\1:<br>[ wee | ,<br>~ | oe”<br>B e tere ] cee eee eee eect<br>d= (( +)<br>| H'<br>| ' '<br>|:<br>| |<br>~ _ | |<br>LY S | |<br>~~ +“NS..~Stee ae ! ;|<br>: ~~~ we eeee eel. ores NS vl<br>eee SE<br>Ta .‘.ee eecoe ~ rid<br>d= /( +)<br><!-- End of picture text -->
**Table 1.** Summary of prior distributions on _g_

||||Bet|a shrinkage priors,<br>|_δ ∼_Be(_b, c_)<br>||||
|---|---|---|---|---|---|---|---|---|
|nr.|name|_b_|_c_|truncated|model spec.|mean|BF|cons.|
|1–3|Benchmark|_c_max_{n, k_<sup>2</sup>_}_<br>|_c_|no|no|_c >_1|no|yes|
|4|Maruyama-George|_n−kj−_1<br>2<br>_−c_|_<_1_/_2|no|yes|no|yes|yes|
|5|Bottolo-Richardson|1|0|_g <_max_{n, k_<sup>2</sup>_}_|no|yes|no|yes|
|6–7|Hyper-_g_|1|(_a/_2)_−_1|no|no|_a >_4|no|no|
|8|F-Z, _a_= 2 + 2_/n_|1|1_/n_|no|no|no|no|yes|
|9|F-Z, _a_= 2 + 2_/k_<sup>2</sup>|1|1_/k_<sup>2</sup>|no|no|no|no|no|
|10|Horseshoe|1_/_2|1_/_2|no|no|no|no|no|
|11|Forte _et al._|1|1_/_2|_g >_<br>_n_+1<br>_k_+3 <sup>_−_1</sup>|yes|no|yes|yes|
||||Beta s|_j _<br>hrinkage/_n_ priors,<br>|_g_<br>_n_+_g _<sup>_∼_Be(</sup><sup>_b, c_</sup><br>|<sup>)</sup>|||
|||_b_|_c_|truncated|model spec.|mean|BF|cons.|
|12|Hyper-_g/n_|1|(_a/_2)_−_1|no|no|_a >_4|no|yes|
|13|Horseshoe_/n_|1_/_2|1_/_2|no|no|no|no|yes|
||||Inve|rted Gamma prior<br>|, _g ∼_IG(_b, c_)<br>||||
|||_b_|_c_|truncated|model spec.|mean|BF|cons.|
|14|Zellner-Siow|1_/_2|_n/_2|no|no|no|no|yes|

Zellner-Siow and hyper- _g/n_ priors are consistent, but the hyper- _g_ prior is not consistent when the null model is the true model. In contrast, the benchmark Beta prior does lead to consistency in this case:

**Proposition 1.** _If data are generated by the null model and we adopt the prior for g in (7), the Bayes factor for any other model Mj versus the null model tends to_

_as the number of observations n tends to infinity. Thus, if we take the benchmark Beta prior settings in Subsection 3.3 (where b →∞ with n and c is fixed), we achieve consistency under the null model._ Actually, Proposition 1 can also be used to show that the hyper- _g_ proposal by Feldkircher and Zeugner (2009) where _a_ = 2 + _w_ ( _n_ ) where _w_ ( _n_ ) tends to zero with _n_ is consistent, as also mentioned in their paper. With this prior, the Bayes factor for any model versus the null model is _w_ ( _n_ ) _/_ ( _w_ ( _n_ ) + _kj_ ) and will thus tend to zero.

The Bottolo-Richardson prior has a hyper- _g_ form but depends on _n_ through the truncation and this can be shown to lead to consistency. However, the Bayes factor for any model _Mj_ versus the true null model is or the order 1 _/_ ln( _n_ ) and the rate of convergence is thus quite slow.

11
# **4. Sample information on** _g_ **and Bayes factors**

When we move from choosing a fixed value for _g_ towards the treatment of _g_ as a random quantity with a host of different possible priors, it makes sense to consider what exactly we can expect to learn from the sample about _g_ . Clearly, if _g_ is close to unidentified, we are not going to move away substantially from the prior and the choice of prior is going to be critical. Of course, that does not preclude inference with a proper prior on _g_ but it does mean we should choose the prior carefully.

The information that the sample provides marginally on _g_ can, in principle, be derived from the likelihood integrated with respect to the prior on model parameters in (2): from which we can write the likelihood marginalized with everything except for _g_ as A plot of _ly_ ( _g_ ) as a function of _g_ describes the marginal information that is present in the sample about _g_ . With the exception of the priors in Maruyama and George (2010) and Forte _et al._ (2010), the priors on model space do not depend on _g_ and easy analytical expressions for _P_ ( _Mj_ ) exist, in the case of fixed and random prior covariate inclusion probabilities (Ley and Steel, 2009). The problem in evaluating (10), however, is the huge amount of terms in the sum, which makes this seemingly simple calculation infeasible for _k >_ 20 or so.<sup>4</sup> A feasible way of calculating _ly_ ( _g_ ) is to simply start from Bayes rule to realise that where we have indicated dependence on the prior used for _g_ by a subscript _i_ . Note that _ly_ ( _g_ ) does not depend on the prior chosen for _g_ . For the evaluation of the sample information, the proportionality constant in (11) does not matter, as we are really only interested in the profile of the (unnormalized) integrated likelihood _ly_ ( _g_ ) as a function of _g_ .

The posterior density needs to be computed numerically on the basis of an MCMC chain (as described in the next subsection), but the simple equation in (11) completely characterizes the information about _g_ contained in the sample. In principle, it should give the same result (up to an additive constant) for any prior on _g_ , but as it is bound to be less precise when both densities tend to zero, an average of (11) will be computed over the

> 4 Note that the sum in (10) is weighted with the prior distribution on model space, so we can not simply approximate it by a sum over the (small subset of) models visited in the MCMC chain used for posterior model inference (see Section 5), but we would typically need complete enumeration.

12
different priors used on _g_ , where we discard the influence of a prior for values of _g_ where this prior has very small density values. To neutralise the effect of the different values of _ci_ in this average, we normalise the profiles by choosing _ci_ = _pi_ ( _g_ 0) _/pi_ ( _g_ 0 _|y_ ) for a central value of _g_ 0 for which all priors and posteriors have nonnegligible mass. The more peaked this information measure, the more information the sample contains about _g_ and the less important prior choice on _g_ becomes.

The expression in (11) is also reminiscent of the so-called Savage-Dickey density ratio (Verdinelli and Wasserman, 1995). Indeed, if we make explicit that the proportionality constant _ci_ = _ly_ (prior _i_ ) = � _ly_ ( _g_ ) _pi_ ( _g_ )d _g_ , then it becomes clear that the ratio _pi_ ( _g|y_ ) _/pi_ ( _g_ ) evaluated at a value _g_ 0 is the Bayes factor of the model with fixed _g_ 0 versus the model with prior _pi_ ( _g_ ).

Another thing that can be easily done is to compare the data support for different fixed values of _g_ given a choice of prior _pi_ ( _g_ ). In particular, if we consider two values, say _g_ 1 and _g_ 2, the Bayes factor is given by Values of _g_ for which the posterior density value is higher relative to the prior density are more strongly supported by the data, in line with intuition. Perhaps even more interesting is the direct comparison of different priors on _g_ , which can be done immediately through the Bayes factors. From (11) the Bayes factor for the model with prior _q_ versus the one with prior _m_ can be computed as where we choose a value _g_ 0 such that none of the density values on the right hand side are very small.<sup>5</sup> Note that the data will tend to support models with priors that take relatively high density values at _g_ 0 compared to the posterior. As the marginal likelihood is the likelihood _ly_ ( _g_ ) integrated out with the prior _pi_ ( _g_ ), it is not surprising that Bayes factors between models that differ in the prior on _g_ will favour those models that concentrate the prior weight on _g_ around values with very high likelihood support.

# **5. Induced Complexity and Lack of Fit Penalties**

In order to make the analysis more robust with respect to the prior assumptions, we have introduced hyperpriors on the inclusion probability of regressors, _θ_ in (5), and on _g_ . Let us now examine how this affects the posterior model probabilities through the penalties for model complexity and lack of fit. Posterior odds between models are the product of prior odds (from the prior over model space) and Bayes factors (from the marginal likelihood

> 5 The use of the basic identity as in (11) evaluated at fixed parameter values to estimate marginal likelihoods also underlies the proposal of Chib (1995).

13
for each model). Since the prior on model space is such that regressors are included independently with probability _θ_ and using the expression in (9), we have, for any two models and given _θ_ and _g_ In what follows, we shall consider each of the three factors in (13) separately, and focus on the case where _ki_ = _kj_ + 1 and _Ri_<sup>2=</sup><sup>_R_</sup> _j_<sup>2toisolatecomplexity(modelsize)penalties</sup> and the case where _ki_ = _kj_ and _Ri_<sup>2</sup><sup>_<R_</sup> _j_<sup>2tostudythepenaltyforlackoffit.Wedefine</sup> penalties as minus the logarithm of the corresponding odds factor,<sup>6</sup> so that if the factor contributing to the odds between two models of different complexities or levels of fit is one, the induced penalty is zero. The complexity penalty induced by the prior odds can be positive (in favour of the smaller _Mj_ ) or negative (favouring the larger model _Mi_ ), whereas the penalties induced by the Bayes factor are always positive. Values for the three penalties can immediately be compared as the penalties are simply added together to form the negative log posterior odds.

Starting with the prior odds, the first factor in (13), this induces a complexity penalty, say _cpM ∈R_ , for adding an extra regressor equal to For example, if we fix _θ_ to be 0.5, this is always zero and there is no penalty for model complexity in the prior over the model space. The use of a value of _θ <_ 0 _._ 5 reflects a prior penalty for model complexity. If we now use the random _θ_ in (5), this induces a distribution on _cpM_ which corresponds to an inverted Beta on exp( _−cpM_ ), and is given by where _m_ denotes the prior mean model size. Figure 3 plots the distribution of this prior complexity penalty factor for _k_ = 41 and two different choices of _m_ : _m_ = 7 (often used in the growth regression context and reflecting a preference for models of size around 7) and _m_ = 20 _._ 5 (the same prior mean model size as implied by the choice of a fixed _θ_ = 0 _._ 5 and reflecting a uniform prior distribution for model size, as explained in Ley and Steel, 2009). Vertical lines in the figure correspond to the fixed complexity penalties implied by fixing _θ_ leading to the same values of _m_ (given by _cpM_ = log[( _k − m_ ) _/m_ ]). Note that the densities corresponding to the rather different cases shown in Figure 3 have considerable overlap.

> 6 The use of the negative log odds ratio as a penalty function seems natural, as interchanging the two models is reflected by a sign change of the penalty and it ties in well with the terminology of classical information criteria, which, in some cases, correspond to the limits of log posterior odds, as shown in Fern ́andez _et al._ (2001a).

14
Even though they are centered around the value corresponding to fixed _θ_ with the same _m_ they allow for quite a bit of variation. In summary, choosing a random _θ_ prior makes this prior complexity penalty itself random, and thus allows for the data to influence this penalty.<sup>7</sup> This renders the analysis more robust with respect to the specification of mean model size _m_ (see Ley and Steel, 2009 for more details and results with some growth data sets).

<!-- Start of picture text --> 0.30<br>0.25<br>0.20<br>0.15<br>0.10<br>0.05<br>0.00<br>�10 �5 0 5 10<br>cpM<br>�<br>M<br>cp<br>�<br>p<br><!-- End of picture text -->

**Fig. 3.** Prior of complexity penalty _cpM k_ = 41, _m_ = 7 (solid) and _m_ = 20 _._ 5 (dashed)

A separate complexity penalty derives from the second factor in (13), which is part of the Bayes factor. This penalty, say _cpBF_ , takes values in _R_ + (always favouring the smaller model) and is given by For fixed _g_ , this illustrates the link between, _e.g._ , _g_ = _n_ and the BIC. Note also that the penalty increases with _g_ and tends to zero as _g_ tends to zero. If _pg_ is the density function corresponding to the hyperprior on _g_ , this induces the following distribution on this complexity penalty: Figure 4 presents the density function of this complexity penalty for the cases of the benchmark prior with _c_ = 0 _._ 01, the hyper- _g/n_ with _a_ = 3 and the Zellner-Siow prior, using _n_ = 72 and _k_ = 41. Vertical lines indicate the fixed penalty values corresponding to _g_ = _n_ (solid) and _g_ = _k_<sup>2</sup> (dashed). Whereas the hyper- _g/n_ and Zellner-Siow priors favour penalties around the one corresponding to _g_ = _n_ and BIC, it is clear that the benchmark prior implies a much more uniform prior on this complexity penalty and also

> 7 The posterior distribution of this prior complexity penalty is, of course, easily obtained from the posterior of _θ_ via a simple variable transformation.

15
<!-- Start of picture text --> 0.5<br>0.4<br>0.3<br>0.2<br>0.1<br>0.0<br>0 2 4 6 8 10<br>cpBF<br>�<br>BF<br>cp<br>�<br>p<br><!-- End of picture text -->

**Fig. 4.** Prior of complexity penalty _cpBF_ ( _n_ = 72 _, k_ = 41) Benchmark _c_ = 0 _._ 01 (solid), hyper- _g/n_ , _a_ = 3 (dashed), and the Zellner-Siow prior (dotted) Vertical lines correspond to _g_ = _n_ (solid) and _g_ = _k_<sup>2</sup> (dashed)

has considerable mass in areas with appreciably larger penalty values. This is, of course, a direct consequence of the very fat right tail for this hyperprior.

The third factor in (13) corresponds to a penalty for lack of fit, which can be denoted as always taking positive values (remember we consider _Ri_<sup>2</sup><sup>_<R_</sup> _j_<sup>2)andthusfavouringthe</sup> better fitting model. Like _cpBF_ this penalty increases with _g_ (with an upper limit equal to [( _n −_ 1) _/_ 2] log[(1 _− Ri_<sup>2)</sup><sup>_/_(1</sup><sup>_−R_</sup> _j_<sup>2)])andtendstozeroas</sup><sup>_g_tendstozero.Figure5presents</sup> the induced distributions of this lack of fit penalty in the same format as Figure 4, using _n_ = 72 _, k_ = 41 _, Ri_<sup>2=0</sup><sup>_._75and</sup><sup>_R_</sup> _j_<sup>2=0</sup><sup>_._8.Theeffectof</sup><sup>_g_decreasesrapidlyasthevalue</sup> of _g_ increases, and the penalty _fpBF_ quickly approaches its upper bound. This is clearly illustrated by the fact that all three priors lead to a distribution that is concentrated towards the upper limit, which is also close to the penalties obtained with the two popular fixed values for _g_ . Thus, the effect of the prior on _g_ is minimal in terms of this lack-of-fit penalty.

In conclusion, the hyperpriors on _θ_ and _g_ have a pronounced effect on the induced penalties for model complexity, and virtually none on the penalty for lack of fit. These hyperpriors allow for the data to affect the model complexity penalties associated with the prior over model space (through _θ_ ) and the marginal likelihood for each model (through _g_ ), and thus make the analysis less dependent on (usually arbitrary) prior assumptions. In particular, the benchmark hyperprior on _g_ induces a very vague prior on the complexity penalty implicit in the Bayes factor.

16
<!-- Start of picture text --> 50<br>40<br>30<br>20<br>10<br>0<br>7.0 7.2 7.4 7.6 7.8<br>fpBF<br>�<br>BF<br>fp<br>�<br>p<br><!-- End of picture text -->

**Fig. 5.** Prior of lack-of-fit penalty _fpBF_ , ( _n_ = 72 _, k_ = 41 _, Ri_<sup>2= 0</sup><sup>_._75and</sup><sup>_R_</sup> _j_<sup>2= 0</sup><sup>_._8)</sup> Benchmark _c_ = 0 _._ 01 (solid), hyper- _g/n_ , _a_ = 3 (dashed), and the Zellner-Siow prior (dotted) Vertical lines correspond to _g_ = _n_ (solid) and _g_ = _k_<sup>2</sup> (dashed)

# **6. Computational Issues**

In typical applications in economics, the number of potential covariates _k_ is so large that the model space is impossible to evaluate exhaustively.<sup>8</sup> So even if Bayes factors between different models can be computed analytically, we still need some sort of numerical method to conduct inference over models. A convenient way to do this is through Markov chain Monte Carlo, in particular the MC<sup>3</sup> algorithm of Madigan and York (1995) has been used in this context with success (Fern ́andez _et al._ , 2001a; and Ley and Steel, 2009). Eicher _et al._ (2011) experiment with various algorithms and find that MC<sup>3</sup> works quite well in this context. Recently, other alternative methods were proposed, such as _Bayesian Adaptive Sampling_ by Clyde _et al._ (2011) and _Evolutionary Stochastic Search_ by Bottolo and Richardson (2010), which is designed to work for situations where _k_ is orders of magnitude larger than _n_ . We retain the simple MC<sup>3</sup> algorithm, which works well for the types of problems we focus on here (see also Garc ́ıa-Donato and Mart ́ınez-Beneito, 2011). Except for the prior of Maruyama and George (2010) and Forte _et al._ (2010), we need to deal with the fact that the integral in (4) does not have a straightforward closed-form solution. Liang _et al._ (2008) approximate this integral in _g_ with a Laplace approximation, but we will opt for a Gibbs sampler approach over model space and _g_ . In the latter, the Bayes factor between any two models given _g_ is

> 8 For example, the applications to growth data discussed in Section 7 involve model spaces with, respectively, 2<sup>41</sup> = 2 _._ 2 _×_ 10<sup>12</sup> and 2<sup>67</sup> = 1 _._ 5 _×_ 10<sup>20</sup> different models.

17
The conditional posterior of _g_ given _Mj_ is simply<sup>9</sup> The advantage of using the Gibbs sampler on ( _g, Mj_ ) is that it does not rely on approximations that are hard to control and makes prediction quite straightforward: for every _g_ drawn in the sampler we predict as with a fixed _g_ (Fern ́andez _et al._ , 2001a), and predictions are simply mixed over values of _g_ in the sampler. With Laplace approximations this seems much less straightforward and the quality of these approximations is not that easy to assess. Also, truncation, as in the Bottolo-Richardson and Forte _et al._ priors, can be dealt with easily in a Gibbs sampling framework.

We use a simple random walk Metropolis-Hastings step for _g_ with a log-Normal proposal centred over the previous value. Finally, we control the acceptance probability for _g_ by making the Metropolis-Hastings step adaptive.

Throughout, we use MCMC chains of length 1,000,000 after a burn-in of 500,000, which was found to be more than sufficient for convergence. The Fortran code used for this paper is available from the authors upon request.<sup>10</sup>

# **7. Simulations**

First, we examine simulated datasets to mimic the kinds of situations typical for many applications in econometrics (and in other areas as well). We generate datasets from three different model structures, which closely correspond to those in Fern ́andez _et al._ (2001a). For Models 1 and 2 we generate an _n × k_ matrix _R_ for _k_ = 15 regressors where the first ten columns ( _r_ (1) _, . . . , r_ (10)) are drawn from independent standard normal distributions, and the next columns are constructed as where _E_ denotes an _n ×_ 5 matrix drawn from independent normal random variables. This induces correlations between the first five and the last five regressors ranging from 0.153 to 0.74. The demeaned version of _R_ is then the matrix of regressors _Z_ = ( _z_ (1) _, . . . , z_ (15)) used to generate the data according to: Model 2: _y_ = _ιn_ + _σε,_

> 9 Indeed, if we take the null model for _Mj_ we get _kj_ = 0 and _Rj_ 2<sup>=0andwegetexactlytheprioron</sup><sup>_g_</sup> back, as we should (since the null model does not involve _g_ ).

> 10 Fortran code for the fixed _g_ case as described in Ley and Steel (2009) is available online in the _Journal of Applied Econometrics_ archive corresponding to the latter paper. Code in R (which accommodates hyper- _g_ priors) is available from http://bms.zeugner.eu/.

18
where _ε_ is a vector of iid standard normal random variables and we take _σ_ = 2 _._ 5. In Model 1 we use _n_ = 50 _,_ 100 and for Model 2 we use _n_ = 50 _,_ 100 _,_ 1000 _,_ 10 _,_ 000 in order to illustrate consistency.

The remaining model structure used in the simulations uses regressors with pairwise correlations of 0.5, generated as _r_ ( _i_ ) = _r_ (<sup>_⋆_</sup> _i_ )<sup>+</sup><sup>_e_where each</sup><sup>_r_</sup> (<sup>_⋆_</sup> _i_ )<sup>and</sup><sup>_e_are vectors of</sup><sup>_n_independent</sup> standard normal elements. After demeaning to obtain _Z_ we generate _n_ observations as Model 3: _y_ = _ιn_ +<sup>�7</sup> _h_ =1<sup>_z_(</sup><sup>_h_) +</sup><sup>_σε,_</sup> where now we use _σ_ = 2. Values of _k_ used are 20, 40 and 80, while we adopt _n_ = 50 and 100.<sup>11</sup> We analyse 100 simulated data sets for each model and value of _k_ and _n_ mentioned above. The prior on model space is constructed with a random _θ_ as in (5) with a prior mean model size _m_ = _k/_ 2. The different priors for _g_ discussed above have been used, as well as two fixed values for _g_ ( _g_ = _n_ and _g_ = _k_<sup>2</sup> ). For the hyper- _g/n_ prior we take _a_ = 3.

Rather than provide exhaustive results from these simulations, we merely highlight the most important findings here. The different priors do lead to rather different posterior distributions for _g_ . Broadly speaking, the hyper- _g_ priors lead to the smallest median posterior median<sup>12</sup> for _g_ , while the benchmark prior with _c_ = 1 often leads to the largest. From Section 5, we know that larger _g_ induces a higher model size penalty ( _cpBF_ ) and this is immediately reflected in the results. Consider, in particular, the average number of regressors in the visited models. Tables 2 and 3 record the median over the 100 samples of the average model size. As expected, average model sizes are smaller for the priors leading to higher values of _g_ , in particular the benchmark prior with _c_ = 1 and the case with fixed _g_ = _k_<sup>2</sup> . On the other hand, the hyper- _g_ priors and the Bottolo-Richardson and horseshoe priors favour larger models. This is a serious drawback in the case of Model 2, where data are generated from the null model. Here the hyper- _g_ and horseshoe priors (and to a slightly lesser extent the Bottolo-Richardson prior) are choosing models that are far too large: marginal inclusion probabilities of all 15 regressors (none of which are used in the true model) are typically over 0.4 with these priors. In contrast, the probability of erroneously including the regressors tends to be under 1% for the benchmark priors with _c_ = 1 and _c_ = 0 _._ 1 and the case with _g_ = _k_<sup>2</sup> . The priors of Maruyama-George, Forte _et al._ and Zellner-Siow and the case with _g_ = _n_ typically lead to marginal inclusion probabilities under 2.5%.

Tables 2 and 3 also present the median posterior probability of the true model. These tend to be smallest for the hyper- _g_ priors throughout and for the Bottolo-Richardson prior in the case of Model 3. The highest probabilities on the true model are typically found

> 11 The situation with _n_ = 50 and _k_ = 80 means that we need to exclude models with _kj ≥ n −_ 1, as the posterior for such a model would no longer be well-defined. This is simply done by imposing a prior on model space that limits the model size to _n −_ 2. The sampler then rejects any proposed model for which _kj > n −_ 2. In the analysis of these simulated data, the sampler never even gets close to models that large.

> 12 Here and in the subsequent discussion we often focus on the median over the 100 samples we generated. We do not consider the median of the posterior means of _g_ , since the posterior mean of _g_ does not exist for all but the Bottolo-Richardson prior (see the discussion in Subsection 3.3).

19
for the benchmark prior with _c_ = 1 and _c_ = 0 _._ 1, the Maruyama-George prior and the case with fixed _g_ = _k_<sup>2</sup> . For Models 1 and 3 differences in the posterior probability of the true model are not huge (covered by a factor of about 5), but for Model 2 these differences are very substantial indeed, ranging from 0.08 to 0.90 for _n_ = 50 and from 0.08 to 0.99 for _n_ = 10 _,_ 000. Table 2 also illustrates the results on consistency, as dealt with in Proposition 1 and summarized in Table 1. The priors not leading to consistency (hyper- _g_ , F-Z with _a_ = _a_ +2 _/k_<sup>2</sup> and horseshoe priors and the case with _g_ = _k_<sup>2</sup> ) lead to posterior probabilities on the true model that are virtually unaffected by sample size _n_ . The consistent priors (benchmark, Maruyama-George, Bottolo-Richardson, F-Z with _a_ = 2 + 2 _/n_ , Forte _et al._ , hyper- _g/n_ , horseshoe/ _n_ , Zellner-Siow and the case with _g_ = _n_ ) see this probability increase to unity with _n_ , although this convergence seems relatively slow for the horseshoe/ _n_ and very slow for the Bottolo-Richardson prior and the F-Z prior with _a_ = 2 + 2 _/n_ .<sup>13</sup> Another consequence of different model size penalties is the number of models that are actually visited by the chain. From Section 5 we know that the implied model size penalty stems from both _θ_ and _g_ . Thus, we would expect the number of models visited by the chain to be affected by the choice of prior on _g_ . Indeed, that is the case, with the hyper- _g_ and horseshoe priors typically leading to large number of models visited and the benchmark prior with _c_ = 1 and the case with _g_ = _k_<sup>2</sup> often resulting in much less model visits. The differences can be quite substantial— _e.g._ , in Model 1 with _n_ = 50 the hyper- _g_ and horseshoe priors lead to a median of over 10,000 model visits whereas the chains for the benchmark prior with _c_ = 1 visit a median of 4128 different models, and if we fix _g_ = _k_<sup>2</sup> this is only 2132 models. For Model 2 we observe even larger differences, with the number of models visited ranging from around 200 for the benchmark prior with _c_ = 1, MaruyamaGeorge, Forte _et al._ and _g_ = _n_ to about 33,000 for the hyper- _g_ and the horseshoe priors if we take _n_ = 1000. As expected, for Model 3 with _k_ = 80 we observe most visits in the chains, ranging from a median visit count of 3280 for the case with _g_ = _k_<sup>2</sup> to around 60,000 for the hyper- _g_ priors.

An interesting aspect that is suggested by Table 3 is the effect of different values of _k_ and _n_ . One would expect that the challenge of finding the model that generated the data is harder as the model space (and thus _k_ ) grows. Indeed, if we have only _n_ = 50 observations, the posterior probability assigned to the true model decreases by around an order of magnitude each time we double the value of _k_ . However, if we have 100 observations to conduct inference with, we see that the posterior probability of the true model is much less affected by the changes in _k_ . There is a slight downward tendency with _k_ , but the median is never even halved by going from _k_ = 20 to _k_ = 80. This seems to suggest that for large enough _n_ we are not too much led astray by the introduction of further unrelated regressors. Interestingly, the marginal posterior inclusion probabilities of the irrelevant regressors is not much smaller for the case with _n_ = 100, but clearly the entire posterior distribution on model space is quite different. As is obvious from the expressions in Section 5, the sample size _n_ only affect the lack-of-fit penalty and not the

> 13 In the case of the latter prior, Proposition 1 tells us that the Bayes factor in favour of the (true) null model against any other model _Mj_ is 1 + _nkj /_ 2, which seems to be sizeable for _n_ = 10 _,_ 000, but we need to keep in mind that there are many alternative models.

20
complexity penalties. In particular, larger _n_ means a larger penalty for lack of fit and this will tend to favour larger models (since they fit at least as well as the models nested in them). Indeed, we notice in Table 3 that the median model size is larger throughout for _n_ = 100. Furthermore, including irrelevant regressors will not change the fit much, so it is understandable that the inclusion probabilities of irrelevant regressors are about the same for both sample sizes. However, the real difference lies in the fact that with the larger sample size we move less frequently to models that lack one of the “true” regressors as this normally implies a substantial lack-of-fit penalty.

**Table 2.** Simulated Data, Models 1 and 2 (null model)—results for model size and inclusion probability of the true model. Number of regressors _k_ = 15.

||||Mod|el 1|||||Mod|el 2||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
||True model size||5||5||0||0||0||0|
|P|rior mean model size||7.5||7.5||7.5||7.5||7.5||7.5|
||_n_||50||100||50||100||1000|10|,000|
|||||me<br>media|dian a<br>n post|verag<br>erior|e post<br> prob.|erior <br> true|mode<br> mode|l size<br>l (in|: _η_<br> %):|_γ_||
||Prior on _g_|_η_|_γ_|_η_|_γ_|_η_|_γ_|_η_|_γ_|_η_|_γ_|_η_|_γ_|
|1|Benchmark _c_= 1|4.2|0.4|5.2|3.1|0.1|88|0.1|87|0.1|94|0.0|98|
|2|Benchmark _c_= 0_._1|5.3|0.5|5.9|2.5|0.1|90|0.1|89|0.0|95|0.0|99|
|3|Benchmark _c_= 0_._01|6.2|0.5|6.1|2.3|1.1|56|0.8|45|0.2|89|0.0|97|
|4|Maruyama-George|5.5|0.5|5.7|2.5|0.3|81|0.1|89|0.0|97|0.0|99|
|5|Bottolo-Richardson|6.4|0.5|6.2|2.2|4.8|18|4.2|20|3.9|28|3.2|31|
|6|Hyper-_g_, _a_= 3|7.4|0.3|6.9|1.7|6.6|9|6.4|9|6.4|9|6.3|9|
|7|Hyper-_g_, _a_= 4|7.7|0.3|7.2|1.5|6.8|8|6.6|8|6.7|8|6.6|8|
|8|F-Z, _a_= 2 + 2_/k_<sup>2</sup>|6.3|0.4|6.1|2.2|2.2|41|2.0|40|2.0|40|2.2|44|
|9|F-Z, _a_= 2 + 2_/n_|6.3|0.4|6.2|2.3|2.5|40|2.1|37|2.0|42|2.1|43|
|10|Horseshoe|6.8|0.4|6.5|2.1|6.5|10|6.2|11|6.3|11|6.3|11|
|11|Forte _et al._|6.0|0.5|6.0|2.4|0.3|78|0.2|86|0.0|96|0.0|99|
|12|Hyper-_g/n_|5.8|0.5|5.8|2.4|1.2|44|0.7|53|0.1|91|0.0|98|
|13|Horseshoe/_n_|6.1|0.4|6.0|2.4|3.9|29|3.1|35|1.7|54|0.6|66|
|14|Zellner-Siow|5.6|0.5|5.9|2.4|0.4|73|0.2|82|0.1|94|0.0|98|
|15|_g_ =_n_|4.8|0.6|5.1|3.1|0.4|71|0.2|80|0.1|94|0.0|98|
|16|_g_ =_k_<sup>2</sup>|3.6|0.3|4.6|3.6|0.2|87|0.2|86|0.2|86|0.1|87|

Combining the results on consistency with those of the posterior probability assigned to the true model, we conclude from this section that the best performing priors seem to be the benchmark, Maruyama-George, Forte _et al._ and the Zellner-Siow priors.

# **8. Applications to Real Data**

In the context of real applications, we will further examine all the different priors, and will also investigate the predictive performance as well as Bayes factors. For each application, we will randomly partition the sample in an estimation sample and a prediction sample, of a fixed size. We do this 50 times and this allows us to assess predictive performance based 21
**Table 3.** Simulated Data, Model 3—results for model size and inclusion probability of the true model.

||||||||Mo|del 3||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
||||_k_ =|20|||_k_ =|40|||_k_ =|80||
||True model size||7||7||7||7||7||7|
|P|rior mean model size||10||10||20||20||40||40|
||_n_||50||100||50||100||50||100|
|||||me<br>media|dian <br>n po|avera<br>sterio|ge pos<br>r prob|terio<br>. tru|r mod<br>e mod|el siz<br>el (in|e: _η_<br> %): _γ_|||
||Prior on _g_|_η_|_γ_|_η_|_γ_|_η_|_γ_|_η_|_γ_|_η_|_γ_|_η_|_γ_|
|1|Benchmark _c_= 1|6.7|1.1|8.1|23|5.3|0.1|7.6|29|4.1|0.00|7.1|30|
|2|Benchmark _c_= 0_._1|7.0|1.0|8.2|21|6.3|0.1|8.1|20|5.3|0.01|7.9|19|
|3|Benchmark _c_= 0_._01|7.1|1.0|8.2|20|6.6|0.1|8.2|18|6.1|0.01|8.3|15|
|4|Maruyama-George|7.1|1.0|8.2|20|6.7|0.1|8.3|18|6.2|0.01|8.4|14|
|5|Bottolo-Richardson|7.9|0.5|9.1|10|7.3|0.1|9.0|9|6.7|0.00|9.2|6|
|6|Hyper-_g_, _a_= 3|7.5|0.8|8.5|16|7.0|0.1|8.5|15|6.5|0.00|8.6|11|
|7|Hyper-_g_, _a_= 4|7.6|0.8|8.6|15|7.1|0.1|8.6|14|6.6|0.01|8.7|10|
|8|F-Z, _a_= 2 + 2_/k_<sup>2</sup>|7.1|1.0|8.2|20|6.7|0.1|8.3|18|6.2|0.01|8.4|13|
|9|F-Z, _a_= 2 + 2_/n_|7.1|1.0|8.2|20|6.7|0.1|8.3|18|6.2|0.01|8.3|14|
|10|Horseshoe|7.3|1.0|8.3|18|6.8|0.1|8.3|17|6.4|0.01|8.5|12|
|11|Forte _et al._|7.3|0.9|8.3|18|6.8|0.1|8.4|17|6.4|0.01|8.5|12|
|12|Hyper-_g/n_|7.1|1.1|8.2|20|6.7|0.1|8.3|18|6.1|0.01|8.4|13|
|13|Horseshoe/_n_|7.2|1.0|8.2|19|6.7|0.1|8.3|18|6.3|0.01|8.4|13|
|14|Zellner-Siow|7.2|1.0|8.3|19|6.8|0.1|8.3|18|6.3|0.01|8.4|13|
|15|_g_ =_n_|7.5|0.8|8.7|14|7.1|0.1|8.6|13|6.6|0.01|8.8|8|
|16|_g_ =_k_<sup>2</sup>|6.2|0.9|7.8|31|4.9|0.0|7.2|36|3.8|0.00|6.6|28|

on the log predictive score, abbreviated to LPS (Fern ́andez _et al._ , 2001a,b).<sup>14</sup> In addition, analyzing the 50 estimation subsamples allows us to get a certain amount of robustness with respect to possible unusual data points, partly addressing some of the concerns voiced in Ciccone and Jaroci ́nski (2010)<sup>15</sup> with respect to data sensitivity. Bayes factors between models with different priors will be computed on the basis of the full sample. We consider four different priors on the model space: fixed _θ_ and random _θ_ as in (5), both with prior mean model sizes of _m_ = 7 and _m_ = _k/_ 2.

# _8.1. Cross-country growth regressions_

The context of cross-country growth regressions is one of the first areas within economics where the use of BMA has become popular. This area of macroeconomics is characterized by a particularly large number of potential drivers for growth and a scarcity of observations,

> 14 Alternative proper scoring rules are discussed in Gneiting and Raftery (2007).

> 15 However, also see the comment in Feldkircher and Zeugner (2010).

22
so this is an ideal candidate for BMA. We consider below two datasets which have been used in many studies on this topic.

# The FLS data

The first dataset we use contains _k_ = 41 potential regressors for modelling average per capita growth over the period 1960-1992 for a sample of _n_ = 72 countries. It was used in Fern ́andez _et al._ (2001b) (FLS), which presents more details.

First, we consider the analysis of the 50 subsamples which each consist of 62 observations. In line with the higher model size penalty implicit in the benchmark Beta prior with _c_ = 1 and the case with fixed _g_ = _k_<sup>2</sup> , we note that marginal posterior inclusion probabilities of the regressors tend to be (sometimes much) smaller for these cases. In keeping with the higher _g_ values induced by these priors, the posterior mean of _βj_ for model _Mj_ will be larger in absolute value (less shrinkage to the prior mean of zero), but this will typically be counteracted by more mass on smaller models. The resulting marginal posterior means of the regression coefficients can be smaller or larger than those for the other priors on _g_ . For the other priors posterior inclusion probabilities and posterior inference on the regression coefficients are quite similar, even though the posterior inference on _g_ can be somewhat different. Figure 6 presents boxplots over the 50 subsamples for the posterior medians of _g_ (on a log scale), the number of visited models and the posterior mean model size. As with the simulated data, posterior medians of _g_ are smallest for the hyper- _g_ priors and largest for the benchmark prior with _c_ = 1 and the case with fixed _g_ = _k_<sup>2</sup> . This leads to most models visited in the chain for the hyper- _g_ priors and least models in the chain for the benchmark with _c_ = 1 and _g_ = _k_<sup>2</sup> cases. Posterior model sizes are lower for the latter two cases, but not that different between the other priors. Note also that the prior assumptions on model space have almost no effect on posterior inference on _g_ , but prior mean model size _m_ does substantially affect the number of visited models and the posterior mean model size when we use a fixed _θ_ approach. In line with the results in Ley and Steel (2009), however, this dependence on _m_ virtually disappears when we use the random _θ_ prior in (5). Thus, for the rest of the section we will only present results for random _θ_ with _m_ = 7.

Prediction based on LPS (for 10 observations in each prediction subsample) is summarized in the top panel of Figure 7, where we present boxplots of the LPS values for BMA over the 50 prediction subsamples. As lower values of LPS are associated with better predictions, this suggests that the benchmark prior with _c_ = 1 and the fixed _g_ = _k_<sup>2</sup> case tend to predict somewhat worse than the rest. The other priors are quite close in predictive performance. This is consistent with these priors leading to quite similar posterior inclusion probabilities for the regression variables, and rather different from those obtained with the priors that result in worse predictions. BMA predicts best (compared with the highest posterior probability model, the null model and the full model) in most of the subsamples (in between 52% and 62%, depending on which prior we choose for _g_ ) and is never beaten by the null model, except for a few cases (up to 6%) with the benchmark priors with _c_ = 1 and _c_ = 0 _._ 1 and the fixed _g_ specifications.

The information contained in the data regarding _g_ is summarized by the marginal likeli23
log ( g ) : Fixed θ , m = 7 Number of Models: Fixed θ , m = 7 Model Size: Fixed θ , m = 7 <!-- Start of picture text --> G<br>G<br>G G<br>G G G G G G G G G G G<br>G<br>G<br>1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15<br>12<br>7 200000<br>11<br>6 150000 10<br>5 9<br>100000<br>8<br>4<br>7<br>50000<br>3 6<br>0<br><!-- End of picture text --> log ( g ) : Fixed θ , m = 20.5 Number of Models: Fixed θ , m = 20.5 Model Size: Fixed θ , m = 20.5 <!-- Start of picture text --> G G G G G G G G G G G G G G G<br>G<br>G G G G G<br>G<br>G G G G G G G G G G G G G<br>G<br>G G G G G G G G G G<br>G G G<br>G<br>G<br>1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15<br>7 5e+05<br>6 4e+05 20<br>5 3e+05<br>15<br>4 2e+05<br>3<br>1e+05 10<br><!-- End of picture text --> log ( g ) : Random θ , m = 7 Number of Models: Random θ , m = 7 Model Size: Random θ , m = 7 <!-- Start of picture text --> G G GG G G GGG GGG GGG GGG GGG GGG GGG GGG GGG GGG G GG GG GG GG GG GG GG GG GG GG GG GG GG GG<br>G G GG<br>G G GG GG GG G GG GG GG GG GG GG GG<br>G<br>G G G<br>G G<br>1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15<br>25<br>7<br>6 3e+05 20<br>5 2e+05 15<br>4<br>10<br>1e+05<br>3<br>5<br>0e+00<br><!-- End of picture text --> log ( g ) : Random θ , m = 20.5 Number of Models: Random θ , m = 20.5 Model Size: Random θ , m = 20.5 <!-- Start of picture text --> G G GG G G G G GG GG GG G GG GG GG GG GG GG GG GG GG GG GG GG GG GG<br>G G G G G G G G G G G GG G G<br>G<br>G G G G G G G G G G G G G G<br>1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15<br>7 4e+05 25<br>6 3e+05 20<br>5<br>15<br>2e+05<br>4<br>10<br>1e+05<br>3<br>5<br>0e+00<br><!-- End of picture text -->

**Fig. 6.** FLS data: log posterior medians of _g_ , number of models visited and posterior mean model size The different priors on _g_ are ordered as in Tables 2 and 3. The top two rows correspond to fixed _θ_ (with different _m_ ), while the bottom two rows are for random _θ_ priors on model space

24
benchmark prior with _c_ = 1 (Prior 1) leads to very small prior and posterior density values at _g_ 0, so that the ratio can really not be computed with any accuracy. However, given that this is indicative of a very small prior density in the region of the mode of _ly_ ( _g_ ), it is safe to assume the associated Bayes factor would be quite small. The same problem affects other models with the other two datasets (indicated with – in Table 4). As commented in Section 4, the priors that put a lot of weight on regions of _g_ with high _ly_ ( _g_ ) do particularly well in this respect. Thus, the Forte _et al._ , Zellner-Siow, Bottolo-Richardson, hyper- _g/n_ and horseshoe/ _n_ priors lead to the highest Bayes factors while the F-Z, hyper- _g_ and benchmark priors do worse. This is in line with the prior density functions in Figures 1 and 2 and the marginal likelihood in Figure 8. In Table 4 we also include the models where we choose a fixed value of _g_ , in which case we can compute the Bayes factor as the ratio _p_ 3( _g | y_ ) _/p_ 3( _g_ ) evaluated at the fixed value of _g_ (as explained in Section 4). Some entries in the table are missing, as they correspond to _p_ 3( _g|y_ ) = 0 for the fixed value of _g_ ; again, this can be taken as a sign that this value is associated with a very small Bayes factor. So, whereas the data do support choosing _g_ = _n_ , the case _g_ = _k_<sup>2</sup> does not receive much support in terms of Bayes factors.

**Table 4.** Bayes Factors of Different Priors vs Benchmark with _c_ = 0 _._ 01 (Prior 3)

(Random _θ_ , _m_ = 7)

|_i_|Prior on _g_|FLS|SDM|Tobias-Li|
|---|---|---|---|---|
|2|Benchmark _c_= 0_._1|0.3|–|0.4|
|4|Maruyama-George|16.3|10.1|–|
|5|Bottolo-Richardson|23.6|57.2|24.4|
|6|Hyper-_g_, _a_= 3|2.3|8.7|3.0|
|7|Hyper-_g_, _a_= 4|0.7|3.0|1.0|
|8|F-Z, _a_= 2 + 2_/k_<sup>2</sup>|0.1|0.1|0.3|
|9|F-Z, _a_= 2 + 2_/n_|2.4|5.6|0.1|
|10|Horseshoe|11.2|34.6|11.9|
|11|Forte _et al._|53.2|123.2|n.a.|
|12|Hyper-_g/n_|21.8|50.8|1.7|
|13|Horseshoe/_n_|25.9|68.5|7.6|
|14|Zellner-Siow|29.8|60.1|–|
||_g_0|25.5|19.5|19.5|
|15|_g_ =_n_|19.7|37.1|–|
|16|_g_ =_k_<sup>2</sup>|–|0.1|–|

# The SDM data

Another popular growth dataset was introduced by Sala-i-Martin _et al._ (2004) (SDM), who model annual GDP growth per capita between 1960 and 1996 for _n_ = 88 countries using _k_ = 67 potential regressors.

The 50 randomly selected estimation subsamples each have 75 observations, and lead to the boxplots in Figure 9. The results are mostly in line with those for the FLS data, but with larger differences between the priors in terms of posterior model size.

26
log ( g ) : Random θ , m = 7 <!-- Start of picture text --> log ( g ) : Random θ , m = 7 Number of Models: Random θ , m = 7 Model Size: Random θ , m = 7<br>G G G G G G G G G G G G<br>G G G G G G G G GG GG G G G G G GG G G G GGG G<br>G GG<br>G G G G G G G G GG G G G<br>G G G G G GG GG G G G G G GG G G G G G G G G G G G G G<br>1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15<br>8 250000 16<br>7 14<br>6 12<br>150000<br>5 10<br>8<br>4<br>6<br>3 50000<br>4<br>2 0<br><!-- End of picture text -->

**Fig. 9.** SDM data: log posterior medians of _g_ , number of models visited and posterior mean model size.

Predictions for these data seem to be quite similar for all priors, as judged by the LPS boxplots in Figure 7. Here we have used prediction subsets of 13 observations each. Interestingly, these comparable prediction accuracies are obtained with regression models that are rather different, since the posterior inclusion probabilities corresponding to the benchmark priors with _c_ = 1 and _c_ = 0 _._ 1 and to the choice of _g_ = _k_<sup>2</sup> ( _i.e._ , the models leading to the highest values for _g_ ) are quite different from those using the other priors. BMA predicts best in 76% to 92% of the subsamples, with the exception of the benchmark priors with _c_ = 1 and _c_ = 0 _._ 1 and the case with _g_ = _k_<sup>2</sup> , where BMA outperforms the highest posterior probability model, the null model and the full model in roughly half the subsamples. BMA is only beaten by the null model in up to 8% of the cases,<sup>16</sup> except for the benchmark priors with _c_ = 1 and _c_ = 0 _._ 1 and the fixed- _g_ specifications, where the null model outperforms BMA in 10% or 12% of the cases.

The marginal information in the data on _g_ , as presented in Figure 8, is quite similar to that for the LPS data, with a slight shift to smaller values and a heavier tail. As a consequence, the pattern of Bayes factors between the models with different priors on _g_ is rather similar to what they were for the FLS data (see Table 4).

# _8.2. Returns to Schooling_

Here we investigate the area of returns to education, where again many potential model specifications have been proposed, and BMA was introduced in this context by Tobias and Li (2004). As these are microeconomic data, the number of potential observations is much larger. We will focus on the log of hourly wages of _n_ = 1190 white males in the U.S. in 1990, which we model as a function of _k_ = 26 possible regressors. In order to simplify the computations, Tobias and Li (2004) always include seven of these regressors in each model and restrict themselves to a smaller model space characterized by 18 possibly included regressors, but we allow for inclusion or exclusion of all regressors. In addition, we have added the local unemployment rate (which only appears in Tobias and Li, 2004, through an interaction term with education) as a potential regressor, giving us a total of 26 candidate regressors. More details on the application can be found in Tobias and Li (2004).

> 16 For the hyper- _g_ prior with _a_ = 4, BMA is actually never beaten by the null model.

27
On the basis of 50 subsamples of 1012 observations, we get the results for _g_ , numbers of models visited and model size presented in Figure 10. Results are now a bit different from those in the growth applications, as a result of the much larger value of _n_ . In particular, the Maruyama-George, Forte _et al._ and Zellner-Siow priors as well as the case with fixed _g_ = _n_ lead to much higher posterior medians of _g_ , a smaller number of models visited in the chain and smaller mean model size than in the previous applications. Of course, all of these priors depend on _n_ , but it is interesting that the F-Z prior with _a_ = 2 + 2 _/n_ and the hyper- _g/n_ and horseshoe _/n_ priors are less affected. Of course, the benchmark priors and the Bottolo-Richardson prior depend on _n_ through max _{n, k_<sup>2</sup> _}_ , which is here of the same order of magnitude as for the FLS data.

<!-- Start of picture text --> log ( g ) : Random θ , m = 7 Number of Models: Random θ , m = 7 Model Size: Random θ , m = 7<br>G<br>G G<br>G<br>G G G G<br>G G<br>G G G G G G<br>G G<br>1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15 1 2 3 4 5 6 7 8 9 11 13 15<br>7<br>4e+05<br>6 3e+05 14<br>5 12<br>2e+05<br>4 10<br>3 1e+05 8<br>0e+00<br><!-- End of picture text -->

**Fig. 10.** Tobias-Li data: log posterior medians of _g_ , number of models visited and posterior mean model size.

Prediction based on prediction subsamples of 178 observations each is summarized in the lower panel of Figure 7. Even though there is some variability in the posterior inclusion probabilities of the regressors over the different priors, we observe virtually identical predictive performance. This is related to the fact that the full model here predicts very similarly to using BMA. This is not surprising as the number of observations _n_ is much larger than the number of potential regressors, so we can hardly overfit and for prediction not much is lost by including all _k_ regressors. In the full model, the only effect that the prior for _g_ has on prediction is the different shrinkage of the coefficients, but as the shrinkage coefficient is very close to one anyway, that will make very little difference. Of course, BMA still has an important role in identifying the important determinants of wages, but is not that critical for prediction. In line with this, the null model does very badly here and never beats BMA or the full model. BMA predicts best in 20% to 46% of the subsamples, with the lowest percentages occurring for the benchmark Beta priors with _c_ = 1 and the fixed- _g_ cases. BMA is beaten by the full model in 42% to 48% of the cases with random _g_ . For the two specifications with fixed _g_ , the full model beats BMA in 52% of the subsamples, while the best model also beats BMA there in over a third of the cases.

Figure 8 tells us that the likelihood marginalized with the prior of everything but _g_ is rather similar to that of the growth datasets. This is perhaps surprising, given that the number of observations _n_ is quite a lot larger in this case. This leads to somewhat similar behaviour of the Bayes factors as for the two growth datasets, with the exception of the 28
Maruyama-George and Zellner-Siow priors (leading to a zero posterior density at _g_ 0), the F-Z with _a_ = 2 + 2 _/n_ , the hyper- _g/n_ and horseshoe- _n_ priors and the case with _g_ = _n_ , as a consequence of the very different value of _n_ for this application. In this application, the truncation induced by the Forte _et al._ prior means that the prior support does not include the chosen value of _g_ 0 (and is truncated quite a bit above that), which certainly makes this prior less appropriate for large values of _n/k_ . Thus, prior and posterior density values are both zero at _g_ 0 and the Bayes factor can not be computed for this case.<sup>17</sup> Let us summarize the information provided by these applications to real economic datasets. Predictive performance seems quite similar across priors, although possibly a bit worse for the benchmark prior with _c_ = 1 and the case with _g_ = _k_<sup>2</sup> . There is more variation in the Bayes factors, with most consistent data support provided for the Bottolo-Richardson, horseshoe, hyper- _g/n_ and horseshoe _/n_ priors, whereas the Maruyama-George, Forte _et al._ and Zellner-Siow priors (as well as the case with fixed _g_ = _n_ ) do quite well in two of the three applications, but rather badly in the one with large _n_ .

# **9. Concluding Remarks and Recommendations**

Combining the properties listed in Table 1 with the evidence from both simulated and real data, we can now come up with a recommendation for practitioners. We assume that users will want their priors to be consistent, to avoid the information paradox and to perform well in a wide variety of situations. The Bottolo-Richardson and horseshoe- _n_ priors are consistent (albeit at a slow rate) and do well on real data, but underperform on the simulated data. The truncation of the Forte _et al._ . prior makes it hard to recommend for situations where _n_ is appreciably larger than _k_ . The benchmark Beta prior with _c_ = 1 does not fare well in prediction for one application, and the benchmark priors with _c_ = 1 and _c_ = 0 _._ 1 do not do well in terms of Bayes factors. The Zellner-Siow and Maruyama-George priors perform quite well, except for the last application (with large _n_ ), where they get very little support from the data. Nevertheless, we feel they do deserve a place in the econometrician’s toolbox, especially if _n_ is relatively small (comparable to _k_ , say). In addition, if one wants to avoid numerical methods to compute Bayes factors between models with different sets of regressors, the Maruyama-George prior should be recommended, although this comes at the (small) cost of making the prior model-specific, which may make it slightly harder to interpret the prior on _g_ .

In our view, the two priors that stand out by not having displayed any truly bad behaviour in our experiments are the benchmark Beta prior with _c_ = 0 _._ 01 and the hyper- _g/n_ prior (with _a_ = 3).<sup>18</sup> Thus, these priors provide an interesting compromise and would be our general recommendations to practitioners.

> 17 Using a much larger value of _g_ 0 ( _g_ 0 _>_ 40) it would be possible, in principle, to compute a Bayes factor, but we can safely assume the Tobias-Li data do not support this prior at all, as it puts no mass whatsoever on areas with large values for the marginal likelihood _ly_ ( _g_ ).

> 18 For the hyper- _g_ prior we find (as do Feldkircher and Zeugner, 2009) that the choice between _a_ = 3 and _a_ = 4 makes little difference. As a consequence, we would also expect the hyper- _g/n_ prior with _a_ = 4 to perform very similarly.

29
The hierarchical Bayesian model explored in this paper has a hyperprior on the covariate inclusion probability _θ_ and a hyperprior on _g_ , which leads to an integral for the marginal likelihood that is solved by running the MCMC sampler over models and _g_ jointly. The advantage of these hyperpriors is to make the analysis more robust with respect to often arbitrary prior assumptions. We now allow the data to inform us on variable inclusion probabilities and the appropriate region for _g_ . This will affect the model size penalty (and, to a much lesser extent, the lack-of-fit penalty) for each given model. Putting a prior on both _θ_ and _g_ makes the analysis naturally adaptive and avoids the information paradox (Liang _et al._ , 2008), which affects analyses with fixed _g_ . It is important to stress that we are not claiming that using hyperpriors on _θ_ and _g_ performs better (according to some criterion) than a model with fixed values of _θ_ and _g_ that are chosen to optimize this same criterion. The point is rather that we don’t know what the “optimal” values are and getting them wrong can lead to very poor performance. We feel the model used here with the recommended priors on _g_ can be considered a safe “automatic” choice for use in Bayesian Model Averaging in the types of linear regression problems that typically arise in a variety of econometric settings.

# **References**

- Andrews, D.R., and C.L. Mallows (1974) “Scale Mixtures of Normal Distributions,” _Journal of the Royal Statistical Society_ , B, 36: 99–102.

- Berger, J.O. (1985) _Statistical Decision Theory and Bayesian Analysis_ , 2nd ed., New York: Springer.

- Berger, J.O. and L. Pericchi (1996) “The Intrinsic Bayes Factor for model selection and prediction,” _Journal of the American Statistical Association_ , 91: 109–122.

- Berger, J.O., L. Pericchi and J. Varshavsky (1998) “Bayes factors and marginal distributions in invariant situations,” _Sankhy ̄a_ , Ser. A, 60: 307–321.

- Bernardo, J.M., and A.F.M. Smith (1994) _Bayesian Theory_ , Chichester: John Wiley.

- Bottolo L., and Richardson S. (2008), “Fully Bayesian Variable Selection Using _g_ -Priors,” Working paper, Imperial College.

- Bottolo L., and Richardson S. (2010), “Evolutionary Stochastic Search for Bayesian Model Exploration,” _Bayesian Analysis_ , 5: 583–618.

- Brock, W., S. Durlauf and K. West (2003) “Policy Evaluation in Uncertain Economic Environments,” (with discussion) _Brookings Papers of Economic Activity_ , 1: 235–322.

- Brown, P.J., M. Vannucci and T. Fearn (1998) “Bayesian Wavelength Selection in Multicomponent Analysis,” _Journal of Chemometrics_ , 12: 173–182.

- Carvalho, C.M., N.G. Polson and J.G. Scott (2010) “The Horseshoe Estimator for Sparse Signals,” _Biometrika_ , 97: 465–480.

- Chib, S. (1995) “Marginal Likelihood from the Gibbs Output,” _Journal of the American Statistical Association_ , 90: 1313–1321.

- Ciccone, A. and Jaroci ́nski, M. (2010). Determinants of Economic Growth: Will Data Tell? _American Economic Journal: Macroeconomics_ , 2: 222–246.

- Clyde, M.A., and E.I. George (2004) “Model Uncertainty,” _Statistical Science_ , 19: 81–94.

- Clyde, M.A., J. Ghosh and M. Littman (2011) “Bayesian adaptive sampling for variable selection and model averaging”, _Journal of Computational and Graphical Statistics_ , 20: 80–101.

- Cui, W., and George, E. I. (2008) “Empirical Bayes vs. fully Bayes variable selection,” _Journal of Statistical Planning and Inference_ , 138: 888–900.

30
- Eicher, T.S., C. Papageorgiou and A.E. Raftery (2011) “Default Priors and Predictive Performance in Bayesian Model Averaging, With Application to Growth Determinants,” _Journal of Applied Econometrics_ , 26: 30-55.

- Feldkircher, M. and S. Zeugner (2009) “Benchmark Priors Revisited: On Adaptive Shrinkage and the Supermodel Effect in Bayesian Model Averaging,” IMF Working Paper 09/202.

- Feldkircher, M. and S. Zeugner (2010) “The Impact of Data Revisions on the Robustness of Growth Determinants: A Note on ‘Determinants of Economic Growth. Will Data Tell’?,” Working Paper 2010-12, University of Salzburg.

- Fern ́andez, C., E. Ley and M.F.J. Steel (2001a) “Benchmark Priors for Bayesian Model Averaging,” _Journal of Econometrics_ , 100: 381–427.

- Fern ́andez, C., E. Ley and M.F.J. Steel (2001b) “Model Uncertainty in Cross-Country Growth Regressions,” _Journal of Applied Econometrics_ , 16: 563–76.

- Fern ́andez, C., and M.F.J. Steel (2000) “Bayesian Regression Analysis With Scale Mixtures of Normals,” _Econometric Theory_ , 16: 80-101

- Foster, D.P., and E.I. George (1994), “The Risk Inflation Criterion for multiple regression,” _Annals of Statistics_ , 22: 1947–1975.

- Forte, A., M.J. Bayarri, J.O. Berger and G. Garc ́ıa-Donato (2010), “Closed-form objective Bayes factors for variable selection in linear models”, poster presentation _Frontiers of Statistical Decision Making and Bayesian Analysis_ in honour of Jim Berger.

- Garc ́ıa-Donato, G. and M.A. Mart ́ınez-Beneito (2011), “Inferences in Bayesian variable selection problems with large model spaces”, Technical Report, arXiv:1101.4368v1.

- Gneiting, T. and A.E. Raftery (2007) ”Strictly Proper Scoring Rules, Prediction and Estimation,” _Journal of the American Statistical Association_ , 102: 359–378.

- Gradshteyn, I.S. and I.M. Ryzhik (1994) _Table of Integrals, Series and Products_ , San Diego: Academic Press, 5th ed.

- Hoeting, J.A., D. Madigan, A.E. Raftery and C.T. Volinsky (1999) “Bayesian model averaging: A tutorial,” _Statistical Science_ 14: 382–401.

- Jeffreys, H. (1961) _Theory of Probability_ , Oxford: Clarendon Press, 3rd ed.

- Kass, R.E. and L. Wasserman (1995) “A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion,” _Journal of the American Statistical Association_ , 90: 928–934.

- Liang, F., R. Paulo, G. Molina, M.A. Clyde, and J.O. Berger (2008) “Mixtures of _g_ -priors for Bayesian Variable Selection,” _Journal of the American Statistical Association_ , 103: 410–423.

- Ley, E. and M.F.J. Steel (2009) “On the Effect of Prior Assumptions in Bayesian Model Averaging With Applications to Growth Regression,” _Journal of Applied Econometrics_ , 24: 651–674.

- Madigan, D. and J. York (1995) “Bayesian graphical models for discrete data,” _International Statistical Review_ , 63: 215–232.

- Maruyama, Y. and E.I. George (2010) “gBF: A Fully Bayes Factor with a Generalized _g_ -prior,” Technical Report, University of Pennsylvania, arXiv:0801.4410.

- Mitchell, T.J. and J.J. Beauchamp (1988) “Bayesian variable selection in linear regression (with discussion), ” _Journal of the American Statistical Association_ , 83: 1023–1036.

- Nott, D.J. and R. Kohn (2005) “Adaptive Sampling for Bayesian Variable Selection,” _Biometrika_ , 92: 747–763

- Raftery, A.E., D. Madigan, and J. A. Hoeting (1997) “Bayesian Model Averaging for Linear Regression Models,” _Journal of the American Statistical Association_ , 92: 179–191.

- Sala-i-Martin, X.X., G. Doppelhofer and R.I. Miller (2004) “Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach,” _American Economic Review_ , 94: 813– 835.

- Strawderman, W. (1971) “Proper Bayes minimax estimators of the multivariate normal mean,” _Annals of Statistics_ , 42: 385–388.

31
- Tobias, J.L. and Li, M. (2004) “Returns to Schooling and Bayesian Model Averaging; A Union of Two Literatures,” _Journal of Economic Surveys_ , 18: 153–180.

- Verdinelli, I. and Wasserman, L. (1995) “Computing Bayes Factors Using a Generalization of the SavageDickey Density Ratio,” _Journal of the American Statistical Association_ , 90: 614-618.

- Zellner, A. (1971) _An Introduction to Bayesian Inference in Econometrics_ , New York: Wiley.

- Zellner, A. (1986) “On assessing prior distributions and Bayesian regression analysis with _g_ -prior distributions,” in _Bayesian Inference and Decision Techniques: Essays in Honour of Bruno de Finetti_ , eds. P.K. Goel and A. Zellner, Amsterdam: North-Holland, pp. 233–243.

- Zellner, A. and Siow, A. (1980) “Posterior odds ratios for selected regression hypotheses,” (with discussion) in _Bayesian Statistics_ , eds. J.M. Bernardo, M.H. DeGroot, D.V. Lindley and A.F.M. Smith, Valencia: University Press, pp. 585–603.

32