metadata: authors: 0: author: Ley,Eduardo
1: author: Steel, Mark F. J.
count: World
docna: 0: docna: On the effect of prior assumptions in
            Bayesian model averaging with applications to growth regression
docty: Policy Research Working Paper
subtopic: Population & Development,Tertiary Education,International Trade and Trade Rules,Cholera,Malaria,Communicable Diseases,Leprosy,Energy and Mining
teratopic: Science and Technology Development,Education
abstracts: cdata!: This paper examines the problem of
            variable selection in linear regression models. Bayesian
            model averaging has become an important tool in empirical
            settings with large numbers of potential regressors and
            relatively limited numbers of observations. The paper
            analyzes the effect of a variety of prior assumptions on the
            inference concerning model size, posterior inclusion
            probabilities of regressors, and predictive performance. The
            analysis illustrates these issues in the context of
            cross-country growth regressions using three datasets with
            41 to 67 potential drivers of growth and 72 to 93
            observations. The results favor particular prior structures
            for use in this and related contexts.
repnme: repnme: On the effect of prior assumptions in
            Bayesian model averaging with applications to growth regression
display_title: On the effect of prior assumptions
            in Bayesian model averaging with applications to growth regression
listing_relative_url: /research/2007/06/7712856/effect-prior-assumptions-bayesian-model-averaging-applications-growth-regression
topicv3: Tertiary Education m1326349 482,Trade m1326924 698,Malaria m1327042 1420,Iron m1503514 2931,Life Expectancy m1504363 2666
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Science and Technology Development,Education
no_of_pages: 25
author_keys: 0: author_key: 913182
model_extractions: WPS4238

# **On the Effect of Prior Assumptions in Bayesian Model Averaging with Applications to Growth Regression**

Eduardo Ley Mark F.J. Steel

**Abstract.** This paper examines the problem of variable selection in linear regression models. Bayesian model averaging has become an important tool in empirical settings with large numbers of potential regressors and relatively limited numbers of observations. The paper analyzes the effect of a variety of prior assumptions on the inference concerning model size, posterior inclusion probabilities of regressors and on predictive performance. The analysis illustrates these issues in the context of cross-country growth regressions using three datasets with 41 to 67 potential drivers of growth and 72 to 93 observations. The results favor particular prior structures for use in this and related contexts.

**Keywords.** Model size; Model uncertainty; Posterior odds; Prediction; Prior odds; Robustness

**JEL Classification System.** C11, O47

**Email.** eley@worldbank.org  and m.f.steel@stats.warwick.ac.uk

World Bank Policy Research Working Paper 4238, June 2007 _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent. Policy Research Working Papers are available online at http://econ.worldbank.org._
# **1. Introduction**

This paper considers model uncertainty associated with variable selection in linear regression models. In particular, we focus on applications to cross-country growth regressions, where we often face a large number of potential drivers of growth with only a limited number of observations. Insightful discussions of model uncertainty in growth regressions can be found in Brock and Durlauf (2001) and Brock, Durlauf and West (2003). Various approaches to deal with this model uncertainty have appeared in the literature, starting with the extreme-bounds analysis in Levine and Renelt (1992) and the confidence-based analysis in Sala-i-Martin (1997). A natural solution, supported by formal probabilistic reasoning, is the use of Bayesian model averaging (BMA, see Hoeting _et al._ , 1999), which assigns probabilities on the model space and deals with model uncertainty by mixing over models, using the posterior model probabilities as weights.

Fern ́andez _et al._ (2001b, FLS henceforth) introduce the use of BMA in growth regressions. Often, the posterior probability is spread widely among many models, which strongly suggests using BMA rather than choosing a single model. Evidence of superior predictive performance of BMA can be found in, _e.g._ , Raftery _et al._ (1997), Fern ́andez _et al._ (2001a) and FLS. Other papers using BMA in the context of growth regression are Le ́on-Gonz ́alez and Montolio (2004) and Masanjala and Papageorgiou (2005, MP henceforth). Alternative ways of dealing with model uncertainty are proposed in Sala-i-Martin _et al._ (2004, SDM henceforth), and Tsangarides (2005). As we will show in the paper, the SDM approach corresponds quite closely to a BMA analysis with a particular choice of prior.

For Bayesian (and approximately Bayesian) approaches to the problem, any differences can typically be interpreted as the use of different prior assumptions. A casual comparison of results can sometimes lead to a misleading sense of “robustness” with respect to such assumptions. In particular, posterior results on inclusion probabilities of regressors reported in SDM were found to be rather close to those obtained with the quite different prior settings of FLS, using the same data; such similarities were noted in MP and Ley and Steel (2007). As we will show here, this is mostly by accident, and prior assumptions can be extremely critical for the outcome of BMA analyses. As BMA or similar approaches are rapidly becoming mainstream tools in this area, we wish to investigate in detail how the (often almost arbitrarily chosen) prior assumptions may affect our inference.

As a general principle, the effect of not strongly held prior opinions should be minimal. This intuitive sense of a “non-informative” or “ignorance” prior is often hard to achieve, especially when we are dealing with model choice, as opposed to inference within a given model (see, _e.g._ , Kass and Raftery, 1995). At the very least, we should be able to trace the effect in order to inform the analyst which prior settings are more informative than others, and in which direction they will influence the result. “Clever” prior structures are robust, in that they protect the analyst against unintended consequences of prior choices. In this paper, we focus on a general prior structure which encompasses most priors used in the growth regression literature and allows for prior choice in two areas: the choice of the precision factor _g_ in the _g_ -prior and the prior assumptions on the model space. On the latter, we elicit the prior in terms of the prior mean model size _m_ , which is a quantity that analysts may have some subjective prior information on. Other aspects of the prior are typically less interpretable for most applied analysts and would require “automatic” 1
settings. However, these choices have to be reasonable and robust. It is important to stress that the dependence on prior assumptions does not disappear if we make those assumptions implicit rather than explicit. We then merely lull the analyst into a false sense of security. Thus, the claim in SDM that their approach “limits the effect of prior information” has to be taken with extreme caution.

To build priors on the model space, we shall advocate the use of hierarchical priors, since this increases flexibility and decreases the dependence on essentially arbitrary prior assumptions. Theoretical results on the distribution of model size and prior odds allow us to shed some light on the relative merits of the priors on model space. Analytical results for the marginal likelihoods (Bayes factors) are used to infer the model size penalties implicit in the various choices of _g_ and allow for an insightful comparison with the BACE procedure of SDM.

Using three different data sets that have been used in the growth literature, we assess the effect of prior settings for posterior inference on model size, but we also consider the spread of model probabilities over the model space, and the posterior inclusion probabilities of the regressors. The latter is especially critical for this literature, as the relative importance of the regressors as drivers of growth is often the key motivation for the analysis.

By repeatedly splitting the samples into an inference part and a prediction part, we also examine the robustness of the inference with respect to changes to the data set and we assess the predictive performance of the model with the various prior settings.

Section 2 describes the Bayesian model, and Section 3 examines the theoretical consequences of the priors in more detail. Empirical results for three data sets are provided in Section 4 (using the full samples) and Section 5 (using 100 randomly generated subsamples of a given size). The final section concludes and provides recommendations for users of BMA in this literature.

# **2. The Bayesian Model**

In keeping with the literature, we adopt a Normal linear regression model for _n_ observations of growth in per capita GDP, grouped in a vector _y_ , using an intercept, _α_ , and explanatory variables from a set of _k_ possible regressors in _Z_ . We allow for any subset of the variables in _Z_ to appear in the model. This results in 2<sup>_k_</sup> possible models, which will thus be characterized by the selection of regressors. This model space will be denoted by _M_ and we call model _Mj_ the model with the 0 _≤ kj ≤ k_ regressors grouped in _Zj_ , leading to where _ιn_ is a vector of _n_ ones, _βj ∈R_<sup>_kj_</sup> groups the relevant regression coefficients and _σ ∈R_ + is a scale parameter.

For the parameters in a given model _Mj_ , we follow Fern ́andez _et al._ (2001a) and adopt a combination of a “non-informative” improper prior on the common intercept and the scale and a so-called _g_ -prior (see Zellner, 1986) on the regression coefficients, leading to the prior density where _fN_<sup>_q_(</sup><sup>_w|m, V_) denotes the density function of a</sup><sup>_q_-dimensional Normal distribution on</sup><sup>_w_with</sup> mean _m_ and covariance matrix _V_ . The regression coefficients not appearing in _Mj_ are exactly

2
zero, represented by a prior point mass at zero. Of course, we need a proper prior on _βj_ in (2), as an improper prior would not allow for meaningful Bayes factors. The general prior structure in (2), sometimes with small changes, is shared by many papers in the growth regression literature, and also in the more general literature on covariate selection in linear models (see, _e.g._ , Clyde and George, 2004 for a recent survey).

Based on theoretical considerations and extensive simulation results in Fern ́andez _et al._ (2001a), FLS choose to use _g_ = 1 _/_ max _{n, k_<sup>2</sup> _}_ in (2). In the sequel, we shall mainly focus on the two choices for _g_ that underlie this recommendation.

- The first choice, _g_ 0 _j_ = 1 _/n_ , roughly corresponds to assigning the same amount of information to the conditional prior of _β_ as is contained in one observation. Thus, it is in the spirit of the “unit information priors” of Kass and Wasserman (1995) and the original _g_ -prior used in Zellner and Siow (1980). Fern ́andez _et al._ (2001a) show that log Bayes factors using this prior behave asymptotically like the Schwarz criterion (BIC), and George and Foster (2000) show that for known _σ_<sup>2</sup> model selection with this prior exactly corresponds to the use of BIC.

- The second choice is _g_ 0 _j_ = 1 _/k_<sup>2</sup> , which is suggested by the Risk Inflation Criterion of Foster and George (1994). In growth regression, we typically have that _k_<sup>2</sup> _≫ n_ (as is the case in all three examples here), so that the recommendation of Fern ́andez _et al._ (2001a) would lead to the use of _g_ = 1 _/k_<sup>2</sup> .

In this paper, we shall not consider other choices for _g_ , but some authors suggest making _g_ random— _i.e._ , putting a hyperprior on _g_ . In fact, the original Zellner-Siow prior can be interpreted as such, and Liang _et al._ (2005) propose the class of hyper- _g_ priors, which still allow for closed form expressions for the marginal likelihoods.

The prior model probabilities are often specified by _P_ ( _Mj_ ) = _θ_<sup>_kj_</sup> (1 _− θ_ )<sup>_k−kj_</sup> , assuming that each regressor enters a model independently of the others with prior probability _θ_ . Raftery _et al._ (1997), Fern ́andez _et al._ (2001a) and FLS choose _θ_ = 0 _._ 5, which can be considered a benchmark choice—implying that _P_ ( _Mj_ ) = 2<sup>_−k_</sup> and that expected model size is _k/_ 2. The next section will consider the prior on the model space _M_ more carefully.

We use a Markov chain Monte Carlo (MCMC) sampler to deal with the very large model space _M_ (already containing 2 _._ 2 _×_ 10<sup>12</sup> models for the smallest example here with _k_ = 41). Since the posterior odds between any two models are analytically available (see Section 3), this sampler moves in model space alone. Thus, the MCMC algorithm is merely a tool to deal with the practical impossibility of exhaustive analysis of _M_ , by only visiting the models which have non-negligible posterior probability.<sup>1</sup>

> 1 In this paper we use Fortran code based on that used for FLS, but updated to account for data sets with more than 52 regressors, as explained in Ley and Steel (2007). This code can deal with up to 104 regressors, corresponding to a model space _M_ containing 2<sup>104</sup> = 2 _×_ 10<sup>31</sup> models, and is available at http://www.warwick.ac.uk/go/msteel/.

3
# **3. Prior Assumptions and Posterior Inference**

# _3.1. Model prior specification and model size_

In order to specify a prior on model space, consider the indicator variable _γi_ , which takes the value 1 if covariate _i_ is included in the regression and 0 otherwise, _i_ = 1 _, . . . , k_ . Given the probability of inclusion, say _θ_ , _γi_ will then have a Bernoulli distribution: _γi ∼_ Bern( _θ_ ), and if the inclusion of each covariate is independent then the _model size W_ will have a Binomial distribution: This implies that, if we fix _θ_ —as was done, _e.g._ , in FLS and SDM, as in most other studies—the prior model size will have mean _θk_ and variance _θ_ (1 _− θ_ ) _k_ .

Typically, the use of a hierarchical prior increases the flexibility of the prior and reduces the dependence of posterior and predictive results (including model probabilities) on prior assumptions. Thus, making _θ_ random rather than fixing it would seem a sensible extension (see, _e.g._ , Clyde and George, 2004 and Nott and Kohn, 2005). An obvious choice for the distribution of _θ_ is a Beta with hyperparameters _a, b >_ 0, _i.e. θ ∼_ Be( _a, b_ ), leading to the following prior moments for model size, as a function of _k, a_ and _b_ : The generated prior model size distribution is, in fact, called a Binomial-Beta distribution (see Bernardo and Smith, 1994, p. 117), and has the probability mass function In the special case where _a_ = _b_ = 1— _i.e._ , we mix with a uniform prior for _θ_ —we obtain a discrete uniform prior for model size with _P_ ( _W_ = _w_ ) = 1 _/_ ( _k_ + 1) for _w_ = 0 _, . . . , k_ .

This prior depends on two parameters, ( _a, b_ ), and it will facilitate prior elicitation to fix _a_ = 1. This allows for a wide range of prior behavior and generally leads to reasonable prior assumptions, as seen below. It is attractive to elicit the prior in terms of the prior mean model size, _m_ . The choice of _m ∈_ (0 _, k_ ) will then determine _b_ through equation (3), which implies _b_ = ( _k − m_ ) _/m_ .

Thus, in this setting, the analyst only needs to specify a prior mean model size, which is exactly the same information one needs to specify for the case with fixed _θ_ , which should then equal _θ_ = _m/k_ . With this Binomial-Beta prior, the prior mode for _W_ will be at zero for _m < k/_ 2 and will be at _k_ for _m > k/_ 2. The former situation is likely to be of most practical relevance, and then the prior puts most mass on the null model, which reflects a mildly conservative prior stance, where we require some data evidence to favor the inclusion of regressors.

4
For the case of _k_ = 67, Figure 1 contrasts the prior model size distributions<sup>2</sup> with fixed _θ_ (solid lines) and random _θ_ (dashed lines), for two choices for mean model size: _m_ = 7, which is used in SDM, and _m_ = 33 _._ 5, which corresponds to a uniform prior in the random _θ_ case. Clearly, the prior with fixed _θ_ is very far from uniform, even for _m_ = _k/_ 2. Generally, the difference between the fixed and random _θ_ cases is striking: prior model size distributions for fixed _θ_ are very concentrated.

<!-- Start of picture text --> 0.15<br>Fixed Θ, m � 7<br>0.125<br>Fixed Θ, m � 33.5<br>0.1<br>0.075<br>0.05 Random Θ, m � 7<br>0.025 Random Θ, m � 33.5<br>0<br>0 10 20 30 40 50 60<br>Number of Regressors<br>Probability Density<br><!-- End of picture text -->

**Fig. 1.** Prior model size for fixed _θ_ (solid) and random _θ_ (dashed), with _m_ = 7 and _m_ = 33 _._ 5

With random _θ_ and the hyperparameter choices discussed above, the prior variance of model size is then equal to which is roughly equal to _m_<sup>2</sup> when _k ≫ m_ . In fact, we can illustrate the added uncertainty introduced by the hierarchical prior structure on _θ_ by the fact that the variance in (5) multiplies Var[ _W_ ] of the corresponding case with fixed _θ_ = _m/k_ by a factor which, for _k >_ 1 is always larger than one and is an increasing function of _m_ , ranging from 1 (in the limit as _m ↓_ 0) to ( _k_ + 1) _/_ 2 (in the limit as _m ↑ k_ ).

Thus, treating _θ_ as random will typically imply a substantial increase in the prior uncertainty about model size. To assess whether this increase is reasonable, consider Pearson’s coefficient of variation, say CV[ _W_ ], given by the standard deviation divided by the mean. For fixed _θ_ = _k/m_ this coefficient of variation equals

> 2 These distributions are, of course, discrete, but for ease of presentation they are depicted through continuous graphs—the same comment applies throughout the paper.

5
which is a rapidly decreasing function of _m_ and is unity for _m_ = _k/_ ( _k_ + 1), which is often close to one. Thus, for any reasonable prior mean model size, the prior with fixed _θ_ will be far too tight. For example, if we take _m_ = 7 in our applications, where _k_ ranges from 41 to 67, CV[ _W_ ] will range from 0 _._ 344 to 0 _._ 358, which is quite small. For _m_ = _k/_ 2, CV[ _W_ ] = ~~�~~ 1 _/k_ and ranges from 0 _._ 122 to 0 _._ 156, clearly reflecting an unreasonable amount of precision in the prior model size distribution.

For random _θ_ with the hyperprior as described above, we obtain which is also decreasing in _m_ , but is much flatter than the previous function over the range of practically relevant values for _m_ .<sup>3</sup> Taking _m_ = 7 in our applications, CV[ _W_ ] will now range from 0 _._ 900 to 0 _._ 963, which is much more reasonable. For _m_ = _k/_ 2, we now have that CV[ _W_ ] = �( _k_ + 2) _/_ 3 _k_ , which ranges from 0 _._ 586 to 0 _._ 591.

Thus, this hierarchical prior seems quite a sensible choice. In addition, both Var[ _W_ ] and CV[ _W_ ] increase with _k_ for a given _m_ , which also seems a desirable property. This holds for both prior settings; however, lim _k→∞_ CV[ _W_ ] = ~~�~~ ( _m_ + 1) _/m_ for the case with random _θ_ whereas this limit is only ~~�~~ 1 _/m_ for the fixed _θ_ case.

# _3.2. Prior odds_

Posterior odds between any two models in _M_ are given by where _ly_ ( _Mi_ ) is the marginal likelihood, which is discussed in the next subsection. Thus, the prior distribution on model space only affects posterior model inference through the prior odds ratio _P_ ( _Mi_ ) _/P_ ( _Mj_ ). For a prior with a fixed _θ_ = 0 _._ 5 prior odds are equal to one (each model is a priori equally probable). If we fix _θ_ at a different value, these prior odds are thus inducing a prior penalty for the larger model if _θ <_ 0 _._ 5 and favoring the larger model for values of _θ >_ 0 _._ 5. Viewed in terms of the corresponding mean model size, _m_ , we obtain (for _θ_ = _m/k_ ):

> 3 Of course, both CV functions tend to zero for _m ↑ k_ since then all prior mass has to be on the full model. In addition, CV in both cases tends to _∞_ as _m ↓_ 0.

6
from which it is clear that the prior favors larger models if _m > k/_ 2. For the hierarchical Be( _a, b_ ) prior on _θ_ , we obtain the prior model probabilities: Using _a_ = 1 and our prior elicitation in terms of E[ _W_ ] = _m_ as above, we obtain the following prior odds Figure 2 compares the log prior odds induced by the fixed and random _θ_ prior structures, in the situation where _k_ = 67 and for _m_ = 7 _,_ 33 _._ 5 and 50. The graphs indicate the prior odds in favor of a model with _ki_ = 10 versus models with varying _kj_ .

<!-- Start of picture text --> Fixed Θ Random Θ<br>125<br>20<br>100 m � 7<br>75 10 m � 33.5<br>m � 7<br>50<br>0<br>25<br>m � 33.5 �10<br>0<br>�25 �20 m � 50<br>�50 m � 50<br>�30<br>0 10 20 30 40 50 60 0 10 20 30 40 50 60<br>Number of Regressors Number of Regressors<br>Log Prior Odds Log Prior Odds<br><!-- End of picture text -->

**Fig. 2.** Log of Prior Odds: _ki_ = 10 vs varying _kj_

Note that the random _θ_ case always leads to downweighting of models with _kj_ around _k/_ 2, irrespectively of _m_ . This counteracts the fact that there are many more models with _kj_ around _k/_ 2 in the model space _M_ than for _kj_ values nearer to 0 or _k_ .<sup>4</sup> In contrast, the prior with fixed _θ_ does not take the number of models at each _kj_ into account and simply always favors larger models when _m > k/_ 2 and the reverse when _m < k/_ 2. Note also the wider range of values that the log prior odds take in the case of fixed _θ_ .

Thus, the choice of _m_ is critical for fixed _θ_ , but much less so for random _θ_ . The latter prior structure is naturally adaptive to the data observed. This is, again, illustrated by Figure 3, which plots the log of the prior odds of a model with _ki_ = ( _kj −_ 1) regressors versus a model with _kj_ regressors as a function of _kj_ .

> 4 _k_ The number of models with _kj_ regressors in _M_ is given by<sup>�</sup> _kj_ �. For example, with _k_ = 67, we have 1 model with _kj_ = 0 and _kj_ = _k_ , 8 _._ 7 _×_ 10<sup>8</sup> models with _kj_ = 7 and _kj_ = 60 and a massive 1 _._ 4 _×_ 10<sup>19</sup> models with _kj_ = 33 and 34.

7
<!-- Start of picture text --> Fixed Θ Random Θ<br>m � 7 4<br>2<br>2<br>m � 7<br>1<br>0<br>0 m � 33.5 �2 m � 33.5<br>m � 50<br>�1 m � 50 �4<br>10 20 30 40 50 60 0 10 20 30 40 50 60<br>Number of Regressors Number of Regressors<br>Log Prior Odds Log Prior Odds<br><!-- End of picture text -->

**Fig. 3.** Log of Prior Odds: _ki_ = ( _kj −_ 1) vs varying _kj_

Whereas the fixed _θ_ prior always favors the smaller model _Mi_ for _m < k/_ 2, the choice of _m_ for random _θ_ only moderately affects the prior odds, which swing towards the larger model when _kj_ gets larger than approximately _k/_ 2. This means that using the prior with fixed _θ_ will have a deceptively strong impact on posterior model size. This prior does not allow for the data to adjust prior assumptions on mean model size that are at odds with the data, making it a much more risky choice.

# _3.3. Bayes factors_

In the previous subsection we mentioned the marginal likelihood, which is defined as the sampling density integrated out with the prior. The marginal likelihood forms the basis for the Bayes factor (ratio of marginal likelihoods) and can be derived analytically for each model with prior structure (2) on the model parameters. Provided _g_ in (2) does not depend on the model size _kj_ , the Bayes factor for any two models from (1)–(2) becomes: where _Ri_<sup>2istheusualcoefficientofdeterminationformodel</sup><sup>_Mi_,</sup><sup>_i.e._,</sup><sup>_R_</sup> _i_<sup>2=1</sup><sup>_−_[</sup><sup>_y′QX_</sup> _i_<sup>_y/_(</sup><sup>_y−_</sup> _yι_ ̄ _n_ )<sup>_′_</sup> ( _y − yι_ ̄ _n_ )] and we have defined _QA_ = _I − A_ ( _A_<sup>_′_</sup> _A_ )<sup>_−_1</sup> _A_<sup>_′_</sup> and _Xi_ = ( _ιn, Zi_ ), the design matrix of _Mi_ , which is always assumed to be of full column rank. The expression in (6) is the relative weight that the data assign to the corresponding models, and depends on sample size _n_ , the factor _g_ of the _g_ -prior and the size and fit of both models, with the latter expressed through _R_<sup>2</sup> .

Let us compare this with the so-called BACE approach of SDM. The BACE approach is not totally Bayesian, as it is not formally derived from a prior-likelihood specification, but relies on an approximation as sample size, _n_ , goes to infinity (which may not be that realistic in the growth context). In fact, BACE uses the Schwarz approximation to compute the Bayes factor, as was earlier used in Raftery (1995) in a very similar context. From equation (6) in SDM, we get the 8
following Bayes factor: This expression is not that different from the one in our equation (6), provided we take _g_ = 1 _/n_ . In that case, the Bayes factor in (6) becomes: which behaves very similarly to the BACE procedure in (7) for practically relevant values of _n_ (as in the examples here). This will be crucial in explaining the similarity of the results with the FLS and SDM prior settings mentioned before.

It also becomes immediately clear that the necessity of choosing prior settings implicit in using BMA is not really circumvented by the use of BACE, in contrast with the claims in SDM. In fact, BACE implicitly fixes _g_ in the context of our BMA framework. The fact that this is hidden to the analyst does not make the results more robust with respect to this choice. It even carries a substantial risk of conveying a false sense of robustness to the applied analyst.

Now we can examine more in detail how the various prior choices translate into model size penalties. From (6) we immediately see that if we have two models that fit equally well ( _i.e._ , _Ri_<sup>2=</sup><sup>_R_</sup> _j_<sup>2), then the Bayes factor will approximately equal</sup><sup>_g_(</sup><sup>_ki−kj_)</sup><sup>_/_2 (as</sup><sup>_g_tends to be quite small).</sup> If one of the models contains one more regressor, this means that the larger model will be penalized by _g_<sup>1</sup><sup>_/_2</sup> .

For _n_ = 88 and _k_ = 67 (as in the SDM data) this means that the choice of _g_ = 1 _/n_ leads to a Bayes factor of 0.107 and choosing _g_ = 1 _/k_<sup>2</sup> implies a Bayes factor of 0.015. Thus, the model size penalty is much more severe for _g_ = 1 _/k_<sup>2</sup> in the context of these types of data. The size penalty implicit in the BACE procedure is the same as for _g_ = 1 _/n_ .

We can also ask how much data evidence is required to exactly compensate for the effects of prior odds under different specifications. Posterior odds will be unity if the Bayes factor equals the inverse of the prior odds, thus if where the prior odds are given in the previous subsection, as a function of prior mean model size _m_ , as well as _k_ and the sizes of both models.

Typically, we have no control over _n_ and _k_ , but we do need to select _g_ and _m_ . To give more insight into the tradeoff between _g_ and _m_ , Figure 4 plots the contours in ( _g, m_ )-space that correspond to _n_ = 88 _, k_ = 67 _, ki_ = 8 _, kj_ = 7, and _Rj_<sup>2= 0</sup><sup>_._75 for different ratios</sup><sup>_R_</sup> _i_<sup>2</sup><sup>_/R_</sup> _j_<sup>2that would</sup> make the two models equally probable in the posterior. These plots are provided for fixed and random _θ_ model priors with the Bayes factor in (6).

It is obvious from the left figure for the fixed _θ_ case that there is a clear trade-off between _m_ and _g_ : larger _m_ , inducing a smaller size penalty, can be compensated by a small _g_ , which increases 9
<!-- Start of picture text --> Fixed Θ Random Θ<br>40 40<br>FLS<br>30 30<br>20 20<br>10<br>SDM 10<br>0<br>-9 -8 -7 -6 -5 -4 -9 -8 -7 -6 -5 -4<br>log�g� log�g�<br>m m<br><!-- End of picture text -->

**Fig. 4.** Equal Probability Contours for Different Ratios _Ri_<sup>2</sup><sup>_/R_</sup> _j_<sup>2</sup> Left panel shows also the choices of (log( _g_ ) _, m_ ) for FLS and SDM ( _n_ = 88 _, k_ = 67 _, ki_ = 8 _, kj_ = 7, and _Rj_<sup>2= 0</sup><sup>_._75)</sup>

the penalty. In this particular case with _k_ and _n_ corresponding to the SDM data, we notice that the combination of _g_ = 1 _/n_ with _m_ = 7, as implicitly assumed in SDM, is on a contour rather close to the one corresponding to the settings in FLS: _g_ = 1 _/k_<sup>2</sup> with _m_ = 33 _._ 5. These combinations are represented in the left panel. This explains the similarity of the results with the SDM and FLS prior settings, mentioned in the Introduction. If we adopt the hierarchical prior with random _θ_ , the trade-off between _g_ and _m_ almost disappears, as the choice of _m_ now only has a very small role to play.

# **4. Some Illustrative Examples: Complete Sample Results**

In this section we will present posterior results for three data sets that have been analyzed in the literature. All results in this section are based on MCMC runs of 2 million retained drawings after a burn-in of 1 million.

We focus, in particular, on the effect of the prior choices on posterior model size distributions, the spread of the posterior mass over model space, posterior model probabilities and the inclusion of individual regressors.

Results are presented for eight combinations of prior settings, taking _g_ = 1 _/n_ and _g_ = 1 _/k_<sup>2</sup> , _m_ = 7 and _m_ = _k/_ 2 and using either a fixed or a random _θ_ . The choice of _m_ = 7 was motivated by the restriction to models with up to 7 regressors in Levine and Renelt (1992) and exactly 7 regressors in Sala-i-Martin (1997). Choosing _m_ = _k/_ 2 corresponds to a “vague” model size prior, which is, as discussed in Subsection 3.1, uniform for random _θ_ , and symmetric (but far from uniform) for fixed _θ_ .

10
# _4.1. The FLS Data_

We first illustrate the effects of our prior choices using the growth data of FLS. The latter data set contains _k_ = 41 potential regressors to model the average per capita GDP growth over 1960-1992 for a sample of _n_ = 72 countries.<sup>5</sup> As expected, results with _m_ = 41 _/_ 2 = 20 _._ 5, fixed _θ_ and _g_ = 1 _/k_<sup>2</sup> (the FLS settings) are virtually identical to those obtained in FLS on the basis of a chain of the same length. Figure 5 provides a complete picture of the posterior model size distributions (overplotted with the priors) for all eight combinations of prior settings. Summaries in the form of the first two moments are provided in ~~Table 1.~~ <!-- Start of picture text --> Fixed q, m = 7 Fixed q, m = 20.5<br>0.3<br>0.25 g= k 12 g= 1 n 0.2 g= k 12 g= 1 n<br>0.2 0.15<br>0.15<br>0.1<br>0.1<br>0.05<br>0.05<br>Prior<br>Prior<br>0 0<br>0 10 20 30 40 0 10 20 30 40<br>Numberof Regressors Numberof Regressors<br>Random q, m = 7 Random q, m = 20.5<br>0.25 0.25<br>g= 1<br>k 2<br>0.2 g= 1 g= 1 0.2<br>k 2 n<br>g= 1<br>0.15 0.15 n<br>0.1 0.1<br>0.05 0.05 Prior<br>Prior<br>0 0<br>0 10 20 30 40 0 10 20 30 40<br>Numberof Regressors Numberof Regressors<br>ytis ytis<br>n n<br>e e<br>D D<br>ytilib ytilib<br>a a<br>b b<br>Pro Pro<br>ytis ytis<br>n n<br>e e<br>D D<br>ytilib ytilib<br>a a<br>b b<br>Pro Pro<br><!-- End of picture text -->

**Fig. 5.** Model Size: Prior and Posterior Distributions for FLS data

From these results, we immediately notice the striking influence that prior assumptions have on model size in the posterior. Even <mark>if we fx the prior m</mark> ean model size, simply changing between random and fixed _θ_ or choosing a different _g_ can substantially alter the posterior distribution of model size, _W_ . Clearly, influence of the choice of _m_ is massive for fixed _θ_ , whereas its effect is much less severe for the case of random _θ_ . This accords with the fact (discussed in Subsection 3.1)

> 5 The dataset and the code used in FLS are available on the _JAE_ website at http://qed.econ.queensu.ca/jae/.

11
**Table 1.** FLS Data—prior and posterior moments of model size Properties of the chain and the best model

|||_m_|= 7|||_m_=_k/_|2 = 20_._5||
|---|---|---|---|---|---|---|---|---|
||fx|ed_θ_|rand|om_θ_|fxe|d_θ_|rand|om_θ_|
||_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|
|Prior mean|7|7|7|7|20.5|20.5|20.5|20.5|
|Prior st. dev.|2.41|2.41|6.30|6.30|3.20|3.20|12.12|12.12|
|Posterior mean|9.17|6.29|10.76|5.73|19.84|9.91|12.87|6.03|
|Posterior st. dev.|1.54|1.30|2.67|1.65|2.57|1.63|4.43|1.64|
|# Models visited|141,980|20,003|329,694|19,275|522,637|150,255|436,602|23,558|
|# Models covering 50%|1,461|40|3,569|27|9,348|1,598|5,489|36|
|Post. Prob. best model|0.91%|6.22%|0.65%|5.61%|0_._21%|1.24%|0.54%|5.21%|
|_kj_ for best model|9|6|10|5|20|10|10|6|
|Corr. visits and PO|0.992|0.999|0.971|0.999|0.748|0.992|0.922|0.998|
|Prob. mass visited|70.7%|95.2%|39.6%|95.1%|18.4%|69.7%|27.2%|93.7%|

that the prior on model size is very concentrated for the fixed _θ_ case, and the relative robustness with respect to the choice of _m_ noted for random _θ_ in Subsections 3.2 and 3.3. Also, it appears that the effects of prior choices are much more pronounced for the prior with _g_ = 1 _/n_ . In fact, if we adopt _g_ = 1 _/k_<sup>2</sup> with a hierarchical prior on _θ_ , the value chosen for _m_ has very little impact, in marked contrast with the other prior settings.

From Figure 5 we also deduce that the posterior of _W_ can display bimodality if data and prior information are in conflict. This happens when _m_ = 20 _._ 5 and we choose _g_ = 1 _/n_ for the case with random _θ_ . In the corresponding fixed _θ_ case the prior is actually so concentrated that it dominates and the posterior is mostly determined by the prior.

Table 1 also records some key properties of the chain: it is clear that the choice of _g_ = 1 _/n_ always leads to the sampler visiting many more models, indicating that the posterior mass is more spread out over the model space. This is obvious when considering how many of the higher-probability models we need to cover 50% of the probability mass. This varies dramatically with the choice of _g_ , especially for random _θ_ (where we need more than 100 times as many models for _g_ = 1 _/n_ ). If we adopt _g_ = 1 _/n_ , many larger models are visited and the posterior mass assigned to the best model is much smaller than with _g_ = 1 _/k_<sup>2</sup> (roughly by a factor 8). In addition, the best model is much larger for _g_ = 1 _/n_ . All this is in keeping with the smaller penalty for increasing model size that is implied by adopting _g_ = 1 _/n_ (see Subsection 3.3). An important practical consequence of this is that the chain should run longer before convergence is ensured. Table 1 also presents the correlation between model visit frequencies and probabilities computed on the basis of the exact posterior odds of the visited models, which is in excess of 0.99 (indicating excellent convergence) for all cases with _g_ = 1 _/k_<sup>2</sup> . For _g_ = 1 _/n_ , where the model probability is spread more evenly over _M_ , the evidence in favor of convergence is slightly less convincing. For the case with _m_ = 20 _._ 5 and fixed _θ_ we would ideally recommend a longer run.<sup>6</sup> However, even in this case, multiple runs

> 6 It is important for our purposes in this section that run lengths are identical, to ensure comparability of the properties of

12
led to very similar findings. For completeness, Table 1 also displays the estimated total posterior model probability visited by the chain, computed as suggested in George and McCulloch (1997).

For the random _θ_ cases, the specification and the posterior probability of the best model is not much affected by the choice of _m_ . However, changing from _g_ = 1 _/n_ to _g_ = 1 _/k_<sup>2</sup> has a dramatic effect on both.

Finally, the size of the best model varies in between 5 and 20, and is not much affected by _m_ for random _θ_ .

Of course, one of the main reasons for using BMA in the first place is to assess which of the regressors are important for modelling growth. Table 2 presents the marginal posterior inclusion probabilities of all regressors that receive an inclusion probability of over 10% under any of the prior settings. It is clear that there is a large amount of variation in which regressors are identified as important, depending on the prior assumptions.

Whereas three variables (past GDP, fraction Confucian and Equipment investment) receive more than 0.75 inclusion probability and a further two (Life expectancy and the Sub-Saharan dummy) are included with at least probability 0.50 in all cases, there are many differences. If we compare cases that only differ in _m_ , the choice of _g_ = 1 _/n_ with fixed _θ_ leads to dramatic differences in inclusion probabilities: Fraction Hindu, the Labour force size, and Higher education enrollment go from virtually always included with _m_ = 20 _._ 5 to virtually never included with _m_ = 7; the Number of years open economy has the seventh largest inclusion probability for _m_ = 7 and drops to the bottom of the 32 variables shown in the table for _m_ = 20 _._ 5. In sharp contrast, the case with _g_ = 1 _/k_<sup>2</sup> and random _θ_ leads to very similar inclusion probabilities for both values of _m_ .

Finally, note that results for model size, chain behavior and inclusion probabilities are quite similar for the cases where _g_ = 1 _/n_ with fixed _θ_ = 7 _/_ 41 (the preferred implied prior in SDM) and where _g_ = 1 _/k_<sup>2</sup> with _θ_ = 0 _._ 5 (the prior used in FLS). This is in line with the negative trade-off between _g_ and _m_ illustrated in Figure 4, which explains the similarity between empirical results using BACE and the FLS prior on the same data. The same behavior is observed for the other datasets presented in the next subsections.

# _4.2. The Data Set of MP_

MP investigate the role of initial conditions at independence from colonial rule on the economic growth of African countries. They focus on the average growth rate in GDP from 1960 to 1992 and construct a dataset for _n_ = 93 countries with _k_ = 54 covariates, obtained by combining 32 original regressors and 22 interaction dummies.

Table 3 records the main characteristics of model size and the MCMC chain, and illustrates that results are quite close to those with the FLS data, leading to the same conclusions. Probably due to the somewhat larger model space, convergence is now even more problematic for the fixed _θ_ case the chains. Longer chains will typically capture marginally more of the posterior probability, but they will only add models with very small posterior probabilities and this will not affect any of the conclusions.

13
**Table 2.** FLS data—Marginal posterior inclusion probabilities of the covariates

|||_m_|= 7|||_m_=_k/_|2 = 20_._5||
|---|---|---|---|---|---|---|---|---|
||fx|ed_θ_|rand|om_θ_|fx|ed_θ_|rand|om_θ_|
|Regressors|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|
|log GDP in 1960|1.00|0.91|1.00|0.79|1.00|1.00|1.00|0.84|
|Fraction Confucian|0.99|0.94|1.00|0.93|1.00|1.00|1.00|0.94|
|Life expectancy|0.92|0.74|0.95|0.63|1.00|0.95|0.97|0.68|
|Equipment investment|0.95|0.98|0.93|0.98|0.97|0.94|0.94|0.98|
|Sub-Saharan dummy|0.70|0.59|0.78|0.53|1.00|0.75|0.85|0.55|
|Fraction Muslim|0.62|0.29|0.63|0.23|0.43|0.66|0.61|0.27|
|Rule of law|0.41|0.17|0.56|0.15|0.93|0.52|0.67|0.17|
|Number of years open economy|0.56|0.60|0.46|0.54|0.07|0.50|0.36|0.56|
|Degree of capitalism|0.36|0.09|0.50|0.08|0.56|0.47|0.56|0.09|
|Fraction Protestant|0.38|0.23|0.49|0.24|0.47|0.46|0.51|0.24|
|Fraction GDP in mining|0.36|0.08|0.51|0.07|0.94|0.44|0.63|0.08|
|Non-Equipment investment|0.33|0.07|0.47|0.06|0.71|0.43|0.56|0.07|
|Latin American dummy|0.18|0.09|0.23|0.07|0.75|0.19|0.34|0.08|
|Primary school enrollment, 1960|0.19|0.10|0.21|0.08|0.63|0.18|0.29|0.09|
|Fraction Buddhist|0.14|0.05|0.20|0.07|0.25|0.17|0.23|0.07|
|Black market premium|0.11|0.02|0.22|0.01|0.69|0.16|0.34|0.02|
|Fraction Catholic|0.09|0.03|0.13|0.02|0.12|0.11|0.14|0.03|
|Civil liberties|0.09|0.03|0.13|0.02|0.54|0.10|0.22|0.03|
|Fraction Hindu|0.06|0.07|0.18|0.01|0.97|0.10|0.36|0.01|
|Political rights|0.06|0.01|0.09|0.01|0.29|0.07|0.13|0.02|
|Exchange rate distortions|0.06|0.03|0.06|0.02|0.12|0.06|0.08|0.03|
|Age|0.06|0.02|0.07|0.02|0.25|0.06|0.10|0.02|
|War dummy|0.05|0.02|0.06|0.02|0.14|0.05|0.08|0.02|
|Fraction of Pop. Speaking English|0.04|0.01|0.07|0.01|0.42|0.05|0.15|0.01|
|Size labor force|0.04|0.01|0.11|0.01|0.95|0.05|0.28|0.01|
|Ethnolinguistic fractionalization|0.03|0.01|0.09|0.01|0.87|0.03|0.24|0.01|
|Spanish Colony dummy|0.03|0.01|0.06|0.01|0.59|0.03|0.17|0.01|
|French Colony dummy|0.03|0.01|0.05|0.01|0.54|0.03|0.15|0.01|
|Higher education enrollment|0.02|0.01|0.08|0.01|0.91|0.02|0.24|0.01|
|British colony dummy|0.02|0.00|0.04|0.00|0.47|0.02|0.13|0.00|
|Outward orientation|0.02|0.01|0.04|0.01|0.42|0.02|0.12|0.01|
|Public education share|0.02|0.00|0.03|0.00|0.30|0.02|0.08|0.00|

with _g_ = 1 _/n_ and _m_ = 54 _/_ 2 = 27. Again, however, longer runs do not lead to appreciably different conclusions (see footnote 6).

# _4.3. The Data Set of SDM and DW_

SDM and Doppelhofer and Weeks (2006) use a larger data set, and model annual GDP growth per capita between 1960 and 1996 for _n_ = 88 countries as a function of _k_ = 67 potential drivers.<sup>7</sup> Despite the larger model space (the number of models in _M_ is now 1 _._ 5 _×_ 10<sup>20</sup> ), Table 4 shows that posterior model probabilities are more concentrated than in the previous two cases; in fact, even in those cases where many models are visited, 50% of the posterior mass is still accounted for through a rather small number of models. Also, model sizes tend to be smaller; in fact, the posterior mean model size is less than 3 for three of the four cases with _g_ = 1 _/k_<sup>2</sup> . If we adopt a random 7 The data and the code used in SDM are available at http://www.econ.cam.ac.uk/faculty/doppelhofer/ 14
**Table 3.** MP Data—prior and posterior moments of model size Properties of the chain and the best model

|||_m_|= 7|||_m_=_k_|_/_2 = 27||
|---|---|---|---|---|---|---|---|---|
||fxe|d_θ_|rand|om_θ_|fx|ed_θ_|rand|om_θ_|
||_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|
|Prior mean|7|7|7|7|27|27|27|27|
|Prior st. dev.|2.47|2.47|6.57|6.57|3.67|3.67|15.88|15.88|
|Posterior mean|8.68|6.05|9.67|5.42|17.90|9.77|10.37|5.75|
|Posterior st. dev.|1.52|1.16|2.09|1.53|2.44|1.75|2.24|1.48|
|# Models visited|106,041|13,103|194,230|11,864|516,479|135,353|241,082|13,88|
|# Models covering 50%|933|31|1,549|23|3,038|1,353|2,049|28|
|Post. Prob. best model|1.65%|10.70%|1.14%|9.60%|0.49%|1.28%|0.90%|9.01%|
|_kj_ for best model|8|6|8|4|19|8|8|6|
|Corr. visits and PO|0.990|0.998|0.973|0.999|0.251|0.985|0.963|0.999|

**Table 4.** SDM Data—prior and posterior moments of model size Properties of the chain and the best model

|||_m_|= 7|||_m_=_k/_|2 = 33_._5||
|---|---|---|---|---|---|---|---|---|
||fxe|d_θ_|rand|om_θ_|fxe|d_θ_|rand|om_θ_|
||_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|
|Prior mean|7|7|7|7|33.5|33.5|33.5|33.5|
|Prior st. dev.|2.50|2.50|6.74|6.74|4.09|4.09|19.63|19.63|
|Posterior mean|6.17|2.84|5.26|2.35|14.20|7.00|5.73|2.40|
|Posterior st. dev.|1.50|0.85|1.86|0.59|1.64|1.44|1.91|0.62|
|# Models visited|102,852|6,141|101,642|2,167|606,792|130,776|129,929|2,594|
|# Models covering 50%|590|4|248|1|485|744|383|1|
|Post. Prob. best model|6.63%|37.72%|5.74%|66.42%|1.43%|6.58%|5.22%|63.64%|
|_kj_ for best model|6|2|2|2|13|6|6|2|
|Corr. visits and PO|0.996|1.000|0.996|1.000|0.029|0.996|0.996|1.000|

_θ_ and _g_ = 1 _/k_<sup>2</sup> , the choice for _m_ has little effect. The twenty best models are the same for both values of _m_ , with very similar posterior probabilities and only slight differences in ordering. The best model (with over 60% posterior probability) in these cases as well as with two other settings, is the model with only the East Asian dummy and Malaria prevalence as regressors. The same model is also second best for the random _θ_ case with _m_ = 33 _._ 5 and _g_ = 1 _/n_ , but is well down the ordering (receiving less than 0.25% of probability) if we fix _θ_ with _m_ = 33 _._ 5. In fact, in the latter case with _g_ = 1 _/n_ , the regressors East Asian dummy and Malaria prevalence never appear together in any model with posterior probability over 0.25%. This further illustrates the huge impact of simply changing between the fixed and random _θ_ cases.

Convergence is excellent, except for the case with fixed _θ_ , _m_ = 33 _._ 5 and _g_ = 1 _/n_ , as in the other examples. The difference in convergence between the cases is even more striking than with the 15
previous examples, and it appears that the case with convergence problems struggles to adequately describe the posterior distribution on the model space _M_ . While most of the mass is covered by a relatively small number of models, the prior assumptions induce a very fat tail of models with little but nonnegligible mass. However, inference on most things of interest, such as the regressor inclusion probabilities is not much affected by running longer chains. For all other prior settings, the very large model space is remarkably well explored by the MCMC chain.

# **5. Robustness and Predictive Analysis: Results from 100 Subsamples**

In the previous section we have illustrated that the choice of, perhaps seemingly innocuous, prior settings can have a dramatic impact on the posterior inference resulting from BMA. Posterior model probabilities and identification of the most important regressors can strongly depend on the prior settings we use in our analysis of growth regressions through BMA.

We now address the issue of whether small changes to the data set would result in large changes in inference— _i.e._ , data robustness. In addition, we want to assess the predictive performance of the model under various prior settings. We will use the same device to investigate both issues, namely the partition of the available data into an inference subsample and a prediction subsample. We will then use the various inference subsamples for the evaluation of robustness and assess prediction on the basis of how well the predictive distribution based on the inference subsamples captures the corresponding prediction subsamples.

We take random partitions of the sample, where the size of the prediction subsample is fixed at 15% of the total number of observations (rounded to an integer), leaving 85% of the sample to conduct inference with. We generate random prediction subsamples of a fixed size by using the algorithm of McLeod and Bellhouse (1983). We use 100 random partitions and compute the posterior results through an MCMC chain of 500,000 drawings with a burn-in of 100,000 for each partition. This led to excellent convergence, with the exception of the cases with _θ_ fixed at 0 _._ 5 and _g_ = 1 _/n_ . Thus, for this combination we have used a chain of length 1,000,000 after a burn-in of 500,000, which leads to reliable results.<sup>8</sup> To increase comparability, we use the same partitions for all prior settings.

# _5.1. Robustness_

Figure 6 indicates the distribution of the posterior mean model size across 100 inference subsamples of the FLS data (left panel). Values corresponding to the full sample are indicated by vertical lines. The right panel of the same Figure shows the posterior inclusion probabilities of the ten regressors with highest posterior inclusion probabilities in the full-sample analysis with the FLS prior setting— _i.e._ , _θ_ = 0 _._ 5 and _g_ = 1 _/_ max _{n, k_<sup>2</sup> _}_ .

A striking characteristic of both panels is the sensitivity of the results to the choice of _m_ for the fixed _θ_ priors, whereas the effect of _m_ is very small for the cases with random _θ_ . The choice of _g_ ,

> 8 In fact, the results are very close to those obtained with 100,000 burn-in and 500,000 retained drawings, with the only noticeable difference in the maximum LPS values.

16
however, always matters: _g_ = 1 _/k_<sup>2</sup> generally leads to smaller mean model sizes and results in very different conclusions on which regressors are important than _g_ = 1 _/n_ , especially for random _θ_ .

Given each choice of prior, the results vary quite a bit across subsamples, especially for the inclusion probabilities where the interquartile ranges can be as high as 60%. It is interesting how the combinations of _g_ = 1 _/n_ with fixed _θ_ = 7 _/_ 41 (the setting implicitly favored in SDM) and _g_ = 1 _/k_<sup>2</sup> with _θ_ = 0 _._ 5 (the prior of FLS) lead to very similar results, both for model sizes and inclusion probabilities, as also noted in Section 4. Overall, the difference between fixed and random _θ_ cases is small for _m_ = 7, but substantial for _m_ = _k/_ 2.

For the MP data, results on posterior model size are quite similar as for the FLS dataset, and we can draw the same conclusions on the basis of the inclusion probabilities.

Finally, Figure 7 shows the robustness results for the SDM data set, where we have _k_ = 67 potential regressors. As before, the choice of _m_ critically affects the fixed _θ_ results, but not the ones for random _θ_ . Differences between fixed and random _θ_ cases are large for _m_ = _k/_ 2, but relatively small for _m_ = 7, as with both previous datasets. Again, the choice of _g_ always affects the results, and inference using _g_ = 1 _/n_ in combination with _θ_ = 7 _/_ 67 is quite similar to that using _g_ = 1 _/k_<sup>2</sup> with _θ_ = 0 _._ 5, for both model size and inclusion probabilities. For _g_ = 1 _/k_<sup>2</sup> inference on model size is quite concentrated on small values, with the exception of the case with fixed _θ_ and _m_ = 33 _._ 5.

Even though many results are not very robust with respect to the particular choice of subsample, it is clear that the differences induced by the various prior assumptions largely persist if we take into account small changes to the data.

17
# Fixed _θ_

<!-- Start of picture text --> m = 7, g = 1 n m = 7, g = 1 k 2 m = 7, g = 1 n m = 7, g = 1 k 2<br>G G G G GG GG G G GG GG GGG GGG GGGG<br>GGG GG G GG G G G G GGGG G GG GG GG GGG G<br>G G G GG G G GG G GG GG GGGG G GG G GGG G GGGG G<br>5 10 15 20 5 10 15 20<br>model size model size<br>m = 20.5, g = 1 n m = 20.5, g = 1 k 2 m = 20.5, g = 1 n m = 20.5, g = 1 k 2<br>G G G G G G GG G GG G G G GG G GG GGGG G G G G GGGG GGG G GG G G GG G<br>G GG GGG G GGGGGGGGGGG GGGGGGG G GGGGGGGG GGGGG GG GGGGGG GGGGG<br>5 10 15 20 5 10 15 20<br>model size model size<br>80 80<br>0.4 0.4<br>60 60<br>0.2 0.2 40 40<br>20 20<br>0.0 0.0 0 0<br>100<br>80 80<br>0.4 0.4<br>60 60<br>0.2 0.2 40 40<br>20 20<br>0.0 0.0 0<br><!-- End of picture text -->

# Random _θ_

<!-- Start of picture text --> m = 7, g = 1 n m = 7, g = 1 k 2 m = 7, g = 1 n m = 7, g = 1 k 2<br>G G GGGGGG GGGGGGGGGGGG GGGGG GGG GGG GGGG G GGGGGG GGG G GGGGGGGGGGGGGGGG G GG G G GGGGGGGGGG G G GGG G GGGG G G G GGGGGGG<br>5 10 15 20 5 10 15 20<br>model size model size<br>m = 20.5, g = 1 n m = 20.5, g = 1 k 2 m = 20.5, g = 1 n m = 20.5, g = 1 k 2<br>GG GGGGG G G G G G G G GG GG GGG G G G G GGG<br>GGG GGGGGG GGGGGGG GGG G GGG GGGGGGGGGGGGGGGGGGGG G G GGGGGGGGGGG GGGG G GGG G GGGG G G<br>5 10 15 20 5 10 15 20<br>model size model size<br>100<br>80 80<br>0.4 0.4<br>60 60<br>0.2 0.2 40 40<br>20 20<br>0.0 0.0 0 0<br>100<br>80 80<br>0.4 0.4<br>60 60<br>0.2 0.2 40 40<br>20 20<br>0.0 0.0 0 0<br><!-- End of picture text -->

**Fig. 6.** Distribution of mean model size and posterior inclusion probabilities of the first ten regressors (FLS data, 100 inference subsamples)

18
# Fixed _θ_

<!-- Start of picture text --> m = 7, g = 1 n m = 7, g = 1 k 2 m = 7, g = 1 n m = 7, g = 1 k 2<br>GG GG G G G G G G G GG G GGGGG GG G G GGGG G GGGGGGGGGGG GGG G GGGG GGGGG GGG<br>G G G GG GGG G GGG G GG GG GG G G GG GGGG G G G GG G G GGG GG G G G G<br>5 10 15 20 25 5 10 15 20 25<br>model size model size<br>m = 33.5, g = 1 n m = 33.5, g = 1 k 2 m = 33.5, g = 1 n m = 33.5, g = 1 k 2<br>GGGG G G GGG GG G GGGGG GGG G G G G G G G G G G G GGGGGG GGG G G GG G G<br>G G G G GG GG GG GGGGGGG<br>G<br>G GGG G G G GG G G GG GG<br>5 10 15 20 25 5 10 15 20 25<br>model size model size<br>Random θ<br>m = 7, g = 1 n m = 7, g = 1 k 2 m = 7, g = 1 n m = 7, g = 1 k 2<br>G<br>GGG GGG GG GG G GG G G G G G GGGGG G G GGGGGGG G G GGGGG G GGGGGGG G G G GGGGGGGGGG G G GG G GGGGGG GG G GG GG GGGGG G G G G G G GGGGGGGG G G G G G GG G G G G G<br>5 10 15 20 25 5 10 15 20 25<br>model size model size<br>m = 33.5, g = 1 n m = 33.5, g = 1 k 2 m = 33.5, g = 1 n m = 33.5, g = 1 k 2<br>G<br>G G G G G G G G<br>G GG G G GGG G G GG G G<br>G G GGGG GG GG G GGGG G G G GG G G G GGGG G GG G G GG G GGG G GGGG G G G G GGG G G G G G G G G G GGGGGGG G GGGG G G GGGG GG G G<br>5 10 15 20 25 5 10 15 20 25<br>model size model size<br>2.0 2.0<br>80 80<br>1.5 1.5<br>60 60<br>1.0 1.0 40 40<br>0.5 0.5 20 20<br>0.0 0.0 0 0<br>2.0 2.0<br>80 80<br>1.5 1.5<br>60 60<br>1.0 1.0 40 40<br>0.5 0.5 20 20<br>0.0 0.0 0 0<br>2.0 2.0<br>80 80<br>1.5 1.5<br>60 60<br>1.0 1.0 40 40<br>0.5 0.5 20 20<br>0.0 0.0 0 0<br>2.0 2.0<br>80 80<br>1.5 1.5<br>60 60<br>1.0 1.0 40 40<br>0.5 0.5 20 20<br>0.0 0.0 0 0<br><!-- End of picture text -->

**Fig. 7.** Distribution of mean model size and posterior inclusion probabilities of the first ten regressors (SDM data, 100 inference subsamples)

19
# _5.2. Prediction_

In order to compare the predictive performance associated with the various prior choices, we turn now to the task of predicting the observable, growth, given the regressors. Of course, the predictive distribution is also derived through model averaging, as explained in, _e.g._ , FLS. As a measure of how well each model predicts the retained observations, we use the log predictive score, which is a strictly proper scoring rule, described in FLS. In the case of i.i.d. sampling, LPS can be given an interpretation in terms of the Kullback-Leibler divergence between the actual sampling density and the predictive density (see Fern ́andez _et al._ 2001a), and smaller values indicate better prediction performance.

We compare the predictions based on: _(i)_ BMA, _(ii)_ the best model (the model with the highest posterior probability), _(iii)_ the full model (with all _k_ regressors), and _(iv)_ the null model (with only the intercept).

Panel A in Table 5 summarizes our findings for the FLS data: the entries indicating “best” or “worst” model or how often a model is beaten by the null are expressed in percentages of the 100 samples for that particular prior setting.

- _(i)_ BMA—The predictive performance of BMA is much superior to that of the other procedures— which corroborates evidence in _e.g._ Raftery _et al._ , 1997, Fern ́andez _et al._ , 2001a and FLS. It is never the worst predictor and leads to the best predictions in more than half of the sampled cases (with the exception of the prior with fixed _θ_ = 0 _._ 5 and _g_ = 1 _/n_ ).

- _(ii)_ Best—Basing predictions solely on the model with highest posterior probability is clearly a lot worse: it almost never gives the best prediction and leads to the worst prediction in 18 to 46% of the cases; moreover, it is beaten by the simple null model in more than 35% of the cases.

- _(iii)_ Full—The use of the full model can lead to good forecasts, but is very risky, as it also has a substantial probability of delivering the worst performance: in fact, for _g_ = 1 _/k_<sup>2</sup> the proportion of the latter always exceeds the fraction of best performances. This behavior is also illustrated by the fact that _{_ Min _,_ Mean _,_ Max _}_ for LPS of the full model is _{_ 0 _._ 74 _,_ 1 _._ 77 _,_ 3 _._ 77 _}_ for _g_ = 1 _/n_ and _{_ 0 _._ 67 _,_ 2 _._ 15 _,_ 5 _._ 82 _}_ for _g_ = 1 _/k_<sup>2</sup> . For comparison, the null model leads to _{_ 1 _._ 67 _,_ 2 _._ 05 _,_ 2 _._ 72 _}_ .

Having established that BMA is the strategy to adopt for prediction, we can focus on the forecast performance of BMA to compare the predictive ability across the different prior settings. Mean values of LPS (over all 100 samples) are not that different, but the maximum values indicate that fixing _θ_ at 0.5 is the most risky strategy. This suggests using random _θ_ . In addition, BMA always performs better with respect to the other prediction strategies for _g_ = 1 _/k_<sup>2</sup> . Indeed, the worst case scenario for BMA appears to be _θ_ = 0 _._ 5 with _g_ = 1 _/n_ . Finally, the choice of _m_ almost leaves the random _θ_ results unaffected, but has a substantial effect on the cases with fixed _θ_ , in line with our expectations.

Panel B in Table 5 presents the same quantities for the MP data (where _k_ = 54), and the _{_ Min _,_ Mean _,_ Max _}_ values for LPS of the full model are _{_ 1 _._ 13 _,_ 2 _._ 55 _,_ 6 _._ 05 _}_ for _g_ = 1 _/n_ and _{_ 1 _._ 12 _,_ 2 _._ 79 _,_ 7 _._ 48 _}_ for _g_ = 1 _/k_<sup>2</sup> , whereas the null model leads to _{_ 1 _._ 70 _,_ 2 _._ 03 _,_ 2 _._ 63 _}_ .

The superiority of BMA is even more pronounced, with the prior setting _θ_ = 0 _._ 5 with _g_ = 1 _/n_ again being the least favorable for BMA. The performance of the full model is now considerably 20
**Table 5.** Predictive performance: Three datasets (100 subsamples)

||||||**Panel A.**|FLS Data||||
|---|---|---|---|---|---|---|---|---|---|
||||_m_|= 7|||_m_=|_k/_2||
|||fxe|d_θ_|rand|om_θ_|fxe|d_θ_|rand|om_θ_|
|||_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|_g_ = 1_/n_|_g_ = 1_/k_<sup>2</sup>|
||Best|59|65|54|66|48|63|54|67|
|BMA|Worst|0|0|0|0|0|0|0|0|
|(%)|Beaten byNull|7|6|11|6|19|11|11|6|
||Best|1|2|1|0|3|2|2|0|
|Best model|Worst|28|18|34|18|46|31|36|17|
|(%)|Beaten bynull|35|35|41|36|52|47|45|36|
||Best|34|28|37|30|37|24|36|29|
|Full model|Worst|16|42|15|41|8|31|15|42|
|(%)|Beaten bynull|25|45|25|45|25|45|25|45|
||Minimum|1.12|1.19|1.12|1.21|0.86|1.11|1.11|1.20|
|LPS of BMA|Mean|1.58|1.61|1.61|1.64|1.65|1.63|1.61|1.63|
||Maximum|2.57|2.52|2.67|2.53|2.76|2.85|2.64|2.47|
||St. dev.|0.30|0.26|0.33|0.25|0.42|0.37|0.34|0.25|
||||||**Panel B.**|MP Data||||
||Best|71|70|75|73|61|78|73|72|
|BMA|Worst|0|0|0|0|0|0|0|1|
|(%)|Beaten bynull|15|16|15|18|32|15|16|18|
||Best|10|8|7|2|2|5|7|3|
|Best model|Worst|18|4|19|6|37|18|20|5|
|(%)|Beaten bynull|42|35|42|33|63|49|43|31|
||Best|4|7|3|9|6|2|4|9|
|Full model|Worst|61|75|59|75|46|66|59|75|
|(%)|Beaten bynull|71|77|71|77|71|77|71|77|
||Minimum|1.15|1.30|1.14|1.39|1.05|1.12|1.13|1.38|
|LPS of BMA|Mean|1.70|1.74|1.71|1.77|1.87|1.72|1.72|1.77|
||Maximum|3.34|3.46|3.26|3.53|3.60|3.31|3.23|3.54|
||St. dev.|0.39|0.37|0.40|0.35|0.51|0.42|0.40|0.36|
||||||**Panel C.**S|DM Data||||
||Best|69|53|64|48|48|63|66|46|
|BMA|Worst|0|0|0|0|0|0|0|0|
|(%)|Beaten bynull|15|11|16|10|51|16|16|10|
||Best|16|36|20|42|1|21|18|44|
|Best model|Worst|3|1|3|0|12|2|3|0|
|(%)|Beaten bynull|38|20|31|19|79|39|32|18|
||Best|0|0|0|0|0|0|0|0|
|Full model|Worst|97|99|97|100|88|98|97|100|
|(%)|Beaten bynull|99|100|99|100|99|100|99|100|
||Minimum|1.36|1.41|1.39|1.43|1.22|1.33|1.38|1.43|
|LPS of BMA|Mean|1.78|1.75|1.78|1.75|2.18|1.81|1.79|1.74|
||Maximum|3.38|2.67|3.31|2.53|3.70|3.63|3.38|2.51|
||St. dev.|0.34|0.26|0.32|0.25|0.60|0.38|0.34|0.25|

21
worse than for the FLS data, as this model quite often leads to the worst behavior and is soundly beaten by the much more conservative null model. This seems in line with the fact that the number of regressors, _k_ , is now larger, so the full model is even more overparameterized than in the FLS case. Considering the LPS values for BMA, priors with _g_ = 1 _/n_ seem to have a slight edge, except for the case with a fixed _θ_ = 0 _._ 5. Finally, the choice of _m_ is again virtually immaterial for the random _θ_ cases, and affects those with fixed _θ_ .

Finally, Panel C in Table 5 collects prediction results for the SDM data, where the number of regressors is even larger with _k_ = 67. This is immediately felt in the behavior of the full model, which is now virtually always the worst model. The _{_ Min _,_ Mean _,_ Max _}_ values for LPS of the full model are _{_ 2 _._ 53 _,_ 5 _._ 16 _,_ 11 _._ 75 _}_ for _g_ = 1 _/n_ and _{_ 2 _._ 65 _,_ 6 _._ 95 _,_ 23 _._ 21 _}_ for _g_ = 1 _/k_<sup>2</sup> , whereas the null model leads to _{_ 1 _._ 72 _,_ 2 _._ 08 _,_ 3 _._ 09 _}_ .

As before, the case with _θ_ = 0 _._ 5 and _g_ = 1 _/n_ leads to the worst BMA performance, allowing the null model to beat it more than half of the time! This is due to those cases where prediction is relatively difficult for which the conservative null model performs better. Throughout, the choice of _g_ = 1 _/k_<sup>2</sup> does better in avoiding large values of LPS, reducing the cases for which BMA is beaten by the null model. On the other hand, it is more conservative and lowers the percentage of cases where BMA leads to the best predictive behavior. As we know from Subsection 3.3, the choice of _g_ = 1 _/k_<sup>2</sup> implies a larger size penalty, and thus stays closer to the rather conservative null model. As in the previous examples, the effect of _m_ is far larger for fixed _θ_ than for random _θ_ . For the combinations of random _θ_ with _g_ = 1 _/k_<sup>2</sup> BMA seems less dominant than for the other cases. This is simply a consequence of the fact that the best model then accounts for more than 60% of the probability mass (see Table 4), so that the best model also predicts very well here (albeit not quite as well as BMA). In fact, either BMA or the best model predict best in 90 of the 100 samples, while the null model does best for the remaining 10 samples.

If we measure robustness as in Subsection 5.1 by the standard deviation of the LPS values for BMA, Table 5 also shows us that cases with random _θ_ and _g_ = 1 _/k_<sup>2</sup> are the most robust. Note that differences in robustness can be quite large, and the case with fixed _θ_ = 0 _._ 5 and _g_ = 1 _/n_ stands out as the least robust combination by far.

To summarize the predictive behavior in terms of the choice of prior setting, we can state that a random _θ_ prior seems preferable in view of the lack of sensitivity to _m_ . Also, the combination of fixed _θ_ = 0 _._ 5 and _g_ = 1 _/n_ is to be avoided as it can lead to relatively bad forecasting behavior and BMA is not as dominant as it is under other priors. For these other priors there seems no clear guidance for the choice of _g_ on the basis of predictive behavior alone.

22
# **6. Concluding Remarks and Recommendations**

The theoretical and empirical evidence provided above shows the critical importance of prior assumptions for BMA: it clearly matters what prior settings we choose. The previously noted similarity of results with BACE and FLS prior settings turns out to be a fluke rather than an indication of prior robustness. Making certain prior assumptions implicit (as the choice of _g_ = 1 _/n_ in BACE) does not, in our view, constitute an improvement over a fully explicit Bayesian analysis and can easily lead to a false sense of prior robustness.

A first clear recommendation on the prior structure is to use random _θ_ rather than fixed _θ_ , since the hierarchical prior is much less sensitive to the (often rather arbitrary) choice of prior mean model size, _m_ . Only in the unlikely situation when you really have very strong prior information on model size can a fixed _θ_ prior be defensible. Therefore, we strongly discourage the use of the fixed _θ_ prior as a “non-informative” prior, as it has clearly been shown to be quite informative.

Secondly, we would recommend to avoid the choice of _g_ = 1 _/n_ , which implies a fairly small model size penalty and can, thus, result in convergence problems (with a very long tail of relatively unimportant models) and has also displayed more sensitivity to _m_ than the alternative _g_ = 1 _/k_<sup>2</sup> . In particular, we strongly advise against choosing _g_ = 1 _/n_ with fixed _θ_ = 0 _._ 5 as this combination can lead to relatively bad predictions and the superiority of BMA (which has been shown to be the best procedure to use for prediction) is less pronounced.

In conclusion: for growth-regression or other linear regression settings where we have a fairly large number of potential regressors with relatively few observations (where _k < n_ but of the same order of magnitude, so that _k_<sup>2</sup> _≫ n_ ), we would recommend to use the prior structure in (2) with _g_ = 1 _/k_<sup>2</sup> for any given model. Also, for the prior over models we strongly advise the use of the hierarchical prior on _θ_ described in Subsection 3.1, whenever analysts have no really strong prior information on model size (which will typically be the case for growth regression). In that situation, the actual choice of the prior mean model size, _m_ , will almost not matter, although we would, of course, advise to use a reasonable value for _m_ .

23
# **7. References**

Bernardo, J.M., and A.F.M. Smith (1994), _Bayesian Theory_ , Chicester: John Wiley.

Brock, W., and S. Durlauf (2001), “Growth Empirics and Reality,” _World Bank Economic Review_ , 15: 229–72.

Brock, W., S. Durlauf and K. West (2003), “Policy Evaluation in Uncertain Economic Environments,” _Brookings Papers of Economic Activity_ , 235–322.

Clyde, M.A., and E.I. George (2004), “Model Uncertainty,” _Statistical Science_ , 19, 81–94.

Doppelhofer, G., and M. Weeks (2006), “Jointness of Growth Determinants,” unpublished (University of Cambridge: mimeo).

Fern ́andez, C., E. Ley and M.F.J. Steel (2001a), “Benchmark Priors for Bayesian Model Averaging,” _Journal of Econometrics_ , 100: 381–427.

Fern ́andez, C., E. Ley and M.F.J. Steel (2001b), “Model Uncertainty in Cross-Country Growth Regressions,” _Journal of Applied Econometrics_ , 16: 563–76.

- Foster, D.P., and E.I. George (1994), “The Risk Inflation Criterion for multiple regression,” _Annals of Statistics_ , 22, 1947–1975.

George, E.I., and D.P. Foster (2000), “Calibration and Empirical Bayes variable selection,” _Biometrika_ , 87, 731–747. George, E.I., and R.E. McCulloch (1997), “Approaches for Bayesian Variable Selection,” _Statistica Sinica_ , 7: 339–373.

- Hoeting, J.A., D. Madigan, A.E. Raftery and C.T. Volinsky (1999), “Bayesian model averaging: A tutorial,” _Statistical Science_ 14: 382–401.

- Kass, R.E. and A.E. Raftery (1995), “Bayes factors”, _Journal of the American Statistical Association_ 90: 773–795.

- Kass, R.E. and L. Wasserman (1995), “A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion,” _Journal of the American Statistical Association_ , 90, 928-934.

- Liang, F., R. Paulo, G. Molina, M.A. Clyde, and J.O. Berger (2005), “Mixtures of _g_ -priors for Bayesian Variable Selection,” ISDS Discussion Paper 2005-12, Duke University.

- Le ́on-Gonz ́alez, R. and D. Montolio (2004), “Growth, Convergence and Public Investment: A BMA Approach,” _Applied Economics_ , 36: 1925–36.

- Levine, R., and D. Renelt (1992), “A Sensitivity Analysis of Cross-Country Growth Regressions,” _American Economic Review_ , 82: 942–963.

- Ley, E. and M.F.J. Steel (2007), “Jointness in Bayesian Variable Selection with Applications to Growth Regression,” _Journal of Macroeconomics_ , doi:10.1016/j.jmacro.2006.12.002.

- Macleod, A.I. and D.R. Bellhouse (1983), “A Convenient Algorithm for Drawing a Simple Random Sample,” _Applied Statistics_ , 32, 182–184.

- Nott, D.J. and R. Kohn (2005), “Adaptive Sampling for Bayesian Variable Selection,” _Biometrika_ , 92, 747–763

- Masanjala, W. and C. Papageorgiou (2005), “Initial Conditions, European Colonialism and Africa’s Growth,” unpublished (Baton Rouge: Department of Economics, Louisiana State University)

- Raftery, A.E. (1995), “Bayesian Model Selection in Social Research,” _Sociological Methodology_ , 25: 111–63.

- Raftery, A.E., D. Madigan, and J. A. Hoeting (1997), “Bayesian Model Averaging for Linear Regression Models,” _Journal of the American Statistical Association_ , 92: 179–91.

- Sala-i-Martin, X.X. (1997), “I Just Ran Two Million Regressions,” _American Economic Review_ , 87: 178–183.

- Sala-i-Martin, X.X., G. Doppelhofer and R.I. Miller (2004), “Determinants of long-term growth: A Bayesian averaging of classical estimates (BACE) approach.” _American Economic Review_ 94: 813–835.

Tsangarides, C.G. (2005) “Growth Empirics under Model Uncertainty: Is Africa Different?,” unpublished (Washington DC: IMF Working Paper: 05/18).

- Zellner, A. (1986), On assessing prior distributions and Bayesian regression analysis with _g_ -prior distributions”, in _Bayesian Inference and Decision Techniques: Essays in Honour of Bruno de Finetti_ , eds. P.K. Goel and A. Zellner, Amsterdam: North-Holland, pp. 233–243.

- Zellner, A. and Siow, A. (1980), “Posterior odds ratios for selected regression hypotheses,” (with discussion) in _Bayesian Statistics_ , eds. J.M. Bernardo, M.H. DeGroot, D.V. Lindley and A.F.M. Smith, Valencia: University Press, pp. 585–603.

24