Empirical Bayes method
Empirical Bayes methods are procedures for statistical inference in which the prior distribution is estimated from the data. This approach stands in contrast to standard Bayesian methods, for which the prior distribution is fixed before any data are observed. Despite this difference in perspective, empirical Bayes may be viewed as an approximation to a fully Bayesian treatment of a hierarchical model wherein the parameters at the highest level of the hierarchy are set to their most likely values, instead of being integrated out. Empirical Bayes, also known as maximum marginal likelihood, represents one approach for setting hyperparameters . Introduction. Empirical Bayes methods can be seen as an approximation to a fully Bayesian treatment of a hierarchical Bayes model. In, for example, a two-stage hierarchical Bayes model, observed data formula_1 are assumed to be generated from an unobserved set of parameters formula_2 according to a probability distribution formula_3. In turn, the parameters formula_4 can be considered samples drawn from a population characterised by hyperparameters formula_5 according to a probability distribution formula_6. In the hierarchical Bayes model, though not in the empirical Bayes approximation, the hyperparameters formula_5 are considered to be drawn from an unparameterized distribution formula_8. Information about  a particular quantity of interest formula_9 therefore comes not only from the properties of those data which directly depend on it, but also from the properties of the population of parameters formula_10 as a whole, inferred from the data as a whole, summarised by the hyperparameters formula_11. Using Bayes' theorem, In general, this integral will not be tractable analytically or symbolically and must be evaluated by numerical methods. Stochastic (random) or deterministic approximations may be used. Example stochastic methods are Markov Chain Monte Carlo and Monte Carlo sampling. Deterministic approximations are discussed in quadrature. Alternatively, the expression can be written  as and the term in the integral can in turn be expressed as These suggest an iterative scheme, qualitatively similar in structure to a Gibbs sampler, to evolve successively improved approximations to formula_15 and formula_16. First, calculate an initial approximation to formula_15 ignoring the formula_18 dependence completely; then calculate an approximation to formula_16 based upon the initial approximate distribution of formula_15; then use this formula_16 to update the approximation for formula_15; then update formula_16; and so on. When the true distribution formula_16 is sharply peaked, the integral determining formula_15 may be not much changed by replacing the probability distribution over  formula_11 with a point estimate formula_27 representing the distribution's peak (or, alternatively, its mean), With this approximation, the above iterative scheme becomes the EM algorithm. The term "Empirical Bayes" can cover a wide variety of methods, but most can be regarded as an early truncation of either the above scheme or something quite like it. Point estimates, rather than the whole distribution, are typically used for the parameter(s) formula_11. The estimates for formula_27 are typically made from the first approximation to formula_15 without subsequent refinement. These estimates for formula_27 are usually made without considering an appropriate prior distribution for  formula_18. Point estimation. Robbins method : non-parametric empirical Bayes (NPEB). Robbins considered a case of sampling from a mixed distribution, where probability for each formula_34 (conditional on formula_35) is specified by a Poisson distribution, where formula_37 denotes "number of". (See also Good–Turing frequency estimation.) Suppose each customer of an insurance company has an "accident rate" Θ and is insured against accidents; the probability distribution of Θ is the underlying distribution, and is unknown. The number of accidents suffered by each customer in a specified time period has a Poisson distribution with expected value equal to the particular customer's  accident rate. The actual number of accidents experienced by a customer is the observable quantity. A crude way to estimate the underlying probability distribution of the accident rate Θ is to estimate the proportion of members of the whole population suffering 0, 1, 2, 3, ... accidents during the specified time period as the corresponding proportion in the observed random sample. Having done so, it is then desired to predict the accident rate of each customer in the sample. As above, one may use the conditional expected value of the accident rate Θ given the observed number of accidents during the  baseline period. Thus, if a customer suffers six accidents during the baseline period, that customer's estimated accident rate is 7 × [the proportion of the sample who suffered 7 accidents] / [the proportion of the sample who suffered 6 accidents]. Note that if the proportion of people suffering "k" accidents is a decreasing function of "k", the customer's predicted accident rate will often be lower than their observed number of accidents. This shrinkage effect is typical of empirical Bayes analyses. Parametric empirical Bayes. If the likelihood and its prior take on simple parametric forms (such as 1- or 2 -dimensional likelihood functions with simple conjugate priors), then the empirical Bayes problem is only to estimate the marginal formula_38 and the hyperparameters formula_18 using the complete set of empirical measurements. For example, one common approach, called parametric empirical Bayes point estimation, is to approximate the marginal using the maximum likelihood estimate (MLE), or a Moments expansion, which allows one to express the hyperparameters formula_18 in terms of the empirical mean and variance. This simplified marginal allows one to plug in the empirical averages into a point estimate for the prior formula_4. The resulting equation for the prior formula_4 is greatly  simplified, as shown below. There are several common parametric empirical Bayes models, including the Poisson–gamma model (below), the Beta-binomial model, the Gaussian–Gaussian model, the Dirichlet-multinomial model, as well specific models for Bayesian linear regression (see below) and Bayesian multivariate linear regression. More advanced approaches include hierarchical Bayes models and Bayesian mixture models. Gaussian–Gaussian model. For an example of empirical Bayes estimation using a Gaussian-Gaussian model, see Empirical Bayes estimators. Poisson–gamma model. For example, in the example above, let the likelihood be a Poisson distribution, and let the prior now be specified by the  conjugate prior, which is a gamma distribution (formula_43) (where formula_44): It is straightforward to show the posterior is also a gamma distribution. Write where the marginal distribution has been omitted since it does not depend explicitly on formula_4. Expanding terms which do depend on formula_4 gives the posterior as: So the posterior density is also a gamma distribution formula_50, where formula_51, and formula_52. Also notice that the marginal is simply the integral of the posterior over all formula_53, which turns out to be a negative binomial distribution. To apply empirical Bayes, we will approximate the marginal using the maximum likelihood  estimate (MLE). But since the posterior is a gamma distribution, the MLE of the marginal turns out to be just the mean of the posterior, which is the point estimate formula_54 we need. Recalling that the mean formula_55 of a gamma distribution formula_56 is simply formula_57, we have To obtain the values of formula_59 and formula_60, empirical Bayes prescribes estimating mean formula_61 and variance formula_62 using the complete set of empirical data. The resulting point estimate formula_63 is therefore like a weighted average of the sample mean formula_64 and the prior mean formula_65. This turns out to be a general  feature of empirical Bayes; the point estimates for the prior (i.e. mean) will look like a weighted averages of the sample estimate and the prior estimate (likewise for estimates of the variance).