Fisher information
In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable "X" carries about an unknown parameter "θ" of a distribution that models "X". Formally, it is the variance of the score, or the expected value of the observed information. In Bayesian statistics, the asymptotic distribution of the posterior mode depends on the Fisher information and not on the prior (according to the Bernstein–von Mises theorem, which was anticipated by Laplace for exponential families). The role of the Fisher information in the asymptotic theory  of maximum-likelihood estimation was emphasized by the statistician Ronald Fisher (following some initial results by Francis Ysidro Edgeworth). The Fisher information is also used in the calculation of the Jeffreys prior, which is used in Bayesian statistics. The Fisher information matrix is used to calculate the covariance matrices associated with maximum-likelihood estimates. It can also be used in the formulation of test statistics, such as the Wald test. Statistical systems of a scientific nature (physical, biological, etc.) whose likelihood functions obey shift invariance have been shown to obey maximum Fisher information. The level of the maximum depends upon  the nature of the system constraints. Definition. The Fisher information is a way of measuring the amount of information that an observable random variable formula_1 carries about an unknown parameter formula_2 upon which the probability of formula_1 depends. Let formula_4 be the probability density function (or probability mass function) for formula_1 conditioned on the value of formula_2. It describes the probability that we observe a given outcome of formula_1, "given" a known value of formula_2. If formula_9 is sharply peaked with respect to changes in formula_2, it is easy to indicate the "correct" value of formula_2 from the data, or  equivalently, that the data formula_1 provides a lot of information about the parameter formula_2. If the likelihood formula_9 is flat and spread-out, then it would take many samples of formula_1 to estimate the actual "true" value of formula_2 that "would" be obtained using the entire population being sampled. This suggests studying some kind of variance with respect to formula_2. Formally, the partial derivative with respect to formula_2 of the natural logarithm of the likelihood function is called the "score". Under certain regularity conditions, if formula_2 is the true parameter (i.e. formula_1 is actually distributed as formula_4), it can be  shown that the expected value (the first moment) of the score, evaluated at the true parameter value formula_2, is 0: The Fisher information is defined to be the variance of the score: Note that formula_25. A random variable carrying high Fisher information implies that the absolute value of the score is often high. The Fisher information is not a function of a particular observation, as the random variable "X" has been averaged out. If is twice differentiable with respect to "θ", and under certain regularity conditions, then the Fisher information may also be written as since and Thus, the Fisher  information may be seen as the curvature of the support curve (the graph of the log-likelihood). Near the maximum likelihood estimate, low Fisher information therefore indicates that the maximum appears "blunt", that is, the maximum is shallow and there are many nearby values with a similar log-likelihood. Conversely, high Fisher information indicates that the maximum is sharp. Regularity conditions. The regularity conditions are as follows: If "θ" is a vector then the regularity conditions must hold for every component of "θ". It is easy to find an example of a density that does not satisfy the regularity conditions : The density of a Uniform(0, "θ") variable fails to satisfy conditions 1 and 3. In this case, even though the Fisher information can be computed from the definition, it will not have the properties it is typically assumed to have. In terms of likelihood. Because the likelihood of "θ" given "X" is always proportional to the probability "f"("X"; "θ"), their logarithms necessarily differ by a constant that is independent of "θ", and the derivatives of these logarithms with respect to "θ" are necessarily equal. Thus one can substitute in a log-likelihood "l"("θ"; "X") instead of in the definitions of  Fisher Information. Samples of any size. The value "X" can represent a single sample drawn from a single distribution or can represent a collection of samples drawn from a collection of distributions. If there are "n" samples and the corresponding "n" distributions are statistically independent then the Fisher information will necessarily be the sum of the single-sample Fisher information values, one for each single sample from its distribution. In particular, if the "n" distributions are independent and identically distributed then the Fisher information will necessarily be "n" times the Fisher information of a single sample from the common distribution . Informal derivation of the Cramér–Rao bound. The Cramér–Rao bound states that the inverse of the Fisher information is a lower bound on the variance of any unbiased estimator of "θ". H.L. Van Trees (1968) and B. Roy Frieden (2004) provide the following method of deriving the Cramér–Rao bound, a result which describes use of the Fisher information. Informally, we begin by considering an unbiased estimator formula_29. Mathematically, "unbiased" means that This expression is zero independent of "θ", so its partial derivative with respect to "θ" must also be zero. By the product rule, this partial derivative is  also equal to For each "θ", the likelihood function is a probability density function, and therefore formula_32. By dividing and multiplying by formula_33, one gets Using these two facts in the above, we get Factoring the integrand gives Squaring the expression in the integral, the Cauchy–Schwarz inequality yields The second bracketed factor is defined to be the Fisher Information, while the first bracketed factor is the expected mean-squared error of the estimator formula_38. By rearranging, the inequality tells us that In other words, the precision to which we can estimate "θ" is fundamentally limited by the Fisher information  of the likelihood function. Single-parameter Bernoulli experiment. A Bernoulli trial is a random variable with two possible outcomes, "success" and "failure", with success having a probability of "θ". The outcome can be thought of as determined by a coin toss, with the probability of heads being "θ" and the probability of tails being . Let "X" be a Bernoulli trial. The Fisher information contained in "X" may be calculated to be Because Fisher information is additive, the Fisher information contained in "n" independent Bernoulli trials is therefore This is the reciprocal of the variance of the mean number of successes  in "n" Bernoulli trials, so in this case, the Cramér–Rao bound is an equality. Matrix form. When there are "N" parameters, so that "θ" is an vector formula_42 then the Fisher information takes the form of an matrix. This matrix is called the Fisher information matrix (FIM) and has typical element The FIM is a positive semidefinite matrix. If it is positive definite, then it defines a Riemannian metric on the "N"-dimensional parameter space. The topic information geometry uses this to connect Fisher information to differential geometry, and in that context, this metric is known as the Fisher information  metric. Under certain regularity conditions, the Fisher information matrix may also be written as The result is interesting in several ways: Orthogonal parameters. We say that two parameters "θ" and "θ" are orthogonal if the element of the "i"th row and "j"th column of the Fisher information matrix is zero. Orthogonal parameters are easy to deal with in the sense that their maximum likelihood estimates are independent and can be calculated separately. When dealing with research problems, it is very common for the researcher to invest some time searching for an orthogonal parametrization of the densities involved in the problem . Singular statistical model. If the Fisher information matrix is positive definite for all , then the corresponding statistical model is said to be "regular"; otherwise, the statistical model is said to be "singular". Examples of singular statistical models include the following: normal mixtures, binomial mixtures, multinomial mixtures, Bayesian networks, neural networks, radial basis functions, hidden Markov models, stochastic context-free grammars, reduced rank regressions, Boltzmann machines. In machine learning, if a statistical model is devised so that it extracts hidden structure from a random phenomenon, then it naturally becomes singular. Multivariate normal distribution. The FIM for a "N"-variate multivariate normal distribution , formula_45 has a special form. Let the "K"-dimensional vector of parameters be formula_46 and the vector of random normal variables be formula_47. Assume that the mean values of these random variables are formula_48, and let formula_49 be the covariance matrix. Then, for formula_50, the ("m", "n") entry of the FIM is: where formula_52 denotes the transpose of a vector, formula_53 denotes the trace of a square matrix, and: Note that a special, but very common, case is the one where formula_55, a constant. Then In this case the Fisher information matrix may be identified with the coefficient matrix of the  normal equations of least squares estimation theory. Another special case occurs when the mean and covariance depend on two different vector parameters, say, "β" and "θ". This is especially popular in the analysis of spatial data, which often uses a linear model with correlated residuals. In this case, where Properties. Chain rule. Similar to the entropy or mutual information, the Fisher information also possesses a chain rule decomposition. In particular, if "X" and "Y" are jointly distributed random variables, it follows that: where formula_60 and formula_61 is the Fisher information of "Y" relative to formula_2 calculated with respect to the  conditional density of "Y" given a specific value "X" = "x". As a special case, if the two random variables are independent, the information yielded by the two random variables is the sum of the information from each random variable separately: Consequently, the information in a random sample of "n" independent and identically distributed observations is "n" times the information in a sample of size 1. Sufficient statistic. The information provided by a sufficient statistic is the same as that of the sample "X". This may be seen by using Neyman's factorization criterion for a sufficient statistic. If "T"("X ) is sufficient for θ", then for some functions "g" and "h". The independence of "h"("X") from "θ" implies and the equality of information then follows from the definition of Fisher information. More generally, if is a statistic, then with equality if and only if "T" is a sufficient statistic. Reparametrization. The Fisher information depends on the parametrization of the problem. If "θ" and "η" are two scalar parametrizations of an estimation problem, and "θ" is a continuously differentiable function of "η", then where formula_68 and formula_69 are the Fisher information measures of "η" and "θ", respectively. In the vector case , suppose formula_70 and formula_71 are "k"-vectors which parametrize an estimation problem, and suppose that formula_70 is a continuously differentiable function of formula_71, then, where the ("i", "j")th element of the "k" × "k" Jacobian matrix formula_75 is defined by and where formula_77 is the matrix transpose of formula_78 In information geometry, this is seen as a change of coordinates on a Riemannian manifold, and the intrinsic properties of curvature are unchanged under different parametrizations. In general, the Fisher information matrix provides a Riemannian metric (more precisely, the Fisher–Rao metric) for the manifold of thermodynamic states, and can be used  as an information-geometric complexity measure for a classification of phase transitions, e.g., the scalar curvature of the thermodynamic metric tensor diverges at (and only at) a phase transition point. In the thermodynamic context, the Fisher information matrix is directly related to the rate of change in the corresponding order parameters. In particular, such relations identify second-order phase transitions via divergences of individual elements of the Fisher information matrix. Isoperimetric inequality. The Fisher information matrix plays a role in an inequality like the isoperimetric inequality. Of all probability distributions with a given entropy, the one whose Fisher information matrix  has the smallest trace is the Gaussian distribution. This is like how, of all bounded sets with a given volume, the sphere has the smallest surface area. The proof involves taking a multivariate random variable formula_1 with density function formula_9 and adding a location parameter to form a family of densities formula_81. Then, by analogy with the Minkowski–Steiner formula, the "surface area" of formula_1 is defined to be where formula_84 is a Gaussian variable with covariance matrix formula_85. The name "surface area" is apt because the entropy power formula_86 is the volume of the "effective support set," so formula_87  is the "derivative" of the volume of the effective support set, much like the Minkowski-Steiner formula. The remainder of the proof uses the entropy power inequality, which is like the Brunn–Minkowski inequality. The trace of the Fisher information matrix is found to be a factor of formula_87. Applications. Optimal design of experiments. Fisher information is widely used in optimal experimental design. Because of the reciprocity of estimator-variance and Fisher information, "minimizing" the "variance" corresponds to "maximizing" the "information". When the linear (or linearized) statistical model has several parameters, the mean of the parameter estimator is a vector  and its variance is a matrix. The inverse of the variance matrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using statistical theory, statisticians compress the information-matrix using real-valued summary statistics; being real-valued functions, these "information criteria" can be maximized. Traditionally, statisticians have evaluated estimators and designs by considering some summary statistic of the covariance matrix (of an unbiased estimator), usually with positive real values (like the determinant or matrix trace). Working with positive real numbers brings several advantages : If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone). For several parameters, the covariance matrices and information matrices are elements of the convex cone of nonnegative-definite symmetric matrices in a partially ordered vector space, under the Loewner (Löwner) order. This cone is closed under matrix addition and inversion, as well as under the multiplication of positive real numbers and matrices. An exposition of matrix  theory and Loewner order appears in Pukelsheim. The traditional optimality criteria are the information matrix's invariants, in the sense of invariant theory; algebraically, the traditional optimality criteria are functionals of the eigenvalues of the (Fisher) information matrix (see optimal design). Jeffreys prior in Bayesian statistics. In Bayesian statistics, the Fisher information is used to calculate the Jeffreys prior, which is a standard, non-informative prior for continuous distribution parameters. Computational neuroscience. The Fisher information has been used to find bounds on the accuracy of neural codes. In that case, "X" is typically the joint responses of many neurons representing  a low dimensional variable "θ" (such as a stimulus parameter). In particular the role of correlations in the noise of the neural responses has been studied. Derivation of physical laws. Fisher information plays a central role in a controversial principle put forward by Frieden as the basis of physical laws, a claim that has been disputed. Machine learning. The Fisher information is used in machine learning techniques such as elastic weight consolidation, which reduces catastrophic forgetting in artificial neural networks. Fisher information can be used as an alternative to the Hessian of the loss function in second-order gradient descent  network training. Relation to relative entropy. Fisher information is related to relative entropy. The relative entropy, or Kullback–Leibler divergence, between two distributions formula_89 and formula_90 can be written as Now, consider a family of probability distributions formula_92 parametrized by formula_93. Then the Kullback–Leibler divergence, between two distributions in the family can be written as If formula_2 is fixed, then the relative entropy between two distributions of the same family is minimized at formula_96. For formula_97 close to formula_2, one may expand the previous expression in a series up to second order: But the second order derivative can be  written as Thus the Fisher information represents the curvature of the relative entropy of a conditional distribution with respect to its parameters. History. The Fisher information was discussed by several early statisticians, notably F. Y. Edgeworth. For example, Savage says: "In it [Fisher information], he [Fisher] was to some extent anticipated (Edgeworth 1908–9 esp. 502, 507–8, 662, 677–8, 82–5 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." There are a number of early historical sources and a number of reviews of this early work. See also. Other measures employed in information theory: