Info-metrics
Info-metrics is an interdisciplinary approach to scientific modeling, inference and efficient information processing. It is the science of modeling, reasoning, and drawing inferences under conditions of noisy and limited information. From the point of view of the sciences, this framework is at the intersection of information theory, statistical methods of inference, applied mathematics, computer science, econometrics, complexity theory, decision analysis, modeling, and the philosophy of science. Info-metrics provides a constrained optimization framework to tackle under-determined or ill-posed problems – problems where there is not sufficient information for finding a unique solution. Such problems are very  common across all sciences: available information is incomplete, limited, noisy and uncertain. Info-metrics is useful for modelling, information processing, theory building, and inference problems across the scientific spectrum. The info-metrics framework can also be used to test hypotheses about competing theories or causal mechanisms. History. Info-metrics evolved from the classical maximum entropy formalism, which is based on the work of Shannon. Early contributions were mostly in the natural and mathematical/statistical sciences. Since the mid 1980s and especially in the mid 1990s the maximum entropy approach was generalized and extended to handle a larger class of problems  in the social and behavioral sciences, especially for complex problems and data. The word ‘info-metrics’ was coined in 2009 by Amos Golan, right before the interdisciplinary Info-Metrics Institute was inaugurated. Preliminary definitions. Consider a random variable formula_1 that can result in one of "K" distinct outcomes. The probability formula_2 of each outcome formula_3 is formula_4 for formula_5. Thus, formula_6 is a "K"-dimensional probability distribution defined for formula_1 such that formula_8 and formula_9. Define the informational content of a single outcome formula_10 to be formula_11 (e.g., Shannon). Observing an outcome at the tails of the distribution (a rare event ) provides much more information than observing another, more probable, outcome. The entropy is the expected information content of an outcome of the random variable "X" whose probability distribution is "P": formula_12 Here formula_13 if formula_14, and formula_15 is the expectation operator. The basic info-metrics problem. Consider the problem of modeling and inferring the unobserved probability distribution of some "K"-dimensional discrete random variable given just the mean (expected value) of that variable. We also know that the probabilities are nonnegative and normalized (i.e., sum up to exactly 1). For all "K" > 2 the problem is underdetermined. Within the info -metrics framework, the solution is to maximize the entropy of the random variable subject to the two constraints: mean and normalization. This yields the usual maximum entropy solution. The solutions to that problem can be extended and generalized in several ways. First, one can use another entropy instead of Shannon’s entropy. Second, the same approach can be used for continuous random variables, for all types of conditional models (e.g., regression, inequality and nonlinear models), and for many constraints. Third, priors can be incorporated within that framework. Fourth, the same framework can be extended to accommodate greater uncertainty: uncertainty about  the observed values and/or uncertainty about the model itself. Last, the same basic framework can be used to develop new models/theories, validate these models using all available information, and test statistical hypotheses about the model. Examples. Six-sided dice. "Inference based on information resulting from repeated independent experiments." The following example is attributed to Boltzmann and was further popularized by Jaynes. Consider a six-sided , where tossing the is the event and the distinct outcomes are the numbers 1 through 6 on the upper face of the . The experiment is the independent repetitions of tossing the same . Suppose you  only observe the empirical mean value, y, of N tosses of a six-sided . Given that information, you want to infer the probabilities that a specific value of the face will show up in the next toss of the . You also know that the sum of the probabilities must be 1. Maximizing the entropy (and using log base 2) subject to these two constraints (mean and normalization) yields the most uninformed solution. formula_16 for formula_17 and formula_18 . The solution is where formula_20 is the inferred probability of event formula_21, formula_22 is the inferred Lagrange multipliers associated with the mean constraint , and formula_23 is the partition (normalization) function. If it’s a fair with mean of 3.5 you would expect that all faces are equally likely and the probabilities are equal. This is what the maximum entropy solution gives. If the is unfair (or loaded) with a mean of 4, the resulting maximum entropy solution will be formula_24. For comparison, minimizing the least squares criterion formula_25 instead of maximizing the entropy yields formula_26. Some cross-disciplinary examples. "Rainfall prediction": Using the expected daily rainfall (arithmetic mean), the maximum entropy framework can be used to infer and forecast the daily rainfall distribution . "Portfolio management": Suppose there is a portfolio manager who needs to allocate some assets or assign portfolio weights to different assets, while taking into account the investor’s constraints and preferences. Using these preferences and constraints, as well as the observed information, such as the market mean return, and covariances, of each asset over some time period, the entropy maximization framework can be used to find the optimal portfolio weights. In this case, the entropy of the portfolio represents its diversity. This framework can be modified to include other constraints such as minimal variance, maximal diversity etc. That model involves  inequalities and can be further generalized to include short sales. More such examples and related code can be found on An extensive list of work related to info-metrics can be found here: http://info-metrics.org/bibliography.html Marco Frittelli. "The minimal entropy martingale measure and the valuation problem in incomplete markets". "Mathematical finance", 10(1):39–52, 2000. Amos Golan and Volker Dose. "A generalized information theoretical approach to tomographic reconstruction". "Journal of Physics A: Mathematical and General", 34(7):1271, 2001.