Total correlation
In probability theory and in particular in information theory, total correlation (Watanabe 1960) is one of several generalizations of the mutual information. It is also known as the "multivariate constraint" (Garner 1962) or "multiinformation" (Studený & Vejnarová 1999). It quantifies the redundancy or dependency among a set of "n" random variables. Definition. For a given set of "n" random variables formula_1, the total correlation formula_2 is defined as the Kullback–Leibler divergence from the joint distribution formula_3 to the independent distribution of formula_4, This divergence reduces to the simpler difference of entropies, where formula_7 is the information entropy of  variable formula_8, and formula_9 is the joint entropy of the variable set formula_1. In terms of the discrete probability distributions on variables formula_11, the total correlation is given by The total correlation is the amount of information "shared" among the variables in the set. The sum formula_13 represents the amount of information in bits (assuming base-2 logs) that the variables would possess if they were totally independent of one another (non-redundant), or, equivalently, the average code length to transmit the values of all variables if each variable was (optimally) coded independently. The term formula_14 is the "actual" amount of  information that the variable set contains, or equivalently, the average code length to transmit the values of all variables if the set of variables was (optimally) coded together. The difference between these terms therefore represents the absolute redundancy (in bits) present in the given set of variables, and thus provides a general quantitative measure of the "structure" or "organization" embodied in the set of variables (Rothstein 1952). The total correlation is also the Kullback–Leibler divergence between the actual distribution formula_15 and its maximum entropy product approximation formula_4. Total correlation quantifies the amount of dependence among a group of variables . A near-zero total correlation indicates that the variables in the group are essentially statistically independent; they are completely unrelated, in the sense that knowing the value of one variable does not provide any clue as to the values of the other variables. On the other hand, the maximum total correlation (for a fixed set of individual entropies formula_17) is given by and occurs when one of the variables determines "all" of the other variables. The variables are then maximally related in the sense that knowing the value of one variable provides complete information about the values of all the  other variables, and the variables can be figuratively regarded as "cogs," in which the position of one cog determines the positions of all the others (Rothstein 1952). It is important to note that the total correlation counts up "all" the redundancies among a set of variables, but that these redundancies may be distributed throughout the variable set in a variety of complicated ways (Garner 1962). For example, some variables in the set may be totally inter-redundant while others in the set are completely independent. Perhaps more significantly, redundancy may be carried in interactions of various degrees: A group of  variables may not possess any pairwise redundancies, but may possess higher-order "interaction" redundancies of the kind exemplified by the parity function. The decomposition of total correlation into its constituent redundancies is explored in a number sources (Mcgill 1954, Watanabe 1960, Garner 1962, Studeny & Vejnarova 1999, Jakulin & Bratko 2003a, Jakulin & Bratko 2003b, Nemenman 2004, Margolin et al. 2008, Han 1978, Han 1980). Conditional total correlation. Conditional total correlation is defined analogously to the total correlation, but adding a condition to each term. Conditional total correlation is similarly defined as a Kullback-Leibler divergence between two conditional probability distributions, Analogous to  the above, conditional total correlation reduces to a difference of conditional entropies, Uses of total correlation. Clustering and feature selection algorithms based on total correlation have been explored by Watanabe. Alfonso et al. (2010) applied the concept of total correlation to the optimisation of water monitoring networks.