Information content
In information theory, the information content, self-information, surprisal, or Shannon information is a basic quantity derived from the probability of a particular event occurring from a random variable. It can be thought of as an alternative way of expressing probability, much like odds or log-odds, but which has particular mathematical advantages in the setting of information theory. The Shannon information can be interpreted as quantifying the level of "surprise" of a particular outcome. As it is such a basic quantity, it also appears in several other settings, such as the length of a message needed to  transmit the event given an optimal source coding of the random variable. The Shannon information is closely related to information theoretic entropy, which is the expected value of the self-information of a random variable, quantifying how surprising the random variable is "on average." This is the average amount of self-information an observer would expect to gain about a random variable when measuring it. The information content can be expressed in various units of information, of which the most common is the "bit" (sometimes also called the "shannon"), as explained below. Definition. Claude Shannon's definition of self-information  was chosen to meet several axioms: The detailed derivation is below, but it can be shown that there is a unique function of probability that meets these three axioms, up to a multiplicative scaling factor. Broadly given an event formula_1 with probability formula_2, the information content is defined as follows: formula_3 The base of the log is left unspecified, which corresponds to the scaling factor above. Different choices of base correspond to different units of information: if the logarithmic base is 2, the unit is a bit or shannon; if the logarithm is the natural logarithm (base "e"), the unit  is the nat, short for "natural"; and if the base is 10, the units are hartleys, decimal digits, or occasionally dits. Formally, given a random variable formula_4 with probability mass function formula_5, the self-information of measuring formula_4 as outcome formula_1 is defined as formula_8 The Shannon entropy of the random variable formula_9 above is defined as formula_10 by definition equal to the expected information content of measurement of formula_9. The use of the notation formula_12 for self-information above is not universal. Since the notation formula_13 is also often used for the related quantity of mutual information, many authors  use a lowercase formula_14 for self-entropy instead, mirroring the use of the capital formula_15 for the entropy. Properties. Monotonically decreasing function of probability. For a given probability space, the measurement of rarer events are intuitively more "surprising," and yield more information content, than more common values. Thus, self-information is a strictly decreasing monotonic function of the probability, or sometimes called an "antitonic" function. While standard probabilities are represented by real numbers in the interval formula_16, self-informations are represented by extended real numbers in the interval formula_17. In particular, we have the following, for any choice of logarithmic  base: From this, we can get a few general properties: Relationship to log-odds. The Shannon information is closely related to the log-odds. In particular, given some event formula_1, suppose that formula_21 is the probability of formula_1 occurring, and that formula_23 is the probability of formula_1 not occurring. Then we have the following definition of the log-odds: formula_25 This can be expressed as a difference of two Shannon informations: formula_26 In other words, the log-odds can be interpreted as the level of surprise if the event 'doesn't' happen, minus the level of surprise if the event  'does' happen. Additivity of independent events. The information content of two independent events is the sum of each event's information content. This property is known as additivity in mathematics, and sigma additivity in particular in measure and probability theory. Consider two independent random variables formula_27 with probability mass functions formula_28 and formula_29 respectively. The joint probability mass function is formula_30 because formula_31 and formula_32 are independent. The information content of the outcome formula_33 isformula_34See below for an example. The corresponding property for likelihoods is that the log-likelihood of independent events is the sum of the log-likelihoods of  each event. Interpreting log-likelihood as "support" or negative surprisal (the degree to which an event supports a given model: a model is supported by an event to the extent that the event is unsurprising, given the model), this states that independent events add support: the information that the two events together provide for statistical inference is the sum of their independent information. Notes. This measure has also been called surprisal, as it represents the "surprise" of seeing the outcome (a highly improbable outcome is very surprising). This term (as a log-probability measure) was coined by Myron Tribus in  his 1961 book "Thermostatics and Thermodynamics". When the event is a random realization (of a variable) the self-information of the variable is defined as the expected value of the self-information of the realization. Self-information is an example of a proper scoring rule. Examples. Fair coin toss. Consider the Bernoulli trial of tossing a fair coin formula_4. The probabilities of the events of the coin landing as heads formula_36 and tails formula_37 (see fair coin and obverse and reverse) are one half each, formula_38. Upon measuring the variable as heads, the associated information gain isformula_39so the information gain  of a fair coin landing as heads is 1 shannon. Likewise, the information gain of measuring formula_37 tails isformula_41 Fair die roll. Suppose we have a fair six-sided die. The value of a dice roll is a discrete uniform random variable formula_42 with probability mass function formula_43The probability of rolling a 4 is formula_44, as for any other valid roll. The information content of rolling a 4 is thusformula_45of information. Two independent, identically distributed dice. Suppose we have two independent, identically distributed random variables formula_46 each corresponding to an independent fair 6-sided dice roll. The joint distribution of  formula_4 and formula_48 isformula_49 The information content of the random variate formula_50 isformula_51and can also be calculated by formula_52 Information from frequency of rolls. If we receive information about the value of the dice without knowledge of which die had which value, we can formalize the approach with so-called counting variables formula_53 for formula_54, then formula_55 and the counts have the multinomial distribution formula_56 To verify this, the 6 outcomes formula_57 correspond to the event formula_58 and a total probability of . These are the only events that are faithfully preserved with identity of which dice rolled which outcome because  the outcomes are the same. Without knowledge to distinguish the dice rolling the other numbers, the other formula_59 combinations correspond to one die rolling one number and the other die rolling a different number, each having probability . Indeed, formula_60, as required. Unsurprisingly, the information content of learning that both dice were rolled as the same particular number is more than the information content of learning that one dice was one number and the other was a different number. Take for examples the events formula_61 and formula_62for formula_63. For example, formula_64 and formula_65. The information contents are formula_66 formula_67 Let formula_68  be the event that both dice rolled the same value and formula_69 be the event that the dice differed. Then formula_70 and formula_71. The information contents of the events are formula_72 formula_73 Information from sum of die. The probability mass or density function (collectively probability measure) of the sum of two independent random variables is the convolution of each probability measure. In the case of independent fair 6-sided dice rolls, the random variable formula_74 has probability mass function formula_75, where formula_76 represents the discrete convolution. The outcome formula_77 has probability formula_78. Therefore, the information asserted isformula_79 General discrete uniform  distribution. Generalizing the example above, consider a general discrete uniform random variable (DURV) formula_80 For convenience, define formula_81. The p.m.f. is formula_82In general, the values of the DURV need not be integers, or for the purposes of information theory even uniformly spaced; they need only be equiprobable. The information gain of any observation formula_83 isformula_84 Special case: constant random variable. If formula_85 above, formula_4 degenerates to a constant random variable with probability distribution deterministically given by formula_87 and probability measure the Dirac measure formula_88. The only value formula_4 can take is deterministically formula_90, so the information content of any measurement  of formula_4 isformula_92In general, there is no information gained from measuring a known value. Categorical distribution. Generalizing all of the above cases, consider a categorical discrete random variable with support formula_93 and given by formula_94 For the purposes of information theory, the values formula_95 do not have to be numbers; they can be any mutually exclusive events on a measure space of finite measure that has been normalized to a probability measure formula_96. Without loss of generality, we can assume the categorical distribution is supported on the set formula_97; the mathematical structure is isomorphic in terms of probability theory and  therefore information theory as well. The information of the outcome formula_98 is given formula_99 From these examples, it is possible to calculate the information of any set of independent DRVs with known distributions by additivity. Relationship to entropy. The entropy is the expected value of the information content of the discrete random variable, with expectation taken over the discrete values it takes. Sometimes, the entropy itself is called the "self-information" of the random variable, possibly because the entropy satisfies formula_100, where formula_101 is the mutual information of formula_4 with itself. For continuous random variables the corresponding concept is differential  entropy. Derivation. By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information a priori. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero. Only when the advance knowledge of the content of the message by the receiver is less than 100% certain does the message actually convey information. For example, quoting a character (the Hippy Dippy Weatherman) of comedian George Carlin, "“Weather forecast for tonight: dark . Continued dark overnight, with widely scattered light by morning.”" Assuming one does not reside near the Earth's poles or polar circles, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night. Accordingly, the amount of self-information contained in a message conveying content informing an occurrence of event, formula_103, depends only on the probability of that event. formula_104 for some function formula_105 to be determined below. If formula_106, then formula_107. If formula_108, then formula_109. Further, by definition, the measure of self-information  is nonnegative and additive. If a message informing of event formula_110 is the intersection of two independent events formula_111 and formula_112, then the information of event formula_110 occurring is that of the compound message of both independent events formula_111 and formula_112 occurring. The quantity of information of compound message formula_110 would be expected to equal the sum of the amounts of information of the individual component messages formula_111 and formula_112 respectively: formula_119 Because of the independence of events formula_111 and formula_112, the probability of event formula_110 is formula_123 However, applying function formula_105 results in formula_125 Thanks to work on Cauchy 's functional equation, the only monotone functions formula_105 having the property such that formula_127 are the logarithm functions formula_128. The only operational difference between logarithms of different bases is that of different scaling constants, so we may assume formula_129 where formula_130 is the natural logarithm. Since the probabilities of events are always between 0 and 1 and the information associated with these events must be nonnegative, that requires that formula_131. Taking into account these properties, the self-information formula_132 associated with outcome formula_103 with probability formula_134 is defined as: formula_135 The smaller the probability of event formula_103, the larger the  quantity of self-information associated with the message that the event indeed occurred. If the above logarithm is base 2, the unit of formula_137 is bits. This is the most common practice. When using the natural logarithm of base formula_138, the unit will be the nat. For the base 10 logarithm, the unit of information is the hartley. As a quick illustration, the information content associated with an outcome of 4 heads (or any specific outcome) in 4 consecutive tosses of a coin would be 4 bits (probability 1/16), and the information content associated with getting a result other than  the one specified would be ~0.09 bits (probability 15/16). See above for detailed examples.