Differential entropy
Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy, a measure of average surprisal of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the limiting density of discrete points (LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of  the LDDP, and one that loses its fundamental association with discrete entropy. Definition. Let formula_1 be a random variable with a probability density function formula_2 whose support is a set formula_3. The "differential entropy" formula_4 or formula_5 is defined as For probability distributions which don't have an explicit density function expression, but have an explicit quantile function expression, formula_6, then formula_7 can be defined in terms of the derivative of formula_6 i.e. the quantile density function formula_9 as As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2  (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure formula_1. For example, the differential entropy of a quantity measured in millimeters will be more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of more than the same quantity divided by 1000. One must take care in trying to apply properties of discrete entropy to  differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution formula_12 has "negative" differential entropy being less than that of formula_14 which has "zero" differential entropy. Thus, differential entropy does not share all properties of discrete entropy. Note that the continuous mutual information formula_15 has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of "partitions" of formula_1 and formula_17 as these partitions become finer and finer. Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible  maps), including linear transformations of formula_1 and formula_17, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values. For the direct analogue of discrete entropy extended to the continuous space, see limiting density of discrete points. Properties of differential entropy. However, differential entropy does not have other desirable properties: A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points). Maximization in the normal distribution. Theorem . With a normal distribution, differential entropy is maximized for a given variance. A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian. Proof. Let formula_48 be a Gaussian PDF with mean μ and variance formula_49 and formula_50 an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that formula_50 has the same mean of formula_52 as formula_48. Consider the Kullback–Leibler divergence between the two distributions Now note that because the result does not  depend on formula_50 other than through the variance. Combining the two results yields with equality when formula_58 following from the properties of Kullback–Leibler divergence. Alternative proof. This result may also be demonstrated using the variational calculus. A Lagrangian function with two Lagrangian multipliers may be defined as: where "g(x)" is some function with mean μ. When the entropy of "g(x)" is at a maximum and the constraint equations, which consist of the normalization condition formula_60 and the requirement of fixed variance formula_61, are both satisfied, then a small variation δ"g"("x") about "g(x)" will produce a variation δ"L" about "L  which is equal to zero: Since this must hold for any small δg"("x"), the term in brackets must be zero, and solving for "g(x)" yields: Using the constraint equations to solve for λ and λ yields the normal distribution: Example: Exponential distribution. Let formula_1 be an exponentially distributed random variable with parameter formula_66, that is, with probability density function Its differential entropy is then Here, formula_68 was used rather than formula_4 to make it explicit that the logarithm was taken to base "e", to simplify the calculation. Relation to estimator error. The differential entropy yields a lower bound on the  expected squared error of an estimator. For any random variable formula_1 and estimator formula_71 the following holds: with equality if and only if formula_1 is a Gaussian random variable and formula_71 is the mean of formula_1. Differential entropies for various distributions. In the table below formula_76 is the gamma function, formula_77 is the digamma function, formula_78 is the beta function, and γ is Euler's constant. Many of the differential entropies are from. Variants. As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant  under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities. A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points). If formula_79 is further constrained to be a probability density, the resulting notion is called relative entropy in information theory: The definition of differential entropy above can be obtained by partitioning the range of formula_1 into bins of length formula_82 with associated sample points formula_83 within the bins, for formula_1 Riemann integrable. This  gives a quantized version of formula_1, defined by formula_86 if formula_87. Then the entropy of formula_86 is The first term on the right approximates the differential entropy, while the second term is approximately formula_90. Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be formula_91.