Likelihood Ratio Exponential Families
Abstract
The exponential family is well known in machine learning and statistical physics as the maximum entropy distribution subject to a set of observed constraints [1], while the geometric mixture path is common in MCMC methods such as annealed importance sampling ( AIS ) [2, 3].
Linking these two ideas, recent work [4] has interpreted the geometric mixture path as an exponential family of distributions to analyse the thermodynamic variational objective ( TVO )
In this work, we extend likelihood ratio exponential families to include solutions to rate-distortion ( RD ) optimization [6, 7], the Information Bottleneck ( IB ) method [8], and recent ‘ RDC ’ approaches which combine RD and IB [9, 10].
This provides a common mathematical framework for understanding these methods via the conjugate duality of exponential families and hypothesis testing.
Further, we collect existing results [11–14] to provide a variational representation of intermediate RD or TVO distributions as a minimizing an expectation of KL divergences.
This solution also corresponds to a size-power tradeoff using the likelihood ratio test and the Neyman Pearson lemma.
In thermodynamic integration ( TI ) bounds [15, 16] such as the TVO , we identify the intermediate distribution whose expected sufﬁcient statistics match the log partition function.
Introduction
Likelihood Ratio Exponential Family Following Brekelmans et al.
[4], we consider the geometric mixture path between a base distribution π 0 ( z ) and a target π 1 ( z ) , or posterior π 1 ( z | x ) , as an exponential family.
We deﬁne the sufﬁcient statistics φ ( z )
= log π 1 ( z ) /π
( z ) as the log likelihood ratio [4], although in practice it is convenient to consider unnormalized distributions such as π 1 ( z ) ∝ ˜
π
1 ( z ) or π 1 ( z | x ) ∝ ˜
π 1 ( x, z ) and adjust the normalization constant accordingly.
Using a natural parameter β and base measure π 0 ( z ) ,
[Equation]
where φ
[Equation]
z
[Equation]
=
[Equation]
Before discussing examples in Sec. 2, we review background on conjugate duality in exponential families, which provides insights which are not evident from writing (1) as a geometric mixture [4].
Legendre Duality in Exponential Families Since the log partition function ψ ( β ) of an exponential family is convex in the natural parameters β , its gradient will be unique and may be used as a dual parameterization for π β [17].
This diffeomorphism between the natural parameters β = { β j } 1 and moment parameters, denoted η = { η j } , also deﬁnes the convex conjugate function ψ ∗ ( η ) , with
[Equation]
Using the Lebesgue or counting measure as π 0 ( z ) , the conjugate ψ ∗ ( η ) corresponds to the negative entropy of the maximum entropy solution π β ( z ) with observable constraint η [17, 18].
With a general base measure (see App.
A), we have
[Equation]
Since the convex conjugate is an involution, ( ψ ∗ ) ∗ =
ψ , we can obtain a similar conjugate optimization for ψ ( β ) =
sup η β · η − ψ ∗ ( η ) .
This leads to the canonical expression for Legendre duality, when the two optimizations are in equilibrium, and β and η β are in correspondence (see App. B.1)
[Equation]
We can also construct a Bregman divergence, which corresponds to the gap in a ﬁrst order Taylor approximation, using any convex function.
For example, using ψ ( β ) or ψ ∗ ( η ) , we have
D
[Equation]
With derivations in App.
B, we can see that the Bregman divergences D ψ , D ψ ∗ are equivalent with the order of the arguments reversed, and correspond to a KL divergence
[Equation]
Examples
Thermodynamic Variational Objective
In the variational autoencoder ( VAE ) setting, the TVO [5, 4] uses the approximate posterior as the initial distribution π
= q ( z | x ) and joint generative model as the unnormalized target ˜ π
1 = p θ ( x, z ) .
Masrani et al.
[5] use TI [15, 16] to express ψ ( x ; 1) =
log Z 1 ( x ) = log p θ ( x ) as an integral over the geometric path (2),
[Equation]
where we use the fact that the (partial) derivative of the log partition function equals the expected sufﬁcient statistics in any exponential family [17].
Since ψ ( x ; β ) is convex in β for any x , the leftand right-Riemann sums will provide lower and upper bounds on the log marginal likelihood,
T −
[Equation]
We derive novel insights on TVO curve via hypothesis testing in Sec. 3.
Note that TI bounds as in (8) may be constructed for any one-dimensional likelihood ratio exponential family, such as in RD , although more care would be required for multiple sufﬁcient statistics as in RDC below [9, 10].
Rate-Distortion Rate-distortion ( RD ) optimization ([6, 19, 8, 7] Ch. 13) formalizes the problem of lossy compression subject to a ﬁdelity constraint.
As in Alemi et al.
[6][10], we measure the rate using the KL divergence to a ﬁxed marginal distribution π
( z ) = m ( z ) , which upper bounds the mutual information in general.
The distortion function d ( x, z ) measures the quality of a code z .
RD optimization seeks the minimum-rate encoding which achieves a desired average distortion D ,
[Equation]
We restrict our attention to a reconstruction loss distortion d ( x, z )
= − log p θ ( x | z ) as in [6].
Introducing β to enforce the constraint, we obtain the unconstrained Lagrangian
[Equation]
whose solution, for a given m ( z ) , has an exponential family form with φ ( x, z ) = − d ( x, z ) (e.g. [8])
[Equation]
From the likelihood ratio perspective, we can choose π 0 ( z ) = m ( z ) and ˜ π 1 ( x, z )
=
p θ ( x | z ) m ( z ) ∝ p θ ( z | x ) .
Absorbing the factor of p θ ( x ) into the normalizer Z β ( x ) , we obtain the sufﬁcient statistics
[Equation]
so that the solution in (12) matches π β ( z | x ) in the likelihood ratio family induced by (13).
The
Lagrange multiplier β is chosen to enforce the distortion constraint D , which, since φ ( x, z ) = − d ( x, z ) , translates to seeking β such that the moment parameters η β = − D .
At this optimal solution, R ( D ) simply matches the conjugate ψ ∗ ( η ) in (4)
[Equation] [Equation]
Huang et al.
[20] use the expression in (16) to estimate the RD curve using AIS [2].
Information Bottleneck and RDC When deﬁning ‘relevant information’ via a random variable such as a label y , the Information Bottleneck ( IB ) method [8, 21, 22] simpliﬁes to an RD problem with a learned classiﬁer providing the distortion function c
( y, z )
= − log p θ ( y | z ) ([8] or App.C).
[Equation]
Recent work [9, 10] considers ‘ RDC ’ optimization using both reconstruction and classiﬁcation loss,
[Equation]
In this case, we may consider two sufﬁcient statistics in our likelihood ratio exponential family.
Similarly to multivariate IB [23, 24], we use an unnormalized target which factorizes as ˜ π 1 ( x, y, z )
=
p θ ( x | z ) p θ ( y | z ) m ( z ) , and consider the likelihood ratio sufﬁcient statistics
[Equation]
where we again absorb p θ ( x ) and p ( y ) into the normalization.
Introducing Lagrange multipliers β = { β D , β C } to enforce η d ( β ) = − D , η c
[Equation]
β )
=
− C at optimality, we obtain the solution of (18) as a geometric mixture [9, 10] belonging to the likelihood ratio family with φ =
[Equation]
With applications in transfer learning, Gao and Chaudhari [9] seek to evolve the model parameters θ and approximate posterior q ( z | x ) along the ‘equilibrium surface’ of optimal solutions to (18).
We interpret their free energy F ( β D , β C ) , where β D , β C are analogous to the intensive variables of a physical system [10], as the negative log partition function − ψ ( β D , β C ) .
Written using the conjugate optimization (3), we seek { θ , q ( z | x ) } that yield the appropriate distortion and classiﬁcation loss
[Equation]
Similarly, for given extensive variables η D , η C , the optimal rate R ( D, C ) corresponds to ψ ∗ ( η D , η C
[Equation]
At optimality on the ‘equilibrium surface’ [9], we obtain equality in the expression for Legendre duality (5).
In other words, for the current decoder and classiﬁer parameters θ , the encoder q ( z | x ) matches π β ( z | x ) in the likelihood ratio family (19), with β = { β D , β C } .
This distribution fulﬁlls the constraints η β = { η D , η C } = {− D, − C } , so that
[Equation]
This expression (22) also translates to the ‘ﬁrst law of learning’ from Alemi and Fischer [10], when ψ ( β D , β C ) is considered a ﬁxed quantity for given a choice of β D , β C .
Variational Representations and Hypothesis Testing
Grosse et al.
[12] note that any distribution along the geometric mixture path can be given a variational representation as the solution to an expected KL divergence minimization
[Equation]
We proceed to interpret (23) as a Bregman information (or gap in Jensen’s inequality) [11], or as describing an optimal decision rule for hypothesis testing using the Neyman Pearson lemma.
We restrict our attention to a one-dimensional likelihood ratio family, as in TVO , RD , or IB , in this section.
Bregman Information Banerjee et al.
[11] deﬁne the Bregman information as the minimum expected Bregman divergence to a representative point in the second argument.
Regardless of the divergence, the optimal representative corresponds to the mean over the arguments.
Since
D KL [ r β ( z ) ||
π 0 ( z )]
=
D ψ [0 : β r ] when optimizing over r β ( z ) in the exponential family, we can rewrite (23) as
[Equation]
At this optimum, the expected KL divergence (24) can be written as a gap in Jensen’s inequality for the convex function ψ ( β ) [11],
[Equation]
We visualize this gap in Jensen’s inequality in Fig.
As shown in [25] or App.
E.1, we can also view J ψ , or the expected KL divergence (23), as a Rényi divergence with order β
[Equation]
Neyman Pearson Lemma Suppose we have access to n i.i.d.
observations from an unknown distribution r ( z ) , and are interested in testing the hypotheses that either H 0 : r ( z ) =
π 0 ( z ) or
H 1 : r ( z ) =
π 1 ( z ) .
The Neyman-Pearson lemma states that the likelihood ratio test is optimal, in the sense that, for any other decision region with type-1 error Pr ( e 1 ) =
R , then the type-2 error is no better than that of the likelihood ratio test ([7] Ch. 11, [14]) 2 .
The decision rule is given by
[Equation]
for some threshold η .
Let a type-1 error occur when n i
[Equation] i [Equation] d. draws { z i } Ni =1 from π 0 ( z ) will yield empirical expectations exceeding the threshold η .
Sanov’s Theorem and large deviation theory ([7]
Ch.
11, [27]) states that the asymptotic error exponent corresponds to a KL divergence
[Equation]
The feasible set M η := { r ( z ) | E r log π 1 ( z ) π 0 ( z ) =
η } reﬂects a expectation constraint corresponding to a given decision threshold, and the error exponent is obtained by minimizing the divergence subject to this constraint.
With ψ ∗ ( η ) =
D KL [ π β η ( z ) ||
π 0 ( z )] as in (4), this exactly matches the conjugate or maximum entropy optimization for a given expected sufﬁcient statistic η , and thus r ∗ ( z ) lies within the likelihood ratio exponential family,
[Equation]
As shown in Fig.
1, Sanov’s Theorem implies a similar expression for the asymptotic type-2 error, when draws from π 1 ( z ) achieve a lower expected likelihood ratio than η .
Expressing the conditions of the Neyman Pearson lemma using these asymptotic error probabilities, we can write
[Equation]
subj.
to D KL
[Equation]
r
[Equation]
z
[Equation]
z
[Equation]
R
[Equation]
Using a Lagrange multiplier λ = 1 − ββ to enforce the constraint, we obtain the variational form (23)
[Equation]
Thus, any distribution in our likelihood ratio exponential family corresponds to a likelihood ratio test with decision threshold η , which is optimal for a type-1 error region of size ψ ∗ ( η ) =
R .
Chernoff Information While each choice of β η determines a likelihood ratio test and error region, how should we choose this parameter?
Regardless of the prior probabilities p 0 , p 1 that we might assign to each hypothesis in a Bayesian setting, the Chernoff information provides the best achievable error exponent in the large sample limit ([13], [7] Ch. 11).
[Equation]
To rewrite (34) as a gap in Jensen’s inequality (35), we have expanded π 0 = ˜ π 0 /Z
and π 1 = ˜
π 1 /Z 1 , with ψ ( β ) = log
[Equation]
We denote the optimum over β as the Chernoff point [13], or β ∗ .
In App.
E.2, we derive the moment-matching condition
[Equation]
which holds between arbitrary β 0 , β 1 and implies η β ∗
= ψ (1) − ψ (0) for β 0 = 0 , β 1 = 1 .
At this critical point, we show in Appendix E.3 the KL divergence to the endpoints is equal,
[Equation]
[Table]
[Equation]
[Table]
fniJfQjpRrXOSuZXQNbz5FHkTYisFiGfyt9EYB1qcEd7pee+ADzgDbzGOL9i5sxWZtslzBTBLb6Ttpf+iyjUjIgS3jWd5ZslFt7d3+ns7Tz+bW/j4Nn00iFr5DvyPdkiHfKIHJAfyRtySDj5RP4if5N/WvutDy1oFVPq1SvnNnfJwmjpfwHOj9mV</latexit>
(See [18] Sec. 1.6, 2.8 for an interpretation in terms of projection and a generalization of the Pythagorean Theorem)
R ( D ) =
⇤ ( ⌘ ) <latexit sha1_base64="JEfImWcd4tzRYs3lL34XtQI6Eds=">AAAHZXichVXdbhM5FDawNF1+dssCe8MFFlWltIqqBooALStVSxGsltWyiJYKJkQez5mJVY/H8ThtUjMPwy08EU/Aa3A8bWgSp7sjJTlzvu9858exHWspSrux8eXc+Qs/XFxoLP546fKVqz/9vHTtl92yGBgOO7yQhdmLWQlSKNixwkrY0wZYHkt4E+8/8fibAzClKNRrO9LQyVmmRCo4s+jqLt181dxepb/TSJfivVurmhFYttpdWt5Y36gfGhrtE2N5a5HUz8vutYWHUVLwQQ7KcsnK8t3dTW1boDLsoddxzFjBJVSXokEJmvF9lsE7NBXLoey4uo+KrqAnoWlh8KMsrb2TEY7lZTnKY2TmzPbKWcw7v2MrU6ls+rDjhNIDC4ofZ0oHktqC+qnQRBjgVo7QYNwILJbyHjOMW5zdVNGv2x3nq/MyU4DEiar2RJKp6mIWg6yminJ66IVKJCaQ4vrV/bpBybsGksq9evZH5dr3H7Tubbbu30OWgkNe5DlTiYtiqPxXJpSDvmLGsFFFTxxMikyhwkwI
[Table]
/JD/Jz5neN1Gq184fU06eOYm6Ssad24w9URbaF</latexit>
Free Energy (  ) =
sup ⌘  · ⌘  ⇤ ( ⌘ ) sup occurs when d ⇤ d⌘ =  =  dR dD <latexit sha1_base64="BdZg5yERDb73UauFiWXKSilchqA=">AAAIDXichVVtcxs1ED4XqNPwlsLHftGQCeNkTCZu02kZ2plMSYEOMJTQtJn2XI9Ot3fWRKeTJTmxI+438Gv4BnzlN/BvWJ3txLYcuJnEe
[Table]
</latexit
[Table]
<
[Table]
+j8Opbr3t4Ap05liKL8FmLRS/sIPMOb5f8fSYOWNU4BCbp3a9WccZdf+AeODIE68b53HBKz
[Table]
+2SjvsE8CDQ8Bfq/I3Oqcn1pu1RnVZt8bfXctG3iFzOiBjhmRTWTyA/eHO/HW61d19ure91Tw8nskbukl9Ik4Rkh+yR38kLsk8YAfIv+
[Table]
/VRvC7Q==</latexit>
⇤ ( ⌘  ⇤ ) <latexit sha
[Table]
="CqGKO5XULoKWsqLr2vdEOk0ApXk=">AAAHp3ichVVtb9s2EGbbrc66vqTdx30RFgRwAsO16wRt0C/B1mErtqHt0KRBK0elqJNMhKJoik7scPoZ+zX7uv6I/ZsdlbixRWcTYOt0z3P3HO9EMVaCl6bX++fGzVtffHm7tfbVna/v3rv/YP3ho8OymGgGB6wQhT6KaQmCSzgw3Ag4UhpoHgt4F5/84PB3p6BLXsi3ZqZgmNNM8pQzatAVrT8ODUxNncdOShZpSCobqpIf2+
[Table]
+
[Table]
+
[Table]
</
[Table]
<
[Table] [Table]
<
[Table]
>
[Table]
<
[Table] [Table]
[Table]
/s1NsctveNgElRhWZKtwRYpJLs2gygZvl9xLi0qSyjHJrbn5Xq9TnGiIDMdWvLUy8ZYWjFXT2mHFGiDHrzR1mm5TwAHgYJvUec7rJzqUu2aiKrcpcXfqGOt/
[Table]
+
[Table]
+
[Table]
</
[Table]
⇤ = max  (1   ) (0)
+  (1)  (  ) <
[Table] [Table]
[Table] [Table]
+Gh2OZENcpfcI23SJw/JIXlCnpMjwsi/5D/ygfzfutd60nrWejGjXr1yEfMTWVmtN58Ak1jTvQ==</latexit>
R ( D ) =
⇤ ( ⌘ )
<latexit sha1_base64="JEfImWcd4tzRYs3lL34XtQI6Eds=">AAAHZXichVXdbhM5FDawNF1+dssCe8MFFlWltIqqBooALStVSxGsltWyiJYKJkQez5mJVY/H8ThtUjMPwy08EU/Aa3A8bWgSp7sjJTlzvu9858exHWspSrux8eXc+Qs/XFxoLP546fKVqz/9vHTtl92yGBgOO7yQhdmLWQlSKNixwkrY0wZYHkt4E+8/8fibAzClKNRrO9LQyVmmRCo4s+jqLt181dxepb/TSJfivVurmhFYttpdWt5Y36gfGhrtE2N5a5HUz8vutYWHUVLwQQ7KcsnK8t3dTW1boDLsoddxzFjBJVSXokEJmvF9lsE7NBXLoey4uo+KrqAnoWlh8KMsrb2TEY7lZTnKY2TmzPbKWcw7v2MrU6ls+rDjhNIDC4ofZ0oHktqC+qnQRBjgVo7QYNwILJbyHjOMW5zdVNGv2x3nq/MyU4DEiar2RJKp6mIWg6yminJ66IVKJCaQ4vrV/bpBybsGksq9evZH5dr3H7Tubbbu30OWgkNe5DlTiYtiqPxXJpSDvmLGsFFFTxxMikyhwkwI+BBAa4zT+u00PEihVJBk7f+z1FFn51kLEh0MK+civ2pxSoezggeHE+hhgI4m0FGAHk2gRwGaTqDpLLovEd3uur9eVLNQAhwx3XWR7eF2qWgTe/iA2VaDaSjP7CNT90TNO0LeMOBloELFFp0nKWUSUgOW9p3r5jwBPZyKnxve9zlcfwyu1D0k7Ez60USXc5OKPjL
[Table] [Table]
[Table] [Table]
/JD/
[Table]
</latexit>
Free Energy (  ) =
sup ⌘  · ⌘  ⇤ ( ⌘ ) sup occurs when d ⇤ d⌘ =  =  dR dD <latexit sha1_
[Table]
="BdZg5yERDb73UauFiWXKSilchqA=">AAAIDXichVVtcxs1ED4XqNPwlsLHftGQCeNkTCZu02kZ2plMSYEOMJTQtJn2XI9Ot3fWRKeTJTmxI+
[Table] [Table]
[Table]
/b+adx45933bjbXbq2//8GHH328cfuTl6YcagbHrBSlPkmoAcElHFtuBZwoDbRIBLxKTr/2+Ksz0IaX8oUdK
[Table] [Table]
[Table]
++bZw9J2tu42NzabWxs11snUdBqVtt1sbWxsNVvb28iTcMjyLKMydiOt5ppyaWEgqdZ0XAbHDip4KlGpFgIuBNCa4kH1dBLuSUjpiaz
[Table]
v0oVdbrOuid0MCqt7brpRkkwqic8OJxBDz10PIOOPfRoBj3y0GQGTerovkD0cWhfvCzrUAwMMRXarunj0SqDNdzDF1S77XVDOuYAmarPK94R8kYeLwXpZ2wGi1IKEftUj6XcztXaogRqNBe
[Table] [Table]
M
[Table]
</latexit> ⌘
[Table]
= <latexit sha1_base64="4RaIMYGMW1R/TXcq8EXp6e2eDx0=">AAAHoXichVVtb9s2EGbbrU67l6bdx34hFgRIA8Ow6wRdMAwItg5dsQ3LiqYNFrsaRZ1kIhRFU3Rih9Vv2K/
[Table]
+137N/sKMeNLTqbAFune56753gnirGWorTd7j
[Table] [Table]
/59LPPH2w+fPSmLCaGwzEvZGFOYlaCFAqOrbASTrQBlscS3sZn33n87TmYUhTqtZ1pGOYsUyIVnFl0RZtPBhamts7jTBFHsZxA5QZgWeQGMd7eud2qot/
[Table] [Table]
BnL4BRNxXIoh66uqaLb6EloWhj8KUtr73KEY3lZzvIYmTmzo7KJeecHbHtFyqZfDZ1QemJB8blSOpHUFtR3iibCALdyhgbjRmCxlI
[Table] [Table]
[Table] [Table]
/tmatX1u/
[Table] [Table]
</
[Table]
/WqISxzjEw5Yo53gbypx8tB+B7xFq9xyXnqUz2WtDuX7XUO5HTFfq352MYw4wW44/aQ0hvpF0u7XBuUjZHxTWRdnnz+s1+n8VUrYvNlbXlet/KlbuVeVahFY5bjY0nxdYImQV0T1FqCq4Z0JKrIZXGaVyIDWV2ngUlRhRrGS+HlC5xLxQqwdJTfYnHc0msHk6AKy5JsDbZwIdmNHkTJ8H3FabXILKEci9iep+vVOsWRgsw0suSp542xtGIun9KOLtAGNXijrdJynwMOAgXfY5wfMHOqS7VnQqpy5xb/w46V/ovIxIKIEs6kfnMC+cKrx93+fvfwx/3to8F8OAWbwYPgYdAO+sGT4Cj4OngRHAdJ8Ffwz8btjTst1vqt9Xvrjzn11salzSfBytf6818heum9</latexit>
Chernoff Point on the TVO Integrand For the unnormalized likelihood ratio log ˜
π 1 ( z ) /π
( z ) , we can intepret the Chernoff point using thermodynamic integration bounds (8)
T −
[Equation]
With π 0 ( z ) =
q ( z | x ) as in TVO [5, 4], we note that the integrand at β 0 = 0 corresponds to the familiar evidence lower bound ( ELBO ), E π
[Equation]
Similarly, at β 1 = 1 , the integrand E π 1 [ · ] = log Z 1 ( x ) + D KL [ π 1 ( z | x ) ||
π 0 ( z )] provides an upper bound.
Since ψ (1) = log p θ ( x ) in (36) in this case, the Chernoff point corresponds to
[Equation]
) or the point where the moment parameters switch from an lower bound to an upper bound.
We visualize this in Fig.
2, noting that the shaded regions corresponding to the KL divergence (see [4]) will have equal area due to (37).
Conclusion
We have presented likelihood ratio exponential families as a common framework for understanding
TVO , RD , IB , and RDC optimizations in terms of conjugate duality and hypothesis testing.
These insights may be useful for improving mutual information estimators which leverage intermediate distributions [28], learn a binary classiﬁer distinguishing samples from ˜ π 0 , ˜ π
1 [29], or involve a neural network ‘critic’ whose optimal function output is the true likelihood ratio [30].
While it is natural to introduce additional sufﬁcient statistics from the exponential family perspective, thermodynamic integration bounds and hypothesis testing interpretations remain to be clariﬁed in higher dimensions as in RDC .
Further exploring the approach of Gao and Chaudhari [9], for evolving model parameters and Lagrange multipliers along the equilibrium surface of solutions to Eq.
(5), is an exciting future direction.
Beyond the applications shown in [9], this could lead to replacing heuristics such as KL annealing in β - VAE with more principled dynamics for β over the course of optimization.
A Conjugate as a KL Divergence
When considering an exponential family of the form
[Equation]
[Equation]
we show that ψ ∗ ( η ) takes the form of a KL divergence when considering a base measure π 0 ( z ) .
[Equation]
) where we have added and subtracted a factor of E π βη log π 0 ( z ) in the fourth line.
If π 0 ( z ) is constant with respect to z using, for example, the uniform measure, then D KL [ π β η ( z )
||
π 0 ( z )] reduces to the familiar deﬁnition of the conjugate ψ ∗ ( η ) as the negative entropy E π βη log π β η ( z ) [17].
B KL Divergence as a Bregman Divergence
For an exponential family with partition function ψ ( β ) and sufﬁcient statistics φ ( z ) over a random variable z , the Bregman divergence D ψ corresponds to a KL divergence.
Recalling that ∇ β ψ ( β ) =
η β = E
[ φ ( z )] , we simplify the deﬁnition of the Bregman divergence to obtain
[Equation]
where we have added and subtracted terms involving the base measure π 0 ( z )
[Equation]
and used the deﬁnition of our exponential family from (40).
The Bregman divergence D ψ is thus equal to the KL divergence with arguments reversed.
B.1 Canonical Divergence
We can also show that the Bregman divergences D ψ , D ψ ∗ are equivalent up to reordering of the arguments
[Equation]
(44) where we abbreviate η (cid:48)
=
η β (cid:48) and η = η β .
The conjugacy relationships
[Equation]
(45) can be used to translate between these dual divergences
[Equation]
The intermediate expression (46) is known as the canonical form of the divergence [18]
[Equation]
Comparing with the expression for Legendre duality in (5), note that the correspondence between β and η implies that the divergence vanishes, since both parameterizations refer to the same distribution
[Equation]
C Information Bottleneck as Rate-Distortion
The Information Bottleneck (IB) method [8] deﬁnes the ‘relevant information’ in a representation,
I ( Y : Z ) , via another variable of interest Y , often taken to be a label.
The IB objective then seeks a minimal encoding Z which maintains a given level of predictive ability about the target.
[Equation]
where we let I q reﬂect the exact mutual information for the true data and label distributions q ( x ) q ( y | x ) with a given encoding function q ( z | x ) .
When the desired information constraint equals the total information I c
= I q ( X ; Y ) that the data source contains about the label, (49) corresponds to the problem of ﬁnding the minimal sufﬁcient statistics z for y with respect to x .
The IB objective generalizes this optimization for smaller values of I c .
Since I q ( Y ; Z )
=
H q ( Y ) − H q
( Y | Z )
= − E q log q ( y ) + E q log q ( y | z ) , we can ignore the label entropy as a constant with respect to z .
While it may be difﬁcult to obtain the true posterior q ( y | z ) of the labels given latent variables , we can instead optimize a variational classiﬁer p ( y | z ) .
This provides an lower bound on the mutual information since D KL [ q ( y | z ) || p ( y | z )] ≥ 0 and is also known as the ‘test channel’ in rate-distortion theory ([7] Ch. 13).
Applying this inequality within the unconstrained
IB Lagrangian,
[Equation]
where y ( x ) indicates the label of a given data point.
As shown in Tishby et al.
[8], the Information Bottleneck is a special case of rate-distortion with
[Equation]
Comparing (50) with (51), note that E q [ q ( y | x )] is a constant, leaving c ( y ( x ) , z )
= − E q ( y ( x ) | z ) [ q ( y | z )] as the effective distortion measure.
If this quantity is intractable, we can instead deﬁne the distortion function using p ( y | z ) as above.
D Bregman Information and Jensen Gaps
Imagine we are interested in minimizing the expected Bregman divergence D f to a single representative point, which may then be thought of as the optimal codeword for R = 0 in a rate-distortion scenario using a Bregman divergence distortion.
Banerjee et al.
[11] show that, regardless of the divergence, the minimizing point will be the mean with respect to a desired measure, and the expected divergence will be a gap in Jensen’s inequality for the function f .
Theorem D.1 (Bregman Information, [11]) .
Let X be a random variable that takes values in X
= { x i } ni
=1 ⊆ S ⊆
R d following a positive probability measure ν such that E ν
[ X ] ∈ ri ( S ) .
Given a Bregman divergence D
[Equation] [Equation]
] , the problem:
[Equation]
has a unique minimizer given by the mean s ∗ = µ = E ν
[ X ] .
At this arg min , J ν,f ( s ∗ ) corresponds to a gap in Jensen’s inequality for the convex function f and expectations with respect to ν .
[Equation]
Proof.
Consider a point s and the mean µ =
E ν [ X ] , both in ri ( S ) so that J ν,f is well deﬁned.
[Equation]
with equality only when s = µ =
E ν
[ X ] if f is strictly convex.
Then,
[Equation]
E Gap in Jensen’s Inequality for ψ ( β )
In this section, we analyse the Bregman Information and Jensen gap associated with the log partition function of the likelihood ratio exponential family.
As in Sec. 3, this corresponds to the variational representation of Grosse et al.
[12], while maximizing the Jensen gap will lead to the Chernoff point.
We give proofs of intermediate results at the end of the section.
With D ψ as the divergence associated with the convex function f ( X ) = ψ ( β ) , we take the expected
Bregman divergence using a convex combination ( { 1 − α, α } ) over arguments { β 0 , β 1 } .
[Equation]
where Theorem D.1 shows that the minimizer µ ( α ) β occurs at the mean of the arguments.
After simplifying, we can see that this corresponds to a Jensen’s inequality for the convex function ψ ( β )
[Equation]
For the case of β 0 = 0 and β 1 = 1 , we see that the optimal parameter is simply µ ( α ) β
= α
so that
[Equation]
Nielsen and Nock [25] demonstrate the following lemma, showing the relationship between the
Jensen’s gap and Rényi divergence within an exponential family.
Lemma E.1.
The Rényi divergence of order α between two distributions (indexed by natural parameters β 0 and β 1 ) within an exponential family (with log partition function ψ ( β ) ) has the form of a gap in Jensen’s inequality J ψ, { 1 − α,α } , { β 0 ,β 1 } , abbreviated J ψ,α
[Equation]
[Equation] [Equation]
Proof.
The ﬁrst equality follows from the deﬁnition of D α , and corresponds to the Chernoff coefﬁcient in (34) or Jensen gap J ψ,α .
We demonstrate the second equality for the likelihood ratio family in App.
E.1, or see Nielsen and Nock [25] for the general case.
Van Erven and Harremos [31] show that the scaled Renyi divergence (1 − α ) D α
[ π β 0 : π β 1 ] is concave.
For given endpoint distributions, we can thus seek to maximize J ψ,α as a function of α .
Lemma E.2.
Maximizing the Jensen’s gap J ψ,α obtained from arguments { β 0 , β 1 } of the convex function ψ ( β ) , with respect to the choice of mixing weight
[Equation]
leads to the following condition
[Equation]
Proof.
See App.
E.2 for proof.
For β 0 = 0 and β 1 = 1 , this suggests that the expected sufﬁcient statistics η α (with natural parameter µ ( α )
β
= α ) should match the marginal likelihood log p θ ( x ) .
[Equation]
Lemma E.3.
At the maximum in Eq.
(60) , consider the distribution π β ∗ α in the same exponential family, with natural parameter β ∗ α = µ ( α ) β
=
[Equation]
1 , the Bregman divergences to each endpoint { β 0 , β 1 } are the same.
[Equation]
[Equation]
Since the Bregman divergence D ψ within an exponential family corresponds to the KL divergence, we can equivalently write
[Equation]
Proof.
See App.
E.
for Bregman divergence derivations.
See discussion around (6) for the relationship between the Bregman and KL divergence.
Note that we could also construct a Jensen gap from the dual divergence ψ ∗ ( η β ) =
D KL [ π β || π 0 ] ,
[Equation]
where η ∗ λ = λ
η 0 + (1 − λ )
η 1 .
E.1 Rényi Divergence as a Jensen Gap
We consider the Rényi α divergence between any two distributions π β 1 and π β 0 in our exponential family, so that π β ( z | x ) =
π 0 ( z ) 1 − β π 1 ( z ) β /Z β ( x ) .
Noting that the scaling factor α − 1 ≤
[Equation]
we proceed to show that the scaled divergence is equal to a gap in Jensen’s inequality:
[Equation]
E.2 Chernoff Point and Maximizing the Jensen Gap
In this section, we derive the optimal solution for the optimization deﬁning the Chernoff information point.
In particular, we optimize the Bregman Information or Jensen gap J
[Equation]
where we use arbitrary endpoints β 0 , β 1 and mixing parameter α to highlight the arithmetic mean in the argument of the ﬁnal term.
This will match (35) when β 0 = 0 , β 1 = 1 and t = β .
Now, we can differentiate with respect to α , letting β α =
(1 − α ) β
+ αβ 1 .
We use the product rule and the identity dψ ( β ) /dβ
=
η β in the last term to obtain
[Equation]
[Equation]
where η β α indicates the expected sufﬁcient statistics, or dual parameter, corresponding to the natural parameter β α
= (1 − α ) β
+ αβ 1 .
E.3 Equal KL Divergences Derivation
We show that the KL divergences that constitute J α,ψ are equal at the critical point η α = ψ ( β 1 ) − ψ ( β 0 ) β 1 − β:
[Equation]
We have shown that the two divergences are equal when our condition on η α holds.
Further, observe that each divergence amounts to a Jensen gap J α,ψ with α = β α − β 0 β 1 − β 0 : This is more apparent for β 0 = 0 and β
= 1 , where this simpliﬁes using α = β α − β 0 β 1 −
β
=
β α :
[Equation] [Equation]