T HE D UAL
I NFORMATION B OTTLENECK
Anonymous authors
Paper under double-blind review
A BSTRACT
The Information Bottleneck (IB) framework is a general characterization of optimal representations in large scale learning that suggests an intriguing interpretation of efﬁcient representations in deep learning in particular.
It is based on the optimal trade off between the representation complexity and accuracy, both of which are quantiﬁed by mutual information.
The problem is solved by alternating projections between the encoder and decoder of the representation, which can be performed locally at each representation level.
The non-parametric IB framework, however, has several drawbacks: (i) the difﬁcult estimation of information measures in high dimensions; (ii) the inability to extend the encoder beyond the training data - the need to to know the full joint distribution of the inputs and labels; (iii) the IB does not necessarily optimize the actual prediction of unseen labels, which can be critical when training the encoder from ﬁnite samples.
Here we present a new framework, the Dual Information Bottleneck (dualIB), which improves all the above drawbacks.
By switching the order in the KL-divergence between the representation decoder and data, formally the dual distortion, the optimal decoder becomes the geometric rather than the arithmetic mean of the input points.
This conversion preserves exponential forms of the original distribution and consequently conserves the original features of the data. We show how to solve this new representation learning formulation and illustrate it’s favorable properties over the original IB when trained from a small sample.
We also analyze the critical points of the dualIB and discuss their importance for the quality of this approach.
I NTRODUCTION 1. T HE I NFORMATION B OTTLENECK METHOD
The Information Bottleneck ( IB ) method (Tishby et al., 1999), is an information-theoretic framework for describing efﬁcient representations of a large scale “input” random variable X (input patterns), for predicting an “output” variable Y (desired label).
In this setting the joint distribution of X and Y, p ( x , y ) deﬁnes the problem, or rule, and the training data are a ﬁnite sample from this distribution.
In general, we assume that p ( y | x ) is strictly stochastic, and hence bounded away from { 0 , 1 } 1 .
The representation variable ˆ X is in general a stochastic function of X which forms a Markov chain Y
→ X → ˆ X, and only depends on Y through the input X. We call the map p (ˆ x | x ) the encoder of the representation and denote by p ( y | ˆ x ) the Bayes optimal decoder for this representation; i.e., the best possible prediction of the desired label Y from the representation ˆ X.
The IB has direct successful applications for representation learning in various domains, from vision and speech processing (Ma et al., 2019), to neuroscience (Schneidman et al., 2001), and Natural
Language Processing (Li & Eisner, 2019).
Due to the notorious difﬁculty in estimating mutual information in high dimension, variational approximations to the IB have been suggested and applied also to Deep Learning (Alemi et al., 2016; Parbhoo et al., 2018).
For large scale learning, when almost all input patterns can be considered typical in the information theoretic sense (Cover & Thomas, 2006), the cardinality of the typical inputs is characterized by the pattern’s entropy, H ( X ) =
− (cid:80) x p ( x ) log p ( x ) .
Under similar typicality assumption, the mutual information (MI) of the encoder, I ( X ; ˆ X ) = (cid:80) x , ˆ x p ( x , ˆ x ) log p ( x | ˆ x ) p ( x ) characterizes the cardinality of the typical representations, ˆ X. On the other hand, the mutual information of the Bayes optimal decoder, I ( Y ; ˆ X ) = (cid:80) y , ˆ x p ( y , ˆ x ) log p ( y | ˆ x ) p ( y ) , is equivalent to the cross-entropy generalization error. The IB trade off between the encoder and decoder mutual information values is deﬁned by the minimization of the Lagrangian:
[Equation]
independently over the convex sets of the normalized distributions, { p β (ˆ x | x ) } , { p β (ˆ x ) } and { p β ( y | ˆ x ) } , given a positive Lagrange multiplier β .
As shown in (Tishby et al., 1999; Shamir et al., 2010), this is a natural generalization of the classical concept of Minimal Sufﬁcient Statistics (Cover & Thomas, 2006), where the estimated parameter is replaced by the output variable Y and exact statistical sufﬁciency is characterized by the mutual information equality: I (ˆ X ; Y )
=
I ( X ; Y ) .
The minimality of the statistics is captured by the minimization of I ( X ; ˆ X ) , due to the Data Processing
Inequality (DPI).
However, non-trivial minimal sufﬁcient statistics only exist for very special parametric distributions known as exponential families (Brown, 1986).
Thus in general, the IB relaxes the minimal sufﬁciency problem to a continuous family of representations ˆ X which are characterized by the trade off between compression, I ( X ; ˆ X ) ≡ I X , and accuracy, I ( Y ; ˆ X ) ≡ I Y , along a convex line in the Information-Plane ( I Y vs. I X ).
When the rule p ( x , y ) is strictly stochastic, the convex optimal line is smooth and each point along the line is uniquely characterized by the value of β .
We can then consider the optimal representations ˆ x = ˆ x ( β ) as encoder-decoder pairs: ( p β ( x | ˆ x ) , p β ( y | ˆ x )) 2a point in the continuous manifold deﬁned by the Cartesian product of these distribution simplexes.
We also consider a small variation of these representations, δ ˆ x, as an inﬁnitesimal change in this (encoder-decoder) continuous manifold (not necessarily on the optimal line(s)).
1. IB AND R ATE -D ISTORTION T HEORY
The IB optimization trade off can be considered as a generalized rate-distortion problem (Cover & Thomas, 2006) with the distortion function between a data point, x and a representation point ˆ x taken as the KL-divergence between their predictions of the desired label y:
[Equation]
The expected distortion E p β ( x , ˆ x ) [ d IB ( x , ˆ x )] for the optimal decoder is simply the label-information loss: I ( X ; Y ) − I (ˆ X ; Y ) , using the Markov chain condition.
Thus minimizing the expected IB distortion is equivalent to maximizing I (ˆ X ; Y ) , or minimizing equation 1.
Minimizing this distortion is equivalent to minimizing the log -loss or the cross-entropy loss, as done in most deep learning applications, and it upper-bounds other loss functions such as the L 1 -loss (due to the Pinsker inequality, or (Painsky & Wornell, 2018)).
The Pinsker inequality shows that both orders of the cross-entropy upper bound to the L 1 -loss, or total-variation distance, min { D [ q || p ] , D [ p || q ] } ≥
[Equation]
1. T HE IB E QUATIONS
For discrete X and Y, a necessary condition for the IB (local) minimization is given by the three self-consistent equations for the optimal encoder-decoder pairs, known as the IB equations :
[Equation]
where Z ( x ; β ) is the normalization function. Iterating these equations is a generalized, BlahutArimoto, alternating projection algorithm (Tusnady & Csiszar, 1984; Cover & Thomas, 2006) and it converges to a stationary point of the Lagrangian, equation 1 (Tishby et al., 1999).
Notice that the minimizing decoder, (equation 3- ( iii ) ), is precisely the Bayes optimal decoder for the representation ˆ x ( β ) , given the Markov chain conditions.
1. C RITICAL POINTS AND CRITICAL SLOWING DOWN
One of the most interesting aspects of the IB equations is the existence of critical points along the optimal line of solutions in the Information-Plane (i.e. the information curve).
At these points the representations change topology and cardinality (number of clusters) (Zaslavsky & Tishby, 2019;
Parker et al., 2003)
and they form the skeleton of the information curve and representation space.
Under the strict stochastic assumption, the information-curve is a smooth function of the Lagrange multiplier β , but its derivative may not be smooth.
Critical points are bifurcations of the solutions, which are values of β for which two different solutions (representations) co-exist.
To identify such points we perform a perturbation analysis of the IB equations, as in (Zaslavsky & Tishby, 2019).
Taking a small perturbation of the representation, denoted for brevity by δ ˆ x, the changes in the log encoder and log decoder that satisfy equation 3 for a given β can be determined through the nonlinear eigenvalues problems 3 :
[Equation]
with the two square matrices C deﬁned by:
[Equation]
As shown in (Zaslavsky & Tishby, 2019), these two matrices have the same eigenvalues and have non-trivial eigenvectors (i.e., different co-existing optimal representations) precisely at the critical values of β , the bifurcation points of the IB solution.
At these points the cardinality of the representation ˆ X (the number of “IB-clusters”) changes due to splits of clusters, resulting in topological phase transitions in the encoder.
These critical points form the “skeleton” of the topology of the optimal representations.
Between critical points the optimal representations change continuously (with β ).
The important computational consequence of critical points is known as critical slowing down (Tredicce et al., 2004).
For binary Y, near a critical point the convergence time, τ β , of the iterations of equation 3 scales like: τ β ∼ 1 / (1 − βλ 2 ) , where λ 2 is the second eigenvalue of either
C IB yy (cid:48) or C IB xx (cid:48) .
At criticality, λ 2 (ˆ x ) = β − 1 and the number of iterations diverges.
This phenomenon dominates any local minimization of equation 3 which is based on alternate encoder-decoder optimization.
To illustrate the importance of the critical points and the critical slowing-down we consider a lowdimensional problem that is easy to visualize.
Taking the joint distribution, p ( y , x ) with a binary Y and only 5 possible inputs. For a given β , the Blahut-Arimoto iterations converge to a stationary solutions of the IB equations , which amounts to discrete points in {+the interval [0 , 1] - the simplex of the p ( y ) distributions - for each value of the Lagrange multiplier, β . The evolution of the optimal decoder, p β ( y = 
 | ˆ x ) , ∀ ˆ x ∈ ˆ X, as a function of β , is a bifurcation diagram (Figure 1 ( a ) ).+}
The critical points of the IB equations correspond to bifurcations in the decoders’ graph. At these critical points the non-trivial eigenvalues of C IB yy (cid:48) ( β ) cross the β − 1 value (Figure 1 ( b ) ), and the number of iterations diverges (critical slowing down) (Figure 1 ( c ) ). 1. D RAWBACKS OF THE IB The main theoretical and practical drawback of the IB is it’s apparent reliance on the explicit knowledge of the joint distribution p ( x , y ) .
In the context of representation learning this is often considered a major ﬂaw, as we are always given only a sample from the distribution. While the generalization properties of the ﬁnite sample IB representations have been studied in the past (Shamir et al., 2010), the main problem remains the extension of the encoder , p (ˆ x | x ) [-log 2 ( β ) 0 .0 .0 .
-], when trained from a small sample, to new unobserved patterns x ∈ X. The standard approach in representation learning is to assume (at least implicitly) the existence of features of x with their topology, or a speciﬁc parametric form for the encoder. The original IB is completely non-parametric and does not assume any topology on the input space, beyond label similarity. A related drawback of the IB is it’s poor scalability with the size of the pattern-space X, which makes the reliable estimation of the encoder mutual information impractical for large learning problems.
[Figure] [Figure] [Figure] [Figure]
[Figure]
p β ( y =
| ˆ x ) 0 .2 .5 .
0 ( b ) log 2 ( β ) 0 .0 .0 .
.λ
( ˆ x ) β −0 .2 .5 .
0 ( c ) log 2 ( β )    l o g 2 ( # i t e r s )
1. C ONTRIBUTIONS OF THIS WORK
Our main contribution is by providing a general formal framework resolving the encoder extension drawback of the IB.
While the use of speciﬁc features and variational approximations to the
IB are not new, here we take a different approach assuming that p ( y | x ) can be approximated by exponential families.
This deﬁnes a new rate-distortion problem, the Dual Infromation Bottleneck (dualIB).
We show that the new dualIB formulation preserves the sufﬁcient statistics of the original distribution for all values of the trade off parameter, β .
A desired property upon predictions of a parametrized distribution. Furthermore, it obtains a signiﬁcant improvement of the ﬁnite sample label prediction. We demonstrate numerically that the dualIB encoder-decoder pairs improve the prediction of unseen labels when trained from small samples, compared to the original IB. We also discuss in detail the critical points of the new formulation and their relation to those of the original IB.
Interestingly, as we elaborate in the following sections, all these desirable properties are achieved by merely switching the order of the arguments in the KL-divergence of the IB distortion.
The remainder of the paper is organized as follows.
We present and solve the dualIB problem in § 2.
We discuss the dualIB’s critical points in § 2.1 and prove a general relation between the critical points of the dual and original IB in § 2.2.
In particular, we show that the dual critical points give the best approximation to the original IB information curve.
In § 3 we discuss the dualIB for exponential families, dualExpIB, and demonstrate the advantages of our framework in this case.
We also demonstrate numerically the improvement in the label prediction of the dualIB when trained from a small sample.
We conclude in § 4 with further extensions and possible applications.
T HE DUAL IB: M AXIMIZING THE P REDICTION I NFORMATION
Supervised learning is generally separated into two phases: the training phase, where the internal representations are formed from the training data, and the prediction phase, where these representations are used to predict labels of new input patterns.
We add to our Markov chain another variable, ˆ Y, the predicted label which obtains the same values as Y, but distributed differently:
[Equation]
The left-hand part of this chain describes the representation training, and the right-hand part is the
Maximum Likelihood (ML) prediction using these representations (Slonim et al., 2006).
So far the prediction variable
ˆ Y has not been part of the IB optimization problem.
It has been implicitly assumed that the Bayes optimal decoder , p β ( y | ˆ x ) , which minimizes the IB distortion at a given β , is also the best choice for making predictions of ˆ Y from the representation ˆ X β through the righthand Markov chain.
That is, denoting p β
(ˆ y | ˆ x ) ≡ p β
( y | ˆ x ) the ML decoder is the mixture over the internal representations:
[Equation]
This, however, is not necessarily optimal when the encoder and decoder are trained from ﬁnite samples (Shamir et al., 2010), or when the decoder is not Bayes optimal.
We deﬁne the dualIB distortion by merely switching the order of the arguments in the KL-divergence of the original IB distortion, namely:
[Equation]
which in geometric terms is known as the dual distortion problem (Felice & Ay, 2019).
That is, given the “input” random variable X (input patterns)
, the ”output” random variable Y (desired label) and the “representation” ˆ X, the ML optimal “predicted label” ˆ Y is given by the mixture distribution, equation 7.
The dualIB optimization can be written as the following rate-distortion problem:
[Equation]
with the average distortion given in terms of mutual information on ˆ Y, I (ˆ X ; ˆ Y ) and I ( X ; ˆ Y ) :
[Equation]
This is similar to the known IB relation: E p ( x , ˆ x ) [ d IB ( x , ˆ x )] =
I ( Y ; X ) − I
( Y ; ˆ X ) with an extra positive term ( b ) .
Both terms, ( a ) and ( b ) , vanish precisely when ˆ X is a sufﬁcient statistic for X with respect to ˆ Y, since we can then reverse the order of X and ˆ X in the Markov chain (equation 6).
This replaces the roles of Y and ˆ Y as the variable for which ˆ X β are approximately minimally sufﬁcient statistics.
In that sense the dualIB shifts the emphasis from the training phase to the prediction phase.
This implies that minimizing the dualIB functional maximizes the mutual information between Y and ˆ Y,
I ( Y ; ˆ Y ) , as well as the mutual information I ( X ; ˆ Y ) .
This is illustrated in ﬁgure 4 ( b ) - c .
The next theorem states the form of the solutions of the Dual Information Bottleneck :
Theorem 1.
The minima of equation 9, can be obtained by generalized Blahut-Arimoto iterations between the encoder and the decoder as in the original IB , with the following modiﬁcations: (i)
Replace the distortion by its dual in the encoder update; (ii) Update the decoder by the encoder’s ”geometric” mean of the data distributions p ( y | x ) .
The proof is given in § A.2.
The alternating projections between the encoder and decoder, which converge to a solution of the dualIB at a given value of the Lagrange multiplier β , are implemented by the following iterative algorithm 5 :
Unless
stated otherwise, log =
ln ≡ log e
Here the Z ’s are standard normalization factors (partition functions).
As in the IB, the encoder update (row 3 ) and decoder update (row 6 ) are the core of the algorithm.
Figure 2 illustrates the behavior of the dualIB algorithm for the same distribution as in Figure 1.
[-) log 2 ( β ) 0 .0 .0 .
-]The overall behavior is very similar, but the critical points (representation splits) slightly change their location. In the following sections we provide a detailed analysis and comparison between the two frameworks. [Figure]
.p
β ( y =
| ˆ x ) 0 .2 .5 .
0 ( b ) log 2 ( β ) 0 .0 .0 .
.λ
( ˆ x ) β −0 .2 .5 .
0 ( c ) log 2 ( β )    l og 2 ( # i t e r s )
2. T HE C RITICAL P OINTS OF THE dualIB
As discussed in § 1.4 the skeleton of the IB optimal bound (the information curve) is constituted by the critical points in which the topology (cardinality) of the representation changes.
A similar stability analysis of the dualIB equations reveals similar conditions for the critical points.
Theorem 2.
The dualIB critical points are detected by non-trivial solutions of the nonlinear eigenvalue problem:
[Equation]
with the matrices C dualIB given by:
[Equation]
The proof to theorem 2 is given in § A. [Equation]
Lemma 3.
The matrices C dualIB xx (cid:48)
(ˆ x ; β ) , C dualIB yy (cid:48)
(ˆ x ; β ) have the same eigenvalues { λ i } , with λ 1 (ˆ x ) = 0 .
With binary Y , the critical points are obtained at λ 2 (ˆ x ) =
β − 1 .
The proof of lemma 3 given in section § A.3.1.
As in the IB, at the critical points, β dualIB c , the partial derivatives of the encoder and decoder with respect to β , ∂ log p
β
( x | ˆ x ) /∂β, ∂ log p β
( y | ˆ x ) /∂β , have multiple (at least two) values.
This results in discontinuities (cusps) in the encoder and decoder mutual information values as functions of β along the optimal line, with an undeﬁned second derivative.
2. T HE I NFORMATION P LANE OF THE dualIB
The Information Plane , I X
= I (ˆ X ; X ) vs. I
Y
=
I (ˆ X ; Y ) , is the standard visualization of the compression-prediction trade off of the IB.
It can be deﬁned for any encoder once the decoder is the Bayes optimal (equation 3- ( iii ) ), for which the I Y is the actual information of the representation on the desired label (Tishby et al., 1999).
Comparing the dualIB information curve to the IB curve shows the quality of this approximation.
Figure 3(a) depicts this comparison.
While we know that I IB Y ( β ) is always higher, the two curves are almost indistinguishable.
To better understand the relationship between these two curves, we look at the values of I X
and I Y as functions of the corresponding β (Figure 3(b),(c)).
The important role of the critical points is revealed as the corresponding cusps along these curves.
As we argue below, the IB information values are strictly below those of the dualIB, but the distance between them is minimized precisely at the dual critical points. [Figure] [Figure] ( Y ; ˆ
[Figure]
) / I (
Y ; X ) [Figure] The functions approach each other for large β . Lemma 4.
I X ( β ) and I Y ( β ) , along the optimal lines, are non-decreasing piece-wise concave, functions of β .
When their second derivative (with respect to β ) is deﬁned, it is strictly negative.
Lemma 5.
For any sub-optimal information curve ( I X , I Y ),
I IB X ( β ) ≤ I X ( β ) and I IB Y ( β ) ≤ I Y ( β ) , for all values of β .
Proofs of the above are given in § A.4.
The information plane properties are summarized by the following theorem.
Theorem 6.
(i)
The critical points of the two algorithms alternate: for each critical point, β dualIB c ≤ β IB c .
(ii)
The distance between the two information curves is minimized precisely at the dualIB critical points β dualIB c .
(iii)
The two curves approach each other as β → ∞ .
Proof.
The proof follows from lemmas 4 and 5, together with the critical points analysis above, and is only sketched here.
As the encoder and decoder at the critical points, β IB c and β dualIB c , have different left and right derivatives, they form cusps in the curves of the mutual information ( I X and
I Y )
as functions of β .
These cusps can only be consistent with the optimality of the IB curves if β dualIB c < β IB c (this is true for any sub-optimal distortion), otherwise the curves intersect.
Moreover, at the dualIB critical points, the distance between the curves is minimized due to the strict concavity of the functions segments between the critical points.
As the critical points imply discontinuity in the derivative, this results in a ”jump” in the information values.
Therefore, at any β dualIB c the distance between the curves has a (local) minimum.
This is depicted in Figure 3, comparing I X ( β ) and I Y ( β ) and their differences for the two algorithms.
The two curves approach each other for large β since the two distortion functions become close in the low distortion limit (as long as p ( y | x ) is bounded away from 0 ).
T HE E XPONENTIAL F AMILY dualIB
As discussed in § 1, one of the major drawbacks of the IB is the extension of the decoder inputs from an empirical sample to new unseen patterns.
To address this issue we can assume the existence of features, or statistics of the pattern, A r ( x ) , which are sufﬁcient or approximately sufﬁcient for the prediction of the label Y.
That is, we consider the rule’s distribution, p ( y | x ) , to be an exponential family with these features.
Exponential families form the elegant theoretical core of parametric statistics and often emerge as maximum entropy (Jaynes, 1957) or stochastic equilibrium distributions, subject to observed expectation constraints.
They also form the class of parametric distributions for which exact, ﬁnite dimensional and additive, Minimal Sufﬁcient Statistics exist (Kullback, 1959).
Such distributions should also best match the rational behind the IB as extracting approximate sufﬁcient statistics.
A key property of exponential families is their invariance under geometric averages. For this technical reason the dualIB encoder and decoder preserve the exponential form of the original data distribution for all values of β . Furthermore, the new (assumed limited to a ﬁnite dimension d ) set of features/statistics are precisely the empirical expectations of these features. These d empirical averages are the only numbers needed for predicting the label at the required accuracy, hence their advantage in reducing the encoder’s computational complexity.
Assuming that the rule distribution is of the form:
[Equation]
where A r ( x ) are d linearly independent functions of the input x and λ r ( y ) are functions of the label y, or the parameters of this exponential family.
The λ ( y ) can also be considered Lagrange multipliers associated with the constraints conditional expectations E p ( x | y )
[ A r ( x )] in entropy maximization.
The normalization factors, Z y | x ( x ) , are written, for brevity, as λ 0 x ≡ log( (cid:80) y (cid:81)
dr =1 e −
λ r ( y )
A r ( x ) ) with A 0 ( x ) ≡ 1 .
We do not constrain the marginal p ( x ) .
The important fact about the exponential form is that all the mutual information, I ( X ; Y ) , is fully captured by the d conditional expectations, E p ( x | y )
[ A r ( x )] , since these are the (minimal) sufﬁcient statistics for the parameters.
This means that all the relevant information (in the training sample) is captured solely by d -dimensional empirical expectations.
This can lead to a huge reduction in computational complexity (from dim( X ) to d ).
We next show that for the dualIB, this dimension reduction is preserved or improved along the dual information curve, for all values of β .
Theorem 7. ( dualExpIB ) For data from an exponential family, equation 13, the optimal decoder of the dualIB , at a given β , is given by:
[Equation]
and the respective encoder:
[Equation]
with the constraints and multipliers expectations,
[Equation]
The complete derivations for this section are given in § A.5.
This deﬁnes a simpliﬁed iterative algorithm to solve the dualExpIB problem, since we can replace the decoders’ update rule at each iteration in the dualIB algorithm (Algorithm 1, row 6 ) with the simpliﬁed expression given in equation 14.
As a concrete illustration to the performance of the dualExpIB we examine a problem in which the rule distribution is given by an exponential family, p ( y | x ) ∼ N (0 , a + b | sin( cπ x ) | ) , as in Globerson & Tishby (2003). Setting the constants a, b, c we produce a sample S (of size N = 1000 ) on which we calculate the empirical distribution, ˆ p ( y | x ) and the empirical sufﬁcient statistics ˆ A r ( x ) . Given the assumption on the rule we assess the exponential form empirical distribution, ˆ p exp ( y | x ; ˆ A r ( x )) .
We then evaluate the IB on both empirical distributions, ˆ p and ˆ p exp , and compare it to the dualExpIB given ˆ A r ( x ) , λ r ( y ) 6 . The comparison between the IB and the dualExpIB, is given in Figure 4 ( a ) as function of the continuous ˆ y for a ﬁxed ˆ x. It can be seen that the dualExpIB better captures the structure of the original data and is much less affected by the sampling noise. Next, we consider the Information Plane with respect to ˆ Y, where the expected maximization by the dualExpIB is observed, Figure 4 ( b ) . To evaluate the prediction ability of the dualExpIB we consider the L 1 - expected loss with respect to the original distribution.
Looking at Figure 4 ( c ) it is apparent that for all values of the trade off parameter, β , the dualExpIB minimizes the loss providing the superiority in prediction.
[Figure]
C ONCLUSION
We presented here the Dual Information Bottleneck (dualIB), a framework to preform the Information Bottleneck (IB) compression in a parametric setting.
We have shown that the dualIB has several interesting properties: (i) it provides a good approximation to the original IB while using the low relevant dimensions of the original data.
This can signiﬁcantly reduce the complexity of ﬁnding good IB representations; (ii) it optimizes the information between the representation and the predicted label rather than the desired label as in the original IB.
As we have shown, this can improve the generalization error when trained on small samples since the predicted label is the one used in practice; (iii) It preserves the exponential form of data from exponential families, while reducing the dimensionality of the compressed representations.
This important property was known to be satisﬁed by the Gaussian IB (Chechik et al., 2005) but not known for other distributions.
The Gaussian case is self-dual in that sense.
Generalizing this property to other exponential families was an open problem for many years; (iv) The exponential form of the optimal encoder-decoder pairs allows for the application to distributions with special symmetries, which can be naturally expressed in this form.
We conclude that the exponential Dual Information Bottleneck (dualExpIB) framework allows us now to approach problems previously considered intractable by the IB.
Furthermore, it may allow better predictions to IB applications using ﬁnite samples or data with low internal dimensionality or special symmetries.
A A PPENDIX A. DUAL IB M ATHEMATICAL FORMULATION
As presented in § 1.6 the dualIB is solved with respect to the full Markov chain (equation 6) in which we introduce the new variable, ˆ Y, the predicted label .
Thus, in analogy to the IB we want to write the optimization problem in term of ˆ Y.
Developing the expected distortion
we ﬁnd:
[Equation]
Allowing the dual optimization problem to be written as:
[Equation]
A. T HE D UAL IB SOLUTIONS
To prove theorem 1 we want to obtain the normalized distributions minimizing the dualIB ratedistortion problem.
Proof.
( i )
Given that the problem is formulated as a rate-distortion problem the encoder’s update rule must be the known minimizer of the distortion function (Cover & Thomas, 2006).
Thus the
IB encoder with the dual distortion is plugged in.
( ii )
For the decoder, by considering a small perturbation in the distortion d dualIB ( x , ˆ x ) , with α (ˆ x ) the normalization Lagrange multiplier, we obtain:
[Equation]
Hence, minimizing the expected distortion becomes:
[Equation]
which yields Algorithm 1, row 6 .
Considering the dualIB encoder-decoder
[Equation]
Algorithm
[Equation]
we ﬁnd that E p β ( x , ˆ x ) [ d dualIB ( x , ˆ x )] reduces to the expectation of the decoder’s log partition function: [Equation] x
[Equation]
x
[Equation]
(cid:35) =
− E p β (
[Equation]
x )
[Equation]
log
[Equation]
y | ˆ x (ˆ x ; β ) (cid:3) .
A. S TABILITY ANALYSIS
Here we provide the detailed stability analysis allowing the deﬁnition of the matrices
[Equation]
(equation 12) and which allows us to claim that they obey the same rules as the C matrices (equation 5) in equation 4.
Considering a variation in ˆ x we get:
[Equation]
Substituting equation 18 into equation 17 and vice versa one obtains:
[Equation]
We now deﬁne the C dualIB matrices as follows:
[Equation]
Using the above deﬁnition we have an equivalence to equation 4 in the form of:
[Equation]
Note that for the binary case, the matrices may be simpliﬁed to:
[Equation]
A.3. P ROOF OF L EMMA
We show that the C dualIB matrices share the same eigenvalues with λ 1 (ˆ x ) = 0 .
Proof.
The matrices, C dualIB xx (cid:48) (ˆ x ; β
[Equation]
, C dualIB yy
[Equation]
(ˆ x ; β ) , are given by:
[Equation]
with:
[Equation]
Given that the matrices are obtained by the multiplication of the same matrices, it follows that they have the same eigenvalues { λ i (ˆ x ; β ) } .
To prove that λ 1 (ˆ x ; β ) =
we show that det( C dualIB yy (cid:48) )
= 0 .
We present the exact calculation for a binary label Y ∈ { y 0 , y 1 } (the argument for general Y follows by encoding the label as a sequence of bits and discussing the ﬁrst bit only, as a binary case):
[Equation]
[Table]
[Equation]
Given that the determinant is 0 implies that λ 1 (ˆ x ) = 0 .
For a binary problem we can describe the non-zero eigenvalue using λ 2 (ˆ x ) =
Tr ( C dualIB yy (cid:48)
(ˆ x ; β )) .
That is:
[Equation]
A. I NFORMATION P LANE ANALYSIS
We rely on known results for the rate-distortion problem and the information plane:
Lemma 8.
I ( X ; ˆ X ) is a non-increasing convex function of the distortion
E p β ( x , ˆ x ) [ d ( x , ˆ x )] with a slope of − β .
We emphasis that this is a general result of rate-distortion thus holds for the dualIB as well.
Lemma 9.
For a ﬁxed encoder p β (ˆ x | x ) and the Bayes optimal decoder p β ( y | ˆ x ) :
[Equation]
Thus, the information curve, I Y vs. I X , is a non-decreasing concave function with a positive slope, β − 1 .
The concavity implies that β increases along the curve.
(Cover & Thomas, 2006; Gilad-bachrach et al., 2003).
A.4. P ROOF OF L EMMA
In the following section we provide a proof to lemma 4, for the IB and dualIB problems.
Proof.
We want to analyze the behavior of I X ( β ) , I Y ( β ) , that is the change in each term as a function of the corresponding β .
From lemma 9, the concavity of the information curve, we can deduce that both are non-decreasing functions of β .
As the two β derivatives are proportional it’s enough to discuss the ﬁrst one.
Next, we focus on their behavior between two critical points.
That is, where the cardinality of ˆ X is ﬁxed (clusters are ”static”).
For ”static” clusters, the β derivative of I X , along the optimal line is given by:
[Equation]
This ﬁrst of all reassures that the function is non-decreasing as Var ( d ( x )) ≥ 0 .
The piece-wise concavity follows from the fact that when the number of clusters is ﬁxed (between the critical points) - increasing β decreases the clusters conditional entropy H (ˆ X | X ) , as the encoder becomes more deterministic.
The mutual information is bounded by H (ˆ X ) and it’s β derivative decreases.
Further, between the critical points there are no sign changes in the second β derivative.
A.4. P ROOF OF L EMMA
Proof.
The information curve has a positive slope, β − 1 , with β increasing along it, lemma 9.
That is, given a value of β , there exists a pair I IB Y ( β ) , I IB X ( β ) such that ∂I IB Y ( β ) /∂I
IB X ( β ) =
β − 1 .
Now, consider a sub-optimal information curve, I ∗ Y , I ∗ X .
There exist values β (cid:48) , β (cid:48)(cid:48) such that:
[Equation]
The optimality of the IB implies that sub-optimal curves lie below it; i.e, the IB slope is steeper:
[Equation]
Thus, given that β increases along the information curve, it holds that:
[Equation]
) = I IB Y ( β ) .
A. D ERIVATION OF THE DUAL E XP IB
We provide elaborate derivations to theorem 7; that is, we obtain the dualIB optimal encoder-decoder under the exponential assumption over the data.
We use the notations deﬁned in § 3.
• The decoder , equation 14.
Substituting the exponential assumption into the dualIB log -decoder yields:
[Equation]
Taking a closer look at the normalization term:
[Equation]
From which it follows that λ 0 β (ˆ x ) is given by:
[Equation]
, and we can conclude that the dualExpIB decoder takes the form:
[Equation]
The encoder
, equation 15.
The core of the encoder is the dual distortion function which may now be written as:
[Equation]
substituting this into the encoder’s deﬁnition we obtain:
[Equation]