C AVEATS FOR INFORMATION BOTTLENECK IN DETERMINISTIC SCENARIOS
Anonymous authors
Paper under double-blind review
A BSTRACT
Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y .
To do so, IB identiﬁes an intermediate “bottleneck” variable T that has low mutual information
I ( X ; T ) and high mutual information I ( Y ; T ) .
The IB curve characterizes the set of bottleneck variables that achieve maximal I ( Y ; T ) for a given I ( X ; T ) , and is typically explored by optimizing the IB Lagrangian , I ( Y ; T ) − βI ( X ; T ) .
In some cases, Y is a deterministic function X , including many supervised classiﬁcation scenarios where the output class Y is a deterministic function of the input X .
We demonstrate several caveats when using IB in any situation where
Y is a deterministic function of X : (1) the IB curve cannot be recovered by optimizing the IB Lagrangian for different values of β ; (2) there are “uninteresting” trivial solutions at all points of the IB curve; and (3) for multi-layer classiﬁers that achieve low error rates, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal.
We also demonstrate that when Y is a small perturbation away from being a deterministic function of
X , these issues arise in an approximate way.
To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases.
We demonstrate these issues on the MNIST dataset.
I NTRODUCTION
The information bottleneck (IB) method (Tishby et al., 1999) provides a principled way to extract information that is present in one variable that is relevant for predicting another variable.
Given two random variables X and Y , IB posits a “bottleneck” variable T that obeys the Markov condition Y − X − T .
By the data processing inequality (DPI) (Cover & Thomas, 2012), this Markov condition implies that I ( X ; T ) ≥ I ( Y ; T ) , meaning the bottleneck variable cannot contain more information about Y than it does about X .
In fact, any particular choice of the bottleneck variable T can be quantiﬁed by two terms: the mutual information I ( X ; T ) , which reﬂects how much T compresses X , and the mutual information
I ( Y ; T ) , which reﬂects how well T can be used to predict
Y .
In IB, bottleneck variables are chosen to maximize prediction given a constraint on compression (Witsenhausen & Wyner, 1975; Ahlswede & Körner, 1975; Gilad-Bachrach et al., 2003),
[Equation]
where ∆ is the set of all random variables T that obey the Markov condition Y − X − T .
The values of F ( r ) for different values of r specify the IB curve .
It is known that F is concave in r , though it may not be strictly concave.
This seemingly minor issue of strict vs. non-strict concavity of the IB curve will play a central role in our analysis.
In practice, the IB curve is almost always explored not via the constrained optimization problem of
Eq.
(1), but rather by maximizing the so-called IB Lagrangian ,
[Equation]
where β ∈ [0 , 1] is a parameter that controls the trade-off between compression and prediction.
The advantage of optimizing L β IB is that it avoids the non-linear constraint in Eq.
Several recent papers have drawn connections between IB and supervised learning (Shamir et al., 2010; Tishby & Zaslavsky, 2015), in particular classiﬁcation using neural networks.
In this context, X represents input vectors, Y represents the output classes, and T represents intermediate representations used by the learning architecture, such as the activity of hidden layer(s).
The application of IB to supervised learning has opened a promising set of research directions.
Some of this research investigates neural network training algorithms that optimize the IB Lagrangian (Kolchinsky et al., 2017; Alemi et al., 2016; Chalk et al., 2016), thereby permitting the application of IB to high-dimensional, continuous random variables.
Other research (Shwartz-Ziv & Tishby, 2017) has suggested, somewhat controversially (Saxe et al., 2018), that stochastic gradient descent (SGD) training dynamics may implicitly favor hidden layer activity that optimally balances compression and prediction, and moreover that earlier hidden layers may favor prediction rather than compression, while latter hidden layers may favor compression rather than prediction.
More generally, there are ideas that intermediate representations that are optimal in the IB sense correspond to “interesting” or “useful” compressions of the input (Amjad & Geiger, 2018).
In some cases, Y maybe is a deterministic function of X , i.e., Y = f ( X ) for some single-valued function f .
For example, this is the case in many classiﬁcation problems, when it is assumed that any given input belongs to a single class.
In this paper, we demonstrate several caveats that appear whenever IB is applied to a situation where Y is a deterministic function of X :
There is no one-to-one mapping between different points on the IB curve and maximizers of the IB Lagrangian L β IB for different β , thus the IB curve cannot be explored by maximizing L β IB while varying β .
This occurs because when Y =
f ( X ) , the IB curve has a piecewise-linear shape and is therefore not strictly concave.
The dependence of the IB Lagrangian on the strict concavity of
F r ) has been previously noted (Gilad-Bachrach et al., 2003; Shwartz-Ziv & Tishby, 2017),
but the
pervasiveness
of
this problem in classiﬁcation scenarios has not been recognized.
We
analyze this
issue
and propose a solution in the
form
of an alternative objective
function
, which can be used to explore the
IB curve
even when Y is a deterministic function of X . 2.
When Y is a deterministic function of X , all points on the IB curve contain “uninteresting” trivial solutions.
In particular, the entire IB curve can be generated as a mixture of trivial solutions.
This suggests that IB-optimality is not sufﬁcient for an intermediate representation to be an interesting or useful compression of input data.
For a neural network with several hidden layers that achieves a low probability of error, the hidden layers cannot display a strict trade-off between compression and prediction (in particular, different layers can only differ in the amount of compression, not prediction).
In addition, in Appendix B, we show that the above caveats also apply to the recently proposed deterministic IB variant of IB (Strouse & Schwab, 2017), in which the compression term is quantiﬁed using the entropy H ( T ) rather than the mutual information I ( X ; T ) .
In that Appendix, we propose an alternative objective function that can be used to resolve the ﬁrst problem for dIB.
We also show, in Appendix C, that our results apply when Y is not exactly a deterministic function of X , but (cid:15) -close to one (in terms of (cid:96) 1 distance between X, Y joint distributions).
In this case: (1) it is hard to explore the IB curve by optimizing the IB Lagrangian, because all optimizers will fall within O ( − (cid:15) log (cid:15) ) of a single “corner” point on the information plane; (2) “uninteresting” trivial solutions are no more than O ( − (cid:15) log (cid:15) ) away from being optimal along all points on the IB curve; (3) different layers of a neural networks can trade-off at most O ( − (cid:15) log (cid:15) ) amount of prediction.
A recent paper (Amjad & Geiger, 2018) also discusses several difﬁculties in using IB to analyze intermediate representations in supervised learning.
That paper does not consider the particular issues that arise when Y is a deterministic function of X , and its arguments are complementary to ours.
Another caveat of IB in deterministic settings is discussed in (Shwartz-Ziv & Tishby, 2017)[Sec.
2.4], regarding the relationship between sufﬁcient statistics and the complexity of the input-output mapping, but that issue is orthogonal to the ones here, and we refer the reader to that paper.
Finally, some recent work (Saxe et al., 2018; Amjad & Geiger, 2018) has noted problems with IB that occur when T is continuous-valued and a deterministic function of X , since then I ( X ; T ) can be unbounded; it is important to note that the issues in this paper are unrelated to this problem.
In the next section, we review some of the connections between supervised learning and IB.
In
Section 3, we show that when Y = f ( X ) , the IB curve has a piecewise-linear (not strictly concave) shape.
In Sections 4, 5 and 6, we discuss the three issues mentioned above.
In Section 7, we demonstrate these issues using a neural-network implementation of IB on the MNIST dataset.
S UPERVISED C LASSIFICATION AND IB
In supervised learning, one is provided with a training dataset { x i , y i } i =1 ..N of inputs and outputs, as well as a parameterized family of conditional distributions, { q θ ( y | x ) } , where θ are parameters.
We use the random variables X and Y to refer to the inputs and outputs respectively, as well as the sets of outcomes of those random variables.
We use ˆ p ( x, y ) to indicate the empirical distribution of inputs and outputs in the training data.
It is usually assumed that the x ’s and y ’s are sampled i.i.d.
from some “true” distribution w ( y | x ) w ( x ) .
The high-level goal of supervised learning is to use the dataset to select parameters θ
such that the distribution
q θ ( y | x ) is a good approximation of w ( y | x ) .
Supervised learning is called classiﬁcation when Y takes a ﬁnite set of values, and regression when
Y is continuous-valued.
Here we focus on classiﬁcation, where the set of possible output values can be written as Y = { 0 , 1 , . . .
, m } .
We expect our results to also apply in some regression scenarios (with some care regarding evaluating MI measures, see Footnote 2), but leave this for future work.
In practice, many supervised learning architectures use some kind of intermediate representation to make predictions about the output, such as hidden layers in neural networks.
We use the random variable T to represent the activity of some particular hidden layer in a neural network.
We assume that T is a (possibly stochastic) function of the inputs, as determined by some parameterized conditional distribution q θ ( t | x ) , thus T obeys the Markov condition Y − X − T .
The mapping from inputs to hidden layer activity can be either deterministic (as traditionally done in neural networks) or stochastic (as used in some architectures (Alemi et al., 2016;
Kolchinsky et al., 2017)) 2 .
The mapping from hidden layer activity to predicted outputs is represented with the parameterized conditional distribution q θ ( y | t ) .
The overall mapping from inputs to predicted outputs implemented by a neural network with a hidden layer can be written as q θ
[Equation]
y
[Equation]
Actually, T does not have to represent a particular hidden layer, but could instead represent the activity of the neurons in a set of contiguous hidden layers, or in fact any arbitrary set of neurons that separate inputs from predicted outputs (in the sense of conditional independence, so that Eq.
( ∗ ) holds).
More generally yet, T could also represent some intermediate representation in a non-neuralnetwork supervised learning architecture, again as long as Eq.
( ∗ ) holds.
For classiﬁcation problems, training often involves selecting θ to minimize cross-entropy loss , sometimes while stochastically sampling the activity of hidden layers.
When T is the last hidden layer, or when the mapping from T to the last hidden layer is deterministic, cross-entropy loss can be written
[Equation]
where E indicates expectation, ˆ p θ ( y, t ) := 1 N (cid:80) i δ ( y, y i )
q θ ( t | x i ) is the empirical distribution of outputs and activity of hidden layer T given the dataset, and δ is the Kronecker delta.
The cross-entropy loss in Eq.
(3) can also be written as
[Equation]
where H is (conditional) Shannon entropy, D KL is Kullback-Leibler (KL) divergence, and
ˆ p θ ( y | t ) is deﬁned via the deﬁnition of ˆ p θ ( y, t ) above.
Since KL is non-negative, L CE ( θ ) is an upper bound on the conditional entropy H (ˆ p θ ( Y | T )) .
During training, one can decrease L CE either by decreasing the conditional entropy H (ˆ p θ ( Y | T )) , or by decreasing the KL term by changing q θ ( Y | T ) to better approximate ˆ p θ ( Y | T ) .
Minimizing L CE thus minimizes an upper bound on
H (ˆ p θ ( Y | T )) .
This is equivalent to maximizing a lower bound on I (ˆ p θ ( Y ; T )) , since I (ˆ p θ ( Y ; T )) =
H (ˆ p ( Y ))
− H (ˆ p θ ( Y | T )) , where H (ˆ p ( Y )) is a constant that doesn’t depend on θ .
We can now make explicit the relationship between supervised learning and IB.
In IB, one is provided with a joint distribution p ( x, y ) , and then seeks a bottleneck variable T that obeys the Markov condition Y − X − T and that minimizes I ( X ; T ) while maximizing I ( Y ; T ) .
In supervised learning, one is provided with an empirical distribution ˆ p ( x, y ) and deﬁnes an intermediate representation T that obeys the Markov condition Y − X − T ; during training, cross-entropy loss is minimized, which is often equivalent to maximizing a lower bound on I ( Y ; T ) .
To make the analogy complete, one might choose θ so as to also minimize I ( X ; T ) , i.e., seek hidden layers that provide compressed representations of input data (Alemi et al., 2016; Chalk et al., 2016; Kolchinsky et al., 2017).
In fact, we use such an approach in our experiments on the MNIST dataset, as reported in Section 7.
Before proceeding, we note that in many supervised classiﬁcation problems, there is only one correct label associated with each possible input.
Formally, this means that given the empirical distribution ˆ p ( y | x ) , one can write Y = f
( X ) for some single-valued function f .
This relationship between
X and Y may arise because it holds under the “true” distribution w ( y | x ) from which the training dataset is sampled, or simply because each input vector x occurs at most once in the training data (as happens in most real-world classiﬁcation training datasets).
Importantly, we make no assumptions about the relationship between X and Y under q θ ( y | x ) , the approximate parameterized distribution implemented by the supervised learning architecture.
In particular, our analysis still holds if q θ ( y | x ) is non-deterministic (e.g., the softmax function).
In addition, our results are based on analyticallyprovable properties of the IB curve (i.e., global optima of Eq.
(1)), and do not concern issues of local optima that may be important in practical scenarios.
Our theoretical results are also independent of the practical problem of estimating MI between different layers of a neural network (an important and active area of recent research, see (Belghazi et al., 2018; Goldfeld et al., 2018; Dai et al., 2018;
Gabrié et al., 2018)), while our empirical experiments rely on the estimator proposed in (Kolchinsky & Tracey, 2017; Kolchinsky et al., 2017).
Finally, our results do not concern issues related to generalization error and ﬁnite data sampling (Shamir et al., 2010; Tishby & Zaslavsky, 2015), which are relevant for analyzing the relationship between IB and out-of-sample performance.
It is important to note that not all classiﬁcation problems are deterministic.
The map from X to
Y may be intrinsically noisy or non-deterministic (e.g., if human labelers cannot agree on a class for some x ), or noise may be intentionally added as a regularization technique (e.g., as done in the label smoothing method (Szegedy et al., 2016)).
Moreover, one of the pioneering papers on the relationship between IB and supervised learning (Shwartz-Ziv & Tishby, 2017) analyzed an artiﬁcially-constructed classiﬁcation problem in which p ( y | x ) was explicitly deﬁned to be noisy (see their Eq. 10).
However, we believe that many if not most real-world classiﬁcation problems do in fact exhibit a deterministic relation between X and Y .
Finally, as we show in Appendix C, even if the relationship between X and Y is not perfectly deterministic but close to being so, then all of the issues discussed in this paper still apply in an approximate sense.
T HE IB CURVE WHEN Y IS A DETERMINISTIC FUNCTION OF X
[Figure]
Consider any X, Y where Y is discrete-valued and a deterministic function of X , so that Y =
f ( X ) .
There are two upper bounds on I ( Y ; T ) : I ( Y ; T ) ≤ I ( X ; T ) by the DPI, and I ( Y ; T ) ≤ H ( Y ) (Cover & Thomas, 2012).
To visualize these two bounds, as well as other results of our analysis, we use so-called “information plane” diagrams (Tishby et al., 1999).
The information plane represents various possible bottleneck variables in terms of their compression ( I ( X ; T ) , horizontal axis) and prediction ( I ( Y ; T ) , vertical axis) values.
The two bounds mentioned above are plotted on an information plane diagram in Fig.
In this section, we show that when Y = f ( X ) , the IB curve saturates both bounds mentioned above, and is therefore not strictly concave but rather piece-wise linear (thick gray line in Fig. 1).
To show that the IB curve saturates the DPI bound
I ( Y ; T ) ≤ I ( X ; T ) for I ( X ; T ) ∈ [0 , H ( Y )] , let B α be a Bernoulli random variable that is equal to 1 with probability α .
Then, deﬁne a manifold of bottleneck variables T α parameterized by α ∈ [0 , 1] ,
[Equation]
Thus, T α is equal to Y with probability α , and equal to 0 with probability 1 − α .
From Eq.
(5), it is clear that T α is a (stochastic) function of both X and Y , thus both Markov conditions Y − X − T α and
X − Y − T α hold.
Applying the DPI to both Markov conditions implies that I ( T α ; X ) =
I ( T α ; Y ) for any T α .
Note that for α = 1 , T α = Y and thus I ( T α ; Y ) =
H ( Y ) , while for α = 0 , T α = and thus I ( T α ; Y ) = 0 .
Since MI is continuous in probabilities, the manifold of bottleneck variables T α sweeps the full range of values (cid:104) I
( X ; T α ) , I ( Y ; T α ) (cid:105) =
(cid:104) 0 , 0 (cid:105) to (cid:104) I ( X ; T α ) , I
( Y ; T α ) (cid:105) =
(cid:104) H ( Y ) , H ( Y ) (cid:105) as α ranges from 0 to 1, while obeying I
( T α ; X ) =
I ( T α ; Y ) .
Thus, the manifold of T α bottleneck variables achieves the DPI bound over I ( T ; X ) ∈ [0 , H ( Y )] .
Given its deﬁnition in Eq.
(1), the IB curve must be monotonically increasing.
Since (cid:104) H ( Y ) , H ( Y ) (cid:105) is on the IB curve (e.g., T α for α = 1 ), it must be that I ( Y ; T ) ≥ H ( Y ) whenever I ( X ; T ) ≥ H ( Y ) .
At the same time, I ( Y ; T ) ≤ H ( Y ) for all T .
Thus, for all optimal bottleneck variables T with
I ( X ; T ) ≥ H ( Y ) , the IB curve is a ﬂat line with I ( Y ; T ) =
H ( Y ) .
We have shown that when Y is a deterministic function of X , the IB curve is piecewise-linear, having the shape of the thick gray line in Fig. 1, and thus not strictly concave.
Note that the IB curve may also be not strictly concave in other situations.
A sufﬁcient condition for the IB curve to be strictly concave is for p ( y | x ) > 0 everywhere (Gilad-Bachrach et al., 2003, Lemma 6).
I SSUE 1: IB C URVE CANNOT BE EXPLORED USING THE IB L AGRANGIAN
In this section, we show that when the IB curve is piecewise-linear, there is no one-to-one mapping between points on the IB curve and optimizers of the IB Lagrangian for different values of β .
This means that one cannot explore the IB curve by solving T (cid:63) ∈ arg max T ∈ ∆
I ( Y ; T ) − βI ( X ; T ) while varying β .
This issue is diagrammed in Fig. 2, and explained formally below.
Assume Y = f
( X ) .
For any bottleneck variable T and all β ∈ [0 , 1] , L β IB is bounded by
[Equation]
where the ﬁrst inequality uses the DPI, and the second I ( Y ; T ) ≤ H ( Y ) .
Now consider the bottleneck variable T copy :
= Y (or, equivalently, any one-to-one transformation of T copy ), which corresponds to T α for α = 1 in Eq.
(5), and is the “corner point” of the IB curve in Fig.
1, achieving (cid:104) I ( X ; T copy ) , I ( Y ; T copy ) (cid:105) =
(cid:104) H ( Y ) , H ( Y ) (cid:105) .
Given the deﬁnition of L β IB ,
L β IB ( T copy ) =
(1 − β ) H ( Y ) , so T copy achieves the bound of Eq.
(6) for all β ∈ [0 , 1] .
Therefore,
T copy optimizes L β IB for all β ∈ [0 , 1] , though it corresponds to only a single point on the IB curve.
Now consider the IB Lagrangian for β = 1 .
For this β , L β IB ( T ) =
I ( Y ; T ) − I ( X ; T ) ≤ 0 , where the inequality follows from the DPI.
Any T that has I ( X ; T )
=
I ( Y ; T ) , such as the manifold of
T α deﬁned in Eq.
(5), achieves the maximum L β IB ( T ) = 0 .
Therefore, all solutions on the non-ﬂat part of IB curve in Fig.
1 simultaneously achieve the same maximum value of L β IB for β = 1 .
Finally, when β = 0 , there is no penalty on I ( X ; T ) , and L β IB ( T ) =
I ( Y ; T ) , which is simultaneously maximized by all solutions that achieve the bound I ( Y ; T ) =
H ( Y ) .
Therefore, all solutions on the ﬂat part of the IB curve in Fig.
1 simultaneously achieve the same optimum value of L β IB for β = 0 .
(Note that supervised learning algorithms with cross-entropy loss can be considered to have β = 0 , and their corresponding intermediate representations will generally fall into this regime.)
The problem is illustrated visually in Fig.
2, which shows two hypothetical IB curves (in thick gray lines): a not strictly concave (piecewise-linear) curve (left), as occurs when Y is a deterministic function of X , and a strictly concave curve (right).
In both subplots, the colored lines indicate isolines of the IB Lagrangian.
The highest point on the IB curve crossed by a given colored line is marked by a star, and represents the (cid:104) I ( X ; T ) , I ( Y ; T ) (cid:105) values for a T which optimizes L β IB for the corresponding β .
For the piecewise-linear IB curve, all β ∈ (0 , 1) only recover a single solution ( T copy ).
For the strictly concave IB curve, different β values recover different solutions.
To summarize, when Y is a deterministic function of X , the IB curve is piecewise-linear and cannot be explored by optimizing L β IB while varying β .
On the ﬂip side, when Y = f ( X ) and one’s goal is only to ﬁnd the corner point (cid:104) H ( Y ) , H ( Y ) (cid:105) (maximum compression at no prediction loss), then our results suggest that the IB Lagrangian provides a robust way to do so, being invariant to the particular choice of β .
If one does wish to recover the full IB curve, in principle it can be explored by solving the constrained optimization of Eq.
(1) for different values of r (Miettinen, 1999, Thm.
In practice, however, solving this constrained optimization problem — even approximately
[Figure] [Figure]
— is far from a simple task, in part due to the non-linear constraint on I ( X ; T ) , and many off-theshelf optimization tools cannot handle this kind of problem.
It is desirable to have an unconstrained objective function that can be used to explore any IB curve, whether strictly or non strictly concave.
For this purpose we propose an alternative objective function, that we call the squared-IB functional ,
[Equation]
We show that maximizing L β sq-IB while varying β recovers all points on the increasing part of the
IB curve.
To do so, we ﬁrst provide a bit of analysis of why the IB Lagrangian fails.
Over the increasing region of the IB curve, the inequality constraint in Eq.
(1) can be replaced by an equality constraint (Witsenhausen & Wyner, 1975, Thm 2.5), and maximizing L β IB can be written as max T
I
( Y ; T ) − βI ( X ; T ) =
max r F ( r ) − βr (i.e., the Legendre transform of − F ( r ) ).
The derivative of F ( r ) − βr is zero when F (cid:48)
( r ) =
β , and any point on the IB curve that has F (cid:48) ( r ) =
β will maximize L β IB for that β .
For a piecewise linear IB curve, all points on the increasing part of the curve have F (cid:48)
( r ) = 1 , and will all simultaneously maximize L β IB when β
= 1 .
Using a similar analysis, we write max T L β sq-IB = max r F ( r ) − βr 2 .
The derivative of F ( r ) − βr 2 iswhen F (cid:48) ( r ) / (2 r ) = β .
Since F is (not strictly) concave, F (cid:48) ( r ) is (not strictly) decreasing in r , and
F (cid:48) ( r ) / (2 r ) is strictly decreasing in r .
Thus, there can be at most one r such that F (cid:48)
( r ) / (2 r ) =
β for a given β , meaning that there can only be one point on the IB curve that optimizes L β sq-IB for a given β .
Therefore, the IB curve can be explored by maximizing L β sq-IB while varying β .
Note that any point that satisﬁes F (cid:48) ( r ) =
β also satisﬁes F (cid:48) ( r ) / (2 r ) =
β/ (2 r ) .
Thus, any point that maximizes L β IB for a given β also maximizes L β sq-IB under the transformation β
(cid:55)→ β/ (2 · I ( X ; T )) , and vice versa.
Importantly, unlike for L β IB , there can be non-trivial maximizers of L β sq-IB for β > 1 .
The effect of optimizing L β sq-IB is illustrated in Fig.
This ﬁgure shows two different IB curves (not strictly concave and strictly concave), isolines of L β sq-IB for different β , and points on the curves that maximize L β sq-IB .
For both IB curves, the maximizers correspond to different points for different β .
Finally, note that the IB curve is the Pareto front of the multi-objective optimization problem, { maximize I ( Y ; T ) , minimize I ( X ; T ) } .
L β sq-IB is one modiﬁcation of L β IB that allows us to explore a non-strictly-concave Pareto front.
However, it is not the only possible modiﬁcation, and other alternatives may be considered (this may be particularly relevant when T lives in a restricted model class).
For a full treatment of multi-objective optimization techniques, see Miettinen (1999).
I SSUE 2: A LL POINTS ON IB CURVE HAVE “ UNINTERESTING ” SOLUTIONS
It is often implicitly or explicitly assumed that optimal bottleneck variables on the IB curve will provide “useful” or “interesting” representations of the relevant information in X about Y (see also discussion and proposed criteria in (Amjad & Geiger, 2018)).
While the concepts like usefulness and interestingness are subjective, the general intuition can be illustrated using the following simple example.
Consider the ImageNet task (Deng et al., 2009), which involves labeling images according to 1000 classes, such as border collie , golden retriever , coffeepot , teapot , etc.
It is natural to expect that as one explores the IB curve for the ImageNet dataset, one will identify useful compressed representations of the space of images.
Such useful representations might, for instance, specify hard-clusterings that merge together inputs belonging to perceptually similar classes, such as border collie and golden retriever (but not border collie and teapot ).
In this section, we show that such intuitions do not necessarily hold whenever Y is a deterministic function of X .
Recall the analysis from Section 3, where Eq.
(5) deﬁnes the manifold of bottleneck variables T α parameterized by α ∈ [0 , 1] .
The manifold of T α , which spans the entire increasing portion of the
IB curve, represents a mixture of two “trivial” solutions: a constant mapping to 0, and an exact copy of Y .
(Note that the ﬂat part of the IB curve can also be explored by a mixture of two trivial solutions, though we do not focus on this part of the curve here.)
While the manifold of T α bottleneck variables is optimal in the IB sense, these bottleneck variables do not offer interesting or useful representations of the input data.
The compression offered by T α arises by “forgetting” the input with some probability 1 − α , rather than performing any kind of useful coarse-graining, while the prediction comes from full knowledge of the mapping from inputs to outputs, f : X → Y .
To summarize, when Y is a deterministic function of X , the fact that a bottleneck variable T is on the IB curve does not necessarily imply that the same bottleneck variable is useful compressed representation of X .
At the same time, we do not claim that all bottleneck variables on the IB curve will have the trivial structure of Eq.
In fact, there may also be IB-optimal bottleneck variables that compress X in interesting and useful ways (however such notions may be formalized).
Nonetheless, when Y is a deterministic function of X , for any “interesting” T , there will also be a “trivial” T α that achieves the same values of compression I ( X ; T ) and prediction I ( Y ; T ) , and IB does not distinguish between the two.
Therefore, identifying useful compressed representations will generally require the use of quality functions other than just IB (Amjad & Geiger, 2018).
In Appendix B, we analyze this issue for deterministic IB (dIB).
We show that intermediate representations T that are hard-clusterings of Y will be optimal in terms of dIB, but these clusterings will not generally obey any intuitions about semantic or perceptual similarity between grouped-together elements of Y , and will not necessarily provide any useful representations of the inputs.
I SSUE 3: N O TRADE - OFF AMONG DIFFERENT NEURAL NETWORK LAYERS
So far we’ve analyzed the behavior of IB in terms of a single intermediate representation T (e.g., a single hidden layer in a neural network).
Recent research in machine learning, however, has focused on “deep” neural networks, with multiple successive hidden layers.
What is the relationship between the compression and prediction achieved by these different layers?
Recently, Shwartz-Ziv & Tishby (2017) suggested that due to SGD training dynamics, different layers will explore a strict trade-off between compression and prediction: early layers will sacriﬁce compression (high I ( T ; X ) ) for good prediction (high I ( T ; Y ) ), while latter layers will sacriﬁce prediction (low I ( T ; Y ) ) for good compression (low I ( T ; X ) ).
Shwartz-Ziv & Tishby (2017) demonstrated a strict trade-off using an artiﬁcial classiﬁcation dataset, in which Y was deﬁned to be a noisy function of X (their Fig. 6).
As we show here, however, such results cannot generally hold when Y is a deterministic function of X .
Recall that we consider IB in terms of the empirical distribution of inputs, hidden layers, and outputs given the training data (Section 2).
Imagine that a classiﬁer achieves 0 probability of error (i.e., classiﬁes every input correctly) on training data.
This event, which can only occur when Y is a deterministic function of X , is in fact commonly observed in real-world deep neural networks (Zhang et al., 2017).
In such cases, we show that while latter layers may have better compression (lower
I ( T ; X ) ) than earlier layers, they can’t have worse prediction (lower I ( T ; Y ) ) than earlier layers.
Therefore, different layers can only demonstrate a weak trade-off between compression and prediction.
(The same arguments also hold if the neural network achieves near-0 probability of error on held-out testing data, as long as the information-theoretic measures are evaluated on the same held-out data distribution.)
Consider a neural network with k hidden layers, where each successive layer is a (possibly stochastic) function of the preceding one, and let T 1 , T 2 , . .
, T k indicate the activity of layer 1 , 2 , . . .
, k .
Typically, the last layer T k is used to make a prediction about the output Y ; we write this predicted
Y as the random variable ˆ Y :
= a θ ( T k ) , where a θ is taken as a deterministic function for simplicity (e.g., T k may be mapped through a softmax to give a vector of output class probabilities q θ ( Y | T k ) , from which the highest probability class is chosen, ˆ Y := arg max y q θ
( y | T k ) ).
The probability of error (the probability of predicting the incorrect output class) is then simply P e =
E [ Y (cid:54)
=
ˆ Y ] , where the expectation is over the empirical distribution of the training dataset as well as any stochasticity generated by the network itself (for instance if the layer-to-layer mappings are stochastic).
Given that the true output Y is a function of X , the above architecture obeys the Markov condition
[Equation]
By the DPI, for any i < j we have the inequalities
[Equation]
Applying Fano’s inequality (Cover & Thomas, 2012, p. 39) to the chain Y − T k − ˆ Y gives
[Equation]
(11) where | Y | is the number of output classes, and H is the binary entropy function, H ( x ) :
= − x log x − (1 − x ) log(1 − x ) .
Given our assumption that the probability of error obeys P e = 0 , Eq.
(11) implies that H ( Y | T k )
=
and I ( T k ; Y ) =
H ( Y ) − H
( Y | T k )
=
H ( Y ) .
Since H ( Y ) is the largest MI that any random variable can have about Y , by Eq.
(9) it must be that I ( T i ; Y )
=
H ( Y ) for all i = 1 ...k .
Our arguments don’t undermine or conﬁrm the high-level idea of Shwartz-Ziv & Tishby (2017), that
SGD dynamics may favor compression of hidden layer activity.
Instead, our point is that if a classiﬁer achieves 0 probability of error — which is possible once Y = f
( X ) — then different layers will have the same amount of prediction about the output, and a strict compression/prediction trade-off between different layers is impossible.
At the same time, by Eq.
(10), latter layers may still compress the input more than earlier layers, and a weak trade-off between prediction and compression is possible.
In terms of Fig. 1, this means that different layers will be on the ﬂat part of the IB curve.
Note that some neural network architectures do not have a simple layer structure, in which each layer’s activity is a stochastic function of the previous layer’s, but rather allow information to ﬂow in parallel across different channel (Szegedy et al., 2015) or to skip across some layers (He et al., 2016).
The analysis in this section can still apply to such cases, as long as the different T 1 , . .
, T k are chosen to correspond to groups of neurons such that the Markov condition in Eq.
(8) is obeyed.
A REAL - WORLD EXAMPLE : MNIST
We demonstrate the three issues discussed above using the MNIST dataset of hand-written digits.
We identify IB solutions for this dataset using the “nonlinear IB” method (Kolchinsky et al., 2017), which uses gradient-descent-based training to minimize cross-entropy loss plus a differentiable nonparametric estimate of I ( X ; T ) (Kolchinsky & Tracey, 2017).
We use this technique to optimize an upper bound on the IB Lagrangian, as explained in (Kolchinsky et al., 2017), as well as (via a small modiﬁcation) an upper bound on the squared-IB functional.
In our architecture, the bottleneck variable, T ∈ R 2 , corresponds to the activity of a hidden layer with two hidden units.
Using a two-dimensional bottleneck variable facilitates easy visual analysis of its activity.
The map from inputs X to bottleneck T has the form T = a θ ( X ) + Z , where
Z ∼ N (0 , I ) is the noise, and a θ is a deterministic function implemented using three fully-connected layers: two layers with 800 ReLU units each, and a third layer with two linear units.
Note that the stochasticity in the mapping from X to T makes our mutual information term, I ( X ; T ) , well-deﬁned and ﬁnite.
The decoding map, q θ ( y | t ) , uses a fully-connected layer with 800 ReLU units, followed by an output layer of 10 softmax units.
Results are reported for training data.
See Appendix A for more details and ﬁgures, including results on testing data.
anonymized provides TensorFlow code.
8 I ( X ; T )
I ( Y ;
T )
A = =0.
=0. =0.
=
IB Lagrangian
Theoretical IB curve Empirical results     8 I ( X ; T )
I ( Y ;
T )
B = =0.
=0. =0.
= squared-IB functional
Like other practical general-purpose IB methods, “nonlinear IB” is not guaranteed to ﬁnd globallyoptimal IB bottleneck variables, due to factors like: (a) difﬁculty of optimizing the non-convex IB objective; (b) error in estimating I ( X ; T ) ; (c) limited model class of T (i.e., T must be expressible in the form of T = a θ ( X ) + Z ); (d) mismatch between actual decoder q θ ( y | x ) and optimal decoder ˆ p θ ( y | x ) (see Section 2); and (e) stochasticity of training due to SGD.
Nonetheless, in practice, the solutions discovered by nonlinear IB were very close to IB-optimal, and are sufﬁcient to demonstrate all of the issues discussed in the previous sections.
Issue 1: IB Curve cannot be explored using the IB Lagrangian
We ﬁrst demonstrate that the IB curve cannot be explored by maximizing the IB Lagrangian, but can be explored by maximizing the squared-IB functional, supporting the arguments in Section 4.
Fig.shows the theoretical IB curve for the MNIST dataset, as well as the IB curve empirically recovered by maximizing these two functionals (see also Fig.
A5 for more details).
By optimizing the IB Lagrangian (Fig. 4A), we are only able to ﬁnd three IB-optimal points:
(1) Maximum prediction accuracy and no compression, I ( Y ; T )
=
H ( Y ) , I ( X ; T ) ≈ 8 nats; (2) Maximum compression possible at maximum prediction, I ( Y ; T )
=
I ( X ; T ) =
H ( Y ) ; (3) Maximum compression and zero prediction, I ( X ; T ) =
I ( Y ; T ) = 0 .
Note that the identiﬁed solutions are all very close to the theoretically-predicted IB-curve.
However, the switch between the 2nd regime (maximal compression possible at maximum prediction) and 3rd regime (maximum compression and zero prediction) in practice happens at β ≈ 0 .
This is different from the theoretical prediction, which states that this switch should occur at β = 1 . 0 .
The deviation from the theoretical prediction arises due to various practical details of the optimization done by the nonlinear IB method, as mentioned above.
The switch from the 1st regime (no compression) to the 2nd regime happened as soon as β > 0 , as predicted theoretically.
In contrast to the IB Lagrangian, by optimizing the squared-IB functional (Fig. 4B), we discover solutions located along different points on the IB curve for different values of β .
Additional insight is provided by visualizing the bottleneck variable T ∈ R 2 (i.e., hidden layer activity) for both types of experiments.
This is shown for different β in the scatter plot insets in
Fig.
As expected, the IB Lagrangian experiments displayed three types of bottleneck variables: non-compressed variables (regime 1), compressed variables where each of the 10 classes is represented by its own compact cluster (regime 2), and a trivial solution where all activity is collapsed to a single cluster (regime 3).
For the squared-IB functional, a different behavior was observed: As β increases, multiple classes become clustered together and the total number of clusters decreased.
Thus, nonlinear IB with the squared-IB functional learned to group X into a varying number of clusters, in this way exploring the full trade-off between compression and prediction.
Issue 2: All points on IB curve have “uninteresting” solutions
By maximizing the squared-IB functional, we could ﬁnd (nearly) IB-optimal solutions along different points of the IB curve.
Here we show that such solutions do not provide particularly useful representations of the input data, supporting the arguments made in Section 5.
Note that “stochastic mixture”-type solutions, in particular the family T α (Eq.
(5)), are not in our model class, since they cannot be expressed in the form T = a θ ( X ) + Z .
Instead, our implementation favors “hard-clusterings” in which all inputs belonging to a given output class are mapped to a compact, well-separated cluster in the space of hidden layer activity (note that inputs belonging to multiple output classes may be mapped to a single cluster).
For instance, Fig.
4B, shows solutions with 10 clusters ( β = 0 . 05 ), 6 clusters ( β = 0 . 2 ), 3 clusters ( β = 0 . 5 ), and 1 cluster ( β = 2 . 0 ).
Interestingly, such hard-clusterings are characteristic of optimal solutions for deterministic IB (dIB), as discussed in Appendix B.
At the same time, in our results, the classes are not clustered in any particularly meaningful or useful way, and clusters contain different combinations of classes for different solutions.
For instance, the solution shown for squared-IB functional with β = 0 .
5 hasclusters, one of which contains the classes { 0 , 2 , 3 , 4 , 5 , 6 , 8 , 9 } , another contains the class 1 , and the last contains the class 7 .
However, in other runs for the same β value, different clusterings of the classes arose.
Moreover, because the different classes appear with close-to-uniform frequency in the
MNIST dataset, any solution that groups the 10 classes into 3 clusters of size { 8 , 1 , 1 } will achieve similar values of I ( X ; T ) , I ( Y ; T ) as the solution shown for β = 0 .
Issue 3: No trade-off among different neural network layers
For both types of experiments, runs with β = 0 minimize cross-entropy loss only, without any regularization term that favors compression.
(Such runs are examples of “vanilla” supervised learning, though with stochasticity in the mapping between the input and the 2-node hidden layer.)
These runs achieved nearly-perfect prediction and I ( T ; Y )
≈
H ( Y ) .
However, as shown in the scatter plots in Fig. 4, these hidden layer activations fell into spread-out clusters, rather than point-like clusters seen for β > 0 .
In general, hidden layer activity was not compressed, in that it retained information about X that was irrelevant for predicting the class Y , and fell onto the ﬂat part of the IB curve.
Recall that our neural network architecture has three earlier hidden layers that precede T , as well as one hidden layer that succeeds it.
Due to the DPI inequalities Eqs.
(9) and (10), the earlier hidden layers must have worse compression than T , while the latter hidden layer must have better compression than T .
At the same time, β = 0 runs achieve nearly 0 probability of error on the training dataset (results not shown), meaning that all layers must achieve I ( Y ; T )
≈ H ( Y ) , the maximum possible.
Thus, for β = 0 runs, the activity of the all hidden layers is located on the ﬂat part of the IB curve, demonstrating a lack of a strict trade-off between prediction and compression.
C ONCLUSION
The information bottleneck principle has attracted a great deal of attention in information theory and machine learning, including in the context of classiﬁcation using neural networks.
In this work, we showed that in any situation where Y is a deterministic function of X — which includes many classiﬁcation scenarios — IB demonstrates behavior that is qualitatively different from the mapping from X to Y is stochastic.
In particular, in such cases: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian
I
( Y ; T ) − βI ( X ; T ) while varying β ; (2) all points on the IB curve contain “uninteresting” representations of inputs (in fact, when Y is a deterministic function of X , one can recover the entire IB curve in a closed-form way, without performing any optimization); (3) multi-layer classiﬁers that achieve vanishing probability of error cannot have a strict trade-off between prediction and compression among successive layers, contrary to recent proposals.
Our results should not be taken to mean that the application of IB to supervised learning is without merit.
First, they do not apply to various non-deterministic classiﬁcation problems where the output is stochastic.
Second, even for deterministic supervised learning, one may still wish to control the amount of compression during training, possibly to improve factors like compressibility (Dai et al., 2018), generalization performance (Shamir et al., 2010; Vera et al., 2018), or robustness to adversarial inputs (Alemi et al., 2016).
In this case, however, our work shows that to achieve varying rates of compression, one should use a different objective function than the IB Lagrangian.
A D ETAILS OF MNIST EXPERIMENTS
In Section 7, we demonstrate our results on the MNIST dataset (LeCun et al., 1998).
This dataset contains a training set of 60,000 images and a test set of 10,000 images, each labeled according to digit.
X ∈ R 784 is deﬁned to be a vector of pixels for a single 28 × 28 image, and
Y ∈ { 0 , 1 , ..., 9 } is deﬁned to be the class label.
Our experiments were carried out using the “nonlinear IB” method (Kolchinsky et al., 2017).
I ( X ; T ) was computed using the kernel-based mutual information upper bound (Kolchinsky & Tracey, 2017; Kolchinsky et al., 2017) and I ( Y ; T ) was computed using the lower bound I ( Y ; T )
≥ H ( Y ) − E ˆ p
θ
( Y,T )
[ − log q θ ( Y | T )]
(see Eq.
The neural network was trained using the Adam algorithm (Kingma & Ba, 2014) with a mini-batch size of 128 and a learning rate of 10 − 4 .
Unlike the implementation in (Kolchinsky et al., 2017), the same mini-batch was used to estimate the gradients of both I θ ( X ; T ) and the cross-entropy term.
Training was run for 200 epochs.
At the beginning of each epoch, the order of training examples was randomized.
To eliminate the effect of the local minima, for each possible value of β , we carried out 20 runs and then selected the run that achieved the best value of the objective function.
TensorFlow code is available at anonymized .
Results for the MNIST dataset are shown in Fig.
4 and Fig.
A5, computed for a range of β values.
Fig.
A5 shows results for both training and testing datasets, though the main text focuses exclusively on training data.
It can be seen that while
the solutions found by IB Lagrangian jump discontinuously from the “fully clustered” solution ( I ( T ; X ) =
I ( T ; Y ) =
H ( Y ) ) to the trivial solution ( I ( T ; X ) =
I ( T ; Y ) = 0 ), solutions found by the squared-IB functional explore the trade-off in a continuous manner.
See ﬁgure captions for details.
B D ETERMINISTIC I NFORMATION B OTTLENECK
Here we show that our analysis also applies to a recently-proposed (Strouse & Schwab, 2017) variant of IB called deterministic IB (dIB) .
dIB replaces the standard IB compression cost, I ( X ; T ) , with the entropy of the bottleneck variable, H ( T ) .
This can be interpreted as operationalizing compression costs via a source-coding, rather than a channel-coding, scenario.
Formally, in dIB one is given random variables X and Y .
One then identiﬁes bottleneck variables
T that obey the Markov condition Y − X − T and maximize the dIB Lagrangian ,
[Equation]
for β ∈ [0 , 1] , which can be considered as a relaxation of the constrained optimization problem
[Equation]
To guarantee that the compression cost is well-deﬁned, T is typically assumed to be discretevalued (Strouse & Schwab, 2017).
We call F dIB the dIB curve .
Before proceeding, we note that the inequality constraint in the deﬁnition of F dIB ( r ) can be replaced by an equality constraint,
[Equation]
We do so by showing that F dIB ( r ) is monotonically increasing in r .
Consider any T which maximizes I ( Y ; T ) subject to the constraint H ( T )
=
r .
By deﬁnition, the Markov condition Y − X − T must hold.
Now imagine some random variable D which obeys the Markov condition Y − X − T − D , and deﬁne a new bottleneck variable T (cid:48) :
= ( T, D ) (i.e., the joint outcome of T and D ).
We have
I ( Y ;
[Equation]
=
[Equation]
Y ; T, D ) =
I ( Y
[Equation]
T )
+ I
( Y
[Equation]
D |
[Equation]
=
I ( Y
[Equation]
T ) , where we’ve used the chain rule for mutual information.
At the same time, D can always be chosen so that H ( T (cid:48) )
=
H ( T, D ) =
r (cid:48) for any r (cid:48) ≥ r .
Thus, we have shown that there are always random variables T (cid:48) that achieve at least I ( Y ; T (cid:48) )
=
max T : H ( T )=
r I ( I ; T ) and have H ( T (cid:48) ) >
r , meaning that F dIB ( r ) is monotonically increasing in r .
This means the inequality constraint in Eq.
(A13) can be replaced with an equality constraint.
As we will see, unlike the standard IB curve, F dIB is not necessarily concave.
Since F dIB can be deﬁned using equality constraints, one can rewrite maximization of L β dIB as max T I
( Y ; T )
− βH
( T ) =
max r F dIB ( r ) − βr , the Legendre-Fenchel transform of − F dIB ( r ) .
By properties of the LegendreFenchel transform, this means that optimizers of L β dIB must lie on the concave envelope of F dIB , which we indicate as F ∗ dIB .
B. T HE D IB CURVE WHEN Y IS A DETERMINISTIC FUNCTION OF X
As in standard IB, for a discrete-valued Y we have the inequality
I ( Y ; T ) ≤ H ( Y ) .
(A15)
However, instead of the standard IB inequality I ( Y ; T ) ≤ I ( X ; T ) , we now employ
I ( Y ; T ) ≤ H ( T ) , (A16) which makes use of the assumption that T is discrete-valued.
The dIB curve will have the same bounds as those shown for the standard IB curve (Fig. 1), except that H ( T ) replaces I ( X ; T ) on the horizontal axis.
Now consider the case where Y is a deterministic function of X , i.e., Y =
f
( X ) .
It is easy to check that T copy := f ( X ) =
Y achieves equality for both Eqs.
(A15) and (A16), and thus lies on the dIB curve.
Since F dIB is monotonically increasing, the dIB curve is ﬂat and achieves I ( Y ; T )
=
H ( Y ) for all H ( T ) ≥ H ( Y ) .
We now consider the increasing part of the curve, H ( T ) ∈ [0 , H ( Y )] .
We call any T which is a deterministic function of Y (that is, any T =
g
( Y ) =
g ( f ( X )) , where g is any deterministic function) a “hard-clustering” of Y .
Any hard-clustering of Y has H ( T ) ≤ H ( Y ) .
At the same time, any hard-clustering of Y has H ( T | Y ) = 0 , thus I ( Y ; T ) =
H ( T ) − H
( T | Y ) =
H ( T ) , achieving the bound of Eq.
(A16).
Thus, any hard-clustering of Y lies on the increasing part of the dIB curve.
(Note that any hard-clustering of Y will also be a deterministic function of X , thus have
H ( T | X )
=
and I ( X ; T ) =
H ( T ) − H
( T | X ) =
H ( T ) =
I ( Y ; T ) , and will therefore also fall on the increasing part of the standard IB curve.)
[Figure]
Further saturates
At the same time, the dIB curve cannot be composed entirely of hard-clustering, since — under the assumption that Y is discrete-valued — there can only be a countable number of T ’s that are hard-clusterings of Y .
Thus, the dIB curve must also contain bottleneck variables that are not deterministic functions of Y , thus have H ( T | Y ) >and I ( Y ; T ) <
H ( T ) , and do not achieve the bound of
Eq.
(A16).
Geometrically-speaking, when Y is a deterministic function of X , the dIB curve must have a “step-like” structure over H ( T ) ∈ [0 , H ( Y )] , rather than increasing smoothly like the standard IB curve.
These results are shown schematically in Fig.
A6, where blue dots indicate hard clusters of Y .
As mentioned, optimizers of L β dIB must lie on the concave envelope of F dIB , indicated by F ∗ dIB .
Clearly, the step-like dIB curve that occurs when Y is a deterministic function of X is not concave, and only hard-clusterings of Y lie on its concave envelope for H ( T ) ∈ [0 , H ( Y )] .
In the analysis below, we will generally concern ourselves with optimizers which are hard-clusterings.
B. T HE THREE CAVEATS
We now brieﬂy consider the three issues discussed in the main text, in the context of deterministic IB.
As usual, we assume that Y is a deterministic function of X .
Issue 1: dIB Curve cannot be explored using the dIB Lagrangian
In analogy to the analysis done in Section 4, we use the inequalities Eqs.
(A15) and (A16) to bound the dIB Lagrangian as
[Equation]
Now consider the bottleneck variable T copy , for which L β dIB ( T copy ) =
(1 − β ) H ( Y ) .
Therefore, T copy (or any one-to-one transformation of T copy ) will maximize L β dIB for all β ∈ [0 , 1] .
It is also straightforward to show that when β = 0 , all bottleneck variables residing on the ﬂat part of the dIB curve will simultaneously optimize the dIB Lagrangian.
Similarly, one can show that all hard-clusterings of Y , which achieve the bound of Eq.
(A16), will simultaneously optimize the dIB Lagrangian for β = 1 .
As before, this means that different points on the dIB curve do not generally correspond to optima of L β dIB for different values of β .
Thus, there is no one-to-one map between points on the dIB curve and optimizers of the dIB Lagrangian for different β .
As in Section 4, we propose to resolve this problem by maximizing an alternative objective function, which we call the squared-dIB functional ,
[Equation]
We ﬁrst demonstrate that any optimizer of the dIB Lagrangian must also be an optimizer of the squared-dIB functional.
Consider that maximization of L β dIB can be written as max T
I
( Y ; T )
− βH
( T ) =
max r F ∗ dIB ( r ) − r .
Then, for the point (cid:104) r, F ∗ dIB ( r ) (cid:105) on the dIB curve to maximize L β dIB , it must have β ∈ ∂ r F ∗ dIB ( r ) , where ∂ r indicates the superderivative with regard to r .
At the same time, for the point (cid:104) r, F ∗ dIB ( r ) (cid:105) to maximize L β sq-dIB , it must satisfy 0 ∈ ∂ r (cid:2) F ∗ dIB ( r ) − βr 2 (cid:3) , or after
H ( T )
H ( Y )
I ( Y ;
T )
I ( Y ; T )
H ( T ) H ( T )
I ( Y ; T )
H ( T )0.
[Equation]
It is easy to see that if β ∈ ∂ r F ∗ dIB ( r ) is satisﬁed, then Eq.
(A18) is also satisﬁed under the transformation
β
(cid:55)→ β 2 r .
Thus, any optimizer of L β dIB will also optimize L β sq-dIB , given β (cid:55)→ β 2 r .
We now show that different hard-clusterings of Y will optimize the squared-dIB functional for different values of β , meaning that we can explore the envelope of the dIB curve by optimizing L β sq-dIB while varying β .
Formally, we show that for any given β > 0 , the point
[Equation]
on the dIB curve will be a unique maximizer of L β sq-dIB for the corresponding β .
Consider the value of L β sq-dIB for any T satisfying Eq.
(A19),
[Equation]
Now consider the value of of L β sq-dIB for any other T (cid:48) on the dIB curve, which has H ( T (cid:48) )
(cid:54) = 12 β ,
[Equation]
Inequality ( a ) comes from the assumption that H ( T (cid:48) )
(cid:54) = 12 β , thus ( H ( T (cid:48) ) − 1 / (2 β )) 2 > 0 , while inequality ( b ) comes from the bound I ( Y ; T (cid:48) )
≤ H ( T (cid:48) ) .
We have shown that L β sq-dIB ( T (cid:48) ) <
L β sq-dIB ( T ) , meaning that any point on the dIB curve satisfying Eq.
(A19) for a given β , assuming it exists, will be the unique maximizer of L β sq-dIB .
The situation is diagrammed visually in Fig.
A7.
Issue 2: All points on dIB curve have “uninteresting” solutions
The family of bottleneck variables T α deﬁned in Eq.
(5), i.e., the mixture of two “trivial” solutions, are no longer optimal from the point of view of dIB.
However, as mentioned above, any T that is a hard-clustering of Y achieves the bound of Eq.
(A16), and is thus on the dIB curve.
However, given a hard-clustering of Y , there is no reason for the clusters to obey any intuitions about semantic or perceptual similarity between grouped-together classes.
To use the ImageNet example from Section 5, there is no reason for dIB to prefer a coarse-graining with “natural” groups like { border collie , golden retriever } and { teapot , coffeepot } , rather than a coarse-graining with groups like { border collie , teapot } and { golden retriever , coffeepot } , assuming those classes are of the same size.
Distinguishing between such different clusterings requires some similarity or distortion measure between classes, which is not provided by the standard information theoretic measures.
Issue 3: No trade-off among different neural network layers
In Section 6, we showed that for a neural network with many hidden layers and vanishing probability of error, the activity of the different layers will lie along the ﬂat part of the standard IB curve, where there is no strict trade-off between compression
I ( X ; T ) and prediction I ( Y ; T ) .
Note that any bottleneck variable that lies along the ﬂat part of a standard IB curve will have I ( X ; T ) ≥
H ( Y ) and I ( Y ; T ) =
H ( Y ) .
Using the standard information-theoretic inequality H ( T ) ≥ I ( X ; T ) , the same bottleneck variable must therefore have H ( T ) ≥ H ( Y ) and I ( Y ; T )
=
H ( Y ) , thus also lying on the ﬂat part of the dIB curve.
This shows that for a neural network with many hidden layers and vanishing probability of error, the activity of the different layers will also lie along the ﬂat part of the dIB curve, and not demonstrate any strict trade-off between compression H ( T ) and prediction
I ( Y ; T ) .
C C AVEATS WHEN Y IS APPROXIMATELY A DETERMINISTIC FUNCTION OF X
In this Appendix, we show that when the output random variable is a small perturbation away from being a deterministic function of X , the three caveats discussed in the main text persist in an approximate manner.
To derive our results, we ﬁrst prove several useful theorems.
In our proofs, we make central use of Thm.
17.3.3 from Cover & Thomas (2012), which states that, for two distributions a and b over the same ﬁnite set of outcomes Y , if the (cid:96) 1 distance is bounded as | a − b | 1 ≤ (cid:15) ≤ 12 , then
| H ( a ) − H ( b ) | ≤ − (cid:15) log (cid:15) | Y | .
We will also use the weaker bound | H ( a ) − H ( b ) | ≤ log | Y | , which is based on the maximum and minimum entropy for any distribution over | Y | outcomes.
Theorem 1.
Let Z be a random variable (continuous or discrete), and Y a random variable with a ﬁnite set of outcomes.
Consider two joint distributions over Z × Y , p ZY and ˜ p ZY , which have the same marginal over Z , p ( z ) =
˜ p
( z ) , and obey | p ZY − ˜ p ZY
| 1 ≤ (cid:15) ≤ 12 .
Then,
[Equation]
Proof.
For this proof, we will assume that Z is a continuous-valued.
In case it is discrete-valued, the below proof applies after replacing all integrals over Z with summations.
Let ˆ (cid:15) :
= | p ZY − ˜ p ZY | 1 = (cid:82) (cid:80) y | p ZY ( z, y ) − ˜ p ZY ( z, y )
| dz be the actual (cid:96) 1 distance between p ZY and ˜ p ZY , and note that ˆ (cid:15) ≤ (cid:15) by assumption.
Without loss of generality, we assume that ˆ (cid:15) > 0 .
Then, deﬁne the following probability distribution,
[Equation]
Note that q ( z ) integrates to 1 and is absolutely continuous with respect to p ( z ) ( p ( z ) = 0 implies q ( z ) = 0 ), and that (cid:12)(cid:12) p Y | Z = z − ˜ p
Y | Z
= z (cid:12)(cid:12)
=
ˆ (cid:15) q ( z ) p ( z ) .
When then have
[Equation]
where the ﬁrst case follows from (Cover & Thomas, 2012, Thm. 17.3.3), and the second case by considering minimum vs. maximum possibly entropy values.
[Equation]
We upper bound the ﬁrst integral in Eq
(A20) as
[Equation]
where in Eq.
(A21) we dropped the Iverson bracket, in Eq.
(A22) we used the deﬁnition of KL divergence, and then its non-negativity.
To upper bound the second integral in Eq.
(A20), observe that
[Equation]
Combining Eqs.
(A20), (A22) and (A23) gives
[Equation]
The Theorem follows by noting that − ˆ (cid:15) log ˆ (cid:15) is monotonically increasing for ˆ (cid:15) ≤ 12 , and by assumption ˆ (cid:15) ≤ (cid:15) ≤ 12 .
In the following theorems, we use the notation
I q ( Z ; Y ) to indicate mutual information between Z and Y evaluated under some joint distribution q over Z × Y .
Theorem 2.
Let Z be a random variable (continuous or discrete), and Y a random variable with a ﬁnite set of outcomes.
Consider two joint distribution over X × Y , p XY and ˜ p XY , which have the same marginal over Z , p ( z ) =
˜ p
( z ) , and obey | p ZY − ˜ p ZY
| 1 ≤ (cid:15) ≤ 12 .
Then,
[Equation]
Proof.
We ﬁrst bound the (cid:96) 1 distance between p Y and ˜ p Y ,
[Equation]
By (Cover & Thomas, 2012, Thm. 17.3.3),
[Equation]
A24)
We now bound the magnitude of the difference of mutual informations as
[Equation]
where in the last line we’ve used Eq.
(A24) and Theorem 1.
Theorem 3.
Let X be a random variable (continuous or discrete), and Y a random variable with a ﬁnite set of outcomes.
Consider two joint distribution over X × Y , p XY and ˜ p XY , which have the same marginal over X , p ( x ) =
˜ p
( x ) and obey | p XY − ˜ p XY | 1 ≤ (cid:15) ≤ 12 .
Let F ( r ) and ˜ F ( r ) indicate the IB curves for p XY and ˜ p XY respectively, as deﬁned in Eq.
Then for any r > 0 ,
[Equation]
Proof.
For any compression level r > 0 , let T and ˜ T be optimal bottleneck variables for p XY and
˜ p XY respectively,
[Equation]
Let T be deﬁned by the stochastic map q ( t | x ) , and note that p T ( t ) =
(cid:82) p ( x ) q ( t | x ) dx = (cid:82)
˜ p (
[Equation]
) q
[Equation]
t | x )
[Equation]
= ˜ p T ( t ) .
Consider two joint distributions over T × Y ,
[Equation]
and observe that
[Equation]
We now apply Theorem 2 with Z = T and Y =
Y to give I p ( T ; Y ) + 2 (cid:15) log (cid:15) | Y | 2 ≤ I ˜ p ( T ; Y ) .
At the same time, by deﬁnition of the IB curve, it must be that I ˜ p ( T ; Y )
≤ ˜ F
( r ) =
I ˜ p
( ˜ T ; Y ) .
Combining the last two inequalities gives
[Equation]
The same argument can be repeated while exchanging the roles of T and ˜ T (thus deﬁning the joint distributions p ˜ TY and ˜ p ˜ TY , etc.).
This gives the inequality
[Equation]
The Theorem follows by combining Eqs.
(A25) and (A26), and using the relations I p ( T ; Y ) =
F ( r ) ,
I ˜ p ( T ; Y ) =
˜ F ( r ) .
It is important to emphasize that having a small (cid:96) 1 distance between the joint distributions p XY and p XY does not imply that the conditional distributions p Y | X = x and ˜ p Y | X = x have to be close for all x .
In fact, the conditional distributions can be arbitrarily different for some x , as long as such x do not have much probability under p ( x ) .
We use these theorems to demonstrate how, if the joint distribution of X and Y is (cid:15) -close to having
Y be a deterministic function of X , then the three caveats discussed in the main text all apply in an approximate manner, meaning that can only be avoided up to order O ( − (cid:15) log (cid:15) ) .
C. T HE THREE CAVEATS WHEN Y IS APPROXIMATELY A DETERMINISTIC FUNCTION OF X
Issue
1: IB curve cannot be explored using the IB Lagrangian
In the main text, we demonstrated that when Y is a deterministic function of X , the single point (cid:104) H ( Y ) , H ( Y ) (cid:105) on the information plane optimizes the IB Lagrangian for all β ∈ [0 , 1] .
We now consider the case where the joint distribution of X and Y is (cid:15) -close to having Y be a deterministic function of X .
The next theorem shows that in this case, optimizers of the IB Lagrangian must still be close to (cid:104) H ( Y ) , H ( Y ) (cid:105) .
Formally, for any ﬁxed β ∈ (0 , 1) , the maximum possible distance from (cid:104) H ( Y ) , H ( Y ) (cid:105) scales as O ( − (cid:15) log (cid:15) ) .
This implies that for small (cid:15) , it will difﬁcult to ﬁnd solutions substantially different from (cid:104) H ( Y ) , H ( Y ) (cid:105) , as one would have to search using β very close to 0 or very close to 1.
Theorem 4.
Let X be a random variable (continuous or discrete), and Y a random variable with a ﬁnite set of outcomes.
Let
˜ p
XY be a joint distribution over X × Y under which Y = f ( X ) .
Let p XY be a joint distribution over X × Y which has the same marginal over X as ˜ p XY , p ( x ) =
˜ p
( x ) , and obeys | p XY − ˜ p XY
| 1 ≤ (cid:15) ≤ 12 .
Then, for any T which maximizes the p XY IB Lagrangian ( I p ( Y ; T ) − βI p ( X ; T ) ) for some β ∈ (0 , 1) ,
[Equation]
[Equation] [Equation]
) where γ := − 3 (cid:15) log (cid:15) + 5 (cid:15) log | Y | .
Proof.
First, note that if | p XY − ˜ p XY | 1 ≤ (cid:15) , then | p Y − ˜ p Y | 1 ≤ (cid:15) and (see proof of Theorem 2)
[Equation]
Let F and ˜ F indicate the IB curves for p XY and ˜ p XY , respectively, deﬁned as in Eq.
Because ˜ p XY is deterministic, the IB curve ˜ F is the piecewise linear function ˜ F ( x ) =
min { x, H (˜ p
( Y )) } .
Given Eq.
(A29), this means that
[Equation]
A
[Equation]
We now to bound F ( H ( p ( Y ))) as
[Equation]
where in the ﬁrst line we’ve used Theorem 3, and in the second line we used Eq.
(A30).
Now consider T , the bottleneck variable that optimizes the p XY IB Lagrangian for some β ∈ (0 , 1) .
By concavity of the IB curve F , all points on the IB curve must fall below the line with slope β that passes through the point (cid:104)
I p ( X ; T ) ,
I p ( Y ; T ) (cid:105) on the information plane, including the point (cid:104) H ( p ( Y )) , F ( H ( p ( Y ))
(cid:105) .
Formally, we write this condition as
[Equation]
where the second inequality uses Eq.
(A31).
Note that by the DPI, I p ( X ; T ) ≥ I p ( Y ; T ) .
Combining with Eq.
(A32) gives
[Equation]
At the same time, by properties of mutual information, H ( p ( Y )) ≥ I p ( Y ; T ) .
Combining again with Eq.
(A32) gives
[Equation]
The result Eq.
(A27) follows immediately from Eq.
(A33) and Eq.
(A34) and rearranging.
We now derive Eq.
(A28).
The upper bound arises from basic properties of mutual information,
H ( p ( Y )) ≥
I p ( Y ; T ) .
To derive the lower bound, we rewrite Eq.
(A32) as
[Equation]
where in the second line we used Eq.
(A27) and then combined terms.
Issue 2: All points on the IB curve are O ( − (cid:15) log (cid:15) ) away from trivial solutions.
In the main text, we show that when Y is a deterministic function of X , there are trivial bottleneck variables which are optimal for all points of the IB curve.
We now consider the case where the joint distribution of X and Y is (cid:15) -close to having Y be a deterministic function of X .
In this case, by using Theorem 3, it is straightforward to show that there are trivial bottleneck variables that are
O ( − (cid:15) log (cid:15) )
away from being optimal along all points on the IB curve.
Theorem 5.
Let X be a random variable (continuous or discrete), and Y a random variable with a ﬁnite set of outcomes.
Let
˜ p
XY be a joint distribution over X × Y under which Y = f ( X ) .
Let p XY be a joint distribution over X × Y which has the same marginal over X as ˜ p XY , p ( x ) =
˜ p
( x ) , and obeys | p XY − ˜ p XY
| 1 ≤ (cid:15) ≤ 12 .
Then, for any r > 0 , there is some value of α ∈ [0 , 1] such that the bottleneck variable T α deﬁned by the conditional distribution q α
( T α = t | X = x )
= αδ ( t, f ( x ))
+ (1 − α ) δ ( t, 0) satisﬁes
[Equation]
where γ := − 3 (cid:15) log (cid:15) + 5 (cid:15) log | Y | .
Proof.
First, note that if | p XY − ˜ p XY | 1 ≤ (cid:15) , then | p Y − ˜ p Y | 1 ≤ (cid:15) and
[Equation]
(See proof of Theorem 2.)
Let F ( r ) and ˜ F ( r ) indicate the IB curve for p XY and ˜ p XY , respectively, as deﬁned in Eq.
By properties of mutual information and Eq.
(A37), we have
[Equation]
We now consider two cases.
The ﬁrst case is when r ≤ H (˜ p ( Y )) .
Recall that because ˜ p XY is deterministic, the IB curve ˜ F is the piecewise linear function ˜ F ( x ) =
min { x, H (˜ p
( Y )) } .
Thus, we choose an α so that I p ( X ; T α )
=
r and, using Theorem 3,
[Equation]
log
[Equation]
F
[Equation]
− γ .
Second, when r > H (˜ p ( Y )) , we choose α = 1 and note that I p ( T α ; X ) =
H (˜ p
( Y ))
≤ r .
It can be veriﬁed (see for instance proof of Theorem 3) that the joint distributions p T α Y ( t, y )
= (cid:82) p XY ( x, y )
q α ( t | x ) dx and ˜ p T α Y ( t, y )
= (cid:82) ˜
p XY ( x, y )
q α ( t | x ) dx satisfy the conditions of Theorem 2 with Z =
T
α , Y = Y .
Applying that Theorem gives
[Equation]
where the last inequality uses Eq.
(A38).
Issue 3: Prediction/compression trade-off can be most of order O ( − (cid:15) log (cid:15) )
In the main text, we consider a neural network consisting of k layers, and show that when Y is a deterministic function of X , there can be no strict prediction/compression trade-off between the different layers.
We now consider the case where the joint distribution of X and Y is (cid:15) -close to having Y be a deterministic function of X .
As before, we let X be a (continuous or discrete) random variable, and Y a random variable with a ﬁnite set of outcomes.
Let
˜ p
XY be a joint distribution over X × Y under which Y = f ( X ) .
Let p XY be a joint distribution over X × Y which has the same marginal over X as ˜ p XY , p ( x ) =
˜ p
( x ) , and obeys
[Equation]
We assume that for each input X , our classiﬁer outputs the prediction ˆ Y = f
( X ) — that is, it predicts according to the deterministic mapping Y =
f ( X ) ).
For small (cid:15) , this is an optimal prediction strategy in terms of P e , the probability of error.
We write P e for this classiﬁer as
[Equation]
We now restate Fano’s inequality (Eq.
for the activity of the last layer, T k , as
[Equation]
where the second inequality is found in Zhang (2007).
As before, we note that latter layers may compress input information more than earlier ones, in that
I ( T k ; X ) may be arbitrarily smaller than I ( T 1 ; X ) .
Moreover, when Y is not exactly a deterministic function of X , then in principle a strict prediction/compression trade-off between different layers is possible (i.e., I ( T i ; Y ) can decrease for latter layers).
However, if the joint distribution of X and
Y is (cid:15) -close to having Y be a deterministic function, then the magnitude this trade-off is limited in size,
[Equation]
where we use non-negativity of conditional entropy and Eq.
(A39).