T HE V ARIATIONAL D EFICIENCY B OTTLENECK
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce a bottleneck method for learning data representations based on channel deﬁciency, rather than the more traditional information sufﬁciency.
A variational upper bound allows us to implement this method efﬁciently.
The bound itself is bounded above by the variational information bottleneck objective, and the two methods coincide in the regime of single-shot Monte Carlo approximations.
The notion of deﬁciency provides a principled way of approximating complicated channels by relatively simpler ones.
The deﬁciency of one channel w.r.t.
another has an operational interpretation in terms of the optimal risk gap of decision problems, capturing classiﬁcation as a special case.
Unsupervised generalizations are possible, such as the deﬁciency autoencoder, which can also be formulated in a variational form.
Experiments demonstrate that the deﬁciency bottleneck can provide advantages in terms of minimal sufﬁciency as measured by information bottleneck curves, while retaining a good test performance in classiﬁcation and reconstruction tasks.
I NTRODUCTION
The information bottleneck (IB) is an approach to learning data representations based on a notion of minimal sufﬁciency.
The general idea is to map an input source into a representation that retains as little information as possible about the input ( minimality ) but retains as much information as possible in relation to a target variable of interest ( sufﬁciency ).
See Figure 1.
For example, in a classiﬁcation problem, the target variable could be the class label of the input data.
In a reconstruction problem, the target variable could be a denoised reconstruction of the input.
Intuitively, a representation which is minimal in relation to a given task, will discard nuisances in the inputs that are irrelevant to the task, and hence distill more meaningful information and allow for a better generalization.
In a typical bottleneck paradigm, an input variable X is ﬁrst mapped to an intermediate representation variable Z , and then Z is mapped to an output variable of interest Y .
We call the mappings, resp., a representation model ( encoder ) and an inference model ( decoder ).
The channel κ models the true relation between the input X and the output Y .
In general, the channel κ is unknown, and only accessible through a set of examples ( x ( i ) , y ( i ) )
Ni =1 .
We would like to obtain an approximation of κ using a probabilistic model that comprises of the encoder-decoder pair .
The IB methods (Witsenhausen & Wyner, 1975; Tishby et al., 1999; Harremo¨es & Tishby, 2007;
Hsu et al., 2018) have found numerous applications, e.g., in representation learning, clustering, classiﬁcation, generative modeling, model selection and analysis in deep neural networks, among others (see, e.g., Shamir et al., 2008; Gondek & Hofmann, 2003; Higgins et al., 2017; Alemi et al., 2018; Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017).
In the traditional IB, minimality and sufﬁciency are measured in terms of the mutual information.
Computing the mutual information can be challenging in practice.
Various recent works have formulated more tractable functions by way of variational bounds on the mutual information (Chalk et al., 2016;
Alemi et al., 2016; Kolchinsky et al., 2017), sandwiching the objective function of interest.
Instead of maximizing the sufﬁciency term of the IB, we formulate a new bottleneck method that minimizes deﬁciency.
Deﬁciencies provide a principled way of approximating complex channels
[Figure]
Perfect maximization of the IB sufﬁciency corresponds to perfect minimization of the DB deﬁciency.
However, when working over a parametrized model and adding the bottleneck regularizer, both methods have different preferences, with the DB being closer to the optimal risk gap.
Experiments on basic data sets show that the VDB is able to obtain more compressed representations than the
VIB while performing equally well or better in terms of test accuracy.
We describe the details of our method in Section 2.
We elaborate on the theory of deﬁciencies in
Section 3.
Experimental results with the VDB are presented in Section 4.
T HE VARIATIONAL DEFICIENCY BOTTLENECK (VDB)
Let X denote an observation or input variable and Y an output variable of interest.
Let p ( x, y )
=
π ( x ) κ ( y | x ) be the true joint distribution, where the conditional distribution or channel κ ( y | x ) describes how the output depends on the input.
We consider the situation where the true channel is unknown, but we are given a set of N independent and identically distributed (i.i.d.)
samples ( x ( i ) , y ( i ) )
Ni =1 from p .
Our goal is to use this data to learn a more structured version of the channel κ , by ﬁrst “compressing” the input X to an intermediate representation variable Z and subsequently mapping the representation back to the output Y .
The presence of an intermediate representation can be regarded as a bottleneck, a model selection problem, or as a regularization strategy.
We deﬁne a representation model and an inference model using two parameterized families of channels e ( z | x ) and d ( y | z ) .
We will refer to e ( z | x ) and d ( y | z ) as an encoder and a decoder .
The encoder-decoder pair induces a model (cid:98) κ
( y | x ) =
(cid:82) d ( y | z ) e ( z | x ) dz .
Equivalently, we write (cid:98) κ
= d ◦ e .
Given a representation, we want the decoder to be as powerful as the original channel κ in terms of ability to recover the output.
The deﬁciency of a decoder d w.r.t.
κ quantiﬁes the extent to which any pre-processing of d (by way of randomized encodings) can be used to approximate κ (in the
KL-distance sense).
Let M ( X ; Y ) denote the space of all channels from X to Y .
We deﬁne the deﬁciency of d w.r.t.
κ as follows.
Deﬁnition 1.
Given the channel κ ∈ M ( X ; Y ) from X to Y , and a decoder d ∈ M ( Z ; Y ) from some Z to Y , the deﬁciency of d w.r.t.
κ is deﬁned as
[Equation]
Here D KL ( ·(cid:107) · |· ) is the conditional KL divergence (Csisz´ar & K¨orner, 2011), and π is an input distribution over X .
The deﬁnition is similar in spirit to Lucien Le Cam’s notion of weighted deﬁ- ciencies of one channel w.r.t.
another (Le Cam, 1964; Torgersen, 1991, Section 6.2) and its recent generalization by Raginsky (2011).
We propose to train the model by minimizing the deﬁciency of d w.r.t.
κ subject to a regularization that penalizes complex representations.
The regularization is achieved by limiting the rate I ( Z ; X ) , the mutual information between the representation and the raw inputs.
We call our method the Deﬁciency Bottleneck (DB).
The DB minimizes the following objective over all tuples ( e ∈ M ( X ; Z ) , d ∈ M ( Z ; Y )) :
[Equation]
The parameter β ≥ 0 allows us to adjust the level of regularization.
For any distribution r ( z ) , the rate term admits a simple variational upper bound (Csisz´ar & K¨orner, 2011, Eq.
[Equation]
Let ˆ p data be the empirical distribution of the data (input-output pairs).
By noting that δ π ( d, κ ) ≤
D KL ( κ (cid:107) d ◦ e | π ) for any e ∈ M ( X ; Z ) , and ignoring (unknown) data-dependent constants, we obtain the following optimization objective which we call the Variational Deﬁciency Bottleneck (VDB) objective:
[Equation]
The computation is simpliﬁed by deﬁning r ( z ) to be a standard multivariate Gaussian distribution N (0 , I ) , and using an encoder of the form e ( z | x ) =
N ( z | f φ ( x )) , where f φ is a neural network that outputs the parameters of a Gaussian distribution.
Using the reparametrization trick (Kingma & Welling, 2013; Rezende et al., 2014), we then write e ( z | x ) dz = p ( (cid:15) ) d(cid:15) , where z = f ( x, ε ) is a function of x and the realization (cid:15) of a standard normal distribution.
This allows us to do stochastic backpropagation through a single sample z .
The KL term admits an analytic expression for a choice of Gaussian r ( z ) and encoders.
We train the model by minimizing the following empirical objective:
[Equation]
For training, we choose a mini-batch size of N = 100 .
For Monte Carlo estimates of the expectation inside the log, we choose M = 3 , 6 , 12 samples from the encoding distribution.
We note that the Variational Information Bottleneck (VIB) (Alemi et al., 2016) leads to a similarlooking objective function, with the only difference that the sum over j is outside of the log.
By
Jensen’s inequality, the VIB loss is an upper bound to our loss.
If one uses a single sample from the encoding distribution (i.e., M = 1 ), the VDB and the VIB objective functions coincide.
The average log-loss and the rate term in the VDB objective equation 4 are the two fundamental quantities that govern the probability of error when the model is a classiﬁer.
For a discussion of these relations, see Appendix A.
B LACKWELL S UFFICIENCY AND C HANNEL D EFICIENCY
In this section, we discuss an intuitive geometric interpretation of the deﬁciency in the space of probability distributions over the output variable.
We also give an operational interpretation of the deﬁciency as a deviation from Blackwell sufﬁciency (in the KL-distance sense).
Finally, we discuss its relation to the log-loss.
3. D EFICIENCY AND D ECISION G EOMETRY
We ﬁrst formulate the learning task as a decision problem.
We show that δ π ( d, κ ) quantiﬁes the gap in the optimal risks of decision problems when using the channel d rather than κ .
Let X , Y denote the space of possible inputs and outputs.
In the following, we assume that X
and
Y are ﬁnite.
Let P Y be the set of all distributions on Y .
For every x ∈ X , deﬁne κ x ∈ P Y as κ x ( y )
=
κ ( y | x ) , ∀ y ∈ Y .
Nature draws x ∼ π and y ∼ κ x .
The learner observes x and quotes a distribution q x ∈ P Y that expresses her uncertainty about the true value y .
The quality of a quote q x in relation to y is measured by an extended real-valued loss function called the score (cid:96) :
Y× P Y → R .
For a background on such special kind of loss functions see, e.g., Gr¨unwald et al., 2004; Gneiting & Raftery, 2007; Parry et al., 2012.
Ideally, the quote q x should to be as close as possible to the true conditional distribution κ x .
This is achieved by minimizing the expected loss L ( κ x , q x ) :=
E y ∼ κ x (cid:96) ( y, q x ) , for all x ∈ X .
The score is called proper if κ x ∈ arg min q x ∈ P Y L ( κ x , q x ) .
Deﬁne the Bayes act against κ x as the optimal quote
[Equation]
If multiple Bayes acts exist then select one arbitrarily.
Deﬁne the Bayes risk for the distribution p XY ( x, y )
=
π ( x ) κ ( y | x ) as R ( p XY , (cid:96) ) :
=
E x ∼ π L ( κ x , q ∗ x ) .
A score is strictly proper if the Bayes act is unique.
An example of a strictly proper score is the log-loss function deﬁned as (cid:96) L ( y, q ) :
= − log q ( y ) .
For the log-loss, the Bayes act is q ∗ x
= κ x and the Bayes risk is just the conditional entropy
[Equation]
Given a representation z ∈ Z (output by some encoder), when using the decoder d , the learner is constrained to quote a distribution from a subset of P Y .
Let C = conv( { d z : z ∈ Z} )
⊂ P Y be the convex hull of the points { d z } z ∈Z ∈ P Y .
The Bayes act against d z is
[Equation]
q ∗ x Z has an interpretation as the reverse I-projection of κ x to the convex set of probability measures C ⊂ P Y (Csisz ´ ar & Matu ˇ s, 2003) 1 .
We call the associated Bayes risk as the projected
Bayes risk R Z ( p XY , (cid:96) L ) and the associated conditional entropy as the projected conditional entropy
H Z
( Y | X ) ,
[Equation] [Equation]
The gap in the optimal risks, ∆ R := R Z ( p XY ,
[Equation]
L ) − R
[Equation]
p XY , (cid:96) L ) when making a decision based on an intermediate representation and a decision based on the input data is just the deﬁciency.
This follows from noting that
[Equation]
∆ R vanishes if and only if the optimal quote against d z , q ∗ x Z matches κ x for all x, y .
This gives an intuitive geometric interpretation of a vanishing deﬁciency in the space of distributions over Y .
Given a decoder channel d , since δ π ( d, κ ) ≤ D KL ( κ (cid:107) d ◦ e | π ) for any e ∈ M ( X ; Z ) , the loss term in the VDB objective is a variational upper bound on the projected conditional entropy H Z
( Y | X ) .
However, this loss is still a lower bound to the standard cross-entropy loss in the VIB objective (Alemi et al., 2016), i.e.,
[Equation]
This follows simply from the convexity of the negative logarithm function.
3. D EFICIENCY AS A KL- DISTANCE FROM I NPUT -B LACKWELL S UFFICIENCY
In a seminal paper David Blackwell (1953) asked the following question: if a learner wishes to make an optimal decision about some target variable of interest and she can choose between two channels with a common input alphabet, which one should she prefer?
She can rank the channels by comparing her optimal risks: she will always prefer one channel over another if her optimal risk when using the former is at most that when using the latter for any decision problem.
She can also rank the variables purely probabilistically: she will always prefer the former if the latter is an output-degraded version of the former, in the sense that she can simulate a single use of the latter by randomizing at the output of the former.
Blackwell showed that these two criteria are equivalent.
Very recently, Nasser (2017) asked the same question, only now the learner has to choose between two channels with a common output alphabet.
Given two channels, κ ∈ M ( X ; Y ) and d ∈ M ( Z ; Y ) , we say that κ is input-degraded from d and write d (cid:23) Y κ if κ = d ◦ e for some e ∈ M
( X ; Z ) .
Stated in another way, d can be reduced to κ by applying a randomization at its input.
Nasser (2017) gave a characterization of input-degradedness that is similar to Blackwell’s theorem (Blackwell, 1953).
We say, d is input-Blackwell sufﬁcient for κ if d (cid:23) Y κ .
Input-Blackwell sufﬁciency induces a preorder on the set of all channels with the same output alphabet.
In practice, most channels are uncomparable, i.e., one cannot be reduced to another by a randomization.
When such is the case, the deﬁciency quantiﬁes how far the true channel κ is from being a randomization (by way of all input encodings) of the decoder d .
See Appendix B for a brief summary of Blackwell-Le Cam theory.
3. D EFICIENCY AND THE L OG -L OSS
When Y − X − Z is a Markov chain, the conditional mutual information I ( Y ; X | Z ) is the Bayes risk gap for the log-loss.
This is apparent from noting that I ( Y ; X | Z ) =
H ( Y | Z ) − H
( Y | XZ )
=
H ( Y | Z ) − H ( Y | X ) =
R ( p ZY , (cid:96) L ) − R ( p XY , (cid:96) L ) .
This risk gap is closely related to Blackwell’s original notion of sufﬁciency.
Since the log-loss is strictly proper, a vanishing I ( Y ; X | Z ) implies that the risk gap is zero for all loss functions.
This suggests that minimizing the log-loss risk gap under a suitable regularization constraint is a potential recipe for constructing representations Z that are approximately sufﬁcient for X w.r.t.
Y , since in the limit when I ( Y ; X | Z ) =
one would achieve I ( Y ; Z )
=
I ( Y ; X ) .
This is indeed the basis for the IB algorithm (Tishby et al., 1999) and its generalization, clustering with Bregman divergences (Banerjee et al., 2005; van Rooyen &
Williamson, 2015; 2014).
One can also approximate a sufﬁcient statistic by minimizing deﬁciencies instead.
This is motivated from noting the following proposition.
Proposition 2.
When Y − X − Z is a Markov chain, δ π ( d, κ ) =
⇐⇒ I
( Y ; X | Z )
= 0 .
In general, for the bottleneck paradigms involving the conditional mutual information (IB) and the deﬁciency (DB), we have the following relationship:
[Equation]
Our experiments corroborate that for achieving the same level of sufﬁciency, one needs to store less information about the input X when minimizing the deﬁciencies than when minimizing the conditional mutual information.
E XPERIMENTS
We present some experiments on the MNIST dataset (LeCun & Cortes, 2010).
Classiﬁcation on
MNIST is a very well studied problem.
The main objective of our experiments is to evaluate the information-theoretic properties of the representations learned by the VDB and whether it can match the classiﬁcation accuracy provided by other bottleneck methods.
For the encoder, we use a fully connected feedforward network with 784 input units–1024 ReLUs– 1024 ReLUs–512 linear output units.
The deterministic output of this network is interpreted as the vector of means and variances of a 256 dimensional Gaussian distribution.
The decoder is a
[Figure]
experiment.
Each curve corresponds to one value of M = 1 , 3 , 6 , 12 .
Notice the generalization gap for small values of β (towards the right of the plot).
The lower right panel plots the corresponding information bottleneck curve.
The lower left panel plots the minimality term vs. β .
Evidently, the levels of compression vary depending on M .
Higher values of M (our method) lead to a more compressed representation.
For M = 1 , the VDB and the VIB models coincide.
[Table]
simple logistic regression model with a softmax layer.
These are the same settings of the model used by Alemi et al.
We implement the algorithm in Tensorﬂow and train for 200 epochs using the Adam optimizer.
As can be seen from the upper left panel in Figure 2, the test accuracy is stable with increasing M .
Here, M is the number of encoder output samples used in the training objective.
We note that M =is just the VIB model (Alemi et al., 2016).
L is the number of encoder output samples used for evaluating the classiﬁer (i.e., we use 1 L (cid:80)
Lj =1 d ( y | z ( j ) ) where z ( j ) ∼ e ( z | x ) ).
Numerical values of the
[Figure]
test accuracies are provided in Table 1 for different values of β and M .
We see a slight improvement in the test accuracies for higher values of M .
See Figure 5 for train and test accuracies for L =and L = 12 in Appendix D.
The traditional IB paradigm traces the mutual information I ( Z ; Y ) between representation and output (sufﬁciency) vs. the mutual information
I ( Z ; X ) between representation and input (minimality), for different values of the regularization parameter β .
This curve is called the information bottleneck curve (Tishby et al., 1999).
In the case of the VDB, we deﬁne the corresponding sufﬁciency term as J ( Z ; Y ) :
= H ( Y ) − E ( x,y ) ∼ [Equation] p data (cid:2) − log( (cid:82) d ( y | z ) e ( z | x ) dz ) (cid:3) .
Here H ( Y ) =
log 2 (10) is the entropy of the output which has 10 classes.
In our method, “more informative” means “less deﬁcient”.
The upper right panel in Figure 2 shows the deﬁciency bottleneck curve which traces J ( Z ; Y ) vs. I ( Z ; X ) for different values of β at the end of training.
For orientation, lower values of β have higher values of I ( Z ; X ) (towards the right of the plot).
For small values of β , when the effect of the regularization is negligible, the bottleneck allows more information from the input through the representation.
In this case, J ( Z ; Y ) increases on the training set, but not necessarily on the test set.
This is manifest in the gap between the train and test curves indicative of a degradation in generalization.
For intermediate values of β , the gap is smaller for larger values of M (our method).
The lower right panel plots the corresponding information bottleneck curve.
The lower left panel in Figure 2 plots the minimality term
I ( Z ; X ) vs. β .
We see that, for β in the range between 10 − 8 and 10 − 4 , for the same level of sufﬁciency, setting M =consistently achieves more compression of the input compared to the setting M = 1 .
The dynamics of the information quantities during training are also interesting.
We provide ﬁgures on these in
Appendix D.
In order to visualize the representations, we also train the VDB on MNIST with a 2 dimensional representation.
We use the same settings as before, with the only difference that the dimension of
[Figure]
[Figure] [Figure]
I (
[Figure] [Figure]
MNIST
[Figure]
K
[Figure]
M = 1 M = 3 M =
M =     epoch 0.
t e s t a cc u r a cy
MNIST, K=2, L=
M = 1 M =
M = 6 M = 50 100 150epoch 0.
t e s t a cc u r a cy
MNIST, K=2, L=
M = 1 M =
M = 6 M =     epoch 0.
t e s t a cc u r a cy
MNIST, K=2, L=
M = 1 M =
M = 6 M = 50 100 150epoch 0.
t e s t a cc u r a cy
MNIST, K=2, L=
M = 1 M =
M = 6 M = the output layer of the encoder is 4 , with two coordinates representing the mean, and two a diagonal covariance matrix.
The results are shown in Figure 3.
For β = 10 − 5 , the representations are well separated, depending on the class.
For related ﬁgures in the setting of unsupervised learning see
Appendix E.
The learning dynamics of the mutual information and classiﬁcation accuracy are shown in Figure 4.
The left panel has an interpretation in terms of a phase where the model is mainly ﬁtting the input-output relationship and hence increasing the mutual information I ( Z ; Y ) , followed by a compression phase, where training is mainly reducing I ( Z ; X ) , leading to a better generalization.
The right panel shows the test accuracy as training progresses.
Higher values of M (our method) usually lead to better accuracy.
An exception is when the number L of posterior samples for classiﬁcation is large.
D ISCUSSION
We have formulated a bottleneck method based on channel deﬁciencies.
The deﬁciency of a decoder with respect to the true channel between input and output quantiﬁes how well a randomization at the decoder input (by way of stochastic encodings) can be used to simulate the true channel.
The VDB has a natural variational formulation which recovers the VIB in the limit of a single sample of the encoder output.
Experiments demonstrate that the VDB can learn more compressed representations while retaining the same discriminative capacity.
The method has a statistical decision-theoretic appeal.
Moreover, the resulting variational objective of the DB can be implemented as an easy modiﬁcation of the VIB, with little to no computational overhead.
Given two channels that convey information about a target variable of interest, two different notions of deﬁciencies arise, depending on whether the target resides at the common input or the common output of the given channels.
When the target is at the common output of the two channels, as is in a typical bottleneck setting (see Figure 1), our Deﬁnition 1 has a natural interpretation as a
KL-divergence from input-Blackwell sufﬁciency (Nasser, 2017).
Here sufﬁciency is achieved by applying a randomization at the input of the decoder with the goal of simulating the true channel.
The notion of input-Blackwell sufﬁciency contrasts with Blackwell’s original notion of sufﬁciency (Blackwell, 1953) in the sense that Blackwell’s theory compares two channels with a common input.
One can again deﬁne a notion of deﬁciency in this setting (see Appendix B for a discussion on deﬁciencies in the classical Blackwell setup).
The associated channels (one from Y to Z and the other from Y to X ) do not however have a natural interpretation in a typical bottleneck setting.
In contrast, the input-Blackwell setup appears to be much more intuitive in this context.
The more detailed view of information emerging from this analysis explains various effects and opens the door to multiple generalizations.
In the spirit of the VDB, one can formulate a deﬁciency autoencoder as well (see sketch in Appendix E).
On a related note, we mention that the deﬁciency is a lower bound to a quantity called the Unique information (Bertschinger et al., 2014; Banerjee et al., 2018a)
(see details in Appendix C).
An alternating minimization algorithm similar in spirit to the classical Blahut-Arimoto algorithm (Blahut, 1972) has been proposed to compute this quantity (Banerjee et al., 2018b).
A deep neural network implementation of such an algorithm remains a challenge.
In the limit β → 0 , the VDB is a step forward towards estimating the unique information.
This might be of independent interest in improving the practicality of the theory of information decompositions.
A PPENDIX A M ISCLASSIFICATION ERROR AND THE AVERAGE LOG - LOSS
In a classiﬁcation task, the goal is to use the training dataset to learn a classiﬁer (cid:98) κ ( y | x ) that minimizes the probability of error under the true data distribution, deﬁned as follows.
[Equation]
It is well known that the optimal classiﬁer that gives the smallest probability of error is the Bayes classiﬁer (Boucheron et al., 2005).
Since we do not know the true data distribution we try to learn based on the empirical error.
Directly minimizing the empirical probability of error over the training dataset is in general a NP-hard problem.
In practice, one minimizes a surrogate loss function that is a convex upper bound on P E .
A natural surrogate is the average log-loss function E ( x,y ) ∼ p [ − log (cid:98) κ
( y | x )] .
When the model is (cid:98) κ
= d ◦ e , the following upper bounds are immediate from using Jensen’s inequality.
[Equation]
The bound using the standard cross-entropy loss is evidently weaker than the average log-loss.
A lower bound on the probability of error is controlled by a convex functional of the mutual information between the representation and the raw inputs
I ( Z ; X ) (Vera et al., 2018, see, e.g., Lemma 4).
The average log-loss and the rate term in the VDB objective equation 4 are two fundamental quantities that govern the probability of error.
B C LASSICAL T HEORY OF C OMPARISON OF C HANNELS
In this section, we discuss the classical theory of comparison of channels due to Blackwell (1953) and its extension by Le Cam (1964); Torgersen (1991) and more recently by Raginsky (2011).
Suppose that a learner wishes to predict the value of a random variable Y that takes values in a set Y .
She has a set of actions A .
Each action incurs a loss (cid:96) ( y, a ) that depends on the true state y of Y and the chosen action a .
Let π Y encode the learners’ uncertainty about the true state y .
The tuple ( π Y , A , (cid:96) ) is called a decision problem .
Before choosing her action, the learner observes a random variable X through a channel κ ∈ M ( Y ; X ) .
An ideal learner chooses a strategy ρ ∈
M ( X ; A ) that minimizes her expected loss or risk R ( π Y , κ, ρ, (cid:96) ) :
=
E y ∼ π Y E a ∼ ρ ◦ κ y (cid:96) ( y, a ) .
The optimal risk when using the channel κ is R ( π Y , κ, (cid:96) ) :
= min ρ ∈ M
( X ; A ) R ( π Y , κ, ρ, (cid:96) ) .
Suppose now that the learner has to choose between X and another random variable Z that she observes through a second channel µ ∈ M ( Y ; Z ) with common input Y .
She can always discard X in favor of Z if, knowing Z , she can simulate a single use of X by randomly sampling
a x (cid:48) ∈ X after each observation z ∈ Z .
Deﬁnition 3.
We say that X is output-degraded from Z w.r.t.
Y , denoted Z (cid:119) (cid:48)
Y X , if there exists a random variable X (cid:48)
such that the pairs ( Y, X ) and ( Y, X (cid:48) ) are stochastically indistinguishable, and
Y − Z − X (cid:48) is a Markov chain.
She can also discard X if her optimal risk when using Z is at most that when using X for any decision problem.
Write Z (cid:119) Y X if R ( π Y , κ, (cid:96) ) ≥ R ( π Y , µ, (cid:96) ) for any decision problem.
Blackwell (1953) showed the equivalence of these two relations.
Theorem 4.
(Blackwell’s Theorem) Z (cid:119) (cid:48)
Y X ⇐⇒ Z
(cid:119) Y X .
Write µ (cid:119) Y κ if κ = λ ◦ µ for some λ ∈ M
( Z ; X ) .
If π Y has full support, then it easy to check that µ (cid:119)
Y κ ⇐⇒ Z (cid:119) (cid:48)
Y X (Bertschinger & Rauh, 2014, Theorem 4).
The learner can also compare κ and µ by comparing the mutual informations
I ( Y ; X ) and I ( Y ; Z ) between the common input Y and the channel outputs X and Z .
Deﬁnition 5.
µ is said to be more capable than κ , denoted µ (cid:119) mc Y κ , if I ( Y ; Z ) ≥ I ( Y ; X ) for all probability distribution on Y .
It follows from the data processing inequality that µ (cid:119) Y κ = ⇒ µ (cid:119)
mc Y κ .
However, the converse implication is not true in general (K¨orner & Marton, 1975).
The converse to the Blackwell’s theorem states that if the relation Z (cid:119) (cid:48)
Y X does not hold, then there exists a set of actions A and a loss function (cid:96) ( y, a ) ∈ R Y×A such that R ( π Y , κ, (cid:96) ) <
R ( π Y , µ, (cid:96) ) .
Le Cam introduced the concept of a deﬁciency of µ w.r.t.
κ to express this deﬁcit in optimal risks (Le Cam, 1964) in terms of an approximation of κ from µ via Markov kernels.
Deﬁnition 6.
The Le Cam deﬁciency of µ w.r.t.
κ is
[Equation]
(14) where (cid:107) ·
(cid:107) TV denotes the total variation distance.
When the distribution of the common input to the channels is ﬁxed, one can deﬁne a weighted deﬁciency (Torgersen, 1991, Section 6.2).
Deﬁnition 7.
Given Y ∼ π Y , the weighted Le Cam deﬁciency of µ w.r.t.
κ is
[Equation]
Le Cam’s randomization criterion (Le Cam, 1964) shows that deﬁciencies quantify the maximal gap in the optimal risks of decision problems when using the channel µ rather than κ .
Theorem 8 (Le Cam (1964)) .
Fix µ ∈ M ( Y ; Z ) , κ ∈ M ( Y ; Z ) and a probability distribution π Y on Y and write (cid:107) (cid:96) (cid:107) ∞ =
max y,a (cid:96) ( y, a ) .
For every (cid:15) > 0 , δ π ( µ, κ ) < (cid:15) if and only if R ( π Y , µ, (cid:96) ) −
R ( π Y , κ, (cid:96) ) < (cid:15) (cid:107) (cid:96) (cid:107) ∞ for any set of actions A and any bounded loss function (cid:96) .
Raginsky (2011) introduced a broad class of deﬁciency-like quantities using the notion of a generalized divergence between probability distributions that satisﬁes a monotonicity property w.r.t.
data processing.
The family of f -divergences due to Csisz´ar belongs to this class (Liese & Vajda, 2006).
Deﬁnition 9.
The f -deﬁciency of µ w.r.t.
κ is
[Equation]
Many common divergences, such as the KL divergence, the reverse-KL divergence, and the total variation distance are f -divergences.
When the channel µ is such that its output is constant, no matter what the input, the corresponding f -deﬁciency is called f -informativity (Csisz´ar, 1972).
The f -informativity associated with the KL divergence is just the channel capacity which has a geometric interpretation as an “information radius” (Csisz ´ ar & K ¨ orner, 2011).
We can also deﬁne a weighted f -deﬁciency of µ w.r.t.
κ .
Deﬁnition 10.
The weighted f -deﬁciency of µ w.r.t.
κ is
[Equation]
Specializing to the KL divergence, we have the following deﬁnition.
Deﬁnition 11.
The weighted output deﬁciency of µ w.r.t.
κ is
[Equation]
where the subscript o in δ π o emphasizes the fact that the randomization is at the output of the channel µ .
Note that δ πo ( µ, κ ) =
if and only if Z (cid:119) (cid:48)
Y X , which captures the intuition that if δ πo ( µ, κ ) is small, then X is approximately output-degraded from Z w.r.t.
Y .
Using Pinsker’s inequality, we have
[Equation]
C T HE U NIQUE I NFORMATION B OTTLENECK
In this section, we give a new perspective on the Information Bottleneck paradigm using nonnegative mutual information decompositions.
The quantity we are interested in is the notion of Unique information proposed in (Bertschinger et al., 2014).
Work in similar vein include (Harder et al., 2013) and more recently (Banerjee et al., 2018a) which gives an operationalization of the unique information.
Consider three jointly distributed random variables Y , X , and Z .
Y is the target variable of interest.
The mutual information between Y and X can be decomposed into information that X has about Y that is unknown to Z (we call this the unique information of X w.r.t.
Z ) and information that X has about Y that is known to Z (we call this the shared information).
[Equation]
Conditioning on Z destroys the shared information but creates complementary or synergistic information from the interaction of X and Z .
[Equation]
Using the chain rule, the total information that the pair ( X, Z ) conveys about the target Y can be decomposed into four terms.
[Equation]
[Equation]
(cid:102) UI , (cid:102) SI , and (cid:102) CI are nonnegative functions that depend continuously on the joint distribution of ( Y, X, Z ) .
For completeness, we rewrite the information decomposition equations below.
[Equation]
The unique information can be interpreted as either the conditional mutual information without the synergy, or as the mutual information without the redundancy.
When Y − X − Z is a Markov chain, the information decomposition is
[Equation]
The Information bottleneck (Tishby et al., 1999) minimizes the following objective
[Equation]
26) over all encoders e ∈ M ( X ; Z ) : Y − X − Z .
Since Y − X − Z is a Markov chain, the sufﬁciency term in the IB objective depends on the pairwise marginals ( Y, X ) and ( Y, Z ) , while the minimality term depends on the ( X, Z ) -marginal.
From equation 25b, it follows that one can equivalently write the IB objective function as
[Equation]
From an information decomposition perspective, the original IB is actually minimizing just the unique information subject to a regularization constraint.
This is a simple consequence of the fact that the synergistic information (cid:102) CI ( Y ; X, Z ) =
(see equation 25d) when we have the Markov chain condition Y − X − Z .
Hence, one might equivalently call the original IB as the Unique information bottleneck .
Appealing to classical Blackwell theory, Bertschinger et al.
(2014) deﬁned a nonnegative decomposition of the mutual information
I ( Y ; XZ ) based on the idea that the unique and shared information should depend only on the pairwise marginals ( Y, X ) and ( Y, Z ) .
Deﬁnition 12.
Let ( Y, X, Z ) ∼ P , Y ∼ π Y and let κ ∈ M
( Y ; X ) , µ ∈ M ( Y ; X ) be two channels with the same input alphabet such that P Y X ( y, x ) =
π Y ( y ) κ y ( x ) and P Y Z ( y, z ) =
π Y ( y ) µ y ( z ) .
Deﬁne
[Equation]
where the subscript Q in I Q denotes that joint distribution on which the quantities are computed.
The functions UI , SI , and CI are nonnegative and satisfy equation 24.
Furthermore, UI and SI depend on the marginal distributions of the pairs ( Y, X ) and ( Y, Z ) .
Only the function CI depends on the full joint distribution P .
UI satisﬁes the following intuitive property in relation to Blackwell’s theorem 4.
Proposition 13.
(Bertschinger et al., 2014, Lemma 6) UI ( Y ; X \ Z ) =
⇐⇒ Z
(cid:119) (cid:48)
Y X .
Proposition 2 follows from noting that δ π ( d, κ ) =
⇐⇒ UI ( Y ; X \ Z ) =
(Bertschinger et al., 2014, Theorem 22) and the fact
that UI ( Y ; X \ Z ) =
I ( Y ; X | Z ) when Y − X − Z is a Markov chain.
D A DDITIONAL FIGURES ON VDB EXPERIMENTS
[Figure]
E U NSUPERVISED REPRESENTATION LEARNING OBJECTIVES
Recent work on variational autoencoders (VAEs) has shown that optimizing the standard evidence lower-bound (ELBO) is not sufﬁcient in itself for learning useful representations (Chen et al., 2016;
Alemi et al., 2018; Zhao et al., 2017).
Generalizing the ELBO by incorporating different bottleneck constraints (Zhao et al., 2017; Higgins et al., 2017; Rezende & Viola, 2018) has shown promise in learning better latent codes.
In this section, we discuss some preliminary results on an unsupervised version of the VDB objective.
We discuss its relation to VAE objectives such as the β -VAE (Higgins et al., 2017) and the importance weighted autoencoder (IWAE)
(Burda et al., 2015).
E. β -VAE AND IWAE
A variational autoencoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014) is a directed probabilistic model that deﬁnes a joint density p θ ( x, z )
= p θ ( x | z ) p ( z ) between some continuous latent variable z and observed data x .
p ( z ) is chosen to be a simple prior distribution over the latents (e.g., isotropic unit Gaussian) and p θ ( x | z ) is a decoder network that models the data generative process.
Maximum likelihood estimation of the model parameters θ is in general intractable.
VAEs instead maximize the evidence lower-bound (ELBO) by jointly training the decoder with an auxiliary encoder network q φ ( z | x ) , parameterized by φ .
The ELBO objective is
[Equation]
The ELBO is optimized by sampling from q φ ( z | x ) using the reparameterization trick to obtain low variance Monte Carlo estimates of the gradient.
The ELBO is tight when q φ ( z | x ) matches the true posterior p θ ( z | x ) .
The tightness of the bound is coupled to the expressiveness of the encoder distribution.
When q φ ( z | x ) is chosen as a simple diagonal Gaussian, minimizing D KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) encourages the model’s posterior to be approximately factorial which limits the capacity of the model.
The importance weighted autoencoder (IWAE)
(Burda et al., 2015) addresses this issue.
The key observation is that the uniﬁed ELBO objective in equation 29 is the log of a single (unnormalized) importance weight p θ ( x,z ) q φ ( z | x ) with a proposal density deﬁned by q φ ( z | x ) .
Using more samples from the proposal can only tighten the bound.
The M -sample IWAE bound is
[Equation]
L 1 is just the ELBO and lim M →∞ L M = log p θ ( x ) .
The M -sample bound can alternatively be written as (Le et al., 2017)
[Equation]
where Q IS and P IS are, resp., proposal and target densities deﬁned on an extended sample space.
[Equation]
Optimizing over this extended sample space allows for more ﬂexible decoders and gives the IWAE additional degrees of freedom to model complex posteriors.
The IWAE bound is in fact equivalent to the ELBO in expectation with a more complex proposal density ˜ q IW deﬁned by importance reweighting (Bachman & Precup, 2015; Cremer et al., 2017; Naesseth et al., 2018).
[Equation]
where the inner expectation is w.r.t.
the unnormalized distribution ˜ q IW deﬁned as follows.
[Equation]
For M = 1 , ˜ q IW ( z | x ) =
q φ ( z | x ) , and the uniﬁed ELBO objective admits the following decomposition.
[Equation]
The ﬁrst term can be interpreted as an expected reconstruction cost and the second term as a regularizer (Kingma & Welling, 2013).
For M > 1 however, the IWAE bound admits no such decomposition.
As M → ∞ , ˜ q IW approaches the true posterior p θ ( z | x ) .
However, the magnitude of the gradient w.r.t.
the encoder parameters also decays to zero as more samples are used (Rainforth et al., 2018).
This potentially limits the IWAE’s ability to learn useful representations.
The β -VAE (Higgins et al., 2017; Burgess et al., 2018) augments the ELBO by incorporating a hyperparameter β for the regularization term.
[Equation]
For β = 1 , we recover the standard ELBO.
Higgins et al.
(2017) showed that when the prior p ( z ) and the encoder q φ ( z | x ) are chosen as diagonal Gaussians, reducing the capacity of the latent bottleneck by choosing β > 1 incentivizes the latent representations to be more disentangled.
Higher values of β also results in a more coherent latent space so that the reconstructions interpolate smoothly on latent traversals.
E. U NSUPERVISED D EFICIENCY B OTTLENECK O BJECTIVE
We now discuss some preliminary results on an unsupervised version of the VDB objective on the
MNIST and Fashion-MNIST datasets.
We consider a standard VAE model with the prior p ( z ) and the encoder q φ ( z | x ) parameterized by diagonal Gaussians.
The encoder has two hidden layers with 200 units each.
The decoder is a factorized Bernoulli parameterized by MLP’s with two hidden layers with 200 units each.
Using factorial decoder densities constrains the model to use the latent code to attain a high likelihood (Chen et al., 2016; Alemi et al., 2018).
This is simple way to achieve a nonvanishing mutual information between the latent variable and the input.
This is important in our setting since we are interested in learning a useful representation.
We train the model by minimizing the following unsupervised version of the VDB objective.
[Equation]
We note that the β -VAE has a similar-looking training objective, with the only difference that the averaging w.r.t.
to the posterior samples is outside the log .
In particular, if M = 1 , this is just the β -VAE objective.
The objective in equation 38 also shares some superﬁcial similarities with the
IWAE objective for β = 1 .
Note however, as discussed in Section E.1, we cannot decompose the
IWAE objective for M > 1 .
In particular, this implies we cannot trade-off reconstruction ﬁdelity for learning more meaningful representations by incorporating bottleneck constraints.
We have not explored if using more complex posteriors such as the ˜ q IW is possible in the bottleneck formulation.
For training, we choose a mini-batch size of N = 100 and draw M = 1 , 3 , 6 samples from the approximate posterior by using the reparameterization trick (Kingma & Welling, 2013).
For our choice of Gaussian prior and encoder, the KL term can be computed and differentiated without estimation.
We estimate the expected loss term using Monte Carlo sampling.
Since the expectation is inside the log , higher values of M increases the variance of gradient estimates.
Also numerically handling the log requires some care.
We used the log-sum-exp trick to compute the expectation.
For values of M beyond 12, we observe some degradation in the visualizations.
[Equation]
[Equation] [Equation]