U npacking I nformation B ottlenecks : S urrogate O bjectives for D eep L earning
Anonymous authors
Paper under double-blind review
A bstract
The Information Bottleneck principle o ﬀ ers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models.
However, multiple competing objectives are proposed in the literature, and the information-theoretic quantities used in these objectives are di ﬃ cult to compute for large deep neural networks, which in turn limits their use as a training objective.
In this work, we review these quantities, and compare and unify previously proposed objectives, which allows us to develop surrogate objectives more friendly to optimization without relying on cumbersome tools such as density estimation.
We ﬁnd that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures.
We demonstrate our insights on MNIST, CIFAR-10 and Imagenette with modern DNN architectures (ResNets).
I ntroduction
The Information Bottleneck (IB) principle, introduced by Tishby et al.
(2000), proposes that training and generalization in deep neural networks (DNNs) can be explained by information-theoretic principles (Tishby and Zaslavsky, 2015; Shwartz-Ziv and Tishby, 2017; Achille and Soatto, 2018a).
This is attractive as the success of DNNs remains largely unexplained by tools from computational learning theory (Zhang et al., 2016; Bengio et al., 2009).
The IB principle suggests that learning consists of two competing objectives: maximizing the mutual information between the latent representation and the label to promote accuracy, while at the same time minimizing the mutual information between the latent representation and the input to promote generalization.
Following this principle, many variations of IB objectives have been proposed (Alemi et al., 2016; Strouse and Schwab, 2017; Fischer and Alemi, 2020; Fischer, 2020; Fisher, 2019; Gondek and Hofmann, 2003; Achille and Soatto, 2018a), which, in supervised learning, have been demonstrated to beneﬁt robustness to adversarial attacks (Alemi et al., 2016; Fisher, 2019) and generalization and regularization against overﬁtting to random labels (Fisher, 2019).
Whether the beneﬁts of training with IB objectives are due to the IB principle , or some other unrelated mechanism, remains unclear (Saxe et al., 2019; Amjad and Geiger, 2019; Tschannen et al., 2019), suggesting that although recent work has also tied the principle to successful results in both unsupervised and self-supervised learning (Oord et al., 2018;
Belghazi et al., 2018; Zhang et al., 2018;
Burgess et al., 2018, among others), our understanding of how IB objectives a ﬀ ect representation learning remains unclear.
Critical to studying this question is the computation of the information-theoretic quantities 1 used.
While progress has been made in developing mutual information estimators for DNNs (Poole et al., 2019; Belghazi et al., 2018; Noshad et al., 2019; McAllester and Stratos, 2018; Kraskov et al., 2004), current methods still face many limitations when concerned with high-dimensional random variables (McAllester and Stratos, 2018) and rely on complex estimators or generative models.
This presents a challenge to training with IB objectives.
In this paper, we analyze information quantities and relate them to surrogate objectives for the IB principle which are more friendly to optimization, showing that complex or intractable IB objectives can be replaced with simple, easy-to-compute surrogates that produce similar performance and similar
[Figure]
e s i
[Figure]
a l I n
[Figure]
o
[Figure]
i
[Figure]
n I [
[Figure]
Y | Z
[Figure] [Figure]
Y | X )
L a b e l U n c e r t
a
i n t y
H ( Z | X )
E n c o d i n g
U n c e r t a i n t y
I(Y;Z) Preserved Relevant Information
I(X;Y|Z)Residual
Information
I(X;Z|Y)Redundant
Information
H(Y|Z)Decoder
Uncertainty
H(Z|Y)Reverse
Decoder Uncertainty
H(X|Y,Z)
I(X;Y)Relevant
Information
I(X;Z) Preserved
Information behaviour of information quantities over training.
Sections 2 & 3 review commonly-used information quantities for which we provide mathematically grounded intuition via information diagrams and unify di ﬀ erent IB objectives by identifying two key information quantities, Decoder Uncertainty
H[Y | Z] and Reverse Decoder Uncertainty H[Z | Y] which act as the main loss and regularization terms in our uniﬁed IB objective.
In particular, Section 3.2 demonstrates that using the Decoder
Uncertainty as a training objective can minimize the training error, and shows how to estimate an upper bound on it e ﬃ ciently for well-known DNN architectures.
We expand on the ﬁndings of
Alemi et al.
(2016) in their variational IB approximation and demonstrate that this upper bound is equal to the commonly-used cross-entropy loss 2 under dropout regularization.
Section 3.3 examines pathologies of di ﬀ erential entropies that hinder optimization and proposes adding Gaussian noise to force di ﬀ erential entropies to become non-negative, which leads to new surrogate terms to optimize the Reverse Decoder Uncertainty.
Altogether this leads to simple and tractable surrogate IB objectives such as the following, which uses dropout, adds Gaussian noise over the feature vectors f
( x ; η ), and uses an L2 penalty over the noisy feature vectors:
[Equation]
Section 4 describes experiments that validate our insights qualitatively and quantitatively on MNIST,
CIFAR-10 and Imagenette, and shows that with objectives like the one in equation (1) we obtain information plane plots (as in ﬁgure 1) similar to those predicted by Tishby and Zaslavsky (2015).
Our simple surrogate objectives thus induce the desired behavior of IB objectives while scaling to large, high-dimensional datasets.
We present evaluations on CIFAR-10 and Imagenette images 3 .
Compared to existing work, we show that we can optimize IB objectives for well-known DNN architectures using standard optimizers, losses and simple regularizers, without needing complex estimators, generative models, or variational approximations.
This will allow future research to make better use of IB objectives and study the IB principle more thoroughly.
B ackground
Information quantities & information diagrams.
We denote entropy H[ · ], joint entropy H[ · , · ], conditional entropy H[ · | · ], mutual information I[ · ; · ] and Shannon’s information content h ( · ) (Cover and Thomas, 2012; MacKay, 2003; Shannon, 1948).
We will further require the Kullback-Leibler divergence D KL ( · || · ) and cross-entropy H( · || · ).
The deﬁnitions can be found in section A.1.
We will use di ﬀ erential entropies interchangeably with entropies: equalities between them are preserved in the di ﬀ erential setting, and inequalities will be covered in section 3.3.
Information diagrams (I-diagrams), like the one depicted in ﬁgure 2, clarify the relationship between information quantities: similar to Venn diagrams, a quantity equals the sum of its parts in the diagram .
Importantly, they o ﬀ er a grounded intuition as Yeung (1991) show that we can deﬁne a signed measure µ * such that information quantities map to abstract sets and are consistent with set operations.
We provide details on how to use I-diagrams and what to watch out for in section A.2.
Probabilistic model.
We will focus on a supervised classiﬁcation task that makes prediction ˆY given data X using a latent encoding Z, while the provided target is Y. We assume categorical Y and ˆY, and continuous X. Our probabilistic model based on these assumptions is as follows:
[Equation]
Thus, Z and Y are independent given X, and ˆY is independent of X and Y given Z.
The data distribution ˆp( x , y ) is only available to us as an empirical sample distribution.
θ are the parameters we would like to learn.
p θ
( z | x ) is the encoder from data X to latent Z, and p θ (ˆ y | z ) the decoder from latent Z to prediction ˆY.
Together, p θ ( z | x ) and p θ (ˆ y | z ) form the discriminative model p θ (ˆ y | x ):
[Equation]
We can derive the cross-entropy loss H(ˆp( y | x ) ||
p θ ( ˆY = y | x ))
(Solla et al., 1988; Hinton, 1990) by minimizing the Kullback-Leibler divergence between the empirical sample distribution ˆp ( x , y ) and the parameterized distribution p θ ( x ) p θ (ˆ y | x ), where we set p θ ( x ) =
ˆp( x ).
See section D.1.
Mickey Mouse I-diagram.
The corresponding I-diagram for X, Y, and Z is depicted in ﬁgure 2.
As some of the quantities have been labelled before, we try to follow conventions and come up with consistent names otherwise.
Section B.1 provides intuitions for these quantities, and section B.2 lists all deﬁnitions and equivalences explicitly .
For categorical Z, all the quantities in the diagram are positive, which allows us to read o ﬀ inequalities from the diagram: only I[X; Y; Z] could be negative, but as Y and Z are independent given X, we have I[Y; Z | X] = 0, and I[X
; Y; Z] = I[Y; Z] − I[Y; Z | X] =
I[Y; Z] ≥ 0 .
Section 3.3 investigates how to preserve inequalities for continuous Z.
S urrogate IB & DIB objectives 3.1 IB O bjectives
Tishby et al.
(2000) introduce the IB objective as a relaxation of a constrained optimization problem: minimize the mutual information between the input X and its latent representation Z while still accurately predicting Y from Z .
An analogous objective which yields deterministic Z , the Deterministic
Information Bottleneck (DIB) was proposed by Strouse and Schwab (2017).
Letting β be a Lagrange multiplier, we arrive at the IB and DIB objectives:
[Equation]
This principle can be recast as a generalization of ﬁnding minimal su ﬃ cient statistics for the labels given the data (Shamir et al., 2010; Tishby and Zaslavsky, 2015; Fisher, 2019): it strives for minimality and su ﬃ ciency of the latent Z. Minimality is achieved by minimizing the Preserved Information
I[X; Z]; while su ﬃ ciency is achieved by maximizing the Preserved Relevant Information I[Y; Z].
We defer an in-depth discussion of the IB principle to the appendix Section C.1.
We discuss the several variants of IB objectives, and justify our focus on IB and DIB, in Section C.2.
The information quantities that appear in the IB objective are not tractable to compute for the representations learned by many function classes of interest, including neural networks; for example,
Strouse and Schwab (2017) only obtain an analytical solution to their Deterministic Information
Bottleneck (DIB) method for the tabular setting.
Alemi et al.
(2016) address this challenge by constructing a variational approximation of the IB objective, but their approach has not been applied to more complex datasets than MNIST variants.
Belghazi et al.
(2018) use a separate statistics network to approximate the mutual information, a computationally expensive strategy that does not easily lend itself to optimization.
In this section, we introduce and justify tractable surrogate losses that are easier to apply in common deep learning pipelines, and which can be scaled to large and high-dimensional datasets.
We begin by proposing the following reformulation of IB and DIB objectives.
Proposition 1.
For IB, we obtain
[Equation]
and, for DIB,
[Equation]
with β (cid:48) :
= 1 β − 1 ∈ [0 , ∞ ) and β (cid:48)(cid:48) :
= 1 β ∈ [0 , 1) .
The derivation can be found in section C.3.
In the next sections, we show that Decoder Uncertainty H[Y | Z] provides a loss term, which minimizes the training error, and DIB’s Reverse Decoder Uncertainty H[Z | Y] and IB’s
Redundant
Information I[X; Z | Y], respectively, provide a regularization term, which helps generalization.
Another perspective can be found by relating the objectives to the Entropy Distance Metric introduced by MacKay (2003), which we detail in section C.4.
3.2 D ecoder U ncertainty H[Y | Z]
The Decoder Uncertainty H[Y | Z] is the ﬁrst term in our reformulated IB and DIB objectives, and captures the data ﬁt component of the IB principle.
This quantity is not easy to compute directly for arbitrary representations Z , so we turn our attention to two related entities instead, where we use θ as subscript to mark dependence on the model: the Prediction Cross-Entropy, denoted H
θ [Y | X] (more commonly known as the model’s cross-entropy loss; see section D.1), and the Decoder Cross-Entropy, denoted H
θ [Y | Z].
Noting that h ( x ) = − ln x , we deﬁne these terms as follows:
[Equation]
Jensen’s inequality yields H
θ
[Y | X] ≤ H θ
[Y | Z], with equality i ﬀ Z is a deterministic function of X .
The notational similarity 4 between H θ [Y | Z] and H[Y | Z] is deliberately suggestive: this cross-entropy bounds the conditional entropy H[Y | Z], as characterized in the following proposition.
Proposition 2.
The Decoder Cross-Entropy provides an upper bound on the Decoder Uncertainty:
[Equation]
and further bounds the training error:
[Equation]
Likewise, for H θ
[Y | X] and H[Y | X] .
See section D.2 for a derivation.
Hence, by bounding D KL (p( y | z ) || p
θ (ˆ y | z )), we can obtain a bound for the training error in terms of
H[Y | Z].
We examine one way of doing so by using optimal decoders p
θ (ˆ y | z ) :
=
p(Y = ˆ y | z ) for the case of categorical Z in section E.
Alemi et al.
(2016) use the Decoder Cross-Entropy bound in equation (9) to variationally approximate p( y | z ).
We make this explicit by applying the reparameterization trick to rewrite the latent z as a parametric function of its input x and some independent auxiliary random variable η , i.e. f θ ( x , η ) D =
z ∼ p θ ( z | x ), yielding
[Equation]
Equation (11) can be applied to many forms of stochastic regularization that turn deterministic models into stochastic ones, in particular dropout.
This allows us to use modern DNN architectures as stochastic encoders.
Dropout regularization When we interpret η as a sampled dropout mask for a DNN, DNNs that use dropout regularization (Srivastava et al., 2014), or variants like DropConnect (Wan et al., 2013a), ﬁt the equation above as stochastic encoders.
Monte-Carlo dropout (Gal and Ghahramani, 2016), for example, even speciﬁcally estimates the predictive mean p θ (ˆ y | x ) from equation (3).
The following result extends the observation by Burda et al.
that sampling yields an unbiased estimator for the Decoder Cross-Entropy H
θ
[Y | Z], while it only yields a biased estimator for the Prediction
Cross-Entropy H θ
[Y | X] (which it upper-bounds).
[Figure] [Figure]
Corollary 1.
Let x , y , z and f θ be deﬁned as previously, with η a sampled stochastic dropout mask.
Then h (cid:16) p θ ( ˆY = y |
z = f θ ( x ; η )) (cid:17) evaluated for a single sample η is an unbiased estimator of the
Decoder Cross-Entropy H θ [Y | Z] , and an estimator of an upper bound on the Prediction CrossEntropy H θ
[Y | X] .
This distinction between the Decoder Cross-Entropy and the Prediction Cross-Entropy has been observed in passing in the literature, but not made explicit.
Multi-sample approaches like Multi-Sample
Dropout (Inoue, 2019), for example, optimize H
θ
[Y | Z], while Importance Weighted Stochastic Gradient Descent (Noh et al., 2017) optimizes H θ
[Y | X].
Dusenberry et al.
(2020) observe empirically in the di ﬀ erent context of rank-1 Bayesian Neural Networks that optimizing H
θ
[Y | Z] instead of
H θ
[Y | X] is both easier and also yields better generalization performance (NLL, accuracy, and ECE), while they also put forward an argument for why the stochastic gradients for H
θ [Y | Z] might beneﬁt from lower variance.
We empirically compare training with either cross-entropy in section G.3.
and show results in ﬁgure G.11 in the appendix.
We conclude this section by highlighting that
H[Y | Z] is therefore already minimized in modern DNN architectures that use dropout together with a cross-entropy loss.
This means that, at least for one half of our reformulation of the IB objective, we can apply o ﬀ -the-shelf, scalable objectives and optimizers for its minimization.
3.3 S urrogates for the regularization terms
In the previous section, we have examined how to tractably estimate the error minimization term
H[Y | Z].
In this section, we will examine tractable optimization of the regularization terms H[Z | Y] and I[X; Z | Y], respectively.
We discuss how to minimize entropies meaningfully and show how this uniﬁes DIB and IB via the inequality I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z] before providing tractable upper-bounds for H[Z | Y] and H[Z].
Di ﬀ erential entropies In most cases, the latent Z is a continuous random variable in many dimensions.
Unlike entropies on discrete probability spaces, di ﬀ erential entropies deﬁned on continuous spaces are not bounded from below.
This means that the DIB objective is not guaranteed to have an optimal solution and allows for pathological optimization trajectories in which the variance of the latent Z can be scaled to be arbitrarily small, achieving arbitrarily high-magnitude negative entropy.
We provide a toy experiment demonstrating this in section G.4.
Intuitively, one can interpret this issue as being allowed to encode information in an arbitrarily-small real number using inﬁnite precision, similar to arithmetic coding (MacKay, 2003; Shwartz-Ziv and
Tishby, 2017) 5 .
In practice, due to ﬂoating point constraints, optimizing DIB naively will invariably end in garbage predictions and underﬂow as activations approach zero.
It is therefore not desirable for training.
This is why Strouse and Schwab (2017) only consider analytical solutions to DIB by evaluating a limit for the tabular case.
MacKay (2003) proposes the introduction of noise to solve this issue in the application of continuous communication channels.
However, here we propose adding speciﬁc noise to the latent representation to lower-bound the conditional entropy of Z , which allows us to enforce non-negativity across all IB information quantities as in the discrete case and transport inequalities to the continuous case: for a continuous ˆZ ∈ R k and independent noise (cid:15) , we set Z :
=
ˆZ + (cid:15) ; the di ﬀ erential entropy then satisﬁes H[Z]
= H[ˆZ + (cid:15) ] ≥ H[ (cid:15) ]; and by using zero-entropy noise (cid:15) ∼ N (0 ,
12 π e I k ) speciﬁcally, we obtain H[Z] ≥ H
[ (cid:15) ]
= 0 .
Proposition 3.
After adding zero-entropy noise, the inequality I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z] also holds for continuous Z , and we can minimize I[X; Z | Y] in the IB objective by minimizing H[Z | Y] or H[Z] , similarly to the DIB objective.
Strictly speaking, zero-entropy noise is not necessary for optimizing the bounds: any Gaussian noise is su ﬃ cient, but zero-entropy noise is aesthetically appealing as it preserves inequalities from the discrete setting.
In a sense, this propostion bounds the IB objective by the DIB objective.
However, adding noise changes the optimal solutions: whereas DIB in Strouse and Schwab (2017) leads to hard clustering in the limit, adding noise leads to soft clustering when optimizing the DIB objective, as is the case with the IB objective.
We show in section F.6 that minimizing the DIB objective with noise leads to soft clustering (for the case of an otherwise deterministic encoder).
Altogether, in addition to Shwartz-Ziv and Tishby (2017), we argue that noise is essential to obtain meaningful di ﬀ erential entropies and to avoid other pathological cases as described further in section F.7.
It is not generally possible to compute H[Z | Y] exactly for continuous latent representations Z , but we can derive an upper bound.
The maximum-entropy distribution for a given covariance matrix Σ is a Gaussian with the same covariance.
Proposition 4.
The Reverse Decoder Uncertainty can be approximately bounded using the empirical variance (cid:99) Var[Z i | y ] :
[Equation]
where Z i are the individual components of Z .
H[Z] can be bounded similarly.
More generally, we can create an even looser upper bound by bounding the mean squared norm of the latent:
[Equation]
with C (cid:48) :
= ke 2 C / k 2 π e for Z ∈ R k .
See section F.2 for proof.
Surrogate objectives These surrogate terms provide us with three di ﬀ erent upper-bounds that we can use as surrogate regularizers.
We refer to them as: conditional log-variance regularizer (log Var[Z | Y]), log-variance regularizer (log Var[Z]) and activation L 2 regularizer ( E (cid:107) Z (cid:107) 2 ).
We can now propose the main results of this paper: IB surrogate objectives that reduce to an almost trivial implementation using the cross-entropy loss and one of the regularizers above while adding zero-entropy noise to the latent Z.
Theorem 1.
Let Z be obtained by adding a single sample of zero-entropy noise to a single sample of the output z of the stochastic encoder.
Then each of the following objectives is an estimator of an upper bound on the IB objective.
In particular, for the surrogate objective E (cid:107) Z (cid:107) 2 , we obtain:
[Equation] [Equation]
for log Var[Z
[Equation]
Y] :
[Equation]
and for log Var[Z] :
[Equation]
For the latter two surrogate regularizers, we can relate their coe ﬃ cient γ to β (cid:48) , β (cid:48)(cid:48) and β from section 3.
However, as regularizing E (cid:107) Z (cid:107) 2 does not approximate an entropy directly, its coe ﬃ cient does not relate to the Lagrange multiplier of any ﬁxed IB objective.
We compare the performance of these objectives in section 4.
[Figure] [Figure]
and BasicIterative attacks of varying (cid:15) values.
E xperiments
We now provide empirical veriﬁcation of the claims made in the previous sections.
Our goal in this section is to highlight two main ﬁndings: ﬁrst, that our surrogate objectives obtain similar behavior to what we expect of exact IB objectives with respect to their e ﬀ ect on robustness to adversarial examples.
In particular, we show that our surrogate IB objectives improve adversarial robustness compared to models trained only on the cross-entropy loss, consistent with the ﬁndings of Alemi et al.
Second, we show the e ﬀ ect of our surrogate objectives on information quantities during training by plotting information plane diagrams, demonstrating that models trained with our objectives trade o ﬀ between I [ X ; Z ] and I [ Y ; Z ] as expected.
We show this by recovering information plane plots similar to the ones in Tishby and Zaslavsky (2015) and qualitatively examine the optimization behavior of the networks through their training trajectories.
We demonstrate the scalability of our surrogate objectives by applying our surrogate IB objectives to the CIFAR-10 and Imagenette datasets, high-dimensional image datasets.
For details about our experiment setup, DNN architectures, hyperparameters and additional insights, see section G. In particular, empirical quantiﬁcation of our observations on the relationship between the Decoder Cross-Entropy loss and the Prediction Cross-Entropy are deferred to the appendix due to space limitations as well as the description of the toy experiment that shows that minimizing H[Z | Y] for continuous latent Z without adding noise does not constrain information meaningfully and that adding noise solves the issue as detailed in section 3.3.
Robustness to adversarial attacks Alemi et al.
(2016) and Fischer and Alemi (2020) observe that their IB objectives lead to improved adversarial robustness over standard training objectives.
We perform a similar evaluation to see whether our surrogate objectives also see improved robustness.
We train a fully-connected residual network on CIFAR-10 for a range of regularization coe ﬃ cients γ using our E (cid:107) Z (cid:107) 2 surrogate objective; we then compare against a similar regularization method that does not have an information-theoretic interpretation: L2 weight-decay.
We inject zero-entropy noise in both cases.
After training, we evaluate the models on adversarially perturbed images using the
FGSM (Szegedy et al., 2013), PGD (Madry et al., 2018), BasicIterative (Kurakin et al., 2017) and
DeepFool (Moosavi-Dezfooli et al., 2016) attacks for varying levels of the perturbation magnitude parameter (cid:15) .
We also compare to a simple unregularized cross-entropy baseline (black dashed line).
To compute overall robustness, we use each attack in turn and only count a sample as robust if it defeats them all.
As depicted in ﬁgure 5, we ﬁnd that our surrogate objectives yield signiﬁcantly more robust models while obtaining similar test accuracy on the unperturbed data whereas weight-decay regularization reduces robustness against adversarial attacks.
Plots for the other two regularizers can be found in the appendix in ﬁgure G.13 and ﬁgure G.14.
[Figure]
I
[Figure]
i
[Figure]
n I [
[Figure]
|
[Figure]
Information plane plots for CIFAR-10 To compare the di ﬀ erent surrogate regularizers, we again use a ResNet18 model on CIFAR-10 with zero-entropy noise added to the ﬁnal layer activations
Z, with K = 256 dimensions, as an encoder and add a single K × 10 linear unit as a decoder.
We train with the surrogate objectives from section 3.3 for various γ , chosen in logspace from di ﬀ erent ranges to compensate for their relationship to β as noted in section 3.3: for log Var[Z], γ ∈ [10 − 5 , 1]; for log Var[Z | Y], γ ∈ [10 − 5 , 10]; and for E (cid:107) Z (cid:107) 2 , by trial and error, γ ∈ [
10 −
We estimate information quantities using the method
Kraskov et al
Figure 6 shows an information plane plot for regularizing with E (cid:107) Z (cid:107) 2 for di ﬀ erent γ over di ﬀ erent epochs for the training set.
Similar to Shwartz-Ziv and Tishby (2017), we observe that there is an initial expansion phase followed by compression.
The jumps in performance (reduction of the
Residual Information) are due to drops in the learning rate.
In ﬁgure 4, we can see that the saturation curves for all 3 surrogate objectives qualitatively match the predicted curve from Tishby and Zaslavsky (2015).
Figure G.1 shows the di ﬀ erence between the regularizers more clearly, and ﬁgure G.3 shows the training trajectories for all three regularizers.
More details in section G.3.1.
Information plane plots for Imagenette To show that our surrogate objectives also scale up to larger datasets, we run a similar experiment on Imagenette (Howard, 2019), which is a subset of ImageNet with 10 classes with 224 × 224 × 3 = 1 .
5 × 10 5 input dimensions, and on which we obtain 90% test accuracy.
See the ﬁgure 1, which shows the trajectories on the test set.
We obtain similar plots to the ones obtained for CIFAR-10, showing that our surrogate objectives scale well to higher-dimensional datasets despite their simplicity.
C onclusion
The contributions of this paper have been threefold: First, we have proposed simple, tractable training objectives which capture many of the desirable properties of IB methods while also scaling to problems of interest in deep learning.
For this we have introduced implicit stochastic encoders, e.g. using dropout, and compared multi-sample dropout approaches to identify the one that approximates the Decoder Uncertainty H θ
[Y | Z], relating them to the cross-entropy loss that is commonly used for classiﬁcation problems.
This widens the range of DNN architectures that can be used with IB objectives considerably. We have demonstrated that our objectives perform well for practical DNNs without cumbersome density models.
Second, we have motivated our objectives by providing insight into limitations of IB training, demonstrating how to avoid pathological behavior in IB objectives, and by endeavouring to provide a unifying view on IB approaches.
Third, we have provided mathematically grounded intuition by using I-diagrams for the information quantities involved in IB, shown common pitfalls when using information quantities and how to avoid them, and examined how the quantities relate to each other.
Future work investigating the practical constraints on the expressivity of a given neural network may provide further insight into how to measure compression in neural networks.
Moreover, the connection to Bayesian Neural Networks remains to be explored.
A I nformation quantities & information diagrams
Here we introduce notation and terminology in greater detail than in the main paper.
We review well-known information quantities and provide more details on using information diagrams (Yeung, 1991).
A.1 I nformation quantities
We denote entropy H[ · ], joint entropy H[ · , · ], conditional entropy H[ · | · ], mutual information I[ · ; · ] and
Shannon’s information content h ( · ) following Cover and Thomas (2012); MacKay (2003); Shannon (1948) :
[Equation]
where X
[Equation]
Y , Z are random variables and x
[Equation]
y , z are outcomes these random variables can take
[Equation]
We use di ﬀ erential entropies interchangeably with entropies.
We can do so because equalities between them hold as can be veriﬁed by symbolic expansions.
For example,
[Equation]
which is valid in both the discrete and continuous case (if the integrals all exist)
The question of how to transfer inequalities in the discrete case to the continuous case is dealt with in section 3.
[Equation]
We will further require the Kullback-Leibler divergence D KL ( · || · ) and cross-entropy H( · || · ):
[Equation]
A.2 I nformation diagrams
Information diagrams (I-diagrams), like the one depicted in ﬁgure 2 (or ﬁgure H.1 for a bigger version), visualize the relationship between information quantities: Yeung (1991) shows that we can deﬁne a signed measure µ * such that these well-known quantities map to abstract sets and are consistent with set operations.
[Equation]
Note that interaction information (McGill, 1954) follows as canonical generalization of the mutual information to multiple variables from that work, whereas total correlation does not.
In other words, equalities can be read o ﬀ directly from I-diagrams: an information quantity is the sum of its parts in the corresponding I-diagram.
This is similar to Venn diagrams.
The sets used in
I-diagrams are just abstract symbolic objects, however.
An important distinction between I-diagrams and Venn diagrams is that while we can always read o ﬀ inequalities in Venn diagrams, this is not true for I-diagrams in general because mutual information terms in more than two variables can be negative.
In Venn diagrams, a set is always larger or equal any subset.
However, if we show that all information quantities are non-negative, we can read o ﬀ inequalities again.
We do this for ﬁgure 2 at the end of section 2 for categorical Z and expand this to continuous
Z in section 3.3.
Thus, we can treat the Mickey Mouse I-diagram like a Venn diagram to read o ﬀ equalities and inequalities.
Nevertheless, caution is warranted sometimes.
As the signed measure can be negative, µ * ( X ∩ Y ) =does not imply X ∩ Y = ∅ : deducing that a mutual information term is 0 does not imply that one can simply remove the corresponding area in the I-diagram.
There could be Z with µ * (( X ∩ Y ) ∩ Z ) < 0, such that µ * ( X ∩ Y ) =
µ * ( X ∩ Y ∩ Z ) + µ * ( X ∩ Y − Z )
=
but X ∩ Y (cid:44) ∅ .
This also means that we cannot drop the term from expressions when performing symbolic manipulations.
This is of particular importance because a mutual information of zero means two random variables are independent, which might invite one drawing them as disjoint areas.
The only time where one can safely remove an area from the diagram is for atomic quantities, which are quantities which reference all the available random variables (Yeung, 1991).
For example, when we only have three variables X , Y , Z , I
[ X ; Y ; Z ] and I[ X ; Y | Z ] are atomic quantities.
We can safely remove atomic quantities from I-diagrams when they are 0 as there are no random variables left to apply that could lead to the problem explored above.
Continuing the example,
= I[ X ; Y ; Z ]
= µ * ( X ∩ Y ∩ Z ) would imply X ∩ Y ∩ Z = ∅ , and we could remove it from the diagram without loss of generality.
Moreover, atomic I[ X ; Y | Z ] = µ * ( X ∩ Y − Z )
=then and could be removed from the diagram as well.
We only use I-diagrams for the three variable case, but they supply us with tools to easily come up with equalities and inequalities for information quantities.
In the general case with multiple variables, they can be di ﬃ cult to draw, but for Markov chains they can be of great use.
B M ickey M ouse I- diagram B.1 I ntuition for the M ickey M ouse information quantities
We base the names of information quantities on existing conventions and come up with sensible extensions.
For example, the name Preserved Relevant Information for I[Y; Z] was introduced by
Tishby and Zaslavsky (2015).
It can be seen as the intersection of I[X; Z] and I[X; Y] in the I-diagram, and hence we denote I[X; Z] Preserved Information and I[X; Y] Relevant Information, which are sensible names as we detail below.
We identify the following six atomic quantities:
Label Uncertainty H[Y | X] quantiﬁes the uncertainty in our labels.
If we have multiple labels for the same data sample, it will be > 0.
It is 0 otherwise.
Encoding Uncertainty H[Z | X] quantiﬁes the uncertainty in our latent encoding given a sample.
When using a Bayesian model with random variable ω for the weights, one can further split this term into H[Z | X] = I[Z; ω | X] + H[Z | X , ω ], so uncertainty stemming from weight uncertainty and independent noise (Houlsby et al.
2011; Kirsch et al.
Preserved Relevant Information I[Y; Z] quantiﬁes information in the latent that is relevant for our task of predicting the labels (Tishby and Zaslavsky, 2015).
Intuitively, we want to maximize it for good predictive performance.
Residual Information I[X
Y | Z] quantiﬁes information for labels that is not captured and Zaslavsky
but would be useful to be captured
Redundant Information I[X; Z | Y] quantiﬁes information in the latent that is not needed for predicting the labels 6 .
We also identify the following composite information quantities:
Relevant Information I[X; Y] =
I[X; Y | Z] + I[Y; Z] quantiﬁes the information in the data that is relevant for the labels and which our model needs to capture to be able to predict the labels.
Preserved Information I[X; Z] =
I[X; Z | Y] + I[Y; Z] quantiﬁes information from the data that is preserved in the latent.
Decoder Uncertainty H[Y | Z] =
I[X; Y | Z] + H[Y | X] quantiﬁes the uncertainty about the labels after learning about the latent Z. If H[Y | Z] reaches 0, it means that no additional information is needed to infer the correct label Y from the latent Z: the optimal decoder can be a deterministic mapping.
Intuitively, we want to minimize this quantity for good predictive performance.
Reverse Decoder Uncertainty H[Z | Y] =
I[X; Z | Y]
+ H[Z | X] quantiﬁes the uncertainty about the latent Z given the label
Y.
We can imagine training a new model to predict Z given Y and minimizing H[Z | Y] to 0 would allow for a deterministic decoder from the latent to given the label.
Nuisance 7 H[X | Y] = H[X | Y , Z] + I[X; Z] quantiﬁes the information in the data that is not relevant for the task (Achille and Soatto, 2018a).
B.2 D efinitions & equivalences
The following equalities can be read o ﬀ from ﬁgure 2.
For completeness and to provide a handy reference, we list them explicitly here.
They can also be veriﬁed using symbolic manipulations and the properties of information quantities.
Equalities for composite quantities:
[Equation]
[Equation] [Equation]
[Equation] [Equation]
We can combine the atomic quantities into the overall Label Entropy and Encoding Entropy:
[Equation]
[Equation]
We can express the Relevant Information I[X; Y], Residual Information I[X; Y | Z], Redundant
Information I[X; Z | Y] and Preserved Information I[X; Z] without X on the left-hand side:
[Equation] [Equation]
[Equation] [Equation]
This simpliﬁes estimating these expressions as X is usually much higher-dimensional and irregular than the labels or latent encodings.
We also can rewrite the Preserved Relevant Information I[Y; Z] as:
[Equation] [Equation]
[Equation]
C I nformation bottleneck & related works C.1 G oals & motivation
The IB principle from Tishby et al.
(2000) can be recast as a generalization of ﬁnding minimal su ﬃ cient statistics for the labels given the data (Shamir et al., 2010; Tishby and Zaslavsky, 2015;
Fisher, 2019): it strives for minimality and su ﬃ ciency of the latent Z. Minimality is about minimizing amount of information necessary of X for the task, so minimizing the Preserved Information I[X; Z]; while su ﬃ ciency is about preserving the information to solve the task, so maximizing the Preserved
Relevant Information I[Y; Z].
From ﬁgure 2, we can read o ﬀ the deﬁnitions of Relevant Information and Preserved Information:
[Equation]
and see that maximizing the Preserved Relevant Information I[Y; Z] is equivalent to minimizing the Residual Information I[X
; Y | Z], while minimizing the Preserved Information I[X; Z] at the same time means minimizing the Redundant Information I[X; Z | Y], too, as I[X; Y] is constant for the given dataset 8 .
Moreover, we also see that the Preserved Relevant Information I[Y; Z] is upper-bounded by Relevant Information I[X; Y], so to capture all relevant information in our latent, we want I[X; Y] = I[Y; Z].
Using the diagram, we can also see that minimizing the Residual Information is the same as minimizing the Decoder Uncertainty H[Y | Z]:
[Equation]
Ideally, we also want to minimize the Encoding Uncertainty H[Z | X] to ﬁnd the most deterministic latent encoding Z. Minimizing the Encoding Uncertainty and the Redundant Information I[X; Z | Y] together is the same as minimizing the Reverse Decoder Uncertainty H[Z | Y].
All in all, we want to minimize both the Decoder Uncertainty H[Y | Z] and the Reverse Decoder
Uncertainty H[Z | Y].
C.2 IB objectives
“T
he I
nformation
B
ottleneck M ethod ” (IB)
Tishby et al.
(2000) introduce MI ( X ; ˆ X ) − β MI ( ˆ X ; Y ) as optimization objective for the Information
Bottleneck.
We can relate this to our notation by renaming ˆ X =
Z, such that the objective becomes “min I[X; Z] − β I[Y; Z]”.
The IB objective minimizes the Preserved Information I[X; Z] and trades it o ﬀ with maximizing the Preserved Relevant Information I[Y; Z].
Tishby and Zaslavsky (2015) mention that the IB objective is equivalent to minimizing I[X; Z] + β I[X; Y | Z], see our discussion above.
Tishby et al.
(2000) provide an optimal algorithm for the tabular case, when X, Y and Z are all categorical.
This has spawned additional research to optimize the objective for other cases and speciﬁcally for DNNs.
“D eterministic I nformation B ottleneck ” (DIB)
Strouse and Schwab (2017) introduce as objective “min H[Z] − β I[Y; Z]”.
Compared to the IB objective, this also minimizes H[Z | X] and encourages determinism.
Vice-versa, for deterministic encoders, H[Z | X] = 0, and their objective matches the IB objective.
Like Tishby et al.
(2000), they provide an algorithm for the tabular case.
To do so, they examine an analytical solution for their objective as it is unbounded: H[Z | X] → −∞ for the optimal solution.
As we discuss in section 3.3, it does not easily translate to a continuous latent representation.
“D eep V ariational I nformation B ottleneck ”
Alemi et al.
(2016) rewrite the terms in the bottleneck as maximization problem “max I[Y; Z] − β I[X; Z]” and swap the β parameter.
Their β would be 1 / β in IB above, which emphasizes that I[Y; Z] is important for performance and I[X; Z] acts as regularizer.
The paper derives the following variational approximation to the IB objective, where z = f θ ( x , (cid:15) ) denotes a stochastic latent embedding with distribution p θ ( z | x ), p θ (ˆ y | z ) denotes the decoder, and r( z ) is some ﬁxed prior distribution on the latent embedding:
[Equation]
In principle, the distributions p θ (ˆ y | z ) and p θ ( z | x ) could be given by arbitrary parameterizations and function approximators.
In practice, the implementation of DVIB presented by Alemi et al.
(2016) constructs p θ ( z | x ) as a multivariate Gaussian with parameterized mean and parameterized diagonal covariance using a neural network, and then uses a simple logistic regression to obtain p θ (ˆ y | z ), while arbitrarily setting r( z ) to be a unit Gaussian around the origin.
The requirement for p θ ( z | x ) to have a closed-form Kullback-Leibler divergence limits the applicability of the DVIB objective.
The DVIB objective can be written more concisely as
[Equation]
in the notation introduced in section 3.
We discuss the regularizer in more detail in section F.3.
“C onditional E ntropy B ottleneck ”
In a preprint, Fisher (2019) introduce their Conditional Entropy Bottleneck as “min I[X; Z | Y] −
I[Y; Z]”.
We can rewrite the objective as I[X; Z | Y] + I[X; Y | Z] − I[X; Y], using equations (30) and (31).
The last term is constant for the dataset and can thus be dropped.
Likewise, the IB objective can be rewritten as minimizing I[X; Z | Y] + ( β − 1)I[X; Y | Z].
The two match for β = 2.
Fisher (2019) provides experimental results that favorably compare to Alemi et al.
(2016), possibly due to additional ﬂexibility as Fisher (2019) do not constrain p( z ) to be a unit Gaussian and employ variational approximations for all terms.
We relate CEB to Entropy Distance Metric in section C.4.
“C onditional E ntropy B ottleneck ” (2020)
In a substantial revision of the preprint, Fischer (2020) change their Conditional Entropy Bottleneck to include a Lagrange multiplier: “min I[X; Z | Y] − γ I[Y; Z]”.
Their VCEB objective can be written more concisely as
[Equation]
where, without writing down the probabilistic model, we introduce variational approximations for the
Reverse Decoder Uncertainty and the Encoding Uncertainty.
They are the ﬁrst to report results on CIFAR-10.
It is not clear how they parameterize the model they use for CIFAR-10.
They use one Gaussian per class to model H
θ [Z | Y].
“CEB I mproves M odel R obustness ”
Fischer and Alemi (2020) take CEB and switch to a deterministic model which they turn it into a stochastic encoder by adding unit Gaussian noise.
They use Gaussians of ﬁxed variance to variationally approximate q( y | z ): for each class, q( y | z ) is modelled as a separate Gaussian.
They are the ﬁrst to report results on ImageNet and report good rebustness against adversarial attacks without adversarial training.
C.3 C anonical IB & DIB objectives
We expand the IB and DIB objectives into “disjoint” terms and drop constant ones to ﬁnd a more canonical form.
This leads us to focus on the optimization of the Decoder Uncertainty H[Y | Z] along with additional regularization terms.
In section 3.2, we discuss the properties of H[Y | Z], and in section 3.3 we examine the regularization terms.
Proposition.
For IB
[Equation]
we obtain
[Equation]
and, for DIB,
[Equation]
[Equation]
with β (cid:48) :
= 1 β − 1 ∈ [0 , ∞ ) and β (cid:48)(cid:48) :
= 1 β ∈ [0 , 1) .
Proof.
For the steps marked with *, we make use of β > 1.
For IB, we obtain arg min I[X; Z] −
[Equation]
I[Y; Z]
[Equation]
X
[Equation]
and, for DIB,
[Equation]
with β (cid:48) :
= 1 β − 1 ∈ [0 , ∞ ).
Similarly, we show for DIB
[Equation]
with β (cid:48)(cid:48) :
= 1 β ∈ [0 , 1), which is relevant in section 3.3.
We limit ourselves to β > 1, because, for β < 1, we would be maximizing the Decoder Uncertainty, which does not make sense: the obvious solution to this is one where Z contains no information on Y, that is p( y | z ) is uniform.
In the case of DIB, it is to map every input deterministically to a single latent; whereas for IB, we only minimize the Redundant Information, and the solution is free to contain noise.
For β = 1, we would not care about Decoder Uncertainty and only minimize
Redundant Information and Reverse Decoder Uncertainty, respectively, which allows for arbitrarily bad predictions.
[Equation]
We note that we have β (cid:48)
=
β (cid:48)(cid:48) 1 − β (cid:48)(cid:48) using the relations above.
C.4 IB objectives and the E ntropy D istance M etric
Another perspective on the IB objectives is by expressing them using the Entropy Distance Metric.
MacKay (2003, p. 140) introduces the entropy distance
[Equation]
[Equation]
as a metric when we identify random variables up to permutations of the labels for categorical variables: if the entropy distance is 0, Y and Z are the same distribution up to a consistent permutation of the labels (independent of X).
If the entropy distance becomes 0, both H[Y | Z] =
= H[Z | Y], and we can ﬁnd a bijective map from Z to Y.
We can express the Reverse Decoder Uncertainty H[Z | Y] using the Decoder Uncertainty H[Y | Z] and the entropies:
[Equation]
and rewrite equation (35) as
[Equation]
For optimization purposes, we can drop constant terms and rearrange:
[Equation]
C.4.1 R ewriting IB and DIB using the E ntropy D istance M etric
For β ≥ 1, we can rewrite equations (IB) and (DIB) as:
[Equation]
for IB, and
[Equation] [Equation]
for DIB and replace β with γ = 1 − 2 β ∈ [ − 1 , 1] which allows for a linear mix between H[Y | Z] and
H[Z | Y].
DIB will encourage the model to match both distributions for γ = 0 ( β = 2), as we obtain a term that matches the Entropy Distance Metric from section C.4, and otherwise trades o ﬀ
Decoder Uncertainty and Reverse Decoder Uncertainty.
IB behaves similarly but tends to maximize Encoding Uncertainty as γ − 1 ∈ [ − 2 , 0].
Fisher (2019) argues for picking this conﬁguration similar to the arguments in section C.1.
DIB will force both distributions to become exactly the same, which would turn the decoder into a permutation matrix for categorical variables.
[Equation]
D.1 C ross - entropy loss
The cross-entropy loss features prominently in section 3.2.
We can derive the usual cross-entropy loss for our model by minimizing the Kullback-Leibler divergence between the empirical sample distribution ˆp( x , y ) and the parameterized distribution p θ ( x ) p θ (ˆ y | x ).
For discriminative models, we are only interested in p
θ (ˆ y | x ), and can simply set p θ ( x )
=
ˆ p( x ):
[Equation]
(cid:123)(cid:122) (cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)(cid:32)
[Equation]
In section 3.2, we introduce the shorthand H θ
[Y | X] for H(ˆp( y | x ) || p
θ ( ˆY = y | x )) and refer to it as
Prediction Cross-Entropy.
D.2 U pper bounds & training error minimization
To motivate that H[Y | Z] (or H θ
[Y | Z]) can be used as main loss term, we show that it can bound the (training) error probability since accuracy is often the true objective when machine learning models are deployed on real-world problems 10 .
Proposition.
The Decoder Cross-Entropy provides an upper bound on the Decoder Uncertainty:
[Equation]
and further bounds the training error:
[Equation]
Likewise, for the Prediction Cross-Entropy H θ
[Y | X] and the Label Uncertainty H[Y | X] .
Proof.
The upper bounds for Decoder Uncertainty H[Y | Z] and Label Uncertainty H[Y | X] follow from the non-negativity of the Kullback-Leibler divergence, for example:
[Equation]
The derivation for the training error probability is as follows:
[Equation]
We can then apply Jensen’s inequality using convex h ( x ) = − ln x :
[Equation]
[Equation]
For small H θ [Y | Z], we note that one can use the approximation e x ≈ 1 + x to obtain:
[Equation]
Finally, we split the Decoder Cross-Entropy into the Decoder Uncertainty and a Kullback-Leibler divergence:
[Equation]
If we upper-bound D KL (p( y | z ) || p θ ( ˆY = y | z )), minimizing the Decoder Uncertainty H[Y | Z] becomes a sensible minimization objective as it reduces the probability of misclassiﬁcation.
We can similarly show that the training error is bounded by the Prediction Cross-Entropy H
θ
[Y | X].
(cid:3)
In the next section, we examine categorical Z for which optimal decoders can be constructed and
D KL (p( y | z ) ||
p θ ( ˆY = y | z )) becomes zero.
E C ategorical Z
For categorical Z, p( y | z ) can be computed exactly for a given encoder p θ ( z | x ) by using the empirical data distribution, which, in turn, allows us to compute H[Y | Z] 11 .
This is similar to computing a confusion matrix between Y and Z but using information content instead of probabilities.
Moreover, if we set p θ (ˆ y | z ) :
=
p(Y = ˆ y | z ) to have an optimal decoder, we obtain equality in equation (9), and obtain H θ [Y | X] ≤ H θ
[Y | Z] = H[Y | Z] .
If the encoder were also deterministic, we would obtain H θ
[Y | X] =
H θ
[Y | Z] = H[Y | Z] .
We can minimize H[Y | Z] directly using gradient descent.
dd θ H[Y | Z] only depends on p( y | z ) and dd θ p θ ( z | x ):
[Equation]
Proof.
d
[Equation]
And now we show that E
[Equation]
Splitting the expectation and reordering of E p( x , y , z ) h (cid:0) p( y | z ) (cid:1) dd θ (cid:2) ln p θ ( z | x ) (cid:3) , we obtain the result.
(cid:3)
The same holds for Reverse Decoder Uncertainty H[Z | Y] and for the other quantities as can be veriﬁed easily.
If we minimize H[Y | Z] directly, we can compute p( y | z ) after every training epoch and ﬁx p θ (ˆ y | z ) :
=
p(Y = ˆ y | z ) to create the discriminative model p θ (ˆ y | x ).
This is a di ﬀ erent perspective on the self-consistent equations from Tishby et al.
; Gondek and Hofmann (2003).
x , y )p θ ( z | x ) (cid:80) x
ˆp( x )p θ ( z | x ) .
N a t s / P r o b a b ili t y
Permutation MNIST
Objective min H
[ Y | Z ] min H θ [ Y | Z ] min H
θ [ Y | X ]
Metric
H [ Y | Z ]
H θ
[ Y | Z ]
H θ
[ Y | X ]   60 Epoch
CIFAR
E.1 E mpirical evaluation of D KL (p( y | z ) || p θ ( ˆY = y | z )) during training
We examine the size of the gap between Decoder Uncertainty and Decoder Cross-Entropy and the training behavior of the two cross-entropies with categorical latent Z on Permutation MNIST and
CIFAR-10.
For Permutation MNIST (Goodfellow et al., 2013), we use the common fully-connected
ReLU 784 − 1024 − 1024 − C encoder architecture, with C = 100 categories for Z.
For CIFAR- (Krizhevsky et al., 2009), we use a standard ResNet18 model with C many output classes as encoder (He et al., 2016a).
See section G for more details about the hyperparameters.
Even though a C ×matrix and a SoftMax would su ﬃ ce to describe the decoder matrix p θ (ˆ y | z ) 12 , we have found that over-parameterization using a separate DNN beneﬁts optimization a lot.
Thus, to parameterize the decoder matrix, we use fully-connected ReLUs C − 1024 − 1024 − 10 with a ﬁnal SoftMax layer.
We compute it once per batch during training and back-propagate into it.
Figure E.1 shows the three metrics as we train with each of them in turn.
Our results do not achieve
SOTA accuracy on the test set—we impose a harder optimization problem as Z is categorical, and we are essentially solving a hard-clustering problem ﬁrst and then map these clusters to ˆY. Results are provided for the training set in order to compare with the optimal decoder.
As predicted, the Decoder Cross-Entropy upper-bounds both the Decoder Uncertainty H[Y | Z] and the Prediction Cross-Entropy in all cases.
Likewise, the gap between H θ
[Y | Z] and H[Y | Z] is tiny when we minimize H
θ
[Y | Z].
On the other hand, minimizing Prediction Cross-Entropy can lead to large gaps between H
θ [Y | Z] and H[Y | Z], as can be seen for CIFAR-10.
Very interestingly, on MNIST Decoder Cross-Entropy provides a better training objective whereas on CIFAR-10 Prediction Cross-Entropy trains lower.
Decoder Uncertainty does not train very well on CIFAR-10, and Prediction Cross-Entropy does not train well on Permutation MNIST at all.
We suspect DNN architectures in the literature have evolved to train well with cross-entropies, but we are surprised by the heterogeneity of the results for the two datasets and models.
F S urrogates for regularization terms F.1 D ifferential entropies
Proposition.
After adding zero-entropy noise, the inequality I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z] also holds in the continuous case, and we can minimize I[X; Z | Y] in the IB objective by minimizing
H[Z | Y] or H[Z] , similarly to the DIB objective.
We present a formal proof in section F.1.
Theorem 2.
For random variables A, B, we have
[Equation]
Proof.
See Bercher and Vignat (2002, section 2.2).
[Equation]
Proposition 1.
Let Y, Z and X be random variables satisfying the independence property Z ⊥ Y | X, and F a possibly stochastic function such that Z = F ( X ) + (cid:15) , with independent noise (cid:15) satisfying (cid:15) ⊥ F ( X ) , (cid:15) ⊥ Y and H ( (cid:15) )
= 0 .
Then the following holds whenever I[Y; Z] is well-deﬁned.
[Equation]
Proof.
First, we note that H[Z | X]
=
H[ F ( X ) + (cid:15) | X ] ≥ H
[ (cid:15) | X ]
= H[ (cid:15) ] with theorem 2, as (cid:15) is independent of X , and thus H[Z | X] ≥ 0.
We have H[Z | X]
=
H[ Z | X , Y ] by the conditional independence assumption, and by the non-negativity of mutual information, I[Y; Z] ≥ 0.
Then:
[Equation]
X
[Equation]
The probabilistic model from section 2 fulﬁlls the conditions exactly, and the two statements motivate our proposition.
It is important to note that while zero-entropy noise is necessary for preserving inequalities like
I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z] in the continuous case, any Gaussian noise will su ﬃ ce for optimization purposes: we optimize via pushing down an upper bound, and constant o ﬀ sets will not a ﬀ ect this.
Thus, if we had H[ (cid:15) ] (cid:44) 0, even though I[X; Z | Y] + H[Z | X] (cid:54)≤ H[Z | Y], we could instead use
I[X; Z | Y]
+ H[Z | X] − H[ (cid:15) ] ≤ H[Z | Y] − H[ (cid:15) ] as upper bound to minimize.
The gradients remain the same.
This also points to the nature of di ﬀ erential entropies as lacking a proper point of origin by themselves.
We choose one by ﬁxing H[ (cid:15) ].
Just like other literature usually only considers mutual information as meaningful, we consider H[Z | X] − H
[ (cid:15) ] as more meaningful than H[Z | X].
However, we can side-step this discussion conveniently by picking a canonical noise as point of origin in the form of zero-entropy noise H[ (cid:15) ]
= 0.
F.2 U pper bounds
We derive this result as follows:
[Equation]
Theorem 3.
Given a k-dimensional random variable X =
( X i ) ki
= 1 with Var[ X i ] > 0 for all i,
[Equation] [Equation]
i ]
[Equation]
Proof.
First, the multivariate normal distribution with same covariance is the maximum entropy distribution for that covariance, and thus H[ X ] ≤ ln det(2 π e Cov
[ X ]), when we substitute the di ﬀ erential entropy for a multivariate normal distribution with covariance Cov[ X ].
Let Σ 0 :
=
Cov[ X ] be the covariance matrix and Σ 1 : =
diag(Var[ X i ])
i the matrix that only contains the diagonal.
Because we add independent noise, Var[ X i ] > 0 and thus Σ − 11 exists.
It is clear that tr( Σ − 11 Σ 0 ) =
k .
Then, we can use the KL-Divergence between two multivariate normal distributions N 0 , N 1 with same mean 0 and
[Equation]
We substitute the deﬁnitions of Σ 0 and Σ 1 , and obtain the second inequality after adding k ln(2 π e ) on both sides.
[Equation]
Theorem 4.
Given a k-dimensional real-valued random variable X =
( X i )
[Equation]
= 1 ∈ R k , we can bound the entropy by the mean squared norm of the latent:
[Equation]
with C (cid:48) :
= ke 2 C / k 2 π e .
Proof.
We begin with the previous bound:
[Equation]
where we use the AM-GM inequality:
[Equation] [Equation]
] and the monotony of the logarithm with: (cid:88)
[Equation]
Bounding using E (cid:107) X (cid:107) 2 ≤ C (cid:48) , we obtain
[Equation]
[Equation]
and solving for C (cid:48) yields the statement.
[Equation]
This theorem provides justiﬁcation for the use of ln E (cid:107) Z (cid:107) 2 as a regularizer, but does not justify the use of E (cid:107) Z (cid:107) 2 directly.
Here, we give two motivations.
We ﬁrst observe that ln x ≤ x − 1 due to ln’s strict convexity and ln 1 = 0, and thus:
[Equation]
We can also take a step back and remind ourselves that IB objectives are actually Lagrangians, and β in min I[X; Z] − β I[Y; Z] is introduced as Lagrangian multiplier for the constrained objective:
[Equation]
We can similarly write our canonical DIB objective H[Y | Z] + β (cid:48)(cid:48) H[Z] as constrained objective
[Equation]
Y |
[Equation]
s.
[Equation] [Equation]
[Z] ≤ C , and use above statement to ﬁnd the approximate form
[Equation]
Reintroducing a Lagrangian multiplier recovers our reguralized E (cid:107) Z (cid:107) 2 objective:
[Equation]
F.3 “D eep V ariational I nformation B ottleneck ” and E (cid:107) Z (cid:107)
Alemi et al.
(2016) model p θ ( z | x ) explicitly as multivariate Gaussian with parameterized mean and parameterized diagonal covariance in their encoder and regularize it to become close to N (0 , I k ) by minimizing the Kullback-Leibler divergence D KL
[Equation]
p
[Equation]
) alongside the cross-entropy:
[Equation]
as detailed in section C.2.
We can expand the regularization term to
[Equation]
After dropping constant terms (as they don’t matter for optimization purposes), we obtain
[Equation]
When we inject zero-entropy noise into the latent Z, we have H[Z | X] ≥ 0 and thus E (cid:107) Z (cid:107) 2 −
H[Z | X] ≤ E (cid:107) Z (cid:107) 2 .
Thus, the E (cid:107) Z (cid:107) 2 regularizer also upper-bounds DVIB’s regularizer in this case.
In particular, we have equality when we use a deterministic encoder.
When we inject zero-entropy noise and use a deterministic encoder, we are optimizing the DVIB objective function when we use the E (cid:107) Z (cid:107) 2 regularizer.
In other words, in this particular case, we could reinterpret “min H θ [Y | Z] + γ E (cid:107) Z (cid:107) 2 ” as optimizing the DVIB objective from Alemi et al.
(2016) if they were using a constant covariance instead of parameterizing it in their encoder.
This does not hold for stochastic encoders.
We empirically compare DVIB and the surrogate objectives from section 3.3 in section G.5.
In the corresponding plot in ﬁgure G.15, we can indeed note that E (cid:107) Z (cid:107) 2 and DVIB are separated by a factor of 2 in the Lagrange multiplier.
F.4 D etailed C omparison to CEB, VCEB & DVIB
In Fisher (2019), the introduced CEB objective "min I[X; Z | Y] − γ I[Y; Z]" is rewritten to "min γ H[Y | Z] + H[Z | Y] − H[Z | X]" similar to the IB objective in proposition 1 in section 3.1.
However, these atomic quantities are not separately examined in detail.
Both Alemi et al.
(2016) and Fischer (2020) focus on the application of variational approximations to these quantities.
Using a slight abuse of notation to denote all variational approximations, we can write the VCEB objective 13 (Fischer, 2020) and the DVIB objective (Alemi et al., 2016) more concisely as
[Equation]
DVIB does not specify how to choose stochastic encoders and picks the variational marginal q( z ) to be a unit Gaussian.
We relate how this choice of marginal relates to the E (cid:107) Z (cid:107) 2 surrogate objective in section F.3.
Alemi et al.
(2016) use VAE-like encoders that output mean and standard deviation for latents that are then sampled from a multivariate Gaussian distribution with diagonal covariance in their experiments.
They run experiments on MNIST and on features extracted from the penultimate layer of pretrained models on ImageNet.
While VCEB as introduced in Fisher (2019) is agnostic to the choice of stochastic encoder, Fischer (2020) mention that stochastic encoders can be similar to encoders and decoders in VAEs (Kingma and Welling, 2013) or like in DVIB mentioned above.
Both VAEs and DVIB explicitly parameterize the distribution of the latent to sample from it before passing samples to the decoder.
Fischer and Alemi (2020) use an existing classiﬁer architecture to output means for a Gaussian distribution with unit diagonal covariance.
They further parameterize the variational approximation for the Reverse Decoder Uncertainty q( y | z ) with one Gaussian of ﬁxed variance per class and learn this reverse decoder during training as well.
Fischer and Alemi (2020) report results on CIFARand ImageNet that show good robustness against adversarial attacks without adversarial training, similar to the results in this paper.
This speciﬁc (and not motivated) instantiation of the VCEB objective in Fischer and Alemi (2020) is similar to the log Var[Z | Y] surrogate objective introduced in section 3.3 with a deterministic encoder and zero-entropy noise injection.
However, the latter uses minibatch statistics instead of learning a reverse decoder, trading variational tightness for ease of computation and optimization.
Compared to this prior literature, this paper examines the usage of implicit stochastic encoders (for example when using dropout) and presents three di ﬀ erent simple surrogate objectives together with a principled motivation for zero-entropy noise injection, which has a dual use in enforcing meaningful compression and in simplifying the estimation of information quantities.
Moreover, multi-sample approaches are examined to di ﬀ erentiate between Decoder Cross-Entropy and Prediction CrossEntropy.
In particular, implicit stochastic encoders together with zero-entropy noise and simple surrogates make it easier to use IB objectives in practice compared to using explicitly parameterized stochastic encoders and variational approaches.
F.5 A n information - theoretic approach to VAE s
While Alemi et al.
(2016) draw a general connection to β -VAEs (Higgins et al., 2016), we can use the insights from this paper to derive a simple VAE objective.
Taking the view that VAEs learn latent representations that compress input samples, we can approach them as entropy estimators.
Using
H[X] + H[Z | X] = H[X | Z] + H[Z], we obtain the ELBO
H[X] = H[X | Z] + H[Z
[Equation]
Z
[Equation]
[Equation]
| Z] +
[Equation]
Z
[Equation]
|
[Equation]
[Equation]
] +
[Equation]
We can also put eq.
(40) into words: we want to ﬁnd latent representations such that the reconstruction cross-entropy H[X | Z] and the latent entropy H[Z], which tell us about the length encoding an input sample, become minimal and approach the true entropy as average optimal encoding length of the dataset distribution.
The ﬁrst inequality (1) stems from introducing a cross-entropy approximation H
θ [X | Z] for the conditional entropy H[X | Z].
The second inequality (2) stems from injection of zero-entropy noise with a stochastic encoder.
For a deterministic encoder, we would have equality.
We also note that (1) is the DVIB objective for a VAE with β = 1, and (2) is the DIB objective for a VAE.
Finally, we can use one of the surrogates introduced in section 3.3 to upper bound H[Z].
For optimization purposes, we can substitute the L2 activation regularizer E (cid:107) Z (cid:107) 2 from proposition 4 and obtain as objective
[Equation]
It turns out that this objective is examined amongst others in the recently published Ghosh et al.
(2019) as a CV-VAE , which uses a deterministic encoder and noise injection with constant variance.
The paper derives this objective by noticing that the explicit parameterizations that are commonly used for
VAEs are cumbersome, and the actual latent distribution does often not necessarily match the induced distribution (commonly a unit Gaussian) which causes sampling to generate out-of-distribution data.
It ﬁts a separate density estimator on p( z ) after training for sampling.
The paper goes on to then examine other methods of regularization, but also provides experimental results on CV-VAE, which are in line with VAEs and WAEs.
The derivation and motivation in the paper is di ﬀ erent and makes no use of information-theoretic principles.
Our short derivation above shows the power of using the insights from section 3.2 and 3.3 for applications outside of supervised learning.
F.6 S oft clustering by entropy M inimization with G aussian noise
Consider the problem of minimizing H[Z | Y] and H[Y | Z], in the setting where Z = f θ ( X ) + (cid:15) ∼
N (0 , σ 2 )—i.e. the embedding Z is obtained by adding Gaussian noise to a deterministic function of the input.
Let the training set be enumerated x 1 , . . .
, x n , with µ i =
f θ ( x i ).
Then the distribution of Z is given by a mixture of Gaussians with the following density, where d ( x , µ i ) :
= (cid:107) x − µ i (cid:107) /σ 2 .
[Equation]
Assuming that each x i has a deterministic label
y i , we then ﬁnd that the conditional distributions p( y | z ) and p( z | y ) are given as follows:
[Equation]
where n y is the number of x i with class y i
= y .
Thus, the conditional Z | Y can be interpreted as a mixture of Gaussians and Y | Z as a Softmax marginal with respect to the distances between Z and the mean embeddings.
We observe that H[Z | Y] is lower-bounded by the entropy of the random noise added to the embeddings:
[Equation]
with equality when the distribution of f θ ( X ) | Y is deterministic – that is f θ is constant for each equivalence class.
Further, the entropy H[Y | Z] is minimized when H[Z] is large compared to H[Z | Y] as we have the decomposition
[Equation]
In particular, when f θ is constant over equivalence classes of the input, then H[Y | Z] is minimized when the entropy H [ f θ ( X ) + (cid:15) ] is large – i.e. the values of f θ ( x i ) for each equivalence class are distant from each other and there is minimal overlap between the clusters.
Therefore, the optima of the information bottleneck objective under Gaussian noise share similar properties to the optima of geometric clustering of the inputs according to their output class.
To gain a better understanding of local optimization behavior, we decompose the objective terms as follows:
[Equation]
To examine how the mean embedding µ k of a single datapoint x k a ﬀ ects this entropy term, we look at the derivative of this expression with respect to µ k = f θ ( x k ).
We obtain:
[Equation]
While these derivatives do not have a simple analytic form, we can use known properties of the KL divergence to develop an intuition on how the gradient will behave.
We observe that in the left-hand sum µ k only a ﬀ ects the distribution of Z | Y
(that is we are di ﬀ erentiating a sum of terms that look like a reverse KL), whereas it has greater inﬂuence on p( z | x k ) in the right-hand term, and so its gradient will more closely resemble that of the forward KL.
The left-hand-side term will therefore push µ k towards the centroid of the means of inputs mapping to y , whereas the right-hand side term is mode-seeking.
F.7 A note on differential and discrete entropies
The mutual information between two random variables can be deﬁned in terms of the KL divergence between the product of their marginals and their joint distribution.
However, the KL divergence is only well-deﬁned when the Radon-Nikodym derivative of the density of the joint with respect to the product exists.
Mixing continuous and discrete distributions—and thus di ﬀ erential and continuous entropies—can violate this requirement, and so lead to negative values of the “mutual information”.
This is particularly worrying in the setting of training stochastic neural networks, as we often assume that an stochastic embedding is generated as a deterministic transformation of an input from a ﬁnite dataset to which a continuous perturbation is added.
We provide an examples where naive computation without ensuring that the product and joint distributions of the two random variables have a well-deﬁned Radon-Nikodym derivative yields negative mutual information.
Let X ∼ U
([0 , 0 . 1]), Z = X + R with R ∼ U ( { 0 , 1 } ).
Then
[Equation]
Generally, given X as above and an invertible function f such that Z = f ( X ), I[ X ; Z ]
=
H
[ X ] and can thus be negative.
In a way, these cases can be reduced to (degenerate) expressions of the form
I[ X ; X ] =
H
[ X ].
We can avoid these cases by adding independent continuous noise.
These examples show that not adding noise can lead to unexpected results.
While they still yield ﬁnite quantities that bear a relation to the entropies of the random variables, they violate some of the core assumptions we have such that mutual information is always positive.
G E xperiment details G.1 DNN architectures and hyperparameters
For our experiments, we use PyTorch (Paszke et al., 2019) and the Adam optimizer (Kingma and Ba, 2014).
In general, we use an initial learning rate of 0 .
5 × 10 − 3 and multiply the learning rate by √ 0 .whenever
the loss plateaus for more than 10 epochs for CIFAR-10.
For MNIST and Permutation
MNIST, we use an initial learning rate of 10 − 4 and multiply the learning rate by 0.8 whenever the loss plateaus for more than 3 epochs.
Sadly, we deviate from this in the following experiments: when optimizing the decoder uncertainty for categorical Z for CIFAR-10, we used 5 epochs patience for the decoder uncertainty objective and a initial learning rate of 10 − 4 .
We do not expect this di ﬀ erence to a ﬀ ect the qualitative results mentioned in section E when comparing to other objectives.
We also only used 5 epochs patience when comparing the two cross-entropies on CIFAR-10 in section 3.2.
As this was used for both sets of experiments, it does not matter.
We train the experiments for creating the information plane plots for 150 epochs.
The toy experiment (ﬁgure 3) is trained for 20 epochs.
All other experiments train for 100 epochs.
We use a batchsize of 128 for most experiments.
We use a batchsize of 32 for comparing the crossentropies for CIFAR-10 (where we take 8 dropout samples each), and a batchsize of 16 for MNIST (where we take 64 dropout samples each).
For MNIST, we use a standard dropout CNN, following https://github.com/pytorch/ examples/blob/master/mnist/main.py .
For Permutation MNIST, we use a fully-connected model (for experiments with categorical Z in section E): 784 × 1024 × 1024 × C .
For CIFAR-10, we use a regular deterministic ResNet18 model (He et al., 2016a) for the experiments in section E.
(As the model outputs a categorical distribution it becomes stochastic through that and we don’t need stochasticity in the weights.)
For the other experiments as well as the Imagenette experiments, we use a ResNet18v2 (He et al., 2016b).
When we need a stochastic model for CIFAR-10 (for continuous
Z), we add DropConnect (Wan et al., 2013b) with rate 0.1 to all but the ﬁrst convolutional layers and dropout with rate 0.1 before the ﬁnal fully-connected layer.
Because of memory issues, we reuse the dropout masks within one batch.
The model trains to 94% accuracy on CIFAR-10.
For CIFAR-10, we always remove the maximum pooling layer and change the ﬁrst convolutional layer to have kernel size 3 with stride 1 and padding 1.
We also use dataset augmentation during training, but not during evaluation on the training set and test set for purposes of computing metrics.
We crop randomly after the padding the training images by 4 pixels in every direction and randomly ﬂip images horizontally.
We generally sample 30 values of γ for the information plane plots from the speciﬁed ranges, using a log scale.
For the ablation studies mentioned below, we sample 10 values of γ each.
We always sample γ = 0 separately and run a trial with it.
Baselines were tuned by hand (without regularization) using grad-student descent and small grid searches.
G.2 C luster setup & used resources
We make use of a local SLURM cluster (Jette et al., 2002).
We run our experiments on GPUs (Geforce
RTX 2080 Ti).
We estimate reproducing all results would take 94 GPU days.
G.3 C omparison of the surrogate objectives
[Figure]
T e s t
S e t
2 shows a stronger initial regularization e ﬀ ect, but is di ﬃ cult to compare quantitatively as its hyperparameter does not map to an equivalent β , unlike regularizing using entropy estimates.
Overall, we ﬁnd log Var[Z] to provide stable training trajectories (and expected visualizations) while also having a more meaningful hyperparameter than E (cid:107) Z (cid:107) 2 , though E (cid:107) Z (cid:107)is trivial to implement and communicate 14 .
log Var[Z | Y] performs worse which we hypothesize is due to the increased variance (given equal batch sizes) from conditioning on Y. It further does not minimize the Preserved Information I[X; Z] as strongly as the other regularizers.
G.3.1 M easurement of information quantities
Measuring information quantities can be challenging.
As mentionend in the introduction, there are many complex ways of measuring entropies and mutual information terms.
We can side-step the issue by making use of the bounds we have established and the zero-entropy noise we are injecting, and design experiments around that.
First, to estimate the Preserved Information I[X; Z], we note that when we use a deterministic model as encoder and only inject zero-entropy noise, we have H[Z | X] = 0 and I[X; Z] =
I[X; Z] + H[Z | X] =
H[Z].
We use the entropy estimator from Kraskov et al.
(2004, equation (20)) to estimate the Encoding
Entropy H[Z] and thus I[X; Z].
[Figure]
To estimate the Residual Information I[X; Y | Z], we similarly note that I[X; Y | Z] =
I[X; Y | Z] +
H[Y | X] = H[Y | Z].
Instead of estimating the entropy using Kraskov et al.
(2004), we can use the Decoder Cross-Entropy H θ
[Y | Z] which provides a tighter bound as long as we also minimize
H θ
[Y | Z] as part of the training objective.
When we use stochastic models as encoder, we cannot easily compute I[X; Z] anymore.
In the ablation study in the next section, we thus change the X axis accordingly.
Similarly, when we look at the trajectories on the test set instead of the training set, for example in ﬁgure G.4, we change the Y axis to signify the Decoder Uncertainty H θ
[Y | Z].
It is still an upper-bound, but we do not minimize it directly anymore.
For the plots in ﬁgure 6, we retrained the decoder on the test set to obtain a tighter bound on H[Y | Z] (while keeping the encoder ﬁxed).
We then sampled the latent using the test set to estimate the trajectories.
We only did this for the CIFAR-10 model without dropout.
For our ablations, we did not retrain the decoder and thus only present plots on the test and training set, respectively.
At this point, it is important to recall that the Decoder Uncertainty is also the negative log-likelihood (when training with a single dropout sample), which provides a di ﬀ erent perspective on the plots.
It makes it clear that we can see how much a model overﬁts by comparing the best and ﬁnal epochs of a trajectory in the plot (marked by a circle and a square, respectively).
G.3.2 A blation study
We perform an ablation study to determine whether injecting noise is necessary.
Furthermore, we investigate the more interesting case of using a stochastic model as encoder, and if we can use a stochastic model without injecting zero-entropy noise.
We also investigate whether log Var[Z | Y] performs better when we increase batchsize as we hypothesized that a batchsize of 128 does not su ﬃ ce as it leaves only ≈ 13 samples per class to approximate
H[Z | Y]).
Figure G.3 shows a larger version of ﬁgure 6 for all three regularizers and also training trajectories on the test set.
As described in the previous section, this allows us to validate that the regularizers prevent overﬁtting on the training set: with increasing γ , the model overﬁts less.
Figure G.6 and ﬁgure G.5 shows that injecting noise is necessary independently of whether we use dropout or not.
Regularizing with E (cid:107) Z (cid:107) 2 still has a very weak e ﬀ ect.
We hypothesize that, similar to the toy experiment depicted in ﬁgure 3, ﬂoating-point precision issues might provide a natural noise source eventually.
This would change the e ﬀ ectiveness of γ and might require much higher values to observe similar regularization e ﬀ ects as when we do inject zero-entropy noise.
Figure G.4 shows trajectories for a stochastic encoder (as described above with DropConnect / dropout rate 0.1).
It overﬁts less than a deterministic one.
Figure G.7 shows the e ﬀ ects of using higher dropout rates (using DropConnect / dropout rates of 0.3 / 0.5).
It overﬁts less than model with DropConnect / dropout rates of 0.1 / 0.1.
The plots in ﬁgure G.10 show the e ﬀ ects of di ﬀ erent γ with di ﬀ erent regularizers more clearly.
On both training and test set, one can clearly see the e ﬀ ects of regularization.
Overall, log Var[Z | Y] performs worse as a regularizer.
In ﬁgure G.8, we compare the e ﬀ ect of doubling batchsize.
Indeed, log Var[Z | Y] performs better with higher batchsize and looks closer to log Var[Z].
G.3.3 C omparison between D ecoder C ross -E ntropy and P rediction C ross -E ntropy
When training deterministic models or dropout models with a single sample (as one usually does), the estimators for both the Decoder Cross-Entropy H θ [Y | Z] and the Prediction Cross-Entropy
H θ
[Y | X] coincide.
In section 3.2, we discuss the di ﬀ erences from a theoretical perspective.
Here, we empirically evaluate the di ﬀ erence between optimizing the estimators for each of the two crossentropy losses, for which we will draw multiple dropout samples during training and inference.
We examine models with continuous Z on MNIST and CIFAR-10 (Lecun et al., 1998; Krizhevsky et al., 2009).
Speciﬁcally, we use a standard dropout CNN as an encoder for MNIST, and a modiﬁed
ResNet18 to which we add DropConnect in each layer for CIFAR-10.
We use K = 100 dimensions for the continuous latent Z in the last fully-connected layer, and use a linear decoder to obtain the ﬁnal 10-dimensional output of class logits.
For MNIST, we compute the cross-entropies usingdropout samples; for CIFAR-10, we use 8.
For the purpose of this examination of training behavior, it is not necessary to achieve SOTA accuracy: our models obtain 99.2% accuracy on MNIST and 93.6% on CIFAR-10.
Figure G.11 shows the training error probability as well as the value of each cross-entropy loss for models trained either with the Decoder Cross-Entropy or the Prediction Cross-Entropy.
The Decoder
Cross-Entropy H θ
[Y | Z] outperforms Prediction Cross-Entropy H
θ
[Y | X] as a training objective: the training error probability and both cross-entropies are lower when minimizing H
θ
[Y | Z] compared to minimizing H θ
[Y | X].
We compare only the training, rather than the test, losses of the models to isolate the e ﬀ ect of each loss term on training performance; we leave the prevention of overﬁtting to the regularization terms considered later.
Recently, Dusenberry et al.
(2020) also observed empirically that the Decoder Cross-Entropy H θ
[Y | Z] as an objective is both easier to optimize and provides better generalization performance.
G.4 D ifferential entropies and noise
We demonstrate the importance of adding noise to continuous latents by constructing a pathological sequence of parameters which attain monotonically improving and unbounded regularized objective values (H[Z]) while all computing the same function .
We use MNIST with a standard dropout CNN as encoder, with K = 128 continuous dimensions in Z, and a K × 10 linear layer as decoder.
After every training epoch, we decrease the entropy of the latent by normalizing and then scaling the latent to bound the entropy.
We multiply the weights of the decoder to not change the overall function.
As can be seen in ﬁgure 3, without noise, entropy can decrease freely during training without change in error rate until it is a ﬀ ected by ﬂoating-point issues; while when adding zero-entropy noise, the error rate starts increasing gradually and meaningfully as the entropy starts to approach zero.
We conclude that entropy regularization is meaningful only when noise is added to the latent.
G.5 C omparison between DVIB and surrogate objectives on P ermutation -MNIST
Comparing DVIB and our surrogate objectives is not straightforward because DVIB uses a VAE-like model that explicitly parameterize mean and standard deviation of the latent whereas the stochastic models we focus on in section 3.2 and beyond are implicit by using dropout.
For this comparison, we use the same architecture and optimization strategy for DVIB as described in Alemi et al.
(2016): the encoder is a ReLU-MLP of the form 796 − 1024 − 1024 − 2 K with K =latent dimensions that outputs mean and standard deviation explicitly and separately.
For the standard deviation, we use a softplus transform with a bias of − 5.
We use Polyak averaging with a decay constant of 0 .
999 (Polyak and Juditsky, 1992).
We train the model for 200 epochs with Adam with learning rate 10 − 4 , β 1 = 0 .
5 , β 2 = − .
999 (Kingma and Ba, 2014) and decay the learning rate by 0.
every 2 epochs.
The marginal is ﬁxed to a unit Gaussian around the origin.
We use a softmax layer as decoder.
We use 12 latent samples during training and test time.
For our surrogate objectives, we use a similar ReLU-MLP of the form 796 − 1024 − 1024 − K with
K = 256 latent dimensions and dropout layers of rate 0.3 after the ﬁrst and second layer.
We use also 12 dropout samples during training and test time.
We train for 75 epochs with Adam and learning rate 0 .
5 × 10 − 4 .
We half the learning rate every time the loss does not decrease for 13 epochs.
We run 5 trials for each experiments.
We were not able to reproduce the baseline of an error of 1 . 13% for β = 10 − 3 from Alemi et al.
We show a comparison in ﬁgure G.15.
Our methods do reach an error of 1 .
13% overall though, so the simpler surrogate objectives perform as well good or better than DVIB.
From section F.3, we know that DVIB’s β would have to be twice the γ frm our section 3.3.
We can see this correspondence in the plot.
This also implies that DVIB’s β is not related to the IB objective’s β from section 3.
This makes sense as DVIB arbitrarily ﬁxes the marginal to be a unit Gaussian.
[Table]
a
[Table] [Table]
] γ
[Table]
E p o c h    l og V a r
[ Z | Y ]
E
[Table]
W e i g h t
D ec a y
Train
[Table]
Test Set
[Table] [Table]
[Table]
P r e s e r v e d I n f o r m a t
i
o
n I [ X ;
Z ]
IP
F i gu r e G . 3 :
W it hou
t
d r opou t bu t w it h
ze r
o - e n t r op
y
no i s e :
I n f o r m a ti on
P l an e P l o t o f t r a i n i ng t r a j ec t o r i e s f o r
R e s N e tm od e l s on
C I
FARand d i ﬀ e r e n t r e gu l a r i z e rs .
T h e t r a j ec t o r
i e s a r e c
o l o r e d by t h e i r
r e
s p ec ti v e γ ; t h e i r t r a n s p a r e n c y c h a ng e s by e po c h .
C o m p r e ss i on ( P r e s e r v e d I n f o r m a ti on ↓ ) t r a d e s - o ﬀ w it h p e rf
o r m a n ce
( R e s i du a l
I n f o r m a ti on ↓ ) .
S ee s ec ti on 4 .
T h
e c i r c l e m a r k s t
h e ﬁ n a l e po
c h
o f a t r a j ec t o r y .
T h
e s qu a r e m a r k
s t h e b e s t e po c h (
R e s i du a l
I n f o r m a ti on (cid:21) ) .
10 − 2 10 −
Decoder Cross-Entropy H θ [ Y | Z ] Residual Information I[ Y ; X | Z ] l og V a r
[ Z ] γ 10 −10 −10 −
E p o c h    l og V a r
[ Z | Y ]
E Z
Train Set   0 .
E n c o d i n g E n t r o p y
H [ Z ]
Test Set
F i gu r e
G . 4 :
W it h
d r opou t a nd w it h
ze r oe n t r opy
no i s e :
I n f o r m a ti on
P l an e P l o t o f t r a i n i ng t r a j ec t o r i e s f o r
R e s N e tm od e l s on
C I
FARand d i ﬀ e r e n t r e gu l a r i z e rs .
T h e t r a j ec t o r
i e s a r e c
o l o r e d by t h e i r
r e
s p ec ti v e γ ; t h e i r t r a n s p a r e n c y c h a ng e s by e po c h .
C o m p r e ss i on ( P r e s e r v e d I n f o r m a ti on ↓ ) t r a d e s - o ﬀ w it h p e rf
o r m a n ce
( R e s i du a l
I n f o r m a ti on ↓ ) .
S ee s ec ti on 4 .
T h
e c i r c l e m a r k s t
h e ﬁ n a l e po
c h
o f a t r a j ec t o r y .
T h
e s qu a r e m a r k
s t h e b e s t e po c h (
R e s i du a l
I n f o r m a ti on (cid:21) ) .
10 − 2 10 −
Decoder Cross-Entropy H θ [ Y | Z ] Residual Information I[ Y ; X | Z ] l og V a r
[ Z ] γ 10 −10 −10 −
E p o c h    l og V a r
[ Z | Y ]
E Z
Train Set - 40200 .
E n c o d i n g E n t r o p y
H [ Z ]
Test Set
F i gu r e G d r opou t bu t w it
hou t
ze r oe n t r
o
py
no i s e :
I n f o r m a ti on
P l an e P l o t o f t r a i n i ng t r a j ec t o r i e s f o r
R e s N e tm od e l s on
C I
FARand d i ﬀ e r e n t r e gu l a r i z e rs .
T h e t r a j ec t o r
i e s a r e c
o l o r e d by t h e i r
r e
s p ec ti v e γ ; t h e i r t r a n s p a r e n c y c h a ng e s by e po c h .
C o m p r e ss i on ( P r e s e r v e d I n f o r m a ti on ↓ ) t r a d e s - o ﬀ w it h p e rf
o r m a n ce
( R e s i du a l
I n f o r m a ti on ↓ ) .
S ee s ec ti on 4 .
T h
e c i r c l e m a r k s t
h e ﬁ n a l e po
c h
o f a t r a j ec t o r y .
T h
e s qu a r e m a r k
s t h e b e s t e po c h (
R e s i du a l
I n f o r m a ti on (cid:21) ) .
[Table]
P r e s e r
[Table]
I n f o
[Table]
i
[Table]
n I
[Table]
X
[Table]
Z ]
[Table]
F i gu r e
G . 6 :
W i t hou t
d r opou t a nd w it hou t
ze r oe n t r opy
no i s e : I n f o r m a ti on
P l an e P l o t o f t r a i n i ng t r a j ec t o r i e s f o r
R e s N e tm od e l s on
C I
FARand d i ﬀ e r e n t r e gu l a r i z e rs .
T h e t r a j ec t o r
i e s a r e c
o l o r e d by t h e i r
r e
s p ec ti v e γ ; t h e i r t r a n s p a r e n c y c h a ng e s by e po c h .
C o m p r e ss i on ( P r e s e r v e d I n f o r m a ti on ↓ ) t r a d e s - o ﬀ w it h p e rf
o r m a n ce
( R e s i du a l I n f o r m a ti on ↓ ) .
S ee s ec ti o n 4 .
T h
e c i r c l e m a r k s t
h e ﬁ n a l e po
c h
o f a t r a j ec t o r y .
T h
e s qu a r e m a r k
s t h e b e s t e po c h (
R e s i du a l
I n f o r m a ti on (cid:21) ) .
[Table]
a
[Table] [Table]
] γ
[Table] [Table]
E
[Table]
c h
[Table]
og V a r
[ Z | Y ]
[Table] [Table]
[Table]
E n c o d i n g E n t r o p y
H [ Z ]
Test Set
F i gu r e
G . 7 :
W it h
m o r e d r opou t a nd ze r oe n
t r o py
no i s e : I n f o r m a ti on
P l an e P l o t o f t r a i n i ng t r a j ec t o r i e s f o r
R e s N e tm od e l s on
C I
FARand l og
V a r[
Z | Y ] r e gu l a r i z e r w it
h ba t c h s i
z e s and 256 .
T h e t r a j e c t o r i e s a r e c
o l o r e d by t h e i r
r e
s p ec ti v e γ ; t h e i r t r a n s p a r e n c y c
h a n g e s by e po c h .
C o m p r e ss i on ( P r e s e r v e d I n f o r m a ti on ↓ ) t r a d e so ﬀ
w it
h p e rf
o r m a n ce
( R e s i du a l I n f o r m a ti on ↓ ) .
S ee s ec ti on 4 .
T h
e c i r c l e m a r k s t
h e ﬁ n a l e po
c h
o f a t r a j ec t o r y .
T h
e s qu a r e m a r k
s t h e b e s t e po c h (
R e s i du a l
I n f o r m a ti on (cid:21) ) .
A D r op
C onn ec t r a t e o f 0 .a
nd d r opou t r a t e o f 0 .
4 w e r e u s e d i n s t ea d o f 0 .
1 f o r eac h .
[Figure]
ec o d e r C r o ssE n t r o p
y H θ
[ Y |
Z ]
R e s i du a l I n f o r m a t
i o n
I [ Y ;
[Figure] [Figure]
Compression (Preserved Information ↓ ) trades-o ﬀ with performance (Residual Information ↓ ).
See section 4.
The circle marks the ﬁnal epoch of a trajectory.
The square marks the best epoch (Residual
Information (cid:21) ).
[Figure] [Figure]
Z
[Figure]
r a i n
[Figure] [Figure]
1e-
[Figure]
1e-
[Figure]
1e-
[Figure]
1eN a t s /
E rr o r
Test Set
Objective min H θ
[ Y | Z ] min H θ [ Y | X ] Metric
E ˆ p ( x,y ) p (ˆ y 6 = y )
H θ
[ Y | Z ]
H θ
[ Y | X ]
Train Set
M N I S T 1e1e-
[Figure]
e r m u t a t i o n M N I
S T
[Figure] [Figure]
o d e r
C r o ss - E n t r o p y
H θ
[ Y | Z ]
R e s i du a l
I n f o r m a t i o n
I [ Y ; X | Z ]
[Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
f o
[Table]
a
[Table]
i
[Table]
n
I [ X
[Figure]
[Table]
F i gu r e
G .
: A ve r ag e ad v e rs a r i a l r obu s t n e ss o ve r (cid:15) ∈
o f R e s N e tm od e l s t r a i n e d on
C I
FARw it h s u rr o ga t
e ob j ec ti ve s i n o r Lw e i gh t - d ec a y ( a s non -I B m e t hod ) c
o m pa r e d t o no r m a l a cc u r a cy f o r d i ﬀ e r e n t a m oun t s o f P r e s e r ve d
I n f o r m a ti on .
◦ m a r k e r s s ho w r obu s t n e ss .
× m a r k e r s s ho w t h e no r m a l acc
u
r
ac
y .
W e s ee t h a t r obu s t n e ss d e p e nd s on t h e
P r e s e r v e d I n f o r m a ti o n .
If t h e l a t e n t i s c o m p r e ss e s e d t oo m u c h , r obu s t n e ss ( a nd acc u r ac y )
a r e l o w .
If t h e l a t e n t
i s no t c o m p r e ss e d e nough , r obu s t n e ss a nd t hu s g e n e r a li za ti on s u ﬀ e r .
[Figure]
H
L a r g e
V e r s i o n o f t
h e
M i c k e y
M o u s e
I- D i ag
r a m
H ( Y |
X )
L a b e l U n c e r t
a
i n t y
H ( Z | X )
E n c o d i n g U n c e r t a i n t y
I (
Y ;
Z )
P r e s e r v e d
R e l e v a n t
I n f o r m a t i o n
I (
X ;
Y |
Z )
R e s i
d u a l
I n f o r m a t i o n
I (
X ;
Z |
Y )
R e d un d a n t
I n f o r m a t i o n
H (
Y |
Z )
D e c o d e r
U n c e r t a i n t y
H (
Z |
Y )
R e v e r s e
D e c o d e r
U n c e r t a i n t y
H (
X |
Y ,
Z )
I (
X ;
Y )
R e l e v a n t
I n f o r m a t i o n
I (
X ;
Z )
P r e s e r v e d
I n f o r m a t i o n
F i gu r e
H . 1 :
M i ckey
M ou s e Id
i ag r a m .
S ee
ﬁ gu r e 2 f o r d e t
a
il s .