I NFORMATION B OTTLENECK : E XACT A NALYSIS OF (Q UANTIZED ) N EURAL N ETWORKS
{lorenzen,igel,madsn}@di.ku.dk
A BSTRACT
The information bottleneck (IB) principle has been suggested as a way to analyze deep neural networks.
The learning dynamics are studied by inspecting the mutual information (MI) between the hidden layers and the input and output.
Notably, separate ﬁtting and compression phases during training have been reported.
This led to some controversy including claims that the observations are not reproducible and strongly dependent on the type of activation function used as well as on the way the MI is estimated.
Our study conﬁrms that different ways of binning when computing the MI lead to qualitatively different results, either supporting or refusing IB conjectures.
To resolve the controversy, we study the IB principle in settings where MI is non-trivial and can be computed exactly.
We monitor the dynamics of quantized neural networks, that is, we discretize the whole deep learning system so that no approximation is required when computing the MI.
This allows us to quantify the information ﬂow without measurement errors.
In this setting, we observed a ﬁtting phase for all layers and a compression phase for the output layer in all experiments; the compression in the hidden layers was dependent on the type of activation function.
Our study shows that the initial IB results were not artifacts of binning when computing the MI.
However, the critical claim that the compression phase may not be observed for some networks also holds true.
I NTRODUCTION
Improving our theoretical understanding of why over-parameterized deep neural networks generalize well is arguably one of main problems in current machine learning research (Poggio et al., 2020).
Tishby & Zaslavsky (2015) suggested to analyze deep neural networks based on their Information
Bottleneck (IB) concept, which is built on measurements of mutual information (MI) between the activations of hidden layers and the input and target (Tishby et al., 1999), for an overview see Geiger (2020).
Shwartz-Ziv & Tishby (2017) empirically studied the IB principle applied to neural networks and made several qualitative observations about the training process; especially, they observed a ﬁtting phase and a compression phase.
The latter information-theoretic compression is conjectured to be a reason for good generalization performance and has widely been considered in the literature (Abrol & Tanner, 2020; Balda et al., 2018; 2019; Chelombiev et al., 2019; Cheng et al., 2019; Darlow & Storkey, 2020; Elad et al., 2019;
Fang et al., 2018; Gabrié et al., 2019; Goldfeld et al., 2019;
Jónsson et al., 2020; Kirsch et al., 2020; Tang Nguyen & Choi, 2019; Noshad et al., 2019; Schiemer & Ye, 2020; Shwartz-Ziv & Alemi, 2020; Wickstrøm et al., 2019; Yu et al., 2020).
The work and conclusions by Shwartz-Ziv & Tishby (2017) received a lot of critique, with the generality of their claims being doubted; especially Saxe et al.
(2018) argued that the results by Shwartz-Ziv & Tishby do not generalize to networks using a different activation function.
Their critique was again refuted by the original authors with counter-claims about incorrect estimation of the MI, highlighting an issue with the approximation of MI in both studies.
Our goal is to verify the claims by Shwartz-Ziv & Tishby and the critique by Saxe et al.
in a setting where the MI can be computed exactly.
These studies consider neural networks as theoretical entities working with inﬁnite precision, which makes computation of the information theoretic quantities problematic (for a detailed discussion we refer to Geiger, 2020, see also Section 3).
Assuming continuous input distributions, a deterministic network using any of the standard activation functions (e.g., R E LU, T ANH ) can be shown to have inﬁnite MI (Amjad & Geiger, 2019).
If an empirical input distribution deﬁned by a data set D is considered (as it is the case in many of the previous studies), then randomly-initialized deterministic neural networks with invertible activation functions will most likely result in trivial measurements of MI in the sense that the MI is ﬁnite but always maximal, that is, equal to log |D| (Goldfeld et al., 2019; Amjad & Geiger, 2019).
In order to obtain non-trivial measurements of MI, real-valued activations are usually discretized by binning the values, throwing away information in the process.
The resulting estimated MI can be shown to be highly dependent on this binning, we refer to Geiger (2020) for a detailed discussion.
Instead of approximating the MI in this fashion, we take advantage of the fact that modern computers – and thus neural networks – are discrete in the sense that a ﬂoating point value can typically take at most 2 32 different values.
Because 32-bit precision networks may still be too precise to observe compression (i.e., information loss), we apply quantization to the neural network system to an extent that we can compute informative quantities, that is, we amplify the effect of the information loss due to the discrete computations in the neural network.
One may argue that we just moved the place where the discretization is applied.
This is true, but leads to a fundamental difference: previous studies applying the discretization post-hoc rely on the in general false assumption that the binned MI approximates the continuous MI well – and thus introduce measurement errors, which may occlude certain phenomena and/or lead to artifactual observations.
In contrast, our computations reﬂect the true information ﬂow in a network during training.
Our study conﬁrms that estimation of MI by binning may lead to strong artifacts in IB analyses and shows that:
• Both ﬁtting and compression phases occur in the output S OFT M AX layer.
• For the hidden layers, the ﬁtting phase occurs for both T ANH and R E LU activations.
• When using T ANH in the hidden layers, compression is only observed in the last hidden layer.
• When using R E LU, we did not observe compression in the hidden layers.
• Even when applying low precision quantization, more complex networks with many neurons in each layer are observed to be too expressive to exhibit compression, as no information is lost.
• Our setting excludes that the MI approximation is the reason for these different IB dynamics.
The next section introduces the IB concept with a focus on its application to neural networks including the critique and controversy as well as related work.
Section 3 discusses issues relating to the estimation of MI, and the idea behind our contribution.
Section 4 presents our experiments, results and discussion before we conclude in Section 5.
T HE I NFORMATION B OTTLENECK
Preliminaries.
Given a continuous random variable (r.v.)
X with density function p ( x ) and support X , the continuous entropy H ( X ) of X is a measure of the uncertainty associated with X and is given by H ( X )
= − (cid:82) X p ( x ) log p ( x ) d x .
Given two r.v.s X and Y with density functions p ( x ) and q ( y ) and supports X and Y , the mutual information I ( X ; Y ) of X and Y is a measure of the mutual “knowledge” between the two variables.
The symmetric I ( X ; Y ) is given by
I ( X ; Y ) =
(cid:82) Y (cid:82) X p ( x, y ) log p ( x,y ) p ( x ) p ( y )
d x d y .
In many cases it is impossible to compute the continuous entropy and MI for continuous r.v.s exactly, due to limited samples or computational limits, or because it may not be ﬁnite (Geiger, 2020).
Instead, we often estimate the quantities by their discrete counterparts.
When X is a discrete r.v., we consider the Shannon entropy
H ( X ) = − (cid:80) P ( x ) log P ( x ) .
Correspondingly, the mutual information I ( X ; Y ) of two discrete r.v.s X , Y , is given by I ( X ; Y ) =
(cid:80) x,y P ( x, y ) log P ( x,y ) P ( x ) P ( y ) .
We have the following useful identity for both the continuous and discrete MI:
[Equation]
where H ( X | Y ) is the conditional entropy of X given Y .
IB Deﬁnition.
The IB method was proposed by Tishby et al.
It is an information theoretic framework for extracting relevant components of an input r.v.
X with respect to an output r.v. Y .
These relevant components are found by “squeezing” the information from X through a bottleneck , in the form of an r.v. T .
In other words, T is a compression of X .
The idea generalizes rate distortion theory , in which we wish to compress X , obtaining T , such that I ( X ; T ) is maximized subject to a constraint on the expected distortion d ( x, t ) wrt.
the joint distribution p ( x, t ) (Tishby et al., 1999).
In the IB framework, the distortion measure d is replaced by the negative loss in MI between T and the output Y , I ( T ; Y ) .
Both IB and rate distortion are lossy compression schemes.
The data processing inequality (DPI) I ( Y ; X ) ≥
I ( Y ; T ) holds for the IB; that is, the bottleneck r.v. cannot contain more information about the label than the input.
One drawback of the information bottleneck method is the dependence on the joint distribution, p ( x, y ) , which is generally not known.
Shamir et al.
(2010) addressed this issue and showed that the
MI, the main ingredient in the method, can be estimated reliably with fewer samples than required for estimating the true joint distribution.
As common in the IB literature, whenever we discuss the
MI computed on a ﬁnite data set D , we assume that p ( x, y ) corresponds to the empirical distribution deﬁned by D , which is true for the experiments in Section 4.1.
In practice, the assumption has to be relaxed to the data being drawn i.i.d.
from p ( x, y ) .
However, any uncertainty resulting from the ﬁnite sample estimation in the latter case is not considered in our discussions.
IB
In Deep Learning.
Tishby & Zaslavsky (2015) applied the IB concept to neural networks.
They view the layers of a deep neural network (DNN) as consecutive compressions of the input.
They consider the Markov chain
[Equation]
where T i denotes the i ’th hidden layer of the L -layer network and T L
= ˆ Y denotes the output of the network.
Again, the bottleneck must satisfy the DPI:
[Equation]
Estimating the MI of continuous variables is difﬁcult (Alemi et al., 2017), as evident from the many different methods proposed (Kraskov et al., 2004; Kolchinsky & Tracey, 2017; Noshad et al., 2019).
In the discrete case, I ( X ; T ) and I ( T ; Y ) can be computed as
[Equation]
following from (1) and using in (4) the assumption that T is a deterministic function of X .
However, for deterministic neural networks the continuous entropies may not be ﬁnite (Goldfeld et al., 2019;
Saxe et al., 2018; Amjad & Geiger, 2019).
Shwartz-Ziv & Tishby (2017) estimate the MI via (4) and (5) by discretizing T and then computing the discrete entropy.
They trained a network (shown in Figure 1a) on a balanced synthetic data set consisting of 12-bit binary inputs and binary labels.
The network was trained for a ﬁxed number of epochs while training and test error were observed.
For every epoch and every layer T , The discretization is done by binning of T :
Given upper and lower bounds b u , b l , and m ∈ N , we let
B : R → [ m ] denote the binning operation, that maps x ∈ [ b l , b u ] to the index of the corresponding bin from the set of m uniformly distributed bins in [ b l , b u ] .
Overloading the notation, we apply
B directly to a vector in R d in order to obtain the resulting vector in [ m ] d of bin indices.
Using discretized T (cid:48)
=
B ( T ) , I ( X ; T (cid:48) ) and I ( T (cid:48) ; Y ) are then computed directly by (4) and (5), using estimates of P ( T (cid:48) ) , P ( Y ) and P ( T (cid:48) | Y ) over all samples D of X .
Shwartz-Ziv & Tishby used the T ANH activation function for the hidden layers, with b l = − 1 , b u =( b l = 0 for the output S OFT M AX layer) and m = 30 bins.
The estimated I ( X ; T ) and I ( T ; Y ) are plotted in the information plane , providing a visual representation of the information ﬂow in
12 X 10 T7 T5 T4 T3 T
ˆ Y = T
Input
Hidden
Output (a)     10 0.
I ( X,T i )
I ( T i , Y )
E p o c h s
F i tt i n g
C o m p r e s s i o n
F i tt i n g
C o m p r e ss i o n ˆ Y =
T
T
T(b) the network during training (see example in Figure 1b). Based on the obtained results, ShwartzZiv & Tishby (2017) make several observations; one notable observation is the occurrence of two phases: an empirical risk minimization phase and a compression phase.
The ﬁrst phase, also referred to as ﬁtting phase , is characterized by increasing I ( T ; Y ) related to a decreasing loss.
The subsequent compression phase is characterized by decreasing I ( X ; T ) , and it has been argued that this compression leads to better generalization.
Critique and Controversy.
The work by Tishby & Zaslavsky (2015) and Shwartz-Ziv & Tishby (2017) has jump-started an increasing interest in the IB method for deep learning, with several papers investigating or extending their contributions, see the review by Geiger (2020).
However, as mentioned, their work also received criticism.
In particular, the compression phase as a general phenomenon has been called into question.
Saxe et al.
(2018) published a paper refuting several of the claims made by Shwartz-Ziv & Tishby (2017).
They based their criticism on a replication of the experiment done by Shwartz-Ziv & Tishby (2017) where they replaced the bounded T ANH activation function with the unbounded R E LU activation function.
When discretizing, they used the maximum activation observed across all epochs for the upper binning bound b u .
The article claims that the two phases observed by Shwartz-Ziv & Tishby occurred because the activations computed using T ANH saturate close to the boundaries − 1 and 1 .
The claim is supported by experiments using R E LU activations and m = 100 bins, in which the two phases are not observed.
The critique paper by Saxe et al.
was published at ICLR 2018, but started a discussion already during the review process the previous year, when Shwartz-Ziv & Tishby defended their paper in the online discussion forum OpenReview.net 2 , posting a response titled “Data falsifying the claims of this ICLR submission all together”
(Saxe et al., 2017).
The response speciﬁcally states that “The authors don’t know how to estimate mutual information correctly”, referring to Saxe et al.
(2018), and goes on to provide an example with a network using R E LU activations, which does indeed exhibit the two phases.
In response, Saxe et al. performed further experiments using different estimators for MI: a state-of-the-art non-parametric KDE approach (Kolchinsky & Tracey, 2017) and a k -NN based estimator (Kraskov et al., 2004).
The authors still did not observe the two phases claimed by
Shwartz-Ziv & Tishby.
Following the discussion on OpenReview.net, several other papers have also commented on the controversy surrounding the information bottleneck.
Noshad et al.
(2019) presented a new MI estimator EDGE, based on dependency graphs, and tested it on the speciﬁc counter example using
E they review by Geiger (2020) provides a nice overview over empirical IB studies and if the compression phase was observed (Darlow & Storkey, 2020; Jónsson et al., 2020;
Kirsch et al., 2020; Noshad et al., 2019;
Raj et al., 2020; Shwartz-Ziv & Tishby, 2017) or not (Abrol & Tanner, 2020; Balda et al., 2018; 2019; Tang Nguyen & Choi, 2019; Shwartz-Ziv & Alemi, 2020; Yu et al., 2020) or the results were mixed (Chelombiev et al., 2019; Cheng et al., 2019;
Elad et al., 2019; Fang et al., 2018; Gabrié et al., 2019; Goldfeld et al., 2019; Saxe et al., 2018; Schiemer & Ye, 2020;
Wickstrøm et al., 2019).
In conclusion, an important part of the controversy surrounding the IB hinges on the estimation of the information-theoretic quantities – this issue has to be solved before researching the information ﬂow.
Related Work.
The effect of estimating MI by binning has been investigated before, we again refer to Geiger (2020) for a good overview and discussion.
Shwartz-Ziv & Alemi (2020) consider inﬁnite ensembles of inﬁnitely-wide networks, which renders MI computation feasible, but do not observe a compression phase.
Chelombiev et al.
(2019) applies adaptive binning, which, while less prone to issues caused by having the “wrong” number of bins, is still an estimation and thus also suffers from the same problems.
Goldfeld et al.
(2019) explores IB analysis by use of stochastic neural networks which allow them to show that the compression phase in these noisy networks occurs due to clustering of the hidden representations.
While theoretically interesting, the stochastic neural networks are still qualitatively different from deterministic ones and thus not directly applicable in practice.
Raj et al.
(2020) conducted an IB analysis of binary networks, where both the activations and weights can only be ± 1 , which allows for exact computation of MI.
The binary networks are signiﬁcantly different from the networks used in the original studies, whereas applying the IB analysis to quantized versions of networks from these studies allows for a more direct comparison.
At a ﬁrst glance, the work by Raj et al.
(2020) could be viewed as taking our approach to the extreme.
However, the computations and training dynamics of the binary networks are qualitatively different from our study and the original
IB work.
For example, the binary networks require modiﬁcations for gradient estimation (e.g., Raj et al.
consider straight-through-estimator , approximate sign , and swish sign ).
E STIMATING E NTROPY AND M UTUAL I NFORMATION IN N EURAL N ETWORKS
The IB principle is deﬁned for continuous and discrete input and output, however continuous MI in neural networks is in general infeasible to compute, difﬁcult to estimate, and conceptually problematic.
3. B INNING AND L OW C OMPRESSION IN P RECISE N ETWORKS
Consider for a moment a deterministic neural network on a computer with inﬁnite precision.
Assuming a constant number of training and test patterns, a constant number of update steps, invertible activation functions in the hidden layers and pairwise different initial weights (e.g., randomly initialized).
Given any two different input patterns, two neurons will have different activations with probability one (Goldfeld et al., 2019; Geiger, 2020).
Thus, on any input, a hidden layer T will be unique, meaning that, when analysing data set D , all observed values of T will be equally likely, and the entropy H ( T ) will be trivial in the sense that it is maximum, that is, log |D| .
However, when applying the binning T (cid:48) =
B ( T ) , information is dropped, as B is surjective and several different continuous states may map to the same discrete state; for m bins and d T neurons in
T the number of different states of T (cid:48) is m d T , and thus more inputs from D will map to the same state.
As a consequence, the estimated entropy is decreased.
Furthermore, the rate of decrease follows the number of bins; the smaller the number of bins m , the smaller the estimated H ( T ) (as more values will map to the same state).
The entropy H ( T (cid:48) ) , and therefore I ( X ; T (cid:48) ) in deterministic networks, is upper bounded by d T log m .
Thus, changing the number of bins used in the estimation changes the results.
We also refer to Geiger (2020) for a more detailed discussion of this phenomenon 3 .
Computations in digital computers are not inﬁnitely precise, a 32-bit ﬂoating point variable, for instance, can take at most 2 32 different values.
In theory, this means that we can actually consider the states of a neural network discrete and compute the exact MI directly.
In practice, we expect the precision to be high enough, that – when using invertible activation functions – the entropy of the layers will be largely uninformative, as almost no information loss/compression will occur.
However, using low precision quantized neural networks, we can compute the exact MI in the network without
H ( T ) being trivial – what we regard as the ideal setting to study IB conjectures.
3. E XACT M UTUAL I NFORMATION IN Q UANTIZED N ETWORKS
Quantization of neural networks was originally proposed as a means to create more efﬁcient networks, space-wise and in terms of inference running time, for instance to make them available on mobile devices (Jacob et al., 2018; Hubara et al., 2017).
The idea is to use low precision k -bit weights (typically excluding biases) and activations, in order to reduce storage and running time.
For our use case, we are mainly interested in quantized activations, as they contain the information that is moving through the network.
The naive approach is to quantize after training ( post-training quantization ).
In terms of estimating the entropy/MI, this corresponds to applying a binning with 2 k bins to the non-quantized activations as done in the previous studies (Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018), as information available to the network at the time of training is dropped, and thus we expect it to show similar artifacts.
Instead, we investigate networks trained using quantization aware training (QAT) , in which activations are quantized on the ﬂy during training (Jacob et al., 2018).
Speciﬁcally, forward passes are quantized while backward passes remain non-quantized.
However, the IB and all MI computations in the quantized networks only refer to quantized quantities (i.e., forward pass computations) and thus measure the true information in the network at the considered training step.
The actual quantization happens after each layer (after application of the activation function).
During training, the algorithm keeps track of the minimum a ( T ) l and maximum a ( T ) u activations in each layer T across the epoch, and the activation of each neuron in T is mapped to one of 2 k uniformly distributed values between a ( T ) l and a ( T ) u , before being passed to the next layer (Jacob et al., 2018).
Thus for each epoch, the neurons in a layer T take at most 2 k different values, meaning at most 2 kd T different states for T (because of the S OFT M AX activation function the effective number of states is 2 k ( d ˆ Y − 1) for the ﬁnal layer ˆ Y ).
E XPERIMENTS
We conducted the exact bottleneck analysis in two different setups:
• The setting of Shwartz-Ziv & Tishby (2017) and Saxe et al.
(2018) using the network shown in Figure 1a with either T ANH or R E LU activations, ﬁtted on the same synthetic data set consisting of |D| = 2 12 12-bit binary input patterns with balanced binary output (Section 4.1).
• Learning a two-dimensional representation for the MNIST handwritten digits, where a fully connected network similar to the one in the previous setting using a bottleneck architecture with R E LU activations for all hidden layers is trained (Section 4.2).
For each setting, we applied k = 8 bit quantization aware training to the given network.
As in previous studies (Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018), we used an 80%/20% training/test split to monitor ﬁtting, while computing the MI based on the entire data set (training and test data).
The networks were trained using mini batches of size 256 and the Adam optimizer with a learning rate of 10 − 4 .
Weights of layer T were initialized using a truncated normal with mean 0 and standard deviation 1 / √ d T .
In addition, we also replicated the experiments of Shwartz-Ziv & Tishby (2017) and Saxe et al.
(2018) using varying number of bins, and we investigated 32- and 4-bit quantization.
We also considered IB analysis of a few other architectures for the MNIST task, including architectures with wider bottlenecks, without a bottleneck, and using convolutions.
These experiments are described in
Supplementary Material A, C, and E.
Our implementation is based on Tensorﬂow (Abadi et al., 2016).
Experiments were run on an Intel
Core i9-9900 CPU with 8 3.10GHz cores and a NVIDIA Quadro P2200 GPU. 4 .
ˆ Y = T
I ( X,T i )
I ( T i , Y )    12 0.
T
I ( X,T i )
I ( T i , Y ) 2 ,4 ,6 ,8 ,
[Figure]
I ( T i , Y ) [Figure]
T
T T T
I ( X,T i )
I ( T i , Y ) 2 ,4 ,6 ,8 ,
4. E XACT M UTUAL I NFORMATION FOR S YNTHETIC DATA
The 8-bit quantized network from Figure 1a was ﬁtted for 8000 epochs to the synthetic data set considered in Shwartz-Ziv & Tishby (2017), using either T ANH or R E LU for the hidden layers.
The experiment was repeated 50 times and the means of obtained measurements are reported.
Figure 2 and Figure 3 show the information planes for the quantized T ANH and R E LU networks respectively; additional plots showing the variance of the repetitions are included in Supplementary
Material B. Both quantized networks obtained good ﬁts with similar accuracies to non-quantized networks; see Figure F.14 and Figure F.15 in the supplementary material.
For the quantized T ANH network, the information plane is very similar to the plane obtained in the standard setting using 2 8 = 256 bins.
We clearly observe a ﬁtting and compression phase in the output layer, while I ( X ; T ) and I ( T ; Y ) in general are large for the input and hidden layers.
Looking at the hidden layer before the output layer, we do observe a ﬁtting and a compression phase, but
I ( X,T i )
I ( T i , Y ) 1 ,1 ,2 ,2 ,3 , 000 Compression in 2D layer
ˆ Y = T 5 T
T
T with the ﬁtting phase interrupted by a small compression phase.
For the remaining layers, the MI measurements with input and label remained almost constant throughout the training process.
Likewise, the information plane for the quantized R E LU network looks very similar to the plane obtained with binning.
The MI is slightly higher, but the most notably difference is the occurrence of a slight increase in I ( X ; T ) after approximately 1800 epochs, observed only for the hidden layer before the output layer.
For the remaining hidden layers, only the ﬁtting phase is observed, with no or very little compression; an observation mirrored in the binning experiments by Saxe et al.
4. E XACT M UTUAL I NFORMATION FOR L EARNING AN MNIST R EPRESENTATION
We considered the task of leaning a two-dimensional representation of the MNIST handwritten digits (Bottou et al., 1994; Deng, 2012).
We ﬁtted a network with four hidden layers with R E LU activations and 16, 8, 4 and 2 neurons, respectively, where the last two neurons feed into a 10 neuron S OFT M AX output layer.
The network was trained for 3000 epochs, and the experiment was repeated 20 times.
The bottleneck architecture for learning a two-dimensional representation leads to a non-trivial IB analysis.
Using more expressive networks (e.g., without or with a wider bottleneck) resulted in trivial, constant maximum MI measurements (see Supplementary Material E).
Figure 4 shows the information plane obtained for the network; additional plots showing the variance between repetitions are included in Supplementary Material
B.
The network exhibits sub-par performance due to the limiting bottleneck structure, obtaining a maximum accuracy of around 90% (see
Figure F.16 in the supplementary material).
However, the architecture does admit a non-trivial analysis with a clear ﬁtting phase in the output layer and the three last hidden layers.
Small compression phases are also observed in the R E LU layers around 1500 epochs, similar to the last hidden layer of the network trained on synthetic data.
The compression indicates that the learnt representations for digits from the same class become more similar.
4. D ISCUSSION
Our experiments largely conﬁrm the ﬁndings by Saxe et al.
(2018): we observe no compression phase in the quantized R E LU network for the synthetic data.
While the quantized networks do not directly correspond to the networks in the original studies (Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018), they amplify the effect of the networks being executed on a digital computer.
Note that using quantization (as often done for mobile and edge computing as well as when working on encrypted data) does not violate any assumption in the IB theory – on the contrary, as argued before, it prevents the information theoretic measures from becoming trivial.
The MI generally being higher in our experiments conﬁrms our discussion from Section 3; information is well-preserved through the network.
There is no longer a loss of information from the estimation method.
All information loss – and hence the dynamics observed – is a result of the neural network learning.
Performing the same experiments with the full 32-bit non-quantized networks, this is even more pronounced, as almost no loss of information is observed between layers.
This setting approaches the continuous case, in which we would get H ( T ) =
log |D| for invertible activation functions.
We include information planes for 32-bit networks with exact MI computation in Supplementary Material C, Figure C.11.
Conversely, when training networks with even lower precision, we see a higher information loss between layers.
This can be expected from H ( T ) ≤ 2 d T k and the small, decreasing number d T of neurons in layer T (see Figure 1b).
Using k = 4 bit precision, more layers are clearly distinguishable, with results similar to those obtained by estimating MI by binning; however, the 4-bit quantized networks also provide a worse ﬁt – indicating that when estimating MI by binning, information vital to the network is neglected in the analysis.
We provide information planes for 4-bit quantized networks in Supplementary Material C, Figure C.10.
A similar observation could be made for the network ﬁtted on MNIST.
Using the low complexity bottleneck architecture, we obtained non-trivial information measurements.
Having only two neurons in the smallest layer limits the network in terms of performance, yielding a sub-optimal accuracy around 90%.
Using more complex networks, for instance, with a wider bottleneck or without a bottleneck, we obtained accuracies over 95% (see Figure F.16), but, even when using 8-bit quantization, the information planes were trivial, with only the last layer discernible (see Supplementary Material E).
We observed similar trivial information planes for a convolutional architecture, further supporting that only very little information is actually lost in the complex networks (even with quantization), making the information bottleneck, when computed exactly, a limited tool for analysing deep learning.
C ONCLUSIONS AND L IMITATIONS
The experiments and discussion of Shwartz-Ziv & Tishby (2017) and Saxe et al.
(2018), as well as several follow-up studies, indicate that estimating MI by binning may lead to strong artifacts.
We empirically conﬁrmed these issues (see Supplementary Material A): varying the number of bins in the discretization procedure, we observed large differences in the qualitative analysis of the same network, conﬁrming that the results of previous studies highly depend on the estimation method.
To alleviate these issues, we computed the exact MI in quantized neural networks.
In our experiments using 8-bit quantized networks, the ﬁtting phase was observed for T ANH and R E LU network architectures.
The compression phase was observed in the output S OFT M AX layer in both cases .
For the T ANH network, the MI was only non-trivial for the last two layers, but not for the other hidden layers.
We observed compression in the last hidden layer when using T ANH activations ; similar to the phase observed by Shwartz-Ziv & Tishby (2017).
For the R E LU network, the MI was below its upper bound in all layers; however, no compression was observed in any of the hidden layers using
R E LU activations .
The latter observation conﬁrms the statements by Saxe et al.
(2018) about the IB in R E LU networks.
Applying the analysis to a network trained on MNIST, we found that a bottleneck architecture learning a low-dimensional representation showed a compression phase.
The compression indicates that the learnt representations for digits from the same class become more similar, as intended for representation learning.
When applied to networks without a bottleneck (e.g., a simple convolutional architecture), the IB analysis produced only trivial information planes, which shows the limitations of the IB analysis (even with exact computations) for deep learning in practice.
The main contribution of our work is that we eliminated measurements artifacts from experiments studying the IB in neural networks .
By doing so, future research on the IB concept for deep learning can focus on discussing the utility of the IB principle for understanding learning dynamics – instead of arguing about measurement errors.
The proposed quantization as such is no conceptual limitation, because for invertible activation functions studying a (hypothetical) truly continuous system would lead to trivial results.
The biggest limitation of our study is obviously that we could not (yet) resolve the controversy about the compression phase (which was, admittedly, our goal), because our experiments conﬁrmed that it depends on the network architecture whether the phase is observed or not.
However, the proposed framework now allows for accurate future studies of this and other IB phenomena.
R EPRODUCIBILITY S TATEMENT
Source code and instructions for reproducing our experiments are available online: https:// github.com/StephanLorenzen/ExactIBAnalysisInQNNs A CKNOWLEDGEMENTS The authors would like to thank the anonymous reviewers for their constructive comments. SSL acknowledges funding by the Danish Ministry of Education and Science, Digital Pilot Hub and
Skylab Digital.
MN acknowledges support through grant NNF20SA0063138 from Novo Nordisk Fonden. CI acknowledges support by the Villum Foundation through the project Deep Learning and Remote Sensing for Unlocking Global Ecosystem Resource Dynamics (DeReEco). The authors acknowledge support by the Pioneer Centre for AI (The Danish National Research Foundation, grant no. P1)
A E STIMATING M UTUAL I NFORMATION BY B INNING
This section presents a replication of the experiments from Shwartz-Ziv & Tishby (2017) and Saxe et al.
but varying the number of bins for estimating MI.
Non-quantized T ANH and R E LU networks were trained in the standard setting
[Figure]
and MI was estimated using 30, 100 and 256 bins.
The number of bins were chosen to match the numbers used in the earlier works (30 used by Shwartz-Ziv & Tishby, 100 used by Saxe et al.) and to match the discretization in the quantized neural networks considered in Section 4.
[Figure]
[Figure]
The network is trained in the standard setting, and the average of 50 runs is plotted.
The resulting information planes are shown for the T ANH and R E LU networks in Figure A.5 and
Figure A.6 respectively.
From Figure A.5, we clearly see the results of Shwartz-Ziv & Tishby (2017) replicated using 30 bins for the T ANH network; two phases are clearly visible in each layer.
As expected, for every layer
T with binned T (cid:48)
=
B ( T ) , I ( X ; T (cid:48) ) and I ( T (cid:48) ; Y ) increase with the number of bins used with the phases still remaining visible in the distinguishable layers.
For the R E LU network with 100 bins, the results of Saxe et al.
(2018) were also replicated, that is, the compression phase was not observed.
When using only 30 bins for the R E LU network, the estimation broke down completely in the sense that the DPI (2) is violated: I ( T (cid:48) ; Y ) is non-monotone, a phenomenon occurring because of the estimation, which has also been observed by Geiger (2020).
Again, we see for any layer T that I ( X ; T (cid:48) ) and I ( T (cid:48) ; Y ) increase with m , although to a lesser degree
[Figure]
T 2 T
T 4 T
[Figure]
T
[Figure]
[Figure]
than for the T ANH network, most likely due to the R E LU activation functions actually dropping information (by zeroing negative activations).
In general, the estimation of MI in either network is highly dependable on the number of bins used, indicating that the interpretation of the information plane is not straight forward, when using binning for estimation.
B MI V ARIANCE
In this work, we investigate the effect of MI approximation in previous IB studies.
These original studies are based on a qualitative analysis of mean curves.
However, the question arises of whether mean curves correctly represent individual learning processes.
While we do not investigate this question in detail, the following section provides plots illustrating the variance between experimental trials.
Figures B.7, B.8 and B.9 show the mean and variance of I ( X ; T ) (left) and
I ( T ; Y ) (center), as well as the information plane for a single trial (right), for the T ANH and R E LU networks in the standard setting, and the bottleneck network applied to MNIST, respectively.
For the single trial, we selected the median deviating repetition, ranked by the L2-distance from the mean.
From the plots, we see that when the mean MI is large, the variance is usually low.
The variance is similar across the different networks (note the different scaling of the y-axes), with slightly larger variance for the R E LU activations.
Together with the median deviating information plane, the plots suggest that the mean information plane, which averages out less pronounced phases of MI increase and decrease, represents individual repetitions fairly well.
[Figure]
[Figure]
I ( T , Y
[Figure]
T
T 2 T
T 4 T
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
I
[Figure]
i
[Figure]
[Figure]
Additionally, the compression phase in the T ANH network can clearly be seen in Figure B.7 (left), while it is absent in the R E LU network (Figure B.8)
[Figure]
C E FFECT OF THE Q UANTIZATION P RECISION
This section presents information planes obtained using 4- and 32-bit quantized neural networks trained in the standard setting, in order to investigate the effect of the precision used in the quantization.
Figure C.10 depicts the resulting information planes for the 4-bit networks.
As each neuron can take only 2 4 = 16 different values, the total number of possible states per layer decreases signiﬁcantly in this setting, and as expected we see lower overall MI measurements.
For the T ANH network, several more layers become distinguishable.
The observed information planes looks similar to those observed in the original experiments by Shwartz-Ziv & Tishby (2017) and Saxe et al.
However, the network accuracy has now degraded compared to the non-quantized networks (see Supplementary
Material F), which indicates that the binning used in the estimation of the MI in previous experiments has discarded information vital to the network.
Figure C.11 shows the resulting information planes for the 32-bit networks.
As expected, we see an overall increase in MI; the information drops only very slowly through the network.
Each layer has many possible states and – given the small data set – we we get closer to the behavior of a continuous system.
I ( X,T i )
I ( T i , Y )
Tanh      12 0.
I ( X,T i )
I ( T i , Y )
ReLU 2 ,4 ,6 ,8 ,
[Figure]
[Figure]
[Figure]
I (
[Figure]
i )
[Figure]
[Figure]
i ,
[Figure]
Tanh      12 0.
I ( X,T i ) I ( T i , Y ) ReLU 2 ,4 ,6 ,8 ,
D Q UANTIZED N ETWORK WITH R ANDOMIZED P REFITTING
This section presents results for the standard setting using 8-bit quantized networks, but with weights preﬁtted to random labels.
First, the network is ﬁtted for 1000 epochs to the synthetic data set with the labels shufﬂed.
The network is then trained for 8000 epochs with correct labels (as done in
Section 4.1).
The experiment is repeated 20 times.
The resulting information planes for the T ANH and the R E LU network are presented in Figure D.12, left and right respectively.
Inspecting the plots, we see that, for all discernible layers, I ( X ; T ) appears to increase when ﬁtted to random labels.
This is not surprising as the input is not shufﬂed.
As a consequence of the increase in I ( X ; T ) , the hidden layers (most notable in the R E LU network) also see an increase in I ( T ; Y ) .
As the ability to distinguish different inputs from a hidden layer T increases, so does I ( T ; Y ) .
As expected, the network accuracy does not increase when training on random labels, and accordingly there is no lateral movement in the information plane for the output layer.
I ( X,T i ) I ( T i , Y ) Bottleneck16 0 .
I ( X,T i )
I ( T i ,
Y )
HourGlass 1 ,1 ,2 ,2 ,3 ,      16 0 .
I ( X,T i )
I ( T i ,
Y ) 4x       16 0 .
I ( X,T i )
I ( T i ,
Y )
Conv
E MNIST WITH O THER N ETWORK A RCHITECTURES
This section presents the results of training a number of 8-bit quantized networks with varying architectures on MNIST.
We consider the same setup as in Section 4.2, but with the following architectures (all using R E LU activations in the hidden layers and a ﬁnal 10 layer S OFT M AX output):
• The B OTTLENECK -4 network with 4 hidden layers of widths 16, 12, 8 and 4, i.e. the bottleneck has width 4.
• The H
OUR G LASS network with 6 hidden layers of widths 16, 8, 4, 2, 4, 8.
The bottleneck width is still 2, but the network expands again after the bottleneck, creating an hourglass shape.
• The 4 X 10 network with 4 hidden layers, each of width 10.
• The C ONV network, a simple convolutional network with structure: CV-MP-CV-MP-FC, where CV denotes a (3,3) 2-channel convolutional layer, MP denotes a (2,2) max pooling and FC denotes a fully connected R E LU layer with 20 neurons
Below, we denote the network used in Section 4.2 by B OTTLENECK -2.
Each experiment is repeated 20 times and the resulting information planes are reported in Figure E.13.
As expected, given the increased complexity, all networks exhibit better performance than the bottleneck structured network considered in Section 4.2 (the accuracy of all networks are presented in Figure F.16 in Supplementary
Material F), obtaining accuracies close to or above 95% for all networks except the H OUR G LASS network.
From Figure E.13, we observe the information curves for the B OTTLENECK -4 network to be similar in shape to the B OTTLENECK -2, but with higher MI measurements in general.
This is not surprising, as the layers are wider and thus more expressive.
As expected, the H OUR G LASS network also exhibits information curves similar to B OTTLENECK - (as the architecture is similar for the ﬁrst few layers).
Not surprisingly, the two expanding layers of widths 4 and 8 almost coincide completely with the bottleneck layer; information cannot increase but is also not decreased signiﬁcantly.
The two more expressive networks without bottlenecks, 4 X 10 and C ONV , have almost completely trivial information curves.
Limited ﬁtting for the output layer in 4 X 10 and compression in the output layers for both networks can be observed.
The experiments indicate the limitations of exact IB analysis for complex, large scale networks.
F N ETWORK A CCURACIES
Figure F.14 and Figure F.15 report the accuracies obtained by the T ANH and R E LU networks when ﬁtted on the synthetic data, respectively.
As can be seen, the networks suffers only slightly when
[Figure]
[Figure]
Figure F.16 reports the accuracy of the networks ﬁtted to MNIST.
Unsurprisingly, the more complex networks (B OTTLENECK -4, 4 X 10, C ONV ) obtain better accuracy with lower variance, compared to the networks with a low-width bottleneck (B OTTLENECK -2, H OUR G LASS ).
[Figure]