A DAPTIVE E STIMATORS S HOW I NFORMATION C OMPRESSION IN D EEP N EURAL N ETWORKS
A BSTRACT
To improve how neural networks function it is crucial to understand their learning process.
The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task.
However, empirical evidence for this theory is conﬂicting, as compression was only observed when networks used saturating activation functions.
In contrast, networks with non-saturating activation functions achieved comparable levels of task performance but did not show compression.
In this paper we developed more robust mutual information estimation techniques, that adapt to hidden activity of neural networks and produce more sensitive measurements of activations from all functions, especially unbounded functions.
Using these adaptive estimation techniques, we explored compression in networks with a range of different activation functions.
With two improved methods of estimation, ﬁrstly, we show that saturation of the activation function is not required for compression, and the amount of compression varies between different activation functions.
We also ﬁnd that there is a large amount of variation in compression between different network initializations.
Secondary, we see that L2 regularization leads to signiﬁcantly increased compression, while preventing overﬁtting.
Finally, we show that only compression of the last layer is positively correlated with generalization.
I NTRODUCTION
Although deep learning (reviewed by Schmidhuber (2015)) has produced astonishing advances in machine learning (Silver et al., 2017), a rigorous statistical explanation for the outstanding performance of deep neural networks (DNNs) is still to be found.
According to the information bottleneck (IB) theory of deep learning (Tishby & Zaslavsky, 2015;
Shwartz-Ziv & Tishby, 2017)
the ability of DNNs to generalize can be seen as a type of representation compression.
The theory proposes that DNNs use compression to eliminate noisy and task-irrelevant information from the input, while retaining information about the relevant segments (Chechik & Tishby, 2003).
The information bottleneck method (Tishby et al., 2000) quantiﬁes the relevance of information by considering an intermediate representation T between the original signal X and the salient data Y .
T is the most relevant representation of X , and is said to be an information bottleneck, when it maximally compresses the input, retaining only the most relevant information, while maximizing the information it shares with the target variable Y .
Formally, the information bottleneck minimizes the Lagrangian:
[Equation]
where I ( · ) is mutual information.
In this Lagrangian β is the Lagrange multiplier, determining the trade-off between compression and retention of information about the target.
In the context of deep learning, T is a layer’s hidden activity represented as a single variable, X is a data set and Y is the set is decrease
( T, ) value, increasing
Fitting behaviour refers to both increasing.
Shwartz-Ziv & Tishby (2017) visualized the dynamic of training a neural network by plotting the values of I ( T, X ) and I ( T, Y ) against each other.
This mapping was named the information plane.
According to IB theory the learning trajectory should move the layer values to the top left of this plane.
In fact what was observed was that a network with tanh activation function had two distinct phases: ﬁtting and compression.
The paper and the associated talks 1 show that the compression phase leads to layers stabilizing on the IB bound.
When this study was replicated by Saxe et al.
(2018) with networks using ReLU (Nair & Hinton, 2010) activation function instead of tanh , the compression phase did not happen, and the information planes only showed ﬁtting throughout the whole training process.
This behaviour required more detailed study, as a constant increase in mutual information between the network and its input implies increasing memorization, an undesired trait that is linked to overﬁtting and poor generalization (Morcos et al., 2018).
Measuring differential mutual information in DNNs is an ill-deﬁned task, as the training process is deterministic (Saxe et al., 2018).
Mutual information of hidden activity T with input X is:
[Equation]
If we consider the hidden activity variable T to be deterministic then entropy is:
[Equation]
However, if T is continuous then the entropy formula is:
[Equation]
In the case of deterministic DNNs, hidden activity T is a continuous variable and p ( T | X ) is distributed as the delta function.
For the delta function:
[Equation]
Thus, the true mutual information value I ( T, X ) is in fact inﬁnite.
However, to observe the dynamics of training in terms of mutual information, ﬁnite values are needed.
The simplest way to avoid trivial inﬁnite mutual information values, is to add noise to hidden activity.
Two ways of adding noise have been explored previously by Shwartz-Ziv & Tishby (2017) and Saxe et al.
One way is to add noise Z directly to T and get a noisy variable ˆ T
= T + Z .
Then
H ( T | X ) =
H ( Z ) and mutual information is I ( ˆ T, X )
=
H ( ˆ T )+ H ( Z ) .
When the additive noise is
Gaussian, the mutual information can be approximated using kernel density estimation (KDE), with an assumption that the noisy variable is distributed as a Gaussian mixture (Kolchinsky & Tracey, 2017).
The second way to add noise is to discretize the continuous variables into bins.
To estimate mutual information, Shwartz-Ziv & Tishby (2017) and Saxe et al.
(2018) primarily relied on binning hidden activity.
The noise comes embedded with the discretization that approximates the probability density function of a random variable.
In context of neural networks, adding noise can be done by binning hidden activity and approximating H ( T ) as a discrete variable.
In this case H ( T | X ) =
since the mapping is deterministic and I ( T, X )
=
H ( T ) .
Generally, when considering mutual information in DNNs, the analyzed values are technically the result of the estimation process and, therefore, are highly sensitive to it.
For this reason it is vital to maintain consistency when estimating mutual information.
The problem is not as acute when working with DNNs implemented with saturating activation functions, since all hidden activity is bounded.
However, with non-saturating functions, and the resulting unbounded hidden activity, the level of noise brought by the estimation procedure has to be proportional and consistent, adapting to the state of every layer of the network at a particular epoch.
In the next section adaptive estimation schemes are presented, both for the binning and KDE estimators.
It is shown that for networks with unbounded activation functions in their hidden layers, the estimates of information change drastically.
Moreover, the adaptive estimators are better able to evaluate different activation functions in a way that allows them to be compared.
This approach shows considerable variation in compression for different activation functions.
It also shows that
L2 regularization leads to more compression and clusters all layers to the same value of mutual information.
When compression in hidden layers is quantiﬁed with a compression metric and compared with generalization, no signiﬁcant correlation is observed.
However, compression of the last softmax layer is correlated with generalization.
1. M ETHODS
In this paper we used a fully connected network consisting of 5 hidden layers with 10-7-5-4-3 units, similar to Shwartz-Ziv & Tishby (2017) and some of the networks described in Saxe et al.
ADAM (Kingma & Ba, 2014) optimizer was used with cross-entropy error function.
We trained the network with a binary classiﬁcation task produced by Shwartz-Ziv & Tishby (2017) for consistency with previous papers.
Inputs were 12-bit binary vectors mapped deterministically to the 1-bit binary output, with the categories equally balanced (see Shwartz-Ziv & Tishby (2017) for details).
Weight initialization was done using random truncated Gaussian initialization from Glorot & Bengio (2010), with 50 instances of this initialization procedure for every network conﬁguration used.
Based on the observed data, even 10 initializations provide a clear picture of the average network behaviour, but for the purpose of consistency with previous studies, we used 50 initializations.
80% of the dataset was used for training, using batches of 512 samples.
During the training process, hidden activity for different epochs was saved.
The calculation of mutual information was done using the saved hidden activity, after training has been completed; the two processes did not interfere with one another.
M UTUAL I NFORMATION E STIMATION
The observed dynamics of a neural network’s training on the information plane depends strongly on the estimation procedure.
Estimating mutual information at different epochs and layers presents a problem as these variables are inherently different from one another. Using more adaptive frameworks makes it possible to choose estimator parameters that would produce comparable estimate values.
2. E NTROPY - BASED B INNING
Binning networks with saturating activation functions is straightforward, since the saturation points deﬁne the range of the binning.
With non-saturating activation functions a range of activation values must be speciﬁed.
In Saxe et al.
(2018) for networks with ReLU units in their hidden layer, all hidden activity was binned using a single range, going from zero to m , where m is the maximum value any
ReLU unit has achieved throughout the training.
The argument behind this choice of binning range is that the whole space explored by the ReLU function must be included in the binning process.
This approach is inherently limited, since a network at every epoch is different.
If the binning range is determined by picking a maximum value across all epochs and all layers, then the information plane values estimated for a given epoch and a given layer depend on factors that are unrelated to their own properties.
In fact, the behaviour of the activation values is markedly different in different layers; the maximum values can differ by as much as two orders of magnitude in different layers and epochs (Figure 1).
Using a single maximum value when binning the data usually means that some layers during some epochs will have all of their values binned together in the lowest bin.
To avoid this we use an entropy-based binning strategy (Kohavi & Sahami, 1996) to choose bin boundaries: the bin boundaries are chosen so that each bin contains the same number of unique observed activation levels in a given layer and epoch.
When there are repeated activation levels, as can happen for any activation function that is not unbounded, these are ignored when calculating the boundaries.
This often occurs near the saturation regions where the activation levels are indistinguishable even as ﬂoat64 values.
However, we calculate the entropy for the whole layer rather than for individual units; in other words, the vector of activation levels of the units in the layer is converted to a vector of bin indices and the entropy is calculated for these vectors.
This binning procedure will be referred to as entropy-based adaptive binning (EBAB).
[Figure]
[Figure]
In this approach there is no need to deﬁne a range as part of the binning procedure.
It has other advantages.
Firstly, the deﬁnition of the bins depends on the distribution of the observed activation values, not on the shape of the activation functions.
This means this binning strategy applies in the same way to all activation functions, allowing different networks with different activation functions to be compared.
Secondly, a ﬁxed width binning approach can be insensitive to the behaviour of the layer because much of the activity occurs in a single bin.
An example of binning where a very small subset of hidden activity achieves high values is shown in Figure 2, as such, most of the hidden activity falls into the ﬁrst bin.
With entropy-based binning, bin edges are more densely distributed closer to the lower bound of binning range, providing a better spread.
The difference in mutual information estimates produced with an adaptive binning procedure is signiﬁcant when visualized on the information plane.
Non-adaptive binning consistently underestimates compression.
Information planes of the same network in Figure 3 using ReLU analyzed with different estimators show how non-adaptive binning consistently underestimates compression.
[Figure]
2. K ERNEL D ENSITY E STIMATION
Saxe et al.
(2018) used non-parametric KDE estimator outlined by Kolchinsky & Tracey (2017) to estimate mutual information, as well as binning.
This method of estimation directly adds small
Gaussian noise to the data to produce estimates that are not inﬁnite.
The vast variation of the level of activation values (Figure 1) makes it inappropriate to use a ﬁxed noise level.
Due to the difference in activation values, noise added to layers with smaller activation values produces a different effect, as the noise of the same magnitude, added to the largest activation.
Hence, KDE estimation with a ﬁxed level of noise produces inconsistent estimations.
Here the noise variable σ 2 was adapted for every layer of every epoch.
To implement this we chose a constant reference rate of noise of σ 20 = 1 · 10 − 3 , and then for a given layer σ 2 was calculated by scaling σby the maximum value of the hidden activity in that layer for every epoch.
With this estimation adjustment, each layer at every epoch received noise of different magnitude, but of similar proportion.
This estimation procedure will be referred to as adaptive KDE (aKDE).
[Figure]
Figure 4 demonstrates that scaling the noise variance σ 2 in unison with the magnitude of the activation values produces estimates that are very similar to EBAB.
Similarly, non-adaptive KDE resembles binning with non-adaptive range and widths.
The network initialization implemented with
ReLU in Figures 3 and 4 exhibits compression when analyzed with EBAB and aKDE.
In the next section we look at ReLU networks with different initializations to see whether this compression always happens.
C OMPRESSION IN D EEP N EURAL N ETWORKS
After deriving adaptive estimation framework, we investigated the occurrence of compression in networks with non-saturating activation functions.
Network initializations showed a large variation of compression behaviour.
Figures 3 and 4 showed a network with signiﬁcant information compression.
However, other network initializations can lead to different behaviour on the information plane.
Figure 5 shows different ways a network can evolve during training.
Based on the observed variation of behaviour, it cannot be said that a network with ReLU always exhibits compression during training, as in the case with the saturating tanh .
Networks are capable to learn without compression.
However, it also cannot be said that using a non-saturating ReLU prevents any compression.
Moreover, unlike networks with tanh activation function, the compression phase can happen ﬁrst, with or without subsequent ﬁtting phase.
[Figure]
Similarly to previous studies, we generated 50 networks with different random initializations for every activation function to investigate the average behaviour.
When one information plot averaged from 50 network initializations is produced, ﬁtting and compression mostly cancel out.
On the resulting information plot, the averaged network does not show strong ﬁtting or compression, the
I ( T, X ) values are mostly stable (Figure 6).
[Figure]
ReLU network averaged over 50 individual network initializations shows no distinct phases of ﬁtting or compression.
To see if this behaviour on the information plane is simply the product of absence of a saturating activation function, we have tested other non-saturating activation functions and show information plots averaged over 50 initializations in Figure 7.
Different unit types tested are absolute value, PReLU (He et al., 2015), ELU (Clevert et al., 2015), softplus (Glorot et al., 2011), centered softplus, (Mishkin et al., 2017) and Swish (Ramachandran et al., 2018;
Elfwing et al., 2018).
[Figure]
The information planes demonstrate that ﬁner properties of activation functions have an effect on training dynamics on the information plane, as few functions look similar to each other.
Even though tanh has the strongest compression, ELU, Swish and centered softplus activation functions also compress information, despite being non-saturating.
This compression behaviour is marked by a decrease in I ( X, T ) values for these activation functions in Figure 7.
3. L2 R EGULARIZATION AND C OMPRESSION
We also considered a commonly used technique to increase testing accuracy: L2 regularization.
It spreads out the learning by preventing a subset of network activations from having very large values.
We implemented L2 regularization on all weights of hidden layers with a network using ReLU.
When we plotted the mutual information estimates on the information plane, L2 regularization induced the network to compress information.
Compression occurred in later stages of training, similarly to networks using tanh , when networks usually go into an overﬁtting regime.
When looking at information planes averaged over 50 initializations, strong compression is visible.
As seen in Figure 8, L2 regularization induces compression in networks with ReLU activation functions, that do not show strong compression on their own.
Therefore, compression of information about the input can be a way for a network to avoid overﬁtting, since networks with L2 regularization increased their test accuracy continuously, without going into an overﬁtting regime.
Moreover, L2 regularization clusters the mutual information of all layers in a single position on the information planes.
Thus it induces more compression in layers that are closer to the data (Figure 8 (d)-(f)).
Even with the smallest penalty, compression is distinguishable on the information plane.
As the L2 penalty increases, layers are more attracted to a single point on the information plane.
[Figure]
[Figure]
3. C OMPRESSION AND G ENERALIZATION
In order to analyze compression in a systematic manner, we have developed a simplistic quantitative compression metric to assign a compression score to every network:
[Equation]
where L ∈ R N × M is a matrix containing I ( T, X ) values of all layers and epochs, k enumerates the layers in { 1 , ..., N } , E is the set of epoch indexes { 1 , ..., M } .
The score ranges from zero to one, with one meaning fully compressed information.
When we compared compression of individual network initializations with accuracy, higher rates of compression did not show a signiﬁcant correlation with generalization.
However, the compression that occurred in the last softmax layer showed signiﬁcant correlation with accuracy.
This is demonstrated in Figure 9a.
We also looked at compression of 50 averaged network initializations with different activation functions.
The last layer was not taken into account, since it always uses saturating softmax function and is not relevant to comparison of compression in non-saturating activation functions.
Similarly to individual ReLU initializations, when compression was compared with the average accuracy of networks, there was no signiﬁcant correlation, as shown in Figure 9b.
These ﬁndings suggest that the information bottleneck is not implemented by the whole network, and generalization can be achieved without compression in the hidden layers.
However, the last layer’s compression is correlated with generalization and compression there should be encouraged.
[Figure]
D ISCUSSION
In this paper we proposed adaptive approaches to estimating mutual information in the hidden layers of DNNs.
These adaptive approaches allowed us to compare behaviour of different activation functions and to observe compression in DNNs with non-saturating activation functions.
However, unlike saturating activation functions, compression is not always present and is sensitive to initialization.
This may be due to the minimal size of the network architecture that was tested.
Experiments with larger convolutional neural networks could be used to explore this possibility.
Different non-saturating activation functions compress information at different rates.
While saturation plays a role in compression rates, we show that its absence does not imply absence of compression.
Even seemingly similar activation functions, such as softplus and centered softplus, gave different compression scores.
Compression does not always happen in later stages of training, but can happen from initialization.
Further work is needed to understand the other factors contributing to compression.
We also found that DNNs implemented with L2 regularization strongly compress information, forcing layers to forget information about the input.
The clustering of mutual information to a single point on the information plane has never been reported previously.
This result could lay the ground for further research to optimize the regularization to stabilize the layers on the information bottleneck bound to achieve better generalization (Achille & Soatto, 2018), as well as linking information compression to memorization in neural networks (Zhang et al., 2016).
There are a few limitations to the analysis presented here.
Principally, for tractability, the networks we explored were much smaller and more straightforward than many state of the art networks used for practical applications.
Furthermore, our methods for computing information, although adaptive for any distribution of network activity, were not rigorously derived.
Finally, our compression metric is ad-hoc.
However, overall we have three main observations: ﬁrst, compression is not restricted to saturating activation functions, second, L2 regularization induces compression, and third, generalization accuracy is positively correlated with the degree of compression only in the last layer and is not signiﬁcantly affected by compression of hidden layers. A CKNOWLEDGMENTS
IC thanks Yobota limited ( https://yobota.xyz ) and Ammar Akhtar for contributing to broadening the reach of this work.
CJH thanks the James S. McDonnell Foundation for support through a Scholar Award in Cognition (JSMF #220020239; https://www.jsmf.org/ ).
Some of this work was carried out using the computational facilities of the Advanced Computing Research Centre, University of Bristol - http://www.bris.ac.uk/acrc/ .