Paper: Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR-2024)

Reviewer comment (Novelty):
Abstract: „has only been verified for NNs of tiny sizes or specific types, such as quantized NNs“: I would not say that this is verified see summary table of [1]. The SNN paper did not saw a compression.

Author response:
Indeed, Table 1 from Geiger (2020) shows that compression phase was observed in some (not all) experiments. For clarification, it is therefore better to replace “has only been verified” with “was only partially verified”. This will be done in the next revision.

Reviewer comment (Theory):
Statement 2: „This statement demonstrates that an arbitrary amount of information can be lost through compression. It arises from the fact that “less significant” in terms of metric spaces does not align with “less significant” in terms of information theory. However, with additional assumptions, a more useful theoretical result can be obtained.“: I am not sure but following the arguments from Goldfeld [3] MI is ill defined in deterministic DNN. So with this in mind no information can be removed in the compression phase. Can you maybe discuss this in the paper?

Author response:
We kindly ask for clarification of the question, as in this particular part of our manuscript, we refer to the compression of data via an autoencoder. This should not be confused with the so-called “compression stage” in the “compression-fitting hypothesis”. We will change this part to “an arbitrary amount of information can be lost through compression **of the data**” in the next revision to avoid any potential confusion.

Reviewer comment (Experiments):
Another issue is the one that authors have also mentioned in the paper. Just providing a toy example of MNIST for a small classification model is not yet convincing to consider the scalability of the MI estimation approach, its accuracy on the high-dimensional data (not just 784), and its applicability in other non-classification problems (regression, unsupervised, generative, etc.).

Author response:
2. Information bottleneck analysis of bigger or other non-classification problems (regression, generative or language modeling) is one of the possible directions for the future work on this topic. The current paper is devoted to the validation of the proposed method via a theoretical analysis, tests on synthetic data and application to some non-synthetic high-dimensional examples. Although the used problem is toy, it still poses difficulties in the task of MI estimation. That is why we believe our results to be a step forward.

Reviewer comment (Writing):
Some details about the functions $f_1$, $f_2$, $f$, $g$, etc do not exist in the paper.

Author response:
3. The exact choice of $ f_i $ and $ g_i $ is not very important. In the main text, we only require $ f_i $ and $ g_i $ to satisfy the conditions of Statement 1 (see Algorithm 1; we will also add this requirement to Figure 2 in the next revision) to properly define the experimental setup. Defining $ f_i $ and $ g_i $ used in the tests requires a lot of space, as these functions smoothly map low-dimensional vectors to high-dimensional images and, thus, are very complex. A Python implementation of the functions in question is available in the supplementary material. We will consider adding the corresponding listings in the Appendix in the next revision.

Reviewer comment (Theory):
Does the function $f$ in statement need to be bijective (both injective and surjective)? I think only being injective is not sufficient.

Author response:
1. If $ n' \neq n $, it is impossible to define $ f $ as a smooth bijective mapping between $ \mathbb{R}^{n'} $ and $ \mathbb{R}^n $. However, as $ f $ is injective, it is a bijection between $ \mathbb{R}^{n'} $ and $ f(\mathbb{R}^{n'}) \subseteq \mathbb{R}^n $. So $ f $ can be considered as a smooth bijection between $ \mathbb{R}^{n'} $ and a corresponding manifold in $ \mathbb{R}^n $.

Reviewer comment (Evaluation):
While it is clear that MI does not change under bijective mappings, the same may not hold for estimators of MI. In other words, I question the validity of the first equality in (4). To be more concrete, I think one can create counterexamples where a KDE of MI differs depending on whether it is obtained from the original data (dimension $n$) or the latent representation (dimension $n'$). Since MI can only be estimated and not measured, this leads me to questioning the validity of the study: If compression affects the results of the estimator, then how can we be sure that the estimate represents the "true" (i.e., uncompressed) behavior of MI?

Author response:
It is, of course, true that a MI estimate obtained from the original data generally is not equal to an estimate obtained from the compressed representations; otherwise there is no point in compressing the data (except, probably, for computational complexity). However, conventional estimators completely fail to correctly estimate MI in high-dimensional cases due to the exponential sample complexity (see Goldfeld et al. (2020); McAllester \& Stratos (2020); $ \textnormal{estimation error} \in O(\textnormal{n.samples}^{-1/\textnormal{dim}}) $).
It might be possible that equation (4) has been misunderstood: we do not claim that estimates for the original and compressed data are equal. Instead, we *define* MI estimate for high-dimensional compressible data through a conventional (e.g., via KDE) MI estimate for compressed representation.
We will definitely improve this part of our work in the next revision to avoid such critical misunderstanding.

   Our main theoretical and practical contribution is showing that the proposed MI estimator (defined by eq. (4)) is valid: we (a) show that MI is not lost via lossless compression, (b) bound MI loss in case of lossy compression, (c) provide experimental results with synthetic data showing good performance of the proposed method. One can be assured that the proposed estimation method represents the “true” behavior of MI via combining our results with a convergence analysis of the third-party estimator, which the compressed representations are fed into. In most of the cases, it can be done via the triangle inequality: $ |I(X;Y) - \hat I(E(X);Y)| \leq |I(X;Y) - I(E(X);Y)| + |I(E(X);Y) - \hat I(E(X);Y)| $, where the first term is analyzed in our article, and the second term should be acquired from a convergence analysis of a particular MI estimator.

Reviewer comment (Experiments):
Along that line, it would be important to see in Figs. 3 and 4 how the MI estimate obtained directly from the high-dimensional images behaves. This would more clearly show that compression is useful in comparison with estimators for high-dimensional data.

Author response:
It is a valid point that the contribution of our work would be more clear if we provide estimates computed directly from the data.
However, as conventional entropy estimators exhibit bias and variance of such a high magnitude in case of high-dimensional data, it is impossible to plot results for the compressed and uncompressed data side by side without a complete loss of readability.

   In particular, weighted Kozachenko-Leonenko (WKL) yields estimates ranging from about $ -3000 $ to $ 3000 $ nats with a variance around $ 2000 $ nats in the test with $ 8 \times 8 $ images of rectangles and the true value of MI varying from $ 0 $ to $ 10 $ nats. Other estimators yield the same nonsensical results for this $ 64 $-dimensional example. Thus, the error is of several orders of magnitude. In contrary, our method works fine even for the $ 32 \times 32 = 1024 $-dimensional data of the same kind.

   One can reproduce these results via running the provided source code. We will add experimental proofs of the inability of the conventional methods to estimate MI of high-dimensional random vectors in the next revision.

We hope that provided clarifications will help to better understand the manuscript and reevaluate the contribution of our work.
We will add all the required evidence in the next revision to stress out that obtaining MI estimates via conventional estimators without compression is practically impossible.
We will also correct all the mentioned minor weaknesses in the next revision.

Reviewer comment (Experiments):
The paper does not show that the compressed representations are "better" for MI estimation than the original layer outputs/data, or that fundamentally different qualitative results can be obtained. While the approach seems reasonable and valid, I would like to see more evidence that it outperforms MI estimation from high-dimensional random variables.

Author response:
1. Previous theoretical results (see Goldfeld et al. (2020); McAllester & Stratos (2020)) and our experimental results (Section E.3 in the new revision) show that estimation without compression is a much harder task, which is expected to result in estimates of a much lower quality.
2. If the compression is performed well, the estimate is close to the real value. It is demonstrated both via the experiments with synthetic data and the acquired bounds on the information lost due to the compression. Thanks to Statements 7,8,9 and Corollary 4, the information-theoretic quality of the compression can (under certain assumptions) be measured via the reconstruction error. We additionally note that the proposed bounds are tight, as they are derived from the data processing inequality.
3. In the work of Pool et al. (2019) it has been shown that other complex parametric NN-based estimators (NJW, JS, InfoNCE, etc.) exhibit poor performance during the estimation of MI between a pair of $ 20 $-dimensional _incompressible_ (i.e., not lying along a manifold) synthetic vectors. These vectors, however, are of much simpler structure than the synthetic datasets used in our work (Gaussian vectors and $ x \mapsto x^3 $ mapping applied to Gaussian vectors in (Pool et al., 2019) versus high-dimensional images of geometric shapes and functions in our work). We interpret these results as follows: if it is impossible to compress high-dimensional data to a lower dimension, not only our method fails, but the other modern parametric NN-based methods fail too.

We hope that this information will (a) help to better understand the benefits of the explicit compression, (b) show that other modern and acknowledged methods exhibit the same limitations in case of incompressible data as our method do. Overall, judging by the tests performed in our work (Sections 5, C, E.3) and by the comprehensive overviews presented in (Poole et al., 2019; McAllester & Stratos, 2020), we think that our method is not flawless, but performs better than the existing MI estimators.

Reviewer comment (Evaluation):
In Fig. 3 and 4, why do the curves for most estimators decrease when compared to the true MI?

Author response:
In Figure 3 we attribute such behavior to the floating point error, as $ I(\xi;\eta) = 10 $ in case of $ n' = m' = 2 $ leads to correlation coefficient $ \sqrt{1 - e^{-10}} \approx 1 - 2 \cdot 10^{-5} $, yielding an ill-posed covariance matrix, which is then inverted on the first step of dataset generation.

   In case of $ n' = m' = 4 $ (Figure 4) the correlation coefficient is $ \sqrt{1 - e^{-10/2}} \approx 1 - 3 \cdot 10^{-3} $,
    and WKL yields better results. However, due to a doubled dimension, KDE and KL estimators exhibit an increased bias, as these methods are more sensitive to high dimensionality.

Reviewer comment (Experiments):
In the same figures, why does MINE require a critic with the same architecture as the autoencoder, if its input are low-dimensional ($n'=2$ or $4$) signals? (Note that the MINE curves are blue.)

Author response:
MINE is fed with uncompressed data. The choice of colors for this particular case is indeed misleading. Please refer to the text (“We also conduct experiments with MINE (without compression)”, Section 5, paragraph 3). We will change the color in the next revision. As MINE is fed with uncompressed data, the critic network should be at least as complex as the autoencoder to allow for fair comparison.

Reviewer comment (Experiments):
In the same figures, there is good agreement between the blue and green curves. However, it can be assumed that the encoders $E_X$ and $E_Y$ act in a way to retrieve $f_1(\xi)$ and $g_1(\eta)$, respectively (or some other bijection thereof). With this in mind, it is not surprising that the curves agree so well. Has this been tested? For example, it could be a good idea to use a higher/smaller dimension for the output of the encoders as for the dimension of the generated signals.

Author response:
This hypothesis has not been tested. However, such good agreement may indeed indicate (almost) bijectivity of $ E_X \circ f_2 $ and $ E_Y \circ g_2 $. We will consider investigating this hypothesis in our future work.

Reviewer comment (Evaluation):
Why does the KL estimator have such a high offset?

Author response:
KL estimator is proved to be biased for dimensions $ \ge 4 $, see Berrett et al. (2019).

Reviewer comment (Experiments):
In Section 6, is noise also added during training or only during computing MI estimates? In any case, since noise is added one must not ignore the potential geometric affects that are induced by it (effects both during training and during estimation).

Author response:
Yes, these geometric effects are also mentioned in the work of Goldfeld et al. (2019). We do not ignore them, as we remember that the compression phase in the “fitting-compression hypothesis” can be caused by such clustering of the latent representations. However, keeping it in mind, no additional modifications to the overall experimental setup are required, as autoencoders should account for any peculiarities of the data while learning to reconstruct it. Moreover, in any case, (a) noise is widely used in modern networks for regularization purposes, (b) stochasticity still has to be introduced into non-quantized networks, otherwise it will be impossible to conduct information-theoretic analysis.

Reviewer comment (Reproducibility):
How where the confidence intervals in Fig. 5 computed? How many networks have been trained?

Author response:
Confidence intervals are computed via a standard deviation derived from the entropy estimator. Around $ 35 $ networks were trained in total. For the final plot, only one network has been used.

Reviewer comment (Evaluation):
In Fig. 6, why is there so much variation/so large CIs at the end of training?

Author response:
We kindly ask for clarification of the question: all the networks used to acquire Figure 6 are autoencoders, which were trained before the MI evaluation on synthetic examples. Perhaps, the question is about Figure 5. In such case, the variation actually stayed almost the same throughout the training; we believe an increased density of points near the end of training might give the impression of an increased variance.

Reviewer comment (Evaluation):
In the same figure, why is there no decrease of "loss delta" at the end of training?

Author response:
This happens because of the NN's loss reaching a plateau at the end of the training.

Reviewer comment (Presentation):
In the same figure, does "loss delta" refer to the difference of losses between two consecutive epochs?

Author response:
Yes, we refer to it as “loss function delta per epoch” at the end of page 9. We will define it more clearly in the next revision to avoid confusion.

Reviewer comment (Evaluation):
In the same figure, the data processing inequality seems to be violated for $I(L;Y)$, see $L_2$ vs. $L_5$. Do you have an explanation for this?

Author response:
As we plot estimates of MI, violation of DPI ranging within CIs is expected. If this violation is systematic, it may be attributed to more refined latent representations at the final layers of the network, which probably allow for a better MI estimation.

Reviewer comment (Reproducibility):
A lot of details of the proposed method are missing.

Author response:
To compress synthetic data (Figures 3-4) and input data in experiments with DNN classifier, we use CNN autoencoders (AEs) trained to minimize MAE (yields less blurry results compared to MSE).
We also use PCA — linear AE with MSE loss (allows for analytic expression of optimal compression mapping) to compress outputs of layers in DNN classifier experiment.
We will add the requested details and more information on used networks to the Appendix in the next revision.

   Statement 3 shows (under corresponding assumptions) the connection between loss of MI under compression and AE manifold learning capabilities (this connection is further investigated in Corollary 7 via deriving explicit bounds on loss of MI for PCA). Thus, the performance of the method is as sensitive to the AEs architecture as the reconstruction error is. Because reconstruction error is easy to track, AE architecture can be relatively easily validated or adjusted to match the needs of the task. We believe this makes our method more robust compared to other NN-based MI estimation approaches, where compression is assumed to be performed implicitly and thus cannot be validated (like MINE).

   We also provide several results regarding much simpler architecture of AE — linear (PCA), see Appendix, Figure 6. As this figure shows, even PCA is capable of achieving almost the same quality as nonlinear AEs in the tests used in Section 5. One can consider it to be a somewhat practical evidence of low sensitivity to AE architecture. However, it is also possible to construct an adversarial example when the AE with insufficiently complex architecture fails (see Figure 6 (c)); such cases correspond to large reconstruction errors, which is a reliable signal to consider other AE architectures.

Reviewer comment (Reproducibility):
What are the structures of the auto-encoder used to compress the data, and how are they trained? Is the result sensitive to the choice of the auto-encoder structure or some hyper-parameters? In algorithm 1, $E_X$, $E_Y$ of the auto-encoder is assumed to satisfy conditions of Corollary 1, so the auto-encoder is an important detail of the proposed method and should be discussed more in the paper or appendix. If the performance is sensitive to the details of auto-encoder choice and requires a lot of tuning, it will be a major drawback of the proposed method.

Author response:
To compress synthetic data (Figures 3-4) and input data in experiments with DNN classifier, we use CNN autoencoders (AEs) trained to minimize MAE (yields less blurry results compared to MSE).
We also use PCA — linear AE with MSE loss (allows for analytic expression of optimal compression mapping) to compress outputs of layers in DNN classifier experiment.
We will add the requested details and more information on used networks to the Appendix in the next revision.

   Statement 3 shows (under corresponding assumptions) the connection between loss of MI under compression and AE manifold learning capabilities (this connection is further investigated in Corollary 7 via deriving explicit bounds on loss of MI for PCA). Thus, the performance of the method is as sensitive to the AEs architecture as the reconstruction error is. Because reconstruction error is easy to track, AE architecture can be relatively easily validated or adjusted to match the needs of the task. We believe this makes our method more robust compared to other NN-based MI estimation approaches, where compression is assumed to be performed implicitly and thus cannot be validated (like MINE).

   We also provide several results regarding much simpler architecture of AE — linear (PCA), see Appendix, Figure 6. As this figure shows, even PCA is capable of achieving almost the same quality as nonlinear AEs in the tests used in Section 5. One can consider it to be a somewhat practical evidence of low sensitivity to AE architecture. However, it is also possible to construct an adversarial example when the AE with insufficiently complex architecture fails (see Figure 6 (c)); such cases correspond to large reconstruction errors, which is a reliable signal to consider other AE architectures.

Reviewer comment (Experiments):
The proposed method is only applied to one real data example, the MNIST data. More experiments and analysis would improve the significance of the paper.

Author response:
This is a valid point, and we are planning to perform more experiments with other neural networks and real datasets. However, we consider the proposed compression step and corresponding theoretical (bounds on MI under compression) and practical (tests on synthetic datasets with known MI) results to be the main contribution of our work. We use the experiment with the DNN classifier as a sanity check, trying to reproduce the compression-fitting pattern on the information plane.

Reviewer comment (Reproducibility):
Readers are directed to the source code about these details, but at least some more details of the proposed method should be provided in the paper or appendix.

Author response:
To compress synthetic data (Figures 3-4) and input data in experiments with DNN classifier, we use CNN autoencoders (AEs) trained to minimize MAE (yields less blurry results compared to MSE).
We also use PCA — linear AE with MSE loss (allows for analytic expression of optimal compression mapping) to compress outputs of layers in DNN classifier experiment.
We will add the requested details and more information on used networks to the Appendix in the next revision.

   Statement 3 shows (under corresponding assumptions) the connection between loss of MI under compression and AE manifold learning capabilities (this connection is further investigated in Corollary 7 via deriving explicit bounds on loss of MI for PCA). Thus, the performance of the method is as sensitive to the AEs architecture as the reconstruction error is. Because reconstruction error is easy to track, AE architecture can be relatively easily validated or adjusted to match the needs of the task. We believe this makes our method more robust compared to other NN-based MI estimation approaches, where compression is assumed to be performed implicitly and thus cannot be validated (like MINE).

   We also provide several results regarding much simpler architecture of AE — linear (PCA), see Appendix, Figure 6. As this figure shows, even PCA is capable of achieving almost the same quality as nonlinear AEs in the tests used in Section 5. One can consider it to be a somewhat practical evidence of low sensitivity to AE architecture. However, it is also possible to construct an adversarial example when the AE with insufficiently complex architecture fails (see Figure 6 (c)); such cases correspond to large reconstruction errors, which is a reliable signal to consider other AE architectures.

Reviewer comment (Theory):
In section 3.1, it says "This hypothesis is believed to hold for a wide range of structured data, and there are datasets known to satisfy this assumption precisely." Are there any references to the hypothesis or some examples of datasets satisfying this assumption precisely? Some details would help understand the hypothesis.

Author response:
In the introduction (page 2) we mention the work of Fefferman et al. (2013) [1]. This work provides a more formal and profound analysis of the manifold hypothesis (MH). We will add the citation to Section 3.1 as well.

   One could think of photogrammetry (3D object reconstruction based on series of photos) as a good example of precise (up to imperfections in the setup) satisfaction of MH, as all the photos can be parametrized by a relative position and orientation of the camera. In particular, one can consider ENRICH dataset [2]. We also note NNs trained on such datasets, see [3].

Reviewer comment (Theory):
The stochastic NN is introduced to help compute the mutual information. Does it also affect model training? If not, what is the relationship of a stochastic NN and a regular trained NN?

Author response:
Yes, stochasticity affects training of NN. Injecting random noise is widely considered as a way to increase model performance and generalization capabilities. For example, dropout layer [4] is widely used to regularize NNs. We also mention Gaussian dropout, which is known to benefit generalization even more than binary dropout [5]. Previous works on information bottleneck analysis also argue that using stochastic NNs does not have negative impact on training (see Goldfeld et al. (2019), Adilova et al. (2023)). In previous and in our work, stochastic NNs are used as a proxy to analyze deterministic NNs.