Paper: Deep Variational Multivariate Information Bottleneck - A Framework for Variational Losses (ICLR-2024)

Reviewer comment (Experiments):
To better assess the performance of DVSIB, it would be very useful to compare it against previous work that also rely on multimodal information-theoretical measures, such as [1] using a multiinformation bottleneck.

Author response:
$\textbf{Comparison to other methods:}$ We appreciate your recommendation to compare against various multimodal architectures and evaluate cross-modal consistency. We had already included multimodal architectures such as CCA and its variants even in the original submission. Furthermore, we now integrated the architectures cited in your suggestions into our methods. As we have stated in the general response, it is important to note that our focus lies in the framework itself, enabling the integration and derivation of various methods. We do not aim for state-of-the-art classification accuracy against much better tuned and bigger AI systems. Yet, even with these caveats, our findings already indicate improved accuracy when the method's structure aligns with the expected statistical dependencies in the dataset.

Reviewer comment (Evaluation):
To better assess the performance of DVSIB, it would be very useful to evaluate not just whether the latents can be used for classification, but also the quality and cross-model consistency of the reconstructed images, for example following standard multi-modal evaluation measures [2].

Author response:
$\textbf{Comparison to other methods:}$ We appreciate your recommendation to compare against various multimodal architectures and evaluate cross-modal consistency. We had already included multimodal architectures such as CCA and its variants even in the original submission. Furthermore, we now integrated the architectures cited in your suggestions into our methods. As we have stated in the general response, it is important to note that our focus lies in the framework itself, enabling the integration and derivation of various methods. We do not aim for state-of-the-art classification accuracy against much better tuned and bigger AI systems. Yet, even with these caveats, our findings already indicate improved accuracy when the method's structure aligns with the expected statistical dependencies in the dataset.

Reviewer comment (Experiments):
To better assess the performance of DVSIB, it would be very useful to consider additional multi-modal datasets beyond noisy MNIST.

Author response:
$\textbf{Evaluation on other datasets:}$ Expanding our method's evaluation to other datasets, indeed, could strengthen our work. To address this, we applied DVSIB to the MIR-Flick dataset (please the the general response). However, the caveats remain that it is hard to benchmark a new, developing method, against state-of-the-art systems, fine-tuned using computational resources not generally available to us.

Reviewer comment (Experiments):
Can the approach be generalised to more than two modalities?

Author response:
$\textbf{Including more than two modalities:}$ Our approach can, indeed, extend to include multiple modalities. We have updated the appendix to include implementations of such architectures, demonstrating the straightforward derivation of loss functions for multimodal architectures mentioned in the literature involving more than two modalities. And yet we can still use the elements of the code that we used in the other methods as well, ensuring a straightforward generalization towards crafting new methods that address specific research questions.

Reviewer comment (Experiments):
How are the $\beta$ values in Table 2 tuned? Does it make sense to use different $\beta$ values for evaluating the classification accuracy for different methods, as I would guess that different $\beta$s impact how much information is encoded into the latent variables.

Author response:
$\textbf{Tuning of $\beta$:}$  We perform a sweep across the specified range of $\beta$, selecting the value that corresponds to the highest classification accuracy. We document these sweeps comprehensively in the supplementary data (in the provided Methods Results.ipynb notebook). The varying $\beta$ values indeed impact the information content differently for each model, necessitating tailored beta values for maximizing their potential and ensuring a fair comparison between different methods.

Reviewer comment (Experiments):
The evaluations and benchmarking felt limited, for a few reasons that I will detail in the next few comments. First, the MNIST example feels somewhat contrived, and MNIST in general has become something of a toy dataset. A more complex, more realistic application dataset would make the applicability of DVSIB more clear.

Author response:
$\textbf{Limited experimental evaluation:}$ Addressing the weaknesses you pointed out, particularly the limited evaluation, we want to reiterate that the core of our contribution lies in the framework itself. We aim to demonstrate how this framework provides unification, generalization, and how it facilitates the creation of new methods. We consider Table 1, as much a  primary result of our work as Table 2, which outlines how different algorithms perform on the MNIST data. We have never aimed to achieve state-of-the-art classification accuracy against much bigger and more fine-tuned learning systems. Tishby (one of the inventors of the now so popular Information Bottleneck framework) famously noted that the "tyranny of percent correct" limits the adoption of conceptual advances in machine learning since new, developing concepts cannot immediately compete with well-tuned older methods. We hope that the Reviewers will share this sentiment with us. Nevertheless, we are now including benchmarking of our DVSIB against well-tuned other methods on the MIR-Flicker dataset (see general response for details).

Reviewer comment (Experiments):
Even on the MNIST application, I felt that the benchmarking was insufficient. In particular, in Figure 3 and Table 2 it seems like there is almost no penalty for making $\beta$ huge as long as it is large enough, in which case the penalty on the encoder essentially does not matter. In the VAE setup this would cause the encoder to concentrate on the MAP instead of the posterior, essentially reverting to an Auto-Encoder. Are only the means of the variational distributions used in the downstream classification task? Can the authors perform any benchmarking where the probabilistic interpretation on the latent space matters?

Author response:
$\textbf{Choice of $\beta$:}$ The reviewer questioned the lack of performance penalty for large $\beta$ values. Our current understanding of this phenomenon is as follows. The training drives the decoder to maximize the information gathered from the obtained samples, influencing the reconstruction process significantly. Specifically, our approach involves starting with a batch of pairs of data points, feeding them into the encoder graph, and obtaining probability distributions of $p(z_X|x)$ and $p(z_Y|y)$. Subsequently, we sample from these distributions to learn the decoder. The parameter $\beta$ plays a crucial role in the decoder's behavior. A larger $\beta$ prompts the decoder to grasp as much information as possible from the samples $z_X$ and $z_Y$, pushing for maximal information retention in the reconstruction process, given the variational model and the dimensionality of the latent representations. Since the decoder has $I(Z_X, Z_Y)$ term, even at very large $\beta$, and with variational and dimensional constraints, the methods, therefore, $\textit{does not}$ revert to an autoencoder since not all of the features of the original samples of views are useable for predicting the representation of the other view.

Reviewer comment (Reproducibility):
In order to compute the mutual information between $Z_X$ and $Z_Y$, the authors essentially use an energy-based model, learning $T(z_x, z_y)$ as an unnormalized log-likelihood (up to multiplication by the marginals). Additional details on how the normalizing constant, $Z_\text{norm}$, is computed or approximated are warranted.

Author response:
$\textbf{Estimating mutual information in the latent space:}$ Many estimators can be used to compute the information between $Z_X$ and $Z_Y$. The only requirement is that the estimator is differentiable. However, as we discussed above, it is important to use an estimator that "matches" the specific task at hand. As we hinted above, we leave the detailed analysis of which estimators should be used when to future work. Finally, in the revision we provided more explicit details regarding the computation of the normalizing constant, $\mathcal{Z}_\text{norm}$, as the Reviewer has requested.

Reviewer comment (Novelty):
The novelty of the proposed Deep Variational Symmetric Information Bottleneck seems quite limited since the objective is quite similar to the existing literature and the main differences are not clearly underlined in the main text.

Author response:
$\textbf{Novelty:}$ We believe that an important contribution of our work is in providing a conceptually clean way to unify a number of dimensionality reduction methods within the information-theoretic language. For example, trade-off parameters that often need to be added in an $\textit{ad-hoc}$ manner with other derivations naturally arise using our framework. As we emphasized in our previous responses, the Deep Variational Symmetric Bottleneck is meant to be one of the applications of our more general framework. It produces two latent variables that retain maximal information about one another, rather than about the the individual views of the data. This is a main distinction of this specific method from the others we reviewed.  Part of the loss of DVSIB anchors the $Z_X$ and $Z_Y$ via decoders to $X$ and $Y$. This is one main difference as compared to your Ref.[2]. Our method produces a generative model for $X$ and $Y$ at the same time as producing $Z_X$ and $Z_Y$ maximally informative of each other. Ref.[2], on the other hand, makes distributions for mapping of the latent variables $Z_X$ and $Z_Y$ as close to one another as possible. In fact, it enforces that $Z_X$ and $Z_Y$ have the same domain. This is different from DVSIB where the latent variables can have different units, dimensions, and domains. Please see our general response for details. Finally, we now include performance of the loss in Ref.[2]  loss in our main results table for comparison, and we thank the Reviewer for the suggestion that we should compare to this and other additional methods.

Reviewer comment (Experiments):
The paper introduces a framework that can in principle be applied to complex graphical models involving multiple variables, but the experimental section (and most of the method) solely focuses on a two-variable system that has been widely explored in the literature.

Author response:
$\textbf{Additional evaluation:}$ We now evaluate our method using the MIR-FLICKR dataset. Please see the general response for the details of what we did. Further, we would like to clarify that the noisy MNIST dataset that we used for our main experiments was not given the data labels. The label information, of course, is indirectly captured in the dataset structure by having pairs of images with the same label. This is precisely the main point of DVSIB and related multiview methods:  if there is some relationship between $X$ and $Y$, we can recover it without telling our model about the relationship (aka, without labeling). We were able to demonstrate that DVSIB  was able to do just that by elevating the performance of a linear SVM to predict the labels from the latent space. As far as we understand, this is a standard way of testing representation learning algorithms (see e.g., Wang et al., 2015; 2016).

Reviewer comment (Experiments):
The experiments revolve solely around the MNIST dataset. Further, the paper claims that "none of the algorithms were given the data labels" even though the training pairs are constructed by pairing digits with the same label. As a result, label information is indirectly captured in the dataset structure.

Author response:
$\textbf{Additional evaluation:}$ We now evaluate our method using the MIR-FLICKR dataset. Please see the general response for the details of what we did. Further, we would like to clarify that the noisy MNIST dataset that we used for our main experiments was not given the data labels. The label information, of course, is indirectly captured in the dataset structure by having pairs of images with the same label. This is precisely the main point of DVSIB and related multiview methods:  if there is some relationship between $X$ and $Y$, we can recover it without telling our model about the relationship (aka, without labeling). We were able to demonstrate that DVSIB  was able to do just that by elevating the performance of a linear SVM to predict the labels from the latent space. As far as we understand, this is a standard way of testing representation learning algorithms (see e.g., Wang et al., 2015; 2016).

Reviewer comment (Experiments):
The paper lacks common baselines based on contrastive learning that can be applied in the same settings [1,2,3]. In particular [2] proposes a similar loss function and demonstrates similar performance without using the labels for pairing images.

Author response:
$\textbf{Novelty:}$ We believe that an important contribution of our work is in providing a conceptually clean way to unify a number of dimensionality reduction methods within the information-theoretic language. For example, trade-off parameters that often need to be added in an $\textit{ad-hoc}$ manner with other derivations naturally arise using our framework. As we emphasized in our previous responses, the Deep Variational Symmetric Bottleneck is meant to be one of the applications of our more general framework. It produces two latent variables that retain maximal information about one another, rather than about the the individual views of the data. This is a main distinction of this specific method from the others we reviewed.  Part of the loss of DVSIB anchors the $Z_X$ and $Z_Y$ via decoders to $X$ and $Y$. This is one main difference as compared to your Ref.[2]. Our method produces a generative model for $X$ and $Y$ at the same time as producing $Z_X$ and $Z_Y$ maximally informative of each other. Ref.[2], on the other hand, makes distributions for mapping of the latent variables $Z_X$ and $Z_Y$ as close to one another as possible. In fact, it enforces that $Z_X$ and $Z_Y$ have the same domain. This is different from DVSIB where the latent variables can have different units, dimensions, and domains. Please see our general response for details. Finally, we now include performance of the loss in Ref.[2]  loss in our main results table for comparison, and we thank the Reviewer for the suggestion that we should compare to this and other additional methods.

Reviewer comment (Evaluation):
The qualitative visualization relies solely on t-SNE even if there is evidence to support that t-SNE visualization could be misleading [4].

Author response:
$\textbf{Concerns about visualization:}$ We recognize that t-SNE is not infallible for showing evidence of clustering or of a quality of the latent space. Hence we use t-SNE for qualitative visualization only, and our main results are quantified instead by training an SVM on our learned latent space for label identification. Our SI includes several plots of visualizations of our method reduced to two dimensions where t-SNE is not used, yielding qualitatively similar results.

Reviewer comment (Writing):
The paper is trying to solve problems that are in the domain of probabilistic modeling and variational inference, but uses techniques based on information theory. For readers that are less familiar with information theory, it would help to have a paragraph that explains more how these methods relate to the literature on VAEs and variational infernece.

Author response:
$\textbf{Presentation:}$ We thank the Reviewer for their suggestions for improving the paper. Existing literature for autoencoders and VAEs typically uses maximum likelihood methods for deriving the loss functions. Here we use an information bottleneck approach to achieve the same thing. The Information in the Decoder graph is maximized, which corresponds to maximizing the log-likelihood, and the information in the encoder graph is minimized corresponding to the approximate posterior.  We follow the notation used by Alemi et al. in "Deep Variational Information Bottleneck", which has become a $\textit{de facto}$ standard in the context of information-based variational models.  Thus we use $p$ to represent the "true" distributions and $q$ and $r$ as variational approximations to the true distributions. We understand that this is not the notation that is used commonly in the VAE literature, which makes our paper difficult to read for some members of the community.  However, for some other members, e.g., Reviewer 4, this choice seems to have made the presentation clear. We have made the choice to stay with the information-based notation. However, to make our work accessible to the other half of the community, we added additional details in the main text describing our notation and how our framework relates to the literature on VAEs. We hope that this will make the paper easier to read for everyone.

Reviewer comment (Experiments):
The experimental results are not very strong in my opinion. First of all, since this is not mainly a theory paper, I feel like the authors should experiment on more than one dataset.

Author response:
$\textbf{Additional experiments:}$ We agree with the Reviewer that evaluation of DVSIB on more complex datasets is welcome. Crucially,  we want to reiterate that we do not intend to compete with the state-of-the-art classification methods, trained using much bigger networks, on hardware beyond those available to us, and hence with the ability to fine-tune all algorithmic parameters. As we explain in our general response, we provide the current implementation of DVSIB as a proof of concept, hoping to illustrate that, even without fine-tuning and large-scale computing, its performance is already promising. To this end, we now evaluate DVSIB on an additional, more complex dataset: the MIR-FLICKR dataset. Please see the general response for more details of what we've done. We also conducted additional experiments on other algorithms suggested by this Reviewer and other Reviewers. The main Table in the manuscript already included several methods that computed one or more latent variable per view. including DVCCA and its variants. The table now includes performance using the loss from "[1] Learning robust representations via multi-view information bottleneck" paper, among others suggested by the Reviewers.

Reviewer comment (Experiments):
Ideally, some of these datasets would be more sophisticated than MNIST. I feel like this method would be very useful for researchers in biology or neuroscience, perhaps exploring applied problems in these fields would make the paper stronger.

Author response:
$\textbf{Additional experiments:}$ We agree with the Reviewer that evaluation of DVSIB on more complex datasets is welcome. Crucially,  we want to reiterate that we do not intend to compete with the state-of-the-art classification methods, trained using much bigger networks, on hardware beyond those available to us, and hence with the ability to fine-tune all algorithmic parameters. As we explain in our general response, we provide the current implementation of DVSIB as a proof of concept, hoping to illustrate that, even without fine-tuning and large-scale computing, its performance is already promising. To this end, we now evaluate DVSIB on an additional, more complex dataset: the MIR-FLICKR dataset. Please see the general response for more details of what we've done. We also conducted additional experiments on other algorithms suggested by this Reviewer and other Reviewers. The main Table in the manuscript already included several methods that computed one or more latent variable per view. including DVCCA and its variants. The table now includes performance using the loss from "[1] Learning robust representations via multi-view information bottleneck" paper, among others suggested by the Reviewers.

Reviewer comment (Experiments):
I am not entirely sure if the set of baselines is the best one. For example, the new method is the only one which defines two separate latents for each of the two views of the data. Is the improvement in performance attributed to the fact that each view gets its own latent (which is not a novel idea from this paper; there are other methods that do this), or to the specific way in which the method generates these latents. In order to determine this, another baseline that computes one latent variable per view would be helpful.

Author response:
$\textbf{Additional experiments:}$ We agree with the Reviewer that evaluation of DVSIB on more complex datasets is welcome. Crucially,  we want to reiterate that we do not intend to compete with the state-of-the-art classification methods, trained using much bigger networks, on hardware beyond those available to us, and hence with the ability to fine-tune all algorithmic parameters. As we explain in our general response, we provide the current implementation of DVSIB as a proof of concept, hoping to illustrate that, even without fine-tuning and large-scale computing, its performance is already promising. To this end, we now evaluate DVSIB on an additional, more complex dataset: the MIR-FLICKR dataset. Please see the general response for more details of what we've done. We also conducted additional experiments on other algorithms suggested by this Reviewer and other Reviewers. The main Table in the manuscript already included several methods that computed one or more latent variable per view. including DVCCA and its variants. The table now includes performance using the loss from "[1] Learning robust representations via multi-view information bottleneck" paper, among others suggested by the Reviewers.

Reviewer comment (Experiments):
In particular, what if I were to fit a VAE-type model with two latents $Z_X, Z_Y$ and two observed variables $X, Y$ such that the q and the p have the same independence structure as DVSIB in Table 1. Would this approach be equivalent to DVSIB? If yes, there should be a discussion. If they are not equivalent, then the VAE-type model should be a baseline (and there should still be a discussion of the pros/cons of each approach).

Author response:
$\textbf{More on relations to VAEs:}$ With regards to your question, "If I were to fit a VAE-type model with two latents $Z_X$, $Z_Y$ and two observed variables $X$, $Y$ such that the $q$ and the $p$ have the same independence structure as DVSIB'', we believe that this would be the same as our approach, with just a different name for the same concept. (This is as long as there is the same dependence between $Z_X$ and $Z_Y$ as shown in our model of DVSIB. If there is no dependence then, the structure you described is the same as two separate VAEs, one for $X$ and one for $Y$). This joint encoding, while keeping information between the compressed variables is crucially different from parallel VAEs in our opinion, as we have emphasized in the general response.

Reviewer comment (Experiments):
How does the method compare to VAE-type model with two latents $Z_X, Z_Y$ and two observed variables $X, Y$ such that the q and the p have the same independence structure as DVSIB in Table 1?

Author response:
$\textbf{More on relations to VAEs:}$ With regards to your question, "If I were to fit a VAE-type model with two latents $Z_X$, $Z_Y$ and two observed variables $X$, $Y$ such that the $q$ and the $p$ have the same independence structure as DVSIB'', we believe that this would be the same as our approach, with just a different name for the same concept. (This is as long as there is the same dependence between $Z_X$ and $Z_Y$ as shown in our model of DVSIB. If there is no dependence then, the structure you described is the same as two separate VAEs, one for $X$ and one for $Y$). This joint encoding, while keeping information between the compressed variables is crucially different from parallel VAEs in our opinion, as we have emphasized in the general response.