Paper: Mutual Information Estimation via $f$-Divergence and Data Derangement Based Learning Models (ICLR-2024)

Reviewer comment (Novelty):
Inadequate literature coverage and missing baselines. Technically the author(s) are proposing a GAN-like MI estimator, where an f-divergence-based function is optimized to learn the density ratio, then plug in that density ratio estimate into the MI definition to approximate the MI. This is similar in spirit to the DIM work [1], where the JSD objective (also an f-divergence) was used. Also, the idea of dropping the partition function to reduce estimation variance for MI has also been explored in the work of FLO [2], where similar Fenchel duality is applied to cast the log(partition) into a nested optimization problem. These closely related works should be discussed and compared. Also highly relevant is the work of RenyiCL [3], where skewed Renyi divergence is used as a more generic mutual information measure that enjoys bounded variance for numerical estimation. 

[1] R. Hjelm, et al. LEARNING DEEP REPRESENTATIONS BY MUTUAL INFORMATION ESTIMATION AND MAXIMIZATION. ICLR 2019  
[2] Q Guo, et al. Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization. NeurIPS 2022  
[3] K Lee and J Shin. RényiCL: Contrastive Representation Learning with Skew Rényi Divergence. NeurIPS 2022

Author response:
Despite the fact that [2] and [3] highlighted by the Reviewer are very recent papers, we also came to know about them recently. They are not cited in our paper because they are non-directly connected with our contribution. In particular, it is true that the work in [1] utilizes the JSD (which is only a particular case of our more general formulation) to maximize the MI. But it is also true that for their application,  they do not need to explicitly estimate the MI value, which is instead the purpose of our paper. Moreover, they also rely on Nowozin et. al., 2016 results as we do. Thus, we thought it was enough for us to cite only the latter since this is the starting point. 
For what matters FLO [2], it is true that those authors used the Fenchel duality for deriving their bound (it is very similar to TUBA Poole et. al, 2019 just in a contrastive fashion, and we cited InfoNCE). However, we do not see any connection with our variance analysis as the partition term in FLO naturally remains for the MI estimation (sum over marginal samples in eq. (8) and (9) of [2]). 
Lastly, while the RenyiCL estimator [3] represents a promising approach, especially the skew divergence analysis, it is fundamentally different from our methodology as we extract the critic to compute the MI making the variance analysis in contrast to the VLB (and skew) based ones.

Reviewer comment (Experiments):
Weak experiments. The author(s) only compared different estimators on the toy multi-dimensional Gaussian data, which is inadequate by today's standard. The variational MI estimators are extensively used in self-supervised representation learning (e.g., CLIP, SimCLR, etc.), and the current work failed to provide convincing evidence that the proposed estimator can reliably handle high-dimensional complex data in such applications.

Author response:
We understand the importance of such applications, and we indeed demonstrated the effectiveness of our MI estimators in high dimensional scenarios in Fig.11 in the Appendix. 
We want to remark that most of the papers in the literature, including the three cited by the Reviewer, utilize the multi-dimensional Gaussian setup to test their estimators as the main experiment. We studied such scenarios by considering the variation of several important parameters (data dimension, batch size, architecture, $f$-function) to show that our method does not work only on cherry-picked situations. We even studied the time complexity which is often neglected. In addition, we offered the code in a unified and comprehensive manner, allowing for reproducibility, quick comparisons and further analysis. We also ran the self-consistency tests in Sec.C.2 of the Appendix.

Reviewer comment (Theory):
The statement that "However, Monte Carlo sampling renders MINE a biased estimator." is inaccurate. The bias comes from Jensen's inequality, not Monte Carlo sampling.

Author response:
The Reviewer is right and we apologize for the confusion. It is indeed true that the bias is a consequence of the gradients being biased, as the logarithmic term in front of the expectation does not allow averaging of mini-batches for training. Simply switching the logarithmic term with the expectation would violate Jensen’s inequality. We rephrase that sentence with “the logarithm before the expectation in the second term renders MINE a biased estimator.”

Reviewer comment (Theory):
The statement "Although MINE provides a tighter bound $I_{MINE} ≥ I_{NWJ}$, the NWJ estimator is unbiased." is also misleading. Both MINE and NWJ can be tight estimators. Note that MINE is an approximation to MI, while NWJ is a formal lower bound, so technically you can not compare the two with an inequality.

Author response:
We agree with the Reviewer, although for the same parameters, MINE provides tighter estimates than NWJ (see Sec. 2.2 in (Belghazi et al., 2018)). We can rephrase that sentence as “Although for a fixed $T$ MINE provides a tighter bound (formula), the NWJ estimator is unbiased.”

Reviewer comment (Writing):
The notation of permutation function $\sigma(y)$ is not common in the statistical literature and causes confusion.

Author response:
We apologize for the confusion. However, as the function $\pi(\cdot)$ was already used before, we decided to use $\sigma(\cdot)$ since it is often used for permutation (see Ch.5 of (Scheinerman, 2012)).

Reviewer comment (Theory):
"The partition function estimation $\mathbb{E}_{p_X^N p_Y^N}[\hat{R}]$ represents the major issue when dealing with variational MI estimators."

Author response:
That sentence is not directly related to the derangement solution, but rather to the two-stage procedure the Reviewer highlighted in the strengths. What we want to convey is that VLB estimators often exhibit high variance and this is mainly due to the fact that they use the variational representation of the MI also for the estimation. Meanwhile, our second step needs the critic for the MI estimation and not the full value function.

Reviewer comment (Writing):
"Viceversa, the derangement of y, denoted as $\sigma(y)$, leads to N new random pairs (xi,yj) such that $i \neq j$,$\forall i$ and $j \in {1,··· ,N}$."

Author response:
Short example: a random derangement of $\mathbf{y} = [y_1, y_2, y_3]$ is  $\mathbf{y} = [y_2, y_3, y_1]$ and by definition we ensure that no elements of $\mathbf{y}$ after derangement ends up in the same initial position.

Reviewer comment (Writing):
"Denote with K the number of points $y_k$, with $k \in {1, . . . , N }$, in the same position after the permutation (i.e., the fixed points)."

Author response:
Fixed points appear only with permutations. In the previous example, a random permutation of $\mathbf{y} = [y_1, y_2, y_3]$ is $[y_2, y_3, y_1]$, and the number of fixed points is 0 as no elements remain in the same position after the permutation. However, another random permutation could be $[y_1, y_3, y_2]$ and in this case $y_1$ is a fixed point, the only one, thus in this example $K=1$.

Reviewer comment (Novelty):
The relation between f-MINE proposed in this paper and previous estimators is still not explained very clearly.

Author response:
It is correct that our framework utilizes the $f$-divergence to construct a family of MI estimators. However, we do not see such framework as a generalization of MINE for the following reasons:
1. MINE is based on the DV representation of the MI and it is structurally different from an $f$-divergence (it cannot be obtained starting from  a particular $f$-divergence)
2. MINE and other similar variational lower bounds based estimators (NWJ, SMILE, etc.), utilize the value function to compute the MI, and they suffer from known problems such as high variance, explained in eq. (14) and shown in Fig. 5 
3. The name of the proposed family of estimators is f-DIME (not f-MINE), and this choice comes from how they work: they use an f-divergence loss function to train a discriminator. At convergence, they use such a discriminator to extract the likelihood and compute the MI.

Reviewer comment (Evaluation):
It is good to explore more about the bias, variance, and the derangement strategy.

Author response:
Bias and variance are widely analyzed in the experimental sections of the main paper (see Sec. 6) and in the Appendix C (see Figs. 7,8,9,10, and Tabs. 1,2,3). Moreover, the variance is also theoretically analyzed in Sec. 4 as the low variance of our estimator constitutes an important strength point. The derangement strategy is also analyzed both in its theoretical section (see Sec. 5) and in the Appendix B and C (see Fig. 12). Indeed, the architecture referred to as “deranged” exploits the derangement technique and we offer comparisons with both other architectures and an architecture that simply uses permutations.

Reviewer comment (Novelty):
Will f-MINE be degenerated to traditional MINE, or other estimators reviewed in Section 2, by choosing different f? If so, then f-MINE is a more general framework. If not, which estimators can be degenerated from f-MINE and which ones cannot? It is good to make these clear in order to connect f-MINE with previous works.

Author response:
As mentioned before, MINE cannot be obtained from $f$-DIME as $f$-DIME utilizes a variational representation of the $f$-divergence for training, meanwhile most of the variational lower bound (VLB) estimators proposed so far in literature utilize a variational representation of the MI for training and inference. One could say that the only case where $f$-DIME coincides with the current VLB estimators is when $f$ is the generator of the KL divergence. But even in such case, our approach uses the variational representation *only* for training, as for the MI estimation we utilize the optimal likelihood ratio (see Th. 1). In conclusion, $f$-DIME is indeed a general framework, but structurally different from the SOTA estimators based on the MI VLB.

Reviewer comment (Theory):
What about the bias of f-MINE? When the variance is analyzed, the readers are always curious about the bias.

Author response:
We empirically studied the bias of the f-DIME estimator for different $f$ functions in different scenarios, please see Figs.7,8,9, and Tables. 1,2. Lemma 3 also provided useful mathematical insights about the effect of $f$ on the bias. We decided to focus an entire section on the variance analysis as the variance represents the major issue for discriminative MI estimators (see Sec. 4 of (Song & Ermon, 2020)).

Reviewer comment (Experiments):
The experiments in Section 6 are all based on the derangement strategy proposed in Section 5. In order to prove the usefulness of derangement, it is good to compare the results without derangement.

Author response:
The Reviewer is right to say that it is interesting to compare the architecture based on derangement with the one without. As a matter of fact, we did such a comparison in the Appendix, please see Fig. 12 where we show how the architecture based on simple permutations leads to bounded estimations, in accordance with the main findings of Lemma 2, Th. 2 and Cor. 2.1

Reviewer comment (Experiments):
The experiments included in the main text are quite limited since they consider solely simple unimodal distributions. The reported experimental evidence seems limited to support the claim that f-DIME offers higher accuracy when compared to other estimators in the literature.

Author response:
1. We used the same experimental setting as the most cited papers in the neural MI estimation domain (Song & Ermon, 2020; Poole et al., 2019), to compare our proposed estimator with the state-of-the-art by using the same benchmark. We studied the performance behavior when varying $d$ and $N$, the consistency tests with image datasets, and the computational time (as stated by the Reviewer in the strengths). We also provide a ready-to-use code that unifies all the discussed estimators in the current paper.

Reviewer comment (Novelty):
The paper mentions the similarity between SMILE, GAN-DIME (and JS in [1]), further discussion is required to better underline the differences between the estimators, especially considering that the performance seems hardly distinguishable.

Author response:
**Comparison with JS-like estimators**

We thank the Reviewer for having appreciated our detailed answers.
The reference “(6)” was referred to the mathematical formula of $I_{SMILE}$ in the old pdf version, since our answer to Reviewer rzBW was posted on 18 Nov at 16.48h, while we updated the pdf (as the Reviewer noticed) on 19 Nov at 22.55h. The same formula is identified as (31) in Appendix B. We provide a more detailed explanation of the sentence “setting to a small value does not significantly impact the MI estimate obtained by using only the first term in (6)” in the following. 

The “first term” in (31) (old (6)) is the expectation over $p_{XY}(\mathbf{x}, \mathbf{y})$. When $\tau$ is small (e.g., $\tau=1$), $e^{\tau}$ and $e^{-\tau}$ are also small (e.g., $e$ and $\frac{1}{e}$). Since the second term of (31) (old (6)) was proven to be characterized by high-variance in (Song & Ermon, 2020), especially for high MI values,  the clipping function helps in controlling the variance of $I_{SMILE}$, keeping it low. Because of the clipping, for high MI values the second term marginally contributes to the MI estimation. 

We will include the more detailed comparison between GAN-DIME, JS, and SMILE as we reported in the previous answer to the Reviewer. However, we think that the space limitations force us to write such a paragraph in the Appendix.

Reviewer comment (Novelty):
Although the proposed method is novel in the context of mutual information estimation, the paper builds upon a well-established theory on the estimation of f-divergences. The comparison between the use of permutations and derangements to produce samples from $p({\bf x})p({\bf y})$ is also novel although similar methods have been already used in practice in existing implementations.

Author response:
3. Since the Reviewer recognizes the novelty of the $f$-divergence formulation for MI estimation and the theoretical analysis that we provide on the derangement strategy, we see this more as a strength rather than weakness. In addition, to the best of our knowledge, no paper has neither theoretically analyzed the impact of derangements on neural MI estimators nor provided an upper bound when using random permutations.

Reviewer comment (Presentation):
Many relevant parts of the theory and the experimental results are not included in the main text. I would recommend restructuring the submission by potentially moving detailed descriptions of existing estimators into the appendix.

Author response:
4. We decided to include the previous work in the main text because it helps the reader understand better our novel contribution. However, following the Reviewer suggestion, we will move part of the related work in the Appendix to include Lemma 3 in the main text.

Reviewer comment (Theory):
The analysis in section 4 focuses on characterizing the variance of the f-DIME estimator (equation 12), but the convergence of the ratio estimator $T({\bf x},{\bf y})$ is determined by the variance of the training objective in equation 10. Does the training variance for the objective in 10 relate to the sub-optimality (bias) of the estimator?

Author response:
1. The Reviewer is right, as in practical scenarios, there is always a numerical bias coming from sub-optimality. The training variance of eq. (10) has an impact on it, but mostly its derivative as it appears in the gradient descent method. More precisely, in Lemma 3, we study the effect of not reaching optimality in the MI estimator and show that a discrepancy in the optimal density ratio (or log-density ratio) reflects into a discrepancy of the instantaneous MI by a factor $\delta^{(n)} \cdot \Biggl[\frac{\mathrm{d}}{\mathrm{d}T} \log \bigl( \bigl(f^{*}\bigr)^{\prime}(T)\bigr) \biggr|_{T=\hat{T}} \Biggr]$, where $\hat{T}(x) = f^{\prime} \bigl(\hat{R}(x)\bigr)$, as in eq.(38). In particular higher variance in the batch gradient estimates have an impact on $\delta^{(n)}$ and this term is modulated (multiplied) by a factor dependent on $f$.

Reviewer comment (Experiments):
What are the main differences between GAN-DIME/JS and KL-DIME/NWJ? I believe that a more direct experimental comparison could provide additional value to the submission.

Author response:
2. The major difference between GAN-DIME/JS and KL-DIME/NWJ is the estimator itself, since the class of $f$-DIME estimators does not rely on the estimation of the partition function (as explained in Sec. 4). The difference between GAN-DIME/JS and KL-DIME/NWJ is significant when looking at the graphs in (Poole et al., 2019; Song & Ermon, 2020) and the ones in our paper, because the variance of $f$-DIME is significantly lower. For instance, by comparing Figs. 1 and 5 (achieved using the same parameters) the variances of KL-DIME and GAN-DIME are extremely lower w.r.t. NWJ and SMILE with $\tau=\infty$. Differently, SMILE with $\tau=1$ has a variance comparable to GAN-DIME, showing the importance of the variance reduction that is obtained by reducing the effect of the partition function.

Reviewer comment (Theory):
The authors claimed that their derangement training strategy achieves the unbounded MI estimation. However, I couldn't found out the theory and proof to justify this claim unless I'm misreading. I can only see that they only showed that the permutation bound is bounded by $\log N$. I read the experimental part, but could not find anything to support this claim either. At this stage, the contributions mentioned in the introduction do not align with the content, and I am compelled to express concerns regarding the validity of this research paper.

Author response:
We appreciate that the Reviewer was hopeful about the unboundedness of the proposed estimator and we want to assure the Reviewer that there is no trick, as the paper does actually show that it is possible to overcome the $\log N$ upper bound via derangements, as discussed in the paper with both theoretical arguments (Lemma 2, Th. 2 and Cor. 2.1 and their proofs ) and experimental results (Figs 1, 2,3 and Fig.12 in the Appendix).  

Firstly, The hypothesis of the main theorem of D. McAllester et. al. do not apply to the proposed estimator ($f$-DIME) with the derangement strategy. The main idea in that theorem is that, given two distributions $p$ and $q$, it is difficult to distinguish samples of $q$ from samples of another skew distribution $\tilde{q} = 1/N p + (1-1/N)q$. Therefore, when we evaluate the KL divergence between $p$ and  $\tilde{q}$, it is immediate to notice that such divergence is upper bounded by $\log N$. However, this is not our case. In fact, let us suppose to have only access to $N$ joint samples $(x,y)\sim p_{XY}$, shuffling $y$ to obtain marginal samples would lead, with high probability (please see Lemma 5 in the Appendix), to one sample $(x_i,y_i)$ which is a joint sample, falling in a similar condition of the one studied by D. McAllester et. al. 
Given that we have access only to $N$ samples, how can we construct marginal samples from joint samples? The answer is derangements, since by definition, it is guaranteed that no $y$ sample will end up in the same position when shuffled, avoiding to fall back into a skew distribution $q$, which would lead to the $\log N$ bound. 

Secondly, please note that Lemma 2 provides an inequality between the two value functions, with permutations and with derangements. In particular, we proved that the value function with permutations is upper bounded by the one with derangements. An even further mathematical connection can be obtained when studying the estimator with the permutation strategy as a sort of variational skew-divergence estimator (see Lee and Shin, 2022), but this goes beyond the scope of this paper.

Thirdly, the experimental results do not show the existence of any bound in our case (see Fig. 1, 2, and 3). As a matter of fact, Fig. 12 in the Appendix also shows the boundedness of the estimator when using the permutations instead of the derangements, which is in line with our theoretical result in Corollary 2.1. 

Now, based on the Reviewer's comment, we decided to slightly rephrase our contribution claim in the introduction as follows “We study the impact of data derangement for the learning model and propose a novel derangement training strategy that overcomes the $\log N$ upper bound on the mutual information estimation (D. McAllester et. al.), contrarily to what happens when using a random permutation strategy.” 

Please also observe that our contribution in this respect is twofold: a) it shows the boundedness of the f-DIME estimator when using permutations to obtain marginal samples; b) it presents a strategy and an architecture to overcome such bound. 

In conclusion, with all the respect, we believe that grading the “soundness” with 1 represents a rather unfair rating. Hopefully, with the clarification above the Reviewer agrees with us.   

[Ref] Lee, Kyungmin, and Jinwoo Shin. "RényiCL: Contrastive representation learning with skew rényi divergence." Advances in Neural Information Processing Systems 35 (2022): 6463-6477.

Reviewer comment (Theory):
Is it possible to overcome the $\log N$ upper bound on the mutual information neural estimation through the maximization of the variational lower bound, perhaps by employing a simple technique such as derangements? The recent paper [1] has shown that, not just CPC, **any distribution-free high-confidence lower bound on mutual information estimated from $N$ samples cannot be larger than $\mathcal{O}(\log N)$**. When I first read the preface of this paper and found the claim that unbounded estimation had been achieved despite such serious statistical limitations, I was both surprised and hopeful. However, I am disappointed to discover that there is no theoretical basis or related discussion to support this assertion.

Author response:
We appreciate that the Reviewer was hopeful about the unboundedness of the proposed estimator and we want to assure the Reviewer that there is no trick, as the paper does actually show that it is possible to overcome the $\log N$ upper bound via derangements, as discussed in the paper with both theoretical arguments (Lemma 2, Th. 2 and Cor. 2.1 and their proofs ) and experimental results (Figs 1, 2,3 and Fig.12 in the Appendix).  

Firstly, The hypothesis of the main theorem of D. McAllester et. al. do not apply to the proposed estimator ($f$-DIME) with the derangement strategy. The main idea in that theorem is that, given two distributions $p$ and $q$, it is difficult to distinguish samples of $q$ from samples of another skew distribution $\tilde{q} = 1/N p + (1-1/N)q$. Therefore, when we evaluate the KL divergence between $p$ and  $\tilde{q}$, it is immediate to notice that such divergence is upper bounded by $\log N$. However, this is not our case. In fact, let us suppose to have only access to $N$ joint samples $(x,y)\sim p_{XY}$, shuffling $y$ to obtain marginal samples would lead, with high probability (please see Lemma 5 in the Appendix), to one sample $(x_i,y_i)$ which is a joint sample, falling in a similar condition of the one studied by D. McAllester et. al. 
Given that we have access only to $N$ samples, how can we construct marginal samples from joint samples? The answer is derangements, since by definition, it is guaranteed that no $y$ sample will end up in the same position when shuffled, avoiding to fall back into a skew distribution $q$, which would lead to the $\log N$ bound. 

Secondly, please note that Lemma 2 provides an inequality between the two value functions, with permutations and with derangements. In particular, we proved that the value function with permutations is upper bounded by the one with derangements. An even further mathematical connection can be obtained when studying the estimator with the permutation strategy as a sort of variational skew-divergence estimator (see Lee and Shin, 2022), but this goes beyond the scope of this paper.

Thirdly, the experimental results do not show the existence of any bound in our case (see Fig. 1, 2, and 3). As a matter of fact, Fig. 12 in the Appendix also shows the boundedness of the estimator when using the permutations instead of the derangements, which is in line with our theoretical result in Corollary 2.1. 

Now, based on the Reviewer's comment, we decided to slightly rephrase our contribution claim in the introduction as follows “We study the impact of data derangement for the learning model and propose a novel derangement training strategy that overcomes the $\log N$ upper bound on the mutual information estimation (D. McAllester et. al.), contrarily to what happens when using a random permutation strategy.” 

Please also observe that our contribution in this respect is twofold: a) it shows the boundedness of the f-DIME estimator when using permutations to obtain marginal samples; b) it presents a strategy and an architecture to overcome such bound. 

In conclusion, with all the respect, we believe that grading the “soundness” with 1 represents a rather unfair rating. Hopefully, with the clarification above the Reviewer agrees with us.   

[Ref] Lee, Kyungmin, and Jinwoo Shin. "RényiCL: Contrastive representation learning with skew rényi divergence." Advances in Neural Information Processing Systems 35 (2022): 6463-6477.

Reviewer comment (Presentation):
The theoretical analysis in Sections 4 and 5 is the most crucial part of this paper as it provides strong support for the favorable properties of the method proposed in this paper. Nevertheless, the presentation in these sections introduces assumptions, notations, and proof sketches abruptly before delving into the discussion of the results obtained. This made it quite challenging to grasp the intent and content when I first read it. I believe that it would be challenging for readers to comprehend the content on first encounter if the section's opening does not provide an outline of what is to be demonstrated and the results obtained before sharing the proof's details.

Author response:
- Considering that the primary interest is the effective and precise communication of our findings, we updated the manuscript in this direction. In particular, given the 9 pages constraint (very little), we decided to add another section in the Appendix where we explain with some examples, for the more interested reader, the idea behind the derangement (which is probably difficult to grasp the first time).
- Based on your comment, we also added at the beginning of Sec. 5 the intuition behind the derangement idea and why such a strategy could be effective in reducing the $\log N$ bound. We decided to incorporate part of our previous reply as we believe it really provides added value.

Reviewer comment (Writing):
I believe that readability is a crucial element in effectively presenting the main theoretical claims. Currently, it is necessary to read the content multiple times to understand it to some extent, which can lead to the final intended message becoming unclear. Additionally, I'm concerned about important theoretical aspects being placed in the appendix. For instance, isn't Lemma 3 an important property of the proposed estimator?

Author response:
- Considering that the primary interest is the effective and precise communication of our findings, we updated the manuscript in this direction. In particular, given the 9 pages constraint (very little), we decided to add another section in the Appendix where we explain with some examples, for the more interested reader, the idea behind the derangement (which is probably difficult to grasp the first time).
- Based on your comment, we also added at the beginning of Sec. 5 the intuition behind the derangement idea and why such a strategy could be effective in reducing the $\log N$ bound. We decided to incorporate part of our previous reply as we believe it really provides added value.
- We deeply thank the Reviewer for allowing us to reconsider the relevance of Lemma 3. As promised, we added this Lemma in the main text and we also made a small comment explaining why it is important. Please find it (new Lemma 1) in the updated version of the paper.

Reviewer comment (Evaluation):
I also have concerns regarding the validity of interpreting the experimental results. The authors claim that from the experimental results presented in Figure 1, their proposed f-DIME estimator achieves lower variance compared to conventional methods (MINE, NWJ, SMILE). However, at least to me, it doesn't seem like there is a significant difference. I've also examined the experimental results in Fig. 2-5 and the Appendix, but it appears that there are no results provided to substantiate this claim. Even when reviewing Table 1, which provides quantitative evaluations, it is clear that C PC consistently achieves good performance in terms of variance, which contradicts this claim entirely. I hope it's my misinterpretation, but it's quite challenging to accept this claim as valid.

Author response:
For what matters the concern about the variance of the proposed estimator, with all the respect, we believe the Reviewer may have misinterpreted the results. We  explain our reasons below:
The variance analysis (Sec. 4) compares $f$-DIME and the variational lower bound estimators. This result is theoretically proved and experimentally verified, as it can be observed by comparing the behavior of $f$-DIME estimators with MINE, NWJ, and SMILE (with $\tau=\infty$) (Fig. 5). We believe that the Reviewer might have overlooked this figure as the variance difference with the results obtained in Fig. 1 is clearly visible.

Reviewer comment (Evaluation):
The precision of the f-DIME estimator doesn't seem to be consistently good overall. However, there's no mention of this, and in fact, it's claimed to be robust to changes in data size and dimension. To me, it appears that NJEE and SMILES are rather stable. For instance, in Fig. 2, the accuracy of HD-DIME w./ separable in the cubic case seems to deteriorate compared to Fig. 1. It's challenging to confirm the proposed method's exceptional robustness based on these experimental results.

Author response:
- Thanks for sharing with us your impression. We agree that the results of Fig. 1 (now Fig. 2) might not seem different at a first glance. However, they look convincing (at least in terms of variance, as mentioned in Sec. 6.2) when compared with the estimators in Fig. 5 (now 6). Please note that we started from this setup as this is the setup used also in other established works (see Song & Ermon, 2020; Poole et al., 2019). Thus, it is fair to offer the comparison in the same scenario. 
Nonetheless, Fig. 2 (now Fig. 3) and 3 (now Fig. 4) are there on purpose to show to the reader that in non cherry-picked scenarios, for either different dimension $d$ or different batch size $N$, $f$-DIME is more robust and exhibits excellent performance, as observed in Sec. 6. To precisely answer the Reviewer and for completeness, we also included the associated Table of Fig.1 (now Fig. 2), with the exact numerical values. See Table 3 in the Appendix.

Reviewer comment (Presentation):
I recommend adhering to the ICLR format. The authors seem to have mistakenly used a different font style from the original ICLR template. This is evident, for instance, in the title where the font appears distinct. Furthermore, there also seems to be a slightly wider character spacing throughout the document. This could be a candidate for a desk reject.

Author response:
- Yes correct, thanks for noticing this. We corrected it in the updated version.
- We totally agree with the Reviewer. Please find in the system an updated version of the manuscript that reflects the Reviewers’ comments.

Reviewer comment (Evaluation):
My main concern comes from the claim that the new estimator has best or excellent bias/variance trade-off, which is not clear to me.

Author response:
1. The bias shown in all the figures is a consequence of the non-optimal convergence of the density ratio estimate to the true density ratio while trying to maximize (10).  Lemma 3 studies the effect of this discrepancy and shows that the convergence of $f$-DIME depends on the choice of the $f$-divergence. In particular, from eq. (75) it can be observed that the discrepancy between the true instantaneous MI and the estimated one is controlled by the product between $\delta^{(n)}$ and $\Biggl[\frac{\mathrm{d}}{\mathrm{d}T} \log \bigl( \bigl(f^{*}\bigr)^{\prime}(T)\bigr) \biggr|_{T=\hat{T}} \Biggr]$, where $\hat{T}(x) = f^{\prime} \bigl(\hat{R}(x)\bigr)$. Thus, the latter factor given a certain $f$ is the one to be analyzed since it impacts the gradient descent method. However, since our estimator does not provide a lower bound on the real MI (differently from the others), it cannot be asserted with certainty that the bias decreases with lower MI value, as this does not appear to be always the case (see Fig.1 for GAN-DIME or Fig.11 for HD-DIME).

Reviewer comment (Evaluation):
According to Figs. 1 and 2, the HD-DIME and KL-DIME do not perform that well, which usually suffer from large bias than NJEE and SMILE. Moreover, it seems there is no consistent winner for all f-divergence MI estimators that always performs better than others.

Author response:
2. We have proposed a class of estimators, for which we studied the properties that hold for any choice of $f$-divergence. It is true, as the Reviewer states, that HD-DIME and KL-DIME do not achieve excellent performance in all the scenarios, but the proposed GAN-DIME does. In fact, among all the considered estimators, GAN-DIME attains better or, in few cases, comparable results. Moreover, GAN-DIME is more robust w.r.t. NJEE. In fact, when the latent dimension of the input distribution or the batch size vary, we show in Fig. 2,3 and Tables 1,2 that the performance of NJEE significantly worsens, while the performance of GAN-DIME does not. In addition, as we explain in Sec. 6.2, as the latent dimension increases, the time complexity of NJEE significantly increases, making it an infeasible MI estimator. Differently, GAN-DIME is extremely fast, when using the deranged or separable architectures. The performance of SMILE and GAN-DIME is similar since, as we state in Sec. 6.2 and Appendix C.1.1, SMILE is trained with a JS maximization trick that is equivalent to the GAN-DIME training objective (this is not mentioned in the original paper).

Reviewer comment (Evaluation):
Hence, it is hard to say which estimator really achieves a good bias-variance tradeoff; given that some other estimators also perform well, e.g., NJEE.

Author response:
2. We have proposed a class of estimators, for which we studied the properties that hold for any choice of $f$-divergence. It is true, as the Reviewer states, that HD-DIME and KL-DIME do not achieve excellent performance in all the scenarios, but the proposed GAN-DIME does. In fact, among all the considered estimators, GAN-DIME attains better or, in few cases, comparable results. Moreover, GAN-DIME is more robust w.r.t. NJEE. In fact, when the latent dimension of the input distribution or the batch size vary, we show in Fig. 2,3 and Tables 1,2 that the performance of NJEE significantly worsens, while the performance of GAN-DIME does not. In addition, as we explain in Sec. 6.2, as the latent dimension increases, the time complexity of NJEE significantly increases, making it an infeasible MI estimator. Differently, GAN-DIME is extremely fast, when using the deranged or separable architectures. The performance of SMILE and GAN-DIME is similar since, as we state in Sec. 6.2 and Appendix C.1.1, SMILE is trained with a JS maximization trick that is equivalent to the GAN-DIME training objective (this is not mentioned in the original paper).

Reviewer comment (Theory):
Are there ways to give more insights on the bias of the new estimator, especially demonstrating why it has smaller bias when the true MI value is also small?

Author response:
1. The bias shown in all the figures is a consequence of the non-optimal convergence of the density ratio estimate to the true density ratio while trying to maximize (10).  Lemma 3 studies the effect of this discrepancy and shows that the convergence of $f$-DIME depends on the choice of the $f$-divergence. In particular, from eq. (75) it can be observed that the discrepancy between the true instantaneous MI and the estimated one is controlled by the product between $\delta^{(n)}$ and $\Biggl[\frac{\mathrm{d}}{\mathrm{d}T} \log \bigl( \bigl(f^{*}\bigr)^{\prime}(T)\bigr) \biggr|_{T=\hat{T}} \Biggr]$, where $\hat{T}(x) = f^{\prime} \bigl(\hat{R}(x)\bigr)$. Thus, the latter factor given a certain $f$ is the one to be analyzed since it impacts the gradient descent method. However, since our estimator does not provide a lower bound on the real MI (differently from the others), it cannot be asserted with certainty that the bias decreases with lower MI value, as this does not appear to be always the case (see Fig.1 for GAN-DIME or Fig.11 for HD-DIME).

Reviewer comment (Evaluation):
Given the inconsistent performance of different $f$-divergence MI estimator, how can we choose the suitable one in practice?

Author response:
3. In an unknown scenario, we recommend the usage of GAN-DIME (as discussed in Appendix C.1.5). The rationale behind this choice is that it achieves excellent performance (in terms of bias and variance) for both low and high values of true MI (see Tables 1 and 2 in Appendix C). Table 3 provides a good overview of the proposed estimators strengths, suggesting which ones can be used for low or high MI and which one can be scaled for high dimensions (See summary in Sec. C.1.5 of the Appendix).