Paper: Cauchy-Schwarz Divergence Information Bottleneck for Regression (ICLR-2024)

Reviewer comment (Novelty):
In Remark 3, the authors clearly state that a kernel width of $\sigma=0$ reduces the CS divergence between the true posterior and the decoder distribution to classical MSE. With this in light, it is not clear in what respect CS divergence is fundamentally different from estimating $I(y;t)$ using a Gaussian variational distribution (limits novelty). Also, it is not clear why the CS-IB outperforms other methods also for $r=0$.

Author response:
## [Q1 The fundamental difference between (conditional) CS divergence and $I(y;t)$ with Gaussian variational assumption]

In fact, “CS+Gaussian” and “KL+Gaussian” are fundamentally different. Please refer to our reply to your [Q2].

We now reply to your second question "why the CS-IB outperforms others also for $r=0$". In fact, when the compression ratio $r=0$, all competing methods do not have the compression term, our CS-IB optimizes $D_{CS}(p(y|x);q_\theta(\hat{y}|x))$ which reduces to MSE when kernel size $\sigma=0$; whereas other methods optimize MSE directly. For our method, we never select kernel size $\sigma=0$. This is because by our analysis in **Appendix B.4** in the revised submission, $\sigma=0$ significantly increases the estimation variance (which make the results highly unstable).

There are two reasons why we are better: 1) we do not make parametric assumption on $q_\theta(\hat{y}|x)$. According to our analysis in **Appendix B.2**, a parametric assumption like Gaussian might not be the optimal choice, and the estimation bias is somewhat hard to control. In fact, our performance is better further justifying that there is a large bias introduced by the Gaussian assumption. Note that, for variational KL divergence estimator, the bias is exactly $D_{KL}(p(y|x);q_\theta(\hat{y}|x))$ or $D_{KL}(p(y|t);q_\theta(\hat{y}|t))$ (please refer to Eq.(96) in **Appendix B.2**).

Another reason why $D_{KL}(p(y|x);q_\theta(\hat{y}|x))$ performs better than MSE when $\sigma\neq0$ is because, CS divergence encourages that if two points $x_i$ and $x_j$ are sufficiently close, their predictions ($\hat{y}_i$ and $\hat{y}_j$) will be close as well, which essentially enhances the numerical stability of the trained model. When $\sigma=0$, there is no such regularization.

Reviewer comment (Theory):
In Remark 4, the authors claim that the use of KL divergence combined with a Gaussian assumption is "likely to induce inductive bias". The same can be said about using CS divergence with Gaussian kernels (which resonates with Remark 3, for which CS divergence reduces to MSE for $\sigma=0$). Specifically, at a very superficial glance it appears that the inductive biases of KL+Gauss and CS+Gauss are not too different. I would appreciate a paragraph explaining the fundamental differences (from the perspective of inductive bias).

Author response:
## [Q2 The inductive bias between KL+Gaussian and CS+Gaussian]
In fact, “KL+Gaussian” and “CS+Gaussian” are fundamentally different: the former is a parametric estimator that assumes the underlying distribution is Gaussian; whereas the latter is a non-parametric estimator that does not make any assumption on underlying distribution. A Gaussian kernel in kernel density estimator (KDE) does not mean we assume the underlying data distribution is Gaussian.

When the underlying distribution is not Gaussian, “CS+Gaussian” still works, whereas “KL+Gaussian” suffers from poor performance. We added a toy example in **Appendix C.5** to illustrate this point. 

During the rebuttal, we also systematically analyze the bias and variance of our non-parametric estimators and show that they are consistent with a rate of convergence of $N\sqrt{\sigma}$, in which $N$ is the number of samples, $\sigma$ is the kernel width. Please refer to our discussion in **Appendix B.4**.

Moreover, we also analyze the bias of previous variational KL divergence-based methods. Please refer to our discussion in **Appendix B.2**. Our argument is that previous literature either has obvious much more bias than ours or similar to us. 

To summarize, for our estimator, the asymptotic property is guaranteed. But the use of Gaussian kernel in KDE may introduce some inductive bias, since other kernels may perform better (e.g., smaller variance) or more robust, such as Epanechnikov kernel. But for “KL+Gaussian”, the Gaussian assumption alone for the underlying distribution makes the estimation is highly likely to be biased.

Reviewer comment (Theory):
While I agree that CS-QMI is a rigorous definition, I do not agree that the proposed estimator measures mutual dependence in bits. As has been known for some time (see, e.g., Th. 1 in Amjad and Geiger or the work of Goldfeld et al.), for deterministic networks true MI between $x$ and $t$ is infinite. I doubt that $D_{CS}$ remains finite in such a setting (e.g., Renyi divergence is infinite if the joint and product of marginal distributions are singular w.r.t. each other, cf. Th. 24 in van Erven and Harremoes). From that perspective, I do not see how your estimators based on CS divergence have substantial mathematical advantages over existing estimators (although I acknowledge that the estimation may still be useful from a practical perspective; also, I understand that your networks are stochastic, because $t$ is obtained by adding Gaussian noise to a decoder output).

Author response:
## [Q3 substantial mathematical advantages by CS divergence]
Thanks for the good question. First, we agree with you that in a deterministic network, the true CS-QMI between $x$ and $t$ may also become infinity. To check why, let us consider the CS-QMI between two Gaussian signals $x$ and $t$ with correlation coefficient $\rho$, in which $I_{CS}(x;t)$ is given by:
$I_{CS}(x;t) = \frac{1}{2}\log\left( \frac{4-\rho^2}{4\sqrt{1-\rho^2}} \right)$.
Suppose $x$ passes through an identity mapping, that is $t=x$, and $\rho=1$, we have $I_{CS}(x;t)\rightarrow\infty$.

However, we would like to make a clarification that the CS divergence is not a special case of the well-known Renyi’s divergence developed by Renyi with the following expression:
$D_\alpha$$(p;q) = \frac{1}{\alpha-1}\log\int p^\alpha q^{1-\alpha}du=\frac{1}{\alpha-1}\log\int p\left(\frac{q}{p}\right)^{1-\alpha}du=\frac{1}{\alpha-1}\log\mathbb{E}_p\left(\frac{q}{p}\right)^{1-\alpha}$

When $\alpha=2$, we obtain a new divergence $-\log(1-\chi^2(p;q))$ that is proportional to the $\chi^2$ divergence, rather than our CS divergence mentioned in the paper.

To our knowledge, there are two big ideas to define divergence: the integral probability metrics (IRM) aim to measure $p-q$, whereas the Renyi’s divergences with order $\alpha$ aim to model the ratio of $\frac{p}{q}$. The general idea of CS divergence is totally different: it neither horizontally computes $p-q$ nor vertically evaluates $\frac{p}{q}$.  Rather, it quantifies the “distance” of two distributions by quantify the tightness (or gap) of an inequality associated with integral of densities (i.e., $\int pq$). That is, CS divergence does not belong to either IRM or the traditional Renyi’s divergence family. Please refer to our discussion on **Appendix B.1** for more details.

This uniqueness makes CS divergence has many desirable properties compared to KL. One of them is that it relaxes the constraint on the support. Note that, $D_{KL}(p;q)$ has finite values if and only if support of $p$ is a subset of support of $q$. This condition is very strong, which also explains why we use KL divergence combined with non-parametric estimator, the model is hard to train (please also refer to our discussion in **Appendix B.3**, especially the last paragraphs marked with blue).

Finally, we would also like to mention two important and special properties of CS divergence over KL divergence (or Renyi divergence), although these two properties are not explicated used in our method, but it would be beneficial and helpful to the community. (1) the CS divergence has closed-form expression for mixture-of-Gaussians (MoG); (2) CS divergence is invariant to scaling. That is, given $\lambda_1,\lambda_2>0$, we always have $D_{\text{CS}} (p;q) = D_{\text{CS}} (\lambda_1 p;\lambda_2 q)$. Please also refer to our **Appendix D** for details.

Reviewer comment (Evaluation):
One of my main concerns is the comparison of methods in Table 2 and Fig. 2. As far as I can see, $r$ is defined via the respective estimators (i.e., CS divergences for CS-IB, variational costs for VIB, etc.). This makes a comparison difficult, as the same $r$ indeed may lead to different compression ratios when compression is measured in a common currency (e.g., CS divergence for all methods). While this is problematic only for Table 2, in Fig. 2 the same holds for $I(y,t)$. Specifically, CS-IB is the only method that does not use the MSE, for the estimation of $I(y;t)$ which explains the difference at $r=0$. Essentially, the red line may be just above all others because $I(y;t)$ is measured differently (namely, using CS divergence). Also here a common currency would be better, such as MSE or RMSE. (Further, using the RMSE would allow for a better comparison with Table 2.)

Author response:
## [Q4 Comparison of methods in Table 2 and Fig. 2]
We would like to clarify that in Fig.2, the $y$-axis that quantities $I(y;t)$ is indeed measured with a common currency (to provide a fair comparison). Specifically, we follow (Kolchinsky et al., 2019b), and measure the true value of $I(y;t)$ by $1/2 \log⁡(\text{var}(y)/MSE)$. Hence, $I(y;t)$ in Fig.2 is not measured with respective estimators of different methods.

Going back to the Table 2, yes, the compression ratio $r$ is indeed computed with respective estimators. However, we would like to clarify here that $r$ is a relative quantity, rather than an absolute quantity. Specifically, $r$ is defined as:
$r = 1- \frac{a}{b}$, in which $a=I(x;t)$ when $\beta=\beta^*$ and $b=I(x;t)$ when $\beta=0$ (i.e., no compression).

Hence, for each method, the physical meaning of $r$ is how much information has been compressed (when $\beta=\beta^*$) relatively to no compression (i.e., $\beta=0$).

We use $r$ rather than $I(x;t)$ in both Table 2 and Fig 2. This is just because different IB approaches differ in the way to (define) or estimate $I(x;t)$. Most literatures use Shannon’s mutual information (MI), our CS-QMI estimates a generalized MI. By contrast, HSIC-bottleneck uses HSIC, which does not have physical meaning of mutual information in terms of bits or nats. Hence, if we use $I(x;t)$, different approaches have different units, which may result in an unfair comparison.

Finally, we would like to emphasize that, for each method, we additionally report the best performance achieved when $r\neq0$ (i.e., $β\neq0$) in Table 7 in **Appendix C.4**. This result suggests that, under an optimal compression ratio (in which the $r$ or $\beta$ is different for different methods), our CS-IB is significantly and consistently better.

Reviewer comment (Theory):
The CS divergence is based on the Cauchy-Schwartz inequality, a special case of Holder inequality. Does this metric can be extended in the general $L_p$ space?

Author response:
Thanks for the good question. Yes, you are right, the general idea of CS divergence can be generalized to $L_p$ space by taking use of the Holder inequality:  
$|\int p(x)q(x)dx|\leq (\int|p(x)|^a d(x))^{1/a} (\int |q(x)|^b d(x))^{1/b}$, in which $a,b>1$ and $\frac{1}{a}+\frac{1}{b}=1$.  

Hence, a generalized way to quantify the closeness between $p$ and $q$ can be expressed as:  
$D = -\log\left( \frac{|\int p(x)q(x)dx|}{(\int|p(x)|^a d(x))^{1/a} (\int |q(x)|^b d(x))^{1/b}} \right)$.  

CS divergence is obtained by setting $a=b=2$. The generalized divergence can also be estimated by kernel density estimator.

Reviewer comment (Theory):
Theorem 1 is restricted to Gaussian distribution. Though this assumption can be relaxed if $p$ and $q$ are sufficiently small, it would be possible to extend to the sub-Gaussian, sub-exponential case?

Author response:
Thanks for the good suggestion. We generalized our Theorem 1 to any two square-integral densities $p$ and $q$. Please refer to **Appendix B.5** and Proposition 5 in the revised submission.

Reviewer comment (Theory):
For the generalization bound, the authors obtain a tighter bound but $I_{CS}(x;t)$ and $I(x;t)$ are still in the same constant order?

Author response:
Yes, the same constant order. By definition, $I_{CS}(x;t) = D_{CS}(p(x;t);p(x)p(t))$ and $I(x;t) = D_{KL}(p(x;t);p(x)p(t))$, our justifications in **Appendix B.5** suggest that for any two square-integral densities $p$ and $q$, $D_{\text{CS}}(p;q) \lesssim D_{\text{KL}}(p;q)$. There is no other scaling factors in this inequality. By letting $p=p(x;t)$ and $q=p(x)p(t)$, we obtain $I_{CS}(x;t) \lesssim I(x;t)$.

Reviewer comment (Theory):
The authors claim that by leveraging the CS divergence, they can perform non-linear IB regression for any distribution $p(x, y)$ without heavy reliance on variational approximations and without making distribution assumptions. However, the independence from the variational distribution seems to be achieved not so much due to the properties of the CS divergence but rather through non-parametric estimation using Kernel Density Estimation (KDE) on the outputs from neural networks, treating them as samples from the training data. Therefore, the motivation for introducing the CS divergence may lack a clear and solid rationale, leaving room for discussion regarding its validity in the context of this presentation.

Author response:
Thanks for raising such a good question. More precisely, we feel that the independence to variational approximation and distributional assumption can be attributed to the estimation of CS divergence in a non-parametric way. Per your request, we analyzed how to non-parametrically implement KL divergence-based IB approach, from which we found the model is hard to train (see our reply to your [Q1]). We also point out that one possible reason why the non-parametric estimation of CS divergence successes is maybe because CS divergence relax the constraint on the support of distributions, in which we seldom has the situation $p(x) \log(\frac{p(x)}{0}) = \infty$.

Reviewer comment (Theory):
While the authors claim that their contribution lies in the ability to estimate without making distribution assumptions, the actual theoretical analysis deals with a limited setting where both the model distribution and variational distribution are assumed to be normal distributions. This limitation is acknowledged as one of the limitations in Appendix D. However, it can be considered a significant weakness that the proposed method's theoretical guarantees are limited, given the broad claim of removing the necessity of distribution assumptions. This disparity should ideally be addressed in the main part of the paper.

Author response:
Theorem 1 can be generalized to arbitrary square-integral distributions without Gaussian assumptions. Please refer to Proposition 5 in the **Appendix B.5** of the revised submission.

Reviewer comment (Theory):
For one of the contributions mentioned, the explanation of the adversarial robustness aspect is rather too concise. Besides Lemma 1, there should be a proper proof for [Wang et al., 2021] Theorem 2's Corollary as well. In the current presentation, while one may vaguely understand the content being discussed, the explanations are too informal, making it difficult to accurately determine whether the validity is established. The theoretical part of the proof should ideally be self-contained within the main paper, rather than relying on related literature.

Author response:
Thanks for your suggestion. We added Proposition 4 in the revised submission, in which we bound demonstrate that the probability that CS-QM, i.e., $I_{CS}(x;t)$ bounds the expected value of $|h_\theta(x+\delta)-h_\theta(x)|$ is at least $0.5$ and can reach to nearly $1$ (given sufficient samples). We also provide the assumptions under which this proposition holds.

Reviewer comment (Theory):
When constructing nonparametric estimators, a critical concern always revolves around whether (asymptotic) unbiasedness or (asymptotic) consistency is guaranteed. This theoretical assurance holds true for nonparametric estimations like Kernel Density Estimation (KDE) and k-Nearest Neighbors (k-NN) based on KL divergence, among others. In the context of the IB method under focus here, unbiasedness and consistency play a significant role in determining its performance. The importance of analyzing this aspect lies in the bias introduced by parameter estimation in variational approaches dependent on the variational distribution, as well as the bias in the estimators of the proposed method. Deciding which of these biases is smaller — the one induced by parameter estimation or the one introduced by the proposed method — is a crucial perspective in determining whether a direct estimation approach, like the proposed method, or a variational estimation approach is more useful. Unfortunately, this paper lacks comprehensive discussions on this aspect and convincing arguments on why a direct estimation approach like the proposed method is effective.

Author response:
First, thanks for raising such a good question. The bias and asymptotic property of our estimator has been discussed in **Appendix B.4**. The bias of existing variational approximation is also discussed in **Appendix B.2** in the revised submission. In general, our argument is that previous methods either have much more obvious bias than ours or similar bias to us. We also provide a toy example in **Appendix C.5** to show CS divergence estimated non-parametrically performs better than the variational KL divergence.

Reviewer comment (Experiments):
When comparing the bias introduced by the variational approach with that introduced by nonparametric estimations such as the proposed method, it would be beneficial to investigate a toy example with an increased sample size to determine which approach exhibits higher estimation accuracy. This paper did not provide such experimental confirmations.

Author response:
First, thanks for raising such a good question. The bias and asymptotic property of our estimator has been discussed in **Appendix B.4**. The bias of existing variational approximation is also discussed in **Appendix B.2** in the revised submission. In general, our argument is that previous methods either have much more obvious bias than ours or similar bias to us. We also provide a toy example in **Appendix C.5** to show CS divergence estimated non-parametrically performs better than the variational KL divergence.

Reviewer comment (Theory):
The theory in Section 3.2.1 assumes that the generalization error upper bound is derived in a manner that includes the objective function of the proposed method, which alone does not guarantee generalization performance. The attempt to experimentally verify this aspect is interesting, but the empirical correlations have only been validated on toy data. To increase persuasiveness regarding the correlation with generalization performance, it would have been necessary to confirm the experimental results on benchmark data adopted in Section 4.

Author response:
Thanks for the good suggestion. We justified the positive correlation on California housing data and also trained nearly 100 models with various depth, width, or mini-batch size. The results are demonstrated in Table 1 and Fig. 12. The experimental details are described in **Appendix C.1**.

Reviewer comment (Evaluation):
Due to the absence of reports on measures of dispersion, such as standard deviations, for all the experimental results, it is difficult to determine whether the proposed method consistently achieves superior performance compared to other methods. This raises concerns about the validity of the claim that CS-IB outperforms other methods. Given that the proposed method is built on nonparametric estimation, the variance of the estimates may become large when there are not enough samples, leading to increased performance variability. Supporting the claim of superiority requires considering factors such as randomness of model initialization, mini-batch datasets, and adversarial perturbations. Experiments should be repeated several times, with average and standard error reported, to evaluate performance considering variability.

Author response:
We added the standard deviation in Tables 8 and 9 in **Appendix C.4**. We explained in reply to your [Q5], there is no bug in the implementation. We also explained how hyperparameters in Table 4 are obtained in reply to your [Q7]. Please also refer to **Appendix C.2**.

Reviewer comment (Evaluation):
Some of the reported numerical values in the experiments are identical in every aspect. For instance, in Table 2, the RMSE for the existing methods when $r=0$ is a complete match across all experimental settings. This further emphasizes the need to report both average prediction accuracy and standard error. Perfect matches should be checked for possible implementation bugs. As it stands, there are concerns regarding the credibility of the reported numerical values.

Author response:
First, there is no bug in our code. The performances of all existing methods are the same when the compression ratio $r=0$, i.e., no compression term exists in the IB objective, i.e., $\beta=0$. In this case, all existing methods just minimize a MSE loss (without any regularization on $I(x;t)$) with the same network architecture. Hence, their performances are the same.

Our CS-IB outperforms consistently when $r=0$. This results also demonstrate the advantage of using $D_{CS} (p(y|x);q_\theta(\hat{y}|x))$ alone over the traditional MSE loss.

Reviewer comment (Reproducibility):
There appears to be a lack of explanation regarding hyperparameter settings, such as in Table 4. This omission can give the impression that there might be arbitrariness in the experimental results. If arbitrariness has been eliminated, more detailed explanations about how hyperparameters were selected should be provided to enhance transparency and credibility.

Author response:
For each competing method, the hyperparameters are selected as follows. We first choose the default values of $\beta$ (the balance parameter to adjust the importance of the compression term) and the learning rate $lr$ mentioned in its original paper. Then, we select hyperparameters within a certain range of these default values that can achieve the best performance in terms of RMSE. For VIB, we search within the range of $\beta \in [10^{-3}, 10^{-5}]$. For NIB, square-NIB, and exp-NIB, we search within the range of $\beta \in [10^{-2}, 10^{-5}]$. For HSIC-bottleneck, we search $\beta \in [10^{-2}, 10^{-5}]$. For our CS-IB, we found that the best performance was always achieved with $\beta$ between $10^{-2}$ to $10^{-3}$. Based on this, we sweep the $\beta$ within this range and select the best one. The learning rate range for all methods was set as $[10^{-3}, 10^{-4}]$. We train all methods for $100$ epochs on all datasets except for $PM2.5$, which requires around $200$ epochs of training until converge. All the hyperparameter tuning experiments are conducted on the validation set.

Reviewer comment (Evaluation):
The improvement over existing approaches is fairly limited. In many tasks, the improvement is as small as 0.1 RMSE (where the relative improvement is close to 0.01~0.04). I am not sure if the limited improvement on these datasets is particularly meaningful.

Author response:
Yes, on some relative simpler data, our prediction performance improvement seems a bit marginal. However, we would like to emphasize the obvious performance improvement on the adversarial robustness in Table 3. 

Moreover, in the rebuttal period, we additionally added a new application: to predict the age of patients based on their brain functional MRI (fMRI) data with a graph neural network (GNN), which, to the best of our knowledge, is seldom investigated (prevalent literatures always rely on structural MRI (sMRI) data with convolutional neural networks) and is a much more challenging task. Our new results in  **Appendix D** also demonstrate an obvious performance gain. Moreover, the success of our method in GNN also showcase the broader applicability and generalizability of our estimator.

Reviewer comment (Theory):
It remains conceptually unclear to me why we want to use the Cauchy-Schwarz divergence. It is shown to be always <= KL divergence, but it is not clear how much smaller would this be (maybe only minimally).

Author response:
$D_{KL}(p;q)$ is easy to reach infinity if the support of $p$ is not a subset of the support of $q$, which means that $D_{KL}(p;q)$ only has finite values if $\text{supp}(p) \subseteq \text{supp}(q)$.
In fact, even if the support of $p$ and $q$ has large overlap, but $\text{supp}(p) \not\subset \text{supp}(q)$ (this situation is common in practice), KL is still infinity. Given two square integral densities $p$ and $q$, its value is infinite for CS divergence if and only if there is no overlap on the supports, i.e., $\text{supp}(p) \cap \text{supp}(q) = \emptyset$. In this sense, the CS divergence actually relaxes the support constraint of the KL divergence. Please also refer to our **Appendix B.1** for more details.

Reviewer comment (Theory):
Could the authors provide a high-level idea/motivation in the Introduction to describe why one should consider the Cauchy-Schwarz divergence instead of the more traditional KL divergence? Why is it better to avoid variational approximations?

Author response:
Thanks for the good question. 

On the one hand, in **Appendix B.2**, we show that the approximation error of variational approximation is hard or intractable to control, or obvious larger than ours. By contrast, our estimation error and other properties can be fully analyzed as shown in **Appendix B.4**. That is, our estimator provides better theoretical guarantee. 

On the other hand, some intrinsic limitations of KL divergence, such as support constraint, also motivates us to use other divergence. However, we would also like to emphasize here that the CS divergence also has some other desirable mathematical properties over KL divergence (as has been discussed in **Appendix D**), which we feel should be interesting and helpful to community.

Reviewer comment (Experiments):
Are there toy problems where Cauchy-Schwarz divergence performs significantly better than other traditional approaches?

Author response:
Thanks for the good question. We consider two toy examples in the revised submission in **Appendix C.5**. The first example demonstrates when the true underlying distribution is a mixture of Gaussian, our CS divergence optimization performs better than the reverse KL divergence with variational approximation. The second example demonstrates that CS-QMI (the Cauchy-Schwarz quadratic mutual information) has similar or slightly better ability to identify complex dependence patterns compared with popular approaches like HSIC and distance covariance (dCov).

Reviewer comment (Experiments):
The CS divergence estimation is based on the Gaussian kernel assumption, which will depend on the parameter $\sigma$. What is the effect of $\sigma$ to the IB performance is not shown clearly.

Author response:
## [Q2 The effect of $\sigma$ to IB performance]
We demonstrate in Fig. 19 in the **Appendix C.4**, showing that our estimators provide consistent performance gain in a reasonable range of kernel size. 

Additionally, we also demonstrate in **Appendix B.4** that the effects of $\sigma$ to the bias and variance of our CS divergence estimator. The conclusion is that estimation bias is shrinking at a rate of $O(\sigma^2)$, and the variance shrinks at rate $O(1/(N^2\sigma))$. So there is a trade-off between bias and variance, which makes sense.

Reviewer comment (Experiments):
The KL IB using variational approach is friendly to optimization based methods (gradient-based approaches). On the other hand, CS IB method is based on Gaussian kernel assumption, which may require the tuning of $\sigma$.

Author response:
## [Q3 The tuning of hyperparameter $\sigma$]
We partially agree with this point. For the basic variational information bottleneck (VIB) based on KL divergence, it is indeed very friendly for optimization. However, we would like to emphasize here that, for advanced KL-based IB approaches such as NIB, square-NIB and HSIC-bottleneck, they all require a careful treatment on the choice of some hyperparameter similar to kernel size $\sigma$. Different to these approaches, we systematically analyzed how $\sigma$ affects the bias and variance of our estimator in **Appendix B.4**. To our knowledge, such analysis is missing in previous literature.

Reviewer comment (Evaluation):
I think it’s better to have a section of identifying what are some potential disadvantages of CS-IB method compared to KL-IB method.

Author response:
## [Q4 Potential disadvantage of CS-IB]
Thanks for the good question. We would like to mention a limitation of CS divergence with respect to the KL divergence. That is, we did not find a dual representation for the CS divergence. The dual representation of KL divergence, such as the well-known Donsker-Varadhan representation, enables the use of neural networks to estimate the divergence itself, which significantly improves estimation accuracy in high-dimensional space. A notable example is the mutual information neural estimator (MINE). We leave it as a future work and mention this point in **Appendix D**.

Reviewer comment (Theory):
The complexity of CS-divergence estimation approach is O(N^2). How is it compared to that of the traditional KL divergence based IB method?

Author response:
## [Q1 The computational complexity of traditional KL-divergence based IB method]
Thanks for the good question. For HSIC-bottleneck, the dependence between $x$ and $t$ is measured with HSIC, which also takes a computational complexity of $O(N^2)$. For nonlinear information bottleneck (NIB) and square-NIB and exp-NIB, $I(x;t)$ is upper bounded as:
$I(x;t) \leq -\frac{1}{N} \sum_{i=1}^N \log \frac{1}{N} \sum_{j=1}^N \exp\left( -\frac{\|h_i-h_j\|_2^2}{2\sigma^2} \right)$.
in with $h$ is the hidden activity corresponding to each input sample, i.e., $t$ equals $h$ plus Gaussian noise (to avoid infinite mutual information value). Hence, the complexity is also $O(N^2)$.