Paper: The Representation Jensen-Shannon Divergence (ICLR-2024)

Reviewer comment (Theory):
The proposed divergence measure, RJSD, is regarded as a discrepancy measure for normalized kernel embeddings using the kernel function, kappa square. I think that there is no clear reason why the kernel embedding should be normalized to be a probability density. One could assess the discrepancy between unnormalized kernel embeddings by divergences for unnormalized functions, (pseudo-)distance measures, etc. What is the benefit of thinking of the normalization or the mapping to the set of probability densities for the non-negative kernel embeddings?

Author response:
We would like to clarify that the kernel used to measure RJSD does not need to be normalized to be a probability density, and that the only restriction is that $\kappa(x,x) = 1$. In definition 1. we normalize the kernel to be a probability density, to be able to compare the covariance operator entropy $S(C_\mathbb{P})$ to Shannon's entropy  $H(p)$. Notice that the result in Theorem 1. shows that the cross-entropy $H(p,\hat{p}) = S(C_\mathbb{P}) + \log(h)$, where $h$ is the normalization constant. However, $S(C_\mathbb{P})$ is independent of the normalization constant and does not require the kernel to be a density function to be computed. Additionally, notice that in Theorem 4 (Theorem 5 in the revised document), we establish the conditions under which $D_{JS}(\mathbb{P},\mathbb{Q}) = D_{JS}^\phi(C_\mathbb{P},C_\mathbb{Q}) $, and notice, that in this case, the normalization constant is not involved in this equivalence. In fact, one of the main benefits of the covariance operator entropy is that it avoids the intermediate step of density estimation.

Reviewer comment (Evaluation):
In numerical experiments, the authors observed that learning algorithms based on the proposed divergence measure outperform existing methods. However, any insight into the numerical results is not sufficiently mentioned. For example, GAN using RJSD is found to be robust to the mode collapse. What makes the RJSD so reliable?

Author response:
The RJSD formulation has several advantages. First, in GANs, the problem of mode collapse is usually attributed to using a classifier to measure the divergence since the generator can learn to create a few realistic samples to fool the discriminator and trivially minimize the loss. Besides, since the discriminator handles each example independently, there is a lack of coordination between the gradients, and consequently, the generator is not penalized by producing similar samples. On the contrary, our framework processes the covariance matrices of entire batches of real and generated data rather than individual samples. This leads to coordinated gradients to either increase or decrease the divergence between the distributions. Second, RJSD avoids entropy regularizers to avoid mode collapse. Specifically, RJSD explicitly matches the entropy of the generated samples and the training distribution. This feature penalizes generative distributions with lower entropy than the real distribution.

Reviewer comment (Experiments):
The data size in numerical experiments is not necessarily large. When the proposed method is used for larger data set, how much is the computation cost?

Author response:
In section 2.2 we mention that the cost of the kernel-based estimator is $\mathcal{O}(N^3)$, and the covariance-based is $\mathcal{O}(D^3)$. The cost of the kernel-based estimator is not really prohibitive, since we are able to compute eigendecompositions of up to 20-30K samples easily on modern computers. However, the covariance-based estimator is independent of sample size, which makes this estimator even more appealing, since even for larger amounts of data, we can estimate this quantity faster. In future work, we aim to test the proposed estimators in large scale datasets.

Reviewer comment (Writing):
The paper frequently uses notation and terminology that has not been previously defined, and seems to assume some familiarity with quantum mechanics/quantum information theory. Reading this paper with backgrounds in information theory and in functional analysis, but not in quantum information theory, I think I was able to make the right educated guesses to understand most of what the paper is doing, but this guesswork shouldn't be necessary to read the paper, and I don't think this would be accessible to much of the ICLR community.

Author response:
*Issues about confusing notation.* We appreciate your thorough review and feedback on the clarity of notation and terminology used in our paper. Your observations regarding the potential challenges for readers without a background in quantum mechanics/quantum information theory are valuable. In response to your concerns, we revised the manuscript to enhance the clarity of notation and provide more detailed explanations of relevant operations such as the tensor product $\otimes$, nuclear norm $\lVert \cdot \rVert_*$, Hilbert-Schmidt norm $\lVert \cdot \rVert_{HS}$, among others. Regarding the bra-ket notation, we changed it to a more familiar notation in Machine learning, using a more standard inner product instead avoiding any confusion among the readers. We also defined properly entropy and cross-entropy and added a footnote to define a density matrix.

Reviewer comment (Writing):
Bra-ket notation is used for inner products (e.g., in Definition 1 and Theorem 4) without any introduction. I happen to have heard of this notation, but have never seen it in a paper, and doubt many people in ICLR are familiar with it.

Author response:
*Issues about confusing notation.* We appreciate your thorough review and feedback on the clarity of notation and terminology used in our paper. Your observations regarding the potential challenges for readers without a background in quantum mechanics/quantum information theory are valuable. In response to your concerns, we revised the manuscript to enhance the clarity of notation and provide more detailed explanations of relevant operations such as the tensor product $\otimes$, nuclear norm $\lVert \cdot \rVert_*$, Hilbert-Schmidt norm $\lVert \cdot \rVert_{HS}$, among others. Regarding the bra-ket notation, we changed it to a more familiar notation in Machine learning, using a more standard inner product instead avoiding any confusion among the readers. We also defined properly entropy and cross-entropy and added a footnote to define a density matrix.

Reviewer comment (Writing):
Section 3, just before Definition 2, "The Quantum counterpart of the Jensen-Shannon divergence (QJSD) between density matrices $\rho$ and $\sigma$ is defined as... where $S(\cdot)$ is von Neumann’s entropy.": What is a "density matrix"?

Author response:
*Issues about confusing notation.* We appreciate your thorough review and feedback on the clarity of notation and terminology used in our paper. Your observations regarding the potential challenges for readers without a background in quantum mechanics/quantum information theory are valuable. In response to your concerns, we revised the manuscript to enhance the clarity of notation and provide more detailed explanations of relevant operations such as the tensor product $\otimes$, nuclear norm $\lVert \cdot \rVert_*$, Hilbert-Schmidt norm $\lVert \cdot \rVert_{HS}$, among others. Regarding the bra-ket notation, we changed it to a more familiar notation in Machine learning, using a more standard inner product instead avoiding any confusion among the readers. We also defined properly entropy and cross-entropy and added a footnote to define a density matrix.

Reviewer comment (Writing):
Page 4, First Sentence: The univariate function $H$ is never defined; only the bivariate cross-entropy function was introduced in Eq. (6).

Author response:
*Issues about confusing notation.* We appreciate your thorough review and feedback on the clarity of notation and terminology used in our paper. Your observations regarding the potential challenges for readers without a background in quantum mechanics/quantum information theory are valuable. In response to your concerns, we revised the manuscript to enhance the clarity of notation and provide more detailed explanations of relevant operations such as the tensor product $\otimes$, nuclear norm $\lVert \cdot \rVert_*$, Hilbert-Schmidt norm $\lVert \cdot \rVert_{HS}$, among others. Regarding the bra-ket notation, we changed it to a more familiar notation in Machine learning, using a more standard inner product instead avoiding any confusion among the readers. We also defined properly entropy and cross-entropy and added a footnote to define a density matrix.

Reviewer comment (Presentation):
I don't understand the point of much of Section 2 (Background). For example, Theorems 1 and 2 are never referenced or related to the remainder of the paper. I suggest reducing some of this material to make space for (a) clear definitions of the notation used throughout the paper and (b) further motivation and details regarding the experiments in Section 5.

Author response:
Theorems 1 and 2 are important for establishing a connection between kernel-based entropy and Shannon's differential entropy. The convergence and consistency of RJSD are addressed later in the document where we leverage the results in Bach, (2022). We added the following:
"Leveraging the convergence results in Bach, (2022)[Proposition 7] of the empirical estimator $S(\mathbf{K_X})$ to $S(C_{\mathbb{P}})$, we can show that $D_{\scriptscriptstyle JS}^{\kappa}(\mathbf{X},\mathbf{Y})$ converges to the population quantity $D_{\scriptscriptstyle JS}^\phi(C_{\mathbb{P}}, C_{\mathbb{Q}})$ at a rate $\mathcal{O}\left(\frac{1}{\sqrt{N}}\right)$, assuming $N=M$. Details of this rate are given in Appendix A.8. Additionally, a direct consequence of Theorem 2 is that, under the same assumptions of the theorem, $D_{\scriptscriptstyle JS}^{\kappa}(\mathbf{X},\mathbf{Y})$ converges to  $D_{\scriptscriptstyle JS}(\mathbb{P},\mathbb{Q})$ as $N\rightarrow \infty$ with probability one."
We acknowledge that this might not have been expressed clearly, and we enhanced the clarity of our paper by explicitly stating these two different results.

Reviewer comment (Theory):
The abstract and contributions section both claim that the paper presents consistency and convergence rate guarantees for the proposed estimators, but I couldn't find any of these guarantees in the paper. The only guarantee presented (Theorem 2) applies only for finite-dimensional feature representations ($D < \infty$), so it doesn't seem to apply to the new estimators proposed.

Author response:
Theorems 1 and 2 are important for establishing a connection between kernel-based entropy and Shannon's differential entropy. The convergence and consistency of RJSD are addressed later in the document where we leverage the results in Bach, (2022). We added the following:
"Leveraging the convergence results in Bach, (2022)[Proposition 7] of the empirical estimator $S(\mathbf{K_X})$ to $S(C_{\mathbb{P}})$, we can show that $D_{\scriptscriptstyle JS}^{\kappa}(\mathbf{X},\mathbf{Y})$ converges to the population quantity $D_{\scriptscriptstyle JS}^\phi(C_{\mathbb{P}}, C_{\mathbb{Q}})$ at a rate $\mathcal{O}\left(\frac{1}{\sqrt{N}}\right)$, assuming $N=M$. Details of this rate are given in Appendix A.8. Additionally, a direct consequence of Theorem 2 is that, under the same assumptions of the theorem, $D_{\scriptscriptstyle JS}^{\kappa}(\mathbf{X},\mathbf{Y})$ converges to  $D_{\scriptscriptstyle JS}(\mathbb{P},\mathbb{Q})$ as $N\rightarrow \infty$ with probability one."
We acknowledge that this might not have been expressed clearly, and we enhanced the clarity of our paper by explicitly stating these two different results.

Reviewer comment (Evaluation):
Given the lack of supporting theoretical results (previous point), I was not very convinced by the experimental results. All of them are on simple, mostly synthetic datasets. While RJSD does perform well on many of the tasks, the improvements are mostly small, and, more importantly, there is no discussion of *why* RJSD might perform better than other metrics on these tasks.

Author response:
Having shown the supporting theoretical results of the proposed RJSD estimator (Points 2 and 3),  we appreciate your feedback regarding the clarity of the performance improvement over other metrics. In response, we would like to highlight Theorem 5 (Theorem 6 in the revised manuscript) in our paper, where we establish that RJSD serves as an upper bound on MMD. This result implies that if MMD can effectively distinguish between two distributions, then RJSD is expected to have similar or superior discriminative power. We argue in the paper that this theoretical relationship suggests that RJSD should be at least as good as MMD in identifying discrepancies between distributions. While it is true that the observed performance gain may not bet substantial, deep kernel MMD is a state-of-the-art approach. In this context, even marginal improvements in performance on the evaluated datasets are noteworthy. To address your concern, in the revised paper, we elaborate further on why RJSD has a superior test power. We agree that these additional explanation will convey more clearly the advantages offered by our proposed method.

Reviewer comment (Evaluation):
Section 5.3 only presents the average power of various tests, and I didn't see any evidence that the tests obey their nominal significance levels.

Author response:
We appreciate your attention to the presentation of the test power results in Section 5.3 and your concern about the significance levels. To address this, we would like to clarify that all tests were conducted with a significance level of $\alpha = 0.05$. Detailed information about the testing procedure was provided Appendix C.3. Additionally, we included figures (3 to 6 in the appendix) displaying the standard deviations of test power for all experiments, offering a more comprehensive view of the test performance. Furthermore, we have made the code used to generate these results available on an anonymous GitHub repository to ensure transparency and reproducibility. This allows interested readers to replicate our experiments and verify the reported results. We believe these additional details and resources strengthen the robustness of our findings.

Reviewer comment (Writing):
Abstract: "Our approach embeds the data in an reproducing kernel Hilbert space (RKHS)... Therefore, we name this measure the representation Jensen-Shannon divergence (RJSD).": I didn't understand the use of "Therefore" here; i.e., how does the second sentence follow from the first?

Author response:
The first sentence finishes saying that our metric uses the "covariance operator in this **representation** space". For this reason (**therefore**), we name our divergence, the **representation** Jensen-Shannon divergence.

Reviewer comment (Writing):
Page 2, Last Sentence, "If we normalize the matrix $K_X$ such that, Tr$(K_X) = 1$...": There are many ways such normalization could be performed. Is this done by scaling (i.e., replacing $K_X$ with $\frac{K_X}{\text{Tr}(K_X)}$)? If so, this should be clearer.

Author response:
We agree that the normalization step should be clearer. As you pointed out the normalization is done by dividing over the trace ($\text{Tr}(\mathbf{K_X}) = N$ , because $\kappa(x,x) = 1 \forall x \in \mathcal{X}$), therefore, in the revised paper, we  make this normalization explicit in the kernel-based estimator as follows: $S\left(\mathbf{K}_{X}\right) = -\text{Tr}{\left(\frac{\mathbf{K}_X}{N}\log \frac{\mathbf{K}_X}{N}\right) } $.

Reviewer comment (Theory):
Page 3, Last Two Sentences, "Let $\hat P_\gamma(x)$ be the empirical kernel density function by a Gaussian kernel with scale parameter $\gamma$. Dmitriev & Tarasenko (1974) demonstrate that $H(\hat P_\gamma)$ converges to $H(\mathbb{P})$ as both, $N$ and $\gamma \to \infty$, with probability one.": I was quite confused by the meaning of $\gamma$ here. Typically, for the kernel density estimate (KDE) to be consistent, the bandwidth (which I think of as synonymous with the "scale parameter") should shrink (approach $0$, not $\infty$). Does "scale parameter" here mean the reciprocal of the bandwidth (i.e., the "precision" of the Gaussian)? Also, are there no additional assumptions required here? For example, consistency of the KDE requires that the bandwidth $h$ not shrink too quickly (e.g., $N h^d \to \infty$).

Author response:
In our paper we use the definition of Gaussian kernel $\kappa(x,x') = \exp(-\gamma\lVert x - x' \rVert^2)$ and we regard $\gamma$ as the scale parameter. Here, we can see that $\gamma = \frac{1}{2\sigma^2}$, where $\sigma$ is the kernel bandwidth. Therefore, as $\gamma \rightarrow \infty $, $\sigma \rightarrow 0$, as you correctly mentioned. To avoid confusion, in the revised paper, we properly define the Gaussian kernel that we use and additional assumptions that guarantee the consistency of the KDE.

Reviewer comment (Writing):
Page 5, Equation (12): Typo: I think $D_{JS}$ should be $D_{JS}^\phi$, correct?

Author response:
It is correct, this is a typo. (Addressed in the revised manuscript)

Reviewer comment (Evaluation):
How exactly is the RJSD metric utilized for two-sample testing in Section 5.3? Usually, either a permutation test or an analytically derived asymptotic null distribution is used, but I didn't see any explanation of this.

Author response:
We would like to mention that in the main paper, page 8, last sentence, we refer readers to appendix C.3 where we explain the testing procedure. Specifically, we address the questions that you ask. Next, we quote appendix C.3. 

> The procedure is the following: for synthetic datasets, we create the sets $X_{train} \in \mathbb{R}^{N\times d}$ and $Y_{train} \in \mathbb{R}^{M \times d}$. Then, we learn the kernel/covariance/classifier for each of the methods on that training set. We then sample a testing set  $X_{test} \in  \mathbb{R}^{N\times d}$ and $Y_{test} \in  \mathbb{R}^{M\times d}$ and perform a permutation test. We compute the statistic on the testing set and perform 100 permutations to generate the surrogate of the distribution of the measurement under the null hypothesis. Finally, we compute the rejection threshold, and if the statistic is greater than this threshold we reject the null hypothesis. This is done for 100 independent testing sets. Finally, we repeat the experiment ten times and compute the average test power.

Reviewer comment (Evaluation):
The paper would have been much stronger if the performance improvement over MMD had been shown more clearly. As it stands, there is a rather small performance gain, and it is not completely clear why.

Author response:
* *"The paper would have been much stronger if the performance improvement over MMD had been shown more clearly. As it stands, there is a rather small performance gain, and it is not completely clear why."* (**Q3**) We appreciate your feedback regarding the clarity of the performance improvement over MMD in our paper. We agree that providing a more explicit demonstration of this improvement is crucial and it would make the article stronger. In response, we would like to highlight Theorem 5 in our paper, where we establish that RJSD serves as an upper bound on MMD. This result implies that if MMD can effectively distinguish between two distributions, then RJSD can achieve at least similar discriminative power. We argue in the paper that this theoretical relationship suggests that RJSD should be at least as good as MMD in identifying discrepancies between distributions. While it is true that the observed performance gain may not be substantial, deep kernel MMD is a state-of-the-art approach. In this context, even marginal improvements in performance on the evaluated datasets are noteworthy. To address your concern, in the revised document, we elaborate further on why RJSD is expected to have a superior test power. We agree that these additional explanation will convey more clearly the advantages offered by our proposed method.

Reviewer comment (Theory):
Regarding the theory, how important is the requirement in def. 2 that the kernel should satisfy \( k(x,x)=1 \)? On the one hand, it is easy to arrange. On the other hand, it seems that consistency results require probability normalized Gaussian kernels (i.e., integrate to 1, so the Parzen estimator would be a density), and these have \( k(x,x) \rightarrow \infty \) with \( \gamma \rightarrow \infty \). Thus it appears the method would not compute the true JSD. Please comment. Note that MMD does not have such problems.

Author response:
(**Q1**) The condition $\kappa(x,x) = 1$ ensures that the covariance operator will have unit trace. Note that from $\kappa(x,x)=1$, we obtain that $\lVert \phi(x)\rVert = 1$. Therefore, $\text{Tr}(\phi(x)\otimes \phi(x)) = \lVert \phi(x) \rVert^2$ for all $x \in \mathcal{X}$. On the other hand, in the results connecting kernel-based entropy with Shannon's differential entropy, as $\gamma \rightarrow \infty$, $\kappa(x,x)$ is still one, since for the Gaussian kernel $\kappa(x,x) = \exp(-\gamma\lVert x - x \rVert^2) = \exp(-\gamma \cdot 0) = 1 $ regardless the value of $\gamma$. To avoid confusion, we explicitly define the Gaussian kernel as above in the revised manuscript.

Reviewer comment (Experiments):
The RFF based estimator is perhaps discussed too soon. I would have liked to read the properties and performance of the kernel matrix based estimator when O(n^3) is not really prohibitive. Typically we are able to compute eigendecompositions of up to 20-30K samples easily on modern laptops.

Author response:
* *"The RFF based estimator is perhaps discussed too soon. I would have liked to read the properties and performance of the kernel matrix based estimator when $O(N^3)$ is not really prohibitive."*
We appreciate your suggestion to elaborate further the properties of the kernel matrix-based estimator. It's important to note that the introduction of the Fourier features approach is not solely aimed at reducing computational cost by approximating the kernel-based estimator. Instead, the Fourier features serve a dual purpose in our methodology. Firstly, they facilitate the construction of empirical covariance matrices, crucial for estimating RJSD. Secondly, these features enable a parameterization of the representation space, leading to a kernel-learning strategy. We treat the Fourier features as learnable parameters within a neural network (Fourier Feature network), optimizing them to maximize divergence and enhance the discriminatory power between sets of samples. Consequently, the Fourier features approach offers a more versatile estimator that extends beyond addressing computational cost.
In the revised manuscript, we provide a clearer exposition of these two functionalities to enhance understanding. Also, we changed the order of the RJSD estimators, presenting first the kernel-based estimator as you suggested.

Reviewer comment (Writing):
The writing has a lot of room for improvement.

Author response:
* *"The writing has a lot of room for improvement. The following notational issues bothered (and confused) me while reading this paper."* Regarding the notational issues, we appreciate the thorough revision of the notation inconsistencies or confusing symbols, and we addressed all the concerns raised in the revised manuscript. Next, we reply individually to some of them that require further clarification.

Reviewer comment (Writing):
Curly brackets are used for {X, B_X} that defines the measure space instead of the standard round brackets. This also appears above equation (8).

Author response:
1. Addressed.

Reviewer comment (Writing):
M_+^1(X) is complicated. Can you use a simpler notation, say \mathcal{P}, for the set of all distributions on (X, B_X)?

Author response:
2. Addressed.

Reviewer comment (Theory):
Is it true that phi(x) = K(x, .) since you are saying H is the RKHS corresponding to K? If phi is mapping into H then it has to be K(x, .). If phi(x) is in some other Hilbert space H' (not necessarily the RKHS H) such that K(x, z) = <phi(x), phi(z)>_{H'}, is this sufficient? If by phi(x) you indeed meant K(x, .) then please clarify. This would mean that C_P is the integral operator of the kernel, which is a well-studied object (see [1] in summary), and you should provide some background on this operator.

Author response:
3. Yes, $\phi(x) = \kappa(x,\cdot)$, and we expressed this relationship through the inner product as $\kappa(x,x')= \langle\phi(x),\phi(x')\rangle_\mathcal{H}$.

Reviewer comment (Theory):
Is E[K(X, X)] < ∞ necessary for the relation E[f(X)] = <f, μ_P>? I think it would hold even otherwise, just that both sides would be infinite.

Author response:
5. $E[ \kappa(X, X)] < \infty$ is a necessary condition as it is shown by Smola et al. (2007) and Gretton
et al. (2012)[Lemma 3]. Particularly, $E[\kappa(X, X)] < \infty$ is the sufficient condition to ensure that $\mu_{\mathbb{P}}$ is an element of the Hilbert space $\mathcal{H}$ which allows the inner product $\langle f, \mu_P \rangle$.

Reviewer comment (Theory):
In section 2.2 you say unit trace. Is this an assumption for everything that follows? Should all kernel matrices also be normalized to have unit trace? Regardless, if it is clear (or even nontrivial) as to what happens to the entropy when the trace is not unity, a clarification would be useful to the reader. I think this is the general relation: S(c * C_P) = c * S(C_P) + c * log(c). Please recall the fact that trace=1 is assumed anywhere else in the paper where that is used. Is the RJSD invariant to the scale of the kernel? For example, if I use K1 = c * K, the RKHSs for K1 and K are the same but their norms are scaled versions of each other. How does this change the RJSD? If it doesn’t, make this clear.

Author response:
6. The unit trace of the covariance operator is a necessary assumption so that its eigen-values sum-up to one and can be used to compute the entropy. Assuming $\kappa(x,x) = 1 \forall x$ guarantee that the trace of the covariance operator is one  (Bach, 2022). Therefore, as you pointed out, all kernel matrices should be normalized to have unit trace. That's why we state that "If we normalize the matrix $\mathbf{K_X}$ such that, $Tr(\mathbf{K_X}) = 1$, $\mathbf{C_X}$ and $\mathbf{K_X}$ have the same non-zero eigenvalues (Sanchez Giraldo et al., 2014)". For the sake of clarity, we made this normalization explicit as follows: "We focus on the case of normalized kernels where $\kappa(x, x) = 1$ for all $x\in \mathcal{X}$. We denote the Gram matrix $\mathbf{K_X}$, consisting of all normalized pairwise kernel evaluations of data points in the sample $\mathbf{X}$, that is $(\mathbf{K_X})_{ij} = \kappa(x_i, x_j)$ for $i, j = 1, \dots, N$. It can be shown that $\mathbf{C_X}$ and $\frac{1}{N}\mathbf{K_X}$ have the same non-zero  eigenvalues (Sanchez Giraldo et al., 2014; Bach, 2022), yielding the kernel-based entropy estimator:
$ S\left(\mathbf{K_X}\right) = -\text{Tr}{\left(\tfrac{1}{N}\mathbf{K_X}\log \tfrac{1}{N}\mathbf{K_X}\right) }
$."

Reviewer comment (Writing):
In the paragraph on covariance based estimator, is this phi the same as the phi used earlier? If not, please use a different letter. Next, please write an equation that describes a shift invariant kernel. Unless that is used, the formula for p(w) using Bochner's theorem is not clear because until then kappa is a bivariate function. Use a different function name in that formula to avoid confusion. Also state that w_i are i.i.d. samples from p(w). Also use a different letter for phi_w since phi is already used in Section 2.1 for something else. State at the end of this paragraph how to choose D (perhaps point to Theorem 2).

Author response:
7. $\phi$ is a general notation to describe any mapping (function) from $\mathcal{X}$ to $\mathcal{H}$. Therefore, we use $\phi_\omega$ to denote a mapping that is parameterized by $\omega$ (Random Fourier Features mapping).

Reviewer comment (Theory):
Formally define entropy H and cross-entropy H(P, Q) otherwise Theorem 1 doesn’t make any sense. Also, is this differential entropy? Please state this clearly. Differential entropy doesn’t enjoy all the properties that entropy from information theory does, e.g., it can be negative.

Author response:
8. Addressed.

Reviewer comment (Writing):
In Definition 1 you use K(x, x) = 1, but that is not true when you are using trace(C_P) = 1. Please make sure things are consistent here. Next, in this definition, please use a different letter than P for the LHS. It is very confusing to see P both on the left and right side. This issue complicates things further when you assume some phi* exists in Theorem 3. Theorem 3 was incomprehensible and needs a significant edit after Definition 1 has new notation for the LHS.

Author response:
9. * Remember that we mention that the kernel matrix should be normalized so that $\text{Tr}(\mathbf{K_X}) = 1$. Since $\kappa(x,x) = 1$, the kernel matrix should be normalized by the number of samples to ensure unit trace. For the sake of clarity, we express the kernel estimator in terms of $\mathbf{K_X}/N$.
    * With respect to definition 1, we use $\mathbb{P}_{\phi}$ to denote the approximation to $\mathbb{P}$ given by a kernel density estimator with a mapping $\phi(x) = \kappa(x, \cdot)$. To make this less confusing, we denote the density function as $p(x)$ and its approximation as $\hat{p}(x)$.

Reviewer comment (Writing):
In Theorem 3, which space does phi* lie in? Please again use a different letter here. Phi has been overloaded by being used three times at this point in the paper.

Author response:
10. To ensure consistent notation we use the standard notation $\phi(\cdot)$ to refer to any mapping to some RKHS $\mathcal{H}$. The reason we use $\phi^*(\cdot)$ is to denote some specific mapping $\phi$ that estimates the density function $p(x)$. If this is the case, RJSD is equal to the true Jensen Shannon divergence.

Reviewer comment (Theory):
Theorem 5 has a ||.||* which is undefined. Here again you are using K(x, x) = 1; does that clash with trace=1? Also, formally define MMD_K clearly before stating the lemma. I feel Theorem 5 should just have two objects: RJSD ≥ MMD. The other objects can be pushed to the proof.

Author response:
11. Addressed

Reviewer comment (Writing):
In Section 4, is the omega in phi_w ○ f_w the same for phi and w? It seems weird that the networks’ earlier and last layers have the same parameterization.

Author response:
13. In section 4, $\omega \in \Omega$ refers to all parameters in the network. Notice, that the Fourier features are treated as parameters of a neural network that can be trained. Essentially, the Fourier features can be seen as a linear layer with cosine and sine activation functions. Here, all parameters are optimized jointly through backpropagation.

Reviewer comment (Presentation):
In Figure 1, please mention that the black line is for the closed form expressions between the Cauchy distributions. Please also mention in the caption why the steps are appearing, i.e., because you are manually changing the parameters of the distributions at those points. Related to this, please give full forms of the different estimators you have used.

Author response:
14. Addressed

Reviewer comment (Presentation):
Figure 3 doesn’t really add much value as such. What are we supposed to make of these letters? Are they good?

Author response:
15. Figure 3 could be removed. We just wanted to show evidence that the RJSD-GAN was producing actual digits and that they were visually similar to the ones in the stacked MNIST dataset.