Paper: Generative Sliced MMD Flows with Riesz Kernels (ICLR-2024)

Reviewer comment (Experiments):
It seems that the proposed framework is not scalable in terms of dimension since the result from CIFAR10 is quite blurry and noisy.

Author response:
Thank you very much for your review.  
We extended the experimental part by including the CelebA dataset which is of higher dimension than the other datasets.  

In order to circumvent the dimensionality effect, we use on CelebA two further techniques.  

- We use the multiscale approach to generate the images. More precisely, we first generate a downsampled version of the CelebA dataset, use an upsampling step and approximate the higher-dimensional version.  
From a theoretical side this correspond to change the latent distribution.  

- Following the references (Du et al. 2023) and (Nguyen and Ho, 2022), we included local projections.  

We describe both techniques in more detail within the implementation details setting in Appendix H.  

To explain the dependence on the dimensions theoretically, we have already included Remark 5 in the main paper.  

References:  

(Du et al, 2023) Nonparametric Generative Modeling with Conditional Sliced-Wasserstein Flows  

(Nguyen and Ho, 2022) Revisiting sliced Wasserstein on images: From vectorization to convolution

Reviewer comment (Experiments):
Could any methods be used to improve the experimental results on high-dimensional datasets?

Author response:
Thank you very much for your review.  
We extended the experimental part by including the CelebA dataset which is of higher dimension than the other datasets.  

In order to circumvent the dimensionality effect, we use on CelebA two further techniques.  

- We use the multiscale approach to generate the images. More precisely, we first generate a downsampled version of the CelebA dataset, use an upsampling step and approximate the higher-dimensional version.  
From a theoretical side this correspond to change the latent distribution.  

- Following the references (Du et al. 2023) and (Nguyen and Ho, 2022), we included local projections.  

We describe both techniques in more detail within the implementation details setting in Appendix H.  

To explain the dependence on the dimensions theoretically, we have already included Remark 5 in the main paper.  

References:  

(Du et al, 2023) Nonparametric Generative Modeling with Conditional Sliced-Wasserstein Flows  

(Nguyen and Ho, 2022) Revisiting sliced Wasserstein on images: From vectorization to convolution

Reviewer comment (Theory):
Concerning Theorem 2: Theorem 2 establishes bounds between MMD and Wasserstein distance of order 1. In my opinion, these results are not very sharp, and there appears to be an important missing reference here. Under the same assumptions of compact support, the article [1, Theorem 1] demonstrates that the Wasserstein distance $W_1$ is bounded by an MMD with the Coulomb kernel $k(x, y) = -|x - y|^{2-d}$ but without the power dependency of $1/(d+1)$. Since the measures are bounded, MMD with the Riesz kernel and the Coulomb kernel are related by a constant (dependent on the dimension) on the support of the measures. Therefore, it seems important to mention this result and discuss its relevance.

Author response:
Concerning Theorem 2: 
the statement of Theorem 2 does not follow from a corresponding statement about the Coulomb kernel as we explain below.
The Coulomb kernel in Chafaï et al. is defined by $K_C(x,y)=\\|x-y\\|^{2-d}$ for $d\ge3$ **without the minus**.
Therefore, we cannot bound the Riesz kernel from below by the Coulomb kernel.
Moreover, Thm 1.1 from Chafaï et al. assumes that $\int\int K_C(x,y)d \mu(x)d \mu(y)<\infty$ and $\int\int K_C(x,y)d\nu(x)d \nu(y)<\infty$.
This assumption is violated whenever $\mu(\\{x\\})>0$ or $\nu(\\{x\\})>0$ for any $x$.
In particular, this assumption excludes the case of empirical measures which is the most important one for our applications.
Nevertheless, since this is an interesting idea and since the bound looks similar at the first glance, we added a comment about that at the end of the proof of Theorem 2 in Appendix B. Even though we do not utilize this bound, we found the relations to the Wasserstein-1 distance quite interesting and would like to keep it in the paper.

Reviewer comment (Novelty):
Concerning Theorem 2: In a more general sense, I'm having trouble grasping the significance of these bounds for the current article. They don't appear to be entirely novel, and more importantly, they are not utilized subsequently, neither theoretically nor practically, nor in the discussion.

Author response:
Concerning the sharpness of the bounds in Theorem 2:
We do not think that a sharper version of Theorem 2 without the dimension scaling holds true for the following reason:
It is well-known that MMD has sample complexity $O(n^{-1/2})$ (Gretton et al. 2012) and Wasserstein-1 has a sample complexity of $O(n^{-1/d})$ (see e.g., Peyré and Cuturi, 2020). If it would hold true that $W_1(\mu,\nu)\leq C\mathcal D_K(\mu,\nu)$ for all $\mu,\nu$ this would imply that also Wasserstein-1 would have a sample complexity of $O(n^{-1/2})$ which is a contradiction.
We included a comment about that in the paper.
For the Coulomb kernel this argumentation does not apply, since it is not bounded on compact sets (which is an assumption for the sample complexity $O(n^{-1/2})$ for MMD) and since the result of Chafai et al. explicitly excludes empirical measures.

Reviewer comment (Theory):
Regarding Theorem 3: Theorem 3 is the main contribution of this article, and I agree that it is interesting and significant. However, there is a minor point that needs clarification: while the function $E$ is shown to be differentiable everywhere in the proof, for the differentiability of $V$, the authors use an argument that doesn't seem rigorous. Indeed, the function $x \to |x-y|$ is not differentiable at $x=y$. In practice, this may not be very important because this event is almost surely zero if the samples come from measures with densities. Still, it's important to note this limitation.

Author response:
Regarding Theorem 3: Note that the potential energy $V$ is convex. 
Consequently, we can use subgradients instead of gradients. We added a comment on this in the paper after Theorem 3.

Reviewer comment (Experiments):
About training a sequence of neural networks (Section 4.2): Furthermore, it seems to me that training a network per step is very costly; I doubt the feasibility of this method. Is this standard practice?

Author response:
About training a sequence of neural networks (Section 4.2):
We train the networks for two reasons. 
First, the neural networks speed up the generation process and allow us to draw a single sample (for the particle flow, we always have to compute a batch of samples).
The second and more important reason is that the particle flow itself converges **exactly** to the training images, see Figure 9 and the corresponding explanation in the Appendix I of the paper for an illustration.
In some sense, we could say that the particle flow works "too well" since it approximates the discrete target measure exactly. The resulting FID of the flow particles would be 1.37 for MNIST, 1.90 for FashionMNIST and 3.90 for CIFAR10.
If we use the sequence of neural networks, we can interpolate between the training points in order to generalize the dataset. Finally, note that we do not train a network per step of the particle flow, but one network for 30000 steps of the flow after a warm start procedure. A detailed explanation towards this is given in Appendix H. Thus the training of the networks is not very costly compared to the simulation of the particle flow.

Reviewer comment (Writing):
Regarding the "related work" section in the introduction: I believe that this section could be improved. The paragraph is somewhat confusing as it introduces a set of articles without providing clear context for the current work or establishing connections between the cited articles. As a result, I find this part to be not very illuminating for understanding the related work.

Author response:
Regarding related work:
We agree that several references from the related work section were relevant for providing a broader context of the topic, but were not directly related to our paper. We moved these references to the general introduction and reorganized the related work section.
We hope that this makes it more clear now.

Reviewer comment (Presentation):
Figures 1 and 2: The tables are not informative and hard to read. Since the information is already contained in the plots, I don't see the purpose of these tables. I suggest removing them and using the available space for more details on the points described earlier. Additionally, Figure 1 lacks a legend for the runtime (in ms?).

Author response:
We followed your suggestion to fuse the graphs from Figure 1 and 2 and omit the tables. Moreover, we included the legend for the runtime, it is in fact in ms.

Reviewer comment (Theory):
The Riesz kernel also defines a valid MMD for $r=2$. So why restrict it to $r \in (0,2)$ with 2 excluded?

Author response:
If we use $r=2$ in the Riesz kernel, the corresponding MMD is no longer a metric. Indeed it holds that
$$
\mathcal D_K^2(\mu,\nu)=\frac12\int\int-\\|x-y\\|^2 \mathrm{d} (\mu-\nu)(x) \mathrm{d} (\mu-\nu)(y) 
$$
$$
=\sum_{i=1}^d \frac12\int\int-(x_i-y_i)^2 \mathrm{d} (\mu-\nu)(x) \mathrm{d} (\mu-\nu)(y) =\sum_{i=1}^d\mathcal D_K^2((\pi_i)\\#\mu,(\pi_i)\\#\nu),
$$
where $\pi_i(x)=x_i$. In particular, we have for $K(x,y)=-\\|x-y\\|^2$ that $\mathcal D_K^2(\mu,\nu)$ is zero whenever the marginals $(\pi_i)\\#\mu$ and $(\pi_i)\\#\nu$ coincide for all $i$. Consequently, $\mathcal D_K$ cannot be a metric in this case.
We added a comment about that in the paper.

Reviewer comment (Theory):
Unless I am mistaken, the result about $D_{K}$ as a metric on $P_{r/2}$ in (Modeste & Dombry, 2023) is not mentioned anywhere in the cited article. Can you provide more details on this fact or provide the correct reference?

Author response:
Since the statement about the metric property of $\mathcal D_{\tilde{K}}$ on $\mathcal P_{\frac r 2}(\mathbb R^d)$ is not directly relevant in the main paper, we moved it to Remark 7 in Appendix A. Moreover, we added the following explanations, how it can be deduced from Proposition 2.14 in (Modeste and Dombry 2023):
By Theorem 4.26 in (Steinwart and Christmann, 2008), the MMD
$$
\mathcal D_{\tilde K}^2(\mu,\nu)=\int\int \tilde K(x,y) d(\mu-\nu)(x)d(\mu-\nu)(y)
$$
is finite on $\mathcal M_{\tilde K}=\\{\mu \in \mathcal M(\mathbb R^d):\int \sqrt{\tilde K(x,x)}\mathrm d |\mu|<\infty\\}$, where $\mathcal M(\mathbb R^d)$ is the space of all signed measures on $\mathbb R^d$.
Since $\tilde K(x,x)=2\\|x\\|^r$, we have that $\mathcal M_{\tilde K}=\\{\mu \in \mathcal M(\mathbb R^d):\int \\|x\\|^{\frac r 2}\mathrm d |\mu|<\infty\\}$ such that $\mathcal M_{\tilde K}\cap\mathcal P(\mathbb R^d)=\mathcal P_{\frac r 2}(\mathbb R^d)$.
Now, inserting $\tilde K$ Proposition 2.14 in (Modeste and Dombry 2023) states that $\mathcal D_{\tilde{K}}$ is a metric on $\mathcal M_{\tilde K}\cap \mathcal P(\mathbb R^d)=\mathcal P_{\frac r 2}(\mathbb R^d)$, where the assumptions of the proposition are checked in Example 2 and 3 of (Modeste and Dombry 2023) and $\tilde K$ is named $k_H$ with $H=\frac r 2$.

Reviewer comment (Theory):
I'm quite curious to know if the approach presented in the article can be generalized to the case of $r \in (0,2]$? Particularly, the fast gradient computation.

Author response:
While Theorem 2 works with any $r\in(0,\infty)$ (even though we loose properties of the MMD for $r\geq 2$), we are quite sure that the one-dimensional sorting algorithm cannot be generalized to other cases. However, we are currently working on a generalization of this approach via Fourier transforms and hope to finish a paper on it until the end of the year.

Reviewer comment (Theory):
It seems that the reference (Numayer & Steidl, 20201, Lemma 3.3) is incorrect, or at least Lemma 3.3 doesn't state that the two MMDs coincide.

Author response:
We are sorry about the confusion regarding the reference (Neumayer and Steidl, 2021). We just recognized that the numbering of the lemmas differs between the arxiv and journal version.
The statement which we refer to is part iii) of Lemma 3.3 in the arxiv version and part iii) of Lemma 1 in the journal version.
We corrected the reference in the paper.