Paper: Improved Generalization of cGAN using Vicinal Estimation and Early Stopping (ICLR-2024)

Reviewer comment (Novelty):
The idea to use an auxiliary distribution is similar to importance sampling, are they related somehow?

Author response:
The "auxiliary distribution" in importance sampling is the distribution function chosen similar to the target distribution to be sampled, whereas auxiliary distribution in our manuscript rather means the mapping between label space and auxiliary label space. Despite sharing the name "auxiliary distribution", we do not consider these two concepts to have anything in common.

Reviewer comment (Writing):
Section 4, which is one of the most important ones is poorly written. The definition of the generator that is analyzed is not given in full detail. In particular, neither the set of the discriminators nor the set of the generators involved in the min-max problem of the GAN are clearly specified. In addition, it is not clear in Eq 19 what is understood by $\tilde p_{X,Y}(y|x')$.

Author response:
The details of the GANs used in the proposed framework are described in Section 4. Specifically, they are as follows:

- Set of the discriminator : $\tilde{D}\_a$ defined by Equation (14) parameterized by $a$
- Set of the generator : For an arbitrary generator $G$, we use **Algorithm 2** to sample from $\tilde{p}\_g(\cdot|\mathbf{x}')$. So the set of generators is not limited.
- $\tilde{p}\_{X,Y}(y|x')$ is a conditional distribution, which is induced from the joint distribution $\tilde{p}\_{X,Y}$ which is a VE of the empirical distribution. We have added the desciption for $\tilde{p}\_{X,Y}$ in the revised manuscript.

Reviewer comment (Writing):
Section 5 is very dense and hard to read. The results are quite technical so it is impossible to say whether they are plausible or not without checking the proofs line by line. Given the limited time I had to review the paper, I could not check the proofs provided in the supplementary material. I believe that it would be much better for such a paper to be submitted to a journal, where more space may be used for stating the main theorems and providing some explanations and intuitions, as well as more time could be left to the reviewers for checking the details of the proofs.

Author response:
To the best of the authors' knowledge, there are many theoretic papers in ICLR, including generalization error analysis which our paper focuses on. While the journal will be more suitable for the papers with complex mathematical proofs, we believe that our manuscript does not contain such complex proofs as the proofs are not extremely long (3 pages in appendices) and do not involve extremely complex mathematical concepts. Our proposed model is validated by the experimental results which also support the theoretical analysis.

Reviewer comment (Theory):
It seems that the results of the paper do not imply that the vicinal estimation method is better than the vanilla cGAN. Indeed, the fact that the rate of convergence does not depend on the dimension of Y might very well be a consequence of the choice of the set of discriminators. In particular, it is well known that the lower bound (1) is due to the fact that the set of discriminators defining the $W_1$ (which is a lower bound on $W_2$) -- the set of 1-Lipschitz functions -- is too large. Replacing this set by an RKHS with a bounded kernel leads to the dimension independent rate $N^{-1/2}$.

Author response:
First of all, we are not aware of any results showing that the generalization error in $W\_1$ can be bounded to $N^{-1/2}$ by RKHS kernel regularization, as you mention in the last sentence. To the best of the authors' knowledge, Bias Potential Model (2021) was almost the first to achieve dimension-independent generalization error for unconditional GANs, except when bounding $p\_g$ itself, and the best we know of is $N^{-1/4}$ for KL divergence and $N^{-1/6}$ for $W\_2$.

Even if dimension-independent generalization error can be easily achieved for unconditional GANs as mentioned, it cannot be extended to conditional GANs (vanilla cGANs?) due to a problem named "Lack of Conditional Sample" : even if a generalization error is obtained for each fixed label value, the conditional samples of that label value will be much smaller than the total number of samples.

One of the key contributions of our work is to introduce the concept of vicinal estimation, which allows us to obtain a sufficient number of conditional samples for arbitrary (auxiliary) label values.

Reviewer comment (Theory):
What is the functional space of the discriminator D(y,x) in eq. 5, and how is it related to the W-1 distance in eq. 6? I suppose you are assuming the (y,x) -> D(y,x) is Lipschitz, but for the W-1 distance, we only need y -> D(y,x) to be Lipschitz for any x.

Author response:
As mentioned, it is enough for $D$ to be Lipschitz continuous with respect to $\mathbf{y}$. We have corrected the sentence.

Reviewer comment (Theory):
The tilde D_alpha(y,x’) defined in eq 14, is it also Lipschitz? I do not see why this is so (even based on eq. 22) and it seems to be inconsistent to your previous definition.

Author response:
In the definition of Equation (14), $D_a(\mathbf{y},\mathbf{x})$ is not Lipschitz. In our proposed model, we do not need the Lipschitzness of $D$ because we are applying RKHS norm regularization rather than enforcing Lipschitz like Wasserstein cGAN.

Reviewer comment (Theory):
What is the definition of tilde p_{x,y} and p(x’) in eq. 20? Is p(x’) an empirical distribution as in eq. 7? It seems not to be the case according to theorem 2 and 3. Then I am confused of the over-all setting if you are the true distribution p(x’) of x’ in your training loss.

Author response:
For $\tilde{p}\_{X,Y}$ , it is a vicinal estimate of the empirical distribution defined in Equation (2), which can be obtained by substituting $\tilde{p}\_{X,Y}$ for $p$ in Equation (8). We clarify it before Equation (19) in the revised manuscript to indicate that $\tilde{p}\_{X,Y}$ is the VE of the empirical distribution.  

What you refer to as $p(\mathbf{x}')$ in Equation (20) is $\rho(\mathbf{x}')$, which means a random distribution, not the actual distribution of auxiliary label values. Once $\rho(\mathbf{x}')$ is known, we can obtain a generalization bound. Moreover, we can consider various $\rho(\mathbf{x}')$ to our main results such as **Corollary 2**. For example, as mentioned in the manuscript, when $\tilde{q}$ is a perturbation, we can substitute $p_r(\mathbf{x})$ into $\rho$ to get a generalization error between $p_g$ and $p_r$.

Reviewer comment (Theory):
Do you need to assume that the kernel is characteristic in Theorem 3? It is not clear what are the above conditions in Theorem 5. Please make it clearer. Could you also explain why there is no d_y in the statement of Theorem 5?

Author response:
In **Theorem 3**, it is not necessary for the kernel to be characteristic. As mentioned, the various conditions assumed in **Theorem 5** seem to be scattered, so we have summarized the exact conditions in **Theorem 5**. 

Finally, the reason why $d_\mathbf{y}$ does not appear in **Theorem 5** and **Corollary 2** is because of the Big-O notation to emphasize the decay rate for a dataset size $N$. If we write the right-hand side correctly, it is multiplied by a sublinear term for $d_\mathbf{y}$, which is omitted in Big-O Notation because it is a term independent of $N$.

Reviewer comment (Theory):
Why did you introduce a RKHS regulation in the loss V’ in eq. 18? What would happen if there is no regulation?

Author response:
The RKHS norm regularization introduced in Equation (18) and elsewhere is a variation of prior work (Weinen & E, 2021) that introduced a dimension-free generalization error in unconditional GANs. If there is no such regularization, i.e. if the error had the form of $V'(G,\tilde{D}) = \mathbb{E}\_{\mathbf{x}' \sim \rho(\mathbf{x}')} \left[ \mathbb{E}\_{\mathbf{y} \sim \tilde{p}\_r(\mathbf{y}|\mathbf{x}')}[\tilde{D}(\mathbf{y},\mathbf{x}')] - \mathbb{E}\_{\mathbf{y} \sim \tilde{p}\_g(\mathbf{y}|\mathbf{x}')}[\tilde{D}(\mathbf{y},\mathbf{x}')] \right]$, it is possible to convert it to a problem of minimizing $\mathbb{E}\_{\mathbf{x}' \sim \rho(\mathbf{x}')} W\_1 (\tilde{p}\_g(\cdot|\mathbf{x}'), \tilde{p}\_r(\cdot|\mathbf{x}'))$ by assuming Lipschitzness on \tilde{D}. In this case, we're still facing the curse of dimensionality and cannot get a reasonable generalization bound.

Reviewer comment (Writing):
What is y used for in eq. 25? I do not see it to appear on the right hand side.

Author response:
Note that $\mathbf{y}$ is used as the first argument of the kernel in Equation (25). We've omitted it because it's a notation commonly used in convolution, but it's technically equivalent to $\int k(\mathbf{y},\mathbf{y}') d(\tilde{p}_r(\mathbf{y}'|\mathbf{x}') - \tilde{p}_g^t (\mathbf{y}'|\mathbf{x}'))(\mathbf{y}')$.

Reviewer comment (Theory):
How do you get the optimal scale of ||tilde q||_inf after eq. 33?

Author response:
If we leave $N$ and $Q=||\tilde{q}||\_\infty$ to find $||\tilde{q}||\_\infty$ that minimizes the right-hand side of Equation (33), it becomes equal to $AN^{-1/6}Q^{1/3} + BQ^{-1/d_{\mathbf{x}}}$. The $Q$ that minimizes this expression is easily obtained via differentiation.

Reviewer comment (Writing):
I found it counterintuitive that $X$ is used for the label and $Y$ is used for the generated image. While this makes sense in terms of the inputs and outputs of a cGAN, it conflicts with much more common settings in the ML literature (e.g., image classification, where $X$ is the image and $Y$ is the label), causing me to be confused through much of the paper until I went back and re-read the beginning of Section 3. I don't necessarily suggest changing this (especially if it is consistent with other paper on conditional generative modeling), but perhaps it is worth adding a sentence to explicitly point out this possible point of confusion.

Author response:
In this study, we followed the notation and considered an error analysis from the perspective of Conditional Density Estimation. For this purpose, we used $\mathbf{x}$ to denote a label and $\mathbf{y}$ to denote an image, as used in the studies related to Conditional Density Estimation. We have added a sentence at the beginning of Section 3 to clarify that $\mathbf{x}$ is a label and $\mathbf{y}$ is an image.

Reviewer comment (Writing):
Just before Eq. (11), a minor wording suggestion: "a VE vanishes the conditional information" -> "a VE reduces the conditional information"

Author response:
We have modified the sentence.

Reviewer comment (Theory):
Theorem 1: Isn't $||\tilde q^L||_\infty = L^{-d_x} ||\tilde q^1||_\infty$? If so, perhaps explanding this would make the statement it a bit more intuitive.

Author response:
Just before **Theorem 1**, we added a sentence to emphasize the fact that $d_{\tilde{q}^L}(\textbf{x}') \propto L^{-1}$ and $||\tilde{q}^L||_{\infty} \propto L^{-d_{\textbf{x}}}$.

Reviewer comment (Experiments):
Although the work is intended to be theoretical, it would be strengthened by any discussion of real-world relevance (e.g., describing some applications where the proposed vicinal estimation approach might be useful). Experiments on real-world data would further strengthen the paper.

Author response:
In Conditional cGAN (CcGAN), Ding et al. (2023) utilizes the concept of perturbing the label value in one-dimensional continuous labels and demonstrates the effectiveness of their proposed method for real-world conditional image generation tasks. As perturbation is a special example of vicinal estimation proposed in our manuscript, we can expect that VE and our proposed method can also improve the performance of cGAN models.

There is a technical issue to undertake a supplementary experiment for real-world data, as we require a more intricate discriminator model based on the Barron space. However, we are currently constructing a model to conduct experiments on a real-world image dataset. We will report the results if we obtain them before the rebuttal period.

Reviewer comment (Writing):
Just after Lemma 1:
> Note that the quantities $m_{\tilde q}(x′)$ and $d_{\tilde q}(x′)$ only depend on the auxiliary distribution $q$, not on the distribution $p$.  
I found this sentence confusing, as $m_{\tilde q}(x′)$ and $d_{\tilde q}(x′)$ depend on the inverse auxiliary distribution $\tilde q$, which is related to the auxiliary distribution $q$ in a way that depends on $p$ (namely, through Eq. (10)). Perhaps this was a typo (i.e., "auxiliary distribution $q$" should have been "inverse auxiliary distribution $\tilde q$")? Even so, I don't really understand the point of this sentence.

Author response:
We have modified "auxiliary distribution $q$" to "inverse auxiliary distribution $\tilde{q}$". The reason we emphasize this fact is that $\tilde{q}$ is the most central function in Vicinal Estimation, as only the condition on $\tilde{q}$ is needed to obtain these values and the Generalization Bound that follows. Moreover, only $\tilde{q}$ is used in actual implementation in **Algorithm 2**. We've modified the sentence after **Lemma 1** to emphasize this fact.