Paper: SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer (ICLR-2024)

Reviewer comment (Novelty):
The article's motivation, which revolves around the examination of whether trained discriminators effectively supply gradient information to optimize generators for reducing dissimilarities, appears to have been explored in prior articles [1-5].

Author response:
Thank you for suggesting the papers. We have carefully checked the papers, and found some of them are helpful to highlight the uniqueness of our theoretical perspective more. 

We would like to first clarify the problem we are tackling to make it clear the difference between our work and the suggested papers. Since practical GAN training cannot exactly maximize the inner problem $\mathcal{V}$ w.r.t. the discriminator, we aim to provide a sufficient condition for **discriminators to function as certain distances**, which is more feasible during the training. The new theoretical framework relieves us from an impractical assumption that discriminators keep optimal throughout the training.

[5] is a pioneering work that first introduces the concept of gradient flows in the GAN problem. The authors suspected that one reason of difficulty of GAN training comes from limitation of the representational power of the generator. Hence they proposed to insert gradient layers into the generator to enhance the representational power. They conducted convergence analysis based on smoothness analysis of generator loss, which does not assume optimal discriminator. However, the theoretical results **do not care about the property of the convergent points**. The work did not discuss whether trained discriminators with the proposed method indeed provide generator optimization with gradients that reduce dissimilarities. [4] is another work improving GAN training with the concept of functional gradient learning. They explicitly assume that **discriminators are optimal**. 
 
[2-3] proposed training schemes of generative models based on variational gradient flow, which is different from but related to GAN. They rely on the primal form of the original divergence minimization problem unlike usual GANs. They still use the discriminator as an **ideal** density estimator, which is obtained by adversarial training as in GANs. It means the proposed algorithm is based on the assumption that **the discriminator keeps optimal**, which is required for having such an ideal density estimator.

Lastly, [1] deals with the problem of variational inference, and is less relevant to the topic of GAN.

We appreciate your listing the papers again. We have added the references [2-4] to the second paragraph of Appendix E.2.

[1] Variational inference via Wasserstein gradient flows

[2] Variational Wasserstein gradient flow

[3] Deep Generative Learning via Variational Gradient Flow

[4] A Framework of Composite Functional Gradient Methods for Generative Adversarial Models

[5] Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models

Reviewer comment (Evaluation):
The article falls short in providing a clear connection between the theoretical results and their practical implications.

Author response:
First, we briefly explain how each of the metrizable conditions works in our theorem by taking Wasserstein GAN as an example. We believe this is helpful to get why the three conditions are crucial. Our strategy to derive Theorem 5.3 is basically to connect GAN problem with a variant of sliced Wasserstein, max-ASW, by using the novel framework, a functional mean divergence\* (FM\*), as depicted in Figure 1. First, *direction optimality* ensures that $\mathcal{J}\_{Wass}$ corresponds to a minimization objective for FM\*. Next, *separability* guarantees the equivalence of FM${}^*$ to the max-ASW. Now, thanks to the two conditions, the max-ASW can be minimized by $\mathcal{J}\_{Wass}$. Lastly, *injectivity* is needed to ensure that the max-ASW is indeed a distance.

Next, we briefly provide our intuitions to the conditions. With the representation of the discriminator $f(x)=\langle\omega,h(x)\rangle$, it is interpreted as extracting features $h(x)$ and projecting them onto a one-dimensional space with the direction $\omega$ (please also refer to the final paragraph of Section 6). *Injectivity* and *separability* are conditions for the feature extractor $h(x)$. 
1. *Injectivity* literally indicates that of function $h(x)$, aimed at preventing the loss of information from the original real/fake samples.
2. *Separability* is characterized by the cumulative density functions of $\mathcal{S}^hI_{\mu_{\theta}}(\cdot,\omega)$ and $\mathcal{S}^hI_{\mu_0}(\cdot,\omega)$ . This condition ensures that optimal transport maps for all the samples from $\mathcal{S}^hI_{\mu_{\theta}}(\cdot,\omega)$ to $\mathcal{S}^hI_{\mu_0}(\cdot,\omega)$ share the same direction. We will elaborate an illustration regarding *separability*.
3. *Direction optimality* is a condition for $\omega$ to project the features with a normalized projection that most effectively distinguishes the sets of real and fake samples. 

We have incorporated the above explanations and additional illustrations into the revised manuscript in order to convey the key concept in a more intuitive way. For the details about how we have reflected the above on the manuscript, please refer to our comments on your questions below.

Reviewer comment (Theory):
The significance of the theoretical findings might benefit from more detailed explanation.

Author response:
First, we briefly explain how each of the metrizable conditions works in our theorem by taking Wasserstein GAN as an example. We believe this is helpful to get why the three conditions are crucial. Our strategy to derive Theorem 5.3 is basically to connect GAN problem with a variant of sliced Wasserstein, max-ASW, by using the novel framework, a functional mean divergence\* (FM\*), as depicted in Figure 1. First, *direction optimality* ensures that $\mathcal{J}\_{Wass}$ corresponds to a minimization objective for FM\*. Next, *separability* guarantees the equivalence of FM${}^*$ to the max-ASW. Now, thanks to the two conditions, the max-ASW can be minimized by $\mathcal{J}\_{Wass}$. Lastly, *injectivity* is needed to ensure that the max-ASW is indeed a distance.

Next, we briefly provide our intuitions to the conditions. With the representation of the discriminator $f(x)=\langle\omega,h(x)\rangle$, it is interpreted as extracting features $h(x)$ and projecting them onto a one-dimensional space with the direction $\omega$ (please also refer to the final paragraph of Section 6). *Injectivity* and *separability* are conditions for the feature extractor $h(x)$. 
1. *Injectivity* literally indicates that of function $h(x)$, aimed at preventing the loss of information from the original real/fake samples.
2. *Separability* is characterized by the cumulative density functions of $\mathcal{S}^hI_{\mu_{\theta}}(\cdot,\omega)$ and $\mathcal{S}^hI_{\mu_0}(\cdot,\omega)$ . This condition ensures that optimal transport maps for all the samples from $\mathcal{S}^hI_{\mu_{\theta}}(\cdot,\omega)$ to $\mathcal{S}^hI_{\mu_0}(\cdot,\omega)$ share the same direction. We will elaborate an illustration regarding *separability*.
3. *Direction optimality* is a condition for $\omega$ to project the features with a normalized projection that most effectively distinguishes the sets of real and fake samples. 

We have incorporated the above explanations and additional illustrations into the revised manuscript in order to convey the key concept in a more intuitive way. For the details about how we have reflected the above on the manuscript, please refer to our comments on your questions below.

Reviewer comment (Writing):
If I understand correctly, "direction optimality" implies that a negative gradient of the metric between the generated distribution and the target distribution guides the generated distribution toward the target. "Separability" pertains to two disjointed distributions, and "injectivity" suggests that the metric between the generated and target distributions is a distance. Is this interpretation accurate?

Author response:
Thank you for elaborating your intuition. The interpretations regarding *direction optimality* and *injectivity* are correct. *Optimal direction* corresponds to the direction increasing the distance most. *Injectivity* ensures the metric induced by the discriminator is indeed a distance, as you mentioned.

*Separability* and “disjointed distributions” are disconnected somewhat. Whether the distributions of $h(x)$ with $x\sim\mu_{\theta}$ and $x\sim\mu_0$ are disjoint depends more on whether $\mu_0$ and $\mu_\theta$ are disjoint from each other. *Separable* h(x) does not necessarily mean such disjoint distributions in the feature space. 

The comment inspired us to provide our intuitive explanations for the conditions. Please kindly refer to our comment in the thread **Preliminary comments**. We have put the intuitions in the revised manuscript as below.

1. *direction optimality* right after Proposition 3.5,

2. *separability* right after Lemma 4.3, and 

3. *injectivity* right before Lemma 4.4

Furthermore, we have incorporated additional figure (Figure 1 in the revised manuscript), which illustrates *direction optimality* and *separability*.

Reviewer comment (Presentation):
Could you clarify the symbol "$*$" in Table 1?

Author response:
Thank you for the question. We apologize for the lack of clarity about the symbol. Whether the trained discriminators are injective is **not directly affected by the selected maximization objective** but by other factors such as discriminator design (e.g., activation functions) and regularization objectives for discriminators. Because the aim of Table 1 is mainly to show the inductive effects of the GAN objectives on the metrizable conditions, we refrained from using cross and check marks in the injectivity column. We have clarified it in the caption of Table 1 of the updated manuscript.

Reviewer comment (Writing):
I recommend that when the article first mentions metrizable conditions, namely, direction optimality, separability, and injectivity, it would be more reader-friendly to provide an intuitive understanding of these terms and indicate which paragraphs explain these definitions. This would help readers quickly grasp the key concepts before delving into the detailed explanations.

Author response:
Thank you for the suggestion. We have reflected the feedback on the new manuscript. Please kindly refer to our reply to the first question for how we have reflected.

Reviewer comment (Theory):
Regarding the statement: "most existing GANs besides Wasserstein GAN do not satisfy direction optimality with the maximizer $w$ of $\mathcal{V}$" — although most existing GANs do not satisfy direction optimality as proposed in the article, the discriminator of these GANs actually provides the generator with gradients that make its distribution close to the target distribution, as discussed in other articles [1-5]. What do you think about this question?

Author response:
Please refer to our reply to **weakness**. There is no guarantee that practical (suboptimal) discriminators, optimized by existing GANs, provide the generator with such desirable gradients. We formulate the fundamental problem of whether suboptimal discriminators can still yield such gradients, by introducing a novel concept: the metrizability of discriminators (see Definition 1.1). We address this question without imposing the optimal discriminator assumption.

Reviewer comment (Theory):
I recommend that the author provide a clear explanation of why Equation (16) satisfies Theorem 5.3. I find it somewhat confusing and would appreciate a more comprehensible clarification.

Author response:
We acknowledge your suggestion, and believe it will make the idea behind our loss design clearer. As you suggested, we have put a sentence right after Equation (16) to explain that the first and second terms in Equation (16) respectively induce *separability* and *direction optimality*.

More intuitively, the loss design can be explained by our observations summarized in Table 1: (1) we employ the general GAN maximization objective, distinct from Wasserstein GAN, to induce the *separability*, and (2) Wasserstein GAN’s maximization objective, with some modifications according to $\mathcal{J}_{GAN}$, is used for $\omega$. Please note that, as in our reply to the Question 2, *injectivity* is not directly affected by the maximization objective but induced by the discriminator architecture design.

Reviewer comment (Reproducibility):
I have reviewed the code in the supplementary materials. There is a discrepancy between the code provided in Listing 2 (SAN discriminator) and lines 92-96 in the file "STYLESAN-XL/pg_modules/san_mondules.py/SANConv2d." To ensure accuracy, please verify which version is the correct one.

Author response:
We sincerely appreciate your checking our source code carefully. Our StyleSAN-XL implementation is consistent with the pseudo code in Listing 2, and there is no essential gap between them. We have added a supplementary explanation as described below to Appendix F.2.

In the implementation, the discriminator is implemented as $f(x)=s\langle\omega,\tilde{h}(x)+b\rangle$, where $s$ is a trainable scalar and $\tilde{h}(x)+b$ is an input of the last convolutional layer. This discriminator is equivalent to $f(x)=\langle\omega,h(x)\rangle$ with $h(x)=s(\tilde{h}(x)+b)$ because of linearity of the inner product. Since just normalizing the last linear layer may deteriorate the representation ability of the overall network, we introduced the scaling factor $s$ to keep the ability. Even with the scaling factor, the discriminator implementation still follows the concept of SAN. We calculated the discriminator outputs with $\omega$ detached for $\langle\omega^-,h\rangle$, while detached both $s$ and $\tilde{h}(x)+b$ to get $\langle\omega,h^-\rangle$.

Reviewer comment (Theory):
I find it quite confusing to understand the importance of the three conditions in your Theorem 5.3. In my opinion, direction optimality should be the most important condition, while the separability and injectivity conditions may not carry the same weight. This is because metrics used in GANs, such as f-divergences or IPM functions, inherently satisfy the distance property. Maintaining the separability condition becomes increasingly challenging as GAN training progresses and the generated distribution approaches the real data distribution. Therefore, I recommend that the author clearly outline the relationships and significance of all three conditions to provide a better understanding.

Author response:
Thank you for the comment. *Direction optimality* is the most important condition for our methodology part because the idea behind SAN is to impose the property on arbitrary GANs. We have stressed that *direction optimality* is particularly crucial for our methodological proposal, right after presenting Theorem 5.3 in the revised manuscript. Adding such a note near to the theorem should be helpful for readers to find the connection of Theorem 5.3 and the methodology section (Section 6). For explanations for how each condition works in the theorem, please refer to our comment in the thread **Preliminary comments**.

Although GAN formulation is based on dual formulation of such metrics, the distances can only be evaluated using the **optimal discriminators**. While the optimal discriminator assumption is convenient for theoretical analysis of GANs, achieving such optimality in practical cases is rarely feasible. In summary, there is no guarantee that practical GANs satisfy the distance property. As explained in our comment on the **weakness**, the aim of our work is to answer the question whether the discriminator can serve as a distance without relying on such strong assumptions.

We agree that the *separability* condition becomes increasingly difficult to achieve as the generator distribution approaches the data distribution. However, as long as $\mu_{\theta}\neq\mu_0$, a class of *separable* functions $h$ includes a broad range of functions, whereas the optimal discriminator is always unique for the given measures $(\mu_{\theta},\mu_0)$. In this sense, the optimal discriminator is much harder than the metrizable conditions to achieve.

We appreciate your valuable suggestions. We have put an illustration (Figure 3 of the updated manuscript) in the updated manuscript to clearly show the relationship and significance. Lastly, we have reflected your suggestions regarding the three key conditions on the manuscript, but we would appreciate your further feedback if there still remains something unclear.

Reviewer comment (Experiments):
The authors are encouraged to conduct experiments on CIFAR-100 using BigGAN and FFHQ using StyleGAN2 to further verify the effectiveness of the proposed method.

Author response:
We greatly appreciate the valuable feedback you have provided, and have addressed each of your comments. Please kindly find our response below. 

**Weakness**
---
> 2. The authors are encouraged to conduct experiments on CIFAR-100 using BigGAN and FFHQ using StyleGAN2 to further verify the effectiveness of the proposed method.

We agree that expanding the evaluation using additional datasets could provide more comprehensive validation of the proposed method's effectiveness. To further extend our experiments, we additionally trained GAN and SAN on CIFAR-100 using BigGAN architecture. We followed the same hyperparameter setting (e.g., learning rate) as employed for CIFAR-10, yielding the numerical results reported below. SAN exhibits superior performance. Especially, FID has been significantly improved by 25%. We have added the results to Section 7.2 in the updated manuscript..

| CIFAR100 | FID ($\downarrow$) | IS ($\uparrow$) |
| --- | --- | --- |
| BigGAN | 10.73+/-0.16 | 10.56+/-0.01 |
| BigSAN | **8.05**+/-0.04  | **10.72**+/-0.05 |

Reviewer comment (Writing):
To enhance accessibility, the authors should consider simplifying the mathematical notation to cater to a wider audience.

Author response:
We greatly appreciate the valuable feedback you have provided, and have addressed each of your comments. Please kindly find our response below. 

**Weakness**
---
> 3. To enhance accessibility, the authors should consider simplifying the mathematical notation to cater to a wider audience.

We agree that some notations are a bit complicated. We have simplified the following notations and their combinations in the revised manuscript. Besides, we have added Table 6 to summarize some key notations, which are not introduced in Section 2.1.

- $\mathcal{J}\_{\text{Wass}}$ $\Longrightarrow$ $\mathcal{J}_{W}$

- $\mathcal{D}^{\text{max-ASW}}\_{\mathcal{F}}$ $\Longrightarrow$ $\mathcal{D}^{\text{mA}}_{\mathcal{F}}$

- $\mathcal{D}\_{\mathcal{F}\_{\text{Inj}}}$ $\Longrightarrow$ $\mathcal{D}\_{\mathcal{F}\_{\text{I}}}$

- $\mathcal{D}\_{\mathcal{F}\_{\text{Sep}(\mu,\nu)}}$ $\Longrightarrow$ $\mathcal{D}\_{\mathcal{F}\_{\text{S}}}$

- $\mathcal{D}\_{\mathcal{F}\_{\text{Inj}}\cap\mathcal{F}\_{\text{Sep}(\mu,\nu)}}$ $\Longrightarrow$ $\mathcal{D}\_{\mathcal{F}\_{\text{I,S}}}$

Reviewer comment (Reproducibility):
In Table 4, are the results shown for BigGAN the ones obtained after replacing ReLU with LeakyReLU?

Author response:
**Questions**
---
> 2. In Table 4, are the results shown for BigGAN the ones obtained after replacing ReLU with LeakyReLU?

We deeply appreciate your keen observation. The discriminator of the original BigGAN is constructed with ReLUs. As our main goal of the experiment is to validate our proposed training scheme proposed in Section 6, we did not change any other conditions including the activation functions. We have clarified this in Appendix F.2 of the revised manuscript.

Inspired by the question, we replaced ReLUs with leakyReLUs in the discriminator, and subsequently trained BigSAN on both CIFAR-10 and CIFAR-100, for ablation of the injectivity induction by the activation functions. We got the results listed in the table below, showing that the use of leakyReLU leads to better or competitive scores than that of ReLU in both cases. The empirical result is also insightful as it is consistent with our claim. We have put the experimental results in Appendix H.3 of the revised manuscript. Thanks again for your valuable comment.

| CIFAR10 | FID ($\downarrow$) | IS ($\uparrow$) |
| --- | --- | --- |
| BigSAN (w/ ReLU) | 6.20+/-0.27  | 9.16+/-0.08 |
| BigSAN (w/ leakyReLU) | **5.97**+/-0.12 | **9.23**+/-0.03 |

| CIFAR100 | FID ($\downarrow$) | IS ($\uparrow$) |
| --- | --- | --- |
| BigSAN (w/ ReLU) | 8.05+/-0.04  | 10.72+/-0.05 |
| BigSAN (w/ leakyReLU) | **7.97**+/-0.06 | **10.84**+/-0.03 |