Paper: SPI-GAN: Denoising Diffusion GANs with Straight-Path Interpolations (ICLR-2024)

Reviewer comment (Novelty):
SPI-GAN is highly grounded by StyleGAN2 (generator's backbone, ADA training), and I am not sure if the score-based components are important. From Fig.6, by replacing the NODEs-based mapping network with StyleGAN2's mapping network, the model's performance is almost unchanged. That new network is pretty much StyleGAN2-Ada, but instead of modeling the clean image only, the model learns to generate the linear-interpolated version between the clean image (i(1)) and its full-noise version (i(0)), conditioning on an interpolation factor u. The NODEs-based mapping network, while being sophisticated, may not be necessary and can be replaced by a simple MLP-based solution. It makes SPI-GAN look like a pure GAN model, disguising itself as a diffusion model. It does not devalue the paper but changes its narrative.

Author response:
> Can you replace the NODEs-based mapping network with StyleGAN2's mapping network, and report the performance of that model in Table 1-3? Does SPI-GAN have noticeable better performance compared with that pure GAN method?

$\to$ In Fig.6, there is a slight degradation in performance even after replacing the NODE-based mapping network with the StyleGAN2's original mapping network. The StyleGAN2's original mapping network is able to produce good images because the condition $u$ is given to it. In other words, we concatenate the time condition $u$ with input to the StyleGAN2's mapping network.
However, there is a difference between our NODE-based mapping network and the StyleGAN2’s mapping network. NODE-based mapping networks have homeomorphic characteristics. Because of this homomorphic characteristic, SPI-GAN with a NODE-based mapping network shows an appropriate denoising process when $\hat{\mathbf{i}}(0)$ is generated using $\mathbf{h}(0)$ to $\mathbf{h}(1)$, given a fixed $z$. In contrast, StyleGAN2's mapping network, which has non-homeomorphic properties, does not show a denoising process even when time point $u$ is used as a condition. A visualization of the difference between the NODE-based mapping network and the StyleGAN2 mapping network  is in Figure 10 in Appendix C in our revised paper.

Reviewer comment (Evaluation):
In Fig.8-Left, given the same initial embedding, latent vectors h(u) at different time-step u produce completely different output images. It does not match the expected behavior of diffusion models.

Author response:
> In Fig.8-Left, given the same initial embedding, latent vectors h(u) at different time-step u produce completely different output image. It does not match the expected behavior of diffusion models. Can you explain why?

$\to$ SPI-GAN learns the data distribution at arbitrary time $u \in [0,1]$. The diffusion model has an auto-regressive denoising process and therefore, one sample is continuously denoised. However, SPI-GAN does not do it but it can learn the ground-truth data distribution at time $u$.

Reviewer comment (Experiments):
The authors should ablate the model performance when adding the time-step (u) input to the generator. Ablation studies on the role of s are also recommended.

Author response:
> The authors should ablate the model performance when adding the time-step (u) input to the generator. Ablation studies on the role of s are also recommended.

$\to$ Thank you for the suggestion. As your suggestion, we report the results of ablate the model performance when adding the time-step ($u$) input to the generator in the following table. Table shows that there is no significant difference from SPI-GAN even if the time point $u$ is explicitly given to the generator as a condition. This can be confirmed in Table 10 in Appendix E.6 in the revision. However, $s$ used as input to the generator is a layer-wise varying noise, which was proposed in StyleGAN2 for stochasticity and is not the contribution of our SPI-GAN. After removing $s$, the stochasticity of our model is removed, which is not appropriate when the original diffusion model is stochastic.

|            CIFAR-10           |  FID |
|:-----------------------------:|:----:|
| SPI-GAN (condition $u$ to generator) | 3.03 |
|            SPI-GAN            | 3.01 |

Reviewer comment (Presentation):
Fig 7: The x ticker labels are too close to each other. The caption should be improved, e.g., "stochastic" -> "stochasticity".

Author response:
> Fig 7: The x ticker labels are too close to each other. The caption should be improved, e.g., "stochastic" -> "stochasticity".

$\to$ Thank you for your comments. We revised Figure 7 and its caption in the revised paper for clarity.

Reviewer comment (Experiments):
The claim of achieving diversity as good as score-based models is made based on datasets that are, in some cases, insufficient for assessing diversity. The resolution of the datasets is often low, and they are primarily single-domain (faces and churches). To validate the diversity claim, it is essential to perform experiments on more extensive, real-world datasets, such as ImageNet, which encompasses a broader range of categories and challenges.

Author response:
$\to$ Thank you for suggesting an experiment that demonstrates the claim that it achieves as much diversity as a score-based model. However, due to lack of time, we were unable to experiment with ImageNet 64x64, but we will add the results after the discussion period.

Reviewer comment (Experiments):
While diversity is addressed in Table 4, there is a noticeable absence of baseline comparisons with vanilla StyleGAN2 or other diffusion models. Such comparisons would help in understanding whether the method genuinely offers improvements in terms of diversity, or if it simply matches existing capabilities.

Author response:
$\to$ Thank you for your suggestion. In Table 4, we selected and compared only the best performing models in LSUN-Church-256: Diffusion-GAN, DD-GAN, and CIPs. We will add vanilla StyleGAN2 or other diffusion models as a baseline comparison.

Reviewer comment (Experiments):
Have the authors considered the potential applicability of the implicit noise schedule produced by their method to other diffusion models? An exploratory comparison with commonly used schedulers in the field could provide valuable insights.

Author response:
$\to$ Yes, we only focused on imitating diffusion models (especially variance preserve) using GAN, but we expect that the optimal denoising process can be applied to models with different noise diffusion schedules such as NCSN [1] and EDM [2]. 

[1] Song, Yang, and Stefano Ermon. “Generative modeling by estimating gradients of the data distribution.” Advances in neural information processing systems 32 (2019).

[2] Karras, Tero, et al. “Elucidating the design space of diffusion-based generative models.” Advances in Neural Information Processing Systems 35 (2022): 26565-26577.

Reviewer comment (Experiments):
Is there a discernible difference between implicitly including the noise level in the latent space, as done in the paper, and explicitly passing it like the discriminator?

Author response:
$\to$ The generator appears to include the noise level implicitly. However, since we use the latent vector generated from a NODE-based mapping network, which explicitly includes the noise level, as input to the generator, the generator explicitly includes the noise level. Even if time point $u$ is additionally given to the generator as a condition, the results with SPI-GAN are not significant. We report the performance of this experiment in the following table. This can be confirmed in Table 10 in Appendix E.6 in the revision.

|            CIFAR-10           |  FID |
|:-----------------------------:|:----:|
| SPI-GAN (condition $u$ to generator) | 3.03 |
|            SPI-GAN            | 3.01 |

Reviewer comment (Theory):
The rationale behind integrating the diffusion process into the generator, particularly in terms of applying it to the latent code input, remains unclear. This might be attributable to the discriminator being exposed to a variety of augmented images, potentially helping to avert overfitting. Meanwhile, the results in Table 3 depict a noticeably inferior performance of SPI-GAN in comparison to both Diffusion-GAN and StyleGANs, raising questions about the soundness of the approach of employing the diffusion process on the generator.

Author response:
$\to$ Our goal is to decrease the large sampling time of the diffusion model by letting a generator learn the straight-path interpolation between latent codes and their corresponding clean images. Given a source and a destination, a Brownian motion used by the diffusion model is inefficient so we propose to **walk** along the straight-path interpolation.

To this end, we first use a neural ODE-based mapping network to convert the straight-path interpolation into the hidden dynamics of $\mathbf{h}(u)$, where $u$ is in $[0,1]$. Therefore, $\mathbf{h}(u)$ contains the information of what to generate at time $u$. Moreover, neural ODEs are theoretically homeomorphic, i.e., continuous and bijective. In Appendix C, we describe the appropriateness of using homeomorphic NODEs for modeling the denoising process of SGMs. For similar purposes, neural ODEs had been already used to learn hidden dynamics of the hidden vectors for videos in [1].

We then train our generator to produce realistic samples from the hidden dynamics of $\mathbf{h}(u)$, where $u$ is in $[0,1]$, being advised by the discriminator. In this way, our generator is able to produce samples along the straight-path interpolation.

[1] Park, Sunghyun, et al. "Vid-ode: Continuous-time video generation with neural ordinary differential equation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 3. 2021.

Reviewer comment (Experiments):
An ablation study could be beneficial, exploring alternatives to using the Neural Ordinary Differential Equation (NODE) on the latent code. Investigating the generator’s performance when directly conditioned on the diffused latent code $h(u)$—without the NODE—or simply conditioned on $u$, could provide valuable insights into the viability of such designs.

Author response:
$\to$ In Figure 6, we have the result of deriving h(u) by using the latent code and $u$ as input to the StyleGAN2 mapping network without NODE. Additionally, as you suggest, we report the results of simply applying $u$ as a condition to the generator, without a mapping network in the following table. The following table shows that a mapping network is essential for SPI-GAN. This can be confirmed in Table 9 in Appendix E.6 in the revision.

|            CIFAR-10           |  FID |
|:-----------------------------:|:----:|
| SPI-GAN (w/o mapping network) | 5.72 |
|            SPI-GAN            | 3.01 |

Reviewer comment (Experiments):
The observed performance gains over the baseline are relatively modest. Particularly in the case of LSUN-Church-256, the proposed method trails behind both Diffusion GANs and StyleGAN2. It's also inconvincing that the paper does not include a comparison of SPI-GAN with DiffusionGAN on CelebA-HQ-256, which could have provided a more comprehensive evaluation.

Author response:
$\to$ In LSUN-Church-256, SPI-GAN is inferior to Diffusion-GAN and StyleGAN in terms of FID performance, but please note that it shows superior to Diffusion-GAN and CIPs (State-of-the-art model in LSUN-Church-256) in terms of Recall and Coverage, which are recently proposed advanced metrics.

Reviewer comment (Reproducibility):
Is $u$ applied directly to the image, or is it integrated into an intermediate layer within the discriminator’s architecture?

Author response:
$\to$ $u$ is the temporal condition of the generated image and is used as input to the discriminator. In other words, $u$ is concatenated with the generated image and used as input to the discriminator. Using this concatenated information, the discriminator is able to know the temporal condition of input images.

Reviewer comment (Experiments):
Given that $h(u)$ can be derived from $i(u)$ in the current design, I'm curious to know if the generator has the ability to create images that are very similar to the real images, $i(u)$.

Author response:
$\to$ As your comment, we compare $\hat{\mathbf{i}}(u)$ derived from $\mathbf{h}(u)$ and the ground truth $\mathbf{i}(u)$ in Figures 15 in the Appendix of our revised paper. As a result, it appears that $\mathbf{i}(u)$ and $\hat{\mathbf{i}}(u)$ have similar distributions.