Paper: On the Analysis of GAN-based Image-to-Image Translation with Gaussian Noise Injection (ICLR-2024)

Reviewer comment (Theory):
The discussion after __Lemma 1__ only covers non-Gaussian noise, but the majority of experiments are about non-Gaussian source. The gap of source distribution should at least be explicitly discussed in the main text not appendix, if not well addressed. Currently the proof of __Corollary 1__ seems to rely on Eq. 6 of __Lemma 1__, which holds only when the source is Gaussian. If I understand correctly, in that case, it becomes unreasonable to derive the optimal noise level $0.08$ for actual image generation, which is obviously non-Gaussian. If the theory is not connected to the experiment, then this paper becomes a little bit unconvincing, as no new empirical approach is proposed and the empirical results alone are not enough for accepting this paper.

Author response:
We appreciate the reviewer raising this point. 

For **Lemma** 1 and **Corollary** 1, as the derivations require closed-form expressions of KL divergences, extension to general non-Gaussian sources cannot be easily achieved. For **Theorem** 2, we generalize some results to non-Gaussian sources in **Theorem** 3. 
**Theorem 3:** Let $\\mathbf{X}$ be a $d$-dimensional random vector. Denote $\\theta(\\sigma^2\_t,\\sigma^2\_e\\boldsymbol{\\Sigma}\_{\\widetilde{e}})$ as the KL-divergence of non-Gaussian source signal for arbitrary noise, $\\theta\_g(\\sigma^2\_t,\\sigma^2\_e\\boldsymbol{\\Sigma}\_{\\widetilde{e}})$ denotes the special case of $\\theta(\\sigma^2_t,\\sigma^2_e\\boldsymbol{\\Sigma}_{\\widetilde{e}})$ when $\\boldsymbol{E}$ is Gaussian noise. Then for small $\\sigma^2_e$ with $\\sigma^2_e\\ll 1$, we have

$\\theta(\\sigma^2\_t,\\sigma^2\_e\\boldsymbol{\\Sigma}\_{\\widetilde{e}})=\\theta\_g(\\sigma^2\_t,\\sigma^2\_e\\boldsymbol{\\Sigma}\_{\\widetilde{e}})+o(\\sigma^2\_e)$

Thus, **Theorem 3** generalizes the result of **Theorem 2** to non-Gaussian sources, demonstrating that an I2I system's resilience to Gaussian noise ensures robustness against noises with the same covariance matrices, regardless of whether the input signals are Gaussian or non-Gaussian. 

We want to emphasize that adopting Gaussian signal models is a common theoretical approach even when practical signals are non-Gaussian. This allows deriving foundational insights that can be obscured in more complex settings. For example, in I2I, the widely used Fréchet Inception Distance (FID) assumes Gaussian distributions, despite natural images not being strictly Gaussian. Our work follows this principled methodology - establishing core theoretical results with Gaussian signals first.

We agree extending the analysis to non-Gaussian sources is an important future direction. In fact, we take a first step in deriving Theorem 3 of the revised paper, showing the covariance-matching result holds for non-Gaussian signals when noise levels are small. However, a full generalization of all results is non-trivial and requires care to handle the increased complexity.

By starting from Gaussian signal analysis, we lay a solid foundation upon which future research can build. This approach provides a rigorous basis for expanding into non-Gaussian settings, offering fundamental insights that can be further enriched to encompass more realistic signal models.

Reviewer comment (Theory):
Is it possible to extend current theory (__Lemma 1__, __Theorem 2__, __Corollary 1__) to mixture of Gaussian source? As we can approximate any distribution, including natural image distribution with GMM, it would be much better if we can extend the theory to GMM, even for finite mixture. In this way, the distance between the current theory and empirical results on natural image can be reduced.

Author response:
Thanks for your suggestions of an extension to GMM signals. We include the discussions in Appendix of our revised paper.  

Consider a source signal $\\boldsymbol{x}$ represented by the Gaussian mixture model that $p\_{\\boldsymbol{X}}(\\boldsymbol{x})=\\sum\_{k=1}^{N}\\pi_k\\cdot\\mathcal{N}(\\boldsymbol{x}|\\boldsymbol{\\mu}\_k,\\boldsymbol{\\Sigma}\_k)$, $\\sum\_{k=1}^{N}\\pi_k=1$ and $\\pi_k>0$.  Although we cannot get a closed-form expression of the KL divergence for the GMM signal source, we can extend our results to its upper-bound, as listed below: 

- **Extention of Lemma 1**:  By using the convexity property of KL divergence, we have $D\_{KL}(\\hat{P}\_{\\boldsymbol{X}+\\boldsymbol{E}}\\|\\bar{P}\_{\\boldsymbol{X}+\\sigma\_t\\boldsymbol{N}})\\leq\\zeta(\\sigma^2\_t,\\boldsymbol{\\Sigma}\_e)$. 
$$
\begin{aligned}
\\zeta(\\sigma^2\_t,\\boldsymbol{\\Sigma}\_e)&= \\sum_{k=1}^{N}\\pi\_k\\cdot D\_{KL}(\\hat{P}\_{\\boldsymbol{X}\_k+\\boldsymbol{E}}\\|\\bar{P}\_{\\boldsymbol{X}\_k+\\sigma\_t\\boldsymbol{N}})\=\\sum_{k=1}^{N}\\pi_k\\cdot\\rho\_{k}(\\sigma_t^2,\\boldsymbol{\\Sigma}\_e),
\end{aligned}
$$
where $\\boldsymbol{X}$ is d-dimensional random variable of the GMM represented source signal $\\boldsymbol{x}$, $\\boldsymbol{X}\_k$ is the $k$-th Gaussian distribution with its weight $\pi_k$ for GMM and $\rho_{k}(\sigma_t^2,\boldsymbol{\Sigma}_e)$ is the KL-divergence of $k$-th Gaussian distribution for arbitrary noise. 
The above equation implies that for GMM fitted source signal, its KL-divergence concerning arbitrary noise is upper-bounded by the weighted sum of the KL-divergence of $N$ Gaussian source signals with respect to arbitrary noise. 

- **Extension of Theorem 2**:  For $\\zeta(\\sigma^2\_t,\\boldsymbol{\\Sigma}\_e)$ given above, when $\\boldsymbol{\\Sigma}\_e=\\sigma^2\_e\\boldsymbol{\\Sigma}\_{\\tilde{e}}$,   $\\zeta(\\sigma^2\_t,\\boldsymbol{\\Sigma}\_e)$ is also convex in $\sigma^2_e$ as each  $\rho_{k}(\sigma_t^2,\boldsymbol{\Sigma}_e)$ is convex in $\sigma^2_e$. Likewise, $\zeta(\sigma^2_t,\boldsymbol{\Sigma}_e)<\zeta(0,\boldsymbol{\Sigma}_e)$ when $\boldsymbol{\Sigma}_e>\sigma^2_t/2$. 

- **Extension of Corollary 1**: Under the assumptions of Corollary 1, the optimal solution 
$$
\\bar{\sigma}\_{t,o}^2 = \\arg \\min\_{\\sigma\_t^2} \\mathbb{E}\_{\\sigma\_e^2\\sim\\mathcal{U}(0,\\lambda\_{\\max})}\\left\\{\\zeta(\\sigma^2\_t,\\sigma\_e^2\\boldsymbol{I}\_d)\\right\\}
$$ 
is still $\\bar{\\sigma}\_{t,o}^2=\\frac{1}{2}\\lambda\_{\\max}$, which remains the same as that of the Gaussian source.

Reviewer comment (Theory):
Current analysis on joint distribution looks quite general, can it be extended into any conditional GAN, beyond I2I?

Author response:
We thank the reviewers for this suggestion. This would be considered in our future work for other applications of conditional GAN.

Reviewer comment (Experiments):
What about more real-life noise, e.g. JPEG compression?

Author response:
Following the reviewers' advice,  we expanded our simulations to include signal-dependent noise and corruptions using the ImageNet-C corruption models. Specifically, we adopted the widely-used image degradation model $\\boldsymbol{\\hat{x}}=\\boldsymbol{A}(\\boldsymbol{x})+\\boldsymbol{n}$, where $\\boldsymbol{x}$ and $\\boldsymbol{\\hat{x}}$ denote clean and corrupted images, respectively, $\\boldsymbol{A}$ represents an operator (e.g., JPEG compression, blurring, pixelation, etc.), and $\\boldsymbol{n}$ denotes the noise component. 

Our expanded simulations covered shot noise, speckle noise, JPEG compression, pixelation, blurring, and combinations of these distortions. Key findings include:

 - When signal-dependent noise $\\boldsymbol{n}$ is present alone, models trained with Gaussian noise injection exhibited substantial improvements over baseline models trained solely on clean images, akin to the results for signal-independent noises.
 - For blurring, pixelation and JPEG compression only (with $\\boldsymbol{n}=0$),  models trained with Gaussian noise injection are inferior to the baselines in some cases. 
 - When both $\\boldsymbol{A}$ and $\\boldsymbol{n}$ were present, models trained with Gaussian noise injections consistently demonstrated significant enhancements over the baseline models. This observation aligns with practical scenarios where both $\\boldsymbol{A}$ and $\\boldsymbol{n}$ coexist in degradation models, underscoring the efficacy of Gaussian noise injection.

Notably, in cases where the corruption model excluded the noise component, the addition of a small amount of noise to the input images proved effective in producing outputs with stable quality. These findings provide further insights into the robustness and potential of Gaussian noise injection in I2I for various degradations.

Reviewer comment (Novelty):
Analysis of the Gaussian noise has been widely investigated, the authors should give some comparison or analysis about them in related works.

Author response:
We appreciate your comment and would like to clarify that we have indeed addressed the context of Gaussian noise injection in related work, specifically in Section 2. In our related work section, we emphasize that existing theoretical analyses of Gaussian noise injection primarily focus on robustness against adversarial attacks in image classification and recognition tasks. These tasks involve discrete output variables, making their derivations less directly applicable to image-to-image (I2I) translation tasks, which operate in the domain of continuous image generation.

The key difference lies in the discrete nature of classification outputs compared to the more intricate image generation performed by I2I models. While some prior studies have explored noise injection in GAN-based I2I approaches, they often treat it as an empirical technique without in-depth theoretical analysis. To the best of our knowledge, our paper represents a pioneering effort in providing a comprehensive theoretical framework for understanding the role of noise injection in robust I2I tasks. For a more detailed discussion of these issues, we kindly refer the reviewer to Section 2 (related work) in the paper. We hope this clarifies the context and positioning of our research within the existing literature.

Reviewer comment (Theory):
This paper has a fatal theoretical flaw, i.e., the authors think too simply and naively about real-world degradations. They set up additive Gaussian noise in the training phase and five synthetic degradations (Gaussian, Uniform, Color, Laplacian, and Salt & Pepper) in the testing phase. However, the real-world degradations are much more complex and fundamentally different from these degradations. To be specific, as for the noise term, the real-camera raw noise produced by photon sensing comes from multiple sources (e.g., short noise, thermal noise, dark current noise, etc.) [1, 2, 3] and is further affected by the in-camera signal processing pipeline to become spatio-chromatically correlated. The real-camera noise contains both signal-dependent and -independent terms. However, the authors only consider very naive and simple signal-independent noise types, which is far from their motivation because the degradation patterns they studied simply do not exist in real-world images. Similar to the face super-resolution problem, the blur kernel cannot be explicitly estimated. The reviewer suggests that to continue studying this valuable topic, the authors should conduct experiments on real degraded images, and refers them to datasets [4], [5], and [6].

Author response:
We acknowledge the significance of modeling real-world image distortions and the development of efficient denoising algorithms. However, while crucial, these areas fall outside our work's primary focus. Our objective is to establish a theoretical framework that guarantees robustness in image-to-image (I2I) translation models when Gaussian noise injection is used during training. We address the concerns regarding the simplicity of our noise models about real-world degradations with our theoretical and empirical findings.
 1. **Theoretical Framework and Applicability:**  In Theorems 2 and 3 of our revised paper, we base our theoretical analysis on the assumption of signal-independent noise with a finite covariance matrix. These foundational assumptions are critical yet do not limit the applicability to potential real-world noise distributions. Common signal-independent noises, such as Gaussian, erasure, and impulsive noises in image transmission, and uniform noise in low-resolution quantization, are prevalent in various applications. The core insight from Theorems 2 and 3 is that certifying robustness to a broader class of signal-independent noise is feasible if an I2I system is designed to withstand Gaussian noise with the same covariance matrix.
 2. **Comparative Analysis with Related Works:** Our research aligns with significant findings in image classification, notably those presented by Cohen, Rosenfeld, and Kolter (2019) in "Certified adversarial robustness via randomized smoothing." Our study provides theoretical and empirical evidence of robustness in I2I translation models under noisy conditions, akin to the certified robustness against common corruptions and adversarial attacks discussed in their work.

 3.  **Extended Empirical Validation:** The experiments in our original paper were designed to substantiate our theoretical claims. Following the reviewers' suggestions, we expanded our simulations to include signal-dependent noise and corruptions using the ImageNet-C corruption models. This expansion included shot noise, speckle noise, JPEG compression, pixelation, blurring, and combinations of these distortions. The results revealed an improved performance with Gaussian noise injection during training, with exceptions in some cases of pure blurring, JPEG compression, and pixelation. Notably, even in these scenarios, a minor addition of noise was found to stabilize the performance. It's important to mention that *ImageNet-C (Hendrycks et al. 2019)*, published in ICLR, is a widely accepted benchmark in related studies.

Hendrycks, D., & Dieterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. ICLR 2019.

In summary, our paper primarily contributes to the theory and guarantees of noise resilience rather than precise noise modeling and the development of denoising algorithms. We recognize the complexity of real-world image degradation and emphasize that our theoretical foundation is a critical step towards understanding the impact of Gaussian noise on I2I models, paving the way for future integration of more complicated, real-world noise models.

Reviewer comment (Novelty):
This work does not have any technical contributions. Training with additive Gaussian noise has been studied in the image denoising I2I task for a long time. It is a very common setting. The robustness of noise has also been validated by many prior works. This paper does not propose any new method.

Author response:
-   While noise injection during training has been explored in prior works, our key contribution is providing the first **rigorous theoretical analysis** of this phenomenon for image-to-image translation tasks.
-  We have derived new analytical results that formally characterize the connections between distribution divergence, score matching, and the impact of Gaussian noise on alignment. To our knowledge, this level of ***mathematical insight*** has not been shown before.
-   Our theoretical findings lead to new proofs and guarantees on the generalized robustness implications of Gaussian noise resilience. We formally show this can imply robustness to broader noise classes beyond the Gaussian distributions.
-   These theoretical contributions go significantly beyond simply validating empirically known heuristics. We provide an in-depth analytical basis to fundamentally understand the mechanisms behind improved noise resilience. 
-   We derived the selection of optimal training noise levels for maximally enhancing real-world robustness. This constitutes a new theoretical contribution.

In summary, while noise injection has been explored before, we provide significant technical contributions through new formal analysis unlocking fundamental insights into this phenomenon for I2I translation. Our rigorous analytical treatment and extensive theoretical results constitute the main value of this work.

Reviewer comment (Novelty):
The idea of adding Gaussian noise and aligning distribution is highly similar to a prior work [7] that also studies real-camera degradations in I2I tasks. There is no discussion or comparison.

Author response:
The primary objective of reference [7] is the generation of pairs of real-world noisy images from clean images through adversarial training. It is crucial to note that this approach is different from our focus, which is to train the GAN with images corrupted by known noise, enabling the translation from noisy images. Our primary goal is the translation of images from the source domain, with a specific emphasis on assessing the I2I system's robustness in the presence of noise, rather than precise noise modeling or denoising algorithm formulation.

The alignment highlighted in 7 includes both image and noise alignment. While these alignments bear similarities to the divergence measure between distributions analyzed in our work, it's noteworthy that [7] does not consider these alignments to theoretical analyses. Instead, their approach relies solely on adversarial training of GANs for distribution alignment. It is widely acknowledged that the interpretability of adversarial learning in GANs often poses challenges. In contrast, our contribution involves the development of a comprehensive theoretical framework for *'alignment'* in this context. We delve into an in-depth analysis of the change in f-divergence with regard to training noise intensity, shedding light on the understanding of Gaussian noise injection in I2I systems.

Reviewer comment (Reproducibility):
Code and pre-trained models are not submitted. The reproducibility cannot be checked.

Author response:
As all baseline models utilized in our study are open source, interested readers can readily reproduce our results. Due to the substantial size of the model weights, it is not feasible to include them as supplementary materials. However, we commit to providing complete access to all codes and pre-trained weights upon the acceptance of our work.

Reviewer comment (Theory):
For the theoretical noise analysis part, if considering signal-independent and -dependent noise terms, what will happen to the analysis? How does it change?

Author response:
Thank you for your insightful comment regarding the inclusion of signal-dependent noise in our theoretical analysis. In our revised manuscript, we present empirical results that demonstrate improved performance under both signal-independent and -dependent noise conditions.

Our current research focuses on establishing a solid theoretical foundation with signal-independent noise, a critical precursor to addressing the more complicated noises in practice.  Extending our theoretical framework to include signal-dependent noise is a complex challenge. It demands a reevaluation of the existing analysis to account for the statistical dependencies introduced by signal-dependent factors. We recognize the importance and practical relevance of such an extension for real-world applications and consider it a significant direction for future research. However, due to the complexity and the need for a more advanced analytical framework, it remains beyond the scope of our current work.

Reviewer comment (Evaluation):
To measure the domain discrepancy, why not use the metric PSNR Gap proposed by the prior work DANet [3]?

Author response:
We appreciate this suggestion provided by you. To make fair comparisons with baseline methods, we have chosen to employ the same objective metrics that are widely recognized and utilized in I2I evaluations. While PSNR-Gap is a valid metric in certain contexts, it is not commonly employed to measure the difference between two probability distributions in the study of I2I tasks.