Paper: Idempotent Generative Network (ICLR-2024)

Reviewer comment (Novelty):
**Motivation:** While the idea is interesting and utilizes foundational structures from mathematics, the motivation for idempotence in generative models is still a little weak. Several different consequences are described in the introduction (IGN provides outputs in a single step, yet allows additional refinements, etc.) Each of these properties is attractive, but already possessed by an existing mature method (as is pointed out in the paper). It was unclear to this reviewer whether starting from scratch with a completely new paradigm is the right way to obtain the desirable properties of GANs, diffusion models, etc. On the other hand, idempotence is a deep idea whose consequences have been explored in a wide range of settings and directions. It feels likely to this reviewer that there are good reasons for wanting idempotence in a model that go beyond properties we already have in alternative methods. However, these may yet need to be uncovered.

Author response:
**Motivation: “Several different consequences are described in the introduction (IGN provides outputs in a single step, yet allows additional refinements, etc.) Each of these properties is attractive, but already possessed by an existing mature method (as is pointed out in the paper). “**  
IGN has some unique properties that do not exist in other models. In addition, IGN has some properties that exist in other models but do not co-exist together. For instance, let us consider the single step + refinement property the reviewer mentions. There are Diffusion based methods with fewer steps, such as consistency models. However IGN always does one step full generation after which it is possible to apply it again for refinement. 
Another example is a consistent latent space. Surely, GANs have that, but IGN allows smooth interpolation across all iterations ($f(z), f(f(z)), f(f(f(z)))$...). (See video in supplementary).
Also, Unlike GANs and Diffusion models, editing of real images in IGN does not require any type of inversion. Simply inputting a real image in a feed-forward fashion into IGN allows manipulation on it (see new figures 9,10 in revised manuscript).
Lastly there is the projection property addressed below in response to your specific question.

Reviewer comment (Experiments):
**Comparison to other methods:** While the reviewer agrees that it is not fair to hold this novel approach to the standards of more mature methods, it would still be useful to be able to compare the performance of IGN with known Approach. As it is, this reviewer was unable to tell how far IGN is from the performance of a comparably sized GAN, VAE, or diffusion model.

Author response:
**Quantitative evaluation and comparison to other methods:**  
We report FID=39 for IGN on CelebA. Naturally, this is not competitive with modern models, but is comparable to early models such as DCGAN (FID=34), while also capable of a wide range of conditional tasks. We updated the draft paper to include this info.

Reviewer comment (Evaluation):
**Quantitative results:** It appears that the only way the reader can evaluate the experiments is through a handful of example generations. While these are helpful for getting a qualitative sense of the model (and look good, as described in the Strengths section), it would have made the analysis stronger if quantitative results were also presented. It would be interesting, for instance, to understand network performance for different types of initial $x$, especially since the domain of the network includes both $\mathcal{P}_z$ and $\mathcal{P}_x$. It would have also been interesting to quantitatively measure differences in $f^{(k)}(x)$ as $k \rightarrow \infty$.

Author response:
**Quantitative evaluation and comparison to other methods:**  
We report FID=39 for IGN on CelebA. Naturally, this is not competitive with modern models, but is comparable to early models such as DCGAN (FID=34), while also capable of a wide range of conditional tasks. We updated the draft paper to include this info.

Reviewer comment (Writing):
Idempotence is spelled wrong in the sentence below equation (7).

Author response:
**Idempotence spelled wrong:** Fixed, thanks!

Reviewer comment (Writing):
The tightness objective is a little subtle. It might make sense to utilize some of the notation already introduced (e.g., $\mathcal{S}$) to help communicate why this loss term is necessary.

Author response:
**Tightness loss clarity:**  
Indeed it is subtle, and challenging to convey. We added the notion of $\mathcal{S}$ to the text around fig.2 and its caption as you suggested. We updated the text about $\mathcal{L}\_{tight}$ and also added Fig.17 that offers further clarification on the two paths to enforce idempotence. We welcome any suggestions to enhance the explanation in the paper.
We have provided some informal perspectives to help understand $\mathcal{L}\_{tight}$ in responses to other reviewers. please check them (not pasting here due to char limit, sorry.).

Reviewer comment (Experiments):
It may be a matter of personal taste, but this reviewer felt that while the PyTorch code was appreciated, it might be better to include in the Appendix and instead include additional analysis of the experiments.

Author response:
**Code better in appendix:**  
The provided PyTorch code is a substitute for an algorithm pseudocode. We felt that it is the cleanest way of presenting the algorithm. That said, we can consider moving it to the appendix and replace it with the regular pseudocode notation. We will also release full code.

Reviewer comment (Novelty):
One of the claimed strengths of IGN is that through the idempotence property, it can be used to project noisy or corrupted images back to the data manifold. Based on the reviewer’s understanding of the capability, there are already a range of methods for doing this (e.g., via image denoising with a diffusion model). Is there some additional nuance that the reviewer is missing? It is certainly the case that existing methods will not map an image plus noise $x + \epsilon$ by to $x$ exactly, but it was our sense from reading the paper that neither would IGN.

Author response:
**“IGN can be used to project corrupted images … already a range of methods for doing this (e.g., via image denoising with a diffusion model).”**  
While many methods handle certain degradations or modifications of images, they will typically be trained for it specifically (e.g., Diffusion-Denoising, Cold Diffusion, image enhancement methods, image-to-image methods), or they will require some manipulation at test time (adding noise, guidance in diffusion and some tricks in GANs). IGN is not trained for specifically on modified images. IGN only sees pure noise and real images during training. No different type of inference is needed. For example, no noise is added to bring the input into distribution, as would be in, for instance, SDEdit. Note that it is not only for corrupted images but also image-to-image translation such as sketches. It can be thought of as a generic  ``make-it-real" button.

Reviewer comment (Theory):
It would be interesting to see the behavior of $f^{k}(x)$ as $k \rightarrow \infty$, does this converge or diverge? If it does converge, what does it converge to?

Author response:
**“ $\mathbf{f^k(z) \quad k\rightarrow\infty}$. Does this converge or diverge?“:**   
We added fig.16 in the appendix of the revised manuscript which demonstrates applications of MNIST trained IGN up to $300000$ iterations. For the first few tens of iterations, the system seems to be stable but around 100 it diverges from the result of the first application. We hypothesize that the size of the model which influences the ability to get low reconstruction loss determines the stability.

Reviewer comment (Writing):
This reviewer did not understand the modified GAN network architecture described in Section 2.3.

Author response:
**“This reviewer did not understand the modified GAN network architecture described in Section 2.3.”**  
In the revised manuscript, we added Fig.15 to illustrate the modification of DCGAN architecture to be used for IGN on CelebA. The exact architecture is in Table.1 in the appendix. 
DCGAN has a generator $G$ and a discriminator $D$. In training $D(G(z))$ is applied. $D$ produces a $1\times1\times1$ result per image. we modify $D$ to end with a $512\times1\times1$ tensor instead, denoted as $\tilde{D}$. Our model in terms of DCGAN is $G(\tilde{D}(z))$.

Reviewer comment (Novelty):
As the authors note, the development of high-performing generative models has come about through numerous iterative improvements. Would it be possible to build an IGN model using the framework of a known generative approach?

Author response:
**“Would it be possible to build an IGN model using the framework of a known generative approach?”, “What are the current drawbacks to scaling this method up?
“**  
Scaling up is a current effort. Indeed, the ways explored are based also on existing methods. Among others, working in latent space as in Stable Diffusion and using transformer architectures are being tried. One challenge we have seen is the need of a bottleneck which makes the popular Unet challenging to work with due to its skip connections.

Reviewer comment (Experiments):
What are the current drawbacks to scaling this method up?

Author response:
**“Would it be possible to build an IGN model using the framework of a known generative approach?”, “What are the current drawbacks to scaling this method up?
“**  
Scaling up is a current effort. Indeed, the ways explored are based also on existing methods. Among others, working in latent space as in Stable Diffusion and using transformer architectures are being tried. One challenge we have seen is the need of a bottleneck which makes the popular Unet challenging to work with due to its skip connections.

Reviewer comment (Novelty):
The biggest issue is that I don't get the motivation for designing such a model. When one designs a new class of generative model that is different from any previous approaches, it is necessary to clarify that it has the potential to address some limitations of previous approach. As a preliminary study, it does not need to have great results, but conceptually the advantage of the model has to be stated. Otherwise, it is just "another class of generative models". Since it suffers the same mode collapse and instability issue of GAN, the motivation becomes more important. Why do we need such a model? What are the potential advantages?

Author response:
**Motivation: “When one designs a new class of generative model that is different from any previous approaches, it is necessary to clarify that it has the potential to address some limitations of previous approach.”**  
IGN has several potential advantages over the current methods:  
1. Unlike DIffusion models, IGN generates in a single step, and then, unlike GANs,  allows for sequential refinement. There are Diffusion based methods with fewer steps, but for IGN the number of reprojections can be determined adaptively (apply again if desired). A single step will always yield good results. 
2. Unlike Diffusion models, IGN maintains a consistent latent space, allowing smooth interpolation across all iterations ($f(z), f(f(z)), f(f(f(z)))$...). (Please see videos in supplementary).
3. IGN can be applied to types of data that it has never ‘seen’ and project them onto the data manifold. While Diffusion models can be guided to get condition signals, IGN does that when regularly applied to degraded/modified input.
4. Unlike GANs and Diffusion models, editing of real images does not require any type of inversion. Simply inputting a real image in a feed-forward fashion into IGN allows manipulation on it (see new figures 9,10 in revised manuscript).

Reviewer comment (Theory):
I don't quite get the point of the tightness loss. Why maximize it is good for regularization? More explanation is needed.

Author response:
**”I don't quite get the point of the tightness loss. Why maximize it is good for regularization? More explanation is needed.”**  
We appreciate the feedback and acknowledge the challenge in conveying the concept of $\mathcal{L}\_{tight}$. In the new draft, we added the notion of $\mathcal{S}$ to the caption of fig.2. We updated the text about $\mathcal{L}\_{tight}$ and also added Fig.17 to the appendix of the revised manuscript. Fig.17 offers further clarification on the two paths to enforce idempotence. We offer additional informal perspectives here for better clarity and welcome any suggestions to enhance the explanation in the paper.

Consider the metaphor of High Jump. The objective is to clear the bar. This can be achieved in two ways: jumping higher, or lowering the bar. If the athlete, $f$ trained directly for this objective, it would become an expert on lowering the bar (tends to become identity). Instead, the athlete tries to get **bigger** gap from the bar when jumping, but trying to get **smaller** gap from the bar (opposite from the objective), when setting it according to its recent jumps. We emphasize that this is the same athlete that both jumps and sets the bar.

Examining the objective $\delta\_\theta = D(f\_\theta(f\_\theta(z)),f\_\theta(z))$, we can split it into two phases:  
(1) $y=f\_\theta(z)$ (jumping)  
(2) $\delta\_\theta = D(f\_\theta(y),y)$ (evaluating the gap from the bar)  
Both are done using the same set of parameters $\theta$. See Fig.17 in the appendix. The instance of $f$ that is in phase (1) sends gradients to $\theta$ that improve the generation (higher jump, lower drift $\delta\_\theta(y) = D(f\_\theta(y), y)$). The Gradients to $\theta$ through the instance of $f$ that is in phase (2) judge generated images and train to exclude them from the manifold (higher drift), just like setting the bar higher according to the recent jumps. 
This aspect can also be thought of as a GAN where the generator and discriminator are the same network.

Reviewer comment (Theory):
I am not fully convinced of the theoretical validity of the method. In particular, Equation (19) does not appear fully rigorous. From my understanding, the value \( M \) is the maximum distance between points in the state space. What if the state space is unbounded? Should we expect states to be restricted to an image hypercube?

Author response:
**The value $M$:**  
$M$ is the supremum of the distance metric for any pair $(y\_1, y\_2)$. There is no bounded space assumption -- M can also be infinity (which is why we did not define it as Maximum. We added a clarification in the paper). We examine both the scenario of unbounded $M$ (which is the case of MSE as used for MNIST and shown in the provided code), and the scenario of bounded $M$ which exists when using the clamp technique as introduced in eq.14. 
Unbounded $M$ is consistent with all the steps of the proof, under the practical assumption that $M \cdot 0 < M$.

Reviewer comment (Theory):
Why is it the case that \( \max_{\theta*} \delta_{\theta*}(y) = M \) for any \( y \) when \( P_x (y) < \lambda_t P_{\theta*} (y) \)? Wouldn't that mean the distance between \( y \) and the furthest point from \( y \) in the state space is \( M \) for each \( y \)? I am not sure that makes intuitive sense. It seems that \( M \) should not be a constant but rather a function \( M(y) \) measuring the distance between \( y \) and the furthest point in the state space, which greatly complicates the analysis that follows.

Author response:
**The role of $M$ in eq.21 (formerly eq.19):**  
Thank you for pointing out the lack of clarity. We have revised the text before the equation (that is eq.21 now).  
Eq.21 describes what happens when $\mathcal{L}\_{rt}$ is minimized, according to how it is expressed in eq.20 (eq.18 second row in pre-revised version):  
$$ \mathcal{L}\_{rt}(\theta;\theta^*)=\int \delta\_{\theta}(y) \Bigl(  
\mathcal{P}\_x(y)  
\- \lambda\_t   
\mathcal{P}\_{\theta^*}(y)  
\Bigr)dy $$  
$\delta\_\theta$ is multiplied by a given expression in the brackets that is not being optimized in this phase.  
To obtain a minimum, $\delta\_\theta\geq0$ needs to be as big as possible when the expression in the brackets is negative and as small as possible when positive. Thus, for any $M>0$ eq.21 holds when  $\mathcal{L}\_{rt}$ obtains its minimum:  
$$ \delta\_{\theta^*}(y) = M \cdot 1\_{\{\mathcal{P}\_{x}(y) < \lambda\_t  \mathcal{P}\_{\theta^*}(y) \}}$$  

**Should it be $M(y)$?:**  
Thank you for identifying this subtle issue. Indeed, for a general metric in a general space, $M$ should be defined using a reference input $M(y\_1) = \sup\_{y\_2} D(y\_1, y\_2)$. An example could be MSE in some bounded space. We should point out that in the metrics used in the paper (MSE, L1, and clamped L1), $M$ would be constant ($\infty$ for unbounded metrics and the clamp value for the clamped metric). Regardless, using $M(y)$ instead of $M$ would not have influence on the subsequent steps of the proof as long as $M(y)>0\quad\forall y$.  
To see this we can examine minimizing $\mathcal{L}\_{idem}$ as in eq.24 modified with $M(y)$:  
$$
    \theta^* = argmin\_\theta  \mathbb{E}\_{z}\bigl[ M(y) \cdot  1\_{\{
    \mathcal{P}\_{x}(y) < \lambda\_t  
    \mathcal{P}\_{\theta}(y) 
    \}} \bigr] 
$$  
$M$ being positive implies that for every value of $y$, $M(y)\cdot 0 < M(y) \cdot 1$. This leads to the same $\theta^*$ for constant $M$ or $M(y)$.  
We are happy to revise and redefine $M(y)$. Another option is to add in the text assumptions on the metrics used.

Reviewer comment (Theory):
The proof heavily relies on the simple form of (19), but I am not sure this simple form is valid. Furthermore, in practice the distance \( M \) is heavily restricted which leaves a gap between the theoretical development of the method and practical implementation. Since this method departs heavily from the established probabilistic principles used to justify generative models, clearly demonstrating the validity of the method is essential. I am willing to raise my score if these points can be addressed.

Author response:
**$M$ imposes gap between theory and practice?:**   
Hopefully, the above explanation was able to show that $M$ does not impose restrictions and that, while the theoretical results make very strong idealistic assumptions, the practical implementation does not diverge more than in any other generative model's theoretical results.  But please let us know if something is still unclear and we will do our best to respond quickly.

Reviewer comment (Evaluation):
Although I understand that the current results are meant to be a proof of concept, it would still be good to provide a more formal comparison with prior generative models e.g. by calculating FID scores on CelebA.

Author response:
**FID on CelebA:**  
We report FID=39 for IGN on CelebA with the architecture in Table.1. Naturally, this is not competitive with modern models, but is comparable to early models such as DCGAN (FID=34), while also capable of a wide range of conditional tasks. We updated the draft paper to include this info.