Paper: Competitive-Collaborative GAN with Performance Guarantee (ICLR-2024)

Reviewer comment (Novelty):
There are existing multiplayer methods to improve the stability of GAN training, which change the original GAN competition setting by introducing additional players. Would like to hear some words from the authors about the advantages or disadvantages compared to methods in this setting.  
- https://arxiv.org/abs/2101.07524  
- https://arxiv.org/abs/1907.02690  
- https://arxiv.org/abs/1709.03831

Author response:
The reviewer suggests providing a more comprehensive comparison with existing methods, specifically addressing how CCGAN performs compared to baseline models in the proposed scenarios. We agree with this suggestion and will expand the comparison section by providing a detailed discussion of CCGAN's performance in contrast to existing methods. This will include a deeper analysis of results, highlighting the advantages of CCGAN in scenarios such as image colorization, image super-resolution, and power data generation.

Reviewer comment (Presentation):
All the visual results are too small. I can’t really see the difference unless I zoom hard on a giant monitor, which is not possible from the printed version.

Author response:
We acknowledge the concern about the size of the visual results. In the revised version, we will ensure that the visual results in Figures 3 and 4, especially in columns 3, 4, and 4b, are presented in a larger size to enhance visibility. We understand the importance of clear and visible results for effective evaluation.

Reviewer comment (Evaluation):
In Figure 3 column 3 and 4, the results are not really better than the baseline.

Author response:
We understand that the reviewer finds Figure 3 columns 3 and 4 not to be significantly better than the baseline. In the revised version, we will reevaluate and potentially provide additional insights or analyses to better convey the improvements made by CCGAN. This may involve revisiting the presentation or providing more explicit comparisons.

Reviewer comment (Experiments):
It sounds like the proposed method should work with the original GAN method. Why not a direct comparison with it on an image generation task with FID / IS scores?

Author response:
Thank you for the valuable feedback. We acknowledge the suggestion to include a direct comparison with the original GAN, specifically employing metrics such as FID (Fréchet Inception Distance) and IS (Inception Score) for image generation tasks. In response to this, we plan to incorporate a dedicated section in the revised manuscript to showcase the performance improvements achieved by the proposed CCGAN method in comparison to the original GAN. Additionally, we appreciate the recommendation to explore combining the CCGAN method with state-of-the-art models like StyleGAN3. In the revised version, we will investigate the feasibility of integrating CCGAN with contemporary generative models to provide a more comprehensive evaluation beyond the original GAN. These enhancements aim to offer readers a clearer understanding of the proposed method's effectiveness and its compatibility with cutting-edge generative models. We value the feedback and will integrate these suggestions into the revised manuscript.

Reviewer comment (Reproducibility):
Training details are missing from the experiment section, such as resolution and sample size.

Author response:
The reviewer rightly suggests providing additional details on the experimental setup, including the rationale behind specific parameter choices, network architectures, and training procedures. We will address this by including a more comprehensive section on experimental details, explaining the reasoning behind the chosen configurations and providing insights into the experimental setup.

Reviewer comment (Writing):
The paper presents various claims together with some concepts which are not popular and without definition or explanation, some of which somewhat quite arbitrary. The authors argued that CCGAN overcomes the limitation of a pure-competitive GAN model and other existing collaborative GANs in terms of beyond the “authenticity” of data distribution, incorporating “additional knowledge”, or “harmonizing” the loss functions, reducing “degree of complexity” in finding the optima and deriving closed-form which “serves as a guidance for training and hyper-parameter tuning”.

Author response:
We appreciate the reviewers' attention to the clarity of our proposed method. The regularization term in CCGAN is strategically designed to address the challenges posed by pure-competition GANs, particularly in applications requiring data generation beyond authentic distribution. The regularization term is not arbitrary but is intended to promote collaboration between the generator and discriminator, allowing the model to learn partial modes and additional restrictions beyond authenticity. In the revised manuscript, we will provide a more detailed explanation of the regularization term's construction, elucidating its connection to the overarching goal of accommodating specific requisites in data generation tasks. Additionally, we will explicitly define terms such as ``authenticity'', ``additional knowledge'', and ``mutual dependency'' to enhance the clarity of key concepts, as suggested in R3-1. These improvements will contribute to a more precise and understandable presentation of our proposed method.

Reviewer comment (Theory):
The authors should clarify what form of “additional knowledge” does the proposed regularization add into? What is “authenticity” of the data distribution? What is “mutual dependency” and why is it helpful for GANs training to break this? In addition, could the authors conduct experiments to support claims of “harmonizing” the loss function, reducing the “degree of complexity” in finding the optima and how does the regularization “serves as a guidance for training and hyper-parameter tuning”?

Author response:
We appreciate the reviewer's request for clarification regarding the form of ``additional knowledge'' introduced by the proposed regularization in CCGAN. In the experimental design, our datasets are carefully selected to represent diverse data types, each with specific collaboration requirements. The regularization in CCGAN is designed to enable the model to collaborate effectively with the discriminator in capturing additional knowledge beyond the authentic data distribution. For instance, in the task of coloring grayscale shapes (dataset \textbf{Syn}), the collaboration requirement involves understanding and applying suitable colors to different geometric shapes. Similarly, in the image super-resolution task (dataset \textbf{SuperRes}), the generator must not only capture the data distribution but also faithfully reproduce high-resolution images from low-resolution counterparts. These examples illustrate the diverse ways in which CCGAN incorporates additional knowledge for specific collaboration requirements in various applications. In the revised manuscript, we will provide more explicit details and examples to ensure a clear understanding of how the proposed regularization facilitates collaboration with additional knowledge in different dataset contexts.

Reviewer comment (Theory):
From mathematical perspective, it is unclear why $JSD (p_{data} || \frac{\lambda - 1}{\lambda} * p_{gen})$ is superior than $JSD (p_{data} || p_{gen})$ of original GAN? As the new JSD appears not optimal, regardless of the selected value of $\lambda$, JSD never converges to optimal values.

Author response:
We appreciate the reviewer's attention to the comparison of the new Jensen–Shannon divergence (JSD) introduced in CCGAN with the original GAN's JSD. The concern raised about the superiority of the new JSD compared to the original JSD and the lack of optimality regardless of the selected value of $\lambda$ is valid, and we appreciate the opportunity to provide clarification. The new JSD in CCGAN is introduced as part of our effort to achieve a smooth integration between collaboration and competition loss components. It is employed to facilitate the transformation of the loss function into a divergence metric, aiming for a closed-form equilibrium. While we acknowledge that the optimality of JSD is contingent on the selected value of $\lambda$, and the introduced modification raises questions, we would like to highlight that our primary focus is on the practical implications and advantages brought by the collaboration-competition mechanism in CCGAN. In the revised manuscript, we will provide additional clarification regarding the role of the new JSD in the proposed framework. We will explicitly address the concerns raised by the reviewer, acknowledging the trade-offs and considerations associated with the choice of $\lambda$ and the new JSD. Furthermore, we will discuss the practical implications of this choice in the context of achieving stable and high-quality GAN training. Your feedback is invaluable, and we are committed to enhancing the clarity and thoroughness of our explanations in the revised manuscript.

Reviewer comment (Experiments):
The regularization appears close to the label smoothing “trick” which is widely used in GAN training. This suggests regularization again might suffice as a GAN loss. Have the authors explored the impact on results when removing the GAN loss function in this study? Can the author explain why the optimal D in Theorem 3 help CCGAN avoid the training collapse?

Author response:
We appreciate the insightful observation regarding the potential similarity between the proposed regularization in CCGAN and the label smoothing ``trick'' commonly used in GAN training. The reviewer rightly suggests exploring the impact on results when removing the GAN loss function in this study and seeks clarification on why the optimal discriminator in Theorem 3 helps CCGAN avoid training collapse. In response to these points, we acknowledge the need for a more in-depth analysis and experimentation. We will conduct an ablation study to investigate the impact of removing the GAN loss function, providing a clearer understanding of the contribution of the proposed regularization to the overall performance of CCGAN. Regarding the optimal discriminator in Theorem 3, we will provide a detailed explanation in the revised manuscript, elucidating how the specific form of the regularization term influences the discriminator's behavior. We will also discuss how this optimal discriminator contributes to avoiding training collapse, emphasizing the importance of breaking the mutual dependency between the generator and discriminator for improved stability.

Reviewer comment (Theory):
For me, the established regularization term is very ad hoc, without clarification on why it is constructed like that. As I read from the introduction, the authors seem to regularize GANs to learn partial modes or addition restrictions, the so-called learning beyond the authentic distributions. However, the established regularization term does not indicate those intensions.

Author response:
We appreciate the reviewers' attention to the clarity of our proposed method. The regularization term in CCGAN is strategically designed to address the challenges posed by pure-competition GANs, particularly in applications requiring data generation beyond authentic distribution. The regularization term is not arbitrary but is intended to promote collaboration between the generator and discriminator, allowing the model to learn partial modes and additional restrictions beyond authenticity. In the revised manuscript, we will provide a more detailed explanation of the regularization term's construction, elucidating its connection to the overarching goal of accommodating specific requisites in data generation tasks. This clarification aims to address concerns about the ad hoc nature of the regularization term raised in R1-1.

Reviewer comment (Theory):
This paper claims that existing regularized GANs do not have closed-form equilibrium, which is proved based on the very simple Gaussian case. This is not very convincing. Even though, in my opinion, for training GANs, the way to approach the equilibrium is more important than the closed-form solution obtained by ad hoc regularization terms. It is well-known that for the vanilla GAN with closed-form equilibrium, we are not always ensured to get this equilibrium.

Author response:
We appreciate the reviewer's insightful observation and comments regarding the theoretical proof in Theorem 1 and the claim about the closed-form equilibrium point in original GANs. The concern raised about the closed-form equilibrium point and its relevance in practical GAN training is duly acknowledged. The closed-form equilibrium point in the original GAN, as discussed in Theorem 1, is presented in the context of the theoretical analysis to establish the motivation for our proposed CCGAN. It serves as a reference point to highlight the challenges associated with the introduction of collaboration-competition mechanisms in existing GAN models. However, we recognize the importance of emphasizing the practical implications of the closed-form equilibrium point in the context of GAN training. In practice, the closed-form solution might not always directly translate into improved training stability or enhanced model performance. We will revise the manuscript to provide a more nuanced and practical discussion, clarifying that the theoretical analysis sets the stage for proposing CCGAN rather than implying a direct practical advantage. Moreover, we will highlight the unique contributions and advantages of CCGAN, such as addressing specific collaboration requirements and avoiding training collapse, which are crucial aspects of the proposed framework. Your feedback helps us refine the clarity and emphasis of our theoretical contributions in the revised manuscript.