Paper: PRISM: Privacy-Preserving Improved Stochastic Masking For Federated Generative Models (ICLR-2024)

Reviewer comment (Evaluation):
There is no DP analysis for the method and no $\varepsilon$'s or $\delta$'s are reported in the experimental results.

Author response:
Thank you for your prompt feedback. We initially set $\epsilon=50$ and $\delta=0.00001$ to align with the baseline parameters used in their original configurations, ensuring a fair comparison. However, in response to your concern about the high epsilon-value, we conducted an additional experiment with a lower privacy cost of $\epsilon=9.8$, commonly adopted in Differential Privacy Stochastic Gradient Descent (DP-SGD). As detailed in Tables 6 and 7, PRISM consistently demonstrated robust performance, even under stricter privacy constraints. In contrast, both DP-FedAvgGAN and GS-WGAN showed performance collapse. These models necessitate higher $\epsilon$ values, as evidenced in Table 2 of the GS-WGAN study [1].

[1] Chen, Dingfan, Tribhuvanesh Orekondy, and Mario Fritz. "Gs-wgan: A gradient-sanitized approach for learning differentially private generators." Advances in Neural Information Processing Systems 33 (2020): 12673-12684.

Reviewer comment (Theory):
If I understand the method correctly, the masks would indeed depend on the data, so the $\varepsilon$ is definitely not $0$.

Author response:
Thanks for providing valuable feedbacks and various concerns that arise in our work.

As can be seen from Appendix A of our manuscript, after clipping the noise-injected $\theta$, the privacy budget we have becomes $\epsilon_{amp}\leq min\{\epsilon, d\gamma_{\alpha}(c)\}$. Here, during the noise injection process, the variance $\sigma^2$ of the Gaussian noise should become a function of $\epsilon$, which controls the level of privacy we guarantee. Overall, the perturbing $\theta$ is necessary to guarantee a certain level of privacy.
Regarding the privacy amplification, our intention was to theoretically show that the privacy bound can become tighter (or at least does not compromise the bound) by taking advantage of Bernoulli sampling. Depending on various parameters and the model size, it may or may not be amplified, but the key point here is that since we have applied $(\epsilon, \delta)$-dp to $\theta$, there is no case that the privacy bound falls below $\epsilon$.

Reviewer comment (Theory):
How would you prove the differential privacy guarantees for the proposed method?

Author response:
Thanks for providing valuable feedbacks and various concerns that arise in our work.

As can be seen from Appendix A of our manuscript, after clipping the noise-injected $\theta$, the privacy budget we have becomes $\epsilon_{amp}\leq min\{\epsilon, d\gamma_{\alpha}(c)\}$. Here, during the noise injection process, the variance $\sigma^2$ of the Gaussian noise should become a function of $\epsilon$, which controls the level of privacy we guarantee. Overall, the perturbing $\theta$ is necessary to guarantee a certain level of privacy.
Regarding the privacy amplification, our intention was to theoretically show that the privacy bound can become tighter (or at least does not compromise the bound) by taking advantage of Bernoulli sampling. Depending on various parameters and the model size, it may or may not be amplified, but the key point here is that since we have applied $(\epsilon, \delta)$-dp to $\theta$, there is no case that the privacy bound falls below $\epsilon$.

Reviewer comment (Theory):
What would be the DP guarantees for the experimental results that you provide?

Author response:
Thank you for your prompt feedback. We initially set $\epsilon=50$ and $\delta=0.00001$ to align with the baseline parameters used in their original configurations, ensuring a fair comparison. However, in response to your concern about the high epsilon-value, we conducted an additional experiment with a lower privacy cost of $\epsilon=9.8$, commonly adopted in Differential Privacy Stochastic Gradient Descent (DP-SGD). As detailed in Tables 6 and 7, PRISM consistently demonstrated robust performance, even under stricter privacy constraints. In contrast, both DP-FedAvgGAN and GS-WGAN showed performance collapse. These models necessitate higher $\epsilon$ values, as evidenced in Table 2 of the GS-WGAN study [1].

[1] Chen, Dingfan, Tribhuvanesh Orekondy, and Mario Fritz. "Gs-wgan: A gradient-sanitized approach for learning differentially private generators." Advances in Neural Information Processing Systems 33 (2020): 12673-12684.

Reviewer comment (Novelty):
It appears that GANs are growing slightly out of fashion in the generative AI community, which slightly affects the significance of this work.

Author response:
**R4-4. GAN is out of fashion**

 The tackiness of GAN is discussed in GC1, GC2. Additionally, the reviewer’s question seems to be intended for DDPM, which has recently shown strong performance. Although we have been empirically confirmed that SLT works successfully, there are still several limitations such as slow convergence. 
We would like to clarify that PRISM uses MMD loss, while the baselines employ GANs. As you mentioned, it’s true that the popularity of GANs has declined with the emergence of diffusion models, but GANs still offer valid advantages such as simplicity and fast inference times [1]. While it’s possible to apply diffusion models to FL setups, the introduction is not straightforward in FL, where communication efficiency becomes a bottleneck. This is because diffusion models require substantial computational resources and slow convergence due to the iterative learning. Importantly, there should be discussions considering various ways in the field of FL for generative models.

[1] Kang, Minguk, et al. "Scaling up gans for text-to-image synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Experiments):
Did the authors experiment with other losses, to further showcase the strength of MDD? It would provide more insight to the results of Tables 1 and 2 if the authors applied the same losses from other methods with their subnetwork strategy. This way one can better understand where the performance improvements are coming from.

Author response:
**R4-1. Experiment with other losses.**

Please see GC1 and R1-1.
We want to clarify the distinction between our approach and baselines. Baselines typically  train the weights using GAN loss, while PRISM employs the MMD loss to find the SLT within the dense network. If an SLT is introduced into a baselines method, it essentially becomes PRISM. In that case, there is a potential question to consider whether strong performance emerges when GAN loss is applied to PRISM. To handle this curiosity, we conducted an experiment which utilizes GAN loss as score function for SLT, not MMD loss. In order to learn the generator by GAN loss instead of MMD loss, each client learns the discriminator and generator adversarially. After the local training is completed, the server aggregates and returns only the generator. As seen in Figure.10, the unstable GAN loss fails to find the SLT and proceed the FL normally, which implies that MMD loss is suitable for decentralized setting and SLT.

Reviewer comment (Experiments):
Is this approach outperforming compared to traditional FL? On one hand, can PRISM attain equivalent efficiency to traditional FL, which directly uploads local parameters? On the other hand, when we consider the attack of inferring the client's local dataset, can PRISM provide robust protection against such attacks? We look forward to the authors conducting relevant experiments to address the questions.

Author response:
Since PRISM uploads binary masks instead of float-type weights, PRISM can save communication costs up to 32 times compared to traditional FL. The comparison of efficiency between PRISM and traditional FL can be found in Table.1,2, and 3.  
On the aspect of security, particularly against attacks aiming to infer clients' local datasets, PRISM offers robust protection. This security is primarily attributed to three factors. First, even if an attacker hijacks the mask used in PRISM, without knowledge of the client's architecture, the information gleaned is effectively meaningless. Second, even if the attacker is aware of the architecture, the lack of precise information about the random initialization renders any attempt to decode the client's data futile. Third, PRISM incorporates differential privacy (DP) techniques. It is well known that DP can prevent various attacks in decentralized environments. For more details on DP, please see GC1 and R3-3.

Reviewer comment (Theory):
How can DP guarantee privacy in generative models? Any proofs?

Author response:
Differential privacy is an algorithm designed to reduce the influence of single elements on output. Let’s suppose M is a mechanism that we want to guarantee privacy, such as neural network. With adjacent datasets D, D’, M is $(\epsilon, \delta)$-differential privacy when $P(M(D)) \leq e^{\epsilon}P(M(D’)) + \delta$ is satisfied. Also, by composability, if all components of M satisfy DP, M itself also satisfies DP. Since generative model is a stacked layer, DP guarantees privacy in generative models.

Reviewer comment (Experiments):
The comparison experiments in the study may not reflect the state-of-the-art, and we anticipate that PRISM will be compared with more advanced methods. For example, whether PRISM's model performance surpasses that of Multi-FLGAN.

Author response:
Multi-FLGAN introduces multi-GAN, competing X generators and Y discriminators to select the best-performing generator. Consequently, there exist X*Y sync servers. While Multi-FLGAN achieves excellent performance and stability, it demands a substantial amount of computation resources. According to the paper, authors used four V100 GPUs for experiments on MNIST and FMNIST. However, we agree that comparing PRISM with SOTA models can enhance the contribution of PRISM. We will apply differential privacy to Multi-FLGAN and update the results when experiments are completed.  

**(updated)**  

As we discussed above, Multi-FLGAN employs a complex architecture involving multiple generators and discriminators ($X\times Y$ of each per client), with the server aggregating $N\times X\times Y$ models, where $N$ is the number of clients. Our comparison between PRISM and Multi-FLGAN[1] involved a scaled-down Multi-FLGAN setup ($X=2, Y=2$) due to its high resource demands.  
The table below details our findings, comparing the standard Multi-FLGAN (default setup introduced in the original paper) and an modified version, Multi-FLGAN+DP version, which incorporates differential privacy ($(\epsilon, \delta)=(9.8, 10^{-5})$) for a fair comparison to PRISM and other baseline models. We observed significant performance degradation in Multi-FLGAN when applying DP, likely due to its complex framework involving both generators and discriminators.  
This raises to two key concerns:  
DP suitability: The complex framework of Multi-FLGAN limits its effectiveness when DP is applied, suggesting that the framework may not be ideally suited for privacy-preserving FL scenarios.  
Resource and Communication Overheads: The architecture demands extensive training resources and incurs high communication overhead, posing challenges for practical deployment, especially in resource-constrained environments.  

|        MNIST              |   Privacy     |      FID      |      P&R                  |      D&C                 |     Cost     |  
|:---------------------------:|:----------------:|:---------------:|:--------------------------:|:-------------------------:|:---------------:|  
|  Multi-FLGAN          |        X        |   44.97      |   0.1461 / 0.3248   |   0.0461 / 0.0868   |    52MB    |          
|  Multi-FLGAN + DP |        O        |  323.721   |         0.0 / 0.0         |         0.0 / 0.0         |    52MB    |  
|  PRISM                    |        O        |  59.4503   |   0.2594 / 0.3884   |   0.0832 / 0.0852   |   5.75MB  |  

[1] Amalan, Akash, et al. "MULTI-FLGANs: Multi-Distributed Adversarial Networks for Non-IID distribution." arXiv preprint arXiv:2206.12178 (2022).

Reviewer comment (Novelty):
The paper borrows almost all the main ideas from [1]. The only noticeable difference seems to be applying FedPM in [1] to generative models other than classifiers. So, I would expect the authors to highlight this properly and give credit to [1]. Right now, [1] is listed in the related work section very briefly as if there is not much similarity between the two works while the framework is exactly the same.

Author response:
**R2-1. Highlight and give credit to FedPM**

As the reviewer suggested, we will more clearly outline the relationship between our research and prior works, including FedPM.

Reviewer comment (Theory):
I am not sure how the privacy claims follow. The paper cites [1] and [2] for the statement that PRISM (or FedPM) should amplify privacy -- which is correct. But PRISM (or FedPM) alone is not sufficient to have any differential privacy guarantee. They can only amplify privacy when there is an explicit differential privacy mechanism somewhere in the framework. In the experiments section, specifically in Table 1 and Figure 3, both PRISM and PRISM-$\alpha$ are put in the "with privacy" category. Can the authors explain what their privacy mechanism actually is? And how much does the Bernoulli sampling process amplify this privacy?

Author response:
Thank you for your insights and constructive opinions. Your efforts have been a great help in developing our work! 

Concerning privacy amplification, we aim to theoretically demonstrate that the privacy bound can be strengthened (or, at the least, remains uncompromised) through the utilization of bernoulli sampling. The degree of amplification may vary depending on different parameters and the model size. However, the crucial aspect is that, given the application of $(\epsilon, \delta)$-dp to $\theta$, there is no scenario where the privacy bound descends below $\epsilon$.

Reviewer comment (Novelty):
What is the reason the authors renamed FedPM as PRISM given that they are the same?

Author response:
**R2-2. Limited novelty. Why rename FedPM as PRISM?**

Regarding the perceived limited novelty and the renaming of FedPM to PRISM, please see GC2 for a detailed explanation. 

As outlined in GC2, our domain faces a significant gap in research that attempts to consider both privacy and communication efficiency simultaneously. This gap, particularly in the context of various losses and frameworks for GANs, results in existing methods underperforming. Our approach addresses this by integrating the MMD loss, which aids in the stable identification of SLT. We also incorporated elements from the decoder structure to improve performance without significantly increasing communication costs. Given these enhancements, we decided to name our method PRISM to reflect its distinctiveness and advancements. Nonetheless, in our revised submission, we will make sure to clarify the evolution from FedPM to PRISM, enabling readers to clearly understand the continuity and progression of ideas.

Reviewer comment (Theory):
Where does the differential privacy guarantee come from given that Bernoulli sampling only amplifies privacy but does not introduce any privacy guarantee alone?

Author response:
Thank you for your insights and constructive opinions. Your efforts have been a great help in developing our work! 

Concerning privacy amplification, we aim to theoretically demonstrate that the privacy bound can be strengthened (or, at the least, remains uncompromised) through the utilization of bernoulli sampling. The degree of amplification may vary depending on different parameters and the model size. However, the crucial aspect is that, given the application of $(\epsilon, \delta)$-dp to $\theta$, there is no scenario where the privacy bound descends below $\epsilon$.

Reviewer comment (Reproducibility):
It is unclear what kind of architecture is used in the paper.

Author response:
**R1-3. Which architecture is used? Does the choice matter?**

We have consistently utilized the ResNet-based generator for both our method and all baseline methods to ensure a fair comparison. Our detailed analysis, shown in Figure 4, compares performance while controlling for the number of parameters. The results demonstrate that PRISM outperforms the other baseline methods, even when matched for capacity and network structure. In addition, our method works generally well for different architectures. Basically, SLT just presumes that a dense network is sufficiently overparameterized to find a useful subnetwork. This implies that SLT has robustness with architecture. To validate our argument, we will evaluate architectures frequently used in generative models. When ongoing experiments are completed, we will update you.

**(updated)**

We applied various generator architectures that have been studied in the field of generative models to the architecture of PRISM and compared their performance. Review can find the results in Table.8. We kept all settings, including DP, consistent with our framework. Our default configuration is a ResNet-based generator, and architectures such as DCGAN and SNGAN show robust performance on MNIST but exhibit a slight advantage depending on the architecture in CelebA. PRISM can somewhat follow the performance of the architecture since it extracts a subnetwork from the dense network. Still, it consistently finds a stable SLT for various structures.

| MNIST     |   ResNet   |   DCGAN   |   SNGAN    |
|:--------------:|:---------------:|:----------------:|:----------------:|
|  FID         |   34.4595  |    36.9683  |   33.4447   |
| Precision |   0.3784    |    0.3694    |   0.4652     |
| Recall      |   0.4213    |    0.4848    |   0.3318     |
| Density    |   0.1418    |    0.1323    |   0.2047     |
| Coverage |   0.1588    |    0.1490   |   0.2019     |

|  CelebA   |   ResNet   |   DCGAN   |   SNGAN    |
|:--------------:|:---------------:|:----------------:|:----------------:|
|  FID         |   51.0857  |    78.4130  |  38.3009  |
| Precision |   0.5267    |    0.4540    |   0.6851     |
| Recall      |   0.0631    |    0.0404    |   0.0925     |
| Density    |   0.2676    |    0.1916    |   0.5378     |
| Coverage |   0.2012    |   0.0947    |   0.3155     |

Reviewer comment (Experiments):
It is unclear whether the choice of architecture affects the performance.

Author response:
**R1-3. Which architecture is used? Does the choice matter?**

We have consistently utilized the ResNet-based generator for both our method and all baseline methods to ensure a fair comparison. Our detailed analysis, shown in Figure 4, compares performance while controlling for the number of parameters. The results demonstrate that PRISM outperforms the other baseline methods, even when matched for capacity and network structure. In addition, our method works generally well for different architectures. Basically, SLT just presumes that a dense network is sufficiently overparameterized to find a useful subnetwork. This implies that SLT has robustness with architecture. To validate our argument, we will evaluate architectures frequently used in generative models. When ongoing experiments are completed, we will update you.

**(updated)**

We applied various generator architectures that have been studied in the field of generative models to the architecture of PRISM and compared their performance. Review can find the results in Table.8. We kept all settings, including DP, consistent with our framework. Our default configuration is a ResNet-based generator, and architectures such as DCGAN and SNGAN show robust performance on MNIST but exhibit a slight advantage depending on the architecture in CelebA. PRISM can somewhat follow the performance of the architecture since it extracts a subnetwork from the dense network. Still, it consistently finds a stable SLT for various structures.

| MNIST     |   ResNet   |   DCGAN   |   SNGAN    |
|:--------------:|:---------------:|:----------------:|:----------------:|
|  FID         |   34.4595  |    36.9683  |   33.4447   |
| Precision |   0.3784    |    0.3694    |   0.4652     |
| Recall      |   0.4213    |    0.4848    |   0.3318     |
| Density    |   0.1418    |    0.1323    |   0.2047     |
| Coverage |   0.1588    |    0.1490   |   0.2019     |

|  CelebA   |   ResNet   |   DCGAN   |   SNGAN    |
|:--------------:|:---------------:|:----------------:|:----------------:|
|  FID         |   51.0857  |    78.4130  |  38.3009  |
| Precision |   0.5267    |    0.4540    |   0.6851     |
| Recall      |   0.0631    |    0.0404    |   0.0925     |
| Density    |   0.2676    |    0.1916    |   0.5378     |
| Coverage |   0.2012    |   0.0947    |   0.3155     |

Reviewer comment (Theory):
It is unclear how the random mask learning benefits from multiple-round federated learning. It seems to me that it is closer to the one-shot learning. The authors should expand this part and provide more insights.

Author response:
**R1-4. How does random mask learning benefit from multiple-round federated learning?**

We interpret your question as probing why we opt for multiple-round learning rather than identifying a Strong Lottery Ticket (SLT) at each client side and then aggregating the final mask in a one-shot manner at the server.
 To answer this, we draw an analogy between each local dataset in our method and a mini-batch in traditional centralized SGD. In this comparison, one-shot aggregation is akin to optimizing with a very large step size. It is well-known in optimization theory that training with a large step size and few epochs can often lead to local minima. Coming back to the perspective of federated learning, by aggregating the scores (or binary masks) of each client through multiple rounds, our goal is to find the global lottery ticket that is not biased towards specific local dataset. To support our opinion, we conducted an experiment in which we performed single aggregation after overfitting MNIST dataset to 500 local iterations. The results can be found in Appendix.C. We observe that it fails to generate proper images.

Reviewer comment (Experiments):
The experiments about differential privacy (DP) are either missing or incomplete. The two most critical parameters in DP, $\varepsilon$ and $\delta$, are missing in the paper. It is also unclear how to apply DP to the proposed method when generating masks.

Author response:
There has been notable concern regarding the experiment results conducted with epsilon=50, as highlighted by Reviewers sDDa and xZ6u. In response to this, we carried out additional experiments using $\epsilon=9.8$, which is frequently used in DP-SGD. The results of these experiments can be found in Appendix.A and Table.6 and 7. Notably, we observed that PRISM robustly performs well, whereas other baselines failed. We would like to notify that all experiments were conducted with identical $\epsilon$ values  for both baselines and PRISM. To avoid any potential confusion, we have also provided information on $\epsilon=9.8$ on GC1. In addition, as can be seen from Table.2 of GS-WGAN[1], they require a large $\epsilon$. This is why we initially picked $\epsilon=50$ in the previous response.

[1] Chen, Dingfan, Tribhuvanesh Orekondy, and Mario Fritz. "Gs-wgan: A gradient-sanitized approach for learning differentially private generators." Advances in Neural Information Processing Systems 33 (2020): 12673-12684.

Reviewer comment (Evaluation):
The experiments regarding DP are not meaningful without mentioning privacy budgets.

Author response:
There has been notable concern regarding the experiment results conducted with epsilon=50, as highlighted by Reviewers sDDa and xZ6u. In response to this, we carried out additional experiments using $\epsilon=9.8$, which is frequently used in DP-SGD. The results of these experiments can be found in Appendix.A and Table.6 and 7. Notably, we observed that PRISM robustly performs well, whereas other baselines failed. We would like to notify that all experiments were conducted with identical $\epsilon$ values  for both baselines and PRISM. To avoid any potential confusion, we have also provided information on $\epsilon=9.8$ on GC1. In addition, as can be seen from Table.2 of GS-WGAN[1], they require a large $\epsilon$. This is why we initially picked $\epsilon=50$ in the previous response.

[1] Chen, Dingfan, Tribhuvanesh Orekondy, and Mario Fritz. "Gs-wgan: A gradient-sanitized approach for learning differentially private generators." Advances in Neural Information Processing Systems 33 (2020): 12673-12684.

Reviewer comment (Experiments):
The DP-related comparison might be unfair.

Author response:
There has been notable concern regarding the experiment results conducted with epsilon=50, as highlighted by Reviewers sDDa and xZ6u. In response to this, we carried out additional experiments using $\epsilon=9.8$, which is frequently used in DP-SGD. The results of these experiments can be found in Appendix.A and Table.6 and 7. Notably, we observed that PRISM robustly performs well, whereas other baselines failed. We would like to notify that all experiments were conducted with identical $\epsilon$ values  for both baselines and PRISM. To avoid any potential confusion, we have also provided information on $\epsilon=9.8$ on GC1. In addition, as can be seen from Table.2 of GS-WGAN[1], they require a large $\epsilon$. This is why we initially picked $\epsilon=50$ in the previous response.

[1] Chen, Dingfan, Tribhuvanesh Orekondy, and Mario Fritz. "Gs-wgan: A gradient-sanitized approach for learning differentially private generators." Advances in Neural Information Processing Systems 33 (2020): 12673-12684.

Reviewer comment (Experiments):
The performance in some experiments is somehow too surprisingly good to convince me. I am not fully convinced that a randomly initialized model without training can outperform a carefully designed and trained generative model.

Author response:
**R1-1. Why can SLT outperform a fully trained generative model? Why is MMD loss effective?**

We interpret your question as probing why the combination of SLT and MMD loss performs so well. We will divide the reviewer’s question into three parts for clarity: 

(1) Effectiveness of SLT: Previous research has theoretically shown that in large, randomly initialized models, there often exists a 'strong lottery ticket'--- a subnetwork capable of matching or even surpassing the performance of the full model [1]. To illustrate, consider the analogy of the Library of Babel, where a vast repository of all possible combinations of text exists. In such a library, the probability of finding meaningful books is not negligible. Similarly, in a large enough model, strong and effective subnetworks are likely to exist. This intuitive analogy serves as a foundation for understanding the concept of SLT. Consequently, when such a strong lottery ticket is identified, it has the potential to provide a comparable or sometimes even better performance than fully trained GAN-based models. 

Of course, while the theoretical existence of SLTs is significant, the practical challenge lies in the methodology to locate these optimal subnetworks. This requires the careful development of a score function and algorithm tailored to the specific task at hand, which is a highly nontrivial endeavor. (Our contribution lies in this realm, developing a framework that effectively addresses the challenge of creating a federative generative model) Understanding this framework sets the stage to address the reviewer's next question: the effectiveness of MMD loss. 

(2) Effectiveness of MMD Loss: 
MMD loss functions by minimizing the mean distance between samples in the Reproducing Kernel Hilbert Space (RKHS). Theoretical evidence suggests that distributions are identical when their statistics in RKHS coincide [2][3]. In practice, this translates to each client computing the L2 distance from the global model to its RKHS statistics during local training, effectively measuring the MMD distance for global statistics. This statistical approach underlies MMD's superiority over traditional GAN-based methods when combined with SLT. This method, as analyzed in the context of WGANs, suggests that MMD, being a distance function, provides more stability in training compared to divergence-based methods like traditional GANs. 

This stability is crucial in federated learning environments, where training can be erratic and unstable due to the inherent challenges such as heterogeneous data distributions across clients. The ability of MMD loss to consistently measure distances in RKHS contributes significantly to mitigating these challenges, thereby enhancing model performance and reliability. Thus, our model leverages the combination of SLTs and the stable training dynamics of MMD loss to achieve its notable performance.


(3) Comparing GAN and MMD loss: GAN loss is known for its instability and requires extensive regularization and careful training management. To further confirm the advantage of using MMD loss in our setup, we have also explored the possibility of using the GAN loss to find SLT (Figure 10). Due to the inherent training instability of GAN loss, particularly in federated learning settings with data heterogeneity, this approach proved ineffective. Experiments using GAN loss instead of MMD in our PRISM framework resulted in collapsed images, emphasizing GAN loss's unsuitability for SLT in decentralized environments. Nevertheless, as you rightly pointed out, MMD does have limitations in handling large, complex datasets. Exploring alternative frameworks to overcome these challenges is an intriguing direction for future research. 

[1] Ramanujan, Vivek, et al. "What's hidden in a randomly weighted neural network?." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Gretton, Arthur, et al. "A kernel two-sample test." The Journal of Machine Learning Research 13.1 (2012): 723-773.

[3] Gretton, Arthur, et al. "A kernel method for the two-sample-problem." Advances in neural information processing systems 19 (2006).

Reviewer comment (Evaluation):
The score selection criterion is MMD, which is known to be suboptimal for image generation.

Author response:
**R1-1. Why can SLT outperform a fully trained generative model? Why is MMD loss effective?**

We interpret your question as probing why the combination of SLT and MMD loss performs so well. We will divide the reviewer’s question into three parts for clarity: 

(1) Effectiveness of SLT: Previous research has theoretically shown that in large, randomly initialized models, there often exists a 'strong lottery ticket'--- a subnetwork capable of matching or even surpassing the performance of the full model [1]. To illustrate, consider the analogy of the Library of Babel, where a vast repository of all possible combinations of text exists. In such a library, the probability of finding meaningful books is not negligible. Similarly, in a large enough model, strong and effective subnetworks are likely to exist. This intuitive analogy serves as a foundation for understanding the concept of SLT. Consequently, when such a strong lottery ticket is identified, it has the potential to provide a comparable or sometimes even better performance than fully trained GAN-based models. 

Of course, while the theoretical existence of SLTs is significant, the practical challenge lies in the methodology to locate these optimal subnetworks. This requires the careful development of a score function and algorithm tailored to the specific task at hand, which is a highly nontrivial endeavor. (Our contribution lies in this realm, developing a framework that effectively addresses the challenge of creating a federative generative model) Understanding this framework sets the stage to address the reviewer's next question: the effectiveness of MMD loss. 

(2) Effectiveness of MMD Loss: 
MMD loss functions by minimizing the mean distance between samples in the Reproducing Kernel Hilbert Space (RKHS). Theoretical evidence suggests that distributions are identical when their statistics in RKHS coincide [2][3]. In practice, this translates to each client computing the L2 distance from the global model to its RKHS statistics during local training, effectively measuring the MMD distance for global statistics. This statistical approach underlies MMD's superiority over traditional GAN-based methods when combined with SLT. This method, as analyzed in the context of WGANs, suggests that MMD, being a distance function, provides more stability in training compared to divergence-based methods like traditional GANs. 

This stability is crucial in federated learning environments, where training can be erratic and unstable due to the inherent challenges such as heterogeneous data distributions across clients. The ability of MMD loss to consistently measure distances in RKHS contributes significantly to mitigating these challenges, thereby enhancing model performance and reliability. Thus, our model leverages the combination of SLTs and the stable training dynamics of MMD loss to achieve its notable performance.


(3) Comparing GAN and MMD loss: GAN loss is known for its instability and requires extensive regularization and careful training management. To further confirm the advantage of using MMD loss in our setup, we have also explored the possibility of using the GAN loss to find SLT (Figure 10). Due to the inherent training instability of GAN loss, particularly in federated learning settings with data heterogeneity, this approach proved ineffective. Experiments using GAN loss instead of MMD in our PRISM framework resulted in collapsed images, emphasizing GAN loss's unsuitability for SLT in decentralized environments. Nevertheless, as you rightly pointed out, MMD does have limitations in handling large, complex datasets. Exploring alternative frameworks to overcome these challenges is an intriguing direction for future research. 

[1] Ramanujan, Vivek, et al. "What's hidden in a randomly weighted neural network?." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Gretton, Arthur, et al. "A kernel two-sample test." The Journal of Machine Learning Research 13.1 (2012): 723-773.

[3] Gretton, Arthur, et al. "A kernel method for the two-sample-problem." Advances in neural information processing systems 19 (2006).