Paper: On the Evaluation of Generative Models in Distributed Learning Tasks (ICLR-2024)

Reviewer comment (Evaluation):
The need for the two modes of evaluation the authors consider (all and avg) is not well motivated in the paper. Among the cited previous works on federated learning for generative models, most if not all rely on some form of FID-all, typically by computing FID scores on a separate test dataset. Are there any examples in the literature where scores similar to FID-avg or KID-avg were considered? I imagine one would be interested in FID-avg in the context of personalization or when privacy concerns prevent the usage of FID-all, for example, but that is never discussed in detail in the current version of the paper.

Author response:
**1- Motivation of our study of the evaluation score aggregations.**

**Re:** As pointed out by the reviewer, the existing literature on federated generative model learning mostly utilize the distribution-level aggregate scores (i.e. FID-all and KID-all). However, in a real federated learning setting, the computation of the FID-all and KID-all scores could be highly challenging due to common privacy and computation constraints in federated learning tasks. For the distributed computation of both KID-all and FID-all, the clients should either share their entire dataset with the server, which would violate the privacy constraints, or solve a completely independent distributed optimization problem which incur great communication and computation costs. Under such constraints, the computation of the averaged score of individual clients (FID-avg and KID-avg) will be significantly less expensive and thus more feasible. 

In our work, we show that although KID-all and KID-avg will take different values, they will result in the same ranking of generative models (Theorem 2). This result reveals an important advantage of KID evaluation in distributed settings, since to obtain the models' KID-all ranking the clients need to share only their individual KID scores with the server. On the other hand, we show that the same ranking consistency does not hold for FID-avg and FID-all. We characterize the gap between the two aggregate scores which depend on the choice of generative model $P_G$. Based on the reviewer’s comments and writing suggestions, we have added one more paragraph to the introduction explaining the above motivation of our study.

Reviewer comment (Evaluation):
In Section 5.1., the authors note that “[…] counterintuitively, the ‘ideal estimator’ did not reach the minimum average of the Fréchet distances”. Could they share any intuition as to why that is the case? Does that indicate taking the average among clients is a poor metric to optimise for?

Author response:
**2- The ideal estimator in Section 5.1** 

**Re:** We note that FD-avg ranks the models in terms of the average Frechet distance with respect to the two Gaussian modes $\mathcal{N}_1,\mathcal{N_2}$ representing the clients. Therefore, the ideal estimator is a mixture distribution $\frac{1}{2}(\mathcal{N}_1+\mathcal{N_2})$ combining the two modes. However, the Frechet distance of the mixture distribution to each of the modes $\mathcal{N}_1,\mathcal{N_2}$ is non-zero, and therefore it is not clear if the mixture distribution can minimize the sum of the Frechet distances. Our experiments show that the ideal estimator’s mixture distribution $\frac{1}{2}(\mathcal{N}_1+\mathcal{N_2})$ is indeed *not the minimizer* of the sum of Frechet distances to $\mathcal{N}_1$ and $\mathcal{N}_2$.

Reviewer comment (Evaluation):
In page 7, the authors comment on the effect of privacy considerations when choosing between FID-all or FID-avg: “[…] a distributed computation of FID-all is more challenging than obtaining FID-avg due to privacy considerations”. This shows an interesting trade-off between the two modes that the authors could explore a bit further. While FID-all can be more challenging to compute from a privacy perspective, wouldn’t FID-avg favor a model that fits the distributions of only one or a few of clients very well, thus potentially “memorizing” the data of these clients?

Author response:
**4-“Wouldn’t FID-avg favor a model that fits the distributions of only one or a few of clients very well?"**

**Re:** As shown in Theorem 1, the optimal mean vectors (in the InceptionNet-based embedding) for FID-all and FID-avg are identical and equal to the averaged mean of clients: $\widehat{\mu}=\sum_{i=1}^k \lambda_i\mu_i$. Please note that $\widehat{\mu}$ is the mean of the collective distribution across clients, and therefore the optimal solution based on FID-avg could not fit only a few clients’ distributions and miss the majority of clients’ distributions.

Reviewer comment (Theory):
For FID scores, while the paper gave theoretical formulations for FID-avg and FID-all, the formulations can only distinguish different rankings of generators based on their distances to corresponding covariance matrices. It would be more informative to characterize the gap between FID-avg and FID-all with the distances between clients’ data distributions. This could allow one to estimate the degree of FID score inconsistency based on how different clients’ datasets are.

Author response:
**Re:** We note that Theorem 1 characterizes the gap between FID-all and FID-avg up to a $\mathrm{constant}_1$ (with respect to generative model $P_G$), as it shows

$$FID_{\mathrm{avg}} - \mathrm{FID}_{\mathrm{all}} =\mathrm{FID}(\widetilde{P}_X;P_G) - \mathrm{FID}(\widehat{P}_X;P_G) + \mathrm{constant}_1$$

Here $\widehat{P}_X$ is the normal distribution with the aggergate mean $\widehat{\mu} = \sum_i \lambda_i \mu_i$ and aggregate covariance matrix $\widehat{C} = \sum_i \lambda_i (C_i + \mu_i\mu_i^\top - \widehat{\mu}\widehat{\mu}^\top )$, and $\widetilde{P}_X$ is the Barycenter normal distribution with the same mean $\widehat{\mu} = \sum_i \lambda_i \mu_i$ and barycenter covaraince matrix $\widetilde{C}$ defined in Theorem 1 (aggergating only $C_1,\ldots, C_k$). Therefore, the gap between FID-avg and FID-all gap can be simplified to the following for a $\mathrm{constant}_2$ (with respect to generative model $P_G$)

$$FID_{\mathrm{avg}} - \mathrm{FID}_{\mathrm{all}}= 2\mathrm{Tr}((C_G\widehat{C})^{1/2}- (C_G\widetilde{C})^{1/2} ) + \mathrm{constant}_2$$

As can be seen, the gap between FID-all and FID-avg depends on the choice of $C_G$ (generative model's covariance matrix) and hence is not independent of $P_G$. We have included this discussion in the revised Remark 3.

Reviewer comment (Experiments):
In experiments, it seems that most results for FID scores come from “simulated” generators, that is, treating a class of images from CIFAR-10 or CIFAR-100 as the output of a generator. Such simulation does not well represent real generators trained on the multi-class dataset, or the general federated learning protocol (even when each client owns one class of images only). Hence the results do not sufficiently reveal if inconsistency between FID-avg and FID-all is an actual issue in practice. Moreover, the FID scores are quite large and far from optimal, in which case the consistency of the evaluation metric may not be a most important property to pursue.

Author response:
**Re:** We note that our main goal in the numerical experiments is to show the potential inconsistencies between FID-avg and FID-all ranking and contrast it with the KID-avg and KID-all consistent rankings. As implied by the theoretical and numerical results, a heterogeneous distributed setting with a significantly smaller intra-client variance than inter-client variance could lead to inconsistent  FID-based rankings of generative models. This scenario could potentially occur in real-world federated learning tasks, because the samples within each client are expected to have a considerably lower variance than the collective set of all clients’ samples.    

To address the reviewer's comment on the lack of real generators in our experiments, we performed an experiment using a pre-trained StyleGAN generator. In the experiments, we simulated a heterogeneous setting by modeling the client distributions as truncated normal distribution for the GAN's latent vector with differently selected mean vectors at different clients. We then evaluated the FID and KID aggregated scores for the evaluation of a truncated StyleGAN generator with different truncation coefficients, which is a similar experimental setting to the related work [1].  Our numerical results in the Appendix showed the same consistency trends that we reported for the simulated generators in the main text. We visualize the client-based samples and report the aggregated scores in the updated supplementary document (Figures 2,3,4).

Reviewer comment (Evaluation):
In Figure 2, do the KID scores imply that the plane generator is better than the DDPM model, while the former generates much less diverse images? This may raise a question of how to properly evaluate the models apart from consistency.

Author response:
**Re:** As pointed out by the reviewer, the KID-score with polynomial kernel prefers the less-diverse simulated plane generator. Please note that the KID-based ranking of generative models highly depends on the choice of kernel similarity function $k(\cdot,\cdot)$ in the KID definition. In fact, we numerically validated that a different choice of kernel function (Gaussian kernel) will prefer the more-diverse DDPM generated samples. We discussed the numerical comparison with different kernel functions in Section 3.3 in the updated supplementary material.

Finally, we note that our result in Theorem 2 on the consistent ranking of KID-all and KID-avg applies to every choice of kernel function $k$. However, the study of the proper choice of kernel functions in different machine learning applications is beyond the scope of our work, and could be an interesting research direction for future exploration.

Reviewer comment (Experiments):
The simple observation would’ve still constituted as a good contribution if the authors provide some actionable insights. For example, one possible conclusion could be that one should use KID rather than FID in distributed settings — however, authors do not compare models obtained via these two methods in detail. Another possible direction could be to modify FID so that the new score behaves well under averaging.

Author response:
**3- Our results and the choice of evaluation metric in distributed settings**

**Re:** A practical implication of our results is the lower privacy and computational costs of ranking generative models in the distributed settings using the KID metric. This is because instead of computing KID-all which requires sharing massive information on the clients' data, we can use KID-avg (requiring each client to only share their KID score), which, as Theorem 2 shows leads, gives the same ranking of generative models. However, the privacy and computational expenses of an FID-based evaluation could be significantly higher, since averaging the clients’ individual FID scores could lead to an inconsistent ranking of the models with the original FID-all metric.